Yosemite Server (Server.app 4.0) supports SMB3 by default. You can add encryption and digital signing as well.
To verify your connection to that specific share is encrypted, run this command in Terminal.app
smbutil statshares -m /Volumes/pathtoshare
Compare a share that has encryption on and one that has it off. You will see additional items on the share that is encrypted.