Our user-data is stored in /export/users on artificial-flavours in a RAID 1 software array running on two 400 GB SATA disks. We export /users via NFSv3 and NFSv4. All of our systems NFSv3 mount /users.
We have also explored additional methods for replicating user-data, including AFS, Coda, and DRBD, but have found all to be unusable or problematic.
NFSv3 has been in long standing use by the CSC as well as almost everyone else on the planet. NFSv4 mounts of /users are currently in the works to CSCF. Unfortunately NFS has a number of problems. Clients become desperately unhappy when disconnected from the NFS server. Also previous to NFSv4 there was no way to client side cache, resulting in poor performance with large files.
On November 8, 2007, we experienced a major NFS failure. An analysis of the logs indicated that the fault was likely caused by NFSv4-specific code. As a result, we have returned to mounting with NFSv3.
There is a patch and utility to get/set NFSv4 ACL's here.
We plan to use ZFS in mirror mode.
Each user directory is stored in a separate zfs file system.
To create a user directory:
zfs create users/$USER
To delete a user directory:
zfs destroy users/$USER
To move/rename a user directory:
zfs rename users/$USER_OLD users/$USER_NEW
To query quota:
zfs get quota users/$USER
To set quota:
zfs set quota=$SIZE users/$USER
where $SIZE could be 2.5G, 100M, etc...
To set no quota:
zfs set quota=none users/$USER
Snapshots can be accessed from /users/$USER/.zfs/snapshot/.
To create a snapshot:
zfs create users/$USER@$SNAPSHOT
To delete a snapshot:
zfs destroy users/$USER@$SNAPSHOT
To rename a snapshot:
zfs rename users/$USER@$SNAPSHOT_OLD users/$USER@$SNAPSHOT_NEW
To list snapshots:
zfs list -t snapshot -r users