Difference between revisions of "NFS/Kerberos"

From CSCWiki
Jump to navigation Jump to search
(Removed old ZFS-related stuff (still preserved in history) - I'm not sure about the NFS/KRB thing, so I've left that for now.)
Line 1: Line 1:
 
Our user-data is stored in /users on [[Machine_List#ginseng|ginseng]] in a RAID 1 mirror running on two 400 GB SATA disks. All of our systems NFS mount /users.
'''''This page is out of date since we moved to using linux on ginseng. We no longer use ZFS.'''''
 
 
 
Our user-data is stored in /users on [[ginseng]] in a RAID 1 mirror running on two 400 GB SATA disks. Ginseng runs Solaris 10 and uses [http://en.wikipedia.org/wiki/ZFS ZFS] as the filesystem for /users. All of our systems NFSv4 mount /users.
 
   
 
We have also explored additional methods for replicating user-data, including AFS, Coda, and DRBD, but have found all to be unusable or problematic.
 
We have also explored additional methods for replicating user-data, including AFS, Coda, and DRBD, but have found all to be unusable or problematic.
Line 14: Line 11:
 
= ZFS =
 
= ZFS =
   
On March 15, 2008, we transitioned to ZFS.
+
On March 15, 2008, we transitioned to ZFS. This move has since been reversed; details are preserved in [[User-data&oldid=2331|a previous revision of this page]].
 
== Overview ==
 
 
Each user directory is stored in a separate zfs file system.
 
 
To create a user directory:
 
zfs create users/$USER
 
 
To delete a user directory:
 
zfs destroy users/$USER
 
 
To move/rename a user directory:
 
zfs rename users/$USER_OLD users/$USER_NEW
 
 
== NFS (server-side) ==
 
 
To disable atime, devices, and setuid:
 
zfs set atime=off users
 
zfs set devices=off users
 
zfs set setuid=off users
 
 
To export over NFS using host-based access-control:
 
zfs set sharenfs="sec=sys,rw=$ACCESS_LIST,nosuid" users
 
where ACCESS_LIST may be as a colon-separated list of any of the following:
 
* hostname (e.g. glucose-fructose.csclub.uwaterloo.ca)
 
* netgroup (e.g. in LDAP)
 
* domain name suffix (e.g. .csclub.uwaterloo.ca)
 
* network (e.g. @129.97.134.0/24)
 
A minus sign (-) may prefix one of the above to indicate that access is to be denied. 'man share_nfs' has full details.
 
 
To make umask work sanely with ACL's:
 
zfs set aclmode=passthrough users
 
 
To make ACL inheritance work sanely:
 
zfs set aclinherit=passthrough users
 
 
Here's what we set this to currently:
 
rw=acesulfame-potassium:artificial-flavours:ascorbic-acid:caffeine:caramel-colour:citric-acid:dextroamphetamine-saccharate:\
 
glucose-fructose:natural-flavours:ozone:perpugilliam:phosphoric-acid:potassium-citrate:sodium-citrate:taurine,root=caffeine
 
   
 
The NFSv4 domain is auto-detected by default, although to be safe, you can explicitly set it in /etc/default/nfs:
 
The NFSv4 domain is auto-detected by default, although to be safe, you can explicitly set it in /etc/default/nfs:
Line 62: Line 20:
 
This documents some important steps that needed to be done once.
 
This documents some important steps that needed to be done once.
   
You need to create an nfs kerberos principal on caffeine:
+
You need to create an NFS Kerberos principal on caffeine:
 
sudo kadmin.local
 
sudo kadmin.local
 
addprinc -randkey zfs/ginseng.csclub.uwaterloo.ca
 
addprinc -randkey zfs/ginseng.csclub.uwaterloo.ca
Line 72: Line 30:
   
 
In order to support NFSv4 ACL's with getfacl/setfacl, you should apply the [http://www.citi.umich.edu/projects/nfsv4/linux/ NFSv4 ACL patch]. You can also compile the [http://www.citi.umich.edu/projects/nfsv4/linux/ nfs4_getfacl/nfs4_setfacl utils].
 
In order to support NFSv4 ACL's with getfacl/setfacl, you should apply the [http://www.citi.umich.edu/projects/nfsv4/linux/ NFSv4 ACL patch]. You can also compile the [http://www.citi.umich.edu/projects/nfsv4/linux/ nfs4_getfacl/nfs4_setfacl utils].
 
== Quota ==
 
 
To query quota:
 
zfs get quota users/$USER
 
 
To set quota:
 
zfs set quota=$SIZE users/$USER
 
where $SIZE could be 2.5G, 100M, etc...
 
 
To set no quota:
 
zfs set quota=none users/$USER
 
 
It's important to note that quotas include all descendants of a filesystem, including snapshots. This means that if a user deletes a file, but the file is contained in a snapshot, the users available quota will not decrease. To get around this, refquota should be used instead of quota. However, refquota is only supported on OpenSolaris.
 
 
== Snapshots ==
 
 
Snapshots can be accessed from /users/$USER/.zfs/snapshot/.
 
 
To create a snapshot:
 
zfs create users/$USER@$SNAPSHOT
 
 
To delete a snapshot:
 
zfs destroy users/$USER@$SNAPSHOT
 
 
To rename a snapshot:
 
zfs rename users/$USER@$SNAPSHOT_OLD users/$USER@$SNAPSHOT_NEW
 
 
To list snapshots:
 
zfs list -t snapshot -r users
 
 
A rolling snapshot script is available in ~dtbartle/csc/zfs/. It should be configured to run as a root cronjob:
 
0 0,2,4,6,8,10,12,14,16,18,20,22 * * * /usr/local/sbin/snapshot-rotate.py users bihourly 7
 
15 2 * * * /usr/local/sbin/snapshot-rotate.py users daily 7
 
30 2 * * 0 /usr/local/sbin/snapshot-rotate.py users weekly 4
 
45 2 1 * 0 /usr/local/sbin/snapshot-rotate.py users monthly 4
 
 
== Miscellaneous ==
 
 
You should occasional scrub (error-check) the zpool:
 
zpool scrub users
 
 
For small files, ZFS+NFS performance really sucks. You can fix this by enabling zil_disable:
 
echo 'set zfs:zil_disable=1' >> /etc/system
 
More information on zil_disable is available [http://blogs.sun.com/erickustarz/entry/zil_disable here] and [http://weblog.etherized.com/?p=130 here].
 
 
To see zpool status and statistics:
 
zpool status -v users
 
zpool iostat -v users
 

Revision as of 21:59, 14 January 2010

Our user-data is stored in /users on ginseng in a RAID 1 mirror running on two 400 GB SATA disks. All of our systems NFS mount /users.

We have also explored additional methods for replicating user-data, including AFS, Coda, and DRBD, but have found all to be unusable or problematic.

NFS

NFSv3 has been in long standing use by the CSC as well as almost everyone else on the planet. NFSv4 mounts of /users are currently in the works to CSCF. Unfortunately NFS has a number of problems. Clients become desperately unhappy when disconnected from the NFS server. Also previous to NFSv4 there was no way to client side cache, resulting in poor performance with large files.

On November 8, 2007, we experienced a major NFS failure. An analysis of the logs indicated that the fault was likely caused by NFSv4-specific code. As a result, we have returned to mounting with NFSv3.

ZFS

On March 15, 2008, we transitioned to ZFS. This move has since been reversed; details are preserved in a previous revision of this page.

The NFSv4 domain is auto-detected by default, although to be safe, you can explicitly set it in /etc/default/nfs:

NFSMAPID_DOMAIN=csclub.uwaterloo.ca

Initial setup

This documents some important steps that needed to be done once.

You need to create an NFS Kerberos principal on caffeine:

sudo kadmin.local
addprinc -randkey zfs/ginseng.csclub.uwaterloo.ca
ktadd -e des-cbc-crc:normal -k /tmp/ginseng.nfs.keytab

You then need to merge that keytab (using ktutil) into /etc/krb5/krb5.keytab on ginseng.

NFS (client-side)

In order to support NFSv4 ACL's with getfacl/setfacl, you should apply the NFSv4 ACL patch. You can also compile the nfs4_getfacl/nfs4_setfacl utils.