Difference between revisions of "NFS/Kerberos"

From CSCWiki
Jump to navigation Jump to search
 
(30 intermediate revisions by 8 users not shown)
Line 1: Line 1:
Our user-data is stored in /export/users on artificial-flavours in a RAID 1 software array running on two 400 GB SATA disks. We export /users via NFSv3 and NFSv4. All of our systems NFSv3 mount /users.
+
Our user-data is stored in /users on [[Machine_List#psilodump|psilodump]] on an ISCSI volume exported to [[Machine_List#aspartame|aspartame]], which exports /users/ via NFS. Plans to add a layer of LVM abstraction so as to support regular snapshot backups of /users/ are currently in-place, but not yet fully implemented. All of our systems NFS mount /users, and most of them do so using [[Kerberos]] for authentication.
 
We are now planning to run Solaris 10 as the disk server's operating system, with [http://en.wikipedia.org/wiki/ZFS ZFS] as the file system, and exporting via NFSv4. See [[Solaris 10]] for current information.
 
   
 
We have also explored additional methods for replicating user-data, including AFS, Coda, and DRBD, but have found all to be unusable or problematic.
 
We have also explored additional methods for replicating user-data, including AFS, Coda, and DRBD, but have found all to be unusable or problematic.
Line 11: Line 9:
 
On November 8, 2007, we experienced a major NFS failure. An analysis of the logs indicated that the fault was likely caused by NFSv4-specific code. As a result, we have returned to mounting with NFSv3.
 
On November 8, 2007, we experienced a major NFS failure. An analysis of the logs indicated that the fault was likely caused by NFSv4-specific code. As a result, we have returned to mounting with NFSv3.
   
  +
In November 2015, we made another attempt at mounting with NFSv4 in the office. This was a huge time suck and failed sporadically. As a result, we have returned to mounting with NFSv3. NFSv4 ACLs/mapping seem to be the culprit. NFSv4, '''just not ready'''.
= ZFS =
 
   
  +
== Troubleshooting ==
We plan to use ZFS in mirror mode.
 
   
  +
* If NFS refuses to mount, with a message similar to "Incorrect mount option was specified", ensure that the "nfs-common" service is running. This is required for Kerberos authentication with NFS.
== Overview ==
 
   
 
= ZFS =
Each user directory is stored in a separate zfs file system.
 
 
To create a user directory:
 
zfs create users/$USER
 
 
To delete a user directory:
 
zfs destroy users/$USER
 
 
To move/rename a user directory:
 
zfs rename users/$USER_OLD users/$USER_NEW
 
 
== NFS (server-side) ==
 
 
To disable atime, devices, and setuid:
 
zfs set atime=off users
 
zfs set devices=off users
 
zfs set setuid=off users
 
 
To export over NFS using host-based access-control:
 
zfs set sharenfs="sec=sys,rw=$ACCESS_LIST,nosuid" users
 
where ACCESS_LIST may be as a colon-separated list of any of the following:
 
* hostname (e.g. glucose-fructose.csclub.uwaterloo.ca)
 
* netgroup (e.g. in LDAP)
 
* domain name suffix (e.g. .csclub.uwaterloo.ca)
 
* network (e.g. @129.97.134.0/24)
 
A minus sign (-) may prefix one of the above to indicate that access is to be denied. 'man share_nfs' has full details.
 
 
== NFS (client-side) ==
 
 
You should install the autofs package. Then edit /etc/auto.master and append the following:
 
/users /etc/auto.users
 
 
Create /etc/auto.users with content:
 
* -fstype=nfs4,soft,intr,nosuid,nodev disk:/users/&
 
 
In order to support NFSv4 ACL's with getfacl/setfacl, you should apply the [http://www.citi.umich.edu/projects/nfsv4/linux/ NFSv4 ACL patch]. You can also compile the [http://www.citi.umich.edu/projects/nfsv4/linux/ nfs4_getfacl/nfs4_setfacl utils].
 
 
== Quota ==
 
 
To query quota:
 
zfs get quota users/$USER
 
 
To set quota:
 
zfs set quota=$SIZE users/$USER
 
where $SIZE could be 2.5G, 100M, etc...
 
 
To set no quota:
 
zfs set quota=none users/$USER
 
 
== Snapshots ==
 
 
Snapshots can be accessed from /users/$USER/.zfs/snapshot/.
 
 
To create a snapshot:
 
zfs create users/$USER@$SNAPSHOT
 
 
To delete a snapshot:
 
zfs destroy users/$USER@$SNAPSHOT
 
 
To rename a snapshot:
 
zfs rename users/$USER@$SNAPSHOT_OLD users/$USER@$SNAPSHOT_NEW
 
 
To list snapshots:
 
zfs list -t snapshot -r users
 
 
== Miscellaneous ==
 
   
  +
On March 15, 2008, we transitioned to ZFS. This move has since been reversed; details are preserved in [http://wiki.csclub.uwaterloo.ca/User-data?oldid=2331 a previous revision of this page]
You should occasional scrub (error-check) the zpool:
 
zpool scrub users
 
   
  +
[[Category:Services]]
For small files, ZFS+NFS performance really sucks. You can fix this by enabling zil_disable:
 
  +
[[Category:Software]]
echo 'set zfs:zil_disable=1' >> /etc/system
 
More information on zil_disable is available [http://blogs.sun.com/erickustarz/entry/zil_disable here] and [http://weblog.etherized.com/?p=130 here].
 

Latest revision as of 01:20, 4 December 2015

Our user-data is stored in /users on psilodump on an ISCSI volume exported to aspartame, which exports /users/ via NFS. Plans to add a layer of LVM abstraction so as to support regular snapshot backups of /users/ are currently in-place, but not yet fully implemented. All of our systems NFS mount /users, and most of them do so using Kerberos for authentication.

We have also explored additional methods for replicating user-data, including AFS, Coda, and DRBD, but have found all to be unusable or problematic.

NFS

NFSv3 has been in long standing use by the CSC as well as almost everyone else on the planet. NFSv4 mounts of /users are currently in the works to CSCF. Unfortunately NFS has a number of problems. Clients become desperately unhappy when disconnected from the NFS server. Also previous to NFSv4 there was no way to client side cache, resulting in poor performance with large files.

On November 8, 2007, we experienced a major NFS failure. An analysis of the logs indicated that the fault was likely caused by NFSv4-specific code. As a result, we have returned to mounting with NFSv3.

In November 2015, we made another attempt at mounting with NFSv4 in the office. This was a huge time suck and failed sporadically. As a result, we have returned to mounting with NFSv3. NFSv4 ACLs/mapping seem to be the culprit. NFSv4, just not ready.

Troubleshooting

  • If NFS refuses to mount, with a message similar to "Incorrect mount option was specified", ensure that the "nfs-common" service is running. This is required for Kerberos authentication with NFS.

ZFS

On March 15, 2008, we transitioned to ZFS. This move has since been reversed; details are preserved in a previous revision of this page