NetApp
As of Fall 2013, the CSC has a NetApp FAS3000 series which is capable of hosting network shares. It was donated to us by CSCF. It is also pretty old. Like, Pentium IV old.
Documentation
All the manuals are hosted in ~sysadmin/netapp-docs/
Relevant docs for storage modification are: smg.pdf, sysadmin.pdf
iSCSI documentation is in ontop/bsag.pdf
Background
While the NetApp supports both NFS and CIFS, neither of these export options provide the versatility nor the options we desire of a network fileshare (for instance, no device authentication is supported). Instead, we have configured the NetApp to export iSCSI block devices to be mounted on aspartame. Therefore, aspartame now replaces ginseng as the primary CSC fileserver.
Terminology
- Filer: the controller unit for the NetApp. Currently psilodump.
- Disk shelf: where the physical disks live. Can be plugged into a filer or directly into another machine.
- RAID: "Redundant Array of Independent Disks", used to improve reliability and protect against disk failures.
- RAID-DP: "Double Parity" RAID, similar to RAID6 (this is probably RAID4 with two parity disks, but the result is the same). It uses two parity disks and can survive up to two disk failures before degradation.
- aggr: An aggregate of disks. This is a list of physical disks, similar to selecting the physical devices used for LVM.
- vol: A volume consisting of some space on an aggregate. In general, we use the whole aggregate for a volume. RAID level is set at the volume. Similar to an LVM volume group.
- lun: "Logical Unit Number" The LUN is a device addressed by the SCSI protocol, and looks like a disk to the user. We usually use the whole volume for a single LUN. This is similar to an LVM logical volume.
Common Commands
aggr status -r aggr_name Shows aggregate status disk show -v Shows disks, and which filer they are owned by (currently all by psilodump) storage storage related things disk assign Assigns orphaned disks to a filer vol Volume stuffs
NetApp Configuration
Should aspartame get totally hosed, or stability is long enough such that all sysadmin folk at the time have graduated, here is how to access, configure, and complete set up iSCSI on the NetApp+aspartame.
Access
Configuration mechanisms are accessible via SSH or serial interface, but through aspartame only, which the machine is directly plugged into. The NetApp is not visible on 134net at all.
The private IP is 10.15.134.130, only available from aspartame on the interface with IP 10.15.134.1. You may have to remove the default route from the routing table in order to successfully contact the machine with ssh.
Disk information
- shelf 1
- 14x136GB 10,000RPM FibreChannel disks
- Currently disconnected, could be connected to psilodump or directly to another machine.
- shelf 2
- 14x136GB 10,000RPM FibreChannel disks
- Currently assigned to psilodump
- shelf 3
- 14x500GB 7,200RPM ATA disks
- Currently assigned to psilodump
- shelf 4
- 14x500GB 7,200RPM ATA disks
- Currently assigned to psilodump
Aggregates
- aggr0
- Root aggregate volume, in RAID-DP
- aggr1
- Music aggregate volume, in RAID-DP
- aggr2
- Users aggregate volume, in RAID-DP
- aggr3
- Backups volume for CSC videos, in RAID-DP
Volumes
- /vol/vol0
- Root volume.
- /vol/vol1music
- Music volume. This volume is not accessible via NFS or CIFS. It contains only the iSCSI LUN /vol/vol1music/lun0 .
- /vol/vol2users
- Users volume. This volume is not accessible via NFS or CIFS. It contains only the iSCSI LUN /vol/vol2users/lun0 .
- /vol/vol3backup
- Backup volume for videos. This volume is not accessible via NFS or CIFS. It contains only the iSCSI LUN /vol/vol3backup/lun0 .
Enabling iSCSI and Auth (one-time setup)
Enable iSCSI and configure default authentication.
options iscsi.enable on iscsi nodename iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca iscsi security default -s CHAP -p yoursecurepassword -n psilodump
where yoursecurepassword is more secure. For iSCSI hosts, the target will be on node iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca with username psilodump and password yoursecurepassword.
Setting up a new disk aggregate, volume, and LUN
1. Login to the NetApp. You'll either need access to the physical serial console or to ssh as root to psilodump's private IP (10.15.134.130). Credentials are stored in /users/sysadmin .
2. To get information on the available disks, run the command:
aggr status -r
This command will return three lists: Active aggregates with their assigned disks, spare disks, and disks managed by the partner. An aggregate is roughly equivalent to an LVM volume group: It is a collection of physical disks, possibly across multiple disk shelves and with various RAID levels applied, which may host one or more logical volumes. Do not proceed if there are fewer than three spare disks of each type available. Refer to the NetApp documentation to add more disks or release disks from existing aggregates.
3. Choose a list of disks for your new aggregate. The available space will be approximately 2/3 of the total disk space.
4. Create the aggregate as follows:
aggr create aggrN -t raid_dp -d [disk-list]
where [disk-list] is a list of the form AA:BB CC:DD ... containing the identifiers for the disks you wish to use to create the aggregate.
5. Retrieve the aggregate information. You will need to know the available space for the next step.
aggr show_space aggrN
6. Create a volume in the aggregate:
vol create volNfoo -s volume aggrN XXXK
where XXX is the total available space in aggrN. You may need to choose a smaller number due to hidden size constraints and rounding. If you can't seem to find the right size, pick one much smaller, and then use the command
vol size volNfoo +XXX
to grow the volume. This command will tell you how much available space remains, unlike `vol create`, so you don't need to keep guessing.
7. Disable snapshotting and access time update. Neither will be needed for exporting an iSCSI LUN.
vol options volNfoo no_atime_update on vol options volNfoo nosnap on snap reserve volNfoo 0
8. Create a LUN on your volume:
lun create -s XXXK -t linux /vol/volNfoo/lun0
where XXXK is the amount of available space on the volume, as shown by the command df.
9. Create an iSCSI initiator group and add all of your hosts to it:
igroup create -i -t linux volNfoo_group igroup add volNfoo_group iqn.1993-08.org.debian:01:123456789 igroup add volNfoo_group iqn.1993-08.org.debian:01:981287231 ...
The node identifiers given to the igroup add command will soon be able to access the iSCSI LUN you created above.
10. Map the LUN to the iSCSI initiator group:
lun map /vol/volNfoo/lun0 volNfoo_group
You're done! Any host in the initiator group should now be able to access the LUN you've created as a block device.
Host Configuration
aspartame Configuration
Install open-iscsi:
apt-get install open-scsi
Edit /etc/iscsi/iscsid.conf:
node.startup = manual discovery.sendtargets.auth.authmethod=CHAP discovery.sendtargets.auth.username=username discovery.sendtargets.auth.password=password node.session.auth.authmethod=CHAP node.session.auth.username=username node.session.auth.password=password
Start open-iscsi service:
service open-iscsi start
Scan for iSCSI devices from the NetApp:
iscsiadm --mode discovery --type st --portal psilodump
This should dump out a ton of information, for example:
[fe80::XXXX:XXXX:XXXX:XXXX]:3260,2001 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca [fe80::XXXX:XXXX:XXXX:XXXX]:3260,2000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca [fe80::XXXX:XXXX:XXXX:XXXX]:3260,2002 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca [fe80::XXXX:XXXX:XXXX:XXXX]:3260,1000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca 10.15.134.131:3260,2002 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca 129.97.134.131:3260,2001 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca 10.15.134.130:3260,2000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca 129.97.134.130:3260,1000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
The .130 IPs correspond to one filer, and the .131 IPs correspond to the other filer. Currently we are only using one of the filers (psilodump).
This also populates the /etc/iscsi/nodes/iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca directory with all possible ways to access the NetApp. For testing purposes (i.e. node.startup = manual), this is okay.
Test to see if you can get the iSCSI device to show up correctly:
iscsiadm --mode node --targetname "iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca" --portal 10.15.134.130:3260 --login
This should produce output similar to:
Logging in to [iface: default, target: iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca, portal: 10.15.134.130,3260] Login to [iface: default, target: iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca, portal: 10.15.134.130,3260]: successful
Check /dev/disk/by-path/ip* to ensure new disks show up:
# ls -l /dev/disk/by-path/ip* /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-0 -> ../../sda /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-0-part1 -> ../../sda1 /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-1 -> ../../sdb /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-1-part1 -> ../../sdb1
If this fails, check all your configuration again.
If this succeeds, you are now ready to try autoconnecting the iSCSI device.
Delete all extraneous entries from /etc/iscsi/nodes/iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca . This prevents the startup script from (a) hanging, and (b) being very upset. All that is left should be the interface you intend to connect through:
# ls -l /etc/iscsi/nodes/iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca/ 10.15.134.130,3260,2000
Edit /etc/iscsi/iscsid.conf:
node.startup = automatic
For the init.d script to work correctly (i.e. properly mount things) we need to add a sleep to allow the device to settle: Edit /etc/init.d/open-iscsi roughly around line 127 to add a "sleep 1":
... # Now let's mount sleep 1 log_daemon_msg "Mounting network filesystems" MOUNT_RESULT=1 if mount -a -O _netdev >/dev/null 2>&1; then MOUNT_RESULT=0 break fi log_end_msg $MOUNT_RESULT ...
Now we can restart the service:
service open-iscsi restart
Now you can configure partitions and mountpoints.
Exporting Kerberized NFS from Debian Sid
The default kernel in Debian sid (stable, 2.6.32) does not support the necessary crypto suites to export kerberized NFS to newer kernels. You MUST upgrade the kernel, nfs-common, and nfs-kernel-server packages to AT LEAST squeeze-backports.
iSCSI block device mount optimizations
tmyklebu made some changes to /sys/block/sda/queue. The following is now in /etc/rc.local on aspartame:
echo 2048 > /sys/block/sda/queue/read_ahead_kb echo 32768 > /sys/block/sda/queue/max_sectors_kb echo 4096 > /sys/block/sda/queue/nr_requests echo noop > /sys/block/sda/queue/scheduler
We should increase the iSCSI configs node.session.queue_depth and node.session.cmds_max during next maintenance window.
Transferring old files from ginseng
Method A
- On ginseng, use parted to set up the mounted iscsi drive as an ext4 primary partition (setting up a partition of size >2TB requires care and a GPT)
- Compiled star in /root on ginseng
- Transferred files with the following Makefile (assuming original user directories in /export/users, destination volume in /mnt/iscsi, make -j8):
foo := $(wildcard /export/users/*) bar := $(patsubst /export/users/%,/mnt/iscsi/%,$(foo)) all: $(bar) /mnt/iscsi/%: /export/users/% # echo $@ $< ~/star-1.5.2/star/OBJ/x86_64-linux-cc/star \ -copy -p -acl artype=exustar \ -C /export/users $(notdir $<) /mnt/iscsi
Method B
- On ginseng, authenticate with iSCSI target (psilodump.csclub.uwaterloo.ca lun0).
- Umount /dev/mapper/vg0-users
- Copy users filesystem directly to iSCSI target:
dd if=/dev/mapper/vg0-users of=/path/to/psilodump:lun0 bs=8M
- Resize users filesystem on destination partition to fit:
resize2fs /path/to/psilodump:lun0