Difference between revisions of "NetApp"

From CSCWiki
Jump to navigation Jump to search
(→‎iSCSI setup procedure: Makefile for copying user homedirs)
Line 7: Line 7:
  
 
iSCSI documentation is in ontop/bsag.pdf
 
iSCSI documentation is in ontop/bsag.pdf
 +
 +
==Background==
 +
While the NetApp supports both NFS and CIFS, neither of these export options provide the versatility nor the options we desire of a network fileshare. Instead, we have configured the NetApp to export iSCSI block devices to be mounted on aspartame. Therefore, aspartame now replaces ginseng as being the primary fileserver in CSC.
  
 
==Access==
 
==Access==
Currently it is accessible via serial interface via ascorbic-acid.
+
Configuration mechanisms are accessible either via SSH or serial interface, but through aspartame only. The NetApp is not visible on 134net at all.
 +
 
 +
Private IP TBD.
 +
 
 +
==Configuration==
 +
Should aspartame get totally hosed, or stability is long enough such that all sysadmin folk at the time have graduated, here is how to set up iSCSI on the NetApp+aspartame.
 +
 
 +
===NetApp Configuration===
 +
1. on the netapp,
 +
* start the iscsi stuff. Set up a client user/pass (m4burns: please fill this part out at some point)
 +
 
 +
===aspartame Configuration===
 +
Install open-iscsi:
 +
apt-get install open-scsi
 +
 
 +
Edit /etc/iscsi/iscsid.conf:
 +
node.startup = manual
 +
discovery.sendtargets.auth.authmethod=CHAP
 +
discovery.sendtargets.auth.username=username
 +
discovery.sendtargets.auth.password=password
 +
node.session.auth.authmethod=CHAP
 +
node.session.auth.username=username
 +
node.session.auth.password=password
 +
 
 +
Start open-iscsi service:
 +
service open-iscsi start
 +
 
 +
Scan for iSCSI devices from the NetApp:
 +
iscsiadm --mode discovery --type st --portal psilodump
 +
 
 +
This should dump out a ton of information, for example:
 +
[fe80::XXXX:XXXX:XXXX:XXXX]:3260,2001 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
 +
[fe80::XXXX:XXXX:XXXX:XXXX]:3260,2000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
 +
[fe80::XXXX:XXXX:XXXX:XXXX]:3260,2002 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
 +
[fe80::XXXX:XXXX:XXXX:XXXX]:3260,1000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
 +
10.15.134.131:3260,2002 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
 +
129.97.134.131:3260,2001 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
 +
10.15.134.130:3260,2000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
 +
129.97.134.130:3260,1000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
 +
 
 +
The .130 IPs correspond to one filer, and the .131 IPs correspond to the other filer. Currently we are only using one of the filers (psilodump).
 +
 
 +
This also populates the /etc/iscsi/nodes/iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca directory with all possible ways to access the NetApp. For testing purposes (i.e. node.startup = manual), this is okay.
 +
 
 +
Test to see if you can get the iSCSI device to show up correctly:
 +
iscsiadm --mode node --targetname "iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca"  --portal 10.15.134.130:3260 --login
 +
 
 +
This should produce output similar to:
 +
Logging in to [iface: default, target: iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca, portal: 10.15.134.130,3260]
 +
Login to [iface: default, target: iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca, portal: 10.15.134.130,3260]: successful
 +
 
 +
Check /dev/disk/by-path/ip* to ensure new disks show up:
 +
# ls -l /dev/disk/by-path/ip*
 +
    /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-0 -> ../../sda
 +
    /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-0-part1 -> ../../sda1
 +
    /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-1 -> ../../sdb
 +
    /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-1-part1 -> ../../sdb1
  
(Later, this will be accessible via aspartame, through its network interface+serial interface).
+
If this fails, check all your configuration again.
  
==Permissions==
+
If this succeeds, you are now ready to try autoconnecting the iSCSI device.
The NetApp does not support extended ACLs on NFSv4, so extended ACLs are supported via CIFS. This is particularly important for club accounts, and for some users hosting git repos out of their homedirs.
+
 
 +
Delete all extraneous entries from /etc/iscsi/nodes/iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca . This prevents the startup script from (a) hanging, and (b) being very upset. All that is left should be the interface you intend to connect through:
 +
# ls -l /etc/iscsi/nodes/iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca/
 +
    10.15.134.130,3260,2000
 +
 
 +
Edit /etc/iscsi/iscsid.conf:
 +
node.startup = automatic
 +
 
 +
For the init.d script to work correctly (i.e. properly mount things) we need to add a sleep to allow the device to settle:
 +
Edit /etc/init.d/open-iscsi roughly around line 127 to add a "sleep 1":
 +
  ...
 +
        # Now let's mount
 +
        sleep 1
 +
        log_daemon_msg "Mounting network filesystems"
 +
        MOUNT_RESULT=1
 +
        if mount -a -O _netdev >/dev/null 2>&1; then
 +
                MOUNT_RESULT=0
 +
                break
 +
        fi
 +
        log_end_msg $MOUNT_RESULT
 +
  ...
 +
 
 +
Now we can restart the service:
 +
service open-iscsi restart
 +
 
 +
Now you can configure partitions and mountpoints.
 +
 
 +
==Other notes==
 +
 
 +
4. set up the netapp filesystem and transfer old files from ginseng:
 +
* on ginseng, use parted to set up the mounted iscsi drive as an ext4 primary partition (setting up a partition of size >2TB requires care)
 +
* installed star as root on ginseng
 +
* transferred files with the following Makefile (make -j8):
 +
foo := $(wildcard /export/users/*)
 +
bar := $(patsubst /export/users/%,/mnt/iscsi/%,$(foo))
 +
all: $(bar)
 +
/mnt/iscsi/%: /export/users/%
 +
# echo $@ $<
 +
~/star-1.5.2/star/OBJ/x86_64-linux-cc/star \
 +
    -copy -p artype=exustar \
 +
    -C /export/users $(notdir $<) /mnt/iscsi
 +
*
  
==Commands==
 
aggr status -r aggr<num>
 
  Shows aggregate status
 
disk show -v
 
  Shows disks, and which filer they are owned by (currently all by psilodump)
 
storage
 
  storage related things
 
disk assign
 
  Assigns orphaned disks to a filer
 
vol
 
  Volume stuffs
 
  
==Layout==
+
==Disk information==
 
* shelf 1
 
* shelf 1
** 10,000 RPM FibreChannel disks
+
** 14x??? 10,000RPM FibreChannel disks
 
** Currently set to standalone filer+shelf, not set up
 
** Currently set to standalone filer+shelf, not set up
 
* shelf 2
 
* shelf 2
** 10,000 RPM FibreChannel disks
+
** 14x??? 10,000RPM FibreChannel disks
 
** Currently assigned to phlogiston, not set up (phlogiston is off)
 
** Currently assigned to phlogiston, not set up (phlogiston is off)
 
* shelf 3
 
* shelf 3
** 7,200 RPM ATA disks
+
** 14x500GB 7,200RPM ATA disks
 
** Currently assigned to psilodump
 
** Currently assigned to psilodump
 
* shelf 4
 
* shelf 4
** 7,200 RPM ATA disks
+
** 14x500GB 7,200RPM ATA disks
 
** Currently assigned to psilodump
 
** Currently assigned to psilodump
  
Line 57: Line 146:
 
* /vol/vol2users
 
* /vol/vol2users
 
** Users volume
 
** Users volume
 +
 +
==Commands==
 +
aggr status -r aggr<num>
 +
  Shows aggregate status
 +
disk show -v
 +
  Shows disks, and which filer they are owned by (currently all by psilodump)
 +
storage
 +
  storage related things
 +
disk assign
 +
  Assigns orphaned disks to a filer
 +
vol
 +
  Volume stuffs
  
 
==Terminology==
 
==Terminology==
 
* RAID-DP - Double Parity RAID6
 
* RAID-DP - Double Parity RAID6
 
==iSCSI setup log==
 
1. on the netapp,
 
* start the iscsi stuff. Set up a client user/pass (m4burns: please fill this part out at some point)
 
 
2. set up aspartame to talk with the netapp. (all commands were run on aspartame)
 
* installed open-iscsi
 
* set discovery.sendtargets.auth.* and node.session.auth.* in /etc/iscsi/iscsid.conf (users and passwords are the same)
 
* start open-iscsi with service open-iscsi start
 
* to verify that the netapp is exporting the iSCSI interface properly,
 
iscsiadm  -m discovery --type st -p psilodump
 
 
iscsiadm  --mode node  --targetname "iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca"  --portal 129.97.134.130:3260 --login
 
 
3. set up ginseng the same way as aspartame
 
 
4. set up the netapp filesystem and transfer old files from ginseng:
 
* on ginseng, use parted to set up the mounted iscsi drive as an ext4 primary partition (setting up a partition of size >2TB requires care)
 
* installed star as root on ginseng
 
* transferred files with the following Makefile (make -j8):
 
foo := $(wildcard /export/users/*)
 
bar := $(patsubst /export/users/%,/mnt/iscsi/%,$(foo))
 
all: $(bar)
 
/mnt/iscsi/%: /export/users/%
 
# echo $@ $<
 
~/star-1.5.2/star/OBJ/x86_64-linux-cc/star \
 
    -copy -p artype=exustar \
 
    -C /export/users $(notdir $<) /mnt/iscsi
 
*
 

Revision as of 01:41, 4 February 2013

As of 2013, the CSC has a NetApp FAS3000 series which is capable of hosting network shares. It was donated to us by CSCF. It is also pretty old.

Documentation

All the manuals are hosted in ~sysadmin/netapp-docs/

Relevant docs for storage modification are: smg.pdf, sysadmin.pdf

iSCSI documentation is in ontop/bsag.pdf

Background

While the NetApp supports both NFS and CIFS, neither of these export options provide the versatility nor the options we desire of a network fileshare. Instead, we have configured the NetApp to export iSCSI block devices to be mounted on aspartame. Therefore, aspartame now replaces ginseng as being the primary fileserver in CSC.

Access

Configuration mechanisms are accessible either via SSH or serial interface, but through aspartame only. The NetApp is not visible on 134net at all.

Private IP TBD.

Configuration

Should aspartame get totally hosed, or stability is long enough such that all sysadmin folk at the time have graduated, here is how to set up iSCSI on the NetApp+aspartame.

NetApp Configuration

1. on the netapp,

  • start the iscsi stuff. Set up a client user/pass (m4burns: please fill this part out at some point)

aspartame Configuration

Install open-iscsi:

apt-get install open-scsi

Edit /etc/iscsi/iscsid.conf:

node.startup = manual
discovery.sendtargets.auth.authmethod=CHAP
discovery.sendtargets.auth.username=username
discovery.sendtargets.auth.password=password
node.session.auth.authmethod=CHAP
node.session.auth.username=username
node.session.auth.password=password

Start open-iscsi service:

service open-iscsi start

Scan for iSCSI devices from the NetApp:

iscsiadm --mode discovery --type st --portal psilodump

This should dump out a ton of information, for example:

[fe80::XXXX:XXXX:XXXX:XXXX]:3260,2001 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
[fe80::XXXX:XXXX:XXXX:XXXX]:3260,2000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
[fe80::XXXX:XXXX:XXXX:XXXX]:3260,2002 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
[fe80::XXXX:XXXX:XXXX:XXXX]:3260,1000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
10.15.134.131:3260,2002 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
129.97.134.131:3260,2001 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
10.15.134.130:3260,2000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca
129.97.134.130:3260,1000 iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca

The .130 IPs correspond to one filer, and the .131 IPs correspond to the other filer. Currently we are only using one of the filers (psilodump).

This also populates the /etc/iscsi/nodes/iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca directory with all possible ways to access the NetApp. For testing purposes (i.e. node.startup = manual), this is okay.

Test to see if you can get the iSCSI device to show up correctly:

iscsiadm --mode node --targetname "iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca"  --portal 10.15.134.130:3260 --login

This should produce output similar to:

Logging in to [iface: default, target: iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca, portal: 10.15.134.130,3260]
Login to [iface: default, target: iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca, portal: 10.15.134.130,3260]: successful

Check /dev/disk/by-path/ip* to ensure new disks show up:

# ls -l /dev/disk/by-path/ip*
   /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-0 -> ../../sda
   /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-0-part1 -> ../../sda1
   /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-1 -> ../../sdb
   /dev/disk/by-path/ip-10.15.134.130:3260-iscsi-iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca-lun-1-part1 -> ../../sdb1

If this fails, check all your configuration again.

If this succeeds, you are now ready to try autoconnecting the iSCSI device.

Delete all extraneous entries from /etc/iscsi/nodes/iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca . This prevents the startup script from (a) hanging, and (b) being very upset. All that is left should be the interface you intend to connect through:

# ls -l /etc/iscsi/nodes/iqn.1992-08.com.netapp:psilodump.csclub.uwaterloo.ca/
    10.15.134.130,3260,2000

Edit /etc/iscsi/iscsid.conf:

node.startup = automatic

For the init.d script to work correctly (i.e. properly mount things) we need to add a sleep to allow the device to settle: Edit /etc/init.d/open-iscsi roughly around line 127 to add a "sleep 1":

 ...
       # Now let's mount
       sleep 1
       log_daemon_msg "Mounting network filesystems"
       MOUNT_RESULT=1
       if mount -a -O _netdev >/dev/null 2>&1; then
               MOUNT_RESULT=0
               break
       fi
       log_end_msg $MOUNT_RESULT
 ...

Now we can restart the service:

service open-iscsi restart

Now you can configure partitions and mountpoints.

Other notes

4. set up the netapp filesystem and transfer old files from ginseng:

  • on ginseng, use parted to set up the mounted iscsi drive as an ext4 primary partition (setting up a partition of size >2TB requires care)
  • installed star as root on ginseng
  • transferred files with the following Makefile (make -j8):
foo := $(wildcard /export/users/*)
bar := $(patsubst /export/users/%,/mnt/iscsi/%,$(foo))
all: $(bar)
/mnt/iscsi/%: /export/users/%
	# echo $@ $<
	~/star-1.5.2/star/OBJ/x86_64-linux-cc/star \
	    -copy -p artype=exustar \
	    -C /export/users $(notdir $<) /mnt/iscsi


Disk information

  • shelf 1
    • 14x??? 10,000RPM FibreChannel disks
    • Currently set to standalone filer+shelf, not set up
  • shelf 2
    • 14x??? 10,000RPM FibreChannel disks
    • Currently assigned to phlogiston, not set up (phlogiston is off)
  • shelf 3
    • 14x500GB 7,200RPM ATA disks
    • Currently assigned to psilodump
  • shelf 4
    • 14x500GB 7,200RPM ATA disks
    • Currently assigned to psilodump

Aggregates

  • aggr0
    • Root aggregate volume, in RAID-DP
  • aggr1
    • Music aggregate volume, in RAID-DP
  • aggr2
    • Users aggregate volume, in RAID-DP

Volumes

  • /vol/vol0
    • Root volume
  • /vol/vol1music
    • Music volume
  • /vol/vol2users
    • Users volume

Commands

aggr status -r aggr<num>
  Shows aggregate status
disk show -v
  Shows disks, and which filer they are owned by (currently all by psilodump)
storage
  storage related things
disk assign
  Assigns orphaned disks to a filer
vol
  Volume stuffs

Terminology

  • RAID-DP - Double Parity RAID6