Tips, Tricks, and Notes on running RAID1 and RAID5 on XCP-NG: Part 2

This is a continuation of a series. See Part 1 here.

Now that I have the base system installed on a RAID1 array, along with a Local Storage Repo residing there, I want to create an array for "bulk" storage. I have 3 8TB disks I want to put into a RAID5 array. This will give me a 16TB block device I can use as a sort of home NAS. Since this is RAID5, we can more efficiently expand this storage just by adding another 8TB+ disk. Additionally, I can use LVM to effectively split this storage between NAS storage and another Local Storage Repo for larger VM disk allocations.

Prep the disks

Before we do anything, we need to adjust the device timeouts on EACH drive you have installed. It's not uncommon for SATA to take a long time to recover from an error, and if this results in a controller reset, you can risk RAID array failure and data loss. For more info on this, I would recommend reading https://strugglers.net/~andy/blog/2015/11/09/linux-software-raid-and-drive-timeouts/ . This article mentions setting the timeout on the disks. In my case, even though the command to set the timeouts didn't fail, it never seemed to work, as the timeout values could not be listed and verified. Instead, to be safe, I'd recommend applying the 180 second (3 minute) timeout to all the drives you have in your system using this command:

for disk in `ls /sys/block/*/device/timeout` ; do echo 180 > $disk ; done

Then add a udev rule so that it applies on startup AND when new drives are inserted:

cat << EOF > /etc/udev/rules.d/60-set-disk-timeout.rules.test

# Set newly inserted disk I/O timeout to 3 minutes
ACTION=="add", SUBSYSTEMS=="scsi", DRIVERS=="sd", ATTRS{timeout}=="?*", RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/timeout'"
EOF

systemctl restart systemd-udevd

Next, I would recommend removing all partitions on each disk and re-setting the disk labels to GPT. Beware! This and pretty much everything to follow is destructive. Be sure you are working with the correct disk! Checking the "dmesg" command output after inserting a new disk is usually how I verify.

parted /dev/sdz

# Respond with Y to confirm all data on the disk will be lost

Partition the drives (replacing /dev/sdz with your disk) . Repeat this for each of your disks:

echo -en "mklabel gpt\nmkpart primary ext4 0% 100%\nset 1 raid on\nalign-check optimal 1\nprint\nquit\n" | parted -a optimal /dev/sdz

RAID5 and LVM Setup

Now, let's build our RAID5 array:

mdadm --create --verbose /dev/md6 --level=5 --raid-devices=3 /dev/sdx /dev/sdy /dev/sdz

# Set up mdmonitor mail
# In /etc/ssmtp/ssmtp.conf:
mailhub=mail.your-smtp-server.com

# In /etc/mdadm.conf:
MAILADDR address@youremail.com

Then partition the resulting md device to have one large LVM partition

gdisk /dev/md6

# Commands within gdisk...
p
# You should see no partitions

n
# Hit Enter (3x) when asked for partition number, first, and last sectors
# Use a Hex code of 8e00 for the filesystem type of Linux LVM
p
# You should then see something like this as your partition table, with a different size, obviously:
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048     31255572446   14.6 TiB    8E00  Linux LVM

# Write partition table and exit gdisk:
w

# Answer "Y" to confirmation to write GPT data

To clarify the above, here are the commands that are run in gdisk:

(p)rint partition tables
(n)ew partition
(enter) Use partition number 1 (default)
(enter) Start at first sector on device (default)
(enter) End at last sector on device (default) 
(8e00) Linux LVM partition code
(p)rint the partition table
(w)rite the partition table and exit 

Next, let's set up LVM using the partition:

# Tell the kernel to reload the partition table
partprobe /dev/md6

# Create the Physical Volume
pvcreate --config global{metadata_read_only=0} /dev/md6p1

# Create the Volume Group "BulkStorage00"
vgcreate --config global{metadata_read_only=0} BulkStorage00 /dev/md6p1

# Set the volume groups active so they come back up on reboot
vgchange --config global{metadata_read_only=0} -ay

### ??? Edit /etc/grub.cfg and remove "nolvm" from the "XCP-ng" menu entry

# Not sure if this is needed, but doesn't hurt (rebuild init ram fs)
dracut --force

Activating Volume Groups on Boot

Disclaimer: I know, I know. This is admittedly a hack. There's probably a better way of doing this that fits within the design of XCP-NG that I'm not aware of. If someone suggests a better way of handling this, I'll gladly update here.

When we need to reboot, our Logical Volumes won't be active. I believe this has something to do with XCP-NG/XenServer's unique handling of LVM. I've banged my head on this problem for far too long, so begrudgingly, here's the workaround to it:

# Make rc.local executable
chmod u+x /etc/rc.local

# Enable the rc.local service
systemctl enable rc-local

# Add our vgchange activation command to rc.local
echo "vgchange --config global{metadata_read_only=0} -ay ; sleep 1" >> /etc/rc.d/rc.local

Local Storage Repository

Now let's create a 200GB logical volume (start small as we can expand later)

lvcreate --config global{metadata_read_only=0} -L 200G -n "RAID_5_Storage_Repo" BulkStorage00

And create the Storage Repo in XCP-NG (using the "ext" filesystem for thin provisioning)

xe sr-create content-type=user device-config:device=/dev/disk/by-id/dm-name-BulkStorage00-RAID_5_Storage_Repo host-uuid=<tab_to_autocomplete_your_host_uuid> name-label="RAID 5 Storage Repo" shared=false type=ext

And that's it! The above creates another set of LVM layers on top of your existing BulkStorage00/RAID_5_Storage_Repo logical volume which consists of a physical volume, volume group, and logical volume that are natively managed by XCP-NG. You should now have a 200GB storage repo named "RAID 5 Storage Repo" in XCP-ng Center and/or XOA. Next, let's set up a bulk store to share out via SMB/CIFS...

Bulk Storage Share Setup

First let's allocate 500GB of that space for another Logical Volume named "BulkVolume00"

lvcreate --config global{metadata_read_only=0} -L 500G -n "BulkVolume00" BulkStorage00

And lets format it with EXT4

mkfs.ext4 /dev/BulkStorage00/BulkVolume00

Let's set up /etc/fstab (edit with vi or nano) to mount the device by adding the following line to the end of the file

/dev/BulkStorage00/BulkVolume00 /opt/BulkVolume00       ext4    rw,noauto        0 0

And lets create a mount point for the device and mount it

mkdir /opt/BulkVolume00
mount /opt/BulkVolume00

We'll also want to set this up to mount on boot (note, as above with our Storage Repository, we have to use a workaround to mount this AFTER the logical volumes have been activated

echo "mount /opt/BulkVolume00" >> /etc/rc.d/rc.local

And lastly, lets get that shared out via SMB

# Install Samba server
yum --enablerepo base,updates install samba

# Make a backup copy of the original SMB configuration (just in case)
cp -rp /etc/samba/smb.conf /etc/samba/smb.conf.orig

Then edit /etc/samba/smb.conf, erasing the contents and using this as your template:

[global]
        workgroup = MYGROUP
        # This must be less than 15 characters.
        # Run "testparm" after editing this file to verify.
        netbios name = XCP-NG-WHATEVER
        security = user
        map to guest = Bad User

[BulkVolume00]
	comment = Bulk RAID5 Storage Volume
	path = /opt/BulkVolume00
	guest ok = yes
	writable = yes
        read only = no
	browseable = yes
        force user = root
	create mode = 0777
	directory mode = 0777
	force create mode = 0777
	force directory mode = 0777

Note that this is set up to be very permissive for use on a secured/home network. Setting this up for more elaborate security is beyond the scope of this tutorial.

Next let's (re)start samba, and poke the required holes in the firewall

# Verify your smb.conf is sane
testparm
# Enable and start samba services
systemctl enable smb.service
systemctl enable nmb.service
systemctl start smb.service
systemctl start nmb.service

# Edit "/etc/sysconfig/iptables" and add the following lines below the port 443 rule:
-A RH-Firewall-1-INPUT -m conntrack --ctstate NEW -m udp -p udp --dport 137 -j ACCEPT
-A RH-Firewall-1-INPUT -m conntrack --ctstate NEW -m udp -p udp --dport 138 -j ACCEPT
-A RH-Firewall-1-INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 139 -j ACCEPT
-A RH-Firewall-1-INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 445 -j ACCEPT

# Then restart iptables to apply
systemctl restart iptables

Finally, test this share in Windows (or your favorite SMB/CIFS client) by opening the URI to your server's IP, ex: \\192.168.1.2\BulkVolume00\

The next installment of this series will be a rough collection of tips on managing this XCP-NG RAID5 combo. Stay tuned.