Tips, Tricks, and Notes on running RAID1 and RAID5 on XCP-NG: Part 1

Note: This article series is specifically written with XCP-NG in mind, however if you are using XenServer, it is possible that some if not all of this is still applicable. YMMV.

Recently when installing XCP-NG on my home server, I ran into some persistent issues with setting up RAID1 on my boot disks, and setting up RAID5 for my secondary/bulk Local Storage Repository. Here are my notes and tips on succeeding with an installation such as this.

I have a chassis with 2x400GB SATA disks in a RAID1 array used for the boot disks that hold the OS and the first Local Storage Repository, and 3 8TB SATA disks in a RAID5 array with LVM on top, serving as both a Local Storage Repository as well as a bulk storage served via SMB/CIFS/NFS as a sort of NAS.

Installation Prep

Starting from scratch, we might want to wipe all the partition tables, boot records, and/or RAID superblocks on each target disk. This is optional, but if you know you don't care about the data on your disks, it should help ensure success in case you had a prior installation of a RAID superblock or the GRUB bootloader on any of the disks. If you care about the data on any of the disks in your system, power off, unplug those disks, and start over. Boot up the installation disk, and type Alt+F2 to get to the console, then run the following:

WARNING, THIS IS DESTRUCTIVE:

for dev in `ls /dev/md*` ; do mdadmin --stop $dev ; done

for dev in `ls /dev/sd[a-z]+` ; do echo "Wiping Partitions and MBR on $dev" ; dd if=/dev/zero of=$dev bs=512 count=1 ; done

Next, reboot to start the installer over again, ensuring we start with a clean slate. Run the following to verify:

lsblk

You should see no partitions or raid labels (ie, md127) on any of your target disks. If you do, you may need to re-run the above and try again.

Installation

Proceed as normal with the installation prompts, selecting the Software RAID option when it comes up.

Select the two (or more) disks you wish to add to the RAID1 array, then enter Create

Select your RAID disk as the install target (usually md127)

Select your RAID disk for your local storage (usually md127)

Proceed as normal with installation

When the install starts, type Alt+F2 to go to the CLI console

Run this to see the RAID resync process:

while true ; do tput clear ; date ; mdadmin --detail /dev/md127 | grep -v ^$ | grep -e ^ -e "Resync Status.*"; sleep 10 ; done

You should see a line starting with "Resync Status" indicating the percentage complete.

You may type Alt+F1 to go back to the installation progress screen.

Once the installation is complete, DO NOT REBOOT YET! I suspect this was the cause of an issue I had on one of my attempts doing this. The RAID1 array needs to finish syncing, or else you may be missing the required MBR/GRUB information on one of your disks, and your system may fail to boot if the non-synced disk happens to be the first in your BIOS/HBA boot order. Type Alt+F2 to go back to the CLI console and re-check the Resync Status progress. Once it reaches 100% complete and is in a State of "clean", you may proceed.

Type Alt+F1 and finish the installation as normal.

Continue on with Part 2 in this series