Categories
General Linux

Installing a software RAID 10 Debian system with LUKS disk encryption

In this guide we will be installing Debian 9 (aka stretch) on a physical server with 4 disks. The role of this machice is to be used as a Storage/NAS system. We will create a software RAID 10 setup, with LVM and LUKS full disk encryption. Our goals:

  • Install a Debian 9 with RAID10/LVM/LUKS.
  • Secure SSH.
  • Enable Firewall (UFW).
  • Setup bonding with the two network cards.
  • Setup remote system unlock with dropbear and initramfs.
  • Setup disk-monitoring with smartmontools and mdadm.
  • Setup kexec for faster reboots.

Computer specs

  • Type: HP ProLiant MicroServer
  • CPU: AMD Turion(tm) II Neo N40L Dual-Core Processor
  • RAM: 2GB
  • Disks: 4x3TB SATA
  • Network:
    • Broadcom Limited NetXtreme BCM5723 Gigabit Ethernet PCIe (on board)
    • Intel Corporation 82574L Gigabit Network Connection (extra)

Assumptions

  • Server IP: 192.168.1.10
  • Netmask: 255.255.255.0
  • Gateway IP: 192.168.1.1
  • DNS IP: 192.168.1.1
  • Hostname: storage.example.com

Install Debian stretch

Basic Settings

It would probably be more clear if there were screenshots for each step, but this was an installation on a physical server and taking photos for each step, opposes my laziness :). Just follow the instructions and you will be fine.

  • Choose: Install
  • Language: English
  • Country: other
  • Europe: Cyprus
  • Country to base default locale settings: United States
  • Keymap to use: American English
  • Primary network interface: enp3s0: Broadcom Limeted NetXtream BCM5723 Gigabit Ethernet PCIe
  • Let it get an IP from DHCP
  • Hostname: storage
  • Domain name: example.com
  • Root password: SomethingBigAndUnpredictable
  • Re-enter password to verify: SomethingBigAndUnpredictable
  • Full name: Sysadmin
  • Username: admin
  • Choose a password for the new user: AlsoSomethingBigAndUnpredictable
  • Re-enter password to verify: AlsoSomethingBigAndUnpredictable
  • Select your time zone: Asia/Nicosia
  • Partitioning method: Manual

Feel free to adjust the above according to your own preferences.

Partitioning

There are 4 disks of 3TB each (3.0 TB SATA):

  • SCSI1 (0,0,0) (sda)
  • SCSI2 (0,0,0) (sdb)
  • SCSI3 (0,0,0) (sdc)
  • SCSI4 (0,0,0) (sdd)

Not really SCSI but SATA in fact.

Partition the devices

Then create a raid partition for /boot:

  • Select the free space of sda and ‘Enter’
  • Create a new partition
  • New partition size: 512 MB
  • Location of new partition: Beginning
  • Use as: physical volume for RAID
  • Done setting up the partition

Lastly create the raid partition to be used by the encrypted volume:

  • Select the free space of sda and ‘Enter’
  • Create a new partition
  • New partition size: 3.0 TB
  • Location of new partition: Beginning
  • Use as: physical volume for RAID
  • Done setting up the partition

Repeat the above steps for sdb, sdc and sdd.

Setup Software RAID 10

First select ‘Configure software RAID’ and follow these steps:

  • Write the changes to the storage devices and configure RAID? Yes

The we create the software RAID (MD) devices. First we create device md0 for /boot:

  • Create MD Device
  • RAID10
  • Number of active devices inn the RAID10 array: 4
  • Number of spare devices inn the RAID10 array: 0
  • Active devices for the RAID10 array (use ‘Space bar’ to select)
    • /dev/sda2
    • /dev/sdb2
    • /dev/sdc2
    • /dev/sdd2
  • Press ‘Continue’ when done.

Then we create the software RAID device to be used for the encrypted volume (md1):

  • Create MD Device
  • RAID10
  • Number of active devices inn the RAID10 array: 4
  • Number of spare devices inn the RAID10 array: 0
  • Active devices for the RAID10 array (use the ‘Space bar’ to select)
    • /dev/sda3
    • /dev/sdb3
    • /dev/sdc3
    • /dev/sdd3
  • Press ‘Continue’ when done.
  • Press ‘Finish’ when done.

Create the /boot volume

When done press ‘Finish partitioning and write changes to disk’.`

When finished you will see a ‘RAID10 device #0 1GB Software RAID device’:

  • Select: #1 1.0GB
  • Use as: Ext4 journaling file system
  • Mount point: /boot
  • Done setting up the partition

Setup the encrypted volume

We will be using the software RAID /dev/md1 device for the encrypted volume.

Now select ‘Configure encrypted volumes’ and follow these steps:

  • Write the changes to disk and configure encrypted volumes? Yes
  • Create encrypted volumes

  • Select: /dev/md1
  • Erase data: yes (this will take a long time)
  • Done setting up the partition

  • Write the changes to disk and configure encrypted volumes? Yes

  • Finish
  • Encryption passphrase: MyVeryLongEncryptionPassphrase
  • Re-enter the passphrase to verify: MyVeryLongEncryptionPassphrase
  • Setup LVM

    Next we select the ‘Configure the Logical Volume Manager’ option and follow these steps:

    • Write the changes to disks and configure LVM? Yes
  • Create volume group

  • Volume group name: VG00
  • Devices for the new volume group:
    • /dev/mapper/md1_crypt
  • Then we create the Logical Volumes (LV). First let’s create a SWAP volume:

    • Create logical volume
    • Volume group: VG00
    • Logical volume name: SWAP
    • Logical volume size: 2048MB (2G is more than enough for this system)

    Lastly we create the system (ROOT) volume. On an enterprise installation we may want to use different volumes for /usr, /home, /var, etc but for a home installation we will be fine to use just one.

    • Create logical volume
    • Volume group: VG00
    • Logical volume name: ROOT
    • Logical volume size: 5996818MB (All available space)

    Press ‘Finish’ when done.

    Start the installation

    After all the steps are completed these Logical Volumes will be present on the system:

    • LVM VG VG00, LV ROOT 0 6.0 TB
    • LVM VG VG00, LV SWAP 0 2.0 GB

    Create the ROOT filesystem

    Under the ‘LVM VG VG00, LV ROOT 0 6.0 TB’ line select the ‘#1 6.0TB’ option:

    • Use as: Ext4 journaling file system
    • Mount point: /
    • Done setting up this partition

    Create the SWAP space

    Under the ‘LVM VG VG00, LV SWAP 0 2.0 GB’ line select the ‘#1 2.0GB’ option:

    • Use as: swap area
    • Done setting up this partition

    Now we are ready to write the changes and start the installation. Press the ‘Finish partitioning and write changes to disk’ option to continue:

    • Write the changes to disks? Yes

    Wait for the base install to finish. Then select a country close to you. No debian mirrors in Cyprus so I use UK:

    • Debian archive mirror country: United Kingdom
    • Debian archive mirror: ftp.uk.debian.org
    • HTTP proxy: (none)

    Wait for the APT configuration to Finish.

    • Participate in the package usage survey: no
  • Choose software to install:

    • SSH server
    • standard system utilities
  • Wait while software is installing

    • Install the GRUB boot loader to the master boot record.
    • Device for boot loader installation:
      • /dev/sda

    Wait for the installation to finish and reboot. Remember to remove the USB during the reboot cycle.

    Post install steps

    During start-up you will see the ‘Please unlock md1_crypt’ prompt. Type your LUKS passphrase to unlock the disk and continue.

    Update and Upgrade

    Login as root:

    # apt update && apt -y dist-upgrade
    

    Install essential packages

    # apt -y install vim htop multitail ntp byobu ufw unattended-upgrades downtimed
    

    Secure ssh

    You need to generate an SSH key pair on you PC, if you don’t have one (you should!):

    $ ssh-keygen -b 4096
    

    Copy the public key:

    $ cat ~/.ssh/id_rsa.pub
    

    Paste the public key at the end of the /root/.ssh/authorized_keys file in your server and try to login from your PC:

    $ ssh root@192.168.1.10
    

    Some final adjustments on your SSH config (/etc/ssh/sshd_config). Change these values:

    Port 2233
    PasswordAuthentication no
    

    Restart SSH:

    # systemctl restart ssh.service
    

    Enable the UFW firewall

    We are using port 2233 for SSH so we need to allow that and enable the firewall:

    # ufw allow 2233/tcp
    # ufw enable
    

    Setup bonding

    Since we have two ethernet cards, we may take advantage of thr Linux bonding feature and join them as one. We will be using the Adaptive load balancing mode which provides load balancing of transmit, load balancing of receive for IPv4 and requires no configuration from the switch side.

    First we need to install ifenslave:

    # apt -y install ifenslave
    

    Set up this in /etc/network/interfaces:

    # This file describes the network interfaces available on your system
    # and how to activate them. For more information, see interfaces(5).
    
    source /etc/network/interfaces.d/*
    
    # The loopback network interface
    auto lo
    iface lo inet loopback
    
    # enp2s0 is manually configured, and slave to the "bond0" bonded NIC
    auto enp2s0
    iface eth0 inet manual
        bond-master bond0
    
    # enp3s01 is also manually configured, thus creating a 2-link bond.
    auto enp3s0
    iface eth1 inet manual
        bond-master bond0
    
    # bond0 is the bonded NIC and can be used like any other normal NIC.
    # bond0 is configured using static network information.
    auto bond0
    iface bond0 inet static
        address 192.168.1.10
        gateway 192.168.1.1
        netmask 255.255.255.0
    
        # bond0 uses adaptive load balancing
        bond-mode 6
        bond-miimon 100
        bond-slaves enp2s0 enp3s0
    

    An ifup bond0 should bring the bonded interface up. Or you can just reboot.

    Setup remote-unlock with dropbear

    The server will be a headless system, located in a difficult to access location. So we need a way to unlock it when a power failure occurs. The most convenient way to do this is to use a mandos server but convenience comes at a cost. A safer and easier way is to use dropbear during boot (initrd). The weak point of this solution is that the server will be basically offline until the sysadmin manually unlocks it, to boot.

    First we install dropbear for initrd:

    # apt -y install dropbear-initramfs
    

    Then we set a custom ssh port for dropbear. This better be different than the custom ssh port we used earlier. Change the dropbear port to 2244 in /etc/dropbear-initramfs/config:

    DROPBEAR_OPTIONS="-p 2244"
    

    Add the static IP in the initramtools configuration (/etc/initramfs-tools/initramfs.conf):

    IP=192.168.1.10::192.168.1.1:255.255.255.0:storage:enp3s0:off
    

    Copy the authorized_keys file in /etc/dropbear-initramfs:

    # cp /root/.ssh/authorized_keys /etc/dropbear-initramfs/
    

    Regenerate the initrd file:

    # update-initramfs -u
    

    Now reboot and ssh to it to test it:

    $ ssh -p 2244 root@192.168.1.10
    

    If your pubkeys are in place you will enter a busybox shell. Enter the crypt-unlock command, supply your unlock passphrase and the system will boot to the encrypted system.

    Setup a local MTA for notifications

    We will be using our main mailserver as a smarthost for mail to go through.

    Install the postfix MTA and the mail utility:

    # apt -y install postfix mailutils
    

    Answer these questions:

    • General type of mail configuration: Internet with smarthost
    • System mail name: storage.example.com
    • SMTP relay host (blank for none): smtp.example.com

    Test it:

    # echo 'Testing #1' | mail -s 'Test #1' user@example.com
    

    If you get a mail in your mailbox then everything is set. If not extra configuration may be needed on the smarthost. Contact the sysadmin of the smarthost, or check the logs if you access to it.

    Setup pro-active disk monitoring

    Install smartmontools:

    # apt -y install smartmontools
    

    Enable S.M.A.R.T, offline testing, attribute autosave, short and long test on all 4 devices. Add these lines in /etc/smartd.conf:

    /dev/sda -a -d sat -o on -S on -s (S/../.././02|L/../../6/03) -m user@example.com -M exec /usr/share/smartmontools/smartd-runner
    /dev/sdb -a -d sat -o on -S on -s (S/../.././02|L/../../6/03) -m user@example.com -M exec /usr/share/smartmontools/smartd-runner
    /dev/sdc -a -d sat -o on -S on -s (S/../.././02|L/../../6/03) -m user@example.com -M exec /usr/share/smartmontools/smartd-runner
    /dev/sdd -a -d sat -o on -S on -s (S/../.././02|L/../../6/03) -m user@example.com -M exec /usr/share/smartmontools/smartd-runner
    

    Restart smartmontools:

    # systemctl restart smartmontools.service
    

    Setup Software RAID10 monitoring

    We also need to setup monitoring for the software raid. Add your email address in the /etc/mdadm/mdadm.conf file:

    MAILADDR user@example.com
    

    Restart the mdmonitor service:

    # systemctl restart mdmonitor.service
    

    Setup kexec for faster reboots

    Kexec is a Linux kernel mechanism that can load a fresh kernel from a running system. This results in a “reboot” without in fact rebooting the computer. The system loads a new kernel, the system appears “rebooted” but skipping the BIOS?UEFI initialization, thus resulting in faster reboots.

    Install kexec-tools:

    # apt -y install kexec-tools
    

    The ‘Should kexec-tools handle reboots (sysvinit only)?’ question is related only to sysvinit systems. Since we are using systemd, it has no effect in our case.

    Now if you want to reboot instead of running reboot you can run systemctl kexec. The latter command will reboot the system without going though BIOS/UEFI, POST etc and your system downtime is minimized.

    And we are done! Store your server in a protected location, add a UPS for power backup and you are ready.

    References

    • https://wiki.debian.org/Bonding
    • https://help.ubuntu.com/community/UbuntuBonding
    • https://www.theo-andreou.org/?p=1579
    • https://wiki.recompile.se/wiki/Mandos
    • http://forums.ayksolutions.com/forum/documentation/knowledgebase/general-server-questions/641-proactively-monitoring-hard-drive-health-using-smartd

    By Theodotos Andreou

    An old school Linux guy, a founding member of Ubuntucy and a founding member of Cyprus FOSS community. Currently working as sysadmin in the Cyprus University of Technology.

    https://www.ubuntucy.org

    https://ellak.org.cy

    13 replies on “Installing a software RAID 10 Debian system with LUKS disk encryption”

    PLEASE LOSE THESE STEPS as they are SUPER confusing, and not offered by Debian9.3(stretch):

    First create the reserved BIOS Boot area:

    Select the free space of sda and ‘Enter’
    Create a new partition
    New partition size: 512 MB
    Location of new partition: Beginning
    Use as: Reserved BIOS boot area
    Done setting up the partition

    Otherwise, this is PRICELESS!!!
    Keep up the great info-share Theo!

    I don’t think GRUB 2 will work on a raid 10, unless it’s been updated very recently. My efforts in the past have forced me to use raid 1 for the /boot volume. Am I just living in the past?

    I remember that yes. How things change… 🙂

    root@storage:~# cat /proc/mdstat 
    Personalities : [raid10] [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] 
    md1 : active raid10 sdb3[1] sda3[0] sdd3[3] sdc3[2]
          5858271232 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
          bitmap: 12/44 pages [48KB], 65536KB chunk
    
    md0 : active raid10 sdb2[1] sda2[0] sdd2[3] sdc2[2]
          997376 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
    
    unused devices: <none>
    
    root@storage:~# df -hT
    Filesystem            Type      Size  Used Avail Use% Mounted on
    udev                  devtmpfs  990M     0  990M   0% /dev
    tmpfs                 tmpfs     201M   21M  180M  11% /run
    /dev/mapper/VG00-ROOT ext4      5,5T  488G  4,7T  10% /
    tmpfs                 tmpfs    1002M   96K 1002M   1% /dev/shm
    tmpfs                 tmpfs     5,0M     0  5,0M   0% /run/lock
    tmpfs                 tmpfs    1002M     0 1002M   0% /sys/fs/cgroup
    /dev/md0              ext4      943M   94M  785M  11% /boot
    tmpfs                 tmpfs     201M     0  201M   0% /run/user/0

    Outstanding! I’ve been waiting for that to happen.

    One more thing. I’m not certain this will work as described with the BIOS partition step removed. Did it just work for you without that step on a BIOS machine, or are you running UEFI. Once again — am I just living in the past?

    Just finished testing, and it didn’t work for me. Using an HP Proliant Microserver Gen8 (BIOS), I followed these steps to the letter. The OS installs, then fails when trying to install GRUB2. For the second test, I created a GRUB BIOS partition on each drive (for failover), tried again, and installation completed successfully.

    Hi Theo, thanks very much for the guide, much appreciated! One issue I ran into: I followed the guide on a Dell T40 with Debian 10.3, to install a raid 1 array with to SSD’s. After trying to finish the partitioning, I get ‘No EFI System Partition Found, return to partioning?”

    This was solved by re-starting all over and deleting the partition table, then creating an extra 512MB FAT32 non-raid partiion at the start of the disk, and marking it for use as EFI. 512MB is probably way too large, but I did not feel like doing this a third time.

    Unfortunately, EFI partitions cannot be software raided it seems. If you have any thoughts on this please let me know.

    After setup, the system boots succesfully with two disks installed. However, then as a test, when removing the disk with the EFI partition on it, and trying to boot, I get ‘no boot device’. This was rather obvious since I had to install the EFI partition manually and could only do this on one disk. What suprised me though was that on testing with the other disk removed, I get “cryptsetup: waiting for encrypted source device xxxx-xxxxx-xxxx-xxxx”. I ended up going without raid.

    Leave a Reply to Joshua Palmer Cancel reply

    Your email address will not be published. Required fields are marked *