How to create custom bootable rescue ISO image for SolusIO?

1. Access your server via SSH.

2. Download the rescue system using debootstrap, and prepare mounts:

   # apt-get install -y binutils debootstrap squashfs-tools xorriso grub-pc-bin grub-efi-amd64-bin mtools && mkdir $HOME/rescue-iso && debootstrap --arch=amd64 --variant=minbase focal $HOME/rescue-iso/chroot && mount --bind /dev $HOME/rescue-iso/chroot/dev && mount --bind /run $HOME/rescue-iso/chroot/run

1. Chroot into the created environment:

   # chroot $HOME/rescue-iso/chroot

   SVM_WARN: The below steps are executed inside chroot

2. Mount proc and sysfs:

   # mount none -t proc /proc && mount none -t sysfs /sys && mount none -t devpts /dev/pts && export HOME=/root && export LC_ALL=C

3. Configure default repositories - in this example for Ubuntu 20.04:

   # cat <<EOF > /etc/apt/sources.list
   deb http://archive.ubuntu.com/ubuntu/ focal main restricted universe multiverse
   deb-src http://archive.ubuntu.com/ubuntu/ focal main restricted universe multiverse
   
   deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted universe multiverse
   deb-src http://archive.ubuntu.com/ubuntu/ focal-updates main restricted universe multiverse
   
   deb http://archive.ubuntu.com/ubuntu/ focal-security main restricted universe multiverse
   deb-src http://archive.ubuntu.com/ubuntu/ focal-security main restricted universe multiverse
   
   deb http://archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
   deb-src http://archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
   
   EOF

4. Install the required packages:

   Warning: cloud-init should be installed and network manager should not be used (and systemd-networkd should be enabled), qemu-guest-agent should be installed. This is required for SolusIO to configure ssh and network during rescue boot.

   # apt update -y && apt install -y uuid-runtime && uuidgen > /etc/machine-id && dpkg-divert --local --rename --add /sbin/initctl && ln -s /bin/true /sbin/initctl && apt update -y && apt install -y ssh cloud-init netplan.io vim less nano iputils-ping curl qemu-guest-agent ubuntu-standard casper lupin-casper linux-generic && systemctl enable systemd-networkd

5. Remove autologin for root user and rebuild initramfs:

   # rm -f /usr/share/initramfs-tools/scripts/casper-bottom/25adduser && rm -f /usr/share/initramfs-tools/scripts/casper-bottom/01integrity_check && update-initramfs -u

6. Make sure that VM can be properly rebooted without waiting for any actions:

   # sed -i '+s+read x < /dev/console+echo "\\n" | read x+' /usr/sbin/casper-stop

7. Clean up the environment and exit it:

   # apt upgrade -y && apt-get autoremove -y && truncate -s 0 /etc/machine-id && rm /sbin/initctl && dpkg-divert --rename --remove /sbin/initctl && apt clean && umount /proc && umount /sys && umount /dev/pts && export HISTSIZE=0 && exit

Warning: The below steps are executed on server, not in rescue environment prepared earlier.

1. Remove bash history, unmount previously mounted system directories:

   # rm -f $HOME/rescue-iso/chroot/root/.bash_history && umount $HOME/rescue-iso/chroot/dev && umount $HOME/rescue-iso/chroot/run

2. Create boot layout for ISO:

   # cd $HOME/rescue-iso && mkdir -p image/{casper,isolinux,install} && cp chroot/boot/vmlinuz-**-**-generic image/casper/vmlinuz && cp chroot/boot/initrd.img-**-**-generic image/casper/initrd && touch image/ubuntu

3. Create grub.cfg for ISO:

   # cat <<EOF > image/isolinux/grub.cfg
   
   search --set=root --file /ubuntu
   
   set default="0"
   set timeout="0"
   
   menuentry "Rescue mode iso" {
   linux /casper/vmlinuz boot=casper ---
   initrd /casper/initrd
   }
   EOF

4. Create manifests to comply bootable ISO standards:

   # cd $HOME/rescue-iso && chroot chroot dpkg-query -W --showformat='${Package} ${Version}\n' | sudo tee image/casper/filesystem.manifest && cp -v image/casper/filesystem.manifest image/casper/filesystem.manifest-rescue && sed -i '/ubiquity/d' image/casper/filesystem.manifest-rescue && sed -i '/casper/d' image/casper/filesystem.manifest-rescue && sed -i '/discover/d' image/casper/filesystem.manifest-rescue && sed -i '/laptop-detect/d' image/casper/filesystem.manifest-rescue && sed -i '/os-prober/d' image/casper/filesystem.manifest-rescue

5. Create squashfs layout:

   # mksquashfs chroot image/casper/filesystem.squashfs && printf $(sudo du -sx --block-size=1 chroot | cut -f1) > image/casper/filesystem.size

6. Create README.diskdefines:

   # cat <<EOF > image/README.diskdefines
   #define DISKNAME Rescue mode iso
   #define TYPE binary
   #define TYPEbinary 1
   #define ARCH amd64
   #define ARCHamd64 1
   #define DISKNUM 1
   #define DISKNUM1 1
   #define TOTALNUM 0
   #define TOTALNUM0 1
   EOF

7. Prepare for grub finalization:

   # cd $HOME/rescue-iso/image && grub-mkstandalone \
   --format=x86_64-efi \
   --output=isolinux/bootx64.efi \
   --locales="" \
   --fonts="" \
   "boot/grub/grub.cfg=isolinux/grub.cfg"

8. Create .img and vfat on top of that:

   # (
   cd isolinux && \
   dd if=/dev/zero of=efiboot.img bs=1M count=10 && \
   sudo mkfs.vfat efiboot.img && \
   LC_CTYPE=C mmd -i efiboot.img efi efi/boot && \
   LC_CTYPE=C mcopy -i efiboot.img ./bootx64.efi ::efi/boot/
   )

9. Finalize grub config:

   # grub-mkstandalone \
   --format=i386-pc \
   --output=isolinux/core.img \
   --install-modules="linux16 linux normal iso9660 biosdisk memdisk search tar ls" \
   --modules="linux16 linux normal iso9660 biosdisk search" \
   --locales="" \
   --fonts="" \
   "boot/grub/grub.cfg=isolinux/grub.cfg"
   
   # cat /usr/lib/grub/i386-pc/cdboot.img isolinux/core.img > isolinux/bios.img
   # /bin/bash -c "(find . -type f -print0 | xargs -0 md5sum | grep -v "\./md5sum.txt" > md5sum.txt)"

10. Finally, create iso:

    # sudo xorriso \
    -as mkisofs \
    -iso-level 3 \
    -full-iso9660-filenames \
    -volid "Rescue mode iso" \
    -eltorito-boot boot/grub/bios.img \
    -no-emul-boot \
    -boot-load-size 4 \
    -boot-info-table \
    --eltorito-catalog boot/grub/boot.cat \
    --grub2-boot-info \
    --grub2-mbr /usr/lib/grub/i386-pc/boot_hybrid.img \
    -eltorito-alt-boot \
    -e EFI/efiboot.img \
    -no-emul-boot \
    -append_partition 2 0xef isolinux/efiboot.img \
    -output "../rescue-latest.iso" \
    -graft-points \
    "." \
    /boot/grub/bios.img=isolinux/bios.img \
    /EFI/efiboot.img=isolinux/efiboot.img

11. Find created ISO at $HOME/rescue-iso/

1. Upload newly created image to any file share, e.g. to AWS S3

2. Log into SolusIO and navigate to Settings > Compute Resources

3. Replace Rescue Image URL with a new one