3 - Ubuntu with LUKS: Backup and Restore with Veeam Part 3 - Creating the Veeam Recovery media

As you would like to be able to do a bare metal restore, you now need to make sure that you have the recovery media ready when you need it. I recommend that you create both the generic and the custom media and then store it somewhere off of the server, so you can create a bootable thumb drive when you need it.

Why this does not work from VBR console

If you try to do this from VBR console, like you would with other hosts (here a Windows computer), 

you will see that, if the host ist Ubuntu 22.04, this option is missing.

Creating the generic recovery media

admin01@testlabubuntu01:~$ sudo veeamconfig downloadiso --output /home/admin01

Downloading Veeam Recovery Media...

Veeam Recovery Media has been downloaded successfully.

admin01@testlabubuntu01:~$

But I don't have access to the Ubuntu computer anymore, because it was hacked/the hard disk died/it went up in flames! What can I do?

You can simply download it from Veeam's servers. I chose to download it on the Ubuntu host, as this will allow me create a custom recovery media later.

Creating the custom recovery media

Depending on the hardware you use, you may need to create a custom recovery media which, in my understanding includes hardware drivers particular to your hardware. You will need to test for yourself if the generic media is enough to do a bare metal recovery on your hardware. Either way, I recommend that you have both the generic and the custom recovery media ready when you need it.

Now, there will be a few issues.

admin01@testlabubuntu01:~$ sudo veeamconfig patchiso \

        --efi \

        --input /home/admin01/veeam-recovery-amd64-5.0.0.iso \

        --output /home/admin01/veeam-recovery-amd64-5.0.0-custom.iso

Creating custom Veeam Recovery Media...

Failed to patch the Recovery Media: 'xorriso' not found.

admin01@testlabubuntu01:~$

Firstly, this fails right away, because the xorriso package is missing. Install it with

sudo apt install xorriso

Secondly, if you try this again with xorriso installed, it will still fail.

admin01@testlabubuntu01:~$ sudo veeamconfig patchiso \

        --efi \

        --input /home/admin01/veeam-recovery-amd64-5.0.0.iso \

        --output /home/admin01/veeam-recovery-amd64-5.0.0-custom.iso

Creating custom Veeam Recovery Media...

FATAL ERROR: write_file: failed to create file /tmp/veeam/livecd-{d20e051c-6e0b-4ca8-a269-a9c0f9a7be70}/rootfs/usr/lib/modules/4.19.0-18-amd64/kernel/drivers/hid/hid-roccat.ko, because Too many open files

Exit code: [1]

Failed to execute script command unsquashfs -no-progress -dest /tmp/veeam/livecd-{d20e051c-6e0b-4ca8-a269-a9c0f9a7be70}/rootfs /tmp/veeam/livecd-{d20e051c-6e0b-4ca8-a269-a9c0f9a7be70}/patched_iso/filesystem.squashfs

Failed to unpack squashfs image [/tmp/veeam/livecd-{d20e051c-6e0b-4ca8-a269-a9c0f9a7be70}/patched_iso/filesystem.squashfs] to directory [/tmp/veeam/livecd-{d20e051c-6e0b-4ca8-a269-a9c0f9a7be70}/rootfs].

admin01@testlabubuntu01:~$

The solution to this is to edit the service configuration.

sudo systemctl edit veeamservice.service

Starting at line 3, add

[Service] LimitNOFILE=524288 LimitNOFILESoft=524288

Like so, and save the file.

Now reload systemctl/systemd and the Veeam service. 

sudo systemctl daemon-reload

sudo service veeamservice restart

This time, it should work.

admin01@testlabubuntu01:~$ sudo veeamconfig patchiso \

        --efi \

        --input /home/admin01/veeam-recovery-amd64-5.0.0.iso \

        --output /home/admin01/veeam-recovery-amd64-5.0.0-custom.iso

Creating custom Veeam Recovery Media...

Custom Veeam Recovery Media has been created successfully

You should now have two ISO files. For obvious reasons, do not store these files on the same host that will be restored with the recovery media.

admin01@testlabubuntu01:~$ ls *recovery*

veeam-recovery-amd64-5.0.0-custom.iso  veeam-recovery-amd64-5.0.0.iso

admin01@testlabubuntu01:~$

Note that the custom media does not boot when Secure Boot is enabled.

But I don't have access to the Ubuntu computer anymore, because it was hacked/doesn't boot anymore/was swallowed by a sinkhole. What can I do?

You can test if the generic media works properly on your system. In that case, you don't need the custom media. If you do the custom media, because the generic media does not work with your hardware, you could

• install a fresh Ubuntu installation
• install all the drivers/modules you need
• install Veeam agent for Linux
• create the custom media
• then wipe everything an restore the backup

However, instead I recommend that you proceed with using an Ubuntu live system as your recovery media.

Using a bootable Ubuntu live system as a Veeam recovery media

There are a few options here. You could

• boot up a real live system and create the recovery media as needed, at the time of the restore. This is the option that I will show.
• use a live system on a thumb drive with persistent partition. That way, you can prepare the Ubuntu live recovery media beforehand.
• do a regular Ubuntu installation but on a thumbdrive and use that as recovery media

In this case, I will show how you can turn Ubuntu Desktop 22.10 live into a working Veeam recovery media.

Boot into Ubuntu live.

Enabling SSH connections into the live system

This part is optional, but I like to SSH into it. Also, the Veeam recovery media also allows you to SSH into it, so I would like to have this functionality here too.

You need to set a password for the "ubuntu" user that is present on Ubuntu live systems but given that in a real world scenario, I would be in a rush to the get the data restored and the computer up and running again, I do not want so deal with complex passwords.

Removing complex password requirements

This may be bad advice and it is also optional of course.. On the desktop of the Ubuntu live system, open the terminal app.

sudo nano /etc/pam.d/common-password

In this file, there will be three lines like this:

#password       requisite                       pam_pwquality.so retry=3

#password       [success=2 default=ignore]      pam_unix.so obscure use_authtok try_first_pass yescrypt

#password       sufficient                      pam_sss.so use_authtok

Add '#' to make turn them into comments. Now add a new line below and save the file.

password        [success=1 default=ignore]      pam_unix.so minlen=1 sha512

Now you can change the password for the ubuntu user

passwd ubuntu

Next, install the SSH package. Do not try to upgrade everything with sudo apt upgrade. You may run out of /tmp space.

sudo apt update

sudo apt install ssh

Now you can find out the live system's IP address and connect remotely via SSH.

admin01@testlabubuntu01:~$ ip a | grep 'inet '

    inet 127.0.0.1/8 scope host lo

    inet 192.168.3.135/24 metric 100 brd 192.168.3.255 scope global dynamic eth0

admin01@testlabubuntu01:~$

 In case of my test lab, the IPv4 is 192.168.3.135.

Installing Veeam agent for Linux

This is similar, but not identical to the installation shown in the previous part.

Download the Veeam repository Debian Software Package.

wget https://repository.veeam.com/backup/linux/agent/dpkg/debian/public/pool/veeam/v/veeam-release-deb/veeam-release-deb_1.0.8_amd64.deb

Add the Veeam repository.

ubuntu@ubuntu:~$ sudo dpkg -i ./veeam-release*

sudo apt update

This time, install the agent without veeamsnap module, as you will not create snapshots (or backups for that matter) in the recovery system.

sudo apt update
sudo apt install veeam-nosnap

This will lead to an error because the libfuse2 package is missing.

The following packages have unmet dependencies:

 veeam-nosnap : Depends: libfuse2 but it is not installable

E: Unable to correct problems, you have held broken packages.

To fix this, type

sudo add-apt-repository universe

sudo apt update

Now, it should install without errors.

sudo apt install veeam-nosnap

Optionally, you can now confirm that Veeam agent is installed.

ubuntu@ubuntu:~$ whereis veeam

veeam: /usr/bin/veeam /usr/lib/veeam /etc/veeam /usr/share/veeam /usr/share/man/man8/veeam.8

ubuntu@ubuntu:~$ sudo apt list --installed | grep veeam

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

veeam-nosnap/stable,now 5.0.2.4567 amd64 [installed]

veeam-release-deb/stable,now 1.0.8 amd64 [installed]

ubuntu@ubuntu:~$

Installing the agent does not make the live system a Veeam recovery media. Although Veeam agent for Linux is able to restore volumes, it does not create volumes that do not already exist. Only the recovery UI will create new partitions on an empty disk.

Enabling Veeam recovery UI

Enable recovery UI editing /etc/veeam/veeam.ini.

sudo nano /etc/veeam/veeam.ini

At the end of the file, add:

[recoveryui]

 enableOnLiveSystem = 1

Restart the veeamservice to apply changes.

sudo service veeamservice restart

Confirm that this worked, by starting the Recovery UI.

sudo veeamconfig recoveryui

It should look like this.

Source: KB4183: Veeam Recovery Media for Linux OS, Workarounds

Now you have a generic recovery media, a custom recovery media and you can make an Ubuntu live based recovery media if needed, but before you proceed to bare-metal restore, there are still a few things to do.

In the next part, I will show what needs to be done to be ready for the bare-metal restore.