Linux Bare Metal Disaster Recovery on USB
LINUX UPGRADES, UPDATES AND ENHANCEMENTS
RELAX-AND-RECOVER (ReaR) LINUX BARE METAL DISASTER RECOVERY
s an added layer to successful backup and recovery planning, USBs play a meaningul role in my strategy. So does Relax-and-Recover (ReaR), a modular and extensible (bash, and open source) Linux bare-metal disaster recovery solution that sits near the center of the most useful tools I deploy as "stock & standard". The main purpose of ReaR is to create bootable images from what is currently installed on a Linux host that can partition disks and retrieve a backup of the system. Options exist for where to save the bootable image and what to do with it after it has been created (the bootable image can be a USB device, an ISO file or a number of other options).
Providing more depth, ReaR (Relax and Recover) is a tool-set aimed at sysadmins for creating disaster recovery images based on the original distribution installed with its original tools. The recovery process remains compatible with the original system and applications, and hardware support is guaranteed. The disaster recovery information can either be stored via the network or locally on hard disks, USB devices, DVD/CD-R, tape or similar.
This is easily one of the most useful tools in my entire kit, but let me first provide a little clarity as to ReaR's core functionality. In using ReaR and recommending it to colleagues and clients, a misconception I often encounter is the expectation that ReaR is a backup tool - it can perform that function, but that is not a primary and is dependent upon your configuration. ReaR focuses on Disaster Recovery, not backups, by instead providing tight integration with common backup softwares (including, but not limited to: internal backup methods like tar and rsync, in addition to a fantastic number of 3rd party backup solutions like CommVault Galaxy, Bareos, Bacula, Duplicity, EMC Networker (Legato), Symantec Netbackup, HP DataProtector and IBM's Tivoli Storage Manager (TSM)).
Just as important as its integration and interoperability, ReaR is fantastically easy to install and use immediately within your Disaster Recovery strategy. In my opinion, ReaR provides such a wealth of applicational use across my portfolio of backup and restore tools/applicatons, coupled with its simplicity in ease-of-use, that it would be a shame for you not to use it in applying your own strategy. Basically, ReaR is a centrifugal force in my Disaster Recovery strategy, allowing me to deploy a range of options to mitigate risk against permanent data loss and ability to quickly recover from hardware failures.
KEEPING YOUR REAR FRONT & CENTER
As you might suspect, you can find more information in the man page (man rear), but since I fear most readers will not peruse (or follow the convenient, direct links I typically provide) I am literally going to enlist verbatim from the ReaR website here. I think it's worth repeating, word for word, and belaboring the point as to what ReaR brings to the table. How's that for gonging you over the head to get you to use this tool?! To follow is a healthy enlistment of features that show why I keep ReaR front and center within my overall disaster recovery tools, and you should too:
Bare metal recovery on dissimilar hardware
support for physical-to-virtual (P2V), virtual-to-physical (V2P)
support for physical-to-physical (P2P) and virtual-to-virtual (V2V)
various virtualization technologies supported (KVM, Xen, Vmware)
Support for various integrated boot media types, including:
Support for various transport methods, including:
Extensive disk layout implementation, including:
HWRAID (HP SmartArray)
LUKS (encrypted partitions and filesystems)
Two phase disk layout recovery, allows for reconfiguration before recovery, e.g.
migrations from e.g. SWRAID to HWRAID, or unencrypted to encrypted partitions
migration from partitions to LVM
Various techniques to help troubleshooting:
structured log files and guided menus
log files are moved to recovery image, and to recovered system (available in every step for debugging)
advanced debugging options to help trace scripts or develop new functionality
Integration with monitoring (e.g., Nagios/Opsview)
Integration with schedulers (e.g., let cron recreate and transfer your images upon disk layout changes)
Various best practices to assist recovery
integrates with local bootloader (in case it is still possible, you can restore from local disk through Grub)
automatic network and ssh configuration (for remote recovery)
automatic serial console support (useful for recovery through iLO or KVM serial console)
shell history-stuffing (stuff shell history with useful commands at every step)
automatic recovery when possible, guided recovery when needed
There. Are these enough reason to be keeping ReaR front and centerin your arsenal for Disaster Recovery?Yes, yes they are.
The Dreaded Disaster Recovery Scenario
Backup tools are great at backups. Restoring? Not so much. All to familiar is this type of dreaded discovery recovery scenario: a file system corruption has rendered a server or individual workstation caput. Not to worry, because you take scheduled backups of your system. But what's next for you here? A way is needed in getting that backup data back onto disk. Anyone that has been down this path knows and can relate (and, if you by chance haven't you will - everyone eventually gets their turn here!) - most backups from traditional backup applications and mechanisms simply do not restore full systems, but instead rebuild them if something goes horribly wrong with the operating system. While there is some value in this approach, it unfortunately and unrealistically relies on you knowing what the state of the system was. Par example:
- What configuration changes have been made since the installation?
- What additional software has been installed?
- What scripts have been put in place?
That list could actually expand and go on for quite a bit. While possible that you are such a diligent and highly proactive sysadmin you are able to manage all that, this is getting into that forbidden territory for me of spending way too much time with something when there exists a more secure and efficient way of going about it. This is a problem that won’t solve itself; if you haven't manually maintained the details of all the customizations then wouldn’t it be really nice to be able to run a command to pull back the contents of all your local file systems as they were at the point of the last backup, allowing you to simply reboot and be back in business? Keeping ReaR front and center in your Disaster Recovery will do that for you!
BOOTING FROM THE REAR ON USB
The wow-factor for me in use of ReaR really comes in exploring all the flexibility it affords within an overall Disaster Recovery strategy, complementing my chosen backup solutions and output media (e.g., fitting in perfectly with my favored use of Duplicity for backups and iPXE network booting with ISO images, or using CommVault, or using Bareos, or using tar and USBs or... the important piece here is the "or"). However, our singular focus here is on bootable USBs (a more explorative article on ReaR within network booting scenarios is absolutely worth it, and forthcoming). If you create a bootable image on a USB device then wouldn't it be nice to create a backup of your system on the same device? Yes, ReaR does that, too. Here's how I've used ReaR to create both backup of my full system and bootable restore image all on the same media for Ubuntu and RHEL.
REAR BOOTABLE USB IN UBUNTU
ReaR depends upon syslinux, but I highly recommend the modernized extlinux boot loaders. Install via Software Center.
The current, time-of-this-writing package for ReaR is rear_1.16.1_all.deb. Download for your distribution and install using the Gdebi Package Installer (I typically prefer this for .deb installs over Software Center as, imho, Gdebi resolves dependencies much better).
Firstly, we need to Format our USB using ReaR. This will rename the USB to REAR-000. And yes, you acknowledge that this will destroy all data currently on the USB :)
sudo /usr/sbin/rear format /dev/sdc1
With a properly formatted USB, we are now ready to create a bootable restore with a backup of the current system, applications and settings/preferences all saved onto to the USB.
Our parameters are setup using /etc/rear/local.conf; this file by default is empty of commands so I'm not going to back it up prior to edits. Using my handy SumoSudo custom launcher I open gEdit with elevated privileges and set params as:
You can find here a list of scenarios and output parameters we could use for ReaR.
To execute ReaR (with verbose output) to create a bootable USB restore from backup saved to our USB, based on the above params we set in local.conf:
sudo /usr/sbin/rear -v mkbackup
You can find here a list of scenarios and output parameters we could use for ReaR.
What now? With your new ReaR USB we can boot the system from rescue media, restore the disk layout (create partitions, RAID configuration and LVM; create and configure file systems), restore the backup data, restore the boot loader, inspect and reboot!
ALWAYS TEST the USB to make sure you can boot and recover!!!
REAR BOOTABLE USB IN RHEL
Let's do the same then for our Red Hat Enterprise Linux system(s). Our trusty EPEL Repository provides us with safe accessibiltiy to the ReaR rpms, but as with Ubuntu we first need to ensure we are using the more modern extlinux:
From and through PackageKit, we then install ReaR (Relax-and-Recover):
Also just as we did within Ubuntu, let's now create a bootable backup of our RHEL system to USB. Format your USB as we did above as REAR-000, and set the correct params in local.conf.
From Terminal, kick off a backup onto USB:
Here's a sample of the ReaR Workflow output based on our input paramaters above:
Do I really need to repeat and reiterate to Test the USB to ensure you can boot and recover? Throw up a test VM and boot and recover to make sure this is working, before you have to use this for real and find out something is amiss!
Relax-and-Recover (ReaR) Demo Video
♠ If you found the above useful, you might also find to be of interest:
KICKING DISASTER RECOVERY IN THE REAR
There are tools I love and then there are tools I really, really love. ReaR is one of the latter. And what I love about ReaR is that it fits my Disaster Recovery strategy almost no matter what approach and combination I'm taking: from backup mechanism, to backup location, to starting the rescue image, ReaR not only fits into, but complements almost any Disaster Recovery plan and tools I use to execute it, and let's face it - any good DR plan is one that is constantly being assessed and evolving to best-fit your environment and requirements. What is really needed here is a more in-depth breakout of what ReaR brings to a full DR implementation (to sate, here is a sample DR scenario); but for now, this is how to create a bootable USB with ReaR for bare metal disaster recovery.