Recovering Linux Systems with mkcdrec
Kerry Thompson
Sooner or later, every sys admin will have to perform a system recovery. One
of my favorite tools to do this task for Linux systems is a package called mkcdrec
(make CD recovery), which was originally written in November 2000 by Gratien
D'haese and has evolved into a compendium of Linux recovery tools.
There are plenty of utilities to back up your system to a writeable CD, and
Linux distribution CDs usually can be booted into some sort of recovery mode,
but mkcdrec does more. mkcdrec builds a recovery CD that is a fully equipped
workshop containing almost every tool you may need to rebuild a failed system.
Furthermore, mkcdrec can also be used to clone systems -- building identical
copies of a system on a new box. This capability is indispensable if you have
a number of boxes to build with identical configurations -- just the thing for
Web server farms, DNS servers, mail servers, and so on.
Mkcdrec is also ideal to recover systems from intrusions. Because the media
is read-only, it is impervious to alteration by attackers. If you know, or even
suspect, that your Web server has been invaded, you can simply shut down the
system and quickly rebuild it from a media that you know to be clean. Alternatively,
you can take a spare box, build it as a clone of the one attacked, and do a
simple replacement.
Overview
Mkcdrec can be used in a number of ways. The first, and most important, is
the rescue CD-ROM. This is an ISO-format CD image that is simply a minimal bootable
(El Torito format) copy of your Linux system. It contains the Linux kernel,
kernel modules, necessary utilities such as a shell, and useful system recovery
tools, such as fdisk, mkfs, debugfs, and others. It also includes network support
and network client programs such as telnet, ftp, rsh, etc. The essential configuration
details of your system are also included -- obvious configuration files like
/etc/hosts and /etc/services -- as well as less obvious things
like your filesystem layout and disk partitioning information. Besides the rescue
CD and its utilities, mkcdrec can also save all of your files in a compressed
tar archive and add them to your recovery CD.
You don't have to write all of the backup information strictly onto a CD;
you also have the option of writing the backup information into another directory
-- such as an NFS shared filesystem -- or onto a local or even remotely served
magnetic tape drive.
Installation and Configuration
To begin, download the mkcdrec package from http://mkcdrec.ota.be.
(The current version at the time of writing is v0.6.1.) Simply unzip this into
the directory of your choice. I use /opt for my tools:
# cd /opt
# wget http://mkcdrec.ota.be/projects/mkCDrec_v0.6.1.tar.gz
# gzip -cd <mkCDrec_v0.6.1.tar.gz | tar xvf -
The files will be extracted into a directory called mkcdrec. Because I
always have multiple versions of these tools, I like to keep each version separate,
so I'll rename it:
# mv mkcdrec mkcdrec-0.6.1
# cd mkcdrec-0.6.1
Some options may need to be set in your kernel configuration; these are described
in the mkcdrec Introduction and Installation pages of the documentation. You may
need to configure these options in your Linux kernel to have mkcdrec working properly.
In particular, you will need the following set:
CONFIG_BLK_DEV_IDECD=y ( or m )
CONFIG_BLK_DEV_LOOP=y ( or m )
CONFIG_ISO9660_FS=y ( or m )
CONFIG_JOLIET=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=4096 ( or 8192 )
CONFIG_BLK_DEV_INITRD=y
CONFIG_MSDOS_FS=y
Additionally, if you have a CD writer on your Linux system, you will also need:
CONFIG_SCSI=y ( or m )
CONFIG_BLK_DEV_SR=y ( or m )
Next, check that everything is in place by running make test, which should
produce a display similar to the following:
# make test
/opt/mkcdrec-0.6.1/scripts/test.sh
make test output of mkCDrec v0.6.1
Test 1: Are we root? Passed
Test 2: missing executables needed by mkCDrec
dd: Found
mount: Found
bc: Found
umount: Found
gcc: Found
ldd: Found
mformat: Found
mkisofs: Found
cdrecord: Found
nasm: Found
mt: Not found
mt: needed with tape back-up!
ash: Found
bzip2: Found
gzip: Found
rsh: Found
ssh: Found
genromfs: Found
file: Found
openssl: Found
Test 3: Filesystem for Initial ramdisk allowed? Passed
Test 4: loopback device works? Passed
Test 5: ram device available Passed
Test 6: romfs supported by the kernel? N/A
Test 7: cramfs supported by the kernel? N/A
Test 8: strip (from binutils) available? Passed
Test 9: BOOT_FLOPPY_DENSITY=ED ok? Passed
Test 10: cdrecord -scanbus N/A
Test 11: Header files present? Passed
Test 12: DEVFS supported by kernel? N/A
Test 13: filesystem tools present?
ext2: Passed
vfat: Passed
Test 14: initrd must be compiled in kernel! Passed
Test 15: Amount of memory available 129 Mb
Test 16: scripts/Config.sh a link? Passed
#
Note that the make test didn't find the mt command on my system.
I don't have any magnetic tape devices, so that's not a problem for this example.
If you note missing items, then the Introduction section of the mkcdrec manual
will assist you in finding them.
After running make test, you'll need to set a number of configuration
parameters in the Config.sh file. I'll cover the most important ones
here:
RAMDISK_SIZE=32 -- Set this to be the size of the ramdisk that will
hold the running Linux recovery system. The default is 24 Mb; set it higher
if you have plenty of memory.
ISOFS_DIR=/var/tmp/backup -- The target Linux system will be assembled
here before being converted into an ISO filesystem. This should be an empty
directory, because mkcdrec will delete any existing files in this directory.
The directory should have at least 32 Mb of free space.
CDREC_ISO_DIR=/var/tmp/iso -- This is where the ISO CD image will be
created. If you are creating a full backup CD, then you'll need a good 700 Mb
of space here; 20 Mb should be plenty for a recovery-only CD. It's a good idea
to create this directory before running make.
LINUX_KERNEL="/boot/vmlinuz" -- This is the name of your current running
kernel, which will be the kernel booted by the recover CD. If this is left blank,
mkcdrec will try to find the current kernel by looking in /etc/lilo.conf;
if you don't use LILO, then you should specify this parameter explicitly or
you will see strange error messages.
BURNCDR=n -- This is an option that defines whether mkcdrec will automatically
burn the ISO image to CD (when BURNCDR=y), or just leave the image in the directory
on disk for you to do manually. If your CD writer is on another system, set
BURNCDR=n so you can copy it to the system where your CD writer resides.
Most CD-writing software supports the ISO CD image format.
CDRECORD=cdrecord
SCSIDEVICE="0,1,0"
WRITERSPEED="2" -- These are parameters for the CD writer device on
your local Linux system. Setting up a CD writer on your Linux system is beyond
the scope of this article, but the only way to get an IDE CD writer working
seems to be through a SCSI emulation kernel module, hence the SCSIDEVICE
parameter used to specify the CD writer. My CD writer is a little old and is
only capable of 2x write speed; yours may be faster.
EXCLUDE_LIST="/tmp" -- A list of directories to be excluded from the
full backup. It's advisable to put temporary directories into this list; also
add those directories that you specified as the mkcdrec "backup" and "iso" directories.
Most of the other parameters in Config.sh are secondary. They include
lists of kernel modules, utility programs, and manual pages that get written
onto the CD image. If you have extra utilities that you want to add to the recovery
image, you can easily add them into the Config.sh file.
After you have modified Config.sh, run make test again to be
sure that everything is still okay. With everything configured and ready to
go, you're ready to make a recovery CD of your system.
Making a Recovery CD
To begin the process of making a recovery CD, just run make while in
the mkcdrec installation directory. mkcdrec will present you with a menu requesting
your choice of recovery CD. See Figure 1.
There are four types of recovery image that you can make:
1. A basic recovery-only CD that will boot to give you a mini-Linux system
with enough tools and features to correct most system booting problems.
2. A recovery CD that includes the above, but also contains compressed backups
of your filesystems. Although this is very useful to have, it takes some time
to create if you have a lot of disk space in use. This option will create a
set of multiple CDs if one isn't big enough to hold all of the data.
3. A recovery CD image and a backup of your filesystems to another directory,
such as an NFS directory on another system. This is useful because the CD-bootable
system has NFS support. Thus, by writing the backups to an NFS server, you will
be saving valuable CD space and writing time.
4. A recovery CD image and a backup of your filesystems to a magnetic tape
drive (if you have one).
It's recommended that you start by making a basic recovery CD image (option
1), which is built quickly and is easy to test. Once you've tested the recovery-only
CD, try making a full backup (option 2). After you have created your recovery
CD, try to boot from it. If you have problems booting (which could be caused
by a multitude of problems) the mkcdrec FAQ and "Installation and Getting Started
Guide" is a good place for advice. These documents are included in the distribution.
Performing a Recovery
To perform a recovery of your Linux system, simply insert the CD into the
drive and boot from it. Most modern PCs will do this, although in some cases
you will need to change the boot order in the BIOS if you want it to try booting
from the CD before booting from the hard disk.
Mkcdrec boots quickly, loading a mini-kernel into an initial RAMDISK, then
it presents you with a rather colorful splash screen and a boot: prompt.
See Figure 2. Just hit <Return> at the boot
prompt and the Linux system will begin booting, which usually takes about a
minute to complete. When done, you should see the root shell prompt, which looks
like:
I have no name!@host:/ #
The reason you see I have no name! is because the /etc/password
file is missing or corrupted (possibly a bug), but it's nothing to worry about.
At this stage, using df or mount will show you that only the
root ramdisk and CD filesystem are mounted -- the recovery boot does not mount
or touch the hard drive at all. You can now use the common utilities such as
fdisk, fsck, debugfs, etc. to work on recovering your failed system. Note that
the manual pages for these utilities are also installed, so you don't have to
hunt for them.
If you look into the /etc/recovery/ directory, you will see a bunch
of configuration files and scripts. Here is a complete record of how your system
was set up. Two scripts in particular here will help you recover, rebuild, or
clone your system. start-restore.sh will completely restore your system
from the information on the CD. It will partition the disks, make filesystems,
and read back the compressed archive of your files. clone-disk.sh will
also do a complete restore but is aimed at restoring onto a system that is not
quite the same as the original, such as a system with a different number of
disk drives, and so on. clone-disk.sh is more flexible and gives you
more options about what to restore.
Using these tools is straightforward, and if you're confident using tools
like fdisk, mkfs, and mount, then you should have no trouble performing a recovery
or building a new system based on the contents of the mkcdrec recovery CD.
Caveats
Although mkcdrec is a wonderful tool for the rapid recovery and cloning of
a Linux system, there are some things to watch for. You may not be able to boot
the recovery CD on a system that has a different architecture from the one that
was used to build it. I discovered this when building a recovery CD on a Celeron
system and finding it would not boot on an old Pentium 200/MMX. I think the
Linux kernel on the Celeron box was compiled for that CPU and would not run
on an older architecture. Be aware of this, and make sure you test your recovery
methods before you need to rely on them.
mkcdrec runs pretty slowly when making a full backup of a large system because
of the way that it makes a gzipped tar of all of your files, which then gets
placed into the ISO image to be burned onto your CD. Most CD-writing software
has quicker ways to do this. Some modules -- particularly the ext2 filesystem
support and the initrd module -- must be compiled into your kernel and not implemented
as loadable modules. The mkcdrec documentation covers all you need to know,
and most Linux systems work fine with no kernel rebuilding required.
Conclusion
mkcdrec can be a real lifesaver. It will take a snapshot of your Linux system
and write it onto a CD to help you recover from the inevitable. This is an ideal
way to back up and recover systems that don't change much, such as firewalls,
DNS servers, email gateways, and so on. Because of its simple structure, and
the fact that it's based on straightforward scripts, it is easy to customize
and use.
References
mkcdrec home page -- http://mkcdrec.ota.be
mkcdrec page at SourceForge -- http://sourceforge.net/projects/mkcdrec
Kerry Thompson is an IT Security Consultant with CISSP certification and
more than 12 years of experience in UNIX systems administration. He lives on
a small plot of land outside of Auckland, New Zealand, and is often found either
performing science experiments on his wife's computer or trying to round up
the sheep. He can be contacted at: kerry@crypt.gen.nz.