Logrotate -- A Backup Solution
Sean D. Conway
After a system failure, tracking down the
media, determining the last known good backup, and starting the recovery process
can be tricky. I implemented the solution described in this article to reduce
the stress of performing backups. Using a program found in Linux called logrotate,
I created a backup solution to consolidate and automate the backup process.
Logrotate's configuration file contains the details to create the backups, compress
the backup files, provide automatic rotation, and remove stale files.
Logrotate is a utility designed to handle the management of log files. Logrotate
maintains a series of log files, automatically rotates the files, and removes
the files based on the parameters provided in a configuration file. You can
specify the name of any file in the logrotate configuration file, it is treated
the same as a log file.
I used logrotate when I worked with Red Hat (RH) 6.0. Since then, I have found
it to be part of a standard RH install. The current version is available from:
http://www.rpmfind.net/linux/RPM/
The program can be installed with an rpm -Uvh logrotate-3.6.4-1.i386.rpm
(current RH 7.3 version) command. Following are a few points of interest in the
logrotate tour:
/usr/sbin/logrotate -- Utility residence
/etc/logrotate.conf -- Default configuration file
/etc/logrotate.d -- Default directory for other configuration files
/var/lib/logrotate.status -- Default state file
On RH, logrotate is run from a daily job directory /etc/cron.daily/logrotate.
When logrotate is called, it obtains information about which files it is processing
by using the default configuration file /etc/logrotate.conf. Alternate configuration
files can be used if optioned. The configuration file defines the file to rotate
and the parameters to apply. A configuration file entry may look like this:
#daily
"/var/log/logfile.log" {
daily
rotate 7
compress
}
Using the configuration shown here, the file logfile.log would be compressed and
rotated and given the name of logfile.log.1.gz. After day two, the file logfile.log.1.gz
would be rotated to logfile.log.2.gz and new copy of logfile.log.1.gz produced
from logfile.log. At the end of the configuration file schedule, the directory
would contain seven copies of the database. On the eighth day, the copy with the
extension logfile.log.7.gz would be replaced with another rotated copy. A long
listing (ls -al) in the directory would display seven copies of the database
files, with dates preserved from when they were first created.
The man pages for logrotate explain the parameters used to control logrotate.
The remainder of this article describes how logrotate can be used to produce
a backup schema.
When planning the backup process, I used a variation on the grandfather-father-son
backup tape rotation method. The backup set consists of grandfather(monthly),
father(weekly), and son(daily) as shown in Figure 1.
Under this grandfather-father-son implementation method, a full backup is
performed every day (Monday-Sunday); a full backup of the oldest daily is preserved
every Sunday; a full backup of the oldest weekly is preserved the last day of
a four-week cycle. This assumes you start the daily cycle on a Monday. After
1 year, 6 dailies, 3 weeklies, and 12 monthly copies are preserved.
Logrotate works similarly to a stack in programming. As each file is rotated,
it is pushed onto the stack and forces other files down the stack. The size
of the stack is determined by the rotate value in the logrotate configuration
file. Once the stack capacity is reached, the first file in gets pushed out.
Figure 1 illustrates the idea.
To produce the backup schema shown in Figure 1,
create the configuration file /etc/logrotate_backup.conf from the following
listing:
##############################
#
#Name:logrotate_backup.conf
#
#Description:script designed to produce backups
#grandfather(monthly),father(weekly),son(daily)
#
#Author: Sean D. Conway
#
#Wed Aug 28 18:12:59 CDT 2002
#
#Revision:
#
#NOTE:
#replace the path to be backed up.
#don't remove "" good for files that have spaces.
#maintain the "." file extensions this will preserve the frequency
################################
#daily (son)
"/home/oracle/backup/database" {
daily
rotate 7
compress
}
#weekly (father)
"/home/oracle/backup/database.7.gz" {
weekly
rotate 4
}
#monthly (grandfather)
"/home/oracle/backup/database.7.gz.4" {
monthly
rotate 12
}
This configuration file is divided into three sections: daily, weekly, and monthly.
The following listing provides some comments to explain the parameter lines found
in the script:
#daily (son) <--comments line
"/home/oracle/backup/database" { <--file to rotate
daily <--rotation schedule
rotate 7 <--rotation interval
compress <--compress the file
}
The parameters here enable the logrotate program to rotate and compress daily
a file called database. The rotation interval is seven. At the end of one week,
the storage directory will contain the following files: database.1.gz, database.2.gz,
database.3.gz, database.4.gz, database.5.gz, and database.6.gz. The weekly configuration
would rotate the file database.7.gz to database.7.gz.1. The file database.1.gz
would be the most current.
The logrotate utility can access the configuration files using two methods:
1. The configuration file can be placed in the default configuration file
directory /etc/logrotate.d. This directory is listed in logrotates default configuration
file /etc/logrotate.conf. All files in /etc/logrotate.d are additional configuration
files that will be processed by the program.
2. An alternate method is to force the logrotate program to use an alternate
configuration file from a command line. The following is the cron entry to support
logrotate running each day at 07:00 using a configuration file called /etc/logrotate_backup.conf.
00 07 * * * /usr/sbin/logrotate -f /etc/logrotate_backup.conf
I prefer the second method because it allows more flexible scheduling.
The logrotate backup mechanism is used to support three database servers as
shown in Figure 2. Each database server produces
a backup file and sends it to a server called a tape_gate. On the tape_gate
server, logrotate runs daily and takes care of producing the rotation schema.
The complete backup series is exported to tape for offsite storage.
Tape_gate also runs a licensed tape client for accessing the tape silo. This
configuration allows the administrators to use the client software to schedule
and access the backups. The administrator can use the backups stored locally
or pull the file back from the tape storage device. All the file creation dates
are preserved, and a simple grep on a month name will produce a list of backup
files available for that month.
To get support for this backup solution, I had to address some design limitations.
This implementation requires a server to process the backups. The horsepower
required for the server is minimal. For this installation, I used a reclaimed
Pentium desktop with an additional cost of two hard drives. It does provide
an alternate point for accessing backups. I find it easier to access a backup
from a disk rather than searching a tape library.
Backup files are moved from the source to tape_gate using secure ftp. At selective
network slow periods the additional network traffic is acceptable. Secure ftp
provides access to a ready backup from tape_gate if needed.
Table 1 shows the complete set of filenames after
a one-year cycle. The drive capacity must be sufficient to support the number
of system backups being handled. The cost for hard drive capacity today is lower
than it was ten years ago. Sixty-gig IDE drives are common. At 14-MB compressed,
I was able to get all the backups on a 60-GB drive. The three-database server
backups total 500 MB compressed.
The feature that sold this design was cost. A client license for accessing
the tape silo costs $1200. Each of the databases runs a different operating
system, so each would have needed a different client license. Instead, two drives
cost $300 and $1200 for one client license for tape_gate; I saved $2100. The
alternative was to create NFS mount on all the servers.
I have been asked to add additional backups to the server. It seems this common
approach to backups for a variety of systems can make life easier for administrators.
Certainly, one common backup mechanism for all systems is easier to remember
than a different backup method for each system.
Sean D. Conway is a former college instructor turned Network System Specialist
for a regional telecommunications company in Canada. Working with administrators,
they are responsible for the care and feeding of operational support systems
inside the telecommunication cloud. He still dabbles in network theory lectures.
His formal electronic engineering technology training and education background
are valuable tools in achieving a goal -- making learning about computers and
networks easier.