I am a danish programmer living in Bangkok.
Read more about me @ rasmus.rummel.dk.
Webmodelling Home > Bacula backup tutorial
Do good

Bacula backup tutorial - install, backup & restore

Updated 23 Apr 2013. This bacula backup tutorial is a step by step guide how to install Bacula, backup & restore Ubuntu and Windows using Bacula - the prime open source and free industrial grade backup solution.

Then you finish this bacula tutorial, you will be able to use Bacula to backup & restore multiple servers and multiple operation systems using one or multiple backup servers - Bacula easily scales to many hundreds of computers in a network.

Index :

Appendixes :

Relevant Links :



Bacula main components

Bacula consist of 5 main parts :

  • Director daemon : Controls backup & restore operations. In /etc/bacula/bacula-dir.conf you configure Jobs, FileSets, Schedules etc.
  • Bacula Console : A command line program that allows you to interact with the Director daemon, eg. then you need to run a Restore job.
  • Storage daemon : The Director hands off the responsibility for writing and reading the physical backup media to the Storage daemon - in our case the physical backup media are files (not tapes) on a harddrive, while the harddrive itself (not a tape drive) is our backup device.
  • File daemon : The File daemon is a program that must be installed on the machine to backup. The Directory communicates with the machines to backup through the File daemons. The File daemon is responsible for the file system dependent parts of the backup and restore processes, therefore the File daemon is specific to the OS - currently there is a Unix/Linux File daemon, a Windows File daemon and a Mac OSX daemon.
  • Catalog : A database that keeps records of all volumes, all jobs and all files in a way that allows the administrator to easily & quickly locate files for restore across full, differential & incremental backups of multiple servers - the Catalog is what sets Bacula apart from your own homegrown tar based backup system no matter how good you are.


Server schema

In this tutorial I am going to backup several servers, the following is the full server schema :

FQDN IP OS Bacula daemons installed Comments
backup.favouritehosting.com 27.254.33.96 Ubuntu Director, Catalog, Storage, Console All backup data are stored here and all the information about the backup data are also stored here and both Director & Console are installed here as well - this is our backup server.
web1.favouritehosting.com 27.254.33.57 Ubuntu Linux File Shows how to backup a linux web server.
db1.favouritehosting.com 27.254.33.58 Ubuntu Linux File Shows how to backup a MySQL database server.
win1.favouritehosting.com 27.254.33.65 Windows 2008r2 Windows File Shows how to backup a windows server.

Bacula can only be installed on a Linux OS (here Ubuntu), however Bacula can easily take backup of Windows and Mac OSX machines.



How to Install Bacula

We have 4 servers to install on.

On the backup server we need to have :

  • The bacula director daemon (controlling backup, restore, pools, volumes, jobs, filesets etc).
  • The bacula console program (command line utility we use to communicate with the director, eg. start a restore job or purge a volume).
  • The bacula storage daemon (responsible for reading & writing the backup media, in our case backup media is the files that contains the backups).
  • A MySQL server for the catalog database (Bacula can alternatively use postgresql or sqllite) - I will not go through how to install & configure MySQL as if you install Bacula on an empty system, MySQL will be automatically installed for you. If you already have MySQL server installed Bacula will prompt you if you want to let Bacula setup the database for you.
  • A mail sending program for sending backup & restore reports to your email account - Bacula comes with a small mail sender called bsmtp, which I will use in this tutorial (so no need to install & configure an MTA).

Let's start the installation :

  1. Install the backup server :
    1. Log on to the backup server either on a local or remote terminal. Here I use Putty on a Windows dev box to log onto the backup server.
    2. backup server shell> apt-get update : update your package information, so you will install the newest version of any package.
    3. Simulate an install of bacula to see what is going to happen :
      • First on an empty system :
        • backup server shell> apt-get -s install bacula : MySQL and email capability are automatically installed and will also be configured to some extent.
      • Second on a system already having MySQL and Postfix : (this is the actual system I am going to install on)
        • backup server shell> apt-get -s install bacula : MySQL and email capability are already available and it seems we need to configure the whole thing ourselves.
    4. backup server shell> apt-get install bacula : start the installation
    5. You are prompted whether you want to create the a database, choose Yes.
    6. You are prompted for the MySQL server root password.
    7. You can supply a password for the bacula-director-mysql, I just leave it blank to get an autogenerated password.
    8. backup server shell> dpkg -l | grep bacula : list the installed packages.
    9. DONE - that concludes the package installation on the backup server.
  2. Install a Linux File Daemon on the web server :
    1. Logon to the web server either on a local or remote terminal. Again I use Putty on a Windows dev box to log onto the web server.
    2. web1 shell> apt-get -s install bacula-fd : simulate (-s) an install of the bacula-fd (bacula file daemon) package to see what's going to happen - 2 packages are going to be installed & configured.
    3. web1 shell> apt-get install bacula-fd : start the actual installation
    4. web1 shell> dpkg -l | grep bacula : confirm the 2 packages are installed.
    5. DONE - bacula file daemon installed on backup target server.
  3. Install a Linux File Daemon on the database server :
    1. Do exactly the same as for the web server.
  4. Install a Windows File Daemon on the windows server :
    1. Log on to a windows server that you want to backup either locally or remote using eg. Remote Desktop.
    2. From SourceForge.net download the bacula windows file daemon that match the version on your backup server (in my case it is 5.0.3).
    3. On your windows server, double click the downloaded installation file to start the bacula file daemon installer.
    4. Press the "Next" button.
    5. Press the "I Agree" button.
    6. Choose "Automatic" installation type and then press the "Next" button.
    7. I choose only the "Client" component as I see no use for the windows local bacula CLI nor the BAT (I do think the BAT have great potential though). Click the "Next" button.
    8. Write the name that you later will give to the Bacula Director (I choose FH-dir for Favourite Hosting Director). Press the "Install" button.
    9. If you like, you can choose to write out a template for the Client{..} section to put in your bacula-dir.conf file on your backup server, however it is more fun less helpful. Press the "Next" button.
    10. Ok, you are finish installing the bacula file daemon on your windows server.
    11. Navigate to Bacula main folder "C:\Program Files\Bacula" and look at the files, especially note the bacula-fd.conf file.
    12. DONE - you are finish installing the bacula windows file daemon.

All necessary programs are now installed, it is time to start configuring the system.



How to Configure Bacula

Bacula configuration is done in 4 .conf files, one for each service :

  • bconsole.conf

    : /etc/bacula/bconsole.conf (on backup server) : configuring the Console program
    • Director section : Describing what Director this Console will access (where the Director is and what credentials to use for logon).
  • bacula-dir.conf

    : /etc/bacula/bacula-dir.conf (on backup server) : configuring the Director daemon
    • Director section : Describing this Director daemon.
    • Pool sections : You group multiple backup media (file volumes or tape volumes) into pools, eg. all tapes or files on which you store your full backup may be grouped into one Pool, while all tapes or files on which you store your incremental backup may be grouped into another Pool.
    • Job sections : We will have many backup & restore jobs, each of them is defined in a Job{..} section.
    • Schedule sections : A Job wil refer to one or more Schedule{..} sections defining when the Job will run.
    • Client sections : Describing the backup target machines (where they are and logon credentials). A Job will refer to one Client{..} section defining on which machine the Job will run (in our case we need Client{..} sections for web1, db1 & win1).
    • FileSet sections : A Job will refer to one FileSet{..} section defining what files to backup on the Client or what files to restore.
    • Messages section : Here you can setup notification emails to your email address so you know what have happened, eg. if a backup job succeeded.
    • Catalog section : Describing the Catalog database (where the database is and logon credentials).
  • bacula-sd.conf

    : /etc/bacula/bacula-sd.conf (on backup server) : configuring the Storage daemon
    • Storage section : Describing this Storage daemon.
    • Director section : Authentication details for the Director that is allowed to control this Storage daemon.
    • Device section : Describing the backup device (in our case the backup device is a harddrive, but it could also be a tape drive).
    • Messages section : Setup which messages are sent back to the Director.
  • bacula-fd.conf

    : /etc/bacula/bacula-fd.conf (on client server) : configuring the File daemons - one for each backup target (here web1, db1 & win1)
    • Client section : Describing this File daemon.
    • Director section : Authentication details for the Director that is allowed to control this File daemon.
    • Messages section : Setup which messages are sent back to the Director.


Bacula Configuring Connections

The first thing we want to achieve is to be able to use the Console to connect to the Director and the Director to connect to the Storage & 3 File daemons.

Here is the official connection credentials overview. I have added the tables to show also the connection addresses :

   

Console

:
bacula-dir.conf bconsole.conf
Director{..} Name==Director{..} Name
Director{..} Password==Director{..} Password
Director{..} DirAddress==Director{..} Address


Storage

:
bacula-dir.conf bacula-sd.conf
Director{..} Name==Director{..} Name
Storage{..} Password==Director{..} Password
Storage{..} Address==Storage{..} SDAddress


File Daemon

:
bacula-dir.conf bacula-fd.conf
Director{..} Name==Director{..} Name
Client{..} Password==Director{..} Password
Client{..} Address==FileDaemon{..} FDAddress

Ok, let's implement it :

  1. backup server shell> cd /etc/bacula : navigate to the Bacula main folder.
  2. backup server shell> ls -l : list all files in the Bacula main folder, especially note bacula-dir.conf, bconsole.conf & bacula-sd.conf (ignore the bacula-fd.conf which we will not use since we do not backup anything on the backup server).
  3. bacula-dir.conf : (on backup server)
    1. backup server shell> nano /etc/bacula/bacula-dir.conf : open bacula-dir.conf on the backup server in the nano editor.
      1. Edit the bacula-dir.conf Director{..} section :
        • Director {
        •     Name = FH-dir : I have changed the name to FH-dir for Favourite Hosting Director.
        •     DIRport = 9101 : Director will listen on this port, eg. the Console program must contact Director on this port.
        •     QueryFile = "/etc/bacula/scripts/query.sql"
        •     WorkingDirectory = "/var/lib/bacula"
        •     PidDirectory = "/var/run/bacula"
        •     Maximum Concurrent Jobs = 1
        •     Password = "dir-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" : if you want to logon to the Director, you need to use this password (eg. then using the Console program).
        •     Messages = Daemon
        •     DirAddress = 27.254.33.96 : changed from 127.0.0.1 (this is not strictly necessary, however it is necessary in other places, so I like to use all global IP addresses). Update : it seems that in Bacula version 5.2.5 that bacula-dir.conf Director{} DirAddres MUST be equal to bconsole.conf Director {} address, so if you change to global IP instead of localhost, you should also do so in bconsole.conf.
        • }
      2. Edit the bacula-dir.conf Catalog{..} section : (Bacula version 5.0.3 have an error in the generated bacula-dir.conf file in the Catalog{..} section)
        • Catalog {
        •     Name = MyCatalog
        •     # Uncomment the following line if you want the dbi driver
        •     # dbdriver = "dbi:sqlite3"; dbaddress = 127.0.0.1; dbport =
        •     dbname = bacula; DB Address = ""; dbuser = "bacula"; dbpassword = "catalog-zzzzzzzzzzzz" : bacula version 5.0.3 have quotes around dbname value "bacula", remove the quotes, otherwise the Director will not be able to connect to the database (keep the other quotes around the DB Address, dbuser & dbpassword values).
        • }
      3. Delete or out-comment ALL bacula-dir.conf Client{..} sections. If bacula-dir.conf have a Client{..} section that have invalid references, the Director cannot start.
      4. Create a bacula-dir.conf Client{..} section for web1 :
        • Client {
        •     Name = web1 : this Client{..} section defines the web1 server, so I think web1 is a good name.
        •     Address = 27.254.33.57 : the IP address of the web1 server.
        •     FDPort = 9102 : the port on which the file daemon on web1 will be listening (that is: the port on which this director should contact the file daemon).
        •     Catalog = MyCatalog
        •     Password = "web1-yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" : password specific for web1 and must match the password specified in bacula-fd.conf Director{..} section on the web1 server.
        •     File Retention = 30 days
        •     Job Retention = 6 months
        •     AutoPrune = yes
        • }
      5. Create a bacula-dir.conf Client{..} section for db1 :
        • Client {
        •     Name = db1
        •     Address = 27.254.33.58 : the IP address of the db1 server.
        •     FDPort = 9102
        •     Catalog = MyCatalog
        •     Password = "db1-yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" : password specific for db1 and must match the password specified in bacula-fd.conf Director{..} section on db1.
        •     File Retention = 30 days
        •     Job Retention = 6 months
        •     AutoPrune = yes
        • }
      6. Create a bacula-dir.conf Client{..} section for win1 :
        • Client {
        •     Name = win1
        •     Address = 27.254.33.65 : the IP address of the win1 server.
        •     FDPort = 9102
        •     Catalog = MyCatalog
        •     Password = "win1-yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" : password specific for win1 and must match the password specified in bacula-fd.conf Director{..} section on win1.
        •     File Retention = 30 days
        •     Job Retention = 6 months
        •     AutoPrune = yes
        • }
      7. Edit the bacula-dir.conf Storage{..} section : (there are 4 storage sections, however 3 of them are out-commented and not interesting for us)
        • Storage {
        •     Name = storage1 : I am going to use 2 devices (disks) for backup, so I like the name of this section to represent the disk number of the device this section is going to point to.
        •     # Do not use "localhost" here
        •     Address = 27.254.33.96 : change localhost to the global IP of the storage server (even if the storage server is indeed localhost).
        •     SDPort = 9103 : the port on which to connect the Storage Daemon.
        •     Password = "storage-ssssssssssssssssssssssss" : this password much match the bacula-sd.conf Director{..} section.
        •     Device = disk1 : for each device (harddrive, DVD drive, tape drive) the Storage daemon uses for storage, there will be a Device{..} section in the bacula-sd.conf file. Here we reference by name, disk1, the Device{..} section in bacula-sd.conf that we want to use for selecting storage device (disk1 is here not an actual device but just a name pointing to a Device{..} section in bacula-sd.conf).
        •     Media Type = File : Bacula supports many different media types, eg. DDS-4 tapes, DVDs etc. We are using files to store the backup data, so we specify the Media Type = File.
        • }
      8. Edit the bacula-dir.conf JobDefs{..} section : (we don't want to define it now, but we need to edit some wrong references, otherwise the Director cannot start)
        • JobDefs {
        •     Name = "DefaultJob"
        •     Type = Backup
        •     Level = Incremental
        •     Client = web1 : here we need to reference the name of one of our Client{..} sections, otherwise Director will not start.
        •     FileSet = "Full Set"
        •     Schedule = "WeeklyCycle"
        •     Storage = storage1 : here we need to reference the name of our Storage{..} section, otherwise Director will not start.
        •     Messages = Standard
        •     Pool = File
        •     Priority = 10
        •     Write Bootstrap = "/var/lib/bacula/%c.bsr"
        • }
      9. One of the bacula-dir.conf Job{..} sections contains a Client reference, set the value of that Client to web1 (we don't want to set up the whole Job{..} section now, we just want to avoid an invalid reference to a Client{..} section that does not exist). If bacula-dir.conf have a Job{..} section with invalid references, the Director cannot start.
    2. Press ctrl+x and then y to exit the nano editor and save the bacula-dir.conf file.
    3. backup server shell> service bacula-director restart : restart the Director daemon to apply the new configurations.
  4. bconsole.conf : (on backup server)
    1. backup server shell> nano /etc/bacula/bconsole.conf : open bconsole.conf on the backup server in the nano editor.
      1. Edit the bconsole.conf Director section :
        • Director {
        •     Name = FH-dir : Here you must use the same name you gave the bacula-dir.conf Director{} section (there the Director defines itself).
        •     DIRport = 9101 : The port on which the Console will contact the Director.
        •     address = 27.254.33.96 : the address on which the Director runs (I changed address from localhost which in our case is not strictly necessary, however I like to use all global IP addresses).
        •     Password = "dir-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" : the password we together with the name use to logon to the Director (must match the password in the bacula-dir Director{} section).
        • }
    2. Press ctrl+x and then y to exit the nano editor and save the bconsole.conf file.
  5. bacula-sd.conf : (on backup server)
    1. backup server shell> nano /etc/bacula/bacula-sd.conf : open bacula-sd.conf on the backup server in the nano editor.
      1. Edit the bacula-sd.conf Director section :
        • Director {
        •     Name = FH-dir : the name of the Director that can administer this Storage daemon.
        •     Password = "storage-ssssssssssssssssssssssss" : the password that the Director must use then administer this Storage daemon. This password is also in the bacula-dir.conf Storage{..} section.
        • }
      2. Edit the bacula-sd.conf Storage section :
        • Storage {
        •     Name = FH-sd : I use FH-sd for Favourite Hosting Storage Daemon. This name is also in the bacula-dir.conf Storage{..} section.
        •     SDPort = 9103 : The port on which this Storage daemon will listen (and therefore the port on which the Director must contact this Storage daemon).
        •     WorkingDirectory = "/var/lib/bacula"
        •     Pid Directory = "/var/run/bacula"
        •     Maximum Concurrent Jobs = 20
        •     SDAddress = 27.254.33.96 : change 127.0.0.1 to global IP of the storage server.
        • }
      3. Edit the bacula-sd.conf Device section : (there are many Device{..} sections, however only one is un-commented and that is the one we are interested in)
        • Device {
        •     Name = dis1 : this Device{..} section will specify a disk device to use for storage, since I use multiple disks, I think disk1 is a suitable name. Also this Device{..} section is referenced by bacula-dir.conf Storage{..} section by name, disk1.
        •     Media Type = File : on this device we will use files to store the backup data, therefore the Media Type = File.
        •     Archive Device = /media/disk1 : here you specify the mount point of the actual device (the actual harddrive I want to use is mounted as /media/disk1). You can also use just a folder to store your backup, it does not need to be a fresh disk.
        •     LabelMedia = yes;
        •     Random Access = Yes;
        •     AutomaticMount = yes;
        •     RemovableMedia = no;
        •     AlwaysOpen = no;
        • }
      4. Edit the bacula-sd.conf Messages section :
        • Messages {
        •     Name = Standard
        •     director = FH-dir = all : change to correct Director name, in my case to FH-dir.
        • }
    2. Press ctrl+x and then y to exit the nano editor and save the bacula-sd.conf file.
    3. backup server shell> service bacula-sd restart : restart the Storage daemon to apply the new configurations.
  6. bacula-fd.conf on web1 :
    1. web1 server shell> cd /etc/bacula : navigate to the Bacula main folder on the web1 server.
    2. web1 server shell> ls -l : list all files in the Bacula main folder, especially note the important bacula-fd.conf file.
    3. web1 server shell> nano /etc/bacula/bacula-fd.conf : open bacula-fd.conf on the web1 server in the nano editor.
      1. Edit the bacula-fd.conf Director section : (defining what Director is allowed to control this this file daemon)
        • Director {
        •     Name = FH-dir : the name I gave the Director in the bacula-dir.conf file.
        •     Password = "web1-yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" : this passwords needs to match the web1 Client{..} section in the bacula-dir.conf file.
        • }
      2. Note the bacula-fd.conf FileDaemon section : (identifies this file daemon, so it can be referenced by the Director)
        • FileDaemon {
        •     Name = web1-fd : the default name, web1-fd, is exactly the name I want, so no need to change it.
        •     FDport = 9102 : the port on which this file daemon will listen (so the Director needs to use this port to contact the file daemon).
        •     WorkingDirectory = /var/lib/bacula
        •     Pid Directory = /var/run/bacula
        •     Maximum Concurrent Jobs = 20
        •     FDAddress = 27.254.33.57 : changed from 127.0.0.1 as I like to use all global IP addresses (even though it is not necessary here).
        • }
      3. Edit the bacula-fd.conf Messages section :
        • Messages {
        •     Name = Standard
        •     Director = FH-dir = all, !skipped, !restored : Director name needs to be the same as specified in the first Director{..} section, in my case FH-dir.
        • }
    4. Save the bacula-fd.conf file (either directly if you use eg. nano or upload if you use ftp).
    5. web1 server shell> service bacula-fd restart : restart the file daemon to apply the configuration changes.
  7. bacula-fd.conf on db1 :
    1. db1 server shell> cd /etc/bacula : navigate to the Bacula main folder on the db1 server.
    2. db1 server shell> ls -l : list all files in the Bacula main folder, especially note the important bacula-fd.conf file.
    3. db1 server shell> nano /etc/bacula/bacula-fd.conf : open bacula-fd.conf on the db1 server in the nano editor.
      1. Edit the bacula-fd.conf Director section : (defining what Director is allowed to control this this file daemon)
        • Director {
        •     Name = FH-dir : the name I gave the Director in the bacula-dir.conf file.
        •     Password = "db1-yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" : this passwords needs to match the db1 Client{..} section in the bacula-dir.conf file.
        • }
      2. Note the bacula-fd.conf FileDaemon section : (identifies this file daemon, so it can be referenced by the Director)
        • FileDaemon {
        •     Name = db1-fd : the default name, db1-fd, is exactly the name I want, so no need to change it.
        •     FDport = 9102 : the port on which this file daemon will listen (so the Director needs to use this port to contact the file daemon).
        •     WorkingDirectory = /var/lib/bacula
        •     Pid Directory = /var/run/bacula
        •     Maximum Concurrent Jobs = 20
        •     FDAddress = 27.254.33.58
        • }
      3. Edit the bacula-fd.conf Messages section :
        • Messages {
        •     Name = Standard
        •     Director = FH-dir = all, !skipped, !restored
        • }
    4. Press ctrl+x and then y to exit the nano editor and save the bacula-sd.conf file.
    5. db1 server shell> service bacula-fd restart : restart the file daemon to apply the configuration changes.
  8. bacula-fd.conf on win1 :
    1. Navigate to C:\Program Files\Bacula, the Bacula main folder on the win1 server.
    2. In C:\Program Files\Bacula folder notice the important bacula-fd.conf file.
    3. Open and edit the bacula-fd.conf file using your favourite windows text editor :
      1. Edit the bacula-fd.conf Director section : (defining what Director is allowed to control this this file daemon)
        • Director {
        •     Name = FH-dir : the name I gave the Director in the bacula-dir.conf file.
        •     Password = "win1-yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" : this passwords needs to match the win1 Client{..} section in the bacula-dir.conf file.
        • }
      2. Note the bacula-fd.conf FileDaemon section : (identifies this file daemon, so it can be referenced by the Director)
        • FileDaemon {
        •     Name = win1 : default Bacula installer will set the name to [hostname]-fd, however I here I just use the hostname, win1.
        •     FDport = 9102 : the port on which this file daemon will listen (so the Director needs to use this port to contact the file daemon).
        •     WorkingDirectory = "C:\\Program Files\\Bacula\\working"
        •     Pid Directory = "C:\\Program Files\\Bacula\\working"
        •     # Plugin Directory = "C:\\Program Files\\Bacula\\plugins"
        •     Maximum Concurrent Jobs = 10
        • }
      3. Edit the bacula-fd.conf Messages section :
        • Messages {
        •     Name = Standard
        •     Director = FH-dir = all, !skipped, !restored
        • }
    4. Save the bacula-fd.conf file.
    5. Restart the file daemon service on win1 :
      1. On the win1 server press the Windows Key + R to open the run box, write services.msc and press the "Ok" button to launch the Services Manager.
      2. In the Services Manager right click on "Bacula File Service" and choose "Restart" from the context menu.


Bacula Configuring Connections - Test

  1. backup server shell> bconsole : start the console program and confirm that you can connect to Director.
  2. *> status storage : test if you can connect.
  3. *> status client : prompt for a client to connect to - try selecting web1-fd to see if you can connect.
  4. *> list clients : you should be able to see the 3 clients web1, db1 & win1.
  5. *> quit : quit Bacula Console to return to OS system prompt.
  6. Just for the fun of it, let's confirm the clients using the Catalog database :
    1. backup server shell> mysql -u bacula -pzzzzzzzzzzzz : logon to MySQL server using the bacula credentials specified in the bacula-dir.conf Catalog{..} section (notice there is no space between -p and the password).
    2. mysql> show databases; : lets see what databases the bacula user have access to.
    3. mysql> use bacula; : set default database to bacula (bacula is the name of the Catalog database).
    4. mysql> show tables; : lets see what tables are in the Catalog database.
    5. mysql> select * from Client : and there you get exactly the same 3 clients we saw before using the bacula list clients command.
    6. mysql> quit : quit MySQL CLI prompt to return to OS system prompt.

Congratulations - your Bacula Console, Director and File Daemons should now be able to connect (however there is no backup going on yet).



Bacula Configuring Storage

On my backup server, I have 3 harddrives, here I am going to use 2 of them as bacula backup devices (the 3. harddrive I use for OS, Bacula, ad-hoc backup etc.).

Indeed Bacula does support multiple storage devices, eg. store backup data on multiple harddrives spread over multiple backup servers. Unfortunately Bacula does not natively support to treat multiple harddrives as one harddrive (it can be setup outside of bacula, but it is too cumbersome for my taste, sere more here). Instead in bacula-sd.conf we need to specify a Device{..} section for each harddrive we want to use as a backup device - in our case 2 such Device{..} sections.

Ok, let's configure the storage :

  1. bacula-sd.conf

    1. backup server shell> nano /etc/bacula/bacula-sd.conf : open bacula-sd.conf on the backup server in the nano editor.
      1. Edit the bacula-sd.conf Device{..} section :
        • Device {
        •     Name = disk1 : on this device we will store backup of web1.
        •     Media Type = File
        •     Archive Device = /media/disk1 : mount point of the harddrive (if you use folders instead of harddrives, just specify a folder here).
        •     LabelMedia = yes; : then a new media volume (in our case a file inside which backup data is stored) is created, we want Bacula to automatically give the volume (file) a name.
        •     Random Access = Yes;
        •     AutomaticMount = yes;
        •     RemovableMedia = no;
        •     AlwaysOpen = no; : this directive is ignored for Media Type = File, therefore in our case yes or no is the same.
        • }
      2. Create a second Device{..} section in the bacula-sd.conf file :
        • Device {
        •     Name = disk2 : on this device we will store backup of db1 and win1.
        •     Media Type = File
        •     Archive Device = /media/disk2 : mount point of the harddrive.
        •     LabelMedia = yes;
        •     Random Access = Yes;
        •     AutomaticMount = yes;
        •     RemovableMedia = no;
        •     AlwaysOpen = no;
        • }
    2. Press ctrl+x and then y to exit the nano editor and save the bacula-sd.conf file.
    3. backup server shell> service bacula-sd restart : restart the storage daemon to apply the configuration changes.
  2. bacula-dir.conf

    1. backup server shell> nano /etc/bacula/bacula-dir.conf : open bacula-dir.conf on the backup server in the nano editor.
      1. Edit the bacula-dir.conf Storage{..} section :
        • Storage {
        •     Name = storage1
        •     Address = 27.254.33.96
        •     SDPort = 9103
        •     Password = "storage-ssssssssssssssssssssssss" : must match bacula-sd.conf Director{..} Password.
        •     Device = disk1 : reference to the Device{..} section in bacula-sd.conf that use the harddrive mounted on /media/disk1.
        •     Media Type = File
        • }
      2. Create a new Storage{..} section in the bacula-dir.conf file : (if you have only 1 harddrive, you can still separate your backup jobs by storing the volume files in separate folders if you should wish)
        • Storage {
        •     Name = storage2
        •     Address = 27.254.33.96
        •     SDPort = 9103
        •     Password = "storage-ssssssssssssssssssssssss" : must match bacula-sd.conf Director{..} Password.
        •     Device = disk2 : reference to the Device{..} section in bacula-sd.conf that use the harddrive mounted on /media/disk2.
        •     Media Type = File
        • }
      3. Delete or out-comment all Pool{..} sections in bacula-dir.conf.
      4. Create a new Pool{..} section in bacula-dir.conf : (here we group volumes for disk1, however we could also group volumes for full backup and another group for incremental backup)
        • Pool {
        •     Name = pool1
        •     Pool Type = Backup
        •     Recycle = yes : yes (default) means that purged Volumes will be recycled then Bacula finds no Volume that are appendable. A purged Volume is a Volume that contains only expired Jobs & files and thus is not in the Catalog.
        •     AutoPrune = yes : yes (default) means that Bacula will automatically apply the Volume Retention period then a new Volume is needed (no appendable Volume exist in the Pool). Volume Pruning means to delete expired Jobs (older than Volume Retention period) from the Catalog permitting possible recycling of the Volume.
        •     Volume Retention = 61 days : I don't want to recycle the volumes before last writing to the volume is 2 month old (that way I should be able to go 2 month back in time).
        •     Maximum Volume Bytes = 25G : this is a tricky size, if the file is corrupted, the data cannot be restored so the file size should be small, but on the other hand if you run out of backup space you will need to purge the volumes one at a time, so the volume size should be big to avoid having to manually purging too many volumes.
        •     Maximum Volumes = 19 : the harddrive is 500G, so I can have 20 volumes of 25G (but to avoid problems, I set it to 19)
        •     Label Format = pool1- : volumes are now created automatic and labelled pool1-0000, pool1-0001 and so forth (if you have different pools for your full and incremental backups, you would probably use a label format like Full- and Incr-).
        •     Catalog Files = yes : yes (default) means that file names backed up will be written to the catalog database. If setting the value to "no", then the catalog database will be much smaller, howerver it will not be possible to produce a catalog listing of files backed up for each Job and the console restore command will not work.
        •     Maximum Volume Jobs = 0 : zero (default) means no limit on the number of Jobs that can be written to the same Volume.
        •     Maximum Volume Files = 0 : zero (default) means no limit on the number of files that can be written to the same Volume.
        •     Volume Use Duration = 0 : zero (default) means no limit on the time that files can be written to the same Volume. A positive value defines a limited number of days that a Volume can be written to starting from the first write.
        • }
      5. Create a second new Pool{..} section in bacula-dir.conf : (to group volumes for incremental backup)
        • Pool {
        •     Name = pool2
        •     Pool Type = Backup
        •     Recycle = yes
        •     AutoPrune = yes
        •     Volume Retention = 62 days : Then the last writing to a volume is more than 61 days old, Bacula is allowed to purge the volume and recycle it if needed for storage space.
        •     Maximum Volume Bytes = 25G : Each volume should be 25G.
        •     Maximum Volumes = 19 : In this Pool we will have 19 volumes.
        •     Label Format = pool2- : In this Pool (group of volumes), I want to prefix all volume names with pool2-
        •     Catalog Files = yes
        •     Maximum Volume Jobs = 0
        •     Maximum Volume Files = 0
        •     Volume Use Duration = 0
        • }
    2. Press ctrl+x and then y to exit the nano editor and save the bacula-dir.conf file.
    3. backup server shell> service bacula-director restart : restart the director daemon to apply the configuration changes.

For the storage daemon we have now configured 2 backup devices and for the director we have configured references to these devices as well as 2 pools (collection of volume files), one pool for each backup device.



Bacula Configuring Schedules

At it's most basic level, a schedule defines then to execute what, in our case then to execute a job (typically only backup jobs & verify jobs, not restore jobs, are scheduled). In our case we are going to slightly complicate this very basic schedule for our backup jobs : (setting the backup level)

  1. bacula-dir.conf

    1. backup server shell> nano /etc/bacula/bacula-dir.conf : open bacula-dir.conf on the backup server in the nano editor.
      1. Create a new Schedule{..} section : (it's possible to define many Scedule{..} sections, however in our case we need only one)
        • Schedule {
        •     Name = "MonthlyCycle"
        •     Run = Level=Full on 1 at 2:05 : the 1. of every month at 2:05 I want to take a full backup.
        •     Run = Level=Incremental on 2-31 at 2:05 : all other days of the month, I want to take an incremental backup.
        • }
    2. Press ctrl+x and then y to exit the nano editor and save the bacula-dir.conf file.
    3. backup server shell> service bacula-director restart : restart the Director to apply the new configurations.

Note that in the above Schedule{..} section we are for any job definition using the schedule actually overwriting the backup level property of the job by saying that on the 1. of every month we want the level to be full and all other days we want the level to be incremental - this means that we don't need to specify one job for full backup and another job for incremental backup with each their schedule, thus saving us one Job{..} section for each client.

While the above schedule is all we need in our setup, the Schedule section is actually a much more powerful construct with the ability to overwrite also other job properties like Level, Pool, Storage & Messages. This fleksibility of the Schedule{..} section may often greatly reduce the amount of jobs that we need to define.

Say that on a single backup device, you wanted to keep full backup in one pool and incremental backup in another pool, you could then have the following schedule : (overwriting the pool to use depending on level).

  • Schedule {
  •     Name = "MonthlyCycle"
  •     Run = Level=Full Pool=MyFullPool on 1 at 2:05 : taking full backup using the MyFullPool for storage media.
  •     Run = Level=Incremental FullPool=MyFullPool IncrementalPool=MyIncrPool on 2-31 at 2:05 : for incremental backup use MyIncrPool, however if an incremental backup is upgraded to full (no prior full backup exists), then use MyFullPool.
  • }

Note that the Pool override does NOT apply if you run the job manually, in which case the Pool will be selected from the Job {} resource (or from you manually modifying the Pool then executing a run)



Bacula Configuring Messages

We want to be able to overview how the jobs are running - imagine our backup jobs failed (it can fail for many reasons, eg. no more backup space or a file daemon have crashed etc.) and we did not know about it and a month later we need to restore - that would be bad, very bad.

Bacula daemons all have the ability to send job execution information to various targets (log files, catalog, emailing, director), however Bacula also makes it possible to aggregate all the information of a job in one report per job. Here I will setup messaging so that we are emailed one report per job and also send that report to the Director so it is available using the Console program :

  1. bacula-dir.conf

    1. backup server shell> nano /etc/bacula/bacula-dir.conf : open bacula-dir.conf on the backup server in the nano editor.
      1. backula-dir.conf contains several Messages{..} sections, find the Messages{..} section with Name = Standard and edit it as following :
        • Messages {
        •     Name = Standard : the name by which a job can reference this section and thereby define the job report.
        •     mailcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<backup@favouritehosting.com\>\" -s \"Bacula: %t %e of %c %l\" %r" : command mailing the job report, the command have the following parts :
          • /usr/lib/bacula/bsmtp -h localhost : use bsmtp on host (-h) localhost to send the email.
          • -f \"\(Bacula\) \<backup@favouritehosting.com\>\" : create from (-f) field "Bacula backup@favouritehosting.com".
          • -s \"Bacula: %t %e of %c %l\" : create subject (-s) field "Bacula: [JobType] (-t) [Exit Code] (-e) of [Client] (-c) [Level] (-l), eg. "Bacula: Backup OK of win1 Incremental".
        •     operatorcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<backup@favouritehosting.com\>\" -s \"Bacula: Intervention needed for %j\" %r" : command mailing maintenance needs.
        •     mail = rasmusrummel@gmail.com = all, !skipped : using the mailcommand mail all message types except skipped to rasmusrummel@gmail.com all (eg. incremental backup will skip files not changed since last backup - do not send me any message that these files are not backup)
        •     operator = rasmusrummel@gmail.com = mount : using the operatorcommand mail all messages of type mount to rasmusrummel@gmail.com.
        •     console = all, !skipped, !saved : send all messages except of type skipped & saved to the console (these messages are held by the console until you request watching them using the messages command).
        •     #append = "/var/lib/bacula/log" = all, !skipped : I like to out-comment the log as I don't use it. If you keep the log file, you need to delete it periodically (it would be great if it was possible to set a maximum size on the log file so that oldest records are deleted then new records are appended).
        •     catalog = all : send all messages to the catalog (bacula) database (in table Log). The records in the Log table are pruned then the Job records are pruned so the table will not grow indefinetely. The Log table is typically used by reporting software, but can also be searched the same way as the file log I just out-commented.
        • }
    2. Press ctrl+x and then y to exit the nano editor and save the bacula-dir.conf file.
    3. backup server shell> service bacula-director restart : restart the Director to apply the new configurations.
  2. Install mail transfer agent (MTA)

    Bacula comes with a mail user agent (MUA) called bsmtp. Default your Message {} section will use /usr/lib/bacula/bsmtp to try to connect to an MTA on localhost as in the operator command above operatorcommand = "/usr/lib/bacula/bsmtp -h localhost ...". While bsmtp supports connecting to an external MTA, I have not tried how to authenticate, so I recommend installing an MTA on the same host as your Director. Ubuntu default installation does not come with any MTA, so if you have not installed an MTA on your Bacula Director host already, you need to install one now :

    1. backup server shell> nmap localhost : first check if any smtp daemon is already listening on port 25, if there are then an MTA is probably already available and you can skip the next step.
    2. backup server shell> apt-get install sendmail-bin : sendmail-bin is a small footprint well working MTA supporting both spools and sockets.

    Note that because of bsmtp capability to connect to an external host, bsmtp does not deliver messages to a spool service like /var/spool/mqueue, but instead connect to your MTA directly using TCP (like in telnet localhost 25).



Bacula Configuring Jobs

There are 3 types of jobs, backup jobs, restore jobs and verify jobs. We will create one backup job and one restore job for each client (each server to backup).

Then configuring a job, we need to specify quite a lot of job parameters :

  • On what client the job should run - we reference a Client{..} section.
  • What files to backup (or restore) - we reference a FileSet{..} section.
  • When to execute the job - we reference a Schedule{..} section (a schedule is only necessary for jobs you want to execute automatically).
  • On what device (storage) to store the backup - we reference a Storage{..} section.
  • What group of volumes to store the backup - we reference a Pool{..} section (a pool is a group of volumes).
  • What type the job is (backup, restore or verify)
  • In case of type = backup then also what level to backup (full, differential or incremental).
  • In case of type = verify then also what to verify (lots of settings there).
  • What priority the job has compared to other jobs

Also since Bacula does not natively support treating multiple physical devices as one logical device (eg. by being able to specify multiple "Archive Device" within one Device{..} section), we will need to spread the backup load by manually deciding which jobs run on which device - that is just not smooth, but that is how we are going to do it :

  1. bacula-dir.conf

    1. backup server shell> nano /etc/bacula/bacula-dir.conf : open bacula-dir.conf on the backup server in the nano editor.
      1. Setup 2 collections of job properties (one for each backup device) so we don't have to write all these properties more than once :
        1. Edit the bacula-dir.conf JobDefs{..} section : (there is default only one JobDefs{..} section)
          • JobDefs {
          •     Name = "jobdefs1" : all jobs including the definitions setup here will be backed up to disk1.
          •     Type = Backup :
          •     Schedule = "MonthlyCycle" :
          •     Storage = storage1 :
          •     Messages = Standard :
          •     Pool = pool1 : in our case we actually use this pool as we don't use the Schedule {..} section to overwrite the pool to use.
          •     Priority = 10 :
          •     Write Bootstrap = "/var/lib/bacula/%c.bsr" :
          • }
        2. Create a new bacula-dir.conf JobDefs{..} section : (to hold default values for all jobs that is associated with disk2)
          • JobDefs {
          •     Name = "jobdefs2"
          •     Type = Backup :
          •     Schedule = "MonthlyCycle" :
          •     Storage = storage2 :
          •     Messages = Standard :
          •     Pool = pool2
          •     Priority = 10 :
          •     Write Bootstrap = "/var/lib/bacula/%c.bsr" :
          • }
      2. Setup the 3 backup jobs (one backup job for each client to backup) :
        1. Create a new bacula-dir.conf Job{..} section :
          • Job {
          •     Name = "web1-backup" : this job will run on web1 and is of type backup.
          •     JobDefs = "jobdefs1" : include all job settings from jobdefs1.
          •     Client = web1 : reference to the web1 Client{..} section to specify what client to run this job on (web1 server).
          •     FileSet = "web1" : reference to the web1 FileSet{..} section to specify which files to backup on the client (which files to backup on the web1 server).
          • }
        2. Create a new bacula-dir.conf Job{..} section :
          • Job {
          •     Name = "db1-backup"
          •     JobDefs = "jobdefs2"
          •     Client = db1
          •     FileSet = "db1"
          • }
        3. Create a new bacula-dir.conf Job{..} section :
          • Job {
          •     Name = "win1-backup"
          •     JobDefs = "jobdefs2"
          •     Client = win1
          •     FileSet = "win1"
          • }
      3. Setup the 3 restore jobs : (one restore job for each client that we may need to restore)
        1. Create a new bacula-dir.conf Job{..} section :
          • Job {
          •     Name = "web1-restore"
          •     Type = Restore" : specifying that this is a restore job.
          •     Client = web1 : which client we want to restore.
          •     FileSet = "web1" : what files we are restoring from.
          •     Storage = "storage1" : on what device to find the backup files.
          •     Pool = "pool1" : in what collection of volumes the backup files are stored.
          •     Messages = "standard" : what messages to send where about the restore job.
          •     Where = "/var/FHRestore" : where to restore the files to, here I restore to a folder /var/FHRestore on the web1 server.
          • }
        2. Create a new bacula-dir.conf Job{..} section :
          • Job {
          •     Name = "db1-restore"
          •     Type = Restore"
          •     Client = db1
          •     FileSet = "db1"
          •     Storage = "storage2"
          •     Pool = "pool2"
          •     Messages = "standard"
          •     Where = "/var/FHRestore"
          • }
        3. Create a new bacula-dir.conf Job{..} section :
          • Job {
          •     Name = "win1-restore"
          •     Type = Restore"
          •     Client = win1
          •     FileSet = "win1"
          •     Storage = "storage2"
          •     Pool = "pool2"
          •     Messages = "standard"
          •     Where = "C:/FHRestore" : notice that for windows clients, we still use the forward slashes navigating a folder structure (though there is of course no beginning slash since windows does not have an equilavent to the linux root).
          • }
      4. Setup the 3 FileSet{..} sections : (one for each client)
        (A FileSet first of all consist of a list of directories & files to backup and a list of directories & files to exclude. However backup options like compression, encryption & signatures are available)
        1. Create a new bacula-dir.conf FileSet{..} section :
          • FileSet {
          •     Name = "web1" : since we are defining what files to backup on web1, I think web1 is a good name for this FileSet.
          •     Include { : the Include subsection (there are also an Exclude subsection which we will not use).
          •         Options { : different options can be specified for the Include subsection.
          •             signature = MD5 : here we specify that the Catalog database should keep a signature of each file, which I think is mostly useful with verify jobs and intrusion detection, however it is default so I keep it.
          •         }
          •         File = /var/www/clients : we want to backup the /var/www/clients folder (which is the parent folder for websites on this server). Here I only specify one File property, however we can have as many File properties as we want (see the win1 FileSet below).
          •     }
          • }
        2. Create a new bacula-dir.conf FileSet{..} section :
          • FileSet {
          •     Name = "db1"
          •     Include {
          •         Options {
          •             signature = MD5
          •         }
          •         File = /var/FHDBDumps : on db1 all databases are backup once a day to this folder - so here we are actually backup the backup.
          •     }
          • }
        3. Create a new bacula-dir.conf FileSet{..} section :
          • FileSet {
          •     Name = "win1"
          •     Include {
          •         Options {
          •             signature = MD5
          •         }
          •         File = "C:/Websites" : I have some asp.net websites here.
          •         File = "C:/Databases/Backup Auto" : I don't use a separate database server for my asp.net websites. All databases runs on the same server and are backup daily to this folder - again here we are actually backup the backup.
          •     }
          • }
    2. Press ctrl+x and then y to exit the nano editor and save the bacula-dir.conf file.
    3. backup server shell> service bacula-director restart : restart the Director to apply the new configurations.

Ok, that concludes the Bacula configuration - we should now have daily backup of our servers, however we better test if the backup jobs can actually run.



Test backup jobs

Here we want to test :

  • That we can manually execute a job (using the run command).
  • That after the job have executed a job report is written and send to the Director.
  • That we receive an email with the same job report.
  1. backup server shell> bconsole : start the Console program (to interact with the Director).
  2. *> run : start the interactive prompt for running a job, you should see all jobs defined (I have a little more than in this tutorial).
  3. Select Job resource (1-16):> 7 : I select 7 which on my system is win1-backup, you should select a number representing a backup job on your own system. You are now presented with the job properties, in my case :
    • JobName: win1-backup
    • Level: Incremental
    • Client: win1
    • FileSet: win1
    • Pool: pool2
    • Storage: storage2
    • When: NOW
    • Priority: 11
  4. OK to run? (yes/mod/no):> yes : you can change the job settings by writing mod, however if this is your first run with this FileSet on this Client, then your backup level will automatically be upgraded from incremental to full.
  5. *> messages : since we were alerted that we had messages, we wrote the messages command to display them. As we can see the Director, FH-dir, have upgraded the backup level to full and ordered the Storage daemon to write to pool2 on disk2 and the Storage daemon, FH-sd, have found a writable volume pool2-0008 in pool2 and is preparing the volume to receive data.
  6. *> status dir : check out the current situation - showing scheduled, running and terminated jobs. Here we can see that the win1-backup job is currently running. We can also see that there are more messages.
  7. *> messages : get the latest messages which is from the win1 file daemon stating that a VSS (Volume Shadow Copy) snapshot is being creating of the files to backup.
  8. Wait a long time ...
  9. *> messages : success, we got the last of the job report and it even terminates OK
  10. *> status dir : and we can see the job in the "Terminated Jobs" table.
  11. success, I got the report sent to my gmail account in which I have setup a Bacula label so I fast can select all emails from Bacula and delete them (mostly you will just want to fast see the termination codes, OK or ERROR).

Congratulations - you are now able to execute backup jobs, being notified with execution reports and most likely your scheduling also works (though that you can only test by waiting until next day or setting the clock appropriate in the schedule).



Restore files

The restore command is very fleksible enabling you to restore from backup latest version or older version of one or many files/folders for different clients restoring the files to different targets. However I find that the 2 most common restore schenarios are :

  • We need to restore the latest version of one or more files - (eg. if the files have been deleted or overwritten by accident).
  • We need to restore an older version of one or more files - (eg. if the files have become virus infected unnoticed for some time).

Note that if you want to cancel an action, you press the dot key '.' - dot is the standard cancel command throughout the Console program.

Restore latest version

Restore type 5 "Select the most recent backup for a client".

  1. backup server shell> bconsole : start the Bacula console program.
  2. *> restore : start the interactive restore process. The Catalog MyCatalog is automatically selected and you are presented with 12 different ways to select from which JobIds you want to restore.
  3. Select item (1 - 13):> 5 : select method 5 "Select the most recent backup for a client" which will prompt you to select what client to restore.
  4. Select the Client (1 - 12):> 9 : here I select the client web1 and the following is now happening :
    1. The only appropriate fileset, web1, is autoselected.
    2. The latest full backup JobId (84) and all subsequent incremental JobIds (87, 98, 107) are selected.
    3. A folder/file tree with the latest version of each file is build - this tree is now ready for navigation and full or partial selection of folders & files.
  5. $> dir : we use dir to see what is in current folder, here we find a subfolder called var.
  6. $> cd var : we use cd to change directory.
  7. Continue browse the file tree until you arrive to some files/folders you want to restore.
  8. $> mark web : we use mark to mark files or folders we want to restore, here we mark the "web" folder which contains 14,052 files. We could now continue to browse the file tree and mark more folders or single files if we would like.
  9. $> done : we use done to tell that we are finished selecting files. The restore dialog will now show what volumes and devices must be online to fetch the files selected, however this have no meaning for disk based storage as volumes are (in theory) always online. Also we are presented with a list of available restore jobs - the restore job we select will set various restore parameters, most importantly where to restore.
  10. Select Restore Job (1-11):> 3 : here we select "web1-restore" which will define several appropriate properties of the restore job, the most important are :
    • Restore Client: web1 : target server on which the restored files will be copied to.
    • Where: /var/FHRestore : path on the target server to where the restored files will be copied.
  11. OK to run? (yes/mod/no):> yes : execute the job with its default parameters as we have defined them in the bacula-dir.conf file (otherwise write mod to modify the parameters). Here the job gets the JobId 116.
  12. *> messages : SUCCESS I expected to see only the preparing setup in the messages, however the job was terminated very fast
  13. The job was also emailed my gmail account even though it was not clear from the above job parameter output that any messages were associated with the job.
  14. web1 server shell> cd /var/FHRestore : the part of the file tree restored (/var/www/clients/client1/web88/web) is copied to /var/FHRestore. All you now need to do is to overwrite the existing files - done.

Restore an older version

Restore type 6 "Select backup for a client before a specified time".

  1. backup server shell> bconsole : start the Bacula console program.
  2. *> restore : start the interactive restore process. The Catalog MyCatalog is automatically selected and you are presented with 12 different ways to select from which JobIds you want to restore.
  3. Select item (1 - 13):> 6 : select method 6 "Select backup for a client before a specified time" which will prompt you to specify a time before which the newest JobIds should be selected.
  4. Enter date as YYYY-MM-DD HH:MM:SS :> 2012-06-07 04:00:00 4 days ago after the nightly backup at 2:05
  5. Select the Client (1 - 12):> 9 : here I select the client web1 and the following is now happening :
    1. The only appropriate fileset, web1, is autoselected.
    2. The latest full backup JobId (84) and all subsequent incremental JobIds (87, 98) up to the specified time are selected. (notice that the incremental backup running 2012-06-08 with JobId 107 is NOT selected if you compare with the JobIds from latest backup)
    3. A folder/file tree with the latest version of each file is build - this tree is now ready for navigation and full or partial selection of folders & files.
  6. $> dir : we use dir to see what is in current folder, here we find a subfolder called var.
  7. $> cd var : we use cd to change directory.
  8. Continue browse the file tree until you arrive to some files/folders you want to restore.
  9. $> mark web : we use mark to mark files or folders we want to restore, here we mark the "web" folder which contains 14,052 files. We could now continue to browse the file tree and mark more folders or single files if we would like.
  10. $> done : we use done to tell that we are finished selecting files. The restore dialog will now show what volumes and devices must be online to fetch the files selected, however this have no meaning for disk based storage as volumes are (in theory) always online. Also we are presented with a list of available restore jobs - the restore job we select will set various restore parameters, most importantly where to restore.
  11. Select Restore Job (1-11):> 3 : here we select "web1-restore" which will define several appropriate properties of the restore job, the most important are :
    • Restore Client: web1 : target server on which the restored files will be copied to.
    • Where: /var/FHRestore : path on the target server to where the restored files will be copied.
  12. OK to run? (yes/mod/no):> yes : execute the job with its default parameters as we have defined them in the bacula-dir.conf file (otherwise write mod to modify the parameters). Here the job gets the JobId 116.
  13. *> messages : SUCCESS I expected to see only the preparing setup in the messages, however the job was terminated very fast
  14. The job was also emailed my gmail account even though it was not clear from the above job parameter output that any messages were associated with the job.
  15. web1 server shell> cd /var/FHRestore : the part of the file tree restored (/var/www/clients/client1/web88/web) is copied to /var/FHRestore. All you now need to do is to overwrite the existing files - done.

Restore a database

Restoring a database is straightforward :

  1. Restore the appropriate database backup file using either of the approaches above.
  2. Use the restored database file to restore your database the way you have always restored your database.


Bacula Console Commands & Management

Here are the commands I use the most : (full command list)

  • Daemon management :
    • backup server shell> service bacula-director restart : instead of restart you can use status, stop or start.
    • backup server shell> service bacula-sd restart : : instead of restart you can use status, stop or start.
    • client server shell> service bacula-fd restart : : instead of restart you can use status, stop or start.
  • backup server shell> bconsole : start the console
    • *> help : display the help menu.
    • *> version : display current Bacula version.
    • *> run : prompt for job to run - very important command.
    • *> restore : start the restore interactive prompt - very important command.
    • status-command :
      • *> status dir : displays todays terminated jobs, scheduled jobs and current jobs.
      • *> status storage :
      • *> status client : prompts to select a client to connect to and then displays list of jobs for that client.
      • *> status clients : connects to Storage daemon.
      • *> status jobs :
    • list-command : (used to show the content of the Catalog)
      • *> list jobs : list execution status of all jobs for some time back (eg. time run, how many files, how many bytes, what type of job and exit code)
      • *> list job=JOBNAME list execution status of a specific job for some time back
      • *> list volumes :
      • *> list pools
      • *> list clients
      • *> list volumes
      • *> list volumes jobid=JOBID
      • *> list volumes jobname=JOBNAME
      • *> list volumes pool=POOLNAME
      • *> list files jobid=JOBID : list all files backed up for a specific job
    • show-command : (used to show the content of the Directors resource records as defined in bacula-dir.conf)
      • *> show devices
      • *> show storages
      • *> show pools
      • *> show clients
      • *> show jobs
      • *> show filesets
      • *> show schedules
    • Management commands :
      • *> purge : prompts for what to purge, most common a volume for which all records should be pruned and VolStatus changed to Purged. Cannot be undone. After purging a volume, the jobs & files will be deleted from the catalog database and the volume record itself will change VolStatus to Purged. Data is still in the volume, however Bacula may now recycle the volume (whiping out the data) then a new volume is needed.
      • *> cancel :
      • *> update : prompts for what to update, eg. update pool will among other update volume retention periods for new volumes (existing volumes will keep their retention periods, bacula.Media.VolRetention, however retention period for individual volumes can be updated using update volume).
      • *> delete : prompts for what to delete (volume, pool or jobid). If deleting eg. a volume, the record will disappear in the catalog database, however the volume file itself will not be deleted. If you have some test pools, this is also a consistent way to delete the pool records from the database.
  • Delete Clients :
    1. backup server shell> mysql -u USERNAME -pPASSWORD : log on to the local mysql server.
      1. mysql> use bacula; : change database to bacula (the Catalog database).
      2. mysql> delete from Client where Name = 'CLIENTNAME'; : delete the relevant record from the Client table.
      3. mysql> quit : exit to system prompt.
    2. backup server shell> dbcheck -f -b -c /etc/bacula/bacula-dir.conf : remove any orphane records from the Catalog database.
  • Delete FileSets :
    1. backup server shell> mysql -u USERNAME -pPASSWORD : log on to the local mysql server.
      1. mysql> use bacula; : change database to bacula (the Catalog database).
      2. mysql> delete from FileSet where FileSet = 'FILESET'; : delete the relevant record from the FileSet table.
      3. mysql> quit : exit to system prompt.
    2. backup server shell> dbcheck -f -b -c /etc/bacula/bacula-dir.conf : remove any orphane records from the Catalog database.
  • Debugging bacula : (eg. why a daemon will not start)
    • backup server shell> tail -15 /var/log/bacula/log : shows the last 15 lines from the bacula log file.
    • backup server shell> service bacula-sd start -d100 : will start the Storage daemon with debug level 100 (default is zero).
  • Testing a new configuration file :
    • backup server shell> /etc/bacula/bacula-dir -t -c bacula-dir.conf : shows if the bacula director configuration file is syntactically correct (the same can be done for SD & FD).


Bacula Concepts

In my experience the most difficult concepts to grasp are : pruning, purging and rentention periods. Let me fast explain these concepts here :

  • Purge >< Prune >< Delete : A purged Volume is a Volume that contains only expired Jobs & files and thus is not in the Catalog. Then pruning a Volume, it means to delete expired jobs & files in the Catalog associated with that Volume. Since not all files & jobs within a Volume may be expired, pruning a Volume does not guarantee the Volume will be purged. You can though purge a Volume which will delete all files and jobs from the Catalog associated with that Volume whatever the timestamps on these records - that will guarantee a purged Volume.
  • Retention Periods : regards how long time Bacula keeps records in the Catalog database. Deleting Catalog file, job & volume records from the Catalog database is called pruning. Then eg. a file record is deleted, it is not any more possible to browse the file using the standard restore command.
    • File Retention : the period that Bacula will keep file records in the Catalog database. If AutoPrune is set to yes, then Bacula will delete file records from the Catalog database after the file retention period (actually these records may be deleted earlier if the Job record to which they belong have been deleted).
    • Job Retention : the period that Bacula will keep job records in the Catalog database. If AutoPrune is set to yes, then Bacula will delete job records from the Catalog database after the job retention period (actually a job record may be deleted earlier if the volume on which the job have been stored have been pruned).
    • Volume Retention : the period that Bacula will keep the volume records in the Catalog database after end time of the last job written to the volume. If AutoPrune is set to yes, then Bacula will delete volume records from the Catalog database after the volume retention period and also delete all jobs and file records associated with the volume. The physical volume can now be recycled (but will first be recycled if no other volume is free to write).
  • Volume Status : (*> list volumes will show all your volumes and their status)
    • Append : volume is appendable so backup data can be written to this volume.
    • Full : volume is full so no more backup data can be written to this volume.
    • Purged : volume contains only expired jobs & files (no content in the catalog points to this volume) and so this volume can be recycled for new backup data.
    • Error : some error have occured, the volume cannot be used to store backup data.
    • Recycle : I am not sure, I once had this status which prohibited backup. I tried to run a backup several times without luck, but then after maybe an hour a backup changed the status to Append and everything worked again.
    • Used : volume cannot be used to store additional backup data and even if jobs & files stored is all expired, the volume will not change status (to Append).
    • Archive :
    • Busy :
    • Cleaning :
    • Disabled :
    • Read-Only :


Appendix : Bacula utility programs

Sometimes disaster strucks and you may not be able to access the director using bconsole or the Catalog have been destroyed etc. In such cases there are different utilities you can use to list and retrieve data from the raw volumes (full list of Bacula utility programs).

  • Disaster utilities :
    • bscan : Can scan a volume and rebuilding the catalog database with the files from the volume. Standard restore of the files within the volume will then be possible. This is usefull in at least 2 scenarios :
      • The Catalog database have become damaged.
      • Information about the volume have been pruned from the Catalog database.
    • bextract : can extract files or file lists from volumes in case restore cannot be used.
    • bls : can do an ls-like listing of a bacula volume.
  • Other utilities :
    • bacula-web : a read-only web based interface to Director & Catalog showing real-time progress of jobs (Bacula-web homepage).
    • bweb : a web based interface to Director as a GUI alternative to the CLI Console.

Read more about Bacula disaster recovery here



Appendix : Bacula folders

  • /etc/bacula : bacula main folder on both backup server and linux backup clients. On backup server containing bacula-dir.conf, bacula-sd.conf, bconsole.conf etc. On backup clients containing bacula-fd.conf.
  • /var/log/bacula : bacula log file is here.
  • /var/lib/bacula : bacula keeps email to be sent here.
  • /usr/sbin : seems to keep several daemons here.
  • /usr/lib/bacula : the bacula smtp daemon is here.
  • C:\Program Files\Bacula : bacula main folder on windows backup clients containing most notably the bacula-fd.conf file.


Appendix : Bacula alternatives

This comparision is a work in progress, email me if you want to help develop it.

Name Free DB Integration Email Integration OS'es Comment
Bacula yes no no Unix, Linux, Windows, Mac
Amanda yes Amanda is free, but comes in a commercial enterprise version sold by Zmanda.
Commvault no Likely among the best enterprise backup software
Quest Netvault no
Arkeia
Crashplan no
Symantic Netbackup no
CA ARCserve Backup no
Tivoli Storage Manager no Is likely very expensive
EMC Networker no
HP Data Protector no
PerfectBackup+ no
FreeFileSync yes Linux, Windows Seems to be intended for personal use as it seems to be controlled from each computer instead of a central management.
Robocopy yes
SyncToy yes Windows Part of Microsoft Power Toys.
Retrospect no

A more detailed comparison of Bacula and other backup solutions here and here.

Another award based comparison by techtarget

Lastly I found this exceptional comparison half down the page by a guy called Mark : what-is-the-best-enterprise-backup-software.



Appendix : Common errors and solutions

  1. shell> bconsole displays Connecting to Director xxx.com:9101 and then returns to system prompt.
  2. shell> tail /var/log/bacula/log shows error Warning: Cannot bind port 9101: ERR=Cannot assign requested address: Retrying ...

Reason : Different problems may lead to error 1, however if you also have error 2, it is most likely because the directory address specified in /etc/bacula/bconsole.conf does not correctly resolve.

Example : In bconsole.conf I had the address of the bacula director specified by a domain name, favouritehosting.com, but then I changed the IP of favouritehosting.com and so bconsole would try to find the director on that new IP (which was a different server).
Say your bconsole.conf file look like this :

  • Director {
  •     Name = localhost-dir
  •     DIRport = 9101
  •     address = favouritehosting.com : favouritehosting.com does not any longer resolve to the IP of the server that bacula director is installed on.
  •     Password = "secret"
  • }

Solution : In /etc/bacula/bconsole.conf change the address field to the correct IP, eg like this :

  • Director {
  •     Name = localhost-dir
  •     DIRport = 9101
  •     address = 27.254.33.96 : ok, now I am sure to get the correct IP.
  •     Password = "secret"
  • }
  1. Bacula director service seems to crash less than a minute after starting it

Reason : There can be a lot of different reasons for this, so the most important is to try do debug the problem.

Debug example :

  • shell> tail -30 /var/log/bacula/log : check for any entry in the bacula log file
  • shell> /usr/sbin/bacula-dir -d 9 -c /etc/bacula/bacula-dir.conf start bacula-dir with debug (-d) maximum (9) using config file (-c) /etc/bacula/bacula-dir.conf.
  1. Director server cannot connect to client server JobId 0: Fatal error: bsock.c:134 Unable to connect to Client: web1-fd on 27.254.33.57:9102. ERR=Connection refused.

Reason 1 : There is a firewall on the client server blocking port 9102 - go to Solution 1.

Reason 2 : no software is listening on port 9102 :

  • Reason 2a : bacula file daemon is not running - go to Solution 2a.
  • Reason 2b : bacula-fd.conf on the client server specifies localhost or 127.0.0.1 for FDAddress in the FileDaemon{..} section - go to Solution 2b. Note that this problem cannot happen on a windows server, but is a very common problem on a linux server.

Solution 1 : unblock port 9102 on the client server

  • Unblock port 9102 on a windows server
    1. Click on "Start | Administrative Tools | Windows Firewall with Advanced Security"
    2. Select "Inbound Rules" and then click on "New Rule..." to create a new rule for Bacula.
    3. Select "Port" for Rule Type and then click the "Next" button.
    4. Set the port to 9102 and then click the "Next" button.
    5. Select the default "Allow the connection" and then click the "Next" button.
    6. Select the default which is all Profiles and then click the "Next" button.
    7. Name the new rule Bacula and then click the "Finish" button.
    8. Ok, the new rule called Bacula should appear in the list of inbound rules.
  • Unblock port 9102 on a Ubuntu server : (UFW : Uncomplicated FireWall : default iptables frontend in Ubuntu)
    1. shell> ufw status verbose : display among other whether the UFW firewall is enabled and if it is also display all the the firewall rules.
    2. shell> ufw allow 9102 : unblock port 9102 from all IP sources (this only have meaning if ufw is enabled).

Solution 2a : start the bacula file daemon : (I assume that bacula file daemon is actually installed on the client server).

  • Start the bacula file daemon on a windows server
    1. On the win1 server press the Windows Key + R to open the run box, write services.msc and press the "Ok" button to launch the Services Manager.
    2. In the Services Manager right click on "Bacula File Service" and choose "Restart" from the context menu.
  • Start the bacula file daemon on a Ubuntu server
    1. client server shell> netstat -anp | grep 9102 : show what if anything is listening on port 9102 (it should show bacula-fd).
    2. client server shell> service bacula-fd status : show whether bacula file daemon is running.
    3. client server shell> service bacula-fd restart : this will start or restart the bacula file daemon.

Solution 2b : edit bacula-fd.conf FileDaemon{..} section on the client server to specify global IP address:

  • On Windows : the problem cannot happen on a windows client.
  • On Linux :
    1. client server shell> nmap localhost : this may show that something is listening on port 9102.
    2. client server shell> nnap 27.254.33.57 : this should show that nothing is listening on port 9102.
    3. client server shell> nano /etc/bacula/bacula-fd.conf : open bacula-fd.conf in the nano editor.
    4. Edit the FileDaemon section :
      • FileDaemon {
      •     Name = web1-fd
      •     FDport = 9102
      •     WorkingDirectory = /var/lib/bacula
      •     Pid Directory = /var/run/bacula
      •     Maximum Concurrent Jobs = 20
      •     FDAddress = 27.254.33.57 : change FDAddress to the global IP address (not localhost or 127.0.0.1).
      • }
    5. Press ctrl+x and then y to close nano and save the changes to bacula-fd.conf.
    6. client server shell> service bacula-fd restart : restart the file daemon to apply the configuration changes.
    7. client server shell> telnet 27.254.33.57 9102 : this time you should get through (remember to use your own IP address)
    8. client server shell> telnet localhost 9102 : this should be refused.
  1. 09-May 14:51 FH-Backup-dir JobId 0: Fatal error: Unable to authenticate with File daemon at "27.254.33.57:9102". Possible causes:
    Passwords or names not the same or
    Maximum Concurrent Jobs exceeded on the FD or
    FD networking messed up (restart daemon).

Reason : Most likely your bacula-fd.conf Director{..} section (Password & Name) does not exactly match the values specified in bacula-dir.conf.

Solution : the following must be true : (values are case sensitive, also the Name value)

  • bacula-dir.conf Director{..} Name == bacula-fd.conf Director{..} Name
  • bacula-dir.conf Client{..} Password == bacula-fd.conf Director{..} Password
  1. Cannot find any appendable volume.
  2. *> status jobs displays Device xxx is not open and Device is blocked waiting to create a volume for:.

Reason 1 : If you get above error 6 or 7, the reason is most likely that there are no more space available on the backup device. Current volume (file) is full, there is no space to create new volumes and due to volume retention periods no existing volume can be purged to allow for new data to be written.

Reason 2 : especially the above error 7 can have other reasons than disk space shortage, eg. problems with the storage daemon.

Solution 1 : Long term solutions to this problem is to either increase the size of your device (bigger harddrive) or to reduce the volume retention period (keep backup data for a shorter period before discarding them). However the here and now solution that will allow you to continue your backup is to manually purge your oldest volumes :

  1. shell> bconsole : start the Console if not already started.
  2. *> purge : start the interactive purge program.
  3. Select Volumes
  4. You will now be allowed to select a volume to purge. Select your oldest volume and continue purging volumes until you have enough space. Purging a volume means to remove information of the content of the volume from the Catalog, not actually deleting the content within the volume. Also then purging a Volume, the VolStatus of the volume in the Catalog changes to Purged. If the volume you purge belong to a Pool that have Recycle=Yes, then Bacula will automatically continue the backup not long after a volume have been purged.

Solution 2 : restart the bacula daemons :

  1. shell> service bacula-sd restart
  2. shell> service bacula-director restart
  1. JOB-NAME is waiting for Client CLIENT-NAME to connect to Storage STORAGE-NAME

Reason 1 : Your firewall on the storage server is not allowing incoming traffic on port 9103 from your file daemon server (remember that the file daemon will have to initiate contact to the storage daemon).

Reason 2 : The file daemon does not know the IP address of the storage daemon - this IP address I think is given to the file daemon by the director from bacula-dir.conf Storage {} address, which under installation could be set to localhost (the Director does NOT translate localhost to actual IP then giving the value to the file daemon).

Solution 1 : Be sure your firewall allow tcp traffic on port 9103 on the storage daemon server : here using UFW

  • storage daemon server shell> ufw status verbose : first see if the UFW firewall is enabled and if it is then list the rules.
  • storage daemon server shell> ufw allow proto tcp to any port 9101:9103 : here I allow all the bacula ports to any IP on on the storage daemon server (I don't specify from what IP, so that is also any IP).

Solution 2 : Be sure to use global IP's and not localhost for daemon addresses :

  1. Write your bacula configuration files so the following is true :
    • bconsole.conf Director {} address = GlobalIP of director server
    • bacula-dir.conf Director {} DirAddress = GlobalIP of director server
    • bacula-dir.conf Storage {} Address = GlobalIP of storage daemon server
    • bacula-sd.conf Storage {} SDAddress = GlobalIP of storage daemon server
  2. Restart your daemons to apply the configuration changes : (normally director machine and storage daemon machine is the same machine)
    1. director machine shell> service bacula-director restart
    2. storage daemon machine shell> service bacula-sd restart

Comments

You can comment without logging in
Profile
Username
Password
Password
Email
Nickpic
Get notified on reply to own posts  (only works if you specify an email address)
Get notified on receiving a PM  (only works if you specify an email address)
Remember my username
Remember my password
signature
Words: Chars: Chars left: 
    


click to top Bacula Backup