Howto: Full DirectAdmin backup

[ProWebUK]

Edit added 09/23/05 by jlasman:
I know this has been posted in a few places, but I can never find it online when I want to refer to it, so here it is:

This is the list of files and directories we recommend backing up for a DA restore:

Directories:
/etc/mail
/etc/virtual
/home/admin
/usr/lib/apache
/usr/local
/usr/share/ssl
/var/log
/var/mail
/var/named
/var/spool/cron
/var/spool/mail
/var/spool/virtual
/var/www
/usr/local/directadmin/data/admin

Files:
/etc/exim.conf
/etc/exim.crt
/etc/exim.key
/etc/exim.pl
/etc/group
/etc/gshadow
/etc/hosts
/etc/httpd/conf/httpd.conf
/etc/httpd/conf/ips.conf
/etc/named.conf
/etc/passwd
/etc/proftpd.conf
/etc/proftpd.passwd
/etc/proftpd.vhosts.conf
/etc/resolv.conf
/etc/shadow
/etc/ssh/sshd_config
/etc/system_filter.exim
/usr/local/directadmin/conf/mysql.conf

Of course we make no guarantees of any errors or omissions in this list; you're responsible for your own server backup.


Edit added 03/0/05 by jlasman:
It's no longer necessary to download and install sysbk into DirectAdmin as it's now included in the DirectAdmin installation and completely configurable through the admin level DirectAdmin control panel.

Well to keep things clean and simple its based on sysbk, a great script released by r-fx networks with 2 modules included for ensim and cpanel aswell as default support for such files as your mysql configs the mysql init script mysql dump and myisamchk binaries aswell as named config files and zone files - finally your apache directory also gets backed up - /etc/httpd

Its a very simple to use system, everything is done for you, basically theres a config file for you the rest is thoroughly checked by a built in md5 checker on most packages within it, a variable to allow the maximum load sysbk will run under (default value is 2 - anything higher sysbk wont make the backup at the time) and also a minimum disk space that you specifiy to perform backups.

Backups can be moved using FTP or SCP (SSH)

Before we get onto the installation and porting for directadmin here is the feature list from the website:

.: Summary of features:
- all backup operations have a companion .md5 file with md5sum of the original files, to ensure data integrity
- capable of using a secondary archiving utility such as bzip2
- easy configuration of backup path and mount points
- configurable load-level threshold for performing backup operations
- priority queing of backup operations to reduce run-time load level
- top level backup of apache path
- multi-stage Bind DNS backup
- multi-stage MySQL backup with MYI/ISM check, mysqldump of all databases to
individual .sql files, and master backup of mysql db path
- remote transfer of backup data via FTP or SCP (SSH)
- incremented transfer mode to conserve disk space during backups
- and much more...

Before we move on just need to make clear, as usual, Itake no responsibility with the use of this guide or the sysbk software

Ok this is basic installation we will move onto the directadmin side in a moment :)

from root:

# mkdir sysbk
# cd sysbk
# wget http://www.rfxnetworks.com/downloads/sysbk-current.tar.gz
# tar -zxvf sysbk-current.tar.gz
# rm -f sysbk-current.tar.gz
# cd sysbk-*
# chmod 750 install.sh
# ./install.sh
# cd
# cd /usr/local/sysbk
# pico -w conf.sysbk

set what you want to backup and what you dont want to backup from the default options there, leave ensim and cpanel areas as '0' so they do not get used.

If you want mysq / dns / apache backups ensure you set them to 1 not 0 on their first option for sysbk to make the backups of them. You mus also ensure that CUSTOM_BK="0" is chnaged to CUSTOM_BK="1" so we can specify the files needed to be backed up for directadmin and your accounts

[ctrl] + [x] to save press [y] to confirm.

# cd mod
# pico -w custom.dirs

# add paths to any additional directorys you wish to include some directadmin use are:

/home
/usr/local/directadmin
/usr/local/frontpage
/var/spool/virtual
/var/spool/mail
/var/www
/var/log
/etc/virtual


[ctrl] + [x] to save press [y] to confirm.

# pico -w custom.files
# add paths to any files you wish to include some directadmin use are:

/etc/exim.conf
/etc/exim.pl
/etc/system_filter.exim
/etc/exim.crt
/etc/exim.key
/etc/proftpd.conf
/etc/proftpd.vhosts.conf
/etc/proftpd.passwd
/etc/hosts
/etc/resolv.conf


[ctrl] + [x] to save press [y] to confirm.

The paths are based on http://directadmin.com/paths.html if you feel the need to add any others or remove any feel free to suggest :)

Before you atempt to use it i suggest you read through /usr/local/sysbk/conf.sysbk to input your ftp / ssh details there to connect / make the backup.... without that it may be unsuccessful with connecting for obvious reasons :)

Finally, the excutable is located at:

/usr/local/sysbk/sysbk

usage /usr/local/sbin/sysbk [-s] [-q]

-s Standard run
-q Quiet run


Chris

[sander815]

is it only possible to backup to ftp, or is 2nd HD also an option?

[ProWebUK]

Dont specify any external locations, and just change your path from /home/backup to an area opn your second HD (whever it is mounted)

As default (if you dont specify any ftp locations) all backups will be organised in the format:

/home/backup/<date>/*backup sections*

Chris

[S2S-Robert]

Will this also include executables and if so, will the backup be capable of being restored on a different RH OS? (9 instead of 7.3)

[ProWebUK]

Originally posted by S2S-Robert
Will this also include executables and if so, will the backup be capable of being restored on a different RH OS? (9 instead of 7.3)

It can include anything you can usually pack with tar basically!.... with default options such as httpd and mysql it backs up the binaries along with all the main files and configs for example

Chris

[S2S-Robert]

Originally posted by ProWebUK
It can include anything you can usually pack with tar basically!.... with default options such as httpd and mysql it backs up the binaries along with all the main files and configs for example

Chris And now for the second part of my question? :D

[ProWebUK]

Well..... anything that you could usually take from 7.3 and place on 9 will work...... if something is compiled specifcally for X then it may not work on another OS.

The upgrade software itself has no effect on whether the file you restore is suitable for whatever system you restore on.

Chris

[netswitch]

don't we have to add
/var/named
in the custom.dirs file in order to backup the zone configuration ?


also somebody aldready tested a restart from scrath with this backup ?

[bjseiler]

I am trying to run this on freebsd and I am getting this error. When I run ./sysbk -s

[: 0: unexpected operator
Archiving /home: [: custom/: unexpected operator
-en \033[62G
[ -en \033[1;32m
$COMPLETED-en \033[0;39m


Any ideas?

[ProWebUK]

Originally posted by netswitch
don't we have to add
/var/named
in the custom.dirs file in order to backup the zone configuration ?



Nope, just ensure you enable DNS backup;

BIND_BK="0"

to:

BIND_BK="1"

It then backs up your whole /var/named directory (all zone files)aswell as your bind configuration file (/etc/named.conf)

I also advise you to enable all options apart from ensim and cpanel specialsed backup options :)

Regarding the restore, it basically just packs and compresses the files then stores them where you specify. locally or externally.. restore procedure should simply be untar the files....... I advise you to play around with some backups and the saved structure if you are unsure about anything.

Chris

Chris

[ProWebUK]

Originally posted by bjseiler
I am trying to run this on freebsd and I am getting this error. When I run ./sysbk -s

[: 0: unexpected operator
Archiving /home: [: custom/: unexpected operator
-en \033[62G
[ -en \033[1;32m
$COMPLETED-en \033[0;39m


Any ideas?

Not sure if it works with FBSD. nothing noted on its website regarding it either.... unable to help when it comes to FBSD and 3rd party software (especially).

Sorry,
Chris

[bjseiler]

Just so I am clear, when you run

./sysbk -s

you don't get any output that looks like what I posted?

[ProWebUK]

Originally posted by bjseiler
Just so I am clear, when you run

./sysbk -s

you don't get any output that looks like what I posted?

Performing sanity checks: [ COMPLETED ]
Checking load average: [ COMPLETED ]
Checking free disk space: [ COMPLETED ]
Performing Apache backup: [ COMPLETED ]
Performing cleanup operations: [ COMPLETED ]


Is basically what you should get, the option to get no output would be quiet run (./sysbk -q)

Deepending what options you select the output above will vary, the example is for a backup of apache only... no custom no dns etc

Chris

[prophecy]

How about restoring all that now? Or is that a whole other ballgame?

[sander815]

i think that involves a lot of handwork....?

[ProWebUK]

Originally posted by prophecy
How about restoring all that now? Or is that a whole other ballgame?

It may sound difficult, however it's actually very easy..... move all of the tar.gz backups to your root directory (/root/) then simply unpack them - tar -zxvf *.tar.gz

Im fairly sure the backup includes symlinks - although it certainly holds the privileges :)

(Also remember to check the md5sums on your backups :) )

Chris

[FarCry]

you might want to add /var/mail to that list

[ProWebUK]

Its /var/spool/mail and its already under the custom directories :)

Chris

[prophecy]

Ok, just went through this process today, and it's now 7:30 am and finally finishing up.

The backup process went smooth and that sysbk is very cool.

Restoring:

I basically, unpacked everything and copied each thing that was unpacked to the new OS build (RH 9). This worked great for everything, but email because permsissions were MESSED!

Issues

Had to remake all user accounts in RH using useradd.

Had to:
chmod -R diradmin.diradmin * in /usr/local/directadmin/data/users/ to dir

chown apache.apache * in /var/www/html/squirrelmail-1.4.2/data

chown admin.admin * in /home/admin/mail

make sure in /var/mail the owner is the username.mail

in /var/spool/virtual, the main user that created the accounts should own the files and dirs and mail should be the group (chown USER.mail)

I lost track of what permissions went where so this may not be right, but the email permissions should be the following thanks to John for this:




drwxrwxrwt 33 mail mail 1024 Jan 19 11:58 /var/spool/virtual

drwxrwx--- 2 username mail 1024 Jan 3 15:34 /var/spool/virtual/domain.com

-rw-rw---- 1 username mail 2010 Jan 20 12:11 /var/spool/virtual/domain.com/char
drwxr-xr-x 38 mail mail 2048 Jan 25 12:02 /etc/virtual

drwx--x--x 4 mail mail 1024 Jan 25 12:02 /etc/virtual/domain.com

/etc/virtual/domain.com/
-rw------- 1 mail mail 171 Jan 20 12:04 aliases
-rw------- 1 mail mail 30 Dec 31 14:55 autoresponder.conf
-rw------- 1 mail mail 2107 Dec 31 14:55 filter
-rw------- 1 mail mail 407 Dec 31 14:55 filter.conf
drwxr-x--x 5 majordom daemon 1024 Jan 23 20:31 majordomo
-rw------- 1 mail mail 57 Dec 31 14:55 passwd
-rw------- 1 mail mail 35 Dec 31 14:55 quota
drwx------ 2 mail mail 1024 Jan 4 14:05 reply
-rw------- 1 mail mail 119 Jan 4 14:03 vacation.conf

How could this have beens smoother?

I got to thinking how things coudl have been smoother and I think that if the users were made before unpacking, then the permissions may have worked out?? No idea, any linux guru know? And if that's the case, the order should have been:
[list=1]
Backup from old machine using this guide and sysbk
Create all users from old machine using useradd and appropriate groups if required.
Unpack all backups into appropriate locations
[/list=1]

And that might have been better, I unpacked first and then created users and have been fixing permissions for the past day.

[ProWebUK]

Originally posted by prophecy
I got to thinking how things coudl have been smoother and I think that if the users were made before unpacking, then the permissions may have worked out?? No idea, any linux guru know? And if that's the case, the order should have been:


Yes, if a user doesn't exist you shouldnt be able to give something ownerships of that user (since it doesnt exist!)

/etc/passwd and /etc/group may be an idea....... I will look into backing up full users.

Thanks for sharing the information :)

Chris

[prophecy]

Originally posted by ProWebUK

/etc/passwd and /etc/group may be an idea....... I will look into backing up full users.

That would be awesome if you could just transfer all the users along too, probalby skipping the root user and all the users and groups that DA makes as well. I guess only the users for the sites themselves should be moved.

[Icheb]

This system seems to be a bit better developed than the system i'm using at this time, but i would really like to be able to transfer the files offsite using rsync, since my company's datatraffic is limited to 7.5 GB/month at the moment.
At the moment i use rsync to first transfer all needed files offsite and than use our intranet server to tar it after which it goes to a certain backup drive...

Or is it just the best solution to use the current version and transfer the /home/backup/ dir ?
I only need to see to it that there isn't an <date> variable since this would require a new transfer of all files if using rsync to do it.

(yes, it's rsync over an ssh tunnel :D, so not the realy unsecure method :D)

[ProWebUK]

You get the option to:

- store data on the local server
- ftp to an external location
- ssh to an external location

With this backup script :)

Chris

[sander815]

when we add /home to the backup config it takes ages for it to finish, and we took it off teh backup...
WHat could cause that? /home isn't that big, 15 gb i think

[ProWebUK]

Originally posted by sander815
when we add /home to the backup config it takes ages for it to finish, and we took it off teh backup...
WHat could cause that? /home isn't that big, 15 gb i think

Give it a few minutes.... the script may appear to die on a few areas however its more than likely working.... you could try using tar -czf to tar it manually and see how long that takes... its more than likely not instant :)

Chris

[prophecy]

Originally posted by sander815
when we add /home to the backup config it takes ages for it to finish, and we took it off teh backup...
WHat could cause that? /home isn't that big, 15 gb i think

It took about half an hour for mine on the home directory, but that is because it's doing two processor intensive tasks

1. Compressing
2. MD5 checksumming

Both are worth the wait.

[prophecy]

Originally posted by Icheb
This system seems to be a bit better developed than the system i'm using at this time, but i would really like to be able to transfer the files offsite using rsync, since my company's datatraffic is limited to 7.5 GB/month at the moment.
At the moment i use rsync to first transfer all needed files offsite and than use our intranet server to tar it after which it goes to a certain backup drive...

Or is it just the best solution to use the current version and transfer the /home/backup/ dir ?
I only need to see to it that there isn't an <date> variable since this would require a new transfer of all files if using rsync to do it.

(yes, it's rsync over an ssh tunnel :D, so not the realy unsecure method :D)

I think you are probably looking for something different, the backup guide here is for entire server backups in general, something you might do once a week or month or when you plan to move servers. This isn't a good backup system for daily backups or anything like that. rsync is much better suited for that and if you just rsynced your /home directory, that would work just fine to keep things backed up daily and it would not take long.

I guess the best thing for keeping things completely backed up would be to run this system say once a week and have an rsync run once each night.

[Icheb]

Originally posted by prophecy
I guess the best thing for keeping things completely backed up would be to run this system say once a week and have an rsync run once each night.

It sounds great in theory, but really, at the moment we don't have the bandwidth to do this. The server bandwidth is ok, but our own company's isn't...
But it seems like a nice enough idea to impliment, a /home only rsync and a weekly backup transfer :)


Edit:
The backuping system seems to be working :)
The only problem i've found so far, is it's not really working on a production server, since all open mysql tables are skipped instead of dumped with current values...
So half my MySQL backup is missing.

[pilpelet]

Chris ,

The .install.sh doesnt work , its more like make install -d ./....

[ProWebUK]

Originally posted by pilpelet
Chris ,

The .install.sh doesnt work , its more like make install -d ./....

typo I have just noticed, should be

./install.sh

not:

.install.sh

Chris

[ClayRabbit]

Nice program. Thank you :)

Just my 2 cents:

1) remove /home from custom.dirs
2) remove all entries in cpanel.dirs and cpanel.files
3) Make CPANEL_BK="1" in conf.sysbk

And you will get nice separate archives for each user's homedir :)
I think that's much better than ONE HUGE home backup.

(But anyway homedirs backup process is taking too much time... :(
I think we need some smart script for backuping userdirs in /home. Script must get latest ctime among user's files (but ignore files in userdir/domains/*/logs and userdir/domains/*/stats) and compare it to mtime of previous backup file. If first < second then we don't need to create new archive - just copy it from previous backup.
Unfortunately, i'm not very famliar with programming in *nix, and don't know how to write such command-file. Of course i can write it in Perl, but...)

[pilpelet]

Hi ,

Worth to mention that the FTP option wont work on most DA installtions without installing "ncftpput" , by default it doesnt .


NcFTP installation : [updated]

wget ftp://ftp.ncftp.com/ncftp/binaries/ncftp-3.1.7-linux-x86-export.tar.gz
tar zxvf ncftp-3.1.7-linux-x86-export.tar.gz
rm -f -R ncftp-3.1.7-linux-x86-export.tar.gz
cd ncftp-3.1.7
make install

The installtion do it all and put all the needed files in place , after that the FTP option of sysbk will work .

If you need to scp to server B an automate the backup then it a bit more complicated , i wrote a full howto in this post :

http://www.directadmin.com/forum/showthread.php?s=&postid=14122#post14122

[XYZed]

Hi,

Just seeing if the main thread has been updated with all the little recommendations/tweaks?

Also is this suitable for moving an entire server? - We are replacing all our hardware and need to move all servers to the new ones. I want to keep the same IP's etc (well 1 new one for the new server so can transfer the files). Yes I could just stick in the old HDD's, but they are PATA whilst new ones are SATA and RAID1 config.

So is this suitable for a complete move on a clean RH install ?

I guess it's treating it like a full restore from a dead HDD or hacked system. Fun part is there is only 20 servers to do :-(

Thanks.

[netswitch]

The main problem is that thee is no automated restore script yet (or I missed it) and restoring a full server by hand is quite a huge piece of work...

[XYZed]

Wonderful, I can see it's time to go and hire a few linux gurus for a few weeks.

or maybe:

cd /
tar -zcvf *.* all.tar.gz

then wget, and untar LOL - if only.

Amazing there isn't a restore script for the backups yet - thankfully never needed it.

[netswitch]

the output of the sysbk is a bit more complex than a full system tar...

[jlasman]

One of the posts in this thread says it doesn't backup up open mysql files, but I've checked the code and it DOES stop mysql before it backs up, so it shouldn't have any problems with mysql database backups. Is anyone still having problems with this?

Also...

I'm testing this backup program as a prelude to offering more serviceability...

One thing I'm not sure of...

Where it wants: MYSQL_ROOTPW=
in conf.sysbk...

Do I use the value of

mysql=

in usr/local/directadmin/scripts/setup.txt ?

Thanks.

Jeff

[ProWebUK]

Originally posted by jlasman
Also...

I'm testing this backup program as a prelude to offering more serviceability...

One thing I'm not sure of...

Where it wants: MYSQL_ROOTPW=
in conf.sysbk...

Do I use the value of

mysql=

in usr/local/directadmin/scripts/setup.txt ?

Unless you have changed that password, which you should, that would be your mysql root password... so yes :)

Chris

[jlasman]

This is a client's machine and he hasn't changed the password.

I searched the forums but I can't find where I have to put the new password if I change it (besides in the sysbk script of course; that I know :) ).

So if I change the password, where do I put the new one?

Thanks.

Jeff

[ProWebUK]

Originally posted by jlasman
So if I change the password, where do I put the new one?

Not sure what you mean, *how* do you chnage the root mysql password? - or if you have changed it..?

Chris

[jlasman]

I found a password n usr/local/directadmin/scripts/setup.txt .

It works for dumping any mysql database on the system, so I guess I'm okay in presuming it's the master password :) .

You wrote:Unless you have changed that password, which you should, that would be your mysql root password.
So my question now is:

If I change the mysql root password, do I have to put the new master password anywhere so DA can use it?

(I know where to put it so sysbk can put it.)

Thanks.

Jeff

[ProWebUK]

No, setup.txt is for the first installation only, and im 99% certain the password is only there for you to know, nothing else :)

Chris

[jlasman]

Thanks, Chris, but what I'm thinking is, then where does PHPmyAdmin get it's password from?

Thanks.

Jeff

[ProWebUK]

Originally posted by jlasman
Thanks, Chris, but what I'm thinking is, then where does PHPmyAdmin get it's password from?

As a user logs into phpmyadmin, it uses mysql itself to check the passwords / login combinations... and grab the data.... pma doesnt require any password/username files unless you specify to use it like that ;)

Chris

[sander815]

can i exclude certains dirs in /home that sysbk doesn't backup

f.i. /home/admin/user_backups is useless, cause they already tarred, and take large amounts of extra diskspace


and why do i get these errors in mysql stage 2 backup?
myisamchk: warning: 1 clients is using or hasn't closed the table properly
myisamchk: MyISAM file /var/lib/mysql/nurburg_forum/phpbb_posts.MYI
m

[Mathieu]

With Freebsd:

Performing sanity checks: -en \033[62G
[ -en \033[1;31m
$FAILED-en \033[0;39m
]
/bin/gunzip not found, aborting...
Performing cleanup operations: [: 1: unexpected operator
-en \033[62G
[ -en \033[1;32m
$COMPLETED-en \033[0;39m
]



Can anyone fix this for me plz????
Thanks

[dxoxb]

Sorry, but is this thread still valid, I am a newbie to linux and I can't seem to get through the 4th line.


# mkdir sysbk
# cd sysbk
# wget http://www.rfxnetworks.com/download...-current.tar.gz
# tar -zxvf sysbk-current.tar.gz
# rm -f sysbk-current.tar.gz.....

[tribal-dolphin]

This function is now include in DirectAdmin.
Tis post is now out of date.

[jlasman]

Thanks for bringing this to my attention; I've added an edit at the top of the thread to indicate that sysbk is now included with DirectAdmin.

Jeff

[jaapvstr]

Originally posted by ProWebUK
It may sound difficult, however it's actually very easy..... move all of the tar.gz backups to your root directory (/root/) then simply unpack them - tar -zxvf *.tar.gz

Shouldn't you unpack using tar -zxpvf *.tar.gz to perserve permissions? i think things will be pretty messed up without the -p switch.

[jlasman]

I believe that by default tar preserves permissions; I don't believe I've never seen otherwise.

Jeff

[jaapvstr]

Odd that it's actually listed as a seperate switch then in the man pages o_O

p Restore the named files to their original modes, and
ACLs if applicable, ignoring the present umask(1).
This is the default behavior if invoked as super-user
with the x function letter specified. If super-user,
SETUID and sticky information are also extracted, and
files are restored with their original owners and per-
missions, rather than owned by root. When this func-
tion modifier is used with the c function, ACLs are
created in the tarfile along with other information.
Errors will occur when a tarfile with ACLs is
extracted by previous versions of tar.

Ah, yep, default if you run it as su

[roarkh]

I was just looking through the file list in the first post of this thread and noticed that one of the files in the list is...

/etc/exim.crt

At least on my installation I do not see that file but I do see this file...

/etc/exim.cert

Is there possibly a typo in the file list?

[jlasman]

possibly.

Jeff

[DavidJacobs]

Hmmm, this thread might be out of date, but Jeff; you don't mention if you found the place where the root PW for MySQL is stored, if you're still wondering; I believe it is in /usr/local/directadmin/conf/mysql.conf

gr david

[jlasman]

Yes, that's been in my list for some time. Thanks for updating the thread.

Jeff