full backup

[rldev]

How do some of you deal with the backup of user home directories? I ask because I currently use the DA Full SYs backup, but it would seem like a real pain to decompress the entire /home dir to get to a clients home directory in order to restore it if the home dir becomes very large. Any info on this topic would be nice to hear. I am interested in how you handle backups in general.

[nobaloney]

We use a customized version of the same program, making use of the CPanel module (not supplied by DA) to back up individual home directories.

We'll eventually make the product available at low cost, but it's not ready yet.

Of course you're welcome to do it yourself; we'll be releasing the list of required files to fully restore a server within a few days now, as we're done with the analysis and we're ready to try it out within the next few days.

Jeff

[rldev]

very good to hear. Seems a bit more functional. Looks like nobaloney is doing a lot of DA work :) I am interested in purchasing this script.

[Chrysalis]

yeah its good but also bad, some of these addon scripts that cost money really should be part of the base directadmin.

[rldev]

Well I don't mind paying if it is something I need. Agreed that DA could have done it this way and may in the future. I'm not sure what other add-on scripts cost money other then installatron(well worth the money) and this.

[vandal]

this feature should be in DA already and should be made the priority. The full backup is great, unless you actually need to restore it, it's half complete and should be completed so that if there is a problem you can have your server back online with a few hours max, not a few days.

[rldev]

What is exactly the problem with restoring the full backup as it stands now?

[nobaloney]

The sysbk implementation in DA has some files left out that are important to the reinstallation of the system. Fortunately the implementation is fully adjustable from the web-based interface DA includes, and you, or I, or both of us, can certainly take the time to resolve the issues.

It's taken us many trial and error hours to determine the proper files and directories to back up and to come up with a list which we're willing to give away as soon as we do one more verification run.

We always honor the licenses under which we work, and since sysbk is distributed under the GNU General Public License (GPL), we will publish any changes we make in the core sysbk code.

With the combination of the list of files that need to be backed up, and a copy of any of the changes we'll finallly make to Sysbk (as of now we haven't made any), anyone should be able to duplicate our work, in a lot less time than we've taken. And of course at that point, should you do so, you may certainly give it away if you want; that's the beauty of Open Source.

We've also invested a lot of hours in determining exactly how to do the restore, and we'll be publishing a how-to as well.

DA is a great low-priced system that manages server hosting. DA (the company) has merged my exim.conf file (on which we also spent many hours of work) into DA (the product), and they're certainly welcome to merge my published and released work on Sysbk into DA as well.

But writing a script to automate the How-To process (which includes a lot of lines such as "Do an intelligent merge between the /etc/passwd file on both systems" is going to be as time-consuming as the rest of the project.

DA (the product) doesn't do everything. Which leaves some room for some niche developers.

As always, for you, it's a matter of time vs money. The DA staff is a certain amount of manpower and is already swamped with work. Perhaps theyll eventually include a fully and properly implemented backup and restore solution.

What I've done, since I needed one now, is create my own without waiting for them.

Now I'm willing to give away a major piece of my work, or sell you the finished solution if you need it.

Many DA users won't buy it, and they'll still offer excellent service to their clients. Because they'll take the time to do it themselves. Perhaps they'll even come up with a better solution than mine.

And the beauty of it all is we can choose.

Jeff

[vandal]

i understand what you are saying, but paying for a control panel without a 1 click/minimal hassel backup/restore solution is like buying a new car without having any warranty.

sure you can do all the reseller backups yourself, but there are still bugs in that and it still uses a lot of the admins time up to restore.

other panels have this implemented, securely and pain free, and i'm a huge fan of DA, but this is so very important and should be made priority.

[nobaloney]

Originally posted by rldev
What is exactly the problem with restoring the full backup as it stands now?

Several problems come to mind:

1) Unless you've added files, DA is not by default saving all the files you need for a painless restore.

2) If you back up the entire home directory, as DA does, you'll be backing up your backup.

3) Some people are having problems determining the backup scenario.

4) Some backed up files shouldn't be copied over to the restore system, but rather should be intelligently compared/merged against files of the same name in the restored system.

Backup/Restore isn't quite the nobrainer some people would like.

But it hardly ever is.

Jeff

[Chrysalis]

yeah I got no problem with you wanting some reward for many hours of work you put into something, I just feel this should be already part of directadmin.

[vandal]

i know it's complex, and they should focus on this before doing anything else besides bug fixes.

[nobaloney]

Have you told them?

Jeff

[vandal]

absolutely.

[nobaloney]

Did they give you a reply?

If they're about to bring it out I can stop working on mine, as our resources here are limited.

Jeff

[vandal]

yes they did "Hello,

I'm not too sure. It would be very difficult/risky to implement due to the system password files (more so for FreeBSD).
I'll start to consider how it might be implemented.

Thank you,"

[Chrysalis]

whats the proper way for emailing suggestions/bugs to directadmin? I only know of this forum or support tickets and tickets dont seem appropriate for this kind of thing.

[sander815]

support@directadmin.com

[nobaloney]

Originally posted by vandal
yes they did
I've got it almost down pat for Linux. To make it work on FreeBSD someone would have to give me access to a full backup (using my specifications) of a FreeBSD box.

But since I'm planning on selling it as opposed to giving it away, no one may be willing to do that.

Jeff

[Chrysalis]

give me a pm of what you need.

[tritting]

As an interim solution, you may want to consider Reoback (http://sourceforge.net/projects/reoback/) to pickup, backup, and transfer any files/dirs you feel are being missed by DA. We used this simple script via cron for quite some time with good results.

Be well, TR

[nobaloney]

Originally posted by Chrysalis
give me a pm of what you need.
I'm still far behind in a lot of work, so it would be a while :( .

But I will probably be publishing the list of files, directories, and paths required for a successful reinstall on RHL by the end of the week, and then perhaps one of the FreeBSD gurus can figure it out.

Jeff

[nobaloney]

Originally posted by tritting
As an interim solution, you may want to consider Reoback (http://sourceforge.net/projects/reoback/) to pickup, backup, and transfer any files/dirs you feel are being missed by DA.
There's no problem with the sysbk script supplied by DA; the problem is with the list of directories and files it's supposed to back up.

We used this simple script via cron for quite some time with good results.
You don't know how successful you are until you've tried a complete restore to a new box without referring back to the old one (in other words a worst-case scenario restore).

It's with the restore that we found problems with the file/directory list.

If you know you have a file/directory list that will create a worst-case scenario restore, then why don't you share?

Thanks.

Jeff

[Chrysalis]

yeah I would agree the directadmin backup tool is quite excellent its just the list of what it backs up is incomplete, and the restore is lacking.

[tritting]

Originally posted by jlasman
There's no problem with the sysbk script supplied by DA; the problem is with the list of directories and files it's supposed to back up.


I didn't think I implied any different; as my post suggested, Reoback is simply a possible intrim solution to grab any files/dirs that are being missed by the DA backup routine until the issue is addressed.



You don't know how successful you are until you've tried a complete restore to a new box without referring back to the old one (in other words a worst-case scenario restore).

It's with the restore that we found problems with the file/directory list.


No argument on this point. Here's how we minimized the impact:
1) I don't believe in automated restores.
2) All boxes when completely setup (including all user accounts) were imaged (dd works quite well for this task) and the image stored.
3) Any custom additions added after the imaging were added to the Reoback file list.
4) /var /etc and certain /usr locations backed up nightly (FTP over SSH to remote location; this way I CAN refer to the old box).
5) Major software upgrades or security patches documented and filed for future reference.
6) Customers responsible for /home dirs. (however, we did back them up from time to time)
7) Dreamed nightly of being able to afford a nice Raid array. ;)

We did get lucky and only had 2 failures over 6 servers in 9 years, but restores were a piece of cake.

Be well, TR

[nobaloney]

I don't see why you'd need Reoback to do that; you can just add the same directories and files to the sysbk backup list.

Jeff

[tritting]

My impression of this thread was that there were files and dirs being left out of the list, and to have them included was something in the DA backend and hence the reason for my Reoback suggestion until this was accomplished by the DA folks. Apologies to all if I've misunderstood...

Be well, TR

[nobaloney]

Point taken.

The list of files and directories is customizable by the admin user in the DA interface.

Jeff

[Chrysalis]

I got a new problem with the backup now, it is crontabbed for every morning at 4am but it isn't finishing it stops at the mysql stage, the logging just stops as well. look below

---

Performing sanity checks: Completed
Checking load average: Completed
Checking free disk space: Completed
Performing Custom backup
Archiving /etc/exim.conf: Completed
Archiving /etc/exim.crt: Completed
Archiving /etc/exim.key: Completed
Archiving /etc/exim.pl: Completed
Archiving /etc/group: Completed
Archiving /etc/hosts: Completed
Archiving /etc/passwd: Completed
Archiving /etc/proftpd.conf: Completed
Archiving /etc/proftpd.passwd: Completed
Archiving /etc/proftpd.vhosts.conf: Completed
Archiving /etc/resolv.conf: Completed
Archiving /etc/shadow: Completed
Archiving /etc/system_filter.exim: Completed
Archiving /etc: Completed
Archiving /etc/mail: Completed
Archiving /etc/virtual: Completed
Archiving /usr/home: Completed
Archiving /usr/local/directadmin: Completed
Archiving /usr/local/etc: Completed
Archiving /usr/local/frontpage: Completed
Archiving /var/log: Completed
Archiving /var/mail: Completed
Archiving /var/spool/cron: Completed
Archiving /var/spool/mail: Completed
Archiving /var/spool/virtual: Completed
Archiving /var/www: Completed
Performing DNS backup
Stage 1 DNS backup: Completed
Stage 2 DNS backup: Completed
Performing Apache backup: Completed
Performing MySQL backup
Stage 1 MySQL backup:<br>

------

thats where it ended.

yet if I do a backup by hitting backup now it works every time.

[Chrysalis]

sorry heres the bit that cutoff

Performing MySQL backup
Stage 1 MySQL backup:MySQL could not be shutdown, aborting...
Performing cleanup operations: Completed

[nobaloney]

If your server doing it's crontab.daily stuff at the same time?

If so then perhaps for some reason mysql is already shut down and the error is causing it to abort.

Note that this is a guess.

Jeff

[Chrysalis]

it worked this morning.

I got 1 other crontab for 4am the license check.

0 4 * * * root echo 'action=check&value=license' >> /usr/local/directadmin/data/task.queue

[Chrysalis]

well I moved the crontab to 4.30 am and noticed it wasnt working but also it wasn't seemingly even trying to work.

So I did a manual backup and seen gzip start in the process list, but now 5 mins later nothing is happening.

1 - There is no new dir for today's date in the backup dir
2 - There is no message for me in directadmin
3 - 'View last backup log' Shows the log from my last succesful backup 5 days ago.
4 - There is plenty of free disk space (well over 90gig)

any idea what could be wrong?

[Chrysalis]

sorry it was working, I told it to delete local file after remote transfer and the dirs exist on the remote server.

But I am getting no backup done messages generated when it finishes.

[vandal]

i noticed similar problems, mine rarely tells me my backup was completed.

[ClayRabbit]

Damn... That DA feature killed my old sysbk config...
It seems their new features becomes more and more painfull for me...

We'll never use that feature because we already using modified version of sysbk to backup config files and mysql databases.
Furthermore, we'll never use sysbk for backing up user's home directories. It's great amount of data to copy over and over every night - so we implemented script for some kind of incremental backup to backup only new/modified data from /home/*.

A list of directories we are backing up with sysbk:
/etc/ssh
/etc/virtual
/usr/local/directadmin/conf
/usr/local/directadmin/data
/usr/local/directadmin/scripts/custom
/var/spool/cron (/var/cron on FreeBSD)
/var/spool/mail (/var/mail on FreeBSD)
/var/spool/virtual

List of files:
/etc/passwd*
/etc/group*
/etc/shadow*
/etc/*.conf
/etc/*.cnf
/etc/proftpd.*
/usr/local/frontpage/*.cnf

(There is no /etc/httpd/conf and /var/named in this list while that files are handled individually by sysbk.)

I believe this set of files is enough to completely restore web-server on new machine. But maybe I have missed something :)

Anyway backup lists need to be customized for your own systems - all depends on which files you are modifying, so maybe you have important and exclusive data in other locations of your hdd ;)

Maybe you'll prefer also to include directories:
/etc/cron.d
/etc/cron.daily
/etc/cron.hourly
/etc/cron.weekly
/etc/cron.monthly
/etc/mail
/root/.ssh
/usr/local/directadmin/plugins
/var/www

and files:
/etc/crontab
/etc/exim.*
/etc/hosts
/root/.*

PS: I wonder why I still use sysbk while all I need is creating tarballs from list of files and directories?... We are not using remote backups so I think there is also no point to use md5sum. As for mysql backup - I already rewrited this sysbk module to make it with mysqlhotcopy.
BTW, why sysbk stores 2 copies of mysql in different format? And both ways are not best - direct copy need to shutdown server, and mysqldump is quite slow epecially on restore.