No backups durring the DAY !!!

[Salmus]

Hello,

I have 15 clients with data on account ~ 15 GB - and they have a stupid need to make backups in the middle of the day - or every evening ! (18:00 -> 22:00 P.M) When my servers are really requested - and they cannot handle 2 concurrent backups with more than 1500 requests per sec.


There is any possibility to set DirectAdmin to queue backups for the midnight for example ? Or - in the interval 02:00 -> 05:00 A.M when all the servers are more idle than busy ?

Please ! I really need this ...

[nobaloney]

In other words, force them to not run backups when they want to, but when you want them to?

Jeff

[scsi]

I could see a need for this.

[Salmus]

In other words: force backups (queue) to be made at specific range of time

For example: between 02:00 -> 05:00 A.M. - when a major of servers are not used so much.


An easy fix for me will be to cat /usr/local/directadmin/data/task.queue end extract lines with "backup" word - and reinsert them in the specific range of time (via cron). But an configuration option in DirectAdmin about this will be awesome.


I do not think that I'm the only one who needs this, no one will want that backups to be made in the middle of day day ... when load is high.

[tillo]

This would be nice indeed.

[jca]

Great idea, so far I've never run into this problem, but I've seen clients do backups during "work" hours (when they are at the office) and the server is busier, but a queue for midnight would rock.

[DirectAdmin Support]

Hmm.. for the backup cron creation, the all_pre.sh would be able to do this... but for the "now" option (immediate backup), it would be more complex. It would be a bit confusing if they issued a "now" backup.. DA accpeted it, but then did nothing for several hours.. I could see that being a bit confusing. Another option would be to have backups placed into a backup.queue file.. and have a dataskq cronjob executed whenever you want to process that queue.. there are several ways of going about it that could be considered.

John

[scsi]

Well usually its not a problem if a user wants to do a now backup. But I do understand if you have a customer with like alot of files. It could put alot of load on the server at the wrong time.

Another idea might be to renice the backup with extremly low priority.

[nobaloney]

I believe it's already running with an extremely low priority.

Jeff

[smtalk]

I think it's easier to run a cronjob and renice the process if load is e.g. >4.

[tillo]

Unfortunately this won't solve all the problems.
For example I've a couple MyISAM DB with 1-2GB of data; mysqldump seems to have exclusive access when dumping MyISAM data, therefore any page or cron script can't access the DB while the backup is running.

[Salmus]

Renice-ing the process will not solve the problem ... MySQL is the biggest problem - it will bring the load upto 15 on my machines for databases ~ 1,5 GB - and it will drastically increase response time of MySQL.

[DirectAdmin Support]

Here's just a quick sample of blocking end Users (Site Backup) from running backups during the day. This example requires the time to be after 10pm and before 6am.

Create:
/usr/local/directadmin/scripts/custom/all_pre.sh

Add the code:

#!/bin/sh
if [ "$command" = "/CMD_SITE_BACKUP" ]; then
NOW=`date '+%H'`
EVE=22
MORN=6

if [ "$EVE" -le "$NOW" ] || [ "$MORN" -gt "$NOW" ]; then
exit 0;
else
echo "please run your backups between hour $EVE pm and $MORN am";
exit 1;
fi
fi
exit 0;chmod the all_pre.sh to 755, and you're in business.

This doesn't address Reseller Backups, specifically cronjobs.. but the same idea could be used for cron creation.. check the variables for the hour set for the cron to ensure it's within the range you want.

Edit: thinking about it, you can use this same concept... if the time is during the day, then still return "exit 1", but with a "backup queued" message, and you could add the username to a list, and have another cronjob to execute the list later that night.. using the task.queue format you see when you create a backup normally... just dump the same command to the task.queue when you want, and you've now got queued backups during the day, and instant ones during the night.

John

[evil_smurf]

It's a nice idea, but why can't you just set the priority for the backups to be as low as possible? That way it may take a few extra minutes (or hours even) to complete, but it would not be impacting your server's ability to handle requests.

[DirectAdmin Support]

It already is set to 19 I think... the problem is that any connection to mysql doesn't have a nice value.. mysqld controlls the priority, not mysqldump (unless there is a way to set mysqld via mysqldump through the socket).. because the load isn't myqldump that's spiking the cpu usage.. it's the mysqld process which is a seperate entity that is already running.. The dump is treated like a connection just like when viewing a forum... except all data is requested at the same time.
Related: http://www.directadmin.com/features.php?id=577

John

[morfargekko]

Hello, should backup_nice=19 be set in directadmin.conf ?

I don't have it there. :confused:

[DirectAdmin Support]

It's already set to 19.. that's the default.
If you want to change it to something other than 19, then you add it to the directadmin.conf (it overrides the internal default).
To see what you've got right now, type:
cd /usr/local/directadmin/
./directadmin c | grep backup_niceand that shows you what's being used (assuming DA was restarted if you've made a change)

John

[morfargekko]

Thanks John.
It would be nice to have some kind of doc that tell us what we can do, how it works and why. Maybe there is one but in that case I havn't been able to find it.

[DirectAdmin Support]

To see all options, type:
cd /usr/local/directadmin
./directadmin cThen simply type in the variable name into:
http://www.directadmin.com/versions.php
And you'll get the docs.

The versions system holds all feature/variables/changes ever done... while the help.directadmin.com contains guides on how to use some of the more commonly used features (faq).

John

[morfargekko]

Thanks again John. This is good info :)

[floyd]

to set DirectAdmin to queue backups for the midnight

This would need to be fully configurable since midnight in one part is noon in another part.

My customers are world wide and there really is no slower time during the day.