how to configure DA to keep 7 days' worth of log lines in all of our log files?

[greentea]

Hi,

Is there any way we can configure DA to keep 7 days' worth of log lines in all of our log files, e.g. apache error log, instead of the default 1 day which is discarded everyday?

We would appreciate any sample code or scripts, if applicable.

Thank you in advance.

[jmstacey]

DirectAdmin rotates apache logs based on size. If the logs are going over that size in a one day period, then they will be rotated. Might want to check in Administrator settings.

It's also possible that logrotate was setup to rotate the apache logs everyday.

[greentea]

Hi,

We just checked our Administrator settings and it shows:

Max File Upload Size (bytes) 10485760
Apache Log Rotate Size (Megabytes) 5
Keep Number of Apache Log Backups 5

We checked admin.site-helper.com but the Administrator settings section said nothing about the above fields.

Could someone please explain what they are, what they do and what settings we should use to ensure that DA keeps 7 days' worth of log lines in all of our log files, e.g. apache error log, instead of the default 1 day which is discarded everyday?

Thank you in advance.

[jmstacey]

The first controls the max size (in bytes) of files that can be uploaded through the filemanager.

The second controls when apache logs are rotated. Currently they are rotated every 5mg. The size is checked and logs are rotated if needed when the tally is run every night.

And the last value controls the number of backlogs you want to keep. So for example, when a log is rotated it is renamed something like log.01 and then log is truncated and starts a new. Then log.01 becomes log.02 and log becomes log.01 and so on.

[rldev]

With the latest update logs are rotated daily, but I don't see where to control this.

[jmstacey]

I have not run accross it yet aswell. Might send DirectAdmin an email if they don't respond here or nobody else knows.

[rldev]

DA just got back to me. Daily log rotation is now forced. You can no longer set file size to rotate logs. You can set how many logs you want to keep. So to keep a week just set this to 7 in admin settings.

[jmstacey]

Ok, that's what I got from reading the features description but it just didn't seem like the direction DirectAdmin was heading on features lately (configurable).

[rldev]

Yeah I foud that strange indeed. I would rather have it this way then the way it was previously. It would be nice to specify day, week, month at least.

[nuclear]

This new method is highly annoying. It screws up my nightly log processing [which can be rescheduled but that's not the point] and is just crazy as to how many files it creates in the domain's log directory...

I can only imagine what happens if I need to search for something. I much prefered having [usually] just one log per month.

I don't see any reason why this has to be forced on users.

At the very least I'd like to see them rotated into the same archive for the month instead of creating one for every day stored.

[jmstacey]

I'm noticing apache.logrotate in /usr/local/directadmin/data/templates

If this script is called when the nightly tally runs it may be possible to workaround this by keeping track of the number of times it had been executed and then execute when it meets the desired threshold.

I haven't tried this, but it could be a possibility.

[DirectAdmin Support]

Hello,

The logs directly in /var/log/httpd are controlled by logrotate.
The logs in /var/log/httpd/domains are controlled by DA.

The reason we had to switch to the nightly rotation is because webalizer requires that the logs not be actively written to while it's running. The only way to accomplish that efficiently (that we could see) is to rename the logs to *.log.1 quickly (all at once), and then issue an HUP to apache to reopen new *.log files, leaving the *.log.1 files as inactive and ready for webalizer to use. Doing it this way has it's advantages as well, for one, allowing webalizer to work as it was intented to (same things applies to tar, it doesn't like active logs while compressing). Secondly, by rotating daily, webalizer doesn't have to reprocess old log data over again which should speed up the nightly tally fairly substantially (webalizer uses up the majority of the resources during the tally due to all the parsing).


There are ways to work around the nightly tally, mainly by editing the httpd.conf templates to not use the path for the logs/errorlogs that DA is expecting, thus allowing you to do whatever you'd like to them (note that it would stop webalizer from showing correct stats). Another option would be to add a 2nd CustomLog line for your own logs, leaving DA's intact, and adding a 2nd log for your own purposes.

As ususal, suggestions are welcome. ;) (Just keep in mind the reason we did it)

John

[albatroz]

At what hour is this logrotate process executed?

I need to execute my awstats over my apache logs before directadmin erases it...

[albatroz]

Dumping webalizer 'cos it sucks is a reasonable suggestion? :)

[davidb]

Ok.. Here's the suggestion.. Make the logging behavior an option. I have written a script and utilize an off-box solution (weblog expert) because it provides fantastic stats for my customers..

However, the new config kills me.. WebLog Expert does not keep history, but did utilize the compressed logs. Also, the # of logs retained kills me on sites that have more than 5 subdomains..

Since it is administrative, give us the option on webalizer friendly (new way) or the old way. Until then, I have nuked the logrotate.d/apache.conf file and am setting retention to 500 to protect the historical data that some of my sites need that is now gone due to the new rotation configuration.

[sander815]

uhm, how do i keep a week of logs as backup
Mine get rotated every 4 hours, so now i only have 5x4hours = 20 hours of logs

-rw-r--r-- 1 root root 152474 Jun 7 00:11 Jun-2005.1.tar.gz
-rw-r--r-- 1 root root 21954 Jun 7 04:10 Jun-2005.2.tar.gz
-rw-r--r-- 1 root root 13096 Jun 7 08:11 Jun-2005.3.tar.gz
-rw-r--r-- 1 root root 84259 Jun 7 12:10 Jun-2005.4.tar.gz
-rw-r--r-- 1 root root 110870 Jun 6 20:11 Jun-2005.tar.gz

[sander815]

Originally posted by davidb

Since it is administrative, give us the option on webalizer friendly (new way) or the old way. Until then, I have nuked the logrotate.d/apache.conf file and am setting retention to 500 to protect the historical data that some of my sites need that is now gone due to the new rotation configuration.

what did you change?

[ClayRabbit]

Originally posted by DirectAdmin Support
The reason we had to switch to the nightly rotation is because webalizer requires that the logs not be actively written to while it's running.Where we can read about that?
There are many control panels using webalizer, but at least 2 of them - CPanel and ISPmanager, doesn't seem to perform daily rotation and doesn't seem to have any problems with that, AFAIK.

So now, we have no way to switch back to rotation scheme used in previous DA versions? There is only two options now:
1) Use daily rotation by DA (rotation=1)
or
2) "make your own script to rotate by size or monthly and to put archives to domain's /stats/ directory" (rotation=0)
Is that so?

(So, in my opinion, that "feature" in v1.24 only made DA worse...)

[magmall]

for legal reasons it is really terrible to remove logs on a daily basis especially for an ecommerce site. My suggestion is to rotate daily as you do now but to append the day's logs to a "master backup log file" and that is the file which you control through the Administrator in terms of "true rotation". this way users can download a month of activity or a week of activity and then wipe out the backup any time. plus if people are using directadmin to charge for hosting it gives them additional revenue for storing log files.

[davidb]

What is the current status of the logging options? I am trying to improve my process and want to get clarification on the current frequency and how they are maintained (subdomains, etc). Also, is there an easy way to retrofit all of the users httpd.conf files with a second logging option line?

[Extrarius]

I'm also interested in the fix for this problem, because as a user I'd like to be able to have decent stats but more important to me is that I have decent access to logs. Many problems with the logs (such as not updating at all sometimes) and stats (right now my subdomain stats are frozen despite the logs 'working'{daily... ick}) on several DA hosts are prompting me to implement php+mysql-based a logging system into my custom site script, but that really isn't an option for a lot of people.

[David C]

Amazingly enough, my Admin Settings doesn't have "Max Log size" at all.

I do have "keep no. of logs" but no "Max Log size"

[albatroz]

I wonder if somebody from DA could make a summary of the current situation of this issue..

[DirectAdmin Support]

Hello,

Currently logs are rotated daily. The admin can specify how many rotations he wishes to keep with the "Keep Number of Apache Log Backups" option. There is no "rotate at size" option for the DA apache logs.

If you wish to change the behavior and move to your own setup, that's fine. You can either change the location of the logs, or also add your own 2nd CustomLog entry. To modify the VirtualHost entries for your Users, use this guide: http://help.directadmin.com/item.php?id=2
There will be 4 virtual_host*.conf templates for your version of apache (virtual_host, virtual_host_secure, virtual_host_sub, virtual_host_secure_sub). Note that if you add extra CustomLog entires, you are responsible for rotating them.

John

[David C]

Problem is, I don't have "max log size" in my admin panel.

[DirectAdmin Support]

Originally posted by David C
Problem is, I don't have "max log size" in my admin panel. There is no such option.

[David C]

Then I'm having a problem.

All my access logs are only like kept 1 day ago.

[Rebecca]

Originally posted by DirectAdmin Support

The logs directly in /var/log/httpd are controlled by logrotate.
The logs in /var/log/httpd/domains are controlled by DA.


So... I have a log in /var/log/httpd/domains that is exceeding 2GB daily and causing Apache to malfunction (very high traffic site). How can I get this file to be rotated more often?

I've looked at this post: http://www.directadmin.com/forum/showthread.php?s=&postid=78741#post78741

but I don't understand if the tally command referred to will also rotate this log.

Alternately, how can I get logrotate to deal with this particular file? I looked at the custom httpd.conf, and I can see where it sets the custom log directory, but it also says that any changes will be overwritten, so how do you change that? If I put that file in the directory above, would logrotate deal with it? And then how would i get logrotate to do it's thing more than once a day... would it have to be hourly? Or if i omit that line would it just not create that log?

Or any other suggestions for keeping the file from getting so big that it kills Apache.

BTW... I nuked webalizer long ago (webalizer = 0 in directadmin.conf) because it was running 24/7, but I didn't add anything about rotate in directadmin.conf, I'm assuming it defaults to true.

OK, any thoughts are greatly appreciated.
Rebecca