HowTo: Custom webalizer wrapper for DA v0.0.1

[markus]

NOTE: Please, consider this a beta version. :rolleyes:

DOWNLOAD:
http://www.phpmix.com/files/directadmin/webalizer

# ======================================================================
# Custom webalizer wrapper for DA - version 1.0.0 (2005-01-11)
#
# Upgrades and further information
# --------------------------------
#
# Please, check out the DirectAdmin (DA) How-To Guides forum:
# http://www.directadmin.com/forum/forumdisplay.php?forumid=40
#
# History: See change log at the end of these notes.
#
#
# Short Description
# -----------------
#
# This script replaces the webalizer executable called by DA.
# By doing so, it allows you to:
#
# a) Use your own webalizer.conf, even using specific settings
# per domain (or even per subdomain), if you wish.
#
# b) Enable/Disable webalizer stats globally or per domain. This
# might be useful, for instance, if you're using some other Apache
# log analizer like AWStats, etc. for some or all your domains.
#
# In short, this script allows you to call the 'original' webalizer
# ONLY for the domains you want, the way you exactly want. :-)
#
#
# Compatibility
# -------------
#
# As of now (see version and date on top), this script supports:
#
# - DirectAdmin 1.23.3
# - Webalizer 2.01
# - Red Hat Linux Enterprise 3
#
# Please, confirm if you can make it run on other systems. Thanks
#
# ======================================================================
# 1. README
# ======================================================================
#
# 1.1. How current DirectAdmin allows you to customize Webalizer?
# ----------------------------------------------------------------
#
# Quoted from here:
# http://help.directadmin.com/item.php?id=27
#
# DirectAdmin uses the default webalizer settings by not specifying a
# webalizer.conf file at all. If you want to adjust the settings for
# webalizer, you can create your own basic webalizer.conf file for DA
# to use. This file must be domain independant because this one file
# will be used for all domains without any modifications to it.
#
# This file should be stored here:
# /usr/local/directadmin/data/template/custom/webalizer.conf
#
#
# 1.2. When DirectAdmin calls Webalizer?
# --------------------------------------
#
# DirectAdmin calls webalizer as part of the tally process which is
# triggered by a daily cron task.
#
# You can manually trigger this procedure by executing the following
# command:
#
# echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue
#
# Please, read this for further information:
# http://help.directadmin.com/item.php?id=48
#
#
# 1.3. How DirectAdmin calls Webalizer?
# -------------------------------------
#
# DirectAdmin uses the root user to call webalizer for each domain and
# subdomain as follows:
#
# webalizer -p -n HOSTNAME -o OUTPUT_DIR INPUT_LOG
#
# where:
#
# HOSTNAME - Might either be domain.tld or subdomain.domain.tld
# OUTPUT_DIR - This is the directory where the webalizer results
# will be stored (HTML and PNG files, etc.)
# INPUT_LOG - This is the input log file that will be processed.
#
# examples (*):
#
# webalizer -p -n example.com \
# -o /home/user1/domains/example.com/stats \
# /var/log/httpd/domains/example.com.log
#
# webalizer -p -n subdom.example.com \
# -o /home/user1/domains/example.com/stats/subdom \
# /var/log/httpd/domains/example.com.subdom.log
#
# (*) Note the \ symbol means the command continues in the next line ;-)
#
#
# 1.4. DirectAdmin limitations regarding Webalizer?
# -------------------------------------------------
#
# a) DA does not allow you to use domain specific tokens in the
# webalizer.conf custom template (see 1.1 above).
#
# b) DA runs the webalizer for all domains and subdomains defined in
# the system. But, what if you want to use a different Apache log
# analizer like AWStats, etc.?
#
#
# 1.5. What can we do?
# --------------------
#
# Well, meanwhile DA does not directly support more features related to
# webalizer... we can replace the webalizer executable with a little
# script, so that we can parse the command line and then:
#
# a) Decide if we really want to run webalizer for this domain. This is
# useful if, instead, you want to run another Apache log analizer.
#
# b) Use a custom webalizer.conf with the ability to use hostname
# related parameters.
#
# c) Use the same webalizer.conf for all domains or use different .conf
# files per domain (or even per subdomain). This also allows your own
# users to create the webalizer.conf file the way they wish.
#
#
# ======================================================================
# 2. INSTALLATION
# ======================================================================
#
# This installation procedure assumes you're logged in as root !!!
#
# 2.1. Create the file we will use to specify the domains we want to
# allow or deny using webalizer.
#
# cd /etc/virtual
# touch webalizer_domains
# chmod 660 webalizer_domains
#
# You should manually edit this file to add or remove domains.
#
# The syntax of the webalizer_domains file is pretty simple. You can
# specify a domain in each line. Blank lines are ignored. You can
# even use comments with the symbol #.
#
# You can customize the 'meaning' of the webalizer_domains file using
# the WEBALIZER_DEFAULT parameter in the custom webalizer script.
#
# WEBALIZER_DEFAULT="allow" means allow webalizer by default. So, you
# just use the webalizer_domains file to specify the domains that you
# want to deny from using webalizer.
#
# WEBALIZER_DEFAULT="deny" means deny webalizer by default. So, you
# just use the webalizer_domains file to specify the domains that you
# want to allow to use webalizer.
#
#
# 2.2. Replace the webalizer executable with this script.
#
# cd /usr/bin
# mv webalizer webalizer-orig
# wget http://files.phpmix.com/directadmin/webalizer
# chmod 750 webalizer
# chown webalizer:root webalizer
#
#
# 2.3. Customize the custom webalizer script (this script).
#
# * Please, check out the 'Configuration Section'.
# * Specially the WEBALIZER_DEFAULT parameter.
#
#
# 2.4. Install the logrotate script for the new webalizer logs.
#
# cd /etc/logrotate.d
# wget -O webalizer http://files.phpmix.com/directadmin/webalizer.logrotate
#
#
# 2.5. Customize your webalizer.conf files. Any of the following
# files is optional. It will be used only if it exists.
#
# The GLOBAL configuration file should be created here:
# /usr/local/directadmin/data/templates/webalizer.conf
#
# DOMAIN configuration files should be created here:
# /home/$USER/domains/$DOMAIN/stats/webalizer.conf
#
# SUBDOMAIN configuration files should be created here:
# /home/$USER/domains/$DOMAIN/stats/$SUBDOMAIN/webalizer.conf
#
# Note, you should replace $USER $DOMAIN and $SUBDOMAIN with valid
# values for your installation.
#
# In those .conf files you can use any of the following tokens:
# |HOSTNAME|
# |DOMAIN|
# |USER|
#
#
# And you're done!
#
# Please, check the chapter 1.3 in README section on how to execute
# webalizer to test the installation.
#
# Remember to check the webalizer_log if something seems to fail.
#
# ----------------------------------------------------------------------
#
# Copyright (c) 2005 Markus, phpmix.com
#
# Note that me nor phpmix.com have any affiliation with DirectAdmin.
#
# This script is released under the GNU GPL v2. You can find a copy of
# the license here: http://www.gnu.org/copyleft/gpl.html
#
# THIS SOFTWARE IS PROVIDED "AS IS" WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, WHETHER ORAL OR WRITTEN, EXPRESS OR IMPLIED.
#
# ======================================================================
# CHANGELOG:
#
# version 1.0.0 (2005-01-11)
# - Added support for subdomains.
# - Added more and better documentation.
# - Added more information messages to the log (troubleshooting).
#
# version 0.0.1 (2005-01-08)
# - First attempt.
# - It does not support subdomains (yet).
# - It needs better documentation.
# - The idea sounds promising, though. ;-)
#
# ======================================================================

[markus]

This is an example of global webalizer.conf, you should place it here:
/usr/local/directadmin/data/templates/webalizer.conf

GMTTime yes
VisitTimeout 1800
DNSCache /var/www/usage/dns_cache.db
DNSChildren 10

PageType htm*
PageType cgi
PageType txt
PageType *html
PageType php*
PageType asp*
PageType pl

HideURL *.gif
HideURL *.GIF
HideURL *.jpg
HideURL *.JPG
HideURL *.png
HideURL *.PNG
HideURL *.ra

TopAgents 30
AllAgents yes
TopCountries 30
TopReferrers 30
AllReferrers yes
TopSites 30
TopKSites 20
AllSites yes
TopURLs 30
TopKURLs 20
AllURLs yes
TopEntry 30
TopExit 30
TopSearch 20
AllSearchStr yes
TopUsers 0
AllUsers no

IgnoreSite *|DOMAIN|
IgnoreSite localhost*
IgnoreReferrer *|DOMAIN|/*
HideReferrer Direct Request


As you'll see, you can use the token |DOMAIN| in this configuration file. Our webalizer script will replace it with the current domain. This is useful, for instance, to create webalizer rules to ignore self referers, etc.

Comments are welcome. :)

[markus]

To test this script, you can trigger DA to run webalizer for all domains with the following command:

echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue

[rldev]

I tried these modifications but when I execute the command for it to run. It does not seem to be updating. How can I troubleshoot this problem? Thanks.

[rldev]

Also,
/etc/virtual/webalizer_domains

does not exist by default, this must be created.

[markus]

Originally posted by rldev
I tried these modifications but when I execute the command for it to run. It does not seem to be updating. How can I troubleshoot this problem? Thanks.
Please, look at /var/log/httpd/webalizer_log (or where your WEBALIZER_LOG parameter points to). All webalizer output should be redirected this this log, plus some messages generated by the script itself to easilly identify each executionor whether a domain has been skipped, etc.

You could also test this script calling it from the command line like this:

/usr/bin/webalizer -p \ (press enter)
-n |DOMAIN| \ (press enter)
-o /home/|USER|/domains/|DOMAIN|/stats \ (press enter)
/var/log/httpd/domains/|DOMAIN|.log

This is how DA calls webalizer. Just replace the tokens (|DOMAIN| and |USER|) with somthing else.


Originally posted by rldev
Also,
/etc/virtual/webalizer_domains

does not exist by default, this must be created.
If this file doesn't exists, the script behaves (as if it was empty) according to the value of the WEBALIZER_DEFAULT variable.



I have updated the first post on this topic with some notes. Please, keep in mind this is still Beta. I just figured it doesn't work correctly with subdomains.


More soon... :)

[rldev]

I get the following error when running webalizer -p

/usr/bin/webalizer: line 112: syntax error near unexpected token `)'
/usr/bin/webalizer: line 112: `below)'

Also there is no log being created.

log location is:

WEBALIZER_LOG="/var/log/httpd/webalizer_log"


I think part of the problem is copying the new webalizer script from the forum. I copied and pasted the script twice and easch time when I run webalizer -p it gives a different line error. Perhaps you can attach this script in a txt file?


After fixing the errors in the webalizer files. I get the following error:

/usr/bin/webalizer: line 83: /etc/virtual/domainowners: Permission denied
/usr/bin/webalizer: line 114: please,: command not found
/usr/bin/webalizer: line 151: logs: command not found

[markus]

I have replaced the code with a link to the file. See top topic. Thanks for taking the time to test this thingy... :)

As per the "permission denied" error:

You could create the file following these steps (as root):

touch /etc/virtual/webalizer_domains
chown mail:mail /etc/virtual/webalizer_domains
chmod 644 /etc/virtual/webalizer_domains

Syntax is just like /etc/virtual/domains. That is a single domain per line.

[rldev]

The problem is:

/usr/bin/webalizer: line 83: /etc/virtual/domainowners: Permission denied

which is referenced in your script.

-rw-r--r-- 1 mail mail 43 Jan 10 10:49 domainowners

[markus]

Try running the script as root.


DA invokes webalizer as root (in my RHLE 3 box).

[rldev]

What do you mean by running the script as root? I am using root. So if I am root shouldn't not matter who owns the files for the purposes of running webalizer -p ?

[rldev]

The problem seems to be the permissions on /etc/virtual/domainowners which is set to 644 by default. If you change the permissions to say 750 the script will start but give the following errors:

/etc/virtual/domainowners: line 1: lsmith.com:: command not found
/etc/virtual/domainowners: line 2: rocksolidnet.com:: command not found

[rldev]

I don't know but something seem up with the file you posted. I recpoied it from the link you provided and I now get the following errors:

[root@tp2 bin]# webalizer
/usr/bin/webalizer: line 42: /etc/virtual/domainowners: Permission denied
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options: GNU long options:
-f progfile --file=progfile
-F fs --field-separator=fs
-v var=val --assign=var=val
-m[fr] val
-W compat --compat
-W copyleft --copyleft
-W copyright --copyright
-W dump-variables[=file] --dump-variables[=file]
-W gen-po --gen-po
-W help --help
-W lint[=fatal] --lint[=fatal]
-W lint-old --lint-old
-W non-decimal-data --non-decimal-data
-W profile[=file] --profile[=file]
-W posix --posix
-W re-interval --re-interval
-W source=program-text --source=program-text
-W traditional --traditional
-W usage --usage
-W version --version

To report bugs, see node `Bugs' in `gawk.info', which is
section `Reporting Problems and Bugs' in the printed version.

gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.

Examples:
gawk '{ sum += $1 }; END { print sum }' file
gawk -F: '{ print $1 }' /etc/passwd
/usr/bin/webalizer: line 2: {print$1}: command not found

[markus]

Yup! ...I believe the error is the command you're using. Sorry if I didn't make it clear. I've rewritten the message where it is posted.

Here it is again:
/usr/bin/webalizer -p \ (press enter)
-n |DOMAIN| \ (press enter)
-o /home/|USER|/domains/|DOMAIN|/stats \ (press enter)
/var/log/httpd/domains/|DOMAIN|.log

The three lines are part of the same command. The symbol \ means the command continues in the next line.

Alternatively you could enter:
/usr/bin/webalizer -p -n |DOMAIN| -o /home/|USER|/domains/|DOMAIN|/stats /var/log/httpd/domains/|DOMAIN|.log
Do you see what I mean?

[rldev]

Yes I see what you mean but the errors are similar:

[root@tp2 bin]# /usr/bin/webalizer -p -n |DOMAIN| -o /home/|USER|/domains/|DOMAIN|/stats /v
ar/log/httpd/domains/|DOMAIN|.log
bash: DOMAIN: command not found
bash: -o: command not found
bash: USER: command not found
bash: /domains/: No such file or directory
bash: DOMAIN: command not found
bash: /stats: No such file or directory
bash: DOMAIN: command not found
bash: .log: command not found
/usr/bin/webalizer: line 42: /etc/virtual/domainowners: Permission denied
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options: GNU long options:
-f progfile --file=progfile
-F fs --field-separator=fs
-v var=val --assign=var=val
-m[fr] val
-W compat --compat
-W copyleft --copyleft
-W copyright --copyright
-W dump-variables[=file] --dump-variables[=file]
-W gen-po --gen-po
-W help --help
-W lint[=fatal] --lint[=fatal]
-W lint-old --lint-old
-W non-decimal-data --non-decimal-data
-W profile[=file] --profile[=file]
-W posix --posix
-W re-interval --re-interval
-W source=program-text --source=program-text
-W traditional --traditional
-W usage --usage
-W version --version

To report bugs, see node `Bugs' in `gawk.info', which is
section `Reporting Problems and Bugs' in the printed version.

gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.

Examples:
gawk '{ sum += $1 }; END { print sum }' file
gawk -F: '{ print $1 }' /etc/passwd
/usr/bin/webalizer: line 2: {print$1}: command not found

domainowners still gives a permission denied so the script can not attatin the proper values to process.

[markus]

Replace the tokens |DOMAIN| and |USER| with proper values.

For instance:
/usr/bin/webalizer -p -n example.com
-o /home/sample/domains/example.com/stats
/var/log/httpd/domains/example.com.logAll should be entered in the same command line. I just splitted the line to avoid breaking the post format.

The "permission denied" error seems like it is trying to write to the file. Using the proper command line should fix it. I believe.

[rldev]

Thanks for the help. Same problems. I'll let a few others test this.

[markus]

Originally posted by rldev
/usr/bin/webalizer: line 42: /etc/virtual/domainowners: Permission denied

line 42? That one should be a comment. This must be an error releted to the copy/paste operation when I first posted the code (on the first message) instead of a link to the file. :confused:


Anyway, thanks again for taking the time to test it. :)

[markus]

Hi all!

I have been rewriting some things in this script and updated the file and top post on this topic. I've changed the version number to 1.0.0

Changes:
- Added support for subdomains.
- Added more and better documentation.
- Added more information messages to the log (troubleshooting).

I should say I started to write bash scripts a couple of months ago or so. So, perhaps there are things that could be done much better.

I tried to create a script that should not eat much resources, though.

Please, let me know if you can see anything that could improved, suggestions, etc.

Thanks

PS: If it works, just enjoy! :D

PS: I've tried to change to topic title as well. But it didn't worked. :eek:

[jlasman]

Originally posted by markus
PS: I've tried to change to topic title as well. But it didn't worked. :eek:
What would you like it changed to?

I'll see what I can do.

Jeff

[markus]

Originally posted by jlasman
What would you like it changed to?
When I wrote some phpBB MOD-ifications I used to use the topic title to also display the current version.

I guess it should be enough without version number here... :rolleyes:


Thanks

[interfasys]

Paths for FreeBSD:
/usr/local/bin/bash
/usr/local/bin/webalizer

Not allowed on FreeBSD (not needed?)
chown webalizer:root webalizer



WARNING
Use this path for the webalizer.conf, otherwise it may get overwritten by directadmin on an update.
/usr/local/directadmin/data/template/custom/webalizer.conf

[markus]

Thanks for the input. :)

I'll be updating the script and notes on top to reflect changes required to fix above "problems", as well as to add support for FreeBSD.

In fact, if there's enough interest I'll be more than glad to provide support/upgrades for this "tool" in the future.

I'll also try to install AWStats, and see if it can be done anything else to integrate both worlds. Any input here would be much appreciated... :)

One thing doesn't work for me is the logrotate script. TBH, I don't know much about its syntax.

[interfasys]

On Freebsd, I don't have a /etc/logrotate.d so I didn't even try to install it.

You can use /etc/newsyslog.conf though.

[markus]

Could you please post a /etc/newsyslog.conf example here?

Sorry, I do not have access to a FreeBSD machine. :(


Thanks :)

[rldev]

I would imagine if we can somehow get this to work well, there would be interest. I guess people would like to be able to offer awstats or webalizer as a choice. This wrapper is a start, but then what happens to the general webalizer stats that DA uses?

[interfasys]

markus, I have no idea how newsyslog.conf works =(

rldev, how is DA using webalizer? To do what?

[rldev]

I was referring to the the Site Summary page. Isn't the information posted by the domain from Webalizer? If it was not why would there be a problem removing webalizer completely and using AWstats. I ask because a lot of people would like to do this.

[interfasys]

I think it would be possible to rewrite that "module" instead of trying to patch it since it's not built within the skin.

open nice css box
for every domain
show link to stats
show Bandwidth
show Disk Usage
show Log Usage
show Subdomains
show Suspended
endif


All those tokens are probably available to skin builders (me not skin builder)

[rldev]

After looking at, I think you may be correct.

[RoRoo]

Link dead? :'(

[mbear]

Yeah, this link is now dead. Anyone got an alternative/the scripts they could attach to a post or something?

Would really like to get rid of Webalizer!