HowTo: MailScanner+Exim for FreeBSD

[Shahid]

MailScanner Installation Guide – Exim (FreeBSD)

Exim: MailScanner + ClamAV + SpamAssasin for FreeBSD

Install HTML-Parser
Install SpamAssasin
Install MailScanner
Install ClamAV
Creating directory
Configuration: Exim
Configuration: MailScanner
Run and test

NOTE: Before you proceeding below, it is recommended that your ports (/usr/ports/) should be updated. If not then go to http://www.bsdguides.org/guides/freebsd/misc/update_ports_tree.php and follow the instruction.

============================
| Install HTML-Parser
============================
HTML-Parser is for parsing text formatted in HTML

# cd /usr/ports/www/p5-HTML-Parser
# make install


============================
| Install SpamAssasin
============================
# cd /usr/ports/mail/p5-Mail-SpamAssassin
# make install

Note: do not load SpamAssassin(spamd) and modify exim.conf to use SpamAssassin


============================
| Install MailScanner
============================
# cd /usr/ports/mail/mailscanner
# make install
# make initial-config


============================
| Install ClamAV
============================
# cd /usr/ports/security/clamav
# make install

Note: Don't worry about ClamAV Update, MailScanner will do it for you. There is no point running ClamAV(clamd) at the background process, MailScanner can still run ClamAV for you.


============================
| Creating directories
============================
We now need to create some directories and chown it.

Exim incoming spool directories:
# mkdir /var/spool/exim.in
# mkdir /var/spool/exim.in/input
# mkdir /var/spool/exim.in/data
# mkdir /var/spool/exim.in/db
# chown mail:mail /var/spool/exim.in
# chown mail:mail /var/spool/exim.in/input
# chown mail:mail /var/spool/exim.in/data
# chown mail:mail /var/spool/exim.in/db

MailScanner spool directories:
# mkdir /var/spool/MailScanner
# mkdir /var/spool/MailScanner/incoming
# mkdir /var/spool/MailScanner/quarantine
# chown mail:mail /var/spool/MailScanner
# chown mail:mail /var/spool/MailScanner/incoming
# chown mail:mail /var/spool/MailScanner/quarantine


============================
| Configuration: Exim
============================
We have to use two separate Exim processes daemons and each of the daemons must have it is own configuration file, so that the spool directories can be different. Incoming mail is accepted into one queue and outgoing mail is sent to different queue.

Create a backup:
# cp /etc/exim.conf /etc/exim.conf-backup

Copy:
#cp /etc/exim.conf /etc/exim_outgoing.conf

Using your favourite text editor (such as, nano, ee, vim, etc), we need to add some lines in configuration file.
# nano -w /etc/exim.conf

Add the following lines (After MAIN CONFIGURATION SETTINGS header):
log_file_path = /var/spool/exim/msglog/%slog
process_log_path = /var/spool/exim/exim-process.info
queue_only = true
queue_only_override = false


Modify startup Exim, we need to add another exim process.
# nano -w /usr/local/etc/rc.d/exim

-------------------
Original:
echo -n "Starting exim: "
daemon /usr/sbin/exim $EXIM_OPTS -oP /var/run/exim.pid
RETVAL=$?
-------------------
Change it to:
echo -n "Starting exim: "
daemon /usr/sbin/exim $EXIM_OPTS -oP /var/run/exim.pid
daemon /usr/sbin/exim -q15m -C /etc/exim_outgoing.conf
RETVAL=$?
-------------------


============================
| Configuration: MailScanner
============================
# nano -w /usr/local/etc/MailScanner/MailScanner.conf

And modify these:
----------------------------
%org-name% = yoursite
(Do not use . (dot) in %org-name%)

Run As User = mail
Run As Group = mail
Incoming Queue Dir = /var/spool/exim.in/input
Outgoing Queue Dir = /var/spool/exim/input
MTA = exim
Sendmail = /usr/sbin/exim -C /etc/exim.conf
Sendmail2 = /usr/sbin/exim -C /etc/exim_outgoing.conf
Virus Scanners = clamav
Always Include SpamAssassin Report = yes
#Spam List = ORDB-RBL SBL+XBL # MAPS-RBL+ costs money (except .ac.uk)
Spam List =
Use SpamAssassin = yes
#Enable Spam Bounce = %rules-dir%/bounce.rules
----------------------------

============================
| Run and Test
============================
# killall exim
# /usr/local/etc/rc.d/exim start

# cd /usr/local/etc/rc.d/
# mv mailscanner.sh.sample mailscanner.sh
# /usr/local/etc/rc.d/mailscanner.sh start

Wait for few seconds, and then check the log:
# tail -f /var/log/maillog

Note: If you use see "MailScanner[98992]: User's home directory /var/mail/mail does not exist" this will show only for few seconds and it will stop – don't worry about this.


and now try to send an email to your box hopefully you won't get any error in maillog :)


When you receive an email – you should get something like this in email header:

X-CompayName-MailScanner: Found to be clean
X-CompayName-MailScanner-SpamCheck: not spam, SpamAssassin (score=2.147, required 6, FROM_ENDS_IN_NUMS 0.87, HTML_40_50 0.47, HTML_MESSAGE 0.00, MIME_MISSING_BOUNDARY 0.80)


If you want to send some test viruses to your box then download some test viruses at http://www.eicar.org/anti_virus_test_file.htm (Download these files at your own risk!)


That it!
Hope you enjoy this HOWTO guide. Any problems, idea, tips or security - please feel free to post here.


Edit: I've tested with FreeBSD 4.9 and fully working, I can't guarantee if it work with 5.x properly.


Shahid Hussain

[joh2900]

shouldnt the spool directory be in exim.conf?

[wdv]

No, MailScanner moves the files from exim.in to exim.

[existenz]

Interesting I tested what you did and found a few issues on my install:

One everything seems to be working and I am able to send and recieve email BUT I don't see any header information that lets me know that the services are working properly. Here is a example of the maillog...

Jun 9 01:51:05 dom MailScanner[42337]: MailScanner E-Mail Virus Scanner version 4.31.6 starting...
Jun 9 01:51:05 dom MailScanner[42337]: User's home directory /home/mail does not exist
Jun 9 01:51:05 dom MailScanner[42337]: User's home directory /home/mail is not writable
Jun 9 01:51:05 dom MailScanner[42337]: You need to set the "SpamAssassin User State Dir" to a directory that the "Run As User" can write to
Jun 9 01:51:09 dom MailScanner[42337]: Using locktype = posix
Jun 9 01:51:09 dom MailScanner[42337]: Creating hardcoded struct_flock subroutine for freebsd (BSD-type)


What should the "SpamAssassin User State Dir" be set to. Is't option is for individual users spamassassin configs?

Jun 9 02:25:23 dom MailScanner[2685]: MailScanner E-Mail Virus Scanner version 4.31.6 starting...
Jun 9 02:25:27 dom MailScanner[2685]: Using locktype = posix
Jun 9 02:25:27 dom MailScanner[2685]: Creating hardcoded struct_flock subroutine for freebsd (BSD-type)


I also don't get mail on the test box when I check for mail but I do just after a restart...

humm something is wrong

I have check a few times that I followed the steps properly.

Any ideas?

[wdv]

Try:

# mkdir /var/spool/MailScanner/spamassassin
# chown mail:mail /var/spool/MailScanner/spamassassin

And set the "SpamAssassin User State Dir" directive to this directory.

[Shahid]

I think this is because you have set something in "SpamAssassin User State Dir" in MailScanner.conf? and it could not write files to that dir.

I don't have anything in "SpamAssassin User State Dir = ".


Take a look, make sure you have:
MTA = exim
SpamAssassin User State Dir = (Dont put anything here)
Always Include SpamAssassin Report = yes

If you have made some change in MailScanner.conf then do:
# /usr/local/etc/rc.d/mailscanner.sh stop
# /usr/local/etc/rc.d/mailscanner.sh start

# killall exim
# /usr/local/etc/rc.d/exim start


Also make sure two running exim process is running at the background
# ps aux | grep exim

should return:
mail 30463 0.0 0.0 3568 0 ?? IWs - 0:00.00 /usr/sbin/exim -bd -q1h -oP /var/run/exim.pid
mail 30465 0.0 0.3 3568 308 ?? Is Mon06PM 0:00.15 /usr/sbin/exim -q15m -C /etc/exim_outgoing.conf

This is normal if you see this, it is not an error:

Jun 9 02:25:23 dom MailScanner[2685]: MailScanner E-Mail Virus Scanner version 4.31.6 starting...
Jun 9 02:25:27 dom MailScanner[2685]: Using locktype = posix
Jun 9 02:25:27 dom MailScanner[2685]: Creating hardcoded struct_flock subroutine for freebsd (BSD-type)



let me know if that fixed out.

[existenz]

I have checked the settings again and that does not seem to be the issue.

Exim appears to be working properly

mail 69095 0.0 0.7 2772 1688 ?? Ss 11:20AM 0:00.00 /usr/sbin/exim -bd -q1h -oP /var/run/exim.pid
mail 69097 0.0 0.7 2772 1688 ?? Ss 11:20AM 0:00.00 /usr/sbin/exim -q15m -C /etc/exim_outgoing.conf
root 69099 0.0 0.7 2784 1688 ?? S 11:20AM 0:00.07 /usr/sbin/exim -q
root 69101 0.0 0.7 2784 1688 ?? S 11:20AM 0:00.02 /usr/sbin/exim -C /etc/exim_outgoing.conf -q
root 69180 0.0 0.7 2908 1780 ?? S 11:20AM 0:00.00 /usr/sbin/exim -C /etc/exim_outgoing.conf -q
mail 69181 0.0 0.7 2908 1800 ?? S 11:20AM 0:00.00 /usr/sbin/exim -C /etc/exim_outgoing.conf -q
root 69470 0.0 0.7 2892 1768 ?? S 11:21AM 0:00.00 /usr/sbin/exim -q
mail 69484 0.0 0.7 2908 1800 ?? S 11:21AM 0:00.00 /usr/sbin/exim -q

Now when I restart the maillog will continue to show over and over:


Jun 9 11:35:32 dom MailScanner[71941]: MailScanner E-Mail Virus Scanner version 4.31.6 starting...
Jun 9 11:35:32 dom MailScanner[71941]: User's home directory /home/mail does not exist
Jun 9 11:35:32 dom MailScanner[71941]: User's home directory /home/mail is not writable
Jun 9 11:35:32 dom MailScanner[71941]: You need to set the "SpamAssassin User State Dir" to a directory that the "Run As User" can write to
Jun 9 11:35:35 dom MailScanner[71941]: Using locktype = posix
Jun 9 11:35:35 dom MailScanner[71941]: Creating hardcoded struct_flock subroutine for freebsd (BSD-type)

Just leave the user state dir alone? Should't be set to something like "/var/spool/MailScanner/spamassassin"?

The mail now gets delayed like 10 minutes or more and it is not being scanned by anything. According to the headers...

Anywhere else I could have messed up your install?

[wdv]

existenz, try what I said. I didn't use this tutorial, but I had the same problem at first when I installed mailscanner.

[Shahid]

existenz: I am not too sure why that happen to you. It work perfect on mine.

Yes I just leave "SpamAssassin User State Dir " alone and it work without error.

If you wanted me to take a look at your box to find a solution but that is your call.


Shahid

[existenz]

I was thinking of adding in the mailscanner directory manually but it is confusing that it is working but not tagging the mail. Just FYI this is a 5.2.x box but should not matter...

I will try to figure out the solution...

[brundle]

I am running 4.9 and having the same issue.

Messages get delivered, slowly, but no scanning appears to be done. Message delivery si quick if I stop/start the daemons.

So far I followed all the steps in this thread.

[Shahid]

Originally posted by brundle
I am running 4.9 and having the same issue.

Messages get delivered, slowly, but no scanning appears to be done. Message delivery si quick if I stop/start the daemons.

So far I followed all the steps in this thread.

How much RAM do you have in your box? you might be running out of memory when you have loaded MailScanner.


-----------------------------------------
# How many MailScanner processes do you want to run at a time?
# There is no point increasing this figure if your MailScanner server
# is happily keeping up with your mail traffic.
# If you are running on a server with more than 1 CPU, or you have a
# high mail load (and/or slow DNS lookups) then you should see better
# performance if you increase this figure.
# If you are running on a small system with limited RAM, you should
# note that each child takes just over 20MB.
#
# As a rough guide, try 5 children per CPU. But read the notes above.
-----------------------------------------


Take a look in mailscanner.conf

Max Children = 5

try change it to: Max Children = 1

then restart MailScanner, see if that make any different.

[brundle]

The server is a dual proc p4 with 1gig ram. It is currently set to 5 child procs. in the MailScanner.conf

[existenz]

Originally posted by wdv
Try:

# mkdir /var/spool/MailScanner/spamassassin
# chown mail:mail /var/spool/MailScanner/spamassassin

And set the "SpamAssassin User State Dir" directive to this directory.

Tested...tested and tested some more. Every box I tried the howto had the same problem. I think dual 2.8 Xeon's with 2GB of Ram would not be that slow. As soon as I manually configured the above it worked. I thought that was the problem and it turned out to be.

I just don't see in the headers where it said the mail is being scanned by MailScanner or SpamAssassin. I know SpamAssassin is working because my Spam mailbox is not getting any mail.

[brundle]

Setting a value for the 'SpammAssassin User State Dir' didn't help for me - same symptoms - extremely slow delivery.

[Quento]

Same here.

On a FREEBSD 4.9 box.

I have updated the ports before the installation (CUP). But on 4.9 I do not get the latest versions of mailscanner.

Also Exim log shows errors. It seems like the 2 exim deamons have time-outs. Like the waiting for port 25 to come free. Hense the slow delivery.

[Shahid]

I am not too sure why you getting problems.. I have repeated this HOWTO on second server without a problem..

if you want me to take a look at your server to fix mailscanner issue for you then email me at shahid[AT]zonewave[DOT]net

Shahid

[synergy]

We had similar problems until we added the following to exim.conf:

spool_directory = /var/spool/exim.in

Otherwise mail was being spooled in the same area that the outgoing exim process was using. Thereby MailScanner was never picking it up. Hope this helps someone.

[wdv]

If you want to have it installed on FreeBSD, please contact me. I'll install it for a low price, I have done it a few times now so it won't be a problem for me.

Contact me at w-w-d-e-v-r-i-e-s@g-m-a-i-l.com

(Remove all the -)

[Quento]

Originally posted by mcc235
spool_directory = /var/spool/exim.in

That did it for me. The tags are available.

Edit:
It works now. I forgot the tag clamav as virusscanner :)

[brundle]

That appears to have worked for me as well - on two servers. One 4.9 and one 4.10.

[fusionictnl]

Hello,

When i installed the whole set, my mail delivery costs about 5 secs a message.

Jun 9 11:35:32 dom MailScanner[71941]: User's home directory /home/mail does not exist
Jun 9 11:35:32 dom MailScanner[71941]: User's home directory /home/mail is not writable

I created this directory and gave mail:mail permissions on it. By doing this spamassassin could write it's logs en statistics in:

.razor .spamassassin

Now my delivery takes < 1 sc ;)

[RosT]

All work with line

spool_directory = /var/spool/exim.in

in /etc/exim.conf.

How to update antivirus databases?

[wdv]

Use freshclam

[RosT]

Many spam-messages I receive marked with {Spam?} in the subject. Why? How to fix this?

[existenz]

I have been trying to get it to work on a 5.x box and it does not appear to be working properly. Anyone having success?

[wdv]

Originally posted by existenz
I have been trying to get it to work on a 5.x box and it does not appear to be working properly. Anyone having success?

Yes I have, and I have installed it on other boxes too with success.

[wdv]

Originally posted by RosT
Many spam-messages I receive marked with {Spam?} in the subject. Why? How to fix this?

This is standard behaviour. If you want that the messages get deleted, check your /usr/local/etc/MailScanner/MailScanner.conf file.

[Firehosting]

In maillog i get:

Aug 19 23:43:44 free1 MailScanner[655]: MailScanner E-Mail Virus Scanner version 4.28.6 starting...
Aug 19 23:43:44 free1 MailScanner[655]: Cannot open ruleset file /usr/local/etc/MailScanner/rules/bounce.rules, No such file or directory

Where can i find these rules so i can place them there?

Many thanx,
Marcel.

[Firehosting]

I have make a blanc file of bounce.rules

Is that oke?
Seems to work...

Marcel.

[wdv]

Yep that's ok. Or just disable the lookup to the rules file in the MailScanner config.

[RosT]

I need Mailscanner don't scan outgoing messages! Only inbound. How to?

[RosT]

All works, but this software (exim+mailscanner+..) make high load to processor. How to fix this?

[jlasman]

You can use the linux nice command:

(see "man nice")

But if you do, your email processing will be delayed.

Might I recommend using RBLs (see the SpamBlocker exim.conf file here (http://www.nobaloney.net/downloads/spamblocker/DirectAdmin/), as it may help by eliminating a lot of the email before MailScanner has to deal with it.

Jeff

[RosT]

How to disable spamassasin support, and leave only virus checks and RBLs using?

[pcoeman]

Error

I receive this error. Any suggestions?

===> p5-Net-DNS-0.48 depends on file: /usr/local/bin/perl5.8.0 - found
===> Patching for p5-Net-DNS-0.48
===> p5-Net-DNS-0.48 depends on file: /usr/local/bin/perl5.8.0 - found
===> Applying FreeBSD patches for p5-Net-DNS-0.48
1 out of 1 hunks failed--saving rejects to Makefile.PL.rej
>> Patch patch-Makefile.Pl failed to apply cleanly.
*** Error code 1

Stop in /usr/ports/dns/p5-Net-DNS.
*** Error code 1

Stop in /usr/ports/mail/p5-Mail-SpamAssassin.
server# make install
===> p5-Mail-SpamAssassin-2.64 depends on file: /usr/local/lib/perl5/site_perl/5.8.0/mach/Net/DNS.pm - not found
===> Verifying install for /usr/local/lib/perl5/site_perl/5.8.0/mach/Net/DNS.pm in /usr/ports/dns/p5-Net-DNS
===> Patching for p5-Net-DNS-0.48
===> p5-Net-DNS-0.48 depends on file: /usr/local/bin/perl5.8.0 - found
===> Applying FreeBSD patches for p5-Net-DNS-0.48
1 out of 1 hunks failed--saving rejects to Makefile.PL.rej
>> Patch patch-Makefile.Pl failed to apply cleanly.
*** Error code 1

Stop in /usr/ports/dns/p5-Net-DNS.
*** Error code 1

Stop in /usr/ports/mail/p5-Mail-SpamAssassin.

[wdv]

Originally posted by RosT
How to disable spamassasin support, and leave only virus checks and RBLs using?

Check the MailScanner config file. Set 'Use SpamAssassin' to 'no'.

[wdv]

Originally posted by pcoeman
Error

I receive this error. Any suggestions?

*error*

I suggest cvsupping the ports tomorrow and try it again, probably a little glitch which will be fixed soon.

[werwin01]

Everything works fine

[westbox]

Hey,

i don't get anymore mails but mails sending works

i got the following error when restarting exim:

server1# /usr/local/etc/rc.d/exim restart
Shutting down exim: [ OK ]
Starting exim: 2005-03-23 02:34:34 Exim configuration error in line 934 of /etc/exim.conf:
unknown retry error name "="
[ OK ]

gonne past the last lines of my config including that 934line :

log_file_path = /var/spool/exim/msglog/%slog
process_log_path = /var/spool/exim/exim-process.info
queue_only = true
queue_only_override = false
spool_directory = /var/spool/exim.in

if i do ps aux |grep exim i see something wrong:

server1# ps aux |grep exim
mail 93005 0.0 0.4 2304 944 ?? Is 2:34AM 0:00.00 /usr/sbin/exim -q15m -C /etc/exim_out
root 93044 0.0 0.3 1304 708 p2 RV 2:35AM 0:00.00 grep exim (csh)

it has to be something like this:

mail 30463 0.0 0.0 3568 0 ?? IWs - 0:00.00 /usr/sbin/exim -bd -q1h -oP /var/run/exim.pid
mail 30465 0.0 0.3 3568 308 ?? Is Mon06PM 0:00.15 /usr/sbin/exim -q15m -C /etc/exim_outgoing.conf

anyone knows what to do?

Greets,
Alex Vanhecke

[icepick]

Hey guys,

I've been running on this setup (from the post) for a good couple of months with no issues, however I'm wanting to add some rules to spamassassin however can't figure out which file to modify.

I want to add something like this:

body THISISATEST /this is a test/i
describe THISISATEST We just testing
score THISISATEST 50


However each and every local.cf I add it to it doesnt work, i've alo tried to add it to /usr/share/spamassassin/10_misc.cf and still doesn't work.

Any help would be much appreciated.

Cheers
Barry

[pucky]

A far more efficient means of running MailScanner is via split directories. This should be configured in Exim as well as in MailScanner. Split directories reduces the overhead of processing mail via MS instead of dumping everyting into one directory.

[icepick]

Hey there,

This has been running just fine for over a year, however I've just encounted an error where a school emails all their students using phplist, all the email (6000 odd) is queued, scanned for viruses and scanned for spam, so I did the following in hope things would speed up:

Virus Scanning = %rules-dir%/virus.scanning.rules

In virus.scanning.rules I added:
FromOrTo: default yes
From: 127.0.0.1 no
From: x.x.x.x no (mail server IP)
From: email@mailinglist.co.nz no (mailing list email address)

I also added the following to spam.whitelist.rules
FromOrTo: default no
From: x.x.x.x yes (mail server IP)
From: 127.0.0.1 yes

Even though email is being whitelisted, it's only doing it in batches of 30 and MailScanner is still using 100% CPU even though it shouldn't be scanning virus/spam which I find weird...

How can I stop outgoing scanning?

[icepick]

I think I found the solution:
root@sophia:/usr/local/etc/MailScanner# grep report.alwaysinclude.conf ./MailScanner.conf
Always Include SpamAssassin Report = %report-dir%/report.alwaysinclude.conf


root@sophia:/usr/local/etc/MailScanner# cat rules/report.alwaysinclude.conf
FromOrTo: default yes
From: 127.0.0.1 no
From: x.x.x. no (IP ranges of mail servers)

[pucky]

If you have 6k in mail waiting to be scanned by MS then you can expect high loads, thats a guarantee and there is nothing you can do about it unless.

1) You renamed /var/spool/exim/incoming/input to something else so that those messages dont get processed or
2) You start controlling the number of incoming mail thats allowed in at once.

No matter what you do though, if you have thousands of messages in the incoming queue your going to get big loads from MS.

Another thing you can do is log into DA and go to the queue. You could try deleting the messages from there, good luck as DA is hopeless in allowing people to remove mass quantities of mail from the mail queue. I suggest 1) above then you will need to recreate input and restart exim.

[icepick]

Don't think you understand what I was trying to do... stop mailscanner scanning messages from the local server or from the LAN...

I.e. if someone uses some form of mailing list script, dont scan all those thousands of emails as I dont care about outbound email only inbound email being scanned.

The above was the solution.

P.s.
exim -bp -C /etc/exim_outgoing.conf | awk '/email@address.co.nz/ {print $3}' | xargs exim -C/etc/exim_outgoing.conf -Mrm -- empty email queue for this email address

[icepick]

Hi,

I've been using this mailscanner for nearly 2 years now and it's been great. I've since had a lot more users added to the system and now want to move mailscanner to another box or 2, has anyone done this and got it working?

Basically the 2 mailscanner boxes in front will accept mail and if not spam forward it to the internal directadmin server to the clients mailboxes. This will load balance the scanning across 2 hosts and free up lots of CPU on the DirectAdmin server