SpamBlocker3.1-beta is ready

[nobaloney]

SpamBlocker3.1-beta is ready; it fixes multiple problems including the runaway spam first noted by some users approximately 10-Feb-2008.

The files and readme details may be found at:

http://www.nobaloney.net/downloads/spamblocker/DirectAdminSpamBlocker3/

Note that this release requires Dovecot and Maildir; if you haven't upgraded yet, now is the time, as it's out of beta, available in new installs, and is much more efficient with large mailboxes than is the older mbox, especially for IMAP users and anyone who stores email on the server.

Note also that the file is no longer available in different versions depending on whether or not you run ClamAV. There's only one copy of the SpamBlocker exim.conf version 3.1-beta available.

It's no longer necessary to study the ReadMe file to find all the changes you have to make to the new file; all the change information is found directly in the file itself; simply use your favorite editor to search for the word EDIT in all caps; the comment will explain all necessary and optional changes.

If you can't install it yourself, we can do it for you; please contact me at my email address below in my sig. ; Note that I don't read private messages as often as I read my email.

Jeff

[nobaloney]

One major change (not well doucmented) is that you can now add netblocks to /etc/virtual/bad_sender_hosts, and they will work as they should.

To resolve the issues many of us have seen over the past few days, add the following:
125.110.0.0/16
This will only work after the new SpamBlocker Version 3.1-beta exim.conf file has been installed, and it WILL solve the problem many of us are having, because the 125.110.0.0/16 netblock appears to be causing much of the current problem.

Jeff

[chatwizrd]

Why does your link says 3.2 but when you go to the site its says 3.1 beta.

[tristan]

Great thanks Jeff! Is there a changelog as well so we can see what changed?

[philmcdonnell]

One major change (not well doucmented) is that you can now add netblocks to /etc/virtual/bad_sender_hosts, and they will work as they should.

To resolve the issues many of us have seen over the past few days, add the following:
125.110.0.0/16
This will only work after the new SpamBlocker Version 3.1-beta exim.conf file has been installed, and it WILL solve the problem many of us are having, because the 125.110.0.0/16 netblock appears to be causing much of the current problem.

Jeff

What is the net block doing that it is hitting the server so hard for the last several days? Was it a botnet or something? Can we remove it later so that real emails will return?

Thanks,
Phil

[ljweb]

Will the spamblocker plugin still work with this update?

[BlueNoteWeb]

Jeff -

I have installed your new spamblocker file but I'm still seeing spams get through by using a fake HELO line that purports to be an IP on the server. If you'd like some log samples for analysis I would be happy to provide those to you privately.

For now I have disabled the popb4smtp system. This caused a few support tickets from users who could not send mail, but once they updated their settings to authenticate when sending everything looks to be cool.

[jlasman]

Why does your link says 3.2 but when you go to the site its says 3.1 beta.
Because I made a mistake :(.

Fixed now.

Jeff

[jlasman]

Great thanks Jeff! Is there a changelog as well so we can see what changed?
Unfortunately no :(. You can do a "diff" but some sections may be too far out of alignment. I was much too busy finishing it to document the changes, but it's well documented internally, including a word EDIT near every suggested/mandatory edit point.

I will document the eventual final product.

Jeff

[jlasman]

What is the net block doing that it is hitting the server so hard for the last several days? Was it a botnet or something? Can we remove it later so that real emails will return?
Remove it at your own risk; if anyone is blocked by it they can contact your whitelist page :).
I'm still not sure what it was doing or why, but it was (and perhaps still is) very aggressive.

In many cases it was using an empty sender, which by RFCs we're supposed to accept since otherwise we can't get MAILER-DAEMON reports.

We were getting as much spam in a day as we usually get in a month; that's a 30x increase. I spent a lot of time making a fix.

Until we had it ready we were brute-force deleting everything in our queues every seven minutes just to give our email a minimal chance.

That's the beauty of the bad_sender_hosts file though; you can change it in realtime while watching your logfiles.

Until the next update or final release note that for highest efficiency the network-block entries should go at the top before any other entries.

Jeff

[jlasman]

Will the spamblocker plugin still work with this update?
When I spoke to Onno yesterday he said he wasn't sure but he is working on it now. Believe me, it's important even before SpamBlocker Plugin is ready.

Jeff

[jlasman]

I have installed your new spamblocker file but I'm still seeing spams get through by using a fake HELO line that purports to be an IP on the server. If you'd like some log samples for analysis I would be happy to provide those to you privately.
I don't need any now; try this first:

The HELO line shouldn't matter if the bad_sender_hosts_ip is doing it's job; make sure you've got netblocks at the top of your /etc/virtual/bad_sender_hosts file in the format:
125.110.0.0/16
We're working on code to block on phoney HELOs but it's not quite ready yet :(. However that would replace netblocks in certain instances; the netblocks should work fine; they do for us. Don't forget to put them at the top of the file.
For now I have disabled the popb4smtp system. This caused a few support tickets from users who could not send mail, but once they updated their settings to authenticate when sending everything looks to be cool.
Again, this shouldn't matter (really) because authentication is based on the actual IP# of the server, and NOT how it HELOs.

Jeff

[webquarry]

Hmm... after installing this version of spamblocker, I have users complaining when they run large lists on phplist (30K recipients and more) that it crawls while getting the mail out.

Indeed, the queue for exim is huge and full of their emails but the machine load is low.

Is there something in this new exim.conf that limits the number of children that can be spawned?

[Splet]

HELO checks should permit SMTP authenticated users to send mail even if HELO is a single word. When my clients send mail via Outlook, they use their windows computer name in HELO command, which is almost always a single word.

Easy to do this in Postfix, but don't know much about Exim.

[drewishus]

Hi,
I'm looking at moving up to the 3.1 code, but saw the info regarding only support for Maildir, no mbox. Are there any plans to move support to mbox? I realize that Maildir is probably a superior way of dealing with large files, but I've got a lot of users who are using OpenWebmail, and that does not support maildir natively.
I'm hoping you'll decide to support mbox! Keep up the great work either way.
Rock,

AC

[chatwizrd]

Hi,
I'm looking at moving up to the 3.1 code, but saw the info regarding only support for Maildir, no mbox. Are there any plans to move support to mbox? I realize that Maildir is probably a superior way of dealing with large files, but I've got a lot of users who are using OpenWebmail, and that does not support maildir natively.
I'm hoping you'll decide to support mbox! Keep up the great work either way.
Rock,

AC

Openwebmail can be patched to work with maildir.

Q: Does Open WebMail support maildir and vpop3mail user by Qmail?
A: The official release does still not support maildir, however,
Laurent Frigault, lfrigault.AT.users.sourceforge.net has made patched
Open WebMail 2.01 to support maildir. It is available at
http://www.agneau.org/openwebmail/openwebmail-2.01-storage-20031027.tgz

[jlasman]

Hmm... after installing this version of spamblocker, I have users complaining when they run large lists on phplist (30K recipients and more) that it crawls while getting the mail out.

Indeed, the queue for exim is huge and full of their emails but the machine load is low.

Is there something in this new exim.conf that limits the number of children that can be spawned?
Yes: check exim.org for the definitive information on what these limits mean:
message_size_limit = 20M
smtp_receive_timeout = 5m
smtp_accept_max = 100
message_body_visible = 3000
print_topbitchars = true
smtp_accept_max_per_host = 10
and also:
deliver_queue_load_max = 3.0
queue_only_load = 5.0
queue_run_max = 5
Note that I never optimized this code for 30,000 emails sent without limiting, and that I don't intend to. I've suggested in another thread how we handle it for our clients. DirectAdmin is meant for use in a shared environment, I use it in a shared environment, and I wrote and tested SpamBlocker exim.conf to use in a shared environment. In a shared environment availability for everyone is more important than optimization for use for any one sender.

Jeff

[webquarry]

We have already looked at those and increased them.

The queue still takes a while to process but that is to be expected.

I'm trying to figure out why it takes so much longer for phplist to inject it's list into the queue.

The rest we can deal with.

[jlasman]

HELO checks should permit SMTP authenticated users to send mail even if HELO is a single word. When my clients send mail via Outlook, they use their windows computer name in HELO command, which is almost always a single word.

Easy to do this in Postfix, but don't know much about Exim.
We currently don't check for (or against :)) single-word helo commands in SpamBlocker3.1-beta; here's the code (note that by default it's commented out):
# IF YOU CHECK FOR VALID HELO:
# UNCOMMENT THIS SECTION
# WARNING THIS IS UNTESTED AND MAY BREAK ABILITY FOR USERS TO SEND EMAIL THROUGH YOUR SERVER
# deny message = Single word server helo name ($sender_helo_name) rather than a FQDN.
# condition = ${if ! match {$sender_helo_name}{\N^[^.].*\.[^.]+$\N}}
# deny message = IP# server helo name ($sender_helo_name) rather than a FQDN.
# condition = ${if match {$sender_helo_name} {^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\$|^\[[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\]\$} {yes}{no}}
Note the note that uncommenting MAY BREAK ABILITY FOR USERS TO SEND EMAIL THROUGH YOUR SERVER.

Before we uncomment we'll have to accept authenticated users first; we don't do that now.

Jeff

[jlasman]

Are there any plans to move support to mbox? I realize that Maildir is probably a superior way of dealing with large files, but I've got a lot of users who are using OpenWebmail, and that does not support maildir natively.
I'm hoping you'll decide to support mbox! Keep up the great work either way.
Thanks for your words of support. I just don't have the time to support multiple versions of the SpamBlocker exim.conf file.

Through the release of version 2, DirectAdmin only supported mbox, so the decision was made for me.

By the time the first 3.1-beta version was out, DirectAdmin supported both mbox and Maildir, and also some folk were using ClamAV and some weren't, so I ended up supporting four versions. Big mistake. Lots of stupid mistakes every time I did an update. The long time between the first and second beta came out is directly attributable to having four versions.

Instead, beginning with 3.1-beta, I decided to go with one version of the file, with sections that users can comment/uncomment. But I didn't do a section for mbox because I no longer run an mbox-based server, and to build one to test it would have taken more time than I currently have.

If someone else does it, and can test it, and verify that it works (remember, editable sections only, not a separate file), I'll be happy to put it into the code as an EDITable section.

Of course SpamBlocker3.1-beta is my product; it's not part of DirectAdmin until/unless John and Mark decide to make it so; perhaps you should ask them; it would be quite easy for them to rewrite the section of code required and turn it over to me for inclusion.

Or you can :).

Jeff

[jlasman]

I'm trying to figure out why it takes so much longer for phplist to inject it's list into the queue.
I'm confused. How is phplist injecting into the queue? If it's actually doing some kind of direct injection, then the issue isn't in control of exim.

Please explain more. Feel free to start another thread if you feel your issue is getting lost in this one.

Jeff

[Splet]

Before we uncomment we'll have to accept authenticated users first; we don't do that now.

Jeff

Any ideas on how to do this? I like single word helo checks, it stops a lot of spammers on my other, postfix based servers. The less we have to feed to SA, the better - and lower load.

[jlasman]

Single word helo checks will also stop your own users with (for example) Outlook. That's why we have to figure out how to accept authenticated users first.

Anyone?

Jeff

[tarquel]

BTW Just wanted to add that I've implemented this new version [had 3.0 beta previous] and this one is doing a much better job :)

If i notice any bugs tho I'll let u know :)

Nath