Greylisting Code? Anti-Dictionary-Attack code?

[jlasman]

Should greylisting code, or code to block IP#s doing dictionary attacks, be added to SpamBlocker3 before the final release?

I can do one or both, but I really don't think the anti-dictionary-attack code is important if the greylisting code is added; I think the anti-dictionary-attack code would then be redundant.

What do you think?

Jeff

[dannygoh]

I know dictionary attack and what is greylisting?

[keefe007]

As long as these are options that can be turned on or off on a per domain basis.

[xemaps]

this is useless ( always only my mind ;) )
i answered on other topics about that on the forum.

reject at smtp time is enough to do the job without heavy load.

[matrixx]

Jeff,

Can you confirm greylisting ?

I understand it as..

Mail from a new email address is temporarily rejected at MTA level and the senders email address added to a database, then when/if the sending MTA attempts delivers a second time the email address is then moved onto a whitelist.

The principal being that spammers mail servers donot attempt resending mail (or even acknowledge failed deliveries)

If this is the case then I'm definately in favour of testing further BUT I am concerned with the size of the evergrowing database that would need to be kept on each server.

Say 1000 email addresses on each server, with an ever growing whitelist and greylist then I really feel that it could eat up resources cross referencing all the time. Is a 2 million record table easy enough to look up on quickly?

I maybe wrong on this (and would be happy to be wrong) - maybe you (Jeff) or someone has experience of this?

Maybe theres a way to limit table sizes (say email addresses are whitelisted for 30 days at a time) and the greylisted email addresses are given 12 hours to re-send before they are deleted from the greylist?

I also don't see how this would limit spam received from exploited php forms on senders servers and spam from servers that have been exploited and the existing server based MTA used to send spam. I'm also not sure what percentage of spam is sent using each method - have you done any tests or seen any sites that have these figures?


Rob

[jlasman]

Originally posted by matrixx
Can you confirm greylisting ?

I understand it as..

Mail from a new email address is temporarily rejected at MTA level and the senders email address added to a database, then when/if the sending MTA attempts delivers a second time the email address is then moved onto a whitelist.

The principal being that spammers mail servers donot attempt resending mail (or even acknowledge failed deliveries)
Yes.
If this is the case then I'm definately in favour of testing further BUT I am concerned with the size of the evergrowing database that would need to be kept on each server.
I am, too, but I havent studied the issue other than to talk to others about it who says it works and saves a lot of server load. If there's interest I'll look into it. I'm probably going to implement it into our servers in any case.
Say 1000 email addresses on each server, with an ever growing whitelist and greylist then I really feel that it could eat up resources cross referencing all the time. Is a 2 million record table easy enough to look up on quickly?
I have no idea. Anyone?
I maybe wrong on this (and would be happy to be wrong) - maybe you (Jeff) or someone has experience of this?
Anyone?
Maybe theres a way to limit table sizes (say email addresses are whitelisted for 30 days at a time) and the greylisted email addresses are given 12 hours to re-send before they are deleted from the greylist?
I found two implementations for exim discussed here (http://www.tldp.org/HOWTO/Spam-Filtering-for-MX/exim-greylisting.html). Care to do the research for us, Rob :) ?
I also don't see how this would limit spam received from exploited php forms on senders servers and spam from servers that have been exploited and the existing server based MTA used to send spam.
Standard MTA servers will of course eventually lresend, but they're a very small amount of the spam on the 'net today. Most spam today comes from spambots, and this appears to be the best way to block them.
I'm also not sure what percentage of spam is sent using each method - have you done any tests or seen any sites that have these figures?
Yes, from various anti-spam lists and newsgroups. I'm not doing formal research and I haven't kept notes. I plan on eventually implementing for me, ahead of blocklists, and see how much lower blocklist use gets.

Jeff

[matrixx]

The link looks ok - I'd definately be interested in this.

Would you plan to permanently add it pre-blocklist?

I can see that adding it pre-blocklist would give a gauge as to it's effectiveness.

Because blocklists filter 'definate' spammers I was wondering if (after testing) it would come after the definate spammers have been weeded out?

R

[jlasman]

I've made no decisions :) . I haven't even decided yet if I'm going to continue to work on SpamBlocker; obviously there are people here who think they can do much better than me; perhaps it's time to let them try.

Jeff

[panamaspace]

Dear Jeff,

If no one else will say it, then I will.

Your work on SpamBlocker is TRULY appreciated by the silent majority of DA users, and we would be extremely saddened if you were to drop the project because there is but ONE vociferous complainer.

In Spanish, and attributed to Cervantes Saavedra, (though not true), we have a saying, "Los Perros Ladran, Sancho, señal que avanzamos", said by El Quijote to his trusty servant Sancho Panza. (The dogs bark which means we are making progress).

That is, we don't go unnoticed in life, because wherever we thread, we leave footprints and the dogs bark to signal our passage.

Let this dog bark, and pay it no attention.

I am willing to give Mr. X. the benefit of the doubt. He has provided some useful pointers, and I have even applied one of his recommendations to good effect. I would like to think, that he, like me, has trouble with a language that is not his native tongue and therefore expresses himself badly.

I'd say, from now on, just ignore his misdirected comments, and let the other THOUSANDS of grateful DA admins and users reward you with their sympathy and appreciation, even if it goes unspoken.

You, Mr. Lasman, hold the higher moral ground. Mr. X only serves to constantly remind us of that.

Do not stop work on the Spamblocker project. We know you are right, end of discussion.


Regards,


Carlos - PanamaSpace
Just one of the many GRATEFUL & SATISFIED SPAMBLOCKER DA users lurking on these boards.:o

p.s. I can't wait for Spamblocker 3 to go Final!!!

[BigWil]

Jeff,

You know I love your work. But I will have to say that having a database that will grow and possibly bloat isn't a good idea. Bloating is going to cause greater resource usage and on email spikes the outcome can take down a machine. Heck with ClamAV and SA running the spikes already cause a few minutes of congestion from time to time.

I think that we should use a RBL dedicated to the greylisted IPs rather than having a database on every server running spamblocker. But of course the chore in that case is that someone would have to write a utility to propagate it. And of course someone has to cough up the cash and resources to host it. Xemaps? LOL!

In the case of the non-dictionary-attack using the right SA rules seems to block that already.

So my votes are in.......

Big Wil

[jlasman]

BigWil,

I think you may misunderstand greylisting. You can't have a central blocklist for greylisting because greylisting is based entirely on email coming to your server. The first time it comes you tell it to go away and come back later.

When it comes back later you accept.

I most likely will include greylisting in the final release (I'm starting to study it tonight) because it has the possibility of taking a lot of load off the server. In fact I believe it's the easiest way to block most spam and if run before blocklists will probably cut their positives almost to zero (I'm looking forward to seeing if I'm right :) ).

I'm probably going to add anti-dictionary-attack code before the greylisting code because I believe it will keep the greylisting database smaller than it would otherwise be.

If possible both will be optional on a per-domain basis.

Jeff

[BigWil]

No that is about what I understood it to be. But the info is stored until it comes back correct? So depending on the sending mail server's settings we are looking at several hours before email delivery is tried again yes? But if the receiving server gets say 300 email delivery attempts in a minute or even worse how bloated can a database get. Pretty darn bloated is my guess.

Secondly, what if a customer wants their email right away?

Any insight is greatly appreciated.

Big Wil

[jlasman]

A few minutes to less than an hour, generally.

We won't ever know the load it will generate on your server. We're testing it on my server.

If we add greylisting you'll have to enable it on a per-domain basis, for the domains on your server. We recommend you offer greylisting to your clients but don't force them to use it.

Jeff

[BigWil]

Ok. Well as long as it is a disable until enabled scenario then I think that will be fine. Have a Happy and Safe New Years.

Big Wil

[jlasman]

I haven't been able to get it running yet; using the solution requiring a separate daemon I can't get the daemon to run on any of our servers :( . And I don't yet have a copy of exim compiled for mysql support.

So moving forward... but slowly.

Jeff

[NSteffens]

I was thinking, this grey-listing, will this happen to every email which is delivered to the mail server?

If so, then all the (real) mail servers would have to send the mail twice to get a succesfull answer right? (please correct me if i'm wrong).

If everybody (or a lot of mail servers) would use this grey listing, then wouldn't all the mail servers be very busy sending mails, checking for an answer and then resend it and then finally get his answer?

That would double the email activity?

Just some of my thoughts...

Niels

[jlasman]

That depends on how long you keep the information in the database. Once in the database, all email will go through the first time.

What would you want to do if you could eliminate almost all the spam you now get by implementing a delay? Would you like it?

Jeff

[NSteffens]

I'm not quite sure, I can't see through all the pro's and cons.

If you'd ask me, would you like a solution for all(most) your spam, offcourse the answer would be yes.

A delay... I'm not quite sure about that... Part of me says yes, but I would however like the email to go through directly... But al this spam...

I understand your difficulty about making the choice ;(...

In my case it's just a private server hosting some domains for me and some friends... So it's not a very big problem for me. Maybe others with large production servers have some good ideas....

Greetings,
Niels

[jlasman]

Originally posted by NSteffens
I'm not quite sure, I can't see through all the pro's and cons.
Neither DA staff nor I will ever propose a default solution that'll cost you problems or clients.
If you'd ask me, would you like a solution for all(most) your spam, offcourse the answer would be yes.

A delay... I'm not quite sure about that... Part of me says yes, but I would however like the email to go through directly... But al this spam...
Contrary to what anyone else may be saying, the anti-spam community consensus appears to be that greylisting is the best way to eliminate spammers today. The delay for most well-behaved servers is about five minutes. Some servers may delay up to an hour, and of course a few badly cofigured servers will give up immediately. While there's no such thing as a world without false positives these false positives will result in the email being returned to the sender with a notification that it didn't go through. Which is much better than solutions that (for example) blackhole the suspected email so no one ever sees it and senders believe you've got it. I don't like it any more than you do; I refused to consider it a few months ago when I was asked to do it.

Hovever times are changing rapidly, and today I believe it's worth the delay.

That said, it won't be in the release versioon of SpamBlocker3 because it requires a copy of exim with mysql support enabled. I can do that for my platform, CentOS, but I can't do it for every platform, so it's not being included.

When it is included it'll be configurable on a per-domain basis.
I understand your difficulty about making the choice ;(...

In my case it's just a private server hosting some domains for me and some friends... So it's not a very big problem for me. Maybe others with large production servers have some good ideas....

Actually for you it's an easier decision to use it than it is for those of us with shared hsoting servers for clients. Historically people with their own servers have always been more conservative about what they accept than those of us with commercial hosting services. We have to be liberal because we can't risk losing important customer emails.

I really hoped others would chime in constructively, but so far we've gotten only complaints but no better solutions :( .

I've got no ulterior motive at all in producting SpamBlocker for DA. Unless you call wanting to keep spam off my servers an ulterior motive.

Jeff

[Scott DeLeury]

Personally, I agree that greylisting is a wonderful front-line defense to spam. Most spammers' servers will give up after the first try when an email is greylisted, while legitimate emails will be re-sent again after a specified period of time.

There are implementations that use MySQL and there are those that don't (here's one: http://blog.adamsweet.org/?p=172)

I've been looking into this for a bit on my own for work purposes on another MTA installation, and will be implementing it when I get back from vacation on Wed.

Greylisting is a legitimate and widely used anti-spam practice these days, so why not make it a 'standard' with DA? :)

[jlasman]

I haven't been able to get greylistd work on a CentOS server, even using an RPM prepared specifically for the version of CentOS we use, which is CentOS 4.4.

In order to use a version which requires a daemon we have to have a daemon we can make work under every OS Distribution on which DA works.

That's the only reason I'm considering the MySQL version.

Jeff

[Scott DeLeury]

I'll see where I can get with it on one of my DA servers (not saying I'm going to get it to work when you didn't ;) ) but it's worth a shot :)

[jlasman]

Thanks, Scott.

It has to be foolproof on every platform DA supports or we can't make it a standard part of DA.

For several years now I've been saying DA supports too many platforms; maybe I'm actually right :D .

Jeff

[Maniak]

My vote went for both, due to the amout of email that will have to be checked, I prefer to make a restriction by greylist.

The dictionnary will tweak a bit more exim, and it is not bad at all.

[jlasman]

I've decided that neither greylisting nor anti-dictionary attack code is going into SpamBlocker3.

I'm going to put greylisting into the next verison of SpamBlocker after DA starts compiling exim with MySQL support.

Jeff

[Chrysalis]

That is a shame I have seen mail servers go down that rely on mysql due to the mysql overhead involved with the high number of connections associated with dictionary attacks.

Isnt this possible without mysql?

[jlasman]

Yes. But finding something that works without problems on all OS Distributions on which DA must work has been problematic for me.

You won't have to run greylisting.

You can certainly create your own solution, but any solution I come up with must work on all versions of DA on all distributions.

Jeff

[floyd]

I have a question and I realize that I am probably the dumbest person here. I am trying to figure out what makes a spammer's server different than any other server. Why would the spammer's server give up after the first try? What makes my servers keep trying? Why wouldn't the spammer just configure their server to resend again to get around the grey listing?

[jlasman]

Most spammer use specially build servers/deamons/worms/viruses to send email. They're specifically designed to not retry.

Why? Because if he didn't, as he sent out millions of spams, he'd be getting hundreds of thousands of undeliverable notices. He really can't be bothered with doing anything with them; it's much more efficient for the spammer to keep sending to new addresses rather than try to figure out what to do with refused email.

Jeff

[tdldp]

Hi jeff,

I was reading with interest your poll and thread concerning greylisting, as we start having difficulties with too restrictive SBL rules, that result in loosing some mails...

Concerning greylisting, i've seen working here in france an interesting method, that i find more efficient than the idea you exposed (note : no criticism of my part, i just love SB, since version 1, but want just to give a point of view)...

Indeed, your idea is that SB will accept mails on second sending from a senders adress (nothing is said, if it's IP or email)... This is not considering that spambox, often resend mails 3-4 times before sending a new email.
Well on my box and domain, i get in 24 hours, up to 10 mails totally identical with same IP / senders adress, which is spam (pharmaceutics / stocks)... If i understand well principles of greylisting, it will bypass blacklisting as soon an adress is in grey list.... Well in this case i'll get spam in my greylist, because it will pass greylisting tests, with 10 mails sent in 24 hours...

What i have seen working, is another greylisting method, based on alternative mail / web authentification...
A non listed (neither whitelist / neither greylist) sender mails to server, and is detected as new sender... A message is sent back to senders adress, specifying that the message is currently on hold, and will be delivered once authentification is done... The email adress is placed on greylisting, accepting all new mails but on hold (not delivering messages yet). In the response mail, is joined a weblink, with a sender Id that needs to be clicked or pasted in browser to validate sending. Sender has 24 - 72 hours (user setted), to respond to email by clicking on the link (sending on a php page)...Once this validation is passed, sender is added to whitelist. Any new mails sent by the same email adress will be accepted by server, with no new authentification process. If no action is undertaken under time given, senders IP and/or adress is added to autoblacklist...
This principle is used in many companies here in france, and announces false positives at zero, and 99.9999 % spam efficiency.

This solution is in france a commercial application, and i am unable to give you more details as not using it (i don't know if it sql based, if it works inside a panel, if it is reproductible), but i find this idea interesting and if you were to do something in this way, i would think this would be the best method to block spam really efficiently... It is post rbl controls and therefore is i suppose usable in exim...

I'll be trying in a few weeks to integrate this by myself in exim, (once all the other work on fire is done) and will keep you up with what i'll have found...

Tdldp

[jlasman]

Lots of companies sell what you describe. It's even got a name:

Challenge-Response

Using it makes you a spammer. And I can't add it to SpamBlocker because then you and I would be spammers.

To find out why I think so, read this wikipedia article (http://en.wikipedia.org/wiki/Challenge-response_spam_filtering#Criticisms).

Jeff

[floyd]

I agree with you 100% Jeff. I had started writing a reply post to this but decided it was too long. Now that it has been a couple of days I have calmed down some.

Challenge response systems cause way too much email traffic and I will not use them even when I am the one being challenged. The only ones that I have seen use them are people sending me email. I have never been challenged when I send the initial email. I have only been challenged on replies to emails. If somebody is not smart enough to whitelist me before they email me then they just will not get a response from me. Half the time I cannot even read the letters and numbers on the graphic. I hate them. I abhor them. I will not use them ever.

[tdldp]

Lots of companies sell what you describe. It's even got a name:

Challenge-Response

Using it makes you a spammer. And I can't add it to SpamBlocker because then you and I would be spammers.

To find out why I think so, read this wikipedia article (http://en.wikipedia.org/wiki/Challenge-response_spam_filtering#Criticisms).

Jeff

After reading your post, and the wiki link, i agree that it is maybe not best solution, yet it is interesting... I think it could be interesting after all solutions you deploy in exim... Personnaly, i'll give it a test, and let you know how it works out on our system...

[floyd]

If you use it you will lose mail because people like me will not answer the challenge.

[jlasman]

Not to mention that many people who can will put your IP# into either SpamCop or SORBS.

SpamCop not so bad, because it comes off after a while. SORBS makes you donate money (up to $25 for each piece of spam they believe you've sent) before they'll remove you.

Jeff

[thor]

Jeff,

I know you, and many others here, hate Challenge Response / Whitelisting, but I have to say for me it has worked great for years. Filters and SA block legit mail and allow some garbage through. C/R does not do that except in a blue moon when a spammer falsely sticks in a legit from address with an auto-response activated (for BoxTrapper which does not use a CAPTCHA webpage, but just a "reply to this message" approach).

I have a cPanel server with BoxTrapper and a DA server. I MUCH prefer DA overall, but cannot move all my domains over due to the BoxTrapper reliance for some address I, and my clients, get slammed with spam on and must have some form of C/R. I wish DA had a solution as such.

I have to disagree with you that it turns me into a spammer, because 90% of the time, spammers use bogus "from" / "reply to" email address and the mail will die after a few failure attempts at delivery (thus the greylisting idea). Do some spammers stick some poor unsuspecting users "from" address in there sometimes, yes. And that is a flaw, agreed. But I don't think it makes me the spammer? It is just like a doorbell on a house with a locked door. Who are you at my door? A salesman? go away... If a few people say, I did not knock on your door, that is a small casualty of something we all abhor.

However, one element of whitelisting that is vital is an auto whitelist of outgoing addresses. If I send a mail to someone (To, CC or BCC) it MUST add those addresses to the whitelist. Otherwise, as Floyd stated, it is entirely rude to send a reply to someone and then have to answer a C/R.

I wish and hope DA would implement a blacklist that just blackholes the mail on the blacklist, a whitelist that everything gets through on the list, an auto whitelist for outgoing mails, and then either a greylist as described in this thread (if you are firmly opposed to C/R), or a C/R with a CATCHA web page.

Make it all optional on a PER ADDRESS basis so I, and my users, can configure their own settings.

And if someone sends me a message that I have never emailed, and they do not want to whitelist themselves, I can always release their email from the holding bin. (and adding this feature to the greylist functionality would allow those "got to get this mail NOW" through if the recipient goes to the web interface to check the queue.)

Adding a configuration to a greylisting feature that says release on attempt 2,3, or 4, would allow you to increase the number of tries needed to resend if a spammers starts resending emails to try and defeat the greylist...

Anyway, just my thoughts. I love Challenge Response, because it keeps my inbox clear, and I can always check my queue. But I agree autowhitelisting MUST be included...

I just widh DA had either greylisting or C/R, with the other features (queue checking, auto whitelisting, etc)

As far as my IP being listed as spam, it has not happened in the years I have used C/R...

For guidelines on how to best deploy C/R to overcome hurdles, from the wikipedia site, http://www.templetons.com/brad/spam/challengeresponse.html

[floyd]

Business mail filtering of any kind is unacceptable. Either accept the mail or reject the mail. There is far too much money at stake to risk losing even one legitimate email. I will spend the 10 minutes per day deleting the spam that gets by the blocklists.

[thor]

I was not speaking of business mail filtering. That is why is should be 100% optional PER ADDRESS. One would be silly to turn that on for a business address. I totally agree too much $ is at stake to do that on a business address. However, for personal addresses.....

Any maybe, a hybrid system would be good too... start out with the greylist function, and on the second delivery attempt, THEN issue the Challenge Response. To cut down on all the excess spam mail being sent challenges...

[jlasman]

We will continue to disagree. Vehemently.

I will not add any kind of challenge response system to my SpamBlocker exim.conf file. If DA decides to use it they can make any changes to it they wish, but I won't use any changes that require challenge response.

I almost never respond to a challenge-response request.

Sending challenge-response requests will at some point get you listed in SORBS and other blocklists as well. SORBS charges money to remove you.

Don't forget that if you use form-to-email forms on your sites, then no forms will ever get to you.

more below...

I know you, and many others here, hate Challenge Response / Whitelisting, but I have to say for me it has worked great for years.
More correctly the entire anti-spam community hates Challenge/Response. If you use it you will eventually be bit by this fact.
Filters and SA block legit mail and allow some garbage through. C/R does not do that except in a blue moon when a spammer falsely sticks in a legit from address with an auto-response activated
Many spammers use real from/to addresses from the same domain; you've probably got those addresses whitelisted already.
I wish DA had a solution as such.
Then write one :) .
I have to disagree with you that it turns me into a spammer, because 90% of the time, spammers use bogus "from" / "reply to" email address and the mail will die after a few failure attempts at delivery (thus the greylisting idea). Do some spammers stick some poor unsuspecting users "from" address in there sometimes, yes. And that is a flaw, agreed.
Your numbers were probably correct a few years ago. Today most spam comes from infected machines. The bots on those machines always use real addresses on those machines as the from address on those spams. If everyone used Challenge/Response then those return addresses would get millions of responses.
But I don't think it makes me the spammer?
You can think what you want. If you send unsolicited email to me in response to spam that didn't come from me, you're a spammer, and will be treated as such. And I'm not in the minority.
It is just like a doorbell on a house with a locked door. Who are you at my door? A salesman? go away... If a few people say, I did not knock on your door, that is a small casualty of something we all abhor.
You don't really see anything wrong with that analogy? Oh well. I can't convince you then, because you're blind to what's really happening.
I wish and hope DA would implement a blacklist that just blackholes the mail on the blacklist, a whitelist that everything gets through on the list,
Both of these are installed now, and have been for some time, as part of the SpamBlocker exim.conf file currently installed in DirectAdmin.
an auto whitelist for outgoing mails,
A serverwide whitelist with automatic additions is going to get very large very fast. Would you recommend that it be implemented in MySQL or as a flat file? Either one could cause large server loads when hit with dictionary attack email, as whitelists have to be checked before blocklists, so all incoming email must first be run through the whitelist. I'm not sure this is viable for many of us.
and then either a greylist as described in this thread (if you are firmly opposed to C/R), or a C/R with a CATCHA web page.
I'm not going to add Challenge/Response; someone else can but I won't use it or support it. If you think that CAPTCHA works, then please explain how automated bots still manage to post spam on these forums.
Make it all optional on a PER ADDRESS basis so I, and my users, can configure their own settings.
DA would have to decide what to add to the Control Panel; I only write the exim.conf file which so far, anyway, DA has chosen to use. There are two sources for plugins controlling SpamBlocker; perhaps one of them would want to do what you're suggesting. Note that per-address checking makes for huge flat files, huge configuration files, or complex MySQL setups.
And if someone sends me a message that I have never emailed, and they do not want to whitelist themselves, I can always release their email from the holding bin.
How does it take less time to look at email in the holding bin than it does in the inbox?
(and adding this feature to the greylist functionality would allow those "got to get this mail NOW" through if the recipient goes to the web interface to check the queue.)
Here you've lost me. Greylisting happens before you know anything about the email, before you've been able to do anything with the mail. Are you saying we should Greylist the mail but accept it anyway? How would we do that?
Adding a configuration to a greylisting feature that says release on attempt 2,3, or 4, would allow you to increase the number of tries needed to resend if a spammers starts resending emails to try and defeat the greylist...
Okay, so now we're talking thousands of hits per minute (on a busy server) to a database or flatfile?
Anyway, just my thoughts. I love Challenge Response, because it keeps my inbox clear, and I can always check my queue.
Which takes as long as checking my inbox.
I just wish DA had either greylisting or C/R, with the other features (queue checking, auto whitelisting, etc)
I plan on adding greylisting to SpamBlocker at some point soon. The rest won't come from me but of course you can do it and offer it to the rest of us, just as I've written SpamBlocker and given it to the rest of us.
As far as my IP being listed as spam, it has not happened in the years I have used C/R...
Which certainly doesn't mean it won't.

Jeff

[smtalk]

jlasman, any release date set for the beta2?

[thor]

I can check the queue once a week if I want, not get interupted on my desktop with a new spam every 10 minutes...

And if you use an C/R on a form address, your silly.

Bottom line, it has worked for me and I love it.

If the anti-spam community hates it... too bad. There is nothing else that works nearly as well for my inbox. And at the end of the day, that is all I care about.

If greylisting works, fine, I will use that. But filters are just not cutting it.

[floyd]

I am curious as to what people call "a lot of spam." I know that in a 24 hour period the spam that makes it through the blocklists to my Inbox is about 300 to 400. I can delete these in about 5 to 10 minutes by simply looking at the subject line. So to me that is not a lot of time to spend filtering my own mail compared to the trouble that a C/R system or any filtering system causes.

So how many spam emails do you have to get to then resort to a C/R system? And when you resorted to a C/R system were you currently using a blocklists system?

[jlasman]

Unfortunately not; we've gotten hung up on some work and far behind in billing :( .

Jeff

[Przemek]

I haven't been able to get greylistd work on a CentOS server, even using an RPM prepared specifically for the version of CentOS we use, which is CentOS 4.4.


I've installed greylistd successfully on CentOS 4 server today. Here's what I did:

rpm -Uvh http://dl.atrpms.net/all/greylistd-0.8.3.2-8.el4.at.noarch.rpm

in /etc/greylistd/config change:
mode = 0660
to
mode = 0666
because greylistd is running as user greylistd and we need to read/write to its socket as user mail.

Then in /etc/exim.conf I've found:
# accept mail to errors@example.com, regardless of source
# accept local_parts = errors
# domains = example.com
and added following code below it:

#GREYLIST
defer message = $sender_host_address is greylisted
log_message = greylisted.
!hosts = +relay_hosts : \
${if exists {/etc/greylistd/whitelist-hosts}\
{/etc/greylistd/whitelist-hosts}{}}
hosts = !+relay_hosts
domains = +relay_domains
!senders = : postmaster@*
set acl_m6 = $sender_host_address $sender_address $local_part@$domain
set acl_m6 = ${readsocket{/var/run/greylistd/socket}{$acl_m6}{5s}{}{false}}
condition = ${if eq {$acl_m6}{grey}{true}{false}}


Then we start greylistd and restart exim:

service greylistd start
service exim restart


Now we can check if greylistd is running:

greylist stats



Statistics since Sat Mar 3 13:35:23 2007 (1 hour and 31 minutes ago)
---------------------------------------------------------------------
49 items, matching 65 requests, are currently whitelisted
0 items, matching 0 requests, are currently blacklisted
1047 items, matching 1158 requests, are currently greylisted

Of 49 items that were initially greylisted:
- 49 (100.0%) became whitelisted
- 0 ( 0.0%) expired from the greylist


Tested successfully on 2 CentOS boxes.

BTW: I also modified some other config settings in order to delay messages for shorter time. In /etc/greylistd/config:

retryMin = 300
retryMax = 86400

[smtalk]

Przemek, great work! I appreciate that.

[jlasman]

I'll give it a try.

Jeff

[GranTW]

Works perfect with my CentOS 4.4 system.

Just a note I put the greylisting code just under the final RBL check so that alot of spam gets caught before ever being greylisted.

[jlasman]

Since greylisting takes less resources I'd run it first.

Jeff

[smtalk]

jlasman, have you tried it (greylisting)? If yes, is there any release date set (for SpamBlocker v3)?

[dec]

I would like to know if it is safe to use this version of Spamblocker in production servers. Any reason not to use it? Should i wait more?

Thanks,

[jlasman]

No, we've not had time to test greylisting. I should get the time to build it on a testbed server next week.

Safe to use which version in production servers? The beta? Absolutely; we've been using it since a week or so before I posted it.

Jeff

[wallacetan]

Greylisting should not be applied to all incoming emails.
It should be use as one of the methods to prevent spam.

I only apply greylisting to suspicious sender's IP.

Suspicious IPs are:
1. Without reverse hostname
2. Reverse hostname does not point back to same IP
3. Reverse hostname is dynamic, e.g. 1-1-168-192.dialuppool.domain

The other cheap spam prevention method I use is checking for valid SMTP HELO.

These 2 methods fiters out more then 90% of spam before SMTP DATA.
The rest can be handled by more expensive process, i.e. ClamAV and SpamAssassin.

Implementation Notes:
I use Berg's exim-greylist, see url:
http://johannes.sipsolutions.net/Projects/exim-greylist/

I've installed exim-greylist on secondary MX server running CentOS 5.0 without DirectAdmin.
Using the default exim 4.63 installed by "yum install exim" which comes with mysql support.

/etc/exim/exim_dynamic_regex is file containing regex matches to dynamic IP's reverse hostname
# Example: (1-1-168-192.dialuppool.domain.)
# Example: (d-1.1.168.192.dialuppool.domai)
See url: http://www.linuxmagic.com/opensource/anti_spam/dynamic_regex/

Example exim.conf

acl_check_rcpt:

accept hosts = : +relay_from_hosts

accept authenticated = *

accept condition = ${if or {\
{eq {$interface_port}{465}}\
{eq {$interface_port}{587}}\
}{yes}{no}}
endpass
message = relay not permitted, authentication required
authenticated = *

deny domains = +local_domains
local_parts = ^[.] : ^.*[@%!/|]

deny domains = !+local_domains
local_parts = ^[./|] : ^.*[@%!] : ^.*/\\.\\./

######################################################################
# HELO checks
######################################################################

# HELO is empty or not sent
deny message = You have sent no HELO! Please see RFC 2821 section 4.1.1.1
log_message = Bad HELO: Empty HELO
condition = ${if eq{$sender_helo_name}{}}
delay = 30s

# HELO is not a fully qualified domain name
deny message = Your mail server announcement ($sender_helo_name) \
is a single word rather than a FQDN. This is in breach of RFC2821
log_message = Bad HELO: Not FQDN
condition = ${if match {$sender_helo_name}{\\.}{no}{yes}}
delay = 30s

# IP Only is sent as the HELO
deny message = Your server announces itself ($sender_helo_name) with a plain IP address which is in breach of RFC2821.
log_message = Bad HELO: IP Only Announce
condition = ${if isip{$sender_helo_name}{yes}{no}}
delay = 30s

# Someone is trying to spoof your own IPs!
deny message = HELO/EHLO IP is local. You are not this server.
log_message = Bad HELO: Local IP Spoof Attempt
condition = ${if eq{$sender_helo_name}{localhost}{yes}{no}}
delay = 30s

# Someone is trying to spoof a domain on the server
deny message = Forged HELO: you are not $sender_helo_name
log_message = Forged HELO: $sender_helo_name Spoof Attempt
condition = ${if match_domain{$sender_helo_name}{+local_domains}{yes}{no}}
delay = 30s

######################################################################
# GREYLIST checks
######################################################################
.ifdef GREYLIST_ENABLED

# Reverse Host Lookup Failed
defer !senders = : postmaster@*
domains = +local_domains : +relay_to_domains
condition = ${if eq{$host_lookup_failed}{1}}
acl = greylist_acl
message = greylisted - try again later
log_message = greylisted_1 - host_lookup_failed [$host_lookup_failed]

# Reverse Host Lookup Deferred
defer !senders = : postmaster@*
domains = +local_domains : +relay_to_domains
condition = ${if eq{$host_lookup_deferred}{1}}
acl = greylist_acl
message = greylisted - try again later
log_message = greylisted_2 - host_lookup_deferred [$host_lookup_deferred]

# Reverse DNS Rejected - dynamic ip
defer !senders = : postmaster@*
domains = +local_domains : +relay_to_domains
condition = ${lookup{$sender_host_name} nwildlsearch {/etc/exim/exim_dynamic_regex} {yes}{no}}
acl = greylist_acl
message = greylisted - try again later
log_message = greylisted_3 - dynamic ip

.endif

[mattb]

Since greylisting takes less resources I'd run it first.

Jeff

Hi Jeff,

There is two general thoughts with greylisting.

1. Run Greylisting then DNSBL.
Pro: This has the advantage of limiting the amount of external callouts the system needs to do.
Con: This can mean your greylisting DB can grow quite large. A decent greylistd will normally 'clean out' redundant entries over a given time.

2. DNSBL then greylisting
Con: Means more external calls are undertaken. This can mean more processing/load is required.
Pro: Less to be processed by the greylisting server, thus a small DB is generated.


It's most common to do the greylisting last, as it means all DNSBL checks have been done (and passed) so that the 2nd attempt the user will breeze on past.

Greylisting known spam (ie: stuff caught be DNSBLs), is just replacing their function.... and likely to just add additional disk usage.

I'm neither for/against either method.
Speaking to those that regularly use/maintain a greylist server, it does seen the 2nd approach is often used more often. :confused:

[mattb]

I've installed greylistd successfully on CentOS 4 server today. Here's what I did:

rpm -Uvh http://dl.atrpms.net/all/greylistd-0.8.3.2-8.el4.at.noarch.rpm


I'm testing this and will be putting up a DirectAdmin plugin that end-users can effectively add their own whitelist entries to. :D

[mattb]

/etc/greylistd/config change:
mode = 0660
to
mode = 0666
because greylistd is running as user greylistd and we need to read/write to its socket as user mail.


I imagine assigning exim to the actual greylist group it would resolve this and not make it world writeable? So you can keep the 0660 perm then.

I'll take a peep later today.

So far so good... it's a nice simple and elegant solution. :D


Statistics since Mon Apr 21 12:00:06 2008 (1 hour and 6 minutes ago)
--------------------------------------------------------------------
3 items, matching 3 requests, are currently whitelisted
0 items, matching 0 requests, are currently blacklisted
8 items, matching 12 requests, are currently greylisted

Of 3 items that were initially greylisted:
- 3 (100.0%) became whitelisted
- 0 ( 0.0%) expired from the greylist


NB: I'm doing DNSBLs first prior to greylisting. So it knocks a lot out first (80-90%).

Cheers,

Matt.

[jlasman]

But doesn't that run all emails through the DNSBLs twice?

Jeff

[Chrysalis]

I played with helo checking and I found legitamate emails blocked, senders particurly outlook users have bad helos that get caught by these checks.

[jlasman]

Which is why by default we do not do helo checking in the SpamBlocker exim.conf files delivered with DirectAdmin.

If you're going to do help checking your users will all have to use port 587 to send outbound email through your server.

Jeff

[wallacetan]

HELO checks can be applied if:
1. Your users are using POP before SMTP
OR
2. Your users are using SMTP authentication

POP before SMTP
-----------------
/etc/virtual/pophosts file is created by DirectAdmin.
It contains IP addresses of authenticated POP3 connections.

SMTP authentication
--------------------
Jeff's SpamBlocker also allows SMTP authentication.
However, your users have to setup SMTP authentication on their email client, e.g. Outlook.
Most ISP I know require SMTP authentication, as this is the preferred method to secure against unauthorized or open relay.

Important Note:
---------------
Do not check HELO in acl_check_helo:
See post: http://www.directadmin.com/forum/showpost.php?p=128834&postcount=45

Check HELO in acl_check_rcpt: after "accept hosts" and "accept authenticated"
See code below:


acl_check_rcpt:
accept hosts = : +relay_from_hosts : net-lsearch;/etc/virtual/pophosts
accept authenticated = *
######################################################################
# HELO checks below
######################################################################

[wallacetan]

Hi Jeff,

While reading your SpamBlocker's exim.pl code and thinking about your concern about the default DirectAdmin's exim rpm is without MySQL support.

I think you can use perl's database(MySQL) support to implement Berg's exim-greylist.

I am new to perl, but I would be glad to help writing a few new functions for exim.pl and some exim.conf ACL rules.

Wallace

exim.pl
http://files.directadmin.com/services/exim.pl

That said, it won't be in the release versioon of SpamBlocker3 because it requires a copy of exim with mysql support enabled. I can do that for my platform, CentOS, but I can't do it for every platform, so it's not being included.

[jlasman]

HELO checks can be applied if:
1. Your users are using POP before SMTP
OR
2. Your users are using SMTP authentication

POP before SMTP
-----------------
/etc/virtual/pophosts file is created by DirectAdmin.
It contains IP addresses of authenticated POP3 connections.

SMTP authentication
--------------------
Jeff's SpamBlocker also allows SMTP authentication.
Perhaps calling it Jeff's SpamBlocker is a bit of a misconception; the exim.conf file provided by DirectAdmin for years now is an earlier version of my file. So by now everyone who's ever updated DirectAdmin since it originally came out should be using a SpamBlocker exim.conf file, though not necessarily the latest version.
However, your users have to setup SMTP authentication on their email client, e.g. Outlook.
Most ISP I know require SMTP authentication, as this is the preferred method to secure against unauthorized or open relay.
I'm not sure which ISPs you write of; in the U.S. there are two models of ISPs: those who use their own networks and those who share networks. Those who use their own networks generally accept as authenticated users all who reach their email servers from within their network(s), while those who share networks (often reseller ISPs) generally require SMTP authentication or POP before SMTP authentication. While yes, POP before SMTP has a few holes in it, they're not easily exploitable and POP before SMTP is still a reasonable method for authentication. If you abandon it, you end up with a huge customer support surge as many of your clients can no longer send mail.

All that said, most of us who use DirectAdmin aren't ISPs, but rather webhosts, and none of our clients come from within our networks, so we must require all our users to authenticate by either SMTP authentication or POP before SMTP.
Important Note:
---------------
Do not check HELO in acl_check_helo:
See post: http://www.directadmin.com/forum/showpost.php?p=128834&postcount=45
The above makes sense.
Check HELO in acl_check_rcpt: after "accept hosts" and "accept authenticated"
See code below:
acl_check_rcpt:
accept hosts = : +relay_from_hosts : net-lsearch;/etc/virtual/pophosts
accept authenticated = *
######################################################################
# HELO checks below
######################################################################

However the above doesn't make sense to me because our exim.conf files (old, new, official, or beta) do NOT include an acl called acl_check_rcpt, so there's nowhere to put the code you mention.

You could probably use +relay_hosts instead of net-lsearch;/etc/virtual/pophosts, but depending on where you put this you could be bypassing some of the checks we consider important.

Would you please be so kind as to actually look at the SpamBlocker version 3.1-beta code here (http://www.nobaloney.net/downloads/spamblocker/DirectAdminSpamBlocker3/exim.conf.3.1-beta), and tell us where to put your code? Instead of creating a new acl_check_rcpt you can probably put it somewhere under the check_recipient acl (defined as acl_smtp_rcpt) I've already implemented. At that time I can probably simplify the rest of that acl quite a bit as well.
Originally Posted by jlasman
That said, it won't be in the release versioon of SpamBlocker3 because it requires a copy of exim with mysql support enabled. I can do that for my platform, CentOS, but I can't do it for every platform, so it's not being included.
That post of mine you quoted is over 16 months old; we need to check with DirectAdmin staff to find out if MySQL support is now included in Exim; my understanding is that it now is.

Is it, John?

wallacetan, I look forward to your further input. I'll take all the help I can get :) .

Jeff

[DirectAdmin Support]

Hello,

No, not to my knowledge.

Since we don't use mysql with exim, and since people update mysql versions all the time, we'd have a huge headache trying to keep up with things. IE: if someone chose mysql 4 instead of mysql 5 at install time..we'd have to have 2 sepeate exim rpms per os... which would be unreasonable. Plus when one particular mysql version gets updated, then a full exim recompile would be needed.

Anyway, bottom line, if you want MySQL in exim, you're more than welcome to compile it yourself, as it is totally open source software.
For us to do it would be nearly impossible anyway due to the vast number of different MySQL install options.

John

[wallacetan]

Hi Jeff,

Thanks for your feedback.

I will base my suggestions on SpamBlocker3,
http://www.nobaloney.net/downloads/spamblocker/DirectAdminSpamBlocker3/exim.conf.3.1-beta

1. Set acl_check_helo: section to below
acl_check_helo:

# In this pass, we do not perform any checks here.
accept

2. In check_recipient: section, insert SMTP HELO checks
# ACL that is used after the RCPT command
check_recipient:

<snip>

# envelope senders whitelist
# accept if envelope sender is in whitelist
accept senders = +whitelist_senders

######################################################################
# Start HELO checks #
######################################################################

# HELO is empty or not sent
deny message = You have sent no HELO! Please see RFC 2821 section 4.1.1.1
log_message = Bad HELO: Empty HELO
condition = ${if eq{$sender_helo_name}{}}
delay = 30s

# HELO is not a fully qualified domain name
deny message = Your mail server announcement ($sender_helo_name) \
is a single word rather than a FQDN. This is in breach of RFC2821
log_message = Bad HELO: Not FQDN
condition = ${if match {$sender_helo_name}{\\.}{no}{yes}}
delay = 30s

# IP Only is sent as the HELO
deny message = Your server announces itself ($sender_helo_name) with a plain IP address which is in breach of RFC2821.
log_message = Bad HELO: IP Only Announce
condition = ${if isip{$sender_helo_name}{yes}{no}}
delay = 30s

# Someone is trying to spoof a local domain on the server
deny message = Forged HELO: you are not $sender_helo_name
log_message = Forged HELO: $sender_helo_name Spoof Attempt
condition = ${if match_domain{$sender_helo_name}{+local_domains}{yes}{no}}
delay = 30s

######################################################################
# End HELO checks #
######################################################################

# accept mail to postmaster in any local domain, regardless of source
accept local_parts = postmaster
domains = +local_domains

# accept mail to abuse in any local domain, regardless of source
accept local_parts = abuse
domains = +local_domains

# accept mail to hostmaster in any local domain, regardless of source
accept local_parts = hostmaster
domains =+local_domains

<snip>

[wallacetan]

Hi John,

How about creating an article in DirectAdmin Knowledge Base for compiling Exim with MySQL from source?

I have found a few posts and enquiries on Exim with MySQL support.

I am planning to re-compile Exim with MySQL support, but the information is somewhat incomplete.

It would be most helpful if there is a mini-howto for this, similar to:

DirectAdmin Knowledge Base - How to compile exim from source
http://help.directadmin.com/item.php?id=125

Exim with mysql support
http://www.directadmin.com/forum/showthread.php?t=24510&highlight=exim+mysql

Thanks!

Wallace

[jlasman]

Hi Jeff,

Thanks for your feedback.

I will base my suggestions on SpamBlocker3,
http://www.nobaloney.net/downloads/spamblocker/DirectAdminSpamBlocker3/exim.conf.3.1-beta

Thanks for your response; the code in your post looks good except that I don't see where you accept authenticated users first.

Please help me see the error of my ways so I can make the change :) .

Thanks.

Jeff

[jlasman]

Hi John,

How about creating an article in DirectAdmin Knowledge Base for compiling Exim with MySQL from source?
I think that's going to be a bit hard for folk to keep up with.

How about the post someone (you?) made to do the mysql support in exim.pl?

Or would that somehow slow down the server?

If I recall correctly the reason I abandoned greylisting was after a few tries I still didn't see an easily install one way for everyone solution.

And as an aside, nolisting (search these forums) seems to work well; will greylisting really add something to the mix after using nolisting?

Jeff

[wallacetan]

Opps.. I missed that, thanks for pointing it out.

I have excluded HELO checks for:
1. relay_hosts (POP before SMTP)
2. authenticated users (SMTP authentication)

Where exim.conf main section is:
hostlist relay_hosts = net-lsearch;/etc/virtual/pophosts : 127.0.0.1

check_recipient:

<snip>

# envelope senders whitelist
# accept if envelope sender is in whitelist
accept senders = +whitelist_senders

######################################################################
# Start HELO checks #
######################################################################

# HELO is empty or not sent
deny message = You have sent no HELO! Please see RFC 2821 section 4.1.1.1
log_message = Bad HELO: Empty HELO
hosts = !+relay_hosts
!authenticated = *
condition = ${if eq{$sender_helo_name}{}}
delay = 30s

# HELO is not a fully qualified domain name
deny message = Your mail server announcement ($sender_helo_name) \
is a single word rather than a FQDN. \
This is in breach of RFC2821
log_message = Bad HELO: Not FQDN
hosts = !+relay_hosts
!authenticated = *
condition = ${if match {$sender_helo_name}{\\.}{no}{yes}}
delay = 30s

# IP Only is sent as the HELO
deny message = Your server announces itself ($sender_helo_name) \
with a plain IP address which is in breach of RFC2821.
log_message = Bad HELO: IP Only Announce
hosts = !+relay_hosts
!authenticated = *
condition = ${if isip{$sender_helo_name}{yes}{no}}
delay = 30s

# Someone is trying to spoof a local domain on the server
deny message = Forged HELO: you are not $sender_helo_name
log_message = Forged HELO: $sender_helo_name Spoof Attempt
hosts = !+relay_hosts
!authenticated = *
condition = ${if match_domain{$sender_helo_name}{+local_domains}{yes}{no}}
delay = 30s

######################################################################
# End HELO checks #
######################################################################

# accept mail to postmaster in any local domain, regardless of source
accept local_parts = postmaster
domains = +local_domains

# accept mail to abuse in any local domain, regardless of source
accept local_parts = abuse
domains = +local_domains

# accept mail to hostmaster in any local domain, regardless of source
accept local_parts = hostmaster
domains =+local_domains

[wallacetan]

As I am new to perl, I am not as confident, but I can give it a try.
I have written some scripts in PHP and BASH.

My preference would be to use Exim with MySQL support.
I would think Exim with MySQL support would perform better then using exim.pl

How about the post someone (you?) made to do the mysql support in exim.pl?

Or would that somehow slow down the server?

[jlasman]

Opps.. I missed that, thanks for pointing it out.

I have excluded HELO checks for:
1. relay_hosts (POP before SMTP)
2. authenticated users (SMTP authentication)

Please test it for a day or two, and if it appears to work, then please post again and I'll try it myself, and then put it into my latest release candidate.

I'm sure HELO checks will get rid of a lot of Spam.

Now ... moving forward...

Can anyone tell me if there's an advantage to using greylisting either instead of or in addition to nolisting?

Jeff

[Chrysalis]

wallacetan sorry your post is confusing me.

you quoted

hostlist relay_hosts = net-lsearch;/etc/virtual/pophosts : 127.0.0.1

But your code goes nowhere near that in the config so why is that quoted?

also your code is above the postmaster, abuse exceptions is there a reason for this?

It may have been easier if you pasted the lot without the snips. As I am also wondering where the authenticated bypass is.

Thanks

[wallacetan]

I was trying to highlight /etc/virtual/pophosts is in relay_hosts hostlist.

postmaster, abuse and hostmaster mail accounts are RFC requirements, so is a proper SMTP HELO.
I believe SMTP HELO should be valid, before accepting email for postmaster, abuse and hostmaster.
But if you believe otherwise, the HELO checks can be placed after accepting email for postmaster, abuse and hostmaster.

In each deny ACL, HELO checks excluded for:
1. relay_hosts (POP before SMTP), not in relay_hosts:
hosts = !+relay_hosts

2. authenticated users (SMTP authentication), not authenticated user or authenticated bypass:
!authenticated = *

Single deny ACL:
# HELO is empty or not sent
deny message = You have sent no HELO! Please see RFC 2821 section 4.1.1.1
log_message = Bad HELO: Empty HELO
hosts = !+relay_hosts
!authenticated = *
condition = ${if eq{$sender_helo_name}{}}
delay = 30s

[Chrysalis]

thanks understood now.