HOWTO: Convert to courier imap using mysql

[ballyn]

Quick and dirty guide to using courier-imap with cram-md5 and mysql under DirectAdmin

Motivation

The imap and imap ssl functionality provided by DA is not as robust as other solutions I've configured, and I much prefer the maildir folder hierarchy for mail storage as opposed to single mbox files. So, I went about determining how to convert my DA system to use courier-imap as the MDA (mail delivery agent).

The challenge
DA has two kinds of users, "real" and "virtual". "real" users currently do their authentication through PAM or through the OS shadow mechanism, while "virtual" users are authenticated against a custom file in DA's directory. Both real and virtual mail users must be able to do a number of things, including get delivery, do smtp-auth, do pop, do imap and utlize pop-before-smtp. The biggest hurdle to the migration is that under courier, the path to the maildir is provided by the authentication layer. So, we need to first use an auth method supported by courier that will provide password checking as well as the path to the maildir for that user. Then, the smtp auth facility must also be told to use that mechanism. The courier pop and imap daemons will take care of delivery.

Courier-imap's most common form of authentication is through a shadow-like dbm userdb system called authuserdb. Username, password, etc. is stored in a flat text file and then converted to a db file by a script included with courier. Unfortunately, on my system, I couldn't get exim to read the shadow file that is created by courier (which appears to be a v1 file while exim's support is for v3/4). The problem, then, is that we can't use the same password file for smtp-auth that we use for pop/imap.

Rather than struggle with versioning and compatibility issues, I decided to take a slightly more complex (but more scalable) approach of storing user data in a mysql table. However, this approach also made it simple to further support CRAM-MD5 authentication for both pop/imap and smtp-auth.

Note:There is additional functionality to be added and more testing to be done before this conversion can be "complete". For example, it is currently impossible to change a user's e-mail password via the DA console. Furthermore, I can't get pop3d-ssl to correctly negotiate an sslv3 session...


Required packages:

Note: This is tested under Centos 3.3... you should be able to use similar or identical packages under RHE 3 flavors. Notes regarding BSD are inline regarding possible issues I'm aware of...

Courier-IMAP:
tested courier-imap-3.0.3-1.i386.rpm

Courier-IMAP mysql authentication:
tested courier-imap-mysql-3.0.3-1

MySQL Compatibility libraries:
tested MySQL-shared-compat-4.0.23-0

Exim with mysql support
tested 4.44-1 RPM from DA by installing the src rpm and editing the makefile

DRAC
http://mail.cc.umanitoba.ca/drac/

Configuration:
INSTALL = install
EBIN = /usr/local/sbin
MAN = /usr/local/man/man
DEFS = -DSOCK_RPC -DFCNTL_LOCK -DGETHOST -DDASH_C -DREQ_HASH -DCIDR_KEY -DTERM_KD
CC = gcc
RANLIB = :
CFLAGS = $(DEFS) -g
LDLIBS = -ldb
TSTLIBS = -L. -ldrac
RPCGENFLAGS = -C -I
MANLIB = 3
MANADM = 8

drac-add
http://mail.cc.umanitoba.ca/drac/courier-exec.txt


Setting up MySql:

Choose a user that will own the "mail" database. Create a database called "mail" and create the user table.
CREATE TABLE users (
id char(128) DEFAULT NOT NULL,
crypt char(128) DEFAULT NOT NULL,
clear char(128) DEFAULT NOT NULL,
name char(128) DEFAULT NOT NULL,
uid int(10) unsigned DEFAULT '65534' NOT NULL,
gid int(10) unsigned DEFAULT '65534' NOT NULL,
home char(255) DEFAULT NOT NULL,
maildir char(255) DEFAULT NOT NULL,
quota char(255) DEFAULT '' NOT NULL,
KEY id (id(128))
);


Insert a test user into it:
INSERT INTO users (id, crypt, clear, name, uid, gid, home, maildir) VALUES (
"user@domain.com", encrypt("asecretpass"), "asecretpass", "atestuser", "8", "12",
"/var/spool/virtual/domain.com/user",
"/var/spool/virtual/domain.com/user/Maildir")


Note: we keep a cleartext version of the password in this table for CRAM-MD5 authentication

Note: MySql's encrypt() function requires crypt() support from the OS. No clue if BSD builds support this.

Note: The uid/gid fields for real users should match the os uid/gid since courier will use them to access the users maildir.



Configuring the MTA (Exim) for delivery:

Exim must be configured in two ways:
delivery of mail to a maildir folder
allow smtp-auth for real and virtual users (see below)
We solve the first issue by modifying the local_delivery and virtual_localdelivery directives in exim.conf:
local_delivery:
driver = appendfile
delivery_date_add
envelope_to_add
#file = /var/mail/$local_part
directory=${home}/Maildir
maildir_format = true
#prefix = ""
group = mail
mode = 0660
return_path_add
user = ${local_part}

## for delivering virtual domains to their own mail spool

virtual_localdelivery:
driver = appendfile
create_directory
delivery_date_add
directory_mode = 700
envelope_to_add
#file = /var/spool/virtual/${domain}/${local_part}
directory=/var/spool/virtual/${domain}/${local_part}/Maildir
maildir_format = true
group = mail
mode = 660
return_path_add
user = "${lookup{$domain}lsearch*{/etc/virtual/domainowners}{$value}}"
quota = ${if exists{/etc/virtual/${domain}/quota}{${lookup{$local_part} \
lsearch*{/etc/virtual/${domain}/quota}{$value}{0}}}{0}}

As you can see, we've told exim to deliver mail to a Maildir "directory" for real users as well as virtual users instead of the mbox "file" described previously.. Now we need to set up the maildirs. You'll want to check permissions on this, too... I think I just set "mail" the owner of the virtual maildirs.

mv /var/spool/virtual/<domain>/<user> /var/spool/virtual/<domain>/<user>.bak
maildirmake /var/spool/virtual/<domain>/<user>/Maildir
or
maildirmake /home/<user>/Maildir


There are a number of scripts to convert an mbox format mail file into a maildir...

On restart, exim will begin delivering mail for real users into /home/<user>/Maildir and virtual users into /var/spool/virtual/<domain>/<user>/Maildir. Try sending a message to a user and then checking in the ~user/Maildir/new directory for the file.

It would be a good idea at this point to create a maildir for your /etc/skel folder with a "Sent", "Drafts", "Trash" structure, etc.


Configuring the MDA (courier-imap) for delivery:

Most courier-imap 3.x versions require fam which requires portmap. If you're using an RPM repository, fam should be configured as a prerequisite and installed. Fam typically runs out of inetd or xineted and chkconfig --list should show you the service. Both portmap and fam should be running.

edit /usr/lib/courier-imap/etc/authdaemonrc to auth against mysql:
authmodulelist="authmysql authcram"

edit /usr/lib/courier-imap/etc/authmysqlrc to contain information about your databse, etc.
MYSQL_SERVER localhost
MYSQL_USERNAME admin_mail
MYSQL_PASSWORD asecretpass
MYSQL_SOCKET /var/lib/mysql/mysql.sock
MYSQL_DATABASE admin_mail
MYSQL_USER_TABLE users
MYSQL_CRYPT_PWFIELD crypt
MYSQL_CLEAR_PWFIELD clear
MYSQL_UID_FIELD uid
MYSQL_GID_FIELD gid
MYSQL_LOGIN_FIELD id
MYSQL_HOME_FIELD home
MYSQL_NAME_FIELD name
MYSQL_MAILDIR_FIELD maildir

While we're here, let's edit imapd and pop3d to support SSL, TLS and CRAM-MD5:

/usr/lib/courier-imap/etc/imapd:
IMAP_CAPABILITY="IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT \
THREAD=REFERENCES SORT QUOTA AUTH=CRAM-MD5 AUTH=CRAM-SHA1 IDLE"
IMAP_CAPABILITY_TLS="$IMAP_CAPABILITY AUTH=PLAIN"
/usr/lib/courier-imap/etc/imapd-ssl:
IMAPDSSLSTART=YES
IMAPDSTARTTLS=YES
/usr/lib/courier-imap/etc/pop3d:
POP3AUTH="LOGIN CRAM-MD5 CRAM-SHA1"
POP3AUTH_TLS="LOGIN PLAIN"

Also make sure to set the IMAPD/POP3DSTART=YES flags for the services you want to enable and enable TLS, etc. in pop3d-ssl. You'll probably also want to generate some new ssl certificates... see the /usr/lib/courier-imap/share folder for some scripts.

Now we can shutdown imap (via xinetd) and vm-pop3d (via DA's console) and start courier-imap (service courier-imap start).

At this point, assuming you've entered an email user's information into mysql, you should be able to receive mail into a maildir via exim and use pop and imap to retrieve those messages via courier. Remember that /var/log/maillog and /var/log/exim/mainlog are your friends. Also remember that all users are now getting delivery to maildir folders and they might have to exist before exim will deliver to them and that all authentication is now being done through mysql, so if you don't have a user/pass in the mysql db, you won't be able to authenticate to the mail system.


Configuring SMTP-AUTH:

To use SMTP-AUTH against mysql, modify your exim.conf to contain lines like this:
MYSQL_AUTH_PLAIN = SELECT crypt FROM users WHERE id = '$2'
MYSQL_AUTH_LOGIN = SELECT crypt FROM users WHERE id = '$1'
MYSQL_AUTH_CRAM = SELECT clear FROM users WHERE id = '$1'

plain:
server_condition = "${if and {{!eq{$2}{}}{!eq{$3}{}}{crypteq{$3}{ \
${lookup mysql {MYSQL_AUTH_PLAIN} {$value}fail}}}}{1}{0}}"
login:
server_condition = "${if and {{!eq{$1}{}}{!eq{$2}{}}{crypteq{$2}{ \
${lookup mysql {MYSQL_AUTH_LOGIN} {$value}fail}}}}{1}{0}}"

cram:
driver = cram_md5
public_name = CRAM-MD5
server_secret = "${lookup mysql{MYSQL_AUTH_CRAM}}"
server_set_id = $1


Restart exim and test smtp-authentication... you should be able to use PLAIN, LOGIN and CRAM-MD5 authentication. Make sure that you are not an open relay!

Configure pop-before-smtp:

pop-before-smtp is currently provided by the popb4smtp process which watches the maillog and writes IPs that authenticate against a flat file. Unfortunately, courier's logging breaks this solution. So, for this guide, we'll use two programs, drac and drac-add, which should actually scale much better than a log scrubbing solution. drac is a process that requires portmap and listens for rpc calls. drac-add is added to courier's pop authentication system.

So, what happens is that when someone authenticates over pop, they go through courier's normal auth sequence and then also pass through drac-add. If they are authenticated, drac-add makes an RPC call to drac which adds the IP address in a database (I edited drac's makefile to use /etc/drac/dracd.db). Exim then consults this database when it consults the relay_hosts. This could probably be done for imap as well, but I figure if you have an imap client you probably have smtp-auth as well.


Ensure that portmap is running and that you have a firewall rule protecting it
Start drac, /usr/local/sbin/rpc.dracd
mv the drac-add binary into /usr/lib/courier-imap/libexec/authlib/
add drac-add the authmodulelist for /usr/lib/courier-imap/etc/pop3d: AUTHMODULES="authdaemon drac-add"

edit exim.conf:
hostlist relay_hosts = net32-dbm;/etc/drac/dracd.db : 127.0.0.1


Notes and TODO

This solution requires that ALL users (real and virtual) store a crypted and a cleartext version of their password in a mysql table. The danger of storing the cleartext value in the database is outweighed, IMHO, by the advantage of never sending a password by utilizing hashes in CRAM-MD5. However, the security of the database as well as the limited support for the scheme must be considered. If you're uncomfortable with storing cleartext passwords, just remove everything related to cram-md5 in this howto.

Requiring all users to exist in this database also requires that DirectAdmin have the ability to alter this data (during account creation/deletion, password change, etc.). I have a plugin and a couple of custom scripts that I'm using successfully so far. The one issue I had was storing the mysql info for the plugin in a file that must be executable (and thus readable) by any user... At John's suggestion, I ended up using a small wrapper that is suid diradmin which calls the actual script that does the work and contains the mysql information.

Forwarding, autoresponders, vacation messages, etc. are lightly tested (and should function).

Quotas for virtual users should be fine since we're using the same filename for the maildir's enclosing folder that was used for the mbox. However, real users mail will now be stored in "~/Maildir" instead of "/var/spool/mail/<username>" and "~/mail". I'm not sure if mail quotas are ever calculated for real users, and I don't know if this affects them.

Since we're already using mysql, it should be simple to replace drac and drac-add with a script that will replace drac-add as a courier authentication process and simply take the user's current IP address and write it to a mysql table that we can then use an exim mysql lookup for pop-before-smtp.

Comments, suggestions, etc. are welcome!

[rldev]

Wow you have come along way. I hope to test this shortly on a test server. Getting closer.

[toml]

ballyn is correct about the challenge. I am spening some of my free time trying to create and authentication module for use with Direct Admin. That should be all that is needed to have a fully functional courier-imap with DA, and still support changing passwords from the Control Panel, virtual and system users. Their layout for creating a new authentiaction module is a little complex, but I think I can massage a lot of the code changes that were made to UW into Courier.

The UW changes use a lot of global variables that are UW specific, but I think I can replace them with local variables and come up with a workable module.

The nice thing about courier, is they separated the authentication and the IMAP server. The authentication part is where you set the Maildir and authenticate the user, and the IMAP server remains untouched.

[ballyn]

So I've looked at skins and plugins... it's pretty straightforward to modify the process to change an email (virtual) user's password. I have a prototype but I need to look at security, etc.

If you don't want to use mysql for real users passwords, it's simple to add "AUTHPAM" to the authmysqlrc file and then use that in addition to mysql. That change, along with a custom skin/plugin for pop accounts (and some quota work) should complete the conversion with the exception of smtp-auth for real users.

However, I would personally prefer to have different passwords for email than I do for ssh. By using SSL for DA, the only way in which a cleartext password could be transmitted is via pop/imap, so using different passwords for the mail accounts adds a lot more security to the system.

The skinning/plugin work becomes much greater at that point, and I'd like to hear the general opinion about using mysql vs. some other method (and DA's thoughts, maybe) before I get into it.

[toml]

I was thinking about this some more and another thing that would need to be done, is to use Courier's POP3 service, since the current DA POP3 server relys on mbox mail files.

I personally would prefer to have a MySQL based system, it would really scale a lot better than parsing passwd files. But would need to be integrated with the skins.

[jmstacey]

Originally posted by toml
I personally would prefer to have a MySQL based system, it would really scale a lot better than parsing passwd files. But would need to be integrated with the skins.

I don't speak for DirectAdmin, but my gut feeling is they won't like this if they ever decide to implement this in the future since that would complicate the installation process as well as make the webmail service dependent on MySQl. Whereas, at the moment, most services are pretty compartmentalized.

[toml]

Originally posted by jmstacey
I don't speak for DirectAdmin, but my gut feeling is they won't like this if they ever decide to implement this in the future since that would complicate the installation process as well as make the webmail service dependent on MySQl. Whereas, at the moment, most services are pretty compartmentalized.

That is the reason I was posting it. I do think MySql authentication would be the best thing, but for little to no impact on DA, I think the current /etc/passwd and ~/.shadow system would be the easiest to support for DA.

As I said before the other thing that would need changing is POP3. I have never used the Courier POP3 server, but I assume they use the same authentication server as IMAP, so it should be able to be just dropped in.

[ballyn]

This guide covers using courier's pop3 daemons and should function for using pop3 over ssl as well... I just can't get the SSL part working right now. :) Regular POP using CRAM-MD5 works fine, however, and also uses drac to allow pop-before-smtp.

While I agree that converting to mysql appears to be a more complicated/risky solution, the current mail solution relies on:

exim, vm-pop3d, popb4smtp, xinetd, uw-imap plus PAM and a flatfile for every domain with email accounts

While assuming we can get smtp-auth working using mysql, this solution relies on:

exim, courier-imap, mysql

The scalability and performance is undoubtedly much higher as well.

[rldev]

Whatever is done should also anticipate the ability to have the mail server on it's own server. Whatever configuration would best lend itself to a seperate mail server should be used. Obviously we are looking for speed, stability and scalability. Either way, I believe this is far less complicated then the current mail system. DA can easily implement any changes needed for the DA skin. It is good to see this has come a long way already and I am sure this is very encouraging news to all DA users as well as DA themselves.

[toml]

I posted some code to allow courier to use the existing /etc/passwd and /etc/virtual/$domain/passwd files. I modifed the custom auth module from the 0.53 version of courier-authlib. You can see the file that was changed HERE (http://www.directadmin.com/forum/showthread.php?s=&threadid=6566) . There may still need to be a litte more work done to it, but it was authenticating fine for me on my Solaris box with some dummy virtual domain files in place.

[ballyn]

I'm almost done putting together a plugin that allows users to change email passwords in a courier-mysql environment. Essentially, I'm completely separating DA/OS passwords from email passwords and adding a "Change Email Password" hook to DA. Admins can change all passwords, Resellers can change their users' passwords and users can change their own password. "Virtual" (pop3) accounts are handled as before and I'm using the custom scripts to create/destroy user and pop3 accounts.

I have noticed, however, the courier does use the UID/GID flags in the mysql table I described when logging in a user. Courier will use that uid to access the users Maildir. So, I have to figure out how to get the UIDs right for the table.

So, my question... when is the users' mbox file created? I haven't touched any of DA's processes regarding account/pop3 creation, but I can't figure out where in the process the mbox file was being created. Will Exim create it when it receives a message for it? Wouldn't this cause a client (like squirrelmail) to fail if the user tried to check the account before a message was sent to it?

Thanks...

[ballyn]

Nice work on that auth code, btw... I haven't tested it but I'll look at it soon.

[ballyn]

So I think my plugin/scripts are complete and functional. I ended up grabbing the uid/gid from a stat on the homedir for real users and using mail for pop3 accounts. I also tweaked exim.conf to create the maildir folders if they don't exist with the correct permissions.

I've also tested autoresponders and a few other things which all worked as expected (since we haven't really touched that code). I'm still stuck on pop3-ssl, but since I don't even use pop I haven't spent much time on it... I'm thinking it's a certificate thing or something broken in the rpm build I used vs. the openssl library it's linking against...

Anyways, if anyone gets to the point of getting courier et errata installed and wants to look at the plugin/scripts, let me know.

[rldev]

How about submitting this to DA for review?

How does this affect current existing mail accounts?

[keefe007]

Is someone planning to make this into a full-fledge DA plugin? If so, I know *a lot* of people here would be eternally thankful!

Keep up the great work!

[ballyn]

It would be REALLY difficult to make this a full plugin, especially if you wanted to support more than one OS.

I'm in the process of writing a script that will try to install the pieces, but it will likely only work with RH/Centos in the first iteration.

I'm currently working on how to migrate accounts... I'd like to make sure that it's possible before I get into automating the install.

[alex2k]

Originally posted by ballyn
It would be REALLY difficult to make this a full plugin, especially if you wanted to support more than one OS.

I'm in the process of writing a script that will try to install the pieces, but it will likely only work with RH/Centos in the first iteration.

I'm currently working on how to migrate accounts... I'd like to make sure that it's possible before I get into automating the install.

We'll wait then :)
Thank's for your effort, really appreciate it.

[ballyn]

I have a preliminary script ready for testing that will install and configure courier, mysql and exim (including per-user spam boxes). I'm kind of waiting on a new exim RPM, but if you have a RHE 3 or Centos 3 DA box that you can use to test the script on (i.e might blow up), please let me know via email or PM.

Thanks!

[thoroughfare]

Hey Ballyn,

Just a suggestion - I don't like the idea of using MySQL to power email services, unless it's a fully redundant external MySQL server being used.

How about using an SQLite database, which is more self-contained?

Matt

[ballyn]

Haven't even considered it. I chose MySql since it already exists on all DA boxes.

The SQL is pretty portable in the scripts, but I don't know if courier has an auth plugin for it.

[rldev]

Good Work Ballyn! Any progress on mbox conversion to maildir for current users? I will test this out once the new Exim package is available from DA. Is there a way to undo this once the mail system has changed?

[ballyn]

Yeah, I'm working on the conversion now. I changed the process a bit so that delivery is to a different directory, so it should be easy to do the mbox to maildir migration. The only question is permissions on those folders.

[thoroughfare]

Just an idea - I totally understand why you chose MySQL, I just personally would like to avoid inter-daemon dependencies where possible.

Matt

[jmstacey]

Originally posted by thoroughfare
Just an idea - I totally understand why you chose MySQL, I just personally would like to avoid inter-daemon dependencies where possible.

Matt

I also mentioned this. However it just occured to me that it might be possible to make adjustments and use sqllite. That would provide the same benefits as well as being a self contained, no dependency library.
Just an idea, I have not actually used sqllite in anything significant.

[rldev]

Well if it can run with mysql, I'm sure it can run with SQLlite. The good thing is that this can be a significant upgrade to DA. Fixing one of it's biggest weknesses. I don't know about any of you, but email is most critical to my business.

[thoroughfare]

Please forgive my ignorance but I'm new to Courier.

If Exim is still used in this CourierIMAP thing, what is Courier actually replacing? Just the IMAP and POP daemons? Is there a difference between an MTA and an MDA?

Thanks!
Matt :)

[ballyn]

Courier is replacing:

uw-imap
vm-pop3d
popb4smtp
perl smtpauth (in exim.pl)

The MTA (transfer agent, exim) sends and receives mail. The MDA (delivery agent, uw-imap or courier, etc.) delivers mail to users.

I should finish the mbox to maildir conversion today. I need to clean up the (ugly) folders that squirrel et. al. create.

[thoroughfare]

I see. So currently, our MDA comprises of vm-pop3d and uw-imap?

Matt :)

[rldev]

yes which are poor at best.

[thoroughfare]

Great, thanks :)

I can't stand vm-pop3d I have to agree.

Matt

[ballyn]

The script is pretty much ready... It will install the courier packages and dependencies including exim 4.51-1 with mysql support, configure the database, configure courier and exim and migrate mboxes to maildirs. It does this without touching anything default in DA besides exim.conf, so you can uninstall the system and roll back via the script as well.

The plugin that hooks into mysql and provides the ability to change real user email passwords (separate from the system passwords) is also done. Virtual user passwords are supported via DA custom scripts and there are also scripts for create domain, etc. These are also installed via the install script.

The new exim.conf uses the real user to run the domain_filter transport, so this system also supports per user (and per pop account) spamassassin delivery. The filter template is also installed via the script.

I really need some folks to test this, so if you're interested, please let me know.

Thanks,
Allyn

[rldev]

I'm interested.

[rldev]

Ballyn,
Since it is now possible with this script to support per email box spam assasin, is it also possible that it can support per email box Clam AV?

[ballyn]

You mean for message delivery of "virus found" or something? I haven't looked at mailscanner so I'm not sure what you mean.

[rldev]

I meant that on some systems I have seen like Hsphere and Plesk, AntiVirus can be turned on and off per mailbox.

[ballyn]

OIC... yes, this should be possible, but I haven't looked closely at the logistics. It would likely be a pretty major modification since exim would have to pass all viruses through and then filter at delivery instead of at receipt.

[rldev]

Don't think I would want to filter at delivery.

[kktsang]

ballyn,

Can I have your script to test ?

[Chrysalis]

this looks very interesting but complex to do, if I find the time I will try this on my test FreeBSD server.

[kktsang]

ballyn,

I pm you my email already.

[dennisc]

Hello,

Is the script|plugin available and is it possible to put it public in this thread? Thanks.

Dennis

[resolveit]

I'm also interested in testing/using this, please let me know where I can find everything nescesary to do this on my server(s).

Regards,
Onno

[jlasman]

I'm happy to make a testbed available for anyone willing to do a through test and then report back to the community.

I don't recommend trying it on a production server.

Jeff

[dennisc]

It would be nice if the DA team would actively help to make this a standard. Till now I don't have any idea how Da staff thinks about moving to Maildir and Courier, which is far superior to uw-imap.

[mmerlin]

Yes I will second that motion. I have had a user email box with 1000 messages in it and they could not access it via webmail (both squirrelmail and uebimiau) even after I did some of the recommended php.ini tweaks like increasing the memory allocation and script timeout.

Conversely, on one of my other servers, which uses maildir format, the mailboxes can be massive (waaaay more than 1000 emails) and the webmail systems work just fine. Plus IMAP access is faster.

Mbox just doesn't cut the mustard.
Concatenating emails into one single huge file is like so 70's man, it's living in the past ;)

Seriously though, the mbox format is the only real hiccup I've had with DirectAdmin. Everything else on the whole is great.

Maildir as an officially DA supported option will elevate DA head and shoulders above the competition.

Not to mention make many of us here (and our clients) a lot happier :)

[keefe007]

Any news on the Plugin or integration from the DA staff?

[rldev]

The DA staff knows a change needs to be made to the mail system. As for when this will ever happen, I don't know.

[hostpc.com]

I'd hope that John & crew would seriously work with ballyn to bring this to reality. Obviously he's done a LOT of work on this, and from what I've seen so far, it's as complete as it can be without the blessings of DA - and possibly made default.

Thanks again ballyn

[Chrysalis]

I believe they workong on dovecot so we switch to maildir, but it wont be using mysql as it is is bad to rely on mysql for email to stay online and functioning.

[jlasman]

I also believe they're working on dovecot.

While I know that MySQL works very well with dedicated mailservers, I sure wouldn't want to use it to control email on a Webhosting server. It's already too busy doing too much.

I prefer plain-text files for everything, because I know how to edit them :) .

Jeff

[ballyn]

I've been spending most of my courier effort these days on the x86_64 version... I've gone to current versions of courier and courier-auth and simplified things a bit by going back to exim's smtp auth mechanism and supporting PAM as well as mysql so that "real" user accounts do not have to exist in the database. I've further stopped supporting pop-before-smtp using drac since it doesn't scale much better than the log scrubber and every email client I've seen recently supports smtp auth...

I have only had a few people interested in the script and I haven't gotten much feedback on it, so I probably won't continue developing it for a general release. I am available to convert new DA installs to courier, but I'm not confident enough in the mbox to maildir conversion to support the conversion of populated systems.

I don't expect that you'll see a plugin version anytime soon... The process is simply too complicated.

I've worked with dovecot for awhile, but I think the push from DA to go to dovecot is because it supports the mbox format, so they don't have to convert from mbox to maildir. While I'm sure the performance would be better than UW, the bottleneck will likely still be the size of the mbox file, so I don't think going to dovecot without going to maildir will help the situation much. And if we're going to maildir, I don't think the conversion to dovecot will be any easier than what we're doing for courier.

The new version of courier-authlib helps things a bit on the authentication front. It should be much easier to utilize the virtual user password file for authentication if you don't want to use mysql. However, from the courier-authlib documentation:
userdb is a simple, straightforward solution that scales to a couple of thousand of mail accounts, depending on the hardware. Beyond that, one of database-based modules will need to be used, such as authldap, authmysql, authpgsql.
Essentially, he's saying that using a single flat file for courier authentication should scale to a couple hundred accounts. Using a nested userdb scheme with multiple domain userdb files should scale to a couple of thousand accounts. And if you need more than that, go to a database auth mechanism.

My thinking is that mysql is as critical as email in a shared hosting environment, and in a busy system it will actually be much less expensive to do a simple database lookup than to parse through a flat file for a password. However, if using mysql is the big hold up here, I'll happily look at implementing something that's compatible with DA's current auth mechanism.

On the DA side, however, I think the hangup is still the mbox to maildir conversion. My script uses the mb2md perl script which seems to work relatively well for this, but if we can come up with a rock-solid mbox to maildir conversion it will probably make the conversion to courier an easier feature to support. If someone wants to volunteer on that front, I'll look at the other auth schemes and update my script for i386 and x86_64.

[dennisc]

Totally agree. In that way people have the possibility to choose which method authlib will use.

I would personally like to see DA with Postfix instead of Exim but I don't know if it's possible to change to another mailserver and if Da can manage another mailserver.

[hostpc.com]

Thank you for your hard work on this... hopefully someday soon we can see a choice of options when setting up - Exim or an alternative (with md)

[toml]

Originally posted by ballyn
My thinking is that mysql is as critical as email in a shared hosting environment, and in a busy system it will actually be much less expensive to do a simple database lookup than to parse through a flat file for a password. However, if using mysql is the big hold up here, I'll happily look at implementing something that's compatible with DA's current auth mechanism.

A while back, I posted changes required to use DA Auth mechanism with courier. It basically does the same auth as DA currently does. It can be located here. (http://www.directadmin.com/forum/showthread.php?s=&threadid=6566&highlight=courier)
I haven't done much testing on it, but it should be a good place to start.

[ballyn]

The authpipe mechanism in the new courier-authlib makes it much easier to use a configurable script in place of something compiled... I've put a script together in perl that should work to authenticate against DA's virtual passwd files.

If this is useful I'll find a dev box to test it on and rework the script.

[toml]

I haven't looked at their authpipe yet, I currrently only use courier on my home server and my wife and I are the only users, so I didn't have the need for anything more elaborate than passwd files. When I get some time, I will look into that method.

[scottnic]

Originally posted by ballyn
The authpipe mechanism in the new courier-authlib makes it much easier to use a configurable script in place of something compiled... I've put a script together in perl that should work to authenticate against DA's virtual passwd files.

If this is useful I'll find a dev box to test it on and rework the script.

I'd be very interested in seing your script. I was planning on doing something like this myself, but why reinvent the wheel?

TIA
Scott

[keefe007]

Great work guys.

I'd still really like to see a plugin for this. I'd be willing to pay for it as well!

[hostpc.com]

Any luck on the move the Dovecot? Any updates or official announcements?

The UW IMAP issue is REALLY becoming an issue, especially with Squirrelmail taking an extremely long time to login/navigate - and most of the time throwing errors to the user "connection dropped" -etc. I've been trolling the SM forums/lists and Exim lists - it's definately an issue with the UW client Exim.

I'm all for a conversion - hell, I'll chip in $ to make it come to fruition, but if DA is already working on this - can we make it happen before the end of the year?

Thanks for your hard work guys...

[Chrysalis]

They working on it but I dont know if they pushing it for it to be their next big change, I sure hope so.

[jlasman]

DA staff has written about switching to Dovecot.

They've not written about any conversion. My understanding is they've decided on Dovecot because it can handle both formats and wouldn't need preexisting mailboxes to be converted.

We may eventually do a conversion based on interest, but only based on interest.

Jeff

[keefe007]

A change such as this would really push DA far beyond Cpanel, Ensim, and Plesk. You, I, and many other people have stated that we would pay extra to get this done. I'm not sure why the DA team hasn't taken more interested in this conversion.

There's always the "rut" affect with software development. Why spend a lot of time making a big change when they are making good money just doing updates here and there.

The fact of the matter is that DA would make a ton more money if this was implemented. If you read about WHT and other forums you'll see this is one of the biggest complaints about the DA structure.

[Chrysalis]

keefe007 is bang on right, word does spread around and if maildir was implemented along with other stuff such as jailed ssh directadmin would be much more popular and highly regarded then it is now.

I think you will find the interest will be apperent when dovecot is implemented their will be little interest if there is no integrated maildir solution available.

[jlasman]

I don't remember saying I'd pay for anything :) .

My understanding is that if/when Dovecot is enabled all new domains will be (perhaps optionally) Maildir based.

So for new sales, that's a good thing.

Jeff

[hostpc.com]

Jeff - any chance of asking John for an official type word on this? Are we all just spinning our heals - or is this truly in the works. I wont hold you to a time frame, but is this something we're likely to see soon?

[jlasman]

I suppoose you could ask him, Joe.

Nevertheless I've saved you the trouble and I just sent him the link to this thread in an email.

Jeff

[hostpc.com]

I could have, but since you were fairly active in this conversation, I thought maybe you had his ear a little more :)

Thanks

[DirectAdmin Support]

I've got dovecot running on a DA system. Works well ;)
I've got DA converted over as well for the new passwd format. For the next release anyone will be able to get DA to recognize/setup/convert the dovecot passwd files with the simple addition of a flag into the directadmin.conf (dovecot=1).

I'm quite impressed with the simplicity of the dovecot configs (after reading the docs for several hours anyway ;)).

So DA itself is 'done' wrt supporting it. There is a convertor for the passwd files in the dataskq options. (action=convert&value=todovecot).

When we release it, I don't want everyone to quickly jump into it blindly. This is a *major* system overhaul so I can predict a few bugs, so only people who absolutely want to test it should try it. The passwd format change is the easy part. The new format requires changes all over the place with regards to backups/restores, and domain pointers (symlinks need to be moved around)

At this stage, I still need a conversion script to convert existing email data (mbox to maildir) for all accounts, as well as a domain pointer conversion script to simply create symlinks where needed. Email data will now all be stored under /home/username, chowned to username:mail, which is great because now the system quotas will monitor everything. No more runaway inboxes under /var ;)

I'll be posting more info as it comes.

Remember, it's not completely done yet, so nothing is for *sure* ;)

John

[dennisc]

At this stage, I still need a conversion script to convert existing email data (mbox to maildir) for all accounts, as well as a domain pointer conversion script to simply create symlinks where needed.

http://batleth.sapienti-sat.org/projects/mb2md/
http://www.akadia.com/services/converting_mbox_mdir.html
http://www.gerg.ca/hacks/mb2md/
http://www.firstpr.com.au/web-mail/mb2md/
http://www.averillpark.net/Maildir/

[hostpc.com]

WHOOOOOO! Go John!!!!

This was absolutely the (2nd) best way I could have gotten woken up this morning :)

Great news, thank you for your hard work.

[interfasys]

Looks like the conversion script is done ;)

http://files.directadmin.com/services/customapache/dovecot/mbox2maildir

[rldev]

This is phenominal news.

[Chrysalis]

excellent news, glad this is done.

[mmerlin]

maildir! This is great news.

It will be a happy day when my clients can use their webmail once again.

[harro]

I am interested in trying this (he says on the last 2 days of his official DA support).

Do I still need to follow the steps at the start of this thread to install the courier functionality, or did John also make an installation script for Dovecot?

Balyn / toml did lots of work, but the actual result of the scripts wasn't shown. Is your script, Balyn, compatible with John's work?
So where do I now start :-) ?


Last question (the actual reason I want to try this):

Does this also make it possible to set up exim / fetchmail to collect mail from various email accounts and sort the mails into different maildirs, according to the account that was checked (using mailfilter, for example)?

[interfasys]

The dovecot install has been added to the build script. Same with the conversion script.

Dovecot replaces vm-Pop3d and wu-imap (or some other imap daemon you have running). There is no need to install anything else.

[harro]

You say that dovecot and the conversion is built into the "build script"... but this does not mean anything to me (yet) :-(

- Where can I find this build script?
- Do I have to reinstall my whole DA for it?
- I looked around in http://files.directadmin.com/services/ but can't find dovecot or a build script - is it there?

Thanks Interfasys, for your fast reply - it helps out :-)

Harro

[interfasys]

http://files.directadmin.com/services/customapache/build

[toml]

Originally posted by harro
I am interested in trying this (he says on the last 2 days of his official DA support).

Do I still need to follow the steps at the start of this thread to install the courier functionality, or did John also make an installation script for Dovecot?

Balyn / toml did lots of work, but the actual result of the scripts wasn't shown. Is your script, Balyn, compatible with John's work?
So where do I now start :-) ?


Last question (the actual reason I want to try this):

Does this also make it possible to set up exim / fetchmail to collect mail from various email accounts and sort the mails into different maildirs, according to the account that was checked (using mailfilter, for example)?

I would hold off until DA releases their version of dovecot. It will be a lot easier to get support on a configuration that DA supports.

The work I did with courier, was to add a new authentication module to handle the DA method of authentication. It did not do any conversion. Again, wait for DA, and I would wait until a decent script(s) is made to do the conversion, it is a lot more work than just running the script on one or two accounts, it has to work with both system and virtual accounts.

This should have nothing to do with your ability to use fetchmail to fetch mail from various email accounts and filter them. The only difference, is that it will now have to use Maildir instead of mbox, fetchmail supports both methods.

[harro]

Thanks toml!

I just got a message back from John as well, stating that the DA release of dovecot is coming up soon.


Note: we have not yet released DA for dovecot, so don't try to install it ;)

We'll be releasing it in somewhere around the 19th, so if you can wait 8 days, it will be available then.


I can wait for that... In the meantime I can read up on the fetchmail + mailfilter. Maybe even write a howto on that. As far as I know, the sorting into directories is dependent on maildir and will only run for a specific user, not for every user that wants that. I want it mainly for myself, so that's not a prolem. But then it might not be useful to write a howto. We'll see.

In the meantime:
- how did people fix their problem with only one instance of exim running and
- only mail from the server itself arriving in /var/spool/exim.in/input ?

Bye,

Harro

[toml]

Originally posted by harro
As far as I know, the sorting into directories is dependent on maildir and will only run for a specific user, not for every user that wants that. I want it mainly for myself, so that's not a prolem. But then it might not be useful to write a howto. We'll see.

In the meantime:
- how did people fix their problem with only one instance of exim running and
- only mail from the server itself arriving in /var/spool/exim.in/input ?

Bye,

Harro
Sorting into directories is dependent on maildir, but it can be sorted into mailboxes, they would both appear the same way to the mail client.

I don't see the problem with exim, you only need one process waiting on a select and when an external program connects to the port selected, exim forks a new process. That new process handles the incoming email, by reading the exim.conf rules, it figures out where to deliver it.

[harro]

toml:

thank you for your reply. I didn't realise that the second exim proces only starts when new mail arrives (but that doesn happen, I checked). Now all is clear.

I 'upgraded' to Dovecot and currently all the mail is held back in the /var/spool/mail dir. I have to check whether the Dovecot modifications have been processed in the exim.conf file!

<edit> It seems that if you use the MailScanner set-up, where the exim mail processing is split into two processes, you need to (also?) patch the SECOND exim_outgoing.conf file.</edit>

As to the sorting into mailboxes: what I want in the end is to set up one account for myself that pop's all my mail accounts en sorts them into separate folders under my one IMAP mailuser account. I understood that this is not possible on the server-side without mailfolders and some filtering program to direct each mail to the right place?

Since I don't know I am asking around :-)

Bye,

Harro