[Mimedefang] Speeding up MIMEDefang

David F. Skoll dfs at roaringpenguin.com
Sun Nov 23 10:20:46 EST 2003

On Sun, 23 Nov 2003, Dave Ellenberger wrote:

> I'm looking for some tricks to speed up MIMEDefang. I have read the
> PDF about MIMEDefang where there is written that someone managed a intel
> based system to proceed 2 million mails a day.

You will never get that kind of performance if you use network-based
tests like Razor, Pyzor, DCC and DNSBL's.

The person who got that performance seriously hacked MIMEDefang and
was not using SpamAssassin or the MIME::Tools modules, and was not
using any network tests.

I have clocked around 12 messages/second, or a million a day, on
a dual-Xeon system with 1MB of RAM.  This was on a fast LAN of course;
real-world performance would be less.

My box was running:

- SpamAssassin
- Clam AV
- PostgreSQL for user preferences

> a) As I can see spamassassin is using /tmp for temporary Files:

Really??  Set TMPDIR in the environment before you invoke MIMEDefang.

> b) There are several external programs used by MD (pyzor, python, dcc,
> razor, Antivirus and such)

That will absolutely kill performance.  No question.  And no workaround.

> c) What can be done to improve the speed of perl?

Not much.  The multiplexor already takes care of keeping the Perl process
running.  2.39 will be out Real Soon Now; you can try the embedded Perl

> d) Net::DNS seems to be a performance problem. Is there a way to move
> DNS RBL Checks from Net::DNS to a C-written, multithreaded, daemonized
> DNS resolver?

That won't help.  The problem is network latency.  The only way to improve
this is to run a local copy of the DNS zones you're interested in.  For
some RBL's, you can get access for zone transfers.  For others, you can't.

> e) There are some DNS RBL zones which can be downloaded. For example
> I have the RBL+ from mail-abuse as BIND db-file on my DNS server. I
> Could export this file into berkeley DB format and put this file into
> shfm FS and use some File_DB Perl-based funktion to lookup?

That might help, but it's a lot of work.

> Are there any other things I could do to speed up MD?

Don't use network-based tests.  Don't execute external programs.  Do
use the embedded Perl interpreter.  If you can avoid calling SpamAssassin,
avoid it.  (Check for bad extensions first, for example; if something is
rejected because of a bad extension, don't run SA.)



More information about the MIMEDefang mailing list