[Mimedefang] Speeding up MIMEDefang
David F. Skoll
dfs at roaringpenguin.com
Sun Nov 23 10:20:46 EST 2003
On Sun, 23 Nov 2003, Dave Ellenberger wrote:
> I'm looking for some tricks to speed up MIMEDefang. I have read the
> PDF about MIMEDefang where there is written that someone managed a intel
> based system to proceed 2 million mails a day.
You will never get that kind of performance if you use network-based
tests like Razor, Pyzor, DCC and DNSBL's.
The person who got that performance seriously hacked MIMEDefang and
was not using SpamAssassin or the MIME::Tools modules, and was not
using any network tests.
I have clocked around 12 messages/second, or a million a day, on
a dual-Xeon system with 1MB of RAM. This was on a fast LAN of course;
real-world performance would be less.
My box was running:
- SpamAssassin
- Clam AV
- PostgreSQL for user preferences
> a) As I can see spamassassin is using /tmp for temporary Files:
Really?? Set TMPDIR in the environment before you invoke MIMEDefang.
> b) There are several external programs used by MD (pyzor, python, dcc,
> razor, Antivirus and such)
That will absolutely kill performance. No question. And no workaround.
> c) What can be done to improve the speed of perl?
Not much. The multiplexor already takes care of keeping the Perl process
running. 2.39 will be out Real Soon Now; you can try the embedded Perl
facility.
> d) Net::DNS seems to be a performance problem. Is there a way to move
> DNS RBL Checks from Net::DNS to a C-written, multithreaded, daemonized
> DNS resolver?
That won't help. The problem is network latency. The only way to improve
this is to run a local copy of the DNS zones you're interested in. For
some RBL's, you can get access for zone transfers. For others, you can't.
> e) There are some DNS RBL zones which can be downloaded. For example
> I have the RBL+ from mail-abuse as BIND db-file on my DNS server. I
> Could export this file into berkeley DB format and put this file into
> shfm FS and use some File_DB Perl-based funktion to lookup?
That might help, but it's a lot of work.
> Are there any other things I could do to speed up MD?
Don't use network-based tests. Don't execute external programs. Do
use the embedded Perl interpreter. If you can avoid calling SpamAssassin,
avoid it. (Check for bad extensions first, for example; if something is
rejected because of a bad extension, don't run SA.)
Regards,
David.
More information about the MIMEDefang
mailing list