[Mimedefang] Speeding up MIMEDefang

[Chris Myers]

----- Original Message ----- 
From: "Dave Ellenberger" <dave at nofuture.ch>
To: <mimedefang at lists.roaringpenguin.com>
Sent: Sunday, November 23, 2003 5:39 AM
Subject: [Mimedefang] Speeding up MIMEDefang


> Hi list!
>
> I'm looking for some tricks to speed up MIMEDefang. I have read the
> PDF about MIMEDefang where there is written that someone managed a intel
> based system to proceed 2 million mails a day. My system (Single CPU P4
> 2.53Ghz, 1GB RAM, SCSI disks using shmfs... full SA checks enabled.
> bayes,
> razor, pyzor, dcc, a total of 18 RBL checks, and 1 antivirus running, no
> sql/db lookups) currently is on it's limit at ~50'000 mails a day.

1) Run a caching DNS server on the local box, and load your RBL's into it
whenever possible.  Every time you don't have to hit the network, you're
saving time.
2) Run a DCC server locally (can't do this with Pyzor or Razor).  The public
DCC servers are way overloaded, which means timeouts and retries.
2a) Use dccifd instead of dccproc, you'll save on at least an exec().
3) If this is a dedicated server, make /tmp a shmfs ... virtually nothing
besides SA will be using it anyway.  Make sure any Cron scripts that do
reporting don't eat up /tmp when they run.
4) Make sure you've got enough RAM on the box for the number of processes
you are running.
5) Use greylisting (http://projects.puremagic.com/greylisting/), you don't
have to implement the full spec (especially the minimum time before
accepting a new sender/recipient pair) to get most of the benefit.  If you
get a lot of spam, this may reduce your load by 50% (or more!).
6) Use HELO validation (some spammers pretend to be "you" in the HELO), you
can kick them early in the SMTP conversation.
7) Drop messages with a high SA score (>5, >7, >10, >20 ... depends on what
you call "high" in your environment).  If you don't deliver the message,
you're saving load.
8) Drop executable attachments without virus-scanning them.  Virus scanning
is killing you, do everything possible to avoid it.
9) Make sure your SpamAssassin database isn't getting HUGE, the keyword
database has a nasty tendency to learn bits of undecoded base64.  I've seen
my database get up to 200MB with older versions of SpamAssassin.  The
Mail::SpamAssassin::Conf manpage describes the configuration keywords you
need to control the database size.
10) Put your SpamAssassin database in shmfs.  Reading/writing a 10MB, or
100MB, file for every message is a big hit to performance.  Obviously you
want to back it up to disk occasionally and restore the backup whenever the
system boots.  Remember that disk is 1000x slower than RAM.

Let me repeat myself on #8: Virus scanning is killing you, do everything
possible to avoid it.  Your box, if you use the suggestions above (esp. #5),
should be able to handle a minimum of 250,000 messages/day and probably
2x-3x that ... without virus scanning.  If you're dropping executable
attachments, you barely need virus scanning anyway (.zip and other archive
files come to mind).

Perlcache and speedycgi-type technologies aren't going to help you.  They
obtain speed by making the perl process persistent (no fork()/exec()/Perl
startup every time you run the perl script), which MIMEDefang already does.
See MX_REQUESTS in the MIMEDefang configuration file, which can safely be in
the hundreds.

You're still looking for solutions to speed up per-message processing by
SECONDS, not milliseconds, so don't spend a lot of effort on Net::DNS yet.

Keep an eye on RAM, as you implement these speedups you'll find that your
system is handling more and more sendmail/MIMEDefang processes
simultaneously during peak times.  If you run out of RAM and start
paging/swapping you're going to be hating life.

Chris Myers
Networks By Design