[Mimedefang] Performance Benchmarking / Experiances

[John Scully]

We are handling well in excess of that volume now, and I can tell you
that the architecture of the complete system is far more important then
raw horsepower.  Poorly structured filters can suck down a powerful
server with fairly low mail volume, while well designed ones can handle
amazing quantities of mail.

During a heavy dictionary attack yesterday we ran slightly over 400
messages per minute to almost 20,000 recipients (50 recipients per
message) for about 2 hours.  The total from that attack was 44,000
messages to over 2 million bad recipients, mixed in with our normal mail
flow of about 10,000 messages during that 2 hour window.

Average load on the filtering servers was 5.4 during the attack, and it
caused no issues.  

Our core philosophy is load balanced clusters of mid power servers with
a focus on ram, ram, ram, processor and disk performance.  Plus extra
ram.

Key to the architecture is to have inbound filtering servers that do
NOTHING but filter inbound mail and do not allow outbound traffic and DO
NOT deliver mail locally. This simplifies your filter logic a great
deal, and lets you bypass the slow and  difficult to control aspects of
sendmail's relay control code.

We have 3 P4 2.4 Ghz servers with 2 gig of ram, shortly to go to 4 gig.
SATA drives on Promise raid controllers and dual Ghz nics, one on the
outside network, one on the inside network.  We are a redhat linux shop
(yes, I know that some other flavors of unix give better performance,
but we have a lot of redhat/linux experience here - better to stay with
what you know best.  We have sendmail set to allow 800 threads,
mimedefang set to 50 min/50 max processes.  

USE A TMPFS (ramdisk) for /var/spool/MIMEDEFANG!  This makes a huge
difference.  Anything you can do to lower diskio lag is important - we
are about to go to remote syslogging in order to eliminate the local
diskio from the logs.

We use sendmail as an MTA only, with a perl/mysql mail platform for
POP3, webmail and IMAP access.

On the filtering servers sendmail is configured as a PROMISCUOUS RELAY
so that it is NOT using the access.db to control relaying.  We use
filter_relay in mimedefang to check against a black list of
spammers...but that is not very effective.  Most of the work is done in
filter_recipient.

There is a local mysql DB containing a replicated copy of the subscriber
mailbox tables, allowing mimedefang to simply look up each recipient in
the tables and check if the mailbox exists and is allowed to receive
more mail (status, qouta etc).  If the recipient does not exist we
answer with "REJECT - no such user" and increment a counter on this
message to track how many non-existent recipients so far.  Once we hit 5
we tell sendmail to discard the message, on the assumption that five
non-existent recipients means it is a dictionary attack.  I can not
stress how important it is to be able to quickly identify the recipient
addresses as OK!  We go to the extant that the mailbox tables are broken
out to mailboxaa mailboxab mailboxac etc.  With mailboxes like aaron@ in
the mailboxaa table.  This means that a STUPID sorted dictionary attack
is hitting one small table over and over, and the index gets cached by
mysql so it is not even touching disk after the first few hits.  See my
post from yesterday for more detail and some code snippets.

Here is a snippet from my log - part of ONE SECOND of the log.  Remember
that after we tell sendmail to discard mimedefang no longer sees the
recipients, so this represents only the first 5 recipients of each
message.  Of course, they are somewhat out of order based on many procs
writing to syslog at once.

Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airhed27 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airforce11874990 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airhed3 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airpatty9 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airosmith at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4691]: filter_recipient rejected
recipient <airosoul at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airforce11947 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airpower137 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airhed3000 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airpatty99 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airosmith1 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airmilu at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4691]: filter_recipient rejected
recipient <airospace at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airforce11boy at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient said
DISCARD: Discarding this message 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airpower3 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airpaul1 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airosmith2002 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient said
DISCARD: Discarding this message 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airlinebob at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airmel2 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airhead122914 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airospace2 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airmilwood at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airindo at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient said
DISCARD: Discarding this message 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient said
DISCARD: Discarding this message 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airpower4u at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airperry99 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airett at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient said
DISCARD: Discarding this message 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airnmyhead at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airlinebound at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airhead123 at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airindusa at zcloud.net> 
Sep 24 16:27:49 atmail2 mimedefang.pl[4690]: filter_recipient rejected
recipient <airpersia at zcloud.net>

If it passes at least some recipients and hits filter_begin and
filter_end, then it gets virus and spam scanned.  So during this attack
yesterday the 40,000 dictionary based messages were discarded, and only
the normal traffic got virus and spam scanned.

Note that if it is discarded by filter_recipient sendmail keeps
accepting recipients, but no longer calls mimedefang, and after the last
recipient it discards before receiving any data.  This means that a
large percentage of junk is rejected during the envelope processing, and
you do not lose bandwidth to the actual message data.

The second key to the process it to use the sendmail mailertable to then
control disposition of the email.  The domains are not in the
local-host-table - they are in the mailertable and tell sendmail which
delivery host to pass the message to (it can not use the MX record
because that would point back to the filtering cluster).

If you would like more detailed info please contact me off list.

John Scully
jscully at isupportisp.com

-----Original Message-----
From: mimedefang-admin at lists.roaringpenguin.com
[mailto:mimedefang-admin at lists.roaringpenguin.com] On Behalf Of Joe
Stevens
Sent: Thursday, September 25, 2003 4:17 AM
To: mimedefang at lists.roaringpenguin.com
Subject: [Mimedefang] Performance Benchmarking / Experiances

Hello,

I am interested in deploying Mimedefang at a fairly large mail site, and

would be interested in hearing how many messages get handled by what 
hardware from any list members that also run large sites (our current 
load is about 100,000 messages a day).

After looking at a number of solutions, Mimedefang was unquestionably my

favorite, but wasn't able to find any hard performance numbers on 
google...

I am sure the exact performance changes with which filters you are 
running, but any experiances as well as performance tuning tips I could 
get would be greatly appreciated.

Thanks,

--
Joe

_______________________________________________
MIMEDefang mailing list
MIMEDefang at lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang