[Mimedefang] large systems

[John Scully]

We have only moved a portion of our incoming traffic to the filtering
cluster as of yet, but I expect to normally run with 3 or 4 servers in
the active cluster, and spike to 10-15 during attacks.  We are
installing IBM blade servers with L4 load balancing and the ability to
dynamically move blades in and out of clusters as needed.  Since we have
7 different clusters involved in the mail system this will add a great
deal of efficiency.  One pool of "extra" hardware available to any
cluster that comes under heavy load instead of sizing each cluster for
peak.

We do run spamassasin, DCC and several in-house "spam signature" based
systems to allow us to block a little over 99% of spam and porn with few
false positives and fairly low hardware needs.

The signature test and DCC are the fastest/lowest resource utilization
tests, so they run first (but after the virus scan). Only if the mail
get's by them does spamassassin run.  This seems to make a big
difference, because we get virtually zero false positives from those
first tests.

I think the input I can give you is that mimedefang is a good way to go
if you have good perl and DB people and engineer the system well.  As
long as you do not let the milter hang waiting for a response from a
sluggish spam check, like an overloaded RBL it really takes very little
time.  Average time for all processing of a message that hits every test
is 300 MS.  

But before I worked to control the ability of an RBL or other internet
resource to slow it down I sometimes saw 10 second times - and would
have 700-800 sendmail threads running per server after a minute or so...

John Scully
jscully at isupportisp.com  

-----Original Message-----
From: mimedefang-admin at lists.roaringpenguin.com
[mailto:mimedefang-admin at lists.roaringpenguin.com] On Behalf Of Cor
Bosman
Sent: Saturday, September 27, 2003 5:48 PM
To: mimedefang at lists.roaringpenguin.com
Subject: Re: [Mimedefang] large systems

> You described our setup perfectly.  We only started moving traffic to
> the filtering cluster a few days ago, but have already peaked to 400
> messages per minute addressed to about 20,000 recipients during a spam
> attack.

The minute before I typed this reply we did 2500 emails a minute. 
Thats after Sobig & co gets filtered :)

> I would strongly recommend using mysql, and designing the DB to have
> many tables, broken down by the first two letters of the mailbox (i.e.
> mailboxaa, mailboxab, mailboxac etc).  At anything up to several
million
> mailboxes this keeps the index to only one or two levels of
redirection
> and allows mysql to cache the more active mailboxes.  It makes the
most
> difference when under a sorted dictionary attack, because thousands of
> hits in a row will go to the same small index.
> 
> You can contact me offline if you want more detail.

It may be interesting to others too. Currently I have 7 boxes doing our
incoming email. I figure if I push all of those through milter with
optional virus/spam scanning i probably have to at least double that.
Maybe even more. It's not a big deal but im trying to get a feeling for
the amount of boxes (freebsd) id need..

Do you do spamassassin for your ISP customers? If so, dont they ask you
about bayesian filtering? 

Cor

_______________________________________________
MIMEDefang mailing list
MIMEDefang at lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang