[Mimedefang] mimedefang installation specs

Jan Pieter Cornet johnpc at xs4all.nl
Tue Apr 19 16:24:33 EDT 2005


On Tue, Apr 19, 2005 at 10:16:21AM -0400, David F. Skoll wrote:
> >      Wow!  Assuming I didn't make any keying errors, that's 785,751
> > incoming messages in one 24 hour period.  Could you give us a brief
> > description of the architecture of your system and how you handle that
> > kind of load (including type of computers {CPU and memory}, disks,
> > number of computers, clustering if any, operating systems, and type of
> > filtering done)?
> 
> We have CanIt-PRO customers doing those levels of traffic.  For
> example, one customer does about a million messages/day on three
> dual-Xeon PCs.  We have another customer doing about four million/day
> on 8 dual-Xeon PCs, with a quad Xeon hosting the database server.  If
> you thrown lots (lots!) of memory at the problem, nice fast disks, and
> decent CPUs, you can scale things up quite well.

Yup, it scales pretty well. Actually, 785751 messages is only those
messages scanned with SpamAssassin. We allow our users to opt-in to the
spamfilter. In total yesterday, we had 4.1 million mail deliveries (or
delivery attempts: due to DNS blacklists, only 2.3 million mails made it
past the DNS blacklists; the delivered mails were addressed to 3.6
million recipients).

The four (equal-priority) MX records of xs4all.nl (and all of our
customer domains) point to two L4 loadbalancers. Those loadbalancers
distribute the load to two clusters of Intel PCs running FreeBSD.
Each PC has two hyperthreaded 2.8GHz Xeon P4 CPUs (so the OS sees 4
CPUs), 4 Gig of RAM, and two 140G SCSI disks in RAID1 internally,
for the mqueue and logs.

The mail itself is stored, via NFS, on two FAS960 Network Appliance
servers, storing the actual data on 2x44 72G FCAL disks.

The cluster of Intel/FreeBSD machines runs sendmail, mimedefang, and
two virus scanners all on the same machine. Some users do post-processing
with procmail, which also happens on the same machine. MD filtering is
recipient-specific, but typically includes DNS blacklist checking,
antivirus, and spamassassin, using a hand-hacked mimedefang-filter
(which is modular, I intend to give the modularisation stuff to you
all, but I want to run it past David first, before unleashing it on
this list).

The cluster is slightly overspecced for the job: we have 45 of those
"mxdrop" machines (divided over the two clusters), where 10 can easily
handle the load we currently have. However, we did the measurements
for the cluster size during the breakout of a particularly aggressive
virus, and now we have room for growth, we want to be able to handle
another virus outbreak at the same time, and we also want to be able to
lose half the servers to a possible power or network failure, since the
PCs are not redundant (apart from the internal RAID). (And we've been
very happy with that design when one of the fuses between us and the no-
break installation blew up... twice).

The rationale behind this is: hardware is a lot cheaper than support
calls about malfunctioning email services, which is undoubtedly one of
the most important services for an ISP.

Mail retrieval is done using another cluster (or several other clusters),
by the way: we have a pop cluster, imap cluster, and batched-SMTP
delivery cluster. Each is a lot smaller than the clusters behind the mx
records, though.

I'm actually going to give a talk about this setup, in some more detail,
at the next spring conference of the dutch unix user group. The talk
will most likely be in Dutch, though (depending on the attendance), and
the information on the conference is also only available in Dutch. For
those still interested, the URL for the conference is:
http://www.nluug.nl/events/vj05/index.html

-- 
#!perl -wpl # mmfppfmpmmpp mmpffm <pmmppfmfpppppfmmmf at fpffmm4mmmpmfpmf.ppppmf>
$p=3-2*/[^\W\dmpf_]/i;s.[a-z]{$p}.vec($f=join('',$p-1?chr(sub{$_[0]*9+$_[1]*3+
$_[2]}->(map{/p|f/i+/f/i}split//,$&)+97):qw(m p f)[map{((ord$&)%32-1)/$_%3}(9,
3,1)]),5,1)='`'lt$&;$f.eig;                                # Jan-Pieter Cornet



More information about the MIMEDefang mailing list