[Mimedefang] SMTP error return after DATA?

[Nels Lindquist]

On 9 May 2003 at 10:44, Michael Sims wrote:

> Quoting John Rowan Littell <littejo at earlham.edu>:
> 
> > I know you said you didn't want to hear about hardware upgrades, but
> > here's a thought (if you're not already doing it): run the MIMEDefang
> > spool directory on a memory filesystem (TMPFS, or whatever your OS
> > calls it).  This could help with the speed of scanning quite a bit.
> 
> The only reason that I haven't bothered with that on the main mail exchanger is 
> that it doesn't seem that disk I/O is the main source of the problem.  For 
> kicks I tried removing the SA check, and when I did this the load problems all 
> but disappeared.  But MIMEDefang still has to process the message in the spool 
> even without SA, so this lead me to believe that the speed of the spooling 
> process wasn't the major factor.

Whatever the case, there's a complex interaction of all the different 
subsystems which contributes to performance problems like you're 
seeing.  We were experiencing exactly the same symptoms you describe, 
and it was because the system was starved for memory.  The IO 
subsystem suddenly became the bottleneck as swapping consumed more 
and more CPU, and the load average spiralled out of control until 
Sendmail was tempfailing mail.  Use vmstat (or whatever diagnostics 
your OS has) during one of these incidents to see if the system 
starts swapping heavily--that's a strong indication that memory is 
your bottleneck.

MD + SA very definitely uses a *lot* more RAM per connection than 
does MD on its own, so:

o The faster an individual message is processed, the sooner that 
slave is freed up for the next connection and the fewer slaves are 
needed to handle the load, which lowers the overall memory footprint, 
which reduces the requirement for swapping, etc. etc.

o Putting the MIMEDefang spool directory on TEMPFS definitely speeds 
things up, even on Linux.  And though SA is CPU (and RAM) intensive, 
MD still loads the entire message from the work directory before 
passing it to SA.

o A stalled SA process waiting for DNS results etc. consumes just as 
much memory as one that is actively running a regexp.  If you use 
DCC, razor, etc. make sure you lower the timeouts in sa-mimedefang.cf 
from their default of 10 seconds.  I'm using 2 seconds as a timeout, 
and in my experience if the DCC servers (for example) haven't 
responded in that time, they're not going to respond at all. So why 
let your slave utilize all that RAM for 10 seconds just for one test 
that's not going to return a result anyway?  Also beware of certain 
DNS tests (like the HABEAS tests) which can cause delays of over 30 
seconds waiting for results if the servers are down.  "spamassassin -
D" is your friend--watch to see which tests take forever to return 
and consider disabling them in sa-mimedefang.cf.  If you're not 
already running a local DNS cache on your mail relay, then you 
should.

o Sendmail has some built-in functionality which can smooth out spam 
storms.  Have a look at some of the cf options like confDELAY_LA and 
confCONNECTION_RATE_THROTTLE.

----
Nels Lindquist <*>
Information Systems Manager
Morningstar Air Express Inc.