[Mimedefang] common practice

[Cor Bosman]

> > > If different recipients have different spam criteria - for
> > > instance if one wants to block everything with a score above 20,
> > > another wants to block everything with a score above 7, and
> > > another wants to block only messages found in Razor - you'll need
> > > to look into using stream_by_recipient so that each can be handled
> > > individually.
> 
> > Surely this is only meant for small sites :) We do 3+ million emails a day.
> > Last thing I want is resend what's probably hundreds of thousands of
> > emails. We get emails with large amounts of recipients.
> 
> No, the streaming facility is meant for everyone.  And although you may get
> a few emails with many recipients; I bet 90%+ of your e-mail is for a single
> recipient only.

Here's the facts. I only counted actually delivered email, not bounces,
crap, spam, more crap, and the extra fine crap you get when running an ISP.
Counting that would merely increase the numbers.

Messages:   1903611
Recipients: 3047772

Of the 1.9 million messages, 1.6 were sent to a single email address.
So about 15% is sent to multiple recipients. But honestly, thats not
what counts. What counts is that that 15% adds 1.1 million recipients.
Thats not peanuts. With a lot of large lists you'd quickly explode into
lots of queued emails, adding large delays.

> Streaming is the only way you can use milter to apply different filtering
> rules for different people.  And the streaming implementation is reasonably
> intelligent -- it resents mail in deferred mode.  So if mail comes in for
> 300 recipients, you don't get the system trying to run 300 copies of
> MIMEDefang in parallel.  You get a lot of queue activity as the mail is
> queued, and then during the next queue run, the 300 copies are delivered
> serially.

Customers whine if their mail gets delivered a bit later once in a while.
If that'd be a regular thing they'd come protesting at the front door :)
Unnecessary queueing is something we need to avoid above all. 

I realise it's basically a non-solvable problem. Thats why I wondered
if silently dropping (removing the recipient) would be acceptable. Im not
convinced it is, unfortunately. 

> I would suggest that it's reasonable to lower MaxRecipientsPerMessage
> to 50 or so; I don't think 300 recipients/e-mail is reasonable.

Once you have several hundred thousand customers the chances that
over a 1000 of them are on the same mailinglist approach 100%. Especially
local lists. Hell, we see emails coming in sent to thousands of customers.
Valid emails :) So our best approach is to make sure our 
MaxRecipientsPerMessage is as close as possible to the regular defaults.
If we go way below the defaults remote mailinglists will queue..and queue
and queue...until the last people get their email hours later.

I think we've got it to 200 or so. At 50 the delays become unacceptable.

Anyways, just battling with whats best thing to do when you want to
allow per-customer settings, including blocking, on content when you
need to consider everything. 

Bouncing seems evil, rejecting seems to add significant resources/costs,
silently dropping is bad when emails were falsely blocked. 

Cor