[Mimedefang] Curious about large mimedefang setups

Nathan F nathan at zenlok.com
Sun Sep 27 23:56:33 EDT 2015

Hi all,

Just wondering how other companies are using Mimedefang and what people do
to handle issues that arise with general email parsing/storage.  My
companies product involves email archiving.  I have been working here for
the past 7 years as (more or less) a solo engineer so I often feel like I
am trapped in a bubble.  It can be a bit difficult at times to not have
other people to turn to when I get stuck on something that is non-trivial.
Anyway, very simple numbers are as follows:

3 million mails a day
24 c3.large servers = 90GB ram (AWS love it/hate it we wouldn't have gotten
this far without it).

Ongoing issues:

1) Retry requests.  Given that this is an archiving service, I generally
don't worry about this too much but I wonder how others feel about retry
requests resulting from the slave/queue being full?  As it stands I think
our current setup can handle quite a bit more traffic on average than we
currently process.  We do, however, often get several unpredictable daily
spikes whose resulting retry requests I have tried to overcome by throwing
hardware at the problem.

2) Premature EOM unexpected close.  I see this sometimes and have tried
looking into it on multiple occasions, but I have yet to find anything to
help me sort out what is happening in my case.  I really dislike not
knowing the why of the what.  Our traffic mostly comes from office 365 and
google apps.  Haproxy is sitting in front of the mail processing servers
with lengthy timeouts.  Otherwise it is a very straightforward setup.

We use ES, Tika, S3 and several other technologies so there is a lot
happening around the simple act of storing email.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.mimedefang.org/pipermail/mimedefang_lists.mimedefang.org/attachments/20150928/17cbdc11/attachment.html>

More information about the MIMEDefang mailing list