[Mimedefang] Graphdefang 0.9 missing events?
jpk at bl.org
Tue Nov 4 09:46:42 EST 2003
> One question, though. What is this I hear about the SummaryDB eating near 1G
> of memory? :( That does not sound good. I expect to process/add ca. 5,000
> entries a day. Any idea how long the SummaryDB will last me, before it
> starts growing in the hundreds of megs?
WHAT? I've never heard of this! :-) Just kidding...
Yes, for very large installations graphdefang can eat up a ton of memory.
I made some design decisions up front when I built graphdefang that were
fine for my use, but not so fine for some other people's use.
ASIDE: The basic problem is that perl does not give memory back to the OS
until it exits -- even if you free it up in your program. So, at specific
points in the program I have 2 copies of the SummaryDB in memory (to do db
backups & when saving the summarydb to disk). And... the fix to this is
to change some of my design decisions. I actually wanted to have the
whole hash in memory when I do my processing, but this has proven to be
problematic! The problem I have with fixing this is simply TIME. I leave
the country for work about once per month (mostly to Singapore & Armenia),
and I have not gotten to fixing this yet.
Back to your issue: You shouldn't see any issues with 5,000 entries per
day. I have graphdefang running very smoothly on sites with 10's of
thousands of messages per day. Make sure you use the "--trim" option
fairly often. It will cut out all old statistics that aren't relevant
More information about the MIMEDefang