[Mimedefang] graphdefang 0.9, --trim, memory usage, other comments

[Stephen L Johnson]

On Tue, 2003-10-28 at 17:24, Kevin A. McGrail wrote:
> > As others have mentioned, memory usage is a problem.  I'm regularly seeing
> > in excess of 1Gb of ram in use when graphdefang runs.  I'm only 12 days
> > into my data-set and have no idea how much more will be eaten when I get a
> > full month's worth of data to graph.  Any ideas for reducing memory usage?
> > One that I've had is to stop loading all of the data from the tie() and
> > working off disk.  Which would have the unfortunate side affect of slowing
> > down processing.  Anything else?
> 
> This is where my theory comes in that an RDBMS is the only way to scale
> graphdefang to these types of large systems.  I have to believe that the
> query systems, memory handling, etc. from an RDBMS is tailor designed for
> these issues.

You don't need to go to a RDMS to scale graphdefang. graphdefang uses 
the Berkerley DB which is perfectly adequate for the dataset. But
graphdefang sucks the entire Summary DB file into memory when it starts
up. This speeds up processing tremendously, but the downside is huge
memory comsumption for large databases.

The fix is to leave the DB on disk and process it there. It should be a
configurable option. Site with low mail loads could continue process
from memory for speed. And sites with high mail loads or memeory
constains would run from disk.

IIRC John mentioned in a previous posting that he would be looking into
reducing memory consumption. I anxiuosly await the reults of his
efforts. I'd like to run graphdefang to update the charts throughout the
day.

-- 
Stephen L Johnson                       stephen.johnson at mail.state.ar.us
Unix Systems Administrator              sjohnson at monsters.org
Department of Information Systems
State of Arkansas
501-682-4339