Somewhat OT: Re: [Mimedefang] graphdefang cores with large amounts of data

[Stephen L Johnson]

On Wed, 2003-10-15 at 16:30, Whatley, Lee (CBA) wrote:
> David F. Skoll wrote:
>  > Isn't it standard (prudent) practice to run this kind of logfile analysis
>  > on a different machine?
> 
> Is it?
> Are most people just running a syslog server on another machine, or are 
> they copying local syslogs to another machine periodically?
> 
> The load on my mailserver rarely goes above about 1.3 and usually hovers 
> around .7 or .8 with only around 200M of memory usage.  The only time it 
> every takes a serious processing/ram hit is when graphdefang is running.
> 
> I don't see why graphdefang should be so resource intensive as to cause 
> the need to dedicate another machine to it. I don't need to process my 
> webserver logs on another machine and those files are much bigger than 
> my mail server logs.  I'm curious as to if the fact that it is a perl 
> script is causing the overhead.  Does anyone thins a rewrite in C would 
> help the problem? Someone else mentioned using a relational database to 
> store the data.  Is that really necessary?

The basic problem with graphdefang is that is reads the entire Summary
database into memory. It works fine for small datasets, but large mail
volume sites run into problems.

graphdefang does use tie to open the DB file, but in the next line it
sucks the entire DB into memory. It might be better to provide an option
to use the tie'd database for data crunching. It would run a lot slower,
but it would allow larger datasets to be run with memory issues.




-- 
Stephen L Johnson                       stephen.johnson at mail.state.ar.us
Unix Systems Administrator              sjohnson at monsters.org
Department of Information Systems
State of Arkansas
501-682-4339