[Mimedefang] Graphdefang + High CPU Load when processing

[Rich West]

Ya know, I was considering the very same thing.  My backup server does 
basically nothing throughout the day (it only runs backups), which makes 
it a great candidate. :)

Just curious, but what, exactly, does the --trim option do?  The 
documentation states that it cuts out old data from the SummaryDB, but 
doesn't that defeat the purpose of graphdefang when looking for 
longer-term trends and such?

With regards to the corruption issue, the problem creeps in when 
Graphdefang has a LOT of information to process.  In my situation, we 
have a medium load mail server which generates maillogs on a daily basis 
anywhere between 2MB and 10MB.  Now, that doesn't seem to be too large, 
but, however, GraphDefang runs, when I was running it on a daily basis, 
after about a month of gathering data (maybe sooner, but it's been 
nearly 10 months since we were running it on a daily vs. every 30 minute 
basis), we noticed that the graphs suddenly flatlined.

A little investigation showed that the database was corrupted (ran 
graphdefang.pl by hand, and no new data would get added to the 
database).  That was the 4th occurance of the same problem, so we opted 
for the more frequent updates to reduce the amount of data it had to 
handle at any one point in time.  This resolved the problem, but, now, 
10 months later, we have seen that graphdefang has started spiking the 
CPU for a minute or more as it processes (this is a new problem for 
us).  The machine is a solid box with 1GB of RAM..

I'll try running it on a remote server..

:-)

-Rich



Chris Gauch wrote:

>I ran into this same issue with Graphdefang, but it was fairly easy to
>resolve.  I set up graphdefang on a remote Linux server that had a low
>average load.
><...snip...>
>You could also set up an rsync process to rsync the maillog onto the remote
>graphdefang server, rather than configuring remote syslog.
>
>On the remote syslog server running graphdefang, you should also add a CRON
>script that runs the graphdefang.pl --trim option.  Just make sure this CRON
>script DOES NOT run while graphdefang is processing the SummaryDB info, or
>you'll corrupt the he|| out of your SummaryDB. 
>
>- Chris 
>
>  
>
> Kevin A. McGrail wrote:
>
>I run it once a day and never have a corruption issue.  If you have
>corruption issues, suggest looking at your DB installation.  There is just
>very little in graphdefang that could really cause this issue.  Are you
>having other DB corruption issues?
>
>I am very wary of using DB on servers because of the numerous issues we see
>but the speed benefits are great compared to something like mySQL.
>
>In other words, all over different mailing lists, I constantly read about DB
>corrupted this, DB corrupted that.  Not to mention that DB is often
>implemented in a way that loads the entire DB into memory.  This is great
>for small databases but the graphdefang database can hit half a gigabyte for
>a server.  This causes huge spikes in the load but makes it process quicker.
>
>Perhaps your machine is running out of memory trying to load the database
>and just crashing?
>
>Regards,
>KAM
>