Somewhat OT: Re: [Mimedefang] graphdefang cores with large amounts of data

[Royce Williams]

Stephen L Johnson wrote:

>The basic problem with graphdefang is that is reads the entire Summary
>database into memory. It works fine for small datasets, but large mail
>volume sites run into problems.
>
This would be fine in some ways, as the trimming should keep the size
relatively manageable over time.  But your suggestion below would have
the most flexibility for the most users.

>graphdefang does use tie to open the DB file, but in the next line it
>sucks the entire DB into memory. It might be better to provide an option
>to use the tie'd database for data crunching. It would run a lot slower,
>but it would allow larger datasets to be run with memory issues.
>
This would be a handy option, though I don't think that lack of memory is
directly causing my problem, as I did the crunching on my syslog box which
has a gig more RAM free than the application was asking for.  This is what
prompted my original question about BerkeleyDB or other thresholds.
It does appear to be consistently around the time that the size of the
database when loaded into memory is at the 600-650M mark for me.

Royce

-- 
------------------------------------------------------------------------
Royce D. Williams               "Insanity destroys reason, but not wit."
IP Engineer III - ACS Internet                        - Nathaniel Emmons
907-565-2267    PGP key: 0x3FC087DB     http://web.acsalaska.net/~royce/