[Mimedefang] Upgrading Bayes DBs

[Richard Whelan]

Hi Nels,

>That's an awfully large database for that number of e-mails.  Have 
>you put any bayes_* options in your sa-mimedefang.cf file which 
>override the defaults for bayes expiry, database size, etc.?  Note 
>that there's no advantage to letting your bayes database grow without 
>bound; it'll end up populated primarily by single-occurrence tokens 
>that never show up again.  My bayes system has processed 
>substantially more messages than yours, but the dabase files only 
>make up around 25 MB, with the tokens file comprising only 5 MB of 
>that.
>  
>
I had a feeling it was a little too large.

I have the following in the sa-mimedefang.cf

use_bayes 1
bayes_auto_learn 1
bayes_path /var/spool/MIMEDefang-bayes/bayes
bayes_file_mode 0666
bayes_journal_max_size 128000
bayes_expiry_max_db_size 50000000
bayes_auto_expire 1
bayes_learn_to_journal 1

>If you *haven't* changed any of the defaults, you may have run into a 
>bug in the SA 2.6x expiry system which resulted in no tokens ever 
>being expired.  Do you have SA 2.6x installed anywhere, or another 
>system you can install it on to do the processing?  
>  
>
I tried to use the second of the new servers. I have a different problem
with that, which I cannot identify.

debug: bayes: 456 tie-ing to DB file R/O
/var/spool/MIMEDefang-bayes/bayes_toks
Cannot open bayes databases /var/spool/MIMEDefang-bayes/bayes_* R/O: tie
failed:
debug: Score set 0 chosen.
debug: Initialising learner
debug: Initialising learner
debug: Syncing Bayes journal and expiring old tokens...
debug: lock: 456 created
/var/spool/MIMEDefang-bayes/bayes.lock.smtp2.gxn.net.456
debug: lock: 456 trying to get lock on /var/spool/MIMEDefang-bayes/bayes
with 0 retries
debug: lock: 456 link to /var/spool/MIMEDefang-bayes/bayes.lock: link ok
debug: bayes: 456 tie-ing to DB file R/W
/var/spool/MIMEDefang-bayes/bayes_toks
debug: unlock: 456 unlink /var/spool/MIMEDefang-bayes/bayes.lock
Cannot open bayes databases /var/spool/MIMEDefang-bayes/bayes_* R/W: tie
failed: No such file or directory
debug: Syncing complete.
debug: bayes: 456 untie-ing
debug: bayes: 456 untie-ing db_toks

All the permissions are correct. I've lost count how many times I've
checked. It using the same config file, same bayes files, as the other
new system, which I've still got 3.0.4 on, and that one as I mentioned
just sits for days trying to update.

What I'm trying at the moment is running the force-expire on the
original machine, although I've copied the DBs elsewhere and created a
new sa-mimedefang.cf to reflect this.

>Try (using the 2.6x version) "sa-learn -D --force-expire" and see if 
>the expiry actually works. 
>
What I got was:

debug: Syncing Bayes journal and expiring old tokens...
debug: lock: 17493 created
/home/richard/MIMEDefang-bayes/bayes.lock.gxmx1.gxn.net.17493
debug: lock: 17493 trying to get lock on
/home/richard/MIMEDefang-bayes/bayes with 0 retries
debug: lock: 17493 link to /home/richard/MIMEDefang-bayes/bayes.lock:
link ok
debug: bayes: 17493 tie-ing to DB file R/W
/home/richard/MIMEDefang-bayes/bayes_toks
debug: bayes: 17493 tie-ing to DB file R/W
/home/richard/MIMEDefang-bayes/bayes_seen
debug: bayes: found bayes db version 2
debug: bayes: expiry check keep size, 75% of max: 37500000
debug: bayes: token count: 2576498, final goal reduction size: -34923502
debug: bayes: reduction goal of -34923502 is under 1,000 tokens.
skipping expire.
debug: Syncing complete.
debug: bayes: 17493 untie-ing
debug: bayes: 17493 untie-ing db_toks
debug: bayes: 17493 untie-ing db_seen
debug: bayes: files locked, now unlocking lock
debug: unlock: 17493 unlink /home/richard/MIMEDefang-bayes/bayes.lock

>Getting the database files down to a more 
>reasonable size will make the conversion to SA 3.x bayes much 
>smoother.
>  
>
Agreed.

>Once you've got a reasonable set of db files you should just be able 
>to transfer them to the appropriate location for SA 3.x and issue a 
>"sa-learn --sync" to convert.
>
>  
>
Thanks,

Richard