[Mimedefang] Re: Spam Assassin Reject

[Kelson Vibber]

For those who were wondering about my overly-elaborate spam handling 
routines for mimedefang-filter, here it is.  As I mentioned last time, 
since we're an ISP we need to balance the spam and virus blocking against 
the risk of customers losing legit mail, so we only drop spam under very 
limited circumstances, and always with a rejection message explaining that 
either the message or source looked like spam:
         1. The message appears in Razor *and* has a minimum score of 10
         2. The sending relay appears in at least 3 RBLs
         3. The message exceeds a user-selected threshold (not finished yet)

I started with the sample someone posted on roaringpenguin.com (or maybe a 
site linked from there), made my own modifications, then added other 
features as they made their way into the default filter.  I admit it could 
probably use some cleanup, but it's been running fine.

1. It only calls SpamAssassin if the message is < 128 KB, no virus has been 
detected, and the message isn't from somewhere "local" (as defined by a 
list earlier in the filter).

2. It defines separate thresholds for plain score, score with Razor, 
minimum number of RBLs.  Code can fit in here to customize thresholds 
per-recipient.  (I had some I was testing, but it's still incomplete and a 
bit of a mess, so I've left it out here.)

3. It only acts if the message gets the minimum score from SA, so if 
something has been listed in Razor incorrectly, it won't drop it unless it 
has a few more characteristics.  (This also prevents people who want all 
their spam from having things dropped because of Razor)

4. It checks how many realtime blackhole lists SA found the sending relay 
in, and if it's in at least 3 of them (easily changed to 2 or even 1, 
though I'm reluctant to trust any one RBL to block only spam without some 
collateral damage) it'll reject the message, indicating that we don't 
accept mail from spam senders.

5. It checks the overall score, and checks for a Razor hit (with its own 
threshold), and rejects the message with a "this looks like spam" warning.

6. It writes the spam score and spam report to a file in the quarantine 
directory.

7. It adds the various headers.  If some other service has already added 
"[spam report ___]" to the subject, it'll replace it instead of adding 
it.  It also places the full sam report in the header.  It's least 
intrusive this way (the message isn't modified at all, so things like razor 
reporting and forwarding are not affected), and we had several people who 
expressed that they liked having the full report in there.  It breaks the 
report up into separate lines because Sendmail started collapsing 
mutli-line headers (and adding a "possible attack!" line to the log, every 
single time), and it seemed more sensible to change the behavior of 
MIMEdefang and keep the extra level of sanity checks on the headers.




     # Spam checks
     my $bounce = "no";
     # Don't run SpamAssassin if message is larger than 100KB or if relay 
in local list
     if ( (-s "./INPUTMSG") <= (128 * 1024) && $VirusFound == 0
                 && $RelayAddr !~ /$LocalRelays/
                 && ! exists($SendmailMacros{'auth_authen'}) ) {
         my ($hits, $req, $names, $report);

         # Threshold above which to bounce messages.
         my $drop_threshold = 100; # never seen anything hit this high, 
should be safe
         my $use_razor = 1;
         my $rbl_threshold = 3;
         my $razor_threshold = 10;

         ($hits, $req, $names, $report) = spam_assassin_check();
         if ($hits >= $req ) {
                 # Separate score for blackhole lists;
                 my @RBL_Tests = ('RCVD_IN_RELAYS_ORDB_ORG', 
'RCVD_IN_OSIRUSOFT_COM', 'RCVD_IN_RFCI', 'RCVD_IN_ORBS', 'RCVD_IN_ORBZ');
                 # 'X_OSIRU_SPAM_SRC', 'X_OSIRU_SPAMWARE_SITE' actually 
check the results of the OTHER tests for specifics.
                 # During testing, a legit email sent through a spamming 
ISP was dropped even though it was listed in only
                 # one of these RBLs, because it counted these two rules in 
the "drop it now" decision.  The whole reason
                 # for doing these tests in SpamAssassin is to avoid 
dropping legit mail from compromised networks, rather
                 # than just blocking everything at the sendmail level.
                 my $RBL_score = 0;
                 foreach my $RBL(@RBL_Tests) {
                         $RBL_score += 1 if( index($names, $RBL) != -1 );
                 }

                 # if we're quite confident it actually IS spam, just bounce it
                 # Criteria: SA Score > 20, listed in Razor, or listed on 
at least 3 RBLs
                 if ( $hits > $drop_threshold || (index($names, 
'RAZOR_CHECK') != -1 && $hits > $razor_threshold) || $RBL_score >= 
$rbl_threshold ) {
                         $bounce = "yes";
                         action_quarantine_entire_message();
                         if ($QuarantineSubdir ne '' && open OUTFILE, 
">$QuarantineSubdir/SPAM_REPORT" ) {
                                 print OUTFILE "$names\n";
                                 print OUTFILE "$hits";
                                 print OUTFILE $report;
                                 close OUTFILE;
                         }
                         if( $RBL_score >= $rbl_threshold ) {
                                 action_bounce('We do not accept mail from 
known spam sources');
                         } else {
                                 action_bounce('Message seems to be spam, 
rejected');
                         }
                 } else {
                         my($score);
                         if ($hits < 40) {
                             $score = "*" x int($hits);
                         } else {
                             $score = "*" x 40;
                         }
                         # We add a header which looks like this:
                         # X-Spam-Score: 6.8 (******) NAME_OF_TEST,NAME_OF_TEST
                         # The number of asterisks in parens is the integer 
part
                         # of the spam score clamped to a maximum of 40.
                         # MUA filters can easily be written to trigger on a
                         # minimum number of asterisks...
                         # action_change_header('X-Spam-Score', "$hits 
($score) $names");
                         action_change_header('X-Spam-Score', "$hits 
($score)");
                         # action_add_part($entity, "text/plain", 
"-suggest", "$report\n", "SpamAssassinReport.txt", "inline");
                         if (index($Subject, '[spam score ') != 0) {
                                 action_change_header('Subject', "[spam 
score $hits] $Subject")
                         }
                         else {
                                 (my $NewSubject = $Subject) =~ s/^\[spam 
score(?:.*?)\]/[spam score $hits]/i;
                                 action_change_header('Subject', $NewSubject);
                         }
                         action_change_header('X-Spam-Warning', 
'SpamAssassin says this message is probably SPAM');
                         my $i = 0;
                         foreach my $line (split(/\n/, $report)) {
                                 $line =~ s/^\s*SPAM://;
                                 action_change_header(sprintf("X-Spam-Report-%02d",$i), 
$line);
                                 $i += 1;
                         }
                 }
         }
     }




>Date: Wed, 3 Jul 2002 11:49:36 -0500 (CDT)
>From: Nate Carlson <natecars at real-time.com>
>Subject: Re: [Mimedefang] Re: Spam Assassin Reject
>
>On Wed, 3 Jul 2002, Kelson Vibber wrote:
> > If anyone's interested in seeing the more elaborate recipe I use, let
> > me know and I'll post it.
>
>I'd love to see it.
>
>--
>Nate Carlson <natecars at real-time.com>   | Phone : (952)943-8700
>http://www.real-time.com                | Fax   : (952)943-8500

Kelson Vibber
SpeedGate Communications, Technical Staff
kelson at speed.net          Phone: (949) 341-0800
http://www.speed.net/     FAX:   (949) 341-0900