[Mimedefang] Sender validation

Jonas Eckerman jonas_lists at frukt.org
Sun Jun 27 17:47:49 EDT 2004


On Thu, 24 Jun 2004 15:39:04 +0200, Jonas Eckerman wrote:

>  If this stuff keeps working without problems, I'll
>  post again if I actually start rejecting based on it.

Ok. Now the test has been running for a week, and it has hit exactly *no* legit mail. I just upgraded it from data collection to actually rejecting based on senders. I'll keep montitoring the stuff to see if I should make more excemptions and some more result checking. And of course I'll keep an eye on the stats to see if it rejects enough to be worth it. I'm also keeping the debug log stuff on for a while so I can see that it works as intended.

As Les Mikesell suggested, I'm now using a database (my greylist databse actually, but with a different key prefix). Also as he suggested, rejected addresses have a shorter cache time initially, but it grows for each reject.

My complete filter is available at:
http://whatever.frukt.org/mimedefang-filter.shtml

Below are some snippets from it. Disclaimer: This stuff is new. It may well have bugs. It seems to work, but...

Some settings. I haven't really spent any time finding the best values for the cache times below.
--8<--
$sendercheck = 1;
$sc_cache_valid = 7*24*60*60;
$sc_cache_invalid = 60*60;
$sc_cache_unknown = 7*24*60*60;
$sc_cache_invalid_add = 60*60;
$sc_cache_invalid_max = 24*60*60;
--8<--

Checking a mail address. Note that only 550, 551, 553 and 554 are taken as rejections (meaning that, for example, the fact that a mailbox is full is not seen as a rejection here). Also, I check for some stuff that can indicate that the other servers rejects because it doesn't like the sender <>.
--8<--
# Check a mail address against a mail server
sub check_mail_address($$) {
	my ($a,$s) = @_;
	my ($ok,$msg,$code,$dsn) = md_check_against_smtp_server('<>',$a,$MyFilterHostName,$s);
	my $txt = "$code $dsn $msg";
	$txt =~ s/\s\s+/ /g;
	$txt =~ s/^\s+//;
	$txt =~ s/\s+$//;
	# Disregard some REJECTs because they aren't really rejecting the recipient address.
	return (3,$txt) if ($ok eq 'REJECT' && ($code !~ /^55[0134]$/ || $msg =~ /(sender|mail from|return|<>)/i));
	return (1,'') if ($ok eq 'CONTINUE');
	return (0,$txt) if ($ok eq 'REJECT');
	return (2,'');
}
--8<--

Find MX servera for an address and check it against them using the above sub. This tests MX servers until it gets a clear reject or ok, wich means that I'll get a clear ok from backup servers if the primary server don't give a clear reject even for invalid addresses.
--8<--
# Check a mail address against it's MX server(s)
sub check_mail_address_mx_i($) {
	my ($a) = @_;
	return (4,'') if ($a =~ /^<?>?$/);
	my $d = $a;
	$d =~ s/^.*@([^@>]*)>?$/$1/;
	return (5,'') if (!$d);
	my $dns = Net::DNS::Resolver->new;
	$dns->defnames(0); # do not search default domain
	$dns->persistent_tcp(0);
	$dns->tcp_timeout(15);
	#$dns->udp_timeout(15);
	my $mx = $dns->query($d, 'MX');
	return (6,'') if (!$mx);
	my %mx;
	foreach my $r ($mx->answer) {
		$mx{$r->preference} = $r->exchange if ($r->type eq 'MX');
	}
	return (7,'') if (!%mx);
	my @rinfs = ();
	foreach my $mp (sort keys %mx) {
		my ($ok,$rinf) = check_mail_address($a,$mx{$mp});
		return (0,$rinf) if (!$ok);
		return (1,$rinf) if ($ok == 1);
		push @rinfs, $rinf if ($rinf);
	}
	return (8,join('; ', at rinfs));
}
--8<--

This stuff uses the above function for checking an address with it's MX servers, cacheing the results in the greylist database according to the cache time settings. The first three fields in the records are the same as in my greylist implementation so that the database cleaner doesn't have to understand different records.
--8<--
# Caches the result in the greylist database.
sub check_mail_address_mx($) {
	my ($a) = @_;
	my $key = "S:".address_strip($a);
	my $rc = 1;
	my $now = time();
	my $created = $now;
	my $modified = $now;
	my $count = 0;
	my ($res,$txt);
	my $ores = -1;
	if (tie(%GDB,'DB_File','/var/spool/MIMEDefang/.greylistdb',O_RDWR|O_CREAT|O_EXLOCK,0600)) {
		if (defined($GDB{$key})) {
			my $data = $GDB{$key};
			($created,$modified,$count,$res,$txt) = split(/;/,$data,5);
			$ores = $res;
			my $ct;
			if (!$res) {
				$ct = $sc_cache_invalid + ($sc_cache_invalid_add * ($count - 1));
				$ct = $sc_cache_invalid_max if ($ct > $sc_cache_invalid_max);
			} elsif ($res == 1) {
				$ct = $sc_cache_valid;
			} else {
				$ct = $sc_cache_unknown;
			}
			$rc = ($now - $modified > $ct);
			debug_log(0,"check_mail_address_mx: found, key=\"$key\" data=\"$data\"");
			if (!$rc) {
				$count ++;
				$data = join(';',$created,$modified,$count,$res,$txt);
				debug_log(0,"check_mail_address_mx: count, key=\"$key\" data=\"$data\"");
				$GDB{$key} = $data;
			} else {
				my $a = $now-$modified;
				debug_log(0,"check_mail_address_mx: renew, key=\"$key\" cache=$ct age=$a\n");
			}
		}
		untie %GDB;
	}
	if ($rc) {
		$modified = $now;
		($res,$txt) = check_mail_address_mx_i($a);
		$count = 0 if ($ores != $res);
		$count ++;
		my $data = join(';',$created,$modified,$count,$res,$txt);
		if (tie(%GDB,'DB_File','/var/spool/MIMEDefang/.greylistdb',O_RDWR|O_CREAT|O_EXLOCK,0600)) {
			$GDB{$key} = $data;
			untie %GDB;
			debug_log(0,"check_mail_address_mx: saved, key=\"$key\" data=\"$data\"");
		}
	}
	return ($res,$txt);
}
--8<--

And this is from filter_sender. As you can see there's a bunch of checks for addresses that I don't validate for different reasons.
--8<--
	# Check if sender address is valid. Exempt a bunch of addresses from the check
	# in order to be less abusive with regards to big list servers and that sort
	# of stuff.
	# If from some of our local domains, check locally.
	if ($sendercheck && $sender !~ /^<?>?$/
			&& $sender !~ /^<?(postmaster|abuse)@/i && $sender !~ /^<?(|.*[-_+=])(daemon|gateway)(|[-_+=].*)@/i &&
			$sender !~ /@(|[^@]+\.)(bounces?|returns?|lists?|newsletters?)\.[^@\.]+\.[^@\.]+[^@]*$/i &&
			($sender !~ /^<?(|.*[-_+=])(anonymous|undisclosed|unspecified|lists?|returns?|users|bounces?|\d+)(|[-_+=].*)@/i ||
			$sender !~ /^<?(|.*[-_+=])$OurDomains(|[-_+=].*)@/i)) {
		if ($sender =~ /^.+\@$OurDomains>?$/i && $sender !~ /^.*@(|[^@]\.)frukt.org>?$/i) {
			my $hst = '127.0.0.1';
			my $dom = address_strip($sender);
			$dom =~ s/.*@([^@]+)$/$1/;
			$hst = $storingservers{$dom} if ($dom && defined($storingservers{$dom}));
			my ($ok,$rinf) = check_mail_address($sender,$hst);
			debug_log(0,"filter_sender: $sender = $ok ($hst) [$rinf] {$dom}");
			if (!$ok) {
				md_syslog('info',"MDLOG,$MsgID,bad_sender,local,$ip,$sender,?,?");
				return ('REJECT',"Bad sender address: $sender! Responsible server said: $rinf");
			}
		} elsif ($sender !~ /^.+\@$OurDomains>?$/i) {
			my ($ok,$rinf) = check_mail_address_mx($sender);
			debug_log(0,"filter_sender: $sender = $ok (MX) [$rinf]");
			if (!$ok) {
				md_syslog('info',"MDLOG,$MsgID,bad_sender,mx,$ip,$sender,?,?");
				return ('REJECT',"Bad sender address: $sender! Responsible server(s) said: $rinf");
			}
		} else {
			debug_log(0,"filter_sender: $sender (unchecked 2)");
		}
	} else {
		debug_log(0,"filter_sender: $sender (unchecked 1)");
	}
--8<--

Regards
/Jonas
-- 
Jonas Eckerman, jonas_lists at frukt.org
http://www.fsdb.org/




More information about the MIMEDefang mailing list