[Mimedefang] Multiple submission queues?

[Hugh Messenger]

On Tue, Dec 9, 2008 at 1:23 PM, Hugh Messenger <hugh.messenger at gmail.com> wrote:
> Is it possible to use multiple clientmqeueue's, and hence multiple
> queue runners?

To answer my own question ... yes.  So for the benefit of anyone
stumbling on this thread ... do the following:

Create as many subqueue dirs as you want in /var/spool/clientmqueue,
with some numbered scheme like nq.1 thru nq.5.  Chown them all to
smmsp.smmsp, and chmod to 770.

Add the following to /etc/mail/submit.mc:

define(`confQUEUE_SORT_ORDER', `random')dnl
dnl define(`confMIN_QUEUE_AGE', `30m')dnl
define(`QUEUE_DIR', `/var/spool/clientmqueue/nq.*')dnl

As for any .mc changes, do a 'make' and restart sendmail.  Note that I
haven't played with tuning the MIN_QUEUE_AGE yet, so it's commented
out above.  If you have a really busy submission queue, you will
probably want to set this depending on how often you run the
submission queue (so any queue submissions which tempfail don't get
tried on every q run).

Take a look at /var/spool/clientmqueue, and if you still have any
queued stuff at the top level, move it all into one of the new nq.*
directories, otherwise it won't get sent (sendmail is now looking in
your nq.* dirs, not at the top level).

I've also upped the frequency of my submission qrunner by adding ...

CLIENTQUEUE=5m

... to /etc/sysconfig/sendmail, and modifying the sendmail startup
script in /etc/init.d/sendmail (YMMV), changing SMQUEUE to CLIENTQUEUE
on the Ac daemon line:

       daemon --check sm-client /usr/sbin/sendmail -L sm-msp-queue -Ac \
                       -q$CLIENTQUEUE $SENDMAIL_OPTARG

Here's how that looks on one of my busier relays where I just
converted from a single queue.  I'd somehow left it with the original
qrunner at 1h instead of 5m, so it had built up a substantial queue
(around 600).  After the above changes, and dumping the 600 odd
existing spooled msgs into nq.1 (as described above) and after 30
minutes or so of q draining, it's down to around 90 msgs, and now
looks like:

[root at pmx1 clientmqueue]# mailq -Ac
                /var/spool/clientmqueue/nq.2 (12 requests)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
mBBMTRde007174      514 Thu Dec 11 16:29 <kbrittnidh at wine.net>
                                         <azallea1 at djs-unlimited.net>
... blah blah ...
                /var/spool/clientmqueue/nq.1 (35 requests)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
mBBIbhSw002859     4406 Thu Dec 11 12:40 MAILER-DAEMON
                 (host map: lookup (group.kr): deferred)
                                         <hyundai at group.kr>
... blah blah ...
                /var/spool/clientmqueue/nq.5 (17 requests)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
mBBMUCgq007655      518 Thu Dec 11 16:30 <maurashana_ec at levy.net>
                                         <az810668.1731 at djs-unlimited.net>
... blah blah ...
                /var/spool/clientmqueue/nq.3 (14 requests)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
... blah blah ...
                /var/spool/clientmqueue/nq.4 (14 requests)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
...blah blah
                Total requests: 92

As you can see, nq.1 has the most, as that's where I dumped the
original ones.  All the other nq's have accumulated since I made the
change, but are all draining quickly.

Looking at the process list, we can see where sendmail is obligingly
firing up multiple q runners automagically.

[root at pmx1 clientmqueue]# ps -ef | grep sendmail | grep nq
smmsp     3248 23625  0 16:17 ?        00:00:01 sendmail:
nq.3/mBBMDNoL001788 [127.0.0.1]: client DATA status
smmsp     5125 23625  0 16:22 ?        00:00:00 sendmail:
nq.3/mBBMErEr002548 [127.0.0.1]: client DATA status
smmsp     6645 23625  0 16:27 ?        00:00:00 sendmail:
nq.3/mBBMDbDJ001908 [127.0.0.1]: client DATA status
smmsp     8376 23625  0 16:32 ?        00:00:00 sendmail:
nq.2/mBBMUbVh007812 [127.0.0.1]: client DATA status
smmsp     9078  9074  2 16:35 ?        00:00:00 sendmail:
nq.1/mBBMZ2ct009078 [127.0.0.1]: client greeting
smmsp    23625     1  0 15:42 ?        00:00:00 sendmail: Queue
runner at 00:05:00 for /var/spool/clientmqueue/nq.*

I still need to do some more tuning on the q run frequency and min
queue age.  Need to work round the issue of the queues getting blocked
by undeliverable NDR's (i.e. bouncing from inside MD to domains which
are unreachable).  I'll update this thread with any findings.

Hey, by the time I finished typing this, that machine had emptied the queues:

[root at pmx1 mail]# mailq -Ac
/var/spool/clientmqueue/nq.4 is empty
/var/spool/clientmqueue/nq.3 is empty
/var/spool/clientmqueue/nq.1 is empty
/var/spool/clientmqueue/nq.2 is empty
/var/spool/clientmqueue/nq.5 is empty
                Total requests: 0

>   -- hugh

   -- hugh