Category: linux

Managing The Exim Queue (For SysAdmins) – Removing Unwanted Messages From the Exim Queue

Have you ever had a rogue user trying to spam the world -on purpose or just because he catched a bad bug? -, a temporary problem with your Carrier, users who endlessly type hotmal, gmal, hotmial, instead of… you got the idea!  Then you’ve probably experienced having an email queue so long you can’t see the end of it and it would take hours to process.

You can see how many messages are in the queue issuing exim -bpc and see the queue with exim -bpu

In order to accelerate the queue delivery, we can give it Exim a hand.

Deleting all Frozen messagesice-34075_640

Frozen messages are just bounce messages, that is, notifications that a certain message delivery failed.

Those messages are stuck and going nowhere (**) so let’s just delete them.

You have two options:

exim -bpr | grep frozen | awk ‘{print $3}’ | xargs exim -Mrm

Basically we just examined the queue, look for the word frozen, got the message ID and pass that id to exim to delete it. Or even better run this to get the same result without affecting emails including the word frozen in some part of the email.

exiqgrep -z -i | xargs exim -Mrm

Now, If you don’t want to type that much just create a script to do that. For example:

#!/bin/bash
exiqgrep -z -i | xargs exim -Mrm

Call it removefrozen.sh or whatever you want; place it in an appropriate path and just call the script everytime you need it without the hassle.

Additionally, you might want to modify this lines in the exim configuration file:

ignore_bounce_errors_after = 2d      -> unfreezes a bounce and exim will try to deliver again
timeout_frozen_after = 7d                 -> deletes frozen messages after 7 days

to

ignore_bounce_errors_after = 12h         
timeout_frozen_after = 1d                    

To expedite the process of elimination of such bounces. Now let’s get rid of other messages.

(**) Frozen messages should be investigated to prevent them.

Deleting old messagesclock-147257_640

Messages that have been in the queue for a while, let’s say 1 day or 2 days are unlikely to be delivered. (*) Sometimes the remote mailbox is full, or the domain is non-existent and the message would be attempted to be delivered for as long as it is configured in the exim “configure” file; usually 4 days.

You can lower that value directly in the configuration file, but if you want to keep the default values and just delete old messages at your command you can run exiqgrep and delete messages based on certain time.

Now, this is something I don’t recommend doing, because you can prevent legitimate mail from being delivered, but if you have no option, the following example command deletes all messages which have been in the queue for more than 1 day (in seconds).

exiqgrep -o 86400 -i | xargs exim -Mrm

Again, you can add the command to a simple script for easier access.

Deleting messages to non-existent domains question-mark-160071_640

Now, there’s no way for exim to know if a domain is valid or not. It only knows if the DNS lookup has failed. (*) That’s why this is a tricky one. However, typos are obvious some times, and if you can identify them you can delete a good amount of emails. Now remember, you should give exim some time to attempt to deliver the message so the sender is notified of the failed delivery. Otherwise, the sender would never know he tried to send a message to a wrong recipient.

Some of the super typos I’ve seen are hotmal.com, hotmial.com, homail.com, yahooo.com, etc. A few of these domains exist but they’re not the ones the sender intended. Other cases would be very particular to your users so you might be able to identify those.

So, let’s delete messages to wrong domains with a script.

#!/bin/bash
exiqgrep -r $@ -i | xargs exim -Mrm

Call it, removerecipient.sh, i.e., and call the script with the domain or full email of the recipient you want to delete as a parameter.  Example: sh removerecipient.sh @hotmal.com

Or if you prefer typing the whole command: exiqgrep -r @hotmal.com -i | xargs exim -Mrm

IMPORTANT: The script above is just a basic example, it’s NOT sanitized and it will match partial coincidences. That is, if you intend to delete messages sent to gmal.com and you just execute removerecipient.sh gmal.com you’ll end up removing messages sent to gmal.com, agmal,com, logmal.com, etc., which may be potentially good domains.

The basic script is useful though if you want to delete thousands emails to russian, korean, etc. accounts and you have no business with them whatsoever, aka malware/SPAM.

Therefore, use with caution, use the whole email address (not just the domain) and verify the option passed as parameter inside the script. Also, ensure proper permissions are in place.

Deleting messages from non-existent senders  question-mark-160071_640

Use with the same precautions as the previous option. In this case, you delete messages from a certain sender, instead of a recipient.

#!/bin/bash
exiqgrep -f $@ -i | xargs exim -Mrm

Which it’s equivalent to exiqgrep -f fake@fakedomain.com -i | xargs exim -Mrm

Call it, removesender.sh, i.e., and call the script with the domain or full email of the recipient you want to delete as a parameter.  Example: sh removesender.sh @hotmal.com

Identifying the culprit of a spam attack magnifying-glass-29398_640

Now, all the delete commands stated above won’t stop a spam attack, they will only help in cleaning the email queue. You should identify the sender of the spam and block him. How?

First, you have to identify one of the spam messages and grabs its ID.

See the queue: exim -bpu

Identify a suspicious email, let’s say with ID 1ZCB3s-0007lp-N6 and check its body and its headers.

Checking the body (to verify your suspicion is correct)

exim -Mvb <message-id>
Example: exim -Mvb 1ZCB3s-0007lp-N6

If you see something like viagra promotions, malicious links, etc., you have a winner. Now, let’s check the headers:

exim -Mvh <message-id>
Example: exim -Mvh 1ZCB3s-0007lp-N6

I recommend using less afterwards to see the very first lines:

exim -Mvh 1ZCB3s-0007lp-N6 | less

Now look for the line starting with -auth_id. There you should see the username used to authenticate that email as valid to your email server. Don´t trust the sender you see in the queue because that’s just the envelope. For example you can see in the queue an email from remote@invalid.com with auth_id local@valid.com.

Once you have the address used to auth the spam, block it from your email server (suspend the account and/or change the password), warn the legitimate user, and adjust your spam filters accordingly. Also, you can block in your firewall the sender origin IP indicated in the line starting with -host_address, i.e. the spammer ip address.

Deleting ALL messages in the queue

If for some very special reason you want to delete ALL the email queue, then just issue:

 exiqgrep -i  |  xargs exim -Mrm

——-

In next posts, more queue management options coming.

(*) Always check if you don’t have a problem with DNS resolution.

Advertisement

Configuring a NTP server in different Linux distributions

To configure a Linux computer as a basic NTP server, we need to go through the following steps:

Installing the NTP package

For CentOs, RedHat and Fedora distributions
yum install ntp

For Ubuntu and Debian distributions
apt-get install ntp
or
sudo apt-get install ntp

Configuring the NTP service

With your favorite text editor open the file /etc/ntp.conf to perform the following:

Specifying OUR external server for time synchronization

Our Time source (or sources) are specified in the lines starting with server. These are the servers to which our local server synchronizes. Here is an example:

server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

Default server addresses vary according to the Linux distribution. You can use the default ones or specify the NTP server of your preference.  It is recommended to use the servers for your country or region specified in pool.ntp.org Server link: http://support.ntp.org/bin/view/Servers/NTPPoolServers.

It is recommended than more than one ntp server is present for redundancy in case one of the servers fail.

You might have noticed the word iburst in every line. The iburst option reduces the time in the initial synchronization and it is the default option.

The specified servers are used in a round-robin fashion. If you wish you to use one server above others, add the options prefer at the end of the server line like this:

server 0.centos.pool.ntp.org prefer

Allow clients restricted time synchronization with our time server

If  you want to allow any client to synchronize to your NTP server just add the line

restrict default kod nomodify notrap nopeer noquery

  • default: every client
  • kod: if access is denied, send a “kiss of death” packet. It’s used to prevent abuse of the server. See more here.
  • nomodify: client can not modify the server
  • notrap: prevents ntpdc control message protocol traps
  • nopeer: deny packets trying to establish a peer association
  • noquery: clients can not query status information from our server (like our Operating System or NTP server version); however, they can still get time sync from our NTP server. If you want to block time synchronization you should add the option noserve.

Allow localhost IP to perform any function in the NTP server

The localhost ip 127.0.0.1 is often used for administrative functions; hence, to bypass the “default” restrict policy, add:

restrict 127.0.0.1
restrict ::1

The above lines give the localhost IP address unrestricted access to the NTP server operations.

Specifying the Location of the Drift File

The driftfile keeps track of clock deviations.

driftfile /var/lib/ntp/ntp.drift

That’s pretty much it for a basic configuration, but let’s just add some extra features for security. (This post doesn’t cover advanced security options like authentication).

Allow only specific clients to synchronize with our NTP server

If you added the noserve option in the restrict default line you might want to add specif hosts or networks which will be allowed to synchronize with your server. They usually are hosts or networks in your LAN, or otherwise known devices. For example:

If you want to allow the network 10.10.10.0/24 to query your NTP server add the following line:

restrict 10.10.10.0 mask 255.255.255.0 nomodify notrap nopeer 

If you want to allow the specific host 10.10.10.2 to query your NTP server add the following line:

restrict 10.10.10.2 nomodify notrap nopeer                -> mask 255.255.255.255 is assumed

That network and that host would be able to query your server for time sync and get status information from your server.

Using Local Clock as Backup

In case your server loses access to the Internet, it’s a good idea to failover to the server internal clock.

server 127.127.1.0                                   -> NTP server’s own pseudo address
fudge 127.127.1.0 stratum 12

Use stratum 10 to 15 so it’s never used unless no external server is reachable.

Configuring Logging Parameters

Specify a file path for the logs, although no mandatory it is very useful for debugging:

logfile /var/log/ntp.log

 

A complete basic /etc/ntp.conf should look like this (As you can imagine the -6 lines, are intended for IPv6 protocol) :


 

server 127.127.1.0                                  
fudge 127.127.1.0 stratum 12               

server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst
server 2.pool.ntp.org iburst
server 3.pool.ntp.org iburst

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

restrict 127.0.0.1
restrict -6 ::1

driftfile /var/lib/ntp/ntp.drift
logfile /var/log/ntp.log


 

Starting the NTP Service

For  CentOs and RHEL/RedHat  distributions
/etc/init.d/ntpd start
or
service ntpd start

For Debian and Ubuntu  distributions (notice there’s a d missing)
/etc/init.d/ntp start
or
service ntp start

For Fedora and CentOS/Redhat/RHEL 7 distributions
 systemctl start ntpd.service

The system will start synchronizing.

Verifying the NTP operation

Enter the command:
# ntpq -p

And you’ll see an output similar as the follow:

ntpq

Values will be moving through time. The entry marked with the * is the server currently in use. If you see errors or no server association in a few minutes, probably the NTP service has not started. Review the log file to obtain more information.

Sync the local clock with the external NTP server for the first time

Issue the following command:
ntpdate –u 18.26.4.105                        -> or any server your NTP server is syncing to

Make the NTP service start when the computer boots

To add the ntp service to the list of daemons which start at startup:

For  CentOs and RHEL/RedHat  distributions
chkconfig ntpd on 

For Debian and Ubuntu  distributions (notice there’s a d missing)
systemctl enable ntp.service
o for Debian Jessie (Debian 8)
systemctl enable ntp

For Fedora and CentOS/Redhat/RHEL 7 distributions
systemctl enable ntpd

 

Configuring Linux NTP client in different distributions

To configure a Linux computer as a basic NTP client, we need to go through the following steps:

Installing the NTP package

For CentOs, RedHat and Fedora distributions
yum install ntp

For Ubuntu and Debian distributions
apt-get install ntp
or
sudo apt-get install ntp

Configuring the NTP service

With your favorite text editor, make sure these lines are present in the file /etc/ntp.conf

server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

It is also wise to make sure the log file is established:

logfile /var/log/ntp.log

Server addresses vary according to the Linux distribution. You can use the default ones or specify your own time server or a public NTP server of your preference.  You can find a list of public ntp servers in this link: http://support.ntp.org/bin/view/Servers/NTPPoolServers.

It is recommended than more than one ntp server is present for redundancy in case one of the servers fail.

You might have noticed the word iburst in every line. The iburst option reduces the time in the initial synchronization and it is the default option.

The specified servers are used in a round-robin fashion. If you wish you to use one server above others, add the options prefer at the end of the server line like this:

server 0.centos.pool.ntp.org prefer

Starting the NTP Service

For  CentOs and RHEL/RedHat  distributions
/etc/init.d/ntpd start
or
service ntpd start

For Debian and Ubuntu  distributions (notice there’s a d missing)
/etc/init.d/ntp start
or
service ntp start

For Fedora and CentOS/Redhat/RHEL 7 distributions
 systemctl start ntpd.service

The system will start synchronizing.

Verifying the NTP operation

Enter the command:
# ntpq -p

And you’ll see an output similar as the follow:

ntpq

Values will be moving through time. The entry marked with the * is the server currently in use. If you see errors or no server association in a few minutes, probably the NTP service has not started. Review the log file to obtain more information.

Sync the local clock with the NTP server for the first time

Issue the following command:
ntpdate –u 18.26.4.105                        -> or any server your computer is syncing to

Make the NTP service start when the computer boots

To add the ntp service to the list of daemons which start at startup:

For  CentOs and RHEL/RedHat  distributions
chkconfig ntpd on 

For Debian and Ubuntu  distributions (notice there’s a d missing)
systemctl enable ntp.service
o for Debian Jessie (Debian 8)
systemctl enable ntp

For Fedora and CentOS/Redhat/RHEL 7 distributions
systemctl enable ntpd

I know, I know, I wish everybody could standardize. Feel free to comment with any possible correction or suggestions to this post.