Prev: Restrict certain mailboxes to a few MTAs
Next: Routing messages according IPs
From: =?ISO-8859-1?Q?Ignacio_Garc=EDa?= on 14 Apr 2010 15:15
Hi there. Some days ago 1 of our postfix servers was abused by bot
networks using one of our customer's stolen credentials, inadvertently
done by a virus/keylogger probably. In few hours more than 20000 spam
messages were in our queue. Looking at the logs I realized all those
outgoing messages came authenticated with the same stolen user
credentials and from many different geolocations. Just changing the
password solved the problem. This is a very disturbing issue for us,
since it is hard to notice there's something going on until the server
is already puking spam all over. Does anybody know of an automatic way
of preventing this (or at least an automatic way of blocking it in early
stages)? We were thinking of something like a script monitoring the logs
for same-user authenticated connections from different IPs to create a
blacklist of some sort...

Thanks in advance.

Ignacio

From: ram on 15 Apr 2010 04:28

On Wed, 2010-04-14 at 21:15 +0200, Ignacio García wrote:

> Hi there. Some days ago 1 of our postfix servers was abused by bot
> networks using one of our customer's stolen credentials, inadvertently
> done by a virus/keylogger probably. In few hours more than 20000 spam
> messages were in our queue. Looking at the logs I realized all those
> outgoing messages came authenticated with the same stolen user
> credentials and from many different geolocations. Just changing the
> password solved the problem. This is a very disturbing issue for us,
> since it is hard to notice there's something going on until the server
> is already puking spam all over. Does anybody know of an automatic way
> of preventing this (or at least an automatic way of blocking it in early
> stages)? We were thinking of something like a script monitoring the logs
> for same-user authenticated connections from different IPs to create a
> blacklist of some sort...
>
> Thanks in advance.
>
> Ignacio



This is very common problem. Search the archives for older
conversations
One of them is here

http://groups.google.com/group/mailing.postfix.users/browse_thread/thread/596a160388faba35/862d6abf348b8962





From: =?UTF-8?B?SWduYWNpbyBHYXJjw61h?= on 15 Apr 2010 11:28
El 15/04/10 12:41, ram escribió:
>
> The points mentioned should help you especially ratelimits , and FBL's
>
> Are you planning to do outgoing scanning.

Hi Ram. I believe ratelimits and FBLs can help, but just partially. FBLs
are of great help, but they work only after much harm has been done. For
instance, right now we use FBLs to get warnings of this kind of problem
(besides checking the logs, of course, which does not happen 24
hours/day). When we got our first warning we had more than 20k spam
messages in the queue. OTOH, ratelimiting could work well. However, we
have several customers with internal/intranet mail servers in their own
facilities (with residnetial connections and dynamic IPs) who use our
mail servers as authenticated SMTP relays to send external mail to the
Internet, so limiting the number of outbound emails can be a problem for us.

The way I think this could be solved is by having a program that:

1.- Checks the logs for authenticated smtp usage and saves
smtp_authenticated_user, originating IPs, and country, which is
dicovered using ip geolocation.
2.- During the following minutes, if IP from same authenticated user is
different, then geolocate new IP, and if country is also different then
set it as possible credential theft.
3.- If Step 2 repeats few times in few minutes (or even worse, if a
third country is detected), then we sure have stolen credentials.
4.- Add smtp_authenticated_user to a blacklist, could add a simple
header_checks entry to reject messages with smtp_authenticated_user
header. That way account is still active and able to receive messages.
However, outbound messaging is disabled.
5.- We could use a granulated scoring system. For instance, we are in
Spain, and 99.9% of our customers are in Spain. So, even if more
different IPs are used in short period of times, but all originate in
Spain, it's fair to assume this person may be having connectivity
problems or several devices connected (computer, 3g phone, pda) and
running at the same time, so we cut them some slack :)

We are already brainstorming this. However, were are good sysadmins but
I cannot say the same about complex programming. We'll see what happens.

Regards,

Ignacio
From: John Fawcett on 16 Apr 2010 17:33
On 15/04/10 17:28, Ignacio García wrote:
> The way I think this could be solved is by having a program that:
>
> 1.- Checks the logs for authenticated smtp usage and saves
> smtp_authenticated_user, originating IPs, and country, which is
> dicovered using ip geolocation.
> 2.- During the following minutes, if IP from same authenticated user
> is different, then geolocate new IP, and if country is also different
> then set it as possible credential theft.
> 3.- If Step 2 repeats few times in few minutes (or even worse, if a
> third country is detected), then we sure have stolen credentials.
> 4.- Add smtp_authenticated_user to a blacklist, could add a simple
> header_checks entry to reject messages with smtp_authenticated_user
> header. That way account is still active and able to receive messages.
> However, outbound messaging is disabled.
> 5.- We could use a granulated scoring system. For instance, we are in
> Spain, and 99.9% of our customers are in Spain. So, even if more
> different IPs are used in short period of times, but all originate in
> Spain, it's fair to assume this person may be having connectivity
> problems or several devices connected (computer, 3g phone, pda) and
> running at the same time, so we cut them some slack :)
>
> We are already brainstorming this. However, were are good sysadmins
> but I cannot say the same about complex programming. We'll see what
> happens.
>
> Regards,
>
> Ignacio
Ignacio
I've been using cbpolicyd to do rate limiting on submission port not
because I want to rate limit legitimate users, but to protect against
stolen credentials.
The approach of scanning the logfile that you outline, though not real
time like cbpolicyd would be almost as good if run often enough so that
the window in which spam could leak out was small.

So I have made a prototype (sorry but it's php not perl) that can parse
log files and write postfix access files. It does not do all you wanted,
for example the part about country lookups, though feasible would not
catch the case of stolen credentials from ips within the same country. I
have only implemented two limits: a limit on number of authentications
within a certain time frame and limit on the number of different ips for
a single user in the time frame. If either limit is reached then the IP
is blacklilsted in an access file. Maybe it is useful or you could
extend it for your needs.

http://www.gufonero.com/postfix/check_auth_log_0.1.tgz

After customizing file locations and limits, you can schedule the script
to run from cron, but you need to also include the postmap of the access
file (makefile -f check_auth_log_makefile all) afterwards. It assumes
you use hash access files. Update of main.cf / master.cf is needed to
add in the check of the access map, e.g. for the submission port. You'd
probably also want a whitelist access file in front of this to opt
certain heavy users out of the check.
regards,
John
From: =?ISO-8859-15?Q?Ignacio_Garc=EDa?= on 18 Apr 2010 11:27
El 16/04/10 23:33, John Fawcett escribió:
> I've been using cbpolicyd to do rate limiting on submission port not
> because I want to rate limit legitimate users, but to protect against
> stolen credentials.
> The approach of scanning the logfile that you outline, though not real
> time like cbpolicyd would be almost as good if run often enough so
> that the window in which spam could leak out was small.
>
> So I have made a prototype (sorry but it's php not perl) that can
> parse log files and write postfix access files. It does not do all you
> wanted, for example the part about country lookups, though feasible
> would not catch the case of stolen credentials from ips within the
> same country. I have only implemented two limits: a limit on number of
> authentications within a certain time frame and limit on the number of
> different ips for a single user in the time frame. If either limit is
> reached then the IP is blacklilsted in an access file. Maybe it is
> useful or you could extend it for your needs.
>
> http://www.gufonero.com/postfix/check_auth_log_0.1.tgz
>
> After customizing file locations and limits, you can schedule the
> script to run from cron, but you need to also include the postmap of
> the access file (makefile -f check_auth_log_makefile all) afterwards.
> It assumes you use hash access files. Update of main.cf / master.cf is
> needed to add in the check of the access map, e.g. for the submission
> port. You'd probably also want a whitelist access file in front of
> this to opt certain heavy users out of the check.
> regards,
> John

John, thanks so much for your interest. I just downloaded it. Tomorrow I
will be meeting my partner (he's the php guy) adn we will take a look at
it. I'll certainly keep you posted of any developments.

Best regards,

Ignacio
 |  Next  |  Last
Pages: 1 2
Prev: Restrict certain mailboxes to a few MTAs
Next: Routing messages according IPs