Prev: Retriction order - a little advice please
Next: How to forward mail when mailbox not localy existing
From: Gary Smith on 1 May 2010 13:58
Lately I have found that my outgoing queues are getting a little clogged for yahoo and sbcglobal.net. This usually coincides with a bulk set of news letters sent out from a couples clients. Typically we are seeing that they dump about 2000msg/per batch, with no more than one batch per week (usually on Friday nights). The problem is they do it in one fatal swoop. When this happens, we seem to get rate limited from yahoo/sbcglobal for a few hours, which directly affects our other users.

Some time ago I put a rate limiter in place for AOL/yahoo/sbcglobal/gmail so we wouldn't be bombarding but I don't think that it's being honored, probably because I missed something. When I do a postconf I don't see my rate limiter, which is in main.cf, listed there. I'm not sure if that's by design or part of a misconfiguration on my part. Mail goes out from different sets of servers, with the same public IP, so rate limiting will only affect the bulk mail queue.

Is there something obvious that I missed?

master.cf:
ratelimit unix - - n - 3 smtp

rate_limit_transport:
aol.com ratelimit:
yahoo.com ratelimit:
sbcglobal.net ratelimit:
gmail.com ratelimit:

main.cf:
ratelimit_destination_concurrency_failed_cohort_limit = 100
ratelimit_destination_concurrency_limit = 10

postconf -n:

alias_maps = hash:/etc/aliases
broken_sasl_auth_clients = yes
command_directory = /usr/sbin
config_directory = /etc/postfix
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
debug_peer_level = 2
html_directory = no
mail_owner = postfix
mailq_path = /usr/bin/mailq
manpage_directory = /usr/share/man
myhostname = ...
mynetworks = /etc/postfix/custom/mynetworks
newaliases_path = /usr/bin/newaliases
queue_directory = /var/spool/postfix
readme_directory = /usr/share/doc/postfix-2.6.3/README_FILES
sample_directory = /usr/share/doc/postfix-2.6.3/samples
sendmail_path = /usr/sbin/sendmail
setgid_group = postdrop
smtp_tls_note_starttls_offer = yes
smtp_use_tls = yes
smtpd_client_restrictions = permit_mynetworks,permit_sasl_authenticated,reject
smtpd_recipient_restrictions = permit_mynetworks,permit_sasl_authenticated,reject
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = yes
smtpd_sasl_security_options = noanonymous
smtpd_tls_cert_file = ...
smtpd_tls_key_file = ...
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
smtpd_use_tls = yes
tls_random_source = dev:/dev/urandom
transport_maps = hash:/etc/postfix/custom/rate_limit_transport, hash:/etc/postfix/maps/transport
unknown_local_recipient_reject_code = 550

From: Noel Jones on 1 May 2010 14:12
On 5/1/2010 12:58 PM, Gary Smith wrote:
> Lately I have found that my outgoing queues are getting a little clogged for yahoo and sbcglobal.net. This usually coincides with a bulk set of news letters sent out from a couples clients. Typically we are seeing that they dump about 2000msg/per batch, with no more than one batch per week (usually on Friday nights). The problem is they do it in one fatal swoop. When this happens, we seem to get rate limited from yahoo/sbcglobal for a few hours, which directly affects our other users.
>
> Some time ago I put a rate limiter in place for AOL/yahoo/sbcglobal/gmail so we wouldn't be bombarding but I don't think that it's being honored, probably because I missed something. When I do a postconf I don't see my rate limiter, which is in main.cf, listed there. I'm not sure if that's by design or part of a misconfiguration on my part. Mail goes out from different sets of servers, with the same public IP, so rate limiting will only affect the bulk mail queue.
>

"custom" parameter names don't display in postconf output by
design.

> Is there something obvious that I missed?
>
> master.cf:
> ratelimit unix - - n - 3 smtp
>
> rate_limit_transport:
> aol.com ratelimit:
> yahoo.com ratelimit:
> sbcglobal.net ratelimit:
> gmail.com ratelimit:
>
> main.cf:
> ratelimit_destination_concurrency_failed_cohort_limit = 100
> ratelimit_destination_concurrency_limit = 10
>

This looks reasonable to me; no more than 3 connections should
be made at a time to any combination of those destinations.
Why don't you think it's working?

Hint: it's helpful to add a custom log name to a master.cf
transport entry to make sure it's being used.
-o syslog_name=postfix-ratelimit
http://www.postfix.org/postconf.5.html#syslog_name

> postconf -n:
[not examined]

-- Noel Jones

From: Gary Smith on 1 May 2010 14:31
> > rate_limit_transport:
> > aol.com ratelimit:
> > yahoo.com ratelimit:
> > sbcglobal.net ratelimit:
> > gmail.com ratelimit:
> >
> This looks reasonable to me; no more than 3 connections should
> be made at a time to any combination of those destinations.
> Why don't you think it's working?

I'm not sure why I think it's not working. Skimming the log file, shortly after the back was launched, we saw several of these messages:

connect to sbcgloabal.net[208.73.210.27]:25: Connection refused
connect to comcst.net[216.240.187.144]:25: Connection refused
connect to eathlink.net[216.65.41.185]:25: Connection refused

(obviously the last two aren't on the list, but will be added). Anyway, I will start logging see if it's working. I also just noticed that the rate limiting file was touched this week, so I need to find out what was touched (which it hasn't been touched in a year since we set it up).

>
> Hint: it's helpful to add a custom log name to a master.cf
> transport entry to make sure it's being used.
> -o syslog_name=postfix-ratelimit
> http://www.postfix.org/postconf.5.html#syslog_name
>
> > postconf -n:
> [not examined]
>
> -- Noel Jones

From: Mike Morris on 1 May 2010 14:49
On 05/01/2010 11:31 AM, Gary Smith wrote:
>>> rate_limit_transport:
>>> aol.com ratelimit:
>>> yahoo.com ratelimit:
>>> sbcglobal.net ratelimit:
>>> gmail.com ratelimit:
>>>
>> This looks reasonable to me; no more than 3 connections should
>> be made at a time to any combination of those destinations.
>> Why don't you think it's working?
>
> I'm not sure why I think it's not working. Skimming the log file, shortly after the back was launched, we saw several of these messages:
>
> connect to sbcgloabal.net[208.73.210.27]:25: Connection refused
> connect to comcst.net[216.240.187.144]:25: Connection refused
> connect to eathlink.net[216.65.41.185]:25: Connection refused
>
> (obviously the last two aren't on the list, but will be added). Anyway, I will start logging see if it's working. I also just noticed that the rate limiting file was touched this week, so I need to find out what was touched (which it hasn't been touched in a year since we set it up).
>

Do you realize the entries you just posted are misspelled domains? They
are not sbcglobal.net, comcast.net, or earthlink.net.

-Mike

From: Noel Jones on 1 May 2010 15:10
On 5/1/2010 1:31 PM, Gary Smith wrote:
>>> rate_limit_transport:
>>> aol.com ratelimit:
>>> yahoo.com ratelimit:
>>> sbcglobal.net ratelimit:
>>> gmail.com ratelimit:
>>>
>> This looks reasonable to me; no more than 3 connections should
>> be made at a time to any combination of those destinations.
>> Why don't you think it's working?
>
> I'm not sure why I think it's not working. Skimming the log file, shortly after the back was launched, we saw several of these messages:
>
> connect to sbcgloabal.net[208.73.210.27]:25: Connection refused
> connect to comcst.net[216.240.187.144]:25: Connection refused
> connect to eathlink.net[216.65.41.185]:25: Connection refused
>

These are dead domains, either owned by squatters or web
redirectors owned by the "real" domain. The domains exist but
will never accept mail, so postfix will retry until the mail
expires. You can instruct postfix to reject common
misspellings like this by adding transport map entries such as:

# transport
eathlink.net error:5.1.2 no such domain
sbcgloabal.net error:5.1.2 no such domain
comcst.net error:5.1.2 no such domain


> (obviously the last two aren't on the list, but will be added). Anyway, I will start logging see if it's working. I also just noticed that the rate limiting file was touched this week, so I need to find out what was touched (which it hasn't been touched in a year since we set it up).

For misspelled domains, either ignore them or add a transport
entry to reject them right away. Obviously you can't add
every possible bad domain, but it's helpful to add the top 10
or so for your users.

-- Noel Jones

 |  Next  |  Last
Pages: 1 2
Prev: Retriction order - a little advice please
Next: How to forward mail when mailbox not localy existing