Prev: Versioned documentation, was Re: postmap -q and ldap
Next: Multiple sites (and mail servers) for one domain
From: Christian Purnomo on 27 Jun 2010 18:50


Subject: Re: performance tuning - relay
Date: Fri, Jun 25, 2010 at 06:21:01PM -0500
Quoting Stan Hoeppner (stan(a)hardwarefreak.com):

: Can you provide some more specs on server2? IIRC you said you had a multidisk
: RAID array on serv2. What RAID level and how many disks? What filesystem?
: Are you running Courier with maildrop or the standalone maildrop with another
: IMAP server? What filtering, if any, are you doing with maildrop? Using mbox
: or maildir storage? IIRC you previously said you're BCC'ing _everything_ into
: a single mailbox (single address) on server2. Is this correct?
:
: And, lastly, was server2 in production for any amount of time before these
: problems occurred, prompting your post, or is this a new server that you just
: brought online?
:
: --
: Stan
:
:

Hi Stan,

All valid questions...

The real issue AFAIK was Server1 not feeding enough emails to Server2
and hence thousands of emails were queued on Server1 waiting to be
delivered to Server 2 via transport-relay. This was the major issue I
was trying to solve.

From your questions above, I could see where you're coming from that if
Server2 has performance problem then it would make sense to see the
queue built up at Server1. I can confirm server2 is very underload at
any time, the server is overspec'ed for what it is intended to do. I
can also confirm while those thousands of emails queued up at Server1,
Server2 was running smooth with 0.1-0.3 load average.

We have had server2 for about 4 years now and we have been having this
issues in the last 1 year where one of our new server happens to be a
mailling list which sends out thousands of emails to subscribers.

Anyway, Server2 spec is HP DL385G4, 4G RAM, 6 SCSI disks RAID 5 and
reiserfs.

The delivery method on Server2 is maildrop - we use some mailfilter rule
to drop certain emails to certain folders. I can understand this is
adding some overhead for the local delivery on Server2 but this is the
cost I'm happy to take on. The queue can build up on Server2 and clear
up overtime without impacting our primary MX (Server1).

CP

From: Stan Hoeppner on 28 Jun 2010 02:23
Christian Purnomo put forth on 6/27/2010 5:50 PM:

> From your questions above, I could see where you're coming from that if
> Server2 has performance problem then it would make sense to see the
> queue built up at Server1. I can confirm server2 is very underload at
> any time, the server is overspec'ed for what it is intended to do. I
> can also confirm while those thousands of emails queued up at Server1,
> Server2 was running smooth with 0.1-0.3 load average.

What piqued my curiosity is why the queue on server2 starting growing, and
rather large at that, _after_ you got the Postfix bottleneck straightened out
on server1.

> We have had server2 for about 4 years now and we have been having this
> issues in the last 1 year where one of our new server happens to be a
> mailling list which sends out thousands of emails to subscribers.
>
> Anyway, Server2 spec is HP DL385G4, 4G RAM, 6 SCSI disks RAID 5 and
> reiserfs.

I would have thought this hardware would be able to get the mails into the
mailboxen as quickly as server1 could push them over, without the queue
building up as you demonstrated in a previous message. Email service is
primarily a disk bound application. IIRC, with the DL385G4 you would have the
Smart Array 6i which is an integrated entry level controller. Even so, with
128MB of read/write cache and 6x10k(15?)rpm drives on a SCSI 320 bus, even in
a slowish RAID5 configuration, you should easily be able to sync to mailboxen
as many messages as server1 could push over either fast or gigabit ethernet.
This server should be able to sync a few hundred emails to disk per second.
Is the 6i just really horrible at RAID5, or is there something in the software
stack slowing things down? Were you peaking the disk subsystem when the queue
was building?

> The delivery method on Server2 is maildrop - we use some mailfilter rule
> to drop certain emails to certain folders. I can understand this is
> adding some overhead for the local delivery on Server2 but this is the
> cost I'm happy to take on. The queue can build up on Server2 and clear
> up overtime without impacting our primary MX (Server1).

I'm not familiar at all with maildrop as I've never used it. That said, I
wouldn't think maildrop alone would cause such a bottleneck. Some versions of
Reiser are known for great speed will lots of small files, at least as far as
delete performance. However, most versions of Reiser do not do so well with
large files. Reiser is normally a good performer with maildir, but doesn't do
so well with mbox, especially once the mbox files get large.

Other disk writes? Is maildrop or any other process you're running creating
extra log stamps per email processed? I assume you're storing the OS, logs,
mail, everything on that RAID5 volume. Is this correct?

As you stated, you're not really concerned with queue growth on server2. I
went through all this simply because I think you're leaving some performance,
maybe quite a bit, on the table WRT server2. I'm guessing it's in the
OS/software stack and not the hardware. You may be able to get this box
screaming with simple changes (reduce logging to only what's necessary), and
maybe one or two more major changes (maildir to mbox or vice versa, switching
from Reiser--defunct now anyway--to XFS). Or a really big change, dumping
Maildrop/Courier for Dovecot/LDA which is quite a bit quicker from everything
I've read. I say read because I've not used Courier but I have used Dovecot,
and still do.

Sorry if I've wasted your time here. I just thought I'd point out a few
things just in case you get the urge to poke around on server2 looking for a
little performance boost.

--
Stan

From: Christian Purnomo on 28 Jun 2010 02:30
Hi Stan

Subject: Re: performance tuning - relay
Date: Mon, Jun 28, 2010 at 01:23:15AM -0500
Quoting Stan Hoeppner (stan(a)hardwarefreak.com):

: What piqued my curiosity is why the queue on server2 starting growing, and
: rather large at that, _after_ you got the Postfix bottleneck straightened out
: on server1.

I was expecting this and don't have a problem with this limitation. The
maildrop rule is rather long and I knew I would get penalized. However
delays on local delivery on Server2 has no impact to production so it's
ok.

: I would have thought this hardware would be able to get the mails into the
: mailboxen as quickly as server1 could push them over, without the queue
: building up as you demonstrated in a previous message. Email service is
: primarily a disk bound application. IIRC, with the DL385G4 you would have the
: Smart Array 6i which is an integrated entry level controller. Even so, with
: 128MB of read/write cache and 6x10k(15?)rpm drives on a SCSI 320 bus, even in
: a slowish RAID5 configuration, you should easily be able to sync to mailboxen
: as many messages as server1 could push over either fast or gigabit ethernet.
: This server should be able to sync a few hundred emails to disk per second.
: Is the 6i just really horrible at RAID5, or is there something in the software
: stack slowing things down? Were you peaking the disk subsystem when the queue
: was building?
:
: I'm not familiar at all with maildrop as I've never used it. That said, I
: wouldn't think maildrop alone would cause such a bottleneck. Some versions of
: Reiser are known for great speed will lots of small files, at least as far as
: delete performance. However, most versions of Reiser do not do so well with
: large files. Reiser is normally a good performer with maildir, but doesn't do
: so well with mbox, especially once the mbox files get large.


Maildrop is just procmail for Maildir. I had to use Maildir
format as there are hundreds of thousands of email to the always_bcc
email on Server2.

: Other disk writes? Is maildrop or any other process you're running creating
: extra log stamps per email processed? I assume you're storing the OS, logs,
: mail, everything on that RAID5 volume. Is this correct?
:
: As you stated, you're not really concerned with queue growth on server2. I
: went through all this simply because I think you're leaving some performance,
: maybe quite a bit, on the table WRT server2. I'm guessing it's in the
: OS/software stack and not the hardware. You may be able to get this box
: screaming with simple changes (reduce logging to only what's necessary), and
: maybe one or two more major changes (maildir to mbox or vice versa, switching
: from Reiser--defunct now anyway--to XFS). Or a really big change, dumping
: Maildrop/Courier for Dovecot/LDA which is quite a bit quicker from everything
: I've read. I say read because I've not used Courier but I have used Dovecot,
: and still do.

Server2 wasn't my concern, Server1 was :)

The issue as far as I could see Server1 was unable to feed enough email
to Server2, I knew there was a limit somewhere on Server1 that prevented
this.

: Sorry if I've wasted your time here. I just thought I'd point out a few
: things just in case you get the urge to poke around on server2 looking for a
: little performance boost.


There is no such thing as wasting time here, I am grateful for anyone to
reply to my question. Thanks *_^

From: "Daniel V. Reinhardt" on 28 Jun 2010 03:32


----- Original Message ----
> From: Stan Hoeppner <stan(a)hardwarefreak.com>
> To: postfix-users(a)postfix.org
> Sent: Mon, June 28, 2010 2:23:15 AM
> Subject: Re: performance tuning - relay
>
> Christian Purnomo put forth on 6/27/2010 5:50 PM:

> From your
> questions above, I could see where you're coming from that if
> Server2
> has performance problem then it would make sense to see the
> queue built
> up at Server1. I can confirm server2 is very underload at
> any
> time, the server is overspec'ed for what it is intended to do. I
>
> can also confirm while those thousands of emails queued up at Server1,
>
> Server2 was running smooth with 0.1-0.3 load average.

What
> piqued my curiosity is why the queue on server2 starting growing, and
rather
> large at that, _after_ you got the Postfix bottleneck straightened out
on
> server1.

> We have had server2 for about 4 years now and we have been
> having this
> issues in the last 1 year where one of our new server
> happens to be a
> mailling list which sends out thousands of emails to
> subscribers.
>
> Anyway, Server2 spec is HP DL385G4, 4G RAM, 6 SCSI
> disks RAID 5 and
> reiserfs.

I would have thought this
> hardware would be able to get the mails into the
mailboxen as quickly as
> server1 could push them over, without the queue
building up as you
> demonstrated in a previous message. Email service is
primarily a disk
> bound application. IIRC, with the DL385G4 you would have the
Smart
> Array 6i which is an integrated entry level controller. Even so,
> with
128MB of read/write cache and 6x10k(15?)rpm drives on a SCSI 320 bus,
> even in
a slowish RAID5 configuration, you should easily be able to sync to
> mailboxen
as many messages as server1 could push over either fast or gigabit
> ethernet.
This server should be able to sync a few hundred emails to disk per
> second.
Is the 6i just really horrible at RAID5, or is there something in the
> software
stack slowing things down? Were you peaking the disk subsystem
> when the queue
was building?

> The delivery method on Server2 is
> maildrop - we use some mailfilter rule
> to drop certain emails to certain
> folders. I can understand this is
> adding some overhead for the
> local delivery on Server2 but this is the
> cost I'm happy to take
> on. The queue can build up on Server2 and clear
> up overtime
> without impacting our primary MX (Server1).

I'm not familiar at all with
> maildrop as I've never used it. That said, I
wouldn't think maildrop
> alone would cause such a bottleneck. Some versions of
Reiser are known
> for great speed will lots of small files, at least as far as
delete
> performance. However, most versions of Reiser do not do so well
> with
large files. Reiser is normally a good performer with maildir, but
> doesn't do
so well with mbox, especially once the mbox files get
> large.

Other disk writes? Is maildrop or any other process you're
> running creating
extra log stamps per email processed? I assume you're
> storing the OS, logs,
mail, everything on that RAID5 volume. Is this
> correct?

As you stated, you're not really concerned with queue growth on
> server2. I
went through all this simply because I think you're leaving
> some performance,
maybe quite a bit, on the table WRT server2. I'm
> guessing it's in the
OS/software stack and not the hardware. You may be
> able to get this box
screaming with simple changes (reduce logging to only
> what's necessary), and
maybe one or two more major changes (maildir to mbox
> or vice versa, switching
from Reiser--defunct now anyway--to XFS). Or a
> really big change, dumping
Maildrop/Courier for Dovecot/LDA which is quite a
> bit quicker from everything
I've read. I say read because I've not used
> Courier but I have used Dovecot,
and still do.

Sorry if I've wasted
> your time here. I just thought I'd point out a few
things just in case
> you get the urge to poke around on server2 looking for a
little performance
> boost.

--
Stan

-----------------

Stan,

Actually you do not need to pay for their mail forwarding services. I have a sever setup to accept email just fine and dandy for a dyndns.org support host, and I do not pay anything for it. I get mail to my system woa.homeip.net just fine without paying.

The paid for services you speak of are for people who want to customize their own dyndns settings.

You can send me an email to cryptodan(a)woa.homeip.net and I will receive it, and I can send out. I would suggest you get a dyndns.org account, and do some research on it.

I have been using dyndns.org since about 2001 when I first my DSL Connection.


Daniel Reinhardt
Website: www.cryptodan.com
Email: cryptodan(a)yahoo.com




From: "Daniel V. Reinhardt" on 28 Jun 2010 03:39





----- Original Message ----
> From: Daniel V. Reinhardt <cryptodan(a)yahoo.com>
> To: postfix-users(a)postfix.org
> Sent: Mon, June 28, 2010 3:32:04 AM
> Subject: Re: performance tuning - relay
>
>

----- Original Message ----
> From: Stan Hoeppner <
> ymailto="mailto:stan(a)hardwarefreak.com"
> href="mailto:stan(a)hardwarefreak.com">stan(a)hardwarefreak.com>
> To:
>
> href="mailto:postfix-users(a)postfix.org">postfix-users(a)postfix.org
>
> Sent: Mon, June 28, 2010 2:23:15 AM
> Subject: Re: performance tuning -
> relay
>
> Christian Purnomo put forth on 6/27/2010 5:50
> PM:

> From your
> questions above, I could see where you're
> coming from that if
> Server2
> has performance problem then it
> would make sense to see the
> queue built
> up at Server1. I
> can confirm server2 is very underload at
> any
> time, the server
> is overspec'ed for what it is intended to do. I
>
> can also
> confirm while those thousands of emails queued up at Server1,
>
>
> Server2 was running smooth with 0.1-0.3 load average.

What
>
> piqued my curiosity is why the queue on server2 starting growing,
> and
rather
> large at that, _after_ you got the Postfix bottleneck
> straightened out
on
> server1.

> We have had server2 for
> about 4 years now and we have been
> having this
> issues in the
> last 1 year where one of our new server
> happens to be a
>
> mailling list which sends out thousands of emails to
>
> subscribers.
>
> Anyway, Server2 spec is HP DL385G4, 4G RAM, 6 SCSI
>
> disks RAID 5 and
> reiserfs.

I would have thought
> this
> hardware would be able to get the mails into the
mailboxen as
> quickly as
> server1 could push them over, without the queue
building
> up as you
> demonstrated in a previous message. Email service
> is
primarily a disk
> bound application. IIRC, with the DL385G4
> you would have the
Smart
> Array 6i which is an integrated entry level
> controller. Even so,
> with
128MB of read/write cache and
> 6x10k(15?)rpm drives on a SCSI 320 bus,
> even in
a slowish RAID5
> configuration, you should easily be able to sync to
> mailboxen
as
> many messages as server1 could push over either fast or gigabit
>
> ethernet.
This server should be able to sync a few hundred emails to disk per
>
> second.
Is the 6i just really horrible at RAID5, or is there
> something in the
> software
stack slowing things down? Were you
> peaking the disk subsystem
> when the queue
was building?

>
> The delivery method on Server2 is
> maildrop - we use some mailfilter
> rule
> to drop certain emails to certain
> folders. I can
> understand this is
> adding some overhead for the
> local delivery
> on Server2 but this is the
> cost I'm happy to take
> on. The
> queue can build up on Server2 and clear
> up overtime
> without
> impacting our primary MX (Server1).

I'm not familiar at all with
>
> maildrop as I've never used it. That said, I
wouldn't think maildrop
>
> alone would cause such a bottleneck. Some versions of
Reiser
> are known
> for great speed will lots of small files, at least as far
> as
delete
> performance. However, most versions of Reiser do not
> do so well
> with
large files. Reiser is normally a good
> performer with maildir, but
> doesn't do
so well with mbox, especially
> once the mbox files get
> large.

Other disk writes? Is
> maildrop or any other process you're
> running creating
extra log
> stamps per email processed? I assume you're
> storing the OS,
> logs,
mail, everything on that RAID5 volume. Is this
>
> correct?

As you stated, you're not really concerned with queue growth on
>
> server2. I
went through all this simply because I think you're
> leaving
> some performance,
maybe quite a bit, on the table WRT
> server2. I'm
> guessing it's in the
OS/software stack and not
> the hardware. You may be
> able to get this box
screaming with
> simple changes (reduce logging to only
> what's necessary), and
maybe
> one or two more major changes (maildir to mbox
> or vice versa,
> switching
from Reiser--defunct now anyway--to XFS). Or a
>
> really big change, dumping
Maildrop/Courier for Dovecot/LDA which is quite a
>
> bit quicker from everything
I've read. I say read because I've
> not used
> Courier but I have used Dovecot,
and still do.

Sorry
> if I've wasted
> your time here. I just thought I'd point out a
> few
things just in case
> you get the urge to poke around on server2
> looking for a
little performance
> boost.

--
>
Stan

-----------------

Stan,

Actually you do not need
> to pay for their mail forwarding services. I have a sever setup to accept
> email just fine and dandy for a dyndns.org support host, and I do not pay
> anything for it. I get mail to my system woa.homeip.net just fine without
> paying.

The paid for services you speak of are for people who want
> to customize their own dyndns settings.

You can send me an email to
> ymailto="mailto:cryptodan(a)woa.homeip.net"
> href="mailto:cryptodan(a)woa.homeip.net">cryptodan(a)woa.homeip.net and I will
> receive it, and I can send out. I would suggest you get a dyndns.org
> account, and do some research on it.

I have been using dyndns.org since
> about 2001 when I first my DSL Connection.


Daniel
> Reinhardt
Website: www.cryptodan.com
Email:
> ymailto="mailto:cryptodan(a)yahoo.com"
> href="mailto:cryptodan(a)yahoo.com">cryptodan(a)yahoo.com



>


Please disregard my reply to this message thread. I was going to reply to another thread, but had this one highlighted. Sorry for butting in.

Daniel Reinhardt
Website: www.cryptodan.com
Email:
cryptodan(a)yahoo.com




First  |  Prev  | 
Pages: 1 2
Prev: Versioned documentation, was Re: postmap -q and ldap
Next: Multiple sites (and mail servers) for one domain