Prev: back-scatter
Next: adding header to outgoing mail
From: Gary Smith on 28 Apr 2010 14:08
We have a custom content filter in place. During the execution of this filter we create a set of files, per message, for the purpose of being processes after the filter is finished. The goal in that was to get the mail back into postfix ASAP.

In the background we have a cronjob that goes through the sets of files that are created via the filter. It is pretty inefficient as it processes these files one at a time in sequential order. We are in the process of streamlining that and have created an application to process the sets one at a time, given the set filename, so we run these in parallel. This cronjob runs on a separate server to reduce load on the postfix boxes. The problem with the cronjob, or any of the process jobs in general, is that this is all on a NFS cluster and we spend most of the disk time searching folders for files to be processed.

What I would ideally like to do is to call the new pipeline at the end of the content filter as a background process. I had first intended to just do "process.sh > /dev/null &" in order to make it a background process. Alternatively I could issue a fork inside the process application and call it like a normal file. I'm not sure what impact either of these will have on postfix since it's kicked off from postfix. If this process that is kicked off fails, I still have backup cronjobs that walks the file system. The process we are talking about here is a TCP connection to a separate server that will listen, but may have a delayed backlog, but shouldn't take any real CPU and limited memory, just time.

Given this, I'm not even sure if I should even attempt to do it at the end of our filter. Anyone have any thoughts on this approach?

From: Gary Smith on 11 May 2010 11:38
Given the message below, if I fork a process inside a content filter (say in python or perl), so I can return the message back to postfix faster (and end the content pipe fast with a success exist code), will this have any impact on postfix?

> We have a custom content filter in place. During the execution of this filter
> we create a set of files, per message, for the purpose of being processes
> after the filter is finished. The goal in that was to get the mail back into
> postfix ASAP.
>
> In the background we have a cronjob that goes through the sets of files that
> are created via the filter. It is pretty inefficient as it processes these
> files one at a time in sequential order. We are in the process of
> streamlining that and have created an application to process the sets one at a
> time, given the set filename, so we run these in parallel. This cronjob runs
> on a separate server to reduce load on the postfix boxes. The problem with
> the cronjob, or any of the process jobs in general, is that this is all on a
> NFS cluster and we spend most of the disk time searching folders for files to
> be processed.
>
> What I would ideally like to do is to call the new pipeline at the end of the
> content filter as a background process. I had first intended to just do
> "process.sh > /dev/null &" in order to make it a background process.
> Alternatively I could issue a fork inside the process application and call it
> like a normal file. I'm not sure what impact either of these will have on
> postfix since it's kicked off from postfix. If this process that is kicked
> off fails, I still have backup cronjobs that walks the file system. The
> process we are talking about here is a TCP connection to a separate server
> that will listen, but may have a delayed backlog, but shouldn't take any real
> CPU and limited memory, just time.
>
> Given this, I'm not even sure if I should even attempt to do it at the end of
> our filter. Anyone have any thoughts on this approach?

From: Wietse Venema on 11 May 2010 13:14
Gary Smith:
> Given the message below, if I fork a process inside a content
> filter (say in python or perl), so I can return the message back
> to postfix faster (and end the content pipe fast with a success
> exist code), will this have any impact on postfix?

If the filter reports success to Postfix before giving the FILTERED
message to the Postfix queue, then Postfix will remove the UNFILTERED
message from the queue too early, and you will lose mail when (not
if) the filter has a problem.

Wietse

From: Gary Smith on 11 May 2010 13:40
> If the filter reports success to Postfix before giving the FILTERED
> message to the Postfix queue, then Postfix will remove the UNFILTERED
> message from the queue too early, and you will lose mail when (not
> if) the filter has a problem.
>

The filter re-injects the message back into the queue prior to what I am doing. Basically, I'm saving a copy of the message and the envelope, re-injecting it back to postfix, then exiting with 0 (similar to a callout to spamassassin). Prior to that exit call, I need to call a script that will process this message. Let's call it the post_content_file script. I don't care if the post_content_filter script succeeds or fails (as I have a backup process to handle some of it), I just need to trigger it.

The idea is at the end of the content_filter, prior to exiting with 0, to call a script that will internally fork, then immediately return back to the original content_filter, so it can exit.

My question is will this fork process cause any problems with postfix itself? I just don't know what the impact of a fork in the content_filter will be.

Gary-

From: Victor Duchovni on 11 May 2010 13:52
On Tue, May 11, 2010 at 10:40:05AM -0700, Gary Smith wrote:

> My question is will this fork process cause any problems with postfix
> itself? I just don't know what the impact of a fork in the content_filter
> will be.

Just make sure to close stdout and stderr, to avoid writing garbage
into the pipe between Postfix and the filter, used to collect filter
error messages.

With this level of complexity, you really should use the advanced (SMTP)
filter approach not pipe(8) based filters.

--
Viktor.

P.S. Morgan Stanley is looking for a New York City based, Senior Unix
system/email administrator to architect and sustain our perimeter email
environment. If you are interested, please drop me a note.

 |  Next  |  Last
Pages: 1 2 3 4
Prev: back-scatter
Next: adding header to outgoing mail