Redirect stdout and stderr preserving the original order [Shell]

Prev: list out thousand of files
Next: unzip | touch | re-zip

From: Radoulov, Dimitre on 30 Dec 2009 05:57

On 30/12/2009 0.27, Icarus Sparry wrote:

Hi Icarus Sparry,
thank you very much for replying!

> On Tue, 29 Dec 2009 18:11:01 +0100, Radoulov, Dimitre wrote:
>
>> Hi all,
>> There was a question on a web forum on how to redirect stdin and stderr
>> in the following manner:
>>
>> - two separate files for both streams - a third file containing both
>> streams in their original order
>>
>> We haven't come up with a solution and I'm curious to hear your opinion.
>
> For typical programs, this is very hard. The reason is that almost all
> programs use the "stdio" (standard I/O) library.
>
> stdio has a default buffering strategy for stderr, that says it is
> unbuffered. So sending anything to stderr makes it appear pretty much
> straight away.
>
> The strategy for stdout depends on if the library thinks it is wtiting to
> a terminal or not. If it is not then the output only gets sent when an
> internal buffer is full or the program is terminating. If it is writing
> to a terminal then output gets sent when a line is completed.
>
> So this means that a program
>
> int main(int ac, char *av[]){ printf("Hello\n");
> fprintf(stderr,"World\n"); return 0;}
>
> will output the two words in a different order depending on if it is
> being run with the output going to a pipe (or file) or if it is going to
> a console.
>
> If we assume that you want the "console" output order, then you need to
> convince the stdio library that your program is talking to a console.
> There are a couple of ways you can do this, I like an old program "pty",
> but for most people "expect" is easier to find. It comes with an example
> script "unbuffer" that does exactly what we want.

As another poster already mentioned, the unbuffer expect script found on
Internet combines the stdout and stderr (I don't know if that script
could be rewritten in order to handle the streams differently, I'll
check that).

> At this point you then just need to have a program that listens on
> multiple inputs and writes to multiple outputs, and a way to tie them
> together. Such a program is "multitee" (from the multitee package in
> Debian).

I didn't know about multitee, thank you!
I'll try it out.

> One way to tie them together is with a named pipe.
>
> # Create a pipe for the stderr of the program.
> mkfifo sepipe
> { sleep 1 # To make sure that the shell sets up a reader for sepipe
> unbuffer program 2>sepipes
> rm sepipe
> } | multitee 3<sepipe>stdout 2>stderr 4>both 0:1,4 3:2,4

This one appears to hang, am I missing something?

zsh-4.3.10[t]% cat s
#!/bin/sh

for i in 1 2; do
for j in 1 2; do
printf '%s\n' "out $i"
done
for k in 1 2; do
printf '%s\n' "err $i" >&2
done
done

zsh-4.3.10[t]% cat unbuffer
#!/bin/sh
#
# Description: unbuffer stdout/stderr of a program
# Author: Don Libes, NIST
#
# Option: -p allow run program to read from stdin (for simplification)
#
# Note that expect can 'continue' a comment line, so the follow re-runs this
# as a expect command regardless of its PATH location.
# \
exec expect -- "$0" ${1+"$@"}

if {[string compare [lindex $argv 0] "-p"] == 0} {
# pipeline
set stty_init "-echo"
eval spawn -noecho [lrange $argv 1 end]
interact
} else {
set stty_init "-opost"
eval spawn -noecho $argv
set timeout -1
expect
}

zsh-4.3.10[t]% cat test3
set -xv

mkfifo sepipe
{ sleep 1 # To make sure that the shell sets up a reader for sepipe
unbuffer ./s 2>sepipes
rm sepipe
} | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4

set +xv

zsh-4.3.10[t]% . ./test3

mkfifo sepipe
+./test3:3> mkfifo sepipe
{ sleep 1 # To make sure that the shell sets up a reader for sepipe
unbuffer ./s 2>sepipes
rm sepipe
} | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4
+./test3:4> sleep 1
+./test3:6> rm sepipe
^C

set +xv
+./test3:9> set +xv

> Shells such as "rc" allow you to tie multiple steams together in a
> pipeline in a reasonably easy way.

Do you mean something like the Z Shell multios option?
I didn't manage to get any better results as far as this test is
concerned (hence my post here).

> You might look at the thread
> http://groups.google.com/group/comp.unix.shell/browse_thread/thread/
> dbe5abeac7635d81/5ed189b9dfb75699

Reading it now,
thanks again!

Regards
Dimitre

From: Radoulov, Dimitre on 30 Dec 2009 06:08

On 30/12/2009 11.57, Radoulov, Dimitre wrote:
> On 30/12/2009 0.27, Icarus Sparry wrote:
>
> Hi Icarus Sparry,
> thank you very much for replying!
>
>> On Tue, 29 Dec 2009 18:11:01 +0100, Radoulov, Dimitre wrote:

[...]

>> One way to tie them together is with a named pipe.
>>
>> # Create a pipe for the stderr of the program.
>> mkfifo sepipe
>> { sleep 1 # To make sure that the shell sets up a reader for sepipe
>> unbuffer program 2>sepipes
>> rm sepipe
>> } | multitee 3<sepipe>stdout 2>stderr 4>both 0:1,4 3:2,4
>
> This one appears to hung, am I missing something?

[...]

> zsh-4.3.10[t]% cat test3
> set -xv
>
> mkfifo sepipe
> { sleep 1 # To make sure that the shell sets up a reader for sepipe
> unbuffer ./s 2>sepipes
> rm sepipe
> } | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4
>
> set +xv
>
>
> zsh-4.3.10[t]% . ./test3
>
> mkfifo sepipe
> +./test3:3> mkfifo sepipe
> { sleep 1 # To make sure that the shell sets up a reader for sepipe
> unbuffer ./s 2>sepipes
> rm sepipe
> } | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4
> +./test3:4> sleep 1
> +./test3:6> rm sepipe
> ^C
>
> set +xv
> +./test3:9> set +xv
[...]

Just to add that the version of multitee used in the above script is 3.0-4.

Regards
Dimitre

From: Radoulov, Dimitre on 30 Dec 2009 09:27

On 30/12/2009 11.57, Radoulov, Dimitre wrote:
> On 30/12/2009 0.27, Icarus Sparry wrote:

[...]

>> On Tue, 29 Dec 2009 18:11:01 +0100, Radoulov, Dimitre wrote:

[...]

>> One way to tie them together is with a named pipe.
>>
>> # Create a pipe for the stderr of the program.
>> mkfifo sepipe
>> { sleep 1 # To make sure that the shell sets up a reader for sepipe
>> unbuffer program 2>sepipes
>> rm sepipe
>> } | multitee 3<sepipe>stdout 2>stderr 4>both 0:1,4 3:2,4
>
> This one appears to hang, am I missing something?

[...]

OK,
I was missing something, the name of the pipe was misspelled (sepipe <>
sepipes).

Anyway, as I already said, this method doesn't appear to be working with
the unbuffer expect script I'm using because of the before mentioned
stderr/stdout streams handling.

Regards
Dimitre

From: John Koy on 30 Dec 2009 12:16

Radoulov, Dimitre wrote:
[...]
>>>
>>> # Create a pipe for the stderr of the program.
>>> mkfifo sepipe
>>> { sleep 1 # To make sure that the shell sets up a reader for sepipe
>>> unbuffer program 2>sepipes
>>> rm sepipe
>>> } | multitee 3<sepipe>stdout 2>stderr 4>both 0:1,4 3:2,4
>>
>> This one appears to hang, am I missing something?
>
> [...]
>
> OK,
> I was missing something, the name of the pipe was misspelled (sepipe <>
> sepipes).
>
> Anyway, as I already said, this method doesn't appear to be working with
> the unbuffer expect script I'm using because of the before mentioned
> stderr/stdout streams handling.
>

All these tricks are useless. One cannot achieve synchronousness across
process boundaries without explicit IPC synchronization, like using
semaphores.
The example above creates 2 pipes between 2 processes, and *assumes*
that when producer writes into 1st pipe at time T and to 2nd pipe at
T+1, the consumer will read from the 1st at, say, T+2 and the 2nd at
T+3. This is wrong. There's no such guaranteed synchronous delivery.
Reading from the 1st could very well be at T+3, and the other at T+2.

From: Icarus Sparry on 30 Dec 2009 13:37

On Wed, 30 Dec 2009 09:56:58 +0000, Han Pingtian wrote:

> On 2009-12-29, Icarus Sparry <usenet(a)icarus.freeuk.com> wrote:
>> On Tue, 29 Dec 2009 18:11:01 +0100, Radoulov, Dimitre wrote:
>>
>>> Hi all,
>>> There was a question on a web forum on how to redirect stdin and
>>> stderr in the following manner:
>>>
>>> - two separate files for both streams - a third file containing both
>>> streams in their original order
>>>
>>> We haven't come up with a solution and I'm curious to hear your
>>> opinion.
>>
>> For typical programs, this is very hard. The reason is that almost all
>> programs use the "stdio" (standard I/O) library.
>>
>> stdio has a default buffering strategy for stderr, that says it is
>> unbuffered. So sending anything to stderr makes it appear pretty much
>> straight away.
>>
>> The strategy for stdout depends on if the library thinks it is wtiting
>> to a terminal or not. If it is not then the output only gets sent when
>> an internal buffer is full or the program is terminating. If it is
>> writing to a terminal then output gets sent when a line is completed.
>>
>> So this means that a program
>>
>> int main(int ac, char *av[]){ printf("Hello\n");
>> fprintf(stderr,"World\n"); return 0;}
>>
>> will output the two words in a different order depending on if it is
>> being run with the output going to a pipe (or file) or if it is going
>> to a console.
>>
>> If we assume that you want the "console" output order, then you need to
>> convince the stdio library that your program is talking to a console.
>> There are a couple of ways you can do this, I like an old program
>> "pty", but for most people "expect" is easier to find. It comes with an
>> example script "unbuffer" that does exactly what we want.
>>
>> At this point you then just need to have a program that listens on
>> multiple inputs and writes to multiple outputs, and a way to tie them
>> together. Such a program is "multitee" (from the multitee package in
>> Debian).
>>
>> One way to tie them together is with a named pipe.
>>
>> # Create a pipe for the stderr of the program. mkfifo sepipe
>> { sleep 1 # To make sure that the shell sets up a reader for sepipe
>> unbuffer program 2>sepipes
>> rm sepipe
>> } | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4
>>
>> Shells such as "rc" allow you to tie multiple steams together in a
>> pipeline in a reasonably easy way.
>>
>> You might look at the thread
>> http://groups.google.com/group/comp.unix.shell/browse_thread/thread/
>> dbe5abeac7635d81/5ed189b9dfb75699
>>
>>
> It seems that the unbuffer would combine the stdout and stderr:
>
> sh-4.0$ cat t.
> t.sh t.txt
> sh-4.0$ cat t.sh
> #!/bin/bash
>
>
> for i in 1 2; do
> for j in 1 2; do
> printf '%s\n' "out $i"
> done
> for k in 1 2; do
> printf '%s\n' "err $i" >&2
> done
> done
> sh-4.0$ ./t.sh 2>&1 >/dev/null
> err 1
> err 1
> err 2
> err 2
> sh-4.0$ unbuffer !!
> unbuffer ./t.sh 2>&1 >/dev/null
> sh-4.0$

Interesting. I will have to look at unbuffer (and expect in general) a
bit more.

Thanks.