Redirect stdout and stderr preserving the original order [Shell]

Prev: list out thousand of files
Next: unzip | touch | re-zip

From: Radoulov, Dimitre on 29 Dec 2009 15:55

On 29/12/2009 21.12, mop2 wrote:
> On Tue, 29 Dec 2009 15:11:01 -0200, Radoulov, Dimitre
> <cichomitiko(a)gmail.com> wrote:
>
>> Hi all,
>> There was a question on a web forum on how to redirect stdin and stderr
>> in the following manner:
>>
>> - two separate files for both streams
>> - a third file containing both streams in their original order
>>
>> We haven't come up with a solution and I'm curious to hear your opinion.
>>
>> For instance, let's say we have the following shell script:
>>
>>
>> % cat s
>> #!/bin/sh
>>
>> for i in 1 2; do
>> for j in 1 2; do
>> printf '%s\n' "out $i"
>> done
>> for k in 1 2; do
>> printf '%s\n' "err $i" >&2
>> done
>> done
>>
>>
>> The script produces the following output on the terminal (which of
>> course includes both stdout and stderr streams):
>>
>> % ./s
>> out 1
>> out 1
>> err 1
>> err 1
>> out 2
>> out 2
>> err 2
>> err 2
>>
>> Note that we use out/err n only for clarity and easy debugging. We
>> assume that in the real world both the content and the order of the
>> stdout and stderr streams are unknown.
>>
>> Now the question is, is it possible to end up with the following files
>> after running the script:
>>
>> out.log
>>
>> out 1
>> out 1
>> out 2
>> out 2
>>
>> err.log
>>
>> err 1
>> err 1
>> err 2
>> err 2
>>
>> both.log
>>
>> out 1
>> out 1
>> err 1
>> err 1
>> out 2
>> out 2
>> err 2
>> err 2
>>
>>
>> Note the order of the entries in both.log.
>>
>> We also assume that it's not possible to modify the script/program that
>> generates the output in any way.

[...]

> Just an idea:
>
>
> $ cat s
> #!/bin/sh
>
> for i in 1 2; do
> for j in 1 2; do
> printf '%s\n' "out $i"
> done
> for k in 1 2; do
> printf '%s\n' "err $i" >&2
> done
> done
>
> $ cat test
>
> printf '' >b
> printf '' >o
> printf '' >e
>
> tail -n0 -f o >>b & To=$!
> tail -n0 -f e >>b & Te=$!
> ./s >o 2>e
> read -t2; kill $To $Te
>
> $ . ./test
> [1]- Terminated tail -n0 -f o >> b
> [2]+ Terminated tail -n0 -f e >> b
>
> $ cat o
> out 1
> out 1
> out 2
> out 2
>
> $ cat e
> err 1
> err 1
> err 2
> err 2
>
> $ cat b
> out 1
> out 1
> err 1
> err 1
> out 2
> out 2
> err 2
> err 2
[...]

Thanks, but I don't get the correct order in b:

% head -10 s test
==> s <==
#!/bin/sh

for i in 1 2; do
for j in 1 2; do
printf '%s\n' "out $i"
done
for k in 1 2; do
printf '%s\n' "err $i" >&2
done
done

==> test <==

printf '' >b
printf '' >o
printf '' >e

tail -n0 -f o >>b & To=$!
tail -n0 -f e >>b & Te=$!
../s >o 2>e
read -t2; kill $To $Te

Here is what I get:

zsh-4.3.10[t]% . ./test
[2] 1851
[3] 1852
zsh-4.3.10[t]%
[3] + terminated tail -n0 -f e >> b
zsh-4.3.10[t]%
[2] + terminated tail -n0 -f o >> b
zsh-4.3.10[t]% head -8 e o b
==> e <==
err 1
err 1
err 2
err 2

==> o <==
out 1
out 1
out 2
out 2

==> b <==
out 1
out 1
out 2
out 2
err 1
err 1
err 2
err 2

zsh-4.3.10[t]% ksh
ksh-JM 93t+ 2009-05-01$ . ./test
[1] 1857
[2] 1858
[2] + Terminated . ./test
[1] - Terminated . ./test
ksh-JM 93t+ 2009-05-01$ head -8 e o b
==> e <==
err 1
err 1
err 2
err 2

==> o <==
out 1
out 1
out 2
out 2

==> b <==
err 1
err 1
err 2
err 2
ksh-JM 93t+ 2009-05-01$

ksh-JM 93t+ 2009-05-01$ bash
bash-4.0.33(1)[t]$ . ./test
bash-4.0.33(1)[t]$ head -8 e o b
==> e <==
err 1
err 1
err 2
err 2

==> o <==
out 1
out 1
out 2
out 2

==> b <==
out 1
out 1
out 2
out 2
err 1
err 1
err 2
err 2
[1]- Terminated tail -n0 -f o >> b
[2]+ Terminated tail -n0 -f e >> b
bash-4.0.33(1)[t]$

What shell and OS are you using?

Regards
Dimitre

From: Icarus Sparry on 29 Dec 2009 18:27

On Tue, 29 Dec 2009 18:11:01 +0100, Radoulov, Dimitre wrote:

> Hi all,
> There was a question on a web forum on how to redirect stdin and stderr
> in the following manner:
>
> - two separate files for both streams - a third file containing both
> streams in their original order
>
> We haven't come up with a solution and I'm curious to hear your opinion.

For typical programs, this is very hard. The reason is that almost all
programs use the "stdio" (standard I/O) library.

stdio has a default buffering strategy for stderr, that says it is
unbuffered. So sending anything to stderr makes it appear pretty much
straight away.

The strategy for stdout depends on if the library thinks it is wtiting to
a terminal or not. If it is not then the output only gets sent when an
internal buffer is full or the program is terminating. If it is writing
to a terminal then output gets sent when a line is completed.

So this means that a program

int main(int ac, char *av[]){ printf("Hello\n");
fprintf(stderr,"World\n"); return 0;}

will output the two words in a different order depending on if it is
being run with the output going to a pipe (or file) or if it is going to
a console.

If we assume that you want the "console" output order, then you need to
convince the stdio library that your program is talking to a console.
There are a couple of ways you can do this, I like an old program "pty",
but for most people "expect" is easier to find. It comes with an example
script "unbuffer" that does exactly what we want.

At this point you then just need to have a program that listens on
multiple inputs and writes to multiple outputs, and a way to tie them
together. Such a program is "multitee" (from the multitee package in
Debian).

One way to tie them together is with a named pipe.

# Create a pipe for the stderr of the program.
mkfifo sepipe
{ sleep 1 # To make sure that the shell sets up a reader for sepipe
unbuffer program 2>sepipes
rm sepipe
} | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4

Shells such as "rc" allow you to tie multiple steams together in a
pipeline in a reasonably easy way.

You might look at the thread
http://groups.google.com/group/comp.unix.shell/browse_thread/thread/
dbe5abeac7635d81/5ed189b9dfb75699

From: mop2 on 29 Dec 2009 20:44

On Tue, 29 Dec 2009 18:55:28 -0200, Radoulov, Dimitre <cichomitiko(a)gmail.com> wrote:
~
> ksh-JM 93t+ 2009-05-01$ bash
> bash-4.0.33(1)[t]$ . ./test
> bash-4.0.33(1)[t]$ head -8 e o b
> ==> e <==
> err 1
> err 1
> err 2
> err 2
>
> ==> o <==
> out 1
> out 1
> out 2
> out 2
>
> ==> b <==
> out 1
> out 1
> out 2
> out 2
> err 1
> err 1
> err 2
> err 2
> [1]- Terminated tail -n0 -f o >> b
> [2]+ Terminated tail -n0 -f e >> b
> bash-4.0.33(1)[t]$
>
> What shell and OS are you using?

My system is a slackware:

$ uname -a
Linux k7 2.6.31.6 #1 PREEMPT Sat Nov 21 05:39:49 hrV 2009 i686 GNU/Linux

$ tail --version|head -n1
tail (GNU coreutils) 8.2

$ $0 --version|head -n1
GNU bash, version 4.0.35(1)-release (i686-pc-linux-gnu)

$ ls -l /lib/libc.so*
lrwxrwxrwx 1 root root 11 2009-11-20 19:47 /lib/libc.so.6 -> libc-2.9.so*

The more important here, I think, is the program tail.
If the "-s" is available you can try it to see if behavior changes for small values:
-s, --sleep-interval=N
with -f, sleep for approximately N seconds (default 1.0) between iterations

I never tried this option, but in the past I had problems with buffering, as said by others.
I really like use exclusively shell when this is not inconvenient , and this option always
solves to me buffering and delay problems.

So, a new test, using 2 shell loops in background an your script "s":

$ cat test1

printf '' >b
printf '' >o
mkfifo fo
printf '' >e
mkfifo fe

while read;do printf "$REPLY\n";printf "$REPLY\n">>b;done >o <fo &
while read;do printf "$REPLY\n";printf "$REPLY\n">>b;done >e <fe &

../s >fo 2>fe

rm fo fe

head o e b

$ . ./test1
[1]- Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >> b;
done > o < fo
[2]+ Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >> b;
done > e < fe
==> o <==
out 1
out 1
out 2
out 2

==> e <==
err 1
err 1
err 2
err 2

==> b <==
out 1
out 1
err 1
err 1
out 2
out 2
err 2
err 2

From: Han Pingtian on 30 Dec 2009 04:56

On 2009-12-29, Icarus Sparry <usenet(a)icarus.freeuk.com> wrote:
> On Tue, 29 Dec 2009 18:11:01 +0100, Radoulov, Dimitre wrote:
>
>> Hi all,
>> There was a question on a web forum on how to redirect stdin and stderr
>> in the following manner:
>>
>> - two separate files for both streams - a third file containing both
>> streams in their original order
>>
>> We haven't come up with a solution and I'm curious to hear your opinion.
>
> For typical programs, this is very hard. The reason is that almost all
> programs use the "stdio" (standard I/O) library.
>
> stdio has a default buffering strategy for stderr, that says it is
> unbuffered. So sending anything to stderr makes it appear pretty much
> straight away.
>
> The strategy for stdout depends on if the library thinks it is wtiting to
> a terminal or not. If it is not then the output only gets sent when an
> internal buffer is full or the program is terminating. If it is writing
> to a terminal then output gets sent when a line is completed.
>
> So this means that a program
>
> int main(int ac, char *av[]){ printf("Hello\n");
> fprintf(stderr,"World\n"); return 0;}
>
> will output the two words in a different order depending on if it is
> being run with the output going to a pipe (or file) or if it is going to
> a console.
>
> If we assume that you want the "console" output order, then you need to
> convince the stdio library that your program is talking to a console.
> There are a couple of ways you can do this, I like an old program "pty",
> but for most people "expect" is easier to find. It comes with an example
> script "unbuffer" that does exactly what we want.
>
> At this point you then just need to have a program that listens on
> multiple inputs and writes to multiple outputs, and a way to tie them
> together. Such a program is "multitee" (from the multitee package in
> Debian).
>
> One way to tie them together is with a named pipe.
>
> # Create a pipe for the stderr of the program.
> mkfifo sepipe
> { sleep 1 # To make sure that the shell sets up a reader for sepipe
> unbuffer program 2>sepipes
> rm sepipe
> } | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4
>
> Shells such as "rc" allow you to tie multiple steams together in a
> pipeline in a reasonably easy way.
>
> You might look at the thread
> http://groups.google.com/group/comp.unix.shell/browse_thread/thread/
> dbe5abeac7635d81/5ed189b9dfb75699
>
>
It seems that the unbuffer would combine the stdout and stderr:

sh-4.0$ cat t.
t.sh t.txt
sh-4.0$ cat t.sh
#!/bin/bash

for i in 1 2; do
for j in 1 2; do
printf '%s\n' "out $i"
done
for k in 1 2; do
printf '%s\n' "err $i" >&2
done
done
sh-4.0$ ./t.sh 2>&1 >/dev/null
err 1
err 1
err 2
err 2
sh-4.0$ unbuffer !!
unbuffer ./t.sh 2>&1 >/dev/null
sh-4.0$

--
Han Pingtian

From: Radoulov, Dimitre on 30 Dec 2009 05:38

On 30/12/2009 2.44, mop2 wrote:
> On Tue, 29 Dec 2009 18:55:28 -0200, Radoulov, Dimitre
> <cichomitiko(a)gmail.com> wrote:
> ~
>> ksh-JM 93t+ 2009-05-01$ bash
>> bash-4.0.33(1)[t]$ . ./test
>> bash-4.0.33(1)[t]$ head -8 e o b
>> ==> e <==
>> err 1
>> err 1
>> err 2
>> err 2
>>
>> ==> o <==
>> out 1
>> out 1
>> out 2
>> out 2
>>
>> ==> b <==
>> out 1
>> out 1
>> out 2
>> out 2
>> err 1
>> err 1
>> err 2
>> err 2
>> [1]- Terminated tail -n0 -f o >> b
>> [2]+ Terminated tail -n0 -f e >> b
>> bash-4.0.33(1)[t]$
>>
>> What shell and OS are you using?
>
> My system is a slackware:
>
> $ uname -a
> Linux k7 2.6.31.6 #1 PREEMPT Sat Nov 21 05:39:49 hrV 2009 i686 GNU/Linux

I tested the script on Ubuntu 9.10 on Sun VirtualBox (Windows XP guest)
and on a real (physical) RHEL 5.1.

> $ tail --version|head -n1
> tail (GNU coreutils) 8.2
>
> $ $0 --version|head -n1
> GNU bash, version 4.0.35(1)-release (i686-pc-linux-gnu)
>
> $ ls -l /lib/libc.so*
> lrwxrwxrwx 1 root root 11 2009-11-20 19:47 /lib/libc.so.6 -> libc-2.9.so*

Just for completeness:

Ubuntu VM:

$ tail --version|head -n1
tail (GNU coreutils) 7.4

$ $0 --version|head -n1
GNU bash, version 4.0.33(1)-release (i486-pc-linux-gnu)

$ ls -l /lib/libc.so*
lrwxrwxrwx 1 root root 14 2009-12-26 11:20 /lib/libc.so.6 -> libc-2.10.1.so

RHEL:

$ tail --version|head -n1
tail (GNU coreutils) 5.97

$ $0 --version|head -n1
GNU bash, version 3.1.17(1)-release (i686-redhat-linux-gnu)

$ ls -l /lib/libc.so*
lrwxrwxrwx 1 root root 11 Jan 7 2008 /lib/libc.so.6 -> libc-2.5.so

> The more important here, I think, is the program tail.
> If the "-s" is available you can try it to see if behavior changes for
> small values:
> -s, --sleep-interval=N
> with -f, sleep for approximately N seconds (default 1.0) between iterations
>
> I never tried this option, but in the past I had problems with
> buffering, as said by others.
> I really like use exclusively shell when this is not inconvenient , and
> this option always
> solves to me buffering and delay problems.

I tried with values from 0.1 to 0.5, but I get the same result.
Later I will try the test again on a different system.

> So, a new test, using 2 shell loops in background an your script "s":
>
> $ cat test1
>
> printf '' >b
> printf '' >o
> mkfifo fo
> printf '' >e
> mkfifo fe
>
> while read;do printf "$REPLY\n";printf "$REPLY\n">>b;done >o <fo &
> while read;do printf "$REPLY\n";printf "$REPLY\n">>b;done >e <fe &
>
> ./s >fo 2>fe
>
> rm fo fe
>
> head o e b
>
>
> $ . ./test1
> [1]- Done while read; do
> printf "$REPLY\n"; printf "$REPLY\n" >> b;
> done > o < fo
> [2]+ Done while read; do
> printf "$REPLY\n"; printf "$REPLY\n" >> b;
> done > e < fe
> ==> o <==
> out 1
> out 1
> out 2
> out 2
>
> ==> e <==
> err 1
> err 1
> err 2
> err 2
>
> ==> b <==
> out 1
> out 1
> err 1
> err 1
> out 2
> out 2
> err 2
> err 2

The order in b is not constant between executions. This is on RHEL 5.1:

$ . ./test1
[1]- Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >>b;
done >o <fo
[2]+ Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >>b;
done >e <fe
==> o <==
out 1
out 1
out 2
out 2

==> e <==
err 1
err 1
err 2
err 2

==> b <==
out 1
out 1
out 2
out 2
err 1
err 1
err 2
err 2
$ . ./test1
[1]- Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >>b;
done >o <fo
[2]+ Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >>b;
done >e <fe
==> o <==
out 1
out 1
out 2
out 2

==> e <==
err 1
err 1
err 2
err 2

==> b <==
out 1
err 1
out 1
err 1
err 2
err 2
out 2
out 2

Thanks again for the ideas anyway.

Regards
Dimitre