Prev: Access lotus notes using Python 2.5.1
Next: regex to remove lines made of only whitespace
From: RG on 11 Aug 2010 03:11
When stdin is not a tty, Python seems to buffer all the input through
EOF before processing any of it:

[ron(a)mickey:~]$ cat | python
print 123
print 456 <hit ctrl-D here>
123
456

Is there a way to get Python to process input line-by-line the way it
does when stdin is a TTY even when stdin is not a TTY?

Thanks,
rg
From: Tim Harig on 11 Aug 2010 04:38
On 2010-08-11, RG <rNOSPAMon(a)flownet.com> wrote:
> When stdin is not a tty, Python seems to buffer all the input through
> EOF before processing any of it:
>
> [ron(a)mickey:~]$ cat | python
> print 123
> print 456 <hit ctrl-D here>
> 123
> 456
>
> Is there a way to get Python to process input line-by-line the way it
> does when stdin is a TTY even when stdin is not a TTY?

It would be much better to know the overall purpose of what you are trying
to achieve. There are may be better ways (ie, sockets) depending what you
are trying to do. Knowing your target platform would also be helpful.

For the python interpeter itself, you can can get interactive behavior by
invoking it with the -i option.

If you want to handle stdin a single line at a time from inside of your
program, you can access it using sys.stdin.readline().
From: Cameron Simpson on 11 Aug 2010 05:42
On 11Aug2010 00:11, RG <rNOSPAMon(a)flownet.com> wrote:
| When stdin is not a tty, Python seems to buffer all the input through
| EOF before processing any of it:
|
| [ron(a)mickey:~]$ cat | python
| print 123
| print 456 <hit ctrl-D here>
| 123
| 456
|
| Is there a way to get Python to process input line-by-line the way it
| does when stdin is a TTY even when stdin is not a TTY?

What you're seeing here is not python's behaviour but cat's behaviour.

Almost all programs do line buffering (flush buffer at newline) when the
file is a terminal (character device) and block buffering (flush when a
fixed size buffer, typically 8192 bytes or some larger power of 2) when
the file is not a terminal. This is default behaviour for the stdio
package.

So "cat" is simply not feeding any data to python until it has a lot of
it; there is nothing python can do about that. We would need to know
more about your specific task to suggest workarounds.

Usually you either
need an option on the upstream program to tell it to line buffer
explicitly or you need to play silly games with pseudo terminals to
convince the upstream program it is attached to a terminal. The latter
is both ugly and generally inadvisable because many programs that change
their buffering when attached to a terminal also change other behaviour,
such as issuing interactiove prompts etc.

Cheers,
--
Cameron Simpson <cs(a)zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

The type syntax for C is essentially unparsable. - Rob Pike
From: Wolfgang Rohdewald on 11 Aug 2010 06:01
On Mittwoch 11 August 2010, Cameron Simpson wrote:
> Usually you either
> need an option on the upstream program to tell it to line
> buffer explicitly

once cat had an option -u doing exactly that but nowadays
-u seems to be ignored

http://www.opengroup.org/onlinepubs/009695399/utilities/cat.html

--
Wolfgang
From: Tim Harig on 11 Aug 2010 06:32
On 2010-08-11, Wolfgang Rohdewald <wolfgang(a)rohdewald.de> wrote:
> On Mittwoch 11 August 2010, Cameron Simpson wrote:
>> Usually you either
>> need an option on the upstream program to tell it to line
>> buffer explicitly
>
> once cat had an option -u doing exactly that but nowadays
> -u seems to be ignored
>
> http://www.opengroup.org/onlinepubs/009695399/utilities/cat.html

I have to wonder why cat knows or cares. Since we are referring to
a single directional pipe, there is no fear of creating any kind of
race condition. In general, I would expect that the shell opens the
pipe (pipe()), fork()s, closes its own 0 or 1 descriptor as appropriate
for each child, copies (dup()) one the file descriptors to the
appropriate file descriptor for the child process, and exec()s to call
the new process. Neither of the processes, in general, needs to know
anything other the to write and read from their given descriptors.
 |  Next  |  Last
Pages: 1 2 3 4 5
Prev: Access lotus notes using Python 2.5.1
Next: regex to remove lines made of only whitespace