Prev: Hello
Next: the ball needs a kick...
From: Les Schaffer on 9 Jul 2010 15:13
i have been asked to guarantee that a proposed Python application will
run continuously under MS Windows for two months time. And i am looking
to know what i don't know.

The app would read instrument data from a serial port, store the data in
file, and display in matplotlib. typical sampling times are about once
per hour. our design would be to read the serial port once a second
(long story, but it worked for ten days straight so far) and just store
and display the values once an hour. so basically we'd have a long
running for-loop. maybe running in a separate thread.

i have thought through the issues on our end: we have to properly
handle the serial port and make sure our GUI doesn't hang easily. we'd
have to be sure we don't have any memory leaks in numpy or matplotlib
and we're not storing data in memory stupidly. i've never looked into
Windows serial port driver reliability over the long term. But we think
if the serial port hangs, we can detect that and react accordingly.

but none of this has anything to do with Python itself. i am sure python
servers have been running reliably for long periods of time, but i've
never had to deal with a two-month guarantee before. is there something
else i am missing here that i should be concerned about on the
pure-Python side of things? something under the hood of the python
interpreter that could be problematic when run for a long time?

or need we only concern ourselves with the nuts behind the wheel:that
is, we the developers?

thanks

Les
From: Michael Torrie on 9 Jul 2010 15:56
On 07/09/2010 01:13 PM, Les Schaffer wrote:
> or need we only concern ourselves with the nuts behind the wheel:that
> is, we the developers?

It never hurts to separate the data collection and
visualization/analysis parts into separate programs. That way you can
keep the critical, long-running data collection program running, and if
you get an exception in the GUI because of some divide by zero
programming error or some other problem, you can restart that part
without impacting the overall system.

From: Christian Heimes on 9 Jul 2010 15:58
> but none of this has anything to do with Python itself. i am sure python
> servers have been running reliably for long periods of time, but i've
> never had to deal with a two-month guarantee before. is there something
> else i am missing here that i should be concerned about on the
> pure-Python side of things? something under the hood of the python
> interpreter that could be problematic when run for a long time?

At our company we have long running Python processes with an uptime of
60, 90 days or more. Python is up to the challenge. ;)

But it's a challenge to keep any process written in any programming
language running for two months or more. If I were in your case I would
split up the work across multiple python process. One guardian process
that keeps the other processes alive and checks if they are still
reacting on events, another process that reads data from the serial port
and writes it to a database or file and a third process to process the
data. Small processes reduce the chance of an error. I assume that the
"read from serial port" part is the mission critical element of your app.

Christian

From: William Heymann on 9 Jul 2010 16:03
On Friday 09 July 2010, Les Schaffer wrote:
> but none of this has anything to do with Python itself. i am sure python
> servers have been running reliably for long periods of time, but i've
> never had to deal with a two-month guarantee before. is there something
> else i am missing here that i should be concerned about on the
> pure-Python side of things? something under the hood of the python
> interpreter that could be problematic when run for a long time?
>
> or need we only concern ourselves with the nuts behind the wheel:that
> is, we the developers?
>
> thanks
>
> Les

I have been running zope apps for about 10 years now and they normally run for
many months between being restarted so python has no inherent problems with
running that long. Your specific python program might though.

You have to make sure you don't have any reference leaks so your program keeps
growing in memory but you also have to deal with windows. The program can not
be any more reliable then the os it is running on. Personally I would never
make a guarantee like that on a windows box. I would have to hand choose every
piece of hardware and run some kind of unix on it with a guarantee like that.

Last I ran a program for a long time on windows it ran into some kind of
address space fragmentation and eventually it would die on windows. There is
some kind of problem with the windows VM system. 64bit windows will solve that
mostly by having an address space so large you can't fragment it enough to
kill the program in any reasonable time frame. If your program is not
allocating and destroying large data structures all the time you probably
don't have to worry about that but you do have to test it.
From: Terry Reedy on 9 Jul 2010 16:16
On 7/9/2010 3:13 PM, Les Schaffer wrote:
> i have been asked to guarantee that a proposed Python application will
> run continuously under MS Windows for two months time. And i am looking
> to know what i don't know.
>
> The app would read instrument data from a serial port, store the data in
> file, and display in matplotlib. typical sampling times are about once
> per hour. our design would be to read the serial port once a second
> (long story, but it worked for ten days straight so far) and just store
> and display the values once an hour. so basically we'd have a long
> running for-loop. maybe running in a separate thread.

Is this a dedicated machine, so you do not have anything else going that
can delay for more than a second?
>
> i have thought through the issues on our end: we have to properly handle
> the serial port and make sure our GUI doesn't hang easily. we'd have to
> be sure we don't have any memory leaks in numpy or matplotlib and we're
> not storing data in memory stupidly. i've never looked into Windows
> serial port driver reliability over the long term. But we think if the
> serial port hangs, we can detect that and react accordingly.

I read the ibmpc serial port bios code in the early 80s. It was pretty
simple then. I would be very surprised if it has been messed up since
and not fixed.

> but none of this has anything to do with Python itself. i am sure python
> servers have been running reliably for long periods of time, but i've
> never had to deal with a two-month guarantee before. is there something
> else i am missing here that i should be concerned about on the
> pure-Python side of things? something under the hood of the python
> interpreter that could be problematic when run for a long time?
>
> or need we only concern ourselves with the nuts behind the wheel:that
> is, we the developers?

Python has been used for long-running background processes, at least on
*nix, so the Python devs are sensitive to memory leak issues and respond
to leak reports. That still leaves memory fragmentation. To try to avoid
that, I would allocate all the needed data arrays immediately on startup
(with dummy None pointers) and reuse them instead of deleting and
regrowing them. Keeping explicit pointers is, of course more tedious and
slightly error prone.

I hope someone with actual experience also answers.
--
Terry Jan Reedy

 |  Next  |  Last
Pages: 1 2 3 4 5 6
Prev: Hello
Next: the ball needs a kick...