Dangerous behavior of list(generator)
in [Python]
|
Prev: Spawning an interactive interpreter in a running python process?
Next: unable to read the __main__ namespace
From: Carl Banks on 14 Dec 2009 17:05 On Dec 14, 7:21 am, exar...(a)twistedmatrix.com wrote: > Note, I know *why* the implementation leads to this behavior. I'm > asking why "the devs" *accept* this. As noted, the problem isn't with generators but with iteration protocol. The devs "allowed" this because it was a necessary evil for correct functionality. As the system is set up it can't discriminate between a legitimate and a spurrious StopIteration. (Which is why Steven and others called you out for suggesting that the compiler has to read your mind.) However, as far as I'm concerned there is no reasonable argument that this behavior is good. So how, hypothetically, would one go about fixing it, short of ripping out and replacing the existing machinery? The first argument is that StopIteration has no place within a generator expression. Therefore a generator expression (but not a generator function) could deliberately catch StopIteration and raise a different exception. I don't like it, though: who says that StopIteration has no place within a generator expression? Currently it's possible to do something like this to terminate a genexp early, and I won't the one saying you shouldn't do it. def stop(): raise StopIteration list(x or stop() for x in stream) (Though personally I'd bite the bullet and write it as a generator function). What else? The way I see it, when you throw StopIteration you are trying to stop a specific generator. Therefore StopIteration should (somehow) contain a reference to the generator that it's being applied to. Perhaps it can be obtained by crawling ths stack, which shouldn't be a significant penalty (it'd be only called once per iteration, and most of the time it'd be only one or two frames up). The looping logic within Python should check whether the reference matches the object it's iterating over; if not it raises a LeakyLoopException or something like that. I haven't thought this out though, I'm just kind of throwing this out there. Any issues? But to answer your question, I think "simple is better than complex" rules the day. Right now StopIteration stops an iteration, simple as that. Any fix would add complexity. Carl Banks
From: Steven D'Aprano on 14 Dec 2009 17:48 On Mon, 14 Dec 2009 14:31:44 +0000, exarkun wrote: > On 06:46 am, tjreedy(a)udel.edu wrote: >>On 12/13/2009 10:29 PM, exarkun(a)twistedmatrix.com wrote: >>>Doesn't matter. Sometimes it makes sense to call it directly. >> >>It only makes sense to call next (or .__next__) when you are prepared to >>explicitly catch StopIteration within a try..except construct. You did >>not catch it, so it stopped execution. >> >>Let me repeat: StopIteration is intended only for stopping iteration. >>Outside that use, it is a normal exception with no special meaning. > > You cut out the part of my message where I wrote that one might have > forgotten the exception handling code that you posit is required, and > that the current behavior makes debugging this situation unnecessarily > challenging. I don't see why you think this is any more challenging to debug than any other equivalent bug. If anything I would think it was easier to debug: if the problem is that you get a StopIteration traceback, well that's easy and straightforward, and if the problem is that you don't (and consequently you end up with fewer items in the list than you expect), the obvious debugging technique is to build the list by hand and inspect each item before adding it to the list: L = [] for i, item in enumerate(iterable): print i, item, value = item() # raises StopIteration print value L.append(value) That will expose the StopIteration exception and reveal the problem. But even if I have missed something, and it is a challenging problem to debug, oh well. It should be a quite unusual situation to come across. -- Steven
From: Carl Banks on 14 Dec 2009 18:26 On Dec 14, 2:48 pm, Steven D'Aprano <st...(a)REMOVE-THIS- cybersource.com.au> wrote: > On Mon, 14 Dec 2009 14:31:44 +0000, exarkun wrote: > > On 06:46 am, tjre...(a)udel.edu wrote: > >>On 12/13/2009 10:29 PM, exar...(a)twistedmatrix.com wrote: > >>>Doesn't matter. Sometimes it makes sense to call it directly. > > >>It only makes sense to call next (or .__next__) when you are prepared to > >>explicitly catch StopIteration within a try..except construct. You did > >>not catch it, so it stopped execution. > > >>Let me repeat: StopIteration is intended only for stopping iteration. > >>Outside that use, it is a normal exception with no special meaning. > > > You cut out the part of my message where I wrote that one might have > > forgotten the exception handling code that you posit is required, and > > that the current behavior makes debugging this situation unnecessarily > > challenging. > > I don't see why you think this is any more challenging to debug than any > other equivalent bug. "Errors should never pass silently." I'm not saying it's necessarily difficult to debug--although building a list by hand to test it is a lot more work than reading an exception traceback--but it'a stark violation of a Zen and common sense, so it is more serious than other sorts of errors. Carl Banks
From: Steven D'Aprano on 15 Dec 2009 01:26 On Mon, 14 Dec 2009 15:26:25 -0800, Carl Banks wrote: > On Dec 14, 2:48 pm, Steven D'Aprano <st...(a)REMOVE-THIS- > cybersource.com.au> wrote: >> On Mon, 14 Dec 2009 14:31:44 +0000, exarkun wrote: >> > On 06:46 am, tjre...(a)udel.edu wrote: >> >>On 12/13/2009 10:29 PM, exar...(a)twistedmatrix.com wrote: >> >>>Doesn't matter. Sometimes it makes sense to call it directly. >> >> >>It only makes sense to call next (or .__next__) when you are prepared >> >>to explicitly catch StopIteration within a try..except construct. You >> >>did not catch it, so it stopped execution. >> >> >>Let me repeat: StopIteration is intended only for stopping iteration. >> >>Outside that use, it is a normal exception with no special meaning. >> >> > You cut out the part of my message where I wrote that one might have >> > forgotten the exception handling code that you posit is required, and >> > that the current behavior makes debugging this situation >> > unnecessarily challenging. >> >> I don't see why you think this is any more challenging to debug than >> any other equivalent bug. > > "Errors should never pass silently." StopIteration isn't an error, it's a signal. The error is *misusing* StopIteration, and the above Zen no more applies than it would if I did x-y instead of y-x and complained that I got no traceback. Some errors are programming mistakes, and they are the deadliest error because they can and do pass silently. There's nothing you can do about that except Don't Make Mistakes. > I'm not saying it's necessarily difficult to debug--although building a > list by hand to test it is a lot more work than reading an exception > traceback Of course. Nobody said the life of a programmer was all beer and skittles :) > --but it'a stark violation of a Zen and common sense, so it is > more serious than other sorts of errors. I'm happy to accept it is a Gotcha, but a bug? I'm not convinced. -- Steven
From: Michele Simionato on 16 Dec 2009 05:31
On Dec 14, 11:05 pm, Carl Banks <pavlovevide...(a)gmail.com> wrote: > But to answer your question, I think "simple is better than complex" > rules the day. Right now StopIteration stops an iteration, simple as > that. Any fix would add complexity. +1 |