Dangerous behavior of list(generator)
in [Python]
|
Prev: Spawning an interactive interpreter in a running python process?
Next: unable to read the __main__ namespace
From: exarkun on 14 Dec 2009 10:21 On 02:58 pm, mal(a)egenix.com wrote: >exarkun(a)twistedmatrix.com wrote: >>On 08:45 am, tjreedy(a)udel.edu wrote: >>>Tom Machinski wrote: >>>>In most cases, `list(generator)` works as expected. Thus, >>>>`list(<generator expression>)` is generally equivalent to >>>>`[<generator >>>>expression>]`. >>>> >>>>Here's a minimal case where this equivalence breaks, causing a >>>>serious >>>>and hard-to-detect bug in a program: >>>> >>>> >>> def sit(): raise StopIteration() >>> >>>StopIteration is intended to be used only within the .__next__ method >>>of iterators. The devs know that other 'off-label' use results in the >>>inconsistency you noted, but their and my view is 'don't do that'. >> >>Which is unfortunate, because it's not that hard to get StopIteration >>without explicitly raising it yourself and this behavior makes it >>difficult to debug such situations. >> >>What's with this view, exactly? Is it just that it's hard to >>implement >>the more desirable behavior? > >I'm not exactly sure what you're asking for. > >The StopIteration exception originated as part of the for-loop >protocol. Later on it was generalized to apply to generators >as well. > >The reason for using an exception is simple: raising and catching >exceptions is fast at C level and since the machinery for >communicating exceptions up the call stack was already there >(and doesn't interfere with the regular return values), this >was a convenient method to let the upper call levels know >that an iteration has ended (e.g. a for-loop 4 levels up the >stack). > >I'm not sure whether that answers your question, but it's the >reason for things being as they are :-) I'm asking about why the behavior of a StopIteration exception being handled from the `expression` of a generator expression to mean "stop the loop" is accepted by "the devs" as acceptable. To continue your comparison to for loops, it's as if a loop like this: for a in b: c actually meant this: for a in b: try: c except StopIteration: break Note, I know *why* the implementation leads to this behavior. I'm asking why "the devs" *accept* this. Jean-Paul
From: Mel on 14 Dec 2009 11:09 exarkun(a)twistedmatrix.com wrote: [ ... ] it's as if a loop like this: > > for a in b: > c > > actually meant this: > > for a in b: > try: > c > except StopIteration: > break > > Note, I know *why* the implementation leads to this behavior. I'm > asking why "the devs" *accept* this. It's part of the price Python pays for letting people get their hands on the controls. Consider also: Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) [GCC 4.3.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class dict2(dict): .... def __getitem__ (self, key): .... if key == 'fatal': .... raise KeyError .... >>> d = dict2() >>> d['fatal'] = 'Hello, world!' >>> print d['fatal'] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in __getitem__ KeyError >>> "KeyError when we just put the item into the dict?" "Yep." Mel. > > Jean-Paul
From: Terry Reedy on 14 Dec 2009 13:00 On 12/14/2009 10:21 AM, exarkun(a)twistedmatrix.com wrote: > I'm asking about why the behavior of a StopIteration exception being > handled from the `expression` of a generator expression to mean "stop > the loop" is accepted by "the devs" as acceptable. Any unhandled exception within a loop stops the loop, and the exception is passed to the surrounding code. > To continue your > comparison to for loops, it's as if a loop like this: > > for a in b: > c > > actually meant this: > > for a in b: > try: > c > except StopIteration: > break No it does not. If c raises any exception, the loop stops *and* the exception is passed up to the surrounding code. > Note, I know *why* the implementation leads to this behavior. You do not seem to know what the behavior is. Read what I wrote last night. Terry Jan Reedy
From: exarkun on 14 Dec 2009 14:05 On 06:00 pm, tjreedy(a)udel.edu wrote: >On 12/14/2009 10:21 AM, exarkun(a)twistedmatrix.com wrote: >>I'm asking about why the behavior of a StopIteration exception being >>handled from the `expression` of a generator expression to mean "stop >>the loop" is accepted by "the devs" as acceptable. > >Any unhandled exception within a loop stops the loop, >and the exception is passed to the surrounding code. >>To continue your >>comparison to for loops, it's as if a loop like this: >> >>for a in b: >>c >> >>actually meant this: >> >>for a in b: >>try: >>c >>except StopIteration: >>break > >No it does not. No what does not? I said "It is as if". This is a hypothetical. I'm not claiming this is the actual behavior of anything. >>Note, I know *why* the implementation leads to this behavior. > >You do not seem to know what the behavior is. >Read what I wrote last night. Well, I'm a bit tired of this thread. Please disregard my question above. I'm done here. Sorry for the confusion. Have a nice day. Jean-Paul
From: Antoine Pitrou on 14 Dec 2009 16:30
Le Mon, 14 Dec 2009 15:21:09 +0000, exarkun a écrit : > > I'm asking about why the behavior of a StopIteration exception being > handled from the `expression` of a generator expression to mean "stop > the loop" is accepted by "the devs" as acceptable. It's not "accepted as acceptable", it's just a side effect of how various means of iterating (including for loops and generators) are implemented in CPython. Seeing how it doesn't seem to prevent or promote any useful programming idiom, there was no incentive to either 1) codify it as official spec or 2) change it. In other words, it should be considered undefined behaviour, and perhaps other Python implementations behave differently. Regards Antoine. |