Dangerous behavior of list(generator)
in [Python]
|
Prev: Spawning an interactive interpreter in a running python process?
Next: unable to read the __main__ namespace
From: Terry Reedy on 13 Dec 2009 03:45 Tom Machinski wrote: > In most cases, `list(generator)` works as expected. Thus, > `list(<generator expression>)` is generally equivalent to `[<generator > expression>]`. > > Here's a minimal case where this equivalence breaks, causing a serious > and hard-to-detect bug in a program: > > >>> def sit(): raise StopIteration() StopIteration is intended to be used only within the .__next__ method of iterators. The devs know that other 'off-label' use results in the inconsistency you noted, but their and my view is 'don't do that'.
From: Gabriel Genellina on 13 Dec 2009 03:53 En Sat, 12 Dec 2009 23:43:20 -0300, Ned Deily <nad(a)acm.org> escribi�: > In article <nad-8CDB63.18012412122009(a)news.gmane.org>, > Ned Deily <nad(a)acm.org> wrote: >> In article >> <ec96e1390912121653w56c3dbe3p859a7b979026bf47(a)mail.gmail.com>, >> Benjamin Kaplan <benjamin.kaplan(a)case.edu> wrote: >> > On Sat, Dec 12, 2009 at 7:15 PM, Tom Machinski >> <tom.machinski(a)gmail.com> >> > wrote: >> > > >>> def sit(): raise StopIteration() >> > > ... >> > > >>> [f() for f in (lambda:1, sit, lambda:2)] >> > > Traceback (most recent call last): >> > > File "<stdin>", line 1, in <module> >> > > File "<stdin>", line 1, in sit >> > > StopIteration >> > > >>> list(f() for f in (lambda:1, sit, lambda:2)) >> > > [1] >> > > In most cases, `list(generator)` works as expected. Thus, >> > > `list(<generator expression>)` is generally equivalent to >> `[<generator >> > > expression>]`. >> > Actually, it's list(generator) vs. a list comprehension. I agree that >> > it can be confusing, but Python considers them to be two different >> > constructs. I think nobody has addressed the OP arguments (as I understand them). First, except the obvious outer delimiters (and some corner cases in 2.x, fixed in Python 3), a list comprehension and a generator expression share the same syntax: (x for x in some_values) vs [x for x in some_values]. Also, *almost* always, both list(<comprehension>) and [<comprehension>], when evaluated, yield the same result [1]. *Almost* because StopIteration is handled differently as the OP discovered: the list comprehension propagates a StopIteration exception to its caller; the list constructor swallows the exception and the caller never sees it. Despite a promise in PEP 289, generator expressions semantics isn't explained in detail in the language reference. I can't tell if the difference is intentional, accidental, undocumented behavior, an implementation accident, a bug, or what... [1] <comprehension> being a syntactic construct like: x**2 for x in range(5) or: f() for f in [lambda:1, sit, lambda:2] -- Gabriel Genellina
From: exarkun on 13 Dec 2009 09:35 On 08:45 am, tjreedy(a)udel.edu wrote: >Tom Machinski wrote: >>In most cases, `list(generator)` works as expected. Thus, >>`list(<generator expression>)` is generally equivalent to `[<generator >>expression>]`. >> >>Here's a minimal case where this equivalence breaks, causing a serious >>and hard-to-detect bug in a program: >> >> >>> def sit(): raise StopIteration() > >StopIteration is intended to be used only within the .__next__ method >of iterators. The devs know that other 'off-label' use results in the >inconsistency you noted, but their and my view is 'don't do that'. Which is unfortunate, because it's not that hard to get StopIteration without explicitly raising it yourself and this behavior makes it difficult to debug such situations. What's with this view, exactly? Is it just that it's hard to implement the more desirable behavior? Jean-Paul
From: Steven D'Aprano on 13 Dec 2009 15:18 On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote: >>StopIteration is intended to be used only within the .__next__ method of >>iterators. The devs know that other 'off-label' use results in the >>inconsistency you noted, but their and my view is 'don't do that'. > > Which is unfortunate, because it's not that hard to get StopIteration > without explicitly raising it yourself and this behavior makes it > difficult to debug such situations. I can't think of any way to get StopIteration without explicitly raising it yourself. It's not like built-ins or common data structures routinely raise StopIteration. I don't think I've *ever* seen a StopIteration that I didn't raise myself. > What's with this view, exactly? Is it just that it's hard to implement > the more desirable behavior? What is that "more desirable behaviour"? That StopIteration is used to signal that Python should stop iterating except when you want it to be ignored? Unfortunately, yes, it's quite hard to implement "do what the caller actually wants, not what he asked for" behaviour -- and even if it were possible, it goes against the grain of the Zen of Python. If you've ever had to debug faulty "Do What I Mean" software, you'd see this as a good thing. -- Steven
From: exarkun on 13 Dec 2009 17:45
On 08:18 pm, steve(a)remove-this-cybersource.com.au wrote: >On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote: >>>StopIteration is intended to be used only within the .__next__ method >>>of >>>iterators. The devs know that other 'off-label' use results in the >>>inconsistency you noted, but their and my view is 'don't do that'. >> >>Which is unfortunate, because it's not that hard to get StopIteration >>without explicitly raising it yourself and this behavior makes it >>difficult to debug such situations. > >I can't think of any way to get StopIteration without explicitly >raising >it yourself. It's not like built-ins or common data structures >routinely >raise StopIteration. I don't think I've *ever* seen a StopIteration >that >I didn't raise myself. Call next on an iterator. For example: iter(()).next() > >>What's with this view, exactly? Is it just that it's hard to >>implement >>the more desirable behavior? > >What is that "more desirable behaviour"? That StopIteration is used to >signal that Python should stop iterating except when you want it to be >ignored? Unfortunately, yes, it's quite hard to implement "do what the >caller actually wants, not what he asked for" behaviour -- and even if >it >were possible, it goes against the grain of the Zen of Python. > >If you've ever had to debug faulty "Do What I Mean" software, you'd see >this as a good thing. I have plenty of experience developing and debugging software, Steven. Your argument is specious, as it presupposes that only two possibilities exist: the current behavior of some kind of magical faerie land behavior. I'm surprised to hear you say that the magical faerie land behavior isn't desirable either, though. I'd love a tool that did what I wanted, not what I asked. The only serious argument against this, I think, is that it is beyond our current ability to create (and so anyone claiming to be able to do it is probably mistaken). You chopped out all the sections of this thread which discussed the more desirable behavior. You can go back and read them in earlier messages if you need to be reminded. I'm not talking about anything beyond what's already been raised. I'm pretty sure I know the answer to my question, though - it's hard to implement, so it's not implemented. Jean-Paul |