Dangerous behavior of list(generator)
in [Python]
|
Prev: Spawning an interactive interpreter in a running python process?
Next: unable to read the __main__ namespace
From: Tom Machinski on 12 Dec 2009 19:15 In most cases, `list(generator)` works as expected. Thus, `list(<generator expression>)` is generally equivalent to `[<generator expression>]`. Here's a minimal case where this equivalence breaks, causing a serious and hard-to-detect bug in a program: >>> def sit(): raise StopIteration() ... >>> [f() for f in (lambda:1, sit, lambda:2)] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in sit StopIteration >>> list(f() for f in (lambda:1, sit, lambda:2)) [1] I was bitten hard by this inconsistency when sit() was returning the idiom `(foo for foo in bar if foo.is_baz()).next()`. The nonexistence of a foo with is_baz() True in that query raises an exception as designed, which expresses itself when I use the list comprehension version of the code above; the generator version muffles the error and silently introduces a subtle, confusing bug: `lambda:2` is never reached, and a truncated list of 1 element (instead of 3) is "successfully" generated.. Just wondered what you guys think, -- Tom
From: Benjamin Kaplan on 12 Dec 2009 19:53 On Sat, Dec 12, 2009 at 7:15 PM, Tom Machinski <tom.machinski(a)gmail.com> wrote: > In most cases, `list(generator)` works as expected. Thus, > `list(<generator expression>)` is generally equivalent to `[<generator > expression>]`. > Actually, it's list(generator) vs. a list comprehension. I agree that it can be confusing, but Python considers them to be two different constructs. >>> list(xrange(10)) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> [xrange(10)] [xrange(10)] > Here's a minimal case where this equivalence breaks, causing a serious > and hard-to-detect bug in a program: > > >>> def sit(): raise StopIteration() > ... > >>> [f() for f in (lambda:1, sit, lambda:2)] > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "<stdin>", line 1, in sit > StopIteration > >>> list(f() for f in (lambda:1, sit, lambda:2)) > [1] > > I was bitten hard by this inconsistency when sit() was returning the > idiom `(foo for foo in bar if foo.is_baz()).next()`. The nonexistence > of a foo with is_baz() True in that query raises an exception as > designed, which expresses itself when I use the list comprehension > version of the code above; the generator version muffles the error and > silently introduces a subtle, confusing bug: `lambda:2` is never > reached, and a truncated list of 1 element (instead of 3) is > "successfully" generated.. > > Just wondered what you guys think, > > -- Tom > -- > http://mail.python.org/mailman/listinfo/python-list >
From: Ned Deily on 12 Dec 2009 21:01 In article <ec96e1390912121653w56c3dbe3p859a7b979026bf47(a)mail.gmail.com>, Benjamin Kaplan <benjamin.kaplan(a)case.edu> wrote: > On Sat, Dec 12, 2009 at 7:15 PM, Tom Machinski <tom.machinski(a)gmail.com> > wrote: > > In most cases, `list(generator)` works as expected. Thus, > > `list(<generator expression>)` is generally equivalent to `[<generator > > expression>]`. > Actually, it's list(generator) vs. a list comprehension. I agree that > it can be confusing, but Python considers them to be two different > constructs. > > >>> list(xrange(10)) > [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] > >>> [xrange(10)] > [xrange(10)] That's not a list comprehension, that's a list with one element. >>> [x for x in xrange(10)] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] <CrocodileDundee> Now *that's* a list comprehension. </CrocodileDundee> -- Ned Deily, nad(a)acm.org
From: Benjamin Kaplan on 12 Dec 2009 21:16 On Sat, Dec 12, 2009 at 9:01 PM, Ned Deily <nad(a)acm.org> wrote: > In article > <ec96e1390912121653w56c3dbe3p859a7b979026bf47(a)mail.gmail.com>, > Benjamin Kaplan <benjamin.kaplan(a)case.edu> wrote: >> On Sat, Dec 12, 2009 at 7:15 PM, Tom Machinski <tom.machinski(a)gmail.com> >> wrote: >> > In most cases, `list(generator)` works as expected. Thus, >> > `list(<generator expression>)` is generally equivalent to `[<generator >> > expression>]`. >> Actually, it's list(generator) vs. a list comprehension. I agree that >> it can be confusing, but Python considers them to be two different >> constructs. >> >> >>> list(xrange(10)) >> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >> >>> [xrange(10)] >> [xrange(10)] > > That's not a list comprehension, that's a list with one element. > >>>> [x for x in xrange(10)] > [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] > > <CrocodileDundee> Now *that's* a list comprehension. </CrocodileDundee> > I know. But the OP was wondering why list(<generator expression>) was behaving differently than [<generator-expression>] and I was pointing out that list comprehensions are considered their own syntax- the list comprehension [x for x in xrange(10)] is different than [(x for x in xrange(10)]. > -- > Ned Deily, > nad(a)acm.org > > -- > http://mail.python.org/mailman/listinfo/python-list >
From: Ned Deily on 12 Dec 2009 21:43
In article <nad-8CDB63.18012412122009(a)news.gmane.org>, Ned Deily <nad(a)acm.org> wrote: > In article > <ec96e1390912121653w56c3dbe3p859a7b979026bf47(a)mail.gmail.com>, > Benjamin Kaplan <benjamin.kaplan(a)case.edu> wrote: > > On Sat, Dec 12, 2009 at 7:15 PM, Tom Machinski <tom.machinski(a)gmail.com> > > wrote: > > > In most cases, `list(generator)` works as expected. Thus, > > > `list(<generator expression>)` is generally equivalent to `[<generator > > > expression>]`. > > Actually, it's list(generator) vs. a list comprehension. I agree that > > it can be confusing, but Python considers them to be two different > > constructs. > > > > >>> list(xrange(10)) > > [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] > > >>> [xrange(10)] > > [xrange(10)] > > That's not a list comprehension, that's a list with one element. > > >>> [x for x in xrange(10)] > [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] > > <CrocodileDundee> Now *that's* a list comprehension. </CrocodileDundee> Which is not quite the point Benjamin was trying to make - sorry! Consulting the adjacent sections on "List displays" and "Generator expressions" in the Language Reference: http://docs.python.org/reference/expressions.html#list-displays for generator expressions "the parentheses can be omitted on calls with only one argument " but the expressions in a list_comprehension are not in a call context. So there is no ambiguity: [<generator expression>] requires parens around the generator expression and that list display produces a list with one element as Benjamin points out. -- Ned Deily, nad(a)acm.org |