Dangerous behavior of list(generator)
in [Python]
|
Prev: Spawning an interactive interpreter in a running python process?
Next: unable to read the __main__ namespace
From: Terry Reedy on 14 Dec 2009 02:08 On 12/13/2009 11:33 PM, exarkun(a)twistedmatrix.com wrote: > But if you mistakenly don't catch it, and you're trying to debug your > code to find this mistake, you probably won't be aided in this pursuit > by the exception-swallowing behavior of generator expressions. As I remember, it was the call to list that swalled the exception, not the generator expression. List() takes an iterable as arg and stopping on StopIteration is what it does and how it knows to stop and return the new list. > The behavior of list comprehensions is pretty good. The behavior of > constructing a list out of a generator expression isn't as good. I think you are confused. A generator expression is a shorthand for a def statement that defines a generator function followed by a call to the generator function to get a generator followed by deletion of the function. When you call list() to make a list, it constructs the list from the generator, not from the expression itself. List has no idea that you used a generator expression or even that it was passed a generator. Leaving error checks out, it operates something like def list(it): res = [] it = iter(it) for item in it: # stops whenever it raises StopIteration res.append(item) return res > The > behavior which is more desirable is for a StopIteration raised out of > the `expression` part of a `generator_expression` to not be treated > identically to the way a StopIteration raised out of the `genexpr_for` > part is. It is not. StopIteration in for part stops the for loop in the generator. StopIteration in the expression part stops the loop in the list() call (sooner than it would have been otherwise). When the generator raises StopIteration, list() has no idea what statement within the body raised it. It MUST stop. > This could provide behavior roughly equivalent to the behavior > of a list comprehension. Impossible. The only serious option for consistency is to special case list comps to also trap StopIteration raised in the expression part, but the devs decided not to do this as doing do is arguably a bug. Terry Jan Reedy
From: Lie Ryan on 14 Dec 2009 05:26 On 12/14/09, exarkun(a)twistedmatrix.com <exarkun(a)twistedmatrix.com> wrote: > On 02:50 am, lie.1296(a)gmail.com wrote: >>On 12/14/2009 9:45 AM, exarkun(a)twistedmatrix.com wrote: >>>On 08:18 pm, steve(a)remove-this-cybersource.com.au wrote: >>>>On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote: >>>>>>StopIteration is intended to be used only within the .__next__ >>>>>>method of >>>>>>iterators. The devs know that other 'off-label' use results in the >>>>>>inconsistency you noted, but their and my view is 'don't do that'. >>>>> >>>>>Which is unfortunate, because it's not that hard to get >>>>>StopIteration >>>>>without explicitly raising it yourself and this behavior makes it >>>>>difficult to debug such situations. >>>> >>>>I can't think of any way to get StopIteration without explicitly >>>>raising >>>>it yourself. It's not like built-ins or common data structures >>>>routinely >>>>raise StopIteration. I don't think I've *ever* seen a StopIteration >>>>that >>>>I didn't raise myself. >>> >>>Call next on an iterator. For example: iter(()).next() >> >>.next() is not meant to be called directly > > Doesn't matter. Sometimes it makes sense to call it directly. And I > was just giving an example of a way to get StopIteration raised without > doing it yourself - which is what Steve said he couldn't think of. >>> >>>I'm surprised to hear you say that the magical faerie land behavior >>>isn't desirable either, though. I'd love a tool that did what I >>>wanted, >>>not what I asked. The only serious argument against this, I think, is >>>that it is beyond our current ability to create (and so anyone >>>claiming >>>to be able to do it is probably mistaken). >> >>In your world, this is what happens: >> >>> list = [a, b, c] >> >>> # print list >> >>> print list >>["a", "b", "c"] >> >>> # make a copy of list >> >>> alist = list(llst) # oops a mistype >> >>> alist = alist - "]" + ", "d"]" >> >>> print alist >>["a", "b", "c", "d"] >> >>> alist[:6] + "i", + alist[6:] >> >>> print alist >>["a", "i", "b", "c", "d"] >> >>> print alist >> >>> # hearing the sound of my deskjet printer... >> >>> C:\fikle.text.write(alist) >> >>> print open("C:\file.txt").read() >><h1>a</h1> >><ul> >><li>i</li> >><li>b</li> >><li>c d</li> >> >>> # great, exactly what I needed > > I don't understand the point of this code listing, sorry. I suspect you > didn't completely understand the magical faerie land I was describing - > where all your programs would work, no matter what mistakes you made > while writing them. Exactly, that's what's happening. It just works. It knows that when I said alist[:6] + "i", + alist[6:] ; I want to insert "i" between the sixth character of the textual representation of the list. It knows to find the correct variable when I made a typo. It correctly guess that I want to print to a paper instead of to screen. It knows that when I wrote to C:\path.write(), it knows I wanted a HTML output in that specific format. It just works (TM), whatever mistakes I made. That's what you wanted, right?
From: Peter Otten on 14 Dec 2009 05:35 Terry Reedy wrote: > On 12/13/2009 11:33 PM, exarkun(a)twistedmatrix.com wrote: >> This could provide behavior roughly equivalent to the behavior >> of a list comprehension. > > Impossible. The only serious option for consistency is to special case > list comps to also trap StopIteration raised in the expression part, but > the devs decided not to do this as doing do is arguably a bug. A viable option might be to introduce a different exception type and translate (expr(v) for v in items if cond(v)) into def gen(items, expr, cond): for v in items: try: if cond(v): yield expr(v) except StopIteration: raise TypeError("StopIteration raised in " "'expr' or 'cond' part of " "a generator expression") Peter
From: exarkun on 14 Dec 2009 09:31 On 06:46 am, tjreedy(a)udel.edu wrote: >On 12/13/2009 10:29 PM, exarkun(a)twistedmatrix.com wrote: >>Doesn't matter. Sometimes it makes sense to call it directly. > >It only makes sense to call next (or .__next__) when you are prepared >to explicitly catch StopIteration within a try..except construct. >You did not catch it, so it stopped execution. > >Let me repeat: StopIteration is intended only for stopping iteration. >Outside that use, it is a normal exception with no special meaning. You cut out the part of my message where I wrote that one might have forgotten the exception handling code that you posit is required, and that the current behavior makes debugging this situation unnecessarily challenging. Jean-Paul
From: M.-A. Lemburg on 14 Dec 2009 09:58
exarkun(a)twistedmatrix.com wrote: > On 08:45 am, tjreedy(a)udel.edu wrote: >> Tom Machinski wrote: >>> In most cases, `list(generator)` works as expected. Thus, >>> `list(<generator expression>)` is generally equivalent to `[<generator >>> expression>]`. >>> >>> Here's a minimal case where this equivalence breaks, causing a serious >>> and hard-to-detect bug in a program: >>> >>> >>> def sit(): raise StopIteration() >> >> StopIteration is intended to be used only within the .__next__ method >> of iterators. The devs know that other 'off-label' use results in the >> inconsistency you noted, but their and my view is 'don't do that'. > > Which is unfortunate, because it's not that hard to get StopIteration > without explicitly raising it yourself and this behavior makes it > difficult to debug such situations. > > What's with this view, exactly? Is it just that it's hard to implement > the more desirable behavior? I'm not exactly sure what you're asking for. The StopIteration exception originated as part of the for-loop protocol. Later on it was generalized to apply to generators as well. The reason for using an exception is simple: raising and catching exceptions is fast at C level and since the machinery for communicating exceptions up the call stack was already there (and doesn't interfere with the regular return values), this was a convenient method to let the upper call levels know that an iteration has ended (e.g. a for-loop 4 levels up the stack). I'm not sure whether that answers your question, but it's the reason for things being as they are :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 14 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ |