A "scopeguard" for Python
in [Python]
|
From: Steve Howell on 5 Mar 2010 11:38 On Mar 5, 8:29 am, Mike Kent <mrmak...(a)gmail.com> wrote: > On Mar 4, 8:04 pm, Robert Kern <robert.k...(a)gmail.com> wrote: > > > No, the try: finally: is not implicit. See the source for > > contextlib.GeneratorContextManager. When __exit__() gets an exception from the > > with: block, it will push it into the generator using its .throw() method. This > > raises the exception inside the generator at the yield statement. > > Wow, I just learned something new. My understanding of context > managers was that the __exit__ method was guaranteed to be executed > regardless of how the context was left. I have often written my own > context manager classes, giving them the __enter__ and __exit__ > methods. I had mistakenly assumed that the @contextmanager decorator > turned a generator function into a context manager with the same > behavior as the equivalent context manager class. Now I learn that, > no, in order to have the 'undo' code executed in the presence of an > exception, you must write your own try/finally block in the generator > function. > > This raises the question in my mind: What's the use case for using > @contextmanager rather than wrapping your code in a context manager > class that defines __enter__ and __exit__, if you still have to > manager your own try/finally block? Unless I am misunderstanding the question, the use case is that you still only have to write the context manager once, and you might get multiple uses out of it where the with-enclosed code blocks work at a higher level of abstraction. I actually don't use @contextmanager yet, mainly because I did not know it existed until recently, but also because I find the __enter__/ __exit__ paradigm straightforward enough to just hand code them that way.
From: Steve Howell on 5 Mar 2010 11:49 On Mar 4, 5:04 pm, Robert Kern <robert.k...(a)gmail.com> wrote: > On 2010-03-04 15:19 PM, Mike Kent wrote: > > > > > On Mar 3, 12:00 pm, Robert Kern<robert.k...(a)gmail.com> wrote: > >> On 2010-03-03 09:39 AM, Mike Kent wrote: > > >>> What's the compelling use case for this vs. a simple try/finally? > > >>> original_dir = os.getcwd() > >>> try: > >>> os.chdir(somewhere) > >>> # Do other stuff > >>> finally: > >>> os.chdir(original_dir) > >>> # Do other cleanup > > >> A custom-written context manager looks nicer and can be more readable. > > >> from contextlib import contextmanager > >> import os > > >> @contextmanager > >> def pushd(path): > >> original_dir = os.getcwd() > >> os.chdir(path) > >> try: > >> yield > >> finally: > >> os.chdir(original_dir) > > >> with pushd(somewhere): > >> ... > > > Robert, I like the way you think. That's a perfect name for that > > context manager! However, you can clear one thing up for me... isn't > > the inner try/finally superfluous? My understanding was that there > > was an implicit try/finally already done which will insure that > > everything after the yield statement was always executed. > > No, the try: finally: is not implicit. See the source for > contextlib.GeneratorContextManager. When __exit__() gets an exception from the > with: block, it will push it into the generator using its .throw() method.. This > raises the exception inside the generator at the yield statement. > See also: http://docs.python.org/library/contextlib.html The closing() helper can be used to automatically call thing.close() even after an exception. If you do not use the closing() helper and take on the responsibility yourself of doing try/finally within your generator, I think you still gain some overall simplicity: 1) You don't need to do try/finally in your with blocks, of course. 2) The generator itself probably reads more straightforwardly then a hand-coded class with __enter__ and __exit__. For point #2, I think there are probably different aesthetics. Generators are more concise, but they are also a bit mind-bending.
From: Robert Kern on 5 Mar 2010 12:00 On 2010-03-05 10:29 AM, Mike Kent wrote: > On Mar 4, 8:04 pm, Robert Kern<robert.k...(a)gmail.com> wrote: > >> No, the try: finally: is not implicit. See the source for >> contextlib.GeneratorContextManager. When __exit__() gets an exception from the >> with: block, it will push it into the generator using its .throw() method. This >> raises the exception inside the generator at the yield statement. > > Wow, I just learned something new. My understanding of context > managers was that the __exit__ method was guaranteed to be executed > regardless of how the context was left. It is. @contextmanager turns a specially-written generator into a context manager with an __exit__ that does different things depending on whether or not and exception was raised. By pushing the exception into the generator, it lets the author decide what to do. It may catch a subset of exceptions, or no exceptions, or use a finally:. They all have use cases although finally: is the usual one. > I have often written my own > context manager classes, giving them the __enter__ and __exit__ > methods. I had mistakenly assumed that the @contextmanager decorator > turned a generator function into a context manager with the same > behavior as the equivalent context manager class. Basically, it does. __exit__() is given the exception information. When you write such a class, you can decide what to do with the exception. You can silence it, immediately reraise it, conditionally reraise it, log it and then reraise it, etc. Pushing the exception into the generator keeps this flexibility and the equivalence. If it removed that choice, then it would not be equivalent. > Now I learn that, > no, in order to have the 'undo' code executed in the presence of an > exception, you must write your own try/finally block in the generator > function. > > This raises the question in my mind: What's the use case for using > @contextmanager rather than wrapping your code in a context manager > class that defines __enter__ and __exit__, if you still have to > manager your own try/finally block? The @contextmanager generator implementations are often shorter and easier to read, in my opinion, partly because they use the try: finally: syntax that most of us are very familiar with. I have to think less when I read it because it looks so similar to the equivalent code that you would normally write. The point of context managers isn't to remove the use of try: finally: entirely, but to implement it once so that it can be reused cleanly. You only have to write the one try: finally: in the generator and reuse it simply with the with: statement in many places. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
From: Alf P. Steinbach on 5 Mar 2010 14:13 * Mike Kent: > On Mar 4, 8:04 pm, Robert Kern <robert.k...(a)gmail.com> wrote: > >> No, the try: finally: is not implicit. See the source for >> contextlib.GeneratorContextManager. When __exit__() gets an exception from the >> with: block, it will push it into the generator using its .throw() method. This >> raises the exception inside the generator at the yield statement. > > Wow, I just learned something new. My understanding of context > managers was that the __exit__ method was guaranteed to be executed > regardless of how the context was left. I have often written my own > context manager classes, giving them the __enter__ and __exit__ > methods. I had mistakenly assumed that the @contextmanager decorator > turned a generator function into a context manager with the same > behavior as the equivalent context manager class. Now I learn that, > no, in order to have the 'undo' code executed in the presence of an > exception, you must write your own try/finally block in the generator > function. > > This raises the question in my mind: What's the use case for using > @contextmanager rather than wrapping your code in a context manager > class that defines __enter__ and __exit__, if you still have to > manager your own try/finally block? Robert Kern and Steve Howell have already given given good answers. As it happened this was news to me also, because I'm not that well-versed in Python and it seems contrary to the purpose of providing a simpler way to write a simple init-cleanup wrapper. But additionally, if you want that, then you can define it, e.g. <code> # Py3 def simplecleanup( generator_func ): class SimpleCleanup: def __init__( self, *args, **kwargs ): self.generator = generator_func( *args, **kwargs ) def __enter__( self ): self.generator.send( None ) return self def __exit__( self, x_type, x_obj, x_traceback ): try: self.generator.send( x_obj ) # x_obj is None if no exception except StopIteration: pass # Expected return SimpleCleanup @simplecleanup def hello_goodbye( name ): print( "Hello, {}!".format( name ) ) yield print( "Goodbye {}!".format( name ) ) try: with hello_goodbye( "Mary" ): print( "Talk talk talk..." ) raise RuntimeError( "Offense" ) except: pass print() @simplecleanup def sensitive_hello_goodbye( name ): print( "Hello, {}!".format( name ) ) x = yield if x is not None: print( "Uh oh, {}!".format( x ) ) print( "Good day {}!".format( name ) ) else: print( "C u, {}!".format( name ) ) try: with sensitive_hello_goodbye( "Jane" ): print( "Talk talk talk..." ) raise RuntimeError( "Offense" ) except: pass </code> Cheers, - Alf
From: Alf P. Steinbach on 5 Mar 2010 14:34
* Steve Howell: > On Mar 3, 7:10 am, "Alf P. Steinbach" <al...(a)start.no> wrote: >> For C++ Petru Marginean once invented the "scope guard" technique (elaborated on >> by Andrei Alexandrescu, they published an article about it in DDJ) where all you >> need to do to ensure some desired cleanup at the end of a scope, even when the >> scope is exited via an exception, is to declare a ScopeGuard w/desired action. >> >> The C++ ScopeGuard was/is for those situations where you don't have proper >> classes with automatic cleanup, which happily is seldom the case in good C++ >> code, but languages like Java and Python don't support automatic cleanup and so >> the use case for something like ScopeGuard is ever present. >> >> For use with a 'with' statement and possibly suitable 'lambda' arguments: >> >> <code> >> class Cleanup: >> def __init__( self ): >> self._actions = [] >> >> def call( self, action ): >> assert( is_callable( action ) ) >> self._actions.append( action ) >> >> def __enter__( self ): >> return self >> >> def __exit__( self, x_type, x_value, x_traceback ): >> while( len( self._actions ) != 0 ): >> try: >> self._actions.pop()() >> except BaseException as x: >> raise AssertionError( "Cleanup: exception during cleanup" ) from >> </code> >> >> I guess the typical usage would be what I used it for, a case where the cleanup >> action (namely, changing back to an original directory) apparently didn't fit >> the standard library's support for 'with', like >> >> with Cleanup as at_cleanup: >> # blah blah >> chdir( somewhere ) >> at_cleanup.call( lambda: chdir( original_dir ) ) >> # blah blah >> >> Another use case might be where one otherwise would get into very deep nesting >> of 'with' statements with every nested 'with' at the end, like a degenerate tree >> that for all purposes is a list. Then the above, or some variant, can help to >> /flatten/ the structure. To get rid of that silly & annoying nesting. :-) >> >> Cheers, >> >> - Alf (just sharing, it's not seriously tested code) > > Hi Alf, I think I understand the notion you're going after here. You > have multiple cleanup steps that you want to defer till the end, and > there is some possibility that things will go wrong along the way, but > you want to clean up as much as you can. And, of course, flatter is > better. > > Is this sort of what you are striving for? > > class Cleanup: > def __init__( self ): > self._actions = [] > > def call( self, action ): > self._actions.append( action ) > > def __enter__( self ): > return self > > def __exit__( self, x_type, x_value, x_traceback ): > while( len( self._actions ) != 0 ): > try: > self._actions.pop()() > except BaseException as x: > raise AssertionError( "Cleanup: exception during > cleanup" ) > > def clean_the_floor(): > print('clean the floor') > > def carouse(num_bottles, accident): > with Cleanup() as at_cleanup: > at_cleanup.call(clean_the_floor) > for i in range(num_bottles): > def take_down(i=i): > print('take one down', i) > at_cleanup.call(take_down) > if i == accident: > raise Exception('oops!') > print ('put bottle on wall', i) > > carouse(10, None) > carouse(5, 3) He he. I think you meant: > def carouse(num_bottles, accident): > with Cleanup() as at_cleanup: > at_cleanup.call(clean_the_floor) > for i in range(num_bottles): > def take_down(i=i): > print('take one down', i) > if i == accident: > raise Exception('oops!') > print ('put bottle on wall', i) > at_cleanup.call(take_down) I'm not sure. It's interesting & fun. But hey, it's Friday evening. Regarding the "clean the floor", Marginean's original ScopeGuard has a 'dismiss' method (great for e.g. implementing transactions). There's probably also much other such functionality that can be added. The original use case for Cleanup, I just posted this in case people could find it useful, was a harness for testing that C++ code /fails/ as it should, <url: http://pastebin.com/NK8yVcyv>, where Cleanup is used at line 479. Some discussion of that in Usenet message and associated thread <hmmcdm$p1i$1(a)news.eternal-september.org>, "Unit testing of expected failures -- what do you use?" in [comp.lang.c++]. Another similar approach was discussed by Carlo Milanesi in <url: http://www.drdobbs.com/cpp/205801074>, but he supplied this reference after I'd done the above. Mainly the difference is that he defines a custom mark-up language with corresponding source preprocessing, while I use the ordinary C++ preprocessor. There are advantages and disadvantages to both approaches. Cheers, - Alf |