Prev: example of multi threads
Next: Another MySQL Problem
From: John Reid on 23 Jun 2010 07:40
Hi,

I've written a decorator that prints exceptions and I'm having some
trouble with garbage collection.

My decorator is:

import sys
def print_exception_decorator(fn):
def decorator(self, *args, **kwds):
try:
return fn(*args, **kwds)
except:
print 'Exception:', sys.exc_info()
raise
return decorator



The class I want to decorate the methods of is:

class InstanceCounted(object):
"A class that keeps track of how many instances there are."
count = 0
def __init__(self):
InstanceCounted.count += 1
def __del__(self):
InstanceCounted.count -= 1

class A(InstanceCounted):
"A class that I want to decorate a method on."
def __init__(self):
super(A, self).__init__()
self.method = print_exception_decorator(self.method)

def __del__(self):
del self.method

def method(self):
pass



When I run the following it does not seem like my object 'a' is garbage
collected:

print 'Have %d instances' % InstanceCounted.count
print 'Creating A'
a = A()
print 'Have %d instances' % InstanceCounted.count
print 'Deleting A'
del a
print 'Have %d instances' % InstanceCounted.count


This is the output:

Have 0 instances
Creating A
Have 1 instances
Deleting A
Have 1 instances


The InstanceCounted.count is 1 at the end. If I omit the call to
"self.method = print_exception_decorator(self.method)" then the instance
count goes down to 0 as desired. I thought that the decorator might be
holding a reference to the instance through the bound method, so I added
the __del__() but it doesn't fix the problem.

Can anyone suggest anything? Is my technique to decorate bound methods
not a good one? How else should I decorate a bound method?

Thanks in advance,
John.


From: Thomas Jollans on 23 Jun 2010 08:27
On 06/23/2010 01:40 PM, John Reid wrote:
> Hi,
>
> I've written a decorator that prints exceptions and I'm having some
> trouble with garbage collection.
>
> My decorator is:
>
> import sys
> def print_exception_decorator(fn):
> def decorator(self, *args, **kwds):
> try:
> return fn(*args, **kwds)
> except:
> print 'Exception:', sys.exc_info()
> raise
> return decorator
>
>
>
> The class I want to decorate the methods of is:
>
> class InstanceCounted(object):
> "A class that keeps track of how many instances there are."
> count = 0
> def __init__(self):
> InstanceCounted.count += 1
> def __del__(self):
> InstanceCounted.count -= 1
>
> class A(InstanceCounted):
> "A class that I want to decorate a method on."
> def __init__(self):
> super(A, self).__init__()
> self.method = print_exception_decorator(self.method)
>
> def __del__(self):
> del self.method
>
> def method(self):
> pass
>
>
>
> When I run the following it does not seem like my object 'a' is garbage
> collected:
>
> print 'Have %d instances' % InstanceCounted.count
> print 'Creating A'
> a = A()
> print 'Have %d instances' % InstanceCounted.count
> print 'Deleting A'
> del a
> print 'Have %d instances' % InstanceCounted.count
>
>
> This is the output:
>
> Have 0 instances
> Creating A
> Have 1 instances
> Deleting A
> Have 1 instances
>
>
> The InstanceCounted.count is 1 at the end. If I omit the call to
> "self.method = print_exception_decorator(self.method)" then the instance
> count goes down to 0 as desired. I thought that the decorator might be
> holding a reference to the instance through the bound method, so I added
> the __del__() but it doesn't fix the problem.

Adding __del__ like this does "fix the problem", but it introduces a new
one: lacking a call to super().__del__, you simply don't decrement the
instance count.

To decorate a method, you'd best just decorate it normally. I doubt your
technique will work anyway, as the function returned by the decorator
isn't bound to the object, you'd need to pass one self reference
implicitly, which is then thrown away.

simply,

def exc_decor(fn):
@functools.wraps(fn)
def wrapper(*args, **keywords):
try:
return fn(*args, **keywords):
except:
#...
raise
return wrapper

class X(...):
@exc_decor
def foo(self, arg):
pass

(if targeting pre-decorator Python, the code would look different of course)

This way, the function itself is decorated, and the function returned by
the decorator is bound to the object. It'll just work as expected, no
trickery required.

-- Thomas

>
> Can anyone suggest anything? Is my technique to decorate bound methods
> not a good one? How else should I decorate a bound method?
>
> Thanks in advance,
> John.
>
>

From: Peter Otten on 23 Jun 2010 08:39
John Reid wrote:

> Hi,
>
> I've written a decorator that prints exceptions and I'm having some
> trouble with garbage collection.
>
> My decorator is:
>
> import sys
> def print_exception_decorator(fn):
> def decorator(self, *args, **kwds):
> try:
> return fn(*args, **kwds)
> except:
> print 'Exception:', sys.exc_info()
> raise
> return decorator
>
>
>
> The class I want to decorate the methods of is:
>
> class InstanceCounted(object):
> "A class that keeps track of how many instances there are."
> count = 0
> def __init__(self):
> InstanceCounted.count += 1
> def __del__(self):
> InstanceCounted.count -= 1
>
> class A(InstanceCounted):
> "A class that I want to decorate a method on."
> def __init__(self):
> super(A, self).__init__()
> self.method = print_exception_decorator(self.method)
>
> def __del__(self):
> del self.method
>
> def method(self):
> pass
>
>
>
> When I run the following it does not seem like my object 'a' is garbage
> collected:
>
> print 'Have %d instances' % InstanceCounted.count
> print 'Creating A'
> a = A()
> print 'Have %d instances' % InstanceCounted.count
> print 'Deleting A'
> del a
> print 'Have %d instances' % InstanceCounted.count
>
>
> This is the output:
>
> Have 0 instances
> Creating A
> Have 1 instances
> Deleting A
> Have 1 instances
>
>
> The InstanceCounted.count is 1 at the end. If I omit the call to
> "self.method = print_exception_decorator(self.method)" then the instance
> count goes down to 0 as desired. I thought that the decorator might be
> holding a reference to the instance through the bound method, so I added
> the __del__() but it doesn't fix the problem.
>
> Can anyone suggest anything? Is my technique to decorate bound methods
> not a good one? How else should I decorate a bound method?

The problem is that cyclic garbage collection cannot cope with __del__()
methods. Quoting http://docs.python.org/library/gc.html#gc.garbage

"""
Objects that have __del__() methods and are part of a reference cycle cause
the entire reference cycle to be uncollectable
"""

The best workaround is to control the object's lifetime explicitly by
turning it into a context manager, see

http://docs.python.org/reference/datamodel.html#with-statement-context-
managers

or providing a close() method and use contextlib:

>>> class A(object):
.... def close(self):
.... print "break cycles here"
....
>>> class A(object):
.... def __init__(self):
.... self.self = self
.... def __del__(self):
.... print "bye"
.... def close(self):
.... print "breaking cycles here"
.... del self.self
....
>>> a = A()
>>> del a
>>> import contextlib
>>> with contextlib.closing(A()) as b:
.... print "using b"
....
using b
breaking cycles here
>>> del b
bye

Peter
From: Christian Heimes on 23 Jun 2010 08:51
> The InstanceCounted.count is 1 at the end. If I omit the call to
> "self.method = print_exception_decorator(self.method)" then the instance
> count goes down to 0 as desired. I thought that the decorator might be
> holding a reference to the instance through the bound method, so I added
> the __del__() but it doesn't fix the problem.

Adding __del__ introduces a whole bunch of new issues with cyclic gc.
http://docs.python.org/library/gc.html#gc.garbage

> Can anyone suggest anything? Is my technique to decorate bound methods
> not a good one? How else should I decorate a bound method?

You shouldn't use __del__ to count your instances. Instead you can use
sys.getrefcount() and gc.get_objects() or use a weakref with a callback.

Christian

From: John Reid on 23 Jun 2010 08:56


Thomas Jollans wrote:
>> The InstanceCounted.count is 1 at the end. If I omit the call to
>> "self.method = print_exception_decorator(self.method)" then the instance
>> count goes down to 0 as desired. I thought that the decorator might be
>> holding a reference to the instance through the bound method, so I added
>> the __del__() but it doesn't fix the problem.
>
> Adding __del__ like this does "fix the problem", but it introduces a new
> one: lacking a call to super().__del__, you simply don't decrement the
> instance count.

Now that's a good point! I've added super().__del__ but the problem
remains for some reason. My A class now looks like:

class A(InstanceCounted):
"A class that I want to decorate a method on."
def __init__(self):
super(A, self).__init__()
self.method = print_exception_decorator(self.method)

def __del__(self):
super(A, self).__del__()
del self.method

def method(self):
pass


Did you try this?

>
> To decorate a method, you'd best just decorate it normally. I doubt your
> technique will work anyway, as the function returned by the decorator
> isn't bound to the object, you'd need to pass one self reference
> implicitly, which is then thrown away.

Looking at the original post, I had included an extra "self" in the
argument list

def decorator(self, *args, **kwds):

should have been

def decorator(*args, **kwds):


With this correction my method decorates instance methods on objects
successfully although I don't know why the garbage collection isn't working.



>
> simply,
>
> def exc_decor(fn):
> @functools.wraps(fn)
> def wrapper(*args, **keywords):
> try:
> return fn(*args, **keywords):
> except:
> #...
> raise
> return wrapper
>
> class X(...):
> @exc_decor
> def foo(self, arg):
> pass
>
> (if targeting pre-decorator Python, the code would look different of course)
>
> This way, the function itself is decorated, and the function returned by
> the decorator is bound to the object. It'll just work as expected, no
> trickery required.

Thanks for this. I remember having some problems decorating instance
methods in the past which is why I started doing it as in the original
post. Your method seems just fine though.

Thanks,
John.

 |  Next  |  Last
Pages: 1 2
Prev: example of multi threads
Next: Another MySQL Problem