Overriding "__setattr__" of a module - possible?
in [Python]
|
From: Gabriel Genellina on 17 Jun 2010 03:25 En Wed, 16 Jun 2010 19:56:39 -0300, Ian Kelly <ian.g.kelly(a)gmail.com> escribi�: > On Wed, Jun 16, 2010 at 3:38 PM, John Nagle <nagle(a)animats.com> wrote: >> That just leaves things in a state where even "sys" and "import" >> are undefined. > > Say what? It works fine for me. > >>>> import proxy_mod >>>> proxy_mod.f() > 1 >>>> proxy_mod.a = 2 > setting a=2 >>>> proxy_mod.f() > 2 >>>> proxy_mod.sys > <module 'sys' (built-in)> It *mostly* works, but not always. Try this function: def g(x): global a print 'global a -> ', x a = x py> import fake # ModuleProxy stuff py> fake.f() 1 py> fake.a = 3 setting a=3 py> fake.f() 3 py> fake.g(8) global a -> 8 py> fake.f() 8 py> fake.a 8 Note the fake.g(8) call: __setattr__ wasn't called. If the OP wants to trace assignments to global variables, this becomes a problem. A function defined in a module holds a reference to the module's __dict__ in its func_globals attribute. Getting and setting global variables goes directly to this dictionary, and does not use the module object at all. Even worse, the LOAD_GLOBAL/STORE_GLOBAL opcodes (which implement getting and setting global variables) assume func_globals is a true dictionary and bypass any overriden __getitem__/__setitem__ methods (an optimization, surely). I'm afraid it will be hard to intercept global variable usage in these circumstances. -- Gabriel Genellina
From: Alf P. Steinbach on 17 Jun 2010 03:52 * Gabriel Genellina, on 17.06.2010 09:25: > En Wed, 16 Jun 2010 19:56:39 -0300, Ian Kelly <ian.g.kelly(a)gmail.com> > escribi�: > >> On Wed, Jun 16, 2010 at 3:38 PM, John Nagle <nagle(a)animats.com> wrote: >>> That just leaves things in a state where even "sys" and "import" >>> are undefined. >> >> Say what? It works fine for me. >> >>>>> import proxy_mod >>>>> proxy_mod.f() >> 1 >>>>> proxy_mod.a = 2 >> setting a=2 >>>>> proxy_mod.f() >> 2 >>>>> proxy_mod.sys >> <module 'sys' (built-in)> > > It *mostly* works, but not always. Try this function: > > def g(x): > global a > print 'global a -> ', x > a = x > > py> import fake # ModuleProxy stuff > py> fake.f() > 1 > py> fake.a = 3 > setting a=3 > py> fake.f() > 3 > py> fake.g(8) > global a -> 8 > py> fake.f() > 8 > py> fake.a > 8 > > Note the fake.g(8) call: __setattr__ wasn't called. > If the OP wants to trace assignments to global variables, this becomes a > problem. > > A function defined in a module holds a reference to the module's > __dict__ in its func_globals attribute. Getting and setting global > variables goes directly to this dictionary, and does not use the module > object at all. > > Even worse, the LOAD_GLOBAL/STORE_GLOBAL opcodes (which implement > getting and setting global variables) assume func_globals is a true > dictionary and bypass any overriden __getitem__/__setitem__ methods (an > optimization, surely). I'm afraid it will be hard to intercept global > variable usage in these circumstances. Great exposition. But who would have thunk that Python *isn't dynamic enough*? :-) Cheers, - Alf -- blog at <url: http://alfps.wordpress.com>
From: Gabriel Genellina on 17 Jun 2010 05:29 En Thu, 17 Jun 2010 04:52:48 -0300, Alf P. Steinbach <alfps(a)start.no> escribi�: > * Gabriel Genellina, on 17.06.2010 09:25: >> En Wed, 16 Jun 2010 19:56:39 -0300, Ian Kelly <ian.g.kelly(a)gmail.com> >> escribi�: >> >>> On Wed, Jun 16, 2010 at 3:38 PM, John Nagle <nagle(a)animats.com> wrote: >>>> That just leaves things in a state where even "sys" and "import" >>>> are undefined. >>> >>> Say what? It works fine for me. >>> >>>>>> import proxy_mod >>>>>> proxy_mod.f() >>> 1 >>>>>> proxy_mod.a = 2 >>> setting a=2 >>>>>> proxy_mod.f() >>> 2 >>>>>> proxy_mod.sys >>> <module 'sys' (built-in)> >> >> It *mostly* works, but not always. Try this function: >> >> def g(x): >> global a >> print 'global a -> ', x >> a = x >> >> py> import fake # ModuleProxy stuff >> py> fake.f() >> 1 >> py> fake.a = 3 >> setting a=3 >> py> fake.f() >> 3 >> py> fake.g(8) >> global a -> 8 >> py> fake.f() >> 8 >> py> fake.a >> 8 >> >> Note the fake.g(8) call: __setattr__ wasn't called. >> If the OP wants to trace assignments to global variables, this becomes a >> problem. >> >> A function defined in a module holds a reference to the module's >> __dict__ in its func_globals attribute. Getting and setting global >> variables goes directly to this dictionary, and does not use the module >> object at all. >> >> Even worse, the LOAD_GLOBAL/STORE_GLOBAL opcodes (which implement >> getting and setting global variables) assume func_globals is a true >> dictionary and bypass any overriden __getitem__/__setitem__ methods (an >> optimization, surely). I'm afraid it will be hard to intercept global >> variable usage in these circumstances. > > Great exposition. > > But who would have thunk that Python *isn't dynamic enough*? :-) Yep... There are other examples too (e.g. the print statement in 2.x bypasses sys.stdout.write; see also a recent thread "Which objects are expanded by double-star ** operator?") Most of them seem to be speed optimizations, some might be considered subtle bugs. But in this case (global variable references) speed is so critical than even the dict lookup is inlined; the code in ceval.c says: /* Inline the PyDict_GetItem() calls. WARNING: this is an extreme speed hack. Do not try this at home. */ Python is dynamic but not so much as to make it crawl like a snail... -- Gabriel Genellina
From: Fuzzyman on 17 Jun 2010 06:12 On Jun 17, 10:29 am, "Gabriel Genellina" <gagsl-...(a)yahoo.com.ar> wrote: > En Thu, 17 Jun 2010 04:52:48 -0300, Alf P. Steinbach <al...(a)start.no> > escribió: > > > But who would have thunk that Python *isn't dynamic enough*? :-) > > Yep... There are other examples too (e.g. the print statement in 2.x > bypasses sys.stdout.write; What do you mean by this? The print statement in 2.x does *not* bypass sys.stdout. It may use other methods besides write (writeln perhaps) but you can *definitely* override sys.stdout to capture the output from print statements. Michael Foord > see also a recent thread "Which objects are > expanded by double-star ** operator?") > > Most of them seem to be speed optimizations, some might be considered > subtle bugs. But in this case (global variable references) speed is so > critical than even the dict lookup is inlined; the code in ceval.c says: > > /* Inline the PyDict_GetItem() calls. > WARNING: this is an extreme speed hack. > Do not try this at home. */ > > Python is dynamic but not so much as to make it crawl like a snail... > > -- > Gabriel Genellina -- http://www.voidspace.org.uk/
From: John Nagle on 17 Jun 2010 13:09 On 6/17/2010 12:25 AM, Gabriel Genellina wrote: > Note the fake.g(8) call: __setattr__ wasn't called. > If the OP wants to trace assignments to global variables, this becomes a > problem. > > A function defined in a module holds a reference to the module's > __dict__ in its func_globals attribute. Getting and setting global > variables goes directly to this dictionary, and does not use the module > object at all. > > Even worse, the LOAD_GLOBAL/STORE_GLOBAL opcodes (which implement > getting and setting global variables) assume func_globals is a true > dictionary and bypass any overriden __getitem__/__setitem__ methods (an > optimization, surely). I'm afraid it will be hard to intercept global > variable usage in these circumstances. OK, thanks. You can't actually replace "__setattr__" for a module, and the "module as class" hack is iffy. I didn't really think this would work, but it was worth a try. I'm trying out a proof of concept implementation for a new approach to safe threading. It's somewhat similar in concept to Alan Olsen's scheme. The basic difference is that once the program goes multi-thread, code objects and some other bindings are locked down and become unchangeable. Olsen was climbing the walls trying to get the locking right for the awful cases like redefining a function while another thread is inside it. I'm trying to lock out some of those cases. If you do that, removing the GIL requires less pain than Olsen experienced. The key idea is that the use cases for most of Python's code dynamism are during setup and initialization. You usually don't change code once the program has gone into its heavy parallel processing phase. This suggests a practical compromise. More on this once I have something people can download and try. I'm doing a test implementation in Python so people can try the concept and see if it works in practice. It won't go fast; it's just to give a feel for what it would be like. John Nagle
First
|
Prev
|
Next
|
Last
Pages: 1 2 3 4 5 Prev: Python editing .txt file Next: [ANN] OpenOpt 0.29, FuncDesigner 0.19, DerApproximator 0.19 |