Serializing functions
in [Python]
|
Prev: Could not import external source folder in Pydev
Next: Efficiency/style issues of import <module> vs. from <module>import <name>, ...
From: Robert Kern on 17 Jun 2010 12:26 On 6/17/10 8:23 AM, Matteo Landi wrote: > Some weeks ago, here on the mailing list I read about picloud[1], a > python library used for cloud-computing; I was impressed by its > simplicity, here is an example: > >>>> import cloud >>>> def square(x): > ... return x * x >>>> cloud.call(square, 10) >>>> cloud.result() > 100 > > So, I tried to figure out how to achieve the same result, i.e. define a > local generic function and send it somewhere for remote execution, and > the get the result back. > So I thought about serialization (pickle): I made a few tries and it > seemed to work.. but I was wrong, because I never tried to load pickled > data from another session different from the one used to pickle data > itself. If you try and pickle a function, it is not pickled as a whole, > indeed, once you unpickle it, it will raise an exception telling you > that the target function was not found in the current module. > > So I'm here, with nothing in my hands; how would you implement this? PiCloud's client library is LGPLed. Go take a look for yourself. It's pretty nifty. [Disclosure: My employer, Enthought, has partnered with PiCloud to provide our Enthought Python Distribution to PiCloud users. However, I say that it's nifty purely because it is.] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
From: Aaron Staley on 17 Jun 2010 16:05 I am one of the developer's of PiCloud. To answer your question, we wrote a custom subclass of Pickler to pickle functions. As Robert pointed out, the library is LGPL, so you can see (and use) the source code. I also presented the details on a poster at PyCon 2010. You can see it here: http://blog.picloud.com/?p=96 On Jun 17, 9:26 am, Robert Kern <robert.k...(a)gmail.com> wrote: > On 6/17/10 8:23 AM, Matteo Landi wrote: > > > > > > > Some weeks ago, here on the mailing list I read about picloud[1], a > > python library used for cloud-computing; I was impressed by its > > simplicity, here is an example: > > >>>> import cloud > >>>> def square(x): > > ... return x * x > >>>> cloud.call(square, 10) > >>>> cloud.result() > > 100 > > > So, I tried to figure out how to achieve the same result, i.e. define a > > local generic function and send it somewhere for remote execution, and > > the get the result back. > > So I thought about serialization (pickle): I made a few tries and it > > seemed to work.. but I was wrong, because I never tried to load pickled > > data from another session different from the one used to pickle data > > itself. If you try and pickle a function, it is not pickled as a whole, > > indeed, once you unpickle it, it will raise an exception telling you > > that the target function was not found in the current module. > > > So I'm here, with nothing in my hands; how would you implement this? > > PiCloud's client library is LGPLed. Go take a look for yourself. It's pretty nifty. > > [Disclosure: My employer, Enthought, has partnered with PiCloud to provide our > Enthought Python Distribution to PiCloud users. However, I say that it's nifty > purely because it is.] > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco
From: Chris Rebert on 17 Jun 2010 21:59
2010/6/17 Andreas Löscher <andreas.loescher(a)s2005.tu-chemnitz.de>: > Am Donnerstag, den 17.06.2010, 18:03 +0200 schrieb Andreas Löscher: >> Am Donnerstag, den 17.06.2010, 08:18 -0700 schrieb Paul Rubin: >> > Matteo Landi <landimatte(a)gmail.com> writes: >> > > I could be wrong, but it seems functions are not marshable objects, is >> > > it right? >> > >> > Hmm, you're right, you can marshal code objects, but you can't marshal a >> > function directly.  It's been a while since I've used marshal and I >> > forgot how it works.  You might be able to concoct something around it, >> > but it could be messy.  It may be simplest to send the other side a >> > python source file that it can import. >> >> There was a similar thread a while ago. >> > > Ah I found it: > Search for: > > "Object serialization: transfer from a to b (non-implemented code on b)" > >  in this mailing list. > > Is there a good way to link a former thread? Link to the python.org archive of python-list or the google groups c.l.p thread. In this case: http://groups.google.com/group/comp.lang.python/browse_thread/thread/6c65564f644e327f/62ef889a6e8ab886 Cheers, Chris -- http://blog.rebertia.com |