Serializing functions
in [Python]
|
Prev: Could not import external source folder in Pydev
Next: Efficiency/style issues of import <module> vs. from <module>import <name>, ...
From: Matteo Landi on 17 Jun 2010 11:20 On Thu, 2010-06-17 at 07:37 -0700, Paul Rubin wrote: > Matteo Landi <landimatte(a)gmail.com> writes: > > If you try and pickle a function, it is not pickled as a whole, > > indeed, once you unpickle it, it will raise an exception telling you > > that the target function was not found in the current module. > > > > So I'm here, with nothing in my hands; how would you implement this? > > Use marshal rather than pickle. Note that requires both ends to be I could be wrong, but it seems functions are not marshable objects, is it right? > running the same python version. Be aware of the obvious security > issues of running code that came from remote machine, either because the > remote machine itself may be untrusted or compromised, or because the > network between the two machines may be compromised (remote machine is > being spoofed or commnications are being modified). -- Matteo Landi http://www.matteolandi.net
From: Stephen Hansen on 17 Jun 2010 11:31 On 6/17/10 6:23 AM, Matteo Landi wrote: > itself. If you try and pickle a function, it is not pickled as a whole, > indeed, once you unpickle it, it will raise an exception telling you > that the target function was not found in the current module. You can pickle functions-- and classes, instances, and such-- just fine. If you're having specific difficulty with one instance of doing so, show the actual code. >>> def test(one): ... print one + 1 ... >>> import pickle >>> outp = pickle.dumps(test) >>> outp 'c__main__\ntest\np0\n.' >>> fn = pickle.loads(outp) >>> fn(1) 2 Now, complex stuff like classes and functions have to "live" in the same place with pickle, and it does need the source to be available and already imported, but it works fine. If by "pickle a function" you mean, "take an arbitrary function and pickle it into some random destination and have it by itself still be enough to be executable", then no, pickle doesn't do that. To get that, you'd have to distribute the source to the destination and import it before-hand. That or do some crazy-complicated sort of code object marshalling and manipulation on the other side. I have no idea where to even begin in that situation (hoooow do you turn a code object into something you can actually pass arguments to, hmm? I only know how to 'exec' a bare code object) -- Stephen Hansen ... Also: Ixokai ... Mail: me+list/python (AT) ixokai (DOT) io ... Blog: http://meh.ixokai.io/
From: Matteo Landi on 17 Jun 2010 11:48 On Thu, 2010-06-17 at 08:31 -0700, Stephen Hansen wrote: > On 6/17/10 6:23 AM, Matteo Landi wrote: > > itself. If you try and pickle a function, it is not pickled as a whole, > > indeed, once you unpickle it, it will raise an exception telling you > > that the target function was not found in the current module. > > You can pickle functions-- and classes, instances, and such-- just fine. > If you're having specific difficulty with one instance of doing so, show > the actual code. > > >>> def test(one): > ... print one + 1 > ... > >>> import pickle > >>> outp = pickle.dumps(test) > >>> outp > 'c__main__\ntest\np0\n.' > >>> fn = pickle.loads(outp) > >>> fn(1) > 2 > > Now, complex stuff like classes and functions have to "live" in the same > place with pickle, and it does need the source to be available and > already imported, but it works fine. > > If by "pickle a function" you mean, "take an arbitrary function and > pickle it into some random destination and have it by itself still be > enough to be executable", then no, pickle doesn't do that. To get that, > you'd have to distribute the source to the destination and import it > before-hand. This is the problem, and I excuse me if I ain't been clear enough while showing it inside my previous email. > > That or do some crazy-complicated sort of code object marshalling and > manipulation on the other side. I have no idea where to even begin in > that situation (hoooow do you turn a code object into something you can > actually pass arguments to, hmm? I only know how to 'exec' a bare code > object) > So it seems one should go for sending the function and the module with its definition in order to make the remote side able to execute it. I'm curious to see how picloud is able to discover the needed modules, sources etc etc to run the function specified inside picloud.call(); it seems not an easy task. -- Matteo Landi http://www.matteolandi.net
From: Andreas Löscher on 17 Jun 2010 12:03 Am Donnerstag, den 17.06.2010, 08:18 -0700 schrieb Paul Rubin: > Matteo Landi <landimatte(a)gmail.com> writes: > > I could be wrong, but it seems functions are not marshable objects, is > > it right? > > Hmm, you're right, you can marshal code objects, but you can't marshal a > function directly. It's been a while since I've used marshal and I > forgot how it works. You might be able to concoct something around it, > but it could be messy. It may be simplest to send the other side a > python source file that it can import. There was a similar thread a while ago. You can marshal the functions code object and create a new function on the remote side using this code object. >>> import marshal >>> def spam(): pass now marshal the code object >>> s = marshal.dumps(spam.func_code) On the remote side: >>> import marshal >>> import types >>> code = marshal.loads(s) >>> spam = types.FunctionType(code, globals()) This works as long as you are using byte code compatible Versions on both side. The creation of functions has also some optional arguments you must be aware of, in order to get the result you want. Look at help(types.FunctionType) for further information.
From: Andreas Löscher on 17 Jun 2010 12:07
Am Donnerstag, den 17.06.2010, 18:03 +0200 schrieb Andreas Löscher: > Am Donnerstag, den 17.06.2010, 08:18 -0700 schrieb Paul Rubin: > > Matteo Landi <landimatte(a)gmail.com> writes: > > > I could be wrong, but it seems functions are not marshable objects, is > > > it right? > > > > Hmm, you're right, you can marshal code objects, but you can't marshal a > > function directly. It's been a while since I've used marshal and I > > forgot how it works. You might be able to concoct something around it, > > but it could be messy. It may be simplest to send the other side a > > python source file that it can import. > > There was a similar thread a while ago. > Ah I found it: Search for: "Object serialization: transfer from a to b (non-implemented code on b)" in this mailing list. Is there a good way to link a former thread? |