Python bindings tutorial
in [Python]
|
Prev: cycling through options
Next: Timer
From: Terry Reedy on 17 Mar 2010 16:12 On 3/17/2010 8:18 AM, Stefan Behnel wrote: > Dave Angel, 17.03.2010 12:14: >> Stefan Behnel wrote: >>> I think the point here is that executable binaries are not supposed to >>> be used as libraries. Libraries are. That's the difference between a >>> DLL and an executable in the first place. To run an executable, >>> execute it. The subprocess module is the tool of choice here. To use a >>> DLL, link against it. >>> >> There's no real reason parts of an exe cannot be exported, same as a >> dll. They are in fact the same structure. And in fact many other files >> in the Windows environment are also the same structure, from fonts to >> ocx's > > So, because you can, you'd also try to link against fonts then, I guess? > > I hope you notice that what you and me said isn't contradictory. But > there's a reason why there are libraries and executables, and there's no > reason you *should* export anything from an executable - that's what > libraries are there for. That's my point. To put it another way, if an executable has functions that could/should be available as 'library' functions, then they can/should be put in a separate library file for use as such and called from a smaller .exe. This is what python itself does. For 3.1 on winxp, python.exe is only 26 Kb, while python31.dll, with all the builtin functions and classes, (in windows/system32) is 2072 KB. > > Besides, nothing guarantees that it's safe to call stuff that an > executable exports. The executable may well require some setup code that > it only executes when it is properly started. Terry Jan Reedy
From: Dave Angel on 18 Mar 2010 19:13 Alf P. Steinbach wrote: > * Dave Angel: >> Stefan Behnel wrote: >>> <div class="moz-text-flowed" style="font-family: >>> -moz-fixed">MikeLisanke(a)gmail.com, 17.03.2010 10:08: >>>> Its interesting you've mentioned the hard work involved in this >>>> interface (binding to an EXE instead of a DLL). A year or more ago I >>>> was looking at interfacing IPMITOOL to python. Do to the problems >>>> incurred with swig/python I switched to a running the process through >>>> its command-line interface. I always felt the problems in interfacing >>>> python to an EXE should be worked on (to minimize them), making the >>>> direct use of an EXE API's a routine task. I understand some of the >>>> problems using an EXE (not running all of its startup code but >>>> enough for its proper operation). Have you found this a recurring >>>> question? Thanks. >>> >>> I think the point here is that executable binaries are not supposed >>> to be used as libraries. Libraries are. That's the difference >>> between a DLL and an executable in the first place. To run an >>> executable, execute it. The subprocess module is the tool of choice >>> here. To use a DLL, link against it. >>> >>> Stefan >>> >> There's no real reason parts of an exe cannot be exported, same as a >> dll. They are in fact the same structure. And in fact many other >> files in the Windows environment are also the same structure, from >> fonts to ocx's >> >> Saying they're "not supposed to be used" is like saying that a python >> module should not have an >> >> if __name__ == "__main__": >> >> section. After all, who could want to both run a file, and import >> the same file?? > > A Windows DLL has defined initialization and cleanup per process and > per thread. > > This means that e.g. static variables can be properly initialized when > you load the DLL in order to use its functions (I'm skipping > discussion of subtle problems, but that's the essence). > > A Windows EXE has (only) a single entry point which is for process > startup. It invokes the EXE's behavior-as-a-program. There is no way > to use it to e.g. initialize static variables in order to use exported > functions. > > Hence Mike Lisanke's idea of "not running all of its startup code but > enough for its proper operation" is generally not possible. > > An EXE can be used as a kind of server, /if/ it is designed for that. > In particular it can be a COM server, allowing access of its > functionality from any COM-enabled binding, which for Python would > mean OLE Automation (COM, OLE, Automation: this is Microsoft > technology, we're talking Windows EXEs here). But a Python binding to > EXEs in general can't, as far as I can see, make assumptions about any > particular kind of server being implemented by the EXE. > > > Cheers & hth., > > - Alf > > I'm not talking about COM servers, which run in a separate process. Only about calling functionality that happens to be encoded in an .EXE file. You're trying to generalize what I said. I never said you could call into any .EXE file, just that calling into one is not infeasible. You're describing using an .EXE written using the Microsoft C compiler and/or libraries, not a Windows .EXE in general. In any case, whenever you link to a module written in a different environment, you'll have restrictions. Chief among them is the use of a compatible memory model, the stack conventions, the allocators, and so on.. Solving the thread issue is probably the easiest one to fix. I'm not recommending it, just refuting the "not supposed to" quote above. DaveA
From: Alf P. Steinbach on 18 Mar 2010 19:56 * Dave Angel: > Alf P. Steinbach wrote: >> * Dave Angel: >>> Stefan Behnel wrote: >>>> <div class="moz-text-flowed" style="font-family: >>>> -moz-fixed">MikeLisanke(a)gmail.com, 17.03.2010 10:08: >>>>> Its interesting you've mentioned the hard work involved in this >>>>> interface (binding to an EXE instead of a DLL). A year or more ago I >>>>> was looking at interfacing IPMITOOL to python. Do to the problems >>>>> incurred with swig/python I switched to a running the process through >>>>> its command-line interface. I always felt the problems in interfacing >>>>> python to an EXE should be worked on (to minimize them), making the >>>>> direct use of an EXE API's a routine task. I understand some of the >>>>> problems using an EXE (not running all of its startup code but >>>>> enough for its proper operation). Have you found this a recurring >>>>> question? Thanks. >>>> >>>> I think the point here is that executable binaries are not supposed >>>> to be used as libraries. Libraries are. That's the difference >>>> between a DLL and an executable in the first place. To run an >>>> executable, execute it. The subprocess module is the tool of choice >>>> here. To use a DLL, link against it. >>>> >>>> Stefan >>>> >>> There's no real reason parts of an exe cannot be exported, same as a >>> dll. They are in fact the same structure. And in fact many other >>> files in the Windows environment are also the same structure, from >>> fonts to ocx's >>> >>> Saying they're "not supposed to be used" is like saying that a python >>> module should not have an >>> >>> if __name__ == "__main__": >>> >>> section. After all, who could want to both run a file, and import >>> the same file?? >> >> A Windows DLL has defined initialization and cleanup per process and >> per thread. >> >> This means that e.g. static variables can be properly initialized when >> you load the DLL in order to use its functions (I'm skipping >> discussion of subtle problems, but that's the essence). >> >> A Windows EXE has (only) a single entry point which is for process >> startup. It invokes the EXE's behavior-as-a-program. There is no way >> to use it to e.g. initialize static variables in order to use exported >> functions. >> >> Hence Mike Lisanke's idea of "not running all of its startup code but >> enough for its proper operation" is generally not possible. >> >> An EXE can be used as a kind of server, /if/ it is designed for that. >> In particular it can be a COM server, allowing access of its >> functionality from any COM-enabled binding, which for Python would >> mean OLE Automation (COM, OLE, Automation: this is Microsoft >> technology, we're talking Windows EXEs here). But a Python binding to >> EXEs in general can't, as far as I can see, make assumptions about any >> particular kind of server being implemented by the EXE. >> > I'm not talking about COM servers, which run in a separate process. > Only about calling functionality that happens to be encoded in an .EXE > file. > > You're trying to generalize what I said. I never said you could call > into any .EXE file, just that calling into one is not infeasible. Well, true. And I'm sorry if my reply sounded as a misrepresentation. I haven't tried this calling-a-function-in-an-exe (since 16-bit Windows, that is!, but that was very different), but I can imagine two such cases: (1) a function f is exported by the EXE, and f doesn't depend on anything but its arguments and does not use any static storage, or (2) a DLL loaded by the EXE calls back into the EXE, which could work because then everything's initialized (however, there are better ways). But these are very special cases where one really has to know what one is doing. As Gabriel Genellina wrote up-thread, "you *could* do that if you work hard enough, but that's not how things are usually done". That said, in Windows the least uncommon reason to load an EXE as a DLL is, IME, to access resource data in the EXE. Tip for that: the module handle, casted to appropriate pointer type, points to the start of the loaded image (great for accessing e.g. version info resource). I think this is still undocumented... > You're describing using an .EXE written using the Microsoft C compiler > and/or libraries, not a Windows .EXE in general. In any case, whenever > you link to a module written in a different environment, you'll have > restrictions. Chief among them is the use of a compatible memory model, > the stack conventions, the allocators, and so on.. Solving the thread > issue is probably the easiest one to fix. > > I'm not recommending it, just refuting the "not supposed to" quote above. Again, I'm sorry if my reply sounded as a misrepresentation. I just tried to make the solution-constraining technical facts available, addressing Mike's "making the direct use of an EXE API's a routine task". I think we're all in (violent?) agreement on this. :-) Cheers, - Alf
From: Tim Roberts on 19 Mar 2010 00:13 Dave Angel <davea(a)ieee.org> wrote: >> >There's no real reason parts of an exe cannot be exported, same as a >dll. They are in fact the same structure. And in fact many other files >in the Windows environment are also the same structure, from fonts to ocx's Well, there IS a fundamental difference. EXEs and DLLs and the like do all have the same format. They all have a "transfer address", where execution begins. That's the key problem. With a DLL, the transfer address goes to a DllMain, where certain DLL initialization is done, preparing the other entry points for use. With an EXE, the transfer address goes to "main". So, when you load an EXE as a DLL, you will be RUNNING the program. That's is usually not what you want. -- Tim Roberts, timr(a)probo.com Providenza & Boekelheide, Inc.
From: Alf P. Steinbach on 19 Mar 2010 03:01
* Tim Roberts: > Dave Angel <davea(a)ieee.org> wrote: >> There's no real reason parts of an exe cannot be exported, same as a >> dll. They are in fact the same structure. And in fact many other files >> in the Windows environment are also the same structure, from fonts to ocx's This is a bit off-topic, but your explanation is incorrect in some key respects, so (no offense) to avoid readers getting an incorrect impression: > Well, there IS a fundamental difference. EXEs and DLLs and the like do all > have the same format. They all have a "transfer address" Commonly called an "entry point". This is the term you need to know about when e.g. linking object files. >, where execution > begins. That's the key problem. With a DLL, the transfer address goes to > a DllMain, where certain DLL initialization is done, preparing the other > entry points for use. Right, modulo terminology: there's only one "entry point" in a PE format file. > With an EXE, the transfer address goes to "main". Sorry, no, the EXE entry point goes to a routine of no arguments. Typically, in C and C++ that's a run time library routine (e.g. with Microsoft's run time library mainCRTStartup or one of its cousins) which in turn calls the C/C++ "main", while in Pascal it runs the main program, so on. Note that the EXE entry point is probably still incorrectly documented as requiring WinMain signature -- it's the most infamous Microsoft documentation screw-up. > So, when you load an EXE as a DLL, you will be RUNNING the program. Sorry, no, that's not what happens. The Windows API LoadLibrary(Ex) knows the difference between an EXE and a DLL. It's documented as handling both sub-formats, and it does. > That's is usually not what you want. If that had happened then it would be a problem, yes. Happily it doesn't happen. I've discussed the real problems else-thread. Cheers & hth., - Alf |