Prev: Is This Open To SQL Injection?
Next: Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP
From: Martin v. Loewis on 7 Jul 2010 15:10
> Python 3.1.1, file [pymem.h]:
>
> PyAPI_FUNC(void *) PyMem_Malloc(size_t);
>
> #define PyMem_MALLOC(n) (((n) < 0 || (n) > PY_SSIZE_T_MAX) ? NULL \
> : malloc((n) ? (n) : 1))
>
> The problem with the latter that it seems that it's intended for safety
> but does the opposite...

Why do you say that? It certainly *does* achieve safety, wrt. to certain
errors, specifically:
- passing sizes that are out-of-range
- supporting malloc(0) on all systems


> Perhaps (if it isn't intentional) this is a bug of the oversight type,
> that nobody remembered to update the macro?

Update in what way?

> Except for the problems with file descriptors I think a practical
> interim solution for extensions implemented in C could be to just link
> the runtime lib statically. For a minimal extension this increased the
> size from 8 KiB to 49 KiB. And generally with MS tools the size is
> acceptably small.

If you think that's fine for your extension module (which may well be
the case), go ahead. But then, you could also just link with a different
DLL version of the CRT instead.

> I think that this would be safe because since the C API has to access
> things in the interpreter I think it's a given that all the relevant
> functions delegate to shared library (DLL) implementations, but I have
> not checked the source code.

There are certainly more cases than the ones mentioned so far, in
particular the time zone and the locale. The CRT carries global
variables for these, so if you set them in the copy of the CRT that
Python links with, you won't see the change in your extension module -
which may or may not be a problem.

> As a more longterm solution, perhaps python.org could make available the
> redistributables for various MSVC versions, and then one could introduce
> some scheme for indicating the runtime lib dependencies of any given
> extension.

My preferred long-term solution is to reduce the usage of the C library
in CPython as much as reasonable, atleast on Windows. Memory management
could directly use the heap functions (or even more directly
VirtualAlloc); filenos could be OS handles, and so on. There are
probably limitations to what you can achieve, but I think it's worth trying.

Regards,
Martin
From: Martin v. Loewis on 7 Jul 2010 15:22
> I presume this problem would go away if future versions of Python
> itself were compiled on Windows with something like MinGW gcc. Also,
> this would solve the pain of Python developers attempting to
> redistribute py2exe versions of their programs (i.e. they have to own
> a Visual Studio license to legally be able to redistribute the
> required C runtime) I don't understand enough to know why Visual
> Studio was chosen instead of MinGW. Can anyone shed any light on that
> decision?

sturlamolden has already given the primary reason: Python,
traditionally, attempts to use and work with the system vendor's
compiler. On Windows, that's MSC. It's typically the one that best knows
about platform details that other compilers might be unaware of.

In addition, it's also the compiler and IDE that Windows developers (not
just Python core people, but also extension developers and embedders)
prefer to use, as it has quite good IDE support (in particular debugging
and code browsing).

Perhaps more importantly, none of the other compilers is really an
alternative. GCC in particular cannot build the Win32 extensions, since
it doesn't support the COM and ATL C++ features that they rely on (and
may not support other MSC extensions, either). So the Win32 extensions
must be built with VS, which means Python itself needs to use the same
compiler.

Likewise important: gcc/mingw is *not* a complete C compiler on Windows.
A complete C compiler would have to include a CRT (on Windows); mingw
doesn't (cygwin does, but I think you weren't proposing that Python be
built for cygwin - you can easily get cygwin Python anyway). Instead,
mingw relies on users having a CRT available to
them - and this will be a Microsoft one. So even if gcc was used, we
would have versioning issues with Microsoft CRTs, plus we would have to
rely on target systems including the right CRT, as we couldn't include
it in the distribution.

HTH,
Martin
From: Alf P. Steinbach /Usenet on 7 Jul 2010 15:41
* Martin v. Loewis, on 07.07.2010 21:10:
>> Python 3.1.1, file [pymem.h]:
>>
>> PyAPI_FUNC(void *) PyMem_Malloc(size_t);
>>
>> #define PyMem_MALLOC(n) (((n)< 0 || (n)> PY_SSIZE_T_MAX) ? NULL \
>> : malloc((n) ? (n) : 1))
>>
>> The problem with the latter that it seems that it's intended for safety
>> but does the opposite...
>
> Why do you say that? It certainly *does* achieve safety, wrt. to certain
> errors, specifically:
> - passing sizes that are out-of-range
> - supporting malloc(0) on all systems

It uses malloc instead of PyMem_Malloc. malloc may well be different and use a
different heap in an extension DLL than in the Python interpreter and other
extensions. That's one thing that the docs (rightly) warn you about.


>> Perhaps (if it isn't intentional) this is a bug of the oversight type,
>> that nobody remembered to update the macro?
>
> Update in what way?

I was guessing that at one time there was no PyMem_Malloc. And that it was
introduced to fix Windows-specific problems, but inadvertently without updating
the macro. It's just a guess as to reasons why the macro uses malloc directly.


>> Except for the problems with file descriptors I think a practical
>> interim solution for extensions implemented in C could be to just link
>> the runtime lib statically. For a minimal extension this increased the
>> size from 8 KiB to 49 KiB. And generally with MS tools the size is
>> acceptably small.
>
> If you think that's fine for your extension module (which may well be
> the case), go ahead.

I have no comment on that except pointing it out as a somewhat stupid, somewhat
evil social inclusion/exclusion argument, talking to the audience. Argh. You're
wasting my time. But anyway, 49 KiB is small by today's standards. For example,
you get 20 of those in a single MiB, and about 20.000 in a single GiB.


> But then, you could also just link with a different
> DLL version of the CRT instead.

When I wrote "link the runtime lib statically" that was an alternative to the
usual link-as-DLL.

It wouldn't make sense to link the runtime lib statically as an alternative to
linking it statically.

As for linking to a different /version/ of the CRT, if you really mean that, I
think that's difficult. It's not necessarily impossible, after all there's
STLPort. But I think that it must at the very least be rather difficult to do
with Microsoft's tools, for otherwise people would have employed that solution
before, and so I wouldn't trust the result, and wouldn't waste the time trying.


Cheers,

- Alf

--
blog at <url: http://alfps.wordpress.com>
From: Martin v. Loewis on 7 Jul 2010 15:47

> Also observe that this macro is very badly written (even illegal) C.
> Consider what this would do:
>
> PyMem_MALLOC(n++)
>
> According to Linus Thorvalds using macros like this is not even legal
> C:
>
> http://www.linuxfocus.org/common/src/January2004_linus.html

[Please don't use "legal" wrt. programs - it's not "illegal" to violate
the language's rules; you don't go to jail when doing so. Linus said
"not allowed"]

You are misinterpreting that statement. Linus said that the isdigit
macro was non-conforming, *and meant that specifically for isdigit()*.
That's because the C standard says that isdigit is a function. Under
the as-if rule, you may implement it as a macro as long as nobody can
tell the difference. However, in the presented implementation, there
is a notable difference.

However, the C standard is silent wrt. to PyMem_MALLOC, and it certainly
allows the definition of macros which use the macro arguments more than
once.

> This would be ok, and safe as long as we use the GIL:

The macro is ok as it stands (minus the issues with multiple heaps).
The Python convention is that you clearly recognize PyMem_MALLOC as
a macro, so you should know not to pass parameters with side effects.

> register Py_ssize_t __pymem_malloc_tmp;
> #define PyMem_MALLOC(n)\
> (__pymem_malloc_tmp = n, (((__pymem_malloc_tmp) < 0 ||
> (__pymem_malloc_tmp) > PY_SSIZE_T_MAX) ? NULL \
> : malloc((__pymem_malloc_tmp) ?
> (__pymem_malloc_tmp) : 1)))

That would partially defeat the purpose, namely it would require the
compiler to put the size into a variable in memory, and possibly prevent
optimizations from taking place that rely on constant propagation
(depending on how smart the compiler is).

Regards,
Martin
From: Grant Edwards on 7 Jul 2010 16:22
On 2010-07-07, Martin v. Loewis <martin(a)v.loewis.de> wrote:
>
>> Also observe that this macro is very badly written (even illegal) C.
>> Consider what this would do:
>>
>> PyMem_MALLOC(n++)
>>
>> According to Linus Thorvalds using macros like this is not even legal
>> C:
>>
>> http://www.linuxfocus.org/common/src/January2004_linus.html
>
> [Please don't use "legal" wrt. programs - it's not "illegal" to violate
> the language's rules; you don't go to jail when doing so. Linus said
> "not allowed"]

Nonsense.

The world "illegal" doesn't mean "you'll go to jail".

"Legal" and "illegal" are used to indicate conformance or
nonconformace with respect to some set of rules -- be they a
programming language standard, FIFA footbal rules, or Forumula 1
technical regulations.

It's perfectly standard usage to refer to an "illegal forward pass" in
American football, to "illegal tires" used during a race, or to an
"illegal operation" in a program.

--
Grant Edwards grant.b.edwards Yow! I feel like I'm
at in a Toilet Bowl with a
gmail.com thumbtack in my forehead!!
 |  Next  |  Last
Pages: 1 2
Prev: Is This Open To SQL Injection?
Next: Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP