Easy questions from a python beginner
in [Python]
|
Prev: Why doesn't python's list append() method return the list itself?
Next: Getting started with python on macintosh snow leopard with mysql - need help
From: Stephen Hansen on 23 Jul 2010 00:23 On 7/22/10 7:47 PM, wheres pythonmonks wrote: > Thanks for pointing out that swap (and my swap2) don't work everywhere > -- is there a way to get it to work inside functions? > > "No offense, but you seem like you're still tying to be a hacker. If > that's what you want, fine, but generally speaking (and particularly > for Python), you are going to have a better experience if you do it > the language's way." > > None taken, but I always think that it is the language's job to > express my thoughts... I don't like to think that my thoughts are > somehow constrained by the language. The thing is, every language is adept at expressing different kinds of abstractions. They have different goals and ways of encouraging certain types of solutions while discouraging (and many times simply not even trying to offer support for) others: you're going up against those 'discouraged' very, very hard-- and you're going to hit a wall :) The Python language does constrain your expressiveness in these sorts of areas, and there's no way around it. If that doesn't suit you-- you're free to use another language which works better to how your brain works. Ruby actually might be a very good suggestion. I don't mean this as a 'Deal with it or f- off, loser' statement. :) I mean that Python uses a very simple model in how it handles namespaces, calling and similar -- a lot of what seems completely natural to express in other languages is either slow, convoluted, complicated or flatly impossible here. On purpose. But, for those things that aren't expressed easily in Python, there's almost always a *different way* to express the actual need and intention-- in a way that is elegant and direct, in Python. You can't do it the other way "Pythonic", because "Pythonic" is to *not* do it that way. To not do things like try to treat variables as if they had box-like-behavior. > The truth is that I don't intend to use these approaches in anything > serious. However, I've been known to do some metaprogramming from > time to time. Depending on how you define "metaprogramming", Python is pretty notoriously ill-suited towards the task (more, its basically considered a feature it doesn't let you go there) -- or, it allows you to do vast amounts of stuff with its few dark magic hooks. I've never had a satisfying definition of metaprogramming that more then 50% of any group agree with, so I'm not sure which you're looking for. :) Python allows you to easily override nearly all operations and interactions between objects, construct classes, instances and such at runtime and modify them at any time. If metaprogramming is this to you, basically runtime tweaking and modification of classes types objects and the like, Python's good to go. But! What it doesn't let you do is get clever with syntax and write a sort of "simplified" or domain specific language to achieve certain sorts of repetitive tasks quickly. You always end up writing normal Python that looks like all other Python. You just might have a few more __methods__ and learn some dark magic like metaclasses and descriptors and decorators. (Not that -all- uses of such are dark magic, but when you start combining those your code can start to resemble a magic spell doing unholy things-- like my library with classes that have argument-based method overloading, so x.test("hi") and x.test("hi", 2) call two totally different functions. Not that I'll ever admit to having written the beast, nor that I actually did more then use it for pure proof of concept, and neither shall I willingly ever show anyone the monstrosity). > In a recent application, I pass in a list of callables (lambdas) to be > evaluated repeatedly. > Clearly, a superior solution is to pass a single lambda that returns a > list. [Less function call dispatches] > However, it might be more efficient to avoid the function call > overhead completely and pass-in a string which is substituted into a > string code block, compiled, and executed. This is the kind of "metaprogramming" where Python says nyet, that first kind I mentioned about being Notoriously Ill Suited. You can do things like compile code dynamically, generate up bytecode even, but it'll never be more efficient. Use a generator instead. :) -- Stephen Hansen ... Also: Ixokai ... Mail: me+list/python (AT) ixokai (DOT) io ... Blog: http://meh.ixokai.io/
From: Carl Banks on 23 Jul 2010 02:26 On Jul 22, 7:47 pm, wheres pythonmonks <wherespythonmo...(a)gmail.com> wrote: > Thanks for pointing out that swap (and my swap2) don't work everywhere > -- is there a way to get it to work inside functions? Impossible without hacking the interpreter. > "No offense, but you seem like you're still tying to be a hacker. If > that's what you want, fine, but generally speaking (and particularly > for Python), you are going to have a better experience if you do it > the language's way." > > None taken, but I always think that it is the language's job to > express my thoughts... I don't. In fact whenever I investigate new languages I make an effort to accommodate its thinking, I find I learn a lot more than I would by it into my preconceived thinking. YMMV. > I don't like to think that my thoughts are > somehow constrained by the language. And yet your previous thoughts seem oddly constrained by Perl.... If you really want a language that can accommodate any thought you could possibly have, you should check out Lisp. (defmacro swap (a b) '(let ((t1 ,a) (t2 ,b)) (setq ,b t1) (setq ,a t2))) hygiene-left-as-an-exercise-ly yr's, Carl Banks
From: Steven D'Aprano on 23 Jul 2010 04:10 On Thu, 22 Jul 2010 22:47:11 -0400, wheres pythonmonks wrote: > Thanks for pointing out that swap (and my swap2) don't work everywhere > -- is there a way to get it to work inside functions? Not in CPython. In IronPython or Jython, maybe, I don't know enough about them. But even if you got it to work, it would be an implementation- dependent trick. [...] > I always think that it is the language's job to express > my thoughts... Ha, no, it's the language's job to execute algorithms. If it did so in a way that is similar to the way people think, that would be scary. Have you *seen* the way most people think??? *wink* > I don't like to think that my thoughts are somehow > constrained by the language. Whether you "like" to think that way, or not, thoughts are influenced and constrained by language. While I don't accept the strong form of the Sapir-Whorf hypothesis (that some thoughts are *impossible* due to lack of language to express them, a position which has been discredited), a weaker form is almost certainly correct. Language influences thought. Turing Award winner and APL creator Kenneth E. Iverson gave a lecture about this theme, "Notation as a tool of thought", and argued that more powerful notations aided thinking about computer algorithms. Paul Graham also discusses similar ideas, such as the "blub paradox". Graham argues that the typical programmer is "satisfied with whatever language they happen to use, because it dictates the way they think about programs". We see this all the time, with people trying to write Java in Python, Perl in Python, and Ruby in Python. And Yukihiro Matsumoto has said that one of his inspirations for creating Ruby was the science fiction novel Babel-17, which in turn is based on the Sapir-Whorf Hypothesis. > The truth is that I don't intend to use these approaches in anything > serious. However, I've been known to do some metaprogramming from time > to time. > > In a recent application, I pass in a list of callables (lambdas) to be > evaluated repeatedly. Are you aware that lambdas are just functions? The only differences between a "lambda" and a function created with def is that lambda is syntactically limited to a single expression, and that functions created with lambda are anonymous (they don't have a name, or at least, not a meaningful name). > Clearly, a superior solution is to pass a single lambda that returns a > list. I don't see why you say this is a superior solution, mostly because you haven't explained what the problem is. > [Less function call dispatches] How? You have to generate the list at some point. Whether you do it like this: functions = (sin, cos, tan) data = (2.3, 4.5, 1.2) result = [f(x) for f, x in zip(functions, data)] or like this: result = (lambda x, y, z: return (sin(x), cos(y), tan(z)) )(2.3, 4.5, 1.2) you still end up with the same number of function calls (four). Any execution time will almost certainly be dominated by the work done inside the lambda (sin, cos and tan) rather than the infrastructure. And unless you have profiled your code, you would be surprised as to where the bottlenecks are. Your intuitions from Perl will not guide you well in Python -- it's a different language, and the bottlenecks are different. > However, it might be more > efficient to avoid the function call overhead completely and pass-in a > string which is substituted into a string code block, compiled, and > executed. See, that's *exactly* what I mean about intuitions. No no no no!!! Using exec or eval in Python code is almost certainly a *pessimation*, not an optimization! I expect this will be an order of magnitude slower to parse, compile and execute a string than it is to execute a function. Using exec or friends to avoid the overhead of function calls is like pushing your car to work to avoid the overhead of having to get in and out of the car. But of course, don't take my word for it. Write your code and profile it, see where the bottlenecks are. I might be wrong. -- Steven
From: Steven D'Aprano on 23 Jul 2010 05:05 On Thu, 22 Jul 2010 21:23:05 -0700, Stephen Hansen wrote: > On 7/22/10 7:47 PM, wheres pythonmonks wrote: [...] >> The truth is that I don't intend to use these approaches in anything >> serious. However, I've been known to do some metaprogramming from time >> to time. > > Depending on how you define "metaprogramming", Python is pretty > notoriously ill-suited towards the task (more, its basically considered > a feature it doesn't let you go there) -- or, it allows you to do vast > amounts of stuff with its few dark magic hooks. I've never had a > satisfying definition of metaprogramming that more then 50% of any group > agree with, so I'm not sure which you're looking for. :) I disagree strongly at your characterisation that Python is notorious for being ill-suited towards metaprogramming. I'd say the complete opposite -- what is considered dark and scary metaprogramming tasks in other languages is considered too ordinary to even mention in Python. If you have a class that defines a single member (or attribute in Python terminology) "spam", and you want to add a second "ham" to a specific instance, such a thing is either deep, dark metaprogramming in some languages, if not outright impossible. In Python it is not even noteworthy: instance.ham = "something" # *yawns* Recently there was a thread started by some Java or C++ refugee who was distressed about attribute access in Python, because it made metaprogramming frighteningly easy: http://mail.python.org/pipermail/python-list/2010-June/1248029.html My response at the time was: Python makes metaprogramming *easy*: http://mail.python.org/pipermail/python-list/2010-June/1248053.html I can't imagine how he would have reacted if we had showed him how easy it is to monkey-patch built-in functions... [...] > But! What it doesn't let you do is get clever with syntax and write a > sort of "simplified" or domain specific language to achieve certain > sorts of repetitive tasks quickly. You always end up writing normal > Python that looks like all other Python. Exactly... 90% of the time that you think you want a DSL, Python beat you to it. It is true that other languages (like Lisp, or Forth) allow you to write your own syntax. That's incredibly powerful, but it also has serious costs. If you can define your own syntax, that means every piece of code you look at is potentially a different language. -- Steven
From: wheres pythonmonks on 23 Jul 2010 05:28
Funny... just spent some time with timeit: I wonder why I am passing in strings if the callback overhead is so light.... More funny: it looks like inline (not passed in) lambdas can cause python to be more efficient! >>> import random >>> d = [ (['A','B'][random.randint(0,1)],x,random.gauss(0,1)) for x in xrange(0,1000000) ] >>> def A1(): j = [ lambda t: (t[2]*t[1],t[2]**2+5) for t in d ] >>> def A2(): j = [ (t[2]*t[1],t[2]**2+5) for t in d ] >>> def A3(l): j = [ l(t) for t in d] >>> import timeit >>> timeit.timeit('A1()','from __main__ import A1,d',number=10); 2.2185971572472454 >>> timeit.timeit('A2()','from __main__ import A2,d',number=10); 7.2615454749912942 >>> timeit.timeit('A3(lambda t: (t[2]*t[1],t[2]**2+5))','from __main__ import A3,d',number=10); 9.4334241349350947 So: in-line lambda possible speed improvement. in-line tuple is slow, passed-in callback, slowest yet? Is this possibly right? Hopefully someone can spot the bug? W On Fri, Jul 23, 2010 at 4:10 AM, Steven D'Aprano <steve(a)remove-this-cybersource.com.au> wrote: > On Thu, 22 Jul 2010 22:47:11 -0400, wheres pythonmonks wrote: > >> Thanks for pointing out that swap (and my swap2) don't work everywhere >> -- is there a way to get it to work inside functions? > > Not in CPython. In IronPython or Jython, maybe, I don't know enough about > them. But even if you got it to work, it would be an implementation- > dependent trick. > > [...] >> I always think that it is the language's job to express >> my thoughts... > > Ha, no, it's the language's job to execute algorithms. If it did so in a > way that is similar to the way people think, that would be scary. Have > you *seen* the way most people think??? > > *wink* > > >> I don't like to think that my thoughts are somehow >> constrained by the language. > > > Whether you "like" to think that way, or not, thoughts are influenced and > constrained by language. While I don't accept the strong form of the > Sapir-Whorf hypothesis (that some thoughts are *impossible* due to lack > of language to express them, a position which has been discredited), a > weaker form is almost certainly correct. Language influences thought. > > Turing Award winner and APL creator Kenneth E. Iverson gave a lecture > about this theme, "Notation as a tool of thought", and argued that more > powerful notations aided thinking about computer algorithms. > > Paul Graham also discusses similar ideas, such as the "blub paradox". > Graham argues that the typical programmer is "satisfied with whatever > language they happen to use, because it dictates the way they think about > programs". We see this all the time, with people trying to write Java in > Python, Perl in Python, and Ruby in Python. > > And Yukihiro Matsumoto has said that one of his inspirations for creating > Ruby was the science fiction novel Babel-17, which in turn is based on > the Sapir-Whorf Hypothesis. > > > >> The truth is that I don't intend to use these approaches in anything >> serious. However, I've been known to do some metaprogramming from time >> to time. >> >> In a recent application, I pass in a list of callables (lambdas) to be >> evaluated repeatedly. > > Are you aware that lambdas are just functions? The only differences > between a "lambda" and a function created with def is that lambda is > syntactically limited to a single expression, and that functions created > with lambda are anonymous (they don't have a name, or at least, not a > meaningful name). > > >> Clearly, a superior solution is to pass a single lambda that returns a >> list. > > I don't see why you say this is a superior solution, mostly because you > haven't explained what the problem is. > > >> [Less function call dispatches] > > How? You have to generate the list at some point. Whether you do it like > this: > > functions = (sin, cos, tan) > data = (2.3, 4.5, 1.2) > result = [f(x) for f, x in zip(functions, data)] > > or like this: > > result = (lambda x, y, z: return (sin(x), cos(y), tan(z)) > )(2.3, 4.5, 1.2) > > you still end up with the same number of function calls (four). Any > execution time will almost certainly be dominated by the work done inside > the lambda (sin, cos and tan) rather than the infrastructure. And unless > you have profiled your code, you would be surprised as to where the > bottlenecks are. Your intuitions from Perl will not guide you well in > Python -- it's a different language, and the bottlenecks are different. > > >> However, it might be more >> efficient to avoid the function call overhead completely and pass-in a >> string which is substituted into a string code block, compiled, and >> executed. > > See, that's *exactly* what I mean about intuitions. No no no no!!! Using > exec or eval in Python code is almost certainly a *pessimation*, not an > optimization! I expect this will be an order of magnitude slower to > parse, compile and execute a string than it is to execute a function. > Using exec or friends to avoid the overhead of function calls is like > pushing your car to work to avoid the overhead of having to get in and > out of the car. > > But of course, don't take my word for it. Write your code and profile it, > see where the bottlenecks are. I might be wrong. > > > > -- > Steven > -- > http://mail.python.org/mailman/listinfo/python-list > |