protobuf
in [TCL]
|
Prev: for loop variable scope
Next: Slimming down a tclkit
From: Uwe Klein on 20 Jan 2010 09:57 Flori wrote: > I need Google protocol-buffers <http://code.google.com/p/protobuf/> > for communication in our current project and was disappointed to see > that there seems to be no Tcl language-binding (but for many other > languages there is). > Since the en-/de-coding is simple <http://code.google.com/intl/de/apis/ > protocolbuffers/docs/encoding.html>, I just wanted to do it myself. > But then I was disappointed again, because to correctly implement it > in Tcl I have to read bytes from a channel not chars and there seems > to be no way to accomplish this in Tcl! > > Do I really have to descend to C? > Shouldn't we have a way to read bytes from a channel in plain Tcl? > You've been shown the [ fconfigure -binary ... ] stuff. > Any suggestions? The next step is to use [ binary scan ..] and [ binary format ... ] and you are set to go. uwe
From: slebetman on 20 Jan 2010 11:47 On Jan 20, 9:41 pm, Alexandre Ferrieux <alexandre.ferri...(a)gmail.com> wrote: > On Jan 20, 1:09 pm, Flori <Florian_M...(a)yahoo.de> wrote: > > > > > in Tcl I have to read bytes from a channel not chars and there seems > > to be no way to accomplish this in Tcl! > > I wonder where you got that myth. Google may be biased towards Python, > they are everything but idiots. Disinformation must come form > elsewhere... > > By the way, thanks for the heads-up on protobuf, never heard of it > before. > While the protocol is thoroughly described and rather simple, the > encoding of varints, the zigzag mapping, etc. don't strike me as the > most machine-efficient thing ever. Of course you're saving bytes here > and there, but the need to shuffle bits at every corner costs you > cycles. I don't predict a world domination by protobuf too soon ;-) Every single google search you trigger spawns potentially hundreds of protobuf packets within google's internal network cluster. Every second of every day millions of google searches get triggered. So for google CPU time* is less a bottleneck than network bandwidth. Even with gigabit ethernet or hundred gigabits infiniband it makes sense to cram as much data as possible into each packet when you're operating at google-scale. Then again if you're not google, flickr, facebook or youtube you probably don't need it. * Their megacluster consists of thousands of CPUs. Legend has it that Google's cluster caught fire once destroying 30% of their equipment but the world didn't notice because the rest of the cluster kept on humming.
From: Flori on 20 Jan 2010 12:59 On Jan 20, 1:59 pm, Eric Hassold <hass...(a)evolane.com> wrote: > Le 20/01/2010 13:09, Flori a écrit : > > > I need Google protocol-buffers<http://code.google.com/p/protobuf/> > > for communication in our current project and was disappointed to see > > that there seems to be no Tcl language-binding (but for many other > > languages there is). > > Since the en-/de-coding is simple<http://code.google.com/intl/de/apis/ > > protocolbuffers/docs/encoding.html>, I just wanted to do it myself. > > But then I was disappointed again, because to correctly implement it > > in Tcl I have to read bytes from a channel not chars and there seems > > to be no way to accomplish this in Tcl! > > > Do I really have to descend to C? > > Shouldn't we have a way to read bytes from a channel in plain Tcl? > > Hello, > > Yes, we should...and we have. > > fconfigure $chan -translation binary -encoding binary > set buf [read $chan $nbytes] > > then buf is a byte array. Use [binary] command to parse it, or string > operations, since EIAS, including bytearrays. Thanks for your replies! I was deceived by the manual of 'read channelId numChars' and didn't realize that this means bytes not chars, if I fconfigure the channel to binary. As for the efficiency of protobufs, I guess that they are very efficient, since most of the varints transferred are probably just a single byte and bit-juggling can be done very fast. What makes protobufs very interesting: It is a glue-technology! We use different platforms and programming languages in our system and have to communicate efficiently in an language-independent way. And protobufs can generate stubs, without the need for everyone to write a parser. I didn't find Tcl on <http://code.google.com/p/protobuf/wiki/ ThirdPartyAddOns> and no mention of protobuf in <http://wiki.tcl.tk/>. Therefore I will probably hack a quick and dirty (and slow) solution for my purposes in plain Tcl. If anyone is interested in it, I could share that. In the long run, we should have a proper and fast protobuf-tcl implementation. Thanks again!
From: Gerry Snyder on 20 Jan 2010 15:36 Flori wrote: >> >>> I need Google protocol-buffers<http://code.google.com/p/protobuf/> >>> for communication in our current project.... > > I didn't find Tcl on <http://code.google.com/p/protobuf/wiki/ > ThirdPartyAddOns> > and no mention of protobuf in <http://wiki.tcl.tk/>. > Therefore I will probably hack a quick and dirty (and slow) solution > for my purposes in plain Tcl. > If anyone is interested in it, I could share that. Please wikify it when you have it working. Since many users may try using it to interface with other systems without knowing the protocol themselves, It sounds like a candidate for tcllib, too. Protobuf may not achieve world domination, but it could be a very useful tool to have. And the speed of "plain Tcl" will likely be sufficient for many purposes. Good luck! Gerry
From: tomk on 21 Jan 2010 11:54
On Jan 20, 10:59 am, Flori <Florian_M...(a)yahoo.de> wrote: > On Jan 20, 1:59 pm, Eric Hassold <hass...(a)evolane.com> wrote: > > > > > Le 20/01/2010 13:09, Flori a écrit : > > > > I need Google protocol-buffers<http://code.google.com/p/protobuf/> > > > for communication in our current project and was disappointed to see > > > that there seems to be no Tcl language-binding (but for many other > > > languages there is). > > > Since the en-/de-coding is simple<http://code.google.com/intl/de/apis/ > > > protocolbuffers/docs/encoding.html>, I just wanted to do it myself. > > > But then I was disappointed again, because to correctly implement it > > > in Tcl I have to read bytes from a channel not chars and there seems > > > to be no way to accomplish this in Tcl! > > > > Do I really have to descend to C? > > > Shouldn't we have a way to read bytes from a channel in plain Tcl? > > > Hello, > > > Yes, we should...and we have. > > > fconfigure $chan -translation binary -encoding binary > > set buf [read $chan $nbytes] > > > then buf is a byte array. Use [binary] command to parse it, or string > > operations, since EIAS, including bytearrays. > > Thanks for your replies! > I was deceived by the manual of 'read channelId numChars' and didn't > realize that > this means bytes not chars, if I fconfigure the channel to binary. > > As for the efficiency of protobufs, I guess that they are very > efficient, since > most of the varints transferred are probably just a single byte and > bit-juggling can be done very fast. > > What makes protobufs very interesting: It is a glue-technology! > We use different platforms and programming languages in our system and > have > to communicate efficiently in an language-independent way. > And protobufs can generate stubs, without the need for everyone to > write a parser. > > I didn't find Tcl on <http://code.google.com/p/protobuf/wiki/ > ThirdPartyAddOns> > and no mention of protobuf in <http://wiki.tcl.tk/>. > Therefore I will probably hack a quick and dirty (and slow) solution > for my > purposes in plain Tcl. > If anyone is interested in it, I could share that. > In the long run, we should have a proper and fast protobuf-tcl > implementation. > > Thanks again! I agree with Gerry, this is an excellent candidate for tcllib. Please don't be shy about asking for help to get it added to tcllib when you're done with the code. Good luck with the coding. tomk |