Parser generator (+ Lexer) that actually works?
in [TCL]
|
From: tom.rmadilo on 6 Jun 2010 12:14 On Jun 6, 2:01 am, Uwe Klein <uwe_klein_habertw...(a)t-online.de> wrote: > tom.rmadilo wrote: > > On Jun 5, 11:28 am, "Gerald W. Lester" <Gerald.Les...(a)KnG- > > Consulting.net> wrote: > > >>Fredrik Karlsson wrote: > > >>>Hi, > > >>>I've been trying to get hold of a parser generator + lexer combination > >>>that would give me a parser implemented in Tcl, that actually works. > > >>>My experience so far: > > >>>taccle + fickle - just outputs nothing. Even the demos fails to give > >>>any ouput. > >>>Yeti - I get bus error every time I use either the parser or the lexer > >>>demos. > > >>>I am using UTF-8 encoding.. Are there other options out there that I > >>>should try before continuing my current work of rolling my own parser > >>>from scratch? > > >>If you have control over the input language, how about using a safe interp > >>and letting the Tcl parser parse it for you. > > > I don't understand, if the Tcl parser can parse the language, wouldn't > > it be Tcl? > > Sure. Then you really aren't talking about a new language to parse. > > But even so, since the Tcl parser doesn't enforce command > > structure, how does this end up working? There must be a simple > > example of this concept, which would be very helpful if you don't > > really need a flex/bison type generator. > > The basic idea about tcl: > Being a framework for most "domain specific language" instances > you could need, Maybe there is some confusion about how tools like flex/bison or Yeti are used and why. At best the Tcl parser is a tokenizer like flex. Each command basically serves as its own parser. If the language has any recursion, it would be, my guess, a very ugly language to represent as Tcl code. Or maybe not, but a specific example would clear things up...and help others avoid the extra work of using a flex/bison type generator.
From: tom.rmadilo on 6 Jun 2010 12:59 On Jun 6, 8:03 am, Will Duquette <w...(a)wjduquette.com> wrote: > On Jun 5, 9:18 pm, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote: > > > > > > > On Jun 5, 11:28 am, "Gerald W. Lester" <Gerald.Les...(a)KnG- > > > Consulting.net> wrote: > > > Fredrik Karlsson wrote: > > > > Hi, > > > > > I've been trying to get hold of a parser generator + lexer combination > > > > that would give me a parser implemented in Tcl, that actually works.. > > > > > My experience so far: > > > > > taccle + fickle - just outputs nothing. Even the demos fails to give > > > > any ouput. > > > > Yeti - I get bus error every time I use either the parser or the lexer > > > > demos. > > > > > I am using UTF-8 encoding.. Are there other options out there that I > > > > should try before continuing my current work of rolling my own parser > > > > from scratch? > > > > If you have control over the input language, how about using a safe interp > > > and letting the Tcl parser parse it for you. > > > I don't understand, if the Tcl parser can parse the language, wouldn't > > it be Tcl? But even so, since the Tcl parser doesn't enforce command > > structure, how does this end up working? There must be a simple > > example of this concept, which would be very helpful if you don't > > really need a flex/bison type generator. > > The trick is, you define a set of Tcl commands that provide the > functionality you're looking for. These commands do all necessary > error checking. Then, you put these commands in a safe interpreter > that contains only the commands in your little language, along with > any standard Tcl commands you want to provide (like expr). You are describing what I call application development, and the little language is what I would call an API. The safe interp part seems totally optional. BTW, this is the type of development that I 100% agree with. > As you say, the resulting language is Tcl--in just the same sense as > little languages built on Lisp remain Lisp. But for many > applications, that's fine. Which is why Gerald asked whether Fredrik > had control of the input syntax or not. It seems like you would need to be able to reduce your language to a set of directives or instructions without recursive structure. Of course lots of important and boring code development can be reduced to this form...and should be when possible. This is similar to reducing objects (classes) so that they are completely described by their attributes (data objects).
From: Uwe Klein on 6 Jun 2010 13:45 tom.rmadilo wrote: > Then you really aren't talking about a new language to parse. What is a new language? other grammar? other keywords ? other commands ? A programmer I worked with used the complete yacc lex bison whatnot suite to create a very limited scriptable instrument control language that did not deviate enough from a DOS polled programming style ( on a multiuser multitasking unix platform ) to achieve the design targets in an acceptable manner. ( ditched it after a short time ) A year later I wrote a scriptable framework ( one scriptable n path message multiplexer, one scriptable checkpoint oriented parallelized sequencer that plugged, like all other functionalities, into the multiplexer as separate processes) to actually fill the usage requirements of our instrument. (forex commanding antenna movements while the spectrometer got serviced with a data read out, continuing when movement _and_ readout where complete ) had I known tcl at the time ( and "firstextension" expect and the event based programming stuff been implemented yet) I would have been able to achieve my goal in a tenth of the time and a much reduced need to maintain a standalone framework. uwe
From: tom.rmadilo on 6 Jun 2010 18:50 On Jun 6, 10:45 am, Uwe Klein <uwe_klein_habertw...(a)t-online.de> wrote: > tom.rmadilo wrote: > > Then you really aren't talking about a new language to parse. > > What is a new language? > > other grammar? Yes. > other keywords ? No. > other commands ? No. > > A programmer I worked with used the complete yacc lex bison whatnot > suite to create a very limited scriptable instrument control language > that did not deviate enough from a DOS polled programming style ( on a > multiuser multitasking unix platform ) to achieve the design targets > in an acceptable manner. ( ditched it after a short time ) Maybe it is important to identify the best tool for the job prior to writing much code. > A year later I wrote a scriptable framework ( one scriptable n path > message multiplexer, one scriptable checkpoint oriented parallelized > sequencer that plugged, like all other functionalities, into the > multiplexer as separate processes) to actually fill the usage > requirements of our instrument. > (forex commanding antenna movements while the spectrometer got > serviced with a data read out, continuing when movement _and_ readout > where complete ) > > had I known tcl at the time ( and "firstextension" expect and > the event based programming stuff been implemented yet) I would > have been able to achieve my goal in a tenth of the time and a > much reduced need to maintain a standalone framework. Almost all I do is write frameworks, toolkits and code generators. I also have one flex/bison based compiler: it compiles template code into a Tcl script. I actually use the templates to format text, html or even additional programming code. It could also be used to produce binary output. Most template languages are notoriously difficult to debug because they are not compiled and usually are not written by programmers. Would I love to avoid using flex/bison: yes. Would the results be acceptable? Not by a long shot. The embedded language looks like Tcl code, but is slightly constrained. There is also the non- language text mixed all over the template. The compiled template is just Tcl code. When executed in the proper context, you (hopefully) get the intended result. It supports a single, generic extension mechanism which allows you to execute approved code, so a safe interp is not required. But since frameworks seem to be your point, which I agree with: programming problems in particular areas should be reduced to proven methods. Since I hate dealing in the abstract, I'll use some of my own code as an example, and many clt'ers are familiar with WSDL and web services in general. I wrote an API called tWSDL. Each API defines some part of something you would find in a WSDL file. Simple types, complexTypes, type derivation, messages, operations, portTypes, ports, servers and services. If you want to define a simple web service you will chafe at the amount of repetition and internal referencing. However, as an API/ framework developer, taking shortcuts which preclude full (or fully useful) expression of the WSDL file will end up making life difficult for users and future developers. So the tWSDL API is verbose. But the API calls actually return generated code. You could save the result or [eval] the result of each API. Given the annoyingly verbose tWSDL API, I wrote another "configuration language" called TWiST. This language/API uses introspection/ defaulting and argument analysis to select and eval tWSDL API. In general the only way to inspect the running code is by using a package I call "inspect", which also serves as part of the documentation features in TWiST. So to boil this down: TWiST is a configuration language which generates tWSDL API calls which return Tcl code. Execute the result to get an application. I'm sure this is the kind of thing that Tcl'ers love, but there is sometimes the need for a flex/bison "not-so- dynamic" language parser. Tcl is under-appreciated as and IDL/Framework description language. tWSDL/TWiST could just as easily write JAXW stubs, or annotations in any number of hackish web service IDLs. But you still have to solve the problem. Trying to adapt your language to Tcl's parser seems like the last thing on the list of things to do. But if you get lucky....priceless.
From: Uwe Klein on 7 Jun 2010 02:35
tom.rmadilo wrote: > But you still have to solve the problem. Trying to adapt your language > to Tcl's parser seems like the last thing on the list of things to do. > But if you get lucky....priceless. "priceless" would that not move "tcl grammar" to the top of my problem solving "shift, reduce" mechanism ;-? uwe |