Re: How to get the text corresponding to a bison rule

[Laurence Finston]

On Mon, 1 Dec 2008, sgaurelius wrote:

> 
> I found out how to do this. I have one other problem though. When I debug, it
> doesn;t go where it should go. I mean, if I have a rule L_BRKT expr R_BRKT,
> in the parser.cc there is a switch statement and this rule is one case of
> it. When I set a breakpoint to this case, it never goes there, even if it
> prints the message of this case. Generally, the debug is really not helpful,
> except for the case I do something wrong.

I don't know if it works very well to try to set breakpoints in the code 
that Bison generates.  Sorry if this is obvious, but you are compiling 
it with the `-g' option, aren't you?  Bison's own debugging feature is very 
useful.  You enable it using `%debug'and have to set a
variable `yydebug' to a non-zero value.  I also believe
there is a command line option for Bison equivalent to using `%debug'.  
This is all documented in the manual.  I usually add a command line
option to my own program for setting `yydebug' to 1.

> 
> Also, I wanted to ask, how yylex is constructed? From what I have read,
> yylex can be constructed from the rules I define in the grammar, can't it?

You either write `yylex' yourself or you generate it using lex, Flex,
or some other package.

> In that case, how do I define some extra specific input arguments for it ?
> (ok, I will search more in the manual, but if you know something, that would
> be useful)

Bison calls `yylex' itself, so you can't just redefine its arguments.
The techniques I describe use the facility of passing an argument to
`yyparse'.  This is described under the topic of "generating a
thread-safe parser" in the manual.  If you define `yylex' yourself,
you can just pass the same parameter to `yylex'.  If you use Flex,
you have to pass an object of a certain specific type to `yyparse' and
it's passed on to `yylex'.  If you want to pass information of your
own choosing to the parser and scanner (i.e., `yyparse' and `yylex'),
you must "attach" it to this object.  Presumably, it contains a
pointer to `void' and the function (or macro) `yyset_extra' 
points it at some object.  

I did all this for my most recent project and it did require little fiddling
until I got it right.  I have dug out the declarations and options and
put them at the bottom of this message.
If you want to use locations, that's an additional argument.  I don't
use them in my current project.

> 
> The yytext you said, is being defined in the scanner, right? 

Yes, automatically by Flex.

> So, in that
> case I would have to find the expression [...] for example and tag it as
> matrix and pass the text. 

If you want to.  The normal way would be to pass tokens corresponding
to more basic building blocks of your language back to `yyparse'.
After all, that's what a parser is for.  For simple applications, Flex
is enough.

> But then, how would I get the number tokens ? 

I recommend strongly against doing this, but you could write a rule
in your scanner to find a regular expression like "\[[0-9]+,[0-9]+\]" 
(no guarantees for correctness here) and then use the string
manipulation facilities of C or C++ to find where the numbers start
and end, where the comma is, etc.  If you have nested expressions like
this, it would get complicated and I don't know if it would work.
This kind of thing is what Bison is for. 

> In
> general, can I have a flex rule, so when it finds a specific pattern to give
> info about it, but not return it as a token. For example, when it meets
> [1,2,3], to say that it is a matrix and not just skip it, but to further
> process it and extract number tokens?

Of course, you don't have to return from every scanner rule, but 
see above.  For what it's worth, I don't think this is a good idea.  
If you return tokens from `yylex', you can pass their semantic values
back to `yyparse' along with them.  What you want to do is much easier
with Bison than with Flex.

In my parser, I use these options:

%pure-parser
%parse-param {yyscan_t parameter}
%lex-param {yyscan_t parameter}
%debug

and these declarations:

int 
yylex(YYSTYPE* lvalp, yyscan_t parameter);

int
yywrap(void);

int
yyerror(void* v, char const* s);

In my scanner (Flex input file), I use these options:

%option header-file="scanner.h++"
%option bison-bridge
%option reentrant

This is the declaration of `yyparse':

int 
yyparse(yyscan_t parameter);

This is how I call it:

Scanner_Node scanner_node = new Scanner_Type;

[Set data members of `scanner_node']

yyscan_t parameter;

yylex_init(&parameter);      

yyset_extra(scanner_node, parameter);

yyset_in(fp, parameter);

int status = yyparse(parameter);

yylex_destroy(parameter);


Laurence