remote_expect gets confused by comments, unlike raw expect

[Pedro Alves]

This is more of a FYI / brain dump than a request for action.

Another remote_expect issue the gdb testsuite hang exposed (see my
previous message to this list, and [1]), is that remote_expect
does not ignore comments like expect does.

If you look at the dejagnu code that implements this underlined
functionality:

      # remote_expect works basically the same as standard expect, but it
      # also takes care of getting the file descriptor from the specified
      # host and also calling the timeout/eof/default section if there is an
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      # error on the expect call.
        ^^^^^^^^^^^^^^^^^^^^^^^^
      #
      proc remote_expect { board timeout args } {

you'll notice that it will pick the last "eof" section as error section,
if there are multiple "eof" sections.  But in GDB, it wasn't picking the
last "eof" section.

The reason is that remote_expect, while parsing the body to select
the default block to execute after an error, is affected by
the comments in the body (since they are also parsed).

Compare this version:

 remote_expect {
   # patterns below apply to any spawn id specified.
   eof {
   }
 }

With this version:

 remote_expect {
   # The patterns below apply to any spawn id specified.
   eof {
   }
 }

In the second version, remote_expect notices the eof block, while in
the first it does not.

Any comment at that same place with an even-numbered of tokens
also works.

Comments in TCL should only appear in places where a command can appear,
and here, remote_expect is parsing a list of options, not commands, so
one might think that it's not unreasonable to not parse comments,
similarly to how this:

  set a_list {
     an_element
     # another_element
  }

results in a list with three elements, not one element.

The fact that comments with an even number of tokens work is just a
coincidence of how remote_expect's little state machine is
implemented, which clearly does not consider comments.

However, raw expect _does_ accept comments in that position.
For example, this works as intended:

  expect {
      # Some comment here.
      -re "match me" {
        puts "got it"
      }
  }

so it's also not unreasonable to expect that they work
correctly with remote_expect too.

I argue that remote_expect should follow raw expect here, and
thus explicitly (and correctly, unlike today) ignore comments.

I tried to come up with a quick hack for this on the gdb side, one
that would strip out comment lines before calling remote_expect, but
I couldn't get it to work 100% correctly.  More details on the thread
I pointed at before [1].


Today I took a look at expect's sources, to see how they do it, and
found that they treat the arguments to expect's code block as commands,
here:

 /* called to execute a command of only one argument - a hack to commands */
 /* to be called with all args surrounded by an outer set of braces */
 /* Returns a list object containing the new set of arguments */
 /* Caller then has to either reinvoke itself, or better, simply replace
  * its current argumnts */
 
 Tcl_Obj*
 exp_eval_with_one_arg(
 ....
     /*
      * Treat the pattern/action block like a series of Tcl commands.
      * For each command, parse the command words, perform substititions
      * on each word, and add the words to an array of values.  We don't
      * actually evaluate the individual commands, just the substitutions.
      */

called from here:

 (gdb) bt
 #0  Tcl_ParseCommand (interp=0x610800, start=0x671060 "\n    # comment\n    
-re \"pattern\" {\n\tputs \"got it\"\n    }\n", numBytes=34, nested=0, 
parsePtr=0x7fffffffcee0) at /home/pedro/src/tcl/src/generic/tclParse.c:321
 #1  0x00007ffff7bb6ac3 in exp_eval_with_one_arg (address@hidden, 
address@hidden, objv=<optimized out>) at ../expect5.45/expect.c:308
 #2  0x00007ffff7bb9221 in Exp_ExpectObjCmd (clientData=0x0, interp=0x610800, 
objc=<optimized out>, objv=<optimized out>) at ../expect5.45/expect.c:2568
 [....]

Since in TCL comments can (only) appear where commands
appear, Tcl_ParseCommand handles/skips comments too.

So I guess that a proper fix for remote_expect would employ a
similar solution, though I don't have a good grasp on how easy it
would be to mimic a minimal version of Tcl_ParseCommand in tcl.

[1] https://sourceware.org/ml/gdb-patches/2019-03/msg00484.html

Thanks,
Pedro Alves