Rethinking remote testing

[Jacob Bachmeyer]

The recent and ongoing discussion over is_remote (and why is that the 
only is* predicate with an underscore in the name?) and its 
less-than-entirely-accurate implementation started me thinking about the 
topic.  What would be a better way to handle remote testing?

We will need to preserve backwards compatibility for testsuites, or at 
least provide for a transition period to move to the new model.  
Ideally, the only changes affecting testsuites should be that existing 
workarounds become unnecessary going forward.  At the very least, the 
GDB workaround Pedro Alves mentioned must not break the new implementation.

Test configuration (global init files) need not remain backwards 
compatible -- a site can update their init files when installing the 
upgraded DejaGnu.  Ideally, "old model" configuration should be 
automatically migrated in memory upon load.


A rough sketch to start a discussion for a future solution:

Add an abstraction layer (that probably will support existing API calls 
-- I *think* DejaGnu already has this) that wraps a logical connection.  
Remote testing uses a "transfer" connection 
(ftp/kermit/tftp/sftp/scp/rcp/rsync/etc.) and a "control" connection 
(telnet/rsh/ssh/kermit/snmp/etc.).  Local testing uses a "local-unix" or 
"local-posix" connection that provides both interfaces, but implements 
"transfer" as no-ops.  Ideally, testsuites should not care whether they 
are running in local or remote mode.

We want to support Canadian cross environments with a separate test 
driver machine (so possibly four different platforms, but DejaGnu itself 
is relatively platform-independent, needing only a POSIX host) and any 
combination of build/host/target for toolchain testing.  Note that 
multiarch machines are likely to be the future:  AMD64 already does this 
today, and RISC-V systems could have all of RV32/RV64/RV128 on the same 
machine, all with varying extensions either by configuration or 
heterogeneous hardware.  Cross toolchains are likely to increase in 
importance.  So we have toolchain "build"/"host"/"target" and a separate 
DejaGnu "driver" in the picture, all of which can be same or different, 
so a testsuite could use up to three remote connections.


This leads to a few scenarios off the top of my head that DejaGnu needs 
to support:

1.  Toolchain:  compilers.  These (notably GCC but possibly others as 
well) can need the full complexity of the system, with possible need to 
build the tested program on one remote system, transfer the executables 
to another remote system, use the executables on "host" to produce test 
programs, transfer the test programs to a third remote system, and run 
the test programs on "target".  The simplifying factor here is than 
compilers are batch programs, so interaction is limited.

2.  Toolchain:  debuggers.  These (probably only GDB, but possibly 
others for specific languages) can be as complex as compilers, but are 
also interactive programs for which the tests can depend on a compiler 
to build test programs on "host" for "target".  The debugger itself also 
runs on "host" (possibly a different "host" from the associated 
compiler!) and connects (by some means independent of DejaGnu) to 
"target" for debugging the test programs.

3.  Unit testing.  These are probably less complex, since there is only 
a need to build (possibly on a remote "build") the unit test programs, 
transfer them to the "host", run them on the "host", and collect the 
results.  Small complexities are introduced when gcov(1) or similar 
tools are used to validate unit test coverage.

4.  General system testing.  These are again similar to unit testing:  
programs are built on "build", transferred, and run on "host".  These 
are to unit tests as debuggers are to compilers:  general system testing 
is likely to be "interactive" rather than specific "run this program".


Debuggers are particularly interesting, since they potentially introduce 
fifth and sixth(!) systems into the picture:  the "build" and "host" 
systems for the associated compiler, which could be different from the 
"build" and "host" systems for the debugger.  This further expands if 
there are multiple associated compilers.  This suggests that we should 
split build/host/target into a hierarchical structure, with plain 
build/host/target as defaults for the "main tool" being tested, but 
allowing "subtools" (like the compiler in the case of a debugger) to 
have their own build/host/target sets.  This could be particularly 
relevant for multi-arch systems, where a compiler notionally built for 
one target may well be capable of producing output either directly for 
or that is compatible with another target.  (I expect RISC-V to be able 
to make extensive use of this eventually.)

The possibility of needing remote builds raises a question of scope:  on 
one hand, it is easy to say that building tested programs is 
out-of-scope for DejaGnu, but on the other hand, DejaGnu already needs 
most of the relevant functionality in order to support remote testing of 
compilers.  The incremental code is small:  a routine that uses the 
already-needed remote connection infrastructure to transfer the sources 
to a build machine, run "configure" and "make", and collect the output 
binaries.


The complexity associated with these changes is probably unmanageable 
with DejaGnu's current design, which is why I advocate for moving the 
DejaGnu core and core libraries out of the global Tcl namespace.  (Doing 
that (backwards compatibly) is on my local TODO list, by the way.)

Eventually, (far future) DejaGnu could offer a "strict" mode, where test 
scripts are run in safe interpreters with access to only the DejaGnu 
API.  If the API is properly designed, a testsuite that runs in "strict" 
mode should be independent of local or remote configuration.  Depending 
on the details of the GDB testsuite, "strict" mode may also require some 
notion of "test groups" -- some test scripts being applicable only on 
certain "boards".

Lastly, it unlikely that this discussion would even be possible if the 
existing DejaGnu, with all its limitations, had not been written.  
Thanks to Rob and Cygnus for laying that groundwork.


-- Jacob