[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card")
|
From: |
Jacob Bachmeyer |
|
Subject: |
PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card") |
|
Date: |
Sat, 29 Dec 2018 23:40:32 -0600 |
|
User-agent: |
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 |
This patch is the long-promised dejagnu-report-card(1) tool, although it
is not yet actually installed under that name, instead being run as
"dejagnu report card" or "dejagnu report-card". All three
implementations are included in this patch. Installing it as
dejagnu-report-card simply requires making a symlink dejagnu-report-card
-> dejagnu; the launcher will handle the rest if called via such a symlink.
While there was some consideration given to eliminating the shell
version, as it lacks some features related to argument handling and
should never actually be used in production (the Awk and Tcl versions
have higher priority, the DejaGnu core requires Tcl for Expect, and
POSIX mandates Awk), I found that its radically different I/O timings
have contributed significantly to the robustness of the testsuite for
this feature. Additionally, while its overhead is higher, (due, among
other things, to invoking grep(1) dozens of times) it seems to be the
least sensitive to the volume of input data. (Its fixed overhead is
very large, but it might actually scale better on certain sufficiently
large testsuites!) Accordingly, it is kept for now and is accessible
using an explicitly undocumented option to dejagnu(1). (Read the
source: there is a comment indicating that that option is for testing
and development, and the testsuite uses it. It is undocumented because
it is neither entirely GNU nor POSIX style. It is a long option, but it
must be the very first argument.)
----
ChangeLog entries:
* Makefile.am (commands_DATA): Add "report-card" scripts.
(dist_man_MANS): Add dejagnu-report-card.1 and split.
(DEJATOOL): Add "report-card" tool.
(TESTSUITE_FILES): Add testsuite for "report-card" tool.
(DISTCLEANFILES): Add files and symlinks left by testsuite.
* commands/report-card.awk: New command script.
* commands/report-card.sh: New command script.
* commands/report-card.tcl: New command script.
* doc/dejagnu.texi (Invoking dejagnu report-card): New node.
* doc/dejagnu-report-card.1: New man page.
* testsuite/lib/bohman_ssd.exp: New file.
* testsuite/lib/multimpl.exp: New file.
* testsuite/lib/report-card.exp: New file.
* testsuite/report-card.all/onetest.exp: New file.
* testsuite/report-card.all/passes.exp: New file.
----
patch:
----
diff --git a/Makefile.am b/Makefile.am
index 098fd92..fbe2a6b 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -26,7 +26,33 @@ EXTRA_DIST = ChangeLog-1992 MAINTAINERS dejagnu runtest \
$(commands_DATA) $(TESTSUITE_FILES) $(TEXINFO_TEX)\
$(CONTRIB)
-DISTCLEANFILES = options-init.exp stats-init.exp
+DISTCLEANFILES = options-init.exp stats-init.exp \
+ testsuite/launcher.all/command/bin/dejagnu \
+ testsuite/launcher.all/command/bin/dejagnu-bar \
+ testsuite/launcher.all/command/bin/dejagnu-bar-baz \
+ testsuite/launcher.all/command/bin/dejagnu-baz \
+ testsuite/launcher.all/command/bin/dejagnu-foo \
+ testsuite/launcher.all/command/share/dejagnu/commands \
+ testsuite/report-card.all/onetest/one-error.sum \
+ testsuite/report-card.all/onetest/one-fail.sum \
+ testsuite/report-card.all/onetest/one-kfail.sum \
+ testsuite/report-card.all/onetest/one-kpass.sum \
+ testsuite/report-card.all/onetest/one-note.sum \
+ testsuite/report-card.all/onetest/one-pass.sum \
+ testsuite/report-card.all/onetest/one-unresolved.sum \
+ testsuite/report-card.all/onetest/one-unsupported.sum \
+ testsuite/report-card.all/onetest/one-untested.sum \
+ testsuite/report-card.all/onetest/one-warning.sum \
+ testsuite/report-card.all/onetest/one-xfail.sum \
+ testsuite/report-card.all/onetest/one-xpass.sum \
+ testsuite/report-card.all/passes/basic-a.sum \
+ testsuite/report-card.all/passes/basic-b.sum \
+ testsuite/report-card.all/passes/kxfail-a.sum \
+ testsuite/report-card.all/passes/kxfail-b.sum \
+ testsuite/report-card.all/passes/kxpass-a.sum \
+ testsuite/report-card.all/passes/kxpass-b.sum \
+ testsuite/report-card.all/passes/unresult-a.sum \
+ testsuite/report-card.all/passes/unresult-b.sum
# Give a reassuring message so that users know the "build" worked.
all-local:
@@ -60,7 +86,10 @@ pkgdata_DATA = \
commandsdir = $(pkgdatadir)/commands
commands_DATA = \
- commands/help.sh
+ commands/help.sh \
+ commands/report-card.awk \
+ commands/report-card.sh \
+ commands/report-card.tcl
configdir = $(pkgdatadir)/config
config_DATA = \
@@ -157,6 +186,8 @@ TESTSUITE_FILES = \
testsuite/launcher.all/help.exp \
testsuite/launcher.all/interp.exp \
testsuite/launcher.all/verbose.exp \
+ testsuite/report-card.all/onetest.exp \
+ testsuite/report-card.all/passes.exp \
testsuite/runtest.libs/topdir/subdir1/subsubdir1/subsubfile1 \
testsuite/runtest.libs/topdir/subdir1/subfile1 \
testsuite/runtest.libs/topdir/subdir1/subfile2 \
@@ -173,14 +204,17 @@ TESTSUITE_FILES = \
testsuite/runtest.main/options/testsuite/null.test/null.exp \
testsuite/runtest.main/stats.exp \
testsuite/runtest.main/stats/testsuite/stat.test/stats-sub.exp \
+ testsuite/lib/bohman_ssd.exp \
testsuite/lib/launcher.exp \
testsuite/lib/libdejagnu.exp \
testsuite/lib/libsup.exp \
+ testsuite/lib/multimpl.exp \
+ testsuite/lib/report-card.exp \
testsuite/lib/runtest.exp \
testsuite/lib/util-defs.exp \
testsuite/libdejagnu/tunit.exp
-DEJATOOL = launcher libdejagnu runtest
+DEJATOOL = launcher libdejagnu report-card runtest
RUNTEST = ${top_srcdir}/runtest
@@ -191,5 +225,8 @@ unit_SOURCES = testsuite/libdejagnu/unit.cc
# Documentation.
TEXINFO_TEX = doc/texinfo.tex
-dist_man_MANS = doc/dejagnu.1 doc/dejagnu-help.1 doc/runtest.1
+dist_man_MANS = doc/dejagnu.1 \
+ doc/dejagnu-help.1 \
+ doc/dejagnu-report-card.1 \
+ doc/runtest.1
info_TEXINFOS = doc/dejagnu.texi
diff --git a/commands/report-card.awk b/commands/report-card.awk
new file mode 100644
index 0000000..2c21e00
--- /dev/null
+++ b/commands/report-card.awk
@@ -0,0 +1,238 @@
+# report-card.awk -- Test summary tool
+# Copyright (C) 2018 Free Software Foundation, Inc.
+#
+# This file is part of DejaGnu.
+#
+# DejaGnu is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# DejaGnu is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with DejaGnu; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA.
+
+# This file was written by Jacob Bachmeyer.
+
+# ##help
+# #Usage: dejagnu report-card [<option>|<tool>|<file>]...
+# #Usage: dejagnu report card [<option>|<tool>|<file>]...
+# # --verbose, -v Emit additional messages
+# ##end
+
+# Arrays storing lists in this program store items in numbered keys, with a
+# count in the "C" key, similar to Awk's ARGV/ARGC.
+
+# The Tools array stores a list of tools in 1..N.
+
+# The Passes array stores a global list of passes seen, a per-tool list of
+# passes seen, and a global index of passes seen if DejaGnu's multipass
+# support is used.
+# Key prefixes:
+# "" -- global list: 1..N; "C"
+# "t", <tool> -- per-tool list: 1..N; "C"
+# Key patterns:
+# "p", <pass> -- count of tools using <pass>
+
+# The Totals array stores counts of test results, indexed by tool and pass.
+# A summarization step adds per-tool, per-pass, and grand totals.
+# Key patterns:
+# "tp", <Tool>, <Pass>, <result>
+# "t", <Tool>, <result>
+# "p", <Pass>, <result>
+# <result>
+
+##
+## Get list of files to scan
+
+BEGIN {
+ Tools["C"] = 1
+ Passes["", "C"] = 1
+ ToolWidth = 0
+ PassWidth = 0
+ Verbose = 0
+ # remove arguments from ARGV
+ for (i = 1; i < ARGC; i++) {
+ if (ARGV[i] ~ /^-/) {
+ if (ARGV[i] ~ /^--?v(erb.*)?$/)
+ Verbose++
+ else if (ARGV[i] == "--")
+ break
+ delete ARGV[i]
+ }
+ }
+ if (ARGV[i] == "--")
+ delete ARGV[i]
+ if (Verbose) print "Verbose level is "Verbose
+ # adjust filenames in ARGV
+ FileCount = 0
+ for (i = 1; i < ARGC; i++) {
+ if (i in ARGV) FileCount++
+ else continue
+ if (ARGV[i] ~ /\.sum$/) continue
+ else if (ARGV[i] ~ /\.log$/) sub(/\.log$/, ".sum", ARGV[i])
+ else if (ARGV[i] ~/\.$/) sub(/\.$/, ".sum", ARGV[i])
+ else ARGV[i] = (ARGV[i]".sum")
+ }
+ if (FileCount == 0) {
+ cmd_ls_files = "ls -1 *.sum"
+ while (cmd_ls_files | getline File) {
+ FileCount++
+ ARGV[ARGC++] = File
+ }
+ close(cmd_ls_files)
+ }
+ if (Verbose > 2) {
+ print "Reading "FileCount" file(s)"
+ for (i = 1; i < ARGC; i++)
+ if (i in ARGV)
+ print " "ARGV[i]
+ }
+}
+
+##
+## Read files and collect data
+
+FNR == 1 {
+ if (Verbose)
+ print "Reading `"FILENAME"' ..."
+ Pass = ""
+ Tool = File = FILENAME
+ sub(/\.sum$/, "", Tool)
+ if (length(Tool) > ToolWidth)
+ ToolWidth = length(Tool)
+ Tools[Tools["C"]++] = Tool
+ Passes["t", Tool, "C"] = 1
+ Passes["t", Tool, 1] = "" # will be overwritten if multipass is used
+}
+
+/^Running pass `[^']*' .../ {
+ Pass = $3
+ sub(/^`/, "", Pass)
+ sub(/'$/, "", Pass)
+ if (("p", Pass) in Passes)
+ Passes["p", Pass]++
+ else {
+ if (length(Pass) > PassWidth)
+ PassWidth = length(Pass)
+ Passes["", Passes["", "C"]++] = Pass
+ Passes["p", Pass] = 1
+ }
+ Passes["t", Tool, Passes["t", Tool, "C"]++] = Pass
+}
+
+$1 ~ /:$/ { sub(/:$/, "", $1); Totals["tp", Tool, Pass, $1]++ }
+
+##
+## Compute totals
+
+END {
+ $0 = ("PASS FAIL KPASS KFAIL XPASS XFAIL UNSUPPORTED UNRESOLVED UNTESTED")
+ for (i = 1; i in Tools; i++)
+ for (j = 1; ("t", Tools[i], j) in Passes; j++)
+ for (k = 1; k <= NF; k++) {
+ Totals[$k] \
+ += Totals["tp", Tools[i], Passes["t", Tools[i], j], $k]
+ Totals["t", Tools[i], $k] \
+ += Totals["tp", Tools[i], Passes["t", Tools[i], j], $k]
+ Totals["p", Passes["t", Tools[i], j], $k] \
+ += Totals["tp", Tools[i], Passes["t", Tools[i], j], $k]
+ }
+}
+
+##
+## Compute total name column width
+
+END {
+ if (Passes["", "C"] > 1)
+ NameWidth = ToolWidth + 3 + PassWidth
+ else
+ NameWidth = ToolWidth
+}
+
+##
+## Emit header
+
+END {
+ printf "%*s __________________________________________________\n", \
+ NameWidth, ""
+ printf "%*s / %6s %6s %6s %6s %6s %6s %6s\n", NameWidth, "", \
+ "PASS", "FAIL", "?PASS", "?FAIL", "UNSUP", "UNRES", "UNTEST"
+ printf "%*s |--------------------------------------------------\n", \
+ NameWidth, ""
+}
+
+##
+## Emit counts
+
+END {
+ for (i = 1; i in Tools; i++) {
+ Tool = Tools[i]
+ for (j = 1; ("t", Tool, j) in Passes; j++) {
+ Pass = Passes["t", Tool, j]
+ if (Passes["t", Tool, "C"] > 1)
+ printf "%*s / %-*s | ", ToolWidth, Tool, PassWidth, Pass
+ else if (Passes["", "C"] > 1)
+ printf "%*s %*s | ", ToolWidth, Tool, PassWidth, ""
+ else
+ printf "%*s | ", NameWidth, Tool
+ # Passes["t", <tool>, 1] is a pass name or a null string if
+ # <tool> did not use multipass.
+ printf " %6d %6d %6d %6d %6d %6d %6d%s%s\n", \
+ Totals["tp", Tool, Pass, "PASS"], \
+ Totals["tp", Tool, Pass, "FAIL"], \
+ Totals["tp", Tool, Pass, "KPASS"] \
+ + Totals["tp", Tool, Pass, "XPASS"], \
+ Totals["tp", Tool, Pass, "KFAIL"] \
+ + Totals["tp", Tool, Pass, "XFAIL"], \
+ Totals["tp", Tool, Pass, "UNSUPPORTED"], \
+ Totals["tp", Tool, Pass, "UNRESOLVED"], \
+ Totals["tp", Tool, Pass, "UNTESTED"], \
+ (Totals["tp", Tool, Pass, "ERROR" ] > 0 ? " !E!" : ""), \
+ (Totals["tp", Tool, Pass, "WARNING"] > 0 ? " !W!" : "")
+ }
+ }
+}
+
+##
+## Emit pass totals
+
+END {
+ if (Passes["", "C"] > 1) {
+ printf "%*s |--------------------------------------------------\n", \
+ NameWidth, ""
+ for (i = 1; ("", i) in Passes; i++)
+ printf "%*s %-*s | %6d %6d %6d %6d %6d %6d %6d\n", \
+ ToolWidth, "", PassWidth, Passes["", i], \
+ Totals["p", Passes["", i], "PASS"], \
+ Totals["p", Passes["", i], "FAIL"], \
+ Totals["p", Passes["", i], "KPASS"] \
+ + Totals["p", Passes["", i], "XPASS"], \
+ Totals["p", Passes["", i], "KFAIL"] \
+ + Totals["p", Passes["", i], "XFAIL"], \
+ Totals["p", Passes["", i], "UNSUPPORTED"], \
+ Totals["p", Passes["", i], "UNRESOLVED"], \
+ Totals["p", Passes["", i], "UNTESTED"]
+ }
+}
+
+##
+## Emit grand totals
+
+END {
+ printf "%*s |--------------------------------------------------\n", \
+ NameWidth, ""
+ printf "%*s | %6d %6d %6d %6d %6d %6d %6d\n", NameWidth, "", \
+ Totals["PASS"], Totals["FAIL"], \
+ Totals["KPASS"] + Totals["XPASS"], Totals["KFAIL"] + Totals["XFAIL"], \
+ Totals["UNSUPPORTED"], Totals["UNRESOLVED"], Totals["UNTESTED"]
+ printf "%*s \\__________________________________________________\n", \
+ NameWidth, ""
+}
+
+#EOF
diff --git a/commands/report-card.sh b/commands/report-card.sh
new file mode 100755
index 0000000..6c09336
--- /dev/null
+++ b/commands/report-card.sh
@@ -0,0 +1,137 @@
+#!/bin/sh
+# shellcheck disable=SC2003
+# report-card.sh -- produce a nice summary of DejaGnu results
+#
+# Copyright (C) 2018 Free Software Foundation, Inc.
+#
+# This file is part of DejaGnu.
+#
+# DejaGnu is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+
+# This file was written by Jacob Bachmeyer.
+
+# ##help
+# #Usage: dejagnu report-card [options...] [<file>...]
+# #Usage: dejagnu report card [options...] [<file>...]
+# # --verbose, -v Emit additional messages
+# ##end
+
+# Note that shellcheck produces quite a few excess warnings, but has also
+# detected actual bugs under those same warnings, so disabling them
+# globally is not appropriate either, unlike SC2003 ("use of expr").
+#
+# Since the GNU Coding Standards say to avoid making the program code ugly
+# to placate static analyzers, shellcheck warnings related to the
+# [fpn]width variables (which can only contain numeric strings) and the
+# uses of "grep -c" on a single file and the "sum_counts" function (both of
+# which can only produce a number in these contexts) are left but ignored.
+#
+# The shellcheck tool also found unquoted uses of the "pass" variable, and
+# those have been quoted to prevent glob expansion if someone actually uses
+# such an odd pass name, but also indicate a limitation of the Bourne shell.
+
+## Get list of files to scan
+
+verbose=0
+for a in "$@"; do
+ case $a in
+ -v|--v|-verb*|--verb*) verbose=$((verbose + 1)); shift ;;
+ --) shift; break ;;
+ *) break ;;
+ esac
+done
+
+if expr "$verbose" \> 0 > /dev/null ; then
+ echo Verbose level is $verbose
+fi
+
+if test x"$1" = x; then
+ set -- *.sum
+fi
+
+## Compute width of filename column
+
+fwidth=$( (for f in "$@"; do \
+ printf '%s' "$f" | sed -e 's/\.[^.]*$//' | wc -c; done) \
+ | sort -nr | sed 1q)
+
+## Get pass names and compute widths of pass column
+
+get_passes () (grep '^Running pass' -- "$@" \
+ | sed -e 's/^.*://' -e 's/Running pass `//' -e "s/' ...//" \
+ | sort | uniq)
+
+pwidth=$( (for p in $(get_passes "$@"); do printf '%s' "$p" | wc -c; done) \
+ | sort -nr | sed 1q)
+
+if test x"$pwidth" = x; then
+ nwidth=$fwidth
+else
+ nwidth=$(expr $fwidth + 3 + $pwidth)
+fi
+
+## Emit header
+
+printf "%$nwidth"'s __________________________________________________\n' ''
+printf "%$nwidth"'s / %6s %6s %6s %6s %6s %6s %6s\n' ' ' \
+ PASS FAIL '?PASS' '?FAIL' UNSUP UNRES UNTEST
+printf "%$nwidth"'s |--------------------------------------------------\n' ''
+
+## Read and emit counts
+
+re_list='PASS FAIL [KX]PASS [KX]FAIL UNSUPPORTED UNRESOLVED UNTESTED'
+
+for file in "$@"; do
+ passes="$(get_passes "${file}")"
+ if test x"$passes" = x; then
+ printf "%$nwidth"'s | ' "$(basename "$file" .sum)"
+ for re in ${re_list}; do
+ printf ' %6s' $(grep -c "^${re}: " -- "${file}")
+ done
+ grep -q '^ERROR:' "${file}" && printf ' !E!'
+ grep -q '^WARNING:' "${file}" && printf ' !W!'
+ echo
+ else
+ for pass in ${passes}; do
+ printf "%$fwidth"'s / %-'"$pwidth"'s | ' \
+ "$(basename "$file" .sum)" "$pass"
+ for re in ${re_list}; do
+ printf ' %6s' $(grep -c "^${re}: ${pass}:" -- "${file}")
+ done
+ grep -q '^ERROR:' "${file}" && printf ' !E!'
+ grep -q '^WARNING:' "${file}" && printf ' !W!'
+ echo
+ done
+ fi
+done
+
+sum_counts () (re="$1"; shift;
+ # shellcheck disable=SC2046
+ # this relies on word splitting to form a valid expr command
+ expr 0 $(grep -c "${re}" -- "$@" | \
+ sed -e '/:/s/^.*:/ + /' -e '/:/!s/^/ + /'))
+
+if test x"$pwidth" != x; then
+ printf \
+ "%$nwidth"'s |--------------------------------------------------\n' ''
+ for pass in $(get_passes "$@"); do
+ printf "%$fwidth"'s %-'"$pwidth"'s | ' '' "$pass"
+ for re in ${re_list}; do
+ printf ' %6s' $(sum_counts "^${re}: ${pass}:" "$@")
+ done
+ echo
+ done
+fi
+
+printf "%$nwidth"'s |--------------------------------------------------\n' ''
+printf "%$nwidth"'s | ' ''
+for re in ${re_list}; do
+ printf ' %6s' $(sum_counts "^${re}: " "$@")
+done
+echo
+printf "%$nwidth"'s \\__________________________________________________\n' ''
+
+#EOF
diff --git a/commands/report-card.tcl b/commands/report-card.tcl
new file mode 100644
index 0000000..309bf92
--- /dev/null
+++ b/commands/report-card.tcl
@@ -0,0 +1,207 @@
+# report-card.tcl -- Test summary tool
+# Copyright (C) 2018 Free Software Foundation, Inc.
+#
+# This file is part of DejaGnu.
+#
+# DejaGnu is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# DejaGnu is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with DejaGnu; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA.
+
+# This file was written by Jacob Bachmeyer.
+
+# ##help
+# #Usage: dejagnu report-card [<option>|<tool>|<file>]...
+# #Usage: dejagnu report card [<option>|<tool>|<file>]...
+# # --verbose, -v Emit additional messages
+# ##end
+
+## Get list of files to scan
+
+set verbose 0
+if { [llength $argv] > 0 } {
+ foreach arg $argv {
+ if { [regexp {^-} $arg] } {
+ if { [regexp {^--?v(?:erb.*)?$} $arg] } { incr verbose }
+ } elseif { [regexp {\.sum$} $arg] } {
+ lappend files $arg
+ } elseif { [regexp {\.log$} $arg] } {
+ lappend files [regsub -- {\.log$} $arg {.sum}]
+ } elseif { [regexp {\.$} $arg] } {
+ lappend files ${arg}sum
+ } else {
+ lappend files ${arg}.sum
+ }
+ }
+}
+if { ![info exists files] } { set files [glob -- *.sum] }
+
+if { $verbose > 0 } { puts [format "Verbose level is %d" $verbose] }
+
+if { $verbose > 2 } {
+ puts [format "Reading %d file(s)" [llength $files]]
+ foreach file $files { puts " ${file}" }
+}
+
+foreach file $files { lappend tools [file rootname $file] }
+set tools [lsort $tools]
+
+## Compute width of filename column
+
+proc compare_by_length {a b} {
+ return [expr [string length $a] - [string length $b]]
+}
+
+set fwidth [string length \
+ [lindex [lsort -command compare_by_length $tools] end]]
+
+## Read files and collect data
+
+array set allballs {
+ PASS 0 FAIL 0 UNSUPPORTED 0
+ KPASS 0 KFAIL 0 UNRESOLVED 0
+ XPASS 0 XFAIL 0 UNTESTED 0
+ NOTE 0 ERROR 0 WARNING 0
+}
+array set totals [array get allballs]
+
+proc read_summary_file { tool } {
+ global allballs allpasses passes totals verbose
+ set pass {}
+ set passes($tool) [list]
+ foreach result [array names allballs] {
+ set totals(t,$tool,$result) 0
+ set totals(tp,$tool,,$result) 0
+ }
+ set chan [open ${tool}.sum]
+ if { $verbose > 0 } { puts [format "Reading `%s' ..." ${tool}.sum] }
+ while { [gets $chan line] >= 0 } {
+ set fields [split $line]
+ if { [regexp {:$} [lindex $fields 0]] } {
+ set result [string range [lindex $fields 0] 0 end-1]
+ if { ![info exists totals($result)] } { continue }
+ incr totals(tp,$tool,$pass,$result)
+ } elseif { [regexp {^Running pass `([^']+)' ...} $line -> pass] } {
+ lappend passes($tool) $pass
+ set allpasses($pass) 1
+ foreach result [array names allballs] {
+ set totals(p,$pass,$result) 0
+ set totals(tp,$tool,$pass,$result) 0
+ }
+ }
+ }
+ close $chan
+}
+foreach tool $tools { read_summary_file $tool }
+
+proc compute_totals {} {
+ global tools allballs allpasses passes totals
+ foreach result [array names allballs] {
+ foreach tool $tools {
+ if { [llength $passes($tool)] > 0 } {
+ foreach pass $passes($tool) {
+ incr totals(t,$tool,$result) $totals(tp,$tool,$pass,$result)
+ incr totals(p,$pass,$result) $totals(tp,$tool,$pass,$result)
+ incr totals($result) $totals(tp,$tool,$pass,$result)
+ }
+ } else {
+ incr totals(t,$tool,$result) $totals(tp,$tool,,$result)
+ incr totals($result) $totals(tp,$tool,,$result)
+ }
+ }
+ }
+}
+compute_totals
+
+## Compute width of passes column
+
+set pwidth [string length \
+ [lindex [lsort -command compare_by_length \
+ [array names allpasses]] end]]
+
+if { [array names allpasses] ne "" } {
+ set nwidth [expr $fwidth + 3 + $pwidth]
+} else {
+ set nwidth $fwidth
+}
+
+## Emit header
+
+puts [format "%*s __________________________________________________" \
+ $nwidth ""]
+puts [format "%*s / %6s %6s %6s %6s %6s %6s %6s" $nwidth "" \
+ PASS FAIL ?PASS ?FAIL UNSUP UNRES UNTEST]
+puts [format "%*s |--------------------------------------------------" \
+ $nwidth ""]
+
+## Emit counts
+
+foreach tool $tools {
+ if { [llength $passes($tool)] > 0 } {
+ foreach pass $passes($tool) {
+ puts [format "%*s / %-*s | %6s %6s %6s %6s %6s %6s %6s%s%s" \
+ $fwidth $tool $pwidth $pass \
+ $totals(tp,$tool,$pass,PASS) \
+ $totals(tp,$tool,$pass,FAIL) \
+ [expr { $totals(tp,$tool,$pass,KPASS)
+ + $totals(tp,$tool,$pass,XPASS) }] \
+ [expr { $totals(tp,$tool,$pass,KFAIL)
+ + $totals(tp,$tool,$pass,XFAIL) }] \
+ $totals(tp,$tool,$pass,UNSUPPORTED) \
+ $totals(tp,$tool,$pass,UNRESOLVED) \
+ $totals(tp,$tool,$pass,UNTESTED) \
+ [expr { $totals(tp,$tool,$pass,ERROR) > 0\
+ ? { !E!} : {} }] \
+ [expr { $totals(tp,$tool,$pass,WARNING) > 0\
+ ? { !W!} : {} }]]
+ }
+ } else {
+ puts [format "%*s | %6s %6s %6s %6s %6s %6s %6s%s%s" $nwidth $tool \
+ $totals(t,$tool,PASS) $totals(t,$tool,FAIL) \
+ [expr { $totals(t,$tool,KPASS) + $totals(t,$tool,XPASS) }] \
+ [expr { $totals(t,$tool,KFAIL) + $totals(t,$tool,XFAIL) }] \
+ $totals(t,$tool,UNSUPPORTED) $totals(t,$tool,UNRESOLVED) \
+ $totals(t,$tool,UNTESTED) \
+ [expr { $totals(t,$tool,ERROR) > 0 ? { !E!} : {} }] \
+ [expr { $totals(t,$tool,WARNING) > 0 ? { !W!} : {} }]]
+ }
+}
+
+## Emit totals
+
+if { [llength [array names allpasses]] > 0 } {
+ puts [format "%*s |--------------------------------------------------" \
+ $nwidth ""]
+}
+
+foreach pass [array names allpasses] {
+ puts [format "%*s %-*s | %6s %6s %6s %6s %6s %6s %6s" \
+ $fwidth "" $pwidth $pass \
+ $totals(p,$pass,PASS) $totals(p,$pass,FAIL) \
+ [expr { $totals(p,$pass,KPASS) + $totals(p,$pass,XPASS) }] \
+ [expr { $totals(p,$pass,KFAIL) + $totals(p,$pass,XFAIL) }] \
+ $totals(p,$pass,UNSUPPORTED) $totals(p,$pass,UNRESOLVED) \
+ $totals(p,$pass,UNTESTED)]
+}
+
+
+puts [format "%*s |--------------------------------------------------" \
+ $nwidth ""]
+puts [format "%*s | %6s %6s %6s %6s %6s %6s %6s" $nwidth "" \
+ $totals(PASS) $totals(FAIL) \
+ [expr { $totals(KPASS) + $totals(XPASS) }] \
+ [expr { $totals(KFAIL) + $totals(XFAIL) }] \
+ $totals(UNSUPPORTED) $totals(UNRESOLVED) $totals(UNTESTED)]
+puts [format "%*s \\__________________________________________________" \
+ $nwidth ""]
+
+#EOF
diff --git a/doc/dejagnu-report-card.1 b/doc/dejagnu-report-card.1
new file mode 100644
index 0000000..34b586e
--- /dev/null
+++ b/doc/dejagnu-report-card.1
@@ -0,0 +1,146 @@
+.\" Copyright (C) 2018 Free Software Foundation, Inc.
+.\" You may distribute this file under the terms of the GNU Free
+.\" Documentation License.
+.Dd December 28, 2018
+.Os GNU
+.Dt DEJAGNU-REPORT-CARD 1 URM
+.Sh NAME
+.Nm dejagnu\ report\ card
+.Nd summarize results from testing multiple tools
+.Sh SYNOPSIS
+.Nm dejagnu\ report\ card
+.Oo Ao Ar option Ac \*(Ba Ao Ar tool Ac \*(Ba Ao Ar file Ac Oc ...
+.Sh DESCRIPTION
+The
+.Nm
+command displays results from testing multiple tools in a tabular format.
+The produced table lists, for each tool (and if multiple passes were run,
+each pass) the number of tests passed, failed, unsupported, unresolved, and
+untested. Tests that are expected to fail are counted in separate columns
+from tests expected to pass, but "known" failures and "expected" failures
+are summarized together. If a test generated warnings or errors, a tag
+.Ql !W!
+or
+.Ql !E!
+is appended at the end of the relevant line.
+.Pp
+Aside from options, the argument list may include tool or file names. The
+.Nm
+command prefers to read DejaGnu summary files and will translate names
accordingly:
+.Bl -tag -width ".Pa *.sum"
+.It Pa *.sum
+Used as-is.
+.It Pa *.log
+Rewritten to
+.Pa *.sum
+with the same stem.
+.It Pa *.
+The string
+.Pa sum
+is appended to select a summary file. This processing is done for
+convenience when using Readline file name completion in a shell, which will
+complete to the dot.
+.It Pa *
+Taken as a tool name;
+.Pa .sum
+is appended.
+.El
+.Sh OPTIONS
+.Bl -tag -width ".Fl v , -verbose"
+.It Fl v , -verbose
+Emit additional output describing the operation of
+.Nm
+itself.
+.El
+.Sh FILES
+The
+.Nm
+command produces its output by reading the summary files produced by
+DejaGnu and counting "PASS", "FAIL", etc.
+.Pp
+If no names are given as arguments, all files matching
+.Pa *.sum
+in the current directory are read.
+.Sh EXAMPLES
+.Ss A simple example from DejaGnu's own testsuite
+.Bd -literal
+$ dejagnu report-card
+\ __________________________________________________
+\ / PASS FAIL ?PASS ?FAIL UNSUP UNRES UNTEST
+\ |--------------------------------------------------
+\ launcher | 52 0 0 0 0 0 0
+libdejagnu | 5 0 0 0 0 0 0
+\ runtest | 135 0 0 0 0 0 0
+\ |--------------------------------------------------
+\ | 192 0 0 0 0 0 0
+\ \\__________________________________________________
+.Ed
+.Pp
+Three tools were tested, with a total of 192 tests, all expected to pass.
+In this example, all tests did pass, so all other columns are zero. The
+.Ql ?PASS
+and
+.Ql ?FAIL
+columns count tests known or expected to fail that either unexpectedly
+passed or failed as expected. The remaining three columns count the
+exceptional results for unsupported tests, unresolved tests and stub tests
+that simply declare themselves untested.
+.Pp
+.ne 16v
+.Ss The same example after tests were added for dejagnu-report-card
+.Bd -literal
+$ dejagnu report card
+\ __________________________________________________
+\ / PASS FAIL ?PASS ?FAIL UNSUP UNRES UNTEST
+\ |--------------------------------------------------
+\ launcher | 52 0 0 0 0 0 0
+\ libdejagnu | 5 0 0 0 0 0 0
+report-card / awk | 36 0 0 0 0 0 0
+report-card / sh | 36 0 0 0 0 0 0
+report-card / tcl | 36 0 0 0 0 0 0
+\ runtest | 135 0 0 0 0 0 0
+\ |--------------------------------------------------
+\ awk | 36 0 0 0 0 0 0
+\ sh | 36 0 0 0 0 0 0
+\ tcl | 36 0 0 0 0 0 0
+\ |--------------------------------------------------
+\ | 300 0 0 0 0 0 0
+\ \\__________________________________________________
+.Ed
+.Pp
+The
+.Ql report-card
+tool has been added, with three passes, one for each implementation. As
+before, all tests passed as expected. The interesting difference from the
+previous example is the use of DejaGnu's multipass testing feature and the
+additional per-pass summary lines added. For this example, only the
+.Ql report-card
+tool uses multipass testing, so each pass total is simply the count of
+tests for
+.Ql report-card
+instead of a distinct total.
+.Pp
+Also note that the command used to invoke
+.Nm
+is slightly different here. The
+.Xr dejagnu 1
+launcher will automatically collect separate arguments until a complete
+command is formed. This allows individual words in a command name to be
+separated with either dashes or spaces on the command line.
+.Sh SEE ALSO
+.Xr dejagnu 1
+.Xr runtest 1
+.Pp
+The full documentation for DejaGnu is maintained as a Texinfo manual. If the
+.Nm info
+program is properly installed at your site, the command
+.Li info dejagnu
+should give you access to the complete manual.
+.Sh AUTHORS
+.An Jacob Bachmeyer
+.Sh BUGS
+The fallback shell version of this command does not support the advanced
+argument processing described in this man page.
+.\" LocalWords: Dt dejagnu URM Nm Ao Oo Oc DejaGnu Xr runtest DejaGnu's Bd Ql
+.\" LocalWords: testsuite UNSUP UNRES UNTEST libdejagnu Readline Ss tcl awk
+.\" LocalWords: ne multipass
diff --git a/doc/dejagnu.texi b/doc/dejagnu.texi
index e07a40a..47c7136 100644
--- a/doc/dejagnu.texi
+++ b/doc/dejagnu.texi
@@ -80,6 +80,7 @@ Running other DejaGnu commands
* Invoking dejagnu:: Command line options for the launcher itself.
* Invoking dejagnu help:: Reading man pages for dejagnu subcommands.
+* Invoking dejagnu report-card:: Summarizing test results from many tools.
Customizing DejaGnu
@@ -1026,7 +1027,8 @@ then runs the requested command.
@menu
* Invoking dejagnu:: Command line options for the launcher itself.
-* Invoking dejagnu help:: Reading man pages for dejagnu subcommands.
+* Invoking dejagnu help:: Reading man pages for dejagnu subcommands.
+* Invoking dejagnu report-card:: Summarizing test results from many tools.
@end menu
@node Invoking dejagnu, Invoking dejagnu help, Running other DejaGnu commands,
Running other DejaGnu commands
@@ -1089,7 +1091,7 @@ invoked command.
All arguments after the command name are passed to the invoked command.
address@hidden Invoking dejagnu help, , Invoking dejagnu, Running other
DejaGnu commands
address@hidden Invoking dejagnu help, Invoking dejagnu report-card, Invoking
dejagnu, Running other DejaGnu commands
@section Invoking @command{dejagnu help}
@cindex dejagnu help, invoking
@@ -1116,6 +1118,48 @@ inner workings of the @command{dejagnu help} command to
be produced.
The @option{--path}, @option{-w}, and @option{-W} options are passed
to @command{man}.
address@hidden Invoking dejagnu report-card, , Invoking dejagnu help, Running
other DejaGnu commands
address@hidden Invoking @command{dejagnu report-card}
address@hidden dejagnu report-card, invoking
address@hidden dejagnu report card, invoking
+
+The @command{dejagnu report-card} tool produces a tabular summary of
+the results from test runs by reading the summary files that DejaGnu
+produces.
+
address@hidden
address@hidden report-card} [<option>|<tool>|<file>]...
address@hidden example
+
+The @option{--verbose} option causes additional output describing the
+inner workings of the @command{dejagnu report-card} command to be produced.
+
+Aside from options, the command may include a list of tools or files.
+Names ending in @samp{.sum} are used as-is. Names ending in
address@hidden are changed to instead refer to the summary file. Names
+ending with a simple dot (@samp{.}) have @samp{sum} appended, for
+convenience when using Readline filename completion in a shell, which
+will complete to the dot, since there are both @samp{.sum} and
address@hidden files produced for each tool tested. Lastly, all other
+names are taken as tool names and @samp{.sum} is appended to refer to
+the summary file produced by DejaGnu.
+
+The relevant summary files are read and an ASCII-art table is
+produced. The table has columns for counts of tests passed, failed,
+unsupported, unresolved, and untested. Tests that are expected to
+pass and tests that are expected to fail are counted in separate
+columns, but known failures (@samp{KFAIL} and @samp{KPASS}) are
+summarized together with expected failures (@samp{XFAIL} and
address@hidden) in two additional columns: @samp{?PASS} and
address@hidden Additionally, if a test produced any warnings or
+errors, tags @samp{!W!} or @samp{!E!} are added at the end of the row.
+
+There is also a shell-script version that is used if neither Tcl nor
+Awk is available. This version has two major limitations: it accepts
+only the @option{--verbose} option and accepts only the names of
+summary files. Its use should not occur on the GNU system, or any
+POSIX system, or indeed any system capable of running DejaGnu.
+
@node Customizing DejaGnu, Extending DejaGnu, Running other DejaGnu commands,
Top
@chapter Customizing DejaGnu
@cindex customization
@@ -5610,4 +5654,5 @@ This makes @code{runtest} exit. Abbreviation: @kbd{q}.
@bye
@c LocalWords: subdirectory prepend prepended testsuite filename Expect's svn
address@hidden LocalWords: DejaGnu CVS RCS SCCS prepending subcommands
address@hidden LocalWords: DejaGnu CVS RCS SCCS prepending subcommands Tcl
Awk Readline
address@hidden LocalWords: POSIX KFAIL KPASS XFAIL XPASS
diff --git a/testsuite/lib/bohman_ssd.exp b/testsuite/lib/bohman_ssd.exp
new file mode 100644
index 0000000..25b1072
--- /dev/null
+++ b/testsuite/lib/bohman_ssd.exp
@@ -0,0 +1,225 @@
+# Copyright (C) 2018 Free Software Foundation, Inc.
+#
+# This file is part of DejaGnu.
+#
+# DejaGnu is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# DejaGnu is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with DejaGnu; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA.
+
+# This file was written by Jacob Bachmeyer.
+
+# This library provides functions for generating subset-sum-distinct sets
+# using a construction published by Tom Bohman in:
+# T. Bohman, A construction for sets of integers with distinct subset sums,
+# The Electronic. Journal of Combinatorics 5 (1998) /#R3
+# <URL:http://www.combinatorics.org/Volume_5/PDF/v5i1r3.pdf>,
+# retrieved 2018-12-28 SHA-1 1c35035427b3406a44f7290f13ec8fbc3d105041
+namespace eval ::math_utils::Bohman_SSD {
+
+ # b_n(i)
+ proc b { n i } {
+ if { $n <= 1 } { error "invalid parameter n: $n" }
+ if { $i <= 2*$n } { error "invalid parameter i: $i" }
+
+ if { $i >= 2*$n + 4 } {
+ return [expr { round(sqrt(2*($i + 2 - 2*$n))) }]
+ } elseif { $i == 2*$n + 3 } {
+ return [expr { $n + 2 }]
+ } else { # $i == 2*$n + 1 || $i == 2*$n + 2
+ return [expr { $n + 1 }]
+ }
+ }
+
+ variable d_memo
+ array unset d_memo
+ array set d_memo {}
+
+ # d_n(i)
+ proc d { n i } {
+ variable d_memo
+ if { [info exists d_memo($n,$i)] } { return $d_memo($n,$i) }
+
+ if { $n <= 1 } { error "invalid parameter n: $n" }
+ if { $i < 1 } { error "invalid parameter i: $i" }
+
+ if { $i == $n } {
+ return 1
+ } elseif { $i < $n } {
+ set j [expr { $n - $i }]
+ return [expr { 2 * round(pow(4,($j - 1))) }]
+ } elseif { $i <= 2*$n } {
+ set j [expr { $i - $n }]
+ return [expr { round(pow(4,($j - 1))) }]
+ } else { # $i > 2*$n
+ set sum 0
+ for { set j [expr { $i - [b $n $i] }] } { $j < $i } { incr j } {
+ incr sum [d $n $j]
+ }
+ set d_memo($n,$i) $sum
+ return $sum
+ }
+ }
+
+ # S_{n,m} returns list
+ proc S { n m } {
+ if { $n <= 1 } { error "invalid parameter n: $n" }
+ if { $m < 2*$n } { error "invalid parameter m: $m" }
+
+ set dv [list]
+ for { set i 1 } { $i <= $m } { incr i } { lappend dv [d $n $i] }
+ set sum 0
+ foreach d $dv { incr sum $d }
+ set result [list]
+ foreach d $dv {
+ lappend result $sum
+ incr sum -$d
+ }
+ return $result
+ }
+
+ # b'_n(i)
+ proc bp { n i } {
+ if { $n < 1 } { error "invalid parameter n: $n" }
+ if { $i <= 2*$n + 1 } { error "invalid parameter i: $i" }
+
+ if { $i >= 2*$n + 5 } {
+ return [expr { round(sqrt(2*($i + 1 - 2*$n))) }]
+ } elseif { $i == 2*$n + 2 } {
+ return [expr { $n + 1 }]
+ } else { # $i == 2*$n + 3 || $i == 2*$n + 4
+ return [expr { $n + 2 }]
+ }
+ }
+
+ variable dp_memo
+ array unset dp_memo
+ array set dp_memo {}
+
+ # d'_n(i)
+ proc dp { n i } {
+ variable dp_memo
+ if { [info exists dp_memo($n,$i)] } { return $dp_memo($n,$i) }
+
+ if { $n < 1 } { error "invalid parameter n: $n" }
+ if { $i < 1 } { error "invalid parameter i: $i" }
+
+ if { $i == $n + 1 } {
+ return 1
+ } elseif { $i < $n + 1 } {
+ set j [expr { $n + 1 - $i }]
+ return [expr { round(pow(4,($j - 1))) }]
+ } elseif { $i <= 2*$n + 1 } {
+ set j [expr { $i - $n - 1 }]
+ return [expr { 2 * round(pow(4,($j - 1))) }]
+ } else { # $i > 2*$n + 1
+ set sum 0
+ for { set j [expr { $i - [bp $n $i] }] } { $j < $i } { incr j } {
+ incr sum [dp $n $j]
+ }
+ set dp_memo($n,$i) $sum
+ return $sum
+ }
+ }
+ # The example for d'_3 in the paper is wrong starting at i=11. The
+ # paper says that it is 200, but it is actually 300.
+
+ # S'_{n,m} returns list
+ proc Sp { n m } {
+ if { $n < 1 } { error "invalid parameter n: $n" }
+ if { $m < 2*$n + 1 } { error "invalid parameter m: $m" }
+
+ set dv [list]
+ for { set i 1 } { $i <= $m } { incr i } { lappend dv [dp $n $i] }
+ set sum 0
+ foreach d $dv { incr sum $d }
+ set result [list]
+ foreach d $dv {
+ lappend result $sum
+ incr sum -$d
+ }
+ return $result
+ }
+
+ # Given a list of numbers, verify that all sums of all subsets are in
+ # fact unique.
+ #
+ # This is a brute force search and not based on Bohman's paper. This
+ # quickly becomes impractical for large lists, requiring inordinate
+ # amounts of both time and space.
+ proc check { base } {
+ set bound [expr { int(pow(2,[llength $base])) }]
+ for { set i 0 } { $i < $bound } { incr i } {
+ set R $i
+ set sum 0
+ foreach v $base {
+ if { $R & 1 } { incr sum $v }
+ set R [expr { $R >> 1 }]
+ }
+ if { [info exists output($sum)] } {
+ # emit counterexample
+ set cexl [list]
+ set R $i
+ foreach v $base {
+ if { $R & 1 } { lappend cexl $v }
+ set R [expr { $R >> 1 }]
+ }
+ set cex [join $cexl "+"]
+ append cex "=" $sum "="
+ set cexl [list]
+ set R $output($sum)
+ foreach v $base {
+ if { $R & 1 } { lappend cexl $v }
+ set R [expr { $R >> 1 }]
+ }
+ append cex [join $cexl "+"]
+ error "list is not subset-sum-distinct: $cex"
+ }
+ set output($sum) $i
+ }
+ return 1
+ }
+
+ # Given a list of numbers and a sum of a subset of that list, find a
+ # subset that produces the given sum. If the list of numbers is
+ # subset-sum-distinct, this will return the unique solution.
+ # Otherwise, an unspecified solution is returned. If the sum is not
+ # actually a sum of a subset of the list, an empty list is returned.
+ #
+ # This is a brute force search and not based on Bohman's paper. This
+ # requires constant space, but quickly becomes impractical for large
+ # lists, requiring inordinate time to complete.
+ proc summands { base goal } {
+ set bound [expr { int(pow(2,[llength $base])) }]
+ for { set i 0 } { $i < $bound } { incr i } {
+ set R $i
+ set sum 0
+ foreach v $base {
+ if { $R & 1 } { incr sum $v }
+ set R [expr { $R >> 1 }]
+ }
+ if { $sum == $goal } {
+ set resl [list]
+ set R $i
+ foreach v $base {
+ if { $R & 1 } { lappend resl $v }
+ set R [expr { $R >> 1 }]
+ }
+ return $resl
+ }
+ }
+ return [list]
+ }
+
+}
+
+#EOF
diff --git a/testsuite/lib/multimpl.exp b/testsuite/lib/multimpl.exp
new file mode 100644
index 0000000..1130a32
--- /dev/null
+++ b/testsuite/lib/multimpl.exp
@@ -0,0 +1,72 @@
+# Copyright (C) 2018 Free Software Foundation, Inc.
+#
+# This file is part of DejaGnu.
+#
+# DejaGnu is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# DejaGnu is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with DejaGnu; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA.
+
+# This file was written by Jacob Bachmeyer.
+
+# Ensure that the dejagnu(1) launcher is available for testing.
+if { ![info exists LAUNCHER] } {
+ set LAUNCHER \
+ [file join [file dirname [testsuite file -source -top]] dejagnu]
+}
+verbose "Using LAUNCHER $LAUNCHER" 2
+
+if { [which $LAUNCHER] == 0 } {
+ perror "Can't find LAUNCHER = $LAUNCHER"
+ exit 2
+}
+
+# Verify that the requested implementation is available.
+proc probe_implementation { implementation_ext } {
+ global errorCode
+
+ # reset errorCode
+ catch { error }
+
+ catch { exec $LAUNCHER --DGTimpl $implementation_ext } output
+
+ if { [lindex $errorCode 0] eq "CHILDSTATUS" } {
+ if { [lindex $errorCode 2] == 1 } {
+ # we do not have it
+ return 0
+ } else {
+ error "Error probing availability of '$implementation_ext'"
+ }
+ } else {
+ # we have it
+ return 1
+ }
+}
+
+# Configure multipass testing for a dejagnu(1) subcommand that has multiple
+# implementations. The implementation will be the pass name and $TOOL_IMPL
+# will be set for each implementation.
+proc init_multiple_implementation_tests { tool_name } {
+ global MULTIPASS
+
+ foreach implementation\
+ [glob -tails -directory\
+ [file join [file dirname [testsuite file -source -top]] commands]\
+ ${tool_name}.*] {
+ regsub "^${tool_name}\\." $implementation "" mode
+ if { [probe_implementation $mode] } {
+ lappend MULTIPASS [list $mode TOOL_IMPL=$mode]
+ }
+ }
+}
+
+#EOF
diff --git a/testsuite/lib/report-card.exp b/testsuite/lib/report-card.exp
new file mode 100644
index 0000000..27df6f6
--- /dev/null
+++ b/testsuite/lib/report-card.exp
@@ -0,0 +1,31 @@
+# Copyright (C) 2018 Free Software Foundation, Inc.
+#
+# This file is part of DejaGnu.
+#
+# DejaGnu is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# DejaGnu is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with DejaGnu; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA.
+
+# This file was written by Jacob Bachmeyer.
+
+load_lib multimpl.exp
+
+init_multiple_implementation_tests report-card
+
+# stub: dejagnu-report-card is non-interactive
+proc report-card_exit {} {}
+
+# stub: dejagnu-report-card does not have a separate version number
+proc report-card_version {} {}
+
+#EOF
diff --git a/testsuite/report-card.all/onetest.exp
b/testsuite/report-card.all/onetest.exp
new file mode 100644
index 0000000..383cfb8
--- /dev/null
+++ b/testsuite/report-card.all/onetest.exp
@@ -0,0 +1,209 @@
+# Copyright (C) 2018 Free Software Foundation, Inc.
+#
+# This file is part of DejaGnu.
+#
+# DejaGnu is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# DejaGnu is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with DejaGnu; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA.
+
+# This file was written by Jacob Bachmeyer.
+
+set header_column_names { PASS FAIL ?PASS ?FAIL UNSUP UNRES UNTEST }
+set separator_count 0
+set re_digit_columns {}
+for { set i 0 } { $i < 7 } { incr i } {
+ append re_digit_columns {[[:space:]]+([[:digit:]]+)}
+}
+
+set test_names { pass fail kpass kfail xpass xfail
+ unsupported unresolved untested
+ note warning error }
+set test_results { PASS FAIL KPASS KFAIL XPASS XFAIL
+ UNSUPPORTED UNRESOLVED UNTESTED
+ NOTE WARNING ERROR }
+
+foreach name $test_names result $test_results {
+ set fd [open [testsuite file -object -test onetest one-${name}.sum] w]
+ puts $fd "${result}: one test"
+ close $fd
+}
+
+set stty_init { -onlcr -onlret }
+
+spawn /bin/sh -c \
+ "cd [testsuite file -object -test onetest]\
+ && exec $LAUNCHER --DGTimpl $TOOL_IMPL report-card"
+
+# check header
+expect {
+ -re {^[[:space:]]+_+[\r\n]+} {
+ # discard initial header line
+ exp_continue
+ }
+ -re {^[[:space:]]+/([^\r\n]*)[\r\n]+} {
+ # check column labels
+ foreach want $header_column_names have $expect_out(1,string) {
+ if { $have eq $want } {
+ pass "header item $want"
+ } else {
+ fail "header item $want"
+ }
+ }
+ exp_continue
+ }
+ -re {^[[:space:]]+\|-+[\r\n]+} {
+ incr separator_count
+ }
+}
+
+# check results
+array unset scoreboard
+array set scoreboard {
+ pass 0 fail 0 kpass 0 kfail 0 xpass 0 xfail 0
+ unsupported 0 unresolved 0 untested 0
+ note 0 warning 0 error 0
+}
+array unset column_subexp_map
+array set column_subexp_map {
+ pass 2 fail 3 kpass 4 kfail 5 xpass 4 xfail 5
+ unsupported 6 unresolved 7 untested 8
+ note 0 warning 9 error 9
+}
+set re_table_row {^[[:space:]]*one-([[:alpha:]]+)[[:space:]]+\|}
+append re_table_row $re_digit_columns
+append re_table_row {((?:[[:space:]]+![EW]!)*)[\r\n]+}
+expect {
+ -re $re_table_row {
+ for { set i 2 } { $i < 9 } { incr i } {
+ if { $expect_out($i,string)\
+ == ( $i == $column_subexp_map($expect_out(1,string))\
+ ? 1 : 0 ) } {
+ incr scoreboard($expect_out(1,string))
+ } else {
+ incr scoreboard($expect_out(1,string)) -1
+ }
+ }
+ set have_warning_tag [string match "*!W!*" $expect_out(9,string)]
+ set have_error_tag [string match "*!E!*" $expect_out(9,string)]
+ if { $column_subexp_map($expect_out(1,string)) == 9 } {
+ # testing an after-row tag
+ switch -- $expect_out(1,string) {
+ warning {
+ incr scoreboard(warning) \
+ [expr { $have_warning_tag ? 1 : -1 }]
+ incr scoreboard(error) \
+ [expr { $have_error_tag ? -1 : 1 }]
+ }
+ error {
+ incr scoreboard(warning) \
+ [expr { $have_warning_tag ? -1 : 1 }]
+ incr scoreboard(error) \
+ [expr { $have_error_tag ? 1 : -1 }]
+ }
+ default { error "unknown tag $expect_out(1,string)" }
+ }
+ } else {
+ incr scoreboard(warning) [expr { $have_warning_tag ? -1 : 1 }]
+ incr scoreboard(error) [expr { $have_error_tag ? -1 : 1 }]
+ }
+ exp_continue
+ }
+ -re {^[[:space:]]+\|-+[\r\n]+} {
+ incr separator_count
+ }
+}
+foreach result [lsort [array names scoreboard]] {
+ verbose -log "scoreboard($result) = $scoreboard($result)"
+}
+foreach result [array names scoreboard] {
+ if { $scoreboard($result) == ( 7 + ( $column_subexp_map($result) == 9\
+ ? [llength $test_names] : 0 ) ) } {
+ pass "count result $result"
+ } else {
+ fail "count result $result"
+ }
+}
+
+# check totals
+set column_totals { pad 1 1 2 2 1 1 1 }
+set re_totals_row {^[[:space:]]+\|}
+append re_totals_row $re_digit_columns
+append re_totals_row {[\r\n]+}
+set totals_matched 0
+expect {
+ -re $re_totals_row {
+ for { set i 1 } { $i < 8 } { incr i } {
+ if { [lindex $column_totals $i] == $expect_out($i,string) } {
+ incr totals_matched
+ }
+ }
+ exp_continue
+ }
+ -re {^[[:space:]]+\|-+[\r\n]+} {
+ incr separator_count
+ }
+ -re {^[[:space:]]+\\_+[\r\n]+} {
+ # all done
+ }
+}
+
+if { $totals_matched == 7 } {
+ pass "expected total count"
+} else {
+ fail "expected total count"
+}
+
+if { $separator_count == 2 } {
+ pass "expected separator lines"
+} else {
+ fail "expected separator lines"
+}
+
+# Ensure that totals map correctly by reading each file one at a time
+foreach name $test_names {
+ set separator_count 0
+ spawn /bin/sh -c \
+ "cd [testsuite file -object -test onetest]\
+ && exec $LAUNCHER --DGTimpl $TOOL_IMPL report-card one-${name}.sum"
+ # skip header
+ expect {
+ -re {^[[:space:]]+_+[\r\n]+} { exp_continue }
+ -re {^[[:space:]]+/([^\r\n]*)[\r\n]+} { exp_continue }
+ -re {^[[:space:]]+\|-+[\r\n]+} { incr separator_count }
+ }
+ # capture the item line
+ expect -re {^one-[^|]+(\|[[:space:][:digit:]]*)[[:space:]!EW]*[\r\n]+} {
+ regsub {[[:space:]]*$} $expect_out(1,string) "" item_line
+ }
+ # skip the separator
+ expect -re {^[[:space:]]+\|-+[\r\n]+} { incr separator_count }
+ # capture the totals line
+ expect -re {^[[:space:]]+(\|[[:space:][:digit:]]*)[\r\n]+} {
+ regsub {[[:space:]]*$} $expect_out(1,string) "" totals_line
+ }
+ # skip the footer
+ expect -re {.+} { exp_continue }
+ # do the item and totals lines match?
+ if { $item_line eq $totals_line } {
+ pass "verify total for $name"
+ } else {
+ fail "verify total for $name"
+ }
+ if { $separator_count == 2 } {
+ pass "expected separator lines for $name"
+ } else {
+ fail "expected separator lines for $name"
+ }
+}
+
+#EOF
diff --git a/testsuite/report-card.all/passes.exp
b/testsuite/report-card.all/passes.exp
new file mode 100644
index 0000000..1ca8dc7
--- /dev/null
+++ b/testsuite/report-card.all/passes.exp
@@ -0,0 +1,276 @@
+# Copyright (C) 2018 Free Software Foundation, Inc.
+#
+# This file is part of DejaGnu.
+#
+# DejaGnu is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# DejaGnu is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with DejaGnu; if not, write to the Free Software Foundation,
+# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA.
+
+# This file was written by Jacob Bachmeyer.
+
+load_lib bohman_ssd.exp
+
+set header_column_names { PASS FAIL ?PASS ?FAIL UNSUP UNRES UNTEST }
+set result_column_map {
+ PASS FAIL { KPASS XPASS } { KFAIL XFAIL }
+ UNSUPPORTED UNRESOLVED UNTESTED
+}
+
+set test_results { PASS FAIL KPASS KFAIL XPASS XFAIL
+ UNSUPPORTED UNRESOLVED UNTESTED }
+
+# each entry: { {mode n} { suffix_tag... } { pass... } { { result name }... } }
+array unset tuplemap
+array set tuplemap {
+ basic { {S 3} { a b } { foo bar }
+ { { PASS pass } { FAIL fail } } }
+ kxpass { {S 2} { a b } { foo bar }
+ { { KPASS kpass } { XPASS xpass } } }
+ kxfail { {Sp 2} { a b } { foo bar }
+ { { KFAIL kfail } { XFAIL xfail } } }
+ unresult { {S 2} { a b } { foo bar }
+ { { UNSUPPORTED unsupported }
+ { UNRESOLVED unresolved } { UNTESTED untested } } }
+}
+
+# Given: TUPLES: { { result ... }... }, PASSES: { pass... }
+# Return: Cartesian product TUPLES x PASSES: { { result pass ... }... }
+proc build_tuple_list { tuples passes } {
+ set result [list]
+ foreach cell $tuples {
+ foreach pass $passes {
+ lappend result [linsert $cell 1 $pass]
+ }
+ }
+ return $result
+}
+
+# Given: TUPLES: { { result pass name }... }, MODE: S | Sp, N
+# Return: { { result pass name count }... } where COUNT is from an SSD-set
+proc annotate_tuple_list { tuples mode n } {
+ set m [llength $tuples]
+ set ssd [switch -- $mode {
+ S { ::math_utils::Bohman_SSD::S $n $m }
+ Sp { ::math_utils::Bohman_SSD::Sp $n $m }
+ }]
+ set result [list]
+ foreach cell $tuples ssdterm $ssd {
+ lappend result [linsert $cell end $ssdterm]
+ }
+ return $result
+}
+
+# Given: TUPLES: { { result pass name count }... }; (RESULT,PASS) not unique
+# Return: { { result pass expected_total }... } where (RESULT,PASS) is unique
+proc compute_expected_pass_totals { tuples } {
+ foreach cell $tuples { set count([lrange $cell 0 1]) 0 }
+ foreach cell $tuples { incr count([lrange $cell 0 1]) [lindex $cell 3] }
+ set result [list]
+ foreach name [lsort [array names count]] {
+ lappend result [concat $name $count($name)]
+ }
+ return $result
+}
+
+# Given: TUPLES: { { result pass name count }... }; (RESULT,PASS) not unique
+# Return: { { result expected_grand_total }... }
+proc compute_expected_grand_totals { tuples } {
+ foreach cell $tuples { set count([lindex $cell 0]) 0 }
+ foreach cell $tuples { incr count([lindex $cell 0]) [lindex $cell 3] }
+ set result [list]
+ foreach name [lsort [array names count]] {
+ lappend result [list $name $count($name)]
+ }
+ return $result
+}
+
+# Given: TUPLES: { { result pass ... }... } where (RESULT,PASS) repeats later
+# Return: { { { result pass ... }... }... }; (RESULT,PASS) unique per sublist
+proc split_tuple_list { tuples } {
+ set result [list]
+ set sublist [list]
+ foreach cell $tuples {
+ if { [info exists seen([lrange $cell 0 1])] } {
+ # split here
+ lappend result $sublist
+ set sublist [list]
+ array unset seen
+ }
+ lappend sublist $cell
+ set seen([lrange $cell 0 1]) 1
+ }
+ lappend result $sublist
+ return $result
+}
+
+# TUPLES is: { { result pass name count }... }
+proc write_file { basename tuples } {
+ set fd [open [testsuite file -object -test passes ${basename}.sum] w]
+ set pass {}
+ foreach cell [lsort -index 1 $tuples] {
+ if { $pass ne [lindex $cell 1] } {
+ puts $fd "Running pass `[lindex $cell 1]' ..."
+ set pass [lindex $cell 1]
+ }
+ for { set i 1 } { $i <= [lindex $cell 3] } { incr i } {
+ puts $fd "[lindex $cell 0]: [lindex $cell 1]:\
+ [lindex $cell 2] test ${i}/[lindex $cell 3]"
+ }
+ }
+ close $fd
+}
+
+proc run_multipass_output_test { filetag } {
+ global LAUNCHER TOOL_IMPL
+ global header_column_names
+ global result_column_map
+ global test_results
+ global tuplemap
+
+ set ssdpar [lindex $tuplemap($filetag) 0]
+ set tags [lindex $tuplemap($filetag) 1]
+ set passes [lindex $tuplemap($filetag) 2]
+ set results {}
+ foreach dummy $tags { lappend results [lindex $tuplemap($filetag) 3] }
+ set results [join $results]
+
+ # initialize totals arrays to zero
+ foreach result $test_results { set have_grand_totals($result) 0 }
+ array set want_grand_totals [array get have_grand_totals]
+ foreach cell [build_tuple_list $test_results $passes] {
+ set have_pass_totals([join [lrange $cell 0 1] ","]) 0
+ }
+ array set want_pass_totals [array get have_pass_totals]
+
+ # get the test list
+ set list [build_tuple_list $results $passes]
+ set list [annotate_tuple_list $list [lindex $ssdpar 0] [lindex $ssdpar 1]]
+
+ # compute expected totals
+ # note that this only fills non-zero array positions
+ foreach cell [compute_expected_pass_totals $list] {
+ set want_pass_totals([join [lrange $cell 0 1] ","]) [lindex $cell 2]
+ }
+ array set want_grand_totals [join [compute_expected_grand_totals $list]]
+
+ # write the test data files and store expected per-file counts
+ foreach tag $tags fileset [split_tuple_list $list] {
+ # write test file
+ write_file "${filetag}-${tag}" $fileset
+ # initialize test results for this file
+ foreach result $test_results {
+ foreach pass $passes {
+ set want_file_counts(${filetag}-${tag},$result,$pass) 0
+ set have_file_counts(${filetag}-${tag},$result,$pass) 0
+ }
+ }
+ # store expected results for this file
+ foreach cell $fileset {
+ set want_file_counts(${filetag}-${tag},[join [lrange $cell 0 1] \
+ ","]) [lindex $cell 3]
+ }
+ }
+
+ # run the dejagnu-report-card tool
+ set separator_count 0
+ spawn /bin/sh -c \
+ "cd [testsuite file -object -test passes]\
+ && exec $LAUNCHER --DGTimpl $TOOL_IMPL report-card ${filetag}-*.sum"
+
+ # skip header
+ expect {
+ -re {^[[:space:]]+_+[\r\n]+} { exp_continue }
+ -re {^[[:space:]]+/([^\r\n]*)[\r\n]+} { exp_continue }
+ -re {^[[:space:]]+\|-+[\r\n]+} { incr separator_count }
+ }
+
+ # read individual file lines
+ set re_file_row {^[[:space:]]*}
+ append re_file_row {(} $filetag {-[[:alpha:]]+)[[:space:]]+}
+ append re_file_row {/[[:space:]]+([[:alpha:]]+)[[:space:]]+\|}
+ append re_file_row {[[:space:]]*([[:digit:][:space:]]+)[\r\n]+}
+ expect {
+ -re $re_file_row {
+ foreach column $result_column_map colname $header_column_names \
+ have $expect_out(3,string) {
+ set want 0
+ foreach rs $column {
+ set tmp $expect_out(1,string),$rs,$expect_out(2,string)
+ incr want $want_file_counts($tmp)
+ }
+ if { $have == $want } {
+ pass "count $colname\
+ for pass $expect_out(2,string)\
+ in file $expect_out(1,string)"
+ } else {
+ fail "count $colname\
+ for pass $expect_out(2,string)\
+ in file $expect_out(1,string)"
+ }
+ }
+ exp_continue
+ }
+ -re {^[[:space:]]+\|-+[\r\n]+} { incr separator_count }
+ }
+
+ # read pass totals lines
+ set re_pass_row {^[[:space:]]+([[:alpha:]]+)[[:space:]]+\|}
+ append re_pass_row {[[:space:]]*([[:digit:][:space:]]+)[\r\n]+}
+ expect {
+ -re $re_pass_row {
+ foreach column $result_column_map colname $header_column_names \
+ have $expect_out(2,string) {
+ set want 0
+ foreach rs $column {
+ incr want $want_pass_totals($rs,$expect_out(1,string))
+ }
+ if { $have == $want } {
+ pass "total $colname for pass $expect_out(1,string)"
+ } else {
+ fail "total $colname for pass $expect_out(1,string)"
+ }
+ }
+ exp_continue
+ }
+ -re {^[[:space:]]+\|-+[\r\n]+} { incr separator_count }
+ }
+
+ # read grand totals line
+ expect -re {^[[:space:]]+\|[[:space:]]*([[:digit:][:space:]]+)[\r\n]+} {
+ foreach column $result_column_map colname $header_column_names \
+ have $expect_out(1,string) {
+ set want 0
+ foreach rs $column { incr want $want_grand_totals($rs) }
+ if { $have == $want } {
+ pass "grand total $colname"
+ } else {
+ fail "grand total $colname"
+ }
+ }
+ }
+
+ # skip the footer
+ expect -re {.+} { exp_continue }
+
+ if { $separator_count == 3 } {
+ pass "expected separator lines"
+ } else {
+ fail "expected separator lines"
+ }
+}
+
+foreach filetag [lsort [array names tuplemap]] {
+ run_multipass_output_test $filetag
+}
+
+#EOF
----
Merry Christmas and Happy New Year
-- Jacob
- PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card"),
Jacob Bachmeyer <=
- Re: PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card"), Ben Elliston, 2018/12/30
- Re: PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card"), Ben Elliston, 2018/12/30
- Re: PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card"), Jacob Bachmeyer, 2018/12/30
- Re: PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card"), Ben Elliston, 2018/12/30
- Re: PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card"), Rainer Orth, 2018/12/30
- Re: PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card"), Jacob Bachmeyer, 2018/12/30
- Re: PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card"), Ben Elliston, 2018/12/31
- Re: PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card"), Jacob Bachmeyer, 2018/12/31
- Re: PATCH: add dejagnu-report-card(1) tool (run as "dejagnu report card"), Jacob Bachmeyer, 2018/12/30