Some time between 0.56 and 0.57 the TAP parser broke when a test exits
with a nonzero status.
The TAP protocol does not specify this behaviour - giving latitude to
implementers, and meson's previous behaviour was to report the exit
status gracefully.
This patch restores the old behaviour and adds a regression test
* mtest: TestResult.SKIP is not a failure
If some but not all tests in a run were skipped, then the overall result
is given by whether there were any failures among the non-skipped tests.
Resolves: https://github.com/mesonbuild/meson/issues/7515
Signed-off-by: Simon McVittie <smcv@debian.org>
* Add test-cases for partially skipped TAP tests
issue7515.txt is the output of one of the real TAP tests in gjs, which
failed as a result of #7515. The version inline in meson.build is
a minimal reproducer.
Signed-off-by: Simon McVittie <smcv@debian.org>
D lang compilers have an option -release (or similar) which turns off
asserts, contracts, and other runtime type checking. This patch wires
that up to the b_ndebug flag.
Fixes#7082
This provides an initial support for parsing TAP output. It detects failures
and skipped tests without relying on exit code, as well as early termination
of the test due to an error or a crash.
For now, subtests are not recorded in the TestRun object. However, because the
TAP output goes on stdout, it is printed by --print-errorlogs when a test does
not behave as expected. Handling subtests as TestRuns, and serializing them
to JSON, can be added later.
The parser was written specifically for Meson, and comes with its own
test suite.
Fixes#2923.
Hard errors also come from the GNU Automake test protocol. They happen when
e.g., the set-up of a test case scenario fails, or when some
other unexpected or highly undesirable condition is encountered.
TAP will use them for parse errors too. Add them to the exitcode protocol
first.
When a test fails due to a signal (e.g., SIGSEGV) it can be somewhat
mysterious why the test failed. Also, even when a test fails due to a
non-zero exit status it would help if the exit status was reported. This
augments the result string to include the non-zero exit status or
signal number and name.
Resolves#3642
Luke Drummond
Simon McVittie
Dylan Baker
Jussi Pakkanen
Paolo Bonzini
Kurtis Rader
Niklas Claesson