For a test to be displayed to stdout without buffering, it has to be
1) in verbose mode 2) not executed in parallel with others
3) not parsed, i.e. not TAP or Rust. Include these three
conditions in a new property of stdout and use it in the
three places where it matters: when printing the initial
and final delimiters of the output, and when deciding the
console mode to use.
Suggested-by: Jussi Pakkanen <jpakkane@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
If --no-rebuild is used, the test program might not exist, spawning a
FileNotFoundError inside a long traceback rooted in subprocess.Popen
trying to run that test program. Current versions of Meson even say it's
an "unhandled python exception".
But we can do one better and actually tell the user what is wrong, why,
and what to do to fix it. And we can do so before getting waist deep in
partially running tests.
Fixes#10006
Verbose, non-parallel tests generally print their output as they run, rather than
after they finish. This however is not the case if stdout of the test is parsed
as is the case for TAP and Rust tests. In this case, the output during the run
is the list of the subtests, but stderr still has to be printed after the test
finishes.
Store in TestSerialisation whether a particular test must always be logged
verbosely. This is particularly useful for long-running tests or when a
single Meson test() is wrapping an external test harness. In this case,
TAP can be used by the external harness and Meson will log each subtest as
it runs.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
While the horizontal line and the other pictograms in mtest have an ASCII-only
version, the spinner does not. This causes mtest to fail with a
UnicodeEncodeError exception on non-Unicode locales.
Fixes: #9538
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Unless parsing TAP output, there is no strict requirement for
"meson test" to process test output one line at a time; it simply
looks nicer to not print a partial line if it can be avoided.
However, in the case of extremely long lines StreamReader.readline
can fail with a ValueError. Use readuntil('\n') instead and
just process whatever pieces of the line it returns.
Fixes: #8591
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
If meson is not a process group leader, a SIGINT will be delivered also to
its parent process (and possibly other processes). The parent process then
will probably exit and mtest will continue running in the background, without
any way to interrupt the run completely.
To fix this, treat SIGINT and SIGTERM the same way unless mtest is a
process group leader.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
(var,) is the correct way to pass values to a percent formatted string,
but not to .format so we would end up printing something like:
unexpected input at line (4,)
Upgrade to an f-string and insert the correct value correctly.
All changes were created by running
"pyupgrade --py3-only"
and committing the results. Although this has been performed in the
past, newer versions of pyupgrade can automatically catch more
opportunities, notably list comprehensions can use generators instead,
in the following cases:
- unpacking into function arguments as function(*generator)
- unpacking into assignments of the form x, y = generator
- as the argument to some builtin functions such as min/max/sorted
Also catch a few creeping cases of new code added using older styles.
This reverts commit 5fcb0e6525.
The commit is a massive change that should have been split in
separate pieces, and it also removes a few features:
* in verbose mode, subtests are not printed as they happen
* in non-verbose mode the progress report does not include the
number of subtests that have been run
* in non-parallel mode, output is batched rather than printed as
it happens
Furthermore, none of these changes are not documented in the release
notes. Revert so that each proposal can be tested, evaluated and
documented individually.
This change set aims to fix various "issues" seen with the current
implementation. The changes can be summarized with the following list:
* Replace emojis and spinners with multiline status displaying the name
and running time of each currently running test.
* The test output (especially in verbose mode or when multiple failing
tests' output gets printed out) can get confusing. Try to make the
output easier to read and grasp. Most notable change here is the
addition of the test number to the beginning of each printed line.
* Print exit details (i.e. exit code) of the test in verbose mode.
* Try to make the verbose "live" output from tests to match the look and
feel of otherwise produced (verbose) test output.
Currently, if the test fails to produce XML (or valid XML) then the test
fails with a backtrace. It's actually pretty easy to get into this
situation, a total failure of the test will result in no XML being
written (this can happen, for example, if rpaths to gtest are not
correctly set up). If we can't read the test, go ahead and complete
using `TestRunExitCode.complete()`, which will fail for the bad exit
code.
"meson setup" is resolving symlinks for the build directory in
validate_core_dirs. For consistency with it, do the same when
the build directory is passed via -C to devenv, dist, init, install
and test.
This ensures for example that the path to test dependencies is
computed correctly in "meson test".
Fixes: #8765
If a test program forks a child, the pipes might remain open and
"await stdo_task"/"await stde_task" will never complete in
SingleTestRunner._run_cmd().
Instead, catch them in TestSubprocess.wait() so that the whole
process group is killed.
Fixes: #8533
Reported-by: Bastien Nocera <hadess@hadess.net>
Dependencies is already a large and complicated package without adding
programs to the list. This also allows us to untangle a bit of spaghetti
that we have.
All changes were created by running
"pyupgrade --py3-only --keep-percent-format"
and committing the results. I have not touched string formatting for
now.
- use set literals
- simplify .format() parameter naming
- remove __future__
- remove default "r" mode for open()
- use OSError rather than compatibility aliases
- remove stray parentheses in function(generator) scopes
Reusing the runners for multiple repeats of the test run gets in the
way of the progress report, which stores runners in an OrderedSet.
Instead, create a separate SingleTestRunner object for each repeat.
While at it, fix the "duplicate suite" assertion as it can fire
with TAP tests and --repeat=N.
Fixes: #8405
Some time between 0.56 and 0.57 the TAP parser broke when a test exits
with a nonzero status.
The TAP protocol does not specify this behaviour - giving latitude to
implementers, and meson's previous behaviour was to report the exit
status gracefully.
This patch restores the old behaviour and adds a regression test
os.path.relpath(f, wd) returns path with \ seperator on Windows, but ninja
targets always uses / separator.
See for example https://gitlab.freedesktop.org/ocrete/libnice/-/jobs/7348274.
Analyzed-by: Xavier Claessens <xavier.claessens@collabora.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
It looks like GitLab ignores the suite name and actually uses
the classname. Adjust the output accordingly.
Fixes: #8316
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Avoid that the tasks linger and SingleTestRunner.run() never terminates.
In order to do this, we need read_decode() and read_decode_lines() to be
cancellable, and to handle the CancelledError gracefully while returning
the output they have collected so far.
For read_decode(), this means always operating on a line-by-line basis,
even if console_mode is not ConsoleUser.STDOUT. For read_decode_lines(),
instead, we cannot return an iterator. Rather, read_decode_lines()
returns the output directly (similar to read_decode) and communication
with the parser is mediated by an asyncio.Queue.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This makes non-parallel tests emit their output on the fly,
similar to ninja console jobs. It also cleans up the code
a bit, avoiding the repetition of "self.options.num_processes"
tests.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
- Ensure the output is terminated with a \n even if the test does not
include one.
- Ensure that stdout is flushed for each reported result
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Fix "meson test --wrapper foo --setup bar", it should work just fine
if the setup does not define a wrapper.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Start the parsing of the output early; this avoids a deadlock
if the test writes to stdout but no one reads from it. It
also reports TAP or Rust subtest results as they happen,
which was the intention all along.
While at it, use a consistent naming conventions for coroutines
vs tasks.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Avoid printing something like "30/-1s" when tests are run without
a timeout or with --timeout-multiplier 0.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This new keyword argument makes it possible to run specific
test setups only on a subset of the tests. For example, to
mark some tests as slow and avoid running them by default:
add_test_setup('quick', exclude_suites: ['slow'], is_default: true)
add_test_setup('slow')
It will then be possible to run the slow tests with either
`meson test --setup slow` or `meson test --suite slow`.
This will be needed to exclude testsuites from test setups (which
are stored in the build data). While at it, since a chdir is
needed simplify a bit the loading of tests and benchmarks.
Print the (shortened) output of the failed tests as they happen.
If neither --verbose nor --print-errorlogs was specified, omit the
summary of failures, because it is pretty much the same as the earlier
output of "meson test".
In non-parallel verbose mode the output of the test/benchmark
is not buffered, therefore the command line is only printed by
ConsoleLogger for failing tests and only after the test has run.
Verbose mode is designed mostly for CI systems, where output must
be human readable but is generally consumed from a browser with "Find"
commands rather than from a terminal. With this usecase in mind, it
is better to provide as much detail as possible, so add more output
and just tell the user which tests have started. Do so, using the
recently introduced TestResult.RUNNING state.
Ensure that all the required modifications are included in the logs.
This makes it possible for users to cut-and-paste from the logs when
trying to reproduce failures outside Meson.
Right now the same code is used to print the logs for both the console
and the text log. Differentiating them lets the important bits of
the console output stand out, and makes the console output a bit more
readable.
Paolo Bonzini
Eli Schwartz
William Toohey
Hemmo Nieminen
Dylan Baker
Daniel Mensinger
Luke Drummond
Xavier Claessens