Print the (shortened) output of the failed tests as they happen.
If neither --verbose nor --print-errorlogs was specified, omit the
summary of failures, because it is pretty much the same as the earlier
output of "meson test".
In non-parallel verbose mode the output of the test/benchmark
is not buffered, therefore the command line is only printed by
ConsoleLogger for failing tests and only after the test has run.
Verbose mode is designed mostly for CI systems, where output must
be human readable but is generally consumed from a browser with "Find"
commands rather than from a terminal. With this usecase in mind, it
is better to provide as much detail as possible, so add more output
and just tell the user which tests have started. Do so, using the
recently introduced TestResult.RUNNING state.
Ensure that all the required modifications are included in the logs.
This makes it possible for users to cut-and-paste from the logs when
trying to reproduce failures outside Meson.
Right now the same code is used to print the logs for both the console
and the text log. Differentiating them lets the important bits of
the console output stand out, and makes the console output a bit more
readable.
Using verbose mode dropped stdout/stderr from the logs, because
it was not captured.
Now that we can easily stick code in the middle of the reading of
stdout/stderr, use that to print stdout and stderr on the fly
while also capturing them for the logs. The output is line-buffered.
As a side effect, this also fixes a possible deadlock due to
not using ensure_future around stdo_task and stde_task. In
particular:
- the stdo_task coroutine would not terminate until the test closed
stdo_task
- the stde_task coroutine would not start until the stdo_task
coroutine finished
Therefore, the test could get stuck waiting for its parent to
read the contents of stderr, but that would not happen because
Meson was still in the stdo_task coroutine.
Move the logic to start the read/decode
tasks to TestSubprocess and keep SingleTestRunner simple.
The lines() inner function is tweaked to produce stdout as a future.
This removes the nonlocal access (which is not possible anymore
when the code is moved out of _run_cmd), and also lets _run_cmd
use "await stdo_task" for both parsed and unparsed output.
After the next patch, we will need to complete parse_task before
stdo_task (because parse_task will not set the "stdo" variable
anymore but it will still collect stdout just like now). Do
the change now to isolate the more complicated changes.
The new behavior of interrupting the longest running test with Ctrl-C is useful
when tests hang, but not when the run is completely broken for some reason.
Psychology tells us that the user will compulsively spam Ctrl-C in this case,
so exit if three Ctrl-C's are detected within a second.
This correctly formats tests with CJK names or, well, emoji. It is not perfect
(for example it does not correctly format emoji that are variations of 1-wide
characters), but it is as good as most terminal emulators.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Instead of slurping in the entire stream, build the TestResult along
the way. This allows reporting the results of TAP and Rust subtests as
they come in, either as part of the progress report or (in the future)
as individual lines of the output.
Instead of creating temporary files, get the StreamReaders from
_run_subprocess's returned object. Through asyncio magic, their
contents will be read as it becomes ready and then returned when
the StreamReader.read future is awaited.
Because of this change, the stdout and stderr can be easily
preserved when TestSubprocess returns an additional_error.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
We would like SingleTestRunner to run code before waiting on the process,
for example starting tasks to read stdout and stderr.
Return a new object that is able to complete _run_subprocess's task.
In the next patch, SingleTestRunner will also use the object to get hold
of the stdout and stderr StreamReaders.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Include the names from the TAP output and the SKIP/TODO explanations
if present. Omit the classname attribute, it is optional.
In order to enable this, TestRun.results becomes a list of TAPParser.Test
objects. If in the future there are other kinds of subtest results a
new class can be introduced, but for now it is enough.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
For now this is just a refactoring that simplifies the next patch. However,
it will also come in handy when we will make the parsing asynchronous, because
it will make it possible to access subtest results while the test runs.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
It is cleaner than collections.namedtuple. It also catches that "count()" is
a method on tuple, so rename the field to num_tests.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Pass the StringIO object to the parse method instead, because
there will be no T.Iterator[str] to use in the asynchronous
case.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This is the first step towards asynchronous parsing of the TAP output.
We will need to call the same code from both a "for" loop (for unit
tests) and an "async for" loop (for mtest itself). Because the same
function cannot be both a generator and an asynchronous generator, we
need to build both on a common core. This commit therefore introduces
a parse_line function that "parse" can call in a loop. All the local
variables of TAPParser.parse move into "self".
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Rust has it's own built in unit test format, which is invoked by
compiling a rust executable with the `--test` flag to rustc. The tests
are then run by simply invoking that binary. They output a custom test
format, which this patch adds parsing support for. This means that we
can report each subtest in the junit we generate correctly, which should
be helpful for orchestration systems like gitlab and jenkins which can
parse junit XML.
for non tap tests we want to associate names with the tests, to that end
store them as a dict. For TAP tests, we'll store the "name" as an
integer string that coresponds to the order that the tests were run in.
Flush after each output line, even if printing to a file, so that each
result is immediately visible down a pipeline.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Add a progress report in the style of "yum". Every second the
report prints a different test among the ones that are running.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Xavier Claessens
Paolo Bonzini
Dylan Baker