The new behavior of interrupting the longest running test with Ctrl-C is useful
when tests hang, but not when the run is completely broken for some reason.
Psychology tells us that the user will compulsively spam Ctrl-C in this case,
so exit if three Ctrl-C's are detected within a second.
This correctly formats tests with CJK names or, well, emoji. It is not perfect
(for example it does not correctly format emoji that are variations of 1-wide
characters), but it is as good as most terminal emulators.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Instead of slurping in the entire stream, build the TestResult along
the way. This allows reporting the results of TAP and Rust subtests as
they come in, either as part of the progress report or (in the future)
as individual lines of the output.
Instead of creating temporary files, get the StreamReaders from
_run_subprocess's returned object. Through asyncio magic, their
contents will be read as it becomes ready and then returned when
the StreamReader.read future is awaited.
Because of this change, the stdout and stderr can be easily
preserved when TestSubprocess returns an additional_error.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
We would like SingleTestRunner to run code before waiting on the process,
for example starting tasks to read stdout and stderr.
Return a new object that is able to complete _run_subprocess's task.
In the next patch, SingleTestRunner will also use the object to get hold
of the stdout and stderr StreamReaders.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Include the names from the TAP output and the SKIP/TODO explanations
if present. Omit the classname attribute, it is optional.
In order to enable this, TestRun.results becomes a list of TAPParser.Test
objects. If in the future there are other kinds of subtest results a
new class can be introduced, but for now it is enough.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
For now this is just a refactoring that simplifies the next patch. However,
it will also come in handy when we will make the parsing asynchronous, because
it will make it possible to access subtest results while the test runs.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
It is cleaner than collections.namedtuple. It also catches that "count()" is
a method on tuple, so rename the field to num_tests.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Pass the StringIO object to the parse method instead, because
there will be no T.Iterator[str] to use in the asynchronous
case.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This is the first step towards asynchronous parsing of the TAP output.
We will need to call the same code from both a "for" loop (for unit
tests) and an "async for" loop (for mtest itself). Because the same
function cannot be both a generator and an asynchronous generator, we
need to build both on a common core. This commit therefore introduces
a parse_line function that "parse" can call in a loop. All the local
variables of TAPParser.parse move into "self".
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Rust has it's own built in unit test format, which is invoked by
compiling a rust executable with the `--test` flag to rustc. The tests
are then run by simply invoking that binary. They output a custom test
format, which this patch adds parsing support for. This means that we
can report each subtest in the junit we generate correctly, which should
be helpful for orchestration systems like gitlab and jenkins which can
parse junit XML.
for non tap tests we want to associate names with the tests, to that end
store them as a dict. For TAP tests, we'll store the "name" as an
integer string that coresponds to the order that the tests were run in.
Flush after each output line, even if printing to a file, so that each
result is immediately visible down a pipeline.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Add a progress report in the style of "yum". Every second the
report prints a different test among the ones that are running.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
The TestLogger class lets us move the code for all those log files
out of TestHarness. The interface is based on JunitBuilder, which
is converted already in this commit. Over the next commits, we
will also convert JSON, text and console output.
The main difference with JunitBuilder is that the completion method is
asynchronous. This can be useful if the logger needs to clean up after
itself and wait for asyncio tasks.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Just reuse the collected_failures collection now that it contains
TestRun objects. Move the code to generate the short form of the log
to TestRun.
Note that the first line of the error log is not included in
get_log()'s return value, so the magic "first four lines are passed
unscathed" is changed to three lines only. The resulting output is
like this:
--- command ---
<command line>
--- Listing only the last 100 lines from a long log. ---
--- stdout ---
...
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
If there's an UnicodeEncodeError while printing the error logs,
TestHarness tries an encode/decode pair to get rid of anything that
is not a 7-bit ASCII character; this however results in "?" characters
that are not very clear. To make it easier to understand what is
going on, use backslashreplace instead.
While at it, fix the decode to use a matching encoding. This will
only matter in the rare case of sys.stdout.encoding not being an
ASCII superset, but that should not matter.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Instead of colorizing the whole status line, only colorize the word
representing the outcome of the test (SKIP, OK, FAIL, etc.). This
is less intrusive, so the patch also does the following changes:
- colorize OK and EXPECTEDFAIL, respectively as green and yellow
- colorize the summary of failures as well.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Instead of storing the string, store the whole TestRun. In the
next patches we'll use this to colorize the summary of failures,
and to allow a few more simplifications.
There is some code duplication between the console and logfile
code, but it won't matter as soon as console and logfile output
will be in two completely separate classes.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Avoid passing them around as parameters; this will be useful when logging
is moved out of TestHarness, because individual loggers will call back
into TestHarness to do common formatting chores.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Place in TestRun everything that is needed in order to
format the result. This avoids passing around the number
and visible test name as arguments.
Test numbers are assigned the first time they are used.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This will provide a way to pass more information from the TestHarness
local variables to the SingleTestRunner and use them outside the
run_test function. For example, the name could be used to report
progress while the tests are running.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
It is a usual workflow to fix something and retest to see if it is fixed using a
particular test. When tests start to become numerous, it becomes time consuming
for "meson test" to relink all of them (and in fact rebuild the whole project)
where the user has already specified the tests they want to run, as well as
the tests' dependencies.
Teach meson to be smart and only build what is needed for the test (or suite)
that were specified.
Fixes: #7473
Related: #7830
Avoid calling self.collected_failures.append twice, and avoid
inflated indentation by adding a "plain" decorator to mlog.
Fixes: ba71fde18 ("mtest: collect failures regardless of colorized console", 2020-10-12)
Paolo Bonzini
Dylan Baker
Oleg B