This makes it clear in the results that tests marked "should_fail"
exist. We also avoid the all caps output and make the classifications
unambigous compared to pytest or autotools' XFAIL/XPASS.
Before:
OK: 329
FAIL: 1
SKIP: 0
TIMEOUT: 0
After:
Ok: 323
Expected Fail: 1
Fail: 6
Unexpected Pass: 0
Skipped: 0
Timeout: 0
as instructed in the python docs, you should not use PIPE here. This can
lead to deadlocks, with massive testsuite output. Which was the case for efl.
For now the output of the tests is redirected into the a temp file, the
content from there can then be used to fill the TestRun structure.
This fixes test running problems in efl.
This has the adventage that "meson --help" shows a list of all commands,
making them discoverable. This also reduce the manual parsing of
arguments to the strict minimum needed for backward compatibility.
We used to immediately try to use whatever exe_wrapper was defined in
the cross file, but some people generate the cross file once and use
it for several projects, most of which do not even need an exe wrapper
to build.
Now we're a bit more resilient. We quietly fall back to using
non-exe-wrapper paths for compiler checks and skip the sanity check.
However, if some code needs the exe wrapper, f.ex., if you run a built
executable using custom_target() or run_target(), we will error out
during setup.
Tests will, of course, continue to error out when you run them if the
exe wrapper was not found. We don't want people's tests to silently
"pass" (aka skip) because of a bad CI setup.
Closes https://github.com/mesonbuild/meson/issues/3562
This commit also adds a test for the behaviour of exe_wrapper in these
cases, and refactors the unit tests a bit for it.
We already have code to fetch and find binaries specified in a cross
file, so use the same code for exe_wrapper. This allows us to handle
the same corner-cases that were fixed for other cross binaries.
When a test fails due to a signal (e.g., SIGSEGV) it can be somewhat
mysterious why the test failed. Also, even when a test fails due to a
non-zero exit status it would help if the exit status was reported. This
augments the result string to include the non-zero exit status or
signal number and name.
Resolves#3642
When the exe runner is `wine` or `wine32` or `wine64`, etc.
This allows people to run tests with wine.
Note that you also have to set WINEPATH to point to your custom
prefix(es) if your tests use external dependencies.
Closes https://github.com/mesonbuild/meson/issues/3620
Replace the logic where a test setup with no project specifier defaults to
the main project with one that takes the test setup from the same
(sub)project from where the to-be-executed test has been read from.
Use $project_name:$test_setup namespace scheme for test setups. This
allows one to choose from which (sub)project a test setup is taken from
should there be several sharing the same name. Defaults to the main
project. E.g. "meson test --setup subproj:valgrind".
Setting MALLOC_PERTURB_ to a non-zero value is fine for regular test
cases. It helps catching bugs, but also comes with some runtime
overhead.
This overhead is noticeable for benchmarks when compared to running them
directly instead of through Meason.
Therefore, MALLOC_PERTURB_ is not touched for benchmarks.
closes#3034
According to Python documentation[1] dirname and basename
are defined as follows:
os.path.dirname() = os.path.split()[0]
os.path.basename() = os.path.split()[1]
For the purpose of better readability split() is replaced
by appropriate function if only one part of returned tuple
is used.
[1]: https://docs.python.org/3/library/os.path.html#os.path.split
When `ninja -C builddir/ test` is run, ninja will change into the build
dir before starting, but `meson test -C builddir/` does not. This is
important because meson does not use (for good reasons) absolute paths,
which means if a test case needs to be passed as an argument a file name
that is part of the build process, it will be relative builddir. Without
changing into the builddir the path will not exist (or worse, point at
the wrong thing), and test will not behave as intended.
To fix this mtest will change directory before starting tests, and will
change back after all tests have been finished.
Fixes#2710
Adding it to the end of PATH means that if an installed instance of a DLL
exists, that would be used instead of the built instance.
Compare with run_exe(), which already gets this right.
Stian Selnes
Dave Reisner
Dylan Baker
Marcel Hollerbach
Xavier Claessens
Jussi Pakkanen
Nirbheek Chauhan
Kurtis Rader
George Koehler
Alberto Sartori
Aleksey Filippov
nyorain
Alicia Boya García
Hemmo Nieminen
Alex Hirsch
Robert Doolittle
Eric Engestrom
Jon Turney
Dima Krasner
Elliott Sales de Andrade