The Meson Build System
http://mesonbuild.com/
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
261 lines
8.1 KiB
261 lines
8.1 KiB
--- |
|
short-description: Meson's own unit-test system |
|
... |
|
|
|
# Unit tests |
|
|
|
Meson comes with a fully functional unit test system. To use it simply build |
|
an executable and then use it in a test. |
|
|
|
```meson |
|
e = executable('prog', 'testprog.c') |
|
test('name of test', e) |
|
``` |
|
|
|
You can add as many tests as you want. They are run with the command `ninja |
|
test`. |
|
|
|
Meson captures the output of all tests and writes it in the log file |
|
`meson-logs/testlog.txt`. |
|
|
|
## Test parameters |
|
|
|
Some tests require the use of command line arguments or environment |
|
variables. These are simple to define. |
|
|
|
```meson |
|
test('command line test', exe, args : ['first', 'second']) |
|
test('envvar test', exe2, env : ['key1=value1', 'key2=value2']) |
|
``` |
|
|
|
Note how you need to specify multiple values as an array. |
|
|
|
### MALLOC_PERTURB_ |
|
|
|
By default, environment variable |
|
[`MALLOC_PERTURB_`](http://man7.org/linux/man-pages/man3/mallopt.3.html) is |
|
set to a random value between 1..255. This can help find memory leaks on |
|
configurations using glibc, including with non-GCC compilers. This feature |
|
can be disabled as discussed in [test()](Reference-manual.md#test). |
|
|
|
## Coverage |
|
|
|
If you enable coverage measurements by giving Meson the command line flag |
|
`-Db_coverage=true`, you can generate coverage reports after running the |
|
tests (running the tests is required to gather the list of functions that get |
|
called). Meson will autodetect what coverage generator tools you have |
|
installed and will generate the corresponding targets. These targets are |
|
`coverage-xml` and `coverage-text` which are both provided by |
|
[Gcovr](http://gcovr.com) (version 3.3 or higher) and `coverage-html`, which |
|
requires [Lcov](https://ltp.sourceforge.io/coverage/lcov.php) and |
|
[GenHTML](https://linux.die.net/man/1/genhtml) or [Gcovr](http://gcovr.com). |
|
As a convenience, a high-level `coverage` target is also generated which will |
|
produce all 3 coverage report types, if possible. |
|
|
|
The output of these commands is written to the log directory `meson-logs` in |
|
your build directory. |
|
|
|
## Parallelism |
|
|
|
To reduce test times, Meson will by default run multiple unit tests in |
|
parallel. It is common to have some tests which can not be run in parallel |
|
because they require unique hold on some resource such as a file or a D-Bus |
|
name. You have to specify these tests with a keyword argument. |
|
|
|
```meson |
|
test('unique test', t, is_parallel : false) |
|
``` |
|
|
|
Meson will then make sure that no other unit test is running at the same |
|
time. Non-parallel tests take longer to run so it is recommended that you |
|
write your unit tests to be parallel executable whenever possible. |
|
|
|
By default Meson uses as many concurrent processes as there are cores on the |
|
test machine. You can override this with the environment variable |
|
`MESON_TESTTHREADS` like this. |
|
|
|
```console |
|
$ MESON_TESTTHREADS=5 meson test |
|
``` |
|
|
|
## Priorities |
|
|
|
*(added in version 0.52.0)* |
|
|
|
Tests can be assigned a priority that determines when a test is *started*. |
|
Tests with higher priority are started first, tests with lower priority |
|
started later. The default priority is 0, meson makes no guarantee on the |
|
ordering of tests with identical priority. |
|
|
|
```meson |
|
test('started second', t, priority : 0) |
|
test('started third', t, priority : -50) |
|
test('started first', t, priority : 1000) |
|
``` |
|
|
|
Note that the test priority only affects the starting order of tests and |
|
subsequent tests are affected by how long it takes previous tests to |
|
complete. It is thus possible that a higher-priority test is still running |
|
when lower-priority tests with a shorter runtime have completed. |
|
|
|
## Skipped tests and hard errors |
|
|
|
Sometimes a test can only determine at runtime that it can not be run. |
|
|
|
For the default `exitcode` testing protocol, the GNU standard approach in |
|
this case is to exit the program with error code 77. Meson will detect this |
|
and report these tests as skipped rather than failed. This behavior was added |
|
in version 0.37.0. |
|
|
|
For TAP-based tests, skipped tests should print a single line starting with |
|
`1..0 # SKIP`. |
|
|
|
In addition, sometimes a test fails set up so that it should fail even if it |
|
is marked as an expected failure. The GNU standard approach in this case is |
|
to exit the program with error code 99. Again, Meson will detect this and |
|
report these tests as `ERROR`, ignoring the setting of `should_fail`. This |
|
behavior was added in version 0.50.0. |
|
|
|
## Testing tool |
|
|
|
The goal of the meson test tool is to provide a simple way to run tests in a |
|
variety of different ways. The tool is designed to be run in the build |
|
directory. |
|
|
|
The simplest thing to do is just to run all tests. |
|
|
|
```console |
|
$ meson test |
|
``` |
|
|
|
### Run subsets of tests |
|
|
|
For clarity, consider the meson.build containing: |
|
|
|
```meson |
|
|
|
test('A', ..., suite: 'foo') |
|
test('B', ..., suite: ['foo', 'bar']) |
|
test('C', ..., suite: 'bar') |
|
test('D', ..., suite: 'baz') |
|
|
|
``` |
|
|
|
Specify test(s) by name like: |
|
|
|
```console |
|
$ meson test A D |
|
``` |
|
|
|
Tests belonging to a suite `suite` can be run as follows |
|
|
|
```console |
|
$ meson test --suite (sub)project_name:suite |
|
``` |
|
|
|
Since version *0.46*, `(sub)project_name` can be omitted if it is the |
|
top-level project. |
|
|
|
Multiple suites are specified like: |
|
|
|
```console |
|
$ meson test --suite foo --suite bar |
|
``` |
|
|
|
NOTE: If you choose to specify both suite(s) and specific test name(s), the |
|
test name(s) must be contained in the suite(s). This however is redundant-- |
|
it would be more useful to specify either specific test names or suite(s). |
|
|
|
### Other test options |
|
|
|
Sometimes you need to run the tests multiple times, which is done like this: |
|
|
|
```console |
|
$ meson test --repeat=10 |
|
``` |
|
|
|
Invoking tests via a helper executable such as Valgrind can be done with the |
|
`--wrap` argument |
|
|
|
```console |
|
$ meson test --wrap=valgrind testname |
|
``` |
|
|
|
Arguments to the wrapper binary can be given like this: |
|
|
|
```console |
|
$ meson test --wrap='valgrind --tool=helgrind' testname |
|
``` |
|
|
|
Meson also supports running the tests under GDB. Just doing this: |
|
|
|
```console |
|
$ meson test --gdb testname |
|
``` |
|
|
|
Meson will launch `gdb` all set up to run the test. Just type `run` in the |
|
GDB command prompt to start the program. |
|
|
|
The second use case is a test that segfaults only rarely. In this case you |
|
can invoke the following command: |
|
|
|
```console |
|
$ meson test --gdb --repeat=10000 testname |
|
``` |
|
|
|
This runs the test up to 10 000 times under GDB automatically. If the program |
|
crashes, GDB will halt and the user can debug the application. Note that |
|
testing timeouts are disabled in this case so `meson test` will not kill |
|
`gdb` while the developer is still debugging it. The downside is that if the |
|
test binary freezes, the test runner will wait forever. |
|
|
|
Sometimes, the GDB binary is not in the PATH variable or the user wants to |
|
use a GDB replacement. Therefore, the invoked GDB program can be specified |
|
*(added 0.52.0)*: |
|
|
|
```console |
|
$ meson test --gdb --gdb-path /path/to/gdb testname |
|
``` |
|
|
|
```console |
|
$ meson test --print-errorlogs |
|
``` |
|
|
|
Meson will report the output produced by the failing tests along with other |
|
useful information as the environmental variables. This is useful, for |
|
example, when you run the tests on Travis-CI, Jenkins and the like. |
|
|
|
For further information see the command line help of Meson by running `meson |
|
test -h`. |
|
|
|
## Legacy notes |
|
|
|
If `meson test` does not work for you, you likely have a old version of |
|
Meson. In that case you should call `mesontest` instead. If `mesontest` |
|
doesn't work either you have a very old version prior to 0.37.0 and should |
|
upgrade. |
|
|
|
## Test outputs |
|
|
|
Meson will write several different files with detailed results of running |
|
tests. These will be written into $builddir/meson-logs/ |
|
|
|
### testlog.json |
|
|
|
This is not a proper json file, but a file containing one valid json object |
|
per line. This is file is designed so each line is streamed out as each test |
|
is run, so it can be read as a stream while the test harness is running |
|
|
|
### testlog.junit.xml |
|
|
|
This is a valid JUnit XML description of all tests run. It is not streamed |
|
out, and is written only once all tests complete running. |
|
|
|
When tests use the `tap` protocol each test will be recorded as a testsuite |
|
container, with each case named by the number of the result. |
|
|
|
When tests use the `gtest` protocol meson will inject arguments to the test |
|
to generate it's own JUnit XML, which meson will include as part of this XML |
|
file. |
|
|
|
*New in 0.55.0*
|
|
|