Plan to replace the hard-coded list of 'may be skipped' framework tests in
skippable() with annotations in test.json which record 'will be skipped
in these specific CI jobs'.
If the value of the MESON_CI_JOBNAME env var (an arbitrary string
expected to be unique for each CI configuration) contains any of the
strings in the `skip_on_jobname` key in test.json, the test is expected
to output MESON_SKIP_TEST.
Unexpected skips or runs are treated as an error.
Future work: Maybe we should add additional count categories 'unexpected
skip' and 'unexpected not skipped', rather than counting those as 'skipped'
and 'failed', respectively.
I'm not sure what the current schema is describing, but it's not doing
anything useful to validate 'matrix' entries, as I discover when I come
to add something to that.
'matrix' is a dict with possible keys 'options' and 'exclude'.
'options' is a dict with arbitrary keys, whose's values are arrays
Once we describe those dicts correctly, also fix that 'val' keys may be
string, boolean or null, and the corresponding 'exclude' keys may be
string or boolean.
v2:
Also allow 'val' and 'exclude' to be an array of strings.
Improve test.json schema to disallow arbitrary keys in all dicts which
have a defined set of keys (the 'installed', matrix 'options' and
'stdout' dicts).
Add 'count' and 'comment' keys to 'stdout' dict.
This is useful for automatically generated docs (doxygen, hotdoc)
with a lot of generated files that may differ with different
versions of the generator.
D lang compilers have an option -release (or similar) which turns off
asserts, contracts, and other runtime type checking. This patch wires
that up to the b_ndebug flag.
Fixes#7082
Update the test.json schema, adding the 'stdout' property.
Also amend the test.json schema so the presence of an unexpected
property on the root object causes a validation error.
v2:
Also add 'tools' property to json schema.
Amend the documentation not to use the word 'list' to describe a dict.
This does a couple of nice things, one is that editors like vscode can
be configured to use this schema to provide auto completion and error
highlighting if invalid values are added or required values are missing.
It also allows us test that the format of the test matrix work in a unit
test, which I've added. It does require that the python jsonschema
package is installed.
Eli Schwartz
Jon Turney
Daniel Mensinger
Dylan Baker