Follow-up on commit 4274e0f42a. We want to
allow tests to be skipped freely in third-party environments, so this
should check the jobname, not whether $CI exists.
We will anyways raise an error when trying to run the project tests, if
$CI is set but no jobname is set.
This is treated by the test harness as though unset, i.e. we do normal
skipping and don't assume we are running in Meson's own project CI.
However, it has one distinction which is that it isn't an error to set
$CI without setting $MESON_CI_JOBNAME, if it is in fact set but to the
ignored value.
This lets automated workflows such as Linux distro testing, particularly
alpine linux, set $CI or have it set for them by default, without
messing things up.
Also it has the advantage of $CI actually enabling useful benefits! We
will still assume that this thirdparty environment wants to force
verbose logging (printing testlogs, running ninja/samu with -v) and
colorize the console.
All changes were created by running
"pyupgrade --py3-only"
and committing the results. Although this has been performed in the
past, newer versions of pyupgrade can automatically catch more
opportunities, notably list comprehensions can use generators instead,
in the following cases:
- unpacking into function arguments as function(*generator)
- unpacking into assignments of the form x, y = generator
- as the argument to some builtin functions such as min/max/sorted
Also catch a few creeping cases of new code added using older styles.
Fixes regression in commit 75688240cf.
Even though this function is *currently* only invoked on Windows, these
environment variables may not actually exist -- and apparently don't in
at least the "UnusedMissingReturn / windows" test run, which... did not
get triggered by that commit, since it only edited the testsuite runner,
not any test cases. \o/
The Environment object constructor accepts None as build_dir (for quite
a while now), so don't bother with creating a temporary directory for
use as the build_dir, if we're not going to need it.
Future work: Environment.__init__() sets scratch_dir to '' if build_dir
is None, which seems a little wonky, as it isn't a path.
Remove hard-coded framework test skip logic in skippable(), instead
annotate test.json with environments in which skip is expected.
(Mainly this is done with by testing the value of MESON_CI_JOBNAME now
set for linux jobs)
Plan to replace the hard-coded list of 'may be skipped' framework tests in
skippable() with annotations in test.json which record 'will be skipped
in these specific CI jobs'.
If the value of the MESON_CI_JOBNAME env var (an arbitrary string
expected to be unique for each CI configuration) contains any of the
strings in the `skip_on_jobname` key in test.json, the test is expected
to output MESON_SKIP_TEST.
Unexpected skips or runs are treated as an error.
Future work: Maybe we should add additional count categories 'unexpected
skip' and 'unexpected not skipped', rather than counting those as 'skipped'
and 'failed', respectively.
Since 25df6e7d split the iteration over tests to start them from the
iteration to collect their results, the variable 'name' is only being
set in the first iteratiorn, so all tests are treated as being in the
last test category read (probably 'wasm') for skipppable() and in the
XML output.
Store the category name in the TestDef object
Use it in skippable()
Use it in classname attribute of XML test results
Split out checking of file format as a separate GitHub workflow, rather
than running it as part of the project tests for every platform and
toolchain combination in CI, so that this test is not effected by the
changed paths constraints which are applied to the project tests.
This now gives a clear error rather than silently passes for
unrecognized categories, like:
python run_project_tests.py --only nonexisting
or
python run_project_tests.py --only objc # should be 'objective c'
We don't always have qmake installed (and it's good to test failure
paths too!) so we can't expect this to succeed in all cases. With the
following commit we'll use a test.json to test both pkg-config and
qmake, so we need to be able to skip.
Eli Schwartz
Mark Bolhuis
Daniel Mensinger
Xavier Claessens
Christian Clauss
Mat Cross
Nirbheek Chauhan
Jussi Pakkanen
Jon Turney
Dylan Baker
Ralf Gommers