Back in November when this broke, we didn't notice because our tests
are run in-process, so we don't check that `msbuild RUN_TESTS.vcxproj`
and `ninja test` actually work.
Now we do.
This change helps us run on older distros such as Ubuntu LTS which is
very lazy in updating even non-core and stable packages such as Ninja.
Ninja 1.6.x is only needed for running the tests.
It seems on Windows, deleting in a loop can cause a race where the
following error is raised:
Traceback (most recent call last):
File "run_project_tests.py", line 550, in <module>
(passing_tests, failing_tests, skipped_tests) = run_tests(all_tests, 'meson-test-run', options.extra_args)
File "run_project_tests.py", line 416, in run_tests
result = result.result()
File "C:\python34-x64\lib\concurrent\futures\_base.py", line 402, in result
return self.__get_result()
File "C:\python34-x64\lib\concurrent\futures\_base.py", line 354, in __get_result
raise self._exception
ValueError: I/O operation on closed file.
https://ci.appveyor.com/project/jpakkane/meson/build/1.0.1559/job/vsek754eu000kg3e
There is never any reason to not do this since this script is supposed
to be run by developers and testers who are concerned with the details
of the problems.
It also helps with intermittent or hard-to-reproduce errors.
We have no test coverage for regeneration at all, which is why issues
like #1246 slide by without us noticing. With this, we will run a regen
on every test during `ninja test` after it has been compiled. This will
not affect test times too much since the regen will not rebuild anything
at all since there have been no source changes.
There is no way to do this in the .appveyor.yml file since it seems that
the appveyor environment is forcibly written after each cmd command that
is run.
Also add new tests for the platform-specific and compiler-specific
versioning scheme.
A rough summary is:
1. A bug in how run_tests.py:validate_install checked for files has been
fixed. Earlier it wasn't checking the install directory properly.
2. Shared libraries are no longer installed in common tests, and the
library name/path testing is now done in platform-specific tests.
3. Executables are now always called something?exe in the
installed_files.txt file, and the suffix automatically corrected
depending on the platform.
4. If a test installs a file called 'no-installed-files', the installed
files for that test are not validated. This is required to implement
compiler-specific tests for library names/paths such as MSVC vs MinGW
5. The platform-specific file renaming in run_tests.py has been mostly
removed since it is broken for shared libraries and isn't needed for
static libraries.
6. run_tests.py now reports all missing and extra files. The logic for
finding these has been reworked.
Print status on single line
only print a single line for succeeded tests and two lines for failed
tests. This makes it easier to scan the output for failed tests.
shutil.rmtree, which is used by tempfile.TemporaryDirectory, randomly fails
on Windows, because the directory is not empty although it should be,
because all files were deleted by shutil.rmtree internals before trying to
remove the directory.
A simple retry approach fixes the issue.
The _run_test method uses several global variables (unity_flags,
backend_flags, compile_commands, install_commands) which are
not set when the method is run by the executor (at least on Windows).
To resolve this, pass the variables as method parameters.
Nirbheek Chauhan
Jussi Pakkanen
Mike Sinkovsky
Igor Gnatenko
Elliott Sales de Andrade
Matthias Klumpp
Nirbheek Chauhan
Nicolas Schneider
Nicolas Schneider