mtest: implement a maxfail option

This allows early exit of the project tests once a certain number of
failures are detected. For example `meson test --maxfail=1` will abort
as soon as a single test fails.

Currently running tests are marked as failed via INTERRUPT.

Resolves #9352

docs/markdown/snippets/test-maxfail.md

@@ -0,0 +1,6 @@
+## Option to allow meson test to fail fast after the first failing testcase
+
+`meson test --maxfail=1` will now cause all pending or in-progress tests to be
+canceled or interrupted after 1 test is marked as failing. This can be used for
+example to quit a CI run and avoid burning additional time as soon as it is
+known that the overall return status will be failing.

mesonbuild/mtest.py

@@ -94,6 +94,9 @@ def determine_worker_count() -> int:
     return num_workers
 
 def add_arguments(parser: argparse.ArgumentParser) -> None:
+    parser.add_argument('--maxfail', default=0, type=int,
+                        help='Number of failing tests before aborting the '
+                        'test run. (default: 0, to disable aborting on failure)')
     parser.add_argument('--repeat', default=1, dest='repeat', type=int,
                         help='Number of times to run the tests.')
     parser.add_argument('--no-rebuild', default=False, action='store_true',
@@ -1912,6 +1915,9 @@ class TestHarness:
                     return
                 res = await test.run(self)
                 self.process_test_result(res)
+                maxfail = self.options.maxfail
+                if maxfail and self.fail_count >= maxfail and res.res.is_bad():
+                    cancel_all_tests()
 
         def test_done(f: asyncio.Future) -> None:
             if not f.cancelled():