interpreter: support for forcibly verbose logging of some tests

Add a new keyword argument to test() and benchmark(), completing the
implementation of the feature.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

docs/markdown/snippets/test-verbose.md

@@ -0,0 +1,6 @@
+## New keyword argument `verbose` for tests and benchmarks
+
+The new keyword argument `verbose` can be used to mark tests and benchmarks
+that must always be logged verbosely on the console.  This is particularly
+useful for long-running tests, or when a single Meson test() is wrapping
+an external test harness.

docs/yaml/functions/benchmark.yaml

@@ -104,3 +104,11 @@ kwargs:
       The starting order of tests with identical priorities is
       implementation-defined. The default priority is 0, negative numbers are
       permitted.
+
+  verbose:
+    type: bool
+    since: 0.62.0
+    default: false
+    description: |
+      if true, forces the test results to be logged as if `--verbose` was passed
+      to `meson test`.

docs/yaml/functions/test.yaml

@@ -33,6 +33,13 @@ description: |
   test(..., env: nomalloc, ...)
   ```
 
+  In addition to running individual executables as test cases, `test()`
+  can also be used to invoke an external test harness.  In this case,
+  it is best to use `verbose: true` *(since 0.62.0)* and, if supported
+  by the external harness, `protocol: 'tap'` *(since 0.50.0)*.  This will
+  ensure that Meson logs each subtest as it runs, instead of including
+  the whole log at the end of the run.
+
   Defined tests can be run in a backend-agnostic way by calling
   `meson test` inside the build dir, or by using backend-specific
   commands, such as `ninja test` or `msbuild RUN_TESTS.vcxproj`.

mesonbuild/backend/backends.py

@@ -1150,7 +1150,7 @@ class Backend:
                                    isinstance(exe, build.Executable),
                                    [x.get_id() for x in depends],
                                    self.environment.coredata.version,
-                                   False)
+                                   t.verbose)
             arr.append(ts)
         return arr
 

mesonbuild/interpreter/interpreter.py

@@ -215,6 +215,7 @@ TEST_KWARGS: T.List[KwargInfo] = [
     ENV_KW,
     DEPENDS_KW.evolve(since='0.46.0'),
     KwargInfo('suite', ContainerTypeInfo(list, str), listify=True, default=['']),  # yes, a list of empty string
+    KwargInfo('verbose', bool, default=False, since='0.62.0'),
 ]
 
 permitted_dependency_kwargs = {
@@ -1972,7 +1973,8 @@ external dependencies (including libraries) must go to "dependencies".''')
                     kwargs['timeout'],
                     kwargs['workdir'],
                     kwargs['protocol'],
-                    kwargs['priority'])
+                    kwargs['priority'],
+                    kwargs['verbose'])
 
     def add_test(self, node: mparser.BaseNode, args: T.List, kwargs: T.Dict[str, T.Any], is_base_test: bool):
         t = self.make_test(node, args, kwargs)

mesonbuild/interpreter/interpreterobjects.py

@@ -636,7 +636,7 @@ class Test(MesonInterpreterObject):
                  cmd_args: T.List[T.Union[str, mesonlib.File, build.Target]],
                  env: build.EnvironmentVariables,
                  should_fail: bool, timeout: int, workdir: T.Optional[str], protocol: str,
-                 priority: int):
+                 priority: int, verbose: bool):
         super().__init__()
         self.name = name
         self.suite = listify(suite)
@@ -651,6 +651,7 @@ class Test(MesonInterpreterObject):
         self.workdir = workdir
         self.protocol = TestProtocol.from_str(protocol)
         self.priority = priority
+        self.verbose = verbose
 
     def get_exe(self) -> T.Union[ExternalProgram, build.Executable, build.CustomTarget]:
         return self.exe