new module "sourceset" to match source file lists against configuration data

In QEMU a single set of source files is built against many different
configurations in order to generate many executable.  Each executable
includes a different but overlapping subset of the source files; some
of the files are compiled separately for each output, others are
compiled just once.

Using Makefiles, this is achieved with a complicated mechanism involving
a combination of non-recursive and recursive make; Meson can do better,
but because there are hundreds of such conditional rules, it's important
to keep meson.build files brief and easy to follow.  Therefore, this
commit adds a new module to satisfy this use case while preserving
Meson's declarative nature.

Configurations are mapped to a configuration_data object, and a new
"source set" object is used to store all the rules, and then retrieve
the desired set of sources together with their dependencies.

The test case shows how extract_objects can be used to satisfy both
cases, i.e. when the object files are shared across targets and when
they have to be separate.  In the real-world case, a project would use
two source set objects for the two cases and then do
"executable(..., sources: ... , objects: ...)".  The next commit
adds such an example.

docs/markdown/SourceSet-module.md

@@ -0,0 +1,196 @@
+---
+short-description: Source set module
+authors:
+    - name: Paolo Bonzini
+      email: pbonzini@redhat.com
+      years: [2019]
+...
+
+# Source set module
+
+This module provides support for building many targets against a single set
+of files; the choice of which files to include in each target depends on the
+contents of a dictionary or a `configuration_data` object.  The module can
+be loaded with:
+
+``` meson
+ssmod = import('sourceset')
+```
+
+A simple example of using the module looks like this:
+
+``` meson
+ss = ssmod.source_set()
+# Include main.c unconditionally
+ss.add(files('main.c'))
+# Include a.c if configuration key FEATURE1 is true
+ss.add(when: 'FEATURE1', if_true: files('a.c'))
+# Include zlib.c if the zlib dependency was found, and link zlib
+# in the executable
+ss.add(when: zlib, if_true: files('zlib.c'))
+# many more rules here...
+ssconfig = ss.apply(config)
+executable('exe', sources: ssconfig.sources(),
+           dependencies: ssconfig.dependencies())
+```
+
+and it would be equivalent to
+
+``` meson
+sources = files('main.c')
+dependencies = []
+if config['FEATURE1'] then
+    sources += [files('a.c')]
+endif
+if zlib.found() then
+    sources += [files('zlib.c')]
+    dependencies += [zlib]
+endif
+# many more "if"s here...
+executable('exe', sources: sources, dependencies: dependencies())
+```
+
+Sourcesets can be used with a single invocation of the `apply` method,
+similar to the example above, but the module is especially useful
+when multiple executables are generated by applying the same rules to
+many different configurations.
+
+*Added 0.51.0*
+
+## Functions
+
+### `source_set()`
+
+``` meson
+ssmod.source_set()
+```
+
+Create and return a new source set object.
+
+**Returns**: a [source set][`source_set` object]
+
+## `source_set` object
+
+The `source_set` object provides methods to add files to a source set and
+to query it.  The source set becomes immutable after any method but `add`
+is called.
+
+### Methods
+
+#### `add()`
+
+``` meson
+source_set.add([when: varnames_and_deps],
+               [if_true: sources_and_deps],
+               [if_false: list_of_alt_sources])
+source_set.add(sources_and_deps)
+```
+
+Add a *rule* to a source set.  A rule determines the conditions under which
+some source files or dependency objects are included in a build configuration.
+All source files must be present in the source tree or they can be created
+in the build tree via `configure_file`, `custom_target` or `generator`.
+
+`varnames_and_deps` is a list of conditions for the rule, which can be
+either strings or dependency objects (a dependency object is anything that
+has a `found()` method).  If *all* the strings evaluate to true and all
+dependencies are found, the rule will evaluate to true; `apply()`
+will then include the contents of the `if_true` keyword argument in its
+result.  Otherwise, that is if any of the strings in the positional
+ arguments evaluate to false or any dependency is not found, `apply()`
+will instead use the contents of the `if_false` keyword argument.
+
+Dependencies can also appear in `sources_and_deps`.  In this case, a
+missing dependency will simply be ignored and will *not* disable the rule,
+similar to how the `dependencies` keyword argument works in build targets.
+
+**Note**: It is generally better to avoid mixing source sets and disablers.
+This is because disablers will cause the rule to be dropped altogether,
+and the `list_of_alt_sources` would not be taken into account anymore.
+
+#### `add_all()`
+
+``` meson
+source_set.add_all(when: varnames_and_deps,
+                   if_true: [source_set1, source_set2, ...])
+source_set.add_all(source_set1, source_set2, ...)
+```
+
+Add one or more source sets to another.
+
+For each source set listed in the arguments, `apply()` will
+consider their rules only if the conditions in `varnames_and_deps` are
+evaluated positively.  For example, the following:
+
+``` meson
+sources_b = ssmod.source_set()
+sources_b.add(when: 'HAVE_A', if_true: 'file.c')
+sources = ssmod.source_set()
+sources.add_all(when: 'HAVE_B', if_true: sources_b)
+```
+
+is equivalent to:
+
+``` meson
+sources = ssmod.source_set()
+sources.add(when: ['HAVE_A', 'HAVE_B'], if_true: 'file.c')
+```
+
+#### `all_sources()`
+
+``` meson
+list source_set.all_sources(...)
+```
+
+Returns a list of all sources that were placed in the source set using
+`add` (including nested source sets) and that do not have a not-found
+dependency.  If a rule has a not-found dependency, only the `if_false`
+sources are included (if any).
+
+**Returns**: a list of file objects
+
+#### `apply()`
+
+``` meson
+source_files source_set.apply(conf_data[, strict: false])
+```
+
+Match the source set against a dictionary or a `configuration_data` object
+and return a *source configuration* object.  A source configuration object
+allows you to retrieve the sources and dependencies for a specific configuration.
+
+By default, all the variables that were specified in the rules have to
+be present in `conf_data`.  However, in some cases the convention is
+that `false` configuration symbols are absent in `conf_data`; this is
+the case for example when the configuration was loaded from a Kconfig file.
+In that case you can specify the `strict: false` keyword argument, which
+will treat absent variables as false.
+
+**Returns**: a [source configuration][`source_configuration` object]
+
+## `source_configuration` object
+
+The `source_configuration` object provides methods to query the result of an
+`apply` operation on a source set.
+
+### Methods
+
+#### `sources()`
+
+``` meson
+source_config.sources()
+```
+
+Return the source files corresponding to the applied configuration.
+
+**Returns**: a list of file objects
+
+#### `dependencies()`
+
+``` meson
+source_config.dependencies()
+```
+
+Return the dependencies corresponding to the applied configuration.
+
+**Returns**: a list of dependency objects

docs/markdown/snippets/sourceset.md

@@ -0,0 +1,8 @@
+## New `sourceset` module
+
+A new module, `sourceset`, was added to help building many binaries
+from the same source files.  Source sets associate source files and
+dependencies to keys in a `configuration_data` object or a dictionary;
+they then take multiple `configuration_data` objects or dictionaries,
+and compute the set of source files and dependencies for each of those
+configurations.

docs/sitemap.txt

@@ -43,6 +43,7 @@ index.md
 			Qt5-module.md
 			RPM-module.md
 			Simd-module.md
+			SourceSet-module.md
 			Windows-module.md
 			Cuda-module.md
 			Kconfig-module.md

mesonbuild/modules/sourceset.py

@@ -0,0 +1,190 @@
+# Copyright 2019 The Meson development team
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+#     http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from collections import namedtuple
+from .. import mesonlib
+from ..mesonlib import listify
+from . import ExtensionModule
+from ..interpreterbase import (
+    noPosargs, noKwargs, permittedKwargs,
+    InterpreterObject, MutableInterpreterObject, ObjectHolder,
+    InterpreterException, InvalidArguments, InvalidCode, FeatureNew,
+)
+from ..interpreter import (
+    GeneratedListHolder, CustomTargetHolder,
+    CustomTargetIndexHolder
+)
+
+SourceSetRule = namedtuple('SourceSetRule', 'keys sources if_false sourcesets dependencies extra_deps')
+SourceFiles = namedtuple('SourceFiles', 'sources dependencies')
+
+class SourceSetHolder(MutableInterpreterObject, ObjectHolder):
+    def __init__(self, environment, subdir):
+        MutableInterpreterObject.__init__(self)
+        ObjectHolder.__init__(self, list())
+        self.environment = environment
+        self.subdir = subdir
+        self.frozen = False
+        self.methods.update({
+            'add': self.add_method,
+            'add_all': self.add_all_method,
+            'all_sources': self.all_sources_method,
+            'apply': self.apply_method,
+        })
+
+    def check_source_files(self, arg, allow_deps):
+        sources = []
+        deps = []
+        for x in arg:
+            if isinstance(x, (str, mesonlib.File,
+                              GeneratedListHolder, CustomTargetHolder,
+                              CustomTargetIndexHolder)):
+                sources.append(x)
+            elif hasattr(x, 'found'):
+                if not allow_deps:
+                    msg = 'Dependencies are not allowed in the if_false argument.'
+                    raise InvalidArguments(msg)
+                deps.append(x)
+            else:
+                msg = 'Sources must be strings or file-like objects.'
+                raise InvalidArguments(msg)
+        mesonlib.check_direntry_issues(sources)
+        return sources, deps
+
+    def check_conditions(self, arg):
+        keys = []
+        deps = []
+        for x in listify(arg):
+            if isinstance(x, str):
+                keys.append(x)
+            elif hasattr(x, 'found'):
+                deps.append(x)
+            else:
+                raise InvalidArguments('Conditions must be strings or dependency object')
+        return keys, deps
+
+    @permittedKwargs(['when', 'if_false', 'if_true'])
+    def add_method(self, args, kwargs):
+        if self.frozen:
+            raise InvalidCode('Tried to use \'add\' after querying the source set')
+        when = listify(kwargs.get('when', []))
+        if_true = listify(kwargs.get('if_true', []))
+        if_false = listify(kwargs.get('if_false', []))
+        if not when and not if_true and not if_false:
+            if_true = args
+        elif args:
+            raise InterpreterException('add called with both positional and keyword arguments')
+        keys, dependencies = self.check_conditions(when)
+        sources, extra_deps = self.check_source_files(if_true, True)
+        if_false, _ = self.check_source_files(if_false, False)
+        self.held_object.append(SourceSetRule(keys, sources, if_false, [], dependencies, extra_deps))
+
+    @permittedKwargs(['when', 'if_true'])
+    def add_all_method(self, args, kwargs):
+        if self.frozen:
+            raise InvalidCode('Tried to use \'add_all\' after querying the source set')
+        when = listify(kwargs.get('when', []))
+        if_true = listify(kwargs.get('if_true', []))
+        if not when and not if_true:
+            if_true = args
+        elif args:
+            raise InterpreterException('add_all called with both positional and keyword arguments')
+        keys, dependencies = self.check_conditions(when)
+        for s in if_true:
+            if not isinstance(s, SourceSetHolder):
+                raise InvalidCode('Arguments to \'add_all\' after the first must be source sets')
+            s.frozen = True
+        self.held_object.append(SourceSetRule(keys, [], [], if_true, dependencies, []))
+
+    def collect(self, enabled_fn, all_sources, into=None):
+        if not into:
+            into = SourceFiles(set(), set())
+        for entry in self.held_object:
+            if all(x.found() for x in entry.dependencies) and \
+               all(enabled_fn(key) for key in entry.keys):
+                into.sources.update(entry.sources)
+                into.dependencies.update(entry.dependencies)
+                into.dependencies.update(entry.extra_deps)
+                for ss in entry.sourcesets:
+                    ss.collect(enabled_fn, all_sources, into)
+                if not all_sources:
+                    continue
+            into.sources.update(entry.if_false)
+        return into
+
+    @noKwargs
+    @noPosargs
+    def all_sources_method(self, args, kwargs):
+        self.frozen = True
+        files = self.collect(lambda x: True, True)
+        return list(files.sources)
+
+    @permittedKwargs(['strict'])
+    def apply_method(self, args, kwargs):
+        if len(args) != 1:
+            raise InterpreterException('Apply takes exactly one argument')
+        config_data = args[0]
+        self.frozen = True
+        strict = kwargs.get('strict', True)
+        if isinstance(config_data, dict):
+            def _get_from_config_data(key):
+                if strict and key not in config_data:
+                    raise InterpreterException('Entry %s not in configuration dictionary.' % key)
+                return config_data.get(key, False)
+        else:
+            config_cache = dict()
+
+            def _get_from_config_data(key):
+                nonlocal config_cache
+                if key not in config_cache:
+                    args = [key] if strict else [key, False]
+                    config_cache[key] = config_data.get_method(args, {})
+                return config_cache[key]
+
+        files = self.collect(_get_from_config_data, False)
+        res = SourceFilesHolder(files)
+        return res
+
+class SourceFilesHolder(InterpreterObject, ObjectHolder):
+    def __init__(self, files):
+        InterpreterObject.__init__(self)
+        ObjectHolder.__init__(self, files)
+        self.methods.update({
+            'sources': self.sources_method,
+            'dependencies': self.dependencies_method,
+        })
+
+    @noPosargs
+    @noKwargs
+    def sources_method(self, args, kwargs):
+        return list(self.held_object.sources)
+
+    @noPosargs
+    @noKwargs
+    def dependencies_method(self, args, kwargs):
+        return list(self.held_object.dependencies)
+
+class SourceSetModule(ExtensionModule):
+    @FeatureNew('SourceSet module', '0.51.0')
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.snippets.add('source_set')
+
+    @noKwargs
+    @noPosargs
+    def source_set(self, interpreter, state, args, kwargs):
+        return SourceSetHolder(interpreter.environment, interpreter.subdir)
+
+def initialize(*args, **kwargs):
+    return SourceSetModule(*args, **kwargs)

test cases/common/220 source set configuration_data/a.c

@@ -0,0 +1,8 @@
+#include <stdlib.h>
+#include "all.h"
+
+int main(void)
+{
+    if (p) abort();
+    f();
+}

test cases/common/220 source set configuration_data/all.h

@@ -0,0 +1,7 @@
+extern void f(void);
+extern void g(void);
+extern void h(void);
+extern void undefined(void);
+
+/* No extern here to get a common symbol */
+void (*p)(void);

test cases/common/220 source set configuration_data/f.c

@@ -0,0 +1,5 @@
+#include "all.h"
+
+void f(void)
+{
+}

test cases/common/220 source set configuration_data/g.c

@@ -0,0 +1,6 @@
+#include "all.h"
+
+void g(void)
+{
+    h();
+}

test cases/common/220 source set configuration_data/meson.build

@@ -0,0 +1,54 @@
+project('a', 'c')
+
+good = declare_dependency(link_with: static_library('good', 'g.c'))
+bad = declare_dependency(link_args: 'nonexistent.a')
+not_found = dependency('invalid', required: false)
+
+source_set = import('sourceset')
+
+sources = source_set.source_set()
+sources.add(when: 'YES',  if_false: ['nope.c'])
+sources.add(when: 'YES1', if_true: files('a.c'))
+subdir('subdir')
+sources.add(when: 'NO',   if_true: 'nope.c', if_false: ['f.c'])
+sources.add(when: 'NO',   if_true: bad,      if_false: ['f.c'])
+
+sources.add(when: 'YES2', if_true: good)
+
+# dependencies as conditions
+sources.add(when: not_found, if_true: 'nope.c')
+
+# test add_all
+sources2 = source_set.source_set()
+sources2.add(when: 'YES1',  if_true: 'nope.c')
+sources.add_all(when: 'NO', if_true: sources2)
+
+# test duplicate items
+sources.add(when: 'YES1',   if_true: files('a.c'))
+
+conf1 = configuration_data()
+conf1.set10('YES', true)
+conf1.set10('YES1', true)
+conf1.set10('YES2', false)
+conf1.set10('NO', false)
+result1 = sources.apply(conf1)
+
+conf2 = configuration_data()
+conf2.set10('YES', true)
+conf2.set10('YES1', false)
+conf2.set10('YES2', true)
+conf2.set10('NO', false)
+result2 = sources.apply(conf2)
+
+# Each target will recompile the objects
+executable('first', sources: result1.sources(), dependencies: result1.dependencies())
+executable('second', sources: result2.sources(), dependencies: result2.dependencies())
+
+# All target will use the same object files
+if meson.is_unity()
+  message('Skipping extraction test because this is a Unity build.')
+else
+  all_objs = static_library('all_objs', sources.all_sources())
+  executable('first_via_lib', objects: all_objs.extract_objects(result1.sources()), dependencies: result1.dependencies())
+  executable('second_via_lib', objects: all_objs.extract_objects(result2.sources()), dependencies: result2.dependencies())
+endif

test cases/common/220 source set configuration_data/nope.c

@@ -0,0 +1,3 @@
+#include "all.h"
+
+void (*p)(void) = undefined;

test cases/common/220 source set configuration_data/subdir/b.c

@@ -0,0 +1,13 @@
+#include <stdlib.h>
+#include "all.h"
+
+void h(void)
+{
+}
+
+int main(void)
+{
+    if (p) abort();
+    f();
+    g();
+}

test cases/common/220 source set configuration_data/subdir/meson.build

@@ -0,0 +1 @@
+sources.add(when: ['YES2', good], if_true: [ files('b.c') ])

test cases/common/221 source set dictionary/a.c

@@ -0,0 +1,8 @@
+#include <stdlib.h>
+#include "all.h"
+
+int main(void)
+{
+    if (p) abort();
+    f();
+}

test cases/common/221 source set dictionary/all.h

@@ -0,0 +1,7 @@
+extern void f(void);
+extern void g(void);
+extern void h(void);
+extern void undefined(void);
+
+/* No extern here to get a common symbol */
+void (*p)(void);

test cases/common/221 source set dictionary/f.c

@@ -0,0 +1,5 @@
+#include "all.h"
+
+void f(void)
+{
+}

test cases/common/221 source set dictionary/g.c

@@ -0,0 +1,6 @@
+#include "all.h"
+
+void g(void)
+{
+    h();
+}

test cases/common/221 source set dictionary/meson.build

@@ -0,0 +1,56 @@
+project('a', 'c')
+
+good = declare_dependency(link_with: static_library('good', 'g.c'))
+bad = declare_dependency(link_args: 'nonexistent.a')
+not_found = dependency('invalid', required: false)
+
+source_set = import('sourceset')
+
+sources = source_set.source_set()
+sources.add(when: 'YES',  if_false: ['nope.c'])
+sources.add(when: 'YES1', if_true: files('a.c'))
+subdir('subdir')
+sources.add(when: 'NO',   if_true: 'nope.c', if_false: ['f.c'])
+sources.add(when: 'NO',   if_true: bad,      if_false: ['f.c'])
+
+sources.add(when: 'YES2', if_true: good)
+
+# dependencies as conditions
+sources.add(when: not_found, if_true: 'nope.c')
+
+# test add_all
+sources2 = source_set.source_set()
+sources2.add(when: 'YES1',  if_true: 'nope.c')
+sources.add_all(when: 'NO', if_true: sources2)
+
+# test duplicate items
+sources.add(when: 'YES1',   if_true: files('a.c'))
+
+conf1 = {
+    'YES': true,
+    'YES1': true,
+    'YES2': false,
+    'NO': false,
+}
+result1 = sources.apply(conf1)
+
+conf2 = {
+    'YES': true,
+    'YES1': false,
+    'YES2': true,
+    'NO': false,
+}
+result2 = sources.apply(conf2)
+
+# Each target will recompile the objects
+executable('first', sources: result1.sources(), dependencies: result1.dependencies())
+executable('second', sources: result2.sources(), dependencies: result2.dependencies())
+
+# All target will use the same object files
+if meson.is_unity()
+  message('Skipping extraction test because this is a Unity build.')
+else
+  all_objs = static_library('all_objs', sources.all_sources())
+  executable('first_via_lib', objects: all_objs.extract_objects(result1.sources()), dependencies: result1.dependencies())
+  executable('second_via_lib', objects: all_objs.extract_objects(result2.sources()), dependencies: result2.dependencies())
+endif

test cases/common/221 source set dictionary/nope.c

@@ -0,0 +1,3 @@
+#include "all.h"
+
+void (*p)(void) = undefined;

test cases/common/221 source set dictionary/subdir/b.c

@@ -0,0 +1,13 @@
+#include <stdlib.h>
+#include "all.h"
+
+void h(void)
+{
+}
+
+int main(void)
+{
+    if (p) abort();
+    f();
+    g();
+}

test cases/common/221 source set dictionary/subdir/meson.build

@@ -0,0 +1 @@
+sources.add(when: ['YES2', good], if_true: [ files('b.c') ])

test cases/common/222 source set custom target/a.c

@@ -0,0 +1,7 @@
+#include "all.h"
+
+int main(void)
+{
+    f();
+    g();
+}

test cases/common/222 source set custom target/all.h

@@ -0,0 +1,2 @@
+extern void f(void);
+extern void g(void);

test cases/common/222 source set custom target/cp.py

@@ -0,0 +1,5 @@
+#! /usr/bin/env python3
+
+import sys
+from shutil import copyfile
+copyfile(*sys.argv[1:])

test cases/common/222 source set custom target/f.c

@@ -0,0 +1,5 @@
+#include "all.h"
+
+void f(void)
+{
+}

test cases/common/222 source set custom target/g.c

@@ -0,0 +1,5 @@
+#include "all.h"
+
+void g(void)
+{
+}

test cases/common/222 source set custom target/meson.build

@@ -0,0 +1,28 @@
+# Try using sourceset with various kinds of generated sources
+
+project('a', 'c')
+
+cp = find_program('cp.py')
+
+source_set = import('sourceset')
+sources = source_set.source_set()
+
+a_c = custom_target('gen-custom-target',
+                    input: 'a.c', output: 'out_a.c',
+                    command: [cp, '@INPUT@', '@OUTPUT@'])
+sources.add(when: 'YES', if_true: a_c)
+sources.add(when: 'YES', if_true: a_c[0])
+
+f_c = configure_file(input: 'f.c', output: 'out_f.c', copy: true)
+sources.add(when: 'YES', if_true: f_c)
+sources.add(when: 'YES', if_true: f_c)
+
+gen = generator(cp, output: 'out_@PLAINNAME@', arguments: ['@INPUT@', '@OUTPUT@'])
+g_c = gen.process(files('g.c'))
+sources.add(when: 'YES', if_true: g_c)
+sources.add(when: 'YES', if_true: g_c)
+
+conf1 = { 'YES': true, }
+result1 = sources.apply(conf1)
+
+executable('first', sources: result1.sources(), dependencies: result1.dependencies())