Merge pull request #3893 from FFY00/master

Add dlang module (dub support)

ciimage/Dockerfile

@@ -12,7 +12,9 @@ RUN apt-get -y update && apt-get -y upgrade \
 && apt-get -y install qt4-linguist-tools \
 && apt-get -y install python-dev \
 && apt-get -y install libomp-dev openssh-client \
-&& python3 -m pip install hotdoc codecov
+&& python3 -m pip install hotdoc codecov \
+&& dub fetch urld \
+&& dub build urld --compiler=gdc
 
 # OpenSSH client is needed to run openmpi binaries.
 

docs/markdown/D.md

@@ -5,7 +5,8 @@ short-description: Compiling D sources
 
 # Compiling D applications
 
-Meson has support for compiling D programs. A minimal `meson.build` file for D looks like this:
+Meson has support for compiling D programs. A minimal `meson.build`
+file for D looks like this:
 
 ```meson
 project('myapp', 'd')
@@ -15,8 +16,8 @@ executable('myapp', 'app.d')
 
 ## Compiling different versions
 
-If you are using the [version()](https://dlang.org/spec/version.html) feature for conditional compilation, you can use it using the `d_module_versions`
+If you are using the [version()](https://dlang.org/spec/version.html) feature for conditional compilation,
-target property:
+you can use it using the `d_module_versions` target property:
 ```meson
 project('myapp', 'd')
 executable('myapp', 'app.d', d_module_versions: ['Demo', 'FeatureA'])
@@ -24,10 +25,14 @@ executable('myapp', 'app.d', d_module_versions: ['Demo', 'FeatureA'])
 
 ## Using embedded unittests
 
-If you are using embedded [unittest functions](https://dlang.org/spec/unittest.html), your source code needs to be compiled twice, once in regular
+If you are using embedded [unittest functions](https://dlang.org/spec/unittest.html), your source code needs
-mode, and once with unittests active. This is done by setting the `d_unittest` target property to `true`.
+to be compiled twice, once in regular
-Meson will only ever pass the respective compiler's `-unittest` flag, and never have the compiler generate an empty main function.
+mode, and once with unittests active. This is done by setting the
-If you need that feature in a portable way, create an empty `main()` function for unittests yourself, since the GNU D compiler
+`d_unittest` target property to `true`.
+Meson will only ever pass the respective compiler's `-unittest` flag,
+and never have the compiler generate an empty main function.
+If you need that feature in a portable way, create an empty `main()`
+function for unittests yourself, since the GNU D compiler
 does not have this feature.
 
 This is an example for using D unittests with Meson:
@@ -43,8 +48,10 @@ test('myapptest', test_exe)
 
 # Compiling D libraries and installing them
 
-Building D libraries is a straightforward process, not different from how C libraries are built in Meson. You should generate a pkg-config file
+Building D libraries is a straightforward process, not different from
-and install it, in order to make other software on the system find the dependency once it is installed.
+how C libraries are built in Meson. You should generate a pkg-config
+file and install it, in order to make other software on the system
+find the dependency once it is installed.
 
 This is an example on how to build a D shared library:
 ```meson
@@ -71,12 +78,17 @@ pkgc.generate(name: 'mylib',
 install_subdir('src/mylib/', install_dir: 'include/d/mylib/')
 ```
 
-It is important to make the D sources install in a subdirectory in the include path, in this case `/usr/include/d/mylib/mylib`.
+It is important to make the D sources install in a subdirectory in the
-All D compilers include the `/usr/include/d` directory by default, and if your library would be installed into `/usr/include/d/mylib`, there
+ include path, in this case `/usr/include/d/mylib/mylib`.
-is a high chance that, when you compile your project again on a machine where you installed it, the compiler will prefer the old installed include over
+All D compilers include the `/usr/include/d` directory by default, and
-the new version in the source tree, leading to very confusing errors.
+ if your library would be installed into `/usr/include/d/mylib`, there
-
+is a high chance that, when you compile your project again on a
-This is an example of how to use the D library we just built and installed in an application:
+machine where you installed it, the compiler will prefer the old
+installed include over the new version in the source tree, leading to
+very confusing errors.
+
+This is an example of how to use the D library we just built and
+installed in an application:
 ```meson
 project('myapp', 'd')
 
@@ -85,5 +97,15 @@ myapp_src = ['app.d', 'alpha.d', 'beta.d']
 executable('myapp', myapp_src, dependencies: [mylib_dep])
 ```
 
-Please keep in mind that the library and executable would both need to be built with the exact same D compiler and D compiler version. The D ABI is not
+Please keep in mind that the library and executable would both need to
-stable across compilers and their versions, and mixing compilers will lead to problems.
+be built with the exact same D compiler and D compiler version. The D
+ABI is not stable across compilers and their versions, and mixing
+compilers will lead to problems.
+
+# Integrating with DUB
+
+DUB is a fully integrated build system for D, but it is also a way to
+provide dependencies. Adding dependencies from the [D package registry](https://code.dlang.org/)
+is pretty straight forward. You can find how to do this in
+[Dependencies](Dependencies.md#Dub). You can also automatically
+generate a `dub.json` file as explained in [Dlang](Dlang-module.md#generatedubfile).

docs/markdown/Dependencies.md

@@ -109,6 +109,38 @@ object. Since they can be used interchangeably, the rest of the build
 definitions do not need to care which one it is. Meson will take care
 of all the work behind the scenes to make this work.
 
+# Dependency method
+
+You can use the keyword `method` to let meson know what method to use
+when searching for the dependency. The default value is `auto`.
+Aditional dependencies methods are `pkg-config`, `config-tool`,
+`system`, `sysconfig`, `qmake`, `extraframework` and `dub`.
+
+```meson
+cups_dep = dependency('cups', method : 'pkg-config')
+```
+
+### Some notes on Dub
+
+Please understand that meson is only able to find dependencies that
+exist in the local Dub repository. You need to manually fetch and
+build the target dependencies.
+
+For `urld`.
+```
+dub fetch urld
+dub build urld
+```
+
+Other thing you need to keep in mind is that both meson and Dub need
+to be using the same compiler. This can be achieved using Dub's
+`-compiler` argument and/or manually setting the `DC` environment
+variable when running meson.
+```
+dub build urld --compiler=dmd
+DC="dmd" meson builddir
+```
+
 # Dependencies with custom lookup functionality
 
 Some dependencies have specific detection logic.
@@ -280,7 +312,7 @@ The `language` keyword may used.
 
 Python3 is handled specially by meson:
 1. Meson tries to use `pkg-config`.
-1. If `pkg-config` fails meson uses a fallback:
+2. If `pkg-config` fails meson uses a fallback:
     - On Windows the fallback is the current `python3` interpreter.
     - On OSX the fallback is a framework dependency from `/Library/Frameworks`.
 

docs/markdown/Dlang-module.md

@@ -0,0 +1,43 @@
+# Dlang module
+
+This module provides tools related to the D programming language.
+
+## Usage
+
+To use this module, just do: **`dlang = import('dlang')`**.
+You can, of course, replace the name `dlang` with anything else.
+
+The module only exposes one fucntion, `generate_dub_file`, used to
+automatically generate Dub configuration files.
+
+### generate_dub_file()
+This method only has two required arguments, the project name and the
+source folder. You can pass other arguments with additional keywords,
+they will be automatically translated to json and added to the
+`dub.json` file.
+
+**Structure**
+```meson
+generate_dub_file("project name", "source/folder", key: "value" ...)
+```
+
+**Example**
+```meson
+dlang = import('dlang')
+dlang.generate_dub_file(meson.project_name().to_lower(), meson.source_root(),
+                        authors: 'Meson Team',
+                        description: 'Test executable',
+                        copyright: 'Copyright © 2018, Meson Team',
+                        license: 'MIT',
+                        sourceFiles: 'test.d',
+                        targetType: 'executable',
+                        dependencies: my_dep
+)
+```
+
+You can manually edit a meson generated `dub.json` file or provide a
+initial one. The module will only update the values specified in
+`generate_dub_file()`.
+
+Although not required, you will need to have a `description` and
+`license` if you want to publish the package in the [D package registry](https://code.dlang.org/).

docs/sitemap.txt

@@ -40,6 +40,7 @@ index.md
 			RPM-module.md
 			Simd-module.md
 			Windows-module.md
+			Dlang-module.md
 		Java.md
 		Vala.md
 		D.md

mesonbuild/dependencies/base.py

@@ -19,6 +19,7 @@ import copy
 import os
 import re
 import stat
+import json
 import shlex
 import shutil
 import textwrap
@@ -31,7 +32,6 @@ from ..compilers import clib_langs
 from ..mesonlib import MesonException, OrderedSet
 from ..mesonlib import Popen_safe, version_compare_many, version_compare, listify
 
-
 # These must be defined in this file to avoid cyclical references.
 packages = {}
 _packages_accept_language = set()
@@ -59,6 +59,8 @@ class DependencyMethods(Enum):
     CUPSCONFIG = 'cups-config'
     PCAPCONFIG = 'pcap-config'
     LIBWMFCONFIG = 'libwmf-config'
+    # Misc
+    DUB = 'dub'
 
 
 class Dependency:
@@ -790,6 +792,162 @@ class PkgConfigDependency(ExternalDependency):
         # a path rather than the raw dlname
         return os.path.basename(dlname)
 
+class DubDependency(ExternalDependency):
+    class_dubbin = None
+
+    def __init__(self, name, environment, kwargs):
+        super().__init__('dub', environment, 'd', kwargs)
+        self.name = name
+        self.compiler = super().get_compiler()
+
+        if 'required' in kwargs:
+            self.required = kwargs.get('required')
+
+        if DubDependency.class_dubbin is None:
+            self.dubbin = self.check_dub()
+            DubDependency.class_dubbin = self.dubbin
+        else:
+            self.dubbin = DubDependency.class_dubbin
+
+        if not self.dubbin:
+            if self.required:
+                raise DependencyException('DUB not found.')
+            self.is_found = False
+            mlog.log('Dependency', mlog.bold(name), 'found:', mlog.red('NO'))
+            return
+
+        mlog.debug('Determining dependency {!r} with DUB executable '
+                   '{!r}'.format(name, self.dubbin.get_path()))
+
+        # Ask dub for the package
+        ret, res = self._call_dubbin(['describe', name])
+
+        if ret != 0:
+            if self.required:
+                raise DependencyException('Dependency {!r} not found'.format(name))
+            self.is_found = False
+            mlog.log('Dependency', mlog.bold(name), 'found:', mlog.red('NO'))
+            return
+
+        j = json.loads(res)
+        comp = self.compiler.get_id().replace('llvm', 'ldc').replace('gcc', 'gdc')
+        for package in j['packages']:
+            if package['name'] == name:
+                if j['compiler'] != comp:
+                    msg = ['Dependency', mlog.bold(name), 'found but it was compiled with']
+                    msg += [mlog.bold(j['compiler']), 'and we are using', mlog.bold(comp)]
+                    mlog.error(*msg)
+                    if self.required:
+                        raise DependencyException('Dependency {!r} not found'.format(name))
+                    self.is_found = False
+                    mlog.log('Dependency', mlog.bold(name), 'found:', mlog.red('NO'))
+                    return
+
+                self.version = package['version']
+                self.pkg = package
+                break
+
+        # Check if package version meets the requirements
+        found_msg = ['Dependency', mlog.bold(name), 'found:']
+        if self.version_reqs is None:
+            self.is_found = True
+        else:
+            if not isinstance(self.version_reqs, (str, list)):
+                raise DependencyException('Version argument must be string or list.')
+            if isinstance(self.version_reqs, str):
+                self.version_reqs = [self.version_reqs]
+            (self.is_found, not_found, found) = \
+                version_compare_many(self.version, self.version_reqs)
+            if not self.is_found:
+                found_msg += [mlog.red('NO'),
+                              'found {!r} but need:'.format(self.version),
+                              ', '.join(["'{}'".format(e) for e in not_found])]
+                if found:
+                    found_msg += ['; matched:',
+                                  ', '.join(["'{}'".format(e) for e in found])]
+                if not self.silent:
+                    mlog.log(*found_msg)
+                if self.required:
+                    m = 'Invalid version of dependency, need {!r} {!r} found {!r}.'
+                    raise DependencyException(m.format(name, not_found, self.version))
+                return
+
+        found_msg += [mlog.green('YES'), self.version]
+
+        if self.pkg['targetFileName'].endswith('.a'):
+            self.static = True
+
+        self.compile_args = []
+        for flag in self.pkg['dflags']:
+            self.link_args.append(flag)
+        for path in self.pkg['importPaths']:
+            self.compile_args.append('-I' + os.path.join(self.pkg['path'], path))
+
+        self.link_args = []
+        for flag in self.pkg['lflags']:
+            self.link_args.append(flag)
+
+        search_paths = []
+        search_paths.append(os.path.join(self.pkg['path'], self.pkg['targetPath']))
+        found, res = self.__search_paths(search_paths, self.pkg['targetFileName'])
+        for file in res:
+            self.link_args.append(file)
+
+        if not found:
+            if self.required:
+                raise DependencyException('Dependency {!r} not found'.format(name))
+                self.is_found = False
+                mlog.log('Dependency', mlog.bold(name), 'found:', mlog.red('NO'))
+                return
+
+        if not self.silent:
+            mlog.log(*found_msg)
+
+    def get_compiler(self):
+        return self.compiler
+
+    def __search_paths(self, search_paths, target_file):
+        found = False
+        res = []
+        if target_file == '':
+            return True, res
+        for path in search_paths:
+                if os.path.isdir(path):
+                    for file in os.listdir(path):
+                        if file == target_file:
+                            res.append(os.path.join(path, file))
+                            found = True
+        return found, res
+
+    def _call_dubbin(self, args, env=None):
+        p, out = Popen_safe(self.dubbin.get_command() + args, env=env)[0:2]
+        return p.returncode, out.strip()
+
+    def check_dub(self):
+        dubbin = ExternalProgram('dub', silent=True)
+        if dubbin.found():
+            try:
+                p, out = Popen_safe(dubbin.get_command() + ['--version'])[0:2]
+                if p.returncode != 0:
+                    mlog.warning('Found dub {!r} but couldn\'t run it'
+                                 ''.format(' '.join(dubbin.get_command())))
+                    # Set to False instead of None to signify that we've already
+                    # searched for it and not found it
+                    dubbin = False
+            except (FileNotFoundError, PermissionError):
+                dubbin = False
+        else:
+            dubbin = False
+        if dubbin:
+            mlog.log('Found DUB:', mlog.bold(dubbin.get_path()),
+                     '(%s)' % out.strip())
+        else:
+            mlog.log('Found DUB:', mlog.red('NO'))
+        return dubbin
+
+    @staticmethod
+    def get_methods():
+        return [DependencyMethods.PKGCONFIG, DependencyMethods.DUB]
 
 class ExternalProgram:
     windows_exts = ('exe', 'msc', 'com', 'bat', 'cmd')
@@ -1097,6 +1255,7 @@ def find_external_dependency(name, env, kwargs):
         raise DependencyException('Keyword "required" must be a boolean.')
     if not isinstance(kwargs.get('method', ''), str):
         raise DependencyException('Keyword "method" must be a string.')
+    method = kwargs.get('method', '')
     lname = name.lower()
     if lname in packages:
         if lname not in _packages_accept_language and 'language' in kwargs:
@@ -1113,6 +1272,11 @@ def find_external_dependency(name, env, kwargs):
     if 'language' in kwargs:
         # Remove check when PkgConfigDependency supports language.
         raise DependencyException('%s dependency does not accept "language" keyword argument' % (lname, ))
+    if 'dub' == method:
+        dubdep = DubDependency(name, env, kwargs)
+        if required and not dubdep.found():
+            mlog.log('Dependency', mlog.bold(name), 'found:', mlog.red('NO'))
+        return dubdep
     pkg_exc = None
     pkgdep = None
     try:

mesonbuild/interpreter.py

@@ -376,6 +376,7 @@ class DependencyHolder(InterpreterObject, ObjectHolder):
         self.methods.update({'found': self.found_method,
                              'type_name': self.type_name_method,
                              'version': self.version_method,
+                             'name': self.name_method,
                              'get_pkgconfig_variable': self.pkgconfig_method,
                              'get_configtool_variable': self.configtool_method,
                              'partial_dependency': self.partial_dependency_method,
@@ -398,6 +399,11 @@ class DependencyHolder(InterpreterObject, ObjectHolder):
     def version_method(self, args, kwargs):
         return self.held_object.get_version()
 
+    @noPosargs
+    @permittedKwargs({})
+    def name_method(self, args, kwargs):
+        return self.held_object.get_name()
+
     @permittedKwargs({'define_variable'})
     def pkgconfig_method(self, args, kwargs):
         args = listify(args)

mesonbuild/modules/dlang.py

@@ -0,0 +1,141 @@
+# Copyright 2018 The Meson development team
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+#     http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file contains the detection logic for external dependencies that
+# are UI-related.
+
+import json
+import os
+
+from . import ExtensionModule
+
+from .. import mlog
+
+from ..mesonlib import (
+    Popen_safe, MesonException
+)
+
+from ..dependencies.base import (
+    ExternalProgram, DubDependency
+)
+
+from ..interpreter import DependencyHolder
+
+class DlangModule(ExtensionModule):
+    class_dubbin = None
+    init_dub = False
+
+    def __init__(self, interpreter):
+        super().__init__(interpreter)
+        self.snippets.add('generate_dub_file')
+
+    def _init_dub(self):
+        if DlangModule.class_dubbin is None:
+            self.dubbin = DubDependency.class_dubbin
+            DlangModule.class_dubbin = self.dubbin
+        else:
+            self.dubbin = DlangModule.class_dubbin
+
+        if DlangModule.class_dubbin is None:
+            self.dubbin = self.check_dub()
+            DlangModule.class_dubbin = self.dubbin
+        else:
+            self.dubbin = DlangModule.class_dubbin
+
+        if not self.dubbin:
+            if not self.dubbin:
+                raise MesonException('DUB not found.')
+
+    def generate_dub_file(self, interpreter, state, args, kwargs):
+        if not DlangModule.init_dub:
+            self._init_dub()
+
+        if len(args) < 2:
+            raise MesonException('Missing arguments')
+
+        config = {
+            'name': args[0]
+        }
+
+        config_path = os.path.join(args[1], 'dub.json')
+        if os.path.exists(config_path):
+            with open(config_path, 'r', encoding='utf8') as ofile:
+                try:
+                    config = json.load(ofile)
+                except ValueError:
+                    mlog.warning('Failed to load the data in dub.json')
+
+        warn_publishing = ['description', 'license']
+        for arg in warn_publishing:
+            if arg not in kwargs and \
+               arg not in config:
+                mlog.warning('Without', mlog.bold(arg), 'the DUB package can\'t be published')
+
+        for key, value in kwargs.items():
+            if key == 'dependencies':
+                config[key] = {}
+                if isinstance(value, list):
+                    for dep in value:
+                        if isinstance(dep, DependencyHolder):
+                            name = dep.method_call('name', [], [])
+                            ret, res = self._call_dubbin(['describe', name])
+                            if ret == 0:
+                                version = dep.method_call('version', [], [])
+                                if version is None:
+                                    config[key][name] = ''
+                                else:
+                                    config[key][name] = version
+                elif isinstance(value, DependencyHolder):
+                    name = value.method_call('name', [], [])
+                    ret, res = self._call_dubbin(['describe', name])
+                    if ret == 0:
+                        version = value.method_call('version', [], [])
+                        if version is None:
+                            config[key][name] = ''
+                        else:
+                            config[key][name] = version
+            else:
+                config[key] = value
+
+        with open(config_path, 'w', encoding='utf8') as ofile:
+            ofile.write(json.dumps(config, indent=4, ensure_ascii=False))
+
+    def _call_dubbin(self, args, env=None):
+        p, out = Popen_safe(self.dubbin.get_command() + args, env=env)[0:2]
+        return p.returncode, out.strip()
+
+    def check_dub(self):
+        dubbin = ExternalProgram('dub', silent=True)
+        if dubbin.found():
+            try:
+                p, out = Popen_safe(dubbin.get_command() + ['--version'])[0:2]
+                if p.returncode != 0:
+                    mlog.warning('Found dub {!r} but couldn\'t run it'
+                                 ''.format(' '.join(dubbin.get_command())))
+                    # Set to False instead of None to signify that we've already
+                    # searched for it and not found it
+                    dubbin = False
+            except (FileNotFoundError, PermissionError):
+                dubbin = False
+        else:
+            dubbin = False
+        if dubbin:
+            mlog.log('Found DUB:', mlog.bold(dubbin.get_path()),
+                     '(%s)' % out.strip())
+        else:
+            mlog.log('Found DUB:', mlog.red('NO'))
+        return dubbin
+
+def initialize(*args, **kwargs):
+    return DlangModule(*args, **kwargs)

test cases/d/11 dub/meson.build

@@ -0,0 +1,23 @@
+project('dub-example', 'd')
+
+dub_exe = find_program('dub', required : false)
+if not dub_exe.found()
+  error('MESON_SKIP_TEST: Dub not found')
+endif
+
+urld_dep = dependency('urld', method: 'dub')
+
+test_exe = executable('test-urld', 'test.d', dependencies: urld_dep)
+test('test urld', test_exe)
+
+# If you want meson to generate/update a dub.json file
+dlang = import('dlang')
+dlang.generate_dub_file(meson.project_name().to_lower(), meson.source_root(),
+                        authors: 'Meson Team',
+                        description: 'Test executable',
+                        copyright: 'Copyright © 2018, Meson Team',
+                        license: 'MIT',
+                        sourceFiles: 'test.d',
+                        targetType: 'executable',
+                        dependencies: urld_dep
+)

test cases/d/11 dub/test.d

@@ -0,0 +1,14 @@
+import std.stdio;
+import url;
+
+void main() {
+    URL url;
+    with (url) {
+        scheme = "soap.beep";
+        host = "beep.example.net";
+        port = 1772;
+        path = "/serverinfo/info";
+    queryParams.add("token", "my-api-token");
+    }
+    writeln(url);
+}