We have two copies of this code, and the python module one is vastly
superior, not just because it allows choosing which python executable to
base itself on. Unify this. Fixes various issues including non-Windows
support for sysconfig, and pypy edge cases.
In preparation for wholly merging the dependency handling from the
python module into dependencies.*, move the unique class definitions
from there into their new home in dependencies.python, which is
semantically convenient.
In preparation for handling more work inside dependencies.*, we need to
be able to run a PythonExternalProgram from the python dependency. Move
most of the definition -- but only the parts that have no interest in a
ModuleState -- and subclass a bit of sanity checking that we need to
handle specially when used in the module.
It can go directly inside the function which immediately uses it.
There's no purpose in looking it up exactly once and using it exactly
once, but looking it up outside the function and complicating the
function signature in order to pass it as a function argument.
We write this out as an embedded string to a tempfile in order to run
it, which is pretty awkward. And usually Meson's files are already files
on disk, not packed into a zip, so we can simply run it directly. Since
python 3.7, which is our new minimum, we can handle this well via the
stdlib. (There's also mesonbuild.mesondata, but we do not need
persistence in the builddir.)
This also solves the problem that has always been there, of giant python
programs inside strings occasionally confusing syntax highlighters. Or
even, it would be nice if we had syntax highlighting for this
introspection program. :D
We pass around a PythonInstallation into the depths of the dependency
factory, solely so that we can get at is_pypy in one particular branch.
We don't need the DSL functions, we don't need access to the
interpreter, let's just use the enhanced ExternalProgram object on its
own.
Add is_pypy to the object, and modify other references to get
information from .info['...'] instead of direct access.
When finding a py.dependency() we try to use pkg-config. We then apply
our own custom base class, which replaces self.name with the informative
comment "override the name from the "real" dependency lookup", to which
I can only say "uhhh why". Why do we want to do that???
It turns out we don't, it was just a really old legacy design because we
had a SystemDependency with a .pkgdep attribute hiding the real
dependency bizarro-land style. We cleaned that up in commit
4d67dd19e5 and as part of that, we
*shifted over* the self.name assignment to preserve the visible effects,
sort of. We didn't have a *reason* to override the name, we just did it
because... we weren't sure whether it mattered.
Unfortunately it very much does matter the other way -- we don't want
it. We can pass this dependency to the pkgconfig module, which uses the
name attribute to fill out the `Requires: ` field. Also, the name should
name what we have. :p
Get rid of this bizarre historic quirk. Since we have proper
dependencies here, we should go all in.
Fixes https://github.com/ufo-kit/ufo-core/pull/185#issuecomment-1328224996
Which adds the `use-set-for-membership` check. It's generally faster in
python to use a set with the `in` keyword, because it's a hash check
instead of a linear walk, this is especially true with strings, where
it's actually O(n^2), one loop over the container, and an inner loop of
the strings (as string comparison works by checking that `a[n] == b[n]`,
in a loop).
Also, I'm tired of complaining about this in reviews, let the tools do
it for me :)
The `py.dependency(embed: false)` method is supposed to consistently
provide a distutils-like `python.pc` / `python-embed.pc` interface
regardless of Python version. It handles both pkg-config and sysconfig
scraping. For the latter, we respect the value of self.link_libpython
as determined by distutils, and construct a fully custom dependency. For
the former, we blindly assume pkg-config is correct.
It isn't correct, not until Python 3.8 when embed was added. Before
then, we need to process the pkg-config dependency based on
link_libpython. We did this, but only inside the extension_module
method, which is obviously wrong.
Delete the special casing from extension_module, and handle it inside
the dependency.
Fixes#11097
On Windows, in Python version prior to 3.8.7, the `sysconfig` modules
provides an extension filename suffix that disagrees the one returned
by `distutils.sysconfig`. Get the more awesome suffix from the latter
when building for a Python version known to present this issue.
Simplify the extension module filename suffix lookup to use the same
method used by `setuptools`.
Adjust project tests accordingly.
Fixes#10960.
Generally plumb through the values of get_option() passed to
install_dir, and use this to establish the install plan name. Fixes
several odd cases, such as:
- {datadir} being prepended to "share" or "include"
- dissociating custom install directories and writing them out as
{prefix}/share/foo or {prefix}/lib/python3.10/site-packages
This is the second half of #9478Fixes#10601
Instead of using FeatureNew/FeatureDeprecated in the module.
The goal here is to be able to handle information about modules in a
single place, instead of having to handle it separately. Each module
simply defines some metadata, and then the interpreter handles the rest.
python compiled extensions should never need to expose any symbol other
than PyInit_* which is declared with default visibility via
PyMODINIT_FUNC on supported compilers.
Thus, a reasonably sane default is to mark any other symbols as hidden,
while still respecting any manually specified visibility.
Gate this on the version of python itself, as not all versions decorate
PyMODINIT_FUNC properly.
On Python 3.8.x and later, if the imported module requires non-system DLLs that
are not installed nor bundled with the module package, os.add_dll_directory()
must be called on every path that contains the required DLLs, so that the module
can be imported successfully by Python.
Make things easier for people by calling os.add_dll_directory() on the
valid directories in %PATH%, so that such module checks can be carried out
successfully with much less manual intervention.
It was originally added because proper detection was not working on
Debian, but that has been fixed since. It was causing annoying warning
by default when prefix is /usr/local that can only be avoided by setting
options.
This reverts commit 79c6075b56.
# Conflicts:
# docs/markdown/snippets/devenv.md
# mesonbuild/modules/python.py
# test cases/unit/91 devenv/test-devenv.py
PYTHONPATH cannot be reliably determined. The standard use case for
installing python modules with Meson is mixed pure sources (at least
`__init__.py`) and compiled extension_modules or configured files.
Unfortunately that doesn't actually work because python will not load
the same package hierarchy from two different directories, one a source
directory and one a (mandatory) out of tree build directory.
(It kind of can, but you need to do what this test case accidentally
stumbled upon, which is namespace packages. Namespace packages are a
very specific use case and you are NOT SUPPOSED to use them outside that
use case, so people are not going to use them just to circumvent Meson
devenv stuff as that would have negative install-time effects.)
Adding PYTHONPATH anyway will just lead to documentation commitments
which we cannot actually uphold, and confusing issues at time of use
because some imports *will* work... and some will *not*. The end result
will be a half-created tree of modules which just doesn't work together
at all, but because it partially works, users attempting to debug it
will spend time wondering why parts of it do import.
For any case where the automatic devenv would work correctly, it will
also work correctly to use `meson.add_devenv()` a single time, which is
very easy to manually get correct and doesn't provide any significant
value to automate.
In the long run, an uninstalled python package environment will require
"editable installs" support.
There are two cases where we can assume we found the python dependency
with its requisite libraries using sysconfig:
- we found the library with find_library and are prepared to link to it
- the library is not actually part of the dependency, so its presence or
absence is irrelevant
In the latter case, we should consider it found if link_libpython is
False. Originally we did this, but the logic was inverted in commit
5b422fce87 in an unrelated change and
without explanation, likely by accident.
Normally this doesn't much matter, since a python invariably comes with
a predictably located libpython and the first condition evaluates true.
But that is not true for pypy, and in fact that is the reason the
link_libpython check was originally added in commit
1bd14b52b2.
Restore that original logic.
Fixes#8570
A bunch of files have several T.TYPE_CHECKING blocks that each do some
things which could just as well be done once, with a single `if`
statement. Make them do so.
The default behavior of installing relative to prefix may be unexpected,
and is definitely wrong in many cases.
Give users control in order to specify that yes, they actually want to
install to a venv.
This is particularly useful for projects that use meson as a build
system for a python module, where *all* files shall be installed into
the python site-packages.
If the found python returns None from sysconfig.get_config_var('LIBPC')
then we cannot (and don't) set PKG_CONFIG_LIBDIR from it. In fact, we
can virtually guarantee we won't find a PkgConfigDependency either,
because any python that doesn't have a LIBPC is presumably not installed
to the system pkg-config directory (maybe it's an isolated relocatable
install, maybe it just doesn't have pkg-config support for who knows
what reason).
Trying to find one anyway using pkg-config's builtin search paths can
unexpectedly succeed, though, by finding a completely unrelated python
installation installed to a system location, which isn't the one we are
actually building for.
Instead, return early so that we use the system dependency class
fallback.
While we are at it, add back the debug messages from #3989 which got
removed.
Virtualenvs do not have their Python DLLs etc in the `sys.prefix`
directory, but in the `sys.base_prefix` directory. This directory is
the same as `sys.prefix` if not in a virtualenv, so is safe for either
case:
https://docs.python.org/3/library/sys.html#sys.base_prefix
Hardcoding the name is fragile, and enabling it based on the existence of
/etc/debian_version (as the is_debianlike helper does) will result in
incorrect paths if the Python binary is not provided by Debian.
Using the deb_system distuils scheme instead makes sure we use the
install path from the current interpreter, which Debian could change
between releases, and gives us the correct value on Python installations
that are not provided by Debian (eg. deadsnakes, Github Action Python,
etc.)
Do notice, though, that there is still no guarantee that these are the
correct paths, as they assume all schemes paths have the install prefix
as a base, see #9284.
Signed-off-by: Filipe Laíns <lains@riseup.net>
The sysconfig paths are, by default, correct for every OS -- they are
supposed to follow the scheme that python knows about per default.
For some reason, this overrode the scheme to posix_prefix, which is the
default for posix OSes like linux and macOS, but wrong on Windows.
Simply deleting this entirely makes everything that used to work, still
work, and a couple new things start working.
Eli Schwartz
Peter Urban
Dylan Baker
Daniele Nicolodi
Chun-wei Fan
Matti Picus
Jussi Pakkanen
Xavier Claessens
Ralf Gommers
Matthew Brett
Thomas Heijligen
Christian Clauss
Filipe Laíns