We have two copies of this code, and the python module one is vastly
superior, not just because it allows choosing which python executable to
base itself on. Unify this. Fixes various issues including non-Windows
support for sysconfig, and pypy edge cases.
In preparation for wholly merging the dependency handling from the
python module into dependencies.*, move the unique class definitions
from there into their new home in dependencies.python, which is
semantically convenient.
In preparation for handling more work inside dependencies.*, we need to
be able to run a PythonExternalProgram from the python dependency. Move
most of the definition -- but only the parts that have no interest in a
ModuleState -- and subclass a bit of sanity checking that we need to
handle specially when used in the module.
It can go directly inside the function which immediately uses it.
There's no purpose in looking it up exactly once and using it exactly
once, but looking it up outside the function and complicating the
function signature in order to pass it as a function argument.
We write this out as an embedded string to a tempfile in order to run
it, which is pretty awkward. And usually Meson's files are already files
on disk, not packed into a zip, so we can simply run it directly. Since
python 3.7, which is our new minimum, we can handle this well via the
stdlib. (There's also mesonbuild.mesondata, but we do not need
persistence in the builddir.)
This also solves the problem that has always been there, of giant python
programs inside strings occasionally confusing syntax highlighters. Or
even, it would be nice if we had syntax highlighting for this
introspection program. :D
We pass around a PythonInstallation into the depths of the dependency
factory, solely so that we can get at is_pypy in one particular branch.
We don't need the DSL functions, we don't need access to the
interpreter, let's just use the enhanced ExternalProgram object on its
own.
Add is_pypy to the object, and modify other references to get
information from .info['...'] instead of direct access.
This was causing a ninja issue where the native headers were always
being generated because io.github.hse-project.hse_Hse.h was being
expected, but io.github.hse_project.hse_Hse.h was actually generated.
In some cases we'll get an `ImmutableListProtocol[str]` anyway (and
actually, we should probably be getting one in call cases), since we
don't mutate it anyway, just store it as immutable.
Mainly thi sis that `state.find_program()` is annotated incorrectly, it
returns `ExternalProgram | Executable | OverrideProgram`, but it's
annotated to return only `ExteranlProgram`, and thus a bunch of the
annotations in the gnome module are wrong.
It's probably not useful to spam the user with warnings that old
versions of software may not behave correctly when the first warning was
perfectly valid.
There are lots of warnings that become fatal, that are simply unfixable
by the end user. Things like using old versions of software (because
they're using some kind of LTS release), warnings about compilers not
supporting certain kinds of checks, or standards being upgraded due to
skipped implementations (MSVC has c++98 and c++14, but not c++11). None
of these should be fatal, they're informative, and too important to
reduce to notices, but not important enough to stop meson if they're
printed.
At least, if you tried to use it when passing an install_dir. Because
T.Sequence is horrible and we should never use it, and the annotations
are a lie that produces bugs.
So, fix the annotations on CustomTarget to never allow this to happen
again, and also fix the function too. Move some definitions elsewhere
inline to satisfy the linter.
Fixes#11157
T.Sequence is a questionable concept. The idea is to hammer out generic,
maximally forgiving APIs that operate on protocols, which is a fancy way
of saying "I don't care if you use tuples or lists". This is rarely
needed, actually, and in exchange for this fancy behavior you get free
bugs.
Specifically, `somestr` is of type `T.Sequence[str]`, and also
`somestr[0]` is another string of type you guessed it. It's ~~turtles~~
strings all the way down.
It's worth noting that trying to code for "protocols" is a broken
concept if the contents have semantic meaning, e.g. it operates on
"the install tags of this object" rather than "an iterable that supports
efficient element access".
The other way to use T.Sequence is "I don't like that T.List is
invariant, but also I don't like that T.Tuple makes you specify exact
ordering". This sort of works. In fact it probably does work as long as
you don't allow str in your sequences, which of course everyone allows
anyway.
Use of Sequence has cute side effects, such as actually passing lists
around, knowing that you are going to get a list and knowing that you
need to pass it on as a list, and then having to re-allocate as
`list(mylist)` "because the type annotations says it could be a str or
tuple".
Except it cannot be a str, because if it is then the application is
fatally flawed and logic errors occur to disastrous end user effects,
and the type annotations:
- do not enforce their promises of annotating types
- fail to live up to "minimal runtime penalties" due to all the `list()`
Shun this broken concept, by hardening the type annotations. As it turns
out, we do not actually need any of this covariance or protocol-ism for
a list of strings! The whole attempt was a slow, buggy waste of time.
There are two problems here: a typing problem, and an algorithm problem.
We expect it to always be passed to CustomTarget() as a list, but we ran
list() on it, which became horribly mangled if you violated the types
and passed a string instead. This caused weird*er* errors and didn't
even do anything. We want to do all validation in the interpreter,
anyway, and make the build level dumb.
Meanwhile we type it as accepting a T.Sequence, which technically
permits... a string, actually. This isn't intentional; the point of
using T.Sequence is out of a misguided idea that APIs are supposed to be
"technically correct" by allowing "anything that fulfills an interface",
which is a flawed concept because we aren't using interfaces here, and
also because "technically string fulfills the same interface as a list,
if we're talking sequences".
Basically:
- mypy is broken by design, because it typechecks "python", not "what we
wish python to be"
- we do not actually need to graciously permit passing tuples instead of
lists
As far as historic implementations of this logic go, we have formerly:
- originally, typeslistified anything
- switched to accepting list from the interpreter, redundantly ran list()
on the list we got, and mishandling API violations passing a string
(commit 11f9638035)
- switched to accepting anything, stringlistifying it if it was not
`None`, mishandling `[None]`, and invoking list(x) on a brand new list
from stringlistify (commit 157d438835)
- stopped stringlistify, just accept T.List[str | None] and re-cast to
list, violates typing because we use/handle plain None too
(commit a8521fef70)
- break typing by declaring we accept a simple string, which still
results in mishandling by converting 'foo' -> ['f', 'o', 'o']
(commit ac576530c4)
All of this. ALL of it. Is because we tried to be fancy and say we
accept T.Tuple; the only version of this logic that has ever worked
correctly is the original untyped do-all-validation-in-the-build-phase
typeslistified version.
Let's just call it what it is. We want a list | None, and we handle it too.
In particular, CUDA 12 removes support for Kepler (3.x) entirely.
Unusually, however, it does not introduce any new architectures,
or even compute capabilities.
When finding a py.dependency() we try to use pkg-config. We then apply
our own custom base class, which replaces self.name with the informative
comment "override the name from the "real" dependency lookup", to which
I can only say "uhhh why". Why do we want to do that???
It turns out we don't, it was just a really old legacy design because we
had a SystemDependency with a .pkgdep attribute hiding the real
dependency bizarro-land style. We cleaned that up in commit
4d67dd19e5 and as part of that, we
*shifted over* the self.name assignment to preserve the visible effects,
sort of. We didn't have a *reason* to override the name, we just did it
because... we weren't sure whether it mattered.
Unfortunately it very much does matter the other way -- we don't want
it. We can pass this dependency to the pkgconfig module, which uses the
name attribute to fill out the `Requires: ` field. Also, the name should
name what we have. :p
Get rid of this bizarre historic quirk. Since we have proper
dependencies here, we should go all in.
Fixes https://github.com/ufo-kit/ufo-core/pull/185#issuecomment-1328224996
It is only used by Environment.get_exe_wrapper() and every callers were
handling None already. Type annotation was wrong, it already could
return None for the case an exe wrapper is needed but none is provided.
Since it's also used in the rust module, it should be in a common place.
Also rename from `TEST_KWARGS` to `TEST_KWS`, which is more in line with
the `*_KW` naming scheme used in the type_checking module.
Which adds the `use-set-for-membership` check. It's generally faster in
python to use a set with the `in` keyword, because it's a hash check
instead of a linear walk, this is especially true with strings, where
it's actually O(n^2), one loop over the container, and an inner loop of
the strings (as string comparison works by checking that `a[n] == b[n]`,
in a loop).
Also, I'm tired of complaining about this in reviews, let the tools do
it for me :)
This finds uses of deny-listed functions, which defaults to map and
filter. These functions should be replaced by comprehensions in
idiomatic python because:
1. comprehensions are more heavily optimized and are often faster
2. They avoid the need for lambdas in some cases, which make them
faster
3. you can do the equivalent in one statement rather than two, which
is faster
4. They're easier to read
5. if you need a concrete instance (ie, a list) then you don't have
to convert the iterator to a list afterwards
The `py.dependency(embed: false)` method is supposed to consistently
provide a distutils-like `python.pc` / `python-embed.pc` interface
regardless of Python version. It handles both pkg-config and sysconfig
scraping. For the latter, we respect the value of self.link_libpython
as determined by distutils, and construct a fully custom dependency. For
the former, we blindly assume pkg-config is correct.
It isn't correct, not until Python 3.8 when embed was added. Before
then, we need to process the pkg-config dependency based on
link_libpython. We did this, but only inside the extension_module
method, which is obviously wrong.
Delete the special casing from extension_module, and handle it inside
the dependency.
Fixes#11097
On Windows, in Python version prior to 3.8.7, the `sysconfig` modules
provides an extension filename suffix that disagrees the one returned
by `distutils.sysconfig`. Get the more awesome suffix from the latter
when building for a Python version known to present this issue.
Simplify the extension module filename suffix lookup to use the same
method used by `setuptools`.
Adjust project tests accordingly.
Fixes#10960.
Since commit 32b14b1bb5, hotdoc is run
during configure as an external program. It turns out though, that in
some cases we passed NoneType in the cmd array, previously to
hotdoc.run_hotdoc.run() and now via subprocesses. The former "worked"
due to ignoring unknown arguments (?) but the latter was broken because
command line interfaces don't accept python NoneType objects, naturally.
End result: when for example building Meson's own documentation, this
fails with a python traceback.
The reason this happens to begin with turns out to be, once again,
because of the legacy debt of homebrewed kwargs parsing. We have a
function for `process_known_args` that handles args and ignores them if
they are NoneType, and then include_paths is handled via a custom
processor that internally adds them, then returns a *list* of NoneType
which then gets appended to the global cmd, because the logic ends up as
`[None, None] is None` which is a failed check, so we go ahead and add
it.
It's odd that we ever attempted to process it twice to begin with, so
let's simply not do that.
Eli Schwartz
Tristan Partin
Peter Urban
Thibault Saunier
X512
Dylan Baker
Paolo Bonzini
Andreas Deininger
Olexa Bilaniuk
Xavier Claessens
Daniele Nicolodi