The setuptools package was added to allow definition of namespaces using
the now outdated (and discouraged from use) pkg_resources-style.
The code here, for a long while now, uses a try/except (`ImportError`)
protection around the setuptools code, and falls back to the more
encoraged pkgutil-style.
Removing this library won't affect any current workflow, and in the case
of setuptools not found, it'll actually use a more modern (and
encouraged) flow.
This change updates the README.md to describe the order of resolution
between `options['generate_py_protobufs']`, `--protoc`, and the `PROTOC`
env var.
Since the `setup.py` script is not the main way to find docs, this change
deletes the parts of the docstring that are redundant with `README.md`.
Currently, the logic in `generate_py_protobufs` cannot find a custom `protoc` (even though it's supposed to be possible). Fixes:
1. Mark the `--protoc` flag as accepting an argument.
2. If the `--protoc` flag was not passed, try finding `PROTOC` in the environment.
3. (Existing behavior) Otherwise, fall back to `spawn.find_executable`.
Hat tip to @bobhancock for uncovering the problem(s).
Using non-versioned scripts for `build_file` means every Python version is
tested (under Tox), and this is multiplied by each Python version running Tox.
For example, when the `python38` config is used to run Tox, the
Tox layer runs on Python 3.8. This then tests all of the Python versions
individually, including, for example, python27 tests.
This change fixes the `build_file` paths to point to the same-pathed build
script. For example, `kokoro/linux/python27/presubmit.cfg` now uses
`kokoro/linux/python27/build.sh`.
Some additional fixes:
* Use `python -m tox` in tests.sh instead of just `tox`. This helps non-site
installations of tox, where the `tox` script may not be on `$PATH`.
* Ensure tox (and other Python build-related packages) are available in
Python testing images. (New images have been pushed.)
* Disable `--warnings_as_errors` due to deprecated function.
* Remove apt lists per [Docker best practices][1].
[1]: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
In #713 and #1296, the `google` package in protobuf sources was found
to cause conflicts with other Google projects, because it was not
properly configured as a namespace package [1]. The initial fix in
786f80f addressed part of the issue, and #1298 fixed the rest.
However, 786f80f (the initial fix) also made `google.protobuf` and
`google.protobuf.pyext` into namespace packages. This was not correct:
they are both regular, non-namespace, sub-subpackages.
However (still), the follow-up #1298 did not nominate them as
namespace packages, so the namespace registration behavior has
remained, but without benefit.
This change removes the unnecessary namespace registration, which has
substantial overhead, thus reducing startup time substantially when
using protobufs.
Because this change affects the import internals, quantifying the
overhead requires a full tear-down/start-up of the Python interpreter.
So, to capture the full cost for every run, I measured the time to
launching a _fresh_ Python instance in a subprocess, varying the
imports and code under test. In other words, I used `timeit` to
measure the time to launch a _fresh_ Python subprocess which actually
performs the imports.
* Reference: normal Python startup (i.e., don't import protobuf at all).
```
% python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "pass"])'
10 loops, best of 3: 27.1 msec per loop
```
* Baseline: cost to import `google.protobuf.descriptor`, with
extraneous namespace packages.
```
% python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "import google.protobuf.descriptor"])'
10 loops, best of 3: 133 msec per loop
```
* This change: cost to import `google.protobuf.descriptor`, without
extraneous namespace packages.
```
% python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "import google.protobuf.descriptor"])'
10 loops, best of 3: 43.1 msec per loop
```
[1]: https://packaging.python.org/guides/packaging-namespace-packages/
wheel is required since version 3.13.0 and
ff92cee10b
This will result in the following build failure when cross-compiling:
Download error on https://pypi.org/simple/wheel/: unknown url type: https -- Some packages may not be found!
Couldn't find index page for 'wheel' (maybe misspelled?)
Download error on https://pypi.org/simple/: unknown url type: https -- Some packages may not be found!
No local packages or working download links found for wheel
Remove wheel requirement from setup.py as it is only needed by
release.sh, not by setup.py
Fixes:
- http://autobuild.buildroot.org/results/371c686a10d6870933011b46d36b1879d29046b9
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
This updates README.md to use an example with pip instead of setup.py.
The README.md is then used as the long_description argument to setup(),
so that the doc shows up on pypi.
This extension allows Python code to be generated from setup.py files, so they
are created as part of a normal Python build. The extension uses an already-existing
protoc binary, which can be explicitly specified if needed.
Fixes#7493
By removing all test files from the distribution, we've reduced the .whl
file size from 1259808B (1.2MB) to 996042B (973K), which is 21% reduction,
And reduced the unpacked size from 5317178B (5.1MB) to 3251811B (3.2MB),
which is 39% reduction.
Size was measured for the protobuf-3.11.3-cp35-cp35m-manylinux1_x86_64.whl release.
* Fix a typo
* Fix lots of spelling errors
* Fix a few more spelling mistakes
* s/parsable/parseable/
* Don't touch the third party files
* Cloneable is the preferred C# term
* Copyable is the preferred C++ term
* Revert "s/parsable/parseable/"
This reverts commit 534ecf7675.
* Revert unparseable->unparsable corrections
Newer versions of virtualenv lack the --no-site-packages option, so I
had to remove it to keep the release.sh script working. I read that this
option has already been the default for a long time, so removing it
shouldn't chany any behavior.
For the setup.py script, I was getting some errors about the bdist_wheel
argument to setup.py, but I was able to fix that by adding 'wheel' to
setup_requires.
Since our (public) generated APIs document their use of repeated fields, we should probably document its API, too.
This includes some changes to generate_docs.py to explicitly allow named modules, since the repeated field implementation lives in the "internal" tree.
* Set execute bit on files if and only if they begin with (#!).
Git only tracks the 'x' (executable) bit on each file. Prior to this
CL, our files were a random mix of executable and non-executable.
This change imposes some order by making files executable if and only
if they have shebang (#!) lines at the beginning.
We don't have any executable binaries checked into the repo, so
we shouldn't need to worry about that case.
* Added fix_permissions.sh script to set +x iff a file begins with (#!).
This change updates docstrings and comments so that they will produce nicer
formatting and cross-references from Sphinx. There are a few broad categories of
changes:
- Paramter and attribute docs are updated so that types will be recognized by
Napoleon (https://sphinxcontrib-napoleon.readthedocs.io/en/latest/) This
usually just means moving a colon in the docstring, so
`name: (type) description` becomes `name (type): description`.
- References to other symbols can be cross-references if they have the right
format. For example, "attr_name" might become ":attr:`attr_name`".
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#cross-referencing-python-objects
- For fenced code blocks, adding a double-colon `::` signifies a literal block.
https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#literal-blocks
- Some bits of docstrings move from docstring to comments. For TODOs, this
means we won't be putting stale (or otherwise unrelated) noise into the docs.
For `Message.DESCRIPTOR`, the change means it gets appropriate documentation.
- There are some wording tweaks for consistency, and some new docstrings
(especially for methods in Message).
For types, I used the convention of `list[Foo]` and `dict(foo, bar)`, which seem
to be common among other Python rst docstrings. Sphinx should generally
recognize both, and cross-links them correctly (both internally and to Python
library documentation). Upgrading to Python3-style type annotations would allow
us to use `sphinx-autodoc-typehints`; the changes in this commit are very
similar to typing-based hints.
Background:
This is a follow-up to the PR that adds sphinx docs.
Read the Docs is a hosting platform for documentation, primarily Python
docs. It supports builds at commit time as well as at specific git
labels to support versioned docs. I have claimed the
protobuf.readthedocs.io project and can add any Googlers who need access
to be able to configure and trigger builds.
https://readthedocs.org/projects/protobuf/builds/ It's also relatively
easy to create a new project to test the documentation builds from a
fork, such as https://readthedocs.org/projects/tswast-protobuf/builds/
About this change:
Once web hooks are configured, Read the Docs will automatically build
the docs for the latest changes on the master branch.
I needed to update `python/setup.py` to support installation from the
root of the repository because Read the Docs does not `cd python` before
installing the protobuf package with `setup.py install`. To support
this, I updated the file paths to use the absolute path to files. The
`__file__` special variable comes in handy for this, as it provides the
path to the `setup.py` file.
A banner is added to the docs when published to readthedocs. This links
to the official documentation and the future home of the stable API
reference on googleapis.dev.
**Background:**
This is a follow-up to the PR that adds sphinx docs.
Googleapis.dev hosts client library documentation for several languages as
well as documentation for core packages such as google-api-core. It is owned
and maintained by the Cloud DevRel Core team.
**About this change:**
The `kokoro/docs/publish-python.sh` script builds the docs with sphinx, then
uses the [googleapis/docuploader](https://github.com/googleapis/docuploader)
tool to publish to googleapis.dev. The publish script is triggered manually
with a Kokoro RELEASE type job. Googlers, see go link
protobuf-docs-python-design for additional internal references.
Joshua Haberman
Misha Seltzer
David L. Jones
Fabrice Fontaine
Adam Cozzette
Peter Newman
listonjesse
Jie Luo
Tim Swast
Rafi Kamal