In #713 and #1296, the `google` package in protobuf sources was found
to cause conflicts with other Google projects, because it was not
properly configured as a namespace package [1]. The initial fix in
786f80f addressed part of the issue, and #1298 fixed the rest.
However, 786f80f (the initial fix) also made `google.protobuf` and
`google.protobuf.pyext` into namespace packages. This was not correct:
they are both regular, non-namespace, sub-subpackages.
However (still), the follow-up #1298 did not nominate them as
namespace packages, so the namespace registration behavior has
remained, but without benefit.
This change removes the unnecessary namespace registration, which has
substantial overhead, thus reducing startup time substantially when
using protobufs.
Because this change affects the import internals, quantifying the
overhead requires a full tear-down/start-up of the Python interpreter.
So, to capture the full cost for every run, I measured the time to
launching a _fresh_ Python instance in a subprocess, varying the
imports and code under test. In other words, I used `timeit` to
measure the time to launch a _fresh_ Python subprocess which actually
performs the imports.
* Reference: normal Python startup (i.e., don't import protobuf at all).
```
% python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "pass"])'
10 loops, best of 3: 27.1 msec per loop
```
* Baseline: cost to import `google.protobuf.descriptor`, with
extraneous namespace packages.
```
% python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "import google.protobuf.descriptor"])'
10 loops, best of 3: 133 msec per loop
```
* This change: cost to import `google.protobuf.descriptor`, without
extraneous namespace packages.
```
% python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "import google.protobuf.descriptor"])'
10 loops, best of 3: 43.1 msec per loop
```
[1]: https://packaging.python.org/guides/packaging-namespace-packages/
wheel is required since version 3.13.0 and
ff92cee10b
This will result in the following build failure when cross-compiling:
Download error on https://pypi.org/simple/wheel/: unknown url type: https -- Some packages may not be found!
Couldn't find index page for 'wheel' (maybe misspelled?)
Download error on https://pypi.org/simple/: unknown url type: https -- Some packages may not be found!
No local packages or working download links found for wheel
Remove wheel requirement from setup.py as it is only needed by
release.sh, not by setup.py
Fixes:
- http://autobuild.buildroot.org/results/371c686a10d6870933011b46d36b1879d29046b9
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
This updates README.md to use an example with pip instead of setup.py.
The README.md is then used as the long_description argument to setup(),
so that the doc shows up on pypi.
This extension allows Python code to be generated from setup.py files, so they
are created as part of a normal Python build. The extension uses an already-existing
protoc binary, which can be explicitly specified if needed.
Fixes#7493
By removing all test files from the distribution, we've reduced the .whl
file size from 1259808B (1.2MB) to 996042B (973K), which is 21% reduction,
And reduced the unpacked size from 5317178B (5.1MB) to 3251811B (3.2MB),
which is 39% reduction.
Size was measured for the protobuf-3.11.3-cp35-cp35m-manylinux1_x86_64.whl release.
* Fix a typo
* Fix lots of spelling errors
* Fix a few more spelling mistakes
* s/parsable/parseable/
* Don't touch the third party files
* Cloneable is the preferred C# term
* Copyable is the preferred C++ term
* Revert "s/parsable/parseable/"
This reverts commit 534ecf7675.
* Revert unparseable->unparsable corrections
Newer versions of virtualenv lack the --no-site-packages option, so I
had to remove it to keep the release.sh script working. I read that this
option has already been the default for a long time, so removing it
shouldn't chany any behavior.
For the setup.py script, I was getting some errors about the bdist_wheel
argument to setup.py, but I was able to fix that by adding 'wheel' to
setup_requires.
Since our (public) generated APIs document their use of repeated fields, we should probably document its API, too.
This includes some changes to generate_docs.py to explicitly allow named modules, since the repeated field implementation lives in the "internal" tree.
* Set execute bit on files if and only if they begin with (#!).
Git only tracks the 'x' (executable) bit on each file. Prior to this
CL, our files were a random mix of executable and non-executable.
This change imposes some order by making files executable if and only
if they have shebang (#!) lines at the beginning.
We don't have any executable binaries checked into the repo, so
we shouldn't need to worry about that case.
* Added fix_permissions.sh script to set +x iff a file begins with (#!).
This change updates docstrings and comments so that they will produce nicer
formatting and cross-references from Sphinx. There are a few broad categories of
changes:
- Paramter and attribute docs are updated so that types will be recognized by
Napoleon (https://sphinxcontrib-napoleon.readthedocs.io/en/latest/) This
usually just means moving a colon in the docstring, so
`name: (type) description` becomes `name (type): description`.
- References to other symbols can be cross-references if they have the right
format. For example, "attr_name" might become ":attr:`attr_name`".
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#cross-referencing-python-objects
- For fenced code blocks, adding a double-colon `::` signifies a literal block.
https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#literal-blocks
- Some bits of docstrings move from docstring to comments. For TODOs, this
means we won't be putting stale (or otherwise unrelated) noise into the docs.
For `Message.DESCRIPTOR`, the change means it gets appropriate documentation.
- There are some wording tweaks for consistency, and some new docstrings
(especially for methods in Message).
For types, I used the convention of `list[Foo]` and `dict(foo, bar)`, which seem
to be common among other Python rst docstrings. Sphinx should generally
recognize both, and cross-links them correctly (both internally and to Python
library documentation). Upgrading to Python3-style type annotations would allow
us to use `sphinx-autodoc-typehints`; the changes in this commit are very
similar to typing-based hints.
Background:
This is a follow-up to the PR that adds sphinx docs.
Read the Docs is a hosting platform for documentation, primarily Python
docs. It supports builds at commit time as well as at specific git
labels to support versioned docs. I have claimed the
protobuf.readthedocs.io project and can add any Googlers who need access
to be able to configure and trigger builds.
https://readthedocs.org/projects/protobuf/builds/ It's also relatively
easy to create a new project to test the documentation builds from a
fork, such as https://readthedocs.org/projects/tswast-protobuf/builds/
About this change:
Once web hooks are configured, Read the Docs will automatically build
the docs for the latest changes on the master branch.
I needed to update `python/setup.py` to support installation from the
root of the repository because Read the Docs does not `cd python` before
installing the protobuf package with `setup.py install`. To support
this, I updated the file paths to use the absolute path to files. The
`__file__` special variable comes in handy for this, as it provides the
path to the `setup.py` file.
A banner is added to the docs when published to readthedocs. This links
to the official documentation and the future home of the stable API
reference on googleapis.dev.
**Background:**
This is a follow-up to the PR that adds sphinx docs.
Googleapis.dev hosts client library documentation for several languages as
well as documentation for core packages such as google-api-core. It is owned
and maintained by the Cloud DevRel Core team.
**About this change:**
The `kokoro/docs/publish-python.sh` script builds the docs with sphinx, then
uses the [googleapis/docuploader](https://github.com/googleapis/docuploader)
tool to publish to googleapis.dev. The publish script is triggered manually
with a Kokoro RELEASE type job. Googlers, see go link
protobuf-docs-python-design for additional internal references.
* python: generate documentation with Sphinx and Read the Docs
Background:
Formerly, the Python protobuf reference documentation was built with
[Epydoc](http://epydoc.sourceforge.net/). This package has not been
updated since 2008, and it has inconsistent formatting (see internal
issue 131415575) with most Python documentation. Sphinx is used for the
official docs.python.org docs as well as most other Python packages,
including the Google client libraries and related packages, such as
https://googleapis.dev/python/google-api-core/latest/
To build the docs with Sphinx:
1. Install the needed packages (`sphinx`, `sphinxcontrib-napoleon` for
Google-style docstring support). I've created a conda environment file
to make this easier:
```
conda env create -f python/docs/environment.yml
```
2. (Optional) Generate reference docs files and regenerate index:
```
cd python
python generate_docs.py
cd ..
```
3. Run Sphinx.
```
cd python/docs
make html
```
About this change:
The script at `python/generate_docs.py` creates a ReStructured Text file
for each public module in the protobuf Python package. The script also
updates the table of contents in `python/docs/index.rst` to point to
these module references.
Future work:
Testing the docs build on PRs requires contributors to actually do some
setup work to configure builds on their fork. It'd be better if CI had a
docs build session to verify that the Sphinx docs generation at least
runs.
There are many warnings due to not-quite-correct docstrings in the
actual Python code itself. I'm choosing to ignore these errors to keep
the PR small, but I recommend you fix these and then enable "fail on
warnings" in the docs build on CI.
* add docs to EXTRA_DIST
* add instructions to build documentation to generate_docs.py
* exclude python/odcs from cpp_distcheck
David L. Jones
Fabrice Fontaine
Joshua Haberman
Misha Seltzer
Adam Cozzette
Peter Newman
listonjesse
Jie Luo
Tim Swast
Rafi Kamal
Paul Yang
Brian Wignall