Follow up from #37917 to add `noexcept` to multiple Cython functions using `nogil` with a `void` return type.
Below are the performance hints that were encountered as part of the Cython translation which are solved by this PR:
The below performance hint still exists, and is not resolved.
This is because `noexcept` expects that the function doesn't raise an exception, or a raised exception is just displayed as a warning and not propagated. But `_poll` raises an `AssertionError` at `QUEUE_TIMEOUT`, and hence cannot use `noexcept` with `_poll`.
As [PR 37917](https://github.com/grpc/grpc/pull/37917) and this PR now solves the Cython Asyncio test timeouts caused by Cython upgrade, this PR also reverts the Bazel Cython downgrade PR #37884
### Testing for timeout
Tested using `bazel test -c dbg --runs_per_test=3000 --test_timeout=10 "//examples/python/auth:_auth_example_test"`
```
//examples/python/auth:_auth_example_test PASSED in 4.6s
Stats over 3000 runs: max = 4.6s, min = 2.6s, avg = 3.2s, dev = 0.2s
```
Closes#37922
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37922 from sreenithi:fix_bazel_cython_asyncio_timeout 2162cd28b1
PiperOrigin-RevId: 686524195
We're seeing some Bazel tests timing out due to this upgrade, revert this for now until we resolved the issue.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37884
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37884 from XuanWang-Amos:revert_cython_upgrade 0071889b5c
PiperOrigin-RevId: 684538459
This PR adds templating for Python versions and updates the maximum supported Python version to 3.13. The following major changes related to templating are added:
- Minimum supported Python version and list of supported versions in `setup.py` are fetched using new template generated files called `python_version.py`
- Dockerfiles for the different Python Linux builds are now template generated.
- The "Supported Python Versions" section from READMEs of ancillary and main packages have been removed
Note: All the `python_version.py` files and Linux build `Dockerfiles` except `tools/dockerfile/grpc_artifact_python_linux_armv7/Dockerfile` in the PR are generated from the respective templates.
Further non-templated additions to add support for Python 3.13:
- install scripts and artifacts for windows, macos and linux are added manually. Later, these can be templated as well.
- updated cython bounds to 3.x
- updated twine version to solve [cgi module import error](https://github.com/pypa/twine/issues/1046)
- the twine update introduces a dependency on cryptography>=2.0. But the cryptography package doesn't support 32-bit Linux images and hence `twine check` has been disabled for x86 manylinux and x86 musllinux artifacts.
Closes#37643
PiperOrigin-RevId: 678954495
This PR is required in order to upgrade to the `master` branch version of protobuf. To do this, we upgrade to the latest version of `rules_python` by:
- Adding in explicit dependencies on all PyPi Bazel test dependencies
- Resolving the circular dependency this creates for `xds-protos` by giving it a Bazel build
- Generating the Python code for `xds-protos` as part of `generate-projects.sh` since doing so directly in Bazel would be highly fragily, dependent on many other projects' Bazel builds
Closes#34945
PiperOrigin-RevId: 599875020
We just found out that our current Bazel setup does not support Python
3.11.
Thus PR updates some dependencies to allow using Bazel in Python 3.11.
Cython:
* Cython [backported Python 3.11 support change to
0.29x](https://github.com/cython/cython/issues/4500), but it appears
that the Cython version we are using in Bazel does not include the fix,
so we're using the latest stable version instead.
Gevent:
* The first version of gevent that supports [Python 3.11 is
22.08.0](https://github.com/gevent/gevent/issues/1903#issuecomment-1303227507).
#### Testing
* Tested locally using Python 3.11 virtual environment, was able to
reproduce the issue and verified that those changes were able to fix it.
* Fix all lint errors in repo.
* Use strict buildifier by default
* Whoops. That file does not exist
* Attempt fix to buildifier invocation
* Add missing copyright
* WIP
* Add gevent test suite run under Bazel.
* Fix things up
* Yapf
* Fix up Bazel files
* Make py_grpc_test fancier
* Attempt to fix Windows RBE
* Attempt to kick GitHub
* Fix Python 2 runs
* Yet more fixes
* And the patch file too
* I am an idiot
* Mark gevent tests flaky
* Try to make rules_python more tolerant
* Typo
* Exclude reconnect test from gevent
* Remove unnecessary parts of patch
* Buildifier
* You saw nothing
* isort
* Move py_grpc_test to an internal-only file
* Review comments
* More reviewer comments
* Review
A breaking C-level API change was made in Python 3.9 which old versions
of Cython cannot handle. Since Bazel prescribes a single version of
dependency instead of ranges, like in setuptools, we must update this to
keep our downstream users working properly.
As a side note, since it was a point of confusion while debugging this,
interpreter versions installed by pyenv must be set *globally* rather
than just locally for Bazel to pick them up, since local pyenv
configuration will not be picked up from within the sandbox.
By popular demand, we'll now be offering separate py_grpc_library and
py_proto_library targets sharing the same interface as within google3.
This change necessitated some modifications to how we pull in our own
Python-level dependencies and how we make those available to those
pulling in our project via Bazel.
There is now a grpc_python_deps() Bazel workspace rule that pulls in the
appropriate dependencies, which should be called from the client
project's WORKSPACE file. A test has been added to the bazel/test/
directory to verify that this behavior works as intended.
It's worth noting that the protobuf repository's usage of Starlark
bind() caused a great deal of trouble in ensuring that we could also
pull in six.
This change also required a change in the way generated proto code is
imported in the channelz and health-check modules, as well as in their
associated tests. We were importing them two different ways, each
relative. This resulted in two different module objects being imported
into the process, which were incompatible. I am not sure exactly what
caused this behavior to begin, as this should have been possible before
this PR. As a workaround, I am simply trying two different absolute
imports and using the one that works. This should function both inside
and outside of Bazel environments.
This commit resolves#18331.
This commit resolves#18256.
This commit resolves... another TODO that apparently didn't have an
associated github issue.
We swap out pubref's implementation of py_proto_library with our own,
which more closely mirrors the interface of the internal
py_proto_library, taking the descriptor file output of a proto_library
rule as input.
One minor change in behavior was introduced for simplicity. When a
py_proto_library depends on a proto_library with a source proto file in
a subdirectory of the bazel package, the import module of the resultant
python library will reflect the package, *not* the full directory of the
proto file, including both the bazel package and the subdirectories, as
pubref did previously. This behavior also more closely mirrors google
internal behavior.
This commit also introduces a slightly more stringent bazel format
script. Buildifier on its own will not take care of long lines, but by
running yapf first, we end up with a more legible file. At the moment,
there is no sanity check associated with this formatter.
Marcel
Sreenithi Sridharan
Xuan Wang
Richard Belleville
Michael P. Nitowski
Lidi Zheng
Richard Belleville
Jan Tattermusch