Implement Python CSM observability.
Design: [go/grpc-python-opentelemetry](http://goto.google.com/grpc-python-opentelemetry)
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36557
PiperOrigin-RevId: 639073741
In Cython 3, cdef functions that really will not raise exceptions must be declared as `noexcept`. Fixed by this commit.
Update requirements to `cython >= 3.0` in requirements*.txt and setup.py.
Fixes issue #33918.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35995
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35995 from badshah400:master b3277bac15
PiperOrigin-RevId: 621214091
This PR is required in order to upgrade to the `master` branch version of protobuf. To do this, we upgrade to the latest version of `rules_python` by:
- Adding in explicit dependencies on all PyPi Bazel test dependencies
- Resolving the circular dependency this creates for `xds-protos` by giving it a Bazel build
- Generating the Python code for `xds-protos` as part of `generate-projects.sh` since doing so directly in Bazel would be highly fragily, dependent on many other projects' Bazel builds
Closes#34945
PiperOrigin-RevId: 599875020
Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.5 to 1.26.18.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/urllib3/urllib3/releases">urllib3's releases</a>.</em></p>
<blockquote>
<h2>1.26.18</h2>
<ul>
<li>Made body stripped from HTTP requests changing the request method to GET after HTTP 303 "See Other" redirect responses. (GHSA-g4mx-q9vg-27p4)</li>
</ul>
<h2>1.26.17</h2>
<ul>
<li>Added the <code>Cookie</code> header to the list of headers to strip from requests when redirecting to a different host. As before, different headers can be set via <code>Retry.remove_headers_on_redirect</code>. (GHSA-v845-jxx5-vc9f)</li>
</ul>
<h2>1.26.16</h2>
<ul>
<li>Fixed thread-safety issue where accessing a <code>PoolManager</code> with many distinct origins would cause connection pools to be closed while requests are in progress (<a href="https://redirect.github.com/urllib3/urllib3/issues/2954">#2954</a>)</li>
</ul>
<h2>1.26.15</h2>
<ul>
<li>Fix socket timeout value when HTTPConnection is reused (<a href="https://redirect.github.com/urllib3/urllib3/issues/2645">urllib3/urllib3#2645</a>)</li>
<li>Remove "!" character from the unreserved characters in IPv6 Zone ID parsing (<a href="https://redirect.github.com/urllib3/urllib3/issues/2899">urllib3/urllib3#2899</a>)</li>
<li>Fix IDNA handling of 'x80' byte (<a href="https://redirect.github.com/urllib3/urllib3/issues/2901">urllib3/urllib3#2901</a>)</li>
</ul>
<h2>1.26.14</h2>
<ul>
<li>Fixed parsing of port 0 (zero) returning None, instead of 0 (<a href="https://redirect.github.com/urllib3/urllib3/issues/2850">#2850</a>)</li>
<li>Removed deprecated <code>HTTPResponse.getheaders()</code> calls in <code>urllib3.contrib</code> module.</li>
</ul>
<h2>1.26.13</h2>
<ul>
<li>Deprecated the <code>HTTPResponse.getheaders()</code> and <code>HTTPResponse.getheader()</code> methods.</li>
<li>Fixed an issue where parsing a URL with leading zeroes in the port would be rejected even when the port number after removing the zeroes was valid.</li>
<li>Fixed a deprecation warning when using cryptography v39.0.0.</li>
<li>Removed the <code><4</code> in the <code>Requires-Python</code> packaging metadata field.</li>
</ul>
<h2>1.26.12</h2>
<ul>
<li>Deprecated the <code>urllib3[secure]</code> extra and the <code>urllib3.contrib.pyopenssl</code> module. Both will be removed in v2.x. See this <a href="https://redirect.github.com/urllib3/urllib3/issues/2680">GitHub issue</a> for justification and info on how to migrate.</li>
</ul>
<h2>1.26.11</h2>
<p><strong>If you or your organization rely on urllib3 consider supporting us via <a href="https://github.com/sponsors/urllib3">GitHub Sponsors</a>.</strong></p>
<p>⚠️ <strong>urllib3 v2.0 will drop support for Python 2</strong>: <a href="https://urllib3.readthedocs.io/en/latest/v2-roadmap.html">Read more in the v2.0 Roadmap</a></p>
<ul>
<li>Fixed an issue where reading more than 2 GiB in a call to HTTPResponse.read would raise an OverflowError on Python 3.9 and earlier.</li>
</ul>
<h2>1.26.10</h2>
<p><strong>If you or your organization rely on urllib3 consider supporting us via <a href="https://github.com/sponsors/urllib3">GitHub Sponsors</a>.</strong></p>
<p>⚠️ <strong>urllib3 v2.0 will drop support for Python 2</strong>: <a href="https://urllib3.readthedocs.io/en/latest/v2-roadmap.html">Read more in the v2.0 Roadmap</a></p>
<p>🔐 <strong>This is the first release to be signed with Sigstore!</strong> You can verify the distributables using the <code>.sig</code> and <code>.crt</code> files included on this release.</p>
<ul>
<li>Removed support for Python 3.5</li>
<li>Fixed an issue where a <code>ProxyError</code> recommending configuring the proxy as HTTP instead of HTTPS could appear even when an HTTPS proxy wasn't configured.</li>
</ul>
<h2>1.26.9</h2>
<p><strong>If you or your organization rely on urllib3 consider supporting us via <a href="https://github.com/sponsors/urllib3">GitHub Sponsors</a>.</strong></p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/urllib3/urllib3/blob/main/CHANGES.rst">urllib3's changelog</a>.</em></p>
<blockquote>
<h1>1.26.18 (2023-10-17)</h1>
<ul>
<li>Made body stripped from HTTP requests changing the request method to GET after HTTP 303 "See Other" redirect responses.</li>
</ul>
<h1>1.26.17 (2023-10-02)</h1>
<ul>
<li>Added the <code>Cookie</code> header to the list of headers to strip from requests when redirecting to a different host. As before, different headers can be set via <code>Retry.remove_headers_on_redirect</code>. (<code>[#3139](https://github.com/urllib3/urllib3/issues/3139) <https://github.com/urllib3/urllib3/pull/3139></code>_)</li>
</ul>
<h1>1.26.16 (2023-05-23)</h1>
<ul>
<li>Fixed thread-safety issue where accessing a <code>PoolManager</code> with many distinct origins
would cause connection pools to be closed while requests are in progress (<code>[#2954](https://github.com/urllib3/urllib3/issues/2954) <https://github.com/urllib3/urllib3/pull/2954></code>_)</li>
</ul>
<h1>1.26.15 (2023-03-10)</h1>
<ul>
<li>Fix socket timeout value when <code>HTTPConnection</code> is reused (<code>[#2645](https://github.com/urllib3/urllib3/issues/2645) <https://github.com/urllib3/urllib3/issues/2645></code>__)</li>
<li>Remove "!" character from the unreserved characters in IPv6 Zone ID parsing
(<code>[#2899](https://github.com/urllib3/urllib3/issues/2899) <https://github.com/urllib3/urllib3/issues/2899></code>__)</li>
<li>Fix IDNA handling of '\x80' byte (<code>[#2901](https://github.com/urllib3/urllib3/issues/2901) <https://github.com/urllib3/urllib3/issues/2901></code>__)</li>
</ul>
<h1>1.26.14 (2023-01-11)</h1>
<ul>
<li>Fixed parsing of port 0 (zero) returning None, instead of 0. (<code>[#2850](https://github.com/urllib3/urllib3/issues/2850) <https://github.com/urllib3/urllib3/issues/2850></code>__)</li>
<li>Removed deprecated getheaders() calls in contrib module. Fixed the type hint of <code>PoolKey.key_retries</code> by adding <code>bool</code> to the union. (<code>[#2865](https://github.com/urllib3/urllib3/issues/2865) <https://github.com/urllib3/urllib3/issues/2865></code>__)</li>
</ul>
<h1>1.26.13 (2022-11-23)</h1>
<ul>
<li>Deprecated the <code>HTTPResponse.getheaders()</code> and <code>HTTPResponse.getheader()</code> methods.</li>
<li>Fixed an issue where parsing a URL with leading zeroes in the port would be rejected
even when the port number after removing the zeroes was valid.</li>
<li>Fixed a deprecation warning when using cryptography v39.0.0.</li>
<li>Removed the <code><4</code> in the <code>Requires-Python</code> packaging metadata field.</li>
</ul>
<h1>1.26.12 (2022-08-22)</h1>
<ul>
<li>Deprecated the <code>urllib3[secure]</code> extra and the <code>urllib3.contrib.pyopenssl</code> module.
Both will be removed in v2.x. See this <code>GitHub issue <https://github.com/urllib3/urllib3/issues/2680></code>_
for justification and info on how to migrate.</li>
</ul>
<h1>1.26.11 (2022-07-25)</h1>
<ul>
<li>Fixed an issue where reading more than 2 GiB in a call to <code>HTTPResponse.read</code> would</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="9c2c2307dd"><code>9c2c230</code></a> Release 1.26.18 (<a href="https://redirect.github.com/urllib3/urllib3/issues/3159">#3159</a>)</li>
<li><a href="b594c5ceac"><code>b594c5c</code></a> Merge pull request from GHSA-g4mx-q9vg-27p4</li>
<li><a href="944f0eb134"><code>944f0eb</code></a> [1.26] Use vendored six in urllib3.contrib.securetransport</li>
<li><a href="c9016bf464"><code>c9016bf</code></a> Release 1.26.17</li>
<li><a href="01220354d3"><code>0122035</code></a> Backport GHSA-v845-jxx5-vc9f (<a href="https://redirect.github.com/urllib3/urllib3/issues/3139">#3139</a>)</li>
<li><a href="e63989f97d"><code>e63989f</code></a> Fix installing <code>brotli</code> extra on Python 2.7</li>
<li><a href="2e7a24d087"><code>2e7a24d</code></a> [1.26] Configure OS for RTD to fix building docs</li>
<li><a href="57181d6ea9"><code>57181d6</code></a> [1.26] Improve error message when calling urllib3.request() (<a href="https://redirect.github.com/urllib3/urllib3/issues/3058">#3058</a>)</li>
<li><a href="3c0148048a"><code>3c01480</code></a> [1.26] Run coverage even with failed jobs</li>
<li><a href="d94029b7e2"><code>d94029b</code></a> Release 1.26.16</li>
<li>Additional commits viewable in <a href="https://github.com/urllib3/urllib3/compare/1.26.5...1.26.18">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/grpc/grpc/network/alerts).
</details>
> **Note**
> Automatic rebases have been disabled on this pull request as it has been open for over 30 days.
Closes#34714
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/34714 from grpc:dependabot/pip/urllib3-1.26.18 3281b127f3
PiperOrigin-RevId: 597334547
This reverts commit 96b9e8d3e3.
[Implement OpenTelemetry PR](https://github.com/grpc/grpc/pull/35292) was [reverted](96b9e8d3e3) because some tests started failing after import the changes to g3.
After investigation, we found root cause, it can be fixed both on our side and on gapic API side, we opened an issue to [gapic API team](https://github.com/googleapis/python-api-core/issues/579), this PR will includes the fixes on our side.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35439
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35439 from XuanWang-Amos:reapply_otel 0133564438
PiperOrigin-RevId: 595746222
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35292
PiperOrigin-RevId: 595188404
### Changes in this PR
* Refactor and remove some Core/C++ dependencies to simplify Python Observability package build process.
* Refactored code to read config at Python layer.
* Enable observability build from source.
* Add observability to run_test.
* Currently it's only enabled in Linux.
* Add error handler in run_test loaders.
* Current framework will always visit modules in test directory then decide which tests to skip.
* Since we're not building Observability for MacOS and Windows this step will fail with error `No module named 'grpc_observability'`.
* After the change we'll just skip those modules.
* We still have `_sanity_test` to make sure all tests are loaded correctly for each platform.
* Remov OC dependency as we're migrating to OTel.
* Also removed trace from testing.
* Note that trace propagation function was also removed because of this.
### Testing
* Passed existing tests.
* Tested locally, able to build observability from source using `GRPC_PYTHON_BUILD_WITH_CYTHON=1 pip install .`.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#34207
PiperOrigin-RevId: 590258014
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35255
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35255 from gnossen:freeze_googleapis_bazel 38fb05d8db
PiperOrigin-RevId: 588965912
Bumps [gevent](https://github.com/gevent/gevent) from 22.08.0 to 23.9.1.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="3977b6b3ec"><code>3977b6b</code></a>
Preparing release 23.9.1</li>
<li><a
href="068645423b"><code>0686454</code></a>
Typo.</li>
<li><a
href="bdc82c9e47"><code>bdc82c9</code></a>
Bump to greenlet 3.0rc3</li>
<li><a
href="ca0f9cb7b4"><code>ca0f9cb</code></a>
Bump to greenlet 3.0rc2 and require it on Py3.11 as well as 3.12. Also,
since...</li>
<li><a
href="0047199e04"><code>0047199</code></a>
Back to development: 23.9.0.post2</li>
<li><a
href="06879924ed"><code>0687992</code></a>
Preparing release 23.9.0.post1</li>
<li><a
href="166ecf1bc0"><code>166ecf1</code></a>
Fix windows wheel builds; ensure mac wheel builds have the universal2
tag</li>
<li><a
href="9b72b8c54e"><code>9b72b8c</code></a>
Back to development: 23.9.1</li>
<li><a
href="693181e8e1"><code>693181e</code></a>
Preparing release 23.9.0</li>
<li><a
href="6fc78989b6"><code>6fc7898</code></a>
Set the cython version; go back to default wheel tags.</li>
<li>Additional commits viewable in <a
href="https://github.com/gevent/gevent/compare/22.08.0...23.9.1">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts page](https://github.com/grpc/grpc/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
[requirements.bazel.txt] :updating wheel package to 0.38.1 to fix the
vulnerability
input: lang/python
release notes: no
settings: release notes:no
updating wheel package to 0.38.1 to fix the vulnerability
#python
please do fix it quickly
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
We just found out that our current Bazel setup does not support Python
3.11.
Thus PR updates some dependencies to allow using Bazel in Python 3.11.
Cython:
* Cython [backported Python 3.11 support change to
0.29x](https://github.com/cython/cython/issues/4500), but it appears
that the Cython version we are using in Bazel does not include the fix,
so we're using the latest stable version instead.
Gevent:
* The first version of gevent that supports [Python 3.11 is
22.08.0](https://github.com/gevent/gevent/issues/1903#issuecomment-1303227507).
#### Testing
* Tested locally using Python 3.11 virtual environment, was able to
reproduce the issue and verified that those changes were able to fix it.
* WIP
* Add gevent test suite run under Bazel.
* Fix things up
* Yapf
* Fix up Bazel files
* Make py_grpc_test fancier
* Attempt to fix Windows RBE
* Attempt to kick GitHub
* Fix Python 2 runs
* Yet more fixes
* And the patch file too
* I am an idiot
* Mark gevent tests flaky
* Try to make rules_python more tolerant
* Typo
* Exclude reconnect test from gevent
* Remove unnecessary parts of patch
* Buildifier
* You saw nothing
* isort
* Move py_grpc_test to an internal-only file
* Review comments
* More reviewer comments
* Review
This commit resolves an interop test currently failing on master. Over
the past couple of weeks, bazel-based tests have been introduced. The
current setup does not appear to automatically handle transitive
dependencies. Instead, transitive dependencies such as `requests` have
been manually added to `requirements.bazel.txt`. It appears that the
build server happened to have the dependencies of the `requests` library
installed already, but later had a configuration wipe.
This was compounded by the google-auth library erroneously reporting
that the `requests` module itself was not installed. In fact, it was the
transitive dependencies of `requests` that were not being installed by
the build file. (third-order dependencies of our test)
I consider this a quick fix to get the build passing. In the long run,
we need to automatically resolve and install transitive dependencies in
our bazel builds.
Resolves: #17170
Add interop tests for gRPC Python. py_proto_library rules are added to
src/proto/grpc/testing/BUILD since grpc_proto_library is not compatible
with py_* rules.
'requests' python module is added to requirements.bazel.txt as it is a
dependency for google-auth. Previously, this was installed through
tools/run_tests/helper_scripts/build_python.sh before running tests.
Add interop tests for gRPC Python. py_proto_library rules are added to
src/proto/grpc/testing/BUILD since grpc_proto_library is not compatible
with py_* rules.
'requests' python module is added to requirements.bazel.txt as it is a
dependency for google-auth. Previously, this was installed through
tools/run_tests/helper_scripts/build_python.sh before running tests.
- Include new rules from bazelbuild/rules_python, which allow for
importing pip dependencies with Bazel.
- Make Cython BUILD file visible to WORKSPACE.
- Use a custom requirements file for Bazel python. This is added to
allow for requirements previously added only during build steps.
Xuan Wang
Atri Bhattacharya
Richard Belleville
dependabot[bot]
gRPC Team Bot
Vignesh Babu
suryadev99
Lidi Zheng
Esun Kim
Naresh
Mehrdad Afshari
Mehrdad Afshari