Implement Python CSM observability.
Design: [go/grpc-python-opentelemetry](http://goto.google.com/grpc-python-opentelemetry)
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36557
PiperOrigin-RevId: 639073741
https://github.com/grpc/grpc/pull/36753 has this ASAN test failure with the following error which doesn't seem to make sense. ([full log](https://btx.cloud.google.com/invocations/a587e5cc-ca1e-46ed-a3c3-199c581583db/targets))
```
Executing tests from //test/cpp/util:grpc_tool_test@poller=epoll1
-----------------------------------------------------------------------------
=================================================================
==15==ERROR: AddressSanitizer: odr-violation (0x7fcfa2961400):
[1] size=66 'typeinfo name for google::protobuf::compiler::java::ImmutableExtensionLiteGenerator' external/com_google_protobuf/src/google/protobuf/compiler/java/lite/extension.cc in /b/f/w/bazel-out/k8-fastbuild/bin/test/cpp/util/grpc_tool_test@poller=epoll1.runfiles/com_github_grpc_grpc/test/cpp/util/../../../_solib_k8/libexternal_Scom_Ugoogle_Uprotobuf_Ssrc_Sgoogle_Sprotobuf_Scompiler_Sjava_Slite_Sliblite.so
[2] size=66 'typeinfo name for google::protobuf::compiler::java::ImmutableExtensionLiteGenerator' external/com_google_protobuf/src/google/protobuf/compiler/java/lite/extension.cc in /b/f/w/bazel-out/k8-fastbuild/bin/test/cpp/util/grpc_tool_test@poller=epoll1.runfiles/com_github_grpc_grpc/test/cpp/util/../../../_solib_k8/libexternal_Scom_Ugoogle_Uprotobuf_Ssrc_Sgoogle_Sprotobuf_Scompiler_Sjava_Slite_Slibfield_Ugenerators.so
These globals were registered at these points:
[1]:
#0 0x5626b4ea1488 in __asan_register_globals /tmp/clang-build/src/compiler-rt/lib/asan/asan_globals.cpp:369:3
#1 0x5626b4ea2559 in __asan_register_elf_globals /tmp/clang-build/src/compiler-rt/lib/asan/asan_globals.cpp:352:3
#2 0x7fcfa3fa4b99 (/lib64/ld-linux-x86-64.so.2+0x11b99) (BuildId: 7ae2aaae1a0e5b262df913ee0885582d2e327982)
[2]:
#0 0x5626b4ea1488 in __asan_register_globals /tmp/clang-build/src/compiler-rt/lib/asan/asan_globals.cpp:369:3
#1 0x5626b4ea2559 in __asan_register_elf_globals /tmp/clang-build/src/compiler-rt/lib/asan/asan_globals.cpp:352:3
#2 0x7fcfa3fa4b99 (/lib64/ld-linux-x86-64.so.2+0x11b99) (BuildId: 7ae2aaae1a0e5b262df913ee0885582d2e327982)
==15==HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_odr_violation=0
SUMMARY: AddressSanitizer: odr-violation: global 'typeinfo name for google::protobuf::compiler::java::ImmutableExtensionLiteGenerator' at external/com_google_protobuf/src/google/protobuf/compiler/java/lite/extension.cc in /b/f/w/bazel-out/k8-fastbuild/bin/test/cpp/util/grpc_tool_test@poller=epoll1.runfiles/com_github_grpc_grpc/test/cpp/util/../../../_solib_k8/libexternal_Scom_Ugoogle_Uprotobuf_Ssrc_Sgoogle_Sprotobuf_Scompiler_Sjava_Slite_Sliblite.so
==15==ABORTING
```
This turned out to be a known issue described at https://github.com/google/sanitizers/issues/1017 and there is much to do other than disabling the ODR test. I gave a try to "-mllvm -asan-use-private-alias=1" option but it didn't change the result so I turned into this way.
Closes#36756
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36756 from veblush:asan-workaround 622581e8d2
PiperOrigin-RevId: 638421967
Bumps [requests](https://github.com/psf/requests) from 2.31.0 to 2.32.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/psf/requests/releases">requests's releases</a>.</em></p>
<blockquote>
<h2>v2.32.2</h2>
<h2>2.32.2 (2024-05-21)</h2>
<p><strong>Deprecations</strong></p>
<ul>
<li>
<p>To provide a more stable migration for custom HTTPAdapters impacted
by the CVE changes in 2.32.0, we've renamed <code>_get_connection</code> to
a new public API, <code>get_connection_with_tls_context</code>. Existing custom
HTTPAdapters will need to migrate their code to use this new API.
<code>get_connection</code> is considered deprecated in all versions of Requests>=2.32.0.</p>
<p>A minimal (2-line) example has been provided in the linked PR to ease
migration, but we strongly urge users to evaluate if their custom adapter
is subject to the same issue described in CVE-2024-35195. (<a href="https://redirect.github.com/psf/requests/issues/6710">#6710</a>)</p>
</li>
</ul>
<h2>v2.32.1</h2>
<h2>2.32.1 (2024-05-20)</h2>
<p><strong>Bugfixes</strong></p>
<ul>
<li>Add missing test certs to the sdist distributed on PyPI.</li>
</ul>
<h2>v2.32.0</h2>
<h2>2.32.0 (2024-05-20)</h2>
<h2>🐍 PYCON US 2024 EDITION 🐍</h2>
<p><strong>Security</strong></p>
<ul>
<li>Fixed an issue where setting <code>verify=False</code> on the first request from a
Session will cause subsequent requests to the <em>same origin</em> to also ignore
cert verification, regardless of the value of <code>verify</code>.
(<a href="https://github.com/psf/requests/security/advisories/GHSA-9wx4-h78v-vm56">https://github.com/psf/requests/security/advisories/GHSA-9wx4-h78v-vm56</a>)</li>
</ul>
<p><strong>Improvements</strong></p>
<ul>
<li><code>verify=True</code> now reuses a global SSLContext which should improve
request time variance between first and subsequent requests. It should
also minimize certificate load time on Windows systems when using a Python
version built with OpenSSL 3.x. (<a href="https://redirect.github.com/psf/requests/issues/6667">#6667</a>)</li>
<li>Requests now supports optional use of character detection
(<code>chardet</code> or <code>charset_normalizer</code>) when repackaged or vendored.
This enables <code>pip</code> and other projects to minimize their vendoring
surface area. The <code>Response.text()</code> and <code>apparent_encoding</code> APIs
will default to <code>utf-8</code> if neither library is present. (<a href="https://redirect.github.com/psf/requests/issues/6702">#6702</a>)</li>
</ul>
<p><strong>Bugfixes</strong></p>
<ul>
<li>Fixed bug in length detection where emoji length was incorrectly
calculated in the request content-length. (<a href="https://redirect.github.com/psf/requests/issues/6589">#6589</a>)</li>
<li>Fixed deserialization bug in JSONDecodeError. (<a href="https://redirect.github.com/psf/requests/issues/6629">#6629</a>)</li>
<li>Fixed bug where an extra leading <code>/</code> (path separator) could lead
urllib3 to unnecessarily reparse the request URI. (<a href="https://redirect.github.com/psf/requests/issues/6644">#6644</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/psf/requests/blob/main/HISTORY.md">requests's changelog</a>.</em></p>
<blockquote>
<h2>2.32.2 (2024-05-21)</h2>
<p><strong>Deprecations</strong></p>
<ul>
<li>
<p>To provide a more stable migration for custom HTTPAdapters impacted
by the CVE changes in 2.32.0, we've renamed <code>_get_connection</code> to
a new public API, <code>get_connection_with_tls_context</code>. Existing custom
HTTPAdapters will need to migrate their code to use this new API.
<code>get_connection</code> is considered deprecated in all versions of Requests>=2.32.0.</p>
<p>A minimal (2-line) example has been provided in the linked PR to ease
migration, but we strongly urge users to evaluate if their custom adapter
is subject to the same issue described in CVE-2024-35195. (<a href="https://redirect.github.com/psf/requests/issues/6710">#6710</a>)</p>
</li>
</ul>
<h2>2.32.1 (2024-05-20)</h2>
<p><strong>Bugfixes</strong></p>
<ul>
<li>Add missing test certs to the sdist distributed on PyPI.</li>
</ul>
<h2>2.32.0 (2024-05-20)</h2>
<p><strong>Security</strong></p>
<ul>
<li>Fixed an issue where setting <code>verify=False</code> on the first request from a
Session will cause subsequent requests to the <em>same origin</em> to also ignore
cert verification, regardless of the value of <code>verify</code>.
(<a href="https://github.com/psf/requests/security/advisories/GHSA-9wx4-h78v-vm56">https://github.com/psf/requests/security/advisories/GHSA-9wx4-h78v-vm56</a>)</li>
</ul>
<p><strong>Improvements</strong></p>
<ul>
<li><code>verify=True</code> now reuses a global SSLContext which should improve
request time variance between first and subsequent requests. It should
also minimize certificate load time on Windows systems when using a Python
version built with OpenSSL 3.x. (<a href="https://redirect.github.com/psf/requests/issues/6667">#6667</a>)</li>
<li>Requests now supports optional use of character detection
(<code>chardet</code> or <code>charset_normalizer</code>) when repackaged or vendored.
This enables <code>pip</code> and other projects to minimize their vendoring
surface area. The <code>Response.text()</code> and <code>apparent_encoding</code> APIs
will default to <code>utf-8</code> if neither library is present. (<a href="https://redirect.github.com/psf/requests/issues/6702">#6702</a>)</li>
</ul>
<p><strong>Bugfixes</strong></p>
<ul>
<li>Fixed bug in length detection where emoji length was incorrectly
calculated in the request content-length. (<a href="https://redirect.github.com/psf/requests/issues/6589">#6589</a>)</li>
<li>Fixed deserialization bug in JSONDecodeError. (<a href="https://redirect.github.com/psf/requests/issues/6629">#6629</a>)</li>
<li>Fixed bug where an extra leading <code>/</code> (path separator) could lead
urllib3 to unnecessarily reparse the request URI. (<a href="https://redirect.github.com/psf/requests/issues/6644">#6644</a>)</li>
</ul>
<p><strong>Deprecations</strong></p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="88dce9d854"><code>88dce9d</code></a> v2.32.2</li>
<li><a href="c98e4d133e"><code>c98e4d1</code></a> Merge pull request <a href="https://redirect.github.com/psf/requests/issues/6710">#6710</a> from nateprewitt/api_rename</li>
<li><a href="92075b330a"><code>92075b3</code></a> Add deprecation warning</li>
<li><a href="aa1461b68a"><code>aa1461b</code></a> Move _get_connection to get_connection_with_tls_context</li>
<li><a href="970e8cec98"><code>970e8ce</code></a> v2.32.1</li>
<li><a href="d6ebc4a2f1"><code>d6ebc4a</code></a> v2.32.0</li>
<li><a href="9a40d12778"><code>9a40d12</code></a> Avoid reloading root certificates to improve concurrent performance (<a href="https://redirect.github.com/psf/requests/issues/6667">#6667</a>)</li>
<li><a href="0c030f78d2"><code>0c030f7</code></a> Merge pull request <a href="https://redirect.github.com/psf/requests/issues/6702">#6702</a> from nateprewitt/no_char_detection</li>
<li><a href="555b870eb1"><code>555b870</code></a> Allow character detection dependencies to be optional in post-packaging steps</li>
<li><a href="d6dded3f00"><code>d6dded3</code></a> Merge pull request <a href="https://redirect.github.com/psf/requests/issues/6700">#6700</a> from franekmagiera/update-redirect-to-invalid-uri-test</li>
<li>Additional commits viewable in <a href="https://github.com/psf/requests/compare/v2.31.0...v2.32.2">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/grpc/grpc/network/alerts).
</details>
Closes#36681
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36681 from grpc:dependabot/pip/tools/internal_ci/helper_scripts/requests-2.32.2 c7600bdbee
PiperOrigin-RevId: 638081350
Introduce "Yodel" - a framework for testing things vaguely related to calls.
This is breaking up some work I did for transport test suites - we've got a nice way of spawning test-only promises and tracking them visually, and support for setting up an environment that can run as a test or a fuzzer. I'm making that piece a little more reusable, and then rebasing the transport test suite atop that infrastructure.
Closes#36635
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36635 from ctiller:transport-refs-6 843a9f4b7e
PiperOrigin-RevId: 637022756
Updated create_matrix_images.py to add an additional tag, infrastructure-public-image-ORIGINAL_TAG to a new image to be excluded from the GCP Vulnerability scanner.
Closes#36695
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36695 from arjan-bal:docker-tag-interop-images 28b6a14dc0
PiperOrigin-RevId: 636996322
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - gpr_log
In this CL we are migrating from gRPCs own gpr logging mechanism to absl logging mechanism. The intention is to deprecate gpr_log in the future.
We have the following mapping
1. gpr_log(GPR_INFO,...) -> LOG(INFO)
2. gpr_log(GPR_ERROR,...) -> LOG(ERROR)
3. gpr_log(GPR_DEBUG,...) -> VLOG(2)
Reviewers need to check :
1. If the above mapping is correct.
2. The content of the log is as before.
gpr_log format strings did not use string_view or std::string . absl LOG accepts these. So there will be some elimination of string_view and std::string related conversions. This is expected.
Closes#36694
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36694 from tanvi-jagtap:python_codegen_gpr_log 54ee362e3d
PiperOrigin-RevId: 636071998
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36651
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36651 from sergiitk:java-1.63.1 409d48e1fb
PiperOrigin-RevId: 634815551
This is to address the following error error thrown when running protobuf-at-head. This is because Bazel needs a C++ compiler to run `tools/codegen/core/gen_upb_api.sh`.
```
Auto-Configuration Error: Cannot find gcc or CC (clang); either correct your path or set the CC environment variable
```
This is not ideal as installing `build-essential` package takes time but let's fix it first and get this installed in the CI image later.
Closes#36628
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36628 from veblush:p-fix-at-head 01ac20f057
PiperOrigin-RevId: 634114867
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36509
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36509 from ctiller:transport-refs-3 2771a2b0e1
PiperOrigin-RevId: 633240374
Not the fastest implementation possible, but it's a log helper so I'm not particularly fussed either -- but a useful utility that we can iterate on later to help debugging.
Closes#36554
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36554 from ctiller:args 33b4802fcc
PiperOrigin-RevId: 632997320
Ban GPR_ASSERT and GPR_DEBUG_ASSERT.
These macros are deprecated and should be replaced with absl CHECK and absl DCHECK, respectively.
PiperOrigin-RevId: 632024499
GCC 7 was disabled due to its gcc internal error while building protobuf C++. But we still need to support gcc7 and there appear to be issues causing gcc 7 to not build grpc so let's reintroduce gcc 7 test.
Closes#36565
PiperOrigin-RevId: 631947203
This is the same change as in #36556, but the change was committed without the second commit that existed in the PR (due to a very wide wait-for-CI-before-syncing race).
The sha here is what's currently in testing-images-public. I don't know where it came from but it has the proper update.
Closes#36562
PiperOrigin-RevId: 631871629
The last time it was rebuilt, the base image (dart:stable) seems not to have been updated. grpc-dart is now requiring newer dart version and is currently failing: https://github.com/grpc/grpc-dart/issues/705Closes#36556
PiperOrigin-RevId: 631850210
Change was created by the release automation script. See go/grpc-release.
Closes#36544
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36544 from yashykt:bump_dev_version_202405061956 69ee5c869e
PiperOrigin-RevId: 631187829
Integrates the new features of the the Kokoro PSM Interop install library introduced in grpc/psm-interop#73.
Nearly all common functionality was moved from per-language/per-branch PSM Interop build scripts to [psm_interop_kokoro_lib.sh](https://github.com/grpc/psm-interop/blob/main/.kokoro/psm_interop_kokoro_lib.sh):
1. The list of tests in the each test suite
2. Per-test-suite flag customization
3. `run_test` methods
4. `build_docker_images_if_needed` methods
5. Generic `build_test_app_docker_images` methods (simple docker build + docker push + docker tag). grpc-java is one exception, as it doesn't run docker directly, but a cloudbuild flow.
Now all PSM Interop jobs share the same buildscripts by all test suites:
1. buildscript that invokes the test: `psm-interop-test-{language}.sh` (configured as `build_file` in the build cfg)
2. buildscript that builds the xDS test client/server and publishes them as a Docker image: `psm-interop-build-{language}.sh` (conventional name called from `psm_interop_kokoro_lib.sh`)
`psm-interop-test-{language}.sh`:
1. Sets `GRPC_LANGUAGE`, `BUILD_SCRIPT_DIR` environment variables.
2. Downloads the shared `psm_interop_kokoro_lib.sh` from the main branch of the psm-interop repo.
3. Sources `psm-interop-build-{language}.sh`
4. Calls `psm::run "${PSM_TEST_SUITE}"` (`PSM_TEST_SUITE` configured in the cfg file).
`psm-interop-build-{language}.sh`:
1. Defines `psm::lang::build_docker_images` which is called from `psm_interop_kokoro_lib.sh`.
2. Invokes any repo-specific logic.
3. May use `psm::build::docker_images_generic` for generic Docker build, tag, push, or provide implement its own build/publish method.
References:
- b/288578634
- See the full list of the new features at grpc/psm-interop#73.
- Additional fixes to the shared lib: grpc/psm-interop#78, grpc/psm-interop#79
Closes#36450
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36450 from sergiitk:psm-interop-common-prod-tests 4b977b7bd8
PiperOrigin-RevId: 630502807
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36516
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36516 from yijiem:interop-matrix-1.63.0 742d1ef527
PiperOrigin-RevId: 630501618
Reasoning:
* This benchmark will need to be rewritten to work with the new transport API by EOY anyhow, and the API is fairly different.
* Deleting this saves us from having to migrate the `grpc_endpoint` implementation to `EventEngine::Endpoint`.
Closes#36514
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36514 from drfloob:nix-bm_chttp2_transport e6c6edf39f
PiperOrigin-RevId: 630408686
This should be able to unblock using grpc on vision os, will add tests later
Closes#36476
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36476 from HannahShiSFB:support-visionos edbde32d23
PiperOrigin-RevId: 630173448
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging GPR_ASSERT
Replacing GPR_ASSERT with absl CHECK.
These changes have been made using string replacement and regex.
Will not be replacing all instances of CHECK with CHECK_EQ , CHECK_NE etc because there are too many callsites. Only ones which are doable using very simple regex with least chance of failure will be replaced.
Given that we have 5000+ instances of GPR_ASSERT to edit, Doing it manually is too much work for both the author and reviewer.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36467
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36467 from tanvi-jagtap:tjagtap_src_core_lib 30d3ff5bbb
PiperOrigin-RevId: 629995895
Mark D. Roth
Xuan Wang
Yash Tibrewal
Paulo Castello da Costa
gRPC Team Bot
Craig Tiller
Esun Kim
dependabot[bot]
Arjan Singh Bal
Tanvi Jagtap
AJ Heller
Eryu Xia
Sergii Tkachenko
Terry Wilson
Tanvi Jagtap
Eric Anderson
Yijie Ma
Hannah Shi