[PH2][Refactor] Creating call_tracer_wrapper.h and cc and moving code as-is into it. The content of the functions or classes has not changed.
In the next few iterations, more code will come into this file.
Closes#37786
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37786 from tanvi-jagtap:ph2_internal_h_split_01 31af14a4e7
PiperOrigin-RevId: 680861154
IDK how many times I've messed this up over the years accidentally and had to spend thoughts trying to fix it... when really we could do that trivially with code.
Closes#37811
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37811 from ctiller:bamboozled 0cfbeae925
PiperOrigin-RevId: 679741188
This is the last piece of gRFC A83 (https://github.com/grpc/proposal/pull/438).
Note that although this is the first use-case for this "blackboard" mechanism, we will also use it in the future for the xDS rate-limiting filter on the gRPC server side.
Closes#37646
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37646 from markdroth:gcp_auth_filter_state 72d0d96c79
PiperOrigin-RevId: 679707134
Building armv7 artifacts after adding Python 3.13 is failing with a timeout. Hence increasing timeout to 2 hours.
Closes#37807
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37807 from sreenithi:increase_armv7_artifact_build_timeout 60034378f5
PiperOrigin-RevId: 679608393
Moving out related code into a separate file.
`internal.h -> stream_lists.h`
Closes#37775
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37775 from tanvi-jagtap:ph2_stream_lists_h 13b6030ceb
PiperOrigin-RevId: 679205824
This PR adds templating for Python versions and updates the maximum supported Python version to 3.13. The following major changes related to templating are added:
- Minimum supported Python version and list of supported versions in `setup.py` are fetched using new template generated files called `python_version.py`
- Dockerfiles for the different Python Linux builds are now template generated.
- The "Supported Python Versions" section from READMEs of ancillary and main packages have been removed
Note: All the `python_version.py` files and Linux build `Dockerfiles` except `tools/dockerfile/grpc_artifact_python_linux_armv7/Dockerfile` in the PR are generated from the respective templates.
Further non-templated additions to add support for Python 3.13:
- install scripts and artifacts for windows, macos and linux are added manually. Later, these can be templated as well.
- updated cython bounds to 3.x
- updated twine version to solve [cgi module import error](https://github.com/pypa/twine/issues/1046)
- the twine update introduces a dependency on cryptography>=2.0. But the cryptography package doesn't support 32-bit Linux images and hence `twine check` has been disabled for x86 manylinux and x86 musllinux artifacts.
Closes#37643
PiperOrigin-RevId: 678954495
1. Added script to install all dependencies then run all unit tests for python grpcio.
Closes#37070
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37070 from sourabhsinghs:feature/developer-setup-script 954892f20a
PiperOrigin-RevId: 678718224
Basic building block for retries, hedging: buffer outgoing messages & metadata, allow for replay whilst buffered (with a single reader able to read once buffering ends)
Closes#37448
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37448 from ctiller:once-again-into-the-breach-my-friends 79cb121054
PiperOrigin-RevId: 677959212
Passing manual build: https://btx.cloud.google.com/invocations/9adb3cd8-628c-42e5-a7a3-c2ba1ab6127f
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37723
PiperOrigin-RevId: 677900305
The following files have been moved:
- src/core/lib/avl/*
- src/core/lib/backoff/*
- src/core/lib/debug/event_log*
- src/core/lib/iomgr/gethostname*
- src/core/lib/iomgr/grpc_if_nametoindex*
- src/core/lib/matchers/*
- src/core/lib/uri/* (renamed from uri_parser.* to uri.*)
- src/core/lib/gprpp/* (existing src/core/util/time.cc was renamed to gpr_time.cc to avoid conflict)
Closes#36792
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36792 from markdroth:reorg_util d4e8996f48
PiperOrigin-RevId: 676947640
`grpc/core/master/macos/grpc_basictests_python` test is flaky, all test cases passed by the test itself [failed with timeout](https://btx.cloud.google.com/invocations/33bdec48-e17b-431b-b473-c8a335e1650c/log):
```
2024-09-11 16:48:57,385 TIMEOUT: py38.asyncio.tests_aio.unit.done_callback_test.TestServerSideDoneCallback [pid=11128, time=486.5sec]
```
This PR does two things:
1. Increase timeout for test case from `480s` to `600s`.
2. Add timestamp to test starting and passing so we know which test case takes a long time to run.
Log before timestamp change:
```
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_stream
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_unary
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_stream
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_unary
Testing gRPC Python...
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_error_in_callback
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_error_in_handler
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_stream
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_unary
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_stream
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_unary
```
Log after timestamp change:
```
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_stream
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_unary
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_stream
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_unary
[2024-09-12 10:47:13.877185]Testing gRPC Python...
[2024-09-12 10:47:13.901135]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_add_after_done
[2024-09-12 10:47:13.933705]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_stream
[2024-09-12 10:47:13.962419]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_unary
[2024-09-12 10:47:13.990368]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_stream
```
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37703
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37703 from XuanWang-Amos:add_test_timestamp 4e13d29021
PiperOrigin-RevId: 675703890
As title.
Bump core version since there were API changes (to logging APIs) since the last release.
Closes#37661
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37661 from apolcyn:bump_core_version_202409092234 190b88342b
PiperOrigin-RevId: 672984841
This is flaky in CI, and is being replaced by the new implementation with event engine, disabling this test.
Will clean them up after all have switched to event engine.
Closes#37636
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37636 from HannahShiSFB:disable-iomgr-cfstream-test b430ee3d7c
PiperOrigin-RevId: 671399515
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37572
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37572 from yijiem:upgrade-clang-7 03f8cdc54e
PiperOrigin-RevId: 670692896
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37572
PiperOrigin-RevId: 670631284
We have base images that still need to be migrated (for instance, from marketplace.gcr.io). This change restores support for GCR, to be maintained until all images are migrated.
Closes#37596
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37596 from paulosjca:ar 9c4c91b9bb
PiperOrigin-RevId: 669396634
Final piece of gRFC A83 (https://github.com/grpc/proposal/pull/438): the GCP authentication filter itself.
Infrastructure changes include:
- Added a general-purpose LRU cache library that can be reused elsewhere.
- Fixed the client channel code to use the channel args returned by the resolver for the dynamic filters. This was necessary so that the GCP auth filter could access the `XdsConfig` object, which is passed via a channel arg.
- Unlike the other xDS HTTP filters we support, the GCP auth filter does not support config overrides, and its configuration includes a cache size parameter that we always need at the channel level, not per-call. As a result, I had to change the xDS HTTP filter API to give it the ability to set top-level fields in the service config, not just per-method fields. (We use the service config as a way of passing configuration down into xDS HTTP filters.) Note that for now, this works only on the client side, because we don't have machinery for a top-level service config on the server side.
- The GCP auth filter is also the first case where the filter needs to know its instance name from the xDS config, so I changed the xDS HTTP filter API to plumb that through.
- Fixed a bug in the HTTP client library that prevented the override functions from declining to override a particular request.
Closes#37550
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37550 from markdroth:xds_gcp_auth_filter 19eaefb52f
PiperOrigin-RevId: 669371249
[Gpr_To_Absl_Logging] Remove gpr_log. Adding absl LOG wrappers
List of changes in this PR
1. Replacing all instances of gpr_log in PHP and RUBY with the new absl wrapper APIs. The replacement mapping is given below
gpr_log(GPR_ERROR, ...)
=> grpc_absl_log_error
gpr_log(GPR_INFO, ...)
=> grpc_absl_log_info - Printing a simple message
=> grpc_absl_log_info_int - Printing a message and a number
=> grpc_absl_log_info_str - Printing 2 strings.
gpr_log(GPR_DEBUG, ...)
=> grpc_absl_vlog - Printing a simple message
=> grpc_absl_vlog_int - Printing a message and a number
=> grpc_absl_vlog_str - Printing 2 strings.
Adding grpc_absl_vlog2_enabled() check around gpr_log(GPR_DEBUG, ...)
2. src/python/grpcio_observability/grpc_observability/observability_util.cc One instance of gpr_log to absl LOG replacement was missed earlier. Fixing that.
3. Deleting deprecated gpr stuff : gpr_log_severity , GPR_DEBUG , GPR_INFO , GPR_ERROR , gpr_log .
4. Adding new APIs for Ruby and PHP. These APIs are very simple wrappers around absl.
5. Removing the legacy functions in platform specific log.cc files. These files are safe to delete now.
6. Fixing the allow list in banned_functions.py . This makes sure that these new wrappers don't get used all over the place by everyone. We carefully only allow list the PHP and RUBY files and allow the use of these wrappers. Everywhere else - using these wrappers should fail Sanity Tests.
Closes#37431
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37431 from tanvi-jagtap:remove_gpr_error 6e5e9bcfcc
PiperOrigin-RevId: 668586873
gcc 7 and 8 docker images are based on Debian 10 with older cmake than gRPC needs. Since it's not easy to change the base Debian image for these docker images, let's install a newer cmake instead. This is a left-over from https://github.com/grpc/grpc/pull/37547.
Closes#37571
PiperOrigin-RevId: 668560102
Add validation of the `Audience` cluster metadata type, as per gRFC A83 (https://github.com/grpc/proposal/pull/438).
I had previously changed the metadata to be represented as JSON in #37468. However, while working on the GCP Authentication filter implementation, I realized that that's not an ideal representation, because it would have required us to validate the JSON on a per-RPC basis, which would be bad for performance. So I've changed the representation of metadata to be an abstract type, and we now store the `Audience` metadata as a simple string. I've also moved metadata into its own type with its own validation code, so that in the future we can use it in places other than CDS (many xDS resource types have metadata fields).
While I was at it, I also add some helper functions for validating the `UInt32Value` and `UInt64Value` wrapper protos.
Closes#37566
PiperOrigin-RevId: 668281729
Artifact Registry requires the registry as an argument for `gcloud auth configure-docker`.
Closes#37576
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37576 from paulosjca:ar 1ab668fef1
PiperOrigin-RevId: 667769875
This change moves the location for the prebuilt images used in gRPC OSS benchmarks from GCR to Artifact Registry.
Images are all saved to the same repo, instead of being organized by a prefix (`kokoro`, `kokoro-test`,
`cxx_experiment/kokoro-test`, etc.).
Closes#37558
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37558 from paulosjca:ar a767ab5d91
PiperOrigin-RevId: 666643972
Previously, `grpc_oauth2_token_fetcher_credentials` provided functionality for on-demand token-fetching, but it was integrated into the oauth2 code, so it was not possible to use that same code for on-demand fetching of (e.g.) JWT tokens. This PR splits that class into two parts:
1. A base `TokenFetcherCredentials` class that provides a framework for on-demand fetching of any arbitrary type of auth token.
2. An `Oauth2TokenFetcherCredentials` subclass that derives from `TokenFetcherCredentials` and provides handling for oauth2 tokens.
The `grpc_compute_engine_token_fetcher_credentials`, `StsTokenFetcherCredentials`, and `grpc_google_refresh_token_credentials` classes that previously derived from `grpc_oauth2_token_fetcher_credentials` now derive from `Oauth2TokenFetcherCredentials` instead, so there's not much change to those classes (other than a cleaner interface with the base class functionality).
The `ExternalAccountCredentials` class and its subclasses got more extensive changes here. Previously, this class inheritted from `grpc_oauth2_token_fetcher_credentials` and fooled the base class into thinking that it directly fetched the oauth2 token, when in fact it actually performed a number of steps to gather data and then constructed a synthetic HTTP response to pass back to the base class. I have changed this to instead derive directly from `TokenFetcherCredentials` to provide a much cleaner interface with the parent class.
In addition, I have changed `grpc_call_credentials` from `RefCounted<>` to `DualRefCounted<>` to provide a clean way to shut down any in-flight token fetch when the credentials are unreffed.
This PR paves the way for subsequent work that will allow implementing an on-demand JWT token fetcher call credential, as part of gRFC A83 (https://github.com/grpc/proposal/pull/438).
Closes#37510
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37510 from markdroth:token_fetcher_call_creds_refactor 3bd398a762
PiperOrigin-RevId: 666547985
Arjan Singh Bal
Tanvi Jagtap
Esun Kim
Craig Tiller
Mark D. Roth
Sreenithi Sridharan
Paulo Castello da Costa
erm-g
Sourabh Singh
Hannah Shi
Xuan Wang
apolcyn
Marcel
Yijie Ma
AJ Heller
Eugene Ostroukhov