Some of our tests failed because RPC was canceled by server, error message:
```
I0000 00:00:1726598495.788119 64528 chttp2_transport.cc:1182] ipv4:127.0.0.1:41825: Got goaway [2] err=UNAVAILABLE:GOAWAY received; Error code: 2; Debug Text: Cancelling all calls {grpc_status:14, http2_error:2, created_time:"2024-09-17T18:41:35.78810597+00:00"}
```
It's possible that the request waited too long to be processed and thus canceled by server (default maximum time gRPC server allows a request to be received by the server but not handled is `30s`, controlled by `grpc.server_max_unrequested_time_in_server`).
This PR will print the test start/finish time so we know if we should increase `grpc.server_max_unrequested_time_in_server` time.
Example before change:
```
Testing gRPC Python...
SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_default
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled
```
Example after change:
```
[2024-09-20 18:32:15.228947]Testing gRPC Python...
[2024-09-20 18:32:15.237855]START tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_default
[2024-09-20 18:32:15.265477]SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_default[Duration: 0:00:00.027622s]
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
[2024-09-20 18:32:15.278142]START tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
[2024-09-20 18:32:15.316065]FAILURE tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled[Duration: 0:00:00.037911]
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled
[2024-09-20 18:32:15.328680]START tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled
[2024-09-20 18:32:16.938745]SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled[Duration: 0:00:01.610065s]
```
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37761
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37761 from XuanWang-Amos:add_time_to_unit_test d2ecd8ce22
PiperOrigin-RevId: 677805375
This is a retry of #37762 with a fix. We no longer hold the C mutex and GIL at the same to avoid a deadlock.
MetadataPluginCallCredentials passes a function pointer to C-core to be called during destruction. This function grabs GIL which may race between GIL destruction during process shutdown. Since GIL destruction happens after Python's exit handlers, we cana mark that Python is shutting down from an exit handler and don't grab GIL in this function afterwards using a C mutex.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37784
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37784 from yousukseung:python-atexit-retry b17addf805
PiperOrigin-RevId: 677280657
The following files have been moved:
- src/core/lib/avl/*
- src/core/lib/backoff/*
- src/core/lib/debug/event_log*
- src/core/lib/iomgr/gethostname*
- src/core/lib/iomgr/grpc_if_nametoindex*
- src/core/lib/matchers/*
- src/core/lib/uri/* (renamed from uri_parser.* to uri.*)
- src/core/lib/gprpp/* (existing src/core/util/time.cc was renamed to gpr_time.cc to avoid conflict)
Closes#36792
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36792 from markdroth:reorg_util d4e8996f48
PiperOrigin-RevId: 676947640
This change appears to cause a deadlock.
This reverts commit 4b53c2bac3.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37769
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37769 from yousukseung:python_atexit_rollback 9218d1066e
PiperOrigin-RevId: 676566209
MetadataPluginCallCredentials passes a function pointer to C-core to be called during destruction. This function grabs GIL which may race between GIL destruction during process shutdown. Since GIL destruction happens after Python's exit handlers, we cana mark that Python is shutting down from an exit handler and don't grab GIL in this function afterwards using a C mutex.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37762
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37762 from yousukseung:gil-shutdown 945eca9c6a
PiperOrigin-RevId: 676471627
`grpc/core/master/macos/grpc_basictests_python` test is flaky, all test cases passed by the test itself [failed with timeout](https://btx.cloud.google.com/invocations/33bdec48-e17b-431b-b473-c8a335e1650c/log):
```
2024-09-11 16:48:57,385 TIMEOUT: py38.asyncio.tests_aio.unit.done_callback_test.TestServerSideDoneCallback [pid=11128, time=486.5sec]
```
This PR does two things:
1. Increase timeout for test case from `480s` to `600s`.
2. Add timestamp to test starting and passing so we know which test case takes a long time to run.
Log before timestamp change:
```
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_stream
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_unary
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_stream
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_unary
Testing gRPC Python...
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_error_in_callback
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_error_in_handler
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_stream
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_unary
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_stream
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_unary
```
Log after timestamp change:
```
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_stream
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_unary
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_stream
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_unary
[2024-09-12 10:47:13.877185]Testing gRPC Python...
[2024-09-12 10:47:13.901135]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_add_after_done
[2024-09-12 10:47:13.933705]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_stream
[2024-09-12 10:47:13.962419]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_unary
[2024-09-12 10:47:13.990368]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_stream
```
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37703
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37703 from XuanWang-Amos:add_test_timestamp 4e13d29021
PiperOrigin-RevId: 675703890
This test is flaky, example failure: https://fusion2.corp.google.com/invocations/31bcd70f-650d-40be-8ec4-6fafc3cd3724, error message:
```
Traceback (most recent call last):
File "/usr/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/lib/python3.9/unittest/case.py", line 593, in run
self._callTestMethod(testMethod)
File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
method()
File "/var/local/git/grpc/src/python/grpcio_tests/tests_aio/unit/_test_base.py", line 31, in wrapper
return loop.run_until_complete(f(*args, **kwargs))
File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/var/local/git/grpc/src/python/grpcio_tests/tests_aio/unit/call_test.py", line 820, in test_cancel_after_done_writing
self.assertFalse(call.done())
File "/usr/lib/python3.9/unittest/case.py", line 676, in assertFalse
raise self.failureException(msg)
AssertionError: True is not false
```
We're expecting call to NOT finish immediately after `await call.done_writing()` which is not correct.
* `call.done_writing()` only notifies server that the client is done sending messages, while `call.done()` checks if the RPC call itself has finished.
Thus removing assert check for `call.done()`.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37651
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37651 from XuanWang-Amos:fix_aio_cancel_after_done_writing 42add84236
PiperOrigin-RevId: 671834305
This test is flaky for a while now, most of the time it failed with timeout error, one [example](https://btx.cloud.google.com/invocations/c31650ae-afdd-4e50-bfc4-b9a43588d3ba/log):
```
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
yield
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 676, in run
self._callTestMethod(testMethod)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 633, in _callTestMethod
method()
File "/Volumes/BuildData/tmpfs/altsrc/github/grpc/workspace_python_macos_opt_native/src/python/grpcio_tests/tests_aio/unit/_test_base.py", line 31, in wrapper
return loop.run_until_complete(f(*args, **kwargs))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/Volumes/BuildData/tmpfs/altsrc/github/grpc/workspace_python_macos_opt_native/src/python/grpcio_tests/tests_aio/unit/connectivity_test.py", line 54, in test_unavailable_backend
await asyncio.wait_for(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
```
Suspect the root cause is the same mentioned here: https://github.com/grpc/grpc/pull/26409
Instead of disable this for MacOS, changing the timeout from `4s` to `8s` seems reasonable too.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37645
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37645 from XuanWang-Amos:fix_aio_connectivity_test 10fdf90635
PiperOrigin-RevId: 671523025
Final piece of gRFC A83 (https://github.com/grpc/proposal/pull/438): the GCP authentication filter itself.
Infrastructure changes include:
- Added a general-purpose LRU cache library that can be reused elsewhere.
- Fixed the client channel code to use the channel args returned by the resolver for the dynamic filters. This was necessary so that the GCP auth filter could access the `XdsConfig` object, which is passed via a channel arg.
- Unlike the other xDS HTTP filters we support, the GCP auth filter does not support config overrides, and its configuration includes a cache size parameter that we always need at the channel level, not per-call. As a result, I had to change the xDS HTTP filter API to give it the ability to set top-level fields in the service config, not just per-method fields. (We use the service config as a way of passing configuration down into xDS HTTP filters.) Note that for now, this works only on the client side, because we don't have machinery for a top-level service config on the server side.
- The GCP auth filter is also the first case where the filter needs to know its instance name from the xDS config, so I changed the xDS HTTP filter API to plumb that through.
- Fixed a bug in the HTTP client library that prevented the override functions from declining to override a particular request.
Closes#37550
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37550 from markdroth:xds_gcp_auth_filter 19eaefb52f
PiperOrigin-RevId: 669371249
[Gpr_To_Absl_Logging] Remove gpr_log. Adding absl LOG wrappers
List of changes in this PR
1. Replacing all instances of gpr_log in PHP and RUBY with the new absl wrapper APIs. The replacement mapping is given below
gpr_log(GPR_ERROR, ...)
=> grpc_absl_log_error
gpr_log(GPR_INFO, ...)
=> grpc_absl_log_info - Printing a simple message
=> grpc_absl_log_info_int - Printing a message and a number
=> grpc_absl_log_info_str - Printing 2 strings.
gpr_log(GPR_DEBUG, ...)
=> grpc_absl_vlog - Printing a simple message
=> grpc_absl_vlog_int - Printing a message and a number
=> grpc_absl_vlog_str - Printing 2 strings.
Adding grpc_absl_vlog2_enabled() check around gpr_log(GPR_DEBUG, ...)
2. src/python/grpcio_observability/grpc_observability/observability_util.cc One instance of gpr_log to absl LOG replacement was missed earlier. Fixing that.
3. Deleting deprecated gpr stuff : gpr_log_severity , GPR_DEBUG , GPR_INFO , GPR_ERROR , gpr_log .
4. Adding new APIs for Ruby and PHP. These APIs are very simple wrappers around absl.
5. Removing the legacy functions in platform specific log.cc files. These files are safe to delete now.
6. Fixing the allow list in banned_functions.py . This makes sure that these new wrappers don't get used all over the place by everyone. We carefully only allow list the PHP and RUBY files and allow the use of these wrappers. Everywhere else - using these wrappers should fail Sanity Tests.
Closes#37431
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37431 from tanvi-jagtap:remove_gpr_error 6e5e9bcfcc
PiperOrigin-RevId: 668586873
Add validation of the `Audience` cluster metadata type, as per gRFC A83 (https://github.com/grpc/proposal/pull/438).
I had previously changed the metadata to be represented as JSON in #37468. However, while working on the GCP Authentication filter implementation, I realized that that's not an ideal representation, because it would have required us to validate the JSON on a per-RPC basis, which would be bad for performance. So I've changed the representation of metadata to be an abstract type, and we now store the `Audience` metadata as a simple string. I've also moved metadata into its own type with its own validation code, so that in the future we can use it in places other than CDS (many xDS resource types have metadata fields).
While I was at it, I also add some helper functions for validating the `UInt32Value` and `UInt64Value` wrapper protos.
Closes#37566
PiperOrigin-RevId: 668281729
Previously, `grpc_oauth2_token_fetcher_credentials` provided functionality for on-demand token-fetching, but it was integrated into the oauth2 code, so it was not possible to use that same code for on-demand fetching of (e.g.) JWT tokens. This PR splits that class into two parts:
1. A base `TokenFetcherCredentials` class that provides a framework for on-demand fetching of any arbitrary type of auth token.
2. An `Oauth2TokenFetcherCredentials` subclass that derives from `TokenFetcherCredentials` and provides handling for oauth2 tokens.
The `grpc_compute_engine_token_fetcher_credentials`, `StsTokenFetcherCredentials`, and `grpc_google_refresh_token_credentials` classes that previously derived from `grpc_oauth2_token_fetcher_credentials` now derive from `Oauth2TokenFetcherCredentials` instead, so there's not much change to those classes (other than a cleaner interface with the base class functionality).
The `ExternalAccountCredentials` class and its subclasses got more extensive changes here. Previously, this class inheritted from `grpc_oauth2_token_fetcher_credentials` and fooled the base class into thinking that it directly fetched the oauth2 token, when in fact it actually performed a number of steps to gather data and then constructed a synthetic HTTP response to pass back to the base class. I have changed this to instead derive directly from `TokenFetcherCredentials` to provide a much cleaner interface with the parent class.
In addition, I have changed `grpc_call_credentials` from `RefCounted<>` to `DualRefCounted<>` to provide a clean way to shut down any in-flight token fetch when the credentials are unreffed.
This PR paves the way for subsequent work that will allow implementing an on-demand JWT token fetcher call credential, as part of gRFC A83 (https://github.com/grpc/proposal/pull/438).
Closes#37510
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37510 from markdroth:token_fetcher_call_creds_refactor 3bd398a762
PiperOrigin-RevId: 666547985
We're seeing segfault in Python CSM tests:
```
2024-08-03T09:49:45.720555997Z *** SIGSEGV received at time=1722678585 on cpu 0 ***
2024-08-03T09:49:45.721761998Z PC: @ 0x7847ffd5c1c9 (unknown) (unknown)
2024-08-03T09:49:45.722070502Z @ 0x7847fa309d8c 64 absl::lts_20240116::WriteFailureInfo()
2024-08-03T09:49:45.722175904Z @ 0x7847fa309a15 272 absl::lts_20240116::AbslFailureSignalHandler()
2024-08-03T09:49:45.722187675Z @ 0x7847ffc3d050 1592 (unknown)
2024-08-03T09:49:45.723432238Z @ 0x7847e97f9390 (unknown) (unknown)
2024-08-03T09:49:45.723487349Z @ ... and at least 1 more frames
2024-08-03T09:49:45.829702781Z [INFO tini (1)] Spawned child process '/xds_interop_client' with pid '7'
2024-08-03T09:49:45.829766869Z [DEBUG tini (1)] Received SIGCHLD
2024-08-03T09:49:45.829778749Z [DEBUG tini (1)] Reaped child with pid: '7'
2024-08-03T09:49:45.829787070Z [INFO tini (1)] Main child exited with signal (with signal 'Segmentation fault')
```
### The issue
After investigation, we found that the call tracer was deleted before `RecordEnd` was called.
### Why this fix
* To fix this, we decide to use arena to manage the life cycle of CallTracer.
* Since CallTracer was created in another shard object library (`grpcio_observability`) which don't have a dependency on grpc core, we can't use `grpc_core::Arena` directly when creating the call tracer.
* As a workaround, we created a wrapper class `ClientCallTracerWrapper` to wrap the CallTracer and created another core API `grpc_call_tracer_set_and_manage` so that we can manage the life cycle of CallTracer use the wrapper class.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37460
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37460 from XuanWang-Amos:fix_otel_segfault 33c0b98c64
PiperOrigin-RevId: 662966853
Closes#36943
This PR adds `original_request` to any `_reflection_pb2.ServerReflectionResponse` generated by `grpc_reflection.v1alpha`
Closes#36944
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36944 from Drarig29:corentin.girard/add-original-request-to-ServerReflectionResponse e6a94789e1
PiperOrigin-RevId: 660746169
We're seeing a segfault issue in observability tests:
```
2024-07-26T09:09:18.422255153Z *** SIGSEGV received at time=1721984958 on cpu 0 ***
2024-07-26T09:09:18.424985750Z PC: @ 0x7e1acccb71c9 (unknown) (unknown)
2024-07-26T09:09:18.425333774Z @ 0x7e1ac714ed8c 64 absl::lts_20240116::WriteFailureInfo()
2024-07-26T09:09:18.425356717Z @ 0x7e1ac714ea15 272 absl::lts_20240116::AbslFailureSignalHandler()
2024-07-26T09:09:18.425368880Z @ 0x7e1accb98050 1584 (unknown)
2024-07-26T09:09:18.426117382Z @ 0x7e1ac77f458c 112 absl::lts_20240116::string_view::operator std::__cxx11::basic_string<><>()
2024-07-26T09:09:18.426647368Z @ 0x7e1ac78008df 688 grpc_observability::PythonOpenCensusCallTracer::PythonOpenCensusCallAttemptTracer::RecordEnd()
```
It points to `absl::string_view::operator std::__cxx11::basic_string<>()` which indicates the issue might be related to string conversion.
The most probable cause is that the `parent_->method_` string object is being destroyed before the `std::string` conversion is completed or used by `emplace_back`:
b056bc41d3/src/python/grpcio_observability/grpc_observability/client_call_tracer.cc (L325-L326)
Since it's difficult to manage the lifecycle of `method` in Python/Cython, this PR changes `method_` and `traget_` from `absl::string_view` to `std::string` so that they'll always be available.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37329
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37329 from XuanWang-Amos:fix_otel_segfault c579022529
PiperOrigin-RevId: 657366034
Change was created by the release automation script. See go/grpc-release.
Closes#37279
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37279 from drfloob:bump_dev_version_202407222027 4e6607411e
PiperOrigin-RevId: 654925894
Make sure call_tracers are deleted as long as c_call is unrefed.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37247
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37247 from XuanWang-Amos:fixHangingCallTracers b22a45cca9
PiperOrigin-RevId: 654865870
Tested by manually introduce a segfault, able to see the backtrace:
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37241
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37241 from XuanWang-Amos:add_signal_handler_to_python_interop_client 7316c0ca51
PiperOrigin-RevId: 653671195
Instead of passing the transport byte counts back up through the filter
stack to be reported to the `CallTracer`, we now have the transport
pass the transport byte counts directly to the `CallTracer` itself.
This will eventually allow us to avoid unnecessarily storing these byte
counts in cases where no `CallTracer` actually cares about the data, which
will reduce per-call memory. (In the short term, it actually increases
memory usage, but we can separately do some work to avoid the memory
usage in the transport by removing the `grpc_transport_stream_stats`
struct from the legacy filter API.)
This is a prereq for supporting `CallTracer` in the new call v3 stack,
which does not include the transport byte counts as part of the
receieve-trailing-metadata hook, unlike the legacy filter stack.
This change is controlled by the `call_tracer_in_transport` experiment,
which is enabled by default.
As part of this experiment, we also fix a couple of related bugs:
- On the client side, the chttp2 transport was incorrectly adding
annotations to the parent `ClientCallTracer` instead of the
`CallAttemptTracer`.
- The OpenCensus `ServerCallTracer` was incorrectly swapping the values
of sent and received bytes.
PiperOrigin-RevId: 650728181
This moves more code out of the monolithic `grpc_xds_client` BUILD target. We still need more work to split it up completely, but this is a nice step in the right direction -- and it unblocks a subsequent PR that I'm working on for xDS authority rewriting.
Closes#37130
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37130 from markdroth:xds_resource_type_build_refactoring a021d9773c
PiperOrigin-RevId: 648742472
When using gRPC client alongside kopf (Python framework for writing K8s operators), the existing code fails because of 9243c8fbb2/kopf/_cogs/aiokits/aiotoggles.py (L40). This is because the existing `if candidate:` condition invokes the `__bool__` magic method of the `Toggle` class from kopf.
Closes#36837
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36837 from mboutet:patch-1 831152ca47
PiperOrigin-RevId: 646539016
Creating a new PR in continuation to #36894 to fix post-submit errors. I've re-added the `from ._metadata import Metadata` statement to fix the post-submit error.
Closes#37022
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37022 from sreenithi:fix-issues/36613 beb5b5c63d
PiperOrigin-RevId: 646142324
Previously, metadata mutations were made by the picker directly, which meant that they would be applied even if the channel winds up discarding the pick due to the returned subchannel having been disconnected by the time the pick result is returned. This changes the API such that pickers return metadata mutations along with the pick result, so that the mutations won't get applied unless the pick result is actually used.
Closes#36968
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36968 from markdroth:lb_metadata_api 2765da6121
PiperOrigin-RevId: 645451869
Updated the type hint for return type of grpc.aio.ServicerContext.invocation_metadata() to Optional[MetadataType] instead of Optional[Metadata].
MetadataType is already defined to be a Sequence[MetadatumType] and hence the type hint will reflect the sequence being returned, hence solving the [user's reported issue](https://github.com/grpc/grpc/issues/36613)
Closes#36894
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36894 from sreenithi:fix-issues/36613 7d3a047309
PiperOrigin-RevId: 645276778
This change brings up the direct channel, and inproc promise based transports.
This work exposed a bug that was very difficult to fix with the current call_filters.cc implementation, so I've substantially revamped that - instead of having a pipe-like object per call element, we now have a big ol' combined state machine for the entire call. It's a touch more code, but substantially easier to reason about individual cases, so I much prefer this form (it's also a slight memory improvement: 12 bytes total to track call state, and 10 of those are wakeup bitmasks...).
Closes#36734
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36734 from ctiller:transport-refs-9 3e2a80b40d
PiperOrigin-RevId: 644034593
Change was created by the release automation script. See go/grpc-release.
Closes#36892
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36892 from XuanWang-Amos:bump_dev_version_202406112301 9b2898d716
PiperOrigin-RevId: 643027546
Also made some minor improvements to the `ConfigSelector` API.
Closes#36877
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36877 from markdroth:client_channel_v3_dynamic_filters 6a539fe320
PiperOrigin-RevId: 642755276
gRPC shutdown may happen asynchronously with `work_serializer_dispatch`, so wait briefly until shutdown completes in the child's post-fork.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36873
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36873 from yousukseung:python-fork-shutdown-wait 7908a6912a
PiperOrigin-RevId: 642066286
All TraceFlags are now configured in `src/core/lib/debug/trace_flags.yaml`. The format is:
```
my_flag:
default: false # the default value; default=false
description: Some Description
debug_only: false # debug_only flags only work in debug builds; default=false
internal: false # internal flags will not show up in documentation; default=false
```
To regenerate the trace flag source code, run `tools/codegen/core/gen_trace_flags.py` (requires mako). This script is also run when sanity checking.
This PR also adds two new features:
### Glob-based flag configuration
Trace flag configuration now supports `?` (single wildcard character) and `*` (one or more wildcard characters). For example, using `GRPC_TRACE='event_engine*'` will enable all flags that match that glob. It expands to:
* event_engine
* event_engine_client_channel_resolver
* event_engine_dns
* event_engine_endpoint
* event_engine_endpoint_data
* event_engine_poller
### A cleaner trace-logging macro in abseil logging format
If your goal is only to add log statements when the `fault_injection_filter` trace flag is enabled, you can use the macro:
```
GRPC_TRACE_LOG(fault_injection, INFO) << "Filtered:" << 42;
```
When the trace flag is enabled, the the log will show something like this:
```
I0000 00:00:1715733657.430042 16 file.cc:174] Filtered:42
```
----
Note: just like with the gpr_log to abseil logging conversion, the pre-existing trace logging usages can be replaced with the new tracing macro across multiple PRs.
Closes#36576
PiperOrigin-RevId: 641295215
We changed the way flag was passed in this PR: https://github.com/grpc/psm-interop/pull/94
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36827
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36827 from XuanWang-Amos:csm_fix_interop a2c2c47f30
PiperOrigin-RevId: 640908453
Also rename the class from `MetadataQuery` to `GcpMetadataQuery`.
Closes#36789
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36789 from markdroth:reorg_gcp_metadata_query d2b84f46be
PiperOrigin-RevId: 639256546
The following attributes were completely unused:
- kOffset
- kIndex
- kSize
- kFilename
- kKey
- kValue
The following attributes were added but never programmatically accessed, and I've moved them into the status messages themselves, which is another step toward #22883:
- kErrorNo
- kTsiCode
- kWsaError
- kHttpStatus
- kOsError
- kSyscall
- kTargetAddress
- kRawBytes
- kTsiError
Closes#36523
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36523 from markdroth:grpc_error_attribute_cleanup b289c399fe
PiperOrigin-RevId: 639147583
Implement Python CSM observability.
Design: [go/grpc-python-opentelemetry](http://goto.google.com/grpc-python-opentelemetry)
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36557
PiperOrigin-RevId: 639073741
Xuan Wang
Yousuk Seung
Mark D. Roth
apolcyn
Tanvi Jagtap
Corentin Girard
Craig Tiller
AJ Heller
Esun Kim
Eugene Ostroukhov
Maxence Boutet
Sreenithi Sridharan
Richard Belleville