We're seeing segfault in Python CSM tests:
```
2024-08-03T09:49:45.720555997Z *** SIGSEGV received at time=1722678585 on cpu 0 ***
2024-08-03T09:49:45.721761998Z PC: @ 0x7847ffd5c1c9 (unknown) (unknown)
2024-08-03T09:49:45.722070502Z @ 0x7847fa309d8c 64 absl::lts_20240116::WriteFailureInfo()
2024-08-03T09:49:45.722175904Z @ 0x7847fa309a15 272 absl::lts_20240116::AbslFailureSignalHandler()
2024-08-03T09:49:45.722187675Z @ 0x7847ffc3d050 1592 (unknown)
2024-08-03T09:49:45.723432238Z @ 0x7847e97f9390 (unknown) (unknown)
2024-08-03T09:49:45.723487349Z @ ... and at least 1 more frames
2024-08-03T09:49:45.829702781Z [INFO tini (1)] Spawned child process '/xds_interop_client' with pid '7'
2024-08-03T09:49:45.829766869Z [DEBUG tini (1)] Received SIGCHLD
2024-08-03T09:49:45.829778749Z [DEBUG tini (1)] Reaped child with pid: '7'
2024-08-03T09:49:45.829787070Z [INFO tini (1)] Main child exited with signal (with signal 'Segmentation fault')
```
### The issue
After investigation, we found that the call tracer was deleted before `RecordEnd` was called.
### Why this fix
* To fix this, we decide to use arena to manage the life cycle of CallTracer.
* Since CallTracer was created in another shard object library (`grpcio_observability`) which don't have a dependency on grpc core, we can't use `grpc_core::Arena` directly when creating the call tracer.
* As a workaround, we created a wrapper class `ClientCallTracerWrapper` to wrap the CallTracer and created another core API `grpc_call_tracer_set_and_manage` so that we can manage the life cycle of CallTracer use the wrapper class.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37460
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37460 from XuanWang-Amos:fix_otel_segfault 33c0b98c64
PiperOrigin-RevId: 662966853
Change was created by the release automation script. See go/grpc-release.
Closes#37279
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37279 from drfloob:bump_dev_version_202407222027 4e6607411e
PiperOrigin-RevId: 654925894
Make sure call_tracers are deleted as long as c_call is unrefed.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37247
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37247 from XuanWang-Amos:fixHangingCallTracers b22a45cca9
PiperOrigin-RevId: 654865870
This moves more code out of the monolithic `grpc_xds_client` BUILD target. We still need more work to split it up completely, but this is a nice step in the right direction -- and it unblocks a subsequent PR that I'm working on for xDS authority rewriting.
Closes#37130
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37130 from markdroth:xds_resource_type_build_refactoring a021d9773c
PiperOrigin-RevId: 648742472
When using gRPC client alongside kopf (Python framework for writing K8s operators), the existing code fails because of 9243c8fbb2/kopf/_cogs/aiokits/aiotoggles.py (L40). This is because the existing `if candidate:` condition invokes the `__bool__` magic method of the `Toggle` class from kopf.
Closes#36837
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36837 from mboutet:patch-1 831152ca47
PiperOrigin-RevId: 646539016
Creating a new PR in continuation to #36894 to fix post-submit errors. I've re-added the `from ._metadata import Metadata` statement to fix the post-submit error.
Closes#37022
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37022 from sreenithi:fix-issues/36613 beb5b5c63d
PiperOrigin-RevId: 646142324
Previously, metadata mutations were made by the picker directly, which meant that they would be applied even if the channel winds up discarding the pick due to the returned subchannel having been disconnected by the time the pick result is returned. This changes the API such that pickers return metadata mutations along with the pick result, so that the mutations won't get applied unless the pick result is actually used.
Closes#36968
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36968 from markdroth:lb_metadata_api 2765da6121
PiperOrigin-RevId: 645451869
Updated the type hint for return type of grpc.aio.ServicerContext.invocation_metadata() to Optional[MetadataType] instead of Optional[Metadata].
MetadataType is already defined to be a Sequence[MetadatumType] and hence the type hint will reflect the sequence being returned, hence solving the [user's reported issue](https://github.com/grpc/grpc/issues/36613)
Closes#36894
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36894 from sreenithi:fix-issues/36613 7d3a047309
PiperOrigin-RevId: 645276778
This change brings up the direct channel, and inproc promise based transports.
This work exposed a bug that was very difficult to fix with the current call_filters.cc implementation, so I've substantially revamped that - instead of having a pipe-like object per call element, we now have a big ol' combined state machine for the entire call. It's a touch more code, but substantially easier to reason about individual cases, so I much prefer this form (it's also a slight memory improvement: 12 bytes total to track call state, and 10 of those are wakeup bitmasks...).
Closes#36734
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36734 from ctiller:transport-refs-9 3e2a80b40d
PiperOrigin-RevId: 644034593
Change was created by the release automation script. See go/grpc-release.
Closes#36892
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36892 from XuanWang-Amos:bump_dev_version_202406112301 9b2898d716
PiperOrigin-RevId: 643027546
Also made some minor improvements to the `ConfigSelector` API.
Closes#36877
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36877 from markdroth:client_channel_v3_dynamic_filters 6a539fe320
PiperOrigin-RevId: 642755276
gRPC shutdown may happen asynchronously with `work_serializer_dispatch`, so wait briefly until shutdown completes in the child's post-fork.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36873
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36873 from yousukseung:python-fork-shutdown-wait 7908a6912a
PiperOrigin-RevId: 642066286
All TraceFlags are now configured in `src/core/lib/debug/trace_flags.yaml`. The format is:
```
my_flag:
default: false # the default value; default=false
description: Some Description
debug_only: false # debug_only flags only work in debug builds; default=false
internal: false # internal flags will not show up in documentation; default=false
```
To regenerate the trace flag source code, run `tools/codegen/core/gen_trace_flags.py` (requires mako). This script is also run when sanity checking.
This PR also adds two new features:
### Glob-based flag configuration
Trace flag configuration now supports `?` (single wildcard character) and `*` (one or more wildcard characters). For example, using `GRPC_TRACE='event_engine*'` will enable all flags that match that glob. It expands to:
* event_engine
* event_engine_client_channel_resolver
* event_engine_dns
* event_engine_endpoint
* event_engine_endpoint_data
* event_engine_poller
### A cleaner trace-logging macro in abseil logging format
If your goal is only to add log statements when the `fault_injection_filter` trace flag is enabled, you can use the macro:
```
GRPC_TRACE_LOG(fault_injection, INFO) << "Filtered:" << 42;
```
When the trace flag is enabled, the the log will show something like this:
```
I0000 00:00:1715733657.430042 16 file.cc:174] Filtered:42
```
----
Note: just like with the gpr_log to abseil logging conversion, the pre-existing trace logging usages can be replaced with the new tracing macro across multiple PRs.
Closes#36576
PiperOrigin-RevId: 641295215
Also rename the class from `MetadataQuery` to `GcpMetadataQuery`.
Closes#36789
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36789 from markdroth:reorg_gcp_metadata_query d2b84f46be
PiperOrigin-RevId: 639256546
The following attributes were completely unused:
- kOffset
- kIndex
- kSize
- kFilename
- kKey
- kValue
The following attributes were added but never programmatically accessed, and I've moved them into the status messages themselves, which is another step toward #22883:
- kErrorNo
- kTsiCode
- kWsaError
- kHttpStatus
- kOsError
- kSyscall
- kTargetAddress
- kRawBytes
- kTsiError
Closes#36523
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36523 from markdroth:grpc_error_attribute_cleanup b289c399fe
PiperOrigin-RevId: 639147583
Implement Python CSM observability.
Design: [go/grpc-python-opentelemetry](http://goto.google.com/grpc-python-opentelemetry)
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36557
PiperOrigin-RevId: 639073741
Remove `@abc.abstractmethod` so we don't break backward compatibility.
Fix: https://github.com/grpc/grpc/issues/36683
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36684
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36684 from XuanWang-Amos:remove_abstractmethod_for_registered_method f748fa6e7a
PiperOrigin-RevId: 636186951
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36509
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36509 from ctiller:transport-refs-3 2771a2b0e1
PiperOrigin-RevId: 633240374
Change was created by the release automation script. See go/grpc-release.
Closes#36544
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36544 from yashykt:bump_dev_version_202405061956 69ee5c869e
PiperOrigin-RevId: 631187829
Add registered method change to sync stack server.
## Note
* We're keeping both generic and registered handler in generated stubs (`*_pb2_grpc.py`) because registered method flow is not implemented in aysncio yet, since the stub is used for both sync and asyncio, we can't remove generic method flow until we fully implement the registered method flow in asyncio.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36371
PiperOrigin-RevId: 629842051
Also begin to eliminate `CallContext` in favor of just exposing `Call` - ultimately there's not really a need to introduce two types here, so I'm going to wind that idea back over a few PRs.
I've avoided making this an experiment as the changes required were quite structural.
Closes#36477
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36477 from ctiller:deadline-time 9856eeebe6
PiperOrigin-RevId: 629599230
Add GRPC_PYTHON_BUILD_WITH_SYSTEMD environment variable to setup.py
The variable sets the required compile time define and adds the required
link time library to the build process
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging GPR_ASSERT
Replacing GPR_ASSERT with absl CHECK.
These changes have been made using string replacement and regex.
Will not be replacing all instances of CHECK with CHECK_EQ , CHECK_NE etc because there are too many callsites. Only ones which are doable using very simple regex with least chance of failure will be replaced.
Given that we have 5000+ instances of GPR_ASSERT to edit, Doing it manually is too much work for both the author and reviewer.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36466
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36466 from tanvi-jagtap:tjagtap_src_core_lob_event_engine d101d5f701
PiperOrigin-RevId: 629105415
Introduce the interception chain type.
Also introduces the real call-v3 call spine based atop CallFilters.
Closes#36414
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36414 from ctiller:interception-chain 90c8e96973
PiperOrigin-RevId: 627784183
Xuan Wang
Craig Tiller
AJ Heller
Esun Kim
Eugene Ostroukhov
Mark D. Roth
Maxence Boutet
Sreenithi Sridharan
Richard Belleville
Yousuk Seung
gRPC Team Bot
Oskar Bunyan
Yash Tibrewal
Rushaan Gandhi
nipil
Tanvi Jagtap