Allows use of the party <-> party wakeup batching stuff, which reduces threadhops drastically for this transport.
Closes#37078
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37078 from ctiller:chaotic-party-3 75c32e6a64
PiperOrigin-RevId: 679685211
Building armv7 artifacts after adding Python 3.13 is failing with a timeout. Hence increasing timeout to 2 hours.
Closes#37807
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37807 from sreenithi:increase_armv7_artifact_build_timeout 60034378f5
PiperOrigin-RevId: 679608393
Moving out related code into a separate file.
`internal.h -> stream_lists.h`
Closes#37775
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37775 from tanvi-jagtap:ph2_stream_lists_h 13b6030ceb
PiperOrigin-RevId: 679205824
This PR adds templating for Python versions and updates the maximum supported Python version to 3.13. The following major changes related to templating are added:
- Minimum supported Python version and list of supported versions in `setup.py` are fetched using new template generated files called `python_version.py`
- Dockerfiles for the different Python Linux builds are now template generated.
- The "Supported Python Versions" section from READMEs of ancillary and main packages have been removed
Note: All the `python_version.py` files and Linux build `Dockerfiles` except `tools/dockerfile/grpc_artifact_python_linux_armv7/Dockerfile` in the PR are generated from the respective templates.
Further non-templated additions to add support for Python 3.13:
- install scripts and artifacts for windows, macos and linux are added manually. Later, these can be templated as well.
- updated cython bounds to 3.x
- updated twine version to solve [cgi module import error](https://github.com/pypa/twine/issues/1046)
- the twine update introduces a dependency on cryptography>=2.0. But the cryptography package doesn't support 32-bit Linux images and hence `twine check` has been disabled for x86 manylinux and x86 musllinux artifacts.
Closes#37643
PiperOrigin-RevId: 678954495
Fix: https://github.com/grpc/grpc/issues/37610
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
1. Added script to install all dependencies then run all unit tests for python grpcio.
Closes#37070
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37070 from sourabhsinghs:feature/developer-setup-script 954892f20a
PiperOrigin-RevId: 678718224
gRPC has been passing the wrong directory of `_gRPC_PROTOBUF_WELLKNOWN_INCLUDE_DIR` to protoc. While protoc often works around this by searching its own installation directory for proto files, this isn't always successful. Let's fix this bug to ensure consistent behavior.
Fixes: https://github.com/grpc/grpc/issues/37411Closes#37627
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37627 from veblush:protobuf-cmake-wkt 412f4a6f87
PiperOrigin-RevId: 678428224
We need to create userguide examples for Xds not involving Observability and currently only the csm observability example has a docker image.
Closes#37778
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37778 from kannanjgithub:docker_xds_example_cpp bc91c8d5da
PiperOrigin-RevId: 678362755
[PH2][Refactor][is_client] Split two functions for better readability
1. send_initial_metadata_locked - Moved out the tracing part
2. grpc_chttp2_config_default_keepalive_args - Split the client and server path to reduce cyclomatic complexity.
Closes#37641
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37641 from tanvi-jagtap:remove_is_client_02 bfbc9361db
PiperOrigin-RevId: 678096501
Basic building block for retries, hedging: buffer outgoing messages & metadata, allow for replay whilst buffered (with a single reader able to read once buffering ends)
Closes#37448
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37448 from ctiller:once-again-into-the-breach-my-friends 79cb121054
PiperOrigin-RevId: 677959212
Log error message instead of crashing for this API misuse.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37764
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37764 from yijiem:report-different-gauge-wont-crash 1b6e912bfc
PiperOrigin-RevId: 677944595
Passing manual build: https://btx.cloud.google.com/invocations/9adb3cd8-628c-42e5-a7a3-c2ba1ab6127f
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37723
PiperOrigin-RevId: 677900305
Fix https://github.com/grpc/grpc/issues/37727.
A better idea might be to set up Fuzzing for these APIs to find those sort of things. Maybe that can be a next step if we want to further hardening things.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37737
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37737 from yijiem:memory-leak-alts-2 6be8a49e63
PiperOrigin-RevId: 677880955
Add a ValidateCredentials API to the TLS certificate provider interface. A user can call this API to check that the credentials currently held by the certificate provider instance are valid. The definition of "valid" depends on provider that is being used. For the static data and file watcher providers, "valid" means that the credentials consist of valid PEM.
~Currently there is no check to ensure that credentials consist of valid PEM blocks before a TLS handshake commences. This PR creates a static factory for FileWatcherCertificateProvider (and marks the constructor as deprecated) which performs this validation check. The analogous work for StaticDataCertificateProvider will be done in a follow-up PR.~
Closes#37565
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37565 from matthewstevenson88:filewatcher f223228023
PiperOrigin-RevId: 677847751
Some of our tests failed because RPC was canceled by server, error message:
```
I0000 00:00:1726598495.788119 64528 chttp2_transport.cc:1182] ipv4:127.0.0.1:41825: Got goaway [2] err=UNAVAILABLE:GOAWAY received; Error code: 2; Debug Text: Cancelling all calls {grpc_status:14, http2_error:2, created_time:"2024-09-17T18:41:35.78810597+00:00"}
```
It's possible that the request waited too long to be processed and thus canceled by server (default maximum time gRPC server allows a request to be received by the server but not handled is `30s`, controlled by `grpc.server_max_unrequested_time_in_server`).
This PR will print the test start/finish time so we know if we should increase `grpc.server_max_unrequested_time_in_server` time.
Example before change:
```
Testing gRPC Python...
SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_default
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled
```
Example after change:
```
[2024-09-20 18:32:15.228947]Testing gRPC Python...
[2024-09-20 18:32:15.237855]START tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_default
[2024-09-20 18:32:15.265477]SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_default[Duration: 0:00:00.027622s]
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
[2024-09-20 18:32:15.278142]START tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
[2024-09-20 18:32:15.316065]FAILURE tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled[Duration: 0:00:00.037911]
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled
[2024-09-20 18:32:15.328680]START tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled
[2024-09-20 18:32:16.938745]SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled[Duration: 0:00:01.610065s]
```
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37761
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37761 from XuanWang-Amos:add_time_to_unit_test d2ecd8ce22
PiperOrigin-RevId: 677805375
Ensure OPENSSL global clean up happens after gRPC shutdown completes. OPENSSL registers an exit handler to clean up global objects, which may happen before gRPC removes all references to OPENSSL.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37768
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37768 from yousukseung:openssl-atexit-wait d3d1c964a8
PiperOrigin-RevId: 677284514
This is a retry of #37762 with a fix. We no longer hold the C mutex and GIL at the same to avoid a deadlock.
MetadataPluginCallCredentials passes a function pointer to C-core to be called during destruction. This function grabs GIL which may race between GIL destruction during process shutdown. Since GIL destruction happens after Python's exit handlers, we cana mark that Python is shutting down from an exit handler and don't grab GIL in this function afterwards using a C mutex.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37784
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37784 from yousukseung:python-atexit-retry b17addf805
PiperOrigin-RevId: 677280657
The following files have been moved:
- src/core/lib/avl/*
- src/core/lib/backoff/*
- src/core/lib/debug/event_log*
- src/core/lib/iomgr/gethostname*
- src/core/lib/iomgr/grpc_if_nametoindex*
- src/core/lib/matchers/*
- src/core/lib/uri/* (renamed from uri_parser.* to uri.*)
- src/core/lib/gprpp/* (existing src/core/util/time.cc was renamed to gpr_time.cc to avoid conflict)
Closes#36792
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36792 from markdroth:reorg_util d4e8996f48
PiperOrigin-RevId: 676947640
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37773
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37773 from yijiem:alts-concurrent-connect-timeout 99e371f3ac
PiperOrigin-RevId: 676895049
Looks like our MSAN build is just taking longer at the moment, so increase sharding to reduce per-shard runtime.
Closes#37780
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37780 from ctiller:flake-fightas-13 07c422e977
PiperOrigin-RevId: 676869265
This change appears to cause a deadlock.
This reverts commit 4b53c2bac3.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37769
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37769 from yousukseung:python_atexit_rollback 9218d1066e
PiperOrigin-RevId: 676566209
In some rare occasions on Win machines (0,2-0,5%), the tests are stuck before the handshake when we execute `grpc_call_start_batch`. We receive OP_COMPLETE with `Deadline Exceeded {grpc_status:4}` for such cases. The PR bumps it from 5 to 30s (no flakes for --runs_per_test=1000).
Closes#37767
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37767 from erm-g:h2test 9d1208ee1f
PiperOrigin-RevId: 676531975
MetadataPluginCallCredentials passes a function pointer to C-core to be called during destruction. This function grabs GIL which may race between GIL destruction during process shutdown. Since GIL destruction happens after Python's exit handlers, we cana mark that Python is shutting down from an exit handler and don't grab GIL in this function afterwards using a C mutex.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37762
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37762 from yousukseung:gil-shutdown 945eca9c6a
PiperOrigin-RevId: 676471627
This log line was printing an incorrect new state, and that led me down a multi-hour garden path whereby I was fixing a bug that did not exist.
Closes#37750
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37750 from ctiller:flake-fightas-12 020dd7379c
PiperOrigin-RevId: 676253730
Previously, if we pulled server trailing metadata *before* the call was added to the client transport then we'd never call `on_done_` on the spine and consequently never remove the call from the map. This change fixes that edge case.
In fixing it, I noticed a state in `CallState` that was both complicating the fix and completely irrelevant because we respecced earlier this year to say that ServerTrailingMetadata processing cannot be asynchronous, so I'm removing that state also.
Closes#37749
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37749 from ctiller:flake-fightas-11 847814a286
PiperOrigin-RevId: 676246259
The gRPC Core API currently requires callers to provide initial metadata before trailing metadata. You can see the C++ Callback API do this bookkeeping, for example. There is an eventual goal to be able to provide these in any order, and have gRPC do the right thing, but core is not there yet.
The proxy fixture in our end2end tests had a rare scenario in which trailing metadata from the server would show up at the proxy before initial metadata. This is part of the proxy's job: to split up batches into singular-operations that can complete in any order. There was, however, a rare flake wherein trailing metadata would complete before initial metadata, and the result was both client and server waiting on each other to respond.
This change adds a way for the proxy to defer sending trailing metadata back to the client, until after initial metadata has been sent to the client. In my testing, this eliminates the flake I had been able to reproduce 1 in 10k times using a single test. It happened more frequently across the full set of tests in our CI test suites.
Closes#37738
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37738 from drfloob:fix-proxy-fixture 6e0d7b7e6f
PiperOrigin-RevId: 676026493
It's possible for `LogStateChange` to be called after the party is destroyed (one thread calls Unref, which calls LogStateChange after the atomic op; another calls Unref concurrently, frees the last ref, deletes the object)
Consequently, it's not safe to call `DebugTag()` here - remove that.
Closes#37748
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37748 from ctiller:flake-fightas-10 ddef399309
PiperOrigin-RevId: 675833221
Craig Tiller
Mark D. Roth
Sreenithi Sridharan
Sourabh Singh
Tanvi Jagtap
Xuan Wang
Paulo Castello da Costa
erm-g
Tanvi Jagtap
Esun Kim
Kannan J
Alisha Nanda
Eugene Ostroukhov
Yijie Ma
Arjan Singh Bal
Matthew Stevenson
Yousuk Seung
Chris Kennelly
Yash Tibrewal
Hannah Shi
AJ Heller