[PH2][Refactor][is_client] Split two functions for better readability
1. send_initial_metadata_locked - Moved out the tracing part
2. grpc_chttp2_config_default_keepalive_args - Split the client and server path to reduce cyclomatic complexity.
Closes#37641
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37641 from tanvi-jagtap:remove_is_client_02 bfbc9361db
PiperOrigin-RevId: 678096501
Basic building block for retries, hedging: buffer outgoing messages & metadata, allow for replay whilst buffered (with a single reader able to read once buffering ends)
Closes#37448
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37448 from ctiller:once-again-into-the-breach-my-friends 79cb121054
PiperOrigin-RevId: 677959212
Log error message instead of crashing for this API misuse.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37764
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37764 from yijiem:report-different-gauge-wont-crash 1b6e912bfc
PiperOrigin-RevId: 677944595
Fix https://github.com/grpc/grpc/issues/37727.
A better idea might be to set up Fuzzing for these APIs to find those sort of things. Maybe that can be a next step if we want to further hardening things.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37737
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37737 from yijiem:memory-leak-alts-2 6be8a49e63
PiperOrigin-RevId: 677880955
Add a ValidateCredentials API to the TLS certificate provider interface. A user can call this API to check that the credentials currently held by the certificate provider instance are valid. The definition of "valid" depends on provider that is being used. For the static data and file watcher providers, "valid" means that the credentials consist of valid PEM.
~Currently there is no check to ensure that credentials consist of valid PEM blocks before a TLS handshake commences. This PR creates a static factory for FileWatcherCertificateProvider (and marks the constructor as deprecated) which performs this validation check. The analogous work for StaticDataCertificateProvider will be done in a follow-up PR.~
Closes#37565
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37565 from matthewstevenson88:filewatcher f223228023
PiperOrigin-RevId: 677847751
Some of our tests failed because RPC was canceled by server, error message:
```
I0000 00:00:1726598495.788119 64528 chttp2_transport.cc:1182] ipv4:127.0.0.1:41825: Got goaway [2] err=UNAVAILABLE:GOAWAY received; Error code: 2; Debug Text: Cancelling all calls {grpc_status:14, http2_error:2, created_time:"2024-09-17T18:41:35.78810597+00:00"}
```
It's possible that the request waited too long to be processed and thus canceled by server (default maximum time gRPC server allows a request to be received by the server but not handled is `30s`, controlled by `grpc.server_max_unrequested_time_in_server`).
This PR will print the test start/finish time so we know if we should increase `grpc.server_max_unrequested_time_in_server` time.
Example before change:
```
Testing gRPC Python...
SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_default
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled
```
Example after change:
```
[2024-09-20 18:32:15.228947]Testing gRPC Python...
[2024-09-20 18:32:15.237855]START tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_default
[2024-09-20 18:32:15.265477]SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_default[Duration: 0:00:00.027622s]
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
[2024-09-20 18:32:15.278142]START tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled
[2024-09-20 18:32:15.316065]FAILURE tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_disabled[Duration: 0:00:00.037911]
Running tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled
[2024-09-20 18:32:15.328680]START tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled
[2024-09-20 18:32:16.938745]SUCCESS tests.unit._metadata_flags_test.MetadataFlagsTest.test_call_wait_for_ready_enabled[Duration: 0:00:01.610065s]
```
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37761
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37761 from XuanWang-Amos:add_time_to_unit_test d2ecd8ce22
PiperOrigin-RevId: 677805375
Ensure OPENSSL global clean up happens after gRPC shutdown completes. OPENSSL registers an exit handler to clean up global objects, which may happen before gRPC removes all references to OPENSSL.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37768
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37768 from yousukseung:openssl-atexit-wait d3d1c964a8
PiperOrigin-RevId: 677284514
This is a retry of #37762 with a fix. We no longer hold the C mutex and GIL at the same to avoid a deadlock.
MetadataPluginCallCredentials passes a function pointer to C-core to be called during destruction. This function grabs GIL which may race between GIL destruction during process shutdown. Since GIL destruction happens after Python's exit handlers, we cana mark that Python is shutting down from an exit handler and don't grab GIL in this function afterwards using a C mutex.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37784
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37784 from yousukseung:python-atexit-retry b17addf805
PiperOrigin-RevId: 677280657
The following files have been moved:
- src/core/lib/avl/*
- src/core/lib/backoff/*
- src/core/lib/debug/event_log*
- src/core/lib/iomgr/gethostname*
- src/core/lib/iomgr/grpc_if_nametoindex*
- src/core/lib/matchers/*
- src/core/lib/uri/* (renamed from uri_parser.* to uri.*)
- src/core/lib/gprpp/* (existing src/core/util/time.cc was renamed to gpr_time.cc to avoid conflict)
Closes#36792
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36792 from markdroth:reorg_util d4e8996f48
PiperOrigin-RevId: 676947640
This change appears to cause a deadlock.
This reverts commit 4b53c2bac3.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37769
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37769 from yousukseung:python_atexit_rollback 9218d1066e
PiperOrigin-RevId: 676566209
MetadataPluginCallCredentials passes a function pointer to C-core to be called during destruction. This function grabs GIL which may race between GIL destruction during process shutdown. Since GIL destruction happens after Python's exit handlers, we cana mark that Python is shutting down from an exit handler and don't grab GIL in this function afterwards using a C mutex.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37762
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37762 from yousukseung:gil-shutdown 945eca9c6a
PiperOrigin-RevId: 676471627
This log line was printing an incorrect new state, and that led me down a multi-hour garden path whereby I was fixing a bug that did not exist.
Closes#37750
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37750 from ctiller:flake-fightas-12 020dd7379c
PiperOrigin-RevId: 676253730
Previously, if we pulled server trailing metadata *before* the call was added to the client transport then we'd never call `on_done_` on the spine and consequently never remove the call from the map. This change fixes that edge case.
In fixing it, I noticed a state in `CallState` that was both complicating the fix and completely irrelevant because we respecced earlier this year to say that ServerTrailingMetadata processing cannot be asynchronous, so I'm removing that state also.
Closes#37749
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37749 from ctiller:flake-fightas-11 847814a286
PiperOrigin-RevId: 676246259
It's possible for `LogStateChange` to be called after the party is destroyed (one thread calls Unref, which calls LogStateChange after the atomic op; another calls Unref concurrently, frees the last ref, deletes the object)
Consequently, it's not safe to call `DebugTag()` here - remove that.
Closes#37748
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37748 from ctiller:flake-fightas-10 ddef399309
PiperOrigin-RevId: 675833221
Without this, we see GOAWAYs with "enter idle" irrespective of the reason being idleness or max connection age.
Closes#37709
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37709 from yashykt:ChannelIdleFilterMessage 236072e7e2
PiperOrigin-RevId: 675762380
`grpc/core/master/macos/grpc_basictests_python` test is flaky, all test cases passed by the test itself [failed with timeout](https://btx.cloud.google.com/invocations/33bdec48-e17b-431b-b473-c8a335e1650c/log):
```
2024-09-11 16:48:57,385 TIMEOUT: py38.asyncio.tests_aio.unit.done_callback_test.TestServerSideDoneCallback [pid=11128, time=486.5sec]
```
This PR does two things:
1. Increase timeout for test case from `480s` to `600s`.
2. Add timestamp to test starting and passing so we know which test case takes a long time to run.
Log before timestamp change:
```
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_stream
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_unary
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_stream
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_unary
Testing gRPC Python...
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_error_in_callback
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_error_in_handler
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_stream
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_unary
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_stream
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_unary
```
Log after timestamp change:
```
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_stream
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_unary
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_stream
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_unary
[2024-09-12 10:47:13.877185]Testing gRPC Python...
[2024-09-12 10:47:13.901135]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_add_after_done
[2024-09-12 10:47:13.933705]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_stream
[2024-09-12 10:47:13.962419]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_unary
[2024-09-12 10:47:13.990368]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_stream
```
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37703
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37703 from XuanWang-Amos:add_test_timestamp 4e13d29021
PiperOrigin-RevId: 675703890
Party contains a state field that contains ref count *and some other things*... so RefIfNonZero needs to pay attention to only consider the ref count portion.
Closes#37726
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37726 from ctiller:flake-fightas-7 f2b3b3ff6d
PiperOrigin-RevId: 674514201
StartCall needs to be called from the party of the call being started, and so here we reorder things such that that's guaranteed.
PiperOrigin-RevId: 674471582
Move the problematic log to be a trace log (not needed for general workflows), and take the opportunity to clean up a few other errors to log every n seconds -- because we principally need the signal that it's happening, not every occurrence.
Closes#37721
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37721 from ctiller:flake-fightas-6 ac18237c51
PiperOrigin-RevId: 674404222
3a138e4369 renamed a bunch .c files to .c.inc, this PR adds them to boringssl podspec file.
Closes#37690
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37690 from HannahShiSFB:fix-objc-build-with-boring-ssl-0-0-37 407ce1464e
PiperOrigin-RevId: 674042284
The timer wasn't taking into account the time for one update, which is necessary to get predictable behavior
Closes#37691
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37691 from ctiller:flake-fightas-5 8fc8fb8398
PiperOrigin-RevId: 673731339
Issue noticed on xds_end2end_test and is made worse worse when reducing `xds_resource_does_not_exist_timeout_ms` to 500 and running it on tsan.
Closes#37678
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37678 from yashykt:XdsClientOnTimerDebugging 1d31e28d2c
PiperOrigin-RevId: 673479242
Pretty sure that the `LoadBalancedCall` instance is getting dropped before return on *some* path through `PickSubchannel`, although it's unclear to me which and it's hard to tell with this implementation.
Ensure that such an event cannot cause a crash by holding a ref to the object we need and calling through that.
This will be marginally worse performance per pick for now, but once work serializer dispatch lands everywhere the additional ref will disappear.
Closes#37663
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37663 from ctiller:flake-fightas-3 d12b2e0540
PiperOrigin-RevId: 673088001
As title.
Bump core version since there were API changes (to logging APIs) since the last release.
Closes#37661
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37661 from apolcyn:bump_core_version_202409092234 190b88342b
PiperOrigin-RevId: 672984841
This change adds an experiment to move time caching from `ExecCtx` (which is the wrong place for this mechanism) and moves it to the party update path (the expectation being that a single poll of a call is the granularity at which we expect time caching to be a useful optimization, whilst avoiding the unbounded hold times associated with the current mechanism).
This requires fixing up a few tests that grew to depend on time caching (would appreciate close eyes on the credentials test, as it's unclear to me why this is required or what the effect is).
This should also fix b/232544809.
Closes#37637
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37637 from ctiller:closer-to-the-sun 8bbde2d0bd
PiperOrigin-RevId: 672574762
This test is flaky, example failure: https://fusion2.corp.google.com/invocations/31bcd70f-650d-40be-8ec4-6fafc3cd3724, error message:
```
Traceback (most recent call last):
File "/usr/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/lib/python3.9/unittest/case.py", line 593, in run
self._callTestMethod(testMethod)
File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
method()
File "/var/local/git/grpc/src/python/grpcio_tests/tests_aio/unit/_test_base.py", line 31, in wrapper
return loop.run_until_complete(f(*args, **kwargs))
File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/var/local/git/grpc/src/python/grpcio_tests/tests_aio/unit/call_test.py", line 820, in test_cancel_after_done_writing
self.assertFalse(call.done())
File "/usr/lib/python3.9/unittest/case.py", line 676, in assertFalse
raise self.failureException(msg)
AssertionError: True is not false
```
We're expecting call to NOT finish immediately after `await call.done_writing()` which is not correct.
* `call.done_writing()` only notifies server that the client is done sending messages, while `call.done()` checks if the RPC call itself has finished.
Thus removing assert check for `call.done()`.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37651
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37651 from XuanWang-Amos:fix_aio_cancel_after_done_writing 42add84236
PiperOrigin-RevId: 671834305
This test is flaky for a while now, most of the time it failed with timeout error, one [example](https://btx.cloud.google.com/invocations/c31650ae-afdd-4e50-bfc4-b9a43588d3ba/log):
```
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
yield
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 676, in run
self._callTestMethod(testMethod)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 633, in _callTestMethod
method()
File "/Volumes/BuildData/tmpfs/altsrc/github/grpc/workspace_python_macos_opt_native/src/python/grpcio_tests/tests_aio/unit/_test_base.py", line 31, in wrapper
return loop.run_until_complete(f(*args, **kwargs))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/Volumes/BuildData/tmpfs/altsrc/github/grpc/workspace_python_macos_opt_native/src/python/grpcio_tests/tests_aio/unit/connectivity_test.py", line 54, in test_unavailable_backend
await asyncio.wait_for(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
```
Suspect the root cause is the same mentioned here: https://github.com/grpc/grpc/pull/26409
Instead of disable this for MacOS, changing the timeout from `4s` to `8s` seems reasonable too.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37645
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37645 from XuanWang-Amos:fix_aio_connectivity_test 10fdf90635
PiperOrigin-RevId: 671523025
Fallback interop test is fully deployed. This variable is no longer needed.
Closes#37620
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37620 from eugeneo:no-fallback-var c21509d0a5
PiperOrigin-RevId: 670738146
Tanvi Jagtap
Alisha Nanda
Craig Tiller
Eugene Ostroukhov
Yijie Ma
Mark D. Roth
Matthew Stevenson
Xuan Wang
Yousuk Seung
Yash Tibrewal
alto-ruby
Nathan Baulch
AJ Heller
Hannah Shi
Vignesh Babu
apolcyn
Tanvi Jagtap
Marcel
Joey Raso