The gRPC Core API currently requires callers to provide initial metadata before trailing metadata. You can see the C++ Callback API do this bookkeeping, for example. There is an eventual goal to be able to provide these in any order, and have gRPC do the right thing, but core is not there yet.
The proxy fixture in our end2end tests had a rare scenario in which trailing metadata from the server would show up at the proxy before initial metadata. This is part of the proxy's job: to split up batches into singular-operations that can complete in any order. There was, however, a rare flake wherein trailing metadata would complete before initial metadata, and the result was both client and server waiting on each other to respond.
This change adds a way for the proxy to defer sending trailing metadata back to the client, until after initial metadata has been sent to the client. In my testing, this eliminates the flake I had been able to reproduce 1 in 10k times using a single test. It happened more frequently across the full set of tests in our CI test suites.
Closes#37738
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37738 from drfloob:fix-proxy-fixture 6e0d7b7e6f
PiperOrigin-RevId: 676026493
It's possible for `LogStateChange` to be called after the party is destroyed (one thread calls Unref, which calls LogStateChange after the atomic op; another calls Unref concurrently, frees the last ref, deletes the object)
Consequently, it's not safe to call `DebugTag()` here - remove that.
Closes#37748
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37748 from ctiller:flake-fightas-10 ddef399309
PiperOrigin-RevId: 675833221
Without this, we see GOAWAYs with "enter idle" irrespective of the reason being idleness or max connection age.
Closes#37709
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37709 from yashykt:ChannelIdleFilterMessage 236072e7e2
PiperOrigin-RevId: 675762380
`grpc/core/master/macos/grpc_basictests_python` test is flaky, all test cases passed by the test itself [failed with timeout](https://btx.cloud.google.com/invocations/33bdec48-e17b-431b-b473-c8a335e1650c/log):
```
2024-09-11 16:48:57,385 TIMEOUT: py38.asyncio.tests_aio.unit.done_callback_test.TestServerSideDoneCallback [pid=11128, time=486.5sec]
```
This PR does two things:
1. Increase timeout for test case from `480s` to `600s`.
2. Add timestamp to test starting and passing so we know which test case takes a long time to run.
Log before timestamp change:
```
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_stream
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_unary
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_stream
Running tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_unary
Testing gRPC Python...
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_error_in_callback
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_error_in_handler
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_stream
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_stream_unary
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_stream
SUCCESS tests_aio.unit.done_callback_test.TestServerSideDoneCallback.test_unary_unary
```
Log after timestamp change:
```
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_stream
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_unary
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_stream
Running tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_unary
[2024-09-12 10:47:13.877185]Testing gRPC Python...
[2024-09-12 10:47:13.901135]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_add_after_done
[2024-09-12 10:47:13.933705]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_stream
[2024-09-12 10:47:13.962419]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_stream_unary
[2024-09-12 10:47:13.990368]SUCCESS tests_aio.unit.done_callback_test.TestClientSideDoneCallback.test_unary_stream
```
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37703
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37703 from XuanWang-Amos:add_test_timestamp 4e13d29021
PiperOrigin-RevId: 675703890
`grpc_distribtests_gcp_python` test is [failing with permission error](https://fusion2.corp.google.com/ci;ids=1930537984/kokoro/prod:grpc%2Fcore%2Fmaster%2Flinux%2Fgrpc_distribtests_gcp_python/activity/2c6d57dd-f504-4834-9fd6-845db47ede01/log):
```
ERROR: (gcloud.functions.deploy) ResponseError: status=[403], code=[Ok], message=[Permission 'run.services.setIamPolicy' denied on resource 'projects/grpc-testing/locations/us-central1/services/grpc-gcf-distribtest-3cd3cc11-1a8f-4b88-9c3b-220f0bdf9fdc' (or resource may not exist).]
```
In the logs, looks like we're creating gen2 functions by default:
```
As of this Cloud SDK release, new functions will be deployed as 2nd gen functions by default. This is equivalent to currently deploying new with the --gen2 flag. Existing 1st gen functions will not be impacted and will continue to deploy as 1st gen functions.
```
Since we don't need functions provided by gen2 functions and use `--no-gen2` fixed the permission issue, we're adding `--no-gen2` flag to our test.
### Test:
* Passed manually run: http://sponge/e52ffb74-5a12-4e67-9f95-4638390ef57c
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37684
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37684 from XuanWang-Amos:fix_gcf 26588c0cc3
PiperOrigin-RevId: 675703636
This test has been flaking for a while with a WSAEACCESS error on the `bind` call.
Change the loop to only create on socket at a time (on Windows) to rule out something windows-specific is not liking the fact that we are opening multiple listen sockets on the same port.
Closes#37669
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37669 from apolcyn:change_loop ffa105ba46
PiperOrigin-RevId: 675172803
Party contains a state field that contains ref count *and some other things*... so RefIfNonZero needs to pay attention to only consider the ref count portion.
Closes#37726
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37726 from ctiller:flake-fightas-7 f2b3b3ff6d
PiperOrigin-RevId: 674514201
StartCall needs to be called from the party of the call being started, and so here we reorder things such that that's guaranteed.
PiperOrigin-RevId: 674471582
Move the problematic log to be a trace log (not needed for general workflows), and take the opportunity to clean up a few other errors to log every n seconds -- because we principally need the signal that it's happening, not every occurrence.
Closes#37721
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37721 from ctiller:flake-fightas-6 ac18237c51
PiperOrigin-RevId: 674404222
3a138e4369 renamed a bunch .c files to .c.inc, this PR adds them to boringssl podspec file.
Closes#37690
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37690 from HannahShiSFB:fix-objc-build-with-boring-ssl-0-0-37 407ce1464e
PiperOrigin-RevId: 674042284
The timer wasn't taking into account the time for one update, which is necessary to get predictable behavior
Closes#37691
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37691 from ctiller:flake-fightas-5 8fc8fb8398
PiperOrigin-RevId: 673731339
Issue noticed on xds_end2end_test and is made worse worse when reducing `xds_resource_does_not_exist_timeout_ms` to 500 and running it on tsan.
Closes#37678
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37678 from yashykt:XdsClientOnTimerDebugging 1d31e28d2c
PiperOrigin-RevId: 673479242
Pretty sure that the `LoadBalancedCall` instance is getting dropped before return on *some* path through `PickSubchannel`, although it's unclear to me which and it's hard to tell with this implementation.
Ensure that such an event cannot cause a crash by holding a ref to the object we need and calling through that.
This will be marginally worse performance per pick for now, but once work serializer dispatch lands everywhere the additional ref will disappear.
Closes#37663
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37663 from ctiller:flake-fightas-3 d12b2e0540
PiperOrigin-RevId: 673088001
As title.
Bump core version since there were API changes (to logging APIs) since the last release.
Closes#37661
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37661 from apolcyn:bump_core_version_202409092234 190b88342b
PiperOrigin-RevId: 672984841
This change adds an experiment to move time caching from `ExecCtx` (which is the wrong place for this mechanism) and moves it to the party update path (the expectation being that a single poll of a call is the granularity at which we expect time caching to be a useful optimization, whilst avoiding the unbounded hold times associated with the current mechanism).
This requires fixing up a few tests that grew to depend on time caching (would appreciate close eyes on the credentials test, as it's unclear to me why this is required or what the effect is).
This should also fix b/232544809.
Closes#37637
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37637 from ctiller:closer-to-the-sun 8bbde2d0bd
PiperOrigin-RevId: 672574762
This test is flaky, example failure: https://fusion2.corp.google.com/invocations/31bcd70f-650d-40be-8ec4-6fafc3cd3724, error message:
```
Traceback (most recent call last):
File "/usr/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/lib/python3.9/unittest/case.py", line 593, in run
self._callTestMethod(testMethod)
File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
method()
File "/var/local/git/grpc/src/python/grpcio_tests/tests_aio/unit/_test_base.py", line 31, in wrapper
return loop.run_until_complete(f(*args, **kwargs))
File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/var/local/git/grpc/src/python/grpcio_tests/tests_aio/unit/call_test.py", line 820, in test_cancel_after_done_writing
self.assertFalse(call.done())
File "/usr/lib/python3.9/unittest/case.py", line 676, in assertFalse
raise self.failureException(msg)
AssertionError: True is not false
```
We're expecting call to NOT finish immediately after `await call.done_writing()` which is not correct.
* `call.done_writing()` only notifies server that the client is done sending messages, while `call.done()` checks if the RPC call itself has finished.
Thus removing assert check for `call.done()`.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37651
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37651 from XuanWang-Amos:fix_aio_cancel_after_done_writing 42add84236
PiperOrigin-RevId: 671834305
This test is flaky for a while now, most of the time it failed with timeout error, one [example](https://btx.cloud.google.com/invocations/c31650ae-afdd-4e50-bfc4-b9a43588d3ba/log):
```
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
yield
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 676, in run
self._callTestMethod(testMethod)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 633, in _callTestMethod
method()
File "/Volumes/BuildData/tmpfs/altsrc/github/grpc/workspace_python_macos_opt_native/src/python/grpcio_tests/tests_aio/unit/_test_base.py", line 31, in wrapper
return loop.run_until_complete(f(*args, **kwargs))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/Volumes/BuildData/tmpfs/altsrc/github/grpc/workspace_python_macos_opt_native/src/python/grpcio_tests/tests_aio/unit/connectivity_test.py", line 54, in test_unavailable_backend
await asyncio.wait_for(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
```
Suspect the root cause is the same mentioned here: https://github.com/grpc/grpc/pull/26409
Instead of disable this for MacOS, changing the timeout from `4s` to `8s` seems reasonable too.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37645
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37645 from XuanWang-Amos:fix_aio_connectivity_test 10fdf90635
PiperOrigin-RevId: 671523025
This is flaky in CI, and is being replaced by the new implementation with event engine, disabling this test.
Will clean them up after all have switched to event engine.
Closes#37636
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37636 from HannahShiSFB:disable-iomgr-cfstream-test b430ee3d7c
PiperOrigin-RevId: 671399515
Fix some flakiness caused by fixing jitter in the backoff library in #37595.
In the xDS retry tests, the additional jitter made it such that an extra attempt snuck in before the call deadline, so I adjusted the knobs to ensure that exactly the expected number of attempts fit into the tests.
In the PF test, I rewrote the test to use a connection injector, so that it can more accurately tell the time between the connection attempts, without seeing skew due to the server startup time.
Closes#37629
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37629 from markdroth:xds_retry_backoff_flake_fix f35ac902a0
PiperOrigin-RevId: 671120341
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37632
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37632 from yijiem:alts-concurrent-connectivity-test-flake 68df4b1327
PiperOrigin-RevId: 671117730
Looks like there are some odd interactions, but call-v3 doesn't (and will never) handle wakeup sets, so disable for now until iomgr is removed.
Closes#37630
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37630 from ctiller:cgg 7c37893667
PiperOrigin-RevId: 671104484
AJ Heller
Craig Tiller
alto-ruby
Nathan Baulch
Yash Tibrewal
Xuan Wang
Mark D. Roth
apolcyn
Hannah Shi
Vignesh Babu
Arjan Singh Bal
Yijie Ma