The changes in #37531 are causing test failures under run_tests.py (but not bazel), and #37544 was built on top of #37531, so both need to be reverted.
Closes#37567
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37567 from markdroth:call_creds_revert d086e066f5
PiperOrigin-RevId: 666978406
As per gRFC A83 (https://github.com/grpc/proposal/pull/438).
For now, I am not exposing this new call creds type via the C-core API or in any C++ or wrapped language public APIs, so there's no way to use it externally. We can easily add that in the future if someone asks, but for now the intent is to use it only internally via the xDS GCP authentication filter, which I'll implement in a subsequent PR.
As part of this, I changed the test framework in credentials_test to check the status code in addition to the message on failure. This exposed several places where existing credential types are returnign the wrong status code (unsurprisingly, because of all of the tech debt surrounding grpc_error). I have not fixed this behavior, but I have added TODOs in the test showing which ones I think need to be fixed.
Closes#37544
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37544 from markdroth:gcp_service_account_identity_call_creds 97e0efc48d
PiperOrigin-RevId: 666869692
This adds functionality that is intended to be used for the new GcpServiceAccountIdentityCallCredentials implementation, as per gRFC A83 (https://github.com/grpc/proposal/pull/438). However, it is also a useful improvement for all token-fetching call credentials types, so I am adding it to the base class.
Closes#37531
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37531 from markdroth:token_fetcher_call_creds_prefetch_and_backoff 0fcdb48465
PiperOrigin-RevId: 666809903
Previously, `grpc_oauth2_token_fetcher_credentials` provided functionality for on-demand token-fetching, but it was integrated into the oauth2 code, so it was not possible to use that same code for on-demand fetching of (e.g.) JWT tokens. This PR splits that class into two parts:
1. A base `TokenFetcherCredentials` class that provides a framework for on-demand fetching of any arbitrary type of auth token.
2. An `Oauth2TokenFetcherCredentials` subclass that derives from `TokenFetcherCredentials` and provides handling for oauth2 tokens.
The `grpc_compute_engine_token_fetcher_credentials`, `StsTokenFetcherCredentials`, and `grpc_google_refresh_token_credentials` classes that previously derived from `grpc_oauth2_token_fetcher_credentials` now derive from `Oauth2TokenFetcherCredentials` instead, so there's not much change to those classes (other than a cleaner interface with the base class functionality).
The `ExternalAccountCredentials` class and its subclasses got more extensive changes here. Previously, this class inheritted from `grpc_oauth2_token_fetcher_credentials` and fooled the base class into thinking that it directly fetched the oauth2 token, when in fact it actually performed a number of steps to gather data and then constructed a synthetic HTTP response to pass back to the base class. I have changed this to instead derive directly from `TokenFetcherCredentials` to provide a much cleaner interface with the parent class.
In addition, I have changed `grpc_call_credentials` from `RefCounted<>` to `DualRefCounted<>` to provide a clean way to shut down any in-flight token fetch when the credentials are unreffed.
This PR paves the way for subsequent work that will allow implementing an on-demand JWT token fetcher call credential, as part of gRFC A83 (https://github.com/grpc/proposal/pull/438).
Closes#37510
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37510 from markdroth:token_fetcher_call_creds_refactor 3bd398a762
PiperOrigin-RevId: 666547985
I observed a case where `epoll_wait` would hang after receiving all available bytes (209 total). Note that all epoll events are configured to be edge-triggered (EPOLLET). The message was an HTTP request where the the HTTP headers indicated the connection would be closed. Presumably the peer would close their end of the connection, but the single epoll event did not contain EPOLLHUP. Regardless, the entire payload could be consumed in one read.
*Without* the TCP_INQ logic, the gRPC endpoint assumed that there was more data to read after the first read of 209 bytes. Since there was sufficient buffer space to read again, the endpoint called `recvmsg` a second time and immediately got EAGAIN. This triggered the downstream code to process the received data. Subsequent endpoint Reads would attempt a `recvmsg` first, and only if that failed, would then opt in to be notified of epoll events on their watched socket.
However, the INQ logic caused a nearly constant hang in this one environment, but non-deterministically. After a successful recvmsg (i.e., some bytes were received without any errors), INQ accurately showed there were no more bytes to read, and `recvmsg` was not called again, so the edge was never seen. The data was returned to the caller and the read was complete. The next endpoint Read would *first* ask to be notified of fd events, since the previous read claimed there was nothing left to read on that socket (this is the bug). Due to the nature of EPOLLET[1], the fd was very likely still in a ready state from the previous read, since it didn't call `recvmsg` until the edge was found. So the subsequent read was stuck waiting on a socket change event that may not come.
This fix resets the endpoint's state after consuming all available bytes according to TCP_INQ. This ensures that subsequent Reads will first attempt a read on the fd before requesting to be notified of new fd events.
[1]: https://www.man7.org/linux/man-pages/man7/epoll.7.html
> Receiving an event from epoll_wait(2) should suggest to you that such file descriptor is ready for the requested I/O operation. You must consider it ready until the next (nonblocking) read/write yields EAGAIN.
PiperOrigin-RevId: 666398394
Prior to this PR, the host and path were passed to the override functions, but the query params were unavailable to them. I have replaced the separate `host` and `path` parameters with a single parameter that passes the URI in as a const reference, which provides access to the query params.
While I was at it, I also changed the PUT and POST override methods to pass in the body as a string_view, which is more ergonomic for tests.
Closes#37540
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37540 from markdroth:httpcli_override_uri 902b0d2097
PiperOrigin-RevId: 665950188
Some applications cannot avoid the dependency on `//:grpc++_reflection` because it is buried by several infrastructural layers. For binaries that need to implement their own reflection server via generic handlers, this makes accepting request impossible as they will instead be handled by the default reflection server added by the server reflection plugin.
It would not be feasible to implement this as a build define because this binary is depended upon by many applications and buried beneath several layers of infrastructure.
Further, we cannot use the flag to control whether or not the plugin is installed as the plugin is installed at static initialization time, meaning that flags have not been parsed yet. Instead, we simply disable the functionality of the plugin with this flag.
CC @temawiCloses#37527
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37527 from gnossen:disable_cpp_reflection 8d7e32961a
PiperOrigin-RevId: 665051996
[Gpr_To_Absl_Logging] Remove gpr logging header from src/core headers.
Some of the cc files are using gpr_log_verbosity_init() functions.
So I removed the `#include <grpc/support/log.h>` from all headers and put it selectively only in the cc files that used gpr_log_verbosity_init()
Closes#37512
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37512 from tanvi-jagtap:remove_gpr_headers_from_headers a2f07f45d6
PiperOrigin-RevId: 664002265
[Gpr_To_Absl_Logging] Remove gpr logging header include from other headers
Some of the cc files are using gpr_log_verbosity_init() functions.
So I removed the #include <grpc/support/log.h> from all headers and put it selectively only in the cc files that used gpr_log_verbosity_init()
Closes#37513
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37513 from tanvi-jagtap:remove_gpr_headers_from_headers_01 612ca6d0f7
PiperOrigin-RevId: 663811895
[Gpr_To_Absl_Logging] Using GRPC_TRACE_LOG instead of GRPC_TRACE_FLAG_ENABLED
Closes#37516
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37516 from tanvi-jagtap:GRPC_TRACE_FLAG_ENABLED_09 c21f60f58a
PiperOrigin-RevId: 663811332
[Gpr_To_Absl_Logging] Fix use of GRPC_TRACE_FLAG_ENABLED in gen code
Closes#37514
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37514 from tanvi-jagtap:fix_gen_code_GRPC_TRACE_FLAG_ENABLED 096004c6d7
PiperOrigin-RevId: 663762507
[Gpr_To_Absl_Logging] Using GRPC_TRACE_LOG instead of GRPC_TRACE_FLAG_ENABLED
Closes#37483
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37483 from tanvi-jagtap:GRPC_TRACE_FLAG_ENABLED_08 ab8b98e8c4
PiperOrigin-RevId: 663618882
[Gpr_To_Absl_Logging] Using GRPC_TRACE_LOG instead of GRPC_TRACE_FLAG_ENABLED
Closes#37484
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37484 from tanvi-jagtap:GRPC_TRACE_FLAG_ENABLED_07 087d5fcfb3
PiperOrigin-RevId: 663606075
[PH2][Refactor][FunctionSplit][chttp2_transport.cc]
Splitting very large function perform_stream_op_locked into smaller functions.
Adding new functions :
1. send_initial_metadata_locked
2. send_message_locked
3. send_trailing_metadata_locked
4. recv_initial_metadata_locked
5. recv_message_locked
6. recv_trailing_metadata_locked
All the above functions get called from perform_stream_op_locked .
Why was this done?
I ran into test failures while working on this : https://github.com/grpc/grpc/pull/37186/files .
The test failed because the function execute_stream_op was too large. So I had to split the function as well in the same PR (which is messy).
Decided to do the same type of splitting here.
Closes#37453
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37453 from tanvi-jagtap:split_function 5e06594e81
PiperOrigin-RevId: 663570972
Split off from https://github.com/grpc/grpc/pull/37425
We are adding and removing callbacks on the OpenTelemetry Async Instruments without synchronization. This opens us to races where we have an AddCallback and RemoveCallback operation happening at the same time. The correct result after these operations is to still have a callback registered with OpenTelemetry at the end, but the two operations could race and we could just decide to remove the OpenTelemetry callback.
The fix delays removing OpenTelemetry callbacks to plugin destruction time.
Closes#37485
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37485 from yashykt:FixRaceOTelGauge 016b0a41b5
PiperOrigin-RevId: 663492598
[Gpr_To_Absl_Logging] Removing absl_vlog2_enabled .
@apolcyn : Please review the Ruby code.
@yashykt : Please review the C++ code. And the python sanity test.
Closes#37476
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37476 from tanvi-jagtap:remove_should_fn c05fd6445e
PiperOrigin-RevId: 663193927
We're seeing segfault in Python CSM tests:
```
2024-08-03T09:49:45.720555997Z *** SIGSEGV received at time=1722678585 on cpu 0 ***
2024-08-03T09:49:45.721761998Z PC: @ 0x7847ffd5c1c9 (unknown) (unknown)
2024-08-03T09:49:45.722070502Z @ 0x7847fa309d8c 64 absl::lts_20240116::WriteFailureInfo()
2024-08-03T09:49:45.722175904Z @ 0x7847fa309a15 272 absl::lts_20240116::AbslFailureSignalHandler()
2024-08-03T09:49:45.722187675Z @ 0x7847ffc3d050 1592 (unknown)
2024-08-03T09:49:45.723432238Z @ 0x7847e97f9390 (unknown) (unknown)
2024-08-03T09:49:45.723487349Z @ ... and at least 1 more frames
2024-08-03T09:49:45.829702781Z [INFO tini (1)] Spawned child process '/xds_interop_client' with pid '7'
2024-08-03T09:49:45.829766869Z [DEBUG tini (1)] Received SIGCHLD
2024-08-03T09:49:45.829778749Z [DEBUG tini (1)] Reaped child with pid: '7'
2024-08-03T09:49:45.829787070Z [INFO tini (1)] Main child exited with signal (with signal 'Segmentation fault')
```
### The issue
After investigation, we found that the call tracer was deleted before `RecordEnd` was called.
### Why this fix
* To fix this, we decide to use arena to manage the life cycle of CallTracer.
* Since CallTracer was created in another shard object library (`grpcio_observability`) which don't have a dependency on grpc core, we can't use `grpc_core::Arena` directly when creating the call tracer.
* As a workaround, we created a wrapper class `ClientCallTracerWrapper` to wrap the CallTracer and created another core API `grpc_call_tracer_set_and_manage` so that we can manage the life cycle of CallTracer use the wrapper class.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37460
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37460 from XuanWang-Amos:fix_otel_segfault 33c0b98c64
PiperOrigin-RevId: 662966853
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37442
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37442 from yijiem:migrate-iomgr-getdnsresolver 7b3ed7d980
PiperOrigin-RevId: 662957279
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37466
PiperOrigin-RevId: 662606899
The peer_state_based_framing experiment is not used. The other experiment has been rolled out to 100% in prod for a while now. The expiry date of a few other experiments are updated.
PiperOrigin-RevId: 662565880
[Gpr_To_Absl_Logging] Using GRPC_TRACE_LOG instead of GRPC_TRACE_FLAG_ENABLED
Closes#37387
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37387 from tanvi-jagtap:GRPC_TRACE_FLAG_ENABLED_06 1b87cfece9
PiperOrigin-RevId: 662349186
[Gpr_To_Absl_Logging] Removing EXECUTOR_TRACE and replace with GRPC_TRACE_LOG
Closes#37433
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37433 from tanvi-jagtap:remove_EXECUTOR_TRACE 2a7bdc97ad
PiperOrigin-RevId: 661498614
[Gpr_To_Absl_Logging] Replacing GRPC_ARES_RESOLVER_TRACE_LOG with GRPC_TRACE_LOG
Also, moved definition of GRPC_ARES_RESOLVER_TRACE_LOG from header to the cpp where it is being used to avoid more people from trying to use this macro
Closes#37434
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37434 from tanvi-jagtap:remove_GRPC_ARES_RESOLVER_TRACE_LOG e7883d70e9
PiperOrigin-RevId: 661297848
[Gpr_To_Absl_Logging] Remove gpr_should_log from the header file
Closes#37420
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37420 from tanvi-jagtap:clean_up_ruby_php_01 ce303fa808
PiperOrigin-RevId: 661118631
[Gpr_To_Absl_Logging] Using GRPC_TRACE_LOG instead of GRPC_TRACE_FLAG_ENABLED
Closes#37389
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37389 from tanvi-jagtap:GRPC_TRACE_FLAG_ENABLED_04 bcf1c6255f
PiperOrigin-RevId: 661111633
Closes#36943
This PR adds `original_request` to any `_reflection_pb2.ServerReflectionResponse` generated by `grpc_reflection.v1alpha`
Closes#36944
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36944 from Drarig29:corentin.girard/add-original-request-to-ServerReflectionResponse e6a94789e1
PiperOrigin-RevId: 660746169
Mark D. Roth
AJ Heller
Evan Brown
Richard Belleville
Tanvi Jagtap
gRPC Team Bot
Vignesh Babu
Yash Tibrewal
Nana Pang
Hannah Shi
Tanvi Jagtap
Xuan Wang
Yijie Ma
apolcyn
Corentin Girard