Builds upon #37765 to support arbitrary connection counts in the transport.
(note: at this point the number of connections is determined at connection establishment - future work will be autotuning this)
Closes#38032
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/38032 from ctiller:tiefling-buffer c7520fd7a9
PiperOrigin-RevId: 698952890
Update the chaotic-good wire format with some learnings from the past year, and set up things for the next round of changes we'd like to make:
* Instead of a composite FRAGMENT frame, split out CLIENT_INITIAL_METADATA, CLIENT_END_OF_STREAM, MESSAGE, SERVER_INITIAL_METADATA, SERVER_TRAILING_METADATA as separate frame types - this eliminates a ton of complexity in the transport, and corresponds to how we used the wire format in practice anyway.
* Switch the frame payload for metadata, settings to be protobuf instead of HPACK - this eliminates the ordering requirements on interpreting these frames between streams, which I expect to open up some flexibility with head of line avoidance in the future. It's a heck of a lot easier to read and reason about the code. It's also easier to predict the size of the frame at encode time, which lets us treat metadata and payloads more uniformly in the protocol.
* Add a connection id field to our header, in preparation for allowing multiple data connections
* Allow payloads to be shipped on the control channel ('connection id 0') and use this for sending small messages
Closes#37765
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37765 from ctiller:tiefling 7b57f72367
PiperOrigin-RevId: 695766541
This removes all xDS protos except for 5 of them that have services. We still have some limitations in our internal build system that make it hard to use the real xDS protos for those files, but we're now using the real xDS protos for the rest.
(Note: discovery.proto is actually a special case. While it does have services, we don't actually use those services, so that's not the reason we need a copy of this file. Unfortunately, the xDS BUILD files group discovery.proto into the same build target as ads.proto, which has services that we actually use, thus requiring us to have our own copy. This means that depending on the real discovery.proto causes us to also depend on the real ads.proto, which causes a conflict in the protobuf registry by linking two copies of ads.proto. However, we *are* using the real discovery.proto in unit tests, which do not depend on ads.proto.)
PiperOrigin-RevId: 693907782
This reverts commit 574b19ec31.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#38074
PiperOrigin-RevId: 693803071
This might not be needed? (From https://github.com/grpc/grpc/issues/37976)
- Boringssl commit requiring -msse2: 56d3ad9d23 (diff-f628d148f94bbab9e22a1ad426ccd94311f1fb87bbe5cc533cb85aee18b07a20R248)
- gRPC change to add -msse2 https://github.com/grpc/grpc/pull/36089
---
Answer to the quesiton above is yes; The -msse2 option remains necessary for gRPC on i686 due to BoringSSL's requirements. However, the existing CMake condition for this option was too broad, potentially including ARM architectures where SSE2 isn't supported, leading to compilation errors. I've refined the condition to specifically target 32-bit x86 architectures.
Furthermore, to ensure accurate architecture detection within our dockerized tests, I've configured x86 tests to utilize the linux32 command. This ensures that uname -a correctly reports i686, allowing gRPC's CMake to identify the architecture and apply the -msse2 option as needed.
It's important to note that RBE overrides the default entrypoint, so RBE-based tests must explicitly invoke linux32 even if the Docker image already has it set.
Fixes https://github.com/grpc/grpc/issues/37976Closes#38024
PiperOrigin-RevId: 693026079
We're about to completely change the wire format here... land one additional copy of the transport and tests as a hedge against bugs. Enable the hedge with an experiment.
Closes#38026
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/38026 from ctiller:legacy-admission 5a32bb105d
PiperOrigin-RevId: 692984545
[PH2][NewFile][ClassStructure][Important] Add client and server class
1. New classes Http2ServerTransport and Http2ClientTransport
2. Similar to the classes in [Chaotic Good Client Transport](https://github.com/grpc/grpc/blob/master/src/core/ext/transport/chaotic_good/client_transport.h) and [Chaotic Good Server Transport](https://github.com/grpc/grpc/blob/master/src/core/ext/transport/chaotic_good/server_transport.h)
3. Added new Test files. For now, the 2 new tests just call the constructor of Http2ServerTransport and Http2ClientTransport.
Tested locally using
```
CC=cc bazel test --test_output=all -c dbg --config=asan --verbose_failures //test/core/transport/chttp2:http2_client_transport_test
```
```
CC=cc bazel test --test_output=all -c dbg --config=asan --verbose_failures //test/core/transport/chttp2:http2_server_transport_test
```
Closes#37840
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37840 from tanvi-jagtap:ph2_add_client_server_class c6c3a0d5fb
PiperOrigin-RevId: 692824127
This migrates all of the xDS unit tests except for the fuzzer, which I'll get in a subsequent PR.
This also does not include the xDS e2e tests, which I will also do separately.
Closes#37896
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37896 from markdroth:xds_tests_use_real_protos2 de568b4e53
PiperOrigin-RevId: 686197812
This eliminates the need for the `grpc_cc_proto_library` bazel BUILD rule introduced in #37863.
To make this work, I had to upgrade several bazel dependencies and apply a patch to rules_go to work around https://github.com/bazelbuild/bazel/issues/11636.
Closes#37902
PiperOrigin-RevId: 685868647
This is a trial baloon to see if we can actually make this work. If it does, I'll change the remaining xDS tests to use the real xDS protos and completely remove our local copies.
Closes#37863
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37863 from markdroth:xds_tests_use_real_protos 3ad2fe12be
PiperOrigin-RevId: 684877750
[PH2][Refactor] Creating call_tracer_wrapper.h and cc and moving code as-is into it. The content of the functions or classes has not changed.
In the next few iterations, more code will come into this file.
Closes#37786
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37786 from tanvi-jagtap:ph2_internal_h_split_01 31af14a4e7
PiperOrigin-RevId: 680861154
This is the last piece of gRFC A83 (https://github.com/grpc/proposal/pull/438).
Note that although this is the first use-case for this "blackboard" mechanism, we will also use it in the future for the xDS rate-limiting filter on the gRPC server side.
Closes#37646
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37646 from markdroth:gcp_auth_filter_state 72d0d96c79
PiperOrigin-RevId: 679707134
Basic building block for retries, hedging: buffer outgoing messages & metadata, allow for replay whilst buffered (with a single reader able to read once buffering ends)
Closes#37448
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37448 from ctiller:once-again-into-the-breach-my-friends 79cb121054
PiperOrigin-RevId: 677959212
The following files have been moved:
- src/core/lib/avl/*
- src/core/lib/backoff/*
- src/core/lib/debug/event_log*
- src/core/lib/iomgr/gethostname*
- src/core/lib/iomgr/grpc_if_nametoindex*
- src/core/lib/matchers/*
- src/core/lib/uri/* (renamed from uri_parser.* to uri.*)
- src/core/lib/gprpp/* (existing src/core/util/time.cc was renamed to gpr_time.cc to avoid conflict)
Closes#36792
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36792 from markdroth:reorg_util d4e8996f48
PiperOrigin-RevId: 676947640
As title.
Bump core version since there were API changes (to logging APIs) since the last release.
Closes#37661
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37661 from apolcyn:bump_core_version_202409092234 190b88342b
PiperOrigin-RevId: 672984841
Final piece of gRFC A83 (https://github.com/grpc/proposal/pull/438): the GCP authentication filter itself.
Infrastructure changes include:
- Added a general-purpose LRU cache library that can be reused elsewhere.
- Fixed the client channel code to use the channel args returned by the resolver for the dynamic filters. This was necessary so that the GCP auth filter could access the `XdsConfig` object, which is passed via a channel arg.
- Unlike the other xDS HTTP filters we support, the GCP auth filter does not support config overrides, and its configuration includes a cache size parameter that we always need at the channel level, not per-call. As a result, I had to change the xDS HTTP filter API to give it the ability to set top-level fields in the service config, not just per-method fields. (We use the service config as a way of passing configuration down into xDS HTTP filters.) Note that for now, this works only on the client side, because we don't have machinery for a top-level service config on the server side.
- The GCP auth filter is also the first case where the filter needs to know its instance name from the xDS config, so I changed the xDS HTTP filter API to plumb that through.
- Fixed a bug in the HTTP client library that prevented the override functions from declining to override a particular request.
Closes#37550
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37550 from markdroth:xds_gcp_auth_filter 19eaefb52f
PiperOrigin-RevId: 669371249
[Gpr_To_Absl_Logging] Remove gpr_log. Adding absl LOG wrappers
List of changes in this PR
1. Replacing all instances of gpr_log in PHP and RUBY with the new absl wrapper APIs. The replacement mapping is given below
gpr_log(GPR_ERROR, ...)
=> grpc_absl_log_error
gpr_log(GPR_INFO, ...)
=> grpc_absl_log_info - Printing a simple message
=> grpc_absl_log_info_int - Printing a message and a number
=> grpc_absl_log_info_str - Printing 2 strings.
gpr_log(GPR_DEBUG, ...)
=> grpc_absl_vlog - Printing a simple message
=> grpc_absl_vlog_int - Printing a message and a number
=> grpc_absl_vlog_str - Printing 2 strings.
Adding grpc_absl_vlog2_enabled() check around gpr_log(GPR_DEBUG, ...)
2. src/python/grpcio_observability/grpc_observability/observability_util.cc One instance of gpr_log to absl LOG replacement was missed earlier. Fixing that.
3. Deleting deprecated gpr stuff : gpr_log_severity , GPR_DEBUG , GPR_INFO , GPR_ERROR , gpr_log .
4. Adding new APIs for Ruby and PHP. These APIs are very simple wrappers around absl.
5. Removing the legacy functions in platform specific log.cc files. These files are safe to delete now.
6. Fixing the allow list in banned_functions.py . This makes sure that these new wrappers don't get used all over the place by everyone. We carefully only allow list the PHP and RUBY files and allow the use of these wrappers. Everywhere else - using these wrappers should fail Sanity Tests.
Closes#37431
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37431 from tanvi-jagtap:remove_gpr_error 6e5e9bcfcc
PiperOrigin-RevId: 668586873
Add validation of the `Audience` cluster metadata type, as per gRFC A83 (https://github.com/grpc/proposal/pull/438).
I had previously changed the metadata to be represented as JSON in #37468. However, while working on the GCP Authentication filter implementation, I realized that that's not an ideal representation, because it would have required us to validate the JSON on a per-RPC basis, which would be bad for performance. So I've changed the representation of metadata to be an abstract type, and we now store the `Audience` metadata as a simple string. I've also moved metadata into its own type with its own validation code, so that in the future we can use it in places other than CDS (many xDS resource types have metadata fields).
While I was at it, I also add some helper functions for validating the `UInt32Value` and `UInt64Value` wrapper protos.
Closes#37566
PiperOrigin-RevId: 668281729
The first commit is a pure revert of the revert, and the second one has the fix.
Closes#37573
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37573 from markdroth:call_creds_roll_forward 2476329534
PiperOrigin-RevId: 667672832
The changes in #37531 are causing test failures under run_tests.py (but not bazel), and #37544 was built on top of #37531, so both need to be reverted.
Closes#37567
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37567 from markdroth:call_creds_revert d086e066f5
PiperOrigin-RevId: 666978406
This will fix timestamps on logs and show all `VLOG(2)` logs on tests by default.
Currently, timestamps on logs are shown as -
```
I0000 00:00:1724385276.681936 1894892 config.cc:262] gRPC experiments enabled: call_tracer_in_transport, event_engine_dns, event_engine_listener, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
```
After invoking `absl::InitializeLog()`, this gets fixed to -
```
I0823 03:55:53.993928 1895644 config.cc:262] gRPC experiments enabled: call_tracer_in_transport, event_engine_dns, event_engine_listener, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
```
Closes#37560
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37560 from yashykt:ImproveLoggingForTests 66433336c8
PiperOrigin-RevId: 666956421
As per gRFC A83 (https://github.com/grpc/proposal/pull/438).
For now, I am not exposing this new call creds type via the C-core API or in any C++ or wrapped language public APIs, so there's no way to use it externally. We can easily add that in the future if someone asks, but for now the intent is to use it only internally via the xDS GCP authentication filter, which I'll implement in a subsequent PR.
As part of this, I changed the test framework in credentials_test to check the status code in addition to the message on failure. This exposed several places where existing credential types are returnign the wrong status code (unsurprisingly, because of all of the tech debt surrounding grpc_error). I have not fixed this behavior, but I have added TODOs in the test showing which ones I think need to be fixed.
Closes#37544
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37544 from markdroth:gcp_service_account_identity_call_creds 97e0efc48d
PiperOrigin-RevId: 666869692
Mark D. Roth
Craig Tiller
Vignesh Babu
Esun Kim
gRPC Team Bot
Alisha Nanda
Tanvi Jagtap
Richard Belleville
AJ Heller
erm-g
apolcyn
Marcel
Yash Tibrewal