Instead of passing the transport byte counts back up through the filter
stack to be reported to the `CallTracer`, we now have the transport
pass the transport byte counts directly to the `CallTracer` itself.
This will eventually allow us to avoid unnecessarily storing these byte
counts in cases where no `CallTracer` actually cares about the data, which
will reduce per-call memory. (In the short term, it actually increases
memory usage, but we can separately do some work to avoid the memory
usage in the transport by removing the `grpc_transport_stream_stats`
struct from the legacy filter API.)
This is a prereq for supporting `CallTracer` in the new call v3 stack,
which does not include the transport byte counts as part of the
receieve-trailing-metadata hook, unlike the legacy filter stack.
This change is controlled by the `call_tracer_in_transport` experiment,
which is enabled by default.
As part of this experiment, we also fix a couple of related bugs:
- On the client side, the chttp2 transport was incorrectly adding
annotations to the parent `ClientCallTracer` instead of the
`CallAttemptTracer`.
- The OpenCensus `ServerCallTracer` was incorrectly swapping the values
of sent and received bytes.
PiperOrigin-RevId: 650728181
As per gRFC A81 (https://github.com/grpc/proposal/pull/435).
A few small improvements along the way:
- Use `DownCast<>` instead of `static_cast<>` in `ServiceConfigCallData::GetCallAttribute()`.
- Add a convenient templated wrapper for `ClientChannelLbCallState::GetCallAttribute()`, similar to the one on `ServiceConfigCallData`.
- Added a `ParseBoolValue()` helper to xds_common_types_parser, and use it in various places in xDS resource parsing.
Closes#37087
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37087 from markdroth:xds_authority_rewriting 3679d2b061
PiperOrigin-RevId: 650293912
As part of the xDS authority rewriting implementation in #37087, I need to write some tests showing the authority being rewritten. However, the fake security connector currently crashes on unexpected authorities. (As a side note, I think the fake security connector is very cumbersome and should be redesigned, but that's a separate project.) As a result, I need a way to use InsecureCreds on a per-test basis.
I thought about just adding an option to `XdsTestType` to trigger use of InsecureCreds, but the logic we use for determining which creds type to use for what is already very cumbersome, and adding another option there would have just made that worse. Instead, I have switched to a simpler approach where the individual tests can decide what creds type to use directly. This both unblocks my other PR and makes the existing code more maintainable.
Closes#37156
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37156 from markdroth:xds_e2e_test_creds_improvement 2e813834d0
PiperOrigin-RevId: 649225248
Since these were disabled they stopped working, and we really need to be tracking overheads here.
Closes#37077
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37077 from ctiller:it-must-work aa19a4aa89
PiperOrigin-RevId: 648469428
This allows CallTracers to be created with parameters dictated by
channel args.
For the moment, I've used the EventEngine `EndpointConfig` API to expose
the channel args here, so as to avoid directly exposing
`grpc_core::ChannelArgs`. We should determine a better API here before
we de-experimentalize the stats APIs.
Also add an experiment to be used in a subsequent PR.
PiperOrigin-RevId: 647730284
As we've learned what configuration is needed for our benchmarks the settings have been growing more and more bespoke for each binary. Try to consolidate that into some useful defaults.
Also ensure we always `linkstatic=1`. `cc_binary` defaults to this, so it's reasonable to assume that's the performance our customers see. It also deeply impacts performance for small microbenchmarks, and so enabling it gives us more apples:apples, and saves chasing things that don't matter.
Closes#37012
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37012 from ctiller:benchmark-bin b84cab58c6
PiperOrigin-RevId: 645483183
Previously, metadata mutations were made by the picker directly, which meant that they would be applied even if the channel winds up discarding the pick due to the returned subchannel having been disconnected by the time the pick result is returned. This changes the API such that pickers return metadata mutations along with the pick result, so that the mutations won't get applied unless the pick result is actually used.
Closes#36968
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36968 from markdroth:lb_metadata_api 2765da6121
PiperOrigin-RevId: 645451869
Specifically:
- use `OrphanablePtr<>` for `grpc_endpoint`
- use `absl::AnyInvocable<>` instead of `grpc_closure`
- use `EventEngine::Run()` instead of `ExecCtx::Run()`
- use `SliceBuffer` instead of `grpc_slice_buffer`
- use `absl::Status` instead of `grpc_error_handle`
- use `absl::string_view` instead of `const char*` for handshaker names
Also pass acceptor via `HandshakerArgs` instead of as a separate parameter.
Also changed chttp2 and httpcli to use `OrphanablePtr<>` for the endpoint.
PiperOrigin-RevId: 644551906
We don't have a guarantee of timeliness for connectivity issues to clear up with resource quota, and so this wait for ready setting is causing very long delays for the test to complete.
Closes#36967
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36967 from ctiller:rq 4e76af9bda
PiperOrigin-RevId: 644502087
Preparation for switching away from `grpc_channel_filter*` to identify channel filters.
Closes#36907
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36907 from ctiller:type-name e7ad4c67a2
PiperOrigin-RevId: 644483948
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - gpr_log
In this CL we are migrating from gRPCs own gpr logging mechanism to absl logging mechanism. The intention is to deprecate gpr_log in the future.
We have the following mapping
1. gpr_log(GPR_INFO,...) -> LOG(INFO)
2. gpr_log(GPR_ERROR,...) -> LOG(ERROR)
3. gpr_log(GPR_DEBUG,...) -> VLOG(2)
Reviewers need to check :
1. If the above mapping is correct.
2. The content of the log is as before.
gpr_log format strings did not use string_view or std::string . absl LOG accepts these. So there will be some elimination of string_view and std::string related conversions. This is expected.
Closes#36941
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36941 from tanvi-jagtap:src_core_xds 9ab8de800d
PiperOrigin-RevId: 644229971
This change brings up the direct channel, and inproc promise based transports.
This work exposed a bug that was very difficult to fix with the current call_filters.cc implementation, so I've substantially revamped that - instead of having a pipe-like object per call element, we now have a big ol' combined state machine for the entire call. It's a touch more code, but substantially easier to reason about individual cases, so I much prefer this form (it's also a slight memory improvement: 12 bytes total to track call state, and 10 of those are wakeup bitmasks...).
Closes#36734
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36734 from ctiller:transport-refs-9 3e2a80b40d
PiperOrigin-RevId: 644034593
Also made some minor improvements to the `ConfigSelector` API.
Closes#36877
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36877 from markdroth:client_channel_v3_dynamic_filters 6a539fe320
PiperOrigin-RevId: 642755276
- ensure ordering of `OnAccept` and `Shutdown` callbacks from thready event engine (previously these could be reordered and this caused spurious failures)
- enable thready_tsan for one C++ e2e test
- don't filter thready_tsan for local builds (only CI)
Closes#36886
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36886 from ctiller:erm e3b88e7d86
PiperOrigin-RevId: 642702724
[Gpr_To_Absl_Logging] Move function to test header form log.h
This is not really needed in log.h
Closes#36860
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36860 from tanvi-jagtap:move_function_to_test_header e6494bd06f
PiperOrigin-RevId: 642080756
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36729
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36729 from yijiem:per-channel-stats-plugin 4786bed42f
PiperOrigin-RevId: 642030366
This gives grpc_endpoint the same destruction-is-shutdown semantic as
EventEngine::Endpoint, which will make the migration easier.
PiperOrigin-RevId: 639867616
The following attributes were completely unused:
- kOffset
- kIndex
- kSize
- kFilename
- kKey
- kValue
The following attributes were added but never programmatically accessed, and I've moved them into the status messages themselves, which is another step toward #22883:
- kErrorNo
- kTsiCode
- kWsaError
- kHttpStatus
- kOsError
- kSyscall
- kTargetAddress
- kRawBytes
- kTsiError
Closes#36523
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36523 from markdroth:grpc_error_attribute_cleanup b289c399fe
PiperOrigin-RevId: 639147583
Make `Arena` be a refcounted object.
Solves a bunch of issues: our stack right now needs a very complicated dance between transport and surface to destroy a call, but with this scheme we can just hold a ref to what we need in each place and everything works out.
Removes some `ifdef`'d out code that had been sitting dormant for a year or two also -- I'd left it in as a hedge against it being maybe a bad idea, but it looks like it's not needed.
Closes#36758
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36758 from ctiller:arena-counting d1b672fe30
PiperOrigin-RevId: 638767768
This fixes a fairly embarrassing bug and lack of testing from #33234. Prior to this fix, attempting to use the "tls" creds type would always cause a crash.
@gtcooke94@matthewstevenson88 Note that the root cause of this bug was that when I wrote this code, I assumed that `grpc_tls_credentials_options` had a reasonable default for the cert verifier. But it turns out that it doesn't do that directly; instead, we are only imposing that default in [`CredentialOptionSanityCheck()`](621aa4e5ce/src/core/lib/security/credentials/tls/tls_credentials.cc (L85)), which is called only when we call [`grpc_tls_credentials_create()`](621aa4e5ce/src/core/lib/security/credentials/tls/tls_credentials.cc (L160)), not when we directly instantiate `TlsCredentials` as my code was doing. As part of the TlsCreds API cleanup you're working on, we should fix this so that callers get the right behavior even if they are internal callers that instantiate the TlsCreds object directly rather than calling the C-core API.
Closes#36726
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36726 from markdroth:xds_bootstrap_mtls_creds_fix dac2789e10
PiperOrigin-RevId: 637993734
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - gpr_log
In this CL we are migrating from gRPCs own gpr logging mechanism to absl logging mechanism. The intention is to deprecate gpr_log in the future.
We have the following mapping
1. gpr_log(GPR_INFO,...) -> LOG(INFO)
2. gpr_log(GPR_ERROR,...) -> LOG(ERROR)
3. gpr_log(GPR_DEBUG,...) -> VLOG(2)
Reviewers need to check :
1. If the above mapping is correct.
2. The content of the log is as before.
gpr_log format strings did not use string_view or std::string . absl LOG accepts these. So there will be some elimination of string_view and std::string related conversions. This is expected.
Closes#36678
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36678 from tanvi-jagtap:test_cpp_gpr_log 81b8f4179f
PiperOrigin-RevId: 636410944
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - gpr_log
In this CL we are migrating from gRPCs own gpr logging mechanism to absl logging mechanism. The intention is to deprecate gpr_log in the future.
We have the following mapping
1. gpr_log(GPR_INFO,...) -> LOG(INFO)
2. gpr_log(GPR_ERROR,...) -> LOG(ERROR)
3. gpr_log(GPR_DEBUG,...) -> VLOG(2)
Reviewers need to check :
1. If the above mapping is correct.
2. The content of the log is as before.
gpr_log format strings did not use string_view or std::string . absl LOG accepts these. So there will be some elimination of string_view and std::string related conversions. This is expected.
Closes#36677
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36677 from tanvi-jagtap:regex_test_end2end dfb803ebff
PiperOrigin-RevId: 635807701
https://github.com/grpc/grpc/pull/36598 made a change to the logging filter which required `CallTracerAnnotationInterface` to always be present on the call. That is only the case when metrics/tracing is enabled, which is not always the case.
Also, modify the test to test that logging works even if metrics/tracing is not enabled.
Internal ref - b/341794662
Closes#36671
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36671 from yashykt:FixLogging 4f736e7e56
PiperOrigin-RevId: 635610705
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - gpr_log
In this CL we are migrating from gRPCs own gpr logging mechanism to absl logging mechanism. The intention is to deprecate gpr_log in the future.
We have the following mapping
1. gpr_log(GPR_INFO,...) -> LOG(INFO)
2. gpr_log(GPR_ERROR,...) -> LOG(ERROR)
3. gpr_log(GPR_DEBUG,...) -> VLOG(2)
Reviewers need to check :
1. If the above mapping is correct.
2. The content of the log is as before.
gpr_log format strings did not use string_view or std::string . absl LOG accepts these. So there will be some elimination of string_view and std::string related conversions. This is expected.
Closes#36636
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36636 from tanvi-jagtap:regex_test_cpp f2cac9c5a4
PiperOrigin-RevId: 634954173
With 1k iterations instead of 10k iterations, this test takes ~90s in the RBE environment. There must have been a regression somewhere, since this test did not used to fail every time.
Closes#36612
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36612 from drfloob:nerf-writes-per-rpc 0ba8af4167
PiperOrigin-RevId: 633677652
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - BUILD
In this CL we are just editing the build and bzl files to add dependencies.
This is done to prevent merge conflict and constantly having to re-make the make files using generate_projects.sh for each set of changes.
Closes#36607
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36607 from tanvi-jagtap:build_test_cpp 3e17d778d0
PiperOrigin-RevId: 633523097
This cleans up the channel trace test to provide better output when tests fail, which will help diagnose the problem with b/339066599.
Closes#36582
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36582 from markdroth:channelz_test_fix fd287182ad
PiperOrigin-RevId: 633358151
Mark D. Roth
Tanvi Jagtap
Craig Tiller
Eugene Ostroukhov
Tanvi Jagtap
Yijie Ma
Yash Tibrewal
Esun Kim
AJ Heller