(Required recursively splitting some other targets too)
Gets us closer to eliminating `grpc_base`, and (more importantly) sets up some pieces of transport to be split out so I can rebuild them in coming months.
Closes#35496
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35496 from ctiller:chipchip 5319fb925d
PiperOrigin-RevId: 597304996
Provide a public experimental API and bazel compatible build target for OpenTelemetry metrics.
Details -
* New `OpenTelemetryPluginBuilder` class that provides the API specified in https://github.com/grpc/proposal/blob/master/A66-otel-stats.md
* The existing `grpc::internal::OpenTelemetryPluginBuilder` class is moved to `grpc::internal::OpenTelemetryPluginBuilderImpl` for disambiguation.
* Renamed `OTel` in some instances to `OpenTelemetry` for consistency.
Closes#35348
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35348 from yashykt:OTelPublicApi e32328825e
PiperOrigin-RevId: 594271246
Adds temporary `call.cc` and `connected_channel.cc` scaffolding to run `CallInterceptor`/`CallHandler` style calls.
This will get ripped out as soon as the v3 transition is completed.
Closes#35312
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35312 from ctiller:v3-accept ae0bf81f8b
PiperOrigin-RevId: 594128029
When an error occurs in `TrySeq`, log the error.
Also, in the trace statements, capture the file/line that the sequence was constructed at, and log that with the traces.
Closes#35319
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35319 from ctiller:trace-seq bc0d47395f
PiperOrigin-RevId: 591999228
### Changes in this PR
* Refactor and remove some Core/C++ dependencies to simplify Python Observability package build process.
* Refactored code to read config at Python layer.
* Enable observability build from source.
* Add observability to run_test.
* Currently it's only enabled in Linux.
* Add error handler in run_test loaders.
* Current framework will always visit modules in test directory then decide which tests to skip.
* Since we're not building Observability for MacOS and Windows this step will fail with error `No module named 'grpc_observability'`.
* After the change we'll just skip those modules.
* We still have `_sanity_test` to make sure all tests are loaded correctly for each platform.
* Remov OC dependency as we're migrating to OTel.
* Also removed trace from testing.
* Note that trace propagation function was also removed because of this.
### Testing
* Passed existing tests.
* Tested locally, able to build observability from source using `GRPC_PYTHON_BUILD_WITH_CYTHON=1 pip install .`.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#34207
PiperOrigin-RevId: 590258014
This avoids storing unnecessary copies of the address list in each node of the LB policy tree.
Closes#34753
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/34753 from markdroth:lb_address_list_iterator 1d39465fbc
PiperOrigin-RevId: 582891475
This commit upgrades gRPC to protobuf v25.0 and makes some fixes to
account for upb changes. One major change is that upb has been merged
into the protobuf repo, so we can now drop the separate `@upb`
dependency. Another is that `.upb.c` files no longer exist and there are
new `.upb_minitable.h` and `.upb_minitable.c` files. The longer
filenames exceeded a Windows restriction, so to work around that I
renamed the `upb-generated` directory to just `upb-gen`, and likewise
for `upbdefs-generated`.
Changes to fake resolver:
- Add `WaitForReresolutionRequest()` method to fake resolver response
generator to allow tests to tell when re-resolution has been requested.
- Change fake resolver response generator API to have only one mechanism
for injecting results, regardless of whether the result is an error or
whether it's triggered by a re-resolution.
Changes to grpclb_end2end_test:
- Change balancer interface such that instead of setting a list of
responses with fixed delays, the test can control exactly when each
response is set.
- Change balancer impl to always send the initial LB response, as
expected by the grpclb protocol.
- Change balancer impl to always read load reports, even if load
reporting is not expected to be enabled. (The latter case will still
cause the test to fail.) Reads are done in a different thread than
writes.
- Allow each test to directly control how many backends and balancers
are started and the client load reporting interval, so that (a) we don't
waste resources starting servers we don't need and (b) there is no need
to arbitrarily split tests across different test classes.
- Add timeouts to `WaitForAllBackends()` functionality, so that tests
will fail with a useful error rather than timing out.
- Improved ergonomics of various helper functions in the test framework.
In the process of making these changes, I found a couple of bugs:
- A bug in pick_first, which I fixed in #34885.
- A bug in grpclb, in which we were using the wrong condition to decide
whether to propagate a re-resolution request from the child policy,
which I've fixed in this PR. (This bug probably originated way back in
#18344.)
This should address a lot of the flakes seen in grpclb_e2e_test
recently.
As such, `alts_zero_copy_grpc_protector_create` will take a
`GsecKeyFactoryInterface` to create `GsecKeyInterface` objects for the
underlying crypter to use.
This enables the caller to control how all the key related buffers are
prepared and protected.
`gsec_aes_gcm_aead_crypter` holds the raw pointer to `GsecKeyInterface`
instead of a `unique_ptr` possibly because somewhere in the test (and
maybe production code as well), the structure is getting copied. A SEGV
error would be caused with `unique_ptr` which doesn't support copy
operations.
Also break the filter stack and promise based versions apart so that I
can re-understand this code.
---------
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
Add TcpTracer interface for TCP instrumentation. It takes no gRPC
dependencies for use in external TCP implementations. Also add
HttpAnnotation for HTTP transport instrumentation using CallTracer.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Reverse dependency edge, so instead of saying `logging_filter _after_
otel`, instead say `otel _before_ logging_filter` - since this doesn't
inadvertently bring otel into builds where it's unnecessary.
Required moving filter class definitions into the header - which mirrors
all other filters, so I think this is fine.
Also required removing the bespoke visibility rules on logging_filter -
which also seems relatively fine (the defaults limit to grpc usage, and
it's hard to see a firm requirement for tighter visibility that that).
Fixes https://github.com/grpc/grpc/issues/34482.
But this is only a bandaid, there is a bigger issue with build
dependencies. AFAICT,
https://github.com/grpc/grpc/blob/master/tools/distrib/fix_build_deps.py
does not work on many targets, e.g. if the target has specified the
`nofixdeps` tag or if a field is set to a variable, e.g.
```
GRPCXX_SRCS = [
"a",
"b",
"c",
]
...
srcs = GRPCXX_SRCS,
```
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Ditch the old priority scheme for ordering filters, instead explicitly
mark up before/after constraints.
---------
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
The basic APIs for the CRL Reloading features.
This adds external types to represent CRL Providers, CRLs, and
CertificateInfo.
Internally we will use `CrlImpl` - this layer is needed to hide OpenSSL
details from the user.
GRFC - https://github.com/grpc/proposal/pull/382
Things Done
* Add external API for `CrlProvider`, `Crl`, `CertInfo` (`CertInfo` is
used during CRL lookup rather than passing the entire certificate).
* Add code paths in `ssl_transport_security` to utilize CRL providers
* Add `StaticCrlProvider`
* Refactor `crl_ssl_transport_security_test.cc` so it is more extensible
and can be used with providers
Instead of fixing a target size for writes, try to adapt it a little to
observed bandwidth.
The initial algorithm tries to get large writes within 100-1000ms
maximum delay - this range probably wants to be tuned, but let's see.
The hope here is that on slow connections we can not back buffer so much
and so when we need to send a ping-ack it's possible without great
delay.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Experiment 1: On RST_STREAM: reduce MAX_CONCURRENT_STREAMS for one round
trip.
Experiment 2: If a settings frame is outstanding with a lower
MAX_CONCURRENT_STREAMS than is configured, and we receive a new incoming
stream that would exceed the new cap, randomly reject it.
---------
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
Cap requests per read, rst_stream handled per read.
If these caps are exceeded, offload processing of the connection to a
backing thread pool, and allow other connections to make progress.
Craig Tiller
Mark D. Roth
Eugene Ostroukhov
Yash Tibrewal
Vignesh Babu
AJ Heller
gRPC Team Bot
Xuan Wang
Adam Cozzette
Alisha Nanda
Luwei Ge
Yousuk Seung
Esun Kim
Gregory Cooke
Yijie Ma
apolcyn