Split off from https://github.com/grpc/grpc/pull/37425
We are adding and removing callbacks on the OpenTelemetry Async Instruments without synchronization. This opens us to races where we have an AddCallback and RemoveCallback operation happening at the same time. The correct result after these operations is to still have a callback registered with OpenTelemetry at the end, but the two operations could race and we could just decide to remove the OpenTelemetry callback.
The fix delays removing OpenTelemetry callbacks to plugin destruction time.
Closes#37485
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37485 from yashykt:FixRaceOTelGauge 016b0a41b5
PiperOrigin-RevId: 663492598
It is hard to reason about tests if multiple callbacks record values for the same metrics with the same label sets.
Closes#37486
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37486 from yashykt:SimplifyOTelPluginTest 3b4d7f90b3
PiperOrigin-RevId: 663457107
Previously the registered callback's duration is set too low (10ms) that 2 different OTel callbacks get called from OTel consecutively would trigger the registered callback again, making the test flaky.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37243
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37243 from yijiem:otel-plugin-test-flake 70a1572cd2
PiperOrigin-RevId: 653717643
Instead of passing the transport byte counts back up through the filter
stack to be reported to the `CallTracer`, we now have the transport
pass the transport byte counts directly to the `CallTracer` itself.
This will eventually allow us to avoid unnecessarily storing these byte
counts in cases where no `CallTracer` actually cares about the data, which
will reduce per-call memory. (In the short term, it actually increases
memory usage, but we can separately do some work to avoid the memory
usage in the transport by removing the `grpc_transport_stream_stats`
struct from the legacy filter API.)
This is a prereq for supporting `CallTracer` in the new call v3 stack,
which does not include the transport byte counts as part of the
receieve-trailing-metadata hook, unlike the legacy filter stack.
This change is controlled by the `call_tracer_in_transport` experiment,
which is enabled by default.
As part of this experiment, we also fix a couple of related bugs:
- On the client side, the chttp2 transport was incorrectly adding
annotations to the parent `ClientCallTracer` instead of the
`CallAttemptTracer`.
- The OpenCensus `ServerCallTracer` was incorrectly swapping the values
of sent and received bytes.
PiperOrigin-RevId: 650728181
This allows CallTracers to be created with parameters dictated by
channel args.
For the moment, I've used the EventEngine `EndpointConfig` API to expose
the channel args here, so as to avoid directly exposing
`grpc_core::ChannelArgs`. We should determine a better API here before
we de-experimentalize the stats APIs.
Also add an experiment to be used in a subsequent PR.
PiperOrigin-RevId: 647730284
Preparation for switching away from `grpc_channel_filter*` to identify channel filters.
Closes#36907
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36907 from ctiller:type-name e7ad4c67a2
PiperOrigin-RevId: 644483948
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36729
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36729 from yijiem:per-channel-stats-plugin 4786bed42f
PiperOrigin-RevId: 642030366
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - gpr_log
In this CL we are migrating from gRPCs own gpr logging mechanism to absl logging mechanism. The intention is to deprecate gpr_log in the future.
We have the following mapping
1. gpr_log(GPR_INFO,...) -> LOG(INFO)
2. gpr_log(GPR_ERROR,...) -> LOG(ERROR)
3. gpr_log(GPR_DEBUG,...) -> VLOG(2)
Reviewers need to check :
1. If the above mapping is correct.
2. The content of the log is as before.
gpr_log format strings did not use string_view or std::string . absl LOG accepts these. So there will be some elimination of string_view and std::string related conversions. This is expected.
Closes#36678
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36678 from tanvi-jagtap:test_cpp_gpr_log 81b8f4179f
PiperOrigin-RevId: 636410944
https://github.com/grpc/grpc/pull/36598 made a change to the logging filter which required `CallTracerAnnotationInterface` to always be present on the call. That is only the case when metrics/tracing is enabled, which is not always the case.
Also, modify the test to test that logging works even if metrics/tracing is not enabled.
Internal ref - b/341794662
Closes#36671
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36671 from yashykt:FixLogging 4f736e7e56
PiperOrigin-RevId: 635610705
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - BUILD
In this CL we are just editing the build and bzl files to add dependencies.
This is done to prevent merge conflict and constantly having to re-make the make files using generate_projects.sh for each set of changes.
Closes#36607
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36607 from tanvi-jagtap:build_test_cpp 3e17d778d0
PiperOrigin-RevId: 633523097
This adds compile-time checking that the type of the value and the size of the labels and optional labels passed when recording a metric must match with the type and the size specified when the metric is registered.
The RegistrationBuilder API idea is credited to @ctiller.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36449
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36449 from yijiem:registration-builder-api a727810136
PiperOrigin-RevId: 632271022
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - gpr_log
In this CL we are migrating from gRPCs own gpr logging mechanism to absl logging mechanism. The intention is to deprecate gpr_log in the future.
We have the following mapping
1. gpr_log(GPR_INFO,...) -> LOG(INFO)
2. gpr_log(GPR_ERROR,...) -> LOG(ERROR)
3. gpr_log(GPR_DEBUG,...) -> VLOG(2)
Reviewers need to check :
1. If the above mapping is correct.
2. The content of the log is as before.
gpr_log format strings did not use string_view or std::string . absl LOG accepts these. So there will be some elimination of string_view and std::string related conversions. This is expected.
Closes#36547
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36547 from tanvi-jagtap:test_cpp_ext_stats_plugin_tjagtap 8e80774fe1
PiperOrigin-RevId: 632020672
Before this change, on a trailers-only response, Metadata Exchange needed by CSM would just be dropped, and hence CSM labels would not be seen.
Changes -
* OTel call attempt tracer populates labels from trailers if it's a trailers-only response.
* HTTP2 layer propagates the Metadata Exchange field from headers to trailers for trailers-only responses.
* Add a test to make sure that retries continue to work when Metadata Exchange is enabled and a trailers-only response is sent. (This verifies that a trailers-only response remains a trailers-only response.)
Closes#36413
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36413 from yashykt:MetadataExchangeInTrailers e7d202685e
PiperOrigin-RevId: 630144618
Currently channel filter construction returns a `StatusOr<T>`, this change makes it return a `StatusOr<P<T>>` where P is `unique_ptr`, `OrphanablePtr`, `RefCountedPtr`, `DualRefCountedPtr`, etc (most of the code really doesn't need to know, so I'm choosing to leave the flexibility).
That smart pointer is then stored in the channel stack instance, and dereferenced when needed.
This means that channel filters no longer need to be movable (which is a nice simplification), and puts these level-1 filters on a similar memory management track as the level-2 filters we have planned.
(this change also converts client load reporting to v3 apis -- it's a bit accidentally picked up, but seems ok to pull through too)
Closes#36355
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36355 from ctiller:objectify-me 0eb054b748
PiperOrigin-RevId: 625390977
As per https://github.com/grpc/proposal/pull/419, the experimental optional label `grpc.lb.locality` is added to the follow per-call metrics -
* grpc.client.attempt.duration
* grpc.client.attempt.sent_total_compressed_message_size
* grpc.client.attempt.rcvd_total_compressed_message_size
Closes#36254
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36254 from yashykt:OTelOptionalLabelsOnPerCall c5390c99a1
PiperOrigin-RevId: 622973959
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36182
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36182 from yijiem:grpc-metrics-async-gauge 8614485f3d
PiperOrigin-RevId: 622182710
Also addressing a TODO from previous PRs where `authority` is not being populated in the `ChannelScope`.
Closes#36189
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36189 from yashykt:OTelChannelScope e76f9ce0ca
PiperOrigin-RevId: 621231764
Forked from #35957
This PR refactors the credentials types to remove Secure and Insecure Channel and Call credentials types. We standardize on a `c_creds()` accessor method for all credentials types, which can now be treated uniformly. This notably removes special-case handling of insecure credentials.
The special code-paths for insecure creds are no longer necessary in the wake of #25586.
Closes#36176
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36176 from drfloob:fork/35957/creds-API fd64d59c23
PiperOrigin-RevId: 621008166
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36070
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36070 from yijiem:grpc-metrics 72653727b1
PiperOrigin-RevId: 618529035
Adding testing for `Size()` on `MeshLabelsIterable` as promised on https://github.com/grpc/grpc/pull/35371
This method doesn't get used the OTel C++ library at present so it was untested earlier, and bugs in its implementation were noticed later. To avoid that issue in the future, adding manual testing of this method.
Changes -
* Moved `MeshLabelsIterable` to the header to be able to test its Size method directly. Also had to move around some internals to be able to do so.
* Replaced `absl::variant<grpc_core::Slice, StructPb> metadata_` from decoding the metadata lazily to just decoding it in the constructor. I don't remember the original thinking behind this. Maybe I wanted the decoding to happen lazily, but there doesn't seem to be much gain from doing it lazily, we instead have to deal with the overhead of `variant`.
Closes#35769
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35769 from yashykt:CsmMeshLabelsIterableSizeTesting ec9033b742
PiperOrigin-RevId: 617901839
Earlier, the tests just had bazel support. With CMake support added in #36063, we can also add CI CMake support for the tests. A major benefit of this is that we also get coverage for the various platforms that we test from our portability test suite.
Closes#36087
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36087 from yashykt:OTelCISupport b28fbe02e5
PiperOrigin-RevId: 615543685
Changes -
* Add CMake build support to `grpcpp_otel_plugin`. Currently, we are only supporting the `find_package CONFIG` method for depending on `opentelemetry-cpp`.
* Since, `grpcpp_otel_plugin` is an extension of gRPC, it will not be built by default. To enable building of this target, a new CMake option `gRPC_BUILD_GRPCPP_OTEL_PLUGIN` is being added.
* Also add `CMakeLists.txt` to the otel example.
The `otel_plugin_test` can also be built through cmake but, for the CI to work, there are some additional changes. Those will be made in an upcoming PR.
Closes#36063
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36063 from yashykt:OTelCMakeSupport 3bc783823b
PiperOrigin-RevId: 613734473
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35573
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35573 from yijiem:enable-oss-ee-dns-posix-real 1df91d1d84
PiperOrigin-RevId: 609193851
As discussed, this change adds scoping to `CsmObservability` such that when that object goes out of scope, new channels and servers don't record metrics. In the documentation, I've talked about how existing channels/servers are going to continue to record metrics but i've left room for us to change that behavior in the future.
The current way of doing this is through a global bool since there can only be one plugin right now, but we'll change this to use the global stats plugin registry in the future.
Closes#35835
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35835 from yashykt:DisableCsmObsOnScope 33a7c2f7bc
PiperOrigin-RevId: 605468117
Changes -
* `CsmObservability` API will now use the `CsmOpenTelemetryPluginOption` internally. After this change, `CsmObservability` will enable observability for all channels and servers. (Earlier, `CsmObservability` only enabled observability for CSM-enabled channels and servers.) CSM labels will still be added just for CSM-enabled channels and servers.
* Also, we no longer need the ability to set `LabelInjector` on the `OpenTelemetryPluginBuilder` directly. Instead, we always use `PluginOption` to inject the `LabelInjector`. This simplifies the code as well.
Note that `SetTargetSelector` and `SetServerSelector` APIs on the `OpenTelemetryPluginBuilderImpl` are not being deleted yet since we might need them shortly. This is also why `OpenTelemetryPluginBuilderImpl` is not being deleted right now.
Closes#35803
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35803 from yashykt:CsmO11yApisUsePluginOption cf3d65900d
PiperOrigin-RevId: 604323898
This reverts commit 6318e9e7e9.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35667
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35667 from ctiller:a 032999b51e
PiperOrigin-RevId: 601495207
Just to be future-proof, I'm amending the `void` return status of `BuildAndRegisterGlobal` in `OpenTelemetryPluginBuilder` to absl::Status.
This will be backported to 1.61 as well.
Closes#35659
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35659 from yashykt:UpdateOtelApiToAddStatus 07d3f41b8a
PiperOrigin-RevId: 601458408
We are no longer sure about this API, so re-experimentalizing it.
This PR will be backported to 1.61 as well.
Closes#35660
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35660 from yashykt:ReexperimentalizeCsmPluginOption 4f114a54d9
PiperOrigin-RevId: 601378856
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35573
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35573 from yijiem:enable-oss-ee-dns-posix-real 017b99312f
PiperOrigin-RevId: 601245249
Yash Tibrewal
Yijie Ma
Esun Kim
Mark D. Roth
Craig Tiller
Tanvi Jagtap
AJ Heller