The very non-trivial upgrade of third_party/protobuf to 22.x
This PR strives to be as small as possible and many changes that were
compatible with protobuf 21.x and didn't have to be merged atomically
with the upgrade were already merged.
Due to the complexity of the upgrade, this PR wasn't created
automatically by a tool, but manually. Subsequent upgraded of
third_party/protobuf with our OSS release script should work again once
this change is merged.
This is best reviewed commit-by-commit, I tried to group changes in
logical areas.
Notable changes:
- the upgrade of third_party/protobuf submodule, the bazel protobuf
dependency itself
- upgrade of UPB dependency to 22.x (in the past, we used to always
upgrade upb to "main", but upb now has release branch as well). UPB
needs to be upgraded atomically with protobuf since there's a de-facto
circular dependency (new protobuf depends on new upb, which depends on
new protobuf for codegen).
- some protobuf and upb bazel rules are now aliases, so `
extract_metadata_from_bazel_xml.py` and `gen_upb_api_from_bazel_xml.py`
had to be modified to be able to follow aliases and reach the actual
aliased targets.
- some protobuf public headers were renamed, so especially
`src/compiler` needed to be updated to use the new headers.
- protobuf and upb now both depend on utf8_range project, so since we
bundle upb with grpc in some languages, we now have to bundle utf8_range
as well (hence changes in build for python, PHP, objC, cmake etc).
- protoc now depends on absl and utf8_range (previously protobuf had
absl dependency, but not for the codegen part), so python's
make_grpcio_tools.py required partial rewrite to be able to handle those
dependencies in the grpcio_tools build.
- many updates and fixes required for C++ distribtests (currently they
all pass, but we'll probably need to follow up, make protobuf's and
grpc's handling of dependencies more aligned and revisit the
distribtests)
- bunch of other changes mostly due to overhaul of protobuf's and upb's
internal build layout.
TODOs:
- [DONE] make sure IWYU and clang_tidy_code pass
- create a list of followups (e.g. work to reenable the few tests I had
to disable and to remove workaround I had to use)
- [DONE in cl/523706129] figure out problem(s) with internal import
---------
Co-authored-by: Craig Tiller <ctiller@google.com>
This PR aims to de-experimentalize the APIs for GCP Observability.
We would have ideally wanted public feedback before declaring the APIs
stable, but we need stable APIs for GA.
Changes made after API review with @markdroth@veblush, @ctiller and the
entire Core/C++ team -
* The old experimental APIs `grpc::experimental::GcpObservabilityInit`
and `grpc::experimental::GcpObservabilityClose` are now deprecated and
will be deleted after v.1.55 release.
* The new API gets rid of the Close method and follows the RAII idiom
with a single `grpc::GcpObservability::Init()` call that returns an
`GcpObservability` object, the lifetime of which controls when
observability data is flushed.
* The `GcpObservability` class could in the future add more methods. For
example, a debug method that shows the current configuration.
* Document that GcpObservability initialization and flushing (on
`GcpObservability` destruction) are blocking calls.
* Document that gRPC is still usable if GcpObservability initialization
failed. (Added a test to prove the same).
* Since we don't have a good way to flush stats and tracing with
OpenCensus, the examples required users to sleep for 25 seconds. This
sleep is now part of `GcpObservability` destruction.
Additional Implementation details -
* `GcpObservability::Init` is now marked with `GRPC_MUST_USE_RESULT` to
make sure that the results are used. We ideally want users to store it,
but this is better than nothing.
* Added a note on GCP Observability lifetime guarantees.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Adds a simple hello-world example for GCP Observability purposes along
with a README.
The microservices observability user guide is not yet up, but this still
refers to it anyway.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
1. All greeter servers now support flag `--port` to customize the listen
port.
2. All client implementation now have `--target` flag to specify the
server location.
3. ABSL is used to parse the flags to reduce the amount of boilerplate
code in the examples.
Fixes: #26989
* Add timeout example
* Add pb2 file to example
* Remove .proto file
* Add keep_alive example
* Add refelction client
* fixes
* Add example for health_check
* Changes based on comments
* Fix pylint
* Add wait_for_ready with client timeout example
* Use with to create channel
* Fix sanity tests
* Changes based on comments
* Change thread to daemon thread
* Changes based on comments
* Revert "Revert "Add support for systemd socket activation (#30485)" (#31617)"
This reverts commit 867dc6cae2.
* Add checks to unix tests
* Ran generate_projects.sh and fixed styling in test
* Fix variable in unit test
* Use reinterpret_cast in test
* Rebase and fix sanity failures
* Add concurrency into python example in route_guide
- replace await single tasks sequentially with await task_group
* Add comments for asyncio.gather for concurrency
* Correct typo
* Update asyncio_route_guide to meet google style
Co-authored-by: Richard Belleville <rbellevi@google.com>
* Update third_party/protobuf
* run tools/distrib/python/make_grpcio_tools.py
* regenerate protos for ruby, php
* update build_handwritten.yaml
* regenerate projects
* Build - Use :well_known_type_protos instead of :well_known_protos
* Fix target
* Update upb
* Update Python for Protobuf 4.21 (#140)
* Update protobuf dependency on grpcio-tools
* Off by one
* Drop python 3.6 support
* Try upgrading pip
* And in the other script
* Try to figure out if we're compatible with abi3
* See what we've already got installed
* Update the requirements.txt file I didn't know existed
* And here too
* See what's installed
* Let's try that again
* Remove
* Try to confirm version
* Let me see the generated code
* Fix non-Bazel test runner
* Work for all test directories
* Regenerate example protos
* Clean up
* Generate .pyi files
* Fix type checking and linting
* Exclude pyi files from isort
* Upgrade to 3.21.4
* Update iwyu to get around messy protobuf IWYU rules
Co-authored-by: Richard Belleville <gnossen@gmail.com>
* replace darwin checks in extconf.rb to exclude TruffleRuby
* inherit RANLIB and STRIP from RbConfig, set LDXX
* enable overriding ranlib command in top-level makefile
* ensure the -no_warning_for_no_symbols flag is only used with Apple's ranlib
* don't embed openssl & zlib on truffleruby
* add RbConfig's cppflag to CPPFLAGS when using TruffleRuby
* this ensure the paths to find the system's OpenSSL are set up correctly with TruffleRuby (includes being able to find an OpenSSL installed via Homebrew etc)
* don't statically link standard libraries on Linux with Truffleruby
* This does not work when compiling to bitcode.
* Prefer SIGTERM to SIGQUIT for graceful shutdown in examples
* Overriding SIGQUIT is suboptimal, for example on JVM where it is very
useful to dump the thread stacktraces.
* Keep the rb_tr_abi_version symbol for TruffleRuby in grpc_c.so
* Otherwise TruffleRuby cannot verify the ABI version is correct.
* See https://github.com/oracle/truffleruby/issues/2386
* Use RbConfig::CONFIG['STRIP'] instead of just `strip`
* Use a local variable for apple_toolchain for consistency
* Add a comment about -static-libgcc -static-libstdc++ and TruffleRuby
* Split comment into two for openssl/zlib
Co-authored-by: Nicolas Laurent <nicolas.laurent@oracle.com>
* Fixing RouteGuide's C++ Reactor example.
The current method involves locking and unlocking a mutex from different
threads, which isn't allowed. Changing the strategy a bit to address
this.
* Automated change: Fix sanity tests
* Switching to absl::Mutex to annotate usage properly.
* Actually, let's not cover the examples with sanity checks.
* Update target frameworks in examples.
* Update target frameworks and NuGets in examples.
* upgrade Xds example
Co-authored-by: Jan Tattermusch <jtattermusch@google.com>
* Add python async example for hellostreamingworld using generator
* add stub.read() client example + fix Copyrights + small improvements
* Format and lint + remove exernal generator
* Apply lint and auto-format
Jan Tattermusch
Yash Tibrewal
Eugene Ostroukhov
Xuan Wang
Richard Belleville
Esun Kim
Ian Rodney
Ilya Lukin
Andres Beltran
Ho Ngok Chao
Stanley Cheung
niquepolice
Craig Tiller
Mark D. Roth
AmosWang
h-vetinari
AJ Heller
Benoit Daloze
dependabot[bot]
Lidi Zheng
Xiao
Nicolas Noble
Jan Tattermusch
rburns1293
Ming-Chuan
Michael Mercier