The very non-trivial upgrade of third_party/protobuf to 22.x
This PR strives to be as small as possible and many changes that were
compatible with protobuf 21.x and didn't have to be merged atomically
with the upgrade were already merged.
Due to the complexity of the upgrade, this PR wasn't created
automatically by a tool, but manually. Subsequent upgraded of
third_party/protobuf with our OSS release script should work again once
this change is merged.
This is best reviewed commit-by-commit, I tried to group changes in
logical areas.
Notable changes:
- the upgrade of third_party/protobuf submodule, the bazel protobuf
dependency itself
- upgrade of UPB dependency to 22.x (in the past, we used to always
upgrade upb to "main", but upb now has release branch as well). UPB
needs to be upgraded atomically with protobuf since there's a de-facto
circular dependency (new protobuf depends on new upb, which depends on
new protobuf for codegen).
- some protobuf and upb bazel rules are now aliases, so `
extract_metadata_from_bazel_xml.py` and `gen_upb_api_from_bazel_xml.py`
had to be modified to be able to follow aliases and reach the actual
aliased targets.
- some protobuf public headers were renamed, so especially
`src/compiler` needed to be updated to use the new headers.
- protobuf and upb now both depend on utf8_range project, so since we
bundle upb with grpc in some languages, we now have to bundle utf8_range
as well (hence changes in build for python, PHP, objC, cmake etc).
- protoc now depends on absl and utf8_range (previously protobuf had
absl dependency, but not for the codegen part), so python's
make_grpcio_tools.py required partial rewrite to be able to handle those
dependencies in the grpcio_tools build.
- many updates and fixes required for C++ distribtests (currently they
all pass, but we'll probably need to follow up, make protobuf's and
grpc's handling of dependencies more aligned and revisit the
distribtests)
- bunch of other changes mostly due to overhaul of protobuf's and upb's
internal build layout.
TODOs:
- [DONE] make sure IWYU and clang_tidy_code pass
- create a list of followups (e.g. work to reenable the few tests I had
to disable and to remove workaround I had to use)
- [DONE in cl/523706129] figure out problem(s) with internal import
---------
Co-authored-by: Craig Tiller <ctiller@google.com>
This reverts commit 7bd9267f32.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Upgrade boringssl to the latest "master-with-bazel"
- use the `'USE_HEADERMAP' => 'NO'` fix for ObjC
- update the key for asm optimizations on mac/apple in python's setup.py
This PR depends on monterey fixes here:
https://github.com/grpc/grpc/pull/32493 and the boringssl's build
simplification
https://boringssl-review.googlesource.com/c/boringssl/+/56465.
---------
Co-authored-by: Hannah Shi <hannahshisfb@gmail.com>
* WRR: port StaticStrideScheduler to OSS
* WIP
* Automated change: Fix sanity tests
* fix build
* remove unused aliases
* fix another type mismatch
* remove unnecessary include
* move benchmarks to their own file, and don't run it on windows
* Automated change: Fix sanity tests
* add OOB reporting
* generate_projects
* clang-format
* add config parser test
* clang-tidy and minimize lock contention
* add config defaults
* add oob_reporting_period config field and add basic test
* Automated change: Fix sanity tests
* fix test
* change test to use basic RR
* WIP: started exposing peer address to LB policy API
* first WRR test passing!
* small cleanup
* port RR fix to WRR
* test helper refactoring
* more test helper refactoring
* WIP: trying to fix test to have the right weights
* more WIP -- need to make pickers DualRefCounted
* fix timer ref handling and get tests working
* clang-format
* iwyu and generate_projects
* fix build
* add test for OOB reporting
* keep only READY subchannels in the picker
* add file missed in a previous commit
* fix sanity
* iwyu
* add weight expiration period
* add tests for weight update period and OOB reporting period
* Automated change: Fix sanity tests
* lower bound for timer interval
* consistently apply grpc_test_slowdown_factor()
* cache time in test
* add blackout_period tests
* avoid some unnecessary copies
* clang-format
* add field to config test
* simplify orca watcher tracking
* attempt to fix build
* iwyu
* generate_projects
* update xds proto dependency
* add xDS LB policy entry to registry
* add "_experimental" suffix to policy name
* update LB policy name and remove debug log
* add env var protection
* generate_projects
* gen_upb_api
* WRR: update tests to cover qps plumbing
* WIP
* Automated change: Fix sanity tests
* more WIP
* basic WRR e2e test working
* add OOB test
* add xDS WRR e2e test
* clang-format
* fix sanity
* ignore duplicate addresses
* Automated change: Fix sanity tests
* add new tracer to doc/environment_variables.md
* retain scheduler state across pickers
* Automated change: Fix sanity tests
* use separate mutexes for scheduler and timer
* sort addresses to avoid index churn
* remove fetch_sub for wrap around in RR case
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
* Upgrade google/benchmark to v1.7.0
Reason: Setup and Teardown methods are nice to have for multithreaded
tests.
* use version tag
* Revert "use version tag"
This reverts commit 6c7f5e0d35.
* Update third_party/protobuf
* run tools/distrib/python/make_grpcio_tools.py
* regenerate protos for ruby, php
* update build_handwritten.yaml
* regenerate projects
* Build - Use :well_known_type_protos instead of :well_known_protos
* Fix target
* Update upb
* Update Python for Protobuf 4.21 (#140)
* Update protobuf dependency on grpcio-tools
* Off by one
* Drop python 3.6 support
* Try upgrading pip
* And in the other script
* Try to figure out if we're compatible with abi3
* See what we've already got installed
* Update the requirements.txt file I didn't know existed
* And here too
* See what's installed
* Let's try that again
* Remove
* Try to confirm version
* Let me see the generated code
* Fix non-Bazel test runner
* Work for all test directories
* Regenerate example protos
* Clean up
* Generate .pyi files
* Fix type checking and linting
* Exclude pyi files from isort
* Upgrade to 3.21.4
* Update iwyu to get around messy protobuf IWYU rules
Co-authored-by: Richard Belleville <gnossen@gmail.com>
* xDS: upgrade the commit of the xDS protos that we're using
* Automated change: Fix sanity tests
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
* Upgrade benchmark to 1.6.0 and remove hacks.
Details:
- GRPC currently uses an old version of benchmark (from Sept 2020). It should probably upgrade because downstream, in google3, everyone is already using 1.6.0)
- Removed the hack added in PR/27629 to allow benchmarks in GRPC to continue to work with both pre-1.6.0 and 1.6.0 benchmarks.
(This was needed to allow importing benchmarks 1.6.0 into google3 without breaking GRPC)
* fix typo
* update third_party/benchmark and check_submodules.sh
* Upmerge from v1.41.x (#27821)
* Bump version to v1.41.0-pre1 (#27371)
* Bump version to v1.41.0-pre1
* Regenerate projects
* [Backport #27373] add testing_version flag (#27385)
* Bump version to v1.41.0-pre2 (#27390)
* Bump version to v1.41.0-pre2
* Regenerate projects
* Core 19: bump core version from 18.0.0 to 19.0.0 (#27394)
* Bump core version to 19.0.0
* Regenerate projects
* fix use-after-free metadata corruption in C# when receiving response headers for streaming response calls (#27398)
* Final release: bump up version to 1.41.0 (#27476)
* Bump version to 1.41.0
* Regenerate projects
* xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580)
* Revert "xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580)" (#27590)
This reverts commit da0c7d680f.
* Update root pem certs (backport of #27539) (#27619)
* Update boringssl to the latest (#27606) (#27625)
* Change boringssl branch name
* update submodule boringssl-with-bazel with origin/main-with-bazel
* update boringssl dependency to main-with-bazel commit SHA
* regenerate files
* Increment podspec version
* generate boringssl prefix headers
* Bumping up version to v1.41.1 (#27699)
* Bump version to v1.41.1
* Regenerate projects
* [Backport][v1.41.x] xds-k8s tests: Use test driver from master branch (#27695)
Backports sourcing the test driver install script from master.
This is a backport of #27389, #27462 and #27658:
* Add missing quatation marks.
These were missed when creating the Python virtual env.
* xds-k8s tests: Use test driver from master branch (#27462)
Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch.
This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches.
All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.
* xds_k8s tests: Fix xlang install script sourcing. (#27658)
This change sources the test driver install script correctly for the xlang tests.
This fixes a mistake in #27462 where this was missed.
* Fix Python Interop (#27620) (#27703)
* WIP. Attempt to fix interop
* Yapf
* Switch Python xDS Example Server to Listen on IPV4 Only (#27679)
* Switch to IPV4
* Update to all hosts
* Fix rvm ruby install failure (#27769)
Co-authored-by: donnadionne <donnadionne@google.com>
Co-authored-by: Lidi Zheng <lidiz@google.com>
Co-authored-by: Jan Tattermusch <jtattermusch@users.noreply.github.com>
Co-authored-by: sanjaypujare <sanjaypujare@users.noreply.github.com>
Co-authored-by: Sergii Tkachenko <sergiitk@google.com>
Co-authored-by: Esun Kim <veblush@google.com>
Co-authored-by: Terry Wilson <terrymwilson@gmail.com>
Co-authored-by: Richard Belleville <rbellevi@google.com>
* added perf_counters.cc manually since the script didn't work
Co-authored-by: Mark D. Roth <roth@google.com>
Co-authored-by: donnadionne <donnadionne@google.com>
Co-authored-by: Lidi Zheng <lidiz@google.com>
Co-authored-by: Jan Tattermusch <jtattermusch@users.noreply.github.com>
Co-authored-by: sanjaypujare <sanjaypujare@users.noreply.github.com>
Co-authored-by: Sergii Tkachenko <sergiitk@google.com>
Co-authored-by: Esun Kim <veblush@google.com>
Co-authored-by: Terry Wilson <terrymwilson@gmail.com>
Co-authored-by: Richard Belleville <rbellevi@google.com>
* Upgrade libuv to v1.42.0
Also eliminates some cruft from the node implementation from 5 years ago
* sanitize.sh added a blank line?? ok then.
* pin to libuv v1.37.0
* Upgrade re2 to fix broken template on gcc-11.
The fix to re2 was made here:
0ceda87618
* Use release 2021-09-01 of re2. SHA256 pending.
* Adds sha256 sum for re2 release.
* Update sha256 sum.
* Update submodule third_party/re2 to 8e08f47b11b413302749c0d8b17a1c94777495d5 (release 2021-09-01)
* Update the submodule sanity check.
Yash Tibrewal
Jan Tattermusch
Luwei Ge
Yousuk Seung
Stanley Cheung
Esun Kim
Mark D. Roth
Alisha Nanda
Mikhail Lappo
Richard Belleville
AJ Heller
apolcyn
Nicolas Noble
donnadionne
Lidi Zheng
Jan Tattermusch
Craig Tiller
Vy Nguyen
Yesudeep Mangalapilly