The very non-trivial upgrade of third_party/protobuf to 22.x
This PR strives to be as small as possible and many changes that were
compatible with protobuf 21.x and didn't have to be merged atomically
with the upgrade were already merged.
Due to the complexity of the upgrade, this PR wasn't created
automatically by a tool, but manually. Subsequent upgraded of
third_party/protobuf with our OSS release script should work again once
this change is merged.
This is best reviewed commit-by-commit, I tried to group changes in
logical areas.
Notable changes:
- the upgrade of third_party/protobuf submodule, the bazel protobuf
dependency itself
- upgrade of UPB dependency to 22.x (in the past, we used to always
upgrade upb to "main", but upb now has release branch as well). UPB
needs to be upgraded atomically with protobuf since there's a de-facto
circular dependency (new protobuf depends on new upb, which depends on
new protobuf for codegen).
- some protobuf and upb bazel rules are now aliases, so `
extract_metadata_from_bazel_xml.py` and `gen_upb_api_from_bazel_xml.py`
had to be modified to be able to follow aliases and reach the actual
aliased targets.
- some protobuf public headers were renamed, so especially
`src/compiler` needed to be updated to use the new headers.
- protobuf and upb now both depend on utf8_range project, so since we
bundle upb with grpc in some languages, we now have to bundle utf8_range
as well (hence changes in build for python, PHP, objC, cmake etc).
- protoc now depends on absl and utf8_range (previously protobuf had
absl dependency, but not for the codegen part), so python's
make_grpcio_tools.py required partial rewrite to be able to handle those
dependencies in the grpcio_tools build.
- many updates and fixes required for C++ distribtests (currently they
all pass, but we'll probably need to follow up, make protobuf's and
grpc's handling of dependencies more aligned and revisit the
distribtests)
- bunch of other changes mostly due to overhaul of protobuf's and upb's
internal build layout.
TODOs:
- [DONE] make sure IWYU and clang_tidy_code pass
- create a list of followups (e.g. work to reenable the few tests I had
to disable and to remove workaround I had to use)
- [DONE in cl/523706129] figure out problem(s) with internal import
---------
Co-authored-by: Craig Tiller <ctiller@google.com>
* Initial structure for RLS
* Adding and building the proto to parse the Any proto for the plugins
* re-org
* Parsing the plugin
* Parsing more into json
* Parsed proto to json
* small cleanup
* Adding prefix
* Added new rls_experimental policy
* build files
* Fixing according to code review comments
* code review comments
* Adding sym changes
* adding action name check
* fixing code review comments.
* fixing unused var error
* clean up
* fixing code review comments
* fixing code review comments
* fixing according to code review comments.
* Remove unnecessary include
* small fix
* generate more, hard-code less
* Moving to using absl::variant
* absl::string_view and absl::variant of vector of std::string are not
playing nice together.
* fixed variant
* Using absl::variant now
* Checkint used plugins
* Refactor Parsing code and separating out Parsing of the plugin
* Fixing code review comments
* code review comments
* fixing code review comments.
* Addressing code review comments
* First end-to-end test
* generated build files
* commit generated files via tools/codegen/core/gen_upb_api.sh
* Fixing rls policy parsing tests
* Restore checks for the test server
* Refactor rls_server
* added keys to rls request
* fixing small logic error
* Complete the test using all the keys
* Separating out RLS test and rls_server thread
* sanity errors
* generated build files
* Complete the rest of the tests and sanity cleanup
* fixing code review comments: using upb_JsonEncode now!
* fixing code review comments
* fixing code review comments
* Fixing code review comments
* misisng fix
* simplifying tests
* simplify tests 2
* Linking in the correct proto for rls_config
* restore metadata check
* Add disable test
* Fixing RLS test and removing environment var that is no longer necessary
* Fixing "Wrong type" type of tests after json parsing change to accept
STRING for number
* adding json_encode.h/c to src/upb/gen_build_yaml.py and generate
necessary files.
* Fixing un-used var error
* fixing sanity errors
* Fixing the upb encoding buffer
* Fixing code review comments.
* Adding nack test for unkonwn plugin proto
* Last bit of code review comments
* fixing unused variable
* Squashed 'third_party/upb/' changes from 60607da72e..2de300726a
2de300726a Merge pull request #401 from philwo/patch-1
0010bd88ff Merge pull request #403 from haberman/google-wyhash
3f7ecfe315 A few fixes in the 32-bit fallback.
2c14e2788d Added #include for Windows instrinsic.
9e618009ec Removed some extraneous whitespace.
c4744c0b21 Updated generated files.
039975556d Merge branch 'master' into google-wyhash
ff9d011f12 Merge pull request #402 from haberman/update-ads-benchmark
c2f2e93990 Fix Bazel at 4.1.0
41bfbca375 Updated ads benchmark to v7 as v5 no longer exists upstream.
91d506ac32 Ported ABSL's wyhash to C.
3328f06117 Remove ubuntu1604 from presubmit.yml
f925acf5f0 Merge pull request #400 from haberman/extreg
897bd2500a Fixed compile error.
6e53de4a03 Addressed PR comments.
cdd6434a31 Introduced upb_extreg and plumbed it into decoder.
69f186bd75 Merge pull request #399 from haberman/msg-extrep
3f8aa6ef20 Define the extension representation in messages and mini-tables.
4ccfab154b Merge pull request #398 from haberman/field-mode
58e158c6fa Changed mini-table to use a custom "mode" instead of descriptor's "label."
7596fa740f Merge pull request #397 from haberman/codegen-refactor
fa4dfc2baa Addressed PR comments.
0fb61eaeb5 Refactored the codegen into smaller functions, in anticipation of extensions.
52be845c39 Merge pull request #396 from haberman/pre-ext-opt
807e7fe9e2 Fixed dense_below logic to be order-independent and consistent between def.c and codegen.
2e8a122fc0 Changed dense_below calculation to use UINT8_MAX as the constant.
6394894b6e Addressed PR comments.
65d7b8ab0c Optimized decoder and paved the way for parsing extensions.
3e035cb553 Merge pull request #395 from haberman/locale
e8ba2a1899 Added a fix for locales that output ',' as decimal separator.
7010a73828 Merge pull request #394 from haberman/file-uniqueness
9482957425 Enforce that filenames are unique when loaded into symtab.
aaad7801bf Merge pull request #393 from haberman/has-attribute
7887dc0c76 Fixed the __has_attribute() check for old versions of Clang.
01e7436ed3 Merge pull request #390 from haberman/musttail
4132034634 Addressed PR comment.
ed708fcd5d Addressed PR comments.
876abae2db Removed some debug printing and simplified checktag slightly.
286441afa7 Fixed a size regression due to inlining UTF-8 verification.
75df4cdaa3 Loosened the check for fasttable to match what we did before.
e84793dd73 Cleaned up debugging artifacts.
a4b35aa388 Everything passes except 4 conformance tests.
6e10b5789d Removed redundant license from port_def in amalgmation.
4f1e48ecc6 Updated amalgamator to avoid duplicating license blocks.
0723bfa700 Merge pull request #387 from haberman/json-null
dda5416569 Update bazel-skylib main branch name.
2fa52e6951 Merge branch 'master' into json-null
a3126b7c7e Merge pull request #388 from haberman/fix-macos
b344a6c582 Fixed the macOS build for XCode 11.3.
a778f2639a Only perform multiple oneof check for non-null fields.
42bdfcb849 Fix BUILD file.
75d6dab37a Merge pull request #384 from haberman/internal
3881393907 Renamed .int.h to _internal.h, for greater clarity.
5084cbe646 Merge pull request #383 from fowles/master
da4b75b979 Fix build rules for more restrive deps checks
a8f9cf54cf Merge pull request #382 from haberman/license-headers
1ba446ccfe Updated Google, Inc. to Google LLC.
823eb09694 Update all 2011 dates to 2021.
ff89277ef0 Merge pull request #380 from fowles/master
5f74d43cf9 Re-add some comment text that was accidentally removed.
0f27b0450a Merge branch 'master' into license-headers
e59d2c8fa7 Added license headers to all files.
be4f64b926 Merge pull request #381 from haberman/mv-msg-public
1674f28dd7 Put public message interface into msg.h and moved internal functions to msg.int.h.
6481f6319e Spell out deps explicitly
f104225a1e Merge pull request #376 from haberman/fix-quadratic-mem
3c3799498d Merge pull request #379 from fowles/master
2e83d5c98f Remove unused file test_util.h
b053fa6991 Merge pull request #378 from fowles/master
7876639e50 remove unused macro
5b97df91dd Restrict fuse to matching block_alloc
e74d6c23de Small renames and use uintptr_t instead of void*
d9a0c58108 Allow arena fuse to fail
53f5d9f69b Merge pull request #377 from haberman/json-parse-null
8370818143 Switch to a macro to avoid signed/const mismatches.
63ad3db980 Fixed "NULL + 0" UB in JSON encoder and decoder.
add9b12f18 Fixed quadratic memory usage in upb_array_append().
97e2aeb7ee Merge pull request #375 from haberman/cleanups
83c0edbd2a A few minor cleanups.
baa7fe7473 Merge pull request #374 from haberman/update-deps
900bd5c426 Merge pull request #355 from haberman/aarch64
89df647460 Updated versions of ABSL and protobuf deps.
e4343f0fa3 Update comment for ARM64.
103d51f102 Merge branch 'master' into aarch64
1b9cc09f0e Removed extraneous copybara directive.
bbd817fdb6 Merge pull request #373 from haberman/table-cleanup
c358829c76 Now that handlers are gone, cleaned up table to use arenas exclusively.
ed5b4108e0 Merge pull request #363 from haberman/delete-handlers
bfa528f0ae Merge pull request #372 from haberman/errmsg-fix
ec9ba3f893 Fixed error message buffer overflow.
eb0fdda14b Merge pull request #371 from haberman/json-any
e58f7a0284 Fix for Any messages with no fields.
7541ba0f38 Merge pull request #364 from catenacyber/cifuzz
81829bacc3 Removed quote chars from error message.
88ed8f5d3b Changed JSON decoding error message to be more clear.
f41c0ec261 Added an internal API to get arena from symtab, for Ruby's use.
b080659eee Merge pull request #367 from haberman/timestamp-minute-offset
f7ed1f27a3 Support non-zero minutes in the timestamp offset for JSON.
451dcbbb9a Adds CIFuzz
f5d2d55007 Deleted the legacy "Handlers" APIs. upb can finally be deserving of its name.
0f40d59258 Merge pull request #362 from haberman/rb-warnings
c7787cbaa1 Fixed a bunch of Clang warnings.
5e53b5bb3d Merge pull request #360 from haberman/default-msgval
7a6e647be1 Merge branch 'master' into default-msgval
edb0a9a2d9 Merge pull request #361 from haberman/arena-bugfix
9175989431 Bugfix for arena cleanup list when passing to upb_decode().
5e550e88f8 Added API for getting fielddef default as a upb_msgval.
de76b31f9c Merge pull request #359 from veblush/fix-upbdefs
3cab8a38fc Fixed upbdefs error
48bb3e5662 Merge pull request #357 from veblush/string-opt
4c67b2086b Merge pull request #358 from veblush/clang-tidy-fix
38b4beed6b Fix clang-tidy function 'upb_encode_ex' has a definition with different parameter names
9b020d8f65 Optimize calls to std::string::find() and friends for a single char.
3c9ae7837a The fasttable parser works on ARM64!
git-subtree-dir: third_party/upb
git-subtree-split: 2de300726a1ba2de9a468468dc5ff9ed17a3215f
* Updated upb for bazel
* Regen upb files
* Updated src/upb/gen_build_yaml.py
* Implement the xDS Config Dump as CSDS in Core
* Revemp the logic of caching and constructing the CSDS response
* Unref created error
* Fix the clang tidy complains
* Resolve comments about symbol preload and Json ctor
* Improve readability
* Merge with new ADS logic && support latest CSDS
* Refactor the version/client_status logic
* Add support for error_state
* Add support for does_not_exist
* Add node information
* Fix the ProtoBuf segfault
* More test cases
* Refactor to use bytes as cache instead of JSON
* Apply clang_tidy's suggestion
* Resolve reviewer's requests
* Tiny stylish fixes && make sanity test happy
* WIP: still working on the matchers
* Update a bunch of matchers
* Improve readability a bit
* Resolve reviewer's suggestions
* Resolve reviewer's comments
* Make Unpack fail fast
* Ran `2to3` on `tools/buildgen` and the `gen_build_yaml.py` files.
* Updated the invocations of python to be explicitly Python 3.
* Changed the loader in `mako_renderer.py` to allow pickle to find the module for pickling `external_version.Version`
* Added a few open-as-binary to placate readers/writers expecting bytes.
* Hand tweaked the templates to remove `iteritems` and fiddle with encodings.
Partially addresses #24359.
All these changes need to go together to make sense
- changes to use new version of upb in bazel
- allowing includes in build target option
- script for generating c code (upb) for protos
- generated code for example protos
- adding changes for non-bazel builds
- change sanity tests to ignore the generated files.
protos
All these changes need to go together to make sense
- changes to use new version of upb in bazel
- allowing includes in build target option
- script for generating c code for protos
- generated code for example build
- adding changes for non-bazel builds
- change sanity tests to ignore the generated files.
Esun Kim
Jan Tattermusch
donnadionne
Lidi Zheng
capstan
Mark D. Roth
Yash Tibrewal
Nicolas "Pixel" Noble
Vishal Powar
Alexander Polcyn
Vishal Powar