Protobuf is beginning to rely more on upb, pulling in more C files. Hence, grpcio_tool needs the following changes to absorb this.
- Changing the setup.py to support both C and C++ by removing explicit language=c++. Rather it can choose the right compiler by its extension.
- Adding build_extensions injection to deal with C/C++ option conflict as grpcio does.
- Adding protobuf and upb directories to the build script so that it can find newly added source files in protobuf.
Tested by https://github.com/grpc/grpc/pull/35796Closes#36074
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36074 from veblush:grpcio-tool-ready 84abce9083
PiperOrigin-RevId: 613975137
This PR is required in order to upgrade to the `master` branch version of protobuf. To do this, we upgrade to the latest version of `rules_python` by:
- Adding in explicit dependencies on all PyPi Bazel test dependencies
- Resolving the circular dependency this creates for `xds-protos` by giving it a Bazel build
- Generating the Python code for `xds-protos` as part of `generate-projects.sh` since doing so directly in Bazel would be highly fragily, dependent on many other projects' Bazel builds
Closes#34945
PiperOrigin-RevId: 599875020
PanCakes to the rescue!
We noticed that our 'sanity' test was going to fail, but we think we can
fix that automatically, so we put together this PR to do just that!
If you'd like to opt-out of these PR's, add yourself to NO_AUTOFIX_USERS
in .github/workflows/pr-auto-fix.yaml
Co-authored-by: Vignesh2208 <Vignesh2208@users.noreply.github.com>
This (or something like it) is necessary for the automated protobuf
upgrade script, to ensure we don't commit a redundant third_party tree
to the repo. Here is an example of the protobuf upgrade without this
fix: https://github.com/grpc/grpc/pull/33625
One alternative could be adding the filepaths to `.gitignore`.
Spun off from https://github.com/grpc/grpc/pull/33695
- Switched from yapf to black
- Reconfigure isort for black
- Resolve black/pylint idiosyncrasies
Note: I used `--experimental-string-processing` because black was
producing "implicit string concatenation", similar to what described
here: https://github.com/psf/black/issues/1837. While currently this
feature is experimental, it will be enabled by default:
https://github.com/psf/black/issues/2188. After running black with the
new string processing so that the generated code merges these `"hello" "
world"` strings concatenations, then I removed
`--experimental-string-processing` for stability, and regenerated the
code again.
To the reviewer: don't even try to open "Files Changed" tab 😄 It's
better to review commit-by-commit, and ignore `run black and isort`.
The very non-trivial upgrade of third_party/protobuf to 22.x
This PR strives to be as small as possible and many changes that were
compatible with protobuf 21.x and didn't have to be merged atomically
with the upgrade were already merged.
Due to the complexity of the upgrade, this PR wasn't created
automatically by a tool, but manually. Subsequent upgraded of
third_party/protobuf with our OSS release script should work again once
this change is merged.
This is best reviewed commit-by-commit, I tried to group changes in
logical areas.
Notable changes:
- the upgrade of third_party/protobuf submodule, the bazel protobuf
dependency itself
- upgrade of UPB dependency to 22.x (in the past, we used to always
upgrade upb to "main", but upb now has release branch as well). UPB
needs to be upgraded atomically with protobuf since there's a de-facto
circular dependency (new protobuf depends on new upb, which depends on
new protobuf for codegen).
- some protobuf and upb bazel rules are now aliases, so `
extract_metadata_from_bazel_xml.py` and `gen_upb_api_from_bazel_xml.py`
had to be modified to be able to follow aliases and reach the actual
aliased targets.
- some protobuf public headers were renamed, so especially
`src/compiler` needed to be updated to use the new headers.
- protobuf and upb now both depend on utf8_range project, so since we
bundle upb with grpc in some languages, we now have to bundle utf8_range
as well (hence changes in build for python, PHP, objC, cmake etc).
- protoc now depends on absl and utf8_range (previously protobuf had
absl dependency, but not for the codegen part), so python's
make_grpcio_tools.py required partial rewrite to be able to handle those
dependencies in the grpcio_tools build.
- many updates and fixes required for C++ distribtests (currently they
all pass, but we'll probably need to follow up, make protobuf's and
grpc's handling of dependencies more aligned and revisit the
distribtests)
- bunch of other changes mostly due to overhaul of protobuf's and upb's
internal build layout.
TODOs:
- [DONE] make sure IWYU and clang_tidy_code pass
- create a list of followups (e.g. work to reenable the few tests I had
to disable and to remove workaround I had to use)
- [DONE in cl/523706129] figure out problem(s) with internal import
---------
Co-authored-by: Craig Tiller <ctiller@google.com>
Protobuf's HEAD has changed the layout of bazel targets we use for
getting the list of source files for our python grpcio.tools build.
With this patch, make_grpcio_tools.py finishes successfully and gives
reasonable results.
- sort source files to ensure stable ordering
- generate one source file per line
together this should produce diffs that are much more readable by humans
when sources get added/removed to/from protobuf (and
make_grpcio_tools.py is used to regenerate).
* Re-add compiler protos to grpcio-tools
* Attempt to break
* Actually fail if protobufs don't compile. Who wrote this thing?
* Add an actual test
* yapf
* Whoops. Thought I'd reverted you
* Please the formatter gods
* Update third_party/protobuf
* run tools/distrib/python/make_grpcio_tools.py
* regenerate protos for ruby, php
* update build_handwritten.yaml
* regenerate projects
* Build - Use :well_known_type_protos instead of :well_known_protos
* Fix target
* Update upb
* Update Python for Protobuf 4.21 (#140)
* Update protobuf dependency on grpcio-tools
* Off by one
* Drop python 3.6 support
* Try upgrading pip
* And in the other script
* Try to figure out if we're compatible with abi3
* See what we've already got installed
* Update the requirements.txt file I didn't know existed
* And here too
* See what's installed
* Let's try that again
* Remove
* Try to confirm version
* Let me see the generated code
* Fix non-Bazel test runner
* Work for all test directories
* Regenerate example protos
* Clean up
* Generate .pyi files
* Fix type checking and linting
* Exclude pyi files from isort
* Upgrade to 3.21.4
* Update iwyu to get around messy protobuf IWYU rules
Co-authored-by: Richard Belleville <gnossen@gmail.com>
* Run 2to3 on tools directory
* Delete github_stats_tracking
* Re-run 2to3
* Remove unused script
* Remove unused script
* Remove unused line count utility
* Yapf. Isort
* Remove accidentally included file
* Migrate tools/distrib directory to python 3
* Remove unnecessary shebang
* Restore line_count directory
* Immediately convert subprocess.check_output output to string
* Take care of Python 2 shebangs
* Invoke scripts using a Python 3 interpreter
* Yapf. Isort
* Try installing Python 3 first
* See if we have any Python 3 versions installed
* Add Python 3.7 to Windows path
* Try adding a symlink
* Try to symlink differently
* Install six for Python 3
* Run run_interop_tests with python 3
* Try installing six in python3.7 explicitly
* Revert "Try installing six in python3.7 explicitly"
This reverts commit 2cf60d72f3.
* And debug some more
* Fix issue with jobset.py
* Add debug for CI failure
* Revert microbenchmark changes
The `make_grpcio_tools.py` rube-goldberg machinery relies on
`bazel` to extract the list of files required to compile the
`grpcio-tools` package that are provided by the `protobuf`
submodule. In order to ensure that list is up to date,
`check_grpcio_tools.py` sanity check does the same `bazel`
query, and checks the full contents against the already
existing list in the repository. This has the downside of
requiring `bazel` to run that particular check at sanity test
time, and flakiness has been seen there.
This commit changes the code generation process to include
the git hash of the submodule at the time `make_grpcio_tools.py`
is invoked and stores it in the generated file and instead of
doing the whole process at test time, the `check_grpcio_tools.py`
sanity test simply checks the submodule version at test time
and verifies it against the version included in the file by
`make_grpcio_tools.py` thus removing the `bazel` dependency
at test time and increasing test robustness and speed.
This grossly simplifies the protoc invocation to:
python -m grpc.protoc.compiler --python_out=... --grpc_python_out=...
[...] --plugin=protoc-gen-python-grpc=grpc_python_protoc_plugin... [...]
Esun Kim
Richard Belleville
github-actions[bot]
AJ Heller
Sergii Tkachenko
Jan Tattermusch
Yash Tibrewal
Mehrdad Afshari
ncteisen
Ken Payson
Masood Malekghassemi