It has been decided that we will no longer try and split Grpc.Tools
NuGet package logic into two packages by adding logic to
Google.Protobuf.Tools. See
https://github.com/protocolbuffers/protobuf/issues/5957
This PR removes some of the TODOs and redundant files.
There is further simplification work that could be done to combine the
props and targets files that were already split. But this is the first
step in the tidying up the code.
This was done manually due to a problem with
`tools/distrib/python/make_grpcio_tools.py`. ~I fixed it in this PR
(depends on cl/547979185), so there is a fair chance this upgrade will
work normally for the next release.~ The fix may be problematic for
upgrading protobuf on older release branches, so the improvement will be
worked on separately. CC @jtattermusch
This also updates the UPB dep to the latest commit on the 23.x branch.
- Switched from yapf to black
- Reconfigure isort for black
- Resolve black/pylint idiosyncrasies
Note: I used `--experimental-string-processing` because black was
producing "implicit string concatenation", similar to what described
here: https://github.com/psf/black/issues/1837. While currently this
feature is experimental, it will be enabled by default:
https://github.com/psf/black/issues/2188. After running black with the
new string processing so that the generated code merges these `"hello" "
world"` strings concatenations, then I removed
`--experimental-string-processing` for stability, and regenerated the
code again.
To the reviewer: don't even try to open "Files Changed" tab 😄 It's
better to review commit-by-commit, and ignore `run black and isort`.
Reverts grpc/grpc#32636
```
src/compiler/csharp_generator_helpers.h:25:7: error: no member named 'compiler' in namespace ...
src/compiler/csharp_generator_helpers.h:25:25: error: no member named 'csharp' in namespace 'compiler' ...
```
Added `base_namespace` experimental option to `grpc_csharp_plugin` as
this has been requested several times by
people not using `Grpc.Tools` to generate their code - see
https://github.com/grpc/grpc/issues/28663
Notes:
- it should not be used with `Grpc.Tools`. That has a different way of
handling duplicate proto file names in different directories. Using this
option will break those builds. It can only be used on the `protoc`
command line.
- it uses common code with the `base_namespace` option for C# in
`protoc`, which unfortunately has a slightly different name mangling
algorithm for converting proto file names to C# camel case names. This
only affects files with punctation or numbers in the name. This should
not matter unless you are expecting specific file names
- See
https://protobuf.dev/reference/csharp/csharp-generated/#compiler_options
for an explanation of this option
The very non-trivial upgrade of third_party/protobuf to 22.x
This PR strives to be as small as possible and many changes that were
compatible with protobuf 21.x and didn't have to be merged atomically
with the upgrade were already merged.
Due to the complexity of the upgrade, this PR wasn't created
automatically by a tool, but manually. Subsequent upgraded of
third_party/protobuf with our OSS release script should work again once
this change is merged.
This is best reviewed commit-by-commit, I tried to group changes in
logical areas.
Notable changes:
- the upgrade of third_party/protobuf submodule, the bazel protobuf
dependency itself
- upgrade of UPB dependency to 22.x (in the past, we used to always
upgrade upb to "main", but upb now has release branch as well). UPB
needs to be upgraded atomically with protobuf since there's a de-facto
circular dependency (new protobuf depends on new upb, which depends on
new protobuf for codegen).
- some protobuf and upb bazel rules are now aliases, so `
extract_metadata_from_bazel_xml.py` and `gen_upb_api_from_bazel_xml.py`
had to be modified to be able to follow aliases and reach the actual
aliased targets.
- some protobuf public headers were renamed, so especially
`src/compiler` needed to be updated to use the new headers.
- protobuf and upb now both depend on utf8_range project, so since we
bundle upb with grpc in some languages, we now have to bundle utf8_range
as well (hence changes in build for python, PHP, objC, cmake etc).
- protoc now depends on absl and utf8_range (previously protobuf had
absl dependency, but not for the codegen part), so python's
make_grpcio_tools.py required partial rewrite to be able to handle those
dependencies in the grpcio_tools build.
- many updates and fixes required for C++ distribtests (currently they
all pass, but we'll probably need to follow up, make protobuf's and
grpc's handling of dependencies more aligned and revisit the
distribtests)
- bunch of other changes mostly due to overhaul of protobuf's and upb's
internal build layout.
TODOs:
- [DONE] make sure IWYU and clang_tidy_code pass
- create a list of followups (e.g. work to reenable the few tests I had
to disable and to remove workaround I had to use)
- [DONE in cl/523706129] figure out problem(s) with internal import
---------
Co-authored-by: Craig Tiller <ctiller@google.com>
Added documentation on how to use `Grpc.Tools` MSBuild integration with
*unsupported* architectures - i.e. third party built `protoc` and
`grpc_csharp_plugin`.
Fix incompatibilities identified when running adhoc runs on the new
custom win2019 image.
After merging this, it should be possible to switch to the new image
without breaking any tests.
- for most fixes I added a comment that explains why they're necessary.
- the new image won't have VS2015 installed, so I'm switching the protoc
artifact build to VS2017
This PR will need to be backported to older release branches to ensure
the windows tests continue working on those branches as well (IMHO I
haven't made any changes that would be difficult to backport and I tried
to keeps the diff as small as possible to avoid issues when
backporting).
After we switch to the new image (and all the windows tests are green),
we can incrementally move the builds that are still using VS2017 to
VS2019.
Further Grpc.Tools tests
- TestCharactersInName - proto files with dots and numbers in their
names
- TestExtraOptions - test setting AdditionalProtocArguments,
OutputOptions and GrpcOutputOptions
- TestGrpcServices - test setting GrpcServices to "none", "client",
"server" and "both"
- TestSetOutputDirs - test setting OutputDir and GrpcOutputDir
* Rewrite Grpc.Tools BUILD-INTEGRATION.md docs
* Minor edits to text
* various adjustments and typos for Grpc.Tools readme
* Updated with review suggestions
* Further refinements to the docs
* Add README for Grpc.Tools package
* Update BUILD-INTEGRATION.md
* Updated from review comments
Co-authored-by: Jan Tattermusch <jtattermusch@google.com>
* Inital Grpc.Tools unit tests
* refining the tests
* Get tests to run on Linux
* further grpc.tools tests changes
* changes for relative/absoulte paths
* Simplify running on just one framework
* Add Protobuf.MSBuild.dll for tests to use
* fix typo in comment
* fix copyright note
* Revert "Add Protobuf.MSBuild.dll for tests to use"
This reverts commit 542756cade.
* cleanup Directory.Build.props
* fix location of tasksAssembly
* cleanup in preparing paths
* add test-out to .gitignore
* many simplifications to MsBuildIntegrationTest
* update expected test results data
* many improvements to fakeprotoc.py
* yapf code
* fix copyright licenses
* reintroduce normalized paths
* minor fixes
* attempt simplify Directory.Build.targets
* newline at EOF
* simplify result comparison
* add a TODO
* further simplify MsBuildIntegrationTest test harness
* avoid internal symbol conflict by diambiguating test protos
* add TODO
* style fixes
* one more fixup
Co-authored-by: Jan Tattermusch <jtattermusch@google.com>
* Fix msbuild failing when '@' is present in path
There was ambiguity in references to Source metadata on some Items. These
are now qualified to remove the ambiguity.
* Revert "Fix msbuild failing when '@' is present in path"
This reverts commit bb5d5ddc61.
* Unescape Indentity when creating Source
* Fix spelling in comment
* issue 27502 - parse warning from libprotobuf
The warnings and errors logged by some of the pluggins such as the csharp
plugin are in a different format from other warnings etc logged by the
protobuf compiler (the plugins use GOOGLE_LOG to log message).
The parsing code in Grpc.Tools has been modified to correctly parse these
so that warnings are no longer handled as errors.
* Fix typos in comments
* Update third_party/protobuf
* run tools/distrib/python/make_grpcio_tools.py
* regenerate protos for ruby, php
* update build_handwritten.yaml
* regenerate projects
* Build - Use :well_known_type_protos instead of :well_known_protos
* Fix target
* Update upb
* Update Python for Protobuf 4.21 (#140)
* Update protobuf dependency on grpcio-tools
* Off by one
* Drop python 3.6 support
* Try upgrading pip
* And in the other script
* Try to figure out if we're compatible with abi3
* See what we've already got installed
* Update the requirements.txt file I didn't know existed
* And here too
* See what's installed
* Let's try that again
* Remove
* Try to confirm version
* Let me see the generated code
* Fix non-Bazel test runner
* Work for all test directories
* Regenerate example protos
* Clean up
* Generate .pyi files
* Fix type checking and linting
* Exclude pyi files from isort
* Upgrade to 3.21.4
* Update iwyu to get around messy protobuf IWYU rules
Co-authored-by: Richard Belleville <gnossen@gmail.com>
AJ Heller
tony
Jan Tattermusch
Sergii Tkachenko
Yijie Ma
Esun Kim
Yash Tibrewal
Stanley Cheung
Mark D. Roth
dependabot[bot]
Richard Belleville
apolcyn
Lidi Zheng
Jan Tattermusch
Mahdi K. Fard