The very non-trivial upgrade of third_party/protobuf to 22.x
This PR strives to be as small as possible and many changes that were
compatible with protobuf 21.x and didn't have to be merged atomically
with the upgrade were already merged.
Due to the complexity of the upgrade, this PR wasn't created
automatically by a tool, but manually. Subsequent upgraded of
third_party/protobuf with our OSS release script should work again once
this change is merged.
This is best reviewed commit-by-commit, I tried to group changes in
logical areas.
Notable changes:
- the upgrade of third_party/protobuf submodule, the bazel protobuf
dependency itself
- upgrade of UPB dependency to 22.x (in the past, we used to always
upgrade upb to "main", but upb now has release branch as well). UPB
needs to be upgraded atomically with protobuf since there's a de-facto
circular dependency (new protobuf depends on new upb, which depends on
new protobuf for codegen).
- some protobuf and upb bazel rules are now aliases, so `
extract_metadata_from_bazel_xml.py` and `gen_upb_api_from_bazel_xml.py`
had to be modified to be able to follow aliases and reach the actual
aliased targets.
- some protobuf public headers were renamed, so especially
`src/compiler` needed to be updated to use the new headers.
- protobuf and upb now both depend on utf8_range project, so since we
bundle upb with grpc in some languages, we now have to bundle utf8_range
as well (hence changes in build for python, PHP, objC, cmake etc).
- protoc now depends on absl and utf8_range (previously protobuf had
absl dependency, but not for the codegen part), so python's
make_grpcio_tools.py required partial rewrite to be able to handle those
dependencies in the grpcio_tools build.
- many updates and fixes required for C++ distribtests (currently they
all pass, but we'll probably need to follow up, make protobuf's and
grpc's handling of dependencies more aligned and revisit the
distribtests)
- bunch of other changes mostly due to overhaul of protobuf's and upb's
internal build layout.
TODOs:
- [DONE] make sure IWYU and clang_tidy_code pass
- create a list of followups (e.g. work to reenable the few tests I had
to disable and to remove workaround I had to use)
- [DONE in cl/523706129] figure out problem(s) with internal import
---------
Co-authored-by: Craig Tiller <ctiller@google.com>
PHP7 build is failing, removing from CI while investigating the failure.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Our test clusters are in the broken/dirty state (except the URL map's
"basic" cluster, which isn't affected).
This PR switches to the newly created clusters to:
1. Get a data point on whether newly created clusters are affected
by the same issues.
2. Allow for descriptive work on the old clusters.
3. Hopefully, bring our tests back to green.
Bonus: more sensible cluster names.
* WIP. A seemingly properly failing test
* WIP. Pre-fork handlers now work
* Roughly working.
* Clean up
* Clean up more
* Add to CI
* Format
* Ugh. Remove swap file
* And another
* clean up
* Add copyright
* Format
* Remove another debug line
* Add stub forkable methods
* Remove use of 3.9+ function
* Remove unintentional double copyright
* drfloob review comments
* Only hold lock during Close once
* Create separate job for fork test
* Bump up gdb timeout
* Format
Apparently there's a difference between bash 3 and bash 4. OSX comes with bash 3 out-of-box, so for whoever wrote this logic it "worked on my machine".
Recently we had an issue with Service Account IAM policy getting too large.
RC is the cleanup script missing GCP SA account.
This PR adds adds missing `gcp_service_account` to the cleanup script.
* remove remains of grpc_csharp_ext artifact build completely
* add back part of C# distribtests
* redirect C# linux distribtests
* comment out some distribtests
* switch updateversion to Grpc.tools
* fix .proto
* fix distribtest complile
* temporarily comment out service in .proto
* cleanup
* improvements
* reenable mac distribtests
* fixes
This PR moves the PSM benchmark away from the benchmark-prod2
cluster to psm-benchmarks-performance cluster. This is to avoid
any unexpected change on the OSS benchmarks due to the tuning
we may perform for PSM benchmarks.
The psm-benchmarks-performance cluster has 2 system nodes,
1 driver-ci and 2 worker-ci nodes. The node type is the same as
the benchmark-prod2 cluster.
To capture the return status of the test in run_test the last command must be the call to the test itself.
This removes `set +x`, which makes the run_test always return success, and not propagate the test status.
I can't find it, but this exact error bit us before. Looks like it leaked to other scripts.
The good thing is if the test was executed, it's failure would still be picked up from the result xml.
However, if the test framework didn't start in the first place, the result will be false positive.
Example: https://source.cloud.google.com/results/invocations/98d3e679-ec8a-40bd-9f36-88179747b0d6/targets
```
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3: Error while finding module specification for 'tests.authz_test' (ModuleNotFoundError: No module named 'tests')
+ set +x
Failed test suites: 0
[ID: 3548168] Command finished after 625 secs, exit value: 0
```
- Enables pod log collection in all PSM interop jobs implemented in https://github.com/grpc/grpc/pull/30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab
Esun Kim
Sergii Tkachenko
Jan Tattermusch
Eugene Ostroukhov
Paulo Castello da Costa
sanjaypujare
Richard Belleville
Yash Tibrewal
apolcyn
Craig Tiller
Alisha Nanda
Wanlin Du