Reintroduce https://github.com/grpc/grpc/pull/33959.
I added a fix for the python arm64 build (which is the reason why the
change has been reverted earlier).
This is so that the newly added dns_test (which uses twisted-based
server) within the posix_event_engine_test can run on those platforms.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
This PR:
- Fixes the xds-protos Python package, which was broken when the `udpa`
submodule was removed
- This required re-adding the protoc-gen-validate submodule
- Adds non-Bazel tests for xds-protos and all of its dependent packages
- Versions xds-protos the same way as the rest of the Python packages
- Fixes Python 3.11 support in `run_tests.py`, which is necessary for
the testing mentioned above
CC @sergiitk You won't be able to consume this in the interop tests
until it makes it into a release. I'm thinking I'll want to backport
this to the 1.57.x branch to make that happen faster.
CC @drfloob to inform him about the likely backport.
It is not uncommon for Android tests to fail due to an NDK download
failure. To address this issue, the Android Docker image has been
updated to include NDK and CMake. This will help to avoid download
problem.
I verified this works as intended to see the invocation log not having
installation log for NDK and CMake such as
```
Checking the license for package NDK (Side by side) 25.1.8937393 in /opt/android-sdk/licenses
License for package NDK (Side by side) 25.1.8937393 accepted.
Preparing "Install NDK (Side by side) 25.1.8937393 (revision: 25.1.8937393)".
"Install NDK (Side by side) 25.1.8937393 (revision: 25.1.8937393)" ready.
Installing NDK (Side by side) 25.1.8937393 in /opt/android-sdk/ndk/25.1.8937393
"Install NDK (Side by side) 25.1.8937393 (revision: 25.1.8937393)" complete.
"Install NDK (Side by side) 25.1.8937393 (revision: 25.1.8937393)" finished.
Checking the license for package CMake 3.10.2.4988404 in /opt/android-sdk/licenses
License for package CMake 3.10.2.4988404 accepted.
Preparing "Install CMake 3.10.2.4988404 (revision: 3.10.2)".
"Install CMake 3.10.2.4988404 (revision: 3.10.2)" ready.
Installing CMake 3.10.2.4988404 in /opt/android-sdk/cmake/3.10.2.4988404
"Install CMake 3.10.2.4988404 (revision: 3.10.2)" complete.
"Install CMake 3.10.2.4988404 (revision: 3.10.2)" finished.
```
- Add a new docker image "rbe_ubuntu2004" that is built in a way that's
analogous to how our other testing docker images are built (this gives
us control over what exactly is contained in the docker image and
ability to fine-tune our RBE configuration)
- Switch RBE on linux to the new image (which gives us ubuntu20.04-based
builds)
For some reason, RBE seems to have trouble pulling the docker image from
Google Artifact Registry (GAR), which is where our public testing images
normally live, so for now, I used a workaround and I upload a copy of
the rbe_ubuntu2004 docker image to GCR as well, and that makes RBE works
just fine (see comment in the `renerate_linux_rbe_configs.sh` script).
More followup items (config cleanup, getting local sanitizer builds
working etc.) are in go/grpc-rbe-tech-debt-2023
Oops I missed important changes from
https://github.com/grpc/grpc/pull/32712. And it turned out that there
are two problems that I couldn't fix at this point.
- Windows Bazel RBE Linker Error: This may be caused by how new Bazel 6
invokes build tools chain but it's not clear. I put workaround to use
Bazel 5 by using `OVERRIDE_BAZEL_VERSION=5.4.1`
- Rule `rules_pods` to fetch CronetFramework from CocoaPod has
incompatibility with sort of built-in apple toolchain.
(https://github.com/bazel-xcode/PodToBUILD/issues/232): I couldn't find
a workaround to fix this so I ended up disabling all tests depending
this target.
Fix `python_alpine` test failure with
```
fatal: detected dubious ownership in repository at '/var/local/jenkins/grpc'
To add an exception for this directory, call:
git config --global --add safe.directory /var/local/jenkins/grpc
```
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
This is to get the latest version of clang 15 (15.0.7) for our docker
images based on that. By doing so, I had to address this new git
security enforcement so I added a new file to tame it. In a nutshell,
this PR is about polishing docker images based on clang 15.
* remove gcc-multilib from base image
* remove useless "apt-get --allow-releaseinfo-change update" from docker images
* get rid of no-longer-useful python_debian11_base.include
* regenerate dockerfiles
* nit in python interop dockerfile
* address TODO in prepare_ccache_symlinks_rc
* enable ccache for C# and ruby distribtest jobs
* use debian11 instead of bullseye consistently
* upgrade ruby docker images to debian 11
* upgrade C# docker images to debian11
* update sanity dockerimage to debian11
* upgrade cxx interop to debian11
* add ccache to python interop images
* enable use of ccache for interop tests
* adjust run_tests.py to new docker images
* add ccache to rake-compiler-dock docker images
* improve prepare_ccache_symlinks_rc
* enable use of ccache in rake-compiler-dock docker containers
* add ccache support for python_manylinux2014_aarch64
* add ccache support for python_linux_armv7
* deduplicate python3.9 install
* ccache for crosscompiled darwin gems is broken
* fix bash -l resetting of PATH in grpc_artifact_python_linux_armv7
* try speeding up linux portability test
* remove gcc8.3 portability test
* c-ares is the default resolver on linux
* dont set parallel run_tests jobs too high
* original parallelism might lead to better results
* Upgrade libuv to v1.42.0
Also eliminates some cruft from the node implementation from 5 years ago
* sanitize.sh added a blank line?? ok then.
* pin to libuv v1.37.0
Jan Tattermusch
Xuan Wang
Esun Kim
Yijie Ma
Richard Belleville
Richard Belleville
apolcyn
Lidi Zheng
Jan Tattermusch
Ming-Chuan
AJ Heller