(I had to first solve a problem with "dubious file ownership" error that
was happening inside the grpc_repo_archive action when using the
concurrent wrapper).
Example:
```
tools/docker_runners/examples/concurrent_bazel.sh --bazelrc=tools/remote_build/linux.bazelrc test --genrule_strategy=remote,local --workspace_status_command=tools/bazelify_tests/workspace_status_cmd.sh //tools/bazelify_tests/test:runtests_csharp_linux_dbg
```
Bumping gcc 7 to 8 to workaround the ongoing gcc segfault problem when
building Protobuf C++. Currently Foundational C++ requires gcc 7 so this
is a temporary measure to make the test green. We need to either make a
decision to change the minimum version of gcc in the Foundational C++ or
find a way to support gcc 7 without gcc segfault soon.
1. Switch to CMake 1.18.
2. Make ergonomic change to push_testing_images.sh to allow building
just a single image.
3. Update packages to reduce a number of vulnerabilities reported.
- Upgrade bazel
- Reduce the number of places where bazel version needs to be upgraded
in future.
- also make sure the list of bazel versions to test by bazelified tests
is loaded from supported_versions.txt (it was hardcoded before).
- ~~Try upgrading windows RBE build to bazel 6.3.2 as well.~~
The core idea:
- the source of truth for supported bazel versions is in
`bazel/supported_versions.txt`
- the first version listed in `bazel/supported_versions.txt` is
considered to be the "primary" bazel version and is going to be used in
most places thoroughout the repo.
- use templates to include the primary bazel version in testing
dockerfiles and in a newly introduced `.bazelversion` files (which gets
loaded by our existing `tools/bazel` wrapper).
~~Supersedes https://github.com/grpc/grpc/pull/33880~~
Based on https://github.com/grpc/grpc/pull/34033
Bunch of cleanup and rebuilding many docker images from scratch
- consolidate the workaround for "dubious ownership" issue reported by
git. Other team members have run into this recently and used similar but
not identical workarounds so some cleanup is due.
- rebuilding many images increases the chance that we fix the "dubious
ownership" git issue early on rather than later on in the one-at-a-time
fashion in the future (and the former will prevent many teammembers from
wasting time on this weird issue).
- Newer version of ccache is needed for some portability tests to be
able to benefit from caching (e.g. the GCC 12 portability test to get
benefits of local disk caching) - this is a prerequisite for reenabling
the bazelified gcc12 portability test.
- upgrade node interop images to debian:11 (since debian jessie is long
past EOL).
Reintroduce https://github.com/grpc/grpc/pull/33959.
I added a fix for the python arm64 build (which is the reason why the
change has been reverted earlier).
This is so that the newly added dns_test (which uses twisted-based
server) within the posix_event_engine_test can run on those platforms.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
This PR:
- Fixes the xds-protos Python package, which was broken when the `udpa`
submodule was removed
- This required re-adding the protoc-gen-validate submodule
- Adds non-Bazel tests for xds-protos and all of its dependent packages
- Versions xds-protos the same way as the rest of the Python packages
- Fixes Python 3.11 support in `run_tests.py`, which is necessary for
the testing mentioned above
CC @sergiitk You won't be able to consume this in the interop tests
until it makes it into a release. I'm thinking I'll want to backport
this to the 1.57.x branch to make that happen faster.
CC @drfloob to inform him about the likely backport.
It is not uncommon for Android tests to fail due to an NDK download
failure. To address this issue, the Android Docker image has been
updated to include NDK and CMake. This will help to avoid download
problem.
I verified this works as intended to see the invocation log not having
installation log for NDK and CMake such as
```
Checking the license for package NDK (Side by side) 25.1.8937393 in /opt/android-sdk/licenses
License for package NDK (Side by side) 25.1.8937393 accepted.
Preparing "Install NDK (Side by side) 25.1.8937393 (revision: 25.1.8937393)".
"Install NDK (Side by side) 25.1.8937393 (revision: 25.1.8937393)" ready.
Installing NDK (Side by side) 25.1.8937393 in /opt/android-sdk/ndk/25.1.8937393
"Install NDK (Side by side) 25.1.8937393 (revision: 25.1.8937393)" complete.
"Install NDK (Side by side) 25.1.8937393 (revision: 25.1.8937393)" finished.
Checking the license for package CMake 3.10.2.4988404 in /opt/android-sdk/licenses
License for package CMake 3.10.2.4988404 accepted.
Preparing "Install CMake 3.10.2.4988404 (revision: 3.10.2)".
"Install CMake 3.10.2.4988404 (revision: 3.10.2)" ready.
Installing CMake 3.10.2.4988404 in /opt/android-sdk/cmake/3.10.2.4988404
"Install CMake 3.10.2.4988404 (revision: 3.10.2)" complete.
"Install CMake 3.10.2.4988404 (revision: 3.10.2)" finished.
```
- Add a new docker image "rbe_ubuntu2004" that is built in a way that's
analogous to how our other testing docker images are built (this gives
us control over what exactly is contained in the docker image and
ability to fine-tune our RBE configuration)
- Switch RBE on linux to the new image (which gives us ubuntu20.04-based
builds)
For some reason, RBE seems to have trouble pulling the docker image from
Google Artifact Registry (GAR), which is where our public testing images
normally live, so for now, I used a workaround and I upload a copy of
the rbe_ubuntu2004 docker image to GCR as well, and that makes RBE works
just fine (see comment in the `renerate_linux_rbe_configs.sh` script).
More followup items (config cleanup, getting local sanitizer builds
working etc.) are in go/grpc-rbe-tech-debt-2023
Oops I missed important changes from
https://github.com/grpc/grpc/pull/32712. And it turned out that there
are two problems that I couldn't fix at this point.
- Windows Bazel RBE Linker Error: This may be caused by how new Bazel 6
invokes build tools chain but it's not clear. I put workaround to use
Bazel 5 by using `OVERRIDE_BAZEL_VERSION=5.4.1`
- Rule `rules_pods` to fetch CronetFramework from CocoaPod has
incompatibility with sort of built-in apple toolchain.
(https://github.com/bazel-xcode/PodToBUILD/issues/232): I couldn't find
a workaround to fix this so I ended up disabling all tests depending
this target.
Fix `python_alpine` test failure with
```
fatal: detected dubious ownership in repository at '/var/local/jenkins/grpc'
To add an exception for this directory, call:
git config --global --add safe.directory /var/local/jenkins/grpc
```
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
This is to get the latest version of clang 15 (15.0.7) for our docker
images based on that. By doing so, I had to address this new git
security enforcement so I added a new file to tame it. In a nutshell,
this PR is about polishing docker images based on clang 15.
* remove gcc-multilib from base image
* remove useless "apt-get --allow-releaseinfo-change update" from docker images
* get rid of no-longer-useful python_debian11_base.include
* regenerate dockerfiles
* nit in python interop dockerfile
* address TODO in prepare_ccache_symlinks_rc
* enable ccache for C# and ruby distribtest jobs
* use debian11 instead of bullseye consistently
* upgrade ruby docker images to debian 11
* upgrade C# docker images to debian11
* update sanity dockerimage to debian11
* upgrade cxx interop to debian11
* add ccache to python interop images
* enable use of ccache for interop tests
* adjust run_tests.py to new docker images
* add ccache to rake-compiler-dock docker images
* improve prepare_ccache_symlinks_rc
* enable use of ccache in rake-compiler-dock docker containers
* add ccache support for python_manylinux2014_aarch64
* add ccache support for python_linux_armv7
* deduplicate python3.9 install
* ccache for crosscompiled darwin gems is broken
* fix bash -l resetting of PATH in grpc_artifact_python_linux_armv7
* try speeding up linux portability test
* remove gcc8.3 portability test
* c-ares is the default resolver on linux
* dont set parallel run_tests jobs too high
* original parallelism might lead to better results
Jan Tattermusch
Eugene Ostroukhov
Esun Kim
Xuan Wang
Yijie Ma
Richard Belleville
Richard Belleville
apolcyn
Lidi Zheng
Jan Tattermusch