Updated `push_testing_images.sh` to add an additional tag, `infrastructure-public-image-HASH` to a new image to be excluded from the GCP Vulnerability scanner.
To reviewer:
- Only `tools/dockerfile/push_testing_images.sh` has an interesting change (adding a new tag)
- Clang 6 and 17 don't have an actual change other than clean-up which is to test if the script is working.
Closes#36341
PiperOrigin-RevId: 623944712
Typo introduced in: https://github.com/grpc/grpc/pull/34450
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
### NOTE
* We shouldn't merge this PR until GCP cloud functions drops support for Python 3.7 ([Currently scheduled for GCF](https://cloud.google.com/functions/docs/runtime-support#python))
As part of supporting Python 3.12, we're now officially drop support for Python 3.7.
This PR:
* Changed supported Python version from 3.7 to 3.8 in README.
* Replaced distribution test image from `debian:buster` to `debian:bullseye` since the default Python version in buster is 3.7.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#34450
PiperOrigin-RevId: 622899511
This commit does the following 3 things for Android bazel rules, which depends on each other
1. Update NDK to 26.2.11394342 (and a few other Android things)
2. Use rules_android_ndk instead of builtin `native.android_ndk_repository`
* `third_party/android/android_configure.bzl` was a workaround for users who don't set `$ANDROID_NDK_HOME` env var. Now with rules_android_ndk, we can declare repo without registering the toolchain. Instead users who need NDK toolchain should use `--extra_toolchains` to manually register the toolchain, as shown in README.md.
3. Migrate to platforms. See https://bazel.build/concepts/platforms
* Currently we declares android platforms that is needed for binder transport APK. Later gRPC repo can gradually migrate to platforms for other platforms.
* The value of `crosstool_top` will still be `//external:android/crosstool` (which is the default value), so existings android `config_settings` will still work. We should migrate them to match with `@platforms//os:android` constraint later.
The platforms migration needs Bazel 7.0 to work so we also override bazel version in testing scripts.
Closes#36116
PiperOrigin-RevId: 617244655
Refactored android-ndk docker image to have
- Replaced a full Android SDK with a Android command line tool (generally good for docker usage) and bumped its version to the latest among ones that support JDK 11
- Upgraded Debian from 10 to 11
- Upgraded Java SDK 8 to 11
- Upgraded Google CLI to the latest
Closes#36091
PiperOrigin-RevId: 615088973
Fix the docker image for C# distribution tests - csharp_ubuntu2204_x64
The package that installed the .NET 7.0 SDK is missing setting up links and thus the `dotnet` command cannot find the SDK. Setting this links has been added to the dockerfile.
**NOTE**: I'm unable to update `tools/dockerfile/distribtest/csharp_ubuntu2204_x64.current_version` as I don't have access to the artifact registry.
Closes#36019
PiperOrigin-RevId: 611590679
The current .NET SDK in these tests is EOL, and we've been seeing strange SSL errors transiently.
Update to a supported version and remove cruft in the docker image that was there to install the older version.
TODO: submitting first with the ubuntu 22.04 image test skipped, need to follow-up with a fix and unskip the test
Closes#35955
PiperOrigin-RevId: 610798209
- Dropped Ruby 2.7 test (Ruby 2.7 hit EOL 1 year ago)
- Upgraded the base image of Ruby docker images from Debian:10 to Debian:11 (Debian:10 hit EOL)
Closes#35911
PiperOrigin-RevId: 607088383
rebuilt `tools/dockerfile/distribtest/csharp_ubuntu2204_x64/Dockerfile` (without actually changing anything in the docker instructions) in an attempt to fix `The SSL connection could not be established, see inner exception.` errors happening in `dotnet restore` commands in distrib tests using this image.
Closes#35674
PiperOrigin-RevId: 601856906
We've been trying to upgrade Cmake to 3.13 or later as OSS policy bumped it. But we couldn't as Android has a weird linker error with Cmake 3.18 (you can see the error from https://github.com/grpc/grpc/pull/34331) This PR instead upgrades it to use 3.22 for Android test.
Closes#35572
PiperOrigin-RevId: 599317285
Updates ruby-compiler-dock to 1.4.0 which brings Ruby 3.3 final support per https://github.com/rake-compiler/rake-compiler-dock/releases/tag/1.4.0 and starts cross-compiling for ruby 3.3.
I can't find obviously where the test infrastructure configuration is to run the tests under Ruby 3.3, so might need pointers or an accompanying PR for the test infra. (I note #31991 from @apolcyn so perhaps currently they are not run against newer versions)
Fixes#35396
- Backport to `1.60` is desirable since currently the Ruby gems cannot be installed with Ruby 3.3 and have to be built from source.
Closes#35399
PiperOrigin-RevId: 599200628
It turned out that the previous change missed two things which this PR
has
- Fix function `_dockerized_genrule` not to have `timeout` and `flaky`
which Bazel doesn't have. (Bazel 7 may either drop these arguments or
become more strict about passing unrecognized ones)
- Disabled python bazel distribution tests with Bazel 7. This needs to
be addressed by https://github.com/grpc/grpc/issues/35391.
- Added Bazel 7 to the support bazel versions.
- Changed the default Bazel version to 7.
- Fixed Android Binder build issue.
Closes#35362
PiperOrigin-RevId: 592946781
Fix: https://github.com/grpc/grpc/issues/35085
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35325
PiperOrigin-RevId: 592635611
Enable OpenSSL 1.0.2 tests and add a container for 1.1.1 so that it is tested during portability testing as well.
Closes#35236
PiperOrigin-RevId: 590345568
The `bazel_build_with_strict_warnings_linux` job was taking nearly an
hour to complete. This change brings it down to ~20m by splitting up the
build targets into 7 separate RBE actions.
---------
Co-authored-by: drfloob <drfloob@users.noreply.github.com>
Reverts grpc/grpc#34667
The change was reverted because it failed to import to g3, after some
changes, now it's safe to reapply those changes.
Tested by importing this PR internally, it passed presubmit:
cl/573836270
Partial rollback of https://github.com/grpc/grpc/pull/34473 for RBE as
RBE needs Ubuntu 20.04 to get clang artifacts which Google publishes.
There aren't artifacts for Ubuntu 22.04 yet so there is no point of
upgrading RBE image to 22.04.
Introduce `grpc-native-debug` gems containing debug symbol packages that
complement the shared libraries shipped in pre-compiled binary gems.
src/ruby/nativedebug/README.md has details on usage.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
We're not running any test at all from `run_test.py` because of the way
we filter test cases:
1d136fd05f/src/python/grpcio_tests/tests/_runner.py (L137)
* `testcase_filter` is read from a json file (like [this
one](https://github.com/grpc/grpc/blob/master/src/python/grpcio_tests/tests/tests.json))
and test name is similar to `unit._metadata_test.MetadataTest`.
* `case.id()` is loaded by `iterate_suite_cases` and will always have a
prefix of `tests`, an example of case id will be:
`tests.unit._metadata_test.MetadataTest`.
Because of the prefix, none of the test case will be matched thus we're
not running any of the tests.
This PR fixes the prefix issue and all the regressions comes from not
running tests using `run_test.py`.
#### Other Changes
* Added couple of `__init__.py` file since it's required to load tests.
* Added `py_status_code` to Aio rpc state.
* `code()` is expecting to return a python gRPC code but current
`status_code` is a Cython code.
* Added `libsqlite3-dev` to our dockers because it's required for
`coverage==7.2.0`.
* Renamed csds and admin test because test case file have to end with
`_test`:
1d136fd05f/src/python/grpcio_tests/tests/_loader.py (L26)
* Removed gevent test from `run_test.py` because Bazel gevent tests
should be good enough for us.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
(I had to first solve a problem with "dubious file ownership" error that
was happening inside the grpc_repo_archive action when using the
concurrent wrapper).
Example:
```
tools/docker_runners/examples/concurrent_bazel.sh --bazelrc=tools/remote_build/linux.bazelrc test --genrule_strategy=remote,local --workspace_status_command=tools/bazelify_tests/workspace_status_cmd.sh //tools/bazelify_tests/test:runtests_csharp_linux_dbg
```
Foundation for being able to bazelify the build artifact -> build
package -> distribtest workflow tests.
Main ideas:
- "build artifact" and "build packages" will be represented by a custom
genrule (that runs the build on RBE under a docker container).
- since genrule doesn't support displaying logs for each target as a
separate "target log" (in the same way that bazel tests do), and we
generally want readable per-target logs for the bazelified test, a pair
of targets will be created for each "build artifact task":
- a genrule that actually performs the build, creates an archive with
artifacts and stores the exitcode and build log as rule outputs
- a corresponding "build_test" sh_test that simply looks at the result
of the genrule and presents the build log and build result a "target
log" for this test.
Bumping gcc 7 to 8 to workaround the ongoing gcc segfault problem when
building Protobuf C++. Currently Foundational C++ requires gcc 7 so this
is a temporary measure to make the test green. We need to either make a
decision to change the minimum version of gcc in the Foundational C++ or
find a way to support gcc 7 without gcc segfault soon.
Support Python 3.12.
### Testing
* Passed all Distribution Tests.
* Also tested locally by installing 3.12 artifact.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Bazelify tests from "linux/grpc_bazel_build" kokoro job by creating 3
bazelified tests - "build with strict warning", "build with no_xds=True"
and "build with no_xds=True negative test".
- also make the original "linux/grpc_bazel_build" kokoro job a no-op
(since bazelified tests now provide the same coverage).
- make C-core basictests use `--build_only` when running as bazelified
tests. This is because the volume of C core tests is expected to grow
very significantly after https://github.com/grpc/grpc/pull/34419 and
currently the non-bazelified counterpart of the tests (the presubmit
grpc_basictests_c_cpp_build_only job) is also "build only".
- make the linux presubmit job `grpc_basictests_c_cpp_build_only` a
noop, since the bazelified tests already give the same coverage on
presubmit.
1. Switch to CMake 1.18.
2. Make ergonomic change to push_testing_images.sh to allow building
just a single image.
3. Update packages to reduce a number of vulnerabilities reported.
Basically run each of the subtests (buildtest, distribtest_cpp,
distribtest_python) as a separate bazel target.
- currently the bazel distribtest are the slowest targets in
grpc_bazel_rbe_nonbazel
- the shards are basically independent tests anyway
- when split into multiple targets, they each get a separate target log
so it's easier debug issues since there isn't multiple bazel invocations
in each log.
- Upgrade bazel
- Reduce the number of places where bazel version needs to be upgraded
in future.
- also make sure the list of bazel versions to test by bazelified tests
is loaded from supported_versions.txt (it was hardcoded before).
- ~~Try upgrading windows RBE build to bazel 6.3.2 as well.~~
The core idea:
- the source of truth for supported bazel versions is in
`bazel/supported_versions.txt`
- the first version listed in `bazel/supported_versions.txt` is
considered to be the "primary" bazel version and is going to be used in
most places thoroughout the repo.
- use templates to include the primary bazel version in testing
dockerfiles and in a newly introduced `.bazelversion` files (which gets
loaded by our existing `tools/bazel` wrapper).
~~Supersedes https://github.com/grpc/grpc/pull/33880~~
Needs https://github.com/grpc/grpc/pull/34035 to be merged first. With
newer ccache in available in the gcc12 test image, the build is now
faster so we can reenable the test.
Esun Kim
Yash Tibrewal
Xuan Wang
Ming-Chuan
tony
apolcyn
Chad Wilson
AJ Heller
Gregory Cooke
Craig Tiller
Yijie Ma
Jan Tattermusch
Eugene Ostroukhov