This PR adds templating for Python versions and updates the maximum supported Python version to 3.13. The following major changes related to templating are added:
- Minimum supported Python version and list of supported versions in `setup.py` are fetched using new template generated files called `python_version.py`
- Dockerfiles for the different Python Linux builds are now template generated.
- The "Supported Python Versions" section from READMEs of ancillary and main packages have been removed
Note: All the `python_version.py` files and Linux build `Dockerfiles` except `tools/dockerfile/grpc_artifact_python_linux_armv7/Dockerfile` in the PR are generated from the respective templates.
Further non-templated additions to add support for Python 3.13:
- install scripts and artifacts for windows, macos and linux are added manually. Later, these can be templated as well.
- updated cython bounds to 3.x
- updated twine version to solve [cgi module import error](https://github.com/pypa/twine/issues/1046)
- the twine update introduces a dependency on cryptography>=2.0. But the cryptography package doesn't support 32-bit Linux images and hence `twine check` has been disabled for x86 manylinux and x86 musllinux artifacts.
Closes#37643
PiperOrigin-RevId: 678954495
Passing manual build: https://btx.cloud.google.com/invocations/9adb3cd8-628c-42e5-a7a3-c2ba1ab6127f
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37723
PiperOrigin-RevId: 677900305
3a138e4369 renamed a bunch .c files to .c.inc, this PR adds them to boringssl podspec file.
Closes#37690
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37690 from HannahShiSFB:fix-objc-build-with-boring-ssl-0-0-37 407ce1464e
PiperOrigin-RevId: 674042284
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37572
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37572 from yijiem:upgrade-clang-7 03f8cdc54e
PiperOrigin-RevId: 670692896
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37572
PiperOrigin-RevId: 670631284
gcc 7 and 8 docker images are based on Debian 10 with older cmake than gRPC needs. Since it's not easy to change the base Debian image for these docker images, let's install a newer cmake instead. This is a left-over from https://github.com/grpc/grpc/pull/37547.
Closes#37571
PiperOrigin-RevId: 668560102
This flag was being ignored in clang in Xcode 15 and below, but is now an error in Xcode 16 and above.
Closes#36904
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36904 from sergiocampama:master 2c9839376b
PiperOrigin-RevId: 644494816
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Similar to https://github.com/grpc/grpc/pull/36755, this PR updates the image used by Go for interop tests that run against master.
Closes#36917
PiperOrigin-RevId: 644024756
Use the latest golang base image for Go interop tests. Presently the latest image points to Go 1.22.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
## Commands run
To render the Dockerfile from its template
```sh
$ tools/buildgen/generate_projects.sh
```
To build and upload the base docker image.
```sh
$ tools/dockerfile/push_testing_images.sh
```
To build and upload the interop docker image for grpc-go 1.64.0
```sh
$ tools/interop_matrix/create_matrix_images.py --git_checkout --release=v1.64.0 --upload_images --language go
```
To verify the image is uploaded
```sh
$ gcloud container images list-tags gcr.io/grpc-testing/grpc_interop_go1.x
DIGEST TAGS TIMESTAMP
b36be4961ec4 infrastructure-public-image-v1.64.0,v1.64.0 2024-06-03T22:26:48
```
Verify the backward compatability tests pass
```sh
$ export docker_image=gcr.io/grpc-testing/grpc_interop_go1.x:v1.64.0
$ tools/interop_matrix/testcases/go__master
Testing gcr.io/grpc-testing/grpc_interop_go1.x:v1.64.0
```
Fixes: https://github.com/grpc/grpc/issues/36060Closes#36755
PiperOrigin-RevId: 641272572
All clang docker images here are based on Debian 11 (Bullseye) but now we're moving to Debian 12 (Bookworm) primarily to use the recent versions of Cmake >= 3.20 to use CXX_STANDARD=23 for some tests ([doc](https://cmake.org/cmake/help/latest/prop_tgt/CXX_STANDARD.html))
Closes#36810
PiperOrigin-RevId: 640688681
Implement Python CSM observability.
Design: [go/grpc-python-opentelemetry](http://goto.google.com/grpc-python-opentelemetry)
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36557
PiperOrigin-RevId: 639073741
Public Changes -
* Add a pkgconfig installer for `grpcpp_otel_plugin`
Example Changes -
* Add example of how to use the pkgconfig for `grpcpp_otel_plugin` with the existing OpenTelemetry example.
* Add another OpenTelemetry example that uses OTel's OStream exporter. This makes it easier to test the pkgconfig file for `grpcpp_otel_plugin` since the OStream exporter does not require any additional dependencies, as opposed to the Prometheus exporter.
Test changes -
* Modify `run_distrib_test_cmake_pkgconfig.sh` test to install opentelemetry and build the example with the OStream exporter.
Closes#36686
PiperOrigin-RevId: 636965475
GCC 7 was disabled due to its gcc internal error while building protobuf C++. But we still need to support gcc7 and there appear to be issues causing gcc 7 to not build grpc so let's reintroduce gcc 7 test.
Closes#36565
PiperOrigin-RevId: 631947203
This should be able to unblock using grpc on vision os, will add tests later
Closes#36476
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36476 from HannahShiSFB:support-visionos edbde32d23
PiperOrigin-RevId: 630173448
Updated `push_testing_images.sh` to add an additional tag, `infrastructure-public-image-HASH` to a new image to be excluded from the GCP Vulnerability scanner.
To reviewer:
- Only `tools/dockerfile/push_testing_images.sh` has an interesting change (adding a new tag)
- Clang 6 and 17 don't have an actual change other than clean-up which is to test if the script is working.
Closes#36341
PiperOrigin-RevId: 623944712
PDB files are consuming a significant amount of storage space on our Windows test machines. With each PDB file being around 200MB and every target having its own corresponding file, the disk usage adds up quickly.
Fortunately, we can address this issue without sacrificing debugging functionality. [Shrink my Program Database (PDB) file](https://devblogs.microsoft.com/cppblog/shrink-my-program-database-pdb-file/) outlines several techniques to reduce PDB file size. We can implement these methods to achieve our goal. (From my experiment, the compression option only can reduce a 200MB PDB file by 60%)
This change only picks options which don't reduce debuggability.
Closes#36259
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36259 from veblush:pdbdiet 5781a789b6
PiperOrigin-RevId: 622232176
Protobuf for Ruby bumped the major version to 4. [package](https://rubygems.org/gems/google-protobuf) so the Ruby gemspec should be updated to have 4 instead of 3.
But gRPC's other dependencies don't work with Protobuf 4.x so I fixed the version of google-protobuf for gRPC Ruby to 3.25 which was the last version gRPC used.
Closes#36215
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36215 from veblush:ruby-protobuf 171a6d63bc
PiperOrigin-RevId: 620987535
This commit does the following 3 things for Android bazel rules, which depends on each other
1. Update NDK to 26.2.11394342 (and a few other Android things)
2. Use rules_android_ndk instead of builtin `native.android_ndk_repository`
* `third_party/android/android_configure.bzl` was a workaround for users who don't set `$ANDROID_NDK_HOME` env var. Now with rules_android_ndk, we can declare repo without registering the toolchain. Instead users who need NDK toolchain should use `--extra_toolchains` to manually register the toolchain, as shown in README.md.
3. Migrate to platforms. See https://bazel.build/concepts/platforms
* Currently we declares android platforms that is needed for binder transport APK. Later gRPC repo can gradually migrate to platforms for other platforms.
* The value of `crosstool_top` will still be `//external:android/crosstool` (which is the default value), so existings android `config_settings` will still work. We should migrate them to match with `@platforms//os:android` constraint later.
The platforms migration needs Bazel 7.0 to work so we also override bazel version in testing scripts.
Closes#36116
PiperOrigin-RevId: 617244655
The stubs generated by grpcio_tools should always be used with [the same or higher version of grpcio](https://github.com/grpc/grpc/blob/master/tools/distrib/python/grpcio_tools/setup.py#L313), this change will add a run time check for this requirement inside the generated stubs and therefor enforce this requirement.
Please note for now we're just printing a warning for incorrect usage, we'll **change it to an error** soon.
Example warning message:
```
/usr/local/google/home/xuanwn/workspace/misc/grpc/examples/python/helloworld/helloworld_pb2_grpc.py:21: RuntimeWarning: The grpc package installed is at version 1.60.1, but the generated code in helloworld_pb2_grpc.py depends on grpcio>=1.63.0.dev0. Please upgrade your grpc module to grpcio>=1.63.0.dev0 or downgrade your generated code using grpcio-tools<=1.60.1. This warning will become an error in 1.64.0, scheduled for release on May 14,2024.
```
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35906
PiperOrigin-RevId: 615659471
Earlier, the tests just had bazel support. With CMake support added in #36063, we can also add CI CMake support for the tests. A major benefit of this is that we also get coverage for the various platforms that we test from our portability test suite.
Closes#36087
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36087 from yashykt:OTelCISupport b28fbe02e5
PiperOrigin-RevId: 615543685
Refactored android-ndk docker image to have
- Replaced a full Android SDK with a Android command line tool (generally good for docker usage) and bumped its version to the latest among ones that support JDK 11
- Upgraded Debian from 10 to 11
- Upgraded Java SDK 8 to 11
- Upgraded Google CLI to the latest
Closes#36091
PiperOrigin-RevId: 615088973
Change was created by the release automation script. See go/grpc-release.
Additional Changes:
* Boring SSL started to [Require SSE2 when targetting 32-bit x86](56d3ad9d23), thus added `-msse2` to fix some build failures.
Closes#36089
PiperOrigin-RevId: 614822548
gRPC swift package depends on [boringSSL-SwiftPM](https://github.com/firebase/boringSSL-SwiftPM), which hasn't been updated since more than two years ago (https://github.com/firebase/boringSSL-SwiftPM/pull/8).
Updating it is not trivial as the `Package.swift` file is maintained manually, a recent [upgrade to swift 5.5](https://github.com/firebase/boringSSL-SwiftPM/pull/16) took good amount of time only to repeatedly exclude one (sometimes two) non-code files and retry build to get new errors, until the build succeeds.
This PR uses the gRPC build system to generate the swift package file from the same boringssl version in third_party.
Will use the output to update the boringSSL-SwiftPM repo after this PR is merged.
Tested with
```
cp BoringSSL-Package.swift Package.swift
swift build --target openssl_grpc
swift test
```
Closes#36062
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36062 from HannahShiSFB:add-boringssl-swift-package ac3e525bf2
PiperOrigin-RevId: 614088802
Changes -
* Add CMake build support to `grpcpp_otel_plugin`. Currently, we are only supporting the `find_package CONFIG` method for depending on `opentelemetry-cpp`.
* Since, `grpcpp_otel_plugin` is an extension of gRPC, it will not be built by default. To enable building of this target, a new CMake option `gRPC_BUILD_GRPCPP_OTEL_PLUGIN` is being added.
* Also add `CMakeLists.txt` to the otel example.
The `otel_plugin_test` can also be built through cmake but, for the CI to work, there are some additional changes. Those will be made in an upcoming PR.
Closes#36063
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36063 from yashykt:OTelCMakeSupport 3bc783823b
PiperOrigin-RevId: 613734473
- PHP 7.4 and 8.0 have been EOL https://www.php.net/supported-versions.php
- See go/drop-php-8.0 for more details
Most of the changes came from `generate_projects.sh` after updating the `tools/buildgen/plugins/expand_version.py` file. The change to `tools/internal_ci/helper_scripts/prepare_build_macos_rc` is manual.
Closes#35964
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35964 from stanley-cheung:php-update-to-8_1 0c24c4eac2
PiperOrigin-RevId: 609495798
This PR is copied from #34276, since I did not have permissions to add commits to it. That PR has been verified to work (see the top-level description). This PR just makes the gRPC tests pass (e.g. adding includes, clang formatting).
Closes#34874
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/34874 from matthewstevenson88:pull_34276 d5fb73e5b2
PiperOrigin-RevId: 609107146
- Dropped Ruby 2.7 test (Ruby 2.7 hit EOL 1 year ago)
- Upgraded the base image of Ruby docker images from Debian:10 to Debian:11 (Debian:10 hit EOL)
Closes#35911
PiperOrigin-RevId: 607088383
Sreenithi Sridharan
Arjan Singh Bal
Hannah Shi
apolcyn
Yijie Ma
Esun Kim
Eugene Ostroukhov
Nathan Baulch
AJ Heller
Sergio Campamá
Xuan Wang
Yash Tibrewal
Mark D. Roth
Craig Tiller
Ming-Chuan
Stanley Cheung
Matthew Stevenson