Fix: https://github.com/grpc/grpc/issues/35085
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35325
PiperOrigin-RevId: 592635611
Enable OpenSSL 1.0.2 tests and add a container for 1.1.1 so that it is tested during portability testing as well.
Closes#35236
PiperOrigin-RevId: 590345568
Starting from Python 3.11, the pipes module produces this warning:
DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
Turns out that in this repo the pipes module is only used for the
"quote" function which is turn directly taken from the shlex module [1].
The shlex module is not deprecated as of today and is already used in
other places in this repo. The function shlex.quote has been around
since the ancient Python 3.3.
[1] https://github.com/python/cpython/blob/3.11/Lib/pipes.py#L64-L66
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#34941
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/34941 from lepistone:deprecate-python-pipes 233c54c135
PiperOrigin-RevId: 588883480
Reverts grpc/grpc#34667
The change was reverted because it failed to import to g3, after some
changes, now it's safe to reapply those changes.
Tested by importing this PR internally, it passed presubmit:
cl/573836270
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
We're not running any test at all from `run_test.py` because of the way
we filter test cases:
1d136fd05f/src/python/grpcio_tests/tests/_runner.py (L137)
* `testcase_filter` is read from a json file (like [this
one](https://github.com/grpc/grpc/blob/master/src/python/grpcio_tests/tests/tests.json))
and test name is similar to `unit._metadata_test.MetadataTest`.
* `case.id()` is loaded by `iterate_suite_cases` and will always have a
prefix of `tests`, an example of case id will be:
`tests.unit._metadata_test.MetadataTest`.
Because of the prefix, none of the test case will be matched thus we're
not running any of the tests.
This PR fixes the prefix issue and all the regressions comes from not
running tests using `run_test.py`.
#### Other Changes
* Added couple of `__init__.py` file since it's required to load tests.
* Added `py_status_code` to Aio rpc state.
* `code()` is expecting to return a python gRPC code but current
`status_code` is a Cython code.
* Added `libsqlite3-dev` to our dockers because it's required for
`coverage==7.2.0`.
* Renamed csds and admin test because test case file have to end with
`_test`:
1d136fd05f/src/python/grpcio_tests/tests/_loader.py (L26)
* Removed gevent test from `run_test.py` because Bazel gevent tests
should be good enough for us.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
The deleted code here was overriding the
[intended](866fc41067/tools/run_tests/run_tests.py (L62))
default test env of `GRPC_VERBOSITY=DEBUG`.
I'm just deleting it because it looks like`GRPC_TRACE=api` is not having
any affect anyways, since it relies on `GRPC_VERBOSITY=DEBUG` which it
happens to be unsetting.
* Remove `fetch_build_eggs` since they're deprecated and those deps will
be installed by `setuptools`.
* Fix indentation on `run_test` so we don't miss `native` test cases.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
~~NB: I haven't tested this at all and am hoping the CI will tell me
where I've (undoubtedly) messed something up.~~ Edit: looks like CI is
now clear!
BoringSSL's gas-compatible assembly files, like its C files, are now
wrapped with preprocessor ifdefs to capture which platforms each file
should be enabled on. This means that, provided the platform can process
.S files it all (i.e. not Windows), we no longer need to detect the
exact CPU architecture in the build.
Switch gRPC's build to take advantage of this. I've retained
BUILD_OVERRIDE_BORING_SSL_ASM_PLATFORM, on the off chance anyone is
using it to cross-compile between Windows and non-Windows, though I
doubt that works particularly well.
As part of this, restore assembly optimizations in a few places where
they were seemingly disabled for issues relating to this:
- https://github.com/grpc/grpc/pull/31747 had to disable the assembly,
because at the time assembly required the library be built differently
for each architecture and then stitched back together. This should now
work.
- tools/run_tests/run_tests.py disabled x86 assembly due to some issues
with CMAKE_SYSTEM_PROCESSOR in a Docker image. This too should now be
moot.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Add "bazelified" non-bazel tests. See tools/bazelify_tests/README.md for
the core idea.
- add a bunch of test targets that run under docker and execute tests
that correspond to `run_tests.py -l LANG ...`
- many more tests can be added in the future
- to enable running some of the C/C++ portability tests easily, added
support for `--cmake_extra_configure_args` in run_tests.py (the change
is fairly small).
Example passing build that shows how test results are structured:
https://source.cloud.google.com/results/invocations/21295351-a3e3-4be1-b6e9-aaf52195a044/targets
This PR:
- Fixes the xds-protos Python package, which was broken when the `udpa`
submodule was removed
- This required re-adding the protoc-gen-validate submodule
- Adds non-Bazel tests for xds-protos and all of its dependent packages
- Versions xds-protos the same way as the rest of the Python packages
- Fixes Python 3.11 support in `run_tests.py`, which is necessary for
the testing mentioned above
CC @sergiitk You won't be able to consume this in the interop tests
until it makes it into a release. I'm thinking I'll want to backport
this to the 1.57.x branch to make that happen faster.
CC @drfloob to inform him about the likely backport.
Adds experimental fork support to gRPC/Ruby
Works towards https://github.com/grpc/grpc/issues/8798 (see caveats for why this wasn't marked fixed yet)
Works towards https://github.com/grpc/grpc/issues/33578 (see caveats for why this wasn't marked fixed yet)
This leverages existing `pthread_atfork` based C-core support for
forking that python/php use, but there's a bit extra involved mainly
because gRPC/Ruby has additional background threads.
New tests under `src/ruby/end2end` show example usage.
Based on https://github.com/grpc/grpc/pull/33495
Caveats:
- Bidi streams are not yet supported (bidi streams spawn background
threads which are not yet fork safe)
- Servers not supported
- Only linux supported
- Switched from yapf to black
- Reconfigure isort for black
- Resolve black/pylint idiosyncrasies
Note: I used `--experimental-string-processing` because black was
producing "implicit string concatenation", similar to what described
here: https://github.com/psf/black/issues/1837. While currently this
feature is experimental, it will be enabled by default:
https://github.com/psf/black/issues/2188. After running black with the
new string processing so that the generated code merges these `"hello" "
world"` strings concatenations, then I removed
`--experimental-string-processing` for stability, and regenerated the
code again.
To the reviewer: don't even try to open "Files Changed" tab 😄 It's
better to review commit-by-commit, and ignore `run black and isort`.
Fix at-head tests (this is a missing piece of
https://github.com/grpc/grpc/pull/32905) with the following error;
```
/var/local/git/grpc/tools/run_tests/helper_scripts/build_python.sh: line 126: python3.8: command not found
```
Followup for https://github.com/grpc/grpc/pull/32649 (which disabled the
tests mentioned below).
Also sets correct path for tests build by ninja on windows, so that they
don't get skipped.
Once merged, I'll backport to 1.54.x and 1.53.x
The original issue with tests being skipped.
```
+ python3 workspace_c_windows_dbg_native/tools/run_tests/run_tests.py -t -j 8 -x run_tests/c_windows_dbg_native/sponge_log.xml --report_suite_name c_windows_dbg_native -l c -c dbg --iomgr_platform native --bq_result_table aggregate_results --measure_cpu_costs
2023-03-20 07:56:53,523 START: tools\run_tests\helper_scripts\build_cxx.bat
2023-03-20 08:04:51,388 PASSED: tools\run_tests\helper_scripts\build_cxx.bat [time=477.9sec, retries=0:0; cpu_cost=0.0; estimated=1.0]
2023-03-20 08:04:52,434 detected port server running version 21
2023-03-20 08:04:52,672 my port server is version 21
2023-03-20 08:04:52,703 SUCCESS: All tests passed
WARNING: binary not found, skipping cmake/build/Debug/bad_server_response_test.exe
WARNING: binary not found, skipping cmake/build/Debug/connection_refused_test.exe
WARNING: binary not found, skipping cmake/build/Debug/goaway_server_test.exe
WARNING: binary not found, skipping cmake/build/Debug/invalid_call_argument_test.exe
WARNING: binary not found, skipping cmake/build/Debug/multiple_server_queues_test.exe
WARNING: binary not found, skipping cmake/build/Debug/no_server_test.exe
WARNING: binary not found, skipping cmake/build/Debug/pollset_windows_starvation_test.exe
WARNING: binary not found, skipping cmake/build/Debug/public_headers_must_be_c89.exe
```
(part of removing support for VS2017)
Also see https://github.com/grpc/grpc/pull/32649
Also see https://github.com/grpc/grpc/pull/32615
The switch to grpc-win2019 windows workers has already happened:
(cl/517400022).
Once this PR lands, I'll backport to 1.53.x branch as well (since that
release removes the VS2017 support).
Fix incompatibilities identified when running adhoc runs on the new
custom win2019 image.
After merging this, it should be possible to switch to the new image
without breaking any tests.
- for most fixes I added a comment that explains why they're necessary.
- the new image won't have VS2015 installed, so I'm switching the protoc
artifact build to VS2017
This PR will need to be backported to older release branches to ensure
the windows tests continue working on those branches as well (IMHO I
haven't made any changes that would be difficult to backport and I tried
to keeps the diff as small as possible to avoid issues when
backporting).
After we switch to the new image (and all the windows tests are green),
we can incrementally move the builds that are still using VS2017 to
VS2019.
This test has been occasionally failing on CI with "Bus Error" crashes
while requiring the grpc shared library.
These crashes have been unreproducible locally. Let's continue debugging
(b/266212253) but skip this on CI.
Make remaining objC jobs compatible with kokoro monterey workers and
prepare for boringssl upgrade.
The changes here are taken from https://github.com/grpc/grpc/pull/32357,
but they should be merged in a separate PR
(we need the changes to be able to upgrade to monterey anyway and
there's no reason to make the boringssl upgrade PR more complicated by
bundling more fixes into it).
I've checked that the grpc_basictests_objc_examples and
grpc_ios_binary_size are green if switched to monterey.
Unfortunately it's hard to make grpc_basictests_objc_examples pass on
both monterey and mojave, so I suggest merging this PR at the same time
as CL to upgrade the kokoro jobs to monterey.
- that way both PR and continuous runs will remain green
- older branches would need a backport anyway
---------
Co-authored-by: Hannah Shi <hannahshisfb@gmail.com>
Cleanup and remove ios cpp test cronet
To test manually:
./tools/bazel test //src/objective-c/tests:CppCronetTests
@sampajano
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
* Update third_party/protobuf
* run tools/distrib/python/make_grpcio_tools.py
* regenerate protos for ruby, php
* update build_handwritten.yaml
* regenerate projects
* Build - Use :well_known_type_protos instead of :well_known_protos
* Fix target
* Update upb
* Update Python for Protobuf 4.21 (#140)
* Update protobuf dependency on grpcio-tools
* Off by one
* Drop python 3.6 support
* Try upgrading pip
* And in the other script
* Try to figure out if we're compatible with abi3
* See what we've already got installed
* Update the requirements.txt file I didn't know existed
* And here too
* See what's installed
* Let's try that again
* Remove
* Try to confirm version
* Let me see the generated code
* Fix non-Bazel test runner
* Work for all test directories
* Regenerate example protos
* Clean up
* Generate .pyi files
* Fix type checking and linting
* Exclude pyi files from isort
* Upgrade to 3.21.4
* Update iwyu to get around messy protobuf IWYU rules
Co-authored-by: Richard Belleville <gnossen@gmail.com>
* stop running objc plugin tests with run_tests.py
* stop running bazel examples as part of run_tests.py
* stop running UnitTests and MacTests as part part of run_tests.py
Esun Kim
Xuan Wang
Gregory Cooke
Hannah Shi
Leonardo Pistone
Craig Tiller
Yijie Ma
apolcyn
David Benjamin
Jan Tattermusch
Richard Belleville
Sergii Tkachenko
Cheng-Yu Chung
Yash Tibrewal
Denny C. Dai