* Revert "Revert "Cleanup run docker machinery in run_tests.py, task_runner.py and elsewhere (#28704)" (#28741)"
This reverts commit 9a79d44e9b.
* fix python distribtest failure on fedora34
* attempt
* slim down build_docker_and_run_tests.sh
* HOST_GIT_ROOT is unused
* config and arch variable no need to propagate to run_tests.py
* unify unused LOCAL_GIT_ROOT to EXTERNAL_GIT_ROOT
* mounting gcloud config seems useless
* reorder docker args
* cleanup
* allow passing args to build_docker_and_run_tests.sh
* converge build_docker_and_run_tests.sh and build_and_run_docker.sh
* more convergence
* convergence of docker runners
* finalize convergence of run_docker scripts
* GRPC_TEST_REPORT_BASE_DIR might not exist
* adjust report copying
* make report_dir and output_dir readable
* alpine linux does not support cp -t
* add cmake support for ccache
* cleanup: use --env-file for docker run invocations
* make python build compatible with using ccache
* enable building using ccache in selected kokoro jobs
* print ccache stats and the end of run_tests.py
* support "move and respawn" for kokoro jobs
* delete_nonartifacts.sh no longer needed
* bazel distribtest dont publish any artifacts
* try the altsrc respawn for selected jobs
* use altsrc for all windows CI jobs
* use altsrc for all macos CI jobs
* only move files under src/github
* fix uploading artifacts on windows
* Provide custom sponge configs
* Correct the script
* Allow test scripts to produce artifacts through Docker
* Overcome the direcotry clone in our docker script
* Expose TESTGRID_EXCLUDE and GIT_ORIGIN_URL through docker run
* Make sanity tests happy
Ideally instead of having names like ruby_jessie_x64_ruby_2_5 and
ruby_jessie_x64_ruby_2_6 they would have the name "ruby" with tags
containing jessie_x64_ruby_2_5/6. But fixing that would be much more
invasive. The sha1 in the tag is producing the worst effects, so this is
a case of the perfect being the enemy of the good.
Fixes#20546
* Use latest pylint in Python 3.7 (they dropped support for PY2)
* Make latest pylint happy
* Forced to upgrade to shellcheck 0.4.4
* Make shellcheck 0.4.4 happy
* Adopt reviewers' advice to reduce global disabled rules
using ccache when building under a docker image isn't useful when
building on kokoro as each build runs on a fresh VM.
Originally ccache builds were used to speed up jenkins builds,
now removing to guarantee build isolation and simplify stuff.
Change `git submodule` copying mechanism from the host
to the docker container to use `git clone` instead of
`git submodule update --init --reference`, which reaches
out to the network and fails on older docker containers
with deprecated openssl versions that do not support
modern TLS versions required by GitHub, e.g. manylinux1.
This fixes artifact builds for protoc and Python on Linux.
Currently when running python test on linux, we are mounting
/tmp/xdg-cache-home to the docker container that runs the tests
in the attempt to prevent unnecessary downloads of pip packages
(with the theory that more downloads leads to increased flakiness)
- the idea is that while there is a new docker container for each
test suite, the xdg cache remains per-VM.
This approach no longer seems to be useful:
* It turns out that XDG cache doesn't work reliably when multiple
docker containers are using it concurrently (the concurrent run can
see corrupt files). We are using concurrent docker containers in our
multilang test suite to speed up the execution and we are currently
getting flakes seeing flakes caused by this.
* support for caching makes our docker_run scripts more complicated
and we really don't want that.
* since we migrated to kokoro, the caching is limited anyway -
as each run gets a fresh VM, we need to download packages anyway
for every build (and that actually seems to causing way less
flakiness the problem with concurrent XDG caching).
Features included in this merge:
- Added script to build and upload docker image for matrix test.
- Added script to create test cases and created go__master testcases based on it.
- Created dictionary for runtimes and gRPC releases for supported languages.
- Added go 1.7 and 1.8 Dockerfile/templates.
See tools/interop_matrix/README.md for details.
Jan Tattermusch
Yash Tibrewal
Lidi Zheng
Eric Anderson
Esun Kim
Jan Tattermusch
Matt Kwong
Jakob Roland Andersen
Mehrdad Afshari
murgatroid99
Adele Zhou
Yong Ni
Ken Payson
Makarand Dharmapurikar