AJ Heller
b516dbfa21
Increase artifact build timeouts for problem targets ( #25695 )
...
Going back 2 weeks, nightly packages have failed to build due to
artifact build timeouts. This change increases timeouts for:
Python Linux Extra: 5hrs -> 7hrs
Python Manylinux: 1hr -> 2hrs
CSharp iOS: 45min -> 1hr
CSharp MacOS x64: 30m -> 45m
This is a stopgap measure to get nightly packages deployed more regularly;
March has seen 3 nightlies published in 11 days.
I've cherry picked some of the artifact build timeout errors below.
2021.03.10
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-10 05:03:40,560 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=7384, time=18000.4sec]
2021-03-10 05:03:40,646 TIMEOUT: build_artifact.python_linux_extra_armv7_2.7 [pid=7375, time=18000.5sec]
2021-03-10 05:03:40,728 TIMEOUT: build_artifact.python_linux_extra_armv7_3.5 [pid=7378, time=18000.6sec]
2021.03.08
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-08 05:03:18,542 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=6853, time=18000.5sec]
2021-03-08 05:03:18,639 TIMEOUT: build_artifact.python_linux_extra_armv7_2.7 [pid=6844, time=18000.6sec]
2021-03-08 05:03:18,741 TIMEOUT: build_artifact.python_linux_extra_armv7_3.5 [pid=6847, time=18000.7sec]
2021.03.07
grpc/core/master/macos/grpc_build_artifacts:
2021-03-07 00:58:02,492 TIMEOUT: build_artifact.csharp_ext_macos_x64 [pid=58673, time=1805.5sec]
2021-03-07 01:13:03,065 TIMEOUT: build_artifact.csharp_ext_macos_ios [pid=58672, time=2706.1sec]
Bumping iOS again from https://github.com/grpc/grpc/pull/25049
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-07 05:03:36,061 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=8867, time=18002.9sec]
2021.03.06
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-06 05:03:12,474 TIMEOUT: build_artifact.python_linux_extra_armv7_3.5 [pid=8307, time=18000.6sec]
2021.03.05
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-05 05:03:48,611 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=7075, time=18008.6sec]
2021.02.27
grpc/core/master/linux/grpc_build_artifacts
2021-02-27 01:09:57,943 TIMEOUT: build_artifact.python_manylinux2010_x64_cp27-cp27mu [pid=21983, time=3608.9sec]
2021.02.23
grpc/core/master/windows/grpc_build_artifacts
```
2021-02-23 01:48:23,729 SUCCESS: All targets built successfully.
T:\src\github\grpc>set RUNTESTS_EXITCODE=0
T:\src\github\grpc>bash tools/internal_ci/helper_scripts/delete_nonartifacts.sh
++ dirname tools/internal_ci/helper_scripts/delete_nonartifacts.sh
+ cd tools/internal_ci/helper_scripts/../../..
+ find . -type f -not -iname '*sponge_log.*' -not -path './reports/*' -not -path './artifacts/*' -not -path './tools/internal_ci/*' -exec rm -f '{}' +
ERROR: Aborting VM command due to timeout of 10800 seconds
```
4 years ago
Jan Tattermusch
bf21b2b3a0
update bazel version in other places too
4 years ago
Lidi Zheng
16157cebe9
Hopefully fixed 27 build on winserver2016
...
* Download the working msys64 and use it to compile 27
* Remove the cygwin detection override logic
* Only install Python interpreters in jobs running on winserver2016
* Fix the batch script substring check
* Increase the timeout for related jobs
* TIL that there is an allow list for Kokoro env vars
4 years ago
Lidi Zheng
511af22c78
Hopefully fixed 27 build on winserver2016
...
* Download the working msys64 and use it to compile 27
* Remove the cygwin detection override logic
* Only install Python interpreters in jobs running on winserver2016
* Fix the batch script substring check
* Increase the timeout for related jobs
* TIL that there is an allow list for Kokoro env vars
4 years ago
Jan Tattermusch
e58f0129d2
Stop relying on rbe-windows-credentials.json
4 years ago
Lidi Zheng
85d0d548c4
Increase the timeout of Python basic tests on Windows
4 years ago
Jan Tattermusch
858797ae77
disable grpc_build_packages.bat
4 years ago
Lidi Zheng
a0a69de4df
Improve Win CI script with fail fast
4 years ago
Jan Tattermusch
424d33415c
upgrade bazel to 2.2.0
4 years ago
Lidi Zheng
db4ce94ebb
Revert "Revert "Release Python3.8 wheels for Windows""
5 years ago
Jan Tattermusch
42039d09b7
Revert "Release Python3.8 wheels for Windows"
5 years ago
Lidi Zheng
ed054d03a1
Fix the hack, hope the test still passes
5 years ago
Lidi Zheng
6b92b4af05
Attempt to drop support for Python 3.4
5 years ago
Jan Tattermusch
e5e3f70a1a
Revert "Revert "Upgrade to bazel 1.0.0""
5 years ago
Lidi Zheng
0765cf940b
Refactor the way AsyncIO is tested under run_tests.py
5 years ago
Lidi Zheng
b231e7f88f
Add three more Kokoro job for Linux, macOS and Windows
5 years ago
Jan Tattermusch
79cfdb525c
increase timeout of bazel RBE jobs
...
if the kokoro job times out and the bazel command is not done, it ends up in
weird test results ("Unevaluated targets") that are hard to interpret.
Setting the timeout for kokoro jobs high enough to create extra reserve
should help.
5 years ago
Nicolas Noble
04d7e0d11f
Revert "Upgrade to bazel 1.0.0"
5 years ago
Jan Tattermusch
27f1a94aa5
upgrade to bazel 1.0.0
5 years ago
Jan Tattermusch
d10fb3ba99
fix win RBE opt build
5 years ago
Jan Tattermusch
17fc384b1e
add job config for Windows RBE opt config, upload test results
5 years ago
Jan Tattermusch
2262e2b8e8
add extra kokoro build info to Win RBE runs
5 years ago
Jan Tattermusch
837cb04f76
win RBE: dont cache on master, use dbg config
5 years ago
Jan Tattermusch
8d81a934ca
run both C and C++ tests on win RBE
5 years ago
Jan Tattermusch
f35678e3b4
run at least core tests
5 years ago
Jan Tattermusch
16f4103269
try running only one test
5 years ago
Jan Tattermusch
6f6fe451be
run tests on win RBE
5 years ago
Jan Tattermusch
0b65b421d3
try fixing the C# win artifact
5 years ago
Jan Tattermusch
4521dea985
try bazel 0.29.1 for windows
5 years ago
Jan Tattermusch
c5662bba80
use bazel 0.29.0 on windows
5 years ago
Jan Tattermusch
98d2d32b9e
upgrade to bazel 0.29.1
5 years ago
Jan Tattermusch
5cd37064cb
try fix win RBE build
5 years ago
Jan Tattermusch
e88553ed17
upgrade windows RBE to bazel 0.29
5 years ago
Jan Tattermusch
4e8ecb9fe6
Revert "Upgrade to Bazel 0.29 (including Windows RBE)"
5 years ago
Jan Tattermusch
08dc331019
try fix win RBE build
5 years ago
Jan Tattermusch
0c3a5990b1
upgrade windows RBE to bazel 0.29
5 years ago
Nicolas "Pixel" Noble
de9c1e8f9c
Upgrading Bazel Windows RBE to 0.26 too.
5 years ago
Jan Tattermusch
ed7e8c53e0
update scripts
6 years ago
Jan Tattermusch
af279949d4
update bazel_rbe.bat
6 years ago
Jan Tattermusch
48d4bcb8ef
create build.cfg for split-up PR jobs
6 years ago
Jan Tattermusch
bb3808d355
job split followup: increase timeout for macos and windows C/C++ jobs
6 years ago
Jan Tattermusch
b18111c618
split multilang jobs by language
6 years ago
Jan Tattermusch
d791dd1775
create build.cfg for split-up PR jobs
6 years ago
Jan Tattermusch
8b0683a015
add build.cfg for windows bazel RBE PR build
6 years ago
Jan Tattermusch
1dab7cf91a
job split followup: increase timeout for macos and windows C/C++ jobs
6 years ago
Jan Tattermusch
2cb63d859d
split multilang jobs by language
6 years ago
Jan Tattermusch
f53eca24f4
add non-internal build.cfg files for bazel RBE jobs
6 years ago
Jan Tattermusch
284597c9cf
standardize bazel rbe credentials 2
6 years ago
Jan Tattermusch
296f42c3e6
improve windows RBE build
6 years ago
Jan Tattermusch
f2f27a0e64
add non-internal build.cfg files for bazel RBE jobs
6 years ago