Increase artifact build timeouts for problem targets (#25695)

Going back 2 weeks, nightly packages have failed to build due to
artifact build timeouts. This change increases timeouts for:

Python Linux Extra: 5hrs -> 7hrs
Python Manylinux: 1hr -> 2hrs
CSharp iOS: 45min -> 1hr
CSharp MacOS x64: 30m -> 45m

This is a stopgap measure to get nightly packages deployed more regularly;
March has seen 3 nightlies published in 11 days.

I've cherry picked some of the artifact build timeout errors below.

2021.03.10
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-10 05:03:40,560 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=7384, time=18000.4sec]
2021-03-10 05:03:40,646 TIMEOUT: build_artifact.python_linux_extra_armv7_2.7 [pid=7375, time=18000.5sec]
2021-03-10 05:03:40,728 TIMEOUT: build_artifact.python_linux_extra_armv7_3.5 [pid=7378, time=18000.6sec]

2021.03.08
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-08 05:03:18,542 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=6853, time=18000.5sec]
2021-03-08 05:03:18,639 TIMEOUT: build_artifact.python_linux_extra_armv7_2.7 [pid=6844, time=18000.6sec]
2021-03-08 05:03:18,741 TIMEOUT: build_artifact.python_linux_extra_armv7_3.5 [pid=6847, time=18000.7sec]

2021.03.07
grpc/core/master/macos/grpc_build_artifacts:
2021-03-07 00:58:02,492 TIMEOUT: build_artifact.csharp_ext_macos_x64 [pid=58673, time=1805.5sec]
2021-03-07 01:13:03,065 TIMEOUT: build_artifact.csharp_ext_macos_ios [pid=58672, time=2706.1sec]

Bumping iOS again from https://github.com/grpc/grpc/pull/25049

grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-07 05:03:36,061 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=8867, time=18002.9sec]

2021.03.06
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-06 05:03:12,474 TIMEOUT: build_artifact.python_linux_extra_armv7_3.5 [pid=8307, time=18000.6sec]

2021.03.05
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-05 05:03:48,611 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=7075, time=18008.6sec]

2021.02.27
grpc/core/master/linux/grpc_build_artifacts
2021-02-27 01:09:57,943 TIMEOUT: build_artifact.python_manylinux2010_x64_cp27-cp27mu [pid=21983, time=3608.9sec]

2021.02.23
grpc/core/master/windows/grpc_build_artifacts
```
2021-02-23 01:48:23,729 SUCCESS: All targets built successfully.

T:\src\github\grpc>set RUNTESTS_EXITCODE=0

T:\src\github\grpc>bash tools/internal_ci/helper_scripts/delete_nonartifacts.sh
++ dirname tools/internal_ci/helper_scripts/delete_nonartifacts.sh
+ cd tools/internal_ci/helper_scripts/../../..
+ find . -type f -not -iname '*sponge_log.*' -not -path './reports/*' -not -path './artifacts/*' -not -path './tools/internal_ci/*' -exec rm -f '{}' +

ERROR: Aborting VM command due to timeout of 10800 seconds
```
pull/25706/head
AJ Heller 4 years ago committed by GitHub
parent 3c9f3972e3
commit b516dbfa21
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 2
      tools/internal_ci/windows/grpc_build_artifacts.cfg
  2. 7
      tools/run_tests/artifacts/artifact_targets.py

@ -16,7 +16,7 @@
# Location of the continuous shell script in repository.
build_file: "grpc/tools/internal_ci/windows/grpc_build_artifacts.bat"
timeout_mins: 180
timeout_mins: 240
action {
define_artifacts {
regex: "**/*sponge_log.*"

@ -133,7 +133,7 @@ class PythonArtifact:
'tools/dockerfile/grpc_artifact_linux_{}'.format(self.arch),
'tools/run_tests/artifacts/build_artifact_python.sh',
environ=environ,
timeout_seconds=60 * 60 * 5,
timeout_seconds=60 * 60 * 7,
docker_base_image='quay.io/grpc/raspbian_{}'.format(self.arch),
extra_docker_args=extra_args)
elif 'manylinux' in self.platform:
@ -171,7 +171,7 @@ class PythonArtifact:
(self.platform, self.arch),
'tools/run_tests/artifacts/build_artifact_python.sh',
environ=environ,
timeout_seconds=60 * 60)
timeout_seconds=60 * 60 * 2)
elif self.platform == 'windows':
if 'Python27' in self.py_version:
environ['EXT_COMPILER'] = 'mingw32'
@ -250,7 +250,7 @@ class CSharpExtArtifact:
return create_jobspec(
self.name,
['tools/run_tests/artifacts/build_artifact_csharp_ios.sh'],
timeout_seconds=45 * 60,
timeout_seconds=60 * 60,
use_workspace=True)
elif self.platform == 'windows':
return create_jobspec(self.name, [
@ -282,6 +282,7 @@ class CSharpExtArtifact:
return create_jobspec(
self.name,
['tools/run_tests/artifacts/build_artifact_csharp.sh'],
timeout_seconds=45 * 60,
environ={'CMAKE_ARCH_OPTION': cmake_arch_option},
use_workspace=True)

Loading…
Cancel
Save