This is inspired by https://github.com/protocolbuffers/protobuf/issues/12306,
but does not show those failures, so clearly it doesn't catch everything, but
figured some coverage is better than none.
PiperOrigin-RevId: 525134469
For now, this only covers linux on the two architectures we have testing support for. However, it serves as a good sanity check and can be expanded in the future.
PiperOrigin-RevId: 514449399
For now, this only covers linux on the two architectures we have testing support for. However, it serves as a good sanity check and can be expanded in the future.
PiperOrigin-RevId: 514449399
This will make PRs from forked repositories significantly less painful, since they'll agree on which version of each action to use. OTOH, we'll have a separate repo that needs to be maintained, and changes to it will need to be coordinated and versioned carefully. This will likely need to be done less often though now that our infrastructure is stable.
PiperOrigin-RevId: 512117705
These will still use MSVC as the compiler, but will no longer generate Visual Studio projects for the builds. Visual Studio is particularly bad at parallelizing builds, and is hostile to ccache.
This change also tweaks the ccache setup to prevent unbounded the growth observed in our github caches. Windows builds have had debug symbols stripped to reduce ccache size by a factor of 2x, and ccache maximum have been tweaked so that we persist fewer older builds.
Before this change, each CMake build took 12 minutes on every run (plus some constant overhead from staleness/gcloud), even with caching or on large multi-core runners. No amount of caching or parallelization made any noticeable difference above noise. With this change, we see the following improvements:
- 12 minutes to build from scratch on normal runners (unchanged)
- 4 minutes on 32-core runners from scratch
- 1 minute with optimal caches available on normal runners.
- 30 seconds on 32-core runners with optimal caches
PiperOrigin-RevId: 508799909
These will still use MSVC as the compiler, but will no longer generate Visual Studio projects for the builds. Visual Studio is particularly bad at parallelizing builds, and is hostile to ccache.
This change also tweaks the ccache setup to prevent unbounded the growth observed in our github caches. Windows builds have had debug symbols stripped to reduce ccache size by a factor of 2x, and ccache maximum have been tweaked so that we persist fewer older builds.
Before this change, each CMake build took 12 minutes on every run (plus some constant overhead from staleness/gcloud), even with caching or on large multi-core runners. No amount of caching or parallelization made any noticeable difference above noise. With this change, we see the following improvements:
- 12 minutes to build from scratch on normal runners (unchanged)
- 4 minutes on 32-core runners from scratch
- 1 minute with optimal caches available on normal runners.
- 30 seconds on 32-core runners with optimal caches
PiperOrigin-RevId: 508799909
The `BuildInParallel` flag only controls parallelism at the *project* level, while `/MP` enables it at the *compilation unit* level.
PiperOrigin-RevId: 507837073
This uses ccache + github caching to substantially decrease the time it takes to run CMake builds. Due to Bazel caching, these are some of our slowest tests, causing one of the biggest presubmit bottlenecks
PiperOrigin-RevId: 507667813
This no longer served its original purpose and was actually testing an *older* version of MacOS than the rest of our test suite. This did lock down CMake builds on Mac though, so we'll keep that coverage.
PiperOrigin-RevId: 506776948
This migrates from reusable workflows to composite GHA actions. This has the following advantages:
1) We can split them into smaller, easier to reason about chunks
2) We can reuse them more freely between each other and workflows
3) They don't complicate the job names
4) In theory they'll be easier to test in isolation. While composite actions can't be unit-tested, we can easily break them up into nodejs or docker actions that can be.
As a proof of concept, some of our non-Bazel tests are migrated to GHA as well (CMake + Composer)
Closes#11718
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/11718 from protocolbuffers:gha-actions 5403307bc00616e94816aa460813939d8f37a1bd
PiperOrigin-RevId: 506375417
This is a proof of concept that we can experiment with before migrating all of our Kokoro tests to GHA. In addition to the migration, this builds out infrastructure for safe handling of both external and internal contributions. Two of the existing GHA workflows have also been migrated to this system to unify how we handle testing.
To test the new workflow introduced here, you can open PRs to the `gha` branch. This feature branch acts as a staging area for GHA. PRs opened up from this repo will automatically have all tests run on each commit. We will also run them all as post-submits, and continuously on a daily schedule. PRs opened from forked repos will need per-commit approval for each test run, which can be given by adding the `safe for tests` label.
Examples (failures are intentional to show that the PR code is being tested):
- Internal PR: https://github.com/protocolbuffers/protobuf/pull/11679
- Fork PR (approved): https://github.com/protocolbuffers/protobuf/pull/11685
- Fork PR (rejected): https://github.com/protocolbuffers/protobuf/pull/11683
- External PR: https://github.com/protocolbuffers/protobuf/pull/11695Closes#11702
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/11702 from protocolbuffers:gha f3a2663896
PiperOrigin-RevId: 506169444
Mike Kruskal
Deanna Garcia
Mike Kruskal
Adam Cozzette
Thomas Van Lenten
John Cater