Merge pull request #11944 from protocolbuffers/gha-port-22.x
Backport GHA fixes and optimizations to 22.xpull/11946/head
Mike Kruskal
2 years ago
committed by
GitHub
commit
b7f7171c31
15 changed files with 397 additions and 196 deletions
@ -0,0 +1,204 @@ |
|||||||
|
This directory contains all of our automatically triggered workflows. |
||||||
|
|
||||||
|
# Test runner |
||||||
|
|
||||||
|
Our top level `test_runner.yml` is responsible for kicking off all tests, which |
||||||
|
are represented as reusable workflows. This is carefully constructed to satisfy |
||||||
|
the design laid out in go/protobuf-gha-protected-resources (see below), and |
||||||
|
duplicating it across every workflow file would be difficult to maintain. As an |
||||||
|
added bonus, we can manually dispatch our full test suite with a single button |
||||||
|
and monitor the progress of all of them simultaneously in GitHub's actions UI. |
||||||
|
|
||||||
|
There are five ways our test suite can be triggered: |
||||||
|
|
||||||
|
- **Post-submit tests** (`push`): These are run over newly submitted code |
||||||
|
that we can assume has been thoroughly reviewed. There are no additional |
||||||
|
security concerns here and these jobs can be given highly privileged access to |
||||||
|
our internal resources and caches. |
||||||
|
|
||||||
|
- **Pre-submit tests from a branch** (`push_request`): These are run over |
||||||
|
every PR as changes are made. Since they are coming from branches in our |
||||||
|
repository, they have secret access by default and can also be given highly |
||||||
|
privileged access. However, we expect *many* of these events per change, |
||||||
|
and likely many from abandoned/exploratory changes. Given the much higher |
||||||
|
frequency, we restrict the ability to *write* to our more expensive caches. |
||||||
|
|
||||||
|
- **Pre-submit tests from a fork** (`push_request_target`): These are run |
||||||
|
over every PR from a forked repository as changes are made. These have much |
||||||
|
more restricted access, since they could be coming from anywhere. To protect |
||||||
|
our secret keys and our resources, tests will not run until a commit has been |
||||||
|
labeled `safe to submit`. Further commits will require further approvals to |
||||||
|
run our test suite. Once marked as safe, we will provide read-only access to |
||||||
|
our caches and Docker images, but will generally disallow any writes to shared |
||||||
|
resources. |
||||||
|
|
||||||
|
- **Continuous tests** (`schedule`): These are run on a fixed schedule. We |
||||||
|
currently have them set up to run daily, and can help identify non-hermetic |
||||||
|
issues in tests that don't get run often (such as due to test caching) or during |
||||||
|
slow periods like weekends and holidays. Similar to post-submit tests, these |
||||||
|
are run over submitted code and are highly privileged in the resources they |
||||||
|
can use. |
||||||
|
|
||||||
|
- **Manual testing** (`workflow_dispatch`): Our test runner can be triggered |
||||||
|
manually over any branch. This is treated similarly to pre-submit tests, |
||||||
|
which should be highly privileged because they can only be triggered by the |
||||||
|
protobuf team. |
||||||
|
|
||||||
|
# Staleness handling |
||||||
|
|
||||||
|
While Bazel handles code generation seamlessly, we do support build systems that |
||||||
|
don't. There are a handful of cases where we need to check in generated files |
||||||
|
that can become stale over time. In order to provide a good developer |
||||||
|
experience, we've implemented a system to make this more manageable. |
||||||
|
|
||||||
|
- Stale files should have a corresponding `staleness_test` Bazel target. This |
||||||
|
should be marked `manual` to avoid getting picked up in CI, but will fail if |
||||||
|
files become stale. It also provides a `--fix` flag to update the stale files. |
||||||
|
|
||||||
|
- Bazel tests will never depend on the checked-in versions, and will generate |
||||||
|
new ones on-the-fly during build. |
||||||
|
|
||||||
|
- Non-Bazel tests will always regenerate necessary files before starting. This |
||||||
|
is done using our `bash` and `docker` actions, which should be used for any |
||||||
|
non-Bazel tests. This way, no tests will fail due to stale files. |
||||||
|
|
||||||
|
- A post-submit job will immediately regenerate any stale files and commit them |
||||||
|
if they've changed. |
||||||
|
|
||||||
|
- A scheduled job will run late at night every day to make sure the post-submit |
||||||
|
is working as expected (that is, it will run all the staleness tests). |
||||||
|
|
||||||
|
The `regenerate_stale_files.sh` script is the central script responsible for all |
||||||
|
the re-generation of stale files. |
||||||
|
|
||||||
|
# Forked PRs |
||||||
|
|
||||||
|
Because we need secret access to run our tests, we use the `pull_request_target` |
||||||
|
event for PRs coming from forked repositories. We do checkout the code from the |
||||||
|
PR's head, but the workflow files themselves are always fetched from the *base* |
||||||
|
branch (that is, the branch we're merging to). Therefore, any changes to these |
||||||
|
files won't be tested, so we explicitly ban PRs that touch these files. |
||||||
|
|
||||||
|
# Caches |
||||||
|
|
||||||
|
We have a number of different caching strategies to help speed up tests. These |
||||||
|
live either in GCP buckets or in our GitHub repository cache. The former has |
||||||
|
a lot of resources available and we don't have to worry as much about bloat. |
||||||
|
On the other hand, the GitHub repository cache is limited to 10GB, and will |
||||||
|
start pruning old caches when it exceeds that threshold. Therefore, we need |
||||||
|
to be very careful about the size and quantity of our caches in order to |
||||||
|
maximize the gains. |
||||||
|
|
||||||
|
## Bazel remote cache |
||||||
|
|
||||||
|
As described in https://bazel.build/remote/caching, remote caching allows us to |
||||||
|
offload a lot of our build steps to a remote server that holds a cache of |
||||||
|
previous builds. We use our GCP project for this storage, and configure |
||||||
|
*every* Bazel call to use it. This provides substantial performance |
||||||
|
improvements at minimal cost. |
||||||
|
|
||||||
|
We do not allow forked PRs to upload updates to our Bazel caches, but they |
||||||
|
do use them. Every other event is given read/write access to the caches. |
||||||
|
Because Bazel behaves poorly under certain environment changes (such as |
||||||
|
toolchain, operating system), we try to use finely-grained caches. Each job |
||||||
|
should typically have its own cache to avoid cross-pollution. |
||||||
|
|
||||||
|
## Bazel repository cache |
||||||
|
|
||||||
|
When Bazel starts up, it downloads all the external dependencies for a given |
||||||
|
build and stores them in the repository cache. This cache is *separate* from |
||||||
|
the remote cache, and only exists locally. Because we have so many Bazel |
||||||
|
dependencies, this can be a source of frequent flakes due to network issues. |
||||||
|
|
||||||
|
To avoid this, we keep a cached version of the repository cache in GitHub's |
||||||
|
action cache. Our full set of repository dependencies ends up being ~300MB, |
||||||
|
which is fairly expensive given our 10GB maximum. The most expensive ones seem |
||||||
|
to come from Java, which has some very large downstream dependencies. |
||||||
|
|
||||||
|
Given the cost, we take a more conservative approach for this cache. Only push |
||||||
|
events will ever write to this cache, but all events can read from them. |
||||||
|
Additionally, we only store three caches for any given commit, one per platform. |
||||||
|
This means that multiple jobs are trying to update the same cache, leading to a |
||||||
|
race. GitHub rejects all but one of these updates, so we designed the system so |
||||||
|
that caches are only updated if they've actually changed. That way, over time |
||||||
|
(and multiple pushes) the repository caches will incrementally grow to encompass |
||||||
|
all of our dependencies. A scheduled job will run monthly to clear these caches |
||||||
|
to prevent unbounded growth as our dependencies evolve. |
||||||
|
|
||||||
|
## ccache |
||||||
|
|
||||||
|
In order to speed up non-Bazel builds to be on par with Bazel, we make use of |
||||||
|
[ccache](https://ccache.dev/). This intercepts all calls to the compiler, and |
||||||
|
caches the result. Subsequent calls with a cache-hit will very quickly |
||||||
|
short-circuit and return the already computed result. This has minimal affect |
||||||
|
on any *single* job, since we typically only run a single build. However, by |
||||||
|
caching the ccache results in GitHub's action cache we can substantially |
||||||
|
decrease the build time of subsequent runs. |
||||||
|
|
||||||
|
One useful feature of ccache is that you can set a maximum cache size, and it |
||||||
|
will automatically prune older results to keep below that limit. On Linux and |
||||||
|
Mac cmake builds, we generally get 30MB caches and set a 100MB cache limit. On |
||||||
|
Windows, with debug symbol stripping we get ~70MB and set a 200MB cache limit. |
||||||
|
|
||||||
|
Because CMake build tend to be our slowest, bottlenecking the entire CI process, |
||||||
|
we use a fairly expensive strategy with ccache. All events will cache their |
||||||
|
ccache directory, keyed by the commit and the branch. This means that each |
||||||
|
PR and each branch will write its own set of caches. When looking up which |
||||||
|
cache to use initially, each job will first look for a recent cache in its |
||||||
|
current branch. If it can't find one, it will accept a cache from the base |
||||||
|
branch (for example, PRs will initially use the latest cache from their target |
||||||
|
branch). |
||||||
|
|
||||||
|
While the ccache caches quickly over-run our GitHub action cache, they also |
||||||
|
quickly become useless. Since GitHub prunes caches based on the time they were |
||||||
|
last used, this just means that we'll see quicker turnover. |
||||||
|
|
||||||
|
## Bazelisk |
||||||
|
|
||||||
|
Bazelisk will automatically download a pinned version of Bazel on first use. |
||||||
|
This can lead to flakes, and to avoid that we cache the result keyed on the |
||||||
|
Bazel version. Only push events will write to this cache, but it's unlikely |
||||||
|
to change very often. |
||||||
|
|
||||||
|
## Docker images |
||||||
|
|
||||||
|
Instead of downloading a fresh Docker image for every test run, we can save it |
||||||
|
as a tar and cache it using `docker image save` and later restore using |
||||||
|
`docker image load`. This can decrease download times and also reduce flakes. |
||||||
|
Note, Docker's load can actually be significantly slower than a pull in certain |
||||||
|
situations. Therefore, we should reserve this strategy for only Docker images |
||||||
|
that are causing noticeable flakes. |
||||||
|
|
||||||
|
## Pip dependencies |
||||||
|
|
||||||
|
The actions/setup-python action we use for Python supports automated caching |
||||||
|
of pip dependencies. We enable this to avoid having to download these |
||||||
|
dependencies on every run, which can lead to flakes. |
||||||
|
|
||||||
|
# Custom actions |
||||||
|
|
||||||
|
We've defined a number of custom actions to abstract out shared pieces of our |
||||||
|
workflows. |
||||||
|
|
||||||
|
- **Bazel** use this for running all Bazel tests. It can take either a single |
||||||
|
Bazel command or a more general bash command. In the latter case, it provides |
||||||
|
environment variables for running Bazel with all our standardized settings. |
||||||
|
|
||||||
|
- **Bazel-Docker** nearly identical to the **Bazel** action, this additionally |
||||||
|
runs everything in a specified Docker image. |
||||||
|
|
||||||
|
- **Bash** use this for running non-Bazel tests. It takes a bash command and |
||||||
|
runs it verbatim. It also handles the regeneration of stale files (which does |
||||||
|
use Bazel), which non-Bazel tests might depend on. |
||||||
|
|
||||||
|
- **Docker** nearly identical to the **Bash** action, this additionally runs |
||||||
|
everything in a specified Docker image. |
||||||
|
|
||||||
|
- **ccache** this sets up a ccache environment, and initializes some |
||||||
|
environment variables for standardized usage of ccache. |
||||||
|
|
||||||
|
- **Cross-compile protoc** this abstracts out the compilation of protoc using |
||||||
|
our cross-compilation infrastructure. It will set a `PROTOC` environment |
||||||
|
variable that gets automatically picked up by a lot of our infrastructure. |
||||||
|
This is most useful in conjunction with the **Bash** action with non-Bazel |
||||||
|
tests. |
||||||
@ -0,0 +1,27 @@ |
|||||||
|
name: Forked PR workflow check |
||||||
|
|
||||||
|
# This workflow prevents modifications to our workflow files in PRs from forked |
||||||
|
# repositories. Since tests in these PRs always use the workflows in the |
||||||
|
# *target* branch, modifications to these files can't be properly tested. |
||||||
|
|
||||||
|
on: |
||||||
|
# safe presubmit |
||||||
|
pull_request: |
||||||
|
branches: |
||||||
|
- main |
||||||
|
- '[0-9]+.x' |
||||||
|
# The 21.x branch still uses Kokoro |
||||||
|
- '!21.x' |
||||||
|
# For testing purposes so we can stage this on the `gha` branch. |
||||||
|
- gha |
||||||
|
paths: |
||||||
|
- '.github/workflows/**' |
||||||
|
|
||||||
|
jobs: |
||||||
|
check: |
||||||
|
name: Check PR source |
||||||
|
runs-on: ubuntu-latest |
||||||
|
steps: |
||||||
|
- run: > |
||||||
|
${{ github.event.pull_request.head.repo.full_name == 'protocolbuffers/protobuf' }} || |
||||||
|
(echo "This pull request is from an unsafe fork (${{ github.event.pull_request.head.repo.full_name }}) and isn't allowed to modify workflow files!" && exit 1) |
||||||
@ -0,0 +1,17 @@ |
|||||||
|
This directory contains CI-specific tooling. |
||||||
|
|
||||||
|
# Clang wrappers |
||||||
|
|
||||||
|
CMake allows for compiler wrappers to be injected such as ccache, which |
||||||
|
intercepts compiler calls and short-circuits on cache-hits. This can be done |
||||||
|
by specifying `CMAKE_C_COMPILER_LAUNCHER` and `CMAKE_CXX_COMPILER_LAUNCHER` |
||||||
|
during CMake's configure step. Unfortunately, X-Code doesn't provide anything |
||||||
|
like this, so we use basic wrapper scripts to invoke ccache + clang. |
||||||
|
|
||||||
|
# Bazelrc files |
||||||
|
|
||||||
|
In order to allow platform-specific `.bazelrc` flags during testing, we keep |
||||||
|
3 different versions here along with a shared `common.bazelrc` that they all |
||||||
|
include. Our GHA infrastructure will select the appropriate file for any test |
||||||
|
and overwrite the default `.bazelrc` in our workspace, which is intended for |
||||||
|
development only. |
||||||
Loading…
Reference in new issue