We will soon be moving utf8_range into the protobuf repo, not as a subtree
anymore but as the real source of truth. This change adds CI coverage in
advance so that there will not be a lapse in coverage.
I also upgraded our pinned versions of rules_fuzzing and rules_python, to fix
some errors that came up with Bazel 6 and Python 3.12. I had to patch
rules_fuzzing but I am working on upstreaming the fixes.
PiperOrigin-RevId: 579987379
setup-gcloud doesn't honor the environment variables set by setup-python, so we need to manually set CLOUDSDK_PYTHON before running it.
PiperOrigin-RevId: 578674246
This should help us catch problems that come up when libprotobuf and libprotoc
are built as shared libraries. The motivating example was that we recently had
build failures due to `PROTOBUF_EXPORT` being missing from a symbol that needed
it, but none of our existing tests caught this. (Technically this test wouldn't
catch it either since that particular issue affected C++17 only, but at least
this should help with similar problems.)
PiperOrigin-RevId: 575218922
These tests were right near the threshold for disk space on default runners, and recently got pushed over (~18G). The cheapest large runner has 150G of SSD storage, and won't likely hit this issue again.
PiperOrigin-RevId: 572596499
This will retry up to 3 times if we hit networks flakes updating our submodules. It will also allow us to easily inject other stability fixes to this step in the future.
PiperOrigin-RevId: 568306356
## Description
I would like to suggest a security practice recommended by the [OpenSSF Scorecard][scorecard-repo] which is to hash pin dependencies to prevent typosquatting and tag renaming attacks.
The change would only be applied to GitHub workflows.
This means hash pinning GitHub Workflow actions.
Along with hash-pinning dependencies, I also recommend adopting dependabot (or other dependency update tool) to help keep the dependencies up to date. Most tools can update hashes and associated semantic version comments.
Any questions or concerns just let me know.
Thanks!
## Additional Context
A tag renaming attack is a type of attack whereby an attacker:
- Hijack an action.
- Upload a malicious version.
- Replace existing tags with malicious versions.
A [typosquatting attack][typosquatting] is a type of attack whereby an attacker:
- Create a malicious package
- Publish it with a similar name of a known package (example: numpi instead of numpy)
For more informations about the dependency-update tools:
- [Dependabot][dependabot]
[scorecard-repo]: https://github.com/ossf/scorecard
[deps-confusion]: https://www.websecuritylens.org/how-dependency-confusion-attack-works-and-how-to-prevent-it/
[typosquatting]: https://snyk.io/blog/typosquatting-attacks/
[dependabot]: https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/
[renovatebot]:https://www.mend.io/renovate/
PiperOrigin-RevId: 561019142
It's not clear what happens if both C++17/C++20 and C++14 are specified, so it's possible this was effectively disabling those tests.
PiperOrigin-RevId: 544763031
This turns the constexpr constructors into templates to silence errors when constexpr isn't valid. We are also switching to 12.2 for GCC/cmake tests to prevent regressions (9.5 and 13.1 are already tested by GCC/bazel tests).
Fixes#12807
PiperOrigin-RevId: 532258101
Note: gcc only supports docker images down to 9.5, and the 7.3 image is very old and problematic. A follow-up change might enable testing for GCC 7.3, which is our minimal supported version
PiperOrigin-RevId: 529885733
Note: gcc only supports docker images down to 9.5, and the 7.3 image is very old and problematic. A follow-up change might enable testing for GCC 7.3, which is our minimal supported version
PiperOrigin-RevId: 529885733
These more closely follow the standard practices of our users, where dependencies are pre-installed instead of using our provided sub-modules. This will prevent issues such as #12201 from reoccuring.
Additionally, this cl bumps our Abseil dependency to the latest release, and fixes a GTest issue that went previously unnoticed.
PiperOrigin-RevId: 529490402
These more closely follow the standard practices of our users, where dependencies are pre-installed instead of using our provided sub-modules. This will prevent issues such as #12201 from reoccuring.
Additionally, this cl bumps our Abseil dependency to the latest release, and fixes a GTest issue that went previously unnoticed.
PiperOrigin-RevId: 529490402
This is inspired by https://github.com/protocolbuffers/protobuf/issues/12306,
but does not show those failures, so clearly it doesn't catch everything, but
figured some coverage is better than none.
PiperOrigin-RevId: 525134469
For now, this only covers linux on the two architectures we have testing support for. However, it serves as a good sanity check and can be expanded in the future.
PiperOrigin-RevId: 514449399
For now, this only covers linux on the two architectures we have testing support for. However, it serves as a good sanity check and can be expanded in the future.
PiperOrigin-RevId: 514449399
This will make PRs from forked repositories significantly less painful, since they'll agree on which version of each action to use. OTOH, we'll have a separate repo that needs to be maintained, and changes to it will need to be coordinated and versioned carefully. This will likely need to be done less often though now that our infrastructure is stable.
PiperOrigin-RevId: 512117705
These will still use MSVC as the compiler, but will no longer generate Visual Studio projects for the builds. Visual Studio is particularly bad at parallelizing builds, and is hostile to ccache.
This change also tweaks the ccache setup to prevent unbounded the growth observed in our github caches. Windows builds have had debug symbols stripped to reduce ccache size by a factor of 2x, and ccache maximum have been tweaked so that we persist fewer older builds.
Before this change, each CMake build took 12 minutes on every run (plus some constant overhead from staleness/gcloud), even with caching or on large multi-core runners. No amount of caching or parallelization made any noticeable difference above noise. With this change, we see the following improvements:
- 12 minutes to build from scratch on normal runners (unchanged)
- 4 minutes on 32-core runners from scratch
- 1 minute with optimal caches available on normal runners.
- 30 seconds on 32-core runners with optimal caches
PiperOrigin-RevId: 508799909
These will still use MSVC as the compiler, but will no longer generate Visual Studio projects for the builds. Visual Studio is particularly bad at parallelizing builds, and is hostile to ccache.
This change also tweaks the ccache setup to prevent unbounded the growth observed in our github caches. Windows builds have had debug symbols stripped to reduce ccache size by a factor of 2x, and ccache maximum have been tweaked so that we persist fewer older builds.
Before this change, each CMake build took 12 minutes on every run (plus some constant overhead from staleness/gcloud), even with caching or on large multi-core runners. No amount of caching or parallelization made any noticeable difference above noise. With this change, we see the following improvements:
- 12 minutes to build from scratch on normal runners (unchanged)
- 4 minutes on 32-core runners from scratch
- 1 minute with optimal caches available on normal runners.
- 30 seconds on 32-core runners with optimal caches
PiperOrigin-RevId: 508799909
The `BuildInParallel` flag only controls parallelism at the *project* level, while `/MP` enables it at the *compilation unit* level.
PiperOrigin-RevId: 507837073
Adam Cozzette
Mike Kruskal
Sandy Zhang
Protobuf Team Bot
Joyce
Deanna Garcia
Mike Kruskal
Thomas Van Lenten
John Cater