macos-12 runner image is deprecated and is EOL 12/3/24 per https://github.com/actions/runner-images/issues/10721
Note that Python 3.8 is EOL and these tests should also be dropped / upgraded in
favor of 3.9 soon anyways.
#test-continuous
PiperOrigin-RevId: 687002433
This also prevents the validate_yaml script from erroring when trying to log that a step is mishandling continuous-run / continuous-prefix
PiperOrigin-RevId: 686952188
Putting it into BUILD files unintentionally forces it on all our downstream users. Instead, we just want to enable this during testing and let them choose for themselves in their builds.
Note, that this expands the scope of -Werror to our entire repo for CI, so a bunch of fixes and opt-outs had to be applied to get this change passing.
Closed#14714
PiperOrigin-RevId: 666903224
Before this PR, we stored a list internally of tests that must pass on presubmit and tried to keep it up to date.
This PR moves that information keeping into GitHub by adding a 'continuous-only' variable to most testing matrices to allow authors to specify which of their tests should be skipped on presubmit. During presubmit, tests that were specified to not run on presubmit will not be run and their names will be prefixed with "[SKIPPED]". All continuous only tests will be suffixed with "(Continuous)".
At the end of running all the tests, we have a single "All Blocking Tests" signal that will tell us whether all of the necessary tests have passed (either for presubmit or continuous based on how the test was triggered).
I've tested this from a different branch [here](https://github.com/protocolbuffers/protobuf/actions/runs/9602443750?pr=17151) and from a different fork [here](https://github.com/protocolbuffers/protobuf/actions/runs/9602554500?pr=17192). These should be the same and are as far as I can tell.
I also have a continuous test run [here](https://github.com/protocolbuffers/protobuf/actions/runs/9603824200) which runs the entire test suite.
Closes#17198
PiperOrigin-RevId: 662940724
In this first stage, we rename the directory from protos_generator to hpb_generator, updating all necessary BUILD files and #includes.
PiperOrigin-RevId: 642600953
There is a special case where message factories can be confused: if a module
written in C++ with pybind11 links against a self-recursive message, and that
message is part of another message loaded from Python, then the confusion
will happen.
Example:
# This one is also linked into the C++ module.
message SelfRecursive {
optional SelfRecursive self_recursive = 1;
}
# This one is used only in Python and not linked.
message OnlyUsedInPython {
optional SelfRecursive self_recursive = 2;
}
The caching through message_factory::RegisterMessageClass then happens on one
instance of the factory, but traversal with the lookup in another.
This occurs in the pure Python and upb implementations that have their own
default descriptor pools (and thus message factory).
Fix this by using the already passed message factory to registering the
message class to cache.
A test accounts for this case to avoid regressions.
PiperOrigin-RevId: 642551744
This will allow them to reuse our bazelrc and remote caching setup. This also silences the non-bzlmod windows test that's hitting the windows path length.
PiperOrigin-RevId: 626390416
We're seeing a high flake rate due to remote cache-misses only on Bazel 7 builds. The key change here seems to be adding the remote_download_output flag, but this also upgrades us to 7.1.1 and adds some retry behavior to caching issues.
PiperOrigin-RevId: 625892332
The latest github runner image of ubuntu-22 includes a kernel update that breaks the sanitizers we use in our docker images. Long-term, we'll likely need to upgrade the images to new sanitizers and clang.
PiperOrigin-RevId: 616319042
There's a test run in test_python.yml that is non-trivial to get working with
Python 3.12 due to some refactoring of our Docker images that would be needed.
But this change updates everything else to add coverage for Python 3.12.
The main changes necessary to get the builds working were to upgrade some Pip
packages via requirements.txt, including in a patch to `rules_fuzzing` that I
plan to upstream soon. I also had to take an explicit dependency on
`setuptools`.
I removed tox.ini, since it was outdated and we have not been actively
maintaining it.
PiperOrigin-RevId: 580548224
Github changed their mac runners to use 3.12 by default, which removed setuptools and is causing breakages in some of our tests.
PiperOrigin-RevId: 578917538
This makes the file layout a bit more consistent with the `protos ->
protos_generator` pattern. I also replaced the `upbc` namespace with
`upb::generator`.
PiperOrigin-RevId: 569264372
This change moves almost everything in the `upb/` directory up one level, so
that for example `upb/upb/generated_code_support.h` becomes just
`upb/generated_code_support.h`. The only exceptions I made to this were that I
left `upb/cmake` and `upb/BUILD` where they are, mostly because that avoids
conflict with other files and the current locations seem reasonable for now.
The `python/` directory is a little bit of a challenge because we had to merge
the existing directory there with `upb/python/`. I made `upb/python/BUILD` into
the BUILD file for the merged directory, and it effectively loads the contents
of the other BUILD file via `python/build_targets.bzl`, but I plan to clean
this up soon.
PiperOrigin-RevId: 568651768
This will retry up to 3 times if we hit networks flakes updating our submodules. It will also allow us to easily inject other stability fixes to this step in the future.
PiperOrigin-RevId: 568306356
This restores the Python wheel CI runs from the old upb repo with only minor
changes. I had to update a path in one of the `py_wheel` rules and also make a
slight tweak to ensure that the `descriptor.upb_minitable.{h,c}` files make it
into the source wheels. The change in text_format_test.py is not strictly
necessary but is a small simplification I made while I was trying to debug an
issue with CRLF newlines.
I had to update test_util.py to use `importlib` to access the golden files from
the installed `protobuftests` package. I suspect the previous incarnation of
thse test runs was somehow reading the goldens from the repo checkout, but I
think the intention is to read them from `protobuftests` instead. This was a
bit tricky to get working because Python versions before 3.9 do not support
`importlib.resources.files()`. I set up the code to fall back on
`importlib.resources.open_binary()` in that case, but that function does not
support subdirectories, so this required putting an `__init__.py` file inside
the `testdata` directory to make sure it is treated as a Python package.
PiperOrigin-RevId: 567366695
This restores a CI run that used to exist in the upb repo. I still need to add
test runs to exercise the wheels, but this change adds the functionality for
building and uploading them.
PiperOrigin-RevId: 564896637
A couple weeks ago we moved upb into the protobuf Git repo, and this change
continues the merger of the two repos by making them into a single Bazel repo.
This was mostly a matter of deleting upb's WORKSPACE file and fixing up a bunch
of references to reflect the new structure.
Most of the changes are pretty mechanical, but one thing that needed more
invasive changes was the Python script for generating CMakeLists.txt,
make_cmakelists.py. The WORKSPACE file it relied on no longer exists with this
change, so I updated it to hardcode the information it needed from that file.
PiperOrigin-RevId: 564810016
Currently all our upb CI coverage is running from within the protobuf workspace
and referring to upb as the external repo `@upb`. This change makes the GCC
test run cd into upb and use its workspace instead, just to make sure that
builds in the upb workspace have some coverage.
PiperOrigin-RevId: 560197956
This test validates that upb Python targets can be built successfully even if
Python is not installed locally.
I also updated our pinned upb version to pull in some recent fixes needed for
this test run.
PiperOrigin-RevId: 559504790
This change adds test runs for GCC, Windows, and Mac. The 32-bit ASAN test
turned out to be somewhat difficult to migrate, so I left a TODO for that one.
I made sure to give each build environment its own cache prefix, so that we
don't end up with artifacts built by different toolchains conflicting with each
other in the same cache.
PiperOrigin-RevId: 559197655
There are still a bunch more test runs to migrate, but this change adds a few
more that were unblocked by the `exclude-targets` flag.
PiperOrigin-RevId: 558206606
This change moves the upb Fastbuild, Optimized, and FastTable test runs over to
the protobuf repo CI in preparation for moving the upb codebase itself. There
are a bunch more test runs to move, but this initial change handles the easy
ones first.
I also updated our pinned upb version to the current head to pick up some
recent fixes.
PiperOrigin-RevId: 557486174
Protobuf Team Bot
Mike Kruskal
Jie Luo
Sandy Zhang
dependabot[bot]
Deanna Garcia
Hong Shin
Eric Salo
Adam Cozzette