This ports upb's WORKSPACE scraping logic to protobuf, and allows us to dynamically fetch our dependencies at the exact same pinned version as in Bazel via protobuf_FETCH_DEPENDENCIES=ON. This is mostly for development purposes, and is preferable to git submodules. In a later cl we will flip the default behavior to "package"
#test-continuous
PiperOrigin-RevId: 686265348
For cross-compilation, we may not want to re-use the same protoc which would be used during a CMake build (these would differ in hostPlatform).
Instead, allow for invocations of `protobuf_generate` pass their preferred programs, similar to how you can do this for grpc plugins.
Optional proposal:
Use `find_program`, however, this is a pretty big behavior change, this PR allows you to "opt-out" of the legacy behavior, but users shouldn't see a difference unless they opt-in to passing their own protoc-exe command/path
Closes#17888
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/17888 from jonringer:configure-protoc-cmake 9ca38aee11
PiperOrigin-RevId: 678383523
This yields several benefits:
1. The code no longer needs to be bootstrapped (since it no longer depends on upb reflection).
2. The upb code generator no longer depends on libprotobuf at all (except for `code_generator_lite.{h,cc}`, which is just one .cc file and has no deps).
PiperOrigin-RevId: 672280579
If there's an unexpected failure message or an unexpected succeeding test from a wildcard expansion users will be made to remove the wildcarded equivalent. Once removed, they must rerun the conformance test to add the failures contained within the removed wildcarded equivalent.
PiperOrigin-RevId: 663040062
**Description**
Added a cmake option to enable Unity builds for libprotobuf-lite.
Added missing symmetric include for port_def.inc
Added preproc guards for PROTOBUF_PRAGMA_INIT_SEG so it does not get included multiple times in unity build.
**Motivation and Context**
For large code base, especially in the game dev industry, Unity builds (https://en.wikipedia.org/wiki/Unity_build) are used to speed up the compilations.
Special care must be taken to enable Unity builds for a project, since many translation units will be merged togheter.
Closes#16454
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/16454 from jslap-ubi:enable-unity-build a387ffd862
PiperOrigin-RevId: 646581094
The generated `protobuf-config-version.cmake` sets the `Protobuf_WITH_ZLIB`, `Protobuf_MSVC_STATIC_RUNTIME` and `Protobuf_BUILD_SHARED_LIBS` unconditionally while executing the `_check_and_save_build_option(...)` macro.
If there are several **Protobuf** installations available the original code may lead to failure. An example with two installations:
+ A system deployed installation with version `3.12.4`.
+ A repo's thirdparty with version `3.19.4`
The user wants it's project to use the thirdparty one by doing:
```cmake
find_package(Protobuf 3.19.4 CONFIG REQUIRED PATHS path/to/thirdparty)
```
The user is not setting the `Protobuf_XXX` variables in its `CMakeLitsts.txt`.
If the first config processed is the system deployed it sets the variables to its desired variables, for example, `Protobuf_WITH_ZLIB=OFF` but is discarded due to the non-matched version.
Then the thirdparty's config is processed but is discarded because requires `Protobuf_WITH_ZLIB=ON` which is not a real requirement and makes the find operation fail (despite of matching version).
Closes#16654
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/16654 from MiguelBarro:fix/cmake-config 119dd25ecb
PiperOrigin-RevId: 632664341
The only public target here is the edition defaults helper macro, which can be used by external runtimes and plugins. None of this code is C++-specific though, and should be organized higher up. Appropriate aliases are also placed at the top level for public targets
PiperOrigin-RevId: 625392504
For now, these are limited to tests of text-form for delimited fields that locks down our problematic behavior in editions. Follow up changes will adjust the behavior to behave better under editions.
PiperOrigin-RevId: 622211473
When building with `-Dprotobuf_BUILD_SHARED_LIBS=ON`, we currently put all test
protos into their own shared library. PR #15887 seems to be pushing us over a
limit on the number of exported symbols so that this library no longer links
successfully, though.
This change fixes that problem by always building the test protos into a static
library. This should be fine since it's purely for testing and not meant to be
installed. The only things that depend on it are executables, so we don't need
to worry about ODR violations.
PiperOrigin-RevId: 617929396
This installs a upb's static library, headers, and three protoc plugin binaries (upb, upbdefs, upb_minitable). These will now be enabled by default, but can be disabled by setting protobuf_BUILD_LIBUPB=OFF.
To qualify this, we hook into our existing install test infrastructure which attempts to build and run our tests without any of the installation artifacts available from the source tree. Public headers are deleted, and builds of exported libraries/binaries are disabled.
PiperOrigin-RevId: 617376961
My org, as part of its reproducible-build hygiene, builds projects with the source trees in read-only mode. I approached the protobuf build in the same way, but encountered this error (among others):
```
[ 30%] Built target statusor
[ 34%] Built target libprotobuf-lite
[ 45%] Built target libprotobuf
Consolidate compiler generated dependencies of target libprotoc
[ 62%] Built target libprotoc
Consolidate compiler generated dependencies of target protoc
[ 62%] Built target protoc
Consolidate compiler generated dependencies of target scoped_mock_log
[ 62%] Built target scoped_mock_log
[ 62%] Generating /home/src/protobuf/src/google/protobuf/util/message_differencer_unittest_proto3.pb.h, /home/src/protobuf/src/google/protobuf/util/message_differencer_unittest_proto3.pb.cc
/home/src/protobuf/src/google/protobuf/util/message_differencer_unittest_proto3.pb.cc: Read-only file system
make[2]: *** [CMakeFiles/libtest_common.dir/build.make:387: /home/src/protobuf/src/google/protobuf/util/message_differencer_unittest_proto3.pb.h] Error 1
make[1]: *** [CMakeFiles/Makefile2:1194: CMakeFiles/libtest_common.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
```
(`/home/src` is a read-only mount)
Generated source files are effectively build artifacts, and should be written to the build tree, not the source tree. This PR accordingly enables a successful build and test run (lite, full, and conformance) with read-only sources.
(Beyond this, I would add that CMake usually does not need source file paths to be qualified with `CMAKE_SOURCE_DIR` or `CMAKE_BINARY_DIR`; in most cases it knows to look in both locations, favoring the latter if a file is in both. The CMake code could be simplified by relying on this behavior.)
Closes#14455
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/14455 from iskunk:feature/ro-source da7510c24a
PiperOrigin-RevId: 607031010
This caused an issue on some platforms where `protobuf/java/core/src/main/java/com/google/protobuf/java_features.proto` was getting turned into the invalid path `java/core/src/main/java/com/google/java_features.proto`.
Fixes#14862
PiperOrigin-RevId: 586527713
Unlike Arena::CreateMessage, Arena::Create creates only the top level object
from arena even if it is arena constructalble; e.g. messages, RepeatedPtrField,
etc. This renders arenas less effective.
Instead of asking users to be aware of such nuances to use the right API for
the right type, this CL makes Arena::Create recognizes and fully supports arena
constructable types.
While extremly rare, some users try to emulate Arena::CreateMessage with
Arena::Create by passing arena parameter twice. For example,
```
auto foo = Arena::Create<Foo>(&arena, &arena); // bad
```
This pattern is not supported and will break after this change. The following
is recommended instead.
```
auto foo = Arena::CreateMessage<Foo>(&arena); // recommended
auto foo = Arena::Create<Foo>(&arena); // after this change
```
PiperOrigin-RevId: 585709990
We transformed the proto2/proto3 protos to editions, and then run the same set of tests over both. This will verify that migrating to editions preserves the same proto2/proto3 behavior. These will not be enabled by default, and require a flag `--maximum_edition=2023`.
Future changes will:
- add more targeted editions-specific tests
- clean up our conformance test framework to allow for more targeted tests
- add wildcards to failure lists in limited cases to reduce noise
- add feature resolution conformance tests
PiperOrigin-RevId: 574570607
These utilities provide a way to embed a FeatureSetDefaults message into generators or runtimes that need to implement feature resolution. They use protoc to handle the tricky reflection-based algorithm over feature protos, leaving only simple merges to be implemented in other languages. See docs/design/editions/editions-life-of-a-featureset.md for more information.
PiperOrigin-RevId: 574554333
This is likely the cause of our macos cmake flakes. Both the lite and full tests trigger codegen of the lite protos, but there's no dependency between them. This creates a race condition, where it's possible for the files to be invalid by the time one of their builds is triggered. This is consistent with the error message we see:
liblibtest_common_lite.a(unittest_lite.pb.cc.o) has no symbols
PiperOrigin-RevId: 573381442
The current logic of finding a suitable include path relies on CMake to always return a meaningful relative path. On Windows, however, if the two paths reside on different drives (e.g. `c:/Users/Me` and `d:/project/`) the absolute path gets returned unaltered. This yields the invalid final path `c:/Users/Me/d:/project/`.
This case is now detected and the corresponding include path is deemed unsuitable.
(fixes#9290)
Closes#14025
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/14025 from cgudrian:issue-9290-wrong-relative-path-in-protobuf-generate 3c0fd37324
PiperOrigin-RevId: 573315544
This will reduce the number of times our test protos are built, and may fix the flakes we're seeing in aarch64. At the very least, it should reduce them by a factor of 3 and marginally speed up our builds.
PiperOrigin-RevId: 568963559
This works for both built-in generators and plugins written in C++. Any feature extensions specified by this virtual method will end up fully resolved in the descriptors.
PiperOrigin-RevId: 561446773
This will allow us to run the CMake test from within the protobuf repo's Bazel
workspace. The directory structure is a little bit different depending on which
workspace the test is invoked in.
PiperOrigin-RevId: 557275295
I found that this test fails when I try to run it from our protobuf repo CI
setup, probably because rsync is not available in the Docker image there. Let's
just use `cp` instead so that the test works even if rsync is unavailable.
PiperOrigin-RevId: 557225733
Mark Hansen
Protobuf Team Bot
Mike Kruskal
Jonathan Ringer
Joshua Haberman
Yamil Morales
Eduardo Menges Mattje
jslap-ubi
MiguelBarro
Adam Cozzette
h-vetinari
Daniel Richard G
Sandy Zhang
Protobuf Team Bot
Christian Gudrian
Eric Salo