My org, as part of its reproducible-build hygiene, builds projects with the source trees in read-only mode. I approached the protobuf build in the same way, but encountered this error (among others):
```
[ 30%] Built target statusor
[ 34%] Built target libprotobuf-lite
[ 45%] Built target libprotobuf
Consolidate compiler generated dependencies of target libprotoc
[ 62%] Built target libprotoc
Consolidate compiler generated dependencies of target protoc
[ 62%] Built target protoc
Consolidate compiler generated dependencies of target scoped_mock_log
[ 62%] Built target scoped_mock_log
[ 62%] Generating /home/src/protobuf/src/google/protobuf/util/message_differencer_unittest_proto3.pb.h, /home/src/protobuf/src/google/protobuf/util/message_differencer_unittest_proto3.pb.cc
/home/src/protobuf/src/google/protobuf/util/message_differencer_unittest_proto3.pb.cc: Read-only file system
make[2]: *** [CMakeFiles/libtest_common.dir/build.make:387: /home/src/protobuf/src/google/protobuf/util/message_differencer_unittest_proto3.pb.h] Error 1
make[1]: *** [CMakeFiles/Makefile2:1194: CMakeFiles/libtest_common.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
```
(`/home/src` is a read-only mount)
Generated source files are effectively build artifacts, and should be written to the build tree, not the source tree. This PR accordingly enables a successful build and test run (lite, full, and conformance) with read-only sources.
(Beyond this, I would add that CMake usually does not need source file paths to be qualified with `CMAKE_SOURCE_DIR` or `CMAKE_BINARY_DIR`; in most cases it knows to look in both locations, favoring the latter if a file is in both. The CMake code could be simplified by relying on this behavior.)
Closes#14455
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/14455 from iskunk:feature/ro-source da7510c24a
PiperOrigin-RevId: 607031010
In CMake >= 3.0 it is more idiomatic to set per-target compiler options than global compiler settings. I have kept these options and defines as `PRIVATE` so they won't be exported with the target.
Closes#12916
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/12916 from coryan:cleanup-cmake-avoid-global-compile-settings 3d586dc0e8
PiperOrigin-RevId: 536517165
Policy CMP0141 broke our hack to remove /Zi, and there doesn't seem to be a way to strip debug symbols with this policy enabled. We can disable the policy when it exists to get ccache working again and speed up our tests.
PiperOrigin-RevId: 534958215
CMake has fairly strict backwards compatibility behavior. Saying `cmake_minimum_required(VERSION 3.10)` has two effects:
- It requires CMake to be version 3.10 or higher
- It configures CMake to behave exactly as it did in version 3.10
The code used to override some policies to get newer behavior if it was available. The alternative syntax in this PR says "require at least 3.10, and use the behavior (when available) up to 3.26". We can omit policy overrides by doing so. It does not seem like any other policies are important, so we can use the simpler syntax.
Closes#12712
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/12712 from coryan:cleanup-cmake-automatically-use-newer-policies 36db0fa17e
PiperOrigin-RevId: 530376446
Allows the use of an external `jsoncpp` library to be used. Replicates the model used by `abseil-cpp` as a "package" or "module" to the `protobuf_JSONCPP_PROVIDER` option.
Resolves: #11827Closes#12577
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/12577 from omenos:omenos/cmake-jsoncpp 073edd69b4
PiperOrigin-RevId: 528492610
For now, this only covers linux on the two architectures we have testing support for. However, it serves as a good sanity check and can be expanded in the future.
PiperOrigin-RevId: 514449399
For now, this only covers linux on the two architectures we have testing support for. However, it serves as a good sanity check and can be expanded in the future.
PiperOrigin-RevId: 514449399
This change also removes policy settings that are now already set to
`NEW` by default. There are still two remaining policies that were
introduced in CMake > 3.10, so I left those, and documented the CMake
version they were introduced in so we know when we can later remove them
(when our min version advances beyond that).
The min CMake version supported by Protobuf is documented here:
https://github.com/google/oss-policies-info/blob/main/foundational-cxx-support-matrix.md
PiperOrigin-RevId: 511897379
These will still use MSVC as the compiler, but will no longer generate Visual Studio projects for the builds. Visual Studio is particularly bad at parallelizing builds, and is hostile to ccache.
This change also tweaks the ccache setup to prevent unbounded the growth observed in our github caches. Windows builds have had debug symbols stripped to reduce ccache size by a factor of 2x, and ccache maximum have been tweaked so that we persist fewer older builds.
Before this change, each CMake build took 12 minutes on every run (plus some constant overhead from staleness/gcloud), even with caching or on large multi-core runners. No amount of caching or parallelization made any noticeable difference above noise. With this change, we see the following improvements:
- 12 minutes to build from scratch on normal runners (unchanged)
- 4 minutes on 32-core runners from scratch
- 1 minute with optimal caches available on normal runners.
- 30 seconds on 32-core runners with optimal caches
PiperOrigin-RevId: 508799909
These will still use MSVC as the compiler, but will no longer generate Visual Studio projects for the builds. Visual Studio is particularly bad at parallelizing builds, and is hostile to ccache.
This change also tweaks the ccache setup to prevent unbounded the growth observed in our github caches. Windows builds have had debug symbols stripped to reduce ccache size by a factor of 2x, and ccache maximum have been tweaked so that we persist fewer older builds.
Before this change, each CMake build took 12 minutes on every run (plus some constant overhead from staleness/gcloud), even with caching or on large multi-core runners. No amount of caching or parallelization made any noticeable difference above noise. With this change, we see the following improvements:
- 12 minutes to build from scratch on normal runners (unchanged)
- 4 minutes on 32-core runners from scratch
- 1 minute with optimal caches available on normal runners.
- 30 seconds on 32-core runners with optimal caches
PiperOrigin-RevId: 508799909
This uses ccache + github caching to substantially decrease the time it takes to run CMake builds. Due to Bazel caching, these are some of our slowest tests, causing one of the biggest presubmit bottlenecks
PiperOrigin-RevId: 507667813
This deletes our old branches of internal Abseil code in favor of their newly open-sourced library. Notably, this removes the ability to turn CHECK crashes into exceptions.
PiperOrigin-RevId: 504460562
This leaves the decision of which C++ version to use up to our users. We still have a static_assert in port_def.inc that will prevent pre-C++14 usage.
PiperOrigin-RevId: 501351066
--
cc1f708d75 by Joey Parrish <joeyparrish@google.com>:
Fix misdetection of libatomic with strict compiler flags
With strict flags and -Werror, the detection for libatomic breaks, causing it to be included in places where it is not needed. And on macOS, this leads to a link error when libatomic cannot be found.
Closes#10899
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/10900 from joeyparrish:fix-libatomic cc1f708d75
PiperOrigin-RevId: 487859041
This cl hit an issue during the shared library cmake build from ODR violations, leading to mismatched absl hash seeds. The problem was pre-existing but didn't manifest until now, and can be traced to the fact that in shared library builds we linked Abseil statically. All of the cmake changes here remove the underlying ODR violation.
PiperOrigin-RevId: 485787671
Daniel Richard G
Dennis Shao
Kevin King
Jie Luo
Protobuf Team Bot
Protobuf Team Bot
ericsalo
Carlos O'Ryan
Mike Kruskal
deannagarcia
Mike Rochefort
zhangskz
Greg Miller
Joey Parrish