These will still use MSVC as the compiler, but will no longer generate Visual Studio projects for the builds. Visual Studio is particularly bad at parallelizing builds, and is hostile to ccache.
This change also tweaks the ccache setup to prevent unbounded the growth observed in our github caches. Windows builds have had debug symbols stripped to reduce ccache size by a factor of 2x, and ccache maximum have been tweaked so that we persist fewer older builds.
Before this change, each CMake build took 12 minutes on every run (plus some constant overhead from staleness/gcloud), even with caching or on large multi-core runners. No amount of caching or parallelization made any noticeable difference above noise. With this change, we see the following improvements:
- 12 minutes to build from scratch on normal runners (unchanged)
- 4 minutes on 32-core runners from scratch
- 1 minute with optimal caches available on normal runners.
- 30 seconds on 32-core runners with optimal caches
PiperOrigin-RevId: 508799909
Use the bash GHA to ensure WKTs are current, and then just directly invoke
Xcode. This means the script full_mac_build.sh can go back to just being
a simpler helper for folks doing local builds and we can remove all the extra
support that was needed for CI.
PiperOrigin-RevId: 508762464
Changes to these files *can't* be tested in forked PRs, so we should explicitly block them with an error message to explain why.
PiperOrigin-RevId: 508701535
Every where the script is invoked from protoc is already built, and this more
follows the model used by similar scripts for other languages.
PiperOrigin-RevId: 508694936
This fixes a v22 regression for Ruby.
Windows builds of the Ruby package are broken in 3.22.0.rc.2. The reason is, `ruby-upb.h` no longer provides the full path when `#include`ing `utf8_range.h`. See https://github.com/protocolbuffers/protobuf/blob/v22.0-rc2/ruby/ext/google/protobuf_c/ruby-upb.h#L10479 (22.0-rc-2), cf. https://github.com/protocolbuffers/protobuf/blob/v21.12/ruby/ext/google/protobuf_c/ruby-upb.h#L5365 (21.12). The `extconf.rb` build configuration tries to compensate by adding `third_party/utf8_range` to the include path, but does not do so on Windows (i.e. `mingw`) platforms. See https://github.com/protocolbuffers/protobuf/blob/v22.0-rc2/ruby/ext/google/protobuf_c/extconf.rb#L9-L10 (22.0-rc-2), cf. https://github.com/protocolbuffers/protobuf/blob/v21.12/ruby/ext/google/protobuf_c/extconf.rb#L9-L10 (21.12).
We could have fixed this by adding another clause to the if statement for the case `RUBY_PLATFORM =~ /mingw/` and adding the appropriate `-I` flag to `CFLAGS`. However, that `CFLAGS` hack is present in the first place due to a related problem: the usage of `$INCFLAGS` is incorrect.
The `$INCFLAGS` constant in Ruby's `mkmf` is a string in a similar format to `CFLAGS`. It's simply appended to compiler invocations. So when you append new flags to it, you have to (1) provide the flag itself, and (2) precede it with a space to delimit it from the previous entry. In 22.0-rc-2 (and indeed in all earlier versions) the usage is incorrect: it's appending a path to the string without the `-I` and without a space. See https://github.com/protocolbuffers/protobuf/blob/v22.0-rc2/ruby/ext/google/protobuf_c/extconf.rb#L22. Hence, not only does the intended include path not get appended correctly, it also clobbers the previous path in the string. Luckily, the previous path is only `-I$(srcdir)` which happens not to matter for our library. But it does mean that the apparent intent of that line, adding `$(srcdir)/third_party/utf8_range` to the include path, isn't working; hence the code that adds it to `CFLAGS` instead.
(Note that the previous line, adding the path to `$VPATH`, _is_ correct as is, because `$VPATH` is an array.)
So what this PR actually does is fix the `$INCFLAGS` usage so `$(srcdir)/third_party/utf8_range` gets added properly to compiler include paths, for all platforms including Windows. Since that should now be working as intended, we also remove the extra `-I` from CFLAGS. Builds for all platforms should now be able to handle the change to `ruby-upb.h`. This has been tested by running the prototype Ruby build Kokoro job against a patched 22.0-rc-2.
This also needs to be backported to the 22.x branch.
/attn @deannagarciaCloses#11882
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/11882 from dazuma:pr/ruby-builds-fix ebb18e0004
PiperOrigin-RevId: 508550039
objc_library really will only build for Apple platforms, so tag it as such.
The objc conformance test really will only build/run on macOS, so tag it as
such.
With these two changes, it should be safer for recursive builds that include
this directory no matter what the host platform is.
PiperOrigin-RevId: 508438744
Since the sources can't reference each other, this is what helps keep them in
sync. Move it out to be a stand alone test in bazel instead of having it wired
into the Xcode projects.
PiperOrigin-RevId: 508429454
This change introduces a dedicated StringBlock class that hosts block allocated string instances which are destroyed when the arena gets destroyed.
Experiments have shown this to save 0.40% in memory and 0.10% of CPU in arena heavy use cases.
PiperOrigin-RevId: 508205452
- General naming up to "Objective-C" that some linters want, so just
use it every where for consistence.
- Add a bazel macOS config
- Includes the conformance test, so drop it from the Xcode flows.
- Tweak the naming on the Xcode flows so they are a little more clear.
- CocoaPods flows:
- Give them (hopefully) clear naming
- Use the same `image` as the Xcode tests
- Pin the Xcode version to match the Xcode tests
PiperOrigin-RevId: 508156130
This should substantially reduce network flakes, since most builds will no longer need to download external dependencies. Because these caches actually get pretty expensive (~300 MB just from 1 openjdk dependency), we've taken some precautions:
1) The caches only get updated on push events (i.e. post-submit jobs)
2) Only a single job will push a new cache for any os/branch/commit combination. This will be whichever job finishes its bazel run first, and *also* has updates for the cache. Over time, the cache should expand to the full set of transitive dependencies as the quicker jobs stop pushing updates.
3) A clear_caches workflow has been introduced to reset the caches in case they grow too large. This will run monthly for now, but can also be run manually.
PiperOrigin-RevId: 508133044
This CL also includes some cleanups to the core FieldVars function, and eliminates what had long-since become vestigial functions for setting variables. This needs to be part of the same CL because those functions used string_view-keyed maps.
PiperOrigin-RevId: 508124647
It isn't worth the effort to try and compute the correct minimal set for each
sub file, and the more Swift folks use on top of this, the less it matters
as the modules will need everything anyways.
PiperOrigin-RevId: 508122902
This code is experimental and should not be expected to emit working code, and callers are liable to break without warning.
It is being released now so that development can occur in the open, but users should not expect this to be supported any time soon.
PiperOrigin-RevId: 508095929
Move Field & Extension support for lack of UsesClassRef into the legacy api
so the only things taking the hit are old generate code.
PiperOrigin-RevId: 508074967
Add a new generator interface to:
- provide the message name directly instead of the overhead needed to try and compute it
- provide the data for the GPBFileDescriptor as a struct.
This allows -[GPBMessage fullName] to much more easily compute the fullname.
It also will allow the creation of all GPBFileDescriptors to be done on demand thus reducing the work in the startup codepaths as well as reducing memory requirements.
The generation changes are in a following commit.
PiperOrigin-RevId: 508063366
In order to flip the flag, all downstream projects should be adapted. However, it is hard to fix them all if there are constant regressions. Adding it to the CI will ensure that once the project can build with incompatible_disallow_empty_glob it can keep building like that.
See: bazelbuild/bazel#15327
PiperOrigin-RevId: 507927389
With gcc (tested versions: 11.3.0, 12.2.0) compiled with asan, the
`type_id` variable is warned about maybe being used uninitialized in
the later call:
```
ptr = ParseFieldMaybeLazily(static_cast<uint64_t(type_id) * 8 + 2, ptr,
extendee, metadata, ctx);
```
This change initializes the variable to a defined default.
PiperOrigin-RevId: 507883964
Mike Kruskal
Protobuf Team Bot
Sandy Zhang
Andrei Tcaci
Martijn Vels
Daniel Azuma
Protobuf Team Bot
Xavier Bonaventura
James Brock