This places all of the tricky reflection logic of feature resolution into a single location that outputs a portable protobuf message. With this message, feature resolution becomes drastically simpler and easy to reimplement in other languages.
Follow-up changes will provide:
- An API to specify generator features from the CodeGenerator class for C++ plugins
- A CLI for building these default mappings
- Bazel/CMake rules to help embed these mappings in target languages (i.e. for runtimes and non-C++ plugins)
PiperOrigin-RevId: 559615720
Disable the alignment check in 32-bit msvc.
This toolchain has a difference between "natural" and "required" alignment of
types and it can have the alignment of members of type `T` to be smaller than
`alignof(T)`.
Also, disable AnyTest.TestPackFromSerializationExceedsSizeLimit there because it can't allocate that much memory.
PiperOrigin-RevId: 559495544
This makes a few changes:
- It changes generated messages to reference message innards as a type in `__runtime` instead of branching on what fields should be there. That results in much less bifurcation in gencode and lets runtime-agnostic code reference raw message innards.
- It adds a generic mechanism for creating vtable-based mutators. These vtables point to thunks generated for interacting with C++ or upb fields. Right now, the design results in 2-word (msg+vtable) mutators for C++ and 3-word mutators (msg+arena+vtable) for UPB. See upb.rs for an explanation of the design options. I chose the `RawMessage+&Arena` design for mutator data as opposed to a `&MessageInner` design because it did not result in extra-indirection layout changes for message mutators. We could revisit this in the future with performance data, since this results in all field mutators being 3 words large instead of the register-friendly 2 words.
- And lastly, as a nearby change that touches on many of the same topics, it adds some extra SAFETY comments for Send/Sync in message gencode.
PiperOrigin-RevId: 559483437
This fixes Python/C++ and upb, and pushes the buggy behavior to pure python. There, it's very difficult to handle options on the bootstrapped proto with the current architecture. Future changes will attempt to address this more isolated issue.
PiperOrigin-RevId: 559450900
This will be used to seed feature resolution, which becomes just an edition lookup followed by proto merges. Runtime and generators that need to resolve their own features will be able to leverage this to avoid reimplementing all of the reflective logic that goes into the default calculation.
PiperOrigin-RevId: 559271634
The issue is that if another component (MediaPipe in this case) calls PackFrom, there'll be linker errors because kTypeGoogleApisComPrefix wasn't found.
PiperOrigin-RevId: 559132962
This new option will cause only the files that provide types to
be imported into the generated code. This helps reduce the inputs
that aren't needed (folks ignoring the protoc warnings) and for
things imported only for custom options as the objc generated code
doesn't capture those and thus the imports aren't needed in the
generated code.
Also do some general import cleanups.
PiperOrigin-RevId: 558867748
Implementing this method allows descriptors to be used in contexts that support
ABSL's formatting, most notably gunit's PrintToString and absl's StrFormat. The
implementations are done on the concrete descriptor type, which appears to be more
idiomatic for AbslStringify. Implementing them on pointers-to-descriptors would be
nicer for users who typically interact with pointers, but that would conflict with
AbslStringify's built-in support for pointers.
The implementation simply forwards to DebugString for simplicity.
PiperOrigin-RevId: 558854354
Arena enabled copy construction provides efficiency gains for the rather common construct where a message is currently copied by first creating a default initialized instance through the regular (arena) constructor, and then recursively filled from the source message using 'MergeFrom' or 'CopyFrom'.
Arena enabled copy construction is feature gated in port_def.inc and currently not enabled by default.
PiperOrigin-RevId: 558854125
It ended up being odd to construct the AccessorGenerators ahead of time given some specific Context<>: since each of the three methods on AccessorGenerators demands a different Context than the initial one so they shouldn't hold that Context anyway. It ended up just being a spooky parallel array to the Fields with an unenforced invariant that all methods would be called with a new Context<> but the same underlying FieldDescriptor later.
PiperOrigin-RevId: 558770643
The new naming will use our new vocabulary, and restrict feature access:
Unresolved source features - source-retention features that generators need to validate their own features. These should not be sent to the runtime or used for any decision-making.
Resolved source features - fully resolved source-retention features. These should be used to make codegen decisions, but shouldn't be sent to runtimes.
PiperOrigin-RevId: 558287302
This allows the compiler to statically detect use-after-free bugs.
This change touches a subset of field types. More changes to follow.
PiperOrigin-RevId: 558180721
This version is a light-weight, inlined version implemented in terms of `MergeFrom` that extends the protocol buffer API with arena enabled copy construction, but without any explicit (optimized) implementation.
Going forward the protocol compiler will provide an optimized implementation that is expected to save a non trivial amount of compute.
PiperOrigin-RevId: 558124029
Since protoc can't know the full set of features that matter to a plugin, sending partially resolved features is unnecessarily confusing. Plugins written in C++ will have automatic access to the fully resolved features via previous enhancements. Plugins written in other languages will need to reimplement feature resolution logic. Future changes will document this algorithm and provide conformance testing.
PiperOrigin-RevId: 557969789
TDP replaced most of the class' use. MapFieldLite does not need it anymore.
We can also simplify MapFieldLite by removing some now unused template parameters.
PiperOrigin-RevId: 557599609
Notably, @jhump identified the following holes:
- Non-messages can have message_encoding set
- Extensions can have implicit presence set
- Repeated fields can have explicit presence set
PiperOrigin-RevId: 557571375
Every language has very different handling of utf8 validation. Any with proto2/proto3 differences will receive language-specific features for edition zero to better model these subtle differences.
PiperOrigin-RevId: 557258923
This helps make the API more complete, since the FeatureSet object will always be fully resolved on any accessible features. This specifically targets C++ plugins though, which will now have their features filled in by default. Before, any proto files that didn't include the language-specific features would result in unresolved extensions in the generators.
PiperOrigin-RevId: 557232654
This addresses #13441. Second try, now with more internal fixes.
This preserves the similar check at the _point of use_ of invalid messages in `DescriptorBuilder` (and there's an existing test that verifies that check still works).
But it adds another check in the parser, to catch this error at the _point of definition_ of an invalid message. And the corresponding test is updated: we no longer need a usage of the message to catch the error, and the reported position is the definition of the option, not the usage site of the message.
The way this works feels kinda gross, but I wasn't sure of a better way to do it. The only place we know for certain that it was an explicit option (vs. auto-added by the parser when synthesizing a map entry message) is when after processing the message body, we can look at the uninterpreted options. So that's what this does. If you have ideas on better/cleaner approaches, I'd be happy to revise.
Closes#13479
PiperOrigin-RevId: 557199190
This avoids a call to mutable(), and results in smaller code, branching on the field being not default, which should be a well predicted, rarely taken branch as the probability of a split being instantiated with more than 1 field is very low.
PiperOrigin-RevId: 557132505
This will replace the frontend string_field_validation field, which we've discovered can't accurately model all of the unique behaviors in each language.
PiperOrigin-RevId: 556996541
This addresses #13442. This also backfills the tests to add tests for the two checks that were already implemented as well as one for the newly added check.
Finally, this fixes the location information so that positions for reserved ranges are correctly reported. (The previous check that already existed, for enums, failed to show line and column info.)
Closes#13474
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/13474 from jhump:jh/validate-msg-reserved-range-order 661adc745a
PiperOrigin-RevId: 556979032
Protobuf Team Bot
Mike Kruskal
Alyssa Haroldsen
Thomas Van Lenten
pizzud
Martijn Vels
Cong Liu
Protobuf Team Bot
Jonas Kvinge
Sandy Zhang
Matt Kulukundis
Joshua Humphries