This required some work to unify map entry messages with regular messages, with respect to presence. Before map entry fields could never have presence. Now they can have presence according to normal rules. Note that this only applies to times that the user constructs a map entry directly.
PiperOrigin-RevId: 490611656
Prior to this CL, there were several different code paths for reading/writing message data. Generated code, MiniTable accessors, and reflection all performed direct manipulation of the bits and bytes in a message, but they all had distinct implementations that did not share much of any code. This divergence meant that they could easily have different behavior, bugs could creep into one but not another, and we would need three different sets of tests to get full test coverage. This also made it very difficult to change the internal representation in any way, since it would require updating many places in the code.
With this CL, the three different APIs for accessing message data now all share a common set of functions. The common functions all take a `upb_MiniTableField` as the canonical description of a field's type and layout. The lowest-level functions are very branchy, as they must test for every possible variation in the field type (field vs oneof, hasbit vs no-hasbit, different field sizes, whether a nonzero default value exists, extension vs. regular field), however these functions are declared inline and designed to be very optimizable when values are known at compile time.
In generated accessors, for example, we can declare constant `upb_MiniTableField` instances so that all values can constant-propagate, and we can get fully specialized code even though we are calling a generic function. On the other hand, when we use the generic functions from reflection, we get runtime branches since values are not known at compile time. But even the function is written to still be as efficient as possible even when used from reflection. For example, we use memcpy() calls with constant length so that the compiler can optimize these into inline loads/stores without having to make an out-of-line call to memcpy().
In this way, this CL should be a benefit to both correctness and performance. It will also make it easier to change the message representation, for example to optimize the encoder by giving hasbits to all fields.
Note that we have not completely consolidated all access in this CL:
1. Some functions outside of get/set such as clear and hazzers are not yet unified.
2. The encoder and decoder still touch the message without going through the common functions. The encoder and decoder require a bit more specialized code to get good performance when reading/writing fields en masse.
PiperOrigin-RevId: 490016095
update upb to use upb_Map_Next()
remove upb_MapIterator_SetValue(), which was declared but not actually implemented
remove upb_MapIterator_Done(), which was implemented but not actually used
PiperOrigin-RevId: 489989481
Remove circular dependencies that were bouncing back and forth between
msg_internal.h and mini_table/, including:
- splitting out each mini table subtype into its own header
- moving the non-reflection message code into message/
- moving the accessors from mini_table/ to message/
PiperOrigin-RevId: 489121042
- replace all instances of the deprecated iterator with the much nicer new one
- fix a bug which caused the new iterator to skip all values in the hash array
- fix a bug which caused the new iterator to skip the first value in the hash table
- delete the old iterator code
- also replace most uses of the deprecated string hash table iterator
PiperOrigin-RevId: 489093240
The next lowest build target to scrub is the hash table. We already have a few
other things called 'table' (mini table, fast table) so let's just go with
'hash' here. Split apart the headers into int and str branches sharing common
definitions. Leave the core functions in a single .c for inlining.
PiperOrigin-RevId: 488388767
Proto3 optional fields should use a hasbit for their presence. But we had been giving them oneof layouts, which makes them unnecessarily large. This CL will shrink messages with proto3 optional fields, by using hasbits instead of oneof cases for them.
PiperOrigin-RevId: 485998527
We need to sharpen the distinction between messages and extensions in the mini
descriptor encoder, so split the code paths for each.
PiperOrigin-RevId: 480675339
- upb_DefPool_FindExtensionByMiniTable() has been promoted to a public function.
- There are now zero uses of the internal reflection headers outside of upb core (but still a few left in generated code).
PiperOrigin-RevId: 480211522
We had _upb_Message_New(), which created a message from a mini table and was
being used outside of upb even though it is an internal-only function.
We also had upb_Message_New(), which created a message from a message def.
Now there is a single public function, upb_Message_New(), which creates a
message from a mini table and covers all use cases. (The internal version has the same definition and is used for inlining.)
PiperOrigin-RevId: 480169804
Prior to this CL, users were relying on `field->descriptortype` to get the field type. This almost works, as `field->descriptortype` is almost, but not quite, the field type of the field. In two special cases we deviate from the true field type, for ease of parsing and serialization:
- For open enums, we use `kUpb_FieldType_Int32` instead of `kUpb_FieldType_Enum`, because from the perspective of the wire format, an open enum field is equivalent to int32.
- For proto2 strings, we use `kUpb_FieldType_Bytes` instead of `kUpb_FieldType_String`, because proto2 strings do not perform UTF-8 validation, which makes them equivalent to bytes.
In this CL we add a public API function:
```
// Returns the true field type for this field.
upb_FieldType upb_MiniTableField_Type(const upb_MiniTable_Field* f);
```
This will provide the actual field type for this field.
Note that this CL changes the MiniDescriptor format. Previously MiniDescriptors did not contain enough information to distinguish between Enum/Int32. To remedy this we added a new encoded field type, `kUpb_EncodedType_ClosedEnum`.
PiperOrigin-RevId: 479387672
- Each def type has its own .c file and its own .h file
- Functions that require a builder context are declared in def_builder.h
- The mini descriptor encoders have also been pulled into upb/reflection/
- upb/def.h, upb/def.hpp, upb/reflection.h, and upb/reflection.hpp are now deprecated stubs that point to the new headers
PiperOrigin-RevId: 474459500
Joshua Haberman
Eric Salo
Protobuf Team Bot