This change moves almost everything in the `upb/` directory up one level, so
that for example `upb/upb/generated_code_support.h` becomes just
`upb/generated_code_support.h`. The only exceptions I made to this were that I
left `upb/cmake` and `upb/BUILD` where they are, mostly because that avoids
conflict with other files and the current locations seem reasonable for now.
The `python/` directory is a little bit of a challenge because we had to merge
the existing directory there with `upb/python/`. I made `upb/python/BUILD` into
the BUILD file for the merged directory, and it effectively loads the contents
of the other BUILD file via `python/build_targets.bzl`, but I plan to clean
this up soon.
PiperOrigin-RevId: 568651768
A couple weeks ago we moved upb into the protobuf Git repo, and this change
continues the merger of the two repos by making them into a single Bazel repo.
This was mostly a matter of deleting upb's WORKSPACE file and fixing up a bunch
of references to reflect the new structure.
Most of the changes are pretty mechanical, but one thing that needed more
invasive changes was the Python script for generating CMakeLists.txt,
make_cmakelists.py. The WORKSPACE file it relied on no longer exists with this
change, so I updated it to hardcode the information it needed from that file.
PiperOrigin-RevId: 564810016
This is the second attempt to fix our Git history. This should allow
"git blame" to work correctly in the upb/ directory even though our
automation unexpectedly blew away that directory.
After this change, `mini_table` only has MiniTable definitions themselves. Everything having to do with the MiniDescriptor wire format is in `mini_descriptor`.
Also rearranged some of the files in mini_table to have better structure for `internal/`.
This CL contains no functional change.
PiperOrigin-RevId: 543529112
The new API upb_MiniTable_Link() links all sub-messages and sub-enums at a single time, by accepting an array of sub-tables and sub-enums. The order of these sub-tables can be queried using a separate function `upb_MiniTable_GetSubList()`, and this information is added to `CodeGeneratorRequest` as part of the upb-specific info.
PiperOrigin-RevId: 513970874
The upb convention is that "_Build()" means to also allocate, which this function does not do, so rename it as "_Init()" to free up the name for a future function that does allocate.
PiperOrigin-RevId: 510282736
Currently these functions are hardwired to always return true, but the upstream
code now checks for failures (which will be implemented soon).
PiperOrigin-RevId: 504943663
_upb_MiniTable_Build() is now the general version of the function;
upb_MiniTable_Build() calls it and sets the platform to default/native.
PiperOrigin-RevId: 491091021
upb_MiniTable_BuildEnum() -> upb_MiniTableEnum_Build()
upb_MiniTable_BuildExtension() -> upb_MiniTableExtension_Build()
also make the status pointer argument optional for the mini table builders
PiperOrigin-RevId: 490992866
Prior to this CL, there were several different code paths for reading/writing message data. Generated code, MiniTable accessors, and reflection all performed direct manipulation of the bits and bytes in a message, but they all had distinct implementations that did not share much of any code. This divergence meant that they could easily have different behavior, bugs could creep into one but not another, and we would need three different sets of tests to get full test coverage. This also made it very difficult to change the internal representation in any way, since it would require updating many places in the code.
With this CL, the three different APIs for accessing message data now all share a common set of functions. The common functions all take a `upb_MiniTableField` as the canonical description of a field's type and layout. The lowest-level functions are very branchy, as they must test for every possible variation in the field type (field vs oneof, hasbit vs no-hasbit, different field sizes, whether a nonzero default value exists, extension vs. regular field), however these functions are declared inline and designed to be very optimizable when values are known at compile time.
In generated accessors, for example, we can declare constant `upb_MiniTableField` instances so that all values can constant-propagate, and we can get fully specialized code even though we are calling a generic function. On the other hand, when we use the generic functions from reflection, we get runtime branches since values are not known at compile time. But even the function is written to still be as efficient as possible even when used from reflection. For example, we use memcpy() calls with constant length so that the compiler can optimize these into inline loads/stores without having to make an out-of-line call to memcpy().
In this way, this CL should be a benefit to both correctness and performance. It will also make it easier to change the message representation, for example to optimize the encoder by giving hasbits to all fields.
Note that we have not completely consolidated all access in this CL:
1. Some functions outside of get/set such as clear and hazzers are not yet unified.
2. The encoder and decoder still touch the message without going through the common functions. The encoder and decoder require a bit more specialized code to get good performance when reading/writing fields en masse.
PiperOrigin-RevId: 490016095
Remove circular dependencies that were bouncing back and forth between
msg_internal.h and mini_table/, including:
- splitting out each mini table subtype into its own header
- moving the non-reflection message code into message/
- moving the accessors from mini_table/ to message/
PiperOrigin-RevId: 489121042
We need to sharpen the distinction between messages and extensions in the mini
descriptor encoder, so split the code paths for each.
PiperOrigin-RevId: 480675339
Prior to this CL, users were relying on `field->descriptortype` to get the field type. This almost works, as `field->descriptortype` is almost, but not quite, the field type of the field. In two special cases we deviate from the true field type, for ease of parsing and serialization:
- For open enums, we use `kUpb_FieldType_Int32` instead of `kUpb_FieldType_Enum`, because from the perspective of the wire format, an open enum field is equivalent to int32.
- For proto2 strings, we use `kUpb_FieldType_Bytes` instead of `kUpb_FieldType_String`, because proto2 strings do not perform UTF-8 validation, which makes them equivalent to bytes.
In this CL we add a public API function:
```
// Returns the true field type for this field.
upb_FieldType upb_MiniTableField_Type(const upb_MiniTable_Field* f);
```
This will provide the actual field type for this field.
Note that this CL changes the MiniDescriptor format. Previously MiniDescriptors did not contain enough information to distinguish between Enum/Int32. To remedy this we added a new encoded field type, `kUpb_EncodedType_ClosedEnum`.
PiperOrigin-RevId: 479387672
Optimizes `upb_MiniTable_Enum` for enums with many values (>64) but with relatively dense packing in numeric space.
This CL optimizes both the size and speed of such enums:
- size: 30x code size reduction
- speed: moved from linear search to a constant-time bit test
Negative enum values are still expensive, as they are never put into the bitfield.
PiperOrigin-RevId: 473259819
An enum MiniDescriptor simply encodes a set of valid `int32_t` values, so that the protobuf parser can test whether a given enum value is known or not.
The format implemented here is novel and needs to be documented. In short, the format is:
1. base92 values 0-31: 5-bit mask indicating presence or absence of the next five enum values.
2. base92 values 60-91: varint indicating skip over a region of enum values.
Negative enum values are encoded as their `uint32_t` equivalent.
PiperOrigin-RevId: 442892799
Adam Cozzette
Hong Shin
Protobuf Team Bot
Joshua Haberman
Mike Kruskal
Eric Salo
Protobuf Team
Joshua Haberman