protobuf

Commit Graph

Author	SHA1	Message	Date
Eric Salo	6f68ba1350	upb: fail if a message set contains any sub-fields This was causing fuzzer test failures. Unrelated: inlined a few calls to upb_MessageDef_FullName() for performance PiperOrigin-RevId: 483389910	2 years ago
Eric Salo	067c7aa3f1	check for malformed map entries PiperOrigin-RevId: 481258673	2 years ago
Eric Salo	36ce2fa7d1	add version/tag chars to the start of all mini descriptors Verified during decoding. The specific values are just placeholders for now. PiperOrigin-RevId: 481009599	2 years ago
Eric Salo	df34b04658	make upb_ExtensionRegistry_AddArray() and upb_ExtensionRegistry_Lookup() public Previously these were internal _upb functions but it turns out that they are also useful outside of upb. PiperOrigin-RevId: 480997511	2 years ago
Eric Salo	40998462d6	add upb_MtDataEncoder_EncodeExtension() We need to sharpen the distinction between messages and extensions in the mini descriptor encoder, so split the code paths for each. PiperOrigin-RevId: 480675339	2 years ago
Eric Salo	85bf4677b2	lock down internal reflection - upb_DefPool_FindExtensionByMiniTable() has been promoted to a public function. - There are now zero uses of the internal reflection headers outside of upb core (but still a few left in generated code). PiperOrigin-RevId: 480211522	2 years ago
Eric Salo	41335a03be	normalize upb_Message_New() We had _upb_Message_New(), which created a message from a mini table and was being used outside of upb even though it is an internal-only function. We also had upb_Message_New(), which created a message from a message def. Now there is a single public function, upb_Message_New(), which creates a message from a mini table and covers all use cases. (The internal version has the same definition and is used for inlining.) PiperOrigin-RevId: 480169804	2 years ago
Joshua Haberman	b33fd88ed3	Added function for getting the type of a MiniTable field Prior to this CL, users were relying on `field->descriptortype` to get the field type. This almost works, as `field->descriptortype` is almost, but not quite, the field type of the field. In two special cases we deviate from the true field type, for ease of parsing and serialization: - For open enums, we use `kUpb_FieldType_Int32` instead of `kUpb_FieldType_Enum`, because from the perspective of the wire format, an open enum field is equivalent to int32. - For proto2 strings, we use `kUpb_FieldType_Bytes` instead of `kUpb_FieldType_String`, because proto2 strings do not perform UTF-8 validation, which makes them equivalent to bytes. In this CL we add a public API function: ``` // Returns the true field type for this field. upb_FieldType upb_MiniTableField_Type(const upb_MiniTable_Field* f); ``` This will provide the actual field type for this field. Note that this CL changes the MiniDescriptor format. Previously MiniDescriptors did not contain enough information to distinguish between Enum/Int32. To remedy this we added a new encoded field type, `kUpb_EncodedType_ClosedEnum`. PiperOrigin-RevId: 479387672	2 years ago
Joshua Haberman	185d4f09d9	Simplified extension table building slightly by avoiding direct mutation PiperOrigin-RevId: 478953396	2 years ago
Eric Salo	44916d7d27	create reflection_internal library internal declarations are now physically removed from the public headers PiperOrigin-RevId: 478921131	2 years ago
Eric Salo	52c1598554	add upb_EnumDef_IsClosed() PiperOrigin-RevId: 478895160	2 years ago
Joshua Haberman	5d0833f48c	Removed obsolete exemption for closed enums The referenced bug was fixed long ago: https://github.com/protocolbuffers/upb/issues/541 PiperOrigin-RevId: 478870590	2 years ago
Eric Salo	0f585c69af	stop requiring extension fields to have a synthetic oneof https://github.com/protocolbuffers/upb/issues/812 PiperOrigin-RevId: 478527073	2 years ago
Joshua Haberman	128ac1c935	Bugfix for when UPB_TREAT_PROTO2_ENUMS_LIKE_PROTO3 is being used This is not easy to test in google3 since we do not use the flag in Google3. It is only used for Ruby. PiperOrigin-RevId: 478071539	2 years ago
Eric Salo	b8bec58e01	pull the mini descriptor encoders into their proper .c files Performance neutral but it simplifies the code, shrinks the public surface, and makes logical sense PiperOrigin-RevId: 478038589	2 years ago
Joshua Haberman	d5bd55cde1	Treat unlinked sub-messages in the MiniTable as unknown This is an observable behavior change in the decoder. After submitting this CL, clients of the decoder can assume that any unlinked sub-messages will be treated as unknown, rather than crashing. Unlinked sub-messages must never have values present in the message. We can verify this with asserts. Since the values are never set, the encoder should never encounter data for any unlinked sub-message. ``` name old cpu/op new cpu/op delta BM_ArenaOneAlloc 18.3ns ± 9% 17.9ns ± 2% ~ (p=0.690 n=5+5) BM_ArenaInitialBlockOneAlloc 6.40ns ± 1% 6.68ns ±10% ~ (p=0.730 n=4+5) BM_LoadAdsDescriptor_Upb<NoLayout> 5.09ms ± 2% 5.03ms ± 3% ~ (p=0.222 n=5+5) BM_LoadAdsDescriptor_Upb<WithLayout> 5.45ms ± 3% 5.43ms ± 1% ~ (p=0.905 n=5+4) BM_LoadAdsDescriptor_Proto2<NoLayout> 10.9ms ± 1% 10.8ms ± 1% -1.09% (p=0.016 n=5+4) BM_LoadAdsDescriptor_Proto2<WithLayout> 11.3ms ± 9% 11.1ms ± 3% ~ (p=0.841 n=5+5) BM_Parse_Upb_FileDesc<UseArena, Copy> 11.2µs ± 3% 11.3µs ± 3% ~ (p=0.222 n=5+5) BM_Parse_Upb_FileDesc<UseArena, Alias> 10.3µs ± 5% 10.5µs ± 5% ~ (p=0.310 n=5+5) BM_Parse_Upb_FileDesc<InitBlock, Copy> 11.4µs ±18% 11.0µs ± 2% ~ (p=1.000 n=5+5) BM_Parse_Upb_FileDesc<InitBlock, Alias> 10.5µs ±17% 10.6µs ±19% ~ (p=0.421 n=5+5) BM_Parse_Proto2<FileDesc, NoArena, Copy> 20.5µs ± 2% 20.2µs ± 2% ~ (p=0.222 n=5+5) BM_Parse_Proto2<FileDesc, UseArena, Copy> 10.8µs ± 2% 10.9µs ± 4% ~ (p=0.841 n=5+5) BM_Parse_Proto2<FileDesc, InitBlock, Copy> 10.5µs ± 3% 10.6µs ± 3% ~ (p=0.690 n=5+5) BM_Parse_Proto2<FileDescSV, InitBlock, Alias> 9.22µs ± 2% 9.23µs ± 3% ~ (p=1.000 n=5+5) BM_SerializeDescriptor_Proto2 6.05µs ± 3% 5.90µs ± 3% ~ (p=0.222 n=5+5) BM_SerializeDescriptor_Upb 10.2µs ± 3% 10.6µs ±14% ~ (p=0.841 n=5+5) name old time/op new time/op delta BM_ArenaOneAlloc 18.3ns ± 9% 17.9ns ± 2% ~ (p=0.841 n=5+5) BM_ArenaInitialBlockOneAlloc 6.42ns ± 1% 6.69ns ±10% ~ (p=0.730 n=4+5) BM_LoadAdsDescriptor_Upb<NoLayout> 5.10ms ± 2% 5.05ms ± 3% ~ (p=0.222 n=5+5) BM_LoadAdsDescriptor_Upb<WithLayout> 5.47ms ± 3% 5.45ms ± 1% ~ (p=0.905 n=5+4) BM_LoadAdsDescriptor_Proto2<NoLayout> 10.9ms ± 1% 10.8ms ± 1% -1.11% (p=0.016 n=5+4) BM_LoadAdsDescriptor_Proto2<WithLayout> 11.4ms ± 9% 11.1ms ± 3% ~ (p=0.841 n=5+5) BM_Parse_Upb_FileDesc<UseArena, Copy> 11.2µs ± 3% 11.3µs ± 3% ~ (p=0.222 n=5+5) BM_Parse_Upb_FileDesc<UseArena, Alias> 10.3µs ± 5% 10.5µs ± 5% ~ (p=0.151 n=5+5) BM_Parse_Upb_FileDesc<InitBlock, Copy> 11.5µs ±18% 11.0µs ± 2% ~ (p=1.000 n=5+5) BM_Parse_Upb_FileDesc<InitBlock, Alias> 10.5µs ±17% 10.7µs ±19% ~ (p=0.421 n=5+5) BM_Parse_Proto2<FileDesc, NoArena, Copy> 20.6µs ± 2% 20.3µs ± 2% ~ (p=0.222 n=5+5) BM_Parse_Proto2<FileDesc, UseArena, Copy> 10.9µs ± 2% 10.9µs ± 4% ~ (p=0.841 n=5+5) BM_Parse_Proto2<FileDesc, InitBlock, Copy> 10.6µs ± 3% 10.6µs ± 3% ~ (p=0.690 n=5+5) BM_Parse_Proto2<FileDescSV, InitBlock, Alias> 9.24µs ± 2% 9.25µs ± 3% ~ (p=1.000 n=5+5) BM_SerializeDescriptor_Proto2 6.07µs ± 3% 5.91µs ± 3% ~ (p=0.222 n=5+5) BM_SerializeDescriptor_Upb 10.3µs ± 3% 10.6µs ±14% ~ (p=0.841 n=5+5) name old INSTRUCTIONS/op new INSTRUCTIONS/op delta BM_ArenaOneAlloc 201 ± 0% 201 ± 0% ~ (p=0.841 n=5+5) BM_ArenaInitialBlockOneAlloc 69.0 ± 0% 69.0 ± 0% ~ (all samples are equal) BM_LoadAdsDescriptor_Upb<NoLayout> 33.9M ± 0% 34.1M ± 0% +0.66% (p=0.008 n=5+5) BM_LoadAdsDescriptor_Upb<WithLayout> 35.6M ± 0% 35.8M ± 0% +0.64% (p=0.008 n=5+5) BM_LoadAdsDescriptor_Proto2<NoLayout> 70.8M ± 0% 70.8M ± 0% ~ (p=0.548 n=5+5) BM_LoadAdsDescriptor_Proto2<WithLayout> 71.6M ± 0% 71.6M ± 0% ~ (p=0.151 n=5+5) BM_Parse_Upb_FileDesc<UseArena, Copy> 137k ± 0% 141k ± 0% +2.87% (p=0.008 n=5+5) BM_Parse_Upb_FileDesc<UseArena, Alias> 125k ± 0% 128k ± 0% +2.83% (p=0.008 n=5+5) BM_Parse_Upb_FileDesc<InitBlock, Copy> 135k ± 0% 139k ± 0% +2.89% (p=0.008 n=5+5) BM_Parse_Upb_FileDesc<InitBlock, Alias> 124k ± 0% 127k ± 0% +2.85% (p=0.016 n=5+4) BM_Parse_Proto2<FileDesc, NoArena, Copy> 201k ± 0% 201k ± 0% ~ (p=0.222 n=5+5) BM_Parse_Proto2<FileDesc, UseArena, Copy> 107k ± 0% 107k ± 0% ~ (p=1.000 n=5+5) BM_Parse_Proto2<FileDesc, InitBlock, Copy> 105k ± 0% 105k ± 0% ~ (p=0.286 n=5+4) BM_Parse_Proto2<FileDescSV, InitBlock, Alias> 86.5k ± 0% 86.5k ± 0% ~ (p=0.222 n=5+5) BM_SerializeDescriptor_Proto2 60.3k ± 0% 60.3k ± 0% ~ (p=0.071 n=5+5) BM_SerializeDescriptor_Upb 111k ± 0% 111k ± 0% ~ (p=0.841 n=5+5) name old CYCLES/op new CYCLES/op delta BM_ArenaOneAlloc 60.0 ± 7% 58.8 ± 0% -2.15% (p=0.016 n=5+5) BM_ArenaInitialBlockOneAlloc 21.0 ± 0% 21.0 ± 0% ~ (p=1.000 n=5+5) BM_LoadAdsDescriptor_Upb<NoLayout> 16.9M ± 0% 16.9M ± 0% ~ (p=0.056 n=5+5) BM_LoadAdsDescriptor_Upb<WithLayout> 17.9M ± 1% 18.0M ± 1% ~ (p=0.095 n=5+5) BM_LoadAdsDescriptor_Proto2<NoLayout> 35.9M ± 1% 35.8M ± 1% ~ (p=0.421 n=5+5) BM_LoadAdsDescriptor_Proto2<WithLayout> 36.5M ± 0% 36.5M ± 0% ~ (p=0.841 n=5+5) BM_Parse_Upb_FileDesc<UseArena, Copy> 37.2k ± 0% 37.3k ± 0% ~ (p=0.222 n=5+5) BM_Parse_Upb_FileDesc<UseArena, Alias> 34.1k ± 0% 34.7k ± 0% +1.66% (p=0.008 n=5+5) BM_Parse_Upb_FileDesc<InitBlock, Copy> 36.4k ± 0% 36.7k ± 0% +0.83% (p=0.008 n=5+5) BM_Parse_Upb_FileDesc<InitBlock, Alias> 33.3k ± 1% 34.1k ± 1% +2.39% (p=0.008 n=5+5) BM_Parse_Proto2<FileDesc, NoArena, Copy> 68.1k ± 1% 68.0k ± 1% ~ (p=0.421 n=5+5) BM_Parse_Proto2<FileDesc, UseArena, Copy> 36.0k ± 1% 36.1k ± 1% ~ (p=0.841 n=5+5) BM_Parse_Proto2<FileDesc, InitBlock, Copy> 35.3k ± 1% 35.5k ± 1% ~ (p=0.151 n=5+5) BM_Parse_Proto2<FileDescSV, InitBlock, Alias> 30.7k ± 0% 30.9k ± 1% ~ (p=0.151 n=5+5) BM_SerializeDescriptor_Proto2 20.3k ± 2% 19.7k ± 3% ~ (p=0.151 n=5+5) BM_SerializeDescriptor_Upb 33.6k ± 0% 33.7k ± 2% ~ (p=1.000 n=5+5) name old allocs/op new allocs/op delta BM_ArenaOneAlloc 1.19 ± 0% 1.19 ± 0% ~ (all samples are equal) BM_ArenaInitialBlockOneAlloc 0.19 ± 0% 0.19 ± 0% ~ (all samples are equal) BM_LoadAdsDescriptor_Upb<NoLayout> 6.00k ± 0% 6.00k ± 0% ~ (all samples are equal) BM_LoadAdsDescriptor_Upb<WithLayout> 5.99k ± 0% 5.99k ± 0% ~ (all samples are equal) BM_LoadAdsDescriptor_Proto2<NoLayout> 77.8k ± 0% 77.8k ± 0% ~ (all samples are equal) BM_LoadAdsDescriptor_Proto2<WithLayout> 79.0k ± 0% 79.0k ± 0% ~ (all samples are equal) BM_Parse_Upb_FileDesc<UseArena, Copy> 7.19 ± 0% 7.19 ± 0% ~ (all samples are equal) BM_Parse_Upb_FileDesc<UseArena, Alias> 7.19 ± 0% 7.19 ± 0% ~ (all samples are equal) BM_Parse_Upb_FileDesc<InitBlock, Copy> 0.19 ± 0% 0.19 ± 0% ~ (all samples are equal) BM_Parse_Upb_FileDesc<InitBlock, Alias> 0.19 ± 0% 0.19 ± 0% ~ (all samples are equal) BM_Parse_Proto2<FileDesc, NoArena, Copy> 765 ± 0% 765 ± 0% ~ (all samples are equal) BM_Parse_Proto2<FileDesc, UseArena, Copy> 10.2 ± 0% 10.2 ± 0% ~ (all samples are equal) BM_Parse_Proto2<FileDesc, InitBlock, Copy> 1.19 ± 0% 1.19 ± 0% ~ (all samples are equal) BM_Parse_Proto2<FileDescSV, InitBlock, Alias> 1.19 ± 0% 1.19 ± 0% ~ (all samples are equal) BM_SerializeDescriptor_Proto2 0.19 ± 0% 0.19 ± 0% ~ (all samples are equal) BM_SerializeDescriptor_Upb 0.19 ± 0% 0.19 ± 0% ~ (all samples are equal) name old peak-mem(Bytes)/op new peak-mem(Bytes)/op delta BM_ArenaOneAlloc 344 ± 0% 344 ± 0% ~ (all samples are equal) BM_ArenaInitialBlockOneAlloc 112 ± 0% 112 ± 0% ~ (all samples are equal) BM_LoadAdsDescriptor_Upb<NoLayout> 9.64M ± 0% 9.64M ± 0% ~ (all samples are equal) BM_LoadAdsDescriptor_Upb<WithLayout> 9.70M ± 0% 9.70M ± 0% ~ (all samples are equal) BM_LoadAdsDescriptor_Proto2<NoLayout> 6.38M ± 0% 6.38M ± 0% ~ (all samples are equal) BM_LoadAdsDescriptor_Proto2<WithLayout> 6.44M ± 0% 6.44M ± 0% ~ (all samples are equal) BM_Parse_Upb_FileDesc<UseArena, Copy> 36.5k ± 0% 36.5k ± 0% ~ (all samples are equal) BM_Parse_Upb_FileDesc<UseArena, Alias> 36.5k ± 0% 36.5k ± 0% ~ (all samples are equal) BM_Parse_Upb_FileDesc<InitBlock, Copy> 112 ± 0% 112 ± 0% ~ (all samples are equal) BM_Parse_Upb_FileDesc<InitBlock, Alias> 112 ± 0% 112 ± 0% ~ (all samples are equal) BM_Parse_Proto2<FileDesc, NoArena, Copy> 35.8k ± 0% 35.8k ± 0% ~ (all samples are equal) BM_Parse_Proto2<FileDesc, UseArena, Copy> 40.8k ± 0% 40.8k ± 0% ~ (all samples are equal) BM_Parse_Proto2<FileDesc, InitBlock, Copy> 112 ± 0% 112 ± 0% ~ (all samples are equal) BM_Parse_Proto2<FileDescSV, InitBlock, Alias> 112 ± 0% 112 ± 0% ~ (all samples are equal) BM_SerializeDescriptor_Proto2 112 ± 0% 112 ± 0% ~ (all samples are equal) BM_SerializeDescriptor_Upb 112 ± 0% 112 ± 0% ~ (all samples are equal) name old speed new speed delta BM_LoadAdsDescriptor_Upb<NoLayout> 147MB/s ± 2% 148MB/s ± 3% ~ (p=0.222 n=5+5) BM_LoadAdsDescriptor_Upb<WithLayout> 137MB/s ± 3% 137MB/s ± 1% ~ (p=0.905 n=5+4) BM_LoadAdsDescriptor_Proto2<NoLayout> 68.6MB/s ± 1% 69.3MB/s ± 1% +1.10% (p=0.016 n=5+4) BM_LoadAdsDescriptor_Proto2<WithLayout> 66.0MB/s ± 9% 67.4MB/s ± 3% ~ (p=0.841 n=5+5) BM_Parse_Upb_FileDesc<UseArena, Copy> 675MB/s ± 3% 667MB/s ± 3% ~ (p=0.222 n=5+5) BM_Parse_Upb_FileDesc<UseArena, Alias> 730MB/s ± 5% 718MB/s ± 5% ~ (p=0.310 n=5+5) BM_Parse_Upb_FileDesc<InitBlock, Copy> 663MB/s ±16% 685MB/s ± 2% ~ (p=1.000 n=5+5) BM_Parse_Upb_FileDesc<InitBlock, Alias> 723MB/s ±15% 712MB/s ±16% ~ (p=0.421 n=5+5) BM_Parse_Proto2<FileDesc, NoArena, Copy> 367MB/s ± 2% 372MB/s ± 2% ~ (p=0.222 n=5+5) BM_Parse_Proto2<FileDesc, UseArena, Copy> 694MB/s ± 2% 691MB/s ± 4% ~ (p=0.841 n=5+5) BM_Parse_Proto2<FileDesc, InitBlock, Copy> 714MB/s ± 3% 709MB/s ± 3% ~ (p=0.690 n=5+5) BM_Parse_Proto2<FileDescSV, InitBlock, Alias> 816MB/s ± 2% 816MB/s ± 3% ~ (p=1.000 n=5+5) BM_SerializeDescriptor_Proto2 1.24GB/s ± 3% 1.28GB/s ± 3% ~ (p=0.222 n=5+5) BM_SerializeDescriptor_Upb 734MB/s ± 3% 713MB/s ±13% ~ (p=0.841 n=5+5) ``` PiperOrigin-RevId: 477770562	2 years ago
Eric Salo	1e3deb013d	move message/field modifiers functions out of mini_descriptor_encode.c PiperOrigin-RevId: 477486367	2 years ago
Eric Salo	efd06e46a4	use mini descriptors to build message defs and extension defs PiperOrigin-RevId: 477332937	2 years ago
Eric Salo	668cebbad9	mini descriptor encode functions now populate a string view Also fixed a few bugs in which output pointer updates were not stored. PiperOrigin-RevId: 474884814	2 years ago
Eric Salo	38d8430923	simplify makejsonname() PiperOrigin-RevId: 474577074	2 years ago
Eric Salo	00765002ff	- All of reflection now lives in upb/reflection/ - Each def type has its own .c file and its own .h file - Functions that require a builder context are declared in def_builder.h - The mini descriptor encoders have also been pulled into upb/reflection/ - upb/def.h, upb/def.hpp, upb/reflection.h, and upb/reflection.hpp are now deprecated stubs that point to the new headers PiperOrigin-RevId: 474459500	2 years ago

21 Commits (6f68ba1350e9edffec3e4cc011de97aa77e50ea4)