protobuf

Commit Graph

Author	SHA1	Message	Date
Joshua Haberman	10b355a6e0	A couple minor fixes. These are exercised by Ruby.	4 years ago
Joshua Haberman	6c30b5fe73	Fixed upb encoder for field numbers > 2**28. The encoder was improperly sign-extending the tag to 64 bits.	4 years ago
Joshua Haberman	e9551022c1	Added depth limit checking to upb_encode(). This can catch infinite recursion due to loops, or just excessively deep message trees. The depth limit is configurable, but defaults to 64.	4 years ago
Joshua Haberman	695b7f4617	Added code to test UPB_JSONENC_EMITDEFAULTS.	4 years ago
Joshua Haberman	6b357607bd	Fixed bugs in JSON encoding with UPB_JSONENC_EMITDEFAULTS. Previously the code would crash on certain empty fields.	4 years ago
Joshua Haberman	ee49a8d7df	Added an accessor to get the symtab from a filedef. This matches an API already present in proto2 (const DescriptorPool* FileDescriptor::pool()). However there is a slightly subtle implication here. In proto2, the relationship between Descriptor and MessageFactory is 1:many. You can create as many DynamicMessageFactory instances as you want, and each one will have its own independent DynamicMessage prototype and computed layout for the same underlying Descriptor. In practice the layouts will all be the same, but one thing that could be distinct is that each can have its own extension pool, which is a DescriptorPool that will be searched for extensions when parsing. In contrast, upb does not have a separate "message factory" abstraction. That means that each upb_msgdef has a single distinct layout, in other words a 1:1 correspondence between descriptor and layout. This means that there is no way to create multiple message types for the same descriptor that have distinct extension pools. If you want a different set of extensions, you must create a separate upb_symtab with a distinct set of descriptors. This change further entrenches that upb_filedef:upb_symtab is a 1:1 relationship. A single upb_filedef cannot be a member of multiple symbol tables. In practice this was already true (there is no way to add a single filedef to multiple symbol tables) but this change codifies this 1:1 relationship.	4 years ago
Tim Gates	7225108720	docs: fix simple typo, posisiton -> position There is a small typo in upb/table.c. Should read `position` rather than `posisiton`.	4 years ago
Joshua Haberman	76764643ac	Added option to binary encoder to skip unknown fields.	4 years ago
Joshua Haberman	a04627abc8	Added map sorting to binary and text encoders. For the binary encoder, sorting is off by default. For the text encoder, sorting is on by default. Both defaults can be explicitly overridden. This grows code size a bit. I think we could potentially shave this (and other map-related code size) by having the generated code inject a function pointer to the map-related parsing/serialization code if maps are present. FILE SIZE VM SIZE -------------- -------------- +86% +1.07Ki +71% +768 upb/msg.c [NEW] +391 [NEW] +344 _upb_mapsorter_pushmap [NEW] +158 [NEW] +112 _upb_mapsorter_cmpstr [NEW] +111 [NEW] +64 _upb_mapsorter_cmpbool [NEW] +110 [NEW] +64 _upb_mapsorter_cmpi32 [NEW] +110 [NEW] +64 _upb_mapsorter_cmpi64 [NEW] +110 [NEW] +64 _upb_mapsorter_cmpu32 [NEW] +110 [NEW] +64 _upb_mapsorter_cmpu64 -3.6% -8 -4.3% -8 _upb_map_new +9.5% +464 +9.2% +424 upb/text_encode.c [NEW] +656 [NEW] +616 txtenc_mapentry +15% +32 +20% +32 upb_text_encode -20.1% -224 -20.7% -224 txtenc_msg +5.7% +342 +5.3% +296 upb/encode.c [NEW] +344 [NEW] +304 encode_mapentry [NEW] +246 [NEW] +208 upb_encode_ex [NEW] +41 [NEW] +16 upb_encode_ex.ch +0.7% +8 +0.7% +8 encode_scalar -1.0% -32 -1.0% -32 encode_message [DEL] -38 [DEL] -16 upb_encode.ch [DEL] -227 [DEL] -192 upb_encode +2.0% +152 +2.2% +152 upb/decode.c +44% +128 +44% +128 [section .rodata] +3.4% +24 +3.4% +24 _GLOBAL_OFFSET_TABLE_ +0.6% +107 +0.3% +48 upb/def.c [NEW] +100 [NEW] +48 upb_fielddef_descriptortype +7.1% +7 [ = ] 0 upb_fielddef_defaultint32 +2.9% +24 +2.9% +24 [section .dynsym] +1.2% +24 [ = ] 0 [section .symtab] +3.2% +16 +3.2% +16 [section .plt] [NEW] +16 [NEW] +16 memcmp@plt +0.5% +16 +0.6% +16 tests/conformance_upb.c +1.5% +16 +1.6% +16 DoTestIo +0.1% +16 +0.1% +16 upb/json_decode.c +0.4% +16 +0.4% +16 jsondec_wellknown +3.0% +8 +3.0% +8 [section .got.plt] +3.0% +8 +3.0% +8 _GLOBAL_OFFSET_TABLE_ +1.6% +7 +1.6% +7 [section .dynstr] +1.8% +4 +1.8% +4 [section .hash] +0.5% +3 +0.5% +3 [LOAD #2 [RX]] +2.8% +2 +2.8% +2 [section .gnu.version] -60.0% -1.74Ki [ = ] 0 [Unmapped] +0.3% +496 +1.4% +1.74Ki TOTAL	4 years ago
Joshua Haberman	e70e488e09	MSVC apparently doesn't support the standard C/C++ defines. Our requirement then is C99, C++11, or MSVC >= 2015.	4 years ago
Joshua Haberman	8d670d8aea	Renamed decode_varint32() to decode_tag().	4 years ago
Joshua Haberman	9abf8e043f	Clamp 32-bit varints to 5 bytes to fix a fuzz failure.	4 years ago
Joshua Haberman	358fa14d0e	Fixed headers and updated benchmark script.	4 years ago
Joshua Haberman	bc200451ce	Use a macro instead of an inline function for setjmp/longjmp.	4 years ago
Joshua Haberman	fbc0639b07	Use _setjmp on mac to avoid saving/restoring the signal mask.	4 years ago
Joshua Haberman	65d166a6ba	Added API for copy vs. alias and added benchmarks to test both. Benchmark output: $ bazel-bin/benchmarks/benchmark '--benchmark_filter=BM_Parse' 2020-11-11 15:39:04 Running bazel-bin/benchmarks/benchmark Run on (72 X 3700 MHz CPU s) CPU Caches: L1 Data 32K (x36) L1 Instruction 32K (x36) L2 Unified 1024K (x36) L3 Unified 25344K (x2) ------------------------------------------------------------------------------------- Benchmark Time CPU Iterations ------------------------------------------------------------------------------------- BM_Parse_Upb_FileDesc<UseArena, Copy> 4134 ns 4134 ns 168714 1.69152GB/s BM_Parse_Upb_FileDesc<UseArena, Alias> 3487 ns 3487 ns 199509 2.00526GB/s BM_Parse_Upb_FileDesc<InitBlock, Copy> 3727 ns 3726 ns 187581 1.87643GB/s BM_Parse_Upb_FileDesc<InitBlock, Alias> 3110 ns 3110 ns 224970 2.24866GB/s BM_Parse_Proto2<FileDesc, NoArena, Copy> 31132 ns 31132 ns 22437 229.995MB/s BM_Parse_Proto2<FileDesc, UseArena, Copy> 21011 ns 21009 ns 33922 340.812MB/s BM_Parse_Proto2<FileDesc, InitBlock, Copy> 17976 ns 17975 ns 38808 398.337MB/s BM_Parse_Proto2<FileDescSV, InitBlock, Alias> 17357 ns 17356 ns 40244 412.539MB/s	4 years ago
Joshua Haberman	9df96874e9	Start arena block doubling at initial block size. If an initial block is provided, we should start our block doubling at the size of the initial block, not 128. This saves us from unnecessary overhead when we overflow the initial block.	4 years ago
Joshua Haberman	e5bdfba92c	Removed accidentally-added .orig file.	4 years ago
Joshua Haberman	982b634bc5	Fixed a few minor bugs found by fuzzing.	4 years ago
Joshua Haberman	a01f3e23a4	Fixes for google3 build, and exclude even more tests from macOS to avoid timeout.	4 years ago
Joshua Haberman	1eb7bd39e7	Some formatting fixes.	4 years ago
Joshua Haberman	6c16cba83f	Removed obsolete port.c file.	4 years ago
Joshua Haberman	5b1f0d86a1	For Kokoro, only build/test -m32 on Linux. Also fixed a bunch of bugs found by gcc's -fanalyzer.	4 years ago
Joshua Haberman	0497f8deed	Fixed a critical bug on 32-bit builds, and added much more Kokoro testing. There was a bug in our arena code where we assumed that sizeof(upb_array) would be a multiple of 8. On i386 it was not, and this was causing memory corruption on 32-bit builds.	4 years ago
Joshua Haberman	64abb5eb11	Amalgamation no longer bundles wyhash, but #includes it. Also fixed a few spelling mistakes.	4 years ago
Joshua Haberman	dd0994d377	Bugfix for JSON decoding: only check real oneofs for duplicates. Also fixed upb_msg_whichoneof() to work properly for synthetic fields, and to be simpler in general.	4 years ago
Joshua Haberman	c9f9668234	symtab: use longjmp() for errors and avoid intermediate table. We used to use a separate "add table" during the upb_symtab_addfile() operation to make it easier to back out the file if it contained errors. But this created unnecessary work of re-adding the same symbols to the main symtab once everything was validated. Instead we directly add symbols to the main symbols table. If there is an error in validation, we remove precisely the set of symbols that were already added. This also requires using a separate arena for each file. We can fuse it with the symtab's main arena if the operation is successful. LoadDescriptor_Upb 61.2µs ± 4% 53.5µs ± 1% -12.50% (p=0.000 n=12+12) LoadAdsDescriptor_Upb 4.43ms ± 1% 3.06ms ± 0% -31.00% (p=0.000 n=12+12) LoadDescriptor_Proto2 257µs ± 0% 259µs ± 0% +1.00% (p=0.000 n=12+12) LoadAdsDescriptor_Proto2 13.9ms ± 1% 13.9ms ± 1% ~ (p=0.128 n=12+12)	4 years ago
Joshua Haberman	c3b5637646	Added benchmark for loading ads descriptor. Generally this seems to track the speed of loading descriptor.proto. ---------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations ---------------------------------------------------------------------------------------------------- BM_LoadDescriptor_Upb 59091 ns 59086 ns 11747 121.182MB/s BM_LoadAdsDescriptor_Upb 4218587 ns 4218582 ns 166 120.544MB/s BM_LoadDescriptor_Proto2 241083 ns 241049 ns 2903 29.7043MB/s BM_LoadAdsDescriptor_Proto2 13442631 ns 13442099 ns 52 34.8975MB/s	4 years ago
Joshua Haberman	acd72c6d3f	WIP.	4 years ago
Joshua Haberman	8113ebd6c7	Added explanatory comment about integer constants.	4 years ago
Joshua Haberman	f2d826b9f3	Got rid of floating-point division in table init. Ideally we would get rid of all floating-point operations in table.c, but that's a job for another day.	4 years ago
Joshua Haberman	154f2c25f4	Added UTF-8 validation for proto3 string fields.	4 years ago
Joshua Haberman	e8f9eac68c	Added #defines UPB_ENABLE_FASTTABLE and UPB_TRY_ENABLE_FASTTABLE. These control whether fasttable decoding is on.	4 years ago
Joshua Haberman	e86541ac1d	Fixed the build after the merge.	4 years ago
Joshua Haberman	8f3ee80d46	Drop C89/C90 support and MSVC prior to Visual Studio 2015. upb previously attempted to support C89 and pre-2015 versions of Visual Studio. This was to support older compilers with limited C99 support (particularly MSVC). But as of last August, even gRPC has dropped support for MSVC prior to 2015 `c87276d058` Therefore it seems safe for upb to no longer attempt C89 support (we were already not truly C89 compliant, with our use of "bool"). We now explicitly require C99 or greater and MSVC 2015 or greater. This cleaned up port_def.inc a fair bit. I took the chance to also remove some obsolete macros.	4 years ago
Joshua Haberman	2c8bb6dd9d	Specify C99 explicitly until/unless we stop using bool.	4 years ago
Joshua Haberman	efd576b698	Added -std=gnu99 for fastdecode and ran Buildifier.	4 years ago
Joshua Haberman	b928696942	A few more fixes, and test fastdecode under Kokoro.	4 years ago
Joshua Haberman	55f3569cd2	A few minor fixes and more assertions.	4 years ago
Joshua Haberman	46eb82467a	Added comment to decode_fast.h.	4 years ago
Joshua Haberman	bd9f8f580d	Fixed a few bugs with the fast decoder. 1. For long tags we were putting table entries in the wrong slot. 2. For repeated strings, when the buffer flipped to no longer alias we were failing to notice and kept aliasing anyway.	4 years ago
Joshua Haberman	021db6fcd5	Allow larger tags into the table if they are unique mod 31. Also fixed a bug with fixed packed in decode_fast.c.	4 years ago
Joshua Haberman	86d9908c55	Fastdecode support for packed fields. This is not very optimized yet. There is a lot of room to optimize it further.	4 years ago
Joshua Haberman	e3e797b680	Added fasttable support for oneofs.	4 years ago
Joshua Haberman	7ffa9c181a	Fixed some small bugs and performance problems in string copying. Before this CL, with alias=false: ------------------------------------------------------------------------------ Benchmark Time CPU Iterations ------------------------------------------------------------------------------ BM_Parse_Upb_FileDesc_WithInitialBlock 3715 ns 3715 ns 188916 1.88206GB/s Performance counter stats for 'bazel-bin/benchmarks/benchmark --benchmark_filter=BM_Parse_Upb_FileDesc_WithInitialBlock': 1,122.92 msec task-clock # 0.979 CPUs utilized 3 context-switches # 0.003 K/sec 0 cpu-migrations # 0.000 K/sec 196 page-faults # 0.175 K/sec 4,144,746,717 cycles # 3.691 GHz 15,351,966,804 instructions # 3.70 insn per cycle 2,590,281,905 branches # 2306.728 M/sec 2,996,157 branch-misses # 0.12% of all branches 1.146615328 seconds time elapsed 1.115578000 seconds user 0.008025000 seconds sys After this CL: ------------------------------------------------------------------------------ Benchmark Time CPU Iterations ------------------------------------------------------------------------------ BM_Parse_Upb_FileDesc_WithInitialBlock 3554 ns 3554 ns 197527 1.9674GB/s Performance counter stats for 'bazel-bin/benchmarks/benchmark --benchmark_filter=BM_Parse_Upb_FileDesc_WithInitialBlock': 1,105.34 msec task-clock # 0.982 CPUs utilized 3 context-switches # 0.003 K/sec 0 cpu-migrations # 0.000 K/sec 197 page-faults # 0.178 K/sec 4,077,736,892 cycles # 3.689 GHz 15,442,709,352 instructions # 3.79 insn per cycle 2,435,131,301 branches # 2203.068 M/sec 2,643,775 branch-misses # 0.11% of all branches 1.125393845 seconds time elapsed 1.097770000 seconds user 0.008012000 seconds sys	4 years ago
Joshua Haberman	e2c709e047	Repeated string and primitive support. Much of the code was adapted from Gerben's code in: `6333031195`	4 years ago
Joshua Haberman	25db40bc30	Fixed upb::InlinedArena, which was compeltely broken.	4 years ago
Joshua Haberman	d81ba58215	Optimized short string copying. This sped up the alias=false case: Before: ------------------------------------------------------------------------------ Benchmark Time CPU Iterations ------------------------------------------------------------------------------ BM_Parse_Upb_FileDesc_WithInitialBlock 4562 ns 4562 ns 153251 1.53276GB/s Performance counter stats for 'bazel-bin/benchmarks/benchmark --benchmark_filter=BM_Parse_Upb_FileDesc_WithInitialBlock': 1,216.65 msec task-clock # 0.936 CPUs utilized 6 context-switches # 0.005 K/sec 0 cpu-migrations # 0.000 K/sec 200 page-faults # 0.164 K/sec 4,490,925,650 cycles # 3.691 GHz 16,516,403,731 instructions # 3.68 insn per cycle 2,828,536,650 branches # 2324.861 M/sec 5,425,830 branch-misses # 0.19% of all branches 1.300178903 seconds time elapsed 1.211475000 seconds user 0.072207000 seconds sys After: ------------------------------------------------------------------------------ Benchmark Time CPU Iterations ------------------------------------------------------------------------------ BM_Parse_Upb_FileDesc_WithInitialBlock 3587 ns 3587 ns 195749 1.94935GB/s Performance counter stats for 'bazel-bin/benchmarks/benchmark --benchmark_filter=BM_Parse_Upb_FileDesc_WithInitialBlock': 1,109.69 msec task-clock # 0.930 CPUs utilized 5 context-switches # 0.005 K/sec 0 cpu-migrations # 0.000 K/sec 198 page-faults # 0.178 K/sec 4,094,010,257 cycles # 3.689 GHz 15,672,677,812 instructions # 3.83 insn per cycle 2,589,291,160 branches # 2333.346 M/sec 3,306,386 branch-misses # 0.13% of all branches 1.193221789 seconds time elapsed 1.102538000 seconds user 0.072166000 seconds sys	4 years ago
Joshua Haberman	f3a2a79349	More optimization, back up to 2.56GB/s.	4 years ago
Joshua Haberman	199c914295	Simplify push/pop when msg fits in the current buffer.	4 years ago

1 2 3 4 5 ...

755 Commits (c5d6ec737ee0e66178dfb334c3818bed861e3e55)