4132034634
Addressed PR comment.
4 years ago
ed708fcd5d
Addressed PR comments.
4 years ago
876abae2db
Removed some debug printing and simplified checktag slightly.
4 years ago
286441afa7
Fixed a size regression due to inlining UTF-8 verification.
...
Overall size/speed impact on fasttable decoder is now:
name old time/op new time/op delta
ArenaOneAlloc 21.5ns ± 0% 21.5ns ± 0% ~ (p=0.060 n=12+12)
ArenaInitialBlockOneAlloc 6.33ns ± 0% 6.33ns ± 0% ~ (p=0.413 n=11+12)
LoadDescriptor_Upb 43.4µs ± 1% 45.5µs ± 1% +4.79% (p=0.000 n=12+12)
LoadAdsDescriptor_Upb 2.50ms ± 0% 2.51ms ± 2% ~ (p=0.512 n=10+11)
LoadDescriptor_Proto2 240µs ± 0% 240µs ± 0% -0.25% (p=0.000 n=12+12)
LoadAdsDescriptor_Proto2 12.9ms ± 0% 12.9ms ± 0% +0.20% (p=0.014 n=10+12)
Parse_Upb_FileDesc<UseArena,Copy> 4.99µs ± 0% 5.04µs ± 0% +0.98% (p=0.000 n=11+10)
Parse_Upb_FileDesc<UseArena,Alias> 4.02µs ± 0% 4.18µs ± 0% +4.16% (p=0.000 n=10+12)
Parse_Upb_FileDesc<InitBlock,Copy> 4.49µs ± 0% 4.54µs ± 0% +1.16% (p=0.000 n=11+10)
Parse_Upb_FileDesc<InitBlock,Alias> 3.60µs ± 0% 3.80µs ± 0% +5.73% (p=0.000 n=12+11)
Parse_Proto2<FileDesc,NoArena,Copy> 29.3µs ± 0% 29.3µs ± 0% ~ (p=0.069 n=11+12)
Parse_Proto2<FileDesc,UseArena,Copy> 20.2µs ± 3% 20.3µs ± 2% ~ (p=0.880 n=12+11)
Parse_Proto2<FileDesc,InitBlock,Copy> 16.5µs ± 0% 16.5µs ± 0% ~ (p=1.000 n=12+12)
Parse_Proto2<FileDescSV,InitBlock,Alias> 16.4µs ± 0% 16.4µs ± 1% ~ (p=0.590 n=12+12)
SerializeDescriptor_Proto2 5.31µs ± 1% 6.65µs ±29% +25.07% (p=0.000 n=12+12)
SerializeDescriptor_Upb 12.4µs ± 0% 12.5µs ± 0% +1.23% (p=0.000 n=12+12)
FILE SIZE VM SIZE
-------------- --------------
+16% +128 [ = ] 0 [Unmapped]
-1.2% -4 -1.2% -4 [section .text]
[NEW] +2 [NEW] +2 fastdecode_isdonefallback
[DEL] -6 [DEL] -6 fastdecode_longstring_noutf8
-0.2% -124 -0.2% -124 upb/decode_fast.c
+5.8% +64 +6.0% +64 upb_pom_1bt_max64b
+2.7% +64 +2.7% +64 upb_ppv8_2bt
+2.7% +32 +2.8% +32 upb_psm_1bt_max256b
+2.8% +32 +3.0% +32 upb_psm_1bt_max64b
+2.8% +32 +3.0% +32 upb_psm_2bt_max64b
+4.0% +24 +4.2% +24 upb_psv8_1bt
+2.0% +16 +2.1% +16 upb_prf4_2bt
+1.3% +16 +1.4% +16 upb_prz8_2bt
-0.3% -4 -0.3% -4 [3 Others]
-1.6% -8 -1.7% -8 upb_cob_1bt
-1.6% -8 -1.7% -8 upb_csb_1bt
-2.5% -16 -2.6% -16 upb_pov4_1bt
-1.3% -16 -1.3% -16 upb_prv8_2bt
-2.5% -16 -2.7% -16 upb_psv4_1bt
-2.5% -16 -2.6% -16 upb_psv4_2bt
-3.0% -32 -3.1% -32 upb_prs_2bt
-2.6% -32 -2.6% -32 upb_prv4_2bt
-4.9% -48 -5.1% -48 upb_prb_2bt
-3.9% -48 -4.0% -48 upb_prv4_1bt
-7.2% -72 -7.5% -72 upb_prb_1bt
-7.8% -88 -8.0% -88 upb_prs_1bt
[ = ] 0 -0.1% -128 TOTAL
There is a bit of speed regression, but it appears there were bigger
CPU regressions prior to this. We probably need some separate
optimization attention again to get back to the performance numbers
we had when fasttable was first submitted.
4 years ago
75df4cdaa3
Loosened the check for fasttable to match what we did before.
4 years ago
e84793dd73
Cleaned up debugging artifacts.
4 years ago
a4b35aa388
Everything passes except 4 conformance tests.
4 years ago
a778f2639a
Only perform multiple oneof check for non-null fields.
...
This allows payloads like:
{"oneofUint32": null, "oneofString": "test"}
4 years ago
3881393907
Renamed .int.h to _internal.h, for greater clarity.
4 years ago
823eb09694
Update all 2011 dates to 2021.
4 years ago
e59d2c8fa7
Added license headers to all files.
4 years ago
1674f28dd7
Put public message interface into msg.h and moved internal functions to msg.int.h.
4 years ago
5b97df91dd
Restrict fuse to matching block_alloc
4 years ago
e74d6c23de
Small renames and use uintptr_t instead of void*
4 years ago
d9a0c58108
Allow arena fuse to fail
...
Track initial blocks to avoid having fuse operate on arenas that cannot
be fused.
4 years ago
8370818143
Switch to a macro to avoid signed/const mismatches.
4 years ago
63ad3db980
Fixed "NULL + 0" UB in JSON encoder and decoder.
4 years ago
add9b12f18
Fixed quadratic memory usage in upb_array_append().
...
We were erroneously calling realloc() instead of resize(),
forcing the entire array to be reallocated for every array
append.
4 years ago
83c0edbd2a
A few minor cleanups.
4 years ago
e4343f0fa3
Update comment for ARM64.
4 years ago
c358829c76
Now that handlers are gone, cleaned up table to use arenas exclusively.
...
Also cleaned up some cruft from table.
4 years ago
ec9ba3f893
Fixed error message buffer overflow.
4 years ago
e58f7a0284
Fix for Any messages with no fields.
4 years ago
81829bacc3
Removed quote chars from error message.
4 years ago
88ed8f5d3b
Changed JSON decoding error message to be more clear.
...
"No such field:" is more clear than "Unknown field:",
because "unknown field" is a term of art within protocol
buffers that implies that we are preserving the field.
Also "No such field:" matches the pre-existing Ruby
error message.
4 years ago
f41c0ec261
Added an internal API to get arena from symtab, for Ruby's use.
4 years ago
f7ed1f27a3
Support non-zero minutes in the timestamp offset for JSON.
4 years ago
f5d2d55007
Deleted the legacy "Handlers" APIs. upb can finally be deserving of its name.
...
This is possible now that all users have been migrated to the new
upb_msg APIs.
4 years ago
c7787cbaa1
Fixed a bunch of Clang warnings.
...
Unfortunately a few of the Clang warnings did not have easy fixes:
../../../../ext/google/protobuf_c/ruby-upb.c: In function ‘fastdecode_err’:
../../../../ext/google/protobuf_c/ruby-upb.c:353:13: warning: function might be candidate for attribute ‘noreturn’ [-Wsuggest-attribute=noreturn]
353 | const char *fastdecode_err(upb_decstate *d) {
| ^~~~~~~~~~~~~~
../../../../ext/google/protobuf_c/ruby-upb.c: In function ‘_upb_decode’:
../../../../ext/google/protobuf_c/ruby-upb.c:867:30: warning: argument ‘buf’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
867 | bool _upb_decode(const char *buf, size_t size, void *msg,
I even tried to suppress the first error, but it still shows up.
4 years ago
9175989431
Bugfix for arena cleanup list when passing to upb_decode().
4 years ago
5e550e88f8
Added API for getting fielddef default as a upb_msgval.
4 years ago
38b4beed6b
Fix clang-tidy function 'upb_encode_ex' has a definition with different parameter names
4 years ago
9b020d8f65
Optimize calls to std::string::find() and friends for a single char.
4 years ago
3c9ae7837a
The fasttable parser works on ARM64!
4 years ago
10b355a6e0
A couple minor fixes. These are exercised by Ruby.
4 years ago
6c30b5fe73
Fixed upb encoder for field numbers > 2**28.
...
The encoder was improperly sign-extending the tag to 64 bits.
4 years ago
e9551022c1
Added depth limit checking to upb_encode().
...
This can catch infinite recursion due to loops,
or just excessively deep message trees.
The depth limit is configurable, but defaults to 64.
4 years ago
695b7f4617
Added code to test UPB_JSONENC_EMITDEFAULTS.
4 years ago
6b357607bd
Fixed bugs in JSON encoding with UPB_JSONENC_EMITDEFAULTS.
...
Previously the code would crash on certain empty fields.
4 years ago
ee49a8d7df
Added an accessor to get the symtab from a filedef.
...
This matches an API already present in proto2
(const DescriptorPool* FileDescriptor::pool()).
However there is a slightly subtle implication here.
In proto2, the relationship between Descriptor and
MessageFactory is 1:many. You can create as many
DynamicMessageFactory instances as you want, and
each one will have its own independent DynamicMessage
prototype and computed layout for the same underlying
Descriptor. In practice the layouts will all be the same,
but one thing that could be distinct is that each can
have its own extension pool, which is a DescriptorPool
that will be searched for extensions when parsing.
In contrast, upb does not have a separate "message
factory" abstraction. That means that each upb_msgdef
has a single distinct layout, in other words a 1:1
correspondence between descriptor and layout. This means
that there is no way to create multiple message types
for the same descriptor that have distinct extension
pools. If you want a different set of extensions, you
must create a separate upb_symtab with a distinct set
of descriptors.
This change further entrenches that upb_filedef:upb_symtab
is a 1:1 relationship. A single upb_filedef cannot be a
member of multiple symbol tables. In practice this was
already true (there is no way to add a single filedef to
multiple symbol tables) but this change codifies this
1:1 relationship.
4 years ago
7225108720
docs: fix simple typo, posisiton -> position
...
There is a small typo in upb/table.c.
Should read `position` rather than `posisiton`.
4 years ago
76764643ac
Added option to binary encoder to skip unknown fields.
4 years ago
a04627abc8
Added map sorting to binary and text encoders.
...
For the binary encoder, sorting is off by default.
For the text encoder, sorting is on by default.
Both defaults can be explicitly overridden.
This grows code size a bit. I think we could potentially
shave this (and other map-related code size) by having
the generated code inject a function pointer to the map-related
parsing/serialization code if maps are present.
FILE SIZE VM SIZE
-------------- --------------
+86% +1.07Ki +71% +768 upb/msg.c
[NEW] +391 [NEW] +344 _upb_mapsorter_pushmap
[NEW] +158 [NEW] +112 _upb_mapsorter_cmpstr
[NEW] +111 [NEW] +64 _upb_mapsorter_cmpbool
[NEW] +110 [NEW] +64 _upb_mapsorter_cmpi32
[NEW] +110 [NEW] +64 _upb_mapsorter_cmpi64
[NEW] +110 [NEW] +64 _upb_mapsorter_cmpu32
[NEW] +110 [NEW] +64 _upb_mapsorter_cmpu64
-3.6% -8 -4.3% -8 _upb_map_new
+9.5% +464 +9.2% +424 upb/text_encode.c
[NEW] +656 [NEW] +616 txtenc_mapentry
+15% +32 +20% +32 upb_text_encode
-20.1% -224 -20.7% -224 txtenc_msg
+5.7% +342 +5.3% +296 upb/encode.c
[NEW] +344 [NEW] +304 encode_mapentry
[NEW] +246 [NEW] +208 upb_encode_ex
[NEW] +41 [NEW] +16 upb_encode_ex.ch
+0.7% +8 +0.7% +8 encode_scalar
-1.0% -32 -1.0% -32 encode_message
[DEL] -38 [DEL] -16 upb_encode.ch
[DEL] -227 [DEL] -192 upb_encode
+2.0% +152 +2.2% +152 upb/decode.c
+44% +128 +44% +128 [section .rodata]
+3.4% +24 +3.4% +24 _GLOBAL_OFFSET_TABLE_
+0.6% +107 +0.3% +48 upb/def.c
[NEW] +100 [NEW] +48 upb_fielddef_descriptortype
+7.1% +7 [ = ] 0 upb_fielddef_defaultint32
+2.9% +24 +2.9% +24 [section .dynsym]
+1.2% +24 [ = ] 0 [section .symtab]
+3.2% +16 +3.2% +16 [section .plt]
[NEW] +16 [NEW] +16 memcmp@plt
+0.5% +16 +0.6% +16 tests/conformance_upb.c
+1.5% +16 +1.6% +16 DoTestIo
+0.1% +16 +0.1% +16 upb/json_decode.c
+0.4% +16 +0.4% +16 jsondec_wellknown
+3.0% +8 +3.0% +8 [section .got.plt]
+3.0% +8 +3.0% +8 _GLOBAL_OFFSET_TABLE_
+1.6% +7 +1.6% +7 [section .dynstr]
+1.8% +4 +1.8% +4 [section .hash]
+0.5% +3 +0.5% +3 [LOAD #2 [RX]]
+2.8% +2 +2.8% +2 [section .gnu.version]
-60.0% -1.74Ki [ = ] 0 [Unmapped]
+0.3% +496 +1.4% +1.74Ki TOTAL
4 years ago
e70e488e09
MSVC apparently doesn't support the standard C/C++ defines.
...
Our requirement then is C99, C++11, or MSVC >= 2015.
4 years ago
8d670d8aea
Renamed decode_varint32() to decode_tag().
4 years ago
9abf8e043f
Clamp 32-bit varints to 5 bytes to fix a fuzz failure.
4 years ago
358fa14d0e
Fixed headers and updated benchmark script.
4 years ago
bc200451ce
Use a macro instead of an inline function for setjmp/longjmp.
4 years ago
fbc0639b07
Use _setjmp on mac to avoid saving/restoring the signal mask.
4 years ago
65d166a6ba
Added API for copy vs. alias and added benchmarks to test both.
...
Benchmark output:
$ bazel-bin/benchmarks/benchmark '--benchmark_filter=BM_Parse'
2020-11-11 15:39:04
Running bazel-bin/benchmarks/benchmark
Run on (72 X 3700 MHz CPU s)
CPU Caches:
L1 Data 32K (x36)
L1 Instruction 32K (x36)
L2 Unified 1024K (x36)
L3 Unified 25344K (x2)
-------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
-------------------------------------------------------------------------------------
BM_Parse_Upb_FileDesc<UseArena, Copy> 4134 ns 4134 ns 168714 1.69152GB/s
BM_Parse_Upb_FileDesc<UseArena, Alias> 3487 ns 3487 ns 199509 2.00526GB/s
BM_Parse_Upb_FileDesc<InitBlock, Copy> 3727 ns 3726 ns 187581 1.87643GB/s
BM_Parse_Upb_FileDesc<InitBlock, Alias> 3110 ns 3110 ns 224970 2.24866GB/s
BM_Parse_Proto2<FileDesc, NoArena, Copy> 31132 ns 31132 ns 22437 229.995MB/s
BM_Parse_Proto2<FileDesc, UseArena, Copy> 21011 ns 21009 ns 33922 340.812MB/s
BM_Parse_Proto2<FileDesc, InitBlock, Copy> 17976 ns 17975 ns 38808 398.337MB/s
BM_Parse_Proto2<FileDescSV, InitBlock, Alias> 17357 ns 17356 ns 40244 412.539MB/s
4 years ago
Joshua Haberman
Matt Kulukundis
Esun Kim
Tim Gates