I added two variants, one which is unbalanced (always fusing a single arena into the group) while the other is balanced (always fusing two groups of equal size).
Unfortunately it is difficult to benchmark `upb_Arena_Free()` separately from fuse, since the only way to get a non-zero refcount is to fuse.
Results on my machine:
```
CPU: Intel Skylake Xeon with HyperThreading (48 cores) dL1:32KB dL2:1024KB dL3:38MB
Benchmark Time(ns) CPU(ns) Iterations
-------------------------------------------------------------------
BM_ArenaFuseUnbalanced/2 52.2 52.2 1000000 38.300M items/s
BM_ArenaFuseUnbalanced/8 416 416 142315 19.243M items/s
BM_ArenaFuseUnbalanced/64 4034 4033 14306 15.868M items/s
BM_ArenaFuseUnbalanced/128 8302 8301 6837 15.420M items/s
BM_ArenaFuseBalanced/2 54.7 54.7 1000000 36.581M items/s
BM_ArenaFuseBalanced/8 425 425 100000 18.845M items/s
BM_ArenaFuseBalanced/64 4029 4029 14632 15.886M items/s
BM_ArenaFuseBalanced/128 8050 8049 7176 15.902M items/s
```
PiperOrigin-RevId: 518292073
"Thunks" are .c files with definitions for the inline functions in the gencode C
API. They are meant to be used from statically compiled languages that cannot
use inline C functions such as Rust.
Experimental, do not use. Our current plan is to delete this code in the order
of months. Without any prior warning.
PiperOrigin-RevId: 516522823
The new API upb_MiniTable_Link() links all sub-messages and sub-enums at a single time, by accepting an array of sub-tables and sub-enums. The order of these sub-tables can be queried using a separate function `upb_MiniTable_GetSubList()`, and this information is added to `CodeGeneratorRequest` as part of the upb-specific info.
PiperOrigin-RevId: 513970874
This fixes an MSAN warning of the form:
```
WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x7fc964f2a597 in arena_findroot third_party/upb/upb/mem/arena.c:64:3
#1 0x7fc964f2a597 in upb_Arena_Free third_party/upb/upb/mem/arena.c:211:7
#2 0x7fc9d2af0028 in std::__msan::unique_ptr<upb_Arena, void (*)(upb_Arena*)>::~unique_ptr() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__memory/unique_ptr.h:277:7
#3 0x7fc9d2aef7a2 in ~Arena third_party/upb/upb/upb.hpp:70:7
#4 0x7fc9d2aef7a2 in ~InlinedArena third_party/upb/upb/upb.hpp:97:7
#5 0x7fc9d2aef7a2 in Cpp_InlinedArena2_Test::TestBody() third_party/upb/upb/test/test_cpp.cc:187:1
#6 0x7fc97da78a57 in testing::Test::Run() third_party/googletest/googletest/src/gtest.cc:2695:5
#7 0x7fc97da7a3e8 in testing::TestInfo::Run() third_party/googletest/googletest/src/gtest.cc:2844:11
#8 0x7fc97da7b897 in testing::TestSuite::Run() third_party/googletest/googletest/src/gtest.cc:3003:30
#9 0x7fc97daa5136 in testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/googletest/src/gtest.cc:5899:44
#10 0x7fc97daa455c in HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/googletest/src/gtest.cc
#11 0x7fc97daa455c in testing::UnitTest::Run() third_party/googletest/googletest/src/gtest.cc:5464:10
#12 0x562a7fb876f0 in RUN_ALL_TESTS third_party/googletest/googletest/include/gtest/gtest.h:2329:73
#13 0x562a7fb876f0 in main testing/base/internal/gunit_main.cc:86:10
#14 0x7fc9ba9b7632 in __libc_start_main (/usr/grte/v5/lib64/libc.so.6+0x61632) (BuildId: 280088eab084c30a3992a9bce5c35b44)
#15 0x562a7fafdbe9 in _start /build/work/ab393f4ac612f9027aae6b1a7226027ba2a2/google3/blaze-out/k8-opt/bin/third_party/grte/v5_src/grte-scratch/BUILD/src/csu/../sysdeps/x86_64/start.S:120
Member fields were destroyed
#0 0x562a7fb0b13d in __sanitizer_dtor_callback_fields third_party/llvm/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:960:5
#1 0x7fc9d2aef79a in ~InlinedArena third_party/upb/upb/upb.hpp:105:8
#2 0x7fc9d2aef79a in ~InlinedArena third_party/upb/upb/upb.hpp:97:7
#3 0x7fc9d2aef79a in Cpp_InlinedArena2_Test::TestBody() third_party/upb/upb/test/test_cpp.cc:187:1
#4 0x7fc97da78a57 in testing::Test::Run() third_party/googletest/googletest/src/gtest.cc:2695:5
#5 0x7fc97da7a3e8 in testing::TestInfo::Run() third_party/googletest/googletest/src/gtest.cc:2844:11
#6 0x7fc97da7b897 in testing::TestSuite::Run() third_party/googletest/googletest/src/gtest.cc:3003:30
#7 0x7fc97daa5136 in testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/googletest/src/gtest.cc:5899:44
#8 0x7fc97daa455c in HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/googletest/src/gtest.cc
#9 0x7fc97daa455c in testing::UnitTest::Run() third_party/googletest/googletest/src/gtest.cc:5464:10
#10 0x562a7fb876f0 in RUN_ALL_TESTS third_party/googletest/googletest/include/gtest/gtest.h:2329:73
#11 0x562a7fb876f0 in main testing/base/internal/gunit_main.cc:86:10
#12 0x7fc9ba9b7632 in __libc_start_main (/usr/grte/v5/lib64/libc.so.6+0x61632) (BuildId: 280088eab084c30a3992a9bce5c35b44)
#13 0x562a7fafdbe9 in _start /build/work/ab393f4ac612f9027aae6b1a7226027ba2a2/google3/blaze-out/k8-opt/bin/third_party/grte/v5_src/grte-scratch/BUILD/src/csu/../sysdeps/x86_64/start.S:120
```
PiperOrigin-RevId: 511849224
This was not a bug -- the previous behavior was correct. However this
change brings our implementation-specific details in line with C++.
PiperOrigin-RevId: 511273853
The upb convention is that "_Build()" means to also allocate, which this function does not do, so rename it as "_Init()" to free up the name for a future function that does allocate.
PiperOrigin-RevId: 510282736
According to https://en.cppreference.com/w/c/program/setjmp automatic variables
modified in a function calling setjmp can have indeterminate values. Instead,
refactor all functions calling setjmp so that the function calling setjmp
doesn’t have any local variables.
Part II: Mini table decoder.
PiperOrigin-RevId: 509644446
Prior to this CL we were allocating a MiniTable for each message and then overwriting it later. This could lead to an inconsistent state, and is unnecessary. This CL adds an extra phase to initialization so that the MiniTable is assigned only one time for each message.
PiperOrigin-RevId: 507617479
Previously we were calling the static method `Equals()` which did not take into effect the custom comparator or string for reporting errors.
PiperOrigin-RevId: 507344425
The initial motivation for this change was to fix a bug found by fuzzing. The old fuzz test (built on `cc_fuzz_target()`) detected an infinite loop if a bytes field default has an unterminated `\x` escape.
To fix this bug while expanding fuzz coverage, I created a fuzz test that verifies that we can do a lossless round trip from descriptor -> DefPool -> descriptor. We use C++ as the source of truth for whether a descriptor is valid or not, and what the canonical serialization back to protobuf form should be.
I wrote the new fuzz test using go/FuzzTest, which makes it easier and more readable to use an arbitrary `FileDescriptorSet` as input, while adding test cases for regressions.
The fuzz test highlighted a handful of errors that I subsequently fixed and added regression tests for:
1. The aforementioned unterminated `\x` bug.
2. We were not propagating the `edition` field.
3. We were missing the CheckIdent() check in a few places.
4. We were rejecting files with empty name, whereas C++ allows this.
5. There were a few bugs with escaping string defaults.
Since FuzzTest is Clang-only, I split the `FUZZ_TEST()` invocation from the regression tests, since the latter are portable and should be run on all platforms. Only `FUZZ_TEST()` itself is in a google3/Clang-only file.
PiperOrigin-RevId: 506997362
We're already doing a proper string sort in SortedEnums as of cl/503574792, but then we follow it up with a sort on the char* pointers.
PiperOrigin-RevId: 506778694
Our pkg_tar rule was missing a few template files for python. Using setup.py will guarantee conformance with python package requirements.
Adds MANIFEST.in to include c header/inc files in the source distribution.
PiperOrigin-RevId: 506638326
Joshua Haberman
Deanna Garcia
Mike Kruskal
Matt Kulukundis
Protobuf Team Bot
Eric Salo
Marcel Hlopko