* Add acquire/release where necessary for all atomic ops
* Add sentinel member to ensure safe publication when tsan is active; tsan will not catch the previous errors without this member.
* For all operations using relaxed memory order, comment why relaxed order is safe
* Add a test that exercises racy fuses and space allocated checks without mutexes or other memory barriers from the test harness. This test proved the existence of several races not caught by the existing tests, including one with a confident comment about why relaxed memory order was safe.
* Add a test that exercises racing allocation and destruction among fused arenas, which doesn't use locks and substitutes a custom allocator that verifies its memory blocks.
Test coverage and assert/tsan instrumentation is now sufficient to cause test failures if any call site is further relaxed.
PiperOrigin-RevId: 712751905
The test wrappers were another way to document nonconformant behaviour between
different python backends. We can achieve the same by removing the wrapper
script and adding an if-condition in the test itself based on
api_implementation.Type(). Since we already do that for nonconformance between
pure Python vs. C++ backends, this change makes it easier to look for UPB
nonconformance instead of going through another layer of indirection.
Temporarily, we will need to hardcode the migrated test name in test_upb.yml
because not all tests under google.protobuf.internal support UPB yet.
(UPB testing for selected tests are added in 21e9aa6cac).
PiperOrigin-RevId: 712672890
implemented on top of `UntypedMapBase` visitation.
Reduces code duplication in large binaries.
More to come in future changes.
PiperOrigin-RevId: 712658107
The unnecessary [< >] around the ident in that specific macro expansion makes it the name as a literal which would then later be reparsed back into an ident, which makes it lose the r# in the process and break.
PiperOrigin-RevId: 712562736
This was disabled because it couldn't pass with the upb C generated accessors. Now that we only use the upb minitables, this can be safely enabled.
PiperOrigin-RevId: 712560415
This fixes:
* MSVC with `/std:c11 /experimental:c11atomics` on recent versions now emits atomics
* Clang with `-std=c11 -fgnuc-version=0` now emits atomics
* Clang and GCC 14 when built with `-std=c99 -pedantic-errors` will now compile, and not emit atomics
PiperOrigin-RevId: 712538312
The test wrappers were another way to document nonconformant behaviour between
different python backends. We can achieve the same by removing the wrapper
script and adding an if-condition in the test itself based on
api_implementation.Type(). Since we already do that for nonconformance between
pure Python vs. C++ backends, this change makes it easier to look for UPB
nonconformance instead of going through another layer of indirection.
Temporarily, we will need to hardcode the migrated test name in test_upb.yml
because not all tests under google.protobuf.internal support UPB yet.
(UPB testing for selected tests are added in 21e9aa6cac).
PiperOrigin-RevId: 711837521
When there is an #if or #ifdef in raw strings, clang-format wants to put it at the start of the line, which can leave a big gap compared to the rest of the code, and throws off indent calculation.
PiperOrigin-RevId: 711828681
Add protobuf local macro similar to the very recently added absl version cd9dd4266c The macro form allows Wnullability-completeness to check (vs absl's template form of annotation)
The macro will be empty unless you define PROTOBUF_TEMPORARY_NULL_ANNOTATIONS. We will switch that on once all annotations are in. The annotations will also be off in open source until we are ready.
PiperOrigin-RevId: 711805580
I just tried removing a bunch of if api_implementation.Type() guards in
message_test.py, and a bunch of them just pass without specialization! The bugs
that used to cause trouble seem to be now fixed.
PiperOrigin-RevId: 711764185
implemented on top of `UntypedMapBase` visitation.
Reduces code duplication in large binaries.
More to come in future changes.
PiperOrigin-RevId: 710756618
We no longer need to traverse the linked list of blocks to check allocated space, which means we also no longer need atomics in the linked list or even its head. This is especially beneficial as the previous implementation contained a race where we could dereference uninitialized memory; because the setting of the `next` pointers did not use release semantics and the reading of them in `SpaceAllocated` reads with relaxed order, there's no guarantee that `size` has actually been initialized - but worse, *there is also no guarantee that `next` has been!*. Simplified:
```
AddBlock:
1 ptr = malloc();
2 ptr->size = 123;
3 ptr->next = ai->blocks;
4 ai->blocks = ptr (release order);
```
```
SpaceAllocated:
5 block = ai->blocks (relaxed order)
6 block->size (acquire, but probably by accident)
7 block = block->next (relaxed order)
```
So I think a second thread calling SpaceAllocated could see the order 1, 4, 5, 6, 7, 2, 3 and read uninitialized memory - there is no data-dependency relationship or happens-before edge that this order violates, and so it would be valid for a compiler+hardware to produce.
In reality, operation 4 will produce an `stlr` on arm (forcing an order of 1, 2, 3 before 4), and `block->next` has a data dependency on `ai->blocks` which would force an ordering in the hardware between 5->6 and 5->7 even for regular `ldr` instructions.
Delete arena contains, it's private and the only user is its own test.
PiperOrigin-RevId: 709918443
This fixes that we accidentally imported absl ascii.h within the protobuf namespace, and avoids unused imports warnings for imports that are only used on windows.
PiperOrigin-RevId: 709846366
While in general Proto enum numbers are more of arbitary wire tag numbers, sometimes the ordering is used for semantic meaning, and this allows for the enums to be used in things that require ordering eg BTreeMap.
Rust defaults to language enums _not_ being Ord with the derive(Ord) opt in, but it is largely for reasons which don't apply to protobuf enums.
PiperOrigin-RevId: 709831238
This avoids installing every header required by protoc, and only installs the ones we've explicitly marked as public in bazel (and their transitive dependencies).
PiperOrigin-RevId: 709815861
This has been replaced with the cpp_string_type helper on FieldDescriptor, which returns the actual behavior of the field rather than the specification. This also handles merging of ctype and string_type in edition 2023 where both are allowed. ctype will be banned from 2024 onward.
PiperOrigin-RevId: 709226325
This will help us avoid further bugs like #19735, where we're typically not very aware of how our files get installed as long as everything builds. With these tests in place, we will need to manually update the golden files whenever we add a new file to users' installations.
PiperOrigin-RevId: 709134471
- Rename `UnsafeShallowSwap`. The operation is called `InternalSwap` in all other types.
- Remove the dynamic dispatch. It is no longer needed.
- Make `InternalSwap` do the whole swap, including the map and the reflection payload.
- Rename `MapFieldBase::SwapImpl` to `SwapPayload` to make it clearer on what it does and to prevent using it directly in the vtable.
PiperOrigin-RevId: 709132805
On copy construct/assign we can:
- guarantee there is enough space for all elements. This avoids rehashing.
- guarantee all elements are unique. This avoids lookups.
PiperOrigin-RevId: 709057139
An upcoming performance improvement in RepeatedPtrField is incompatible with this API. The improvement is projected to accelerate repeated access to the elements of `RepeatedPtrField`, in particular and especially sequential access.
PA: https://protobuf.dev/news/2024-12-13/
PiperOrigin-RevId: 708439051
Protobuf Team Bot
Protobuf Team Bot
なつき
Tony Liao
Sandy Zhang
Chris Kennelly
Jie Luo
y-yagi
Mike Kruskal