Chiebot-Mirror
/
protobuf
8
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Allow for fuse/free races in `upb_Arena`.

Browse Source 
Implementation is by kfm@, I only added the portability code around it.

`upb_Arena` was designed to be only thread-compatible.  However, fusing of arenas muddies the waters somewhat, because two distinct `upb_Arena` objects will end up sharing state when fused.  This causes a `upb_Arena_Free(a)` to interfere with `upb_Arena_Fuse(b, c)` if `a` and `b` were previously fused.

It turns out that we can use atomics to fix this with about a 35% regression in fuse performance (see below).  Arena create+free does not regress, thanks to special-case logic in Free().

`upb_Arena` is still a thread-compatible type, and it is still never safe to call `upb_Arena_xxx(a)` and `upb_Arena_yyy(a)` in parallel.  However you can at least now call `upb_Arena_Free(a)` and `upb_Arena_Fuse(b, c)` in parallel, even if `a` and `b` were previously fused.

Note that `upb_Arena_Fuse(a, b)` and `upb_Arena_Fuse(c, d)` is still not allowed if `b` and `c` were previously fused.  In practice this means that fuses must still be single-threaded within a single fused group.

Performance results:

```
name                                           old cpu/op  new cpu/op  delta
BM_ArenaOneAlloc                                 18.6ns ± 1%  18.6ns ± 1%     ~     (p=0.726 n=18+17)
BM_ArenaInitialBlockOneAlloc                     6.28ns ± 1%  5.73ns ± 1%   -8.68%  (p=0.000 n=17+20)
BM_ArenaFuseUnbalanced/2                         44.1ns ± 2%  60.4ns ± 1%  +37.05%  (p=0.000 n=18+19)
BM_ArenaFuseUnbalanced/8                          370ns ± 2%   500ns ± 1%  +35.12%  (p=0.000 n=19+20)
BM_ArenaFuseUnbalanced/64                        3.52µs ± 1%  4.71µs ± 1%  +33.80%  (p=0.000 n=18+19)
BM_ArenaFuseUnbalanced/128                       7.20µs ± 1%  9.72µs ± 2%  +34.93%  (p=0.000 n=16+19)
BM_ArenaFuseBalanced/2                           44.4ns ± 2%  61.4ns ± 1%  +38.23%  (p=0.000 n=20+17)
BM_ArenaFuseBalanced/8                            373ns ± 2%   509ns ± 1%  +36.57%  (p=0.000 n=19+17)
BM_ArenaFuseBalanced/64                          3.55µs ± 2%  4.79µs ± 1%  +34.80%  (p=0.000 n=19+19)
BM_ArenaFuseBalanced/128                         7.26µs ± 1%  9.76µs ± 1%  +34.45%  (p=0.000 n=17+19)
BM_LoadAdsDescriptor_Upb<NoLayout>               5.66ms ± 1%  5.69ms ± 1%   +0.57%  (p=0.013 n=18+20)
BM_LoadAdsDescriptor_Upb<WithLayout>             6.30ms ± 1%  6.36ms ± 1%   +0.90%  (p=0.000 n=19+18)
BM_LoadAdsDescriptor_Proto2<NoLayout>            12.1ms ± 1%  12.1ms ± 1%     ~     (p=0.118 n=18+18)
BM_LoadAdsDescriptor_Proto2<WithLayout>          12.2ms ± 1%  12.3ms ± 1%   +0.50%  (p=0.006 n=18+18)
BM_Parse_Upb_FileDesc<UseArena, Copy>            12.7µs ± 1%  12.7µs ± 1%     ~     (p=0.194 n=20+19)
BM_Parse_Upb_FileDesc<UseArena, Alias>           11.6µs ± 1%  11.6µs ± 1%     ~     (p=0.192 n=20+20)
BM_Parse_Upb_FileDesc<InitBlock, Copy>           12.5µs ± 1%  12.5µs ± 0%     ~     (p=0.750 n=18+14)
BM_Parse_Upb_FileDesc<InitBlock, Alias>          11.4µs ± 1%  11.3µs ± 1%   -0.34%  (p=0.046 n=19+19)
BM_Parse_Proto2<FileDesc, NoArena, Copy>         25.4µs ± 1%  25.7µs ± 2%   +1.37%  (p=0.000 n=18+18)
BM_Parse_Proto2<FileDesc, UseArena, Copy>        12.1µs ± 2%  12.1µs ± 1%     ~     (p=0.143 n=18+18)
BM_Parse_Proto2<FileDesc, InitBlock, Copy>       11.9µs ± 3%  11.9µs ± 1%     ~     (p=0.076 n=17+19)
BM_Parse_Proto2<FileDescSV, InitBlock, Alias>    13.2µs ± 1%  13.2µs ± 1%     ~     (p=0.053 n=19+19)
BM_SerializeDescriptor_Proto2                    5.97µs ± 4%  5.90µs ± 4%     ~     (p=0.093 n=17+19)
BM_SerializeDescriptor_Upb                       10.4µs ± 1%  10.4µs ± 1%     ~     (p=0.909 n=17+18)

name                                           old time/op             new time/op             delta
BM_ArenaOneAlloc                                 18.7ns ± 2%             18.6ns ± 0%     ~           (p=0.607 n=18+17)
BM_ArenaInitialBlockOneAlloc                     6.29ns ± 1%             5.74ns ± 1%   -8.71%        (p=0.000 n=17+19)
BM_ArenaFuseUnbalanced/2                         44.1ns ± 1%             60.6ns ± 1%  +37.21%        (p=0.000 n=17+19)
BM_ArenaFuseUnbalanced/8                          371ns ± 2%              500ns ± 1%  +35.02%        (p=0.000 n=19+16)
BM_ArenaFuseUnbalanced/64                        3.53µs ± 1%             4.72µs ± 1%  +33.85%        (p=0.000 n=18+19)
BM_ArenaFuseUnbalanced/128                       7.22µs ± 1%             9.73µs ± 2%  +34.87%        (p=0.000 n=16+19)
BM_ArenaFuseBalanced/2                           44.5ns ± 2%             61.5ns ± 1%  +38.22%        (p=0.000 n=20+17)
BM_ArenaFuseBalanced/8                            373ns ± 2%              510ns ± 1%  +36.58%        (p=0.000 n=19+16)
BM_ArenaFuseBalanced/64                          3.56µs ± 2%             4.80µs ± 1%  +34.87%        (p=0.000 n=19+19)
BM_ArenaFuseBalanced/128                         7.27µs ± 1%             9.77µs ± 1%  +34.40%        (p=0.000 n=17+19)
BM_LoadAdsDescriptor_Upb<NoLayout>               5.67ms ± 1%             5.71ms ± 1%   +0.60%        (p=0.011 n=18+20)
BM_LoadAdsDescriptor_Upb<WithLayout>             6.32ms ± 1%             6.37ms ± 1%   +0.87%        (p=0.000 n=19+18)
BM_LoadAdsDescriptor_Proto2<NoLayout>            12.1ms ± 1%             12.2ms ± 1%     ~           (p=0.126 n=18+19)
BM_LoadAdsDescriptor_Proto2<WithLayout>          12.2ms ± 1%             12.3ms ± 1%   +0.51%        (p=0.002 n=18+18)
BM_Parse_Upb_FileDesc<UseArena, Copy>            12.7µs ± 1%             12.7µs ± 1%     ~           (p=0.149 n=20+19)
BM_Parse_Upb_FileDesc<UseArena, Alias>           11.6µs ± 1%             11.6µs ± 1%     ~           (p=0.211 n=20+20)
BM_Parse_Upb_FileDesc<InitBlock, Copy>           12.5µs ± 1%             12.5µs ± 1%     ~           (p=0.986 n=18+15)
BM_Parse_Upb_FileDesc<InitBlock, Alias>          11.4µs ± 1%             11.3µs ± 1%     ~           (p=0.081 n=19+18)
BM_Parse_Proto2<FileDesc, NoArena, Copy>         25.4µs ± 1%             25.8µs ± 2%   +1.41%        (p=0.000 n=18+18)
BM_Parse_Proto2<FileDesc, UseArena, Copy>        12.1µs ± 2%             12.1µs ± 1%     ~           (p=0.558 n=19+18)
BM_Parse_Proto2<FileDesc, InitBlock, Copy>       12.0µs ± 3%             11.9µs ± 1%     ~           (p=0.165 n=17+19)
BM_Parse_Proto2<FileDescSV, InitBlock, Alias>    13.2µs ± 1%             13.2µs ± 1%     ~           (p=0.070 n=19+19)
BM_SerializeDescriptor_Proto2                    5.98µs ± 4%             5.92µs ± 3%     ~           (p=0.138 n=17+19)
BM_SerializeDescriptor_Upb                       10.4µs ± 1%             10.4µs ± 1%     ~           (p=0.858 n=17+18)
```

PiperOrigin-RevId: 518573683
pull/13171/head
Joshua Haberman 2 years ago committed by Copybara-Service
parent ea4cb79f66
commit d450990631
8 changed files with 295 additions and 43 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 1
      BUILD
  2. 161
      upb/mem/arena.c
  3. 44
      upb/mem/arena_internal.h
  4. 7
      upb/mem/arena_test.cc
  5. 111
      upb/port/atomic.h
  6. 7
      upb/port/def.inc
  7. 2
      upb/port/undef.inc
  8. 5
      upb/wire/decode.c

1
BUILD
Unescape View File

@ -104,6 +104,7 @@ package_group(
cc_library(
name = "port",
hdrs = [
"upb/port/atomic.h",
"upb/port/vsnprintf_compat.h",
],
copts = UPB_DEFAULT_COPTS,

161
upb/mem/arena.c
Unescape View File

@ -26,6 +26,7 @@
*/
#include "upb/mem/arena_internal.h"
#include "upb/port/atomic.h"
// Must be last.
#include "upb/port/def.inc"
@ -58,19 +59,42 @@ typedef struct cleanup_ent {
static const size_t memblock_reserve =
UPB_ALIGN_UP(sizeof(_upb_MemBlock), UPB_MALLOC_ALIGN);
static upb_Arena* arena_findroot(upb_Arena* a) {
/* Path splitting keeps time complexity down, see:
* https://en.wikipedia.org/wiki/Disjoint-set_data_structure */
while (a->parent != a) {
upb_Arena* next = a->parent;
a->parent = next->parent;
static upb_Arena* _upb_Arena_FindRoot(upb_Arena* a) {
uintptr_t poc = upb_Atomic_LoadAcquire(&a->parent_or_count);
while (_upb_Arena_IsTaggedPointer(poc)) {
upb_Arena* next = _upb_Arena_PointerFromTagged(poc);
uintptr_t next_poc = upb_Atomic_LoadAcquire(&next->parent_or_count);
if (_upb_Arena_IsTaggedPointer(next_poc)) {
// To keep complexity down, we lazily collapse levels of the tree. This
// keeps it flat in the final case, but doesn't cost much incrementally.
//
// Path splitting keeps time complexity down, see:
// https://en.wikipedia.org/wiki/Disjoint-set_data_structure
//
// We can safely use a relaxed atomic here because all threads doing this
// will converge on the same value and we don't need memory orderings to
// be visible.
//
// This is true because:
// - If no fuses occur, this will eventually become the root.
// - If fuses are actively occuring, the root may change, but the
// invariant is that `parent_or_count` merely points to *a* parent.
//
// In other words, it is moving towards "the" root, and that root may move
// further away over time, but the path towards that root will continue to
// be valid and the creation of the path carries all the memory orderings
// required.
upb_Atomic_StoreRelaxed(&a->parent_or_count, next_poc);
}
a = next;
poc = next_poc;
}
return a;
}
size_t upb_Arena_SpaceAllocated(upb_Arena* arena) {
arena = arena_findroot(arena);
arena = _upb_Arena_FindRoot(arena);
size_t memsize = 0;
_upb_MemBlock* block = arena->freelist;
@ -83,8 +107,15 @@ size_t upb_Arena_SpaceAllocated(upb_Arena* arena) {
return memsize;
}
uint32_t upb_Arena_DebugRefCount(upb_Arena* arena) {
return arena_findroot(arena)->refcount;
uint32_t upb_Arena_DebugRefCount(upb_Arena* a) {
// These loads could probably be relaxed, but given that this is debug-only,
// it's not worth introducing a new variant for it.
uintptr_t poc = upb_Atomic_LoadAcquire(&a->parent_or_count);
while (_upb_Arena_IsTaggedPointer(poc)) {
a = _upb_Arena_PointerFromTagged(poc);
poc = upb_Atomic_LoadAcquire(&a->parent_or_count);
}
return _upb_Arena_RefCountFromTagged(poc);
}
static void upb_Arena_addblock(upb_Arena* a, upb_Arena* root, void* ptr,
@ -108,7 +139,7 @@ static void upb_Arena_addblock(upb_Arena* a, upb_Arena* root, void* ptr,
}
static bool upb_Arena_Allocblock(upb_Arena* a, size_t size) {
upb_Arena* root = arena_findroot(a);
upb_Arena* root = _upb_Arena_FindRoot(a);
size_t block_size = UPB_MAX(size, a->last_size * 2) + memblock_reserve;
_upb_MemBlock* block = upb_malloc(root->block_alloc, block_size);
@ -139,8 +170,7 @@ static upb_Arena* arena_initslow(void* mem, size_t n, upb_alloc* alloc) {
n -= sizeof(*a);
a->block_alloc = alloc;
a->parent = a;
a->refcount = 1;
upb_Atomic_Init(&a->parent_or_count, _upb_Arena_TaggedFromRefcount(1));
a->freelist = NULL;
a->freelist_tail = NULL;
a->cleanup_metadata = upb_cleanup_metadata(NULL, false);
@ -172,8 +202,7 @@ upb_Arena* upb_Arena_Init(void* mem, size_t n, upb_alloc* alloc) {
a = UPB_PTR_AT(mem, n - sizeof(*a), upb_Arena);
a->block_alloc = alloc;
a->parent = a;
a->refcount = 1;
upb_Atomic_Init(&a->parent_or_count, _upb_Arena_TaggedFromRefcount(1));
a->last_size = UPB_MAX(128, n);
a->head.ptr = mem;
a->head.end = UPB_PTR_AT(mem, n - sizeof(*a), char);
@ -186,8 +215,7 @@ upb_Arena* upb_Arena_Init(void* mem, size_t n, upb_alloc* alloc) {
static void arena_dofree(upb_Arena* a) {
_upb_MemBlock* block = a->freelist;
UPB_ASSERT(a->parent == a);
UPB_ASSERT(a->refcount == 0);
UPB_ASSERT(_upb_Arena_RefCountFromTagged(a->parent_or_count) == 1);
while (block) {
/* Load first since we are deleting block. */
@ -208,8 +236,32 @@ static void arena_dofree(upb_Arena* a) {
}
void upb_Arena_Free(upb_Arena* a) {
a = arena_findroot(a);
if (--a->refcount == 0) arena_dofree(a);
uintptr_t poc = upb_Atomic_LoadAcquire(&a->parent_or_count);
retry:
while (_upb_Arena_IsTaggedPointer(poc)) {
a = _upb_Arena_PointerFromTagged(poc);
poc = upb_Atomic_LoadAcquire(&a->parent_or_count);
}
// compare_exchange or fetch_sub are RMW operations, which are more
// expensive then direct loads. As an optimization, we only do RMW ops
// when we need to update things for other threads to see.
if (poc == _upb_Arena_TaggedFromRefcount(1)) {
arena_dofree(a);
return;
}
if (upb_Atomic_CompareExchangeStrongAcqRel(
&a->parent_or_count, &poc,
_upb_Arena_TaggedFromRefcount(_upb_Arena_RefCountFromTagged(poc) -
1))) {
// We were >1 and we decremented it successfully, so we are done.
return;
}
// We failed our update, so someone has done something, retry the whole
// process, but the failed exchange reloaded `poc` for us.
goto retry;
}
bool upb_Arena_AddCleanup(upb_Arena* a, void* ud, upb_CleanupFunc* func) {
@ -234,33 +286,80 @@ bool upb_Arena_AddCleanup(upb_Arena* a, void* ud, upb_CleanupFunc* func) {
}
bool upb_Arena_Fuse(upb_Arena* a1, upb_Arena* a2) {
upb_Arena* r1 = arena_findroot(a1);
upb_Arena* r2 = arena_findroot(a2);
if (r1 == r2) return true; /* Already fused. */
/* Do not fuse initial blocks since we cannot lifetime extend them. */
// SAFE IN THE PRESENCE OF FUSE/FREE RACES BUT NOT IN THE
// PRESENCE OF FUSE/FUSE RACES!!!
//
// `parent_or_count` has two disctint modes
// - parent pointer mode
// - refcount mode
//
// In parent pointer mode, it may change what pointer it refers to in the
// tree, but it will always approach a root. Any operation that walks the
// tree to the root may collapse levels of the tree concurrently.
//
// In refcount mode, any free operation may lower the refcount.
//
// Only a fuse operation may increase the refcount.
// Only a fuse operation may switch `parent_or_count` from parent mode to
// refcount mode.
//
// Given that we do not allow fuse/fuse races, we may rely on the invariant
// that only refcounts can change once we have found the root. Because the
// threads doing the fuse must hold references, we can guarantee that no
// refcounts will reach zero concurrently.
upb_Arena* r1 = _upb_Arena_FindRoot(a1);
upb_Arena* r2 = _upb_Arena_FindRoot(a2);
if (r1 == r2) return true; // Already fused.
// Do not fuse initial blocks since we cannot lifetime extend them.
if (upb_cleanup_has_initial_block(r1->cleanup_metadata)) return false;
if (upb_cleanup_has_initial_block(r2->cleanup_metadata)) return false;
/* Only allow fuse with a common allocator */
// Only allow fuse with a common allocator
if (r1->block_alloc != r2->block_alloc) return false;
/* We want to join the smaller tree to the larger tree.
* So swap first if they are backwards. */
if (r1->refcount < r2->refcount) {
uintptr_t r1_poc = upb_Atomic_LoadAcquire(&r1->parent_or_count);
uintptr_t r2_poc = upb_Atomic_LoadAcquire(&r2->parent_or_count);
UPB_ASSERT(_upb_Arena_IsTaggedRefcount(r1_poc));
UPB_ASSERT(_upb_Arena_IsTaggedRefcount(r2_poc));
// Keep the tree shallow by joining the smaller tree to the larger.
if (_upb_Arena_RefCountFromTagged(r1_poc) <
_upb_Arena_RefCountFromTagged(r2_poc)) {
upb_Arena* tmp = r1;
r1 = r2;
r2 = tmp;
uintptr_t tmp_poc = r1_poc;
r1_poc = r2_poc;
r2_poc = tmp_poc;
}
/* r1 takes over r2's freelist and refcount. */
r1->refcount += r2->refcount;
// r1 takes over r2's freelist, this must happen before we update
// refcounts since the refcount carriers the memory dependencies.
if (r2->freelist_tail) {
UPB_ASSERT(r2->freelist_tail->next == NULL);
r2->freelist_tail->next = r1->freelist;
r1->freelist = r2->freelist;
}
r2->parent = r1;
// The moment we install `r1` as the parent for `r2` all racing frees may
// immediately begin decrementing `r1`'s refcount. So we must install all the
// refcounts that we know about first to prevent a premature unref to zero.
uint32_t r2_refcount = _upb_Arena_RefCountFromTagged(r2_poc);
upb_Atomic_AddRelease(&r1->parent_or_count, ((uintptr_t)r2_refcount) << 1);
// When installing `r1` as the parent for `r2` racing frees may have changed
// the refcount for `r2` so we need to capture the old value to fix up `r1`'s
// refcount based on the delta from what we saw the first time.
r2_poc = upb_Atomic_ExchangeAcqRel(&r2->parent_or_count,
_upb_Arena_TaggedFromPointer(r1));
UPB_ASSERT(_upb_Arena_IsTaggedRefcount(r2_poc));
uint32_t delta_refcount = r2_refcount - _upb_Arena_RefCountFromTagged(r2_poc);
if (delta_refcount != 0) {
upb_Atomic_SubRelease(&r1->parent_or_count, ((uintptr_t)delta_refcount)
<< 1);
}
return true;
}

44
upb/mem/arena_internal.h
Unescape View File

@ -49,21 +49,47 @@ struct upb_Arena {
/* When multiple arenas are fused together, each arena points to a parent
* arena (root points to itself). The root tracks how many live arenas
* reference it. */
uint32_t refcount; /* Only used when a->parent == a */
struct upb_Arena* parent;
* reference it.
*
* The low bit is tagged:
* 0: pointer to parent
* 1: count, left shifted by one
*/
UPB_ATOMIC uintptr_t parent_or_count;
/* Linked list of blocks to free/cleanup. */
_upb_MemBlock *freelist, *freelist_tail;
};
#ifdef __cplusplus
extern "C" {
#endif
UPB_INLINE bool _upb_Arena_IsTaggedRefcount(uintptr_t parent_or_count) {
return (parent_or_count & 1) == 1;
}
#ifdef __cplusplus
} /* extern "C" */
#endif
UPB_INLINE bool _upb_Arena_IsTaggedPointer(uintptr_t parent_or_count) {
return (parent_or_count & 1) == 0;
}
UPB_INLINE uint32_t _upb_Arena_RefCountFromTagged(uintptr_t parent_or_count) {
UPB_ASSERT(_upb_Arena_IsTaggedRefcount(parent_or_count));
return parent_or_count >> 1;
}
UPB_INLINE uintptr_t _upb_Arena_TaggedFromRefcount(uint32_t refcount) {
uintptr_t parent_or_count = (((uintptr_t)refcount) << 1) | 1;
UPB_ASSERT(_upb_Arena_IsTaggedRefcount(parent_or_count));
return parent_or_count;
}
UPB_INLINE upb_Arena* _upb_Arena_PointerFromTagged(uintptr_t parent_or_count) {
UPB_ASSERT(_upb_Arena_IsTaggedPointer(parent_or_count));
return (upb_Arena*)parent_or_count;
}
UPB_INLINE uintptr_t _upb_Arena_TaggedFromPointer(upb_Arena* a) {
uintptr_t parent_or_count = (uintptr_t)a;
UPB_ASSERT(_upb_Arena_IsTaggedPointer(parent_or_count));
return parent_or_count;
}
#include "upb/port/undef.inc"

7
upb/mem/arena_test.cc
Unescape View File

@ -136,8 +136,9 @@ TEST(ArenaTest, FuzzSingleThreaded) {
}
}
// Disabled because this test currently fails.
TEST(ArenaTest, DISABLED_FuzzFuseFreeRace) {
#ifdef UPB_USE_C11_ATOMICS
TEST(ArenaTest, FuzzFuseFreeRace) {
Environment env;
absl::Notification done;
@ -184,4 +185,6 @@ TEST(ArenaTest, DISABLED_FuzzFuseFuseRace) {
for (auto& t : threads) t.join();
}
#endif
} // namespace

111
upb/port/atomic.h
Unescape View File

@ -0,0 +1,111 @@
/*
* Copyright (c) 2023, Google LLC
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Google LLC nor the
* names of its contributors may be used to endorse or promote products
* derived from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL Google LLC BE LIABLE FOR ANY DIRECT,
* INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef UPB_PORT_ATOMIC_H_
#define UPB_PORT_ATOMIC_H_
#include "upb/port/def.inc"
#ifdef UPB_USE_C11_ATOMICS
#include <stdatomic.h>
#include <stdbool.h>
UPB_INLINE void upb_Atomic_Init(_Atomic uintptr_t* addr, uintptr_t val) {
atomic_init(addr, val);
}
UPB_INLINE uintptr_t upb_Atomic_LoadAcquire(_Atomic uintptr_t* addr) {
return atomic_load_explicit(addr, memory_order_acquire);
}
UPB_INLINE void upb_Atomic_StoreRelaxed(_Atomic uintptr_t* addr,
uintptr_t val) {
atomic_store_explicit(addr, val, memory_order_relaxed);
}
UPB_INLINE void upb_Atomic_AddRelease(_Atomic uintptr_t* addr, uintptr_t val) {
atomic_fetch_add_explicit(addr, val, memory_order_release);
}
UPB_INLINE void upb_Atomic_SubRelease(_Atomic uintptr_t* addr, uintptr_t val) {
atomic_fetch_sub_explicit(addr, val, memory_order_release);
}
UPB_INLINE uintptr_t upb_Atomic_ExchangeAcqRel(_Atomic uintptr_t* addr,
uintptr_t val) {
return atomic_exchange_explicit(addr, val, memory_order_acq_rel);
}
UPB_INLINE bool upb_Atomic_CompareExchangeStrongAcqRel(_Atomic uintptr_t* addr,
uintptr_t* expected,
uintptr_t desired) {
return atomic_compare_exchange_strong_explicit(
addr, expected, desired, memory_order_release, memory_order_acquire);
}
#else // !UPB_USE_C11_ATOMICS
UPB_INLINE void upb_Atomic_Init(uintptr_t* addr, uintptr_t val) { *addr = val; }
UPB_INLINE uintptr_t upb_Atomic_LoadAcquire(uintptr_t* addr) { return *addr; }
UPB_INLINE void upb_Atomic_StoreRelaxed(uintptr_t* addr, uintptr_t val) {
*addr = val;
}
UPB_INLINE void upb_Atomic_AddRelease(uintptr_t* addr, uintptr_t val) {
*addr += val;
}
UPB_INLINE void upb_Atomic_SubRelease(uintptr_t* addr, uintptr_t val) {
*addr -= val;
}
UPB_INLINE uintptr_t upb_Atomic_ExchangeAcqRel(uintptr_t* addr, uintptr_t val) {
uintptr_t ret = *addr;
*addr = val;
return ret;
}
UPB_INLINE bool upb_Atomic_CompareExchangeStrongAcqRel(uintptr_t* addr,
uintptr_t* expected,
uintptr_t desired) {
if (*addr == *expected) {
*addr = desired;
return true;
} else {
*expected = *addr;
return false;
}
}
#endif
#include "upb/port/undef.inc"
#endif // UPB_PORT_ATOMIC_H_

7
upb/port/def.inc
Unescape View File

@ -190,6 +190,13 @@
#define UPB_LONGJMP(buf, val) longjmp(buf, val)
#endif
#ifdef __GNUC__
#define UPB_USE_C11_ATOMICS
#define UPB_ATOMIC _Atomic
#else
#define UPB_ATOMIC
#endif
/* UPB_PTRADD(ptr, ofs): add pointer while avoiding "NULL + 0" UB */
#define UPB_PTRADD(ptr, ofs) ((ofs) ? (ptr) + (ofs) : (ptr))

2
upb/port/undef.inc
Unescape View File

@ -68,3 +68,5 @@
#undef UPB_DESCRIPTOR_UPB_H_FILENAME
#undef UPB_DESC
#undef UPB_IS_GOOGLE3
#undef UPB_ATOMIC
#undef UPB_USE_C11_ATOMICS

5
upb/wire/decode.c
Unescape View File

@ -31,8 +31,10 @@
#include "upb/collections/array_internal.h"
#include "upb/collections/map_internal.h"
#include "upb/mem/arena_internal.h"
#include "upb/mini_table/common.h"
#include "upb/mini_table/enum_internal.h"
#include "upb/port/atomic.h"
#include "upb/wire/common_internal.h"
#include "upb/wire/decode_internal.h"
#include "upb/wire/encode.h"
@ -1305,7 +1307,8 @@ upb_DecodeStatus upb_Decode(const char* buf, size_t size, void* msg,
state.arena.head = arena->head;
state.arena.last_size = arena->last_size;
state.arena.cleanup_metadata = arena->cleanup_metadata;
state.arena.parent = arena;
upb_Atomic_Init(&state.arena.parent_or_count,
_upb_Arena_TaggedFromPointer(arena));
state.status = kUpb_DecodeStatus_Ok;
return upb_Decoder_Decode(&state, buf, msg, l, arena);

Write Preview
Loading…
Cancel
Save