Also changed `space_allocated` to a `uintptr_t` since it's a sum of `size_t`s, and unfortunately due to a lack of `_Generic` availability in default msvc it needs a conversion when being added.
PiperOrigin-RevId: 712994963
* Add acquire/release where necessary for all atomic ops
* Add sentinel member to ensure safe publication when tsan is active; tsan will not catch the previous errors without this member.
* For all operations using relaxed memory order, comment why relaxed order is safe
* Add a test that exercises racy fuses and space allocated checks without mutexes or other memory barriers from the test harness. This test proved the existence of several races not caught by the existing tests, including one with a confident comment about why relaxed memory order was safe.
* Add a test that exercises racing allocation and destruction among fused arenas, which doesn't use locks and substitutes a custom allocator that verifies its memory blocks.
Test coverage and assert/tsan instrumentation is now sufficient to cause test failures if any call site is further relaxed.
PiperOrigin-RevId: 712751905
This fixes:
* MSVC with `/std:c11 /experimental:c11atomics` on recent versions now emits atomics
* Clang with `-std=c11 -fgnuc-version=0` now emits atomics
* Clang and GCC 14 when built with `-std=c99 -pedantic-errors` will now compile, and not emit atomics
PiperOrigin-RevId: 712538312
After using UPB I noticed that field accessors were not getting inlined properly using Clang. The source appears to be the UPB_FORCEINLINE macro falling back to just being `static` when underneath clang despite clang having full support of the required attributes.
Closes#17433
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/17433 from Vadmeme:main 57204e78cd
PiperOrigin-RevId: 699195477
The goal of the `names.h` convention is to have a single canonical place where a code generator can define the set of symbols it exports to other code generators, and a canonical place where the name mangling logic is implemented.
Each upb code generator now has its own `names.h` file defining the symbols that it owns & exports:
* `third_party/upb/upb_generator/c/names.h` (for `foo.upb.h` files)
* `third_party/upb/upb_generator/minitable/names.h` (for `foo.upb_minitable.h` files)
* `third_party/upb/upb_generator/reflection/names.h` (for `foo.upbdefs.h` files)
This is a significant improvement over the previous situation where the name mangling functions were co-mingled in `common.h`/`mangle.h`, or sprinkled throughout the generators, with no clear structure for which code generator owns which symbols.
With this structure in place, the visibility lists for the various `names.h` files provide a clear dependency graph for how different generators depend on each other. In general, we want to keep dependencies on the "C" code generator to a minimum, since it is the largest and most complicated of upb's generated APIs, and is also the most prone to symbol name clashes.
Note that upb's `names.h` headers are somewhat unusual, in that we do not want them to depend on C++'s reflection or upb's reflection. Most `names.h` headers in protobuf would use types like `proto2::Descriptor`, but we don't want upb to depend on C++ reflection, especially during its bootstrapping process. We also don't want to force users to build upb defs just to use these name mangling functions. So we use only plain string types like `absl::string_view` and `std::string`.
PiperOrigin-RevId: 672397247
This also increases compliance by adding `default_applicable_licenses` to several `BUILD` files that previously did not have it.
PiperOrigin-RevId: 670784686
This CL is mostly a no-op, except that now google3-only code is actually stripped from OSS, instead of being preserved in `# begin:google_only` blocks.
This follows the conventions of the greater Copybara ecosystem.
PiperOrigin-RevId: 669513564
Our bootstrapping setup compiles multiple versions of the generated code for `descriptor.proto` and `plugin.proto`, one for each stage of the bootstrap. For source files (`.c`), we can always select the correct version of the file in the BUILD rules, but for header files we need to make sure the correct stage's file is always selected via `#include`.
Previously we used `cc_library(includes=[])` to make it appear as though our bootstrapped headers had the same names as the "real" headers. This allowed a lot of the code to be agnostic to whether a bootstrap header was being used, which simplified things because we did not have to change the code performing the `#include`.
Unfortunately, due to build system limitations, this sometimes led to the incorrect header getting included. This should not have been possible, because we had a clean BUILD graph that should have removed all ambiguity about which header should be available. But in non-sandboxed builds, the compiler was able to find headers that were not actually in `deps=[]`, and worse it preferred those headers over the headers that actually were in `deps=[]`. This led to unintended results and errors about layering check violations.
This CL fixes the problem by removing all use of `includes=[]`. We now spell a full pathname to all bootstrap headers, so this class of errors is no longer possible. Unfortunately this adds some complexity, as we have to hard-code these full paths in several places.
A nice improvement in this CL is that `bootstrap_upb_proto_library()` can now only be used for bootstrapping; it only exposes the `descriptor_bootstrap.h` / `plugin_bootstrap.h` files. Anyone wanting to use the normal `net/proto2/proto/descriptor.upb.h` file should depend on `//net/proto2/proto:descriptor_upb_c_proto` target instead.
PiperOrigin-RevId: 664953196
The functionality is enabled when the proto_one_output_per_message option used by C++ Lite is enabled.
This mirrors the behavior of C++ lite protos.
PiperOrigin-RevId: 642327960
This is needed to make protobuf/bazel package minimal for other proto rules.
Keep 4 public bzl files in upb/bazel. They end up under protobuf/bazel, and they are legitimately used by other repositories.
Move upb_proto_library_internal/* under bazel/private. Those are utilities used in the rules. Moving them one level deeper makes protobuf/bazel package clean for other rules.
Move build_defs.bzl and amalgamation under /upb/bazel. Those are utilities used in the build.
Move lua.BUILD and python* uner /python/dist. Those are used in the WORKSPACE dependency setup.
PiperOrigin-RevId: 621442236
Three of these runtimes are based on upb, and the fourth is based on the Java runtime. Both of these already have editions support, so this was mostly just a matter of:
- Advertising support to allow editions codegen
- Stripping features from the runtime options
- Hooking up conformance tests
- Adding some lightweight editions tests
There are also a few minor orthogonal fixes included here:
- Ruby's upb hack for treating all enums as open enums needed tweaking
- The `enable_editions` flag is no longer needed in our internal proto rules
PiperOrigin-RevId: 616256211
Creates a tracing helper function to associate mini tables with proto full names to use for upcoming upb metrics collectors.
PiperOrigin-RevId: 613290255
To satisfy the layering check, we need to depend on :gtest for the headers, in
addition to :gtest_main which provides the main() function.
There are a bunch of formatting changes as a side effect of this, but they
should be harmless.
PiperOrigin-RevId: 594318263
gRPC auto-generates CMake and other build configs from the Bazel build graph,
but the logic for doing this does not know how to handle filegroups. This
change works around that problem by making the `:port` target refer directly to
the `.inc` files instead of going through a filegroup. This solution is not
ideal but I think it's probably the best way to unblock progress for now.
PiperOrigin-RevId: 578333954
gRPC auto-generates CMake and other build configs from the Bazel build graph,
but the logic for doing this does not know how to handle filegroups. This
change works around that problem by making the `:port` target refer directly to
the `.inc` files instead of going through a filegroup. This solution is not
ideal but I think it's probably the best way to unblock progress for now.
PiperOrigin-RevId: 578333954
This change moves almost everything in the `upb/` directory up one level, so
that for example `upb/upb/generated_code_support.h` becomes just
`upb/generated_code_support.h`. The only exceptions I made to this were that I
left `upb/cmake` and `upb/BUILD` where they are, mostly because that avoids
conflict with other files and the current locations seem reasonable for now.
The `python/` directory is a little bit of a challenge because we had to merge
the existing directory there with `upb/python/`. I made `upb/python/BUILD` into
the BUILD file for the merged directory, and it effectively loads the contents
of the other BUILD file via `python/build_targets.bzl`, but I plan to clean
this up soon.
PiperOrigin-RevId: 568651768
This will allow us to run all the upb tests from the protobuf repo, which
already uses this flag. I just had to remove a couple glob patterns that did
not match any files.
PiperOrigin-RevId: 559434154
Clang and GCC differ on how they detect Address Sanitizer. Support both.
Closes#1424
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/upb/pull/1424 from protocolbuffers:asan-clang 491a5ee4cfd24c8eb281f894de0cf4384525c46a
PiperOrigin-RevId: 553805994
The fields of upb_MiniTableField are intended to be internal-only, accessed only through public functions like `upb_MiniTable_GetSubMessageTable()`. But over time, clients have started accessing many of these fields directly. This is an easy mistake to make, as there is no clear signal that the fields should not be used in applications. This makes the implementation difficult to change without breaking users.
The new `UPB_PRIVATE()` macro appends an unpredictable string to each private symbol. This makes it very difficult to accidentally use a private symbol, since users would need to write something like `field->submsg_index_dont_copy_me__upb_internal_use_only`. This is still possible to do, but it leaves a clear wart in the code showing that an an encapsulation break has occurred. The `UPB_PRIVATE()` macro itself is defined in `port/def.inc`, which users cannot include directly.
Once we land this, more such CLs will follow for the other fields of `upb_MiniTable*`. We will add inline functions as needed to provide the semantic functionality needed by users.
PiperOrigin-RevId: 523166901
Protobuf Team Bot
y-yagi
Mike Kruskal
Vadmeme
Joshua Haberman
Dave MacLachlan
Eric Salo
Adam Cozzette
Jason Lunn
Matt Kulukundis