Joshua Haberman
3 years ago
commit
6f18f3e57e
39 changed files with 798 additions and 452 deletions
@ -0,0 +1,39 @@ |
|||||||
|
|
||||||
|
load( |
||||||
|
"//bazel:build_defs.bzl", |
||||||
|
"UPB_DEFAULT_COPTS", |
||||||
|
) |
||||||
|
|
||||||
|
def py_extension(name, srcs, deps=[]): |
||||||
|
version_script = name + "_version_script.lds" |
||||||
|
symbol = "PyInit_" + name |
||||||
|
native.genrule( |
||||||
|
name = "gen_" + version_script, |
||||||
|
outs = [version_script], |
||||||
|
cmd = "echo 'message { global: " + symbol + "; local: *; };' > $@", |
||||||
|
) |
||||||
|
|
||||||
|
native.cc_binary( |
||||||
|
name = name, |
||||||
|
srcs = srcs, |
||||||
|
copts = UPB_DEFAULT_COPTS + [ |
||||||
|
# The Python API requires patterns that are ISO C incompatible, like |
||||||
|
# casts between function pointers and object pointers. |
||||||
|
"-Wno-pedantic", |
||||||
|
], |
||||||
|
# We use a linker script to hide all symbols except the entry point for |
||||||
|
# the module. |
||||||
|
linkopts = select({ |
||||||
|
"@platforms//os:linux": ["-Wl,--version-script,$(location :" + version_script + ")"], |
||||||
|
"@platforms//os:macos": [ |
||||||
|
"-Wl,-exported_symbol", |
||||||
|
"-Wl,_" + symbol, |
||||||
|
], |
||||||
|
}), |
||||||
|
linkshared = True, |
||||||
|
linkstatic = True, |
||||||
|
deps = deps + [ |
||||||
|
":" + version_script, |
||||||
|
"@system_python//:python_headers", |
||||||
|
], |
||||||
|
) |
||||||
@ -0,0 +1,34 @@ |
|||||||
|
|
||||||
|
# copybara:strip_for_google3_begin |
||||||
|
|
||||||
|
def pyproto_test_wrapper(name): |
||||||
|
src = name + "_wrapper.py" |
||||||
|
native.py_test( |
||||||
|
name = name, |
||||||
|
srcs = [src], |
||||||
|
legacy_create_init = False, |
||||||
|
main = src, |
||||||
|
data = ["@com_google_protobuf//:testdata"], |
||||||
|
deps = [ |
||||||
|
"//python:message_ext", |
||||||
|
"@com_google_protobuf//:python_common_test_protos", |
||||||
|
"@com_google_protobuf//:python_specific_test_protos", |
||||||
|
"@com_google_protobuf//:python_srcs", |
||||||
|
], |
||||||
|
) |
||||||
|
|
||||||
|
# copybara:replace_for_google3_begin |
||||||
|
# |
||||||
|
# def pyproto_test_wrapper(name): |
||||||
|
# src = name + "_wrapper.py" |
||||||
|
# native.py_test( |
||||||
|
# name = name, |
||||||
|
# srcs = [src], |
||||||
|
# main = src, |
||||||
|
# deps = [ |
||||||
|
# "//net/proto2/python/internal:" + name + "_for_deps", |
||||||
|
# "//net/proto2/python/public:use_upb_protos", |
||||||
|
# ], |
||||||
|
# ) |
||||||
|
# |
||||||
|
# copybara:replace_for_google3_end |
||||||
@ -0,0 +1,254 @@ |
|||||||
|
|
||||||
|
# upb vs. C++ Protobuf Design |
||||||
|
|
||||||
|
[upb](https://github.com/protocolbuffers/upb) is a small C protobuf library. |
||||||
|
While some of the design follows in the footsteps of the C++ Protobuf Library, |
||||||
|
upb departs from C++'s design in several key ways. This document compares |
||||||
|
and contrasts the two libraries on several design points. |
||||||
|
|
||||||
|
## Design Goals |
||||||
|
|
||||||
|
Before we begin, it is worth calling out that upb and C++ have different design |
||||||
|
goals, and this motivates some of the differences we will see. |
||||||
|
|
||||||
|
C++ protobuf is a user-level library: it is designed to be used directly by C++ |
||||||
|
applications. These applications will expect a full-featured C++ API surface |
||||||
|
that uses C++ idioms. The C++ library is also willing to add features to |
||||||
|
increase server performance, even if these features would add size or complexity |
||||||
|
to the library. Because C++ protobuf is a user-level library, API stability is |
||||||
|
of utmost importance: breaking API changes are rare and carefully managed when |
||||||
|
they do occur. The focus on C++ also means that ABI compatibility with C is not |
||||||
|
a priority. |
||||||
|
|
||||||
|
upb, on the other hand, is designed primarily to be wrapped by other languages. |
||||||
|
It is a C protobuf kernel that forms the basis on which a user-level protobuf |
||||||
|
library can be built. This means we prefer to keep the API surface as small and |
||||||
|
orthogonal as possible. While upb supports all protobuf features required for |
||||||
|
full conformance, upb prioritizes simplicity and small code size, and avoids |
||||||
|
adding features like lazy fields that can accelerate some use cases but at great |
||||||
|
cost in terms of complexity. As upb is not aimed directly at users, there is |
||||||
|
much more freedom to make API-breaking changes when necessary, which helps the |
||||||
|
core to stay small and simple. We want to be compatible with all FFI |
||||||
|
interfaces, so C ABI compatibility is a must. |
||||||
|
|
||||||
|
Despite these differences, C++ protos and upb offer [roughly the same core set |
||||||
|
of features](https://github.com/protocolbuffers/upb#features). |
||||||
|
|
||||||
|
## Arenas |
||||||
|
|
||||||
|
upb and C++ protos both offer arena allocation, but there are some key |
||||||
|
differences. |
||||||
|
|
||||||
|
### C++ |
||||||
|
|
||||||
|
As a matter of history, when C++ protos were open-sourced in 2008, they did not |
||||||
|
support arenas. Originally there was only unique ownership, whereby each |
||||||
|
message uniquely owns all child messages and will free them when the parent is |
||||||
|
freed. |
||||||
|
|
||||||
|
Arena allocation was added as a feature in 2014 as a way of dramatically |
||||||
|
reducing allocation and (especially) deallocation costs. But the library was |
||||||
|
not at liberty to remove the unique ownership model, because it would break far |
||||||
|
too many users. As a result, C++ has supported a **hybrid allocation model** |
||||||
|
ever since, allowing users to allocate messages either directly from the |
||||||
|
stack/heap or from an arena. The library attempts to ensure that there are |
||||||
|
no dangling pointers by performing automatic copies in some cases (for example |
||||||
|
`a->set_allocated_b(b)`, where `a` and `b` are on different arenas). |
||||||
|
|
||||||
|
C++'s arena object itself `google::protobuf::Arena` is **thread-safe** by |
||||||
|
design, which allows users to allocate from multiple threads simultaneously |
||||||
|
without external synchronization. The user can supply an initial block of |
||||||
|
memory to the arena, and can choose some parameters to control the arena block |
||||||
|
size. The user can also supply block alloc/dealloc functions, but the alloc |
||||||
|
function is expected to always return some memory. The C++ library in general |
||||||
|
does not attempt to handle out of memory conditions. |
||||||
|
|
||||||
|
### upb |
||||||
|
|
||||||
|
upb uses **arena allocation exclusively**. All messages must be allocated from |
||||||
|
an arena, and can only be freed by freeing the arena. It is entirely the user's |
||||||
|
responsibility to ensure that there are no dangling pointers: when a user sets a |
||||||
|
message field, this will always trivially overwrite the pointer and will never |
||||||
|
perform an implicit copy. |
||||||
|
|
||||||
|
upb's `upb::Arena` is **thread-compatible**, which means it cannot be used |
||||||
|
concurrently without synchronization. The arena can be seeded with an initial |
||||||
|
block of memory, but it does not explicitly support any parameters for choosing |
||||||
|
block size. It support a custom alloc/dealloc function, and this function is |
||||||
|
allowed to return `NULL` if no dynamic memory is available. This allows upb |
||||||
|
arenas to have a max/fixed size, and makes it possible in theory to write code |
||||||
|
that is tolerant to out-of-memory errors. |
||||||
|
|
||||||
|
upb's arena also supports a novel operation known as **fuse**, which joins two |
||||||
|
arenas together into a single lifetime. Though both arenas must still be freed |
||||||
|
separately, none of the memory will actually be freed until *both* arenas have |
||||||
|
been freed. This is useful for avoiding dangling pointers when reparenting a |
||||||
|
message with one that may be on a different arena. |
||||||
|
|
||||||
|
### Comparison |
||||||
|
|
||||||
|
**hybrid allocation vs. arena-only**: |
||||||
|
* The C++ hybrid allocation model introduces a great deal of complexity and |
||||||
|
unpredictability into the library. upb benefits from having a much simpler |
||||||
|
and more predictable design. |
||||||
|
* Some of the complexity in C++'s hybrid model arises from the fact that arenas |
||||||
|
were added after the fact. Designing for a hybrid model from the outset |
||||||
|
would likely yield a simpler result. |
||||||
|
* Unique ownership does support some usage patterns that arenas cannot directly |
||||||
|
accommodate. For example, you can reparent a message and the child will precisely |
||||||
|
follow the lifetime of its new parent. An arena would require you to either |
||||||
|
perform a deep copy or extend the lifetime. |
||||||
|
|
||||||
|
**thread-compatible vs. thread-safe arena** |
||||||
|
* A thread-safe arena (as in C++) is safer and easier to use. A thread-compatible |
||||||
|
arena requires that the user prove that the arena cannot be used concurrently. |
||||||
|
* [Thread Sanitizer](https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual) |
||||||
|
is far more accessible than it was in 2014 (when C++ introduced a thread-safe |
||||||
|
arena). We now have more tools at our disposal to ensure that we do not trigger |
||||||
|
data races in a thread-compatible arena like upb. |
||||||
|
* Thread-compatible arenas are more performant. |
||||||
|
* Thread-compatible arenas have a far simpler implementation. The C++ thread-safe |
||||||
|
arena relies on thread-local variables, which introduce complications on some |
||||||
|
platforms. It also requires far more subtle reasoning for correctness and |
||||||
|
performance. |
||||||
|
|
||||||
|
**fuse vs. no fuse** |
||||||
|
* The `upb_Arena_Fuse()` operation is a key part of how upb supports reparenting |
||||||
|
of messages when the parent may be on a different arena. Without this, upb has |
||||||
|
no way of supporting `foo.bar = bar` in dynamic languages without performing a |
||||||
|
deep copy. |
||||||
|
* A downside of `upb_Arena_Fuse()` is that passing an arena to a function can allow |
||||||
|
that function to extend the lifetime of the arena in potentially |
||||||
|
unpredictable ways. This can be prevented if necessary, as fuse can fail, eg. if |
||||||
|
one arena has an initial block. But this adds some complexity by requiring callers |
||||||
|
to handle the case where fuse fails. |
||||||
|
|
||||||
|
## Code Generation vs. Tables |
||||||
|
|
||||||
|
The C++ protobuf library has always been built around code generation, while upb |
||||||
|
generates only tables. In other words, `foo.pb.cc` files contain functions, |
||||||
|
whereas `foo.upb.c` files emit only data structures. |
||||||
|
|
||||||
|
### C++ |
||||||
|
|
||||||
|
C++ generated code emits a large number of functions into `foo.pb.cc` files. |
||||||
|
An incomplete list: |
||||||
|
|
||||||
|
* `FooMsg::FooMsg()` (constructor): initializes all fields to their default value. |
||||||
|
* `FooMsg::~FooMsg()` (destructor): frees any present child messages. |
||||||
|
* `FooMsg::Clear()`: clears all fields back to their default/empty value. |
||||||
|
* `FooMsg::_InternalParse()`: generated code for parsing a message. |
||||||
|
* `FooMsg::_InternalSerialize()`: generated code for serializing a message. |
||||||
|
* `FooMsg::ByteSizeLong()`: calculates serialized size, as a first pass before serializing. |
||||||
|
* `FooMsg::MergeFrom()`: copies/appends present fields from another message. |
||||||
|
* `FooMsg::IsInitialized()`: checks whether required fields are set. |
||||||
|
|
||||||
|
This code lives in the `.text` section and contains function calls to the generated |
||||||
|
classes for child messages. |
||||||
|
|
||||||
|
### upb |
||||||
|
|
||||||
|
upb does not generate any code into `foo.upb.c` files, only data structures. upb uses a |
||||||
|
compact data table known as a *mini table* to represent the schema and all fields. |
||||||
|
|
||||||
|
upb uses mini tables to perform all of the operations that would traditionally be done |
||||||
|
with generated code. Revisiting the list from the previous section: |
||||||
|
|
||||||
|
* `FooMsg::FooMsg()` (constructor): upb instead initializes all messages with `memset(msg, 0, size)`. |
||||||
|
Non-zero defaults are injected in the accessors. |
||||||
|
* `FooMsg::~FooMsg()` (destructor): upb messages are freed by freeing the arena. |
||||||
|
* `FooMsg::Clear()`: can be performed with `memset(msg, 0, size)`. |
||||||
|
* `FooMsg::_InternalParse()`: upb's parser uses mini tables as data, instead of generating code. |
||||||
|
* `FooMsg::_InternalSerialize()`: upb's serializer also uses mini-tables instead of generated code. |
||||||
|
* `FooMsg::ByteSizeLong()`: upb performs serialization in reverse so that an initial pass is not required. |
||||||
|
* `FooMsg::MergeFrom()`: upb supports this via serialize+parse from the other message. |
||||||
|
* `FooMsg::IsInitialized()`: upb's encoder and decoder have special flags to check for required fields. |
||||||
|
A util library `upb/util/required_fields.h` handles the corner cases. |
||||||
|
|
||||||
|
### Comparison |
||||||
|
|
||||||
|
If we compare compiled code size, upb is far smaller. Here is a comparison of the code |
||||||
|
size of a trivial binary that does nothing but a parse and serialize of `descriptor.proto`. |
||||||
|
This means we are seeing both the overhead of the core library itself as well as the |
||||||
|
generated code (or table) for `descriptor.proto`. (For extra clarity we should break this |
||||||
|
down by generated code vs core library in the future). |
||||||
|
|
||||||
|
|
||||||
|
| Library | `.text` | `.data` | `.bss` | |
||||||
|
|------------ |---------|---------|--------| |
||||||
|
| upb | 26Ki | 0.6Ki | 0.01Ki | |
||||||
|
| C++ (lite) | 187Ki | 2.8Ki | 1.25Ki | |
||||||
|
| C++ (code size) | 904Ki | 6.1Ki | 1.88Ki | |
||||||
|
| C++ (full) | 983Ki | 6.1Ki | 1.88Ki | |
||||||
|
|
||||||
|
"C++ (code size)" refers to protos compiled with `optimize_for = CODE_SIZE`, a mode |
||||||
|
in which generated code contains reflection only, in an attempt to make the |
||||||
|
generated code size smaller (however it requires the full runtime instead |
||||||
|
of the lite runtime). |
||||||
|
|
||||||
|
## Bifurcated vs. Optional Reflection |
||||||
|
|
||||||
|
upb and C++ protos both offer reflection without making it mandatory. However |
||||||
|
the models for enabling/disabling reflection are very different. |
||||||
|
|
||||||
|
### C++ |
||||||
|
|
||||||
|
C++ messages offer full reflection by default. Messages in C++ generally |
||||||
|
derive from `Message`, and the base class provides a member function |
||||||
|
`Reflection* Message::GetReflection()` which returns the reflection object. |
||||||
|
|
||||||
|
It follows that any message deriving from `Message` will always have reflection |
||||||
|
linked into the binary, whether or not the reflection object is ever used. |
||||||
|
Because `GetReflection()` is a function on the base class, it is not possible |
||||||
|
to statically determine if a given message's reflection is used: |
||||||
|
|
||||||
|
```c++ |
||||||
|
Reflection* GetReflection(const Message& message) { |
||||||
|
// Can refer to any message in the whole binary. |
||||||
|
return message.GetReflection(); |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
The C++ library does provide a way of omitting reflection: `MessageLite`. We can |
||||||
|
cause a message to be lite in two different ways: |
||||||
|
|
||||||
|
* `optimize_for = LITE_RUNTIME` in a `.proto` file will cause all messages in that |
||||||
|
file to be lite. |
||||||
|
* `lite` as a codegen param: this will force all messages to lite, even if the |
||||||
|
`.proto` file does not have `optimize_for = LITE_RUNTIME`. |
||||||
|
|
||||||
|
A lite message will derive from `MessageLite` instead of `Message`. Since |
||||||
|
`MessageLite` has no `GetReflection()` function, this means no reflection is |
||||||
|
available, so we can avoid taking the code size hit. |
||||||
|
|
||||||
|
### upb |
||||||
|
|
||||||
|
upb does not have the `Message` vs. `MessageLite` bifurcation. There is only one |
||||||
|
kind of message type `upb_Message`, which means there is no need to configure in |
||||||
|
a `.proto` file which messages will need reflection and which will not. |
||||||
|
Every message has the *option* to link in reflection from a separate `foo.upbdefs.o` |
||||||
|
file, without needing to change the message itself in any way. |
||||||
|
|
||||||
|
upb does not provide the equivalent of `Message::GetReflection()`: there is no |
||||||
|
facility for retrieving the reflection of a message whose type is not known statically. |
||||||
|
It would be possible to layer such a facility on top of the upb core, though this |
||||||
|
would probably require some kind of code generation. |
||||||
|
|
||||||
|
### Comparison |
||||||
|
|
||||||
|
* Most messages in C++ will not bother to declare themselves as "lite". This means |
||||||
|
that many C++ messages will link in reflection even when it is never used, bloating |
||||||
|
binaries unnecessarily. |
||||||
|
* `optimize_for = LITE_RUNTIME` is difficult to use in practice, because it prevents |
||||||
|
any non-lite protos from `import`ing that file. |
||||||
|
* Forcing all protos to lite via a codegen parameter (for example, when building for |
||||||
|
mobile) is more practical than `optimize_for = LITE_RUNTIME`. But this will break |
||||||
|
the compile for any code that tries to upcast to `Message`, or tries to use a |
||||||
|
non-lite method. |
||||||
|
* The one major advantage of the C++ model is that it can support `msg.DebugString()` |
||||||
|
on a type-erased proto. For upb you have to explicitly pass the `upb_MessageDef*` |
||||||
|
separately if you want to perform an operation like printing a proto to text format. |
||||||
|
|
||||||
|
## Explicit Registration vs. Globals |
||||||
|
|
||||||
|
TODO |
||||||
Loading…
Reference in new issue