protobuf/DESIGN.md

μpb Design

NOTE: the design described here is being implemented currently, but is not yet complete. The repo is in heavy transition right now.

μpb has the following design goals:

• C89 compatible.
• small code size (both for the core library and generated messages).
• fast performance (hundreds of MB/s).
• idiomatic for C programs.
• easy to wrap in high-level languages (Python, Ruby, Lua, etc) with good performance and all standard protobuf features.
• hands-off about memory management, allowing for easy integration with existing VMs and/or garbage collectors.
• offers binary ABI compatibility between apps, generated messages, and the core library (doesn't require re-generating messages or recompiling your application when the core library changes).
• provides all features that users expect from a protobuf library (generated messages in C, reflection, text format, etc.).
• layered, so the core is small and doesn't require descriptors.
• tidy about symbol references, so that any messages or features that aren't used by a C program can have their code GC'd by the linker.
• possible to use protobuf binary format without leaking message/field names into the binary.

μpb accomplishes these goals by keeping a very small core that does not contain descriptors. We need some way of knowing what fields are in each message and where they live, but instead of descriptors, we keep a small/lightweight summary of the .proto file. We call this a upb_msglayout. It contains the bare minimum of what we need to know to parse and serialize protobuf binary format into our internal representation for messages, upb_msg.

The core then contains functions to parse/serialize a message, given a upb_msg* and a const upb_msglayout*.

This approach is similar to nanopb which also compiles message definitions to a compact, internal representation without names. However nanopb does not aim to be a fully-featured library, and has no support for text format, JSON, or descriptors. μpb is unique in that it has a small core similar to nanopb (though not quite as small), but also offers a full-featured protobuf library for applications that want reflection, text format, JSON format, etc.

Without descriptors, the core doesn't have access to field names, so it cannot parse/serialize to protobuf text format or JSON. Instead this functionality lives in separate modules that depend on the module implementing descriptors. With the descriptor module we can parse/serialize binary descriptors and validate that they follow all the rules of protobuf schemas.

To provide binary compatibility, we version the structs that generated messages use to create a upb_msglayout*. The current initializers are upb_msglayout_msginit_v1, upb_msglayout_fieldinit_v1, etc. Then upb_msglayout* uses these as its internal representation. If upb changes its internal representation for a upb_msglayout*, it will also include code to convert the old representation to the new representation. This will use some more memory/CPU at runtime to convert between the two, but apps that statically link μpb will never need to worry about this.

TODO

The current state of the repo is quite different than what is described above. Here are the major items that need to be implemented.

1. implement the core generic protobuf binary encoder/decoder that uses a upb_msglayout*.
2. remove all mention of handlers, sink, etc. from core into their own module. All of the handlers stuff needs substantial revision, but moving it out of core is the first priority.
3. move all of the def/refcounted stuff out of core. The defs also need substantial revision, but moving them out of core is the first priority.
4. revise our generated code until it is in a state where we feel comfortable committing to API/ABI stability for it. This may involve moving different parts of the generated code into separate files, like keeping the serialized descriptor in a separate file from the compact msglayout.
5. revise all of the existing encoders/decoders and handlers. We probably will want to keep handlers, since they let us decouple encoders/decoders from upb_msg, but we need to simplify all of that a LOT. Likely we will want to make handlers only per-message instead of per-field, except for variable-length fields.