Merge pull request #404 from haberman/readme
Updated some docs and removed/rearranged some obsolete stuff.pull/13171/head
Joshua Haberman
4 years ago
committed by
GitHub
commit
723c9723ea
11 changed files with 89 additions and 304 deletions
@ -1,7 +1,37 @@ |
|||||||
## <a name="cla"></a> Signing the CLA |
|
||||||
|
|
||||||
Please sign the [Google Contributor License Agreement |
# How to Contribute |
||||||
(CLA)](https://cla.developers.google.com/) |
|
||||||
before sending pull requests. For any code changes to be |
We'd love to accept your patches and contributions to this project. There are |
||||||
accepted, the CLA must be signed. It's a quick process, I |
just a few small guidelines you need to follow. |
||||||
promise! |
|
||||||
|
## Get in touch |
||||||
|
|
||||||
|
If your idea will take you more than, say, 30 minutes to |
||||||
|
implement, please get in touch first via the issue tracker |
||||||
|
to touch base about your plan. That will give an |
||||||
|
opportunity for early feedback and help avoid wasting your |
||||||
|
time. |
||||||
|
|
||||||
|
## Contributor License Agreement |
||||||
|
|
||||||
|
Contributions to this project must be accompanied by a Contributor License |
||||||
|
Agreement. You (or your employer) retain the copyright to your contribution; |
||||||
|
this simply gives us permission to use and redistribute your contributions as |
||||||
|
part of the project. Head over to <https://cla.developers.google.com/> to see |
||||||
|
your current agreements on file or to sign a new one. |
||||||
|
|
||||||
|
You generally only need to submit a CLA once, so if you've already submitted one |
||||||
|
(even if it was for a different project), you probably don't need to do it |
||||||
|
again. |
||||||
|
|
||||||
|
## Code Reviews |
||||||
|
|
||||||
|
All submissions, including submissions by project members, require review. We |
||||||
|
use GitHub pull requests for this purpose. Consult |
||||||
|
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more |
||||||
|
information on using pull requests. |
||||||
|
|
||||||
|
## Community Guidelines |
||||||
|
|
||||||
|
This project follows [Google's Open Source Community |
||||||
|
Guidelines](https://opensource.google/conduct/). |
||||||
|
|||||||
@ -1,72 +0,0 @@ |
|||||||
|
|
||||||
μpb Design |
|
||||||
---------- |
|
||||||
|
|
||||||
μpb has the following design goals: |
|
||||||
|
|
||||||
- C89 compatible. |
|
||||||
- small code size (both for the core library and generated messages). |
|
||||||
- fast performance (hundreds of MB/s). |
|
||||||
- idiomatic for C programs. |
|
||||||
- easy to wrap in high-level languages (Python, Ruby, Lua, etc) with |
|
||||||
good performance and all standard protobuf features. |
|
||||||
- hands-off about memory management, allowing for easy integration |
|
||||||
with existing VMs and/or garbage collectors. |
|
||||||
- offers binary ABI compatibility between apps, generated messages, and |
|
||||||
the core library (doesn't require re-generating messages or recompiling |
|
||||||
your application when the core library changes). |
|
||||||
- provides all features that users expect from a protobuf library |
|
||||||
(generated messages in C, reflection, text format, etc.). |
|
||||||
- layered, so the core is small and doesn't require descriptors. |
|
||||||
- tidy about symbol references, so that any messages or features that |
|
||||||
aren't used by a C program can have their code GC'd by the linker. |
|
||||||
- possible to use protobuf binary format without leaking message/field |
|
||||||
names into the binary. |
|
||||||
|
|
||||||
μpb accomplishes these goals by keeping a very small core that does not contain |
|
||||||
descriptors. We need some way of knowing what fields are in each message and |
|
||||||
where they live, but instead of descriptors, we keep a small/lightweight summary |
|
||||||
of the .proto file. We call this a `upb_msglayout`. It contains the bare |
|
||||||
minimum of what we need to know to parse and serialize protobuf binary format |
|
||||||
into our internal representation for messages, `upb_msg`. |
|
||||||
|
|
||||||
The core then contains functions to parse/serialize a message, given a `upb_msg*` |
|
||||||
and a `const upb_msglayout*`. |
|
||||||
|
|
||||||
This approach is similar to [nanopb](https://github.com/nanopb/nanopb) which |
|
||||||
also compiles message definitions to a compact, internal representation without |
|
||||||
names. However nanopb does not aim to be a fully-featured library, and has no |
|
||||||
support for text format, JSON, or descriptors. μpb is unique in that it has a |
|
||||||
small core similar to nanopb (though not quite as small), but also offers a |
|
||||||
full-featured protobuf library for applications that want reflection, text |
|
||||||
format, JSON format, etc. |
|
||||||
|
|
||||||
Without descriptors, the core doesn't have access to field names, so it cannot |
|
||||||
parse/serialize to protobuf text format or JSON. Instead this functionality |
|
||||||
lives in separate modules that depend on the module implementing descriptors. |
|
||||||
With the descriptor module we can parse/serialize binary descriptors and |
|
||||||
validate that they follow all the rules of protobuf schemas. |
|
||||||
|
|
||||||
To provide binary compatibility, we version the structs that generated messages |
|
||||||
use to create a `upb_msglayout*`. The current initializers are |
|
||||||
`upb_msglayout_msginit_v1`, `upb_msglayout_fieldinit_v1`, etc. Then |
|
||||||
`upb_msglayout*` uses these as its internal representation. If upb changes its |
|
||||||
internal representation for a `upb_msglayout*`, it will also include code to |
|
||||||
convert the old representation to the new representation. This will use some |
|
||||||
more memory/CPU at runtime to convert between the two, but apps that statically |
|
||||||
link μpb will never need to worry about this. |
|
||||||
|
|
||||||
TODO |
|
||||||
---- |
|
||||||
|
|
||||||
1. revise our generated code until it is in a state where we feel comfortable |
|
||||||
committing to API/ABI stability for it. In particular there is an open |
|
||||||
question of whether non-ABI-compatible field accesses should have a |
|
||||||
fastpath different from the ABI-compatible field access. |
|
||||||
1. Add missing features (maps, extensions, unknown fields). |
|
||||||
1. Flesh out C++ wrappers. |
|
||||||
1. *(lower-priority)*: revise all of the existing encoders/decoders and |
|
||||||
handlers. We probably will want to keep handlers, since they let us decouple |
|
||||||
encoders/decoders from `upb_msg`, but we need to simplify all of that a LOT. |
|
||||||
Likely we will want to make handlers only per-message instead of per-field, |
|
||||||
except for variable-length fields. |
|
||||||
@ -1,124 +1,62 @@ |
|||||||
|
|
||||||
# μpb - a small protobuf implementation in C |
# μpb: small, fast C protos |
||||||
|
|
||||||
|Platform|Build Status| |
|
||||||
|--------|------------| |
|
||||||
|macOS|[](https://fusion.corp.google.com/projectanalysis/summary/KOKORO/prod%3Aupb%2Fmacos%2Fcontinuous)| |
|
||||||
|ubuntu|[](https://fusion.corp.google.com/projectanalysis/summary/KOKORO/prod%3Aupb%2Fubuntu%2Fcontinuous)| |
|
||||||
|
|
||||||
μpb (often written 'upb') is a small protobuf implementation written in C. |
|
||||||
|
|
||||||
upb generates a C API for creating, parsing, and serializing messages |
|
||||||
as declared in `.proto` files. upb is heavily arena-based: all |
|
||||||
messages always live in an arena (note: the arena can live in stack or |
|
||||||
static memory if desired). Here is a simple example: |
|
||||||
|
|
||||||
```c |
|
||||||
#include "conformance/conformance.upb.h" |
|
||||||
|
|
||||||
void foo(const char* data, size_t size) { |
|
||||||
upb_arena *arena; |
|
||||||
|
|
||||||
/* Generated message type. */ |
|
||||||
conformance_ConformanceRequest *request; |
|
||||||
conformance_ConformanceResponse *response; |
|
||||||
|
|
||||||
arena = upb_arena_new(); |
|
||||||
request = conformance_ConformanceRequest_parse(data, size, arena); |
|
||||||
response = conformance_ConformanceResponse_new(arena); |
|
||||||
|
|
||||||
switch (conformance_ConformanceRequest_payload_case(request)) { |
|
||||||
case conformance_ConformanceRequest_payload_protobuf_payload: { |
|
||||||
upb_strview payload = conformance_ConformanceRequest_protobuf_payload(request); |
|
||||||
// ... |
|
||||||
break; |
|
||||||
} |
|
||||||
|
|
||||||
case conformance_ConformanceRequest_payload_NOT_SET: |
|
||||||
fprintf(stderr, "conformance_upb: Request didn't have payload.\n"); |
|
||||||
break; |
|
||||||
|
|
||||||
default: { |
|
||||||
static const char msg[] = "Unsupported input format."; |
|
||||||
conformance_ConformanceResponse_set_skipped( |
|
||||||
response, upb_strview_make(msg, sizeof(msg))); |
|
||||||
break; |
|
||||||
} |
|
||||||
} |
|
||||||
|
|
||||||
/* Frees all messages on the arena. */ |
|
||||||
upb_arena_free(arena); |
|
||||||
} |
|
||||||
``` |
|
||||||
|
|
||||||
API and ABI are both subject to change! Please do not distribute |
μpb (often written 'upb') is a small |
||||||
as a shared library for this reason (for now at least). |
[protobuf](https://github.com/protocolbuffers/protobuf) implementation written |
||||||
|
in C. |
||||||
|
|
||||||
## Using upb in your project |
upb is the core runtime for protobuf languages extensions in |
||||||
|
[Ruby](https://github.com/protocolbuffers/protobuf/tree/master/ruby), |
||||||
|
[PHP](https://github.com/protocolbuffers/protobuf/tree/master/php), and (soon) |
||||||
|
Python. |
||||||
|
|
||||||
Currently only Bazel is supported (CMake support is partial and incomplete |
While upb offers a C API, the C API & ABI **are not stable**. For this reason, |
||||||
but full CMake support is an eventual goal). |
upb is not generally offered as a C library for direct consumption, and there |
||||||
|
are no releases. |
||||||
|
|
||||||
To use upb in your Bazel project, first add upb to your `WORKSPACE` file, |
## Features |
||||||
either as a `git_repository()` or as a `new_local_repository()` with a |
|
||||||
Git Submodule. (For an example, see `examples/bazel/ in this repo). |
|
||||||
|
|
||||||
```python |
upb has comparable speed to protobuf C++, but is an order of magnitude smaller |
||||||
# Add this to your WORKSPACE file. |
in code size. |
||||||
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository") |
|
||||||
|
|
||||||
git_repository( |
Like the main protobuf implementation in C++, it supports: |
||||||
name = "upb", |
|
||||||
remote = "https://github.com/protocolbuffers/upb.git", |
|
||||||
commit = "d16bf99ac4658793748cda3251226059892b3b7b", |
|
||||||
) |
|
||||||
|
|
||||||
load("@upb//bazel:workspace_deps.bzl", "upb_deps") |
- a generated API (in C) |
||||||
|
- reflection |
||||||
|
- binary & JSON wire formats |
||||||
|
- text format serialization |
||||||
|
- all standard features of protobufs (oneofs, maps, unknown fields, etc.) |
||||||
|
- full conformance with the protobuf conformance tests |
||||||
|
|
||||||
upb_deps() |
upb also supports some features that C++ does not: |
||||||
``` |
|
||||||
|
|
||||||
Then in your BUILD file you can add `upb_proto_library()` rules that |
- **optional reflection:** generated messages are agnostic to whether |
||||||
generate code for a corresponding `proto_library()` rule. For |
reflection will be linked in or not. |
||||||
example: |
- **no global state:** no pre-main registration or other global state. |
||||||
|
- **fast reflection-based parsing:** messages loaded at runtime parse |
||||||
```python |
just as fast as compiled-in messages. |
||||||
# Add this to your BUILD file. |
|
||||||
load("@upb//bazel:upb_proto_library.bzl", "upb_proto_library") |
However there are some features it does not support: |
||||||
|
|
||||||
proto_library( |
|
||||||
name = "foo_proto", |
|
||||||
srcs = ["foo.proto"], |
|
||||||
) |
|
||||||
|
|
||||||
upb_proto_library( |
|
||||||
name = "foo_upbproto", |
|
||||||
deps = [":foo_proto"], |
|
||||||
) |
|
||||||
|
|
||||||
cc_binary( |
|
||||||
name = "test_binary", |
|
||||||
srcs = ["test_binary.c"], |
|
||||||
deps = [":foo_upbproto"], |
|
||||||
) |
|
||||||
``` |
|
||||||
|
|
||||||
Then in your `.c` file you can #include the generated header: |
- proto2 extensions (coming soon!) |
||||||
|
- text format parsing |
||||||
|
- deep descriptor verification: upb's descriptor validation is not as exhaustive |
||||||
|
as `protoc`. |
||||||
|
|
||||||
```c |
## Install |
||||||
#include "foo.upb.h" |
|
||||||
|
|
||||||
/* Insert code that uses generated types. */ |
For Ruby, use [RubyGems](https://rubygems.org/gems/google-protobuf): |
||||||
|
|
||||||
|
``` |
||||||
|
$ gem install google-protobuf |
||||||
``` |
``` |
||||||
|
|
||||||
## Lua bindings |
For PHP, use [PECL](https://pecl.php.net/package/protobuf): |
||||||
|
|
||||||
This repo has some Lua bindings for the core library. These are |
``` |
||||||
experimental and very incomplete. These are currently included in |
$ sudo pecl install protobuf |
||||||
order to validate that the C API is suitable for wrapping. As the |
``` |
||||||
project matures these Lua bindings may become publicly available. |
|
||||||
|
|
||||||
## Contact |
## Contributing |
||||||
|
|
||||||
Author: Josh Haberman ([jhaberman@gmail.com](mailto:jhaberman@gmail.com), |
Please see [CONTRIBUTING.md](CONTRIBUTING.md). |
||||||
[haberman@google.com](mailto:haberman@google.com)) |
|
||||||
|
|||||||
@ -1,46 +0,0 @@ |
|||||||
# Copyright (c) 2009-2021, Google LLC |
|
||||||
# All rights reserved. |
|
||||||
# |
|
||||||
# Redistribution and use in source and binary forms, with or without |
|
||||||
# modification, are permitted provided that the following conditions are met: |
|
||||||
# * Redistributions of source code must retain the above copyright |
|
||||||
# notice, this list of conditions and the following disclaimer. |
|
||||||
# * Redistributions in binary form must reproduce the above copyright |
|
||||||
# notice, this list of conditions and the following disclaimer in the |
|
||||||
# documentation and/or other materials provided with the distribution. |
|
||||||
# * Neither the name of Google LLC nor the |
|
||||||
# names of its contributors may be used to endorse or promote products |
|
||||||
# derived from this software without specific prior written permission. |
|
||||||
# |
|
||||||
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND |
|
||||||
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED |
|
||||||
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE |
|
||||||
# DISCLAIMED. IN NO EVENT SHALL Google LLC BE LIABLE FOR ANY |
|
||||||
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES |
|
||||||
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; |
|
||||||
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND |
|
||||||
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
|
||||||
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
|
||||||
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
|
||||||
|
|
||||||
load("@rules_proto//proto:defs.bzl", "proto_library") |
|
||||||
load("@upb//bazel:upb_proto_library.bzl", "upb_proto_library") |
|
||||||
|
|
||||||
licenses(["notice"]) |
|
||||||
|
|
||||||
proto_library( |
|
||||||
name = "foo_proto", |
|
||||||
srcs = ["foo.proto"], |
|
||||||
) |
|
||||||
|
|
||||||
upb_proto_library( |
|
||||||
name = "foo_upbproto", |
|
||||||
deps = [":foo_proto"], |
|
||||||
) |
|
||||||
|
|
||||||
cc_binary( |
|
||||||
name = "test_binary", |
|
||||||
srcs = ["test_binary.c"], |
|
||||||
copts = ["-std=c99"], |
|
||||||
deps = [":foo_upbproto"], |
|
||||||
) |
|
||||||
@ -1,14 +0,0 @@ |
|||||||
|
|
||||||
workspace(name = "upb_example") |
|
||||||
|
|
||||||
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository") |
|
||||||
|
|
||||||
git_repository( |
|
||||||
name = "upb", |
|
||||||
remote = "https://github.com/protocolbuffers/upb.git", |
|
||||||
commit = "d16bf99ac4658793748cda3251226059892b3b7b", |
|
||||||
) |
|
||||||
|
|
||||||
load("@upb//bazel:workspace_deps.bzl", "upb_deps") |
|
||||||
|
|
||||||
upb_deps() |
|
||||||
@ -1,7 +0,0 @@ |
|||||||
|
|
||||||
syntax = "proto2"; |
|
||||||
|
|
||||||
message Foo { |
|
||||||
optional int64 time = 1; |
|
||||||
optional string greeting = 2; |
|
||||||
} |
|
||||||
@ -1,43 +0,0 @@ |
|||||||
/*
|
|
||||||
* Copyright (c) 2009-2021, Google LLC |
|
||||||
* All rights reserved. |
|
||||||
* |
|
||||||
* Redistribution and use in source and binary forms, with or without |
|
||||||
* modification, are permitted provided that the following conditions are met: |
|
||||||
* * Redistributions of source code must retain the above copyright |
|
||||||
* notice, this list of conditions and the following disclaimer. |
|
||||||
* * Redistributions in binary form must reproduce the above copyright |
|
||||||
* notice, this list of conditions and the following disclaimer in the |
|
||||||
* documentation and/or other materials provided with the distribution. |
|
||||||
* * Neither the name of Google LLC nor the |
|
||||||
* names of its contributors may be used to endorse or promote products |
|
||||||
* derived from this software without specific prior written permission. |
|
||||||
* |
|
||||||
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND |
|
||||||
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED |
|
||||||
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE |
|
||||||
* DISCLAIMED. IN NO EVENT SHALL Google LLC BE LIABLE FOR ANY |
|
||||||
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES |
|
||||||
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; |
|
||||||
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND |
|
||||||
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
|
||||||
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
|
||||||
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
|
||||||
*/ |
|
||||||
|
|
||||||
#include <time.h> |
|
||||||
|
|
||||||
#include "examples/bazel/foo.upb.h" |
|
||||||
|
|
||||||
int main() { |
|
||||||
upb_arena *arena = upb_arena_new(); |
|
||||||
Foo* foo = Foo_new(arena); |
|
||||||
const char greeting[] = "Hello, World!\n"; |
|
||||||
|
|
||||||
Foo_set_time(foo, time(NULL)); |
|
||||||
/* Warning: the proto will not copy this, the string data must outlive
|
|
||||||
* the proto. */ |
|
||||||
Foo_set_greeting(foo, upb_strview_makez(greeting)); |
|
||||||
|
|
||||||
upb_arena_free(arena); |
|
||||||
} |
|
||||||
Loading…
Reference in new issue