diff --git a/docs/implementing_proto3_presence.md b/docs/implementing_proto3_presence.md index 08f9c51321..fae2eb589a 100644 --- a/docs/implementing_proto3_presence.md +++ b/docs/implementing_proto3_presence.md @@ -5,6 +5,8 @@ proto3. Proto3 optional fields track presence like in proto2. For background information about what presence tracking means, please see [docs/field_presence](field_presence.md). +## Document Summary + This document is targeted at developers who own or maintain protobuf code generators. All code generators will need to be updated to support proto3 optional fields. First-party code generators developed by Google are being @@ -14,6 +16,7 @@ independently by their authors. This includes: - implementations of Protocol Buffers for other languges. - alternate implementations of Protocol Buffers that target specialized use cases. +- RPC code generators that create generated APIs for service calls. - code generators that implement some utility code on top of protobuf generated classes. @@ -21,6 +24,53 @@ While this document speaks in terms of "code generators", these same principles apply to implementations that dynamically generate a protocol buffer API "on the fly", directly from a descriptor, in languages that support this kind of usage. +## Background + +Presence tracking was added to proto3 in response to user feedback, both from +inside Google and [from open-source +users](https://github.com/protocolbuffers/protobuf/issues/1606). The [proto3 +wrapper +types](https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto) +were previously the only supported presence mechanism for proto3. Users have +pointed to both efficiency and usability issues with the wrapper types. + +Presence in proto3 uses exactly the same syntax and semantics as in proto2. +Proto3 Fields marked `optional` will track presence like proto2, while fields +without any label (known as "singular fields"), will continue to omit presence +information. The `optional` keyword was chosen to minimize differences with +proto2. + +Unfortunately, for the current descriptor protos and `Descriptor` API (as of +3.11.4) it is not possible to use the same representation as proto2. Proto3 +descriptors already use `LABEL_OPTIONAL` for proto3 singular fields, which do +not track presence. There is a lot of existing code that reflects over proto3 +protos and assumes that `LABEL_OPTIONAL` in proto3 means "no presence." Changing +the semantics now would be risky, since old software would likely drop proto3 +presence information, which would be a data loss bug. + +To minimize this risk we chose a descriptor representation that is semantically +compatible with existing proto3 reflection. Every proto3 optional field is +placed into a one-field `oneof`. We call this a "synthetic" oneof, as it was not +present in the source `.proto` file. + +Since oneof fields in proto3 already track presence, existing proto3 +reflection-based algorithms should correctly preserve presence for proto3 +optional fields with no code changes. For example, the JSON and TextFormat +parsers/serializers in C++ and Java did not require any changes to support +proto3 presence. This is the major benefit of synthetic oneofs. + +This design does leave some cruft in descriptors. Synthetic oneofs are a +compatibility measure that we can hopefully clean up in the future. For now +though, it is important to preserve them across different descriptor formats and +APIs. It is never safe to drop synthetic oneofs from a proto schema. Code +generators can (and should) skip synthetic oneofs when generating a user-facing +API or user-facing documentation. But for any schema representation that is +consumed programmatically, it is important to keep the synthetic oneofs around. + +In APIs it can be helpful to offer separate accessors that refer to "real" +oneofs (see [API Changes](#api-changes) below). This is a convenient way to omit +synthetic oneofs in code generators. + ## Updating a Code Generator When a user adds an `optional` field to proto3, this is internally rewritten as