protobuf/docs/implementing_proto3_presence.md

How To Implement Field Presence for Proto3

Protobuf release 3.12 adds experimental support for optional fields in proto3. Proto3 optional fields track presence like in proto2. For background information about what presence tracking means, please see docs/field_presence.

Document Summary

This document is targeted at developers who own or maintain protobuf code generators. All code generators will need to be updated to support proto3 optional fields. First-party code generators developed by Google are being updated already. However third-party code generators will need to be updated independently by their authors. This includes:

• implementations of Protocol Buffers for other languages.
• alternate implementations of Protocol Buffers that target specialized use cases.
• RPC code generators that create generated APIs for service calls.
• code generators that implement some utility code on top of protobuf generated classes.

While this document speaks in terms of "code generators", these same principles apply to implementations that dynamically generate a protocol buffer API "on the fly", directly from a descriptor, in languages that support this kind of usage.

Background

Presence tracking was added to proto3 in response to user feedback, both from inside Google and from open-source users. The proto3 wrapper types were previously the only supported presence mechanism for proto3. Users have pointed to both efficiency and usability issues with the wrapper types.

Presence in proto3 uses exactly the same syntax and semantics as in proto2. Proto3 Fields marked optional will track presence like proto2, while fields without any label (known as "singular fields"), will continue to omit presence information. The optional keyword was chosen to minimize differences with proto2.

Unfortunately, for the current descriptor protos and Descriptor API (as of 3.11.4) it is not possible to use the same representation as proto2. Proto3 descriptors already use LABEL_OPTIONAL for proto3 singular fields, which do not track presence. There is a lot of existing code that reflects over proto3 protos and assumes that LABEL_OPTIONAL in proto3 means "no presence." Changing the semantics now would be risky, since old software would likely drop proto3 presence information, which would be a data loss bug.

To minimize this risk we chose a descriptor representation that is semantically compatible with existing proto3 reflection. Every proto3 optional field is placed into a one-field oneof. We call this a "synthetic" oneof, as it was not present in the source .proto file.

Since oneof fields in proto3 already track presence, existing proto3 reflection-based algorithms should correctly preserve presence for proto3 optional fields with no code changes. For example, the JSON and TextFormat parsers/serializers in C++ and Java did not require any changes to support proto3 presence. This is the major benefit of synthetic oneofs.

This design does leave some cruft in descriptors. Synthetic oneofs are a compatibility measure that we can hopefully clean up in the future. For now though, it is important to preserve them across different descriptor formats and APIs. It is never safe to drop synthetic oneofs from a proto schema. Code generators can (and should) skip synthetic oneofs when generating a user-facing API or user-facing documentation. But for any schema representation that is consumed programmatically, it is important to keep the synthetic oneofs around.

In APIs it can be helpful to offer separate accessors that refer to "real" oneofs (see API Changes below). This is a convenient way to omit synthetic oneofs in code generators.

Updating a Code Generator

When a user adds an optional field to proto3, this is internally rewritten as a one-field oneof, for backward-compatibility with reflection-based algorithms:

syntax = "proto3";

message Foo {
  // Experimental feature, not generally supported yet!
  optional int32 foo = 1;

  // Internally rewritten to:
  // oneof _foo {
  //   int32 foo = 1 [proto3_optional=true];
  // }
  //
  // We call _foo a "synthetic" oneof, since it was not created by the user.
}

As a result, the main two goals when updating a code generator are:

1. Give optional fields like foo normal field presence, as described in docs/field_presence If your implementation already supports proto2, a proto3 optional field should use exactly the same API and internal implementation as proto2 optional.
2. Avoid generating any oneof-based accessors for the synthetic oneof. Its only purpose is to make reflection-based algorithms work properly if they are not aware of proto3 presence. The synthetic oneof should not appear anywhere in the generated API.

Satisfying the Experimental Check

If you try to run protoc on a file with proto3 optional fields, you will get an error because the feature is still experimental:

$ cat test.proto
syntax = "proto3";

message Foo {
  // Experimental feature, not generally supported yet!
  optional int32 a = 1;
}
$ protoc --cpp_out=. test.proto
test.proto: This file contains proto3 optional fields, but --experimental_allow_proto3_optional was not set.

There are two options for getting around this error:

1. Pass --experimental_allow_proto3_optional to protoc.
2. Make your filename (or a directory name) contain the string test_proto3_optional. This indicates that the proto file is specifically for testing proto3 optional support, so the check is suppressed.

These options are demonstrated below:

# One option:
$ protoc test.proto --cpp_out=. --experimental_allow_proto3_optional

# Another option:
$ cp test.proto test_proto3_optional.proto
$ protoc test_proto3_optional.proto --cpp_out=.
$

The experimental check will be removed in a future release, once we are ready to make this feature generally available. Ideally this will happen for the 3.13 release of protobuf, sometime in mid-2020, but there is not a specific date set for this yet. Some of the timing will depend on feedback we get from the community, so if you have questions or concerns please get in touch via a GitHub issue.

Signaling That Your Code Generator Supports Proto3 Optional

If you now try to invoke your own code generator with the test proto, you will run into a different error:

$ protoc test_proto3_optional.proto --my_codegen_out=.
test_proto3_optional.proto: is a proto3 file that contains optional fields, but
code generator --my_codegen_out hasn't been updated to support optional fields in
proto3. Please ask the owner of this code generator to support proto3 optional.

This check exists to make sure that code generators get a chance to update before they are used with proto3 optional fields. Without this check an old code generator might emit obsolete generated APIs (like accessors for a synthetic oneof) and users could start depending on these. That would create a legacy migration burden once a code generator actually implements the feature.

To signal that your code generator supports optional fields in proto3, you need to tell protoc what features you support. The method for doing this depends on whether you are using the C++ google::protobuf::compiler::CodeGenerator framework or not.

If you are using the CodeGenerator framework:

class MyCodeGenerator : public google::protobuf::compiler::CodeGenerator {
  // Add this method.
  uint64_t GetSupportedFeatures() const override {
    // Indicate that this code generator supports proto3 optional fields.
    // (Note: don't release your code generator with this flag set until you
    // have actually added and tested your proto3 support!)
    return FEATURE_PROTO3_OPTIONAL;
  }
}

If you are generating code using raw CodeGeneratorRequest and CodeGeneratorResponse messages from plugin.proto, the change will be very similar:

void GenerateResponse() {
  CodeGeneratorResponse response;
  response.set_supported_features(CodeGeneratorResponse::FEATURE_PROTO3_OPTIONAL);

  // Generate code...
}

Once you have added this, you should now be able to successfully use your code generator to generate a file containing proto3 optional fields:

$ protoc test_proto3_optional.proto --my_codegen_out=.

Updating Your Code Generator

Now to actually add support for proto3 optional to your code generator. The goal is to recognize proto3 optional fields as optional, and suppress any output from synthetic oneofs.

If your code generator does not currently support proto2, you will need to design an API and implementation for supporting presence in scalar fields. Generally this means:

• allocating a bit inside the generated class to represent whether a given field is present or not.
• exposing a has_foo() method for each field to return the value of this bit.
• make the parser set this bit when a value is parsed from the wire.
• make the serializer test this bit to decide whether to serialize.

If your code generator already supports proto2, then most of your work is already done. All you need to do is make sure that proto3 optional fields have exactly the same API and behave in exactly the same way as proto2 optional fields.

From experience updating several of Google's code generators, most of the updates that are required fall into one of several patterns. Here we will show the patterns in terms of the C++ CodeGenerator framework. If you are using CodeGeneratorRequest and CodeGeneratorReply directly, you can translate the C++ examples to your own language, referencing the C++ implementation of these methods where required.

To test whether a field should have presence

Old:

bool MessageHasPresence(const google::protobuf::FieldDescriptor* field) {
  return field->has_presence();
}

New:

// Presence is no longer a property of a message, it's a property of individual
// fields.
bool FieldHasPresence(const google::protobuf::FieldDescriptor* field) {
  return field->has_presence();
  // Note, the above will return true for fields in a oneof.
  // If you want to filter out oneof fields, write this instead:
  //   return field->has_presence && !field->real_containing_oneof()
}

To test whether a field is a member of a oneof

Old:

bool FieldIsInOneof(const google::protobuf::FieldDescriptor* field) {
  return field->containing_oneof() != nullptr;
}

New:

bool FieldIsInOneof(const google::protobuf::FieldDescriptor* field) {
  // real_containing_oneof() returns nullptr for synthetic oneofs.
  return field->real_containing_oneof() != nullptr;
}

To iterate over all oneofs

Old:

bool IterateOverOneofs(const google::protobuf::Descriptor* message) {
  for (int i = 0; i < message->oneof_decl_count(); i++) {
    const google::protobuf::OneofDescriptor* oneof = message->oneof(i);
    // ...
  }
}

New:

bool IterateOverOneofs(const google::protobuf::Descriptor* message) {
  // Real oneofs are always first, and real_oneof_decl_count() will return the
  // total number of oneofs, excluding synthetic oneofs.
  for (int i = 0; i < message->real_oneof_decl_count(); i++) {
    const google::protobuf::OneofDescriptor* oneof = message->oneof(i);
    // ...
  }
}

Updating Reflection

If your implementation offers reflection, there are a few other changes to make:

API Changes

The API for reflecting over fields and oneofs should make the following changes. These match the changes implemented in C++ reflection.

1. Add a FieldDescriptor::has_presence() method returning bool (adjusted to your language's naming convention). This should return true for all fields that have explicit presence, as documented in docs/field_presence. In particular, this includes fields in a oneof, proto2 scalar fields, and proto3 optional fields. This accessor will allow users to query what fields have presence without thinking about the difference between proto2 and proto3.
2. As a corollary of (1), please do not expose an accessor for the FieldDescriptorProto.proto3_optional field. We want to avoid having users implement any proto2/proto3-specific logic. Users should use the has_presence() function instead.
3. You may also wish to add a FieldDescriptor::has_optional_keyword() method returning bool, which indicates whether the optional keyword is present. Message fields will always return true for has_presence(), so this method can allow a user to know whether the user wrote optional or not. It can occasionally be useful to have this information, even though it does not change the presence semantics of the field.
4. If your reflection API may be used for a code generator, you may wish to implement methods to help users tell the difference between real and synthetic oneofs. In particular:
   • OneofDescriptor::is_synthetic(): returns true if this is a synthetic oneof.
   • FieldDescriptor::real_containing_oneof(): like containing_oneof(), but returns nullptr if the oneof is synthetic.
   • Descriptor::real_oneof_decl_count(): like oneof_decl_count(), but returns the number of real oneofs only.

Implementation Changes

Proto3 optional fields and synthetic oneofs must work correctly when reflected on. Specifically:

1. Reflection for synthetic oneofs should work properly. Even though synthetic oneofs do not really exist in the message, you can still make reflection work as if they did. In particular, you can make a method like Reflection::HasOneof() or Reflection::GetOneofFieldDescriptor() look at the hasbit to determine if the oneof is present or not.
2. Reflection for proto3 optional fields should work properly. For example, a method like Reflection::HasField() should know to look for the hasbit for a proto3 optional field. It should not be fooled by the synthetic oneof into thinking that there is a case member for the oneof.

Once you have updated reflection to work properly with proto3 optional and synthetic oneofs, any code that uses your reflection interface should work properly with no changes. This is the benefit of using synthetic oneofs.

In particular, if you have a reflection-based implementation of protobuf text format or JSON, it should properly support proto3 optional fields without any changes to the code. The fields will look like they all belong to a one-field oneof, and existing proto3 reflection code should know how to test presence for fields in a oneof.

So the best way to test your reflection changes is to try round-tripping a message through text format, JSON, or some other reflection-based parser and serializer, if you have one.

Validating Descriptors

If your reflection implementation supports loading descriptors at runtime, you must verify that all synthetic oneofs are ordered after all "real" oneofs.

Here is the code that implements this validation step in C++, for inspiration:

  // Validation that runs for each message.
  // Synthetic oneofs must be last.
  int first_synthetic = -1;
  for (int i = 0; i < message->oneof_decl_count(); i++) {
    const OneofDescriptor* oneof = message->oneof_decl(i);
    if (oneof->is_synthetic()) {
      if (first_synthetic == -1) {
        first_synthetic = i;
      }
    } else {
      if (first_synthetic != -1) {
        AddError(message->full_name(), proto.oneof_decl(i),
                 DescriptorPool::ErrorCollector::OTHER,
                 "Synthetic oneofs must be after all other oneofs");
      }
    }
  }

  if (first_synthetic == -1) {
    message->real_oneof_decl_count_ = message->oneof_decl_count_;
  } else {
    message->real_oneof_decl_count_ = first_synthetic;
  }