protobuf/docs/design/editions/protobuf-editions-design-features.md

# Protobuf Editions Design: Features

**Author:** [@haberman](https://github.com/haberman),
[@fowles](https://github.com/fowles)

**Approved:** 2022-10-13

A proposal to use custom options as our way of defining and representing
features.

## Background

The [Protobuf Editions](what-are-protobuf-editions.md) project uses "editions"
to allow Protobuf to safely evolve over time. An edition is formally a set of
"features" with a default value per feature. The set of features or a default
value for a feature can only change with the introduction of a new edition.
Features define the specific points of change and evolution on a per entity
basis within a .proto file (entities being files, messages, fields, or any other
lexical element in the file). The design in this doc supplants an earlier design
which used strings for feature definition.

Protobuf already supports
[custom options](https://protobuf.dev/programming-guides/proto2#customoptions)
and we will leverage these to provide a rich syntax without introducing new
syntactic forms into Protobuf.

## Sample Usage

Here is a small sample usage of features to give a flavor for how it looks

```
edition = "2023";

package experimental.users.kfm.editions;

import "net/proto2/proto/features_cpp.proto";

option features.repeated_field_encoding = EXPANDED;
option features.enum = OPEN;
option features.(pb.cpp).string_field_type = STRING;
option features.(pb.cpp).namespace = "kfm::proto_experiments";

message Lab {
  // `Mouse` is open as it inherits the file's value.
  enum Mouse {
    UNKNOWN_MOUSE = 0;
    PINKY = 1;
    THE_BRAIN = 2;
  }
  repeated Mouse mice = 1 [features.repeated_field_encoding = PACKED];

  string name = 2;
  string address = 3 [features.(pb.cpp).string_field_type = CORD];
  string function = 4 [features.(pb.cpp).string_field_type = STRING_VIEW];
}

enum ColorChannel {
  // Turn off the option from the surrounding file
  option features.enum = CLOSED;

  UNKNOWN_COLOR_CHANNEL = 0;
  RED = 1;
  BLUE = 2;
  GREEN = 3;
  ALPHA = 4;
}
```

## Language-Specific Features

We will use extensions to manage features specific to individual code
generators.

```
// In net/proto2/proto/descriptor.proto:
syntax = "proto2";
package proto2;

message Features {
  ...
  extensions 1000;  // for features_cpp.proto
  extensions 1001;  // for features_java.proto
}

```

This will allow third-party code generators to use editions for their own
evolution as long as they reserve a single extension number in
`descriptor.proto`. Using this from a .proto file would look like this:

```
edition = "2023";

import "third_party/protobuf/compiler/cpp/features_cpp.proto"

message Bar {
  optional string str = 1 [features.(pb.cpp).string_field_type = true];
}
```

## Inheritance

To support inheritance, we will specify a single `Features` message that extends
every kind of option:

```
// In net/proto2/proto/descriptor.proto:
syntax = "proto2";
package proto2;

message Features {
  ...
}

message FileOptions {
  optional Features features = ..;
}

message MessageOptions {
  optional Features features = ..;
}
// All the other `*Options` protos.
```

At the implementation level, feature inheritance is exactly the behavior of
`MergeFrom`

```
void InheritFrom(const Features& parent, Features* child) {
  Features tmp(parent);
  tmp.MergeFrom(child);
  child->Swap(&tmp);
}
```

which means that custom backends will be able to faithfully implement
inheritance without difficulty.

## Target Attributes

While inheritance can be useful for minimizing changes or pushing defaults
broadly, it can be overused in ways that would make simple refactoring of
`.proto` files harder. Additionally, not all features are meaningful on all
entities (for example `features.enum = OPEN` is meaningless on a field).

To avoid these issues, we will introduce "target" attributes on features
(similar in concept to the "target" attribute on Java annotations).

```
enum FeatureTargetType {
  FILE = 0;
  MESSAGE = 1;
  ENUM = 2;
  FIELD = 3;
  ...
};
```

These will restrict the set of entities to which a feature may be attached.

```
message Features {
  ...

  enum EnumType {
    OPEN = 0;
    CLOSED = 1;
  }
  optional EnumType enum = 2 [
      target = ENUM
  ];
}
```

## Retention

To reduce the size of descriptors in protobuf runtimes, features will be
permitted to specify retention rules (again similar in concept to "retention"
attributes on Java annotations).

```
enum FeatureRetention {
  SOURCE = 0;
  RUNTIME = 1;
}
```

## Specification of an Edition

An edition is, effectively, an instance of the `Feature` proto which forms the
base for performing inheritance using `MergeFrom`. This allows `protoc` and
specific language generators to leverage existing formats (like text-format) for
specifying the value of features at a given edition.

Although naively we would think that field defaults are the right approach, this
does not quite work, because the default is editions-dependent. Instead, we
propose adding the following to the protoc-provided `features.proto`:

```
message Features {
  // ...
  message EditionDefault {
    optional string edition = 1;
    optional string default = 2;  // Textproto value.
  }

  extend FieldOptions {
    // Ideally this is a map, but map extensions are not permitted...
    repeated EditionDefault edition_defaults = 9001;
  }
}
```

To build the edition defaults for a particular edition `current` in the context
of a particular file `foo.proto`, we execute the following algorithm:

1.  Construct a new `Features feats;`.
2.  For each field in `Features`, take the value of the
    `Features.edition_defaults` option (call it `defaults`), and sort it by the
    value of `edition` (per the total order for edition names,
    [Life of an Edition](life-of-an-edition.md)).
3.  Binsearch for the latest edition in `defaults` that is earlier or equal to
    `current`.
    1.  If the field is of singular, scalar type, use that value as the value of
        the field in `feats`.
    2.  Otherwise, the value of the field in `feats` is given by merging all of
        the values less than `current`, starting from the oldest edition.
4.  For the purposes of this algorithm, `Features`'s fields all behave as if
    they were `required`; failure to find a default explicitly via the editions
    default search mechanism should result in a compilation error, because it
    means the file's edition is too old.
5.  For each extension of `Features` that is visible from `foo.proto` via
    imports, perform the same algorithm as above to construct the editions
    default for that extension message, and add it to `feat`.

This algorithm has the following properties:

*   Language-scoped features are discovered via imports, which is how they need
    to be imported for use in a file in the first place.
*   Every value is set explicitly, so we correctly reject too-old files.
*   Files from "the future" will not be rejected out of hand by the algorithm,
    allowing us to provide a flag like `--allow-experimental-editions` for ease
    of allowing backends to implement a new edition.

## Edition Zero Features

Putting the parts together, we can offer a potential `Feature` message for
edition zero: [Edition Zero Features](edition-zero-features.md).

```
message Features {
  enum FieldPresence {
    EXPLICIT = 0;
    IMPLICIT = 1;
    LEGACY_REQUIRED = 2;
  }
  optional FieldPresence field_presence = 1 [
      retention = RUNTIME,
      target = FIELD,
      (edition_defaults) = {
        edition: "2023", default: "EXPLICIT"
      }
  ];

  enum EnumType {
    OPEN = 0;
    CLOSED = 1;
  }
  optional EnumType enum = 2 [
      retention = RUNTIME,
      target = ENUM,
      (edition_defaults) = {
        edition: "2023", default: "OPEN"
      }
  ];

  enum RepeatedFieldEncoding {
    PACKED = 0;
    EXPANDED = 1;
  }
  optional RepeatedFieldEncoding repeated_field_encoding = 3 [
      retention = RUNTIME,
      target = FIELD,
      (edition_defaults) = {
        edition: "2023", default: "PACKED"
      }
  ];

  enum StringFieldValidation {
    REQUIRED = 0;
    HINT = 1;
    SKIP = 2;
  }
  optional StringFieldValidation string_field_validation = 4 [
      retention = RUNTIME,
      target = FIELD,
      (edition_defaults) = {
        edition: "2023", default: "REQUIRED"
      }
  ];

  enum MessageEncoding {
    LENGTH_PREFIXED = 0;
    DELIMITED = 1;
  }
  optional MessageEncoding message_encoding = 5 [
      retention = RUNTIME,
      target = FIELD,
      (edition_defaults) = {
        edition: "2023", default: "LENGTH_PREFIXED"
      }
  ];

  extensions 1000;  // for features_cpp.proto
  extensions 1001;  // for features_java.proto
}
```