protobuf/docs/design/prototiller/prototiller-reqs-for-edition-zero.md

Prototiller Requirements for Edition Zero

Authors: @mkruskal-google

Approved: 2023-07-07

Background

Edition Zero Features lays out the design for our first edition, which will unify proto2 and proto3 via features. In order to migrate internal Google repositories (and aid OSS migrations), we will be leveraging Prototiller to upgrade from legacy syntax to the new editions model. We will also be using Prototiller for every edition bump, but that's out of scope for this document (more general requirements are laid out in Prototiller Requirements for Editions).

Overview

The way the edition zero features were derived, there will always exist a no-op transformation from proto2/proto3. However, the details of this transformation can be fairly complex and depend on a lot of different factors.

A temporary script has been created as a placeholder, which manages to get fairly good coverage of these rules. Notably, it can't handle groups inside extensions or oneofs. This, along with its golden tests, can serve as a useful benchmark for Prototiller.

Feature Optimization

One important piece of Prototiller that will ease the friction of the Edition Zero large-scale change is a feature optimization phase. If certain features aren't necessary to make the upgrade a no-op, we shouldn't add them and should instead rely on the edition defaults for future changes. Similarly, we should try to minimize the total size of the change by collapsing a feature specification to a higher level (e.g. file-level defaults of a field feature).

Frontend Feature Transformations

This section details all of our frontend features and how to transform from proto2/proto3 to edition zero.

Field Presence

The field_presence feature defaults to EXPLICIT, which matches proto2/proto3 optional behavior. The LEGACY_REQUIRED value corresponds to proto2 required fields, and IMPLICIT corresponds to non-optional proto3 fields. In order to minimize changes, file-level defaults should be utilized.

Example transformations:

syntax = "proto2"; message Foo { optional string bar = 1; required string baz = 2; }

edition = "2023"; message Foo { string bar = 1; string baz = 2 [ features.field_presence = LEGACY_REQUIRED]; }

syntax = "proto3"; message Foo { optional string bar = 1; string baz = 2; string bam = 3; }

edition = "2023"; features.field_presence = IMPLICIT; message Foo { string bar = 1 [features.field_presence = EXPLICIT]; string baz = 2; string bam = 3; }

syntax = "proto3"; message Foo { optional string bar = 1; optional string baz = 2; }

edition = "2023"; message Foo { string bar = 1; string baz = 2; }

Enum Type

The enum_type feature defaults to OPEN, which matches proto3 behavior. The CLOSED value corresponds to typical proto2 behavior. In order to minimize changes, file-level defaults should be utilized.

Example transformations:

syntax = "proto2"; enum Foo { VALUE1 = 0; VALUE2 = 1; }

edition = "2023"; features.enum_type = CLOSED; enum Foo { VALUE1 = 0; VALUE2 = 1; }

syntax = "proto3"; enum Foo { VALUE1 = 0; VALUE2 = 1; }

edition = "2023"; enum Foo { VALUE1 = 0; VALUE2 = 1; }

Repeated Field Encoding

The repeated_field_encoding feature defaults to PACKED, which matches proto3 behavior. The EXPANDED value corresponds to the default proto2 behavior. Both proto2 and proto3 can have the default behavior overridden by using the packed field option. All of these should be replaced in the migration to edition zero. Minimization of changes will be a little more complicated here, since there could exist files where the majority of repeated fields have been overridden.

Example transformations:

syntax = "proto2"; message Foo { repeated int32 bar = 1; repeated int32 baz = 2 [packed = true]; repeated int32 bam = 3; }

edition = "2023"; features.repeated_field_encoding = EXPANDED; message Foo { repeated int32 bar = 1; repeated int32 baz = 2 [ features.repeated_field_encoding = PACKED]; repeated int32 bar = 3; }

syntax = "proto3"; message Foo { repeated int32 bar = 1; repeated int32 baz = 2 [packed = false]; }

edition = "2023"; message Foo { repeated int32 bar = 2; repeated int32 baz = 2 [ features.repeated_field_encoding = EXPANDED]; }

syntax = "proto2"; message Foo { repeated int32 x = 1 [packed = true]; // Strings are never packed. repeated string z = 1; repeated string w = 2; }

edition = "2023"; message Foo { repeated int32 x = 1; repeated string z = 1; repeated string w = 2; }

Message Encoding

The message_encoding feature is designed to replace the proto2-only group syntax (with value DELIMITED), with a default that will always be LENGTH_PREFIXED. This is a somewhat awkward transformation in the general case, since we allow group definitions anywhere fields exist even if message definitions can't. The basic transformation is to create a new message type in the nearest enclosing scope with the same name as the field, and lowercase the field and give it that type.

Example transformations:

syntax = "proto2"; message Foo { optional group Bar = 1 { optional int32 x = 1; } optional Bar baz = 2; }

edition = "2023"; message Foo { message Bar { int32 x = 1; } Bar bar = 1 [features.message_encoding = DELIMITED]; Bar baz = 2; }

syntax = "proto2"; message Foo { oneof foo { group Bar = 1 { optional int32 x = 1; } } }

edition = "2023"; message Foo { message Bar { int32 x = 1; } oneof foo { Bar bar = 1 [ features.message_encoding = DELIMITED]; } }

JSON Format

The json_format feature is a bit of an outlier, because (at least for edition zero) it only affects the frontend build of the proto file. The ALLOW value (proto3 behavior) enables all JSON mapping conflict checks on field names, unless deprecated_legacy_json_field_conflicts is set. The LEGACY_BEST_EFFORT value (proto2 behavior) disables these checks. The ideal minimal transformation would be to switch to ALLOW in all cases except where deprecated_legacy_json_field_conflicts is set or there exist JSON mapping conflicts. In those cases we can fallback to LEGACY_BEST_EFFORT.

Alternatively, if it's difficult for Prototiller to handle, we could do a followup large-scale change to remove all LEGACY_BEST_EFFORT instances that pass build.

Example transformations:

syntax = "proto2"; message Foo { optional string bar = 1; optional string baz = 2; }

edition = "2023"; message Foo { string bar = 1; string baz = 2; }

syntax = "proto3"; message Foo { string bar = 1; string baz = 2; }

edition = "2023"; features.field_presence = IMPLICIT; message Foo { string bar = 1; string baz = 2; }

syntax = "proto2"; message Foo { // Warning only string bar = 1; string bar_ = 2; }

edition = "2023"; features.json_format = LEGACY_BEST_EFFORT; message Foo { string bar = 1; string bar_ = 2; }

syntax = "proto3"; message Foo { option deprecated_legacy_json_field_conflicts = true; string bar = 1; string baz = 2 [json_name = "bar"]; }

edition = "2023"; features.field_presence = IMPLICIT; features.json_format = LEGACY_BEST_EFFORT; message Foo { string bar = 1; string baz = 2; }

Backend Feature Transformations

This section details our backend-specific features and how to transform from proto2/proto3 to edition zero.

In order to limit bloat, it would be ideal if we could check whether or not a proto file is ever used to generate code in the target language. If it's not, there's no reason to add backend-specific features.

Legacy Closed Enum

Java and C++ by default treat proto3 enums as closed if they're used in a proto2 message. The internal cc_open_enum field option can override this, but it has very limited use and may not be worth considering. While the enum type behavior should still be determined by Enum Type, we'll need to add this feature to proto2 files using proto3 messages (the reverse is disallowed).

Example transformations:

syntax = "proto2"; import "some_proto3_file.proto" enum Proto2Enum { BAR = 0; } message Foo { optional Proto3Enum bar = 1; optional Proto2Enum baz = 2; }

edition = "2023"; import "third_party/protobuf/cpp_features.proto" import "third_party/protobuf/java_features.proto" import "some_proto3_file.proto" features.enum_type = CLOSED; message Foo { Proto3Enum bar = 1 [ features.(pb.cpp).legacy_closed_enum = true, features.(pb.java).legacy_closed_enum = true]; Proto2Enum baz = 2; }

UTF8 Validation

This feature is pending approval of Editions Zero Feature: utf8_validation (not available externally).

Other Transformations

In addition to features, there are some other changes we've made for edition zero.

Reserved Identifier Syntax

With Protobuf Change Proposal: Reserved Identifiers (not available externally), we've decided to switch from strings to identifiers for reserved fields. This should be a trivial change, but if the proto file contains strings that aren't valid identifiers there's some ambiguity. They're currently ignored today, but they could be typos we wouldn't want to just blindly delete. So instead, we'll leave behind a comment.

Example transformations:

syntax = "proto2"; message Foo { reserved "bar", "baz"; }

edition = "2023"; message Foo { reserved bar, baz; }

syntax = "proto2"; message Foo { reserved "bar", "1"; }

edition = "2023"; message Foo { reserved bar; /reserved "1";/ }