protobuf/docs/design/prototiller/editions-tooling.md

Editions Tooling

Authors: @mcy

Approved: 2022-08-09

Overview

Protobuf Editions aims to introduce new semantics for Protobuf, but with a major emphasis on mechanical, incremental upgradability, to avoid the two systems problem of proto2 and proto3. The first edition (likely "2023") will introduce converged semantics for Protobuf that permit everything that proto2 and proto3 permitted: any non-editions file can become an editions file with minimal human intervention.

We plan to achieve this with a strong tooling story. These tools are intended to fully automate major steps in editions-related upgrade operations, for both large-scale changes and open source software strategic reasons. In particular:

• Non-automated large-scale change work in the editions space can be constrained to fixing uses of generated code and flipping features on specific fields (or other declarations).
• We can give our external users the most painless migration possible, which consists of "run this tool and commit the results".

This document describes the detailed design of the tools we need. This document presupposes Protochangifier Backend Design Doc (not available externally) integrated into protoc as a prerequisite, so we can ship the tooling as part of protoc. Because the tooling must know the full definition of an edition to work (see below), it seems to more-or-less place a hard requirement of being linked to protoc.

There are three tools we will build.

1. The "features janitor". This is a mode of protoc which consumes a .proto file and produces a ProtoChangeSpec that describes how to add and remove features such that the resulting janitor'ed file has fewer explicit features, but is not semantically different.
2. The "editions adopter". This is another mode of protoc, which produces a ProtoChangeSpec that describes how to bring a proto2 or proto3 file into editions mode, starting at a specific edition.
3. The "editions upgrader". This is a generalization of the adopter, which takes an editions file and produces a ProtoChangeSpec that brings it into a newer edition.

These tools will fundamentally speak ProtoChangeSpec, but we should also provide in-place versions, since those will likely be more useful to OSS users that just want to run the tool atomically on their entire project.

The Janitor

The features janitor is intended to be used as part of migrations to periodically clean up any messes made by flipping lots of features. Conceptually, it turns this proto file

edition = "2023";
message Foo {
  optional string a = 1 [features.(pb.cpp).string_type = VIEW];
  optional string b = 2 [features.(pb.cpp).string_type = VIEW];
  optional string c = 3 [features.(pb.cpp).string_type = VIEW];
  optional string d = 4 [features.(pb.cpp).string_type = VIEW];
  optional string e = 5 [features.(pb.cpp).string_type = VIEW];
}
message Bar {
  optional string a = 1 [features.(pb.cpp).string_type = VIEW];
  optional string b = 2;
  optional string c = 3;
  optional string d = 4;
  optional string e = 5;
}

into this one:

edition = "2023";
message Foo {
  option features.(pb.cpp).string_type = VIEW;
  optional string a = 1;
  optional string b = 2;
  optional string c = 3;
  optional string d = 4;
  optional string e = 5;
}
message Bar {
  optional string a = 1 [features.(pb.cpp).string_type = VIEW];
  optional string b = 2;
  optional string c = 3;
  optional string d = 4;
  optional string e = 5;
}

Specifically, the janitor tries to minimize the number of explicit features on the Protobuf schema. Actually doing this minimally feels like it's nonlinear, so we should invent a heuristic. A sketch of what this could look like:

1. Each feature that can appear explicitly on an AST node is either critical for that node or only for grouping. For example, string_type is critical for fields but not for messages.
2. Propagate features explicitly to every node, including edition defaults.
3. For each feature f, for each node n that f is non-critical for that contains (recursively) nodes that it is critical for (in DFS order):
   1. Set f for n to the value for f that the plurality of its direct children have, and remove the explicit f from those. If tied, choose the edition default if it is among the plurals, or else choose randomly.
4. Once repeated up to the root, delete all explicit features that are reachable from the root without crossing another explicit feature that isn't the edition default. I.e., those features which are implied by the edition defaults.

It is easy to construct cases where this is not optimal, but that is not important. This merely exists to make files prettier while keeping them equivalent. It is easy to see that, by construction, this algorithm satisfies the "semantic no-op" requirement.

The Adopter and the Updater

The adopter is merely a special case of the updater where proto2 and proto3 are viewed as editions (in the sense that an edition is a set of defaults), so we will only describe the updater.

To update one edition ("old") to another ("new", although not necessarily a newer edition):

1. Features that are not already explicitly set at the top level are set to the default given by "old"; they are only set on the outermost scope that does not have an explicit feature. For example, for file-level features, this means making all features explicit at the file level. For message-level features that are not file-level, this means placing an explicit feature on all top-level messages. This is a no-op, because edition = "old"; implies this.
2. The file's edition is set from "old" to "new". Because every feature that could be explicit is explicit, this is a no-op.
3. Feature janitor runs. This explicitly propagates all features (all of which are set explicitly at the top level), and then cleans them up with respect to the "new" edition; note that feature janitor gives preference to editions defaults. This is a no-op, because feature janitor is a no-op.

UX Concerns

Bundling the editions tooling with protoc ensures that it is easy to find. The following will be the pattern for all Protochangifier tooling bundled into protoc:

• There is a flag --change_spec=changespec.pb which will cause protoc to apply a changespec to the passed-in .proto file, e.g. protoc --change_spec=spec.pb --change_out=foo-changed.proto foo.proto. This writes the change to foo-changed.proto. This may be the same file as foo.proto for in-place updates; it may be left out to have the change printed to stdout. This is the core entry-point for Protochanfigier.
• There is a flag --my_analysis for the given analysis, e.g. --janitor. This flag can have an optional argument: if set, it will output the change spec to that path, e.g. --janitor=spec.pb. If it is not passed in, the change is applied in place without the need to use protoc --change_spec.

Alternatively, we could provide these as standalone tools. However, it seems useful from a distribution perspective and user education perspective to say "this is just part of the compiler". We expect to produce new migration tooling with Protochangifier on an ongoing basis, so teaching users that every analysis looks the same is important. Compare rustfix, the tool that Rust uses for things like upgrading editions. Although it is a separate binary, it is accessible through cargo fix, and in a lot of ways cargo is the user-facing interface to Rust; having it be part of the "swiss army knife" helps put it in front of users.