protobuf/docs/design/editions/what-are-protobuf-editions.md

What are Protobuf Editions?

Authors: @mcy, @fowles

Summary

This document is an introduction to the Protobuf Editions project, an ambitious re-imagining of how we migrate Protobuf users into the future.

Goal

Enable incremental evolution of Protobuf across the entire ecosystem without introducing permanent forks in the Protobuf language.

TL;DR

1. We are replacing syntax = ... with edition = ....
   • We plan to produce a new "edition" on a roughly yearly basis.
   • We plan to regularly deprecate and remove old editions after a wide horizon.
   • This gradual churn is enabled by the Protobuf Breaking Changes policy.
2. "Features" are a special kind of file/message/field/enum/etc option.
   • Features control the individual codegen and runtime behavior of fields, messages, enums, etc.
   • Features cannot introduce changes that would directly break existing binaries.
   • We expect heavy churn of features in .proto files, so their design is optimized to minimize diffs to .proto files while permitting fine-grained control.
   • Features are usually attached to the field/message/enum they apply to.
     • Features can be specified at a higher-level entity, such as a file, to apply to all definitions inside of that entity. This is called feature inheritance.
     • Inheritance is intended to allow us to factor frequently-occurring feature declarations, minimizing clutter during migrations.
3. Editions change only the defaults of features and do not otherwise introduce new behavior.
   • New behavior is fundamentally controlled by features (explicitly set or implicit from an edition).
   • Editions allow us to ratchet the ecosystem forward.
     • Editions can be incremented on a per .proto file basis; projects can upgrade incrementally.
4. Messages with any permutation of features are always interoperable (they can import each other freely and use messages from each other).
   • Editions do not split the ecosystem, and migration is largely automated.
   • Directly inspired by Rust editions.
   • Carbon has a similar philosophy
5. The proto2/proto3 distinction is going away.
   • Editions will support everything from both and allow mixed semantics even within the same message or field.
   • Undesirable features will be LSC'd away, using the same template as any other feature/edition migration.

Motivation

Arguably the biggest hard-earned lesson among Software Foundations is that successful migrations are incremental. Most of our experience with these has been for internal migrations. Externally, progress has often ossified because of a lack of established evolution mechanisms. More recently large projects have started planning incremental evolution into their structure. For example, Carbon is heavily focused on evolution as a core precept, and Rust has built language evolution via editions into its core design..

Protobuf is one of Google's oldest and most successful toolchain projects. However, it was designed before we learned and internalized this lesson, making modernization difficult and haphazard. We still have required and group, packed is not everywhere, and string accessors in C++ still return const std::string&. The last radical change to Protobuf (syntax = "proto3";) split the ecosystem.

Editions and features are new language features that will allow us to incrementally evolve Protobuf into the future. This will be done by introducing a new syntax, hopefully the last syntax addition we will ever need.

This high-level document is intended as an introduction to Protobuf Editions for engineers not familiar with the background and the set of tradeoffs that lead us here. Low-level technical details are skipped in preference to describing the kernel of our proposed design. This document reflects the approximate consensus of protobuf-team members who have been developing Protobuf Editions, but please beware: many open questions remain.

What is a feature?

A feature, in the narrow context of Protobuf Editions, is an option on any syntax entity of a .proto file that has the following properties:

• It is a field or extension of a top-level option named features, which is present on every syntax entity (file, message, enum, field, etc). It can be of any type, but bool and enum are the most common.
• If a syntax entity's lexical parent has a particular value for a feature, then the child has the same value, unless the feature has a new value specified on the child, explicitly. This is called feature inheritance, and applies recursively. Features can specify a new value at any of the points where a feature can be added.
• It explicitly specifies what syntax entities it can be set on, similar to Java annotations (although this does not preclude inheritance to or through an entity that it cannot be set on).

Features allow us to control the behavior of protoc, its backends, and the Protobuf runtimes at arbitrary granularity. This is critical for large-scale changes: if a message has few usages, features can be changed at a bigger scope, minimizing diff churn, but if it has heavy usage and the CL to migrate a single field is large, cleanups can happen at the field level, as necessary.

Features won't change a message’s serialization formats (binary, text, or json) in incompatible ways except for extreme circumstances that will always be managed directly by protobuf-team. It is critical for migrations that any behavioral change coming from a feature is the result of a textual change to a .proto file (either an edition bump or a feature change).

ctype is an existing field option that looks exactly like a feature: it controls the behavior of the codegen backend, although it does not have the nice ratcheting properties of editions.

Because features can be extensions, language backends can specify language-scoped features. For example, [ctype = CORD] could instead be phrased as [features.(pb.cpp).string_type = CORD]. Codegen backends own the definitions of their features.

What is an Edition?

An edition is a collection of defaults for features understood by protoc and its backends. Editions are year-numbered, although we have defined a breakout in case we need multiple editions in a particular year.

Instead of writing syntax = "...";, a Protobuf Editions-enabled .proto file begins with edition = "2022"; or similar. edition implies syntax = "editions";, and the syntax keyword itself becomes deprecated. This is to ensure that old tools not owned by protobuf-team, which only work for old Protobuf syntaxes, crash or fail quickly and noticeably, instead of wandering into a descriptor that they cannot understand (we will attempt to migrate what we can, of course).

protoc specifies which editions it understands, and will reject .proto files "from the future", since it cannot meaningfully parse them. protoc backends, which can specify their own set of language-scoped features, must advertise the defaults for a particular edition that they understand (and reject editions that they don't). Runtimes must be able to handle descriptors "from the future"; this only means that upon encountering a descriptor with an edition or feature it does not understand, there must be a reasonable fallback for the runtime's behavior.

What is an Edition used for?

Editions provide the fundamental increments for the lifecycle of a feature. At this point it is important to reiterate that most features will be specific to particular code generators. What follows is an example life cycle for a theoretical feature–features.(pb.cpp).opaque_repeated_fields.

1. Edition “2025” creates features.(pb.cpp).opaque_repeated_fields with a default value of false. This value is equivalent to the behavior from editions less than “2025”.

   a. The migration to edition “2025” across google will move very fast as it is a no-op.

2. Migration begins for features.(pb.cpp).opaque_repeated_fields (each change in this migration will add features.(pb.cpp).opaque_repeated_fields = true and be paired with code changes required to C++ code). It is not anticipated that protos shared between repos will undergo field by field migrations like this as that would cause a large stream of breaking changes, see Protobuf Editions for schema producers for more details.

3. Edition “2027” switches the default of features.(pb.cpp).opaque_repeated_fields to true.

   a. The migration to “2027” will remove explicit uses of features.(pb.cpp).opaque_repeated_fields = true and add explicit uses of features.(pb.cpp).opaque_repeated_fields = false where they were implicit before. As above, this migration will be a no-op, so it will move very fast.

   b. Externally, we will release tools and migration guides for OSS customers. The tools will not be fully turnkey, but should provide a strong starting point for user migrations.

4. Migration continues for features.(pb.cpp).opaque_repeated_fields (each change in this migration will remove features.(pb.cpp).opaque_repeated_fields = false and be paired with code changes required to C++ code).

5. At some point, usage will be officially roped off internally, and externally.

   a. Internally, features.(pb.cpp).opaque_repeated_fields usage will be blocked with allowlists while we remove the hardest to migrate case.

   b. Externally, features.(pb.cpp).opaque_repeated_fields will be marked deprecated in a public edition and removed in a later one. When a feature is removed, the code generators for that behavior and the runtime libraries that support it may also be removed. In this hypothetical, that might be deprecated in “2029” and removed in “2031”. Any release that removes support for a feature would be a major version bump.

The key point to note here is that any .proto file that does not use deprecated features has a no-op upgrade from one edition to the next and we will provide tools to effect that upgrade. Internal users will be migrated centrally before a feature is deprecated. External users will have the full window of the Google migration as well as the deprecation window to upgrade their own code.

It is also important to note that external users will not receive compiler warnings until the feature is actually deprecated, so we provide a period of deprecation to ensure that they have time to update their code before forcing them to upgrade for an edition update.

Separately from feature evolution, protoc itself may remove support for old editions entirely after a suitably long window (like 10 years).

Edition Zero

The first edition of Protobuf Editions, the so-called "edition zero", will effectively be a "proto4" that introduces the new syntax, and merges the semantics of proto2 and proto3. In editions mode, everything that was possible in proto2 and proto3 will be possible, and the handful of irreconcilable differences will be expressed as features.

For example, whether values not specified in an enum go into unknown fields vs producing an enum value outside of the bounds of the specified values in the .proto file (i.e., so-called closed and open enums) will be controlled by feature.enum = OPEN or feature.enum = CLOSED.

Edition Zero should be viewed as the "completion" of the union of proto2 and proto3: it contains both syntaxes as subsets (although with different spellings to disambiguate things) as well as new behavior that was previously inexpressible but which is an obvious consequence of allowing everything from both. For example, proto3-style non-optional singular fields could allow non-zero defaults.

Edition Zero is designed in such a way that we can mechanically migrate an arbitrary .proto file from either proto2 or proto3 with no behavioral changes, by replacing syntax with edition and adding features in the appropriate locations.

This will form the foundation of Protobuf Editions and the torrent of parallel migrations that will follow.

FAQ

I only interact with protos by moving them around and editing schemata. How does this affect me?

This will manifest as a handful of new options appearing at the top of your files. Going forward, expect new options to appear and disappear from your .proto files as LSCs march across the codebase. We intend to minimize disruption, and you should be able to safely ignore them.

In general, you should not need to add options yourself unless we say so in documentation. We will try to make sure tooling recommends the latest edition when creating new files.

Are you taking away <thing>?

Everything expressible today will remain so in Edition Zero. Some syntax will change: we will have only one way of spelling a singular field (with optional vs. the proto3 behavior vs. required controlled by a feature), groups will turn into sub message fields with a special encoding.

I think <thing> from proto{2,3} is bad. Why are you letting people use it in my files?

Long-term bifurcation of the language has resulted in significant damage to the. ecosystem and engineers' mental model of Protobuf. There are features we think are questionable, too, and we want to remove them. But we need to break some eggs to make an omelet.

As stewards of the Protobuf language, we believe this is the best way to get rid of features that were a good idea at the time, but which history has shown to have had poor outcomes.

I manipulate protos reflectively, or have some other complicated use-case

We plan to upgrade reflection to be feature-aware in a way that minimizes code we need to change. We do not expect anyone to implement feature-inheritance logic themselves; feature inheritance should be fully transparent to users, behaving as if features had been placed explicitly everywhere. (Owners of code generators should be the only ones that need to know how to correctly propagate features.)

We will be partnering with use-cases that are known risks for migration, such as storage providers, to minimize toil and disruption on all sides.

I want to use features to fix a defect in Protobuf

Generally, the owner of the relevant component that ingests a particular feature (protoc or the appropriate language backend) will own it. We will try to make it as straightforward as we can to add a language-scoped feature, but it may require some degree of coordination with us to get it into an edition.

Even if it's about one of protobuf-team's backends, we'd love to hear what you think we can fix, within the constraints of editions.

What's your OSS strategy?

We want to share a variant of this document with the OSS community. We plan to publish migration guides and, where feasible, any migration tooling, such as the proto2/proto3 -> edition migrator.

As stated above, we want to minimize friction for non-protobuf-team-owned backends, and this ties into helping third party code generators minimize their pain.

I like Protobuf as it is. Can I keep my old files?

Yes, but you get to keep both pieces. Failing to migrate off of old use-cases and into newer versions that fix known defects is a risk for the entire ecosystem: C++'s disastrous standardization process is a solemn warning of failing to do so.

Trying to stay on proto2 or proto3 will eventually cease to be supported, and old editions (e.g. 5 years) will also cease to be supported. Evolution is at the heart of Protobuf, and we want to make it as easy as possible for users to keep up with our progress towards a better Protobuf.

What do you hope to use editions to change in the short/mid term?

An incomplete list of ideas, which should be taken as non-committal.

• Eliminate required completely by making a particular field be optional but serialized unconditionally.
• Make all uses of string require UTF-8 checking, and all uses that don't want/need it bytes, fulfilling the original proto3 vision.
• Make every string and bytes accessor in C++ return absl::string_view, unlocking performance optimizations.
• Make all scalar repeated fields packed, improving throughput.
• Make enum enumerators in C++ use kName instead of NAME.
• Make enum declarations in C++ into scoped enum class.
• Make CTYPE into a language-scoped feature.
• Replace per-language, file-level options with language-scoped features.
• Make reflection opt-in for some languages (C++).