# Stricter Schemas with Editions **Author:** [@mcy](https://github.com/mcy) **Approved:** 2022-11-28 ## Overview The Protobuf language is surprisingly lax in what it allows in some places, even though these corners of the syntax space are rarely exercised in real use, and which add complexity to backends and runtimes. This document describes several such corners in the language, and how we might use Editions to fix them (spoiler: we'll add a feature for each one and then ratchet the features). This is primarily a memo on a use-case for Editions, and not a design doc per se. ## Potential Lints ### Entity Names Protobuf does not enforce any constraints on names other than the "ASCII identifier" rule: they must match the regex `[A-Za-z_][A-Za-z0-9_]*`. This results in problems for backends: * Backends need to be able to convert between PascalCase, camelCase, snake_case, and SHOUTY_CASE. Doing so correctly is surprisingly tricky. * Extraneous underscores, such as underscores in names that want to be PascalCase, trailing underscores, leading underscores, and repeated underscores create problems for case conversion and can clash with private names generated by backends. * Protobuf does not support non-ASCII identifiers, mostly out of inertia more than anything else. Because some languages (Java most prominent among them) do not support them, we can never support them, but we are not particularly clear on this point. The Protobuf language should be as strict as possible in what patterns it accepts for identifiers, since these need to be transformed to many languages. Thus, we propose the following regexes for the three casings used in Protobuf: * `([A-Z][a-zA-Z0-9]*)+` for PascalCase. We require this case for: * Messages. * Enums. * Services. * Methods. * `[a-z][a-z0-9]*(_[a-z0-9]+)*` for snake_case. We require this case for: * Fields (including extensions). * Package components. * `[A-Z][A-Z0-9]*(_[A-Z0-9]+)*` for SHOUTY_CASE. We require this case for: * Enum values. These patterns are intended to reject extraneous underscores, and to make casing of ASCII letters consistent. We explicitly only support ASCII for maximal portability to target languages. Note that option names are not included, since those are defined as fields in a proto, and would be subject to this rule automatically. To migrate, we would introduce a bool feature `feature.relax_identifier_rules`, which can be applied to any entity. When set, it would cause the compiler to reject `.proto` files which contain identifiers that don't match the above constraints. It would default to true and would switch to false in a future edition. ### Keywords as Identifiers Currently, the Protobuf language allows using keywords as identifiers. This makes the parser somewhat more complicated than it has to be for minimal benefit, and shadowing behavior is not well-specified. For example, what does the following compile as? ``` message Foo { message int32 {} optional int32 foo = 1; } ``` This is particularly fraught in places where either a keyword or a type name can follow. For example, `optional foo = 1;` is a proto3 non-optional with type `optional`, but the parser can't tell until it sees the `=` sign. To avoid this and eventually stop supporting this in the parser, we make the following set of keywords true reserved names that cannot be used as identifiers: ``` bool bytes double edition enum extend extensions fixed32 fixed64 float group import int32 int64 map max message oneof option optional package public repeated required reserved returns rpc service sfixed32 sfixed64 sint32 sint64 stream string syntax to uint32 uint64 weak ``` Additionally, we introduce the syntax `#optional` for escaping a keyword as an identifier. This may *only* be used on keywords, and not non-keyword identifiers. To migrate, we would introduce a bool feature `feature.keywords_as_identifiers`, which can be applied to any entity. When set, it would cause the compiler to reject `.proto` files which contain identifiers that use the names of keywords. It would migrate true->false in a future edition. The `#optional` syntax would not need to be feature-gated. From time to time we may introduce new keywords. The best procedure for doing so is to add a `feature.xxx_is_a_keyword` feature, start it out as true, and then switch it to false in an edition, which would cause it to be treated as a keyword for the purposes of this check. There's nothing stopping us from starting to use it in the syntax without an edition if it would be relatively unambiguous (i.e., a "contextual" keyword). Rust provides guidance here: they really hate contextual keywords since it complicates the parser, so keywords start out as contextual and become properly reserved in the next Rust edition. ### Nonempty Package Right now, an empty package is technically permitted. We should remove this functionality from the language completely and require every file to declare a package. We would introduce a feature like `feature.allow_missing_package`, start it out as true, and switch it to false. ### Invalid Names in `reserved` Currently, `reserved "foo-bar";` is accepted. It is not a valid name for a field and thus should be rejected. Ideally we should remove this syntax altogether and only permit the use of identifiers in this position, such as `reserved foo, bar;`. We would introduce a feature like `feature.allow_strings_in_reserved`, start it out as true, and then switch it to false. ### Almost All Names are Fully Qualified Right now, Protobuf defines a complicated name resolution scheme that involves matching subsets of names inspired by that of C++ (which is even more complicated than ours!). Instead, we should require that every name be either a single identifier OR fully-qualified. This is an attempt to move to Go-style name resolution, which is significantly simpler to implement and explain. In particular, if a name is a single identifier, then: * It must be the name of a type defined at the top level of the current file. * If it is the name of a message or enum for a field's type, it may be the name of a type defined in the current message. This does *not* apply to extension fields. Because any multi-component path must be fully qualified, we no longer need the `.foo.Bar` syntax anymore, except to refer to messages defined in files without a package. We forbid `.`-prefixed names except in that case. We would introduce a feature like `features.use_cpp_style_name_resolution`, start it out as true, and then switch it to false. Ideally, if we get strict identifier names, we can tell that `Foo.Bar` is rooted at a message, rather than a package. In that case, we could go as far as saying that "names that start with a lower-case letter are fully-qualified, otherwise they are relative to the current package, and will only find things defined in the current file." Unlike Go, we do not allow finding things in other packages without being fully-qualified; this mostly comes from doing source-diving in very large packages, like the Go runtime, where it is very hard to find where something is defined. ### Unique Enum Values Right now, we allow aliases in enums: ``` enum Foo { BAR = 5; BAZ = 5; } ``` This results in significant complexity in some parts of the backend, and weird behavior in textproto and JSON. We should disallow this. We would introduce a feature like `features.allow_enum_aliases`, which would switch from true to false. ### Imports are Used We should adopt the Go rule that all non-public imports are used (i.e, every import provides at least one type referred to in the file). We would introduce a feature like `features.allow_unused_imports`, which would switch from true to false. ### Next Field # is Reserved There's a few idioms for this checked by linters, such as `// Next ID: N`. We should codify this in the language by rewriting that every message begin with `reserved N to max;`, with the intent that `N` is the next never-used field number. Because it is required to be the first production in the message, it can be We could, additionally, require that *every* field number be either used or reserved, in addition to having a single `N to max;` reservation. Alternatively, we could require that every field number up to the largest one used be reserved; gaps between message numbers are usually a smell. This applies equally to message fields and enum values. We would introduce a feature like `features.allow_unused_numbers`, which we would switch from true to false. ### Disallow Implicit String Concatenation Protobuf will implicitly concatenate two adjacent strings in any place it allows quoted strings, e.g. `option foo = "bar " "baz;`. This has caused interesting problems around `reserved` in the past, if a comma is omitted: `reserved "foo" "bar";` is `reserved "foobar";`. We would introduce a feature like `features.concatenate_adjacent_strings`, which would switch from true to false. ### Package Is First The `package` declaration can appear anywhere in the file after `syntax` or `edition`. We should take cues from Go and require it to be the first thing in the file, after the edition. We would introduce a feature like `features.package_anywhere`, which would switch from true to false. ### Strict Boolean Options Boolean options can use true, false, True, False, T, or F as a value: `option my_bool = T;`. We should restrict to only `true` and `false`. We would introduce a feature like `features.loose_bool_options`, which would switch from true to false. ### Decimal Field Numbers We permit non-decimal integer literals for field numbers, e.g. `optional int32 x = 0x01;`. Thankfully(?) we do not already permit a leading + or -. We should require decimal literals, since there is very little reason to allow other literals and makes the Protobuf language harder to parse. We would introduce a feature like `features.non_decimal_field_numbers`, which would switch from true to false.