diff --git a/API_VERSIONING.md b/API_VERSIONING.md new file mode 100644 index 00000000..71f28eb8 --- /dev/null +++ b/API_VERSIONING.md @@ -0,0 +1,199 @@ +# API versioning guidelines + +The Envoy project (and in the future [UDPA](https://github.com/cncf/udpa)) takes API stability and +versioning seriously. Providing stable APIs is a necessary step in ensuring API adoption and success +of the ecosystem. Below we articulate the API versioning guidelines that aim to deliver this +stability. + +# API semantic versioning + +The Envoy APIs consist of a family of packages, e.g. `envoy.admin.v2alpha`, +`envoy.service.trace.v2`. Each package is independently versioned with a protobuf semantic +versioning scheme based on https://cloud.google.com/apis/design/versioning. + +The major version for a package is captured in its name (and directory structure). E.g. version 2 +of the tracing API package is named `envoy.service.trace.v2` and its constituent protos are located +in `api/envoy/service/trace/v2`. Every protobuf must live directly in a versioned package namespace, +we do not allow subpackages such as `envoy.service.trace.v2.somethingelse`. + +Minor and patch versions will be implemented in the future, this effort is tracked in +https://github.com/envoyproxy/envoy/issues/8416. + +In everyday discussion and GitHub labels, we refer to the `v2`, `v3`, `vN`, `...` APIs. This has a +specific technical meaning. Any given message in the Envoy API, e.g. the `Bootstrap` at +`envoy.config.bootstrap.v3.Boostrap`, will transitively reference a number of packages in the Envoy +API. These may be at `vN`, `v(N-1)`, etc. The Envoy API is technically a DAG of versioned package +namespaces. When we talk about the `vN xDS API`, we really refer to the `N` of the root +configuration resources (e.g. bootstrap, xDS resources such as `Cluster`). The +v3 API bootstrap configuration is `envoy.config.bootstrap.v3.Boostrap`, even +though it might might transitively reference `envoy.service.trace.v2`. + +# Backwards compatibility + +In general, within a package's major API version, we do not allow any breaking changes. The guiding +principle is that neither the wire format nor protobuf compiler generated language bindings should +experience a backward compatible break on a change. Specifically: + +* Fields should not be renumbered or have their types changed. This is standard proto development + procedure. + +* Renaming of fields or package namespaces for a proto must not occur. This is inherently dangerous, + since: + * Field renames break wire compatibility. This is stricter than standard proto development + procedure in the sense that it does not break binary wire format. However, it **does** break + loading of YAML/JSON into protos as well as text protos. Since we consider YAML/JSON to be first + class inputs, we must not change field names. + + * For service definitions, the gRPC endpoint URL is inferred from package namespace, so this will + break client/server communication. + + * For a message embedded in an `Any` object, the type URL, which the package namespace is a part + of, may be used by Envoy or other API consuming code. Currently, this applies to the top-level + resources embedded in `DiscoveryResponse` objects, e.g. `Cluster`, `Listener`, etc. + + * Consuming code will break and require source code changes to match the API changes. + +* Some other changes are considered breaking for Envoy APIs that are usually considered safe in + terms of protobuf wire compatibility: + * Upgrading a singleton field to a repeated, e.g. `uint32 foo = 1;` to `repeated uint32 foo = 1`. + This changes the JSON wire representation and hence is considered a breaking change. + + * Wrapping an existing field with `oneof`. This has no protobuf or JSON/YAML wire implications, + but is disruptive to various consuming stubs in languages such as Go, creating unnecessary + churn. + + * Increasing the strictness of + [protoc-gen-validate](https://github.com/envoyproxy/protoc-gen-validate) annotations. Exceptions + may be granted for scenarios in which these stricter conditions model behavior already implied + structurally or by documentation. + +The exception to the above policy is for API versions tagged `vNalpha`. Within an alpha major +version, arbitrary breaking changes are allowed. + +Note that changes to default values for wrapped types, e.g. `google.protobuf.UInt32Value` are not +governed by the above policy. Any management server requiring stability across Envoy API or +implementations within a major version should set explicit values for these fields. + +# API lifecycle + +The API lifecycle follows a calendar clock. At the end of Q3 each year, a major API version +increment may occur for any Envoy API package, in concert with the quarterly Envoy release. + +Envoy will support at most three major versions of any API package at all times: +* The current stable major version, e.g. v3. +* The previous stable major version, e.g. v2. This is needed to ensure that we provide at least 1 + year for a supported major version to sunset. By supporting two stable major versions + simultaneously, this makes it easier to coordinate control plane and Envoy + rollouts as well. This previous stable major version will be supported for 1 + year after the introduction of the new current stable major version. +* Optionally, the next experimental alpha major version, e.g. v4alpha. This is a release candidate + for the next stable major version. This is only generated when the current stable major version + requires a breaking change at the next cycle, e.g. a deprecation or field rename. This release + candidate is mechanically generated via the + [protoxform](https://github.com/envoyproxy/envoy/tree/master/tools/protoxform) tool from the + current stable major version, making use of annotations such as `deprecated = true`. This is not a + human editable artifact. + +An example of how this might play out is that at the end of September in 2020, we will freeze +`envoy.config.bootstrap.v4alpha` and this package will become the current stable major version +`envoy.config.bootstrap.v4`. The `envoy.config.bootstrap.v3` package will become the previous stable +major version and support for `envoy.config.bootstrap.v2` will be dropped from the Envoy +implementation. Note that some transitively referenced package, e.g. +`envoy.config.filter.network.foo.v2` may remain at version 2 during this release, if no changes were +made to the referenced package. + +The implication of this API lifecycle and clock is that any deprecated feature in the Envoy API will retain +implementation support for 1-2 years (1.5 years on average). + +# New API features + +The Envoy APIs can be [safely extended](https://cloud.google.com/apis/design/compatibility) with new +packages, messages, enums, fields and enum values, while maintaining [backwards +compatibility](#backwards-compatibility). Additions to the API for a given package should normally +only be made to the *current stable major version*. The rationale for this policy is that: +* The feature is immediately available to Envoy users who consume the current stable major version. + This would not be the case if the feature was placed in `vNalpha`. +* `vNalpha` can be mechanically generated from `vN` without requiring developers to maintain the new + feature in both locations. +* We encourage Envoy users to move to the current stable major version from the previous one to + consume new functionality. + +# When can an API change be made to a package's previous stable major version? + +As a pragmatic concession, we allow API feature additions to the previous stable major version for a +single quarter following a major API version increment. Any changes to the previous stable major +version must be manually reflected in a consistent manner in the current stable major version as +well. + +# How to make a breaking change across major versions + +We maintain [backwards compatibility](#backwards-compatibility) within a major version but allow +breaking changes across major versions. This enables API deprecations, cleanups, refactoring and +reorganization. The Envoy APIs have a stylized workflow for achieving this. There are two prescribed +methods, depending on whether the change is mechanical or manual. + +## Mechanical breaking changes + +Field deprecations, renames, etc. are mechanical changes that will be supported by the +[protoxform](https://github.com/envoyproxy/envoy/tree/master/tools/protoxform) tool. These are +guided by annotations in protobuf. +* Deprecations are specified with the built-in protobuf deprecated option set on a message, enum, + field or enum value. No field may be marked as deprecated unless a replacement for this + functionality exists and the corresponding Envoy implementation is production ready. + +* Renames are specified with a `[#rename-at-next-major-version: ]` protobuf comment + annotation. + +* We anticipate that `protoxform` will also support `oneof` promotion, package movement, etc. via + similar annotations. + +## Manual breaking changes + +A manual breaking change is distinct from the mechanical changes such as field deprecation, since in +general it requires new code and tests to be implemented in Envoy by hand. For example, if a developer +wants to unify `HeaderMatcher` with `StringMatcher` in the route configuration, this is a likely +candidate for this class of change. The following steps are required: +1. The new version of the feature, e.g. the `NewHeaderMatcher` message should be added, together + with referencing fields, in the current stable major version for the route configuration proto. +2. The Envoy implementation should be changed to consume configuration from the fields added in (1). + Translation code (and tests) should be written to map from the existing field and messages to + (1). +3. The old message/enum/field/enum value should be annotated as deprecated. +4. At the next major version, `protoxform` will remove the deprecated version automatically. + +This approach ensures that API major version releases are predictable and mechanical, and has the +bulk of the Envoy code and test changes owned by feature developers, rather than the API owners. +There will be no major `vN` initiative to address technical debt beyond that enabled by the above +process. + +# One Definition Rule (ODR) + +To avoid maintaining more than two stable major versions of a package, and to cope with diamond +dependency, we add a restriction on how packages may be referenced transitively; a package may have +at most one version of another package in its transitive dependency set. This implies that some +packages will have a major version bump during a release cycle simply to allow them to catch up to +the current stable version of their dependencies. + +Some of this complexity and churn can be avoided by having strict rules on how packages may +reference each other. Package organization and `BUILD` visibility constraints should be used +restrictions to maintain a shallow depth in the dependency tree for any given package. + +# Minimizing the impact of churn + +In addition to stability, the API versioning policy has an explicit goal of minimizing the developer +overhead for the Envoy community, other clients of the APIs (e.g. gRPC), management server vendors +and the wider API tooling ecosystem. A certain amount of API churn between major versions is +desirable to reduce technical debt and to support API evolution, but too much creates costs and +barriers to upgrade. + +We consider deprecations to be *mandatory changes*. Any deprecation will be removed at the next +stable API version. + +Other mechanical breaking changes are considered *discretionary*. These include changes such as +field renames and are largely reflected in protobuf comments. The `protoxform` tool may decide to +minimize API churn by deferring application of discretionary changes until a major version cycle +where the respective message is undergoing a mandatory change. + +The Envoy API structure helps with minimizing churn between versions. Developers should architect +and split packages such that high churn protos, e.g. HTTP connection manager, are isolated in +packages and have a shallow reference hierarchy. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d07e1820..b11d3ddf 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -70,8 +70,3 @@ The following are some general guidelines around documentation. * Prefer *italics* for emphasis as `backtick` emphasis is somewhat jarring in our Sphinx theme. * All documentation is expected to use proper English grammar with proper punctuation. If you are not a fluent English speaker please let us know and we will help out. -* Tag messages/enum/files with `[#proto-status: draft|experimental|frozen]` to - reflect their [API - status](https://www.envoyproxy.io/docs/envoy/latest/configuration/overview/v2_overview#status). - Frozen entities do not need to be tagged except when overriding an outer scope - draft or experimental status. diff --git a/STYLE.md b/STYLE.md index fd1bb2c8..14be4fd1 100644 --- a/STYLE.md +++ b/STYLE.md @@ -7,48 +7,11 @@ particular for proto3 as described at: * https://cloud.google.com/apis/design/naming_convention * https://developers.google.com/protocol-buffers/docs/style -In addition, the following conventions should be followed: - -* For protos that are [frozen](https://www.envoyproxy.io/docs/envoy/latest/configuration/overview/v2_overview#status), - the following guidelines are followed: - - * Fields should not be renumbered or have their types changed. This is standard proto development - procedure. - * If fields are deleted, the following syntax should be put in their place: - - ```proto - reserved ; - ``` - - E.g., - - ```proto - reserved 15; - ``` - - * Renaming of fields or package namespaces for a proto must not occur. This is inherently dangerous, since: - * Fields renames break wire compatibility. This is stricter than standard proto development procedure - in the sense that it does not break binary wire format. However, it **does** break loading - of YAML/JSON into protos as well as text protos. Since we consider YAML/JSON to be first class - inputs, we must not change field names. - - * For service definitions, the gRPC endpoint URL is inferred from package - namespace, so this will break client/server communication. - - * For a message embedded in an `Any` object, the type URL, which the package - namespace is a part of, may be used by Envoy or other API consuming code. - Currently, this applies to the top-level resources embedded in - `DiscoveryResponse` objects, e.g. `Cluster`, `Listener`, etc. - - * Consuming code will break and require source change to match the changes. +A key aspect of our API style is maintaining stability by following the [API versioning +guidelines](API_VERSIONING.md). All developers must familiarize themselves with these guidelines, +any PR which makes breaking changes to the API will not be merged. -* Non-frozen fields should be tagged with `[#not-implemented-hide:]`, `[#not-implemented-warn:]`, - `[#proto-status: draft]` or `[#proto-status: experimental]`. - -* Protos for configs and services that are not implemented immediately in - Envoy, or are under active design and development should be versioned - "vNalpha". See the [stable API versioning - policy](https://github.com/envoyproxy/envoy/issues/6271). +In addition, the following conventions should be followed: * Every proto directory should have a `README.md` describing its content. See for example [envoy.service](envoy/service/README.md). @@ -68,6 +31,10 @@ In addition, the following conventions should be followed: logic, e.g. enable vs. disable for a `bool` field, such that the proto3 defaults work, but only where this doesn't result in API gymnastics. +* Use a `[#not-implemented-hide:]` `protodoc` annotation in comments for fields that lack Envoy + implementation. These indicate that the entity is not implemented in Envoy and the entity + should be hidden from the Envoy documentation. + * Always use plural field names for `repeated` fields, such as `filters`. * Due to the fact that we consider JSON/YAML to be first class inputs, we cannot easily change a @@ -121,10 +88,6 @@ In addition, the following conventions should be followed: // [#comment:next free field: 28] ``` -* The [Breaking Change - Policy](https://github.com/envoyproxy/envoy/blob/master/CONTRIBUTING.md#breaking-change-policy) describes - API versioning, deprecation and compatibility. - ## Package organization API definitions are layered hierarchically in packages from top-to-bottom: