xds: add support for TTLs (#13201)

Adds support for per resource TTL for both Delta and SOTW xDS. This allows the server to direct Envoy to remove the resources in the case of control plane unavailability.

Signed-off-by: Snow Pettersen <snowp@lyft.com>

Co-authored-by: Bill Gallagher <bgallagher@lyft.com>

Mirrored from https://github.com/envoyproxy/envoy @ b2b3978afb25c5d2c3ca6b344071b285e8820da1

envoy/service/discovery/v3/discovery.proto

@@ -5,6 +5,7 @@ package envoy.service.discovery.v3;
 import "envoy/config/core/v3/base.proto";
 
 import "google/protobuf/any.proto";
+import "google/protobuf/duration.proto";
 import "google/rpc/status.proto";
 
 import "udpa/core/v1/resource_locator.proto";
@@ -252,7 +253,7 @@ message DeltaDiscoveryResponse {
   string nonce = 5;
 }
 
-// [#next-free-field: 6]
+// [#next-free-field: 7]
 message Resource {
   option (udpa.annotations.versioning).previous_message_type = "envoy.api.v2.Resource";
 
@@ -272,4 +273,19 @@ message Resource {
 
   // The resource being tracked.
   google.protobuf.Any resource = 2;
+
+  // Time-to-live value for the resource. For each resource, a timer is started. The timer is
+  // reset each time the resource is received with a new TTL. If the resource is received with
+  // no TTL set, the timer is removed for the resource. Upon expiration of the timer, the
+  // configuration for the resource will be removed.
+  //
+  // The TTL can be refreshed or changed by sending a response that doesn't change the resource
+  // version. In this case the resource field does not need to be populated, which allows for
+  // light-weight "heartbeat" updates to keep a resource with a TTL alive.
+  //
+  // The TTL feature is meant to support configurations that should be removed in the event of
+  // a management server // failure. For example, the feature may be used for fault injection
+  // testing where the fault injection should be terminated in the event that Envoy loses contact
+  // with the management server.
+  google.protobuf.Duration ttl = 6;
 }

envoy/service/discovery/v4alpha/discovery.proto

@@ -5,6 +5,7 @@ package envoy.service.discovery.v4alpha;
 import "envoy/config/core/v4alpha/base.proto";
 
 import "google/protobuf/any.proto";
+import "google/protobuf/duration.proto";
 import "google/rpc/status.proto";
 
 import "udpa/core/v1/resource_locator.proto";
@@ -254,7 +255,7 @@ message DeltaDiscoveryResponse {
   string nonce = 5;
 }
 
-// [#next-free-field: 6]
+// [#next-free-field: 7]
 message Resource {
   option (udpa.annotations.versioning).previous_message_type =
       "envoy.service.discovery.v3.Resource";
@@ -276,4 +277,19 @@ message Resource {
 
   // The resource being tracked.
   google.protobuf.Any resource = 2;
+
+  // Time-to-live value for the resource. For each resource, a timer is started. The timer is
+  // reset each time the resource is received with a new TTL. If the resource is received with
+  // no TTL set, the timer is removed for the resource. Upon expiration of the timer, the
+  // configuration for the resource will be removed.
+  //
+  // The TTL can be refreshed or changed by sending a response that doesn't change the resource
+  // version. In this case the resource field does not need to be populated, which allows for
+  // light-weight "heartbeat" updates to keep a resource with a TTL alive.
+  //
+  // The TTL feature is meant to support configurations that should be removed in the event of
+  // a management server // failure. For example, the feature may be used for fault injection
+  // testing where the fault injection should be terminated in the event that Envoy loses contact
+  // with the management server.
+  google.protobuf.Duration ttl = 6;
 }

xds_protocol.rst

@@ -656,6 +656,42 @@ adding/removing/updating clusters. On the other hand, routes are not
 warmed, i.e., the management plane must ensure that clusters referenced
 by a route are in place, before pushing the updates for a route.
 
+.. _xds_protocol_TTL:
+
+TTL
+~~~
+
+In the event that the management server becomes unreachable, the last known configuration received
+by Envoy will persist until the connection is reestablished. For some services, this may not be
+desirable. For example, in the case of a fault injection service, a management server crash at the
+wrong time may leave Envoy in an undesirable state. The TTL setting allows Envoy to remove a set of
+resources after a specified period of time if contact with the management server is lost. This can
+be used, for example, to terminate a fault injection test when the management server can no longer
+be reached.
+
+For clients that support the *xds.config.supports-resource-ttl* client feature, A TTL field may
+be specified on each :ref:`Resource <envoy_api_msg_Resource>`. Each resource will have its own TTL
+expiry time, at which point the resource will be expired. Each xDS type may have different ways of
+handling such an expiry.
+
+To update the TTL associated with a *Resource*, the management server resends the resource with a
+new TTL. To remove the TTL, the management server resends the resource with the TTL field unset.
+
+To allow for lightweight TTL updates ("heartbeats"), a response can be sent that provides a
+:ref:`Resource <envoy_api_msg_Resource>` with the :ref:`resource <envoy_api_field_Resource.resource>`
+unset and version matching the most recently sent version can be used to update the TTL. These
+resources will not be treated as resource updates, but only as TTL updates.
+
+SotW TTL
+^^^^^^^^
+
+In order to use TTL with SotW xDS, the relevant resources must be wrapped in a
+:ref:`Resource <envoy_api_msg_Resource>`. This allows setting the same TTL field that is used for
+Delta xDS with SotW, without changing the SotW API. Heartbeats are supported for SotW as well:
+any resource within the response that look like a heartbeat resource will only be used to update the TTL.
+
+This feature is gated by the *xds.config.supports-resource-in-sotw* client feature.
+
 .. _xds_protocol_ads:
 
 Aggregated Discovery Service