You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
857 lines
51 KiB
857 lines
51 KiB
.. _xds_protocol: |
|
|
|
xDS REST and gRPC protocol |
|
========================== |
|
|
|
Envoy discovers its various dynamic resources via the filesystem or by |
|
querying one or more management servers. Collectively, these discovery |
|
services and their corresponding APIs are referred to as *xDS*. |
|
Resources are requested via *subscriptions*, by specifying a filesystem |
|
path to watch, initiating gRPC streams, or polling a REST-JSON URL. The |
|
latter two methods involve sending requests with a :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` |
|
proto payload. Resources are delivered in a |
|
:ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` |
|
proto payload in all methods. We discuss each type of subscription |
|
below. |
|
|
|
Resource Types |
|
-------------- |
|
|
|
Every configuration resource in the xDS API has a type associated with it. Resource types follow a |
|
:repo:`versioning scheme <api/API_VERSIONING.md>`. Resource types are versioned independent of the |
|
transports described below. |
|
|
|
The following v3 xDS resource types are supported: |
|
|
|
- :ref:`envoy.config.listener.v3.Listener <envoy_v3_api_msg_config.listener.v3.Listener>` |
|
- :ref:`envoy.config.route.v3.RouteConfiguration <envoy_v3_api_msg_config.route.v3.RouteConfiguration>` |
|
- :ref:`envoy.config.route.v3.ScopedRouteConfiguration <envoy_v3_api_msg_config.route.v3.ScopedRouteConfiguration>` |
|
- :ref:`envoy.config.route.v3.VirtualHost <envoy_v3_api_msg_config.route.v3.VirtualHost>` |
|
- :ref:`envoy.config.cluster.v3.Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` |
|
- :ref:`envoy.config.endpoint.v3.ClusterLoadAssignment <envoy_v3_api_msg_config.endpoint.v3.ClusterLoadAssignment>` |
|
- :ref:`envoy.extensions.transport_sockets.tls.v3.Secret <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.Secret>` |
|
- :ref:`envoy.service.runtime.v3.Runtime <envoy_v3_api_msg_service.runtime.v3.Runtime>` |
|
|
|
The concept of `type URLs <https://developers.google.com/protocol-buffers/docs/proto3#any>`_ |
|
appears below, and takes the form ``type.googleapis.com/<resource type>`` -- e.g., |
|
``type.googleapis.com/envoy.config.cluster.v3.Cluster`` for a ``Cluster`` resource. In various requests from |
|
Envoy and responses by the management server, the resource type URL is stated. |
|
|
|
|
|
Filesystem subscriptions |
|
------------------------ |
|
|
|
The simplest approach to delivering dynamic configuration is to place it |
|
at a well known path specified in the :ref:`ConfigSource <envoy_v3_api_msg_config.core.v3.ConfigSource>`. |
|
Envoy will use ``inotify`` (``kqueue`` on macOS) to monitor the file for |
|
changes and parse the |
|
:ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` proto in the file on update. |
|
Binary protobufs, JSON, YAML and proto text are supported formats for |
|
the |
|
:ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`. |
|
|
|
There is no mechanism available for filesystem subscriptions to ACK/NACK |
|
updates beyond stats counters and logs. The last valid configuration for |
|
an xDS API will continue to apply if an configuration update rejection |
|
occurs. |
|
|
|
.. _xds_protocol_streaming_grpc_subscriptions: |
|
|
|
Streaming gRPC subscriptions |
|
---------------------------- |
|
|
|
API flow |
|
~~~~~~~~ |
|
|
|
For typical HTTP routing scenarios, the core resource types for the client's configuration are |
|
``Listener``, ``RouteConfiguration``, ``Cluster``, and ``ClusterLoadAssignment``. Each ``Listener`` resource |
|
may point to a ``RouteConfiguration`` resource, which may point to one or more ``Cluster`` resources, |
|
and each ``Cluster`` resource may point to a ``ClusterLoadAssignment`` resource. |
|
|
|
Envoy fetches all ``Listener`` and ``Cluster`` resources at startup. It then fetches whatever |
|
``RouteConfiguration`` and ``ClusterLoadAssignment`` resources that are required by the ``Listener`` and |
|
``Cluster`` resources. In effect, every ``Listener`` or ``Cluster`` resource is a root to part of Envoy's |
|
configuration tree. |
|
|
|
A non-proxy client such as gRPC might start by fetching only the specific ``Listener`` resources |
|
that it is interested in. It then fetches the ``RouteConfiguration`` resources required by those |
|
``Listener`` resources, followed by whichever ``Cluster`` resources are required by those |
|
``RouteConfiguration`` resources, followed by the ``ClusterLoadAssignment`` resources required |
|
by the ``Cluster`` resources. In effect, the original ``Listener`` resources are the roots to |
|
the client's configuration tree. |
|
|
|
Variants of the xDS Transport Protocol |
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
Four Variants |
|
^^^^^^^^^^^^^ |
|
|
|
There are four variants of the xDS transport protocol used via streaming gRPC, which cover all |
|
combinations of two dimensions. |
|
|
|
The first dimension is State of the World (SotW) vs. incremental. The SotW approach was the |
|
original mechanism used by xDS, in which the client must specify all resource names it is |
|
interested in with each request (except when making a wildcard request in LDS/CDS), and the server |
|
must return all resources the client has subscribed to in each request (in LDS/CDS). This means |
|
that if the client is already subscribing to 99 resources and wants to add an additional one, it |
|
must send a request with all 100 resource names, rather than just the one new one. And the server |
|
must then respond by sending all 100 resources, even if the 99 that were already subscribed to have |
|
not changed (in LDS/CDS). This mechanism can be a scalability limitation, which is why the |
|
incremental protocol variant was introduced. The incremental approach allows both the client and |
|
server to indicate only deltas relative to their previous state -- i.e., the client can say that |
|
it wants to add or remove its subscription to a particular resource name without resending those |
|
that have not changed, and the server can send updates only for those resources that have changed. |
|
The incremental protocol also provides a mechanism for lazy loading of resources. For details on |
|
the incremental protocol, see :ref:`Incremental xDS <xds_protocol_delta>` below. |
|
|
|
The second dimension is using a separate gRPC stream for each resource type vs. aggregating all |
|
resource types onto a single gRPC stream. The former approach was the original mechanism used by |
|
xDS, and it offers an eventual consistency model. The latter approach was added for environments |
|
in which explicit control of sequencing is required. For details, see :ref:`Eventual consistency |
|
considerations <xds_protocol_eventual_consistency_considerations>` below. |
|
|
|
So, the four variants of the xDS transport protocol are: |
|
|
|
1. State of the World (Basic xDS): SotW, separate gRPC stream for each resource type |
|
2. Incremental xDS: incremental, separate gRPC stream for each resource type |
|
3. Aggregated Discovery Service (ADS): SotW, aggregate stream for all resource types |
|
4. Incremental ADS: incremental, aggregate stream for all resource types |
|
|
|
RPC Services and Methods for Each Variant |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
For the non-aggregated protocol variants, there is a separate RPC service for each resource type. |
|
Each of these RPC services can provide a method for each of the SotW and Incremental protocol |
|
variants. Here are the RPC services and methods for each resource type: |
|
|
|
- Listener: Listener Discovery Service (LDS) |
|
- SotW: ListenerDiscoveryService.StreamListeners |
|
- Incremental: ListenerDiscoveryService.DeltaListeners |
|
- RouteConfiguration: Route Discovery Service (RDS) |
|
- SotW: RouteDiscoveryService.StreamRoutes |
|
- Incremental: RouteDiscoveryService.DeltaRoutes |
|
- ScopedRouteConfiguration: Scoped Route Discovery Service (SRDS) |
|
- SotW: ScopedRouteDiscoveryService.StreamScopedRoutes |
|
- Incremental: ScopedRouteDiscoveryService.DeltaScopedRoutes |
|
- VirtualHost: Virtual Host Discovery Service (VHDS) |
|
- SotW: N/A |
|
- Incremental: VirtualHostDiscoveryService.DeltaVirtualHosts |
|
- Cluster: Cluster Discovery Service (CDS) |
|
- SotW: ClusterDiscoveryService.StreamClusters |
|
- Incremental: ClusterDiscoveryService.DeltaClusters |
|
- ClusterLoadAssignment: Endpoint Discovery Service (EDS) |
|
- SotW: EndpointDiscoveryService.StreamEndpoints |
|
- Incremental: EndpointDiscoveryService.DeltaEndpoints |
|
- Secret: Secret Discovery Service (SDS) |
|
- SotW: SecretDiscoveryService.StreamSecrets |
|
- Incremental: SecretDiscoveryService.DeltaSecrets |
|
- Runtime: Runtime Discovery Service (RTDS) |
|
- SotW: RuntimeDiscoveryService.StreamRuntime |
|
- Incremental: RuntimeDiscoveryService.DeltaRuntime |
|
|
|
In the aggregated protocol variants, all resource types are multiplexed on a single gRPC stream, |
|
where each resource type is treated as a separate logical stream within the aggregated stream. |
|
In effect, it simply combines all of the above separate APIs into a single stream by treating |
|
requests and responses for each resource type as a separate sub-stream on the single aggregated |
|
stream. The RPC service and methods for the aggregated protocol variants are: |
|
|
|
- SotW: AggregatedDiscoveryService.StreamAggregatedResources |
|
- Incremental: AggregatedDiscoveryService.DeltaAggregatedResources |
|
|
|
For all of the SotW methods, the request type is :ref:`DiscoveryRequest |
|
<envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` and the response type is :ref:`DiscoveryResponse |
|
<envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`. |
|
|
|
For all of the incremental methods, the request type is :ref:`DeltaDiscoveryRequest |
|
<envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>` and the response type is :ref:`DeltaDiscoveryResponse |
|
<envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryResponse>`. |
|
|
|
Configuring Which Variant to Use |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
In the xDS API, the :ref:`ConfigSource <envoy_v3_api_msg_config.core.v3.ConfigSource>` message indicates how to |
|
obtain resources of a particular type. If the :ref:`ConfigSource <envoy_v3_api_msg_config.core.v3.ConfigSource>` |
|
contains a gRPC :ref:`ApiConfigSource <envoy_v3_api_msg_config.core.v3.ApiConfigSource>`, it points to an |
|
upstream cluster for the management server; this will initiate an independent bidirectional gRPC |
|
stream for each xDS resource type, potentially to distinct management servers. If the |
|
:ref:`ConfigSource <envoy_v3_api_msg_config.core.v3.ConfigSource>` contains a :ref:`AggregatedConfigSource |
|
<envoy_v3_api_msg_config.core.v3.AggregatedConfigSource>`, it tells the client to use :ref:`ADS |
|
<xds_protocol_ads>`. |
|
|
|
Currently, the client is expected to be given some local configuration that tells it how to obtain |
|
the :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resources. |
|
:ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` resources may include a |
|
:ref:`ConfigSource <envoy_v3_api_msg_config.core.v3.ConfigSource>` that indicates how the |
|
:ref:`RouteConfiguration <envoy_v3_api_msg_config.route.v3.RouteConfiguration>` resources are obtained, and |
|
:ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resources may include a |
|
:ref:`ConfigSource <envoy_v3_api_msg_config.core.v3.ConfigSource>` that indicates how the |
|
:ref:`ClusterLoadAssignment <envoy_v3_api_msg_config.endpoint.v3.ClusterLoadAssignment>` resources are obtained. |
|
|
|
Client Configuration |
|
"""""""""""""""""""" |
|
|
|
In Envoy, the bootstrap file contains two :ref:`ConfigSource <envoy_v3_api_msg_config.core.v3.ConfigSource>` |
|
messages, one indicating how :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` resources are obtained and |
|
another indicating how :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resources are obtained. It also |
|
contains a separate :ref:`ApiConfigSource <envoy_v3_api_msg_config.core.v3.ApiConfigSource>` message indicating |
|
how to contact the ADS server, which will be used whenever a :ref:`ConfigSource |
|
<envoy_v3_api_msg_config.core.v3.ConfigSource>` message (either in the bootstrap file or in a :ref:`Listener |
|
<envoy_v3_api_msg_config.listener.v3.Listener>` or :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resource obtained from a |
|
management server) contains an :ref:`AggregatedConfigSource |
|
<envoy_v3_api_msg_config.core.v3.AggregatedConfigSource>` message. |
|
|
|
In a gRPC client that uses xDS, only ADS is supported, and the bootstrap file contains the name of |
|
the ADS server, which will be used for all resources. The :ref:`ConfigSource |
|
<envoy_v3_api_msg_config.core.v3.ConfigSource>` messages in the :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and |
|
:ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resources must contain :ref:`AggregatedConfigSource |
|
<envoy_v3_api_msg_config.core.v3.AggregatedConfigSource>` messages. |
|
|
|
The xDS transport Protocol |
|
~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
Transport API version |
|
^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
In addition the resource type version described above, the xDS wire protocol has a |
|
transport version associated with it. This provides type versioning for messages such as |
|
:ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` and :ref:`DiscoveryResponse |
|
<envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`. It is also encoded in the gRPC method name, so a server |
|
can determine which version a client is speaking based on which method it calls. |
|
|
|
Basic Protocol Overview |
|
^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
Each xDS stream begins with a :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` from the |
|
client, which specifies the list of resources to subscribe to, the type URL corresponding to the |
|
subscribed resources, the node identifier, and an optional resource type instance version |
|
indicating the most recent version of the resource type that the client has already seen (see |
|
:ref:`ACK/NACK and resource type instance version <xds_ack_nack>` for details). |
|
|
|
The server will then send a :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` containing |
|
any resources that the client has subscribed to that have changed since the last resource type |
|
instance version that the client indicated it has seen. The server may send additional responses |
|
at any time when the subscribed resources change. |
|
|
|
Whenever the client receives a new response, it will send another request indicating whether or |
|
not the resources in the response were valid (see |
|
:ref:`ACK/NACK and resource type instance version <xds_ack_nack>` for details). |
|
|
|
All server responses will contain a :ref:`nonce<envoy_v3_api_field_service.discovery.v3.DiscoveryResponse.nonce>`, and |
|
all subsequent requests from the client must set the |
|
:ref:`response_nonce <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.response_nonce>` field to the most recent |
|
nonce received from the server on that stream. This allows servers to determine which response a |
|
given request is associated with, which avoids various race conditions in the SotW protocol |
|
variants. Note that the nonce is valid only in the context of an individual xDS stream; it does |
|
not survive stream restarts. |
|
|
|
Only the first request on a stream is guaranteed to carry the node identifier. |
|
The subsequent discovery requests on the same stream may carry an empty node |
|
identifier. This holds true regardless of the acceptance of the discovery |
|
responses on the same stream. The node identifier should always be identical if |
|
present more than once on the stream. It is sufficient to only check the first |
|
message for the node identifier as a result. |
|
|
|
.. _xds_ack_nack: |
|
|
|
ACK/NACK and resource type instance version |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
Every xDS resource type has a version string that indicates the version for that resource type. |
|
Whenever one resource of that type changes, the version is changed. |
|
|
|
In a response sent by the xDS server, the |
|
:ref:`version_info<envoy_v3_api_field_service.discovery.v3.DiscoveryResponse.version_info>` field indicates the current |
|
version for that resource type. The client then sends another request to the server with the |
|
:ref:`version_info<envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.version_info>` field indicating the most |
|
recent valid version seen by the client. This provides a way for the server to determine when |
|
it sends a version that the client considers invalid. |
|
|
|
(In the :ref:`incremental protocol variants <xds_protocol_delta>`, the resource type instance |
|
version is sent by the server in the |
|
:ref:`system_version_info<envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryResponse.system_version_info>` field. |
|
However, this information is not actually used by the client to communicate which resources are |
|
valid, because the incremental API variants have a separate mechanism for that.) |
|
|
|
The resource type instance version is separate for each resource type. When using the aggregated |
|
protocol variants, each resource type has its own version even though all resource types are being |
|
sent on the same stream. |
|
|
|
The resource type instance version is also separate for each xDS server (where an xDS server is |
|
identified by a unique :ref:`ConfigSource <envoy_v3_api_msg_config.core.v3.ConfigSource>`). When obtaining |
|
resources of a given type from multiple xDS servers, each xDS server will have a different notion |
|
of version. |
|
|
|
Note that the version for a resource type is not a property of an individual xDS stream but rather |
|
a property of the resources themselves. If the stream becomes broken and the client creates a new |
|
stream, the client's initial request on the new stream should indicate the most recent version |
|
seen by the client on the previous stream. Servers may decide to optimize by not resending |
|
resources that the client had already seen on the previous stream, but only if they know that the |
|
client is not subscribing to a new resource that it was not previously subscribed to. For example, |
|
it is generally safe for servers to do this optimization for wildcard LDS and CDS requests, and it |
|
is safe to do in environments where the clients will always subscribe to exactly the same set of |
|
resources. |
|
|
|
An example EDS request might be: |
|
|
|
.. code:: yaml |
|
|
|
version_info: |
|
node: { id: envoy } |
|
resource_names: |
|
- foo |
|
- bar |
|
type_url: type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment |
|
response_nonce: |
|
|
|
The management server may reply either immediately or when the requested |
|
resources are available with a :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`, e.g.: |
|
|
|
.. code:: yaml |
|
|
|
version_info: X |
|
resources: |
|
- foo ClusterLoadAssignment proto encoding |
|
- bar ClusterLoadAssignment proto encoding |
|
type_url: type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment |
|
nonce: A |
|
|
|
After processing the :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`, Envoy will send a new |
|
request on the stream, specifying the last version successfully applied |
|
and the nonce provided by the management server. The version provides Envoy and the |
|
management server a shared notion of the currently applied configuration, |
|
as well as a mechanism to ACK/NACK configuration updates. |
|
|
|
ACK |
|
^^^ |
|
|
|
If the update was successfully applied, the |
|
:ref:`version_info <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.version_info>` will be **X**, as indicated |
|
in the sequence diagram: |
|
|
|
.. figure:: diagrams/simple-ack.svg |
|
:alt: Version update after ACK |
|
|
|
NACK |
|
^^^^ |
|
|
|
If Envoy had instead rejected configuration |
|
update **X**, it would reply with :ref:`error_detail <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.error_detail>` |
|
populated and its previous version, which in this case was the empty |
|
initial version. The :ref:`error_detail <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.error_detail>` has |
|
more details around the exact error message populated in the message field: |
|
|
|
.. figure:: diagrams/simple-nack.svg |
|
:alt: No version update after NACK |
|
|
|
In the sequence diagrams, the following format is used to abbreviate messages: |
|
|
|
- *DiscoveryRequest*: (V=version_info,R=resource_names,N=response_nonce,T=type_url) |
|
- *DiscoveryResponse*: (V=version_info,R=resources,N=nonce,T=type_url) |
|
|
|
After a NACK, an API update may succeed at a new version **Y**: |
|
|
|
|
|
.. figure:: diagrams/later-ack.svg |
|
:alt: ACK after NACK |
|
|
|
The preferred mechanism for a server to detect a NACK is to look for the presence of the |
|
:ref:`error_detail <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.error_detail>` field in the request sent by |
|
the client. Some older servers may instead detect a NACK by looking at both the version and the |
|
nonce in the request: if the version in the request is not equal to the one sent by the server with |
|
that nonce, then the client has rejected the most recent version. However, this approach does not |
|
work for APIs other than LDS and CDS for clients that may dynamically change the set of resources |
|
that they are subscribing to, unless the server has somehow arranged to increment the resource |
|
type instance version every time any one client subscribes to a new resource. Specifically, |
|
consider the following example: |
|
|
|
.. figure:: diagrams/error-detail-nack.svg |
|
:alt: detecting NACK from error_detail instead of version and nonce |
|
|
|
ACK and NACK semantics summary |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
- The xDS client should ACK or NACK every :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` |
|
received from the management server. The :ref:`response_nonce |
|
<envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.response_nonce>` field tells the server which of its responses |
|
the ACK or NACK is associated with. |
|
- ACK signifies successful configuration update and contains the |
|
:ref:`version_info <envoy_v3_api_field_service.discovery.v3.DiscoveryResponse.version_info>` from the |
|
:ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`. |
|
- NACK signifies unsuccessful configuration and is indicated by the presence of the |
|
:ref:`error_detail <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.error_detail>` field. The :ref:`version_info |
|
<envoy_v3_api_field_service.discovery.v3.DiscoveryResponse.version_info>` indicates the most recent version that the |
|
client is using, although that may not be an older version in the case where the client has |
|
subscribed to a new resource from an existing version and that new resource is invalid (see |
|
example above). |
|
|
|
.. _xds_protocol_resource_update: |
|
|
|
When to send an update |
|
^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
The management server should only send updates to the Envoy client when |
|
the resources in the :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` have changed. Envoy replies |
|
to any :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` with a :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` containing the |
|
ACK/NACK immediately after it has been either accepted or rejected. If |
|
the management server provides the same set of resources rather than |
|
waiting for a change to occur, it will cause needless work on both the client and the management |
|
server, which could have a severe performance impact. |
|
|
|
Within a stream, new :ref:`DiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` supersede any prior |
|
:ref:`DiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` having the same resource type. This means that |
|
the management server only needs to respond to the latest |
|
:ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` on each stream for any given resource type. |
|
|
|
.. _xds_protocol_resource_hints: |
|
|
|
How the client specifies what resources to return |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
xDS requests allow the client to specify a set of resource names as a hint to the server about |
|
which resources the client is interested in. In the SotW protocol variants, this is done via the |
|
:ref:`resource_names <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.resource_names>` specified in the |
|
:ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>`; in the incremental protocol variants, |
|
this is done via the :ref:`resource_names_subscribe |
|
<envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_subscribe>` and |
|
:ref:`resource_names_unsubscribe |
|
<envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_unsubscribe>` fields in the |
|
:ref:`DeltaDiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>`. |
|
|
|
Normally (see below for exceptions), requests must specify the set of resource names that the |
|
client is interested in. The management server must supply the requested resources if they exist. |
|
The client will silently ignore any supplied resources that were not explicitly requested. When |
|
the client sends a new request that changes the set of resources being requested, the server must |
|
resend any newly requested resources, even if it previously sent those resources without having |
|
been asked for them and the resources have not changed since that time. If the list of resource |
|
names becomes empty, that means that the client is no longer interested in any resources of the |
|
specified type. |
|
|
|
For :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resource |
|
types, there is also a "wildcard" mode, which is triggered when the initial request on the stream |
|
for that resource type contains no resource names. In this case, the server should use |
|
site-specific business logic to determine the full set of resources that the client is interested |
|
in, typically based on the client's :ref:`node <envoy_v3_api_msg_config.core.v3.Node>` identification. Note |
|
that once a stream has entered wildcard mode for a given resource type, there is no way to change |
|
the stream out of wildcard mode; resource names specified in any subsequent request on the stream |
|
will be ignored. |
|
|
|
Client Behavior |
|
""""""""""""""" |
|
|
|
Envoy will always use wildcard mode for :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and |
|
:ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resources. However, other xDS clients (such as gRPC clients |
|
that use xDS) may specify explicit resource names for these resource types, for example if they |
|
only have a singleton listener and already know its name from some out-of-band configuration. |
|
|
|
Grouping Resources into Responses |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
In the incremental protocol variants, the server sends each resource in its own response. This |
|
means that if the server has previously sent 100 resources and only one of them has changed, it |
|
may send a response containing only the changed resource; it does not need to resend the 99 |
|
resources that have not changed, and the client must not delete the unchanged resources. |
|
|
|
In the SotW protocol variants, all resource types except for :ref:`Listener |
|
<envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` are grouped into responses |
|
in the same way as in the incremental protocol variants. However, |
|
:ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resource types |
|
are handled differently: the server must include the complete state of the world, meaning that all |
|
resources of the relevant type that are needed by the client must be included, even if they did |
|
not change since the last response. This means that if the server has previously sent 100 |
|
resources and only one of them has changed, it must resend all 100 of them, even the 99 that were |
|
not modified. |
|
|
|
Note that all of the protocol variants operate on units of whole named resources. There is |
|
no mechanism for providing incremental updates of repeated fields within a named resource. |
|
Most notably, there is currently no mechanism for incrementally updating individual |
|
endpoints within an EDS response. |
|
|
|
Duplicate Resource Names |
|
^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
It is an error for a server to send a single response that contains the same resource name |
|
twice. Clients should NACK responses that contain multiple instances of the same resource name. |
|
|
|
Deleting Resources |
|
^^^^^^^^^^^^^^^^^^ |
|
|
|
In the incremental proocol variants, the server signals the client that a resource should be |
|
deleted via the :ref:`removed_resources <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryResponse.removed_resources>` |
|
field of the response. This tells the client to remove the resource from its local cache. |
|
|
|
In the SotW protocol variants, the criteria for deleting resources is more complex. For |
|
:ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resource types, |
|
if a previously seen resource is not present in a new response, that indicates that the resource |
|
has been removed, and the client must delete it; a response containing no resources means to delete |
|
all resources of that type. However, for other resource types, the API provides no mechanism for |
|
the server to tell the client that resources have been deleted; instead, deletions are indicated |
|
implicitly by parent resources being changed to no longer refer to a child resource. For example, |
|
when the client receives an LDS update removing a :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` |
|
that was previously pointing to :ref:`RouteConfiguration <envoy_v3_api_msg_config.route.v3.RouteConfiguration>` A, |
|
if no other :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` is pointing to :ref:`RouteConfiguration |
|
<envoy_v3_api_msg_config.route.v3.RouteConfiguration>` A, then the client may delete A. For those resource types, |
|
an empty :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` is effectively a no-op |
|
from the client's perspective. |
|
|
|
.. _xds_protocol_resource_not_existed: |
|
|
|
Knowing When a Requested Resource Does Not Exist |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
The SotW protocol variants do not provide any explicit mechanism to determine when a requested |
|
resource does not exist. |
|
|
|
Responses for :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` |
|
resource types must include all resources requested by the client. However, it may not be possible |
|
for the client to know that a resource does not exist based solely on its absence in a response, |
|
because the delivery of the updates is eventually consistent: if the client initially sends a |
|
request for resource A, then sends a request for resources A and B, and then sees a response |
|
containing only resource A, the client cannot conclude that resource B does not exist, because |
|
the response may have been sent on the basis of the first request, before the server saw the |
|
second request. |
|
|
|
For other resource types, because each resource can be sent in its own response, there is no way |
|
to know from the next response whether the newly requested resource exists, because the next |
|
response could be an unrelated update for another resource that had already been subscribed to |
|
previously. |
|
|
|
As a result, clients are expected to use a timeout (recommended duration is 15 seconds) after |
|
sending a request for a new resource, after which they will consider the requested resource to |
|
not exist if they have not received the resource. In Envoy, this is done for |
|
:ref:`RouteConfiguration <envoy_v3_api_msg_config.route.v3.RouteConfiguration>` and :ref:`ClusterLoadAssignment |
|
<envoy_v3_api_msg_config.endpoint.v3.ClusterLoadAssignment>` resources during :ref:`resource warming |
|
<xds_protocol_resource_warming>`. |
|
|
|
Note that this timeout is not strictly necessary when using wildcard mode for :ref:`Listener |
|
<envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resource types, because |
|
in that case every response will contain all existing resources that are relevant to the |
|
client, so the client can know that a resource does not exist by its absence in the next |
|
response it sees. However, using a timeout is still recommended in this case, since it protects |
|
against the case where the management server fails to send a response in a timely manner. |
|
|
|
Note that even if a requested resource does not exist at the moment when the client requests it, |
|
that resource could be created at any time. Management servers must remember the set of resources |
|
being requested by the client, and if one of those resources springs into existence later, the |
|
server must send an update to the client informing it of the new resource. Clients that initially |
|
see a resource that does not exist must be prepared for the resource to be created at any time. |
|
|
|
Unsubscribing From Resources |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
In the incremental protocol variants, resources can be unsubscribed to via the |
|
:ref:`resource_names_unsubscribe |
|
<envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_unsubscribe>` field. |
|
|
|
In the SotW protocol variants, each request must contain the full list of resource names being |
|
subscribed to in the :ref:`resource_names <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.resource_names>` field, |
|
so unsubscribing to a set of resources is done by sending a new request containing all resource |
|
names that are still being subscribed to but not containing the resource names being unsubscribed |
|
to. For example, if the client had previously been subscribed to resources A and B but wishes to |
|
unsubscribe from B, it must send a new request containing only resource A. |
|
|
|
Note that for :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` |
|
resource types where the stream is in "wildcard" mode (see :ref:`How the client specifies what |
|
resources to return <xds_protocol_resource_hints>` for details), the set of resources being |
|
subscribed to is determined by the server instead of the client, so there is no mechanism |
|
for the client to unsubscribe from resources. |
|
|
|
Requesting Multiple Resources on a Single Stream |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
For EDS/RDS, Envoy may either generate a distinct stream for each |
|
resource of a given type (e.g. if each :ref:`ConfigSource <envoy_v3_api_msg_config.core.v3.ConfigSource>` has its own |
|
distinct upstream cluster for a management server), or may combine |
|
together multiple resource requests for a given resource type when they |
|
are destined for the same management server. While this is left to |
|
implementation specifics, management servers should be capable of |
|
handling one or more :ref:`resource_names <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.resource_names>` for a given resource type in |
|
each request. Both sequence diagrams below are valid for fetching two |
|
EDS resources ``{foo, bar}``: |
|
|
|
|Multiple EDS requests on the same stream| |Multiple EDS requests on |
|
distinct streams| |
|
|
|
Resource updates |
|
^^^^^^^^^^^^^^^^ |
|
|
|
As discussed above, Envoy may update the list of :ref:`resource_names <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.resource_names>` it |
|
presents to the management server in each :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` that |
|
ACK/NACKs a specific :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`. In addition, Envoy may later |
|
issue additional :ref:`DiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` at a given :ref:`version_info <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.version_info>` to |
|
update the management server with new resource hints. For example, if |
|
Envoy is at EDS version **X** and knows only about cluster ``foo``, but |
|
then receives a CDS update and learns about ``bar`` in addition, it may |
|
issue an additional :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` for **X** with ``{foo,bar}`` as |
|
``resource_names``. |
|
|
|
.. figure:: diagrams/cds-eds-resources.svg |
|
:alt: CDS response leads to EDS resource hint update |
|
|
|
There is a race condition that may arise here; if after a resource hint |
|
update is issued by Envoy at **X**, but before the management server |
|
processes the update it replies with a new version **Y**, the resource |
|
hint update may be interpreted as a rejection of **Y** by presenting an |
|
**X** :ref:`version_info <envoy_v3_api_field_service.discovery.v3.DiscoveryResponse.version_info>`. To avoid this, the management server provides a |
|
``nonce`` that Envoy uses to indicate the specific :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` |
|
each :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` corresponds to: |
|
|
|
.. figure:: diagrams/update-race.svg |
|
:alt: EDS update race motivates nonces |
|
|
|
The management server should not send a :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` for any |
|
:ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` that has a stale nonce. A nonce becomes stale |
|
following a newer nonce being presented to Envoy in a |
|
:ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`. A management server does not need to send an |
|
update until it determines a new version is available. Earlier requests |
|
at a version then also become stale. It may process multiple |
|
:ref:`DiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` at a version until a new version is ready. |
|
|
|
.. figure:: diagrams/stale-requests.svg |
|
:alt: Requests become stale |
|
|
|
An implication of the above resource update sequencing is that Envoy |
|
does not expect a :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` for every :ref:`DiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` |
|
it issues. |
|
|
|
.. _xds_protocol_resource_warming: |
|
|
|
Resource warming |
|
~~~~~~~~~~~~~~~~ |
|
|
|
:ref:`Clusters <arch_overview_cluster_warming>` and |
|
:ref:`Listeners <config_listeners_lds>` |
|
go through warming before they can serve requests. This process |
|
happens both during :ref:`Envoy initialization <arch_overview_initialization>` |
|
and when the ``Cluster`` or ``Listener`` is updated. Warming of |
|
``Cluster`` is completed only when a ``ClusterLoadAssignment`` response |
|
is supplied by management server. Similarly, warming of ``Listener`` is |
|
completed only when a ``RouteConfiguration`` is supplied by management |
|
server if the listener refers to an RDS configuration. Management server |
|
is expected to provide the EDS/RDS updates during warming. If management |
|
server does not provide EDS/RDS responses, Envoy will not initialize |
|
itself during the initialization phase and the updates sent via CDS/LDS |
|
will not take effect until EDS/RDS responses are supplied. |
|
|
|
.. _xds_protocol_eventual_consistency_considerations: |
|
|
|
Eventual consistency considerations |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
Since Envoy's xDS APIs are eventually consistent, traffic may drop |
|
briefly during updates. For example, if only cluster **X** is known via |
|
CDS/EDS, a ``RouteConfiguration`` references cluster **X** and is then |
|
adjusted to cluster **Y** just before the CDS/EDS update providing |
|
**Y**, traffic will be blackholed until **Y** is known about by the |
|
Envoy instance. |
|
|
|
For some applications, a temporary drop of traffic is acceptable, |
|
retries at the client or by other Envoy sidecars will hide this drop. |
|
For other scenarios where drop can't be tolerated, traffic drop could |
|
have been avoided by providing a CDS/EDS update with both **X** and |
|
**Y**, then the RDS update repointing from **X** to **Y** and then a |
|
CDS/EDS update dropping **X**. |
|
|
|
In general, to avoid traffic drop, sequencing of updates should follow a |
|
make before break model, wherein: |
|
|
|
- CDS updates (if any) must always be pushed first. |
|
- EDS updates (if any) must arrive after CDS updates for the respective clusters. |
|
- LDS updates must arrive after corresponding CDS/EDS updates. |
|
- RDS updates related to the newly added listeners must arrive after CDS/EDS/LDS updates. |
|
- VHDS updates (if any) related to the newly added RouteConfigurations must arrive after RDS updates. |
|
- Stale CDS clusters and related EDS endpoints (ones no longer being referenced) can then be removed. |
|
|
|
xDS updates can be pushed independently if no new |
|
clusters/routes/listeners are added or if it's acceptable to temporarily |
|
drop traffic during updates. Note that in case of LDS updates, the |
|
listeners will be warmed before they receive traffic, i.e. the dependent |
|
routes are fetched through RDS if configured. Clusters are warmed when |
|
adding/removing/updating clusters. On the other hand, routes are not |
|
warmed, i.e., the management plane must ensure that clusters referenced |
|
by a route are in place, before pushing the updates for a route. |
|
|
|
.. _xds_protocol_TTL: |
|
|
|
TTL |
|
~~~ |
|
|
|
In the event that the management server becomes unreachable, the last known configuration received |
|
by Envoy will persist until the connection is reestablished. For some services, this may not be |
|
desirable. For example, in the case of a fault injection service, a management server crash at the |
|
wrong time may leave Envoy in an undesirable state. The TTL setting allows Envoy to remove a set of |
|
resources after a specified period of time if contact with the management server is lost. This can |
|
be used, for example, to terminate a fault injection test when the management server can no longer |
|
be reached. |
|
|
|
For clients that support the *xds.config.supports-resource-ttl* client feature, A TTL field may |
|
be specified on each :ref:`Resource <envoy_v3_api_msg_service.discovery.v3.Resource>`. Each resource will have its own TTL |
|
expiry time, at which point the resource will be expired. Each xDS type may have different ways of |
|
handling such an expiry. |
|
|
|
To update the TTL associated with a *Resource*, the management server resends the resource with a |
|
new TTL. To remove the TTL, the management server resends the resource with the TTL field unset. |
|
|
|
To allow for lightweight TTL updates ("heartbeats"), a response can be sent that provides a |
|
:ref:`Resource <envoy_v3_api_msg_service.discovery.v3.Resource>` with the :ref:`resource <envoy_v3_api_msg_service.discovery.v3.Resource>` |
|
unset and version matching the most recently sent version can be used to update the TTL. These |
|
resources will not be treated as resource updates, but only as TTL updates. |
|
|
|
SotW TTL |
|
^^^^^^^^ |
|
|
|
In order to use TTL with SotW xDS, the relevant resources must be wrapped in a |
|
:ref:`Resource <envoy_v3_api_msg_service.discovery.v3.Resource>`. This allows setting the same TTL field that is used for |
|
Delta xDS with SotW, without changing the SotW API. Heartbeats are supported for SotW as well: |
|
any resource within the response that look like a heartbeat resource will only be used to update the TTL. |
|
|
|
This feature is gated by the *xds.config.supports-resource-in-sotw* client feature. |
|
|
|
.. _xds_protocol_ads: |
|
|
|
Aggregated Discovery Service |
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
It's challenging to provide the above guarantees on sequencing to avoid |
|
traffic drop when management servers are distributed. ADS allow a single |
|
management server, via a single gRPC stream, to deliver all API updates. |
|
This provides the ability to carefully sequence updates to avoid traffic |
|
drop. With ADS, a single stream is used with multiple independent |
|
:ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>`/:ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` sequences multiplexed via the |
|
type URL. For any given type URL, the above sequencing of |
|
:ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` and :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` messages applies. An |
|
example update sequence might look like: |
|
|
|
.. figure:: diagrams/ads.svg |
|
:alt: EDS/CDS multiplexed on an ADS stream |
|
|
|
A single ADS stream is available per Envoy instance. |
|
|
|
An example minimal ``bootstrap.yaml`` fragment for ADS configuration is: |
|
|
|
.. literalinclude:: ../_include/ads.yaml |
|
|
|
.. _xds_protocol_delta: |
|
|
|
Incremental xDS |
|
~~~~~~~~~~~~~~~ |
|
|
|
Incremental xDS is a separate xDS endpoint that: |
|
|
|
- Allows the protocol to communicate on the wire in terms of |
|
resource/resource name deltas ("Delta xDS"). This supports the goal |
|
of scalability of xDS resources. Rather than deliver all 100k |
|
clusters when a single cluster is modified, the management server |
|
only needs to deliver the single cluster that changed. |
|
- Allows the Envoy to on-demand / lazily request additional resources. |
|
For example, requesting a cluster only when a request for that |
|
cluster arrives. |
|
|
|
An Incremental xDS session is always in the context of a gRPC |
|
bidirectional stream. This allows the xDS server to keep track of the |
|
state of xDS clients connected to it. There is no REST version of |
|
Incremental xDS yet. |
|
|
|
In the delta xDS wire protocol, the nonce field is required and used to |
|
pair a :ref:`DeltaDiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryResponse>` |
|
to a :ref:`DeltaDiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>` |
|
ACK or NACK. Optionally, a response message level :ref:`system_version_info <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryResponse.system_version_info>` |
|
is present for debugging purposes only. |
|
|
|
:ref:`DeltaDiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>` can be sent in the following situations: |
|
|
|
- Initial message in a xDS bidirectional gRPC stream. |
|
- As an ACK or NACK response to a previous :ref:`DeltaDiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryResponse>`. In this case the :ref:`response_nonce <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.response_nonce>` is set to the nonce value in the Response. ACK or NACK is determined by the absence or presence of :ref:`error_detail <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.error_detail>`. |
|
- Spontaneous :ref:`DeltaDiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>` from the client. This can be done to dynamically add or remove elements from the tracked :ref:`resource_names <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.resource_names>` set. In this case :ref:`response_nonce <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.response_nonce>` must be omitted. |
|
|
|
In this first example the client connects and receives a first update |
|
that it ACKs. The second update fails and the client NACKs the update. |
|
Later the xDS client spontaneously requests the "wc" resource. |
|
|
|
.. figure:: diagrams/incremental.svg |
|
:alt: Incremental session example |
|
|
|
On reconnect the Incremental xDS client may tell the server of its known |
|
resources to avoid resending them over the network by sending them in |
|
:ref:`initial_resource_versions <envoy_v3_api_field_service.discovery.v3.deltadiscoveryrequest.initial_resource_versions>`. |
|
Because no state is assumed to be preserved from the previous stream, the reconnecting |
|
client must provide the server with all resource names it is interested in. Note that for wildcard |
|
requests (CDS/LDS/SRDS), the request must have no resources in both |
|
:ref:`resource_names_subscribe <envoy_v3_api_field_service.discovery.v3.deltadiscoveryrequest.resource_names_subscribe>` and |
|
:ref:`resource_names_unsubscribe <envoy_v3_api_field_service.discovery.v3.deltadiscoveryrequest.resource_names_unsubscribe>`. |
|
|
|
.. figure:: diagrams/incremental-reconnect.svg |
|
:alt: Incremental reconnect example |
|
|
|
Resource names |
|
^^^^^^^^^^^^^^ |
|
|
|
Resources are identified by a resource name or an alias. Aliases of a |
|
resource, if present, can be identified by the alias field in the |
|
resource of a :ref:`DeltaDiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryResponse>`. The resource name will be |
|
returned in the name field in the resource of a |
|
:ref:`DeltaDiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryResponse>`. |
|
|
|
.. _xds_protocol_delta_subscribe: |
|
|
|
Subscribing to Resources |
|
^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
The client can send either an alias or the name of a resource in the |
|
:ref:`resource_names_subscribe <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_subscribe>` field of a :ref:`DeltaDiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>` in |
|
order to subscribe to a resource. Both the names and aliases of |
|
resources should be checked in order to determine whether the entity in |
|
question has been subscribed to. |
|
|
|
A :ref:`resource_names_subscribe <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_subscribe>` field may contain resource names that the |
|
server believes the client is already subscribed to, and furthermore has |
|
the most recent versions of. However, the server *must* still provide |
|
those resources in the response; due to implementation details hidden |
|
from the server, the client may have "forgotten" those resources despite |
|
apparently remaining subscribed. |
|
|
|
.. _xds_protocol_unsubscribe: |
|
|
|
Unsubscribing from Resources |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
When a client loses interest in some resources, it will indicate that |
|
with the :ref:`resource_names_unsubscribe <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_unsubscribe>` field of a |
|
:ref:`DeltaDiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>`. As with :ref:`resource_names_subscribe <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_subscribe>`, these |
|
may be resource names or aliases. |
|
|
|
A :ref:`resource_names_unsubscribe <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_unsubscribe>` field may contain superfluous resource |
|
names, which the server thought the client was already not subscribed |
|
to. The server must cleanly process such a request; it can simply ignore |
|
these phantom unsubscriptions. |
|
|
|
Knowing When a Requested Resource Does Not Exist |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
When a resource subscribed to by a client does not exist, the server will send a :ref:`Resource |
|
<envoy_v3_api_msg_service.discovery.v3.Resource>` whose :ref:`name <envoy_v3_api_field_service.discovery.v3.Resource.name>` field matches the |
|
name that the client subscribed to and whose :ref:`resource <envoy_v3_api_msg_service.discovery.v3.Resource>` |
|
field is unset. This allows the client to quickly determine when a resource does not exist without |
|
waiting for a timeout, as would be done in the SotW protocol variants. However, clients are still |
|
encouraged to use a timeout to protect against the case where the management server fails to send |
|
a response in a timely manner. |
|
|
|
REST-JSON polling subscriptions |
|
------------------------------- |
|
|
|
Synchronous (long) polling via REST endpoints is also available for the |
|
xDS singleton APIs. The above sequencing of messages is similar, except |
|
no persistent stream is maintained to the management server. It is |
|
expected that there is only a single outstanding request at any point in |
|
time, and as a result the response nonce is optional in REST-JSON. The |
|
`JSON canonical transform of |
|
proto3 <https://developers.google.com/protocol-buffers/docs/proto3#json>`__ |
|
is used to encode :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` and :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` |
|
messages. ADS is not available for REST-JSON polling. |
|
|
|
When the poll period is set to a small value, with the intention of long |
|
polling, then there is also a requirement to avoid sending a |
|
:ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` unless a change to the underlying resources has |
|
occurred via a :ref:`resource update <xds_protocol_resource_update>`. |
|
|
|
.. |Multiple EDS requests on the same stream| image:: diagrams/eds-same-stream.svg |
|
.. |Multiple EDS requests on distinct streams| image:: diagrams/eds-distinct-stream.svg
|
|
|