* api: Update xds_protocol doc to remove v2 and cleanup
Signed-off-by: Ryan Northey <ryan@synca.io>
Mirrored from https://github.com/envoyproxy/envoy @ 7cd615c262dc1b871812209e37c15a49daea5173
- NACK signifies unsuccessful configuration and is indicated by the presence of the
:ref:`error_detail <envoy_api_field_DiscoveryRequest.error_detail>` field. The :ref:`version_info
<envoy_api_field_DiscoveryResponse.version_info>` indicates the most recent version that the
:ref:`error_detail <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.error_detail>` field. The :ref:`version_info
<envoy_v3_api_field_service.discovery.v3.DiscoveryResponse.version_info>` indicates the most recent version that the
client is using, although that may not be an older version in the case where the client has
subscribed to a new resource from an existing version and that new resource is invalid (see
example above).
@ -401,17 +390,17 @@ When to send an update
^^^^^^^^^^^^^^^^^^^^^^
The management server should only send updates to the Envoy client when
the resources in the :ref:`DiscoveryResponse <envoy_api_msg_DiscoveryResponse>` have changed. Envoy replies
to any :ref:`DiscoveryResponse <envoy_api_msg_DiscoveryResponse>` with a :ref:`DiscoveryRequest <envoy_api_msg_DiscoveryRequest>` containing the
the resources in the :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` have changed. Envoy replies
to any :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` with a :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` containing the
ACK/NACK immediately after it has been either accepted or rejected. If
the management server provides the same set of resources rather than
waiting for a change to occur, it will cause needless work on both the client and the management
server, which could have a severe performance impact.
Within a stream, new :ref:`DiscoveryRequests <envoy_api_msg_DiscoveryRequest>` supersede any prior
:ref:`DiscoveryRequests <envoy_api_msg_DiscoveryRequest>` having the same resource type. This means that
Within a stream, new :ref:`DiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` supersede any prior
:ref:`DiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` having the same resource type. This means that
the management server only needs to respond to the latest
:ref:`DiscoveryRequest <envoy_api_msg_DiscoveryRequest>` on each stream for any given resource type.
:ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` on each stream for any given resource type.
.._xds_protocol_resource_hints:
@ -420,13 +409,13 @@ How the client specifies what resources to return
xDS requests allow the client to specify a set of resource names as a hint to the server about
which resources the client is interested in. In the SotW protocol variants, this is done via the
:ref:`resource_names <envoy_api_field_DiscoveryRequest.resource_names>` specified in the
:ref:`DiscoveryRequest <envoy_api_msg_DiscoveryRequest>`; in the incremental protocol variants,
:ref:`resource_names <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.resource_names>` specified in the
:ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>`; in the incremental protocol variants,
this is done via the :ref:`resource_names_subscribe
<envoy_api_field_DeltaDiscoveryRequest.resource_names_subscribe>` and
<envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_subscribe>` and
:ref:`resource_names_unsubscribe
<envoy_api_field_DeltaDiscoveryRequest.resource_names_unsubscribe>` fields in the
In the SotW protocol variants, each request must contain the full list of resource names being
subscribed to in the :ref:`resource_names <envoy_api_field_DiscoveryRequest.resource_names>` field,
subscribed to in the :ref:`resource_names <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.resource_names>` field,
so unsubscribing to a set of resources is done by sending a new request containing all resource
names that are still being subscribed to but not containing the resource names being unsubscribed
to. For example, if the client had previously been subscribed to resources A and B but wishes to
unsubscribe from B, it must send a new request containing only resource A.
Note that for :ref:`Listener <envoy_api_msg_Listener>` and :ref:`Cluster <envoy_api_msg_Cluster>`
Note that for :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>`
resource types where the stream is in "wildcard" mode (see :ref:`How the client specifies what
resources to return <xds_protocol_resource_hints>` for details), the set of resources being
subscribed to is determined by the server instead of the client, so there is no mechanism
@ -570,14 +559,14 @@ Requesting Multiple Resources on a Single Stream
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For EDS/RDS, Envoy may either generate a distinct stream for each
resource of a given type (e.g. if each :ref:`ConfigSource <envoy_api_msg_core.ConfigSource>` has its own
resource of a given type (e.g. if each :ref:`ConfigSource <envoy_v3_api_msg_config.core.v3.ConfigSource>` has its own
distinct upstream cluster for a management server), or may combine
together multiple resource requests for a given resource type when they
are destined for the same management server. While this is left to
implementation specifics, management servers should be capable of
handling one or more :ref:`resource_names <envoy_api_field_DiscoveryRequest.resource_names>` for a given resource type in
handling one or more :ref:`resource_names <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.resource_names>` for a given resource type in
each request. Both sequence diagrams below are valid for fetching two
EDS resources `{foo, bar}`:
EDS resources ``{foo, bar}``:
|Multiple EDS requests on the same stream| |Multiple EDS requests on
distinct streams|
@ -585,15 +574,15 @@ distinct streams|
Resource updates
^^^^^^^^^^^^^^^^
As discussed above, Envoy may update the list of :ref:`resource_names <envoy_api_field_DiscoveryRequest.resource_names>` it
presents to the management server in each :ref:`DiscoveryRequest <envoy_api_msg_DiscoveryRequest>` that
ACK/NACKs a specific :ref:`DiscoveryResponse <envoy_api_msg_DiscoveryResponse>`. In addition, Envoy may later
issue additional :ref:`DiscoveryRequests <envoy_api_msg_DiscoveryRequest>` at a given :ref:`version_info <envoy_api_field_DiscoveryRequest.version_info>` to
As discussed above, Envoy may update the list of :ref:`resource_names <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.resource_names>` it
presents to the management server in each :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` that
ACK/NACKs a specific :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`. In addition, Envoy may later
issue additional :ref:`DiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` at a given :ref:`version_info <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.version_info>` to
update the management server with new resource hints. For example, if
Envoy is at EDS version **X** and knows only about cluster ``foo``, but
then receives a CDS update and learns about ``bar`` in addition, it may
issue an additional :ref:`DiscoveryRequest <envoy_api_msg_DiscoveryRequest>` for **X** with `{foo,bar}` as
`resource_names`.
issue an additional :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` for **X** with ``{foo,bar}`` as
``resource_names``.
..figure:: diagrams/cds-eds-resources.svg
:alt:CDS response leads to EDS resource hint update
@ -602,26 +591,26 @@ There is a race condition that may arise here; if after a resource hint
update is issued by Envoy at **X**, but before the management server
processes the update it replies with a new version **Y**, the resource
hint update may be interpreted as a rejection of **Y** by presenting an
**X**:ref:`version_info <envoy_api_field_DiscoveryResponse.version_info>`. To avoid this, the management server provides a
``nonce`` that Envoy uses to indicate the specific :ref:`DiscoveryResponse <envoy_api_msg_DiscoveryResponse>`
each :ref:`DiscoveryRequest <envoy_api_msg_DiscoveryRequest>` corresponds to:
**X**:ref:`version_info <envoy_v3_api_field_service.discovery.v3.DiscoveryResponse.version_info>`. To avoid this, the management server provides a
``nonce`` that Envoy uses to indicate the specific :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`
each :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` corresponds to:
..figure:: diagrams/update-race.svg
:alt:EDS update race motivates nonces
The management server should not send a :ref:`DiscoveryResponse <envoy_api_msg_DiscoveryResponse>` for any
:ref:`DiscoveryRequest <envoy_api_msg_DiscoveryRequest>` that has a stale nonce. A nonce becomes stale
The management server should not send a :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` for any
:ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` that has a stale nonce. A nonce becomes stale
following a newer nonce being presented to Envoy in a
:ref:`DiscoveryResponse <envoy_api_msg_DiscoveryResponse>`. A management server does not need to send an
:ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`. A management server does not need to send an
update until it determines a new version is available. Earlier requests
at a version then also become stale. It may process multiple
:ref:`DiscoveryRequests <envoy_api_msg_DiscoveryRequest>` at a version until a new version is ready.
:ref:`DiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` at a version until a new version is ready.
..figure:: diagrams/stale-requests.svg
:alt:Requests become stale
An implication of the above resource update sequencing is that Envoy
does not expect a :ref:`DiscoveryResponse <envoy_api_msg_DiscoveryResponse>` for every :ref:`DiscoveryRequests <envoy_api_msg_DiscoveryRequest>`
does not expect a :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` for every :ref:`DiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>`
it issues.
.._xds_protocol_resource_warming:
@ -633,10 +622,10 @@ Resource warming
:ref:`Listeners <config_listeners_lds>`
go through warming before they can serve requests. This process
happens both during :ref:`Envoy initialization <arch_overview_initialization>`
and when the `Cluster` or `Listener` is updated. Warming of
`Cluster` is completed only when a `ClusterLoadAssignment` response
is supplied by management server. Similarly, warming of `Listener` is
completed only when a `RouteConfiguration` is supplied by management
and when the ``Cluster`` or ``Listener`` is updated. Warming of
``Cluster`` is completed only when a ``ClusterLoadAssignment`` response
is supplied by management server. Similarly, warming of ``Listener`` is
completed only when a ``RouteConfiguration`` is supplied by management
server if the listener refers to an RDS configuration. Management server
is expected to provide the EDS/RDS updates during warming. If management
server does not provide EDS/RDS responses, Envoy will not initialize
Since Envoy's xDS APIs are eventually consistent, traffic may drop
briefly during updates. For example, if only cluster **X** is known via
CDS/EDS, a `RouteConfiguration` references cluster **X** and is then
CDS/EDS, a ``RouteConfiguration`` references cluster **X** and is then
adjusted to cluster **Y** just before the CDS/EDS update providing
**Y**, traffic will be blackholed until **Y** is known about by the
Envoy instance.
@ -695,7 +684,7 @@ be used, for example, to terminate a fault injection test when the management se
be reached.
For clients that support the *xds.config.supports-resource-ttl* client feature, A TTL field may
be specified on each :ref:`Resource <envoy_api_msg_Resource>`. Each resource will have its own TTL
be specified on each :ref:`Resource <envoy_v3_api_msg_service.discovery.v3.Resource>`. Each resource will have its own TTL
expiry time, at which point the resource will be expired. Each xDS type may have different ways of
handling such an expiry.
@ -703,7 +692,7 @@ To update the TTL associated with a *Resource*, the management server resends th
new TTL. To remove the TTL, the management server resends the resource with the TTL field unset.
To allow for lightweight TTL updates ("heartbeats"), a response can be sent that provides a
:ref:`Resource <envoy_api_msg_Resource>` with the :ref:`resource <envoy_api_field_Resource.resource>`
:ref:`Resource <envoy_v3_api_msg_service.discovery.v3.Resource>` with the :ref:`resource <envoy_v3_api_msg_service.discovery.v3.Resource>`
unset and version matching the most recently sent version can be used to update the TTL. These
resources will not be treated as resource updates, but only as TTL updates.
@ -711,7 +700,7 @@ SotW TTL
^^^^^^^^
In order to use TTL with SotW xDS, the relevant resources must be wrapped in a
:ref:`Resource <envoy_api_msg_Resource>`. This allows setting the same TTL field that is used for
:ref:`Resource <envoy_v3_api_msg_service.discovery.v3.Resource>`. This allows setting the same TTL field that is used for
Delta xDS with SotW, without changing the SotW API. Heartbeats are supported for SotW as well:
any resource within the response that look like a heartbeat resource will only be used to update the TTL.
@ -727,9 +716,9 @@ traffic drop when management servers are distributed. ADS allow a single
management server, via a single gRPC stream, to deliver all API updates.
This provides the ability to carefully sequence updates to avoid traffic
drop. With ADS, a single stream is used with multiple independent
:ref:`DiscoveryRequest <envoy_api_msg_DiscoveryRequest>`/:ref:`DiscoveryResponse <envoy_api_msg_DiscoveryResponse>` sequences multiplexed via the
:ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>`/:ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` sequences multiplexed via the
type URL. For any given type URL, the above sequencing of
:ref:`DiscoveryRequest <envoy_api_msg_DiscoveryRequest>` and :ref:`DiscoveryResponse <envoy_api_msg_DiscoveryResponse>` messages applies. An
:ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` and :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` messages applies. An
example update sequence might look like:
..figure:: diagrams/ads.svg
@ -739,42 +728,7 @@ A single ADS stream is available per Envoy instance.
An example minimal ``bootstrap.yaml`` fragment for ADS configuration is:
..code:: yaml
node:
id: <node identifier>
dynamic_resources:
cds_config: {ads: {}}
lds_config: {ads: {}}
ads_config:
api_type: GRPC
grpc_services:
envoy_grpc:
cluster_name: ads_cluster
static_resources:
clusters:
- name: ads_cluster
connect_timeout: { seconds: 5 }
type: STATIC
hosts:
- socket_address:
address: <ADS management server IP address>
port_value: <ADS management server port>
lb_policy: ROUND_ROBIN
# It is recommended to configure either HTTP/2 or TCP keepalives in order to detect
# connection issues, and allow Envoy to reconnect. TCP keepalive is less expensive, but
# may be inadequate if there is a TCP proxy between Envoy and the management server.
# HTTP/2 keepalive is slightly more expensive, but may detect issues through more types
# of intermediate proxies.
http2_protocol_options:
connection_keepalive:
interval: 30s
timeout: 5s
upstream_connection_options:
tcp_keepalive:
...
admin:
...
..literalinclude:: ../_include/ads.yaml
.._xds_protocol_delta:
@ -798,16 +752,16 @@ state of xDS clients connected to it. There is no REST version of
Incremental xDS yet.
In the delta xDS wire protocol, the nonce field is required and used to
pair a :ref:`DeltaDiscoveryResponse <envoy_api_msg_DeltaDiscoveryResponse>`
to a :ref:`DeltaDiscoveryRequest <envoy_api_msg_DeltaDiscoveryRequest>`
ACK or NACK. Optionally, a response message level :ref:`system_version_info <envoy_api_field_DeltaDiscoveryResponse.system_version_info>`
pair a :ref:`DeltaDiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryResponse>`
to a :ref:`DeltaDiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>`
ACK or NACK. Optionally, a response message level :ref:`system_version_info <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryResponse.system_version_info>`
is present for debugging purposes only.
:ref:`DeltaDiscoveryRequest <envoy_api_msg_DeltaDiscoveryRequest>` can be sent in the following situations:
:ref:`DeltaDiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>` can be sent in the following situations:
- Initial message in a xDS bidirectional gRPC stream.
- As an ACK or NACK response to a previous :ref:`DeltaDiscoveryResponse <envoy_api_msg_DeltaDiscoveryResponse>`. In this case the :ref:`response_nonce <envoy_api_field_DiscoveryRequest.response_nonce>` is set to the nonce value in the Response. ACK or NACK is determined by the absence or presence of :ref:`error_detail <envoy_api_field_DiscoveryRequest.error_detail>`.
- Spontaneous :ref:`DeltaDiscoveryRequests <envoy_api_msg_DeltaDiscoveryRequest>` from the client. This can be done to dynamically add or remove elements from the tracked :ref:`resource_names <envoy_api_field_DiscoveryRequest.resource_names>` set. In this case :ref:`response_nonce <envoy_api_field_DiscoveryRequest.response_nonce>` must be omitted.
- As an ACK or NACK response to a previous :ref:`DeltaDiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryResponse>`. In this case the :ref:`response_nonce <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.response_nonce>` is set to the nonce value in the Response. ACK or NACK is determined by the absence or presence of :ref:`error_detail <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.error_detail>`.
- Spontaneous :ref:`DeltaDiscoveryRequests <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>` from the client. This can be done to dynamically add or remove elements from the tracked :ref:`resource_names <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.resource_names>` set. In this case :ref:`response_nonce <envoy_v3_api_field_service.discovery.v3.DiscoveryRequest.response_nonce>` must be omitted.
In this first example the client connects and receives a first update
that it ACKs. The second update fails and the client NACKs the update.
@ -830,9 +784,9 @@ Resource names
Resources are identified by a resource name or an alias. Aliases of a
resource, if present, can be identified by the alias field in the
resource of a :ref:`DeltaDiscoveryResponse <envoy_api_msg_DeltaDiscoveryResponse>`. The resource name will be
resource of a :ref:`DeltaDiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryResponse>`. The resource name will be
The client can send either an alias or the name of a resource in the
:ref:`resource_names_subscribe <envoy_api_field_DeltaDiscoveryRequest.resource_names_subscribe>` field of a :ref:`DeltaDiscoveryRequest <envoy_api_msg_DeltaDiscoveryRequest>` in
:ref:`resource_names_subscribe <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_subscribe>` field of a :ref:`DeltaDiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>` in
order to subscribe to a resource. Both the names and aliases of
resources should be checked in order to determine whether the entity in
question has been subscribed to.
A :ref:`resource_names_subscribe <envoy_api_field_DeltaDiscoveryRequest.resource_names_subscribe>` field may contain resource names that the
A :ref:`resource_names_subscribe <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_subscribe>` field may contain resource names that the
server believes the client is already subscribed to, and furthermore has
the most recent versions of. However, the server *must* still provide
those resources in the response; due to implementation details hidden
@ -858,11 +812,11 @@ Unsubscribing from Resources
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When a client loses interest in some resources, it will indicate that
with the :ref:`resource_names_unsubscribe <envoy_api_field_DeltaDiscoveryRequest.resource_names_unsubscribe>` field of a
:ref:`DeltaDiscoveryRequest <envoy_api_msg_DeltaDiscoveryRequest>`. As with :ref:`resource_names_subscribe <envoy_api_field_DeltaDiscoveryRequest.resource_names_subscribe>`, these
with the :ref:`resource_names_unsubscribe <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_unsubscribe>` field of a
:ref:`DeltaDiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DeltaDiscoveryRequest>`. As with :ref:`resource_names_subscribe <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_subscribe>`, these
may be resource names or aliases.
A :ref:`resource_names_unsubscribe <envoy_api_field_DeltaDiscoveryRequest.resource_names_unsubscribe>` field may contain superfluous resource
A :ref:`resource_names_unsubscribe <envoy_v3_api_field_service.discovery.v3.DeltaDiscoveryRequest.resource_names_unsubscribe>` field may contain superfluous resource
names, which the server thought the client was already not subscribed
to. The server must cleanly process such a request; it can simply ignore
these phantom unsubscriptions.
@ -871,8 +825,8 @@ Knowing When a Requested Resource Does Not Exist
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When a resource subscribed to by a client does not exist, the server will send a :ref:`Resource
<envoy_api_msg_Resource>` whose :ref:`name <envoy_api_field_Resource.name>` field matches the
name that the client subscribed to and whose :ref:`resource <envoy_api_field_Resource.resource>`
<envoy_v3_api_msg_service.discovery.v3.Resource>` whose :ref:`name <envoy_v3_api_field_service.discovery.v3.Resource.name>` field matches the
name that the client subscribed to and whose :ref:`resource <envoy_v3_api_msg_service.discovery.v3.Resource>`
field is unset. This allows the client to quickly determine when a resource does not exist without
waiting for a timeout, as would be done in the SotW protocol variants. However, clients are still
encouraged to use a timeout to protect against the case where the management server fails to send
@ -888,12 +842,12 @@ expected that there is only a single outstanding request at any point in
time, and as a result the response nonce is optional in REST-JSON. The
is used to encode :ref:`DiscoveryRequest <envoy_api_msg_DiscoveryRequest>` and :ref:`DiscoveryResponse <envoy_api_msg_DiscoveryResponse>`
is used to encode :ref:`DiscoveryRequest <envoy_v3_api_msg_service.discovery.v3.DiscoveryRequest>` and :ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>`
messages. ADS is not available for REST-JSON polling.
When the poll period is set to a small value, with the intention of long
polling, then there is also a requirement to avoid sending a
:ref:`DiscoveryResponse <envoy_api_msg_DiscoveryResponse>` unless a change to the underlying resources has
:ref:`DiscoveryResponse <envoy_v3_api_msg_service.discovery.v3.DiscoveryResponse>` unless a change to the underlying resources has
occurred via a :ref:`resource update <xds_protocol_resource_update>`.
..|Multiple EDS requests on the same stream|image:: diagrams/eds-same-stream.svg
data-plane-api(Azure Pipelines)
4 years ago