ads: aggregated discovery service. (#124)

See README.md for motivation and usage.

README.md

@@ -50,7 +50,11 @@ closed issue should also be included.
 
 * xDS APIs should support eventual consistency. For example, if RDS references a
   cluster that has not yet been supplied by CDS, it should be silently ignored
-  and traffic not forwarded until the CDS update occurs.
+  and traffic not forwarded until the CDS update occurs. Stronger consistency
+  guarantees are possible if the management server is able to sequence the xDS
+  APIs carefully (for example by using the ADS API below). By following the
+  `[CDS, EDS, LDS, RDS]` sequence for all pertinent resources, it will be
+  possible to avoid traffic outages during configuration update.
 
 * The API is primarily intended for machine generation and consumption. It is
   expected that the management server is responsible for mapping higher level
@@ -59,6 +63,11 @@ closed issue should also be included.
   used to generate xDS configuration are beyond the scope of the definitions in
   this repository.
 
+* REST-JSON API equivalents will be provided for the basic singleton xDS
+  subscription services CDS/RDS/RDS/LDS/RLDS. Advanced APIs such as HDS, ADS and
+  EDS multi-dimensional LB will be gRPC only. This avoids having to map
+  complicated bidirectional stream semantics onto REST.
+
 * Listeners will be immutable. Any updates to a listener via LDS will require
   the draining of existing connections for the specific bound IP/port. As a
   result, new requests will only be guaranteed to observe the new configuration
@@ -92,6 +101,34 @@ Unless otherwise stated, the APIs with the same names as v1 APIs have a similar
 * Rate Limit Discovery Service (RLDS). This is the same as RLS in v1.
 * [Route Discovery Service (RDS)](api/rds.proto).
 
+In addition to the above APIs, an aggregation API will be provided to allow for
+fine grained control over the sequencing of API updates across discovery
+services:
+
+* [Aggregated Discovery Service (ADS)](api/ads.proto). While fundamentally Envoy
+  employs an eventual consistency model, ADS provides an opportunity to sequence
+  API update pushes and ensure affinity of a single management server for an
+  Envoy node for API updates. ADS allows one or more APIs to be delivered on a
+  single gRPC bidi stream by the management server, and within an API to have all
+  resources aggregated onto a single stream. Without this, some APIs such as RDS
+  and EDS may require the management of multiple streams and connections to
+  distinct management servers.
+
+  ADS will allow for hitless updates of configuration by appropriate sequencing.
+  For example, suppose *foo.com* was mappped to cluster *X*. We wish to change
+  the mapping in the route table to point *foo.com* at cluster *Y*. In order to
+  do this, a CDS/EDS update must first be delivered containing both clusters *X*
+  and *Y*.
+
+  Without ADS, the CDS/EDS/RDS streams may point at distinct management servers,
+  or when on the same management server at distinct gRPC streams/connections
+  that require coordination. The EDS resource requests may be split across two
+  distinct streams, one for *X* and one for *Y*. ADS allows these to be
+  coalesced to a single stream to a single management server, avoiding the need
+  for distributed synchronization to correctly sequence the update. With ADS,
+  the management server would deliver the CDS, EDS and then RDS updates on a
+  single stream.
+
 ## Terminology
 
 Some relevant [existing terminology](https://lyft.github.io/envoy/docs/intro/arch_overview/terminology.html) is

api/BUILD

@@ -39,6 +39,19 @@ api_proto_library(
     srcs = ["tls_context.proto"],
 )
 
+api_proto_library(
+    name = "ads",
+    srcs = ["ads.proto"],
+    has_services = 1,
+    deps = [
+        ":base",
+        ":cds",
+        ":eds",
+        ":lds",
+        ":rds",
+    ],
+)
+
 api_proto_library(
     name = "cds",
     srcs = ["cds.proto"],

api/ads.proto

@@ -0,0 +1,24 @@
+syntax = "proto3";
+
+package envoy.api.v2;
+
+import "api/base.proto";
+import "api/cds.proto";
+import "api/eds.proto";
+import "api/lds.proto";
+import "api/rds.proto";
+
+import "google/api/annotations.proto";
+
+// See https://github.com/lyft/envoy-api#apis for a description of the role of
+// ADS and how it is intended to be used by a management server. ADS requests
+// have the same structure as their singleton xDS counterparts, but can
+// multiplex many resource types on a single stream. The type_url in the
+// DiscoveryRequest/DiscoveryResponse provides sufficient information to recover
+// the multiplexed singleton APIs at the Envoy instance and management server.
+service AggregatedDiscoveryService {
+  // This is a gRPC-only API.
+  rpc StreamAggregatedResources(stream DiscoveryRequest)
+      returns (stream DiscoveryResponse) {
+  }
+}

api/base.proto

@@ -110,13 +110,19 @@ message DiscoveryRequest {
   // the first request. It is expected that no new request is sent after a
   // response is received until the Envoy instance is ready to ACK/NACK the new
   // configuration. ACK/NACK takes place by returning the new API config version
-  // as applied or the previous API config version respectively.
+  // as applied or the previous API config version respectively. Each type_url
+  // (see below) has an independent version associated with it.
   string version_info = 1;
   Node node = 2;
   // List of resources to subscribe to, e.g. list of cluster names or a route
   // configuration name. If this is empty, all resources for the API are
   // returned.
   repeated string resource_names = 3;
+  // Type of the resource that is being requested, e.g.
+  // "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment". This is implicit
+  // in requests made via singleton xDS APIs such as CDS, LDS, etc. but is
+  // required for ADS.
+  string type_url = 4;
 }
 
 message DiscoveryResponse {
@@ -153,6 +159,10 @@ message ApiConfigSource {
   google.protobuf.Duration refresh_delay = 3;
 }
 
+// ADS will be used to fetch resources.
+message AggregatedConfigSource {
+}
+
 // Configuration for listeners, clusters, routes, endpoints etc. may either be
 // sourced from the filesystem or from an API source. Filesystem configs are
 // watched with inotify for updates.
@@ -160,5 +170,6 @@ message ConfigSource {
   oneof config_source_specifier {
     string path = 1;
     ApiConfigSource api_config_source = 2;
+    AggregatedConfigSource ads = 3;
   }
 }

api/bootstrap.proto

@@ -18,10 +18,12 @@ message Bootstrap {
   // All post-bootstrap Cluster definitions are provided by a single CDS
   // configuration source.
   ConfigSource cds_config = 3;
+  // A single ADS source may be optionally specified. This must have api_type GRPC.
+  ApiConfigSource ads_config = 4;
   // If a network based configuration source is specified for cds_config, it's
   // necessary to have some initial cluster definitions available to allow Envoy
   // to know how to speak to the management server. These cluster definitions
   // may not use EDS (i.e. they should be static IP or DNS-based).
-  repeated Cluster bootstrap_clusters = 4;
+  repeated Cluster bootstrap_clusters = 5;
   // TODO(htuch): Add support for HDS.
 }

test/build/BUILD

@@ -6,6 +6,7 @@ api_cc_test(
     name = "build_test",
     srcs = ["build_test.cc"],
     proto_deps = [
+        "//api:ads",
         "//api:cds",
         "//api:eds",
         "//api:hds",

test/build/build_test.cc

@@ -11,6 +11,7 @@
 // Basic C++ build/link validation for the v2 xDS APIs.
 int main(int argc, char *argv[]) {
   const auto methods = {
+    "envoy.api.v2.AggregatedDiscoveryService.StreamAggregatedResources",
     "envoy.api.v2.ClusterDiscoveryService.FetchClusters",
     "envoy.api.v2.ClusterDiscoveryService.StreamClusters",
     "envoy.api.v2.EndpointDiscoveryService.FetchEndpoints",