cluster: allow ignoring health check status during eds removal (#3302)

This PR adds a configuration flag that allows disabling the "eventually consistent" aspect of endpoint updates: instead of waiting for the endpoints to go unhealthy before removing them from the cluster, do so immediately. This gives greater control to the control plane in cases where one might want to divert traffic from an endpoint
without having it go unhealthy. The flag goes on the cluster and so applies to all endpoints within that cluster.

Risk Level:
Low: Small configurable feature which reuses existing behavior (identical to the behavior when no health checker is configured). Defaults to disabled, so should have no impact on existing configurations.

Testing:
Added unit test for the case when endpoints are healthy then removed from the ClusterLoadAssignment in a subsequent config update.

Docs Changes:
Docs were added to the proto field.

Release Notes:
Added cluster: Add :ref:`option <envoy_api_field_Clister.drain_connections_on_eds_removal>` to drain endpoints after they are removed through EDS, despite health status. to the release notes.

[Optional Fixes #Issue]
#440 and #3276 (note that this last issue also asks for more fine grained control over endpoint removal. The solution this PR provides was brought up as a partial solution to #3276).

Signed-off-by: Snow Pettersen <snowp@squareup.com>

Mirrored from https://github.com/envoyproxy/envoy @ 08712e93b07695f53d192a2601cfa2ccc7a20f33

envoy/api/v2/cds.proto

@@ -442,6 +442,14 @@ message Cluster {
   //   to an upstream host that becomes unhealthy, Envoy may spend a substantial amount of
   //   time exclusively closing these connections, and not processing any other traffic.
   bool close_connections_on_host_health_failure = 31;
+
+  // If this cluster uses EDS or STRICT_DNS to configure its hosts, immediately drain
+  // connections from any hosts that are removed from service discovery.
+  //
+  // This only affects behavior for hosts that are being actively health checked.
+  // If this flag is not set to true, Envoy will wait until the hosts fail active health
+  // checking before removing it from the cluster.
+  bool drain_connections_on_host_removal = 32;
 }
 
 // An extensible structure containing the address Envoy should bind to when