upstream: add option to scale locality weights in subset lb (#5074)

Adds an option that allows smoothing out the locality weights when used
with subset lb: weights are scaled by the number of hosts that were
removed due to the subset filter. This allows for less of a step
function when a small number of hosts in a locality are updated to be
included in the subset.

Fixes #4837

Signed-off-by: Snow Pettersen snowp@squareup.com

Risk Level: Low, new optional feature
Testing: UTs
Docs Changes: Inline in protos
Release Notes: added release note

Signed-off-by: Snow Pettersen <snowp@squareup.com>

Mirrored from https://github.com/envoyproxy/envoy @ b094017026ce57d52762513bd5c0f774da0fc39e

envoy/api/v2/cds.proto

@@ -376,6 +376,12 @@ message Cluster {
     // host selected but Y having 100, then a lot more load is being dumped on the single host in X
     // than originally anticipated in the load balancing assignment delivered via EDS.
     bool locality_weight_aware = 4;
+
+    // When used with locality_weight_aware, scales the weight of each locality by the ratio
+    // of hosts in the subset vs hosts in the original subset. This aims to even out the load
+    // going to an individual locality if said locality is disproportionally affected by the
+    // subset predicate.
+    bool scale_locality_weight = 5;
   }
 
   // Configuration for load balancing subsetting.