lb_subset_config: new fallback policy for selectors: KEYS_SUBSET (#8890)

Introduce new fallback policy for subset load balancer's selectors: KEYS_SUBSET and related LbSubsetSelector config parameter: fallback_keys_subset.

When context metadata matches given selector on keys, but there is no matching subset and KEYS_SUBSET fallback policy is set for that selector, there will be another attempt on subset selector matching. For that consecutive attempt, the context metadata will be reduced to keys included in fallback_keys_subset.

Risk Level: low (no changes in existing features, adding a new feature that is disabled by default). But there is also a small bugfix that can affect existing behaviour.
Testing: Unit tests and manual testing using envoy with static config
Docs: added
Release Notes: added
Fixes: #8767
Fixes: #8874

Signed-off-by: Marcin Falkowski <marcin.falkowski@allegro.pl>

Mirrored from https://github.com/envoyproxy/envoy @ b7bef67c256090919a4585a1a06c42f15d640a09

envoy/api/v2/cds.proto

@@ -222,6 +222,13 @@ message Cluster {
         // If DEFAULT_SUBSET is selected, load balancing is performed over the
         // endpoints matching the values from the default_subset field.
         DEFAULT_SUBSET = 3;
+
+        // If KEYS_SUBSET is selected, subset selector matching is performed again with metadata
+        // keys reduced to
+        // :ref:`fallback_keys_subset<envoy_api_field_Cluster.LbSubsetConfig.LbSubsetSelector.fallback_keys_subset>`.
+        // It allows for a fallback to a different, less specific selector if some of the keys of
+        // the selector are considered optional.
+        KEYS_SUBSET = 4;
       }
 
       // List of keys to match with the weighted cluster metadata.
@@ -231,6 +238,17 @@ message Cluster {
       // metadata.
       LbSubsetSelectorFallbackPolicy fallback_policy = 2
           [(validate.rules).enum = {defined_only: true}];
+
+      // Subset of
+      // :ref:`keys<envoy_api_field_Cluster.LbSubsetConfig.LbSubsetSelector.keys>` used by
+      // :ref:`KEYS_SUBSET<envoy_api_enum_value_Cluster.LbSubsetConfig.LbSubsetSelector.LbSubsetSelectorFallbackPolicy.KEYS_SUBSET>`
+      // fallback policy.
+      // It has to be a non empty list if KEYS_SUBSET fallback policy is selected.
+      // For any other fallback policy the parameter is not used and should not be set.
+      // Only values also present in
+      // :ref:`keys<envoy_api_field_Cluster.LbSubsetConfig.LbSubsetSelector.keys>` are allowed, but
+      // `fallback_keys_subset` cannot be equal to `keys`.
+      repeated string fallback_keys_subset = 3;
     }
 
     // The behavior used when no endpoint subset matches the selected route's

envoy/api/v3alpha/cds.proto

@@ -216,6 +216,13 @@ message Cluster {
         // If DEFAULT_SUBSET is selected, load balancing is performed over the
         // endpoints matching the values from the default_subset field.
         DEFAULT_SUBSET = 3;
+
+        // If KEYS_SUBSET is selected, subset selector matching is performed again with metadata
+        // keys reduced to
+        // :ref:`fallback_keys_subset<envoy_api_field_api.v3alpha.Cluster.LbSubsetConfig.LbSubsetSelector.fallback_keys_subset>`.
+        // It allows for a fallback to a different, less specific selector if some of the keys of
+        // the selector are considered optional.
+        KEYS_SUBSET = 4;
       }
 
       // List of keys to match with the weighted cluster metadata.
@@ -225,6 +232,18 @@ message Cluster {
       // metadata.
       LbSubsetSelectorFallbackPolicy fallback_policy = 2
           [(validate.rules).enum = {defined_only: true}];
+
+      // Subset of
+      // :ref:`keys<envoy_api_field_api.v3alpha.Cluster.LbSubsetConfig.LbSubsetSelector.keys>` used
+      // by
+      // :ref:`KEYS_SUBSET<envoy_api_enum_value_api.v3alpha.Cluster.LbSubsetConfig.LbSubsetSelector.LbSubsetSelectorFallbackPolicy.KEYS_SUBSET>`
+      // fallback policy.
+      // It has to be a non empty list if KEYS_SUBSET fallback policy is selected.
+      // For any other fallback policy the parameter is not used and should not be set.
+      // Only values also present in
+      // :ref:`keys<envoy_api_field_api.v3alpha.Cluster.LbSubsetConfig.LbSubsetSelector.keys>` are
+      // allowed, but `fallback_keys_subset` cannot be equal to `keys`.
+      repeated string fallback_keys_subset = 3;
     }
 
     // The behavior used when no endpoint subset matches the selected route's