redis: fault injection (#12551)

Un-Reverts 048583b, with fix for high cpu consumption.

This PR implements fault injection for Redis; specifically delay and error faults (which themselves can have delays added). I chose not to implement a separate filter after discussing with Henry; we concluded that the faults we felt were useful didn't need many levels- just a delay on top of the original fault, if any. In addition, as the Redis protocol doesn't support headers that makes it a bit different again from Envoy's http fault injection. This PR fixes the issue previously seen with redis fault injection where excessive cpu was used on connection creation.

Faults record metrics on the original request- and the delay fault adds extra latency which is included in the command latency for that request. Also, faults can apply only to certain commands.
Future work: Add several other faults, including cache misses and connection failures.

Signed-off-by: FAYiEKcbD0XFqF2QK2E4viAHg8rMm2VbjYKdjTg <nflacco@lyft.com>

Mirrored from https://github.com/envoyproxy/envoy @ e319b7c2a93783cd8bde45fa4c002ae02a4cb13f

envoy/extensions/filters/network/redis_proxy/v3/redis_proxy.proto

@@ -23,7 +23,7 @@ option (udpa.annotations.file_status).package_version_status = ACTIVE;
 // Redis Proxy :ref:`configuration overview <config_network_filters_redis_proxy>`.
 // [#extension: envoy.filters.network.redis_proxy]
 
-// [#next-free-field: 8]
+// [#next-free-field: 9]
 message RedisProxy {
   option (udpa.annotations.versioning).previous_message_type =
       "envoy.config.filter.network.redis_proxy.v2.RedisProxy";
@@ -183,6 +183,31 @@ message RedisProxy {
     Route catch_all_route = 4;
   }
 
+  // RedisFault defines faults used for fault injection.
+  message RedisFault {
+    enum RedisFaultType {
+      // Delays requests. This is the base fault; other faults can have delays added.
+      DELAY = 0;
+
+      // Returns errors on requests.
+      ERROR = 1;
+    }
+
+    // Fault type.
+    RedisFaultType fault_type = 1 [(validate.rules).enum = {defined_only: true}];
+
+    // Percentage of requests fault applies to.
+    config.core.v3.RuntimeFractionalPercent fault_enabled = 2
+        [(validate.rules).message = {required: true}];
+
+    // Delay for all faults. If not set, defaults to zero
+    google.protobuf.Duration delay = 3;
+
+    // Commands fault is restricted to, if any. If not set, fault applies to all commands
+    // other than auth and ping (due to special handling of those commands in Envoy).
+    repeated string commands = 4;
+  }
+
   reserved 2;
 
   reserved "cluster";
@@ -236,6 +261,35 @@ message RedisProxy {
   // AUTH, but no password is set" error will be returned.
   config.core.v3.DataSource downstream_auth_password = 6 [(udpa.annotations.sensitive) = true];
 
+  // List of faults to inject. Faults currently come in two flavors:
+  // - Delay, which delays a request.
+  // - Error, which responds to a request with an error. Errors can also have delays attached.
+  //
+  // Example:
+  //
+  // .. code-block:: yaml
+  //
+  //    faults:
+  //    - fault_type: ERROR
+  //      fault_enabled:
+  //        default_value:
+  //          numerator: 10
+  //          denominator: HUNDRED
+  //        runtime_key: "bogus_key"
+  //        commands:
+  //        - GET
+  //      - fault_type: DELAY
+  //        fault_enabled:
+  //          default_value:
+  //            numerator: 10
+  //            denominator: HUNDRED
+  //          runtime_key: "bogus_key"
+  //        delay: 2s
+  //
+  // See the :ref:`fault injection section
+  // <config_network_filters_redis_proxy_fault_injection>` for more information on how to configure this.
+  repeated RedisFault faults = 8;
+
   // If a username is provided an ACL style AUTH command will be required with a username and password.
   // Authenticate Redis client connections locally by forcing downstream clients to issue a `Redis
   // AUTH command <https://redis.io/commands/auth>`_ with this username and the *downstream_auth_password*