Document the new xff_num_trusted_hops feature (#479)

Envoy issue #2503
Corresponding code PR #2559

Signed-off-by: Brian Pane <bpane@pinterest.com>

docs/root/configuration/http_conn_man/headers.rst

@@ -65,13 +65,12 @@ the  :option:`--service-node` option.
 x-envoy-external-address
 ------------------------
 
-It is a common case where a service wants to perform analytics based on the client IP address. Per
-the lengthy discussion on :ref:`XFF <config_http_conn_man_headers_x-forwarded-for>`, this can get
-quite complicated. A proper implementation involves forwarding XFF, and then choosing the first non
-RFC1918 address *from the right*. Since this is such a common occurrence, Envoy simplifies this by
-setting *x-envoy-external-address* during decoding if and only if the request ingresses externally
-(i.e., it's from an external client). *x-envoy-external-address* is not set or overwritten for
-internal requests. This header can be safely forwarded between internal services for analytics
+It is a common case where a service wants to perform analytics based on the origin client's IP
+address. Per the lengthy discussion on :ref:`XFF <config_http_conn_man_headers_x-forwarded-for>`,
+this can get quite complicated, so Envoy simplifies this by setting *x-envoy-external-address*
+to the :ref:`trusted client address <config_http_conn_man_headers_x-forwarded-for_trusted_client_address>`
+if the request is from an external client. *x-envoy-external-address* is not set or overwritten
+for internal requests. This header can be safely forwarded between internal services for analytics
 purposes without having to deal with the complexities of XFF.
 
 .. _config_http_conn_man_headers_x-envoy-force-trace:
@@ -156,8 +155,10 @@ operates in a transparent mode where it does not modify XFF.
 .. attention::
 
   In general, *use_remote_address* should be set to true when Envoy is deployed as an edge
-  node, whereas it may need to be set to false when Envoy is used as an internal service node
-  in a mesh deployment.
+  node (aka a front proxy), whereas it may need to be set to false when Envoy is used as
+  an internal service node in a mesh deployment.
+
+.. _config_http_conn_man_headers_x-forwarded-for_trusted_client_address:
 
 The value of *use_remote_address* controls how Envoy determines the *trusted client address*.
 Given an HTTP request that has traveled through a series of zero or more proxies to reach
@@ -166,24 +167,116 @@ accurate. The source IP address of the immediate downstream node's connection to
 trusted. XFF *sometimes* can be trusted. Malicious clients can forge XFF, but the last
 address in XFF can be trusted if it was put there by a trusted proxy.
 
-Envoy's rules for determining the trusted client address are:
+Envoy's default rules for determining the trusted client address (*before* appending anything
+to XFF) are:
 
 * If *use_remote_address* is false and an XFF containing at least one IP address is
   present in the request, the trusted client address is the *last* (rightmost) IP address in XFF.
 * Otherwise, the trusted client address is the source IP address of the immediate downstream
   node's connection to Envoy.
 
+In an environment where there are one or more trusted proxies in front of an edge
+Envoy instance, the *xff_num_trusted_hops* configuration option can be used to trust
+additional addresses from XFF:
+
+* If *use_remote_address* is false and *xff_num_trusted_hops* is set to a value *N* that is
+  greater than zero, the trusted client address is the (N+1)th address from the right end
+  of XFF. (If the XFF contains fewer than N+1 addresses, Envoy falls back to using the
+  immediate downstream connection's source address as trusted client address.)
+* If *use_remote_address* is true and *xff_num_trusted_hops* is set to a value *N* that is
+  greater than zero, the trusted client address is the Nth address from the right end
+  of XFF. (If the XFF contains fewer than N addresses, Envoy falls back to using the
+  immediate downstream connection's source address as trusted client address.)
+
 Envoy uses the trusted client address contents to determine whether a request originated
 externally or internally. This influences whether the
 :ref:`config_http_conn_man_headers_x-envoy-internal` header is set.
 
-Testing IPv6 in a large multi-hop system can be difficult from a change management perspective. For
-testing IPv6 compatibility of upstream services which parse XFF header values,
-:ref:`represent_ipv4_remote_address_as_ipv4_mapped_ipv6
-<envoy_api_field_config.filter.network.http_connection_manager.v2.HttpConnectionManager.represent_ipv4_remote_address_as_ipv4_mapped_ipv6>`
-can be enabled in the v2 API. Envoy will append an IPv4 address in mapped IPv6 format, e.g.
-::FFFF:50.0.0.1. This change will also apply to
-:ref:`config_http_conn_man_headers_x-envoy-external-address`.
+Example 1: Envoy as edge proxy, without a trusted proxy in front of it
+    Settings:
+      | use_remote_address = true
+      | xff_num_trusted_hops = 0
+
+    Request details:
+      | Downstream IP address = 192.0.2.5
+      | XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1"
+
+    Result:
+      | Trusted client address = 192.0.2.5 (XFF is ignored)
+      | X-Envoy-External-Address is set to 192.0.2.5
+      | XFF is changed to "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5"
+      | X-Envoy-Internal is removed (if it was present in the incoming request)
+
+Example 2: Envoy as internal proxy, with the Envoy edge proxy from Example 1 in front of it
+    Settings:
+      | use_remote_address = false
+      | xff_num_trusted_hops = 0
+
+    Request details:
+      | Downstream IP address = 10.11.12.13 (address of the Envoy edge proxy)
+      | XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5"
+
+    Result:
+      | Trusted client address = 192.0.2.5 (last address in XFF is trusted)
+      | X-Envoy-External-Address is not modified
+      | X-Envoy-Internal is removed (if it was present in the incoming request)
+
+Example 3: Envoy as edge proxy, with two trusted external proxies in front of it
+    Settings:
+      | use_remote_address = true
+      | xff_num_trusted_hops = 2
+
+    Request details:
+      | Downstream IP address = 192.0.2.5
+      | XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1"
+
+    Result:
+      | Trusted client address = 203.0.113.10 (2nd to last address in XFF is trusted)
+      | X-Envoy-External-Address is set to 203.0.113.10
+      | XFF is changed to "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5"
+      | X-Envoy-Internal is removed (if it was present in the incoming request)
+
+Example 4: Envoy as internal proxy, with the edge proxy from Example 3 in front of it
+    Settings:
+      | use_remote_address = false
+      | xff_num_trusted_hops = 2
+
+    Request details:
+      | Downstream IP address = 10.11.12.13 (address of the Envoy edge proxy)
+      | XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5"
+
+    Result:
+      | Trusted client address = 203.0.113.10
+      | X-Envoy-External-Address is not modified
+      | X-Envoy-Internal is removed (if it was present in the incoming request)
+
+Example 5: Envoy as an internal proxy, receiving a request from an internal client
+    Settings:
+      | use_remote_address = false
+      | xff_num_trusted_hops = 0
+
+    Request details:
+      | Downstream IP address = 10.20.30.40 (address of the internal client)
+      | XFF is not present
+
+    Result:
+      | Trusted client address = 10.20.30.40
+      | X-Envoy-External-Address remains unset
+      | X-Envoy-Internal is set to "true"
+
+Example 6: The internal Envoy from Example 5, receiving a request proxied by another Envoy
+    Settings:
+      | use_remote_address = false
+      | xff_num_trusted_hops = 0
+
+    Request details:
+      | Downstream IP address = 10.20.30.50 (address of the Envoy instance proxying to this one)
+      | XFF = "10.20.30.40"
+
+    Result:
+      | Trusted client address = 10.20.30.40
+      | X-Envoy-External-Address remains unset
+      | X-Envoy-Internal is set to "true"
 
 A few very important notes about XFF:
 
@@ -206,6 +299,12 @@ A few very important notes about XFF:
      Envoy will not consider it internal. This is a known "bug" due to the simplification of how
      XFF is parsed to determine if a request is internal. In this scenario, do not forward XFF and
      allow Envoy to generate a new one with a single internal origin IP.
+3. Testing IPv6 in a large multi-hop system can be difficult from a change management perspective.
+   For testing IPv6 compatibility of upstream services which parse XFF header values,
+   :ref:`represent_ipv4_remote_address_as_ipv4_mapped_ipv6 <envoy_api_field_config.filter.network.http_connection_manager.v2.HttpConnectionManager.represent_ipv4_remote_address_as_ipv4_mapped_ipv6>`
+   can be enabled in the v2 API. Envoy will append an IPv4 address in mapped IPv6 format, e.g.
+   ::FFFF:50.0.0.1. This change will also apply to
+   :ref:`config_http_conn_man_headers_x-envoy-external-address`.
 
 .. _config_http_conn_man_headers_x-forwarded-proto:
 

envoy/config/filter/network/http_connection_manager/v2/http_connection_manager.proto

@@ -161,7 +161,6 @@ message HttpConnectionManager {
   // :ref:`config_http_conn_man_headers_x-envoy-external-address` for more information.
   google.protobuf.BoolValue use_remote_address = 14;
 
-  // [#not-implemented-hide:]
   // The number of additional ingress proxy hops from the right side of the
   // :ref:`config_http_conn_man_headers_x-forwarded-for` HTTP header to trust when
   // determining the origin client's IP address. The default is zero if this option