This filter is configured with request_rules that will be matched against requests. A field_selector of a rule represents the head of a linked list, each node of the linked list has a name for logging and an id for matching. The field_selector is tied to a payload field when the linked list corresponds to a downward path which rooted in the top-level of the request message structure. on_present is triggered when corresponding the payload is present. Otherwise, on_missing is triggered.
This filter is designed to support payload passthrough. By performing payload to metadata filter can do deserialization once, and pass the metadata to other filters. This means that load balancing decisions, consumed from log and routing could all use payload information with a single parse. Also notably performing the parsing in payload passthrough buffer will mean deserialization once and not re-serializing, which is the most performant outcome.
Risk Level: low
Testing: unit
Docs Changes: multiple rst
Fixes#23322
Signed-off-by: kuochunghsu <kuochunghsu@pinterest.com>
Mirrored from https://github.com/envoyproxy/envoy @ cd208a5dbc281dcc27a8155a210037267c08ff6f
remove an extra dot in docs
Signed-off-by: Peter Jausovec <peter.jausovec@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 82667905a934826576e817a8222b0e1117b79b15
Add dubbo codec for generic proxy to support proxy dubbo traffic by the generic proxy. It's simple wrapper to the common dubbo codec (`/source/extensions/common/dubbo`).
Risk Level: n/a. new feature.
Testing: Unit.
Docs Changes: n/a.
Release Notes: Added.
Platform Specific Features: n/a.
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Mirrored from https://github.com/envoyproxy/envoy @ ab0abb640b1e501c66762c45b164529378c6bf66
make QUIC connection ID generation an extension point with currently in-use EnvoyDeterministicConnectionIdGenerator as the default implementation.
Additional Description: fix some previously unused QUICHE build targets.
Risk Level: low, control plane change
Testing: added new unit tests
Docs Changes: docs/root/api-v3/config/quic/quic_extensions.rst
Release Notes: N/A
API Considerations: interface naming and documentation
Signed-off-by: Dan Zhang <danzh@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 02ebc03205cfae5b26ce890050b9d1b6e0c2d1f5
It is observed that Envoy crash in ASSERT() due to regex rewrite path string contains invalid character '
'.
We should prohibit null characters ,
,
in the regex rewrite substitution string. This is well guarded in most other cases like RouteAction:prefix_rewrite, but is missing in RouteAction:regex_rewrite:substitution.
Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 23f3ee57b012e28aea0d6e7bdd6a991edd392741
* xds: add config for random and maglev load balancer extensions
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* update extension type to envoy.load_balancing_policies because envoy.clusters is extension type for custom cluster
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Mirrored from https://github.com/envoyproxy/envoy @ e4352708f7991872159dd8db12d379d190da1e26
Added support for replacing dots of gRPC service name with underscores in the gRPC stats by
the config grpc_stats.v3.FilterConfig.replace_dots_in_grpc_service_name.
Risk Level: Low
Testing:
1) Added Unit tests in context_impl_test and config_test.
2) Compiled and tested the Envoy locally and verified via output in /stats page.
Docs Changes:
Added inline doc description in the grpc_stats config.proto.
Release Notes:
Added the note in version history
Platform Specific Features: N/A
Fixes https://github.com/envoyproxy/envoy/issues/16384
Signed-off-by: Junpeng Luo <ljpeng1001@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 6c4c0b1556d421e786c993061d9c304f0bc6e1d5
Since the ipv4 and ipv6 have different socket option flags, when using multiple addresses, the user has to specify different socket options for the ipv4 address and the ipv6 address.
For the listener, the additional address can be the Ipv6 address, then it should be able to set an Ipv6 flag corresponding to the ipv4 one. Add socket_option field for each additional address.
For the upstream, the endpoint can be ipv4 or ipv6, currently, the user can specify the ipv4 and ipv6 local bind address in the bind config, but there is only a global socket_options that apply to both the ipv4 and ipv6 addresses. Add socket_options for each extra source address. https://envoyproxy.slack.com/archives/C78HA81DH/p1664228598624269
Risk Level: low
Testing: n/a
Docs Changes: API doc
Release Notes: n/a
Platform Specific Features: n/a
Signed-off-by: He Jie Xu <hejie.xu@intel.com>
Mirrored from https://github.com/envoyproxy/envoy @ 601cf012144a6d212879b315efa51e9cdf177878
Adds a note that the ``cipher_suites`` setting needs to be manually set if using ``tls_minimum_protocol_version`` below ``TLSv1_2``.
Signed-off-by: gsalisbury <gsalisbury@apnic.net>
Mirrored from https://github.com/envoyproxy/envoy @ 538c0bb9bc0dc960677269512560d60c564b5f26
Risk Level: low
Testing: n/a
Signed-off-by: He Jie Xu <hejie.xu@intel.com>
Mirrored from https://github.com/envoyproxy/envoy @ b1208ec4fd311d86086a99fb5f9f76d16af3a9ee
Add a "canonical suffix" list to the Alt-Svc cache so that Alt-Svc entries can be shared across origins which share the same hostname suffix.
Risk Level: Low
Testing: New unit tests
Docs Changes: Update proto docs
Release Notes: Updated
Signed-off-by: Ryan Hamilton <rch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 2b67ce314da75f304d7f65d05155bcee7c7d18e1
Commit Message: Currently, there can be multiple built-in regexes targeting the same tag name, and in fact there's at least one case where this occurs:
466e78586a/source/common/config/well_known_names.cc (L133)466e78586a/source/common/config/well_known_names.cc (L136)
This change prevents a second tag value for a given name being from being extracted, to meet Prometheus' requirements.
Having two alternate ways of generating the same tag value allows them to be expressed using two distinct regexes, which are easier to understand, and possible for the infrastructure to optimize with the prefix-map. This situation also occurs with Istio/Wasm, which for reasons that elude me, generate stats with two very different syntaxes both meaning HTTP Response Code, and adds those extractors using configuration.
An alternate approach is to add complexity to the regex processing to allow matches in an ORed regex, which is a bit confusing, and results in regexes that cannot be optimized well by our current system. There is no one prefix that can be used to reduce the set of regexes that need to be evaluated against every stat, and the long regexes with captures are hard for humans to read. See https://github.com/envoyproxy/envoy/pull/22791
The disadvantage of allowing multiple regexes to generate the same tag, is that it may create more scenarios where a stats sink like Prometheus may be given multiple tags with the same name, and it would be good to get some notion that this is OK. Currently such cases would be rejected during process startup (for CLI-based tags) or during config processing.
I opened this up for review to initiate this discussion, but want to make sure various stakeholders have a chance to weigh in. Though no protobufs were changed structurally in this PR, it's kind of an API change (with .proto comments) and should probably be approved as one.
Additional Description:
Risk Level: medium
Testing: //test/...
Docs Changes: changed comments in proto file that previously indicated dups were not allowed
Release Notes:
Platform Specific Features:
Fixes: https://github.com/envoyproxy/envoy/issues/22591
Signed-off-by: Joshua Marantz <jmarantz@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 45f062466a40216d29117320ede012d087ca1318
* generic proxy: make the l7 filter could be configured repeatly
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Mirrored from https://github.com/envoyproxy/envoy @ 5a14b1a5b4b35de6dbb764fd1f5cdfb50614b095
Updating documentation to make clearer this issue: #3058
Risk Level: Low since it's just documentation?
Testing: Didn't do any -- happy to but was too lazy to set up my environment 😅
Docs Changes: Specifically for outlier detection, I was confused that 5xx mapped -- for TCP traffic -- to connection failures
Release Notes: Updated documentation on outlier detection
Fixes#3058
Signed-off-by: Steven Chu <stevenc1@gmail.com>
Signed-off-by: Steven Chu <stevenchu@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ 118b15a6b2491d46731a27f3a6b8eed3f643fa00
An xDS delegate extension point was added in
#22473 to enable custom behavior upon
receiving and loading xDS resources. This change creates an implementation of
the XdsResourcesDelegate interface that is backed by a KeyValueStore.
The intended use case is to enable persisting xDS resources and loading them on
startup in Envoy Mobile, in the event that the xDS control plane is unreachable.
Signed-off-by: Ali Beyad abeyad@google.com
Risk Level: Low
Testing: Unit & Integration tests
Docs Changes: N/A (hidden experimental feature)
Release Notes: N/A
Platform Specific Features: N/A
Signed-off-by: Ali Beyad <abeyad@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ f28db324be3fcb6e91eb5a89d90c08617f76a2b2
Update documentation to indicate that auto_host_rewrite works with STATIC clusters, provided that the hostname attribute of the endpoint is configured.
Signed-off-by: Klearchos Chaloulos <chaloulo@nokia.com>
Mirrored from https://github.com/envoyproxy/envoy @ 6d293a901a51f46289ef2b5491c248d93b974d81
Add support to save response headers from CONNECT tunnels in tcp_proxy.
The use case is saving "baggage" header which provides additional metadata about the upstream endpoint for telemetry purposes.
Fixes#23116
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ caa81fa49acff4d793c96d8f61f7cc99f32b59ac
Hi, I would like to suggest five changes/additions to gRPC access loger and accesslog proto file. These changes mainly benefit using Envoy as a TLS transparent proxy. I am not an experienced c++ developer and any feedback is welcome.
The problem is explained here: [#22407](https://github.com/envoyproxy/envoy/issues/22407)
After digging into the code, the proposal is to make five modifications/additions to the gRPC logger. A detailed description per modification/addition is listed below.
1. Set tls_sni_hostname also when TLS connection is not terminated.
2. Add duration
3. Add upstream_request_attempt_count
4. Add connection_termination_details
5. Add ja3_fingerprint when ja3_fingerprinting is enabled
Details
1. Set tls_sni_hostname also when TLS connection is not terminated.
This enables it to log the requested hostname when the tls listener filter is used, but the connection is simply forwarded to the upstream cluster. This is for example useful to see if envoy forwards to the correct cluster based on the requested hostname.
2. Add duration
The existing duration fields in the acceslog.proto are not present when using the tcp listener. The complete duration, based on stream_info.requestComplete(), however is available. A separate duration field is in line with the file access log.
3. Add upstream_request_attempt_count
Logs the number of times the connection request is attempted upstream. Note that the field is omitted when the connect request was never attempted upstream. This is particularly useful when the max_connect_attempts field is set in the tcp_proxy.
4. Add connection_termination_details
Connection termination details may provide additional information about why the connection was terminated by Envoy for L4 reasons. This is especiall usefull when network rbac envoy.filters.network.rbac . The connection termination details contains the info why a connection was blocked/allowed.
5. Add ja3_fingerprint when ja3_fingerprinting is enabled
The [tls inspector listner filter]( https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/listener/tls_inspector/v3/tls_inspector.proto#envoy-v3-api-msg-extensions-filters-listener-tls-inspector-v3-tlsinspector) has an option to generate ja3 fingerprint (default false). When available this logs the ja3 fingerprint. This field is omitted when the ja3 fingerprint is not available. The field is present when eiter envoy terminates the tls connection or acts as a transparent proxy.
Risk Level:
Low/Medium - four additional fields are added, no fields removed or changed (no behavior changes expected)
The only change is that tls_properties is also added when it is not a terminated tls connection. In this scenario, of the tls_properties only the tls_sni_hostname is available. The other tls_properties fields are not present. As far as I can see, this is consistent with the behavior of the tls listener filter.
This could be a breaking change if the receiving application of the access log message checks for the presence of the tls_properries and when present expect all the fields to be there, or assumes this must be a tls terminated connection.
Testing:
unit tests
Development build using mac os x (m1 if that info is relevant), but with ‘default’ extensions.
All tests pass.
manual testing
Using a sample grpc log service. Displaying accesslog stream using golang protojson.Marshal()
Tested
- Tcp with tls
- Tcp no tls
- Http no tls
Docs Changes:
I have added the documentation in the proto file [accesslog.proto](https://github.com/envoyproxy/envoy/blob/main/api/envoy/data/accesslog/v3/accesslog.proto) for data.accesslog.v3.AccessLogCommon, and data.accesslog.v3.TLSProperties.
Am I right in assuming that the documentation is generated from this file? If not, I have to modify the documentation.
Release Notes:
**access_log:** log `duration`, `upstream_request_attempt_count`, `connection_termination_details` and tls `ja3` field in the grpc access log and also log the tls `sni` and `ja3` field in the grpc access log when envoy is configured as a tls forward proxy.
Platform Specific Features: N/A
Issues:
Fixes: [#22407](https://github.com/envoyproxy/envoy/issues/22407)
[Optional API Considerations:]
ls_sni_hostname:
- See risk level above. The other option would be to add a requested_server_name field to the access_log_common properties. That would be present when either the tls connection is terminated or forwarded.
- The downside of this solution would be that this would create a duplicate field for tls_sni_hostname when envoy terminates the tls connection.
- The proposed solution: Filling tls_properties->tls_sni_hostame either when envoy terminates or forwards the tls connection is consistent with file access logging.
upstream_request_attempt_count
- In file logging this field is 0 when there is no upstream connection attempt made. I believe that omitting this field when there is no value is more in line with the grcp ALS service.
- In most scenarios the value of this field is 1. This field could also be adapted to a retry count. In that solution the field would be present only if the attempt > 1. This is however not consistent with file logging. The proposed solution is.
Signed-off-by: Niek Temme <niek@bubl.cloud>
Mirrored from https://github.com/envoyproxy/envoy @ 06625e63d1e780c5affe7938c9d7debe13c36caf
Risk Level: N/A
Testing: N/A
Docs Changes: This is a small docs change
Signed-off-by: Dhi Aurrahman <dio@rockybars.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4c0e53d8cee46d9d886ceed011b1a52000d261cf
This PR will implement issue detailed here and described below: #7763
Match Patterns and Templates
Wildcard support based on match patterns and templates.
A match pattern matches an incoming URL path.
Match patterns support glob operators to match URL text and variable definitions to bind matched text to names.
Template patterns are used to re-write URLs.
Template patterns build new URLs and may reference variables bound by a match pattern.
Match Examples
/**.m3u8 would match /foo.m3u8 and /foo/bar.m3u8.
/{dir_name}/*.ts would match /example/file.ts and bind dir_name="example" for a later template match to use.
/{dir_name}/**.ts would match /example/path/file.ts and bind dir_name="example" for a later template match to use. This would also match /example/.ts, which may or may not be a desired behavior.
/{path=v1/*}/{file=*.ts} would match /v1/example/movie.ts (binding path="v1/example" and file="movie"), but would not match /v0/example/movie.ts.
See post for full details and example:
#7763 (comment)
Risk Level:
Testing:
Unit tests. Both both internal matching/rewrite library and config/data plane changes.
Signed-off-by: silverstar195 <seanmaloney@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8cfc61f916cf52ce8bce6710686e9d4fca2c06bd
* initial draft for the meta protocol proxy
Signed-off-by: wbpcode <comems@msn.com>
* minor update
Signed-off-by: wbpcode <comems@msn.com>
* add match implemented
Signed-off-by: wbpcode <comems@msn.com>
* add some simple test
Signed-off-by: wbpcode <wbphub@live.com>
* add more test for route matcher
Signed-off-by: wbpcode <wbphub@live.com>
* partial commit
Signed-off-by: wbpcode <wbphub@live.com>
* complete basic unit test
Signed-off-by: wbpcode <comems@msn.com>
* fix format
Signed-off-by: wbpcode <comems@msn.com>
* fix error after merge
Signed-off-by: wbpcode <wbphub@live.com>
* add some more test
Signed-off-by: wbpcode <wbphub@live.com>
* minor fix
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix test
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* just make it run
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* just make it run
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* first integration test
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* complete almost all the tests and fix docs and format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix proto format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* minor update
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* add cleanup
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* update for clang tidy and type error
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* lower coverage threshould
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix unexpected include
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix window build
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* rename to generic proxy to avoid name conflict
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix docs
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* remove unnecessary readme
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* update comments and name of matcher
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* add name method
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* move everything to contrib
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* simple release note
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <comems@msn.com>
Signed-off-by: wbpcode <wbphub@live.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Mirrored from https://github.com/envoyproxy/envoy @ 51c0d6f47c98087c7e3288205cbf8edf50ae0196
Signed-off-by: John Howard <howardjohn@google.com>
Co-authored-by: phlax <phlax@users.noreply.github.com>
Mirrored from https://github.com/envoyproxy/envoy @ 466e78586afaeb8ecb5f92351cd5ffdee3871f49
The data is already available in the ClusterInfo class. (getter function; data generation)
Risk Level: Low
Testing: Unit Test
Fixes#22903
Signed-off-by: Alex Ding <39857214+nmdzl@users.noreply.github.com>
Mirrored from https://github.com/envoyproxy/envoy @ b5b291526288b3b6914cda436de1583b68550b58
This changes the Upstream Filter code to creating the codec filter via factory which
requires adding all UpstreamRequest/CodecFilter interactions to an UpstreamCallback interface accessible through the filter manager
requires unhiding the configuration because the presubmit cross-checks for the registered factory require all the config be unhidden
allows configuring the codec filter in configuration, in case of eventual alternate terminal filter
allows fixing up the filter dependency validator to validate the entire upstream filter chain.
Upstream filters flipped on for CI, SHOULD BE FLIPPED OFF BEFORE SUBMITTING
Risk Level: low assuming it's flipped back off
Testing: updated unit tests, covered by filter integration tests
Docs Changes: n/a
Release Notes: n/a (off by default)
Part of #10455
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 978004308203aa9e86886512adf6b26376958405
data-plane-api(Azure Pipelines)