Commit Message: Currently, there can be multiple built-in regexes targeting the same tag name, and in fact there's at least one case where this occurs:
466e78586a/source/common/config/well_known_names.cc (L133)466e78586a/source/common/config/well_known_names.cc (L136)
This change prevents a second tag value for a given name being from being extracted, to meet Prometheus' requirements.
Having two alternate ways of generating the same tag value allows them to be expressed using two distinct regexes, which are easier to understand, and possible for the infrastructure to optimize with the prefix-map. This situation also occurs with Istio/Wasm, which for reasons that elude me, generate stats with two very different syntaxes both meaning HTTP Response Code, and adds those extractors using configuration.
An alternate approach is to add complexity to the regex processing to allow matches in an ORed regex, which is a bit confusing, and results in regexes that cannot be optimized well by our current system. There is no one prefix that can be used to reduce the set of regexes that need to be evaluated against every stat, and the long regexes with captures are hard for humans to read. See https://github.com/envoyproxy/envoy/pull/22791
The disadvantage of allowing multiple regexes to generate the same tag, is that it may create more scenarios where a stats sink like Prometheus may be given multiple tags with the same name, and it would be good to get some notion that this is OK. Currently such cases would be rejected during process startup (for CLI-based tags) or during config processing.
I opened this up for review to initiate this discussion, but want to make sure various stakeholders have a chance to weigh in. Though no protobufs were changed structurally in this PR, it's kind of an API change (with .proto comments) and should probably be approved as one.
Additional Description:
Risk Level: medium
Testing: //test/...
Docs Changes: changed comments in proto file that previously indicated dups were not allowed
Release Notes:
Platform Specific Features:
Fixes: https://github.com/envoyproxy/envoy/issues/22591
Signed-off-by: Joshua Marantz <jmarantz@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 45f062466a40216d29117320ede012d087ca1318
* generic proxy: make the l7 filter could be configured repeatly
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Mirrored from https://github.com/envoyproxy/envoy @ 5a14b1a5b4b35de6dbb764fd1f5cdfb50614b095
Updating documentation to make clearer this issue: #3058
Risk Level: Low since it's just documentation?
Testing: Didn't do any -- happy to but was too lazy to set up my environment 😅
Docs Changes: Specifically for outlier detection, I was confused that 5xx mapped -- for TCP traffic -- to connection failures
Release Notes: Updated documentation on outlier detection
Fixes#3058
Signed-off-by: Steven Chu <stevenc1@gmail.com>
Signed-off-by: Steven Chu <stevenchu@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ 118b15a6b2491d46731a27f3a6b8eed3f643fa00
An xDS delegate extension point was added in
#22473 to enable custom behavior upon
receiving and loading xDS resources. This change creates an implementation of
the XdsResourcesDelegate interface that is backed by a KeyValueStore.
The intended use case is to enable persisting xDS resources and loading them on
startup in Envoy Mobile, in the event that the xDS control plane is unreachable.
Signed-off-by: Ali Beyad abeyad@google.com
Risk Level: Low
Testing: Unit & Integration tests
Docs Changes: N/A (hidden experimental feature)
Release Notes: N/A
Platform Specific Features: N/A
Signed-off-by: Ali Beyad <abeyad@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ f28db324be3fcb6e91eb5a89d90c08617f76a2b2
Update documentation to indicate that auto_host_rewrite works with STATIC clusters, provided that the hostname attribute of the endpoint is configured.
Signed-off-by: Klearchos Chaloulos <chaloulo@nokia.com>
Mirrored from https://github.com/envoyproxy/envoy @ 6d293a901a51f46289ef2b5491c248d93b974d81
Add support to save response headers from CONNECT tunnels in tcp_proxy.
The use case is saving "baggage" header which provides additional metadata about the upstream endpoint for telemetry purposes.
Fixes#23116
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ caa81fa49acff4d793c96d8f61f7cc99f32b59ac
Hi, I would like to suggest five changes/additions to gRPC access loger and accesslog proto file. These changes mainly benefit using Envoy as a TLS transparent proxy. I am not an experienced c++ developer and any feedback is welcome.
The problem is explained here: [#22407](https://github.com/envoyproxy/envoy/issues/22407)
After digging into the code, the proposal is to make five modifications/additions to the gRPC logger. A detailed description per modification/addition is listed below.
1. Set tls_sni_hostname also when TLS connection is not terminated.
2. Add duration
3. Add upstream_request_attempt_count
4. Add connection_termination_details
5. Add ja3_fingerprint when ja3_fingerprinting is enabled
Details
1. Set tls_sni_hostname also when TLS connection is not terminated.
This enables it to log the requested hostname when the tls listener filter is used, but the connection is simply forwarded to the upstream cluster. This is for example useful to see if envoy forwards to the correct cluster based on the requested hostname.
2. Add duration
The existing duration fields in the acceslog.proto are not present when using the tcp listener. The complete duration, based on stream_info.requestComplete(), however is available. A separate duration field is in line with the file access log.
3. Add upstream_request_attempt_count
Logs the number of times the connection request is attempted upstream. Note that the field is omitted when the connect request was never attempted upstream. This is particularly useful when the max_connect_attempts field is set in the tcp_proxy.
4. Add connection_termination_details
Connection termination details may provide additional information about why the connection was terminated by Envoy for L4 reasons. This is especiall usefull when network rbac envoy.filters.network.rbac . The connection termination details contains the info why a connection was blocked/allowed.
5. Add ja3_fingerprint when ja3_fingerprinting is enabled
The [tls inspector listner filter]( https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/listener/tls_inspector/v3/tls_inspector.proto#envoy-v3-api-msg-extensions-filters-listener-tls-inspector-v3-tlsinspector) has an option to generate ja3 fingerprint (default false). When available this logs the ja3 fingerprint. This field is omitted when the ja3 fingerprint is not available. The field is present when eiter envoy terminates the tls connection or acts as a transparent proxy.
Risk Level:
Low/Medium - four additional fields are added, no fields removed or changed (no behavior changes expected)
The only change is that tls_properties is also added when it is not a terminated tls connection. In this scenario, of the tls_properties only the tls_sni_hostname is available. The other tls_properties fields are not present. As far as I can see, this is consistent with the behavior of the tls listener filter.
This could be a breaking change if the receiving application of the access log message checks for the presence of the tls_properries and when present expect all the fields to be there, or assumes this must be a tls terminated connection.
Testing:
unit tests
Development build using mac os x (m1 if that info is relevant), but with ‘default’ extensions.
All tests pass.
manual testing
Using a sample grpc log service. Displaying accesslog stream using golang protojson.Marshal()
Tested
- Tcp with tls
- Tcp no tls
- Http no tls
Docs Changes:
I have added the documentation in the proto file [accesslog.proto](https://github.com/envoyproxy/envoy/blob/main/api/envoy/data/accesslog/v3/accesslog.proto) for data.accesslog.v3.AccessLogCommon, and data.accesslog.v3.TLSProperties.
Am I right in assuming that the documentation is generated from this file? If not, I have to modify the documentation.
Release Notes:
**access_log:** log `duration`, `upstream_request_attempt_count`, `connection_termination_details` and tls `ja3` field in the grpc access log and also log the tls `sni` and `ja3` field in the grpc access log when envoy is configured as a tls forward proxy.
Platform Specific Features: N/A
Issues:
Fixes: [#22407](https://github.com/envoyproxy/envoy/issues/22407)
[Optional API Considerations:]
ls_sni_hostname:
- See risk level above. The other option would be to add a requested_server_name field to the access_log_common properties. That would be present when either the tls connection is terminated or forwarded.
- The downside of this solution would be that this would create a duplicate field for tls_sni_hostname when envoy terminates the tls connection.
- The proposed solution: Filling tls_properties->tls_sni_hostame either when envoy terminates or forwards the tls connection is consistent with file access logging.
upstream_request_attempt_count
- In file logging this field is 0 when there is no upstream connection attempt made. I believe that omitting this field when there is no value is more in line with the grcp ALS service.
- In most scenarios the value of this field is 1. This field could also be adapted to a retry count. In that solution the field would be present only if the attempt > 1. This is however not consistent with file logging. The proposed solution is.
Signed-off-by: Niek Temme <niek@bubl.cloud>
Mirrored from https://github.com/envoyproxy/envoy @ 06625e63d1e780c5affe7938c9d7debe13c36caf
Risk Level: N/A
Testing: N/A
Docs Changes: This is a small docs change
Signed-off-by: Dhi Aurrahman <dio@rockybars.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4c0e53d8cee46d9d886ceed011b1a52000d261cf
This PR will implement issue detailed here and described below: #7763
Match Patterns and Templates
Wildcard support based on match patterns and templates.
A match pattern matches an incoming URL path.
Match patterns support glob operators to match URL text and variable definitions to bind matched text to names.
Template patterns are used to re-write URLs.
Template patterns build new URLs and may reference variables bound by a match pattern.
Match Examples
/**.m3u8 would match /foo.m3u8 and /foo/bar.m3u8.
/{dir_name}/*.ts would match /example/file.ts and bind dir_name="example" for a later template match to use.
/{dir_name}/**.ts would match /example/path/file.ts and bind dir_name="example" for a later template match to use. This would also match /example/.ts, which may or may not be a desired behavior.
/{path=v1/*}/{file=*.ts} would match /v1/example/movie.ts (binding path="v1/example" and file="movie"), but would not match /v0/example/movie.ts.
See post for full details and example:
#7763 (comment)
Risk Level:
Testing:
Unit tests. Both both internal matching/rewrite library and config/data plane changes.
Signed-off-by: silverstar195 <seanmaloney@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8cfc61f916cf52ce8bce6710686e9d4fca2c06bd
* initial draft for the meta protocol proxy
Signed-off-by: wbpcode <comems@msn.com>
* minor update
Signed-off-by: wbpcode <comems@msn.com>
* add match implemented
Signed-off-by: wbpcode <comems@msn.com>
* add some simple test
Signed-off-by: wbpcode <wbphub@live.com>
* add more test for route matcher
Signed-off-by: wbpcode <wbphub@live.com>
* partial commit
Signed-off-by: wbpcode <wbphub@live.com>
* complete basic unit test
Signed-off-by: wbpcode <comems@msn.com>
* fix format
Signed-off-by: wbpcode <comems@msn.com>
* fix error after merge
Signed-off-by: wbpcode <wbphub@live.com>
* add some more test
Signed-off-by: wbpcode <wbphub@live.com>
* minor fix
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix test
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* just make it run
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* just make it run
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* first integration test
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* complete almost all the tests and fix docs and format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix proto format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* minor update
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* add cleanup
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* update for clang tidy and type error
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* lower coverage threshould
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix unexpected include
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix window build
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* rename to generic proxy to avoid name conflict
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix docs
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* remove unnecessary readme
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* update comments and name of matcher
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* add name method
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* move everything to contrib
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* simple release note
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <comems@msn.com>
Signed-off-by: wbpcode <wbphub@live.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Mirrored from https://github.com/envoyproxy/envoy @ 51c0d6f47c98087c7e3288205cbf8edf50ae0196
Signed-off-by: John Howard <howardjohn@google.com>
Co-authored-by: phlax <phlax@users.noreply.github.com>
Mirrored from https://github.com/envoyproxy/envoy @ 466e78586afaeb8ecb5f92351cd5ffdee3871f49
The data is already available in the ClusterInfo class. (getter function; data generation)
Risk Level: Low
Testing: Unit Test
Fixes#22903
Signed-off-by: Alex Ding <39857214+nmdzl@users.noreply.github.com>
Mirrored from https://github.com/envoyproxy/envoy @ b5b291526288b3b6914cda436de1583b68550b58
This changes the Upstream Filter code to creating the codec filter via factory which
requires adding all UpstreamRequest/CodecFilter interactions to an UpstreamCallback interface accessible through the filter manager
requires unhiding the configuration because the presubmit cross-checks for the registered factory require all the config be unhidden
allows configuring the codec filter in configuration, in case of eventual alternate terminal filter
allows fixing up the filter dependency validator to validate the entire upstream filter chain.
Upstream filters flipped on for CI, SHOULD BE FLIPPED OFF BEFORE SUBMITTING
Risk Level: low assuming it's flipped back off
Testing: updated unit tests, covered by filter integration tests
Docs Changes: n/a
Release Notes: n/a (off by default)
Part of #10455
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 978004308203aa9e86886512adf6b26376958405
This updates the version of buf.build/googleapis/googleapis to the
latest pushed version. The BSR had a change in the googleapis module to
to concerns with the number of files in BSR, requiring previous commits
to be removed. See https://docs.buf.build/faq#googleapis-failure for
more details. This PR fixes the issue for envoy.
Signed-off-by: bufdev <bufdev-github@buf.build>
Signed-off-by: bufdev <bufdev-github@buf.build>
Mirrored from https://github.com/envoyproxy/envoy @ 68758cef311f65e8813df6b7235ead660cd952f8
Signed-off-by: Marcin Falkowski <marcin.falkowski@allegro.pl>
Mirrored from https://github.com/envoyproxy/envoy @ 2bee66e427f26680f799643ba9ce82ffd5541895
The HeaderAppendAction was added in the #18246. But the implementation never done. This PR try to complete this feature.
Further work of #18246. To close#22713.
Risk Level: Low.
Testing: Unit Test.
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Mirrored from https://github.com/envoyproxy/envoy @ 5183dbf7f11b23f1f7013a3e42cbce71eb63252a
Currently stats are not available in lua filter, which makes it hard to
track/alert for the script errors. This change adds error stat.
Signed-off-by: Suresh Kumar <sureshkumar.pp@gmail.com>
Signed-off-by: Suresh Kumar <suresh.ponnusamy@freshworks.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8c88943b747bd74517eaf257cd75dd686aa7bbb8
Signed-off-by: Kuat Yessenov <kuat@google.com>
Commit Message: Expose internal listener functionality. Marks the fields as implemented. Adds the following functional changes prior to publishing:
* do not use the `address` field per https://github.com/envoyproxy/envoy/pull/21132, and instead use the listener `name` field as originally expected.
* add an `endpoint_id` field to upstream address to fix https://github.com/envoyproxy/envoy/pull/22420; it's not possible to have multiple internal addresses to the same listener in a single pool without some extra identifier beyond the listener name, because pools identify addresses using `asString()`.
* fixes https://github.com/envoyproxy/envoy/issues/20665 by using stat prefix `listener.envoy_internal_<listener_name>`.
Adds a few sample configs (and fixes https://github.com/envoyproxy/envoy/issues/20335):
* minimal two chained `tcp_proxy` listeners;
* encap HTTP in HTTP CONNECT;
* decap HTTP in HTTP CONNECT;
Risk Level: low, hidden prior
Testing: yes
Docs Changes: yes
Release Notes: yes
Mirrored from https://github.com/envoyproxy/envoy @ 02489bbf8990faddb6be1f2e0f22851b64c3fc85
adding hidden config for upstream filters, and setting the cluster up to create upstream filters.
upstream filters are as yet unused, still cleaning up (#22434) but this makes it a much cleaner PR
(Also adding 2 random tweaks to router code because coverage was failing despite there previously being no changes to router code)
Risk Level: low (no-op if not configured)
Testing: unit tests
Docs Changes: n/a
Release Notes: n/a
part of #10455
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 5261b4285d1487f93e7aece94786de9a8d3a2529
Commit Message: contrib: add Hyperscan regex engine
Additional Description: Hyperscan has been introduced as an input matcher earlier this year. Since the regex engine interface has been completed, the patch extent the usage of Hyperscan into a contrib regex engine.
Risk Level: Low
Testing: Unit
Docs Changes: API
Release Notes: N/A
Platform Specific Features: Requires processor with SSSE3 support (nearly any modern x86 processor)
Signed-off-by: Xie Zhihao <zhihao.xie@intel.com>
Mirrored from https://github.com/envoyproxy/envoy @ 725d0b4b3e6fb84aa7dfd1ca4d5792f2dc4558b2
* cors: add PNA support
Signed-off-by: Loong Dai <loong.dai@intel.com>
* fix field
Signed-off-by: Loong Dai <loong.dai@intel.com>
* use runtime guard rather than api change
Signed-off-by: Loong Dai <loong.dai@intel.com>
* fix CI
Signed-off-by: Loong Dai <loong.dai@intel.com>
* update changelog
Signed-off-by: Loong Dai <loong.dai@intel.com>
* revert unrelated changes
Signed-off-by: Loong Dai <loong.dai@intel.com>
* update API
Signed-off-by: Loong Dai <loong.dai@intel.com>
* feedback and update tests
Signed-off-by: Loong Dai <loong.dai@intel.com>
* update tests
Signed-off-by: Loong Dai <loong.dai@intel.com>
* update tests
Signed-off-by: Loong Dai <loong.dai@intel.com>
* add tests
Signed-off-by: Loong Dai <loong.dai@intel.com>
* feedback
Signed-off-by: Loong Dai <loong.dai@intel.com>
* remove useless test
Signed-off-by: Loong Dai <loong.dai@intel.com>
* feedback
Signed-off-by: Loong Dai <loong.dai@intel.com>
* fix CI
Signed-off-by: Loong Dai <loong.dai@intel.com>
Mirrored from https://github.com/envoyproxy/envoy @ bf61b077e77197a28fefa5798902fd94008e4137
This PR is the first of a series to implement pattern matching and rewrite functionality for Envoy.
These specific proto changes:
Add the foundation for two extension configs
Change the API to expose the extension configs to end users
Risk Level: Low
Signed-off-by: silverstar195 <seanmaloney@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 25654b97cdd2647c258ae459e6607f51e99c99cc
data-plane-api(Azure Pipelines)