This PR will implement issue detailed here and described below: #7763
Match Patterns and Templates
Wildcard support based on match patterns and templates.
A match pattern matches an incoming URL path.
Match patterns support glob operators to match URL text and variable definitions to bind matched text to names.
Template patterns are used to re-write URLs.
Template patterns build new URLs and may reference variables bound by a match pattern.
Match Examples
/**.m3u8 would match /foo.m3u8 and /foo/bar.m3u8.
/{dir_name}/*.ts would match /example/file.ts and bind dir_name="example" for a later template match to use.
/{dir_name}/**.ts would match /example/path/file.ts and bind dir_name="example" for a later template match to use. This would also match /example/.ts, which may or may not be a desired behavior.
/{path=v1/*}/{file=*.ts} would match /v1/example/movie.ts (binding path="v1/example" and file="movie"), but would not match /v0/example/movie.ts.
See post for full details and example:
#7763 (comment)
Risk Level:
Testing:
Unit tests. Both both internal matching/rewrite library and config/data plane changes.
Signed-off-by: silverstar195 <seanmaloney@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8cfc61f916cf52ce8bce6710686e9d4fca2c06bd
* initial draft for the meta protocol proxy
Signed-off-by: wbpcode <comems@msn.com>
* minor update
Signed-off-by: wbpcode <comems@msn.com>
* add match implemented
Signed-off-by: wbpcode <comems@msn.com>
* add some simple test
Signed-off-by: wbpcode <wbphub@live.com>
* add more test for route matcher
Signed-off-by: wbpcode <wbphub@live.com>
* partial commit
Signed-off-by: wbpcode <wbphub@live.com>
* complete basic unit test
Signed-off-by: wbpcode <comems@msn.com>
* fix format
Signed-off-by: wbpcode <comems@msn.com>
* fix error after merge
Signed-off-by: wbpcode <wbphub@live.com>
* add some more test
Signed-off-by: wbpcode <wbphub@live.com>
* minor fix
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix test
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* just make it run
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* just make it run
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* first integration test
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* complete almost all the tests and fix docs and format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix proto format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* minor update
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* add cleanup
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* update for clang tidy and type error
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* lower coverage threshould
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix unexpected include
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix window build
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* rename to generic proxy to avoid name conflict
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix docs
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* remove unnecessary readme
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* update comments and name of matcher
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* add name method
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* move everything to contrib
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* simple release note
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <comems@msn.com>
Signed-off-by: wbpcode <wbphub@live.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Mirrored from https://github.com/envoyproxy/envoy @ 51c0d6f47c98087c7e3288205cbf8edf50ae0196
This changes the Upstream Filter code to creating the codec filter via factory which
requires adding all UpstreamRequest/CodecFilter interactions to an UpstreamCallback interface accessible through the filter manager
requires unhiding the configuration because the presubmit cross-checks for the registered factory require all the config be unhidden
allows configuring the codec filter in configuration, in case of eventual alternate terminal filter
allows fixing up the filter dependency validator to validate the entire upstream filter chain.
Upstream filters flipped on for CI, SHOULD BE FLIPPED OFF BEFORE SUBMITTING
Risk Level: low assuming it's flipped back off
Testing: updated unit tests, covered by filter integration tests
Docs Changes: n/a
Release Notes: n/a (off by default)
Part of #10455
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 978004308203aa9e86886512adf6b26376958405
Commit Message: contrib: add Hyperscan regex engine
Additional Description: Hyperscan has been introduced as an input matcher earlier this year. Since the regex engine interface has been completed, the patch extent the usage of Hyperscan into a contrib regex engine.
Risk Level: Low
Testing: Unit
Docs Changes: API
Release Notes: N/A
Platform Specific Features: Requires processor with SSSE3 support (nearly any modern x86 processor)
Signed-off-by: Xie Zhihao <zhihao.xie@intel.com>
Mirrored from https://github.com/envoyproxy/envoy @ 725d0b4b3e6fb84aa7dfd1ca4d5792f2dc4558b2
This PR is the first of a series to implement pattern matching and rewrite functionality for Envoy.
These specific proto changes:
Add the foundation for two extension configs
Change the API to expose the extension configs to end users
Risk Level: Low
Signed-off-by: silverstar195 <seanmaloney@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 25654b97cdd2647c258ae459e6607f51e99c99cc
Next generation Intel® QAT support with Intel® Xeon® Scalable processors
will feature an Intel® QAT cryptography and compression acceleration
engine.
QAT private key provider extension will use qatlib library
(https://github.com/intel/qatlib) to accelerate RSA operations in
handshakes. The extension will look a bit like the existing cryptomb
private key provider. The use case is to move the expensive
cryptographic operations away from the CPU to the accelerator device,
leaving CPU cycles for other use.
Support for Intel® QAT is already present in the mainline Linux kernel
and in Kubernetes device plugins (to expose the device files to
containers). There are previous generations of Intel QAT® hardware
devices, but they are not supported by this extension.
Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>
Mirrored from https://github.com/envoyproxy/envoy @ 9447ff5bfa8081fc2ddd5918b8ee9c1fd6720c7a
This can be used when using the system resolver is desired. For
example, on Android.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 516b3f304bccf451691224c71a357f9c62840b2b
Creates a transport socket for HTP/1.1 proxy support.
With the combination of the transport socket, and a filter putting the proxy stream info in place this will
redirect TCP connections to the proxy IP address
prefix TLS connections with cleartext CONECT headers to the destination host, and strip CONNECT response
change cleartext HTTP/1.1 to send fully qualified URLs
Risk Level: medium (intended as a no op but it does have data plane refactory)
Testing: new unit, integration tests
Docs Changes: n/a
Part of envoyproxy/envoy-mobile#1622
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 095f4ca336d3d705e629b207fb2cbbc22d29db8f
Signed-off-by: Sergii Tkachenko <sergiitk@google.com>
Co-authored-by: Yan Avlasov <yavlasov@google.com>
Co-authored-by: Matt Klein <mattklein123@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 259d76ef89bc008db6620497c424a3ca3fe97d88
Implements a special transport socket for transferring state (metadata and filter state) over the internal connection.
This transport socket captures a subset of endpoint metadata, cluster metadata, and stream filter state in the user space socket. When an internal listener accepts a user space socket connection, it immediately merges this passthrough state into the connection stream info. Because the state can be transferred from HTTP stream to TCP stream, this transport socket also participates in the hashing decisions in the HTTP connection pools.
Commit Message: Add passhtrough state over internal connection.
Risk Level: low, new extension
Testing: WIP
Docs Changes: yes
Release Notes: yes
Platform Specific Features:
Fixes: #19274
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 637a92a56e2739b5f78441c337171968f18b46ee
Add support for injecting regex engine with its options on startup in a bootstrap option. These are some API, runtime and implementation changes.
* google_re2 in safe_regex will be deprecated and no longer be required. Regex::parseRegex will choose the registered regex engine to parse expressions into matchers.
* A new bootstrap option default_regex_engine will be introduced for regex engine selection.
* For compatibility, GoogleRE2 will be chosen as default regex engine if no regex engine is designated.
Signed-off-by: Xie Zhihao <zhihao.xie@intel.com>
Mirrored from https://github.com/envoyproxy/envoy @ 0a92cc86e7f8b88d5af0eb2e27b5c7ef64719e56
API for Unified Header Validators
Signed-off-by: Yan Avlasov <yavlasov@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e569ce06101932cfe3ccb710c3c60b92dd09464b
udp: Add a udp_packet_packet_writer_config option to UdpListenerConfig.
Fixes#21200
Risk Level: Low
Testing: Existing and additional tests
Docs Changes: None
Release Notes: Added
Platform Specific Features: N/A
Signed-off-by: Ryan Hamilton <rch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 3a7b26aa3909488a8561a232b82845b8b4cd38bb
Commit Message: make HTTP/3 upstream sends 0-RTT (early data) requests if it has cached 0-RTT credentials. Add a config knob in RouteAction to specify which request can be sent over early data, which by default are HTTP safe methods.
Risk Level: high, changes to conn pool behavior though should only take effect for h3 pool
Testing: added h3 upstream integration tests.
Docs Changes: N/A
Release Notes: changes to docs/root/version_history/current.rst
Platform Specific Features: N/A
Runtime guard: envoy.reloadable_features.http3_sends_early_data
Fixes#18715, #19542
Signed-off-by: Dan Zhang <danzh@google.com>
Signed-off-by: Dan Zhang <danzh@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8ce13d75a982ddd347db5a333a4bb080922f7514
* Adding custom config-validation interface, extension, and a CDS validator.
Signed-off-by: Adi Suissa-Peleg <adip@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e58c890ded5c13e60eba3285c96ffefd6f7215cf
Introduce data inputs for connection matching as part of #18871
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ a1181348a4f6df6011843a141814e28840d23eb5
Replace code in mutation_utils.cc with code that calls into the new
mutation_rules library. Add the ability to keep a counter of failed
header mutations, and also add the ability for an invalid mutation
to cause the HTTP request to fail.
Signed-off-by: Gregory Brail <gregbrail@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 1bea8aea1590ffab5bb185d32232f9bf265e6b1e
Risk Level: Medium
Testing: unit test & manual testing
Docs Changes: update sip proxy docs
Release Notes:
Platform Specific Features:
Signed-off-by: Mingling Ding <mingling.ding01@gmail.com>
Co-authored-by: Felix Du <durd07@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7174c148a0763f2ed5863c3ab341d4a9ce01c54b
Introduce a new stateful session extension. This is a change after #17848#17290.
In #17290, we added a new cross-priority host map for fast host searching. In #17848, we extend `LoadBalancerContext` interface to provide an override host and to select the upstream host by the override host.
Finally, in this change, we expand a new API to allow users to extract the state of the session from the request and change the result of load balancing by setting the override host value.
Related doc: https://docs.google.com/document/d/1IU4b76AgOXijNa4sew1gfBfSiOMbZNiEt5Dhis8QpYg/edit?usp=sharing.
Risk Level: Medium
Testing: Added
Docs Changes: Added
Release Notes: Added
Platform-Specific Features: N/A.
Signed-off-by: wbpcode <wbphub@live.com>
Mirrored from https://github.com/envoyproxy/envoy @ fb2aad0303bbbcb0daabf6940c6ad97252995b98
This change introduces a MetaProtocol proxy that provides a framework for layer-7 protocols. The common capabilities, including routing, tracing, metrics, logging, etc., will be built into the MetaProtocol proxy.
Signed-off-by: zhaohuabing <zhaohuabing@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ de51441bfe1b6b5eeab8981037d66cde9ac5b0d7
This PR establishes the ability to filter access log production via CEL expressions over the set of Envoy attributes. This can simply the creation of Envoy access log filters, allowing complex tailoring.
Risk Level: low
Testing: unit
Docs Changes: included
Release Notes: updated
Signed-off-by: Douglas Reid <douglas-reid@users.noreply.github.com>
Mirrored from https://github.com/envoyproxy/envoy @ 77ca6cc0d9aaf0892aec3e2025fe2ad7cf0c39ff
This should reduce the binary size, which is particularly important for Envoy Mobile. Looking at a
local opt build with debug symbols, I'm seeing a drop from ~400MB to ~380MB, so maybe 5% saving. @Reflejo indicates that optimized Envoy Mobile without symbols is observing ~20% improvement.
Related to #10943
Risk level: Low
Testing: bazel query deps to confirm no more v2 API deps.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 53fca618e47702f6c8dbc323db9bd39d15725457
This adds a new transport socket wrapper for collecting OS-level TCP stats and publishing them as envoy metrics, in either the cluster or listener stats namespace (depending on upstream or downstream use). This allows attributing the TCP stats to specific listeners and clusters.
Signed-off-by: Greg Greenway <ggreenway@apple.com>
Mirrored from https://github.com/envoyproxy/envoy @ b959a46661ec87bba8268a1750506695125f2b06
Based on the dicsussion in the issue https://github.com/envoyproxy/envoy/issues/17410, this PR adds
implementation for ability to add rbac policy to filter upstream ip addresses for dynamically
resolved domains (by dynamic proxy filter).
Signed-off-by: Jojy George Varghese <jojy_varghese@apple.com>
Mirrored from https://github.com/envoyproxy/envoy @ 53f95570a7e9d7fa5414a4bc145852524fa017ed
Intel's IPP (Integrated Performance Primitives) crypto library has support for multi-buffer crypto operations. Briefly, multi-buffer
cryptography is implemented with AVX-512 instructions using a SIMD (single instruction, multiple data) mechanism. Up to eight RSA or ECDSA operations are gathered together into a buffer and processed at the same time, providing potentially improved performance. The AVX-512 instructions are available on recently launched 3rd generation Xeon Scalable server processors (Ice Lake server) processors.
This commit adds a private key provider to accelerate RSA and ECDSA crypto operations on recent Intel Xeon processors. Every worker thread has a queue of up-to-eight crypto operations. When the queue is full or when the timer is triggered, the queue is processed and all the pending handshakes are notified.
The potential performance benefit depends on many factors: the size of the cpuset Envoy is running on, incoming traffic pattern, encryption type (RSA or ECDSA), and key size. In my own testing I saw the biggest performance increase when long RSA keys were used on an Envoy running in a fairly limited environment serving lots of new incoming TLS requests. For more details, see this Intel whitepaper which contains some more information about the AVX-512 instructions and potential performance increase: https://www.intel.com/content/www/us/en/architecture-and-technology/crypto-acceleration-in-xeon-scalable-processors-wp.html
Additional Description:
One new dependency is introduced: Intel’s ipp-crypto library. Currently the PR is using a development version of ipp-crypto because BoringSSL support is not yet part of any release. The ipp-crypto team has indicated that BoringSSL version will be included in future ipp-crypto releases.
Basic tests are provided, and a fake library interface is included for testing on systems without the required AVX-512 instruction set.
Risk Level: Medium (TLS security feature, not enabled by default)
Testing: Unit tests
Docs Changes: API interface is documented
Release Notes: Added CryptoMB private key provider to contrib.
Platform Specific Features: Requires Intel 3rd generation Xeon Scalable server processor for the AVX-512 IFMA instruction set.
Fixes: #15871
Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>
Co-authored-by: Greg Greenway <ggreenway@apple.com>
Mirrored from https://github.com/envoyproxy/envoy @ 2144166ca7a3f100ecae16700bc82920b2de4871
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 0c8942dd82e3991d5162c4c53ec5b2e5c1d71010
This PR is a continuation to #17515 - it adds a script that uses buf to check for breaking changes on proto files in the api folder. It does so by comparing the current api folder against the api folder at the git commit computed by tools/git/last_github_commit.sh - that should hopefully represent the most recent commit on main (if there is a better method to obtain the base branch commit, let me know!).
Adding the script also required re-organizing some of the breaking change detector logic from the previous pr: some levels of abstraction were added, and the detector now expects a git repository and ref as the input for initial state (rather than a proto file).
The script is invoked in do_ci.sh if bazel.api_compat is specified as the CI_TARGET.
This PR also bumps the buf bazel dependency to 0.53.0. If this is preferred to be in a separate PR, let me know and I would be happy to do so
Risk Level: low (hopefully) - the CI script will be invoked in a separate CI pipeline job that can be set to be optional on github. The azure pipeline has been added but needs to be set to optional by a CI maintainer
Testing: New scripts and logic were tested manually; also ran tests from the previous PR and they still pass. hoping to observe more output of this tool through reading CI logs of other PRs once this is merged (this PR should not affect the existing PR workflow - refer to Risk Level)
Docs Changes: n/a
Release Notes: n/a
Platform Specific Features: CI script uses a linux binary for buf so it cannot be run outside of docker on non-linux systems
Fixes#3368
Signed-off-by: Yaseen Alkhafaji <yaseena@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ f30c289e456df11d549d7236c56f22da00500e9e
data-plane-api(Azure Pipelines)