Hi, I would like to suggest five changes/additions to gRPC access loger and accesslog proto file. These changes mainly benefit using Envoy as a TLS transparent proxy. I am not an experienced c++ developer and any feedback is welcome.
The problem is explained here: [#22407](https://github.com/envoyproxy/envoy/issues/22407)
After digging into the code, the proposal is to make five modifications/additions to the gRPC logger. A detailed description per modification/addition is listed below.
1. Set tls_sni_hostname also when TLS connection is not terminated.
2. Add duration
3. Add upstream_request_attempt_count
4. Add connection_termination_details
5. Add ja3_fingerprint when ja3_fingerprinting is enabled
Details
1. Set tls_sni_hostname also when TLS connection is not terminated.
This enables it to log the requested hostname when the tls listener filter is used, but the connection is simply forwarded to the upstream cluster. This is for example useful to see if envoy forwards to the correct cluster based on the requested hostname.
2. Add duration
The existing duration fields in the acceslog.proto are not present when using the tcp listener. The complete duration, based on stream_info.requestComplete(), however is available. A separate duration field is in line with the file access log.
3. Add upstream_request_attempt_count
Logs the number of times the connection request is attempted upstream. Note that the field is omitted when the connect request was never attempted upstream. This is particularly useful when the max_connect_attempts field is set in the tcp_proxy.
4. Add connection_termination_details
Connection termination details may provide additional information about why the connection was terminated by Envoy for L4 reasons. This is especiall usefull when network rbac envoy.filters.network.rbac . The connection termination details contains the info why a connection was blocked/allowed.
5. Add ja3_fingerprint when ja3_fingerprinting is enabled
The [tls inspector listner filter]( https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/listener/tls_inspector/v3/tls_inspector.proto#envoy-v3-api-msg-extensions-filters-listener-tls-inspector-v3-tlsinspector) has an option to generate ja3 fingerprint (default false). When available this logs the ja3 fingerprint. This field is omitted when the ja3 fingerprint is not available. The field is present when eiter envoy terminates the tls connection or acts as a transparent proxy.
Risk Level:
Low/Medium - four additional fields are added, no fields removed or changed (no behavior changes expected)
The only change is that tls_properties is also added when it is not a terminated tls connection. In this scenario, of the tls_properties only the tls_sni_hostname is available. The other tls_properties fields are not present. As far as I can see, this is consistent with the behavior of the tls listener filter.
This could be a breaking change if the receiving application of the access log message checks for the presence of the tls_properries and when present expect all the fields to be there, or assumes this must be a tls terminated connection.
Testing:
unit tests
Development build using mac os x (m1 if that info is relevant), but with ‘default’ extensions.
All tests pass.
manual testing
Using a sample grpc log service. Displaying accesslog stream using golang protojson.Marshal()
Tested
- Tcp with tls
- Tcp no tls
- Http no tls
Docs Changes:
I have added the documentation in the proto file [accesslog.proto](https://github.com/envoyproxy/envoy/blob/main/api/envoy/data/accesslog/v3/accesslog.proto) for data.accesslog.v3.AccessLogCommon, and data.accesslog.v3.TLSProperties.
Am I right in assuming that the documentation is generated from this file? If not, I have to modify the documentation.
Release Notes:
**access_log:** log `duration`, `upstream_request_attempt_count`, `connection_termination_details` and tls `ja3` field in the grpc access log and also log the tls `sni` and `ja3` field in the grpc access log when envoy is configured as a tls forward proxy.
Platform Specific Features: N/A
Issues:
Fixes: [#22407](https://github.com/envoyproxy/envoy/issues/22407)
[Optional API Considerations:]
ls_sni_hostname:
- See risk level above. The other option would be to add a requested_server_name field to the access_log_common properties. That would be present when either the tls connection is terminated or forwarded.
- The downside of this solution would be that this would create a duplicate field for tls_sni_hostname when envoy terminates the tls connection.
- The proposed solution: Filling tls_properties->tls_sni_hostame either when envoy terminates or forwards the tls connection is consistent with file access logging.
upstream_request_attempt_count
- In file logging this field is 0 when there is no upstream connection attempt made. I believe that omitting this field when there is no value is more in line with the grcp ALS service.
- In most scenarios the value of this field is 1. This field could also be adapted to a retry count. In that solution the field would be present only if the attempt > 1. This is however not consistent with file logging. The proposed solution is.
Signed-off-by: Niek Temme <niek@bubl.cloud>
Mirrored from https://github.com/envoyproxy/envoy @ 06625e63d1e780c5affe7938c9d7debe13c36caf
Risk Level: N/A
Testing: N/A
Docs Changes: This is a small docs change
Signed-off-by: Dhi Aurrahman <dio@rockybars.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4c0e53d8cee46d9d886ceed011b1a52000d261cf
This PR will implement issue detailed here and described below: #7763
Match Patterns and Templates
Wildcard support based on match patterns and templates.
A match pattern matches an incoming URL path.
Match patterns support glob operators to match URL text and variable definitions to bind matched text to names.
Template patterns are used to re-write URLs.
Template patterns build new URLs and may reference variables bound by a match pattern.
Match Examples
/**.m3u8 would match /foo.m3u8 and /foo/bar.m3u8.
/{dir_name}/*.ts would match /example/file.ts and bind dir_name="example" for a later template match to use.
/{dir_name}/**.ts would match /example/path/file.ts and bind dir_name="example" for a later template match to use. This would also match /example/.ts, which may or may not be a desired behavior.
/{path=v1/*}/{file=*.ts} would match /v1/example/movie.ts (binding path="v1/example" and file="movie"), but would not match /v0/example/movie.ts.
See post for full details and example:
#7763 (comment)
Risk Level:
Testing:
Unit tests. Both both internal matching/rewrite library and config/data plane changes.
Signed-off-by: silverstar195 <seanmaloney@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8cfc61f916cf52ce8bce6710686e9d4fca2c06bd
* initial draft for the meta protocol proxy
Signed-off-by: wbpcode <comems@msn.com>
* minor update
Signed-off-by: wbpcode <comems@msn.com>
* add match implemented
Signed-off-by: wbpcode <comems@msn.com>
* add some simple test
Signed-off-by: wbpcode <wbphub@live.com>
* add more test for route matcher
Signed-off-by: wbpcode <wbphub@live.com>
* partial commit
Signed-off-by: wbpcode <wbphub@live.com>
* complete basic unit test
Signed-off-by: wbpcode <comems@msn.com>
* fix format
Signed-off-by: wbpcode <comems@msn.com>
* fix error after merge
Signed-off-by: wbpcode <wbphub@live.com>
* add some more test
Signed-off-by: wbpcode <wbphub@live.com>
* minor fix
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix test
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* just make it run
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* just make it run
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* first integration test
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* complete almost all the tests and fix docs and format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix proto format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* minor update
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* add cleanup
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* update for clang tidy and type error
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* lower coverage threshould
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix unexpected include
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix window build
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* rename to generic proxy to avoid name conflict
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix docs
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* remove unnecessary readme
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* update comments and name of matcher
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* add name method
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* move everything to contrib
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* simple release note
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
* fix format
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <comems@msn.com>
Signed-off-by: wbpcode <wbphub@live.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Mirrored from https://github.com/envoyproxy/envoy @ 51c0d6f47c98087c7e3288205cbf8edf50ae0196
Signed-off-by: John Howard <howardjohn@google.com>
Co-authored-by: phlax <phlax@users.noreply.github.com>
Mirrored from https://github.com/envoyproxy/envoy @ 466e78586afaeb8ecb5f92351cd5ffdee3871f49
The data is already available in the ClusterInfo class. (getter function; data generation)
Risk Level: Low
Testing: Unit Test
Fixes#22903
Signed-off-by: Alex Ding <39857214+nmdzl@users.noreply.github.com>
Mirrored from https://github.com/envoyproxy/envoy @ b5b291526288b3b6914cda436de1583b68550b58
This changes the Upstream Filter code to creating the codec filter via factory which
requires adding all UpstreamRequest/CodecFilter interactions to an UpstreamCallback interface accessible through the filter manager
requires unhiding the configuration because the presubmit cross-checks for the registered factory require all the config be unhidden
allows configuring the codec filter in configuration, in case of eventual alternate terminal filter
allows fixing up the filter dependency validator to validate the entire upstream filter chain.
Upstream filters flipped on for CI, SHOULD BE FLIPPED OFF BEFORE SUBMITTING
Risk Level: low assuming it's flipped back off
Testing: updated unit tests, covered by filter integration tests
Docs Changes: n/a
Release Notes: n/a (off by default)
Part of #10455
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 978004308203aa9e86886512adf6b26376958405
This updates the version of buf.build/googleapis/googleapis to the
latest pushed version. The BSR had a change in the googleapis module to
to concerns with the number of files in BSR, requiring previous commits
to be removed. See https://docs.buf.build/faq#googleapis-failure for
more details. This PR fixes the issue for envoy.
Signed-off-by: bufdev <bufdev-github@buf.build>
Signed-off-by: bufdev <bufdev-github@buf.build>
Mirrored from https://github.com/envoyproxy/envoy @ 68758cef311f65e8813df6b7235ead660cd952f8
Signed-off-by: Marcin Falkowski <marcin.falkowski@allegro.pl>
Mirrored from https://github.com/envoyproxy/envoy @ 2bee66e427f26680f799643ba9ce82ffd5541895
The HeaderAppendAction was added in the #18246. But the implementation never done. This PR try to complete this feature.
Further work of #18246. To close#22713.
Risk Level: Low.
Testing: Unit Test.
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Mirrored from https://github.com/envoyproxy/envoy @ 5183dbf7f11b23f1f7013a3e42cbce71eb63252a
Currently stats are not available in lua filter, which makes it hard to
track/alert for the script errors. This change adds error stat.
Signed-off-by: Suresh Kumar <sureshkumar.pp@gmail.com>
Signed-off-by: Suresh Kumar <suresh.ponnusamy@freshworks.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8c88943b747bd74517eaf257cd75dd686aa7bbb8
Signed-off-by: Kuat Yessenov <kuat@google.com>
Commit Message: Expose internal listener functionality. Marks the fields as implemented. Adds the following functional changes prior to publishing:
* do not use the `address` field per https://github.com/envoyproxy/envoy/pull/21132, and instead use the listener `name` field as originally expected.
* add an `endpoint_id` field to upstream address to fix https://github.com/envoyproxy/envoy/pull/22420; it's not possible to have multiple internal addresses to the same listener in a single pool without some extra identifier beyond the listener name, because pools identify addresses using `asString()`.
* fixes https://github.com/envoyproxy/envoy/issues/20665 by using stat prefix `listener.envoy_internal_<listener_name>`.
Adds a few sample configs (and fixes https://github.com/envoyproxy/envoy/issues/20335):
* minimal two chained `tcp_proxy` listeners;
* encap HTTP in HTTP CONNECT;
* decap HTTP in HTTP CONNECT;
Risk Level: low, hidden prior
Testing: yes
Docs Changes: yes
Release Notes: yes
Mirrored from https://github.com/envoyproxy/envoy @ 02489bbf8990faddb6be1f2e0f22851b64c3fc85
adding hidden config for upstream filters, and setting the cluster up to create upstream filters.
upstream filters are as yet unused, still cleaning up (#22434) but this makes it a much cleaner PR
(Also adding 2 random tweaks to router code because coverage was failing despite there previously being no changes to router code)
Risk Level: low (no-op if not configured)
Testing: unit tests
Docs Changes: n/a
Release Notes: n/a
part of #10455
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 5261b4285d1487f93e7aece94786de9a8d3a2529
Commit Message: contrib: add Hyperscan regex engine
Additional Description: Hyperscan has been introduced as an input matcher earlier this year. Since the regex engine interface has been completed, the patch extent the usage of Hyperscan into a contrib regex engine.
Risk Level: Low
Testing: Unit
Docs Changes: API
Release Notes: N/A
Platform Specific Features: Requires processor with SSSE3 support (nearly any modern x86 processor)
Signed-off-by: Xie Zhihao <zhihao.xie@intel.com>
Mirrored from https://github.com/envoyproxy/envoy @ 725d0b4b3e6fb84aa7dfd1ca4d5792f2dc4558b2
* cors: add PNA support
Signed-off-by: Loong Dai <loong.dai@intel.com>
* fix field
Signed-off-by: Loong Dai <loong.dai@intel.com>
* use runtime guard rather than api change
Signed-off-by: Loong Dai <loong.dai@intel.com>
* fix CI
Signed-off-by: Loong Dai <loong.dai@intel.com>
* update changelog
Signed-off-by: Loong Dai <loong.dai@intel.com>
* revert unrelated changes
Signed-off-by: Loong Dai <loong.dai@intel.com>
* update API
Signed-off-by: Loong Dai <loong.dai@intel.com>
* feedback and update tests
Signed-off-by: Loong Dai <loong.dai@intel.com>
* update tests
Signed-off-by: Loong Dai <loong.dai@intel.com>
* update tests
Signed-off-by: Loong Dai <loong.dai@intel.com>
* add tests
Signed-off-by: Loong Dai <loong.dai@intel.com>
* feedback
Signed-off-by: Loong Dai <loong.dai@intel.com>
* remove useless test
Signed-off-by: Loong Dai <loong.dai@intel.com>
* feedback
Signed-off-by: Loong Dai <loong.dai@intel.com>
* fix CI
Signed-off-by: Loong Dai <loong.dai@intel.com>
Mirrored from https://github.com/envoyproxy/envoy @ bf61b077e77197a28fefa5798902fd94008e4137
This PR is the first of a series to implement pattern matching and rewrite functionality for Envoy.
These specific proto changes:
Add the foundation for two extension configs
Change the API to expose the extension configs to end users
Risk Level: Low
Signed-off-by: silverstar195 <seanmaloney@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 25654b97cdd2647c258ae459e6607f51e99c99cc
Commit Message: The documentation in the protobuf comments for stats tag extractors has been stale since 2018. This corrects the documentation to match the current behavior.
Additional Description: This documents a change that was done in Feb 2018: #2515 -- obviously we should've fixed the doc then but it was overlooked.
Risk Level: low
Testing: none
Docs Changes: this is a doc change only, to reflect current reality
Release Notes: n/a
Platform Specific Features: n/a
Signed-off-by: Joshua Marantz <jmarantz@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 076bc515b3e6fca54592174928fc7ab06763b812
Commit Message: This PR entirely removes the Lightstep tracer. Lightstep is looking forward to the OpenTelemetry tracer integration.
Additional Description: The owner of this code left Lightstep and we have no plans to maintain this code now that OpenTelemetry is ready.
Risk Level: Low
Docs Changes:
Release Notes: Remove Lightstep tracer.
Co-authored-by: alyssawilk <alyssar@google.com>
Co-authored-by: alyssawilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ eb521f42e760b9e5cb6ca544e5c87523f0592fa9
Signed-off-by: Jacek Ewertowski <jacek.ewertowski1@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ c2dab844230197a1109e62f1d735384b67d74c9c
Next generation Intel® QAT support with Intel® Xeon® Scalable processors
will feature an Intel® QAT cryptography and compression acceleration
engine.
QAT private key provider extension will use qatlib library
(https://github.com/intel/qatlib) to accelerate RSA operations in
handshakes. The extension will look a bit like the existing cryptomb
private key provider. The use case is to move the expensive
cryptographic operations away from the CPU to the accelerator device,
leaving CPU cycles for other use.
Support for Intel® QAT is already present in the mainline Linux kernel
and in Kubernetes device plugins (to expose the device files to
containers). There are previous generations of Intel QAT® hardware
devices, but they are not supported by this extension.
Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>
Mirrored from https://github.com/envoyproxy/envoy @ 9447ff5bfa8081fc2ddd5918b8ee9c1fd6720c7a
data-plane-api(Azure Pipelines)