* accesslog: add fields to ALS proto
Signed-off-by: Lex Herbert <lex.herbert@gmail.com>
* update code to match proto changes
Signed-off-by: Lex Herbert <lex.herbert@gmail.com>
* address PR feedback
Signed-off-by: Lex Herbert <lex.herbert@gmail.com>
---------
Signed-off-by: Lex Herbert <lex.herbert@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ a565b6e68ab269540a16c03fa5a6183d0cb3094f
Hi, I would like to suggest five changes/additions to gRPC access loger and accesslog proto file. These changes mainly benefit using Envoy as a TLS transparent proxy. I am not an experienced c++ developer and any feedback is welcome.
The problem is explained here: [#22407](https://github.com/envoyproxy/envoy/issues/22407)
After digging into the code, the proposal is to make five modifications/additions to the gRPC logger. A detailed description per modification/addition is listed below.
1. Set tls_sni_hostname also when TLS connection is not terminated.
2. Add duration
3. Add upstream_request_attempt_count
4. Add connection_termination_details
5. Add ja3_fingerprint when ja3_fingerprinting is enabled
Details
1. Set tls_sni_hostname also when TLS connection is not terminated.
This enables it to log the requested hostname when the tls listener filter is used, but the connection is simply forwarded to the upstream cluster. This is for example useful to see if envoy forwards to the correct cluster based on the requested hostname.
2. Add duration
The existing duration fields in the acceslog.proto are not present when using the tcp listener. The complete duration, based on stream_info.requestComplete(), however is available. A separate duration field is in line with the file access log.
3. Add upstream_request_attempt_count
Logs the number of times the connection request is attempted upstream. Note that the field is omitted when the connect request was never attempted upstream. This is particularly useful when the max_connect_attempts field is set in the tcp_proxy.
4. Add connection_termination_details
Connection termination details may provide additional information about why the connection was terminated by Envoy for L4 reasons. This is especiall usefull when network rbac envoy.filters.network.rbac . The connection termination details contains the info why a connection was blocked/allowed.
5. Add ja3_fingerprint when ja3_fingerprinting is enabled
The [tls inspector listner filter]( https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/listener/tls_inspector/v3/tls_inspector.proto#envoy-v3-api-msg-extensions-filters-listener-tls-inspector-v3-tlsinspector) has an option to generate ja3 fingerprint (default false). When available this logs the ja3 fingerprint. This field is omitted when the ja3 fingerprint is not available. The field is present when eiter envoy terminates the tls connection or acts as a transparent proxy.
Risk Level:
Low/Medium - four additional fields are added, no fields removed or changed (no behavior changes expected)
The only change is that tls_properties is also added when it is not a terminated tls connection. In this scenario, of the tls_properties only the tls_sni_hostname is available. The other tls_properties fields are not present. As far as I can see, this is consistent with the behavior of the tls listener filter.
This could be a breaking change if the receiving application of the access log message checks for the presence of the tls_properries and when present expect all the fields to be there, or assumes this must be a tls terminated connection.
Testing:
unit tests
Development build using mac os x (m1 if that info is relevant), but with ‘default’ extensions.
All tests pass.
manual testing
Using a sample grpc log service. Displaying accesslog stream using golang protojson.Marshal()
Tested
- Tcp with tls
- Tcp no tls
- Http no tls
Docs Changes:
I have added the documentation in the proto file [accesslog.proto](https://github.com/envoyproxy/envoy/blob/main/api/envoy/data/accesslog/v3/accesslog.proto) for data.accesslog.v3.AccessLogCommon, and data.accesslog.v3.TLSProperties.
Am I right in assuming that the documentation is generated from this file? If not, I have to modify the documentation.
Release Notes:
**access_log:** log `duration`, `upstream_request_attempt_count`, `connection_termination_details` and tls `ja3` field in the grpc access log and also log the tls `sni` and `ja3` field in the grpc access log when envoy is configured as a tls forward proxy.
Platform Specific Features: N/A
Issues:
Fixes: [#22407](https://github.com/envoyproxy/envoy/issues/22407)
[Optional API Considerations:]
ls_sni_hostname:
- See risk level above. The other option would be to add a requested_server_name field to the access_log_common properties. That would be present when either the tls connection is terminated or forwarded.
- The downside of this solution would be that this would create a duplicate field for tls_sni_hostname when envoy terminates the tls connection.
- The proposed solution: Filling tls_properties->tls_sni_hostame either when envoy terminates or forwards the tls connection is consistent with file access logging.
upstream_request_attempt_count
- In file logging this field is 0 when there is no upstream connection attempt made. I believe that omitting this field when there is no value is more in line with the grcp ALS service.
- In most scenarios the value of this field is 1. This field could also be adapted to a retry count. In that solution the field would be present only if the attempt > 1. This is however not consistent with file logging. The proposed solution is.
Signed-off-by: Niek Temme <niek@bubl.cloud>
Mirrored from https://github.com/envoyproxy/envoy @ 06625e63d1e780c5affe7938c9d7debe13c36caf
in #17645 there was a bunch of discussion around the DNS cache returning null addresses and how to handle it. After discussion on #19461 we agreed to keep sending null updates, but to fast-fail if no address was resolved.
Risk Level: Medium (data plane change)
Testing: updated integration tests, unit tests
Docs Changes: n/a
Release Notes: inline
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 2efe480d6b0a1501674780cfd7c343ef332010a1
This should reduce the binary size, which is particularly important for Envoy Mobile. Looking at a
local opt build with debug symbols, I'm seeing a drop from ~400MB to ~380MB, so maybe 5% saving. @Reflejo indicates that optimized Envoy Mobile without symbols is observing ~20% improvement.
Related to #10943
Risk level: Low
Testing: bazel query deps to confirm no more v2 API deps.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 53fca618e47702f6c8dbc323db9bd39d15725457
As of today no route found and route is found but the cluster in that route is not ready are sharing the same error detail.
However, this make big difference. The latter is likely a RDS config problem, or explicit blackhole.
The latter might be a slow CDS.
Address #15448
Signed-off-by: Yuchen Dai <silentdai@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ a11d48368c62eb2c238d6d3087c16ffa355243d1
Commit Message: Added max_connection_duration for tcp_proxy
Additional Description: Added max_connection_duration for downstream connection. After reaching max_connection_duration the connection will be closed.
Risk Level:
Testing:
Docs Changes:
Release Notes:
includes partial fix to #12077
Signed-off-by: Manish Kumar <manish.kumar1@india.nec.com>
Mirrored from https://github.com/envoyproxy/envoy @ 84f538f1ddcb5c16d98627df8b5e8c38f0f26720
Changing the HCM to not adjust the date header when serving (side-car) cached responses.
This includes adding a status flag for cached responses which propagates to access logs etc, as well as some test cleanup to support polling for multiple log entries.
Risk Level: Low (only affects pre-alpha cache filter)
Testing: new UT, enhanced IT
Docs Changes: n/a
Release Notes: n/a
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ bb74a91ac413d81f4b1874ed6474df93e4ca2782
To resolve#10274, adding max stream duration for upstream connection.
Signed-off-by: shikugawa <rei@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 6151a69f9c0dc4aa7938d987036ec00eedb818d5
This patch performs a major version freeze and bump by modifying
package_version_status, using the tooling developed in #10636.
Specifically:
v2 APIs are frozen (except for where they are the latest in their
package history and still active)
v3 APIs are transitioned to be active
Candidate v4alpha APIs are generated (not used by Envoy yet)
Fixes#10355
Risk level: medium (entire API's files are modified, visually verified
to ensure things look sane, all tests pass)
Testing: CI
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 549164c42cae84b59154ca4c36009e408aa10b52
Part of https://github.com/envoyproxy/envoy/issues/10355, this patch introduces additional
annotations to the API to support automatic inference (and developer documentation) of where the
active developer editable version of a file is, and which files are frozen or machine generated.
Risk level: Low (API annotations only)
Testing: CI
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4c5f4310bb8a19a38e5377d7a2d2dc6aa4560f47
In which we convert every v3alpha reference to v3. In future revs of the
stable API versioning policy, we will develop better tooling to support
> 2 alpha and stable versions. For v3, it seems reasonable to just mv
v3alpha to v3, since there should be no external consumers yet.
Risk level: Low
Testing: bazel test //test/..., CI.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 5248a4fb7d4c2a3d1fa151f944d3a63f6b7a06cf
Description:
Move packages around for #8120 and #8121
Risk Level: Med around messing up build.
Testing: CI
Docs Changes: in API/STYLE.md
Release Notes: N/A (v3alpha is not in use yet)
Fixes#8120
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 1371f2ef46582a72b5b3971147bd87c534011731
In order to get file level move annotation, import has to be before options.
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 062c895f499382ae61dead16db2a7e78b9146525
Instead of formatting options heuristically, which will erase new annotations without changing protoxform, use proto descriptor to format options, and enforce its order as well.
Risk Level: Low
Testing: CI
Docs Changes: N/A
Release Notes: N/A
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ dfe687d49574ef7eb1bf84867bf571e805a2bf97
* api: link to previous message type package in API BUILD files.
We need to include the descriptors from the previous message version in
the build. We opt to do this transitively; when you include v3 of a
package, you get the v2 via a transitive dep. This should work based on
alwayslink semantics for cc_library.
The computation of the deps is based on the previous_message_type
annotation, which will allow cross package migrations.
Part of #8082.
Risk level: Low
Testing: Disabled ip_tagging v2 descriptor hack, observed
version_integration_test. After the BUILD changes, this now passes
again.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7f8fb9509d3189819dd253e25ec76e939ae106e7
This PR avoids having to include an API type database in the Envoy build
by introducing a message annotation option that allows Envoy to
determine earlier corresponding message types via descriptor inspection.
The ApiTypeDb is now ApiTypeOracle and utilizes these annotations.
Risk level: Low
Testing: Existing API and verison upgrade tests pass.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 297f7a73b3f93bccf8af73c0a555ae52bce6cecb
Description:
Adds serialization method to filter state and use from logger if specified.
Risk Level: Low
Testing: CI
Docs Changes: Added
Release Notes: Added
Fixes#8790
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ cf74f816933d1350d7c588a3b8478dd399ce3d18
Fix PGV location references and TODOs. Remove some unused imports.
Risk Level: Low (locations in comments and simple TODOs as outlined by @htuch)
Testing: bazel build @envoy_api//envoy/..., bazel test //test/...
Signed-off-by: Michael Payne <michael@sooper.org>
Mirrored from https://github.com/envoyproxy/envoy @ 68ca6746d168c6cea26d21479ef50b2be9aaa25d
Generate or format next free field annotation via protoxform.
Risk Level: low
Testing: N/A
Docs Changes: N/A
Release Notes: N/A
Fixes#8429
Signed-off-by: Yi Tang <ssnailtang@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 986173ed516dcc1c3dea7db90659ed993d0aad75
This provides canonical BUILD formatting and puts protoxform in charge
of being able to determine import paths, without having to worry about
Bazel implications.
Part of #8082.
Risk level: Low
Testing: tools/proto_sync.py, visual inspection of diffs.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e53f40f0e5ccc84fca5cd350416fe0f2accf8229
As part of #8082, we want to be able to (1) automatically generate BUILD
files and (2) treat packages as atomic from a "upgrade / do not upgrade"
decision perspective. This is simplified by having our BUILD targets at
package granularity, since this is what the protoxform plugin operates
on.
This PR broadens the package-level treatment that was already introduced
for Go in #8003 to Python and C++. This simplifies BUILD files
significantly and opens the way to automated generation.
There is some technical debt introduced, since all visibility controls
have been removed. This is slated for reintroduction in
https://github.com/envoyproxy/envoy/issues/8491.
As a bonus (useful for BUILD file generation), also removed the
inconsistency in BUILD package target naming for packages in envoy.api.*
and envoy.type.*. E.g. //envoy/api/v2:v2 is now //envoy/api/v2:pkg.
Risk level: Low (but this will break internal builds and require BUILD
fixups to consuming projects).
Testing: bazel test //test/... @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4e858f17fe08224c9c089240908ccd0c518e01a7
Via ./api/migration/v3alpha.sh. This picks up the changes since the last
sync, in particular the major reformat in #8309.
Risk level: Low (not used yet).
Testing: bazel build @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ c41cfbf6a33b8115a7e29a2b4a926aad4cd062be
This patch introduces a new tool, protoxform, that will be the basis of
the v2 -> v3 migration tooling. It operates as a Python protoc plugin,
within the same framework as protodoc, and provides the ability to
operate on protoc AST input and generate proto output.
As a first step, the tool is applied reflexively on v2, and functions as
a formatting tool. In later patches, this will be added to
check_format/fix_format scripts and CI.
Part of #8082.
Risk level: medium (it's possible that some inadvertent wire changes
occur, if they do, this patch should be rolled back).
Testing: manual inspection of diff, bazel test //test/..., some
grep/diff scripts to ensure we haven't lost any comments.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 08b123a8321d359ea66cbbc0e2926545798dabd3
Remove gogoproto annotations. They can be replaced with a custom gogoproto compiler (e.g. something like https://github.com/gogo/googleapis/tree/master/protoc-gen-gogogoogleapis). I have an experimental version of it to validate that it's possible to re-apply important annotations in the compiler.
Risk Level: low
Testing: builds
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e7f0b7176efdc65f96eb1697b829d1e6187f4502
Fixes#7982
Defines a package level proto library and its associated internal go_proto_library.
Deletes all existing api_go_proto_library, api_go_grpc_library, and go_package annotations in protos (they are not required and pollute the sources).
I deliberately avoided touching anything under udpa since it's being moved to another repository.
Risk Level: low
Testing: build completes
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ d504fde0ffd97017d1ddff8caa9a3b46bba9ae48
This patch establishes a v3alpha baseline API, by doing a simple copy of
v2[alpha] dirs and some sed-style heuristic fixups of BUILD dependencies
and proto package namespaces.
The objective is provide a baseline which we can compare the output from
tooling described in #8083 in later PRs, providing smaller visual diffs.
The core philosophy of the API migration is that every step will be
captured in a script (at least until the last manual steps),
api/migration/v3alpha.sh. This script will capture deterministic
migration steps, allowing v2[alpha] to continue to be updated until we
finalize v3.
There is likely to be significant changes, e.g. in addition to the work
scoped for v3, we might want to reduce the amount of API churn by
referring back to v2 protos where it makes sense. This will be done via
tooling in later PRs.
Part of #8083.
Risk level: Low
Testing: build @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 085d72b490c124a02849812798f5513a8df9ae72
Description:
Before this change, Envoy would silently ignore the `x-envoy-*` header if a
client specifies an invalid value for this header (e.g. `x-envoy-max-retries: 3.0`).
Introduce a `strict_check_headers` config option for `envoy.router` that adds
optional support to reject requests with invalid values for the following headers:
- x-envoy-upstream-rq-timeout-ms
- x-envoy-upstream-rq-per-try-timeout-ms
- x-envoy-max-retries
- x-envoy-retry-on
- x-envoy-retry-grpc-on
On rejection, Envoy responds with HTTP status 400 and sets a new response flag
`IH` to indicate the reason was due to an invalid header.
Risk Level: Low/medium
Testing: unit tests
- unit test: `FilterUtility::StrictHeaderChecker`
- test that router rejects request with HTTP status 400 + setting the `IH` response flag
- test that config validation rejects unsupported values
- manual end-to-end test `client -> envoy -> upstream server` to verify that
Envoy returns a 400 and sets the response flag in the logs
Docs Changes:
- add inline docs to `router.proto` for `strict_check_headers`
- add inline docs to `accesslog.proto` for `IH` response flag
Release Notes: updated for router and accesslog
Fixes#6482
Signed-off-by: Xiao Yu <xyu@stripe.com>
Mirrored from https://github.com/envoyproxy/envoy @ ecd03a4eed07e1cfea9e9844e519b7fffada437a
The TLS session ID is now accessible as a method on the Ssl::ConnectionInfo object.
Add a new %DOWNSTREAM_TLS_SESSION_ID% directive to the file access logger, and a
tls_session_id field to the gRPC access logger. Update unit tests to validate the
new access log fields.
Signed-off-by: Mike Grass <mgrass@salesforce.com>
Mirrored from https://github.com/envoyproxy/envoy @ cbd8d57e9fc2921d26236db24b6bf91246ba3b4f
Passes along the Ssl::ConnectionInfo (renamed from Ssl::Connection)
to StreamInfo, allowing it to be read in access loggers. Updates both
the formatter and gRPC access logger with uri san/subject details,
and adds the SNI to the gRPC access log.
Risk Level: Low
Testing: Unit tests
Docs Changes: Added description for new access log formats
Release Notes: n/a
#4926
Signed-off-by: Snow Pettersen <snowp@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ 65c05cad96db7f7874e1e5a2b70829dcfad10e88
Using proto.MarshalAny results in unstable output due to non-deterministic map ordering. This in turn causes Envoy's diff to reload a config since the hash of the structure changes.
Enable stable marshaler for gogoproto to avoid this problem. See #6252
Risk Level: low
Testing: n/a
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 15a19b9cb1cc8bd5a5ec71d125177b3f6c9a3cf5
data-plane-api(Azure Pipelines)