This patch moves away from the paradigm of sed-style upgrading of every v2
package to v3alpha. Instead, an additional type analysis phase is
performed prior to protoxform by a protoc plugin known as the "type
whisperer".
The type whisperer produces structured type dependency information for
each .proto. The tools/type_whisperer/typedb_gen.py tool then knits
these together to provide an API-wide type dependency graph. This is
then used to determine which types need upgrading (either they have
breaking changes or transitively depend on types with such changes).
Only packages with upgraded types now undergo the v2 -> v3alpha
transition.
The API type database is checked into
source/common/config/api_type_db.pb. This may seem a strange location,
but in the future we will include the type database as a build artifact
for the Envoy binary, as it will be used by the reflection-based version
converter to find the type upgrade path for input proto.
Risk level: Low (the v3alpha protos are not used yet).
Testing: fix_format, manual inspection of diffs, bazel test //test/...,
docs build.
Part of #8082Fixes#8490
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ ad57b58cfbb256af41a467260dce2a8013b7a7fa
As part of #8082, we want to be able to (1) automatically generate BUILD
files and (2) treat packages as atomic from a "upgrade / do not upgrade"
decision perspective. This is simplified by having our BUILD targets at
package granularity, since this is what the protoxform plugin operates
on.
This PR broadens the package-level treatment that was already introduced
for Go in #8003 to Python and C++. This simplifies BUILD files
significantly and opens the way to automated generation.
There is some technical debt introduced, since all visibility controls
have been removed. This is slated for reintroduction in
https://github.com/envoyproxy/envoy/issues/8491.
As a bonus (useful for BUILD file generation), also removed the
inconsistency in BUILD package target naming for packages in envoy.api.*
and envoy.type.*. E.g. //envoy/api/v2:v2 is now //envoy/api/v2:pkg.
Risk level: Low (but this will break internal builds and require BUILD
fixups to consuming projects).
Testing: bazel test //test/... @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4e858f17fe08224c9c089240908ccd0c518e01a7
These were missed in #8125.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 1b3b4ae1180b67bee6395fab5c075896fb1964ec
This avoids having to list new docs protos in both docs/build.sh and
api/docs/BUILD. This technical debt cleanup is helpful in v3 proto work
to simplify collecting proto artifacts from a Bazel aspect.
Risk level: Low
Testing: docs/build.sh, visual inspection of docs.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ b06e2b5c473b4b88e23ed66abd501e29f6bddbee
The libstdc++ std::regex implementation is not safe in all cases
for user provided input. This change deprecates the used of std::regex
in all user facing paths and introduces a new safe regex matcher with
an explicitly configurable engine, right now limited to Google's re2
regex engine. This is not a drop in replacement for std::regex as all
language features are not supported. As such we will go through a
deprecation period for the old regex engine.
Fixes https://github.com/envoyproxy/envoy/issues/7728
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ eff020170c6267e6c8dc235473f7fc85c5b1e07d
This allows using Envoy as a generic HTTP proxy without any
prior configuration of DNS targets. See the included documentation
for more information.
Part of https://github.com/envoyproxy/envoy/issues/1606
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 79e53f21c680b4a02695b1761e7d91868c3481d4
The current /listeners admin endpoint only returns a JSON-encoded list of addresses/ports. If a listener config binds to port 0, the endpoint will return the port that was assigned, but if there are multiple listeners binding to port 0, there's no way to tell which listener received which port.
This PR adds a proto that contains the listener's name and address. The /listeners endpoint returns this data in both text and JSON format, similar to the /clusters admin endpoint.
Risk Level: Low (according to #6959 the /listeners admin endpoint is only used for testing)
Testing:
I updated integration_admin_test.cc to test the new output formats for the /listeners endpoint.
I also updated the tools/socket_passing.py script that's used for integration/hotrestart_test.
Docs Changes:
new inline docs in listeners.proto
updated the admin menu description of /listeners
Release Notes:
admin: /listener endpoint now returns :ref:listener.proto<api/envoy/admin/v2alpha/listeners.proto>
Fixes#6959
Signed-off-by: Kaisen Chen <kaisen@yelp.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4461f431112876932e0328f6af31de7cf34fadfc
Add an http filter to extract the original source on a per-http-request basis
Signed-off-by: Kyle Larose <kyle@agilicus.com>
Mirrored from https://github.com/envoyproxy/envoy @ 712a72a718969ef37eb05a57a7e11c94d5fe6253
1) Add request/response body tapping
2) Add buffered body limits (TBI for transport socket)
3) Add the JSON_BODY_AS_BYTES and JSON_BODY_AS_STRING output
formats for convenience when the body is known to be human
readable.
4) Add JSON output for the file per tap sink.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 9a06dc0777d2809195cb1fc414b05ae7c0660193
This commit refactors the tap transport socket to use the common
tap extension configuration and tap matching infrastructure. More
match conditions will be added in a future PR as well as additional
cleanups that have been marked with TODOs.
One result of this PR is that the HTTP tap filter can now have a static
configuration as well as write to a file per tap sink.
All future tap PRs should be smaller and more targeted after this one.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ f37ebdc14f4c0adf0e90aabddae833355c0cec1b
This is a rename PR only. It renames the capture transport socket
and associated tools to the tap transport socket. It also updates
some documentation. In a subsequent PR I'm going to refactor the
tap transport socket to use the new common tap framework so that
the tap transport socket can be configured via admin, the HTTP
tap filter can write to a file, the tap transport socket can have
matching, etc.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7a5849f2a8bcc55fa16da3eaee94d9c99a11147c
This is a MVP for the HTTP tap filter. It includes minimal
infrastructure for the following:
1. Generic tap configuration which in the future will be used for
static config, XDS config, etc. In this MVP the tap can be
configured via a /tap admin endpoint.
2. Generic output configuration which in the future will be used for
different output sinks such as files, gRPC API, etc. In this MVP
the tap results are streamed back out the /tap admin endpoint.
3. Matching infrastructure. In this MVP only matching on request and
response headers are implemented. Both logical AND and logical OR
matches are possible.
4. In this MVP request/response body is not considered at all.
5. All docs are included and with all the caveats the filter is ready
to use for the limited cases it supports (which are likely still to
be useful).
There is a lot of follow on work which I will do in subsequent PRs.
This includes:
1. Merging the existing capture transport socket into this framework.
2. Implementing body support, both for matching on body contents as
well as outputting body data.
3. Tap rate limiting so too many streams do not get tapped.
4. gRPC matching. Using reflection and loaded proto definitions, it will
be possible to match on gRPC fields.
5. JSON matching. If the body parses as JSON, we can allow matching on
JSON fields.
Part of https://github.com/envoyproxy/envoy/issues/1413.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ cf80045587240d494e54e9772949bc9af5eda61f
We no longer allow running v1 config, and users can look at old
docs if they want to see the v1 docs still.
Also fix a doc issue in the /server_info admin endpoint.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8aba108eda816ea29dc80b973a1387257cd209c6
Introduces a rate limiter filter for thrift requests, implemented as a
decoder filter that works in conjunction with the thrift router filter.
Risk Level: low
Testing: unit tests
Doc Changes: inline with proto definitions
Release notes: updated
Signed-off-by: Stephan Zuercher <stephan@turbinelabs.io>
Mirrored from https://github.com/envoyproxy/envoy @ 6ac936f2750c39a8b4fb232d6ddc4802f4e6aeee
*Description*: Adds support for displaying Subject Alternate Names as a comma separated list of string in `/certs` admin end point
*Risk Level*: Low
*Testing*: Automated
*Docs Changes*: Updated
*Release Notes*: Updated
Signed-off-by: Rama <rama.rao@salesforce.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7403314b0b255a89ab284b7ed0eead5a5f4aecfd
Description: While working on certs changes, I realized that memory proto doc links were missing. This PR adds them.
Risk Level: Low
Signed-off-by: Rama <rama.rao@salesforce.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7600419d74aaa0d52c5f56f1a2b35c7ce05a76a0
update doc for jwt_authn http filter
format config.proto comment for doc
add a new rst file: docs/root/configuration/http_filters/jwt_authn_filter.rst
Risk Level: None
Docs Changes: Yes
Signed-off-by: Wayne Zhang <qiwzhang@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ d101ae7decfdae148f53ea9d2220444c726dfbfd
This allows injecting a synthetic resource pressure from a file, primarily intended for use in integration tests to force envoy into an overloaded state.
Testing: unit tests
Docs Changes: docs updated to reference new resource monitor
Signed-off-by: Elisha Ziskind eziskind@google.com
Mirrored from https://github.com/envoyproxy/envoy @ 12957a29bd2627bc0bba921603bd5869a449be47
We use the new extension_protocol_options field on Cluster to allow clusters
to be configured with a transport and/or protocol. Downstream requests are
automatically translated to the upstream dialect and upstream responses are
translated back to the downstream's dialect.
Moves the TransportType and ProtocolType protobuf enums out of the
ThriftProxy message to allow their re-use in ThriftProtocolOptions.
*Risk Level*: low
*Testing*: integration test
*Docs Changes*: added thrift filter docs
*Release Notes*: n/a
Signed-off-by: Stephan Zuercher <stephan@turbinelabs.io>
Mirrored from https://github.com/envoyproxy/envoy @ c91625ed829a4ec4123bab8b4e6b223f67d88e4a
Added the /clusters?format=json admin endpoint along with a proto representation of /clusters.
Risk Level: Low
Testing: Added a unit test for the new format.
Docs Changes: Added a brief description on the admin docs and linked to the more detailed proto definition.
Release Notes: Added release notes.
Fixes#2020
Signed-off-by: Matt Rice <mattrice@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 64605338ef040a949c5ea205bdd472a8fe42306c
Add support for extracting dynamic metadata from requests. This can then
be used as static metadata would be used (e.g.: for subset load balancer
metadata matches, logging, etc).
Risk Level: Low
Testing: unit-test
Docs Changes: Basic docs.
Release Notes: N/A
Signed-off-by: Raul Gutierrez Segales <rgs@pinterest.com>
Mirrored from https://github.com/envoyproxy/envoy @ 827c0a548ab38d55debe00587ee27253786befad
Adds a file based grpc credentials extension. See issue #3392 for more details.
Risk Level: Low: extension for grpc credentials loaded by explicit configuration options
Testing: tests included in PR
Docs Changes: Inline docs via comments and proto docs
Release Notes: N/A
Fixes#3392
Signed-off-by: Michael Wozniak <wozz@koh.ms>
Mirrored from https://github.com/envoyproxy/envoy @ 230d2216fdd520a182dea9b5152522756853cd90
Fixes https://github.com/envoyproxy/envoy/issues/743
This is a general cleanup of all of the access logging documentation.
I have reorganized a bunch of things and hidden the various gRPC logging
fields that are not implemented yet.
I've also moved the existing tap protos into a new "output" directory. This
is the best name I could come up for cleanly separating output data that might
be stored outside of any service or configuration.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ c15019e79c832d9f0a09468affaadabc4be3e115
* tap/fuzz: transport socket extension for traffic capture.
This PR introduces a transport socket extension that wraps a given transport socket, interposes on its
plain text traffic and records it into a proto trace file on the filesystem. This can be used for a
number of purposes:
1. As a corpus for fuzzing the data plane.
2. Converted to PCAP using a soon-to-be-written utility, allowing existing tools such as Wireshark
to be used to decode L4/L7 protocol history in the trace. Essentially this lets us take advantage
of the PCAP ecosystem.
Relates to #1413 and #508.
Risk Level: Low (opt-in).
Testing: New SSL integration tests, demonstrating plain text intercept.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 6c7a91733469f76381487f9ca78bdece6825c8c9
data-plane-api(CircleCI)
Marcelo Juchem
Shakti Das
htuch
Guy Lichtman
Joshua Marantz