This patch introduces the tools/docs for a new flow for protoxform: freeze.
This performs a major version freeze and bump by modifying
package_version_status. Tooling now needs to deal with 3 possible versions (v2,
v3, v4alpha). The following contributions are made in the patch:
* Tooling added to support v2 APIs freeze, transitioning v3 to active and generating v4alpha.
* A new tool, merge_active_shadow.py, is introduced to combine hand-edited active v3 API protos with the v2 implied shadows for deprecated fields.
* The type database now spans v2 and v3 (and implied v4alpha files)
* api_proto_plugins can now take extra args on the Bazel CLI; this is used to plumb in the freeze arg to protoxform.
Docs are updated to reflect the v3 flow that developers should now
follow.
Part of #10355
This is the tools/docs component of #10601. See #10601 for the resulting
API protos generated by a freeze/fix cycle.
Risk level: Low (tooling only, no API changes yet)
Testing: new golden protos tests for protoxform, unit tests for merge_active_shadow.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ de222c3acf1fbf124ca44f50495fc4b8e9aba525
This commit is this base structure and api definition
for the DNS filter. The code itself takes no action
on packets. Tests will be added later.
Signed-off-by: Alvin Baptiste <alvinsb@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ b3949eaf2080809b8a3a6cf720eba2cfdf864472
* Automatically derive active protos target used in proto_format.sh from the packages with files
annotated as active.
* Replace some sites with hardcoded v2/v3 with active/next_major_version_candidate.
* Move //docs:BUILD to //versioning:BUILD in API tree. This has already ceased to be used directly
in the docs build, so this makes things a bit clearer.
Risk level: Low (API tooling and build only)
Testing: fix_format (mostly idempotent)
Part of #10355.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4403003b9c0888ce81606a2f45fcc1da126514e7
This filter transform HTTP requests to AWS Lambda invocations.
The filter supports pass-through only. Meaning, the request body
is passed to Lambda as is. Note: Lambda requires the request to be in
JSON format.
In a later iteration, we'll wrap the headers the body in a JSON string
before passing it to Lambda.
The filter requires the ARN of the Lambda function and supports
per-filter-config. When the per-filter configuration is used, the target
cluster must be tagged with specific metadata. This indicates to the
filter whether to process the request or to skip it.
Lambda supports two invocation modes:
- Synchronous (Request-Response)
- Asynchronous (Event)
This initial version of the filter supports the synchronous mode only.
In a later iteration I'll add support for the asynchronous (Event-based)
version.
Signed-off-by: Marco Magdy <mmagdy@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 807401004d500899e9aa4c78fce007cf83b538cd
This new alpha filter injects authentication headers for requests
directed at AWS services that require authentication.
Note:
Requests over plain HTTP aren't handled yet, since the message body
needs to be signed.
Fixes#9708
Signed-off-by: Raul Gutierrez Segales <rgs@pinterest.com>
Mirrored from https://github.com/envoyproxy/envoy @ ee2306673b79215641be02893cb4d8b2b256c466
Add Client Status Discovery Service (CSDS) API definition. This can be used by debug tools to obtain config information for specific clients from control plane.
Risk Level: Low
Testing: N/A
Signed-off-by: Fuqiang Gao <fuqianggao@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 10f756efa17e56c8d4d1033be7b4286410db4e01
Currently supported retry host predicates only allow omitting either previously tried or canary hosts. This enhancement is to add a new host predicate that looks at the endpoint metadata match and omits the host in case of a match with the host metadata. See #9531
Risk Level: Low
Testing: Unit testing
Docs Changes: Added new proto for omit_hosts
Signed-off-by: Prakhar Gautam <prakhag@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ e2fdf70f0fca0f9a9a66046fd80b280981b3f0ed
Implements on-demand resolution of VirtualHosts via VHDS
Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8e2d909ad22f84d9eb055f06890924a5879bad76
Define empty config protos for all filters expecting google::protobuf::Empty
Risk Level: medium (change of config type)
Testing: unit
Docs Changes: done
Release Notes: define config protos for all extensions
Co-authored-by: Derek Argueta <dereka@pinterest.com>
Mirrored from https://github.com/envoyproxy/envoy @ 2d5a4e94720cc195324f79ca68f0e7a7dc83ee9e
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ a60f6853a2c2ebbbfed79dfff0b5b644fd735980
Description:
Move packages around for #8120 and #8121
Risk Level: Med around messing up build.
Testing: CI
Docs Changes: in API/STYLE.md
Release Notes: N/A (v3alpha is not in use yet)
Fixes#8120
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 1371f2ef46582a72b5b3971147bd87c534011731
This is a beachhead PR for a Clang Libtooling based workflow that automagically updates Envoy's
source tree to the latest API version for every referenced package. So far, ths tool is only capable
of inferring types and performing header fixups, later PRs will expand this.
Risk level: Low
Testing: Manual cleanup of all headers in source/ test/ and include/, all tests pass.
Part of #8082
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ b54d1c366744314dda38aed5f14e7e93323acb41
* Add an explicit threat model to the end user facing docs, link to this from SECURITY.md
* Switch all Envoy extensions to use a new macro `envoy_cc_extension`, mandating that extensions declare a security posture. Extensions can also optionally declare `alpha` or `wip` status.
* Tag all documentation sites with their well-known Envoy names.
* Introduce tooling to automagically populate a list of known trusted/untrusted extensions in the threat model docs.
* Generate API docs for extensions that depend on `google.protobuf.Empty`. This pattern is deprecated as per https://github.com/envoyproxy/envoy/issues/8933, but we need these for tooling support meanwhile.
This work was motivated by oss-fuzz issue https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18370
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 90d1094b32aa017f90cc8efcd379aeb143acabfc
This patch moves away from the paradigm of sed-style upgrading of every v2
package to v3alpha. Instead, an additional type analysis phase is
performed prior to protoxform by a protoc plugin known as the "type
whisperer".
The type whisperer produces structured type dependency information for
each .proto. The tools/type_whisperer/typedb_gen.py tool then knits
these together to provide an API-wide type dependency graph. This is
then used to determine which types need upgrading (either they have
breaking changes or transitively depend on types with such changes).
Only packages with upgraded types now undergo the v2 -> v3alpha
transition.
The API type database is checked into
source/common/config/api_type_db.pb. This may seem a strange location,
but in the future we will include the type database as a build artifact
for the Envoy binary, as it will be used by the reflection-based version
converter to find the type upgrade path for input proto.
Risk level: Low (the v3alpha protos are not used yet).
Testing: fix_format, manual inspection of diffs, bazel test //test/...,
docs build.
Part of #8082Fixes#8490
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ ad57b58cfbb256af41a467260dce2a8013b7a7fa
As part of #8082, we want to be able to (1) automatically generate BUILD
files and (2) treat packages as atomic from a "upgrade / do not upgrade"
decision perspective. This is simplified by having our BUILD targets at
package granularity, since this is what the protoxform plugin operates
on.
This PR broadens the package-level treatment that was already introduced
for Go in #8003 to Python and C++. This simplifies BUILD files
significantly and opens the way to automated generation.
There is some technical debt introduced, since all visibility controls
have been removed. This is slated for reintroduction in
https://github.com/envoyproxy/envoy/issues/8491.
As a bonus (useful for BUILD file generation), also removed the
inconsistency in BUILD package target naming for packages in envoy.api.*
and envoy.type.*. E.g. //envoy/api/v2:v2 is now //envoy/api/v2:pkg.
Risk level: Low (but this will break internal builds and require BUILD
fixups to consuming projects).
Testing: bazel test //test/... @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4e858f17fe08224c9c089240908ccd0c518e01a7
These were missed in #8125.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 1b3b4ae1180b67bee6395fab5c075896fb1964ec
This avoids having to list new docs protos in both docs/build.sh and
api/docs/BUILD. This technical debt cleanup is helpful in v3 proto work
to simplify collecting proto artifacts from a Bazel aspect.
Risk level: Low
Testing: docs/build.sh, visual inspection of docs.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ b06e2b5c473b4b88e23ed66abd501e29f6bddbee
The libstdc++ std::regex implementation is not safe in all cases
for user provided input. This change deprecates the used of std::regex
in all user facing paths and introduces a new safe regex matcher with
an explicitly configurable engine, right now limited to Google's re2
regex engine. This is not a drop in replacement for std::regex as all
language features are not supported. As such we will go through a
deprecation period for the old regex engine.
Fixes https://github.com/envoyproxy/envoy/issues/7728
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ eff020170c6267e6c8dc235473f7fc85c5b1e07d
This allows using Envoy as a generic HTTP proxy without any
prior configuration of DNS targets. See the included documentation
for more information.
Part of https://github.com/envoyproxy/envoy/issues/1606
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 79e53f21c680b4a02695b1761e7d91868c3481d4
The current /listeners admin endpoint only returns a JSON-encoded list of addresses/ports. If a listener config binds to port 0, the endpoint will return the port that was assigned, but if there are multiple listeners binding to port 0, there's no way to tell which listener received which port.
This PR adds a proto that contains the listener's name and address. The /listeners endpoint returns this data in both text and JSON format, similar to the /clusters admin endpoint.
Risk Level: Low (according to #6959 the /listeners admin endpoint is only used for testing)
Testing:
I updated integration_admin_test.cc to test the new output formats for the /listeners endpoint.
I also updated the tools/socket_passing.py script that's used for integration/hotrestart_test.
Docs Changes:
new inline docs in listeners.proto
updated the admin menu description of /listeners
Release Notes:
admin: /listener endpoint now returns :ref:listener.proto<api/envoy/admin/v2alpha/listeners.proto>
Fixes#6959
Signed-off-by: Kaisen Chen <kaisen@yelp.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4461f431112876932e0328f6af31de7cf34fadfc
Add an http filter to extract the original source on a per-http-request basis
Signed-off-by: Kyle Larose <kyle@agilicus.com>
Mirrored from https://github.com/envoyproxy/envoy @ 712a72a718969ef37eb05a57a7e11c94d5fe6253
1) Add request/response body tapping
2) Add buffered body limits (TBI for transport socket)
3) Add the JSON_BODY_AS_BYTES and JSON_BODY_AS_STRING output
formats for convenience when the body is known to be human
readable.
4) Add JSON output for the file per tap sink.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 9a06dc0777d2809195cb1fc414b05ae7c0660193
This commit refactors the tap transport socket to use the common
tap extension configuration and tap matching infrastructure. More
match conditions will be added in a future PR as well as additional
cleanups that have been marked with TODOs.
One result of this PR is that the HTTP tap filter can now have a static
configuration as well as write to a file per tap sink.
All future tap PRs should be smaller and more targeted after this one.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ f37ebdc14f4c0adf0e90aabddae833355c0cec1b
This is a rename PR only. It renames the capture transport socket
and associated tools to the tap transport socket. It also updates
some documentation. In a subsequent PR I'm going to refactor the
tap transport socket to use the new common tap framework so that
the tap transport socket can be configured via admin, the HTTP
tap filter can write to a file, the tap transport socket can have
matching, etc.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7a5849f2a8bcc55fa16da3eaee94d9c99a11147c
This is a MVP for the HTTP tap filter. It includes minimal
infrastructure for the following:
1. Generic tap configuration which in the future will be used for
static config, XDS config, etc. In this MVP the tap can be
configured via a /tap admin endpoint.
2. Generic output configuration which in the future will be used for
different output sinks such as files, gRPC API, etc. In this MVP
the tap results are streamed back out the /tap admin endpoint.
3. Matching infrastructure. In this MVP only matching on request and
response headers are implemented. Both logical AND and logical OR
matches are possible.
4. In this MVP request/response body is not considered at all.
5. All docs are included and with all the caveats the filter is ready
to use for the limited cases it supports (which are likely still to
be useful).
There is a lot of follow on work which I will do in subsequent PRs.
This includes:
1. Merging the existing capture transport socket into this framework.
2. Implementing body support, both for matching on body contents as
well as outputting body data.
3. Tap rate limiting so too many streams do not get tapped.
4. gRPC matching. Using reflection and loaded proto definitions, it will
be possible to match on gRPC fields.
5. JSON matching. If the body parses as JSON, we can allow matching on
JSON fields.
Part of https://github.com/envoyproxy/envoy/issues/1413.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ cf80045587240d494e54e9772949bc9af5eda61f
We no longer allow running v1 config, and users can look at old
docs if they want to see the v1 docs still.
Also fix a doc issue in the /server_info admin endpoint.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8aba108eda816ea29dc80b973a1387257cd209c6
Introduces a rate limiter filter for thrift requests, implemented as a
decoder filter that works in conjunction with the thrift router filter.
Risk Level: low
Testing: unit tests
Doc Changes: inline with proto definitions
Release notes: updated
Signed-off-by: Stephan Zuercher <stephan@turbinelabs.io>
Mirrored from https://github.com/envoyproxy/envoy @ 6ac936f2750c39a8b4fb232d6ddc4802f4e6aeee
data-plane-api(CircleCI)