This patch introduces a new tool, protoxform, that will be the basis of
the v2 -> v3 migration tooling. It operates as a Python protoc plugin,
within the same framework as protodoc, and provides the ability to
operate on protoc AST input and generate proto output.
As a first step, the tool is applied reflexively on v2, and functions as
a formatting tool. In later patches, this will be added to
check_format/fix_format scripts and CI.
Part of #8082.
Risk level: medium (it's possible that some inadvertent wire changes
occur, if they do, this patch should be rolled back).
Testing: manual inspection of diff, bazel test //test/..., some
grep/diff scripts to ensure we haven't lost any comments.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 08b123a8321d359ea66cbbc0e2926545798dabd3
Remove gogoproto annotations. They can be replaced with a custom gogoproto compiler (e.g. something like https://github.com/gogo/googleapis/tree/master/protoc-gen-gogogoogleapis). I have an experimental version of it to validate that it's possible to re-apply important annotations in the compiler.
Risk Level: low
Testing: builds
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e7f0b7176efdc65f96eb1697b829d1e6187f4502
Fixes#7982
Defines a package level proto library and its associated internal go_proto_library.
Deletes all existing api_go_proto_library, api_go_grpc_library, and go_package annotations in protos (they are not required and pollute the sources).
I deliberately avoided touching anything under udpa since it's being moved to another repository.
Risk Level: low
Testing: build completes
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ d504fde0ffd97017d1ddff8caa9a3b46bba9ae48
This patch establishes a v3alpha baseline API, by doing a simple copy of
v2[alpha] dirs and some sed-style heuristic fixups of BUILD dependencies
and proto package namespaces.
The objective is provide a baseline which we can compare the output from
tooling described in #8083 in later PRs, providing smaller visual diffs.
The core philosophy of the API migration is that every step will be
captured in a script (at least until the last manual steps),
api/migration/v3alpha.sh. This script will capture deterministic
migration steps, allowing v2[alpha] to continue to be updated until we
finalize v3.
There is likely to be significant changes, e.g. in addition to the work
scoped for v3, we might want to reduce the amount of API churn by
referring back to v2 protos where it makes sense. This will be done via
tooling in later PRs.
Part of #8083.
Risk level: Low
Testing: build @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 085d72b490c124a02849812798f5513a8df9ae72
As per #6651, this PR plumbs in CLI options to allow independent control over static/dynamic unknown
field validation.
The defaults are the same for static as today (strict) and for dynamic we are by default permissive.
This permits easy rollout of new API minor versions, including those related to security fixes.
Fixes a regression that occurred in #7200 where strict/permissive checking CLI options were
inverted.
As per #6818, added stats/warning for any unknown fields encountered.
Risk level: Low (strictly more permissive by default)
Testing: additional unit and integration tests added, exercising both permissive/strict checking
over various parts of the API (bootstrap, listeners, clusters, xDS, network filters, etc).
Fixes#6651Fixed#6818
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 0418a855d9f9e37ec70b4c6d1942688fc8bb5751
Due to a seg fault issue with the gogo protobuf library
[https://github.com/gogo/protobuf/issues/568], non nullable repeated
fields in a proto will cause proto.Merge(dst, src) to panic.
The nullable field setting was first added by @kyessenov when he was
re-organizing the protos. Unfortunately, people have been copy pasting it
across several areas in the Envoy proto. To keep the impact radius to a minimum,
I have updated only the fields that are currently causing the segfault
(in go-control-plane) for us.
Its also partly against proto principles. You should be able to determine if
a field is set or not. This non-nullable setting in gogo will insist on initializing
the field to default values.
Risk Level: to go control plane users
Signed-off-by: Shriram Rajagopalan <rshriram@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ b22d2b5cf09f779962cfedaaab24969f384cbc48
Add host priority to cluster response in admin server.
Risk Level: low
Testing: unit test
Docs Changes: N/A
Release Notes: updated
Signed-off-by: Yan Xue <yxyan@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 815c506c96ef441d99341775af2125d58d644b8f
The current /listeners admin endpoint only returns a JSON-encoded list of addresses/ports. If a listener config binds to port 0, the endpoint will return the port that was assigned, but if there are multiple listeners binding to port 0, there's no way to tell which listener received which port.
This PR adds a proto that contains the listener's name and address. The /listeners endpoint returns this data in both text and JSON format, similar to the /clusters admin endpoint.
Risk Level: Low (according to #6959 the /listeners admin endpoint is only used for testing)
Testing:
I updated integration_admin_test.cc to test the new output formats for the /listeners endpoint.
I also updated the tools/socket_passing.py script that's used for integration/hotrestart_test.
Docs Changes:
new inline docs in listeners.proto
updated the admin menu description of /listeners
Release Notes:
admin: /listener endpoint now returns :ref:listener.proto<api/envoy/admin/v2alpha/listeners.proto>
Fixes#6959
Signed-off-by: Kaisen Chen <kaisen@yelp.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4461f431112876932e0328f6af31de7cf34fadfc
Implement the scoped RDS (SRDS) API config subscription and provider based on the config protos introduced in #6675 and the ConfigProvider framework introduced in #5243 and #6781.
NOTES:
See parent PR #5839 for full context into these changes. PRs 2a (#6675) and 2b (#6781) have already been merged.
The API is not yet fully implemented. This PR introduces static and dynamic (xDS config subscription) handling of scoped routing configuration, but the new L7 multi tenant routing logic (see #4704) has not yet been introduced.
The API is not yet plumbed into the HttpConnectionManager, that will be done in the next PR.
This PR includes unit tests only; integration tests will follow in the next PR.
Risk Level: Low (this DS API is not yet integrated into the HCM and can not be enabled via config).
Testing: Unit tests added.
Docs Changes: N/A.
Signed-off-by: Andres Guedez <aguedez@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 831d0cbb389f241fd4d1a8b224ffa43e4257a680
This adds an option to allow hosts to be excluded in lb calculations until they have been health checked
for the first time. This will make it possible to scale up the number of hosts quickly (ie large increase
relative to current host set size) without triggering panic mode/spillover (as long as the initial health check
is succeeds).
While these hosts are excluded from the lb calculations, they are still eligible for routing when panic
mode is triggered.
Signed-off-by: Snow Pettersen <snowp@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4c80194bf82193261aa52a4ca64c4e6a461881c0
When using active health checking, hosts are not removed from
dynamic clusters if they are still passing health checks. This
creates a situation in which hosts might not be removed for a
very long time if the sequence is reversed; removal followed by
health check failure. This change handles the second case so that
any time a host is both removed AND failing active health check,
in any order, it will be removed.
This has been an issue "forever" but is more obvious when using
streaming EDS or very long polling DNS.
Fixes https://github.com/envoyproxy/envoy/issues/6625
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 41eefffcd728d071037a57a1accd402ec188bcd5
This PR replaces deprecated HTTP links by HTTPS links for security and consistency.
Signed-off-by: Vu Cong Tuan <tuanvc@vn.fujitsu.com>
Mirrored from https://github.com/envoyproxy/envoy @ b47112f62b5186bb087bdf335f6df6fb072e2a46
This is a MVP for the HTTP tap filter. It includes minimal
infrastructure for the following:
1. Generic tap configuration which in the future will be used for
static config, XDS config, etc. In this MVP the tap can be
configured via a /tap admin endpoint.
2. Generic output configuration which in the future will be used for
different output sinks such as files, gRPC API, etc. In this MVP
the tap results are streamed back out the /tap admin endpoint.
3. Matching infrastructure. In this MVP only matching on request and
response headers are implemented. Both logical AND and logical OR
matches are possible.
4. In this MVP request/response body is not considered at all.
5. All docs are included and with all the caveats the filter is ready
to use for the limited cases it supports (which are likely still to
be useful).
There is a lot of follow on work which I will do in subsequent PRs.
This includes:
1. Merging the existing capture transport socket into this framework.
2. Implementing body support, both for matching on body contents as
well as outputting body data.
3. Tap rate limiting so too many streams do not get tapped.
4. gRPC matching. Using reflection and loaded proto definitions, it will
be possible to match on gRPC fields.
5. JSON matching. If the body parses as JSON, we can allow matching on
JSON fields.
Part of https://github.com/envoyproxy/envoy/issues/1413.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ cf80045587240d494e54e9772949bc9af5eda61f
* api: add proto options for java
* add ci for checking proto options
Signed-off-by: Penn (Dapeng) Zhang <zdapeng@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 02659d411332e9f20d229f482931c15304ea17fd
This allows a host to be marked as degraded by having x-envoy-degraded
returned from the HTTP active health checking response.
Signed-off-by: Snow Pettersen <snowp@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7b6d7a2706ac5290e5e87535c23c11def49275db
This adds the host's weight to the `/clusters?format=json` response.
Signed-off-by: Venil Noronha <veniln@vmware.com>
Mirrored from https://github.com/envoyproxy/envoy @ 9039f24198c35939af411c4102fe64b9aba7d6b1
Adds init states to ServerInfo.State to make it easier to determine if
an Envoy instance is stuck initializing.
Fixes#4405
Signed-off-by: Snow Pettersen <snowp@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ 755154f96e146ab5e03093ef8a450ff118348d31
Converts the existing /server_info admin endpoint to be represented by a protobuf. This will make it easier to extend with new values in the future.
Risk Level: Low
Testing: Updated the existing unit test
Docs Changes: n/a
Release Notes: n/a
Part of #4405
Signed-off-by: Snow Pettersen <snowp@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ 71bd095297ba64712bfad30d0aee1f019fbd32d8
*Description*: Adds support for displaying Subject Alternate Names as a comma separated list of string in `/certs` admin end point
*Risk Level*: Low
*Testing*: Automated
*Docs Changes*: Updated
*Release Notes*: Updated
Signed-off-by: Rama <rama.rao@salesforce.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7403314b0b255a89ab284b7ed0eead5a5f4aecfd
Adds a /memory endpoint to the admin panel for fast inspection of Envoy::Memory's heap statistics, without needing to query stats.
Risk Level: Low
Testing: Added a test to admin_test.cc.
Docs Changes: Added a description of the option to admin.rst.
Release Notes: N/A
Signed-off-by: James Buckland <jbuckland@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ b5ba7f89a4433d283549c61a659c4e61ed564d16
Currently host level stats in clusters proto uses map so they are outputted in random order. This PR changes it to list so that the order is predictable.
Risk Level: Low
Testing: Added automated tests
Docs Changes: N/A
Release Notes: N/A
Signed-off-by: Rama <rama.rao@salesforce.com>
Mirrored from https://github.com/envoyproxy/envoy @ 69474b39878d74354a1fb0b14d41f362839540e8
Fixes the order of config_dump elements - bootstrap, clusters, listeners and routes will come in that order. Note that inside each of them, ordering is not guaranteed. This is a step in moving towards complete ordering.
Risk Level: Low
Testing: Added Automated tests
Docs Changes: Updated
Release Notes: N/A
Signed-off-by: Rama <rama.rao@salesforce.com>
Mirrored from https://github.com/envoyproxy/envoy @ 51d274be76119efff76fdafaa2f49fdd479f2def
I think this broke in a recent refactor.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 866597fcb8cc3cdd53a767d66755506036261f3c
adds the required visibility rules and delegates the rest to the generic
api_proto_library. I tested the change by doing the following without
getting errors.
./ci/run_envoy_docker.sh './ci/do_ci.sh docs'
I changed the BUILD files using the following commands.
/envoy/api$ find . -type f -name BUILD | xargs sed -i -e 's/api_proto_library(/api_proto_library_internal(/g'
envoy/api$ find . -type f -name BUILD | xargs sed -i -e 's/"api_proto_library"/"api_proto_library_internal"/g'
Signed-off-by: mickey <mickeyju@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4b871c0ab9350882271a490adcee44e613ed9807
Added the /clusters?format=json admin endpoint along with a proto representation of /clusters.
Risk Level: Low
Testing: Added a unit test for the new format.
Docs Changes: Added a brief description on the admin docs and linked to the more detailed proto definition.
Release Notes: Added release notes.
Fixes#2020
Signed-off-by: Matt Rice <mattrice@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 64605338ef040a949c5ea205bdd472a8fe42306c
This adds a new google.protobuf.Timestamp last_updated field to BootstrapConfigDump, ListenersConfigDump, ClustersConfigDump, and RoutesConfigDump, representing the time when the configs under that category were last updated.
Additionally, the ProdSystemTimeSource::instance_ singleton is passed through several layers of interfaces. This is to make testing easier, but I would love to find a workaround for injecting a MockSystemTimeSource closer to the point where it is needed in the tests.
Risk Level: Low
Testing:
Several tests were modified / updated to compare the dumped time (always 1234567890 seconds since epoch time) by using a MockSystemTimeSource which always returned 1234567890.
Recreated PR based off #3641.
Signed-off-by: James Buckland <jbuckland@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ a22d96046d5292d99bd7f3c4ccce5f05fb52ffc7
This change does several things:
1) Clarifies how we handle xDS version_info in responses and sets us up
for both top-level/transactional versions as well as per-resource
versions in the future.
2) Moves the config_dump admin endpoint to the v2alpha namespace so that
we can iterate on it in the future.
3) Fills out the config dump proto for the remaining resource types.
These are not implemented but are here to force a discussion about
how we want to handle versions moving forward.
4) Fixes RDS static config dump to actually work and add better tests.
5) Wire up version for the RDS config dump on a per-resource basis.
Once we agree on the general version semantics I will be following up
with dump capability of the remaining resource types.
Part of https://github.com/envoyproxy/envoy/issues/2421
Part of https://github.com/envoyproxy/envoy/issues/2172
Fixes https://github.com/envoyproxy/envoy/issues/3141
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ ada758739907628b50079b9adfccf5481ec9fc5f
Validated the full push-to-master to data-plane-api, so this time it should work.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ bf1073f04db95b674575c9f96fc821a5537774b2
See envoyproxy/envoy#2771 for context
I will add documentation once the interfaces and placement of these things is settled. Current location (new admin package) was agreed upon as a good start with @htuch but i don't feel strongly about that or naming.
Signed-off-by: James Sedgwick <jsedgwick@lyft.com>
data-plane-api(CircleCI)
James Sedgwick