This patch performs a major version freeze and bump by modifying
package_version_status, using the tooling developed in #10636.
Specifically:
v2 APIs are frozen (except for where they are the latest in their
package history and still active)
v3 APIs are transitioned to be active
Candidate v4alpha APIs are generated (not used by Envoy yet)
Fixes#10355
Risk level: medium (entire API's files are modified, visually verified
to ensure things look sane, all tests pass)
Testing: CI
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 549164c42cae84b59154ca4c36009e408aa10b52
Part of https://github.com/envoyproxy/envoy/issues/10355, this patch introduces additional
annotations to the API to support automatic inference (and developer documentation) of where the
active developer editable version of a file is, and which files are frozen or machine generated.
Risk level: Low (API annotations only)
Testing: CI
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4c5f4310bb8a19a38e5377d7a2d2dc6aa4560f47
This PR introduces a parallel v3 API reference documentation tree to the
existing v2 one.
The docs/root/api-v3/ tree was copied from docs/root/api-v2 and the
necessary package path fixups were made manually. As a result, the tree
largely resembles the v2 docs. Long term this is likely to evolve to
reflect the shape of the new extensions tree.
The message type, field and enum anchors are sed'ed to be distinct and
self-consistent inside api-v3/.
There were a number of API proto changes that were made to obtain a
successful Sphinx build:
* References to deprecated fields were replaced by references to the replacement field.
* clang-format line wrapping in protoxform was removed, this breaks RST in some v3 protos.
* Some packages (type/metadata/v2, data/cluster/v2alpha) were force upgraded to v3, to deal with references to types that are distinct in v2/v3. This is OK as these packages probably make sense to bump for v3, in general we're going to have to think about how to do this more
cleanly, supporting dual v2/v3 references alongside each other.
* Some evil hacks for field renaming added to migrate.py for RouteAction.
There's also some additional machinery added to compute distinct v3/v3
build targets to point protodoc at.
Risk level: Low
Testing: Docs build, manual inspection.
Fixes#8087
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ ac88316892cd47b6a9b58e3736e20e8863cd0d27
In which we convert every v3alpha reference to v3. In future revs of the
stable API versioning policy, we will develop better tooling to support
> 2 alpha and stable versions. For v3, it seems reasonable to just mv
v3alpha to v3, since there should be no external consumers yet.
Risk level: Low
Testing: bazel test //test/..., CI.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 5248a4fb7d4c2a3d1fa151f944d3a63f6b7a06cf
Changing from relative name to absolute name, and fixing the fatal-by-defaults that were broken by the v3 switch.
The old way to allow fatal-by-defaults was
envoy.deprecated_features:proto_file.proto:field_name
the new way is
envoy.deprecated_features:full.namespace.field_name
When we switched to v3, all the hard-coded v2 names stopped working. This reinstates them via hopefully more permanent proto annotation.
The only remaining ugly bit is that unfortunately the full namespace and field name are the v3 versions even if the original config was v2. Between @htuch and I we should fix that before merging.
Risk Level: Medium
Testing: added new unit tests
Docs Changes: updated
Release Notes: n/a
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 156d7c90083c196a206c07fc03b2de6be8260bd3
To better support config dump, deprecated field detection and other debug, it's helpful to leave a type name breadcrumb and be able to synthesize a Protobuf::Message that corresponds to what was delivered on the wire.
While working on this PR, it became apparent that config dump is broken post v3alpha, since a single config dump might have both v2 and v3 Listeners, etc. The only way to resolve this generically is to make the inner resources in config dump Any. This is a breaking API change, but these are v2alpha/v3alpha at this point, so allowed.
Risk level: Low
Testing: new version converter unit test, config dump tests now verify that the correct versioned inner resource is returned.
Fixes#9612
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 233838a39fb8310ea145e7d758d428d5a37b0306
This allows for a clean separation of config/service in v3. This is a
continuation of #9548.
Risk level: Low
Testing: bazel test //test/...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ c3bddaee1912fcd1fedc4786aee830b2e4a7c599
Description:
Move packages around for #8120 and #8121
Risk Level: Med around messing up build.
Testing: CI
Docs Changes: in API/STYLE.md
Release Notes: N/A (v3alpha is not in use yet)
Fixes#8120
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 1371f2ef46582a72b5b3971147bd87c534011731
This commit adds query parameters to the /config_dump admin endpoint
that allow listing keys and dumping only config associated with a passed
key.
You can now filter /config_dump endpoint with the key query parameter, e.g. /config_dump?key=clusters. This uses keys such as "listeners" and "clusters" that are set in places like here. You can view all keys with /config_dump?list_keys.
Risk Level: low
Testing: unit tests
Docs Changes: updated config dump proto docs and admin interface docs
Release Notes: added
Fixes#8743
Signed-off-by: Spencer Lewis <slewis@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ 091a06f5126021cbff8629cba5e4bafa24e6603c
In order to get file level move annotation, import has to be before options.
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 062c895f499382ae61dead16db2a7e78b9146525
*Description:*
Add a new `--disable-extensions` flag that hides the named extensions
from their corresponding factory registry. The extensions are retained
in the registry factory, but the factory pointer is nulled so that
configuration attempts to use the extensions will fail.
*Risk Level:* Medium
*Testing:* Added unit tests, manual verification.
*Docs Changes:* Added to CLI reference.
*Release Notes:* Added release note.
*Fixes:* #9096
Signed-off-by: James Peach <jpeach@apache.org>
Mirrored from https://github.com/envoyproxy/envoy @ 445d0ee4a64738f1de072537fb364d9096478915
Instead of formatting options heuristically, which will erase new annotations without changing protoxform, use proto descriptor to format options, and enforce its order as well.
Risk Level: Low
Testing: CI
Docs Changes: N/A
Release Notes: N/A
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ dfe687d49574ef7eb1bf84867bf571e805a2bf97
* api: link to previous message type package in API BUILD files.
We need to include the descriptors from the previous message version in
the build. We opt to do this transitively; when you include v3 of a
package, you get the v2 via a transitive dep. This should work based on
alwayslink semantics for cc_library.
The computation of the deps is based on the previous_message_type
annotation, which will allow cross package migrations.
Part of #8082.
Risk level: Low
Testing: Disabled ip_tagging v2 descriptor hack, observed
version_integration_test. After the BUILD changes, this now passes
again.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7f8fb9509d3189819dd253e25ec76e939ae106e7
This PR avoids having to include an API type database in the Envoy build
by introducing a message annotation option that allows Envoy to
determine earlier corresponding message types via descriptor inspection.
The ApiTypeDb is now ApiTypeOracle and utilizes these annotations.
Risk level: Low
Testing: Existing API and verison upgrade tests pass.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 297f7a73b3f93bccf8af73c0a555ae52bce6cecb
Currently, application logs are not sanitized of c-style escape sequences. If any filter logs a message that contains newline characters, the logs will be printed to a new line. This breaks log formats set by the --log-format option, breaking integration with log viewers.
This change adds a command line option --log-format-escaped to escape c-style escape characters in application logs before they are outputted. Enabling this flag ensures newline characters in logs are ignored, meaning that each call to ENVOY_LOG will result in at most 1 line outputted. This flag works for both Stderr and File loggers.
Risk Level: Low
Testing:
Unit tests
Fuzz test
Manual verification (see comments in PR)
Performance Impact: As long as production environments are running with the default log level, this will only slightly impact startup time (only when --log-format-escaped is set). The critical section for each request/response will not be impacted.
Docs Changes: Added docs to command line options about new flag and possible use cases, like Stackdriver Logging integration on GKE.
Release Notes: Added release notes about new flag
Fixes#8637
Signed-off-by: Teju Nareddy <nareddyt@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 20ca0ae3bdd9c2a69194203f5e1d2eca92ce2b48
Description: Tracking load status for LDS using the ConfigDump protos
Risk Level: Medium: major changes to an existing (alpha) system
Testing: new unit tests
Docs Changes: API notes inline
Release Notes: n/a
#8039
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 0aed05aecd17576b0dd96f3e4126acd1c24a02bc
Modifies the pedantic spell checker to better handle camel-case words.
Each part the word is now treated as a separate error in both the check
and fix modes.
Disables run-together mode for aspell, which allowed typos such as
"mananger" (man + anger). Fixes the resulting spelling errors.
Miscellaneous other fixes:
* Provides an option to replace a word without adding the word to the
dictionary (in case aspell's suggestions do not contain the correct
replacement).
* Fixes a bug in the script when no suggestions are returned by aspell.
* Checks the dictionary and added words for invalid characters that cause
aspell errors at dictionary load time.
* Sets the mark flag when in CI runs so that misspelled words are indicated.
* Culls words from the dictionary that are no longer in the codebase, or are
otherwise not needed.
Risk Level: low (comments only)
Testing: n/a
Doc Changes: n/a
Release Notes: n/a
Fixes: #8481
Signed-off-by: Stephan Zuercher <zuercher@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ de70fe54cdba6b022b3971379afa535f402f2ffe
This reverts commit 596cd4894c8ecd536c1da1dddecaae3531f269ea.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8c4afa6fee5e104c7b195657a3c22e23f062eb11
Modifies the pedantic spell checker to better handle camel-case words.
Each part the word is now treated as a separate error in both the check
and fix modes.
Disables run-together mode for aspell, which allowed typos such as
"mananger" (man + anger). Fixes the resulting spelling errors.
Miscellaneous other fixes:
* Provides an option to replace a word without adding the word to the
dictionary (in case aspell's suggestions do not contain the correct
replacement).
* Fixes a bug in the script when no suggestions are returned by aspell.
* Checks the dictionary and added words for invalid characters that cause
aspell errors at dictionary load time.
* Sets the mark flag when in CI runs so that misspelled words are indicated.
* Culls words from the dictionary that are no longer in the codebase, or are
otherwise not needed.
Risk Level: low (comments only)
Testing: n/a
Doc Changes: n/a
Release Notes: n/a
Fixes: #8481
Signed-off-by: Stephan Zuercher <zuercher@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 596cd4894c8ecd536c1da1dddecaae3531f269ea
Generate or format next free field annotation via protoxform.
Risk Level: low
Testing: N/A
Docs Changes: N/A
Release Notes: N/A
Fixes#8429
Signed-off-by: Yi Tang <ssnailtang@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 986173ed516dcc1c3dea7db90659ed993d0aad75
This patch moves away from the paradigm of sed-style upgrading of every v2
package to v3alpha. Instead, an additional type analysis phase is
performed prior to protoxform by a protoc plugin known as the "type
whisperer".
The type whisperer produces structured type dependency information for
each .proto. The tools/type_whisperer/typedb_gen.py tool then knits
these together to provide an API-wide type dependency graph. This is
then used to determine which types need upgrading (either they have
breaking changes or transitively depend on types with such changes).
Only packages with upgraded types now undergo the v2 -> v3alpha
transition.
The API type database is checked into
source/common/config/api_type_db.pb. This may seem a strange location,
but in the future we will include the type database as a build artifact
for the Envoy binary, as it will be used by the reflection-based version
converter to find the type upgrade path for input proto.
Risk level: Low (the v3alpha protos are not used yet).
Testing: fix_format, manual inspection of diffs, bazel test //test/...,
docs build.
Part of #8082Fixes#8490
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ ad57b58cfbb256af41a467260dce2a8013b7a7fa
This provides canonical BUILD formatting and puts protoxform in charge
of being able to determine import paths, without having to worry about
Bazel implications.
Part of #8082.
Risk level: Low
Testing: tools/proto_sync.py, visual inspection of diffs.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e53f40f0e5ccc84fca5cd350416fe0f2accf8229
As part of #8082, we want to be able to (1) automatically generate BUILD
files and (2) treat packages as atomic from a "upgrade / do not upgrade"
decision perspective. This is simplified by having our BUILD targets at
package granularity, since this is what the protoxform plugin operates
on.
This PR broadens the package-level treatment that was already introduced
for Go in #8003 to Python and C++. This simplifies BUILD files
significantly and opens the way to automated generation.
There is some technical debt introduced, since all visibility controls
have been removed. This is slated for reintroduction in
https://github.com/envoyproxy/envoy/issues/8491.
As a bonus (useful for BUILD file generation), also removed the
inconsistency in BUILD package target naming for packages in envoy.api.*
and envoy.type.*. E.g. //envoy/api/v2:v2 is now //envoy/api/v2:pkg.
Risk level: Low (but this will break internal builds and require BUILD
fixups to consuming projects).
Testing: bazel test //test/... @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4e858f17fe08224c9c089240908ccd0c518e01a7
We should start tagging messages with "option deprecated = true;" if we
want them to be auto-deprecated by protoxform going forward.
Risk level: Low (v3alpha is not used yet)
Testing: bazel test @envoy_api//..., manual diff inspection.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e3247539bcc39f3111287b479a85e51655a6f1db
Via ./api/migration/v3alpha.sh. This picks up the changes since the last
sync, in particular the major reformat in #8309.
Risk level: Low (not used yet).
Testing: bazel build @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ c41cfbf6a33b8115a7e29a2b4a926aad4cd062be
This patch introduces a new tool, protoxform, that will be the basis of
the v2 -> v3 migration tooling. It operates as a Python protoc plugin,
within the same framework as protodoc, and provides the ability to
operate on protoc AST input and generate proto output.
As a first step, the tool is applied reflexively on v2, and functions as
a formatting tool. In later patches, this will be added to
check_format/fix_format scripts and CI.
Part of #8082.
Risk level: medium (it's possible that some inadvertent wire changes
occur, if they do, this patch should be rolled back).
Testing: manual inspection of diff, bazel test //test/..., some
grep/diff scripts to ensure we haven't lost any comments.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 08b123a8321d359ea66cbbc0e2926545798dabd3
Remove gogoproto annotations. They can be replaced with a custom gogoproto compiler (e.g. something like https://github.com/gogo/googleapis/tree/master/protoc-gen-gogogoogleapis). I have an experimental version of it to validate that it's possible to re-apply important annotations in the compiler.
Risk Level: low
Testing: builds
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e7f0b7176efdc65f96eb1697b829d1e6187f4502
Fixes#7982
Defines a package level proto library and its associated internal go_proto_library.
Deletes all existing api_go_proto_library, api_go_grpc_library, and go_package annotations in protos (they are not required and pollute the sources).
I deliberately avoided touching anything under udpa since it's being moved to another repository.
Risk Level: low
Testing: build completes
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ d504fde0ffd97017d1ddff8caa9a3b46bba9ae48
This patch establishes a v3alpha baseline API, by doing a simple copy of
v2[alpha] dirs and some sed-style heuristic fixups of BUILD dependencies
and proto package namespaces.
The objective is provide a baseline which we can compare the output from
tooling described in #8083 in later PRs, providing smaller visual diffs.
The core philosophy of the API migration is that every step will be
captured in a script (at least until the last manual steps),
api/migration/v3alpha.sh. This script will capture deterministic
migration steps, allowing v2[alpha] to continue to be updated until we
finalize v3.
There is likely to be significant changes, e.g. in addition to the work
scoped for v3, we might want to reduce the amount of API churn by
referring back to v2 protos where it makes sense. This will be done via
tooling in later PRs.
Part of #8083.
Risk level: Low
Testing: build @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 085d72b490c124a02849812798f5513a8df9ae72
As per #6651, this PR plumbs in CLI options to allow independent control over static/dynamic unknown
field validation.
The defaults are the same for static as today (strict) and for dynamic we are by default permissive.
This permits easy rollout of new API minor versions, including those related to security fixes.
Fixes a regression that occurred in #7200 where strict/permissive checking CLI options were
inverted.
As per #6818, added stats/warning for any unknown fields encountered.
Risk level: Low (strictly more permissive by default)
Testing: additional unit and integration tests added, exercising both permissive/strict checking
over various parts of the API (bootstrap, listeners, clusters, xDS, network filters, etc).
Fixes#6651Fixed#6818
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 0418a855d9f9e37ec70b4c6d1942688fc8bb5751
Due to a seg fault issue with the gogo protobuf library
[https://github.com/gogo/protobuf/issues/568], non nullable repeated
fields in a proto will cause proto.Merge(dst, src) to panic.
The nullable field setting was first added by @kyessenov when he was
re-organizing the protos. Unfortunately, people have been copy pasting it
across several areas in the Envoy proto. To keep the impact radius to a minimum,
I have updated only the fields that are currently causing the segfault
(in go-control-plane) for us.
Its also partly against proto principles. You should be able to determine if
a field is set or not. This non-nullable setting in gogo will insist on initializing
the field to default values.
Risk Level: to go control plane users
Signed-off-by: Shriram Rajagopalan <rshriram@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ b22d2b5cf09f779962cfedaaab24969f384cbc48
Add host priority to cluster response in admin server.
Risk Level: low
Testing: unit test
Docs Changes: N/A
Release Notes: updated
Signed-off-by: Yan Xue <yxyan@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 815c506c96ef441d99341775af2125d58d644b8f
The current /listeners admin endpoint only returns a JSON-encoded list of addresses/ports. If a listener config binds to port 0, the endpoint will return the port that was assigned, but if there are multiple listeners binding to port 0, there's no way to tell which listener received which port.
This PR adds a proto that contains the listener's name and address. The /listeners endpoint returns this data in both text and JSON format, similar to the /clusters admin endpoint.
Risk Level: Low (according to #6959 the /listeners admin endpoint is only used for testing)
Testing:
I updated integration_admin_test.cc to test the new output formats for the /listeners endpoint.
I also updated the tools/socket_passing.py script that's used for integration/hotrestart_test.
Docs Changes:
new inline docs in listeners.proto
updated the admin menu description of /listeners
Release Notes:
admin: /listener endpoint now returns :ref:listener.proto<api/envoy/admin/v2alpha/listeners.proto>
Fixes#6959
Signed-off-by: Kaisen Chen <kaisen@yelp.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4461f431112876932e0328f6af31de7cf34fadfc
Implement the scoped RDS (SRDS) API config subscription and provider based on the config protos introduced in #6675 and the ConfigProvider framework introduced in #5243 and #6781.
NOTES:
See parent PR #5839 for full context into these changes. PRs 2a (#6675) and 2b (#6781) have already been merged.
The API is not yet fully implemented. This PR introduces static and dynamic (xDS config subscription) handling of scoped routing configuration, but the new L7 multi tenant routing logic (see #4704) has not yet been introduced.
The API is not yet plumbed into the HttpConnectionManager, that will be done in the next PR.
This PR includes unit tests only; integration tests will follow in the next PR.
Risk Level: Low (this DS API is not yet integrated into the HCM and can not be enabled via config).
Testing: Unit tests added.
Docs Changes: N/A.
Signed-off-by: Andres Guedez <aguedez@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 831d0cbb389f241fd4d1a8b224ffa43e4257a680
This adds an option to allow hosts to be excluded in lb calculations until they have been health checked
for the first time. This will make it possible to scale up the number of hosts quickly (ie large increase
relative to current host set size) without triggering panic mode/spillover (as long as the initial health check
is succeeds).
While these hosts are excluded from the lb calculations, they are still eligible for routing when panic
mode is triggered.
Signed-off-by: Snow Pettersen <snowp@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4c80194bf82193261aa52a4ca64c4e6a461881c0
When using active health checking, hosts are not removed from
dynamic clusters if they are still passing health checks. This
creates a situation in which hosts might not be removed for a
very long time if the sequence is reversed; removal followed by
health check failure. This change handles the second case so that
any time a host is both removed AND failing active health check,
in any order, it will be removed.
This has been an issue "forever" but is more obvious when using
streaming EDS or very long polling DNS.
Fixes https://github.com/envoyproxy/envoy/issues/6625
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 41eefffcd728d071037a57a1accd402ec188bcd5
data-plane-api(CircleCI)