Description: Taking advantage of the new feature introduced in [#12035](https://github.com/envoyproxy/envoy/pull/12035), which introduced quick visibility for init managers to check unready targets, this pull request adds protobuf message for unready targets and enables admin to dump configs of unready targets. An example of config dump for listeners’ unready targets is given in this pull request.
Introduce ```InitDumpHandler``` with ```handlerInitDump``` method to help dump information of unready targets.
Add ```dumpUnreadyTargets``` function for ```init::manager```.
Risk Level: Low
Docs Changes: protodoc
Release Notes: Added
Signed-off-by: pingsun <pingsun@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8aef76370877c66b09f7791f0577ca83aad7d608
This was added for gRPC server support, but we've decided to use resource names instead to explicitly request the listeners we want by name. This is more in-line with the new naming scheme described in the "xDS Transport Next Steps" design.
Signed-off-by: Mark D. Roth <roth@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 05cbb309b828dc86737c51fd2c79d30e48e397a4
Add log control (list and modify log level) in admin interface for Fancy Logger, a new fine-grained logger for Envoy, and provide command line option --enable-fine-grain-logging for developers.
Additional Description: A doc of overview is provided here: source/docs/fancy_logger.md.
Risk Level: Medium
Testing: Unit tests.
Docs Changes: Added a new option --enable-fine-grain-logging and doc it.
Release Notes: Added to current.rst.
Signed-off-by: Jinhui Song <jinhuisong@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ b7138814dde530b8c2957e806ea40879a4fdce32
/config_dump API now supports dumping out EDS while using parameter ?include_eds
Add help method dumpEndpointConfigs() to dump out EDS in /config_dump by calling this method in the handler handlerConfigDump()
This will dump out envoy::admin::v3::EndpointsConfigDump by generating envoy::config::endpoint::v3::ClusterLoadAssignment based on data stored in server_.clusterManager().clusters()
Missing Field:
- ClusterLoadAssignment
- Policy
- endpoint_stale_after
- StaticEndpointConfig
- last_updated
- DynamicEndpointConfig
- version_info
- last_updated
Risk Level: Medium
Testing: add unit test, integration test
Docs Changes: operations_admin_interface
Release Notes: N/A
Part of fixing #3362
Signed-off-by: Yutong Li <yutongli@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 3cec62a096584894e8d1552ba915e47cffd8bf49
Add DrainStrategy enum to Options with Graceful and Immediate
Disable probabilistic drain in DrainManager if DrainStrategy == Immediate
Add integration tests
Risk Level: Low.
Testing: Integration tests, verify that the race condition from #11240 does not occur if the probabilistic drain is disabled.
Signed-off-by: Auni Ahsan <auni@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8c7df0f08bd86bbeebaee3773b79cf6e3949fe3c
Provides a --use-dynamic-base-id flag to select an unused base-id.
Primarily useful for testing, but generally available. Adds a
--base-id-path flag where Envoy writes the base id to a file.
Converts tests to use the dynamic base id selection rather than
trying to keep all base ids unique.
Signed-off-by: Stephan Zuercher <zuercher@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 69f2dfcae15401f14d7cc3c829fd32ff2efc9f0c
Add EndpointsConfigDump message to support EDS in config_dump.proto
(not implemented in Envoy)
Additional Description:
Risk Level: Low
Testing: N/A
Docs Changes: N/A
Release Notes: N/A
This is the first step to solve #3362
Signed-off-by: Yutong Li <yutongli@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 192cd1341643f3a01bb27f992bbb6f0950eef867
In order to speed up eds, don't necessarily visit every proto field to count its validity as WarningValidationVisitor does. This yields a ~30% speed improvement in processing very large updates in EDS.
Risk Level: medium, new feature behind a command line flag.
Testing: Unit and bechmark tests.
Docs Changes: These are probably wrong, thus the draft-ness.
Release Notes: EDS can now ignore unknown dynamic fields, for a ~30% improvement in update processing time. Behind --ignore-unknown-dynamic-fields
Co-authored-by: Joshua Marantz <jmarantz@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 703f2fbdd4d2b11db901a74d15726bc6c017189b
Deprecates GoogleRE2.max_program_size field so that client does not need to check this.
Risk Level: Low
Testing: N/A
Docs Changes: Included in PR
Release Notes: Included in PR
Signed-off-by: Mark D. Roth <roth@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 10125161be0d0a759c3ffb02ddcdf8abc0bc6060
Adds a --boostrap_version flag that can be used to determine which API version the bootstrap should be parsed as.
Risk Level: Low
Testing: UTs
Docs Changes: Flag docs
Release Notes: n/a
Fixes#10343
Signed-off-by: Snow Pettersen <aickck@gmail.com>
Co-authored-by: Snow Pettersen <kpettersen@netflix.com>
Mirrored from https://github.com/envoyproxy/envoy @ 62777e87edc9f16af5c7920bb86f198585789a3d
This patch performs a major version freeze and bump by modifying
package_version_status, using the tooling developed in #10636.
Specifically:
v2 APIs are frozen (except for where they are the latest in their
package history and still active)
v3 APIs are transitioned to be active
Candidate v4alpha APIs are generated (not used by Envoy yet)
Fixes#10355
Risk level: medium (entire API's files are modified, visually verified
to ensure things look sane, all tests pass)
Testing: CI
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 549164c42cae84b59154ca4c36009e408aa10b52
Part of https://github.com/envoyproxy/envoy/issues/10355, this patch introduces additional
annotations to the API to support automatic inference (and developer documentation) of where the
active developer editable version of a file is, and which files are frozen or machine generated.
Risk level: Low (API annotations only)
Testing: CI
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4c5f4310bb8a19a38e5377d7a2d2dc6aa4560f47
This PR introduces a parallel v3 API reference documentation tree to the
existing v2 one.
The docs/root/api-v3/ tree was copied from docs/root/api-v2 and the
necessary package path fixups were made manually. As a result, the tree
largely resembles the v2 docs. Long term this is likely to evolve to
reflect the shape of the new extensions tree.
The message type, field and enum anchors are sed'ed to be distinct and
self-consistent inside api-v3/.
There were a number of API proto changes that were made to obtain a
successful Sphinx build:
* References to deprecated fields were replaced by references to the replacement field.
* clang-format line wrapping in protoxform was removed, this breaks RST in some v3 protos.
* Some packages (type/metadata/v2, data/cluster/v2alpha) were force upgraded to v3, to deal with references to types that are distinct in v2/v3. This is OK as these packages probably make sense to bump for v3, in general we're going to have to think about how to do this more
cleanly, supporting dual v2/v3 references alongside each other.
* Some evil hacks for field renaming added to migrate.py for RouteAction.
There's also some additional machinery added to compute distinct v3/v3
build targets to point protodoc at.
Risk level: Low
Testing: Docs build, manual inspection.
Fixes#8087
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ ac88316892cd47b6a9b58e3736e20e8863cd0d27
In which we convert every v3alpha reference to v3. In future revs of the
stable API versioning policy, we will develop better tooling to support
> 2 alpha and stable versions. For v3, it seems reasonable to just mv
v3alpha to v3, since there should be no external consumers yet.
Risk level: Low
Testing: bazel test //test/..., CI.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 5248a4fb7d4c2a3d1fa151f944d3a63f6b7a06cf
Changing from relative name to absolute name, and fixing the fatal-by-defaults that were broken by the v3 switch.
The old way to allow fatal-by-defaults was
envoy.deprecated_features:proto_file.proto:field_name
the new way is
envoy.deprecated_features:full.namespace.field_name
When we switched to v3, all the hard-coded v2 names stopped working. This reinstates them via hopefully more permanent proto annotation.
The only remaining ugly bit is that unfortunately the full namespace and field name are the v3 versions even if the original config was v2. Between @htuch and I we should fix that before merging.
Risk Level: Medium
Testing: added new unit tests
Docs Changes: updated
Release Notes: n/a
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 156d7c90083c196a206c07fc03b2de6be8260bd3
To better support config dump, deprecated field detection and other debug, it's helpful to leave a type name breadcrumb and be able to synthesize a Protobuf::Message that corresponds to what was delivered on the wire.
While working on this PR, it became apparent that config dump is broken post v3alpha, since a single config dump might have both v2 and v3 Listeners, etc. The only way to resolve this generically is to make the inner resources in config dump Any. This is a breaking API change, but these are v2alpha/v3alpha at this point, so allowed.
Risk level: Low
Testing: new version converter unit test, config dump tests now verify that the correct versioned inner resource is returned.
Fixes#9612
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 233838a39fb8310ea145e7d758d428d5a37b0306
This allows for a clean separation of config/service in v3. This is a
continuation of #9548.
Risk level: Low
Testing: bazel test //test/...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ c3bddaee1912fcd1fedc4786aee830b2e4a7c599
Description:
Move packages around for #8120 and #8121
Risk Level: Med around messing up build.
Testing: CI
Docs Changes: in API/STYLE.md
Release Notes: N/A (v3alpha is not in use yet)
Fixes#8120
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 1371f2ef46582a72b5b3971147bd87c534011731
This commit adds query parameters to the /config_dump admin endpoint
that allow listing keys and dumping only config associated with a passed
key.
You can now filter /config_dump endpoint with the key query parameter, e.g. /config_dump?key=clusters. This uses keys such as "listeners" and "clusters" that are set in places like here. You can view all keys with /config_dump?list_keys.
Risk Level: low
Testing: unit tests
Docs Changes: updated config dump proto docs and admin interface docs
Release Notes: added
Fixes#8743
Signed-off-by: Spencer Lewis <slewis@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ 091a06f5126021cbff8629cba5e4bafa24e6603c
In order to get file level move annotation, import has to be before options.
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 062c895f499382ae61dead16db2a7e78b9146525
*Description:*
Add a new `--disable-extensions` flag that hides the named extensions
from their corresponding factory registry. The extensions are retained
in the registry factory, but the factory pointer is nulled so that
configuration attempts to use the extensions will fail.
*Risk Level:* Medium
*Testing:* Added unit tests, manual verification.
*Docs Changes:* Added to CLI reference.
*Release Notes:* Added release note.
*Fixes:* #9096
Signed-off-by: James Peach <jpeach@apache.org>
Mirrored from https://github.com/envoyproxy/envoy @ 445d0ee4a64738f1de072537fb364d9096478915
Instead of formatting options heuristically, which will erase new annotations without changing protoxform, use proto descriptor to format options, and enforce its order as well.
Risk Level: Low
Testing: CI
Docs Changes: N/A
Release Notes: N/A
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ dfe687d49574ef7eb1bf84867bf571e805a2bf97
* api: link to previous message type package in API BUILD files.
We need to include the descriptors from the previous message version in
the build. We opt to do this transitively; when you include v3 of a
package, you get the v2 via a transitive dep. This should work based on
alwayslink semantics for cc_library.
The computation of the deps is based on the previous_message_type
annotation, which will allow cross package migrations.
Part of #8082.
Risk level: Low
Testing: Disabled ip_tagging v2 descriptor hack, observed
version_integration_test. After the BUILD changes, this now passes
again.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7f8fb9509d3189819dd253e25ec76e939ae106e7
This PR avoids having to include an API type database in the Envoy build
by introducing a message annotation option that allows Envoy to
determine earlier corresponding message types via descriptor inspection.
The ApiTypeDb is now ApiTypeOracle and utilizes these annotations.
Risk level: Low
Testing: Existing API and verison upgrade tests pass.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 297f7a73b3f93bccf8af73c0a555ae52bce6cecb
Currently, application logs are not sanitized of c-style escape sequences. If any filter logs a message that contains newline characters, the logs will be printed to a new line. This breaks log formats set by the --log-format option, breaking integration with log viewers.
This change adds a command line option --log-format-escaped to escape c-style escape characters in application logs before they are outputted. Enabling this flag ensures newline characters in logs are ignored, meaning that each call to ENVOY_LOG will result in at most 1 line outputted. This flag works for both Stderr and File loggers.
Risk Level: Low
Testing:
Unit tests
Fuzz test
Manual verification (see comments in PR)
Performance Impact: As long as production environments are running with the default log level, this will only slightly impact startup time (only when --log-format-escaped is set). The critical section for each request/response will not be impacted.
Docs Changes: Added docs to command line options about new flag and possible use cases, like Stackdriver Logging integration on GKE.
Release Notes: Added release notes about new flag
Fixes#8637
Signed-off-by: Teju Nareddy <nareddyt@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 20ca0ae3bdd9c2a69194203f5e1d2eca92ce2b48
Description: Tracking load status for LDS using the ConfigDump protos
Risk Level: Medium: major changes to an existing (alpha) system
Testing: new unit tests
Docs Changes: API notes inline
Release Notes: n/a
#8039
Signed-off-by: Alyssa Wilk <alyssar@chromium.org>
Mirrored from https://github.com/envoyproxy/envoy @ 0aed05aecd17576b0dd96f3e4126acd1c24a02bc
Modifies the pedantic spell checker to better handle camel-case words.
Each part the word is now treated as a separate error in both the check
and fix modes.
Disables run-together mode for aspell, which allowed typos such as
"mananger" (man + anger). Fixes the resulting spelling errors.
Miscellaneous other fixes:
* Provides an option to replace a word without adding the word to the
dictionary (in case aspell's suggestions do not contain the correct
replacement).
* Fixes a bug in the script when no suggestions are returned by aspell.
* Checks the dictionary and added words for invalid characters that cause
aspell errors at dictionary load time.
* Sets the mark flag when in CI runs so that misspelled words are indicated.
* Culls words from the dictionary that are no longer in the codebase, or are
otherwise not needed.
Risk Level: low (comments only)
Testing: n/a
Doc Changes: n/a
Release Notes: n/a
Fixes: #8481
Signed-off-by: Stephan Zuercher <zuercher@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ de70fe54cdba6b022b3971379afa535f402f2ffe
This reverts commit 596cd4894c8ecd536c1da1dddecaae3531f269ea.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8c4afa6fee5e104c7b195657a3c22e23f062eb11
Modifies the pedantic spell checker to better handle camel-case words.
Each part the word is now treated as a separate error in both the check
and fix modes.
Disables run-together mode for aspell, which allowed typos such as
"mananger" (man + anger). Fixes the resulting spelling errors.
Miscellaneous other fixes:
* Provides an option to replace a word without adding the word to the
dictionary (in case aspell's suggestions do not contain the correct
replacement).
* Fixes a bug in the script when no suggestions are returned by aspell.
* Checks the dictionary and added words for invalid characters that cause
aspell errors at dictionary load time.
* Sets the mark flag when in CI runs so that misspelled words are indicated.
* Culls words from the dictionary that are no longer in the codebase, or are
otherwise not needed.
Risk Level: low (comments only)
Testing: n/a
Doc Changes: n/a
Release Notes: n/a
Fixes: #8481
Signed-off-by: Stephan Zuercher <zuercher@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 596cd4894c8ecd536c1da1dddecaae3531f269ea
Generate or format next free field annotation via protoxform.
Risk Level: low
Testing: N/A
Docs Changes: N/A
Release Notes: N/A
Fixes#8429
Signed-off-by: Yi Tang <ssnailtang@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 986173ed516dcc1c3dea7db90659ed993d0aad75
This patch moves away from the paradigm of sed-style upgrading of every v2
package to v3alpha. Instead, an additional type analysis phase is
performed prior to protoxform by a protoc plugin known as the "type
whisperer".
The type whisperer produces structured type dependency information for
each .proto. The tools/type_whisperer/typedb_gen.py tool then knits
these together to provide an API-wide type dependency graph. This is
then used to determine which types need upgrading (either they have
breaking changes or transitively depend on types with such changes).
Only packages with upgraded types now undergo the v2 -> v3alpha
transition.
The API type database is checked into
source/common/config/api_type_db.pb. This may seem a strange location,
but in the future we will include the type database as a build artifact
for the Envoy binary, as it will be used by the reflection-based version
converter to find the type upgrade path for input proto.
Risk level: Low (the v3alpha protos are not used yet).
Testing: fix_format, manual inspection of diffs, bazel test //test/...,
docs build.
Part of #8082Fixes#8490
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ ad57b58cfbb256af41a467260dce2a8013b7a7fa
This provides canonical BUILD formatting and puts protoxform in charge
of being able to determine import paths, without having to worry about
Bazel implications.
Part of #8082.
Risk level: Low
Testing: tools/proto_sync.py, visual inspection of diffs.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e53f40f0e5ccc84fca5cd350416fe0f2accf8229
As part of #8082, we want to be able to (1) automatically generate BUILD
files and (2) treat packages as atomic from a "upgrade / do not upgrade"
decision perspective. This is simplified by having our BUILD targets at
package granularity, since this is what the protoxform plugin operates
on.
This PR broadens the package-level treatment that was already introduced
for Go in #8003 to Python and C++. This simplifies BUILD files
significantly and opens the way to automated generation.
There is some technical debt introduced, since all visibility controls
have been removed. This is slated for reintroduction in
https://github.com/envoyproxy/envoy/issues/8491.
As a bonus (useful for BUILD file generation), also removed the
inconsistency in BUILD package target naming for packages in envoy.api.*
and envoy.type.*. E.g. //envoy/api/v2:v2 is now //envoy/api/v2:pkg.
Risk level: Low (but this will break internal builds and require BUILD
fixups to consuming projects).
Testing: bazel test //test/... @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4e858f17fe08224c9c089240908ccd0c518e01a7
We should start tagging messages with "option deprecated = true;" if we
want them to be auto-deprecated by protoxform going forward.
Risk level: Low (v3alpha is not used yet)
Testing: bazel test @envoy_api//..., manual diff inspection.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e3247539bcc39f3111287b479a85e51655a6f1db
Via ./api/migration/v3alpha.sh. This picks up the changes since the last
sync, in particular the major reformat in #8309.
Risk level: Low (not used yet).
Testing: bazel build @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ c41cfbf6a33b8115a7e29a2b4a926aad4cd062be
This patch introduces a new tool, protoxform, that will be the basis of
the v2 -> v3 migration tooling. It operates as a Python protoc plugin,
within the same framework as protodoc, and provides the ability to
operate on protoc AST input and generate proto output.
As a first step, the tool is applied reflexively on v2, and functions as
a formatting tool. In later patches, this will be added to
check_format/fix_format scripts and CI.
Part of #8082.
Risk level: medium (it's possible that some inadvertent wire changes
occur, if they do, this patch should be rolled back).
Testing: manual inspection of diff, bazel test //test/..., some
grep/diff scripts to ensure we haven't lost any comments.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 08b123a8321d359ea66cbbc0e2926545798dabd3
Remove gogoproto annotations. They can be replaced with a custom gogoproto compiler (e.g. something like https://github.com/gogo/googleapis/tree/master/protoc-gen-gogogoogleapis). I have an experimental version of it to validate that it's possible to re-apply important annotations in the compiler.
Risk Level: low
Testing: builds
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e7f0b7176efdc65f96eb1697b829d1e6187f4502
Fixes#7982
Defines a package level proto library and its associated internal go_proto_library.
Deletes all existing api_go_proto_library, api_go_grpc_library, and go_package annotations in protos (they are not required and pollute the sources).
I deliberately avoided touching anything under udpa since it's being moved to another repository.
Risk Level: low
Testing: build completes
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ d504fde0ffd97017d1ddff8caa9a3b46bba9ae48
This patch establishes a v3alpha baseline API, by doing a simple copy of
v2[alpha] dirs and some sed-style heuristic fixups of BUILD dependencies
and proto package namespaces.
The objective is provide a baseline which we can compare the output from
tooling described in #8083 in later PRs, providing smaller visual diffs.
The core philosophy of the API migration is that every step will be
captured in a script (at least until the last manual steps),
api/migration/v3alpha.sh. This script will capture deterministic
migration steps, allowing v2[alpha] to continue to be updated until we
finalize v3.
There is likely to be significant changes, e.g. in addition to the work
scoped for v3, we might want to reduce the amount of API churn by
referring back to v2 protos where it makes sense. This will be done via
tooling in later PRs.
Part of #8083.
Risk level: Low
Testing: build @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 085d72b490c124a02849812798f5513a8df9ae72
data-plane-api(CircleCI)