This patch performs a major version freeze and bump by modifying
package_version_status, using the tooling developed in #10636.
Specifically:
v2 APIs are frozen (except for where they are the latest in their
package history and still active)
v3 APIs are transitioned to be active
Candidate v4alpha APIs are generated (not used by Envoy yet)
Fixes#10355
Risk level: medium (entire API's files are modified, visually verified
to ensure things look sane, all tests pass)
Testing: CI
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 549164c42cae84b59154ca4c36009e408aa10b52
This patch introduces the tools/docs for a new flow for protoxform: freeze.
This performs a major version freeze and bump by modifying
package_version_status. Tooling now needs to deal with 3 possible versions (v2,
v3, v4alpha). The following contributions are made in the patch:
* Tooling added to support v2 APIs freeze, transitioning v3 to active and generating v4alpha.
* A new tool, merge_active_shadow.py, is introduced to combine hand-edited active v3 API protos with the v2 implied shadows for deprecated fields.
* The type database now spans v2 and v3 (and implied v4alpha files)
* api_proto_plugins can now take extra args on the Bazel CLI; this is used to plumb in the freeze arg to protoxform.
Docs are updated to reflect the v3 flow that developers should now
follow.
Part of #10355
This is the tools/docs component of #10601. See #10601 for the resulting
API protos generated by a freeze/fix cycle.
Risk level: Low (tooling only, no API changes yet)
Testing: new golden protos tests for protoxform, unit tests for merge_active_shadow.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ de222c3acf1fbf124ca44f50495fc4b8e9aba525
Allow using a custom host name for endpoints and health checks. This enables auto host re-write to work with EDS
Risk Level: Low, new opt in field.
Testing: Unit tests
Docs Changes: inline docs in the protocs.
Release Notes: added to version_history.rst
Fixes#10408
Signed-off-by: Yuval Kohavi <yuval.kohavi@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ b5a3405b2725e979a93e7410fce7c57a0c66d8db
Part of https://github.com/envoyproxy/envoy/issues/10355, this patch introduces additional
annotations to the API to support automatic inference (and developer documentation) of where the
active developer editable version of a file is, and which files are frozen or machine generated.
Risk level: Low (API annotations only)
Testing: CI
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4c5f4310bb8a19a38e5377d7a2d2dc6aa4560f47
In which we convert every v3alpha reference to v3. In future revs of the
stable API versioning policy, we will develop better tooling to support
> 2 alpha and stable versions. For v3, it seems reasonable to just mv
v3alpha to v3, since there should be no external consumers yet.
Risk level: Low
Testing: bazel test //test/..., CI.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 5248a4fb7d4c2a3d1fa151f944d3a63f6b7a06cf
This allows for a clean separation of config/service in v3. This is a
continuation of #9548.
Risk level: Low
Testing: bazel test //test/...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ c3bddaee1912fcd1fedc4786aee830b2e4a7c599
Description:
Move packages around for #8120 and #8121
Risk Level: Med around messing up build.
Testing: CI
Docs Changes: in API/STYLE.md
Release Notes: N/A (v3alpha is not in use yet)
Fixes#8120
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 1371f2ef46582a72b5b3971147bd87c534011731
In order to get file level move annotation, import has to be before options.
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 062c895f499382ae61dead16db2a7e78b9146525
Instead of formatting options heuristically, which will erase new annotations without changing protoxform, use proto descriptor to format options, and enforce its order as well.
Risk Level: Low
Testing: CI
Docs Changes: N/A
Release Notes: N/A
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ dfe687d49574ef7eb1bf84867bf571e805a2bf97
This has not been implemented, so hide the API from the docs.
Signed-off-by: Snow Pettersen <snowp@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8e368e4bdfa8f220c5f4cb03ca61587ae1a3d118
Generate or format next free field annotation via protoxform.
Risk Level: low
Testing: N/A
Docs Changes: N/A
Release Notes: N/A
Fixes#8429
Signed-off-by: Yi Tang <ssnailtang@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 986173ed516dcc1c3dea7db90659ed993d0aad75
This patch moves away from the paradigm of sed-style upgrading of every v2
package to v3alpha. Instead, an additional type analysis phase is
performed prior to protoxform by a protoc plugin known as the "type
whisperer".
The type whisperer produces structured type dependency information for
each .proto. The tools/type_whisperer/typedb_gen.py tool then knits
these together to provide an API-wide type dependency graph. This is
then used to determine which types need upgrading (either they have
breaking changes or transitively depend on types with such changes).
Only packages with upgraded types now undergo the v2 -> v3alpha
transition.
The API type database is checked into
source/common/config/api_type_db.pb. This may seem a strange location,
but in the future we will include the type database as a build artifact
for the Envoy binary, as it will be used by the reflection-based version
converter to find the type upgrade path for input proto.
Risk level: Low (the v3alpha protos are not used yet).
Testing: fix_format, manual inspection of diffs, bazel test //test/...,
docs build.
Part of #8082Fixes#8490
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ ad57b58cfbb256af41a467260dce2a8013b7a7fa
This provides canonical BUILD formatting and puts protoxform in charge
of being able to determine import paths, without having to worry about
Bazel implications.
Part of #8082.
Risk level: Low
Testing: tools/proto_sync.py, visual inspection of diffs.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e53f40f0e5ccc84fca5cd350416fe0f2accf8229
As part of #8082, we want to be able to (1) automatically generate BUILD
files and (2) treat packages as atomic from a "upgrade / do not upgrade"
decision perspective. This is simplified by having our BUILD targets at
package granularity, since this is what the protoxform plugin operates
on.
This PR broadens the package-level treatment that was already introduced
for Go in #8003 to Python and C++. This simplifies BUILD files
significantly and opens the way to automated generation.
There is some technical debt introduced, since all visibility controls
have been removed. This is slated for reintroduction in
https://github.com/envoyproxy/envoy/issues/8491.
As a bonus (useful for BUILD file generation), also removed the
inconsistency in BUILD package target naming for packages in envoy.api.*
and envoy.type.*. E.g. //envoy/api/v2:v2 is now //envoy/api/v2:pkg.
Risk level: Low (but this will break internal builds and require BUILD
fixups to consuming projects).
Testing: bazel test //test/... @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4e858f17fe08224c9c089240908ccd0c518e01a7
This patch introduces a new tool, protoxform, that will be the basis of
the v2 -> v3 migration tooling. It operates as a Python protoc plugin,
within the same framework as protodoc, and provides the ability to
operate on protoc AST input and generate proto output.
As a first step, the tool is applied reflexively on v2, and functions as
a formatting tool. In later patches, this will be added to
check_format/fix_format scripts and CI.
Part of #8082.
Risk level: medium (it's possible that some inadvertent wire changes
occur, if they do, this patch should be rolled back).
Testing: manual inspection of diff, bazel test //test/..., some
grep/diff scripts to ensure we haven't lost any comments.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 08b123a8321d359ea66cbbc0e2926545798dabd3
Remove gogoproto annotations. They can be replaced with a custom gogoproto compiler (e.g. something like https://github.com/gogo/googleapis/tree/master/protoc-gen-gogogoogleapis). I have an experimental version of it to validate that it's possible to re-apply important annotations in the compiler.
Risk Level: low
Testing: builds
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e7f0b7176efdc65f96eb1697b829d1e6187f4502
Fixes#7982
Defines a package level proto library and its associated internal go_proto_library.
Deletes all existing api_go_proto_library, api_go_grpc_library, and go_package annotations in protos (they are not required and pollute the sources).
I deliberately avoided touching anything under udpa since it's being moved to another repository.
Risk Level: low
Testing: build completes
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ d504fde0ffd97017d1ddff8caa9a3b46bba9ae48
Due to a seg fault issue with the gogo protobuf library
[https://github.com/gogo/protobuf/issues/568], non nullable repeated
fields in a proto will cause proto.Merge(dst, src) to panic.
The nullable field setting was first added by @kyessenov when he was
re-organizing the protos. Unfortunately, people have been copy pasting it
across several areas in the Envoy proto. To keep the impact radius to a minimum,
I have updated only the fields that are currently causing the segfault
(in go-control-plane) for us.
Its also partly against proto principles. You should be able to determine if
a field is set or not. This non-nullable setting in gogo will insist on initializing
the field to default values.
Risk Level: to go control plane users
Signed-off-by: Shriram Rajagopalan <rshriram@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ b22d2b5cf09f779962cfedaaab24969f384cbc48
Changes: Fixes coverage test hang.
No change to core logic from #6692.
Signed-off-by: Karthik Reddy <rekarthik@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 27b3e48e8d754dcb956dca58888c7dbfde7ceec8
Description: Adding API support to report Locality proximity information as part of Locality Assignment. This does not need to be documented yet, and should be documented as part of LB algorithm implementation which will use this information.
Risk Level: LOW
Testing: None
Signed-off-by: Vishal Powar <vishalpowar@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 5e039a6581cf5fa1f7b70329afeb2e3d35b8189c
total_issued_requests field tracks the count of requests issued since
the last report. This field will be used for global load balancing
decisions.
Envoy maintains a stats counter rq_total to track total
requests made. By latching this counter when load reporting period
begins, we are able to count the total requests issued in a load
reporting interval. This information is then used to populate the field
total_issued_requests.
Risk Level: Low
Testing: //test/integration:load_stats_reporter passes.
Signed-off-by: Karthik Reddy <rekarthik@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ cdddf54e8548036703ee928335a44c3b503b988b
This new field gives LRS management server the extra info that two clusters may be using the same set of hosts to serve, and it's the management server's call whether to aggregate on the clusterStats.
Risk Level: LOW (new field)
Testing: unit test
Fixes#5986
Signed-off-by: Xin Zhuang <stevenzzz@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ f5bb174c5f2f598df497cec04b0e144af9612570
Adds the API for an additional EDS indirection that allows endpoints to
specified outside the LB structure. This opens up for being able to
reference the same endpoint multiple times in a single CLA.
Risk Level: Low, only API changes for now
Testing: n/a
Docs Changes: n/a
Release Notes: n/a
#4280
Signed-off-by: Snow Pettersen <snowp@squareup.com>
Mirrored from https://github.com/envoyproxy/envoy @ b49e37989160913e91a42824c91b664de82e7a05
* api: add proto options for java
* add ci for checking proto options
Signed-off-by: Penn (Dapeng) Zhang <zdapeng@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 02659d411332e9f20d229f482931c15304ea17fd
Description: This field is used to carry opaque and implementation
dependent information of the upstream endpoints. The information may
be used by management server for debugging purposes.
Example: Consider a requirement of per 'user' load stastics for
debugging. Envoy will embed user info into the metadata field for
every upstream endpoint it sends load to. This user information will
be used by management server for debugging.
Sample message:
message ClusterStats {
cluster_name = ...
message UpstreamLocalityStats {
locality = ...
message UpstreamEndpointStats {
address = "endpoint1"
metadata = { "user" : "alice"}
...
}
message UpstreamEndpointStats {
address = "endpoint1"
metadata = { "user" : "bob"}
...
}
message UpstreamEndpointStats {
address = "endpoint2"
metadata = { "user" : "alice"}
...
}
message UpstreamEndpointStats {
address = "endpoint3"
metadata = { "user" : "bob"}
...
}
}
}
Risk Level: Low
Testing: Compiles successfully.
Signed-off-by: Karthik Reddy <rekarthik@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8d8cb4a7c63b74633b0b459af9104143b13f657f
The issue is due to on the crash of Envoy::Network::Utility::getAddressWithPort because of the invalid port_value. Added max constraint validate rule to the port_value field.
Risk Level: Low
Testing: Tested unit tests (bazel test //test/server/config_validation:config_fuzz_test), built and ran fuzzers with oss-fuzz.
Signed-off-by: Anirudh M <m.anirudh18@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7d61b00171fb3cc349a9d834dafdf10f0c70adda
This PR contains changes to implement feature requested in issue #3823
- Adding DropOverload in eds policy which can be used to specify
drop_percentage per category.
- Adding DroppedRequests in load_report which can report deliberately
dropped requests for each category.
Signed-off-by: vishalpowar <vishal.powar@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ 3ee3aa34d826fa1783c201fbf99dfec3a21eed53
This patch implements load_assigment field in CDS' Cluster.
This change specifically adds the implementation of the new load_assigment field
for clusters with discovery-type: STATIC, STRICT_DNS and LOGICAL_DNS.
While adding this load_assigment field implementation to Cluster,
this patch also allows specifying optional (active) health check config per specified upstream host.
Risk Level: medium
Testing: unit tests
Docs Changes:
This unhides docs for endpoint health check config
Release Notes: N/A
Fixes#439
Signed-off-by: Dhi Aurrahman <dio@rockybars.com>
Mirrored from https://github.com/envoyproxy/envoy @ b32eabfc141760ec14622a4a2a2f0ab0a741cd6c
This PR avoids a situation in which we were losing track of data plane requests as follows:
1. Management server asks Envoy to track load stats for cluster Foo in a LoadStatsResponse for a 10s period. Envoy resets stats for Foo.
2. After the 10s timer, Envoy responds with Foo's stats, resetting them.
3. Management server asks Envoy to track load stats for cluster Foo in a LoadStatsResponse for a 10s period. Envoy resets stats for Foo.
4. After the 10s timer, Envoy responds with Foo's stats, resetting them.
Between 2 and 3, any stats for Foo requests that arrive were previously unaccounted for. We resolve
this (in a relatively backward compatible way) by not making any protocol changes except to require
Envoy to not reset stats for already tracked clusters.
If we were to design LRS from scratch to avoid this, there are better approaches, e.g. making it a
periodic reporting service rather than request-response, but we probably already have a bunch of
existing users of LRS and don't want to break them.
Rist Level: Low
Testing: Modified load_stats_integration_test.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 27c99de910788c2c5ca95a87cee00e55a33da638
adds the required visibility rules and delegates the rest to the generic
api_proto_library. I tested the change by doing the following without
getting errors.
./ci/run_envoy_docker.sh './ci/do_ci.sh docs'
I changed the BUILD files using the following commands.
/envoy/api$ find . -type f -name BUILD | xargs sed -i -e 's/api_proto_library(/api_proto_library_internal(/g'
envoy/api$ find . -type f -name BUILD | xargs sed -i -e 's/"api_proto_library"/"api_proto_library_internal"/g'
Signed-off-by: mickey <mickeyju@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4b871c0ab9350882271a490adcee44e613ed9807
Add load_assignment field in Cluster
This patch introduces load_assigment field in CDS' Cluster. This is an API change only.
This is part of effort on breaking #3261 into multiple PRs.
Risk Level:
Low, since it is hidden.
Testing:
Build api and envoy-static without error
Docs Changes:
Add load_assignment in Cluster of cds.proto.
Signed-off-by: Dhi Aurrahman <dio@rockybars.com>
Mirrored from https://github.com/envoyproxy/envoy @ 79bce5fe1cd8d1ab03dc6085497fcda653320a67
*Risk Level*: None
*Testing*: bazel test //test/...
*Docs Changes*: n/a
*Release Notes*: n/a
Found with buildifier.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 0e8964c83f359916ecbf9c01a03ade3c92aac479
This provides an alternative health check port, if set it allows an
upstream host to have different health check address port.
Signed-off-by: Dhi Aurrahman <dio@rockybars.com>
Introduce the concept of locality weighted LB (as distinct from zone
aware LB) in the docs and a new field in Cluster, locality_weighted_lb,
for configuring this behavior.
Signed-off-by: Harvey Tuch <htuch@google.com>
There are several main changes in this PR:
Create envoy.api.v2.core packages to break circular dependencies from xDS on to subpackages on to base protos.
Create individual packages for each filter and add independent versioning to each filter.
Add visibility constraints to prevent formation of dependency cycles.
Add gogoproto annotations to improve go code generation.
After moving xDS service definitions and top-level resource protos back to envoy.core.api.v2, cycles were created, since the second-level definitions depend on base protobuf definitions, and are in turn included from xDS; however xDS and base definitions are in the same package.
The solution is to split the base protos into another package, envoy.api.v2.core. That eliminates dependency cycles (validated using go-control-plane).
Added a few gogoproto annotations to improve golang code generation.
Signed-off-by: Kuat Yessenov <kuat@google.com>
data-plane-api(CircleCI)
htuch
Dhi Aurrahman
Daniel Hochman
Kuat