This patch moves away from the paradigm of sed-style upgrading of every v2
package to v3alpha. Instead, an additional type analysis phase is
performed prior to protoxform by a protoc plugin known as the "type
whisperer".
The type whisperer produces structured type dependency information for
each .proto. The tools/type_whisperer/typedb_gen.py tool then knits
these together to provide an API-wide type dependency graph. This is
then used to determine which types need upgrading (either they have
breaking changes or transitively depend on types with such changes).
Only packages with upgraded types now undergo the v2 -> v3alpha
transition.
The API type database is checked into
source/common/config/api_type_db.pb. This may seem a strange location,
but in the future we will include the type database as a build artifact
for the Envoy binary, as it will be used by the reflection-based version
converter to find the type upgrade path for input proto.
Risk level: Low (the v3alpha protos are not used yet).
Testing: fix_format, manual inspection of diffs, bazel test //test/...,
docs build.
Part of #8082Fixes#8490
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ ad57b58cfbb256af41a467260dce2a8013b7a7fa
This provides canonical BUILD formatting and puts protoxform in charge
of being able to determine import paths, without having to worry about
Bazel implications.
Part of #8082.
Risk level: Low
Testing: tools/proto_sync.py, visual inspection of diffs.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e53f40f0e5ccc84fca5cd350416fe0f2accf8229
As part of #8082, we want to be able to (1) automatically generate BUILD
files and (2) treat packages as atomic from a "upgrade / do not upgrade"
decision perspective. This is simplified by having our BUILD targets at
package granularity, since this is what the protoxform plugin operates
on.
This PR broadens the package-level treatment that was already introduced
for Go in #8003 to Python and C++. This simplifies BUILD files
significantly and opens the way to automated generation.
There is some technical debt introduced, since all visibility controls
have been removed. This is slated for reintroduction in
https://github.com/envoyproxy/envoy/issues/8491.
As a bonus (useful for BUILD file generation), also removed the
inconsistency in BUILD package target naming for packages in envoy.api.*
and envoy.type.*. E.g. //envoy/api/v2:v2 is now //envoy/api/v2:pkg.
Risk level: Low (but this will break internal builds and require BUILD
fixups to consuming projects).
Testing: bazel test //test/... @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4e858f17fe08224c9c089240908ccd0c518e01a7
Via ./api/migration/v3alpha.sh. This picks up the changes since the last
sync, in particular the major reformat in #8309.
Risk level: Low (not used yet).
Testing: bazel build @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ c41cfbf6a33b8115a7e29a2b4a926aad4cd062be
This patch introduces a new tool, protoxform, that will be the basis of
the v2 -> v3 migration tooling. It operates as a Python protoc plugin,
within the same framework as protodoc, and provides the ability to
operate on protoc AST input and generate proto output.
As a first step, the tool is applied reflexively on v2, and functions as
a formatting tool. In later patches, this will be added to
check_format/fix_format scripts and CI.
Part of #8082.
Risk level: medium (it's possible that some inadvertent wire changes
occur, if they do, this patch should be rolled back).
Testing: manual inspection of diff, bazel test //test/..., some
grep/diff scripts to ensure we haven't lost any comments.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 08b123a8321d359ea66cbbc0e2926545798dabd3
Remove gogoproto annotations. They can be replaced with a custom gogoproto compiler (e.g. something like https://github.com/gogo/googleapis/tree/master/protoc-gen-gogogoogleapis). I have an experimental version of it to validate that it's possible to re-apply important annotations in the compiler.
Risk Level: low
Testing: builds
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ e7f0b7176efdc65f96eb1697b829d1e6187f4502
Fixes#7982
Defines a package level proto library and its associated internal go_proto_library.
Deletes all existing api_go_proto_library, api_go_grpc_library, and go_package annotations in protos (they are not required and pollute the sources).
I deliberately avoided touching anything under udpa since it's being moved to another repository.
Risk Level: low
Testing: build completes
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ d504fde0ffd97017d1ddff8caa9a3b46bba9ae48
This patch establishes a v3alpha baseline API, by doing a simple copy of
v2[alpha] dirs and some sed-style heuristic fixups of BUILD dependencies
and proto package namespaces.
The objective is provide a baseline which we can compare the output from
tooling described in #8083 in later PRs, providing smaller visual diffs.
The core philosophy of the API migration is that every step will be
captured in a script (at least until the last manual steps),
api/migration/v3alpha.sh. This script will capture deterministic
migration steps, allowing v2[alpha] to continue to be updated until we
finalize v3.
There is likely to be significant changes, e.g. in addition to the work
scoped for v3, we might want to reduce the amount of API churn by
referring back to v2 protos where it makes sense. This will be done via
tooling in later PRs.
Part of #8083.
Risk level: Low
Testing: build @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 085d72b490c124a02849812798f5513a8df9ae72
Introduces a generic expression-based admission filter using https://github.com/google/cel-cpp.
This is a follow-up to discussion in https://github.com/envoyproxy/envoy/issues/6751.
The advantage of this approach is:
1. Un-opinionated about the policy structure since the only config is an expression. This is friendly towards control planes which can bear the complexity of translation, analysis, and evolution of policies.
2. Multi-language, CEL supports go, java, and c++ runtimes.
3. Inter-operability with other filters using request `metadata`. Companion filters can populate metadata about requests and resources that affect policy decisions.
4. Generic utility, it can be used for custom metric labels, access log entries, etc.
The dis-advantage of this approach is that its performance is lower than domain-optimized interpreters. On a fair example, the interpreter evaluates in around 1ms (see https://github.com/google/cel-cpp/blob/master/eval/tests/benchmark_test.cc#L591) vs ~150ns for hand-written C++ native code. There is space for improvement (especially if WASM can be used as a compilation target), but ultimately the generic expression form carries a cost.
Conditions are added to support RBAC filter for complementing the existing principal/permission model. They add support for the extended checks (e.g. time of query, resource-bound), but add no cost unless used.
Description: add expression-based admission filter
Risk Level: low
Testing:
Docs Changes:
Release Notes:
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ f90e1b08ac5b4973c45a6529780ebdd211ff901f
Description: Adds support for DNS SAN as Principal in RBAC filter.
Risk Level: Low
Testing: Added automated tests
Docs Changes: Updated
Release Notes: Added
Fixes#7836
Signed-off-by: Rama Chavali <rama.rao@salesforce.com>
Mirrored from https://github.com/envoyproxy/envoy @ 882a30677619856446f7e1b9d28c6ab319b21d1b
Using proto.MarshalAny results in unstable output due to non-deterministic map ordering. This in turn causes Envoy's diff to reload a config since the hash of the structure changes.
Enable stable marshaler for gogoproto to avoid this problem. See #6252
Risk Level: low
Testing: n/a
Signed-off-by: Kuat Yessenov <kuat@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 15a19b9cb1cc8bd5a5ec71d125177b3f6c9a3cf5
* api: add proto options for java
* add ci for checking proto options
Signed-off-by: Penn (Dapeng) Zhang <zdapeng@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 02659d411332e9f20d229f482931c15304ea17fd
This commit adds support for SNI based permissions by matching over a
connection's requested server name.
Signed-off-by: Venil Noronha <veniln@vmware.com>
Mirrored from https://github.com/envoyproxy/envoy @ 811ee0dc52951acf2f66190587f92394473c245c
Correct commonly misspelled English words
Signed-off-by: zyfjeff <tianqian.zyf@alibaba-inc.com>
Mirrored from https://github.com/envoyproxy/envoy @ 31d0611d0ffb982414974adaf6940afd1863fa95
This PR added a new principal_name of type StringMatcher to rbac Authenticated and mark the existing user field as deprecated. This gives us more flexibility to express more matching rules against peer certificate.
Risk Level: Low
Testing: Added unit tests
Signed-off-by: Yangmin Zhu <ymzhu@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 5d731878fd0134ca15d5904450a64dab0ff577a9
Fix the description of an `RBAC` Protobuf message example, so that it
matches the changes made to the YAML code block in PR #3477.
Risk Level: Low
Testing: N/A
Docs Changes: N/A
Release Notes: N/A
Signed-off-by: Tal Nordan <tal.nordan@solo.io>
Mirrored from https://github.com/envoyproxy/envoy @ 0f68948434039eb5e2fe3610e9b315989d843296
adds the required visibility rules and delegates the rest to the generic
api_proto_library. I tested the change by doing the following without
getting errors.
./ci/run_envoy_docker.sh './ci/do_ci.sh docs'
I changed the BUILD files using the following commands.
/envoy/api$ find . -type f -name BUILD | xargs sed -i -e 's/api_proto_library(/api_proto_library_internal(/g'
envoy/api$ find . -type f -name BUILD | xargs sed -i -e 's/"api_proto_library"/"api_proto_library_internal"/g'
Signed-off-by: mickey <mickeyju@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4b871c0ab9350882271a490adcee44e613ed9807
Fixes https://github.com/envoyproxy/envoy/issues/743
This is a general cleanup of all of the access logging documentation.
I have reorganized a bunch of things and hidden the various gRPC logging
fields that are not implemented yet.
I've also moved the existing tap protos into a new "output" directory. This
is the best name I could come up for cleanly separating output data that might
be stored outside of any service or configuration.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ c15019e79c832d9f0a09468affaadabc4be3e115
*Risk Level*: None
*Testing*: bazel test //test/...
*Docs Changes*: n/a
*Release Notes*: n/a
Found with buildifier.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 0e8964c83f359916ecbf9c01a03ade3c92aac479
Added protos to support Role Based Access Control in Envoy.
Also removed existing auth.proto because the new RBAC proto is a replacement of it.
Ealier discussions at
envoyproxy/data-plane-api#586.
Signed-off-by: Limin Wang <liminwang@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 13de384ab34428af99c53201f6b3c95991b7ae10
data-plane-api(CircleCI)