This is a first step towards https://github.com/envoyproxy/envoy/issues/17920
A single proto (kafka mesh) has been swapped from using the udpa
file_status annotation to the xds file_status annotation to avoid a
large amount of churn and a forthcoming migration of many alpha/wip
protos to non alpha/wip. The rest will be audited and swapped in
future PRs. This single one was done to make sure the doc machinary
works properly.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ f0f17a3caa75106a9e28b99edc27dd09c1bed488
This PR is a continuation to #17515 - it adds a script that uses buf to check for breaking changes on proto files in the api folder. It does so by comparing the current api folder against the api folder at the git commit computed by tools/git/last_github_commit.sh - that should hopefully represent the most recent commit on main (if there is a better method to obtain the base branch commit, let me know!).
Adding the script also required re-organizing some of the breaking change detector logic from the previous pr: some levels of abstraction were added, and the detector now expects a git repository and ref as the input for initial state (rather than a proto file).
The script is invoked in do_ci.sh if bazel.api_compat is specified as the CI_TARGET.
This PR also bumps the buf bazel dependency to 0.53.0. If this is preferred to be in a separate PR, let me know and I would be happy to do so
Risk Level: low (hopefully) - the CI script will be invoked in a separate CI pipeline job that can be set to be optional on github. The azure pipeline has been added but needs to be set to optional by a CI maintainer
Testing: New scripts and logic were tested manually; also ran tests from the previous PR and they still pass. hoping to observe more output of this tool through reading CI logs of other PRs once this is merged (this PR should not affect the existing PR workflow - refer to Risk Level)
Docs Changes: n/a
Release Notes: n/a
Platform Specific Features: CI script uses a linux binary for buf so it cannot be run outside of docker on non-linux systems
Fixes#3368
Signed-off-by: Yaseen Alkhafaji <yaseena@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ f30c289e456df11d549d7236c56f22da00500e9e
Follow-up to #17375 where it was agreed that protolock is not actively maintained enough to depend on. This PR "migrates" the tests from that PR to use buf instead, and also cleans some of the code per a few of the review comments. Still a few outstanding points:
- buf build on the envoy/api folder requires several protobuf dependencies such as udpa to be available to buf to consume. Suggested solution by buf is to point buf's config to necessary BSR modules that the buf team is hosting.
- These lines are commented out in this PR as I had some trouble automating it for the tests, and it is not necessary for the tests to pass
- May introduce issues if buf is not pointing to the same version of modules that bazel builds for envoy. May need to introduce some way to couple them, or (ideally) find a way to run the breaking change detector without building the dependencies
- Currently bazel is using a binary release of buf (for linux). Goal is to move to building it from source in the near future
- It may be in our interest to expand the list of API-breaking-change rules (buf provides an extensive list of rules we could adopt)
Risk Level: Low
Testing: Tests that evaluate buf against "allowed" and "breaking" protobuf API changes. Currently 4 tests are skipped - 3 of them are PGV-related (we need to communicate our desired PGV rules to the buf team so they may add them in the near future). The 4th is a test I had originally written to evaluate protolock but may not apply to buf ("forcing" a breaking change) - refer to comments
Docs Changes:
Release Notes:
Platform Specific Features: buf binary imported by bazel is linux-only. Hopefully the ["manual"] tags attribute prevents any issues for non-linux users
Signed-off-by: Yaseen Alkhafaji <yaseena@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ c74cebb3cc7dfb26488f7cac8eb9258cf9a7588e
In order to unblock a dependency between route_components.proto and the matcher tree, move the alpha Matcher to
its own package. To provide an upgrade path for users using this, we'll keep the old copy around for a single deprecation
cycle.
Risk Level: Low
Testing: n/a
Docs Changes: n/a
Release Notes: Deprecation note
Signed-off-by: Snow Pettersen <snowp@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 40ed33327c23a9e4e88aec448694eb1d03098efd
Risk Level: Low; the protobuf types are not changing at all.
Testing: proto_format and tests
Signed-off-by: Tyler Dixon <tdixon@buf.build>
Co-authored-by: Snow Pettersen <snowp@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ c1bc5e78fa68b86236c8d6237e2db15ce1743459
- update to PGV 0.6.1 (release notes)
- update API versions to corresponding PGV and gRPC versions
- Bazel build tools to 4.0.1
- Zipkin API to 1.0.0
- OpenTelemetry Proto to 0.9.0
Risk Level: Low
Testing:
bazel --nohome_rc test //test/...
bazel --nohome_rc test @envoy_api_canonical//test/... @envoy_api_canonical//tools/...
bazel --nohome_rc build @envoy_api_canonical//envoy/...
tools/dependency/release_dates.py api/bazel/repository_locations.bzl
Signed-off-by: Michael Payne <michael@sooper.org>
Mirrored from https://github.com/envoyproxy/envoy @ d5ec0351f2a5dc28fc98c1d34deb90e48cf30083
Fixes a few issues with the generated Go protobuf code:
a) the OT protos are not copied over, which means that their importpath
needs to reference a path where it can pull the protos down from. The
"logs" importpath was nonsensical and couldn't be resolved
b) Bunching several proto packages into one target did not work well
with the go package system. This splits the common protos out into its
own target so that it can reference a different importpath than the
other logs protos
Signed-off-by: Snow Pettersen <snowp@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8037370ce45d8c10654114b2b46e27377525c156
this includes latest xds version which will reduce the warning noise when building protos
Signed-off-by: Ryan Northey <ryan@synca.io>
Mirrored from https://github.com/envoyproxy/envoy @ 25d50667f96ee98d7f6cad8e190a3798ea5cd66f
This PR adds an OTLP logger to Envoy, which allows users to flexibly configure their log structure (according to OpenTelemetry protos).
The API users the common gRPC configuration, used in ALS, for gRPC settings. The flexible OT fields (body, attributes) are completely configurable and are filled with request/response information based on Envoy access log formatting.
There are still a couple of TODOs -
The 'filter_state_objects_to_log' from the common gRPC config isn't added to the log
Resource attributes are not configurable, and only Envoy node info is added.
Note: this extension was added as v3alpha API.
Risk Level: Low - a new extension, so unlikely to break any existing setups (marked as v3alpha API).
Testing: Unit tests.
Docs Changes:
Release Notes: access log: added a new :ref:OpenTelemetry access logger <envoy_v3_api_msg_extensions.access_loggers.open_telemetry.v3alpha.OpenTelemetryAccessLogConfig> extension, allowing a flexible log structure with native Envoy access log formatting.
Signed-off-by: Itamar Kaminski <itamark@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4f52f9ef554e1522e763c0eaf6ef6833aab283e5
This reverts commit 498d65b2fabceebbda9c0b993ba63c127fd16f27.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 3bddd1a8f8f67174d487a06c690f67b944b9e0da
Moved the db generation into a bazel job - which will allow other tasks that also require this data to make use of it more easily/bazely
this also adds buildozer into the bazel env, so we might want to make use of that elsewhere and potentially remove buildozer from the build-image env
Touch #13229
Signed-off-by: Ryan Northey <ryan@synca.io>
Mirrored from https://github.com/envoyproxy/envoy @ a03209d5caf77b36ca4c40671ade8e328cb93fad
Fix#13799. This means integration with external Apache SkyWalking SDK for C++ and destroyed redundant parts of implementations. Hired library is https://github.com/skyAPM/cpp2sky. But it is not released and some of missing parts of implementations which are necessary to achieve clean integration. Currently working private fork of that SDK here. https://github.com/Shikugawa/cpp2sky-envoy.
Additional Description:
cpp2sky dependency policy check list
### MUST
- [x] Cloud Native Computing Foundation (CNCF) approved license
- [x] Dependencies must not substantially increase the binary size unless they are optional (i.e. confined to specific extensions)
- [x] No duplication of existing dependencies
Maybe, the dependencies are described [here](https://github.com/SkyAPM/cpp2sky/blob/main/bazel/repositories.bzl)
- [x] Hosted on a git repository and the archive fetch must directly reference this repository. We will NOT support intermediate artifacts built by-hand located on GCS, S3, etc.
- [x] CVE history appears reasonable, no pathological CVE arcs
- [x] Code review (ideally PRs) before merge
- [x] Security vulnerability process exists, with contact details and reporting/disclosure process
- [x] Tests run in CI
### SHOULD
- [ ] > 1 contributor responsible for a non-trivial number of commits
- [ ] High test coverage (also static/dynamic analysis, fuzzing)
Parts which is used in Envoy have higher test coverage. But gRPC async segment reporter(which is not used in Envoy) has less test.
- [ ] Envoy can obtain advanced notification of vulnerabilities or of security releases
- [ ] Do other significant projects have shared fate by using this dependency?
- [x] Releases (with release notes)
- [x] Commits/releases in last 90 days
Risk Level: Mid
Testing: Unit/Integration
Docs Changes:
Release Notes:
Platform Specific Features:
Signed-off-by: Shikugawa <rei@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 032a8e75928304e011833babea2ca231f32866dd
Specifically interested in
872b28c457
which reduces the startup cost on golang
Signed-off-by: John Howard <howardjohn@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4cb14ea2da6edb9acd4ad96da387d16d859ca4aa
There is an unfortunate change in 3.14 that changed go_package for WKT, which necessitates updating several go dependencies as well:
protoc-gen-validate to 2020-11-30
transitive dependencies from protoc-gen-validate
gazelle to 0.22.2
rules_go to 0.25.0
go to 1.15.5
Risk Level: low
Signed-off-by: Kuat Yessenov <kuat@google.com>
Co-authored-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ b3bb0f9847497bc97951172d248fad3c88096829
As per the decision to move the cncf/udpa repository to cncf/xds branding.
Also updated cncf/udpa hash and updated identifier handling (moved from repeated to a flat string).
Risk level: Low (the only breaking API changes affect not-implemented-hide fields).
Testing: New unit tests for path components.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8c4a3c77a7de016a118aacc4cea933951b85e589
This patch adds a new tracer to support the SkyWalking tracing mechanism and format.
Risk Level: Low, a new extension.
Testing: Unit
Docs Changes: Added
Release Notes: Added
Signed-off-by: wbpcode <comems@msn.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7d0f89b1011503ecd22f28e347cf7f76cba73057
The use of last_updated was ambiguous (is it when an Envoy contributor
creates a PR, merges a commit, or when the dependency is released?).
We really are after the release date as a measure of how stale the
dependency is.
This patch introduces a tool, tools/dependency/release_dates.py, that
uses the GitHub API to compute release date. If a mismatch is detected, an
error is raised.
This patch also introduces a dependency validation CI job that gathers existing
scripts and the release_dates.py script into a single job.
Signed-off-by: Harvey Tuch <htuch@google.com>
Co-authored-by: Michael Payne <michael@sooper.org>
Mirrored from https://github.com/envoyproxy/envoy @ 91f2bb75a34e1068dcc91de1cafca9dad92feecb
A few changes that wrapup #12673.
* Python/Go dependencies that aren't part of the Envoy binary build
don't make sense to track in repository_locations.bzl, since they
have their own language specific metadata (e.g. requirements.txt)
or are in many cases transitively implied.
* Ensure that the full set of dependencies visible to bazel query
is now validated. This requires that we explicitly call out
transitive dependencies that are implied by direct dependencies
in repository_locations.bzl. A new annotation `implied_untracked_deps`
is used.
Fixes#12673
Risk level: Low
Testing: validate.py.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 499f46a24f59eeb4fc3b28e24f6b9191b009580d
* dependencies: CVE scanner for repository_locations.bzl metadata.
This is a custom CVE scanner that consumes the NIST CVE database and
heuristically matches against the CPEs, versions and last update stamps
in repository_locations.bzl.
Future PRs will create a CI job that runs this on a periodic basis
(every few hours) to provide a CVE early warning system.
Example output:
Based on heuristic matching with the NIST CVE database, Envoy may be vulnerable to:
CVE ID: CVE-2019-19391
CVSS v3 score: 9.1
Severity: CRITICAL
Published date: 2019-11-29
Last modified date: 2019-12-19
Dependencies: com_github_luajit_luajit
Description: ** DISPUTED ** In LuaJIT through 2.0.5, as used in Moonjit before
2.1.2 and other products, debug.getinfo has a type confusion issue
that leads to arbitrary memory write or read operations, because
certain cases involving valid stack levels and > options are
mishandled. NOTE: The LuaJIT project owner states that the debug
libary is unsafe by definition and that this is not a vulnerability.
When LuaJIT was originally developed, the expectation was that the
entire debug library had no security guarantees and thus it made no
sense to assign CVEs. However, not all users of later LuaJIT
derivatives share this perspective.
Affected CPEs:
- cpe:2.3🅰️moonjit_project:moonjit:*
- cpe:2.3🅰️luajit:luajit:*
Risk level: Low
Testing: cve_scan_test.py unit tests, manual.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7d50518ce44aacc702d252eb05eb603c69461834
- Refactor code responsible for processing repository location specs, i.e. checking for the presence of fields like last_updated and interpolation of version. The same code is now usable by both API repository_locations.bzl and bazel/repository_locations.bzl.
- Cleanup reference to repo locations in repository_locations.bzl, now using a consistent set of macros.
- Add API dependencies to dependency dashboard.
Risk level: Low
Testing: Docs build.
Part of #12673
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 9181790e3ab21b53ec268470202986b9517c3723
This reverts commit ca7dc5344765f3e7fc3c1a47927e8c1e3e6cefd3.
This PGV upgrade is breaking my local build.
Risk Level: Low
Testing: N/a
Signed-off-by: James Buckland <jbuckland@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7feaadb004dc3d577fb1c0f4f383094a0732ee36
Add CheckedExpr as oneof expression in RBAC proto
Additional Description: CheckedExpr is type check CEL expression. It is future proof and offers better evaluation performance (not for C++ yet, but for other languages). There will be a separate PR to plumb checked expr to RBAC filter.
Risk Level: low
Testing: existing tests
Signed-off-by: jiangtaoli2016 <jiangtao@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 3b8c61b5e5eb4dfddf3c4c0cae9e04eafe1ac61a
This patch introduces support for encoding/decoding
udpa::core::v1::ResourceLocator, in addition to the existing support for
udpa::core::v1::ResourceName. Some refactoring and attention to the
URI/URN/URL distinction (as per RFC3986) is introduced.
Part of #11264
Risk level: Low (not used)
Testing: Additional unit tests for URLs added.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 76525d8d9e3fb8b0b979f8ada5e152a3ddf14019
These map between the structured udpa::core::v1::ResourceName message and flat udpa:// URI
representations of resource names.
Risk level: Low
Testing: Unit tests added.
Part of #11264.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ a707b460b12920e79161e0a4ed5033164b683b3e
This PR replaces #11058, taking a slightly different approach. We
utilize field options to annotate fields that should be set for
untrusted environments with [configure_for_untrusted_downstream,
configure_for_untrusted_downstream]. Defaults are provided out-of-band,
in a manifest files in docs/edge_defaults_manifest.yaml.
Protodoc glues the manifest and options together when generating field
documentation, providing an additional notice for sensitive fields.
This PR depends on #11108 first merging to provide the pip3 build
infrastructure.
Risk level: Low (docs only).
Testing: Inspection of generated docs.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 31128e7dc22355876020188bc8feb99304663041
This patch introduces the tools/docs for a new flow for protoxform: freeze.
This performs a major version freeze and bump by modifying
package_version_status. Tooling now needs to deal with 3 possible versions (v2,
v3, v4alpha). The following contributions are made in the patch:
* Tooling added to support v2 APIs freeze, transitioning v3 to active and generating v4alpha.
* A new tool, merge_active_shadow.py, is introduced to combine hand-edited active v3 API protos with the v2 implied shadows for deprecated fields.
* The type database now spans v2 and v3 (and implied v4alpha files)
* api_proto_plugins can now take extra args on the Bazel CLI; this is used to plumb in the freeze arg to protoxform.
Docs are updated to reflect the v3 flow that developers should now
follow.
Part of #10355
This is the tools/docs component of #10601. See #10601 for the resulting
API protos generated by a freeze/fix cycle.
Risk level: Low (tooling only, no API changes yet)
Testing: new golden protos tests for protoxform, unit tests for merge_active_shadow.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ de222c3acf1fbf124ca44f50495fc4b8e9aba525
Part of https://github.com/envoyproxy/envoy/issues/10355, this patch introduces additional
annotations to the API to support automatic inference (and developer documentation) of where the
active developer editable version of a file is, and which files are frozen or machine generated.
Risk level: Low (API annotations only)
Testing: CI
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4c5f4310bb8a19a38e5377d7a2d2dc6aa4560f47
This enables "non-strict" header validations, that match the ones in place with Envoy's ASSERT(valid()) code. The default strict: true checks checked for RFC-compliance, which may break previously valid configs.
Part of #10318
Signed-off-by: Asra Ali <asraa@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 97a76d991766d96df2e84a1f2e33b69cae844471
This is the new style for indicating a file is WiP and subject to
breaking changes. Rather than rely on alpha major versions, which are
coarse grained and introduce migration difficulties for operators, we
use a file-level annotation.
Risk level: Low
Testing: API/docs build, manual inspection of docs.
Fixes#9769.
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Co-authored-by: htuch <htuch@users.noreply.github.com>
Mirrored from https://github.com/envoyproxy/envoy @ 423fe76d5572bb4f1505391ccaaacf39b2bf2c85
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Mirrored from https://github.com/envoyproxy/envoy @ a60f6853a2c2ebbbfed79dfff0b5b644fd735980
Description:
Move packages around for #8120 and #8121
Risk Level: Med around messing up build.
Testing: CI
Docs Changes: in API/STYLE.md
Release Notes: N/A (v3alpha is not in use yet)
Fixes#8120
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Mirrored from https://github.com/envoyproxy/envoy @ 1371f2ef46582a72b5b3971147bd87c534011731
Inspired (but no longer required) by #9500. This is a relatively simple
API migration feature to support, so worth adding for future flexibility
Risk level: Low
Testing: New protoxform golden tests added.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4efbf45c10e2d2afd84fee2466e476cb8fc0a804
This is intended to simplify the internal handling of deprecations during API boosting.
See https://docs.google.com/document/d/1mGO9LtVo7t4Lph7WlmyGCxXye3h6j29z3JZvIBbs_D0/edit
Ultimately the plan is to hide this all as a build artifact in Bazel cache, but due to the technical complexity of the pure Bazel solution (involving changes spanning PGV, protoxform, API build rules), we will use checked-in artifacts for 1.13.0.
Risk level: Low
Testing: Additional API test and protoxform golden test.
Part of #8082
This should be unwound in the future with #9479
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ da5fb5ce6ec7cc6e4693656a5ba2161107f6ac4a
data-plane-api(Azure Pipelines)