As part of #8082, we want to be able to (1) automatically generate BUILD
files and (2) treat packages as atomic from a "upgrade / do not upgrade"
decision perspective. This is simplified by having our BUILD targets at
package granularity, since this is what the protoxform plugin operates
on.
This PR broadens the package-level treatment that was already introduced
for Go in #8003 to Python and C++. This simplifies BUILD files
significantly and opens the way to automated generation.
There is some technical debt introduced, since all visibility controls
have been removed. This is slated for reintroduction in
https://github.com/envoyproxy/envoy/issues/8491.
As a bonus (useful for BUILD file generation), also removed the
inconsistency in BUILD package target naming for packages in envoy.api.*
and envoy.type.*. E.g. //envoy/api/v2:v2 is now //envoy/api/v2:pkg.
Risk level: Low (but this will break internal builds and require BUILD
fixups to consuming projects).
Testing: bazel test //test/... @envoy_api//...
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 4e858f17fe08224c9c089240908ccd0c518e01a7
In addition to updating protobuf to 3.8.0, this PR also
Removes old protobuf patch now included in 3.8.0
- Patches protocolbuffers/protobuf#6333 that fixes a UBSAN error in the protobuf library.
- Patches protobuf's BUILD to depend on foreign_cc zlib
Risk level: low/medium
Testing: bazel test //test/...
Signed-off-by: Asra Ali <asraa@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 8246167b9d238797cbc6c03dccc9e3921c37617d
Users can now choose between buffered tapping (simpler) and
streaming tapping (more flexible but harder to work with).
Streaming tapping for the transport socket will be added in a
follow-up.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ e2e4baaa85a98b14f2bee6ea5aa16dd79cb832d4
1) Add request/response body tapping
2) Add buffered body limits (TBI for transport socket)
3) Add the JSON_BODY_AS_BYTES and JSON_BODY_AS_STRING output
formats for convenience when the body is known to be human
readable.
4) Add JSON output for the file per tap sink.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 9a06dc0777d2809195cb1fc414b05ae7c0660193
This commit refactors the tap transport socket to use the common
tap extension configuration and tap matching infrastructure. More
match conditions will be added in a future PR as well as additional
cleanups that have been marked with TODOs.
One result of this PR is that the HTTP tap filter can now have a static
configuration as well as write to a file per tap sink.
All future tap PRs should be smaller and more targeted after this one.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ f37ebdc14f4c0adf0e90aabddae833355c0cec1b
This is a rename PR only. It renames the capture transport socket
and associated tools to the tap transport socket. It also updates
some documentation. In a subsequent PR I'm going to refactor the
tap transport socket to use the new common tap framework so that
the tap transport socket can be configured via admin, the HTTP
tap filter can write to a file, the tap transport socket can have
matching, etc.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ 7a5849f2a8bcc55fa16da3eaee94d9c99a11147c
Legacy print statements are syntax errors in Python 3 but print() function works as expected in both Python 2 and Python 3.
Resolves syntax errors and name errors that have the potential to halt the runtime.
Risk Level: Minimal
Testing: http://flake8.pycqa.org
Signed-off-by: cclauss <cclauss@me.com>
Mirrored from https://github.com/envoyproxy/envoy @ abd82f7c078b5c38faaecb885b46e0ac053e0353
Fixes https://github.com/envoyproxy/envoy/issues/743
This is a general cleanup of all of the access logging documentation.
I have reorganized a bunch of things and hidden the various gRPC logging
fields that are not implemented yet.
I've also moved the existing tap protos into a new "output" directory. This
is the best name I could come up for cleanly separating output data that might
be stored outside of any service or configuration.
Signed-off-by: Matt Klein <mklein@lyft.com>
Mirrored from https://github.com/envoyproxy/envoy @ c15019e79c832d9f0a09468affaadabc4be3e115
* tap/fuzz: transport socket extension for traffic capture.
This PR introduces a transport socket extension that wraps a given transport socket, interposes on its
plain text traffic and records it into a proto trace file on the filesystem. This can be used for a
number of purposes:
1. As a corpus for fuzzing the data plane.
2. Converted to PCAP using a soon-to-be-written utility, allowing existing tools such as Wireshark
to be used to decode L4/L7 protocol history in the trace. Essentially this lets us take advantage
of the PCAP ecosystem.
Relates to #1413 and #508.
Risk Level: Low (opt-in).
Testing: New SSL integration tests, demonstrating plain text intercept.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mirrored from https://github.com/envoyproxy/envoy @ 6c7a91733469f76381487f9ca78bdece6825c8c9
There are several main changes in this PR:
Create envoy.api.v2.core packages to break circular dependencies from xDS on to subpackages on to base protos.
Create individual packages for each filter and add independent versioning to each filter.
Add visibility constraints to prevent formation of dependency cycles.
Add gogoproto annotations to improve go code generation.
After moving xDS service definitions and top-level resource protos back to envoy.core.api.v2, cycles were created, since the second-level definitions depend on base protobuf definitions, and are in turn included from xDS; however xDS and base definitions are in the same package.
The solution is to split the base protos into another package, envoy.api.v2.core. That eliminates dependency cycles (validated using go-control-plane).
Added a few gogoproto annotations to improve golang code generation.
Signed-off-by: Kuat Yessenov <kuat@google.com>
* protodoc: make protodoc fast again.
Some low hanging fruit optimizations. This takes address.proto.rst build
from ~21s to ~1.7s.
Signed-off-by: Harvey Tuch <htuch@google.com>
These were lost in my backlog. Required a PGV fix for cross-package enum
validation to deal with the TODOs, see
https://github.com/lyft/protoc-gen-validate/issues/42.
Signed-off-by: Harvey Tuch <htuch@google.com>
* Added PGV C++ generation support. This (hopefully temporarily)
abandons using native proto_library in favor of pgv_cc_proto_library.
We maintain build support for proto_library for the glorious future in
which we write a Bazel aspect to run PGV against the native
proto_library shadow graph.
* Replace min_len with min_bytes on strings, until PGV gets not-empty or
min_len support for C++.
* Various fixups for places where the PGV plugin objected to
annotations.
Signed-off-by: Harvey Tuch <htuch@google.com>
This patch adds an overview page introduced the v2 API concepts via a
worked example. Brought in the entire transitive dep set of protos from
bootstrap.proto, none of these have been cleaned up beyond the minimum
required to have them build under Sphinx.
Also added the ability to link to the underlying proto in messages/enums
from protodoc.py generated RST.
Signed-off-by: Harvey Tuch <htuch@google.com>
I will defer RDS into another PR. I will also do another cleanup
pass over all of this later but want to get most of this in place
first.
Signed-off-by: Matt Klein <mklein@lyft.com>
This should provide an example of how to do the .proto doc linking,
refactoring and constraint addition for the full API.
Signed-off-by: Harvey Tuch <htuch@google.com>
* [#not-implemented-hide:] will hide a message/enum/field from docs.
* [#not-implemented-warn:] will add a "Not implemented yet" warning to a
message/enum/field.
* [#v2-api-diff:<text>] will add a note indicating v2 API difference to
a message/enum/field.
* Switched title annotation to [#protodoc-title:<text>] for consistency.
Signed-off-by: Harvey Tuch <htuch@google.com>
* Import @com_lyft_protoc_gen_validate for validate.proto annotations.
* Example annotation in address.proto (BindConfig).
* Process optional/required annotations in protodoc.
Signed-off-by: Harvey Tuch <htuch@google.com>
Instead of having TOC elements like api/base.proto, we could add a title to protos with prefix
protodoc-title: <title>. These will be converted into page titles and stripped from the docs before rendering.
Signed-off-by: Shriram Rajagopalan <shriram@us.ibm.com>
This takes us to the point where address.proto format in a style fairly
similar to the existing docs. There's some missing bits, e.g. oneof/enum
support, nested messages, optional/required, these will come as later
PRs.
Signed-off-by: Harvey Tuch <htuch@google.com>
Mostly Bazel hacking to get a protoc plugin running against the
proto_library graph. The Python plugin doesn't actually do any RST
generation yet, it just runs against each file and dumps the input
proto.
The tool can be run with:
bazel build //api --aspects \
tools/protodoc/protodoc.bzl%proto_doc_aspect --output_groups=rst
There's a snafu with unsandboxed runs in CI, where I can only get it to
work on direct leaf invocations, will fix this in a followup PR.
Signed-off-by: Harvey Tuch <htuch@google.com>
This PR follows thru on https://github.com/envoyproxy/envoy/issues/1873
on the data-plane-api side.
Also, update the Python validation script and added a test to ensure
this is captured in CI to avoid future bit rot.
Signed-off-by: Harvey Tuch <htuch@google.com>
This tool will take a LDS response message with holes where opaque
filter configs should go, and fill them with the Struct equivalent of
the protos supplied for the filter configs. It emits both output
text proto and JSON.
To generate example output listeners.pb and listeners.json, run bazel
build //examples/service_envoy:listeners_files.
data-plane-api(CircleCI)
Rama Chavali
Daniel Hochman
Kuat
htuch
alyssawilk
Matt Klein
tschroed
Shriram Rajagopalan
Matt Klein