Building out a new framing layer for chttp2.
The central idea here is to have the framing layer be solely responsible
for serialization of frames, and their deserialization - the framing
layer can reject frames that have invalid syntax - but the enacting of
what that frame means is left to a higher layer.
This class will become foundational for the promise conversion of chttp2
- by eliminating action from the parsing of frames we can reuse this
sensitive code.
Right now the new layer is inactive - there's a test that exercises it
relatively well, and not much more. In the next PRs I'll add an
experiments to enable using this layer or the existing code in the
writing and reading paths.
---------
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
This is the initial implementation of the chaotic-good client transport
write path. There will be a follow-up PR to fulfill the read path.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Previously black wouldn't install, as it required newer `packaging`
package.
This fixes `pip install -r requirements-dev.txt`. In addition, `black`
in dev dependencies file is changed to `black[d]`, which bundles
`blackd` binary (["black as a
server"](https://black.readthedocs.io/en/stable/usage_and_configuration/black_as_a_server.html)).
Fixes an issue when an active context selected automatically picked up
as context for `secondary_k8s_api_manager`.
This was introducing an error in GAMMA Baseline PoC
```
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('100.71.2.143', 56723), raddr=('35.199.174.232', 443)>
```
Here's how the secondary context is incorrectly falls back to the
default context when `--secondary_kube_context` is not set:
```
k8s.py:142] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-security", active host: https://35.202.85.90
k8s.py:142] Using kubernetes context "None", active host: https://35.202.85.90
```
- Add Github Action to conditionally run PSM Interop unit tests:
- Only run when changes are detected in
`tools/run_tests/xds_k8s_test_driver` or any of the proto files used by
the driver
- Only run against PRs and pushes to `master`, `v1.*.*` branches
- Runs using `python3.9` and `python3.10`
- Ready to be added to the list of required GitHub checks
- Add `tools/run_tests/xds_k8s_test_driver/tests/unit/__main__.py` test
loader that recursively discovers all unit tests in
`tools/run_tests/xds_k8s_test_driver/tests/unit`
- Add basic coverage for `XdsTestClient` and `XdsTestServer` to verify
the test loader picks up all folders
Related:
- First unit tests without automated CI added in #34097
The tests are skipped incorrectly because `config.server_lang` is
incorrectly compared with the string value "java", instead of
`skips.Lang.JAVA`.
This has been broken since #26998.
```
xds_url_map_testcase.py:372] ----- Testing TestTimeoutInRouteRule -----
xds_url_map_testcase.py:373] Logs timezone: UTC
skips.py:121] Skipping TestConfig(client_lang='java', server_lang='java', version='v1.57.x')
[ SKIPPED ] setUpClass (timeout_test.TestTimeoutInRouteRule)
xds_url_map_testcase.py:372] ----- Testing TestTimeoutInApplication -----
xds_url_map_testcase.py:373] Logs timezone: UTC
skips.py:121] Skipping TestConfig(client_lang='java', server_lang='java', version='v1.57.x')
[ SKIPPED ] setUpClass (timeout_test.TestTimeoutInApplication)
```
This is to make sure upgrading packaging module won't break our logic on
version-based version skipping.
This also fixes a small issue with `dev-` prefix - it should only be
allowed on the left side of the comparison.
Context: packaging module needs to be upgraded to be compatible with
`blackd`.
Update from gtcooke94:
This PR adds support to build gRPC and it's tests with OpenSSL3. There were some
hiccups with tests as the tests with openssl haven't been built or exercised in a
few months, so they needed some work to fix.
Right now I expect all test files to pass except the following:
- h2_ssl_cert_test
- ssl_transport_security_utils_test
I confirmed locally that these tests fail with OpenSSL 1.1.1 as well,
thus we are at least not introducing regressions. Thus, I've added compiler directives around these tests so they only build when using BoringSSL.
---------
Co-authored-by: Gregory Cooke <gregorycooke@google.com>
Co-authored-by: Esun Kim <veblush@google.com>
- add debug-only `WorkSerializer::IsRunningInWorkSerializer()` method
and use it in client_channel to verify that subchannels are destroyed in
the `WorkSerializer`
- note: this mechanism uses `std:🧵:id`, so I had to exclude
work_serializer.cc from the core_banned_constructs check
- fix `WorkSerializer::Run()` to unref the callback before releasing
ownership of the `WorkSerializer`, so that any refs captured by the
`std::function<>` will be released before releasing ownership
- fix the WRR timer callback to hop into the `WorkSerializer` to release
its ref to the picker, since that transitively releases refs to
subchannels
- fix subchannel connectivity state notifications to unref the watcher
inside the `WorkSerializer`, since the watcher often transitively holds
refs to subchannels
Currently the bazel invocations provide a link back to the original
kokoro jobs for resultstore and sponge UIs.
This adds another back link to Fusion UI, which has the advantage of
- being able to navigate to the kokoro job overview
- there is a button to trigger a new build (in case the job needs to be
re-run).
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Reintroduce https://github.com/grpc/grpc/pull/33959.
I added a fix for the python arm64 build (which is the reason why the
change has been reverted earlier).
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Disables the warning produced by kubernetes/client/rest.py calling the
deprecated `urllib3.response.HTTPResponse.getheaders`,
`urllib3.response.HTTPResponse.getheader` methods:
```
venv-test/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
```
This issue introduced by openapi-generator, and solved in `v6.4.0`. To
fix the issue properly, kubernetes/python folks need to regenerate the
library using newer openapi-generator. The most recent release `v27.2.0`
still used openapi-generator
[`v4.3.0`](https://github.com/kubernetes-client/python/blob/v27.2.0/kubernetes/.openapi-generator/VERSION).
Since they release two times a year, and the 2 major version difference
of openapi-generator, the fix may take a while.
Created an issue in their repo:
https://github.com/kubernetes-client/python/issues/2101.
Not adding CMake support yet
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Addresses some issues of the initial triage hint PR:
https://github.com/grpc/grpc/pull/33898.
1. Print unhealthy backend name before the health info - previously it
was unclear health status of which backend is dumped
2. Add missing `retry_err.add_note(note)` calls
3. Turn off the highlighter in triager hints, which isn't rendered
properly in the stack trace saved to junit.xml
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
I was hoping this would solve the issue with `DeprecationWarning:
HTTPResponse.getheaders() is deprecated`, but it didn't.
Anyway, we should be updating this from time to time.
Changelog:
https://github.com/kubernetes-client/python/blob/release-27.0/CHANGELOG.md
The client library changes from `25.3.0` to `27.2.0` are minimal.
The majority of the changelog is API updates pulled from k8s upstream.
This clearly indicates which errors are "blanket" errors and are not a
root cause on their own.
This also moves the debug info with the last known status of an object
the framework was waiting for, but bailed out due to a timeout.
Previously it was printed as the last error message in the test, and
this PR prints it after the stack trace that caused the test failure.
In addition, I added a similar debug information to the "wait for NEGs
to become healthy". Now it prints the statuses of unhealthy backends
To achieve that, I mimicked upcoming [PEP
678](https://peps.python.org/pep-0678/) Exception notes feature. When
we're upgrade to py11, we'll be able to remove `add_note()` methods, and
get the same functionality for free.
This PR adds in delegating call tracers that work like so -
If this is the first call tracer that is being added onto the call
context, just add it as earlier.
If this is the second call tracer that is being added onto the call
context, create a delegating call tracer that contains a list of call
tracers (including the first call tracer).
Any more call tracers added, just get added to the list of tracers in
the delegating call tracer.
(This is not yet used other than through tests.)
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
~~NB: I haven't tested this at all and am hoping the CI will tell me
where I've (undoubtedly) messed something up.~~ Edit: looks like CI is
now clear!
BoringSSL's gas-compatible assembly files, like its C files, are now
wrapped with preprocessor ifdefs to capture which platforms each file
should be enabled on. This means that, provided the platform can process
.S files it all (i.e. not Windows), we no longer need to detect the
exact CPU architecture in the build.
Switch gRPC's build to take advantage of this. I've retained
BUILD_OVERRIDE_BORING_SSL_ASM_PLATFORM, on the off chance anyone is
using it to cross-compile between Windows and non-Windows, though I
doubt that works particularly well.
As part of this, restore assembly optimizations in a few places where
they were seemingly disabled for issues relating to this:
- https://github.com/grpc/grpc/pull/31747 had to disable the assembly,
because at the time assembly required the library be built differently
for each architecture and then stitched back together. This should now
work.
- tools/run_tests/run_tests.py disabled x86 assembly due to some issues
with CMAKE_SYSTEM_PROCESSOR in a Docker image. This too should now be
moot.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Add "bazelified" non-bazel tests. See tools/bazelify_tests/README.md for
the core idea.
- add a bunch of test targets that run under docker and execute tests
that correspond to `run_tests.py -l LANG ...`
- many more tests can be added in the future
- to enable running some of the C/C++ portability tests easily, added
support for `--cmake_extra_configure_args` in run_tests.py (the change
is fairly small).
Example passing build that shows how test results are structured:
https://source.cloud.google.com/results/invocations/21295351-a3e3-4be1-b6e9-aaf52195a044/targets
This adds a new GKE benchmark job, which runs the set of "dashboard"
scenarios for every gRPC experiment configured in the script. Results
are published to BigQuery at
`e2e_benchmarks.ci_cxx_experiment_results_${N}core.${experiment}`
See https://github.com/grpc/grpc/pull/33907 for the scenario config.
This PR fixes the bootstrap generator interop test by making the node
metadata flag dependent on version, which was causing a breakage
previously as all bootstrap generator version's don't necessarily
support the deexpiermentalized flag.
This PR covers C++ only. Other language owners: feel free to ping me if
this would be useful for you.
This allows scenario_config to produce a small superset of tests
required to generate the performance dashboard
https://grafana-dot-grpc-testing.appspot.com/. This only adds a category
to some existing scenarios, and to my knowledge, should not affect any
current automation that uses scenario_config.
I plan to use this category to run gRPC-core experiments and produce
equivalent dashboards. It seems worth landing this independently of
those job configurations, but I'm flexible.
---------
Co-authored-by: drfloob <drfloob@users.noreply.github.com>
The most important change here is to setting `resource_suffix` and
`server_xds_port` flags to "generate randomly" by default. Previously we
were suggesting static values, and devs ended up with resource conflict
errors.
Why: Cleanup for chttp2_transport ahead of promise conversion - lots of
logic has become interleaved throughout chttp2, so some effort to
isolate logic out is warranted ahead of that conversion.
What: Split configuration and policy tracking for each of ping rate
throttling and abuse detection into their own modules. Add tests for
them.
Incidentally: Split channel args into their own header so that we can
split the policy stuff into separate build targets.
---------
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
This PR:
- Fixes the xds-protos Python package, which was broken when the `udpa`
submodule was removed
- This required re-adding the protoc-gen-validate submodule
- Adds non-Bazel tests for xds-protos and all of its dependent packages
- Versions xds-protos the same way as the rest of the Python packages
- Fixes Python 3.11 support in `run_tests.py`, which is necessary for
the testing mentioned above
CC @sergiitk You won't be able to consume this in the interop tests
until it makes it into a release. I'm thinking I'll want to backport
this to the 1.57.x branch to make that happen faster.
CC @drfloob to inform him about the likely backport.
Michael Lumish
Craig Tiller
nanahpang
Sergii Tkachenko
Eugene Ostroukhov
Esun Kim
jrandolf
Mark D. Roth
Jan Tattermusch
Xuan Wang
Yash Tibrewal
Yijie Ma
alto-ruby
Paulo Castello da Costa
David Benjamin
AJ Heller
Zach Reyes
Richard Belleville