Makes some awkward fixes to compression filter, call, connected channel
to hold the semantics we have upheld now in tests.
Once the fixes described here
https://github.com/grpc/grpc/blob/master/src/core/lib/channel/connected_channel.cc#L636
are in this gets a lot less ad-hoc, but that's likely going to be
post-landing promises client & server side.
We specifically need special handling for server side cancellation in
response to reads wrt the inproc transport - which doesn't track
cancellation thoroughly enough itself.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
---------
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
1. `GrpcAuthorizationEngine` creates the logger from the given config in
its ctor.
2. `Evaluate()` invokes audit logging when needed.
---------
Co-authored-by: rockspore <rockspore@users.noreply.github.com>
See `event_engine.h` for the contract change. All other changes are
cleanup.
I confirmed that both the Posix and Windows implementations comply with
this already.
On Windows, the `WindowsEventEngineListener` will only call
`on_shutdown` after all `SinglePortSocketListener`s have been destroyed,
which ensures that no `on_accept` callback will be executed, even if
there is still trailing overlapped activity on the listening socket.
On Posix, the `PosixEngineListenerImpl` will only call `on_shutdown`
after all `AsyncConnectionAcceptor`s have been destroyed, which ensures
`EventHandle::OrphanHandle` has been called. The `OrphanHandle` contract
indicates that all existing notify closures must have already run. The
implementation looks to comply, so if it does not, that's a bug.
3aae08d25e/src/core/lib/event_engine/posix_engine/event_poller.h (L48-L50)
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Whilst the per cpu counters probably help single channel contention, we
think it's likely that they're a pessimization when taken fleetwide.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Spin off from https://github.com/grpc/grpc/pull/32701.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Spin off from https://github.com/grpc/grpc/pull/32701.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Add audit condition and audit logger config into `grpc_core::Rbac`.
Support translation of audit logging options from authz policy to it.
Audit logging options in authz policy looks like:
```json
{
"audit_logging_options": {
"audit_condition": "ON_DENY",
"audit_loggers": [
{
"name": "logger",
"config": {},
"is_optional": false
}
]
}
}
```
which is consistent with what's in the xDS RBAC proto but a little
flattened.
---------
Co-authored-by: rockspore <rockspore@users.noreply.github.com>
Without this change, new users get this warning:
```
CMake Warning (dev) at $MY_INSTALL_DIR/lib64/cmake/protobuf/protobuf-options.cmake:6 (option):
Policy CMP0077 is not set: option() honors normal variables. Run "cmake
--help-policy CMP0077" for policy details. Use the cmake_policy command to
set the policy and suppress this warning.
For compatibility with older versions of CMake, option is clearing the
normal variable 'protobuf_MODULE_COMPATIBLE'.
Call Stack (most recent call first):
$MY_INSTALL_DIR/lib64/cmake/protobuf/protobuf-config.cmake:2 (include)
$MY_SRC_PATH/examples/cpp/cmake/common.cmake:99 (find_package)
CMakeLists.txt:24 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
```
release notes: no
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
- Add a new docker image "rbe_ubuntu2004" that is built in a way that's
analogous to how our other testing docker images are built (this gives
us control over what exactly is contained in the docker image and
ability to fine-tune our RBE configuration)
- Switch RBE on linux to the new image (which gives us ubuntu20.04-based
builds)
For some reason, RBE seems to have trouble pulling the docker image from
Google Artifact Registry (GAR), which is where our public testing images
normally live, so for now, I used a workaround and I upload a copy of
the rbe_ubuntu2004 docker image to GCR as well, and that makes RBE works
just fine (see comment in the `renerate_linux_rbe_configs.sh` script).
More followup items (config cleanup, getting local sanitizer builds
working etc.) are in go/grpc-rbe-tech-debt-2023
Reverts grpc/grpc#32636
```
src/compiler/csharp_generator_helpers.h:25:7: error: no member named 'compiler' in namespace ...
src/compiler/csharp_generator_helpers.h:25:25: error: no member named 'csharp' in namespace 'compiler' ...
```
Added `base_namespace` experimental option to `grpc_csharp_plugin` as
this has been requested several times by
people not using `Grpc.Tools` to generate their code - see
https://github.com/grpc/grpc/issues/28663
Notes:
- it should not be used with `Grpc.Tools`. That has a different way of
handling duplicate proto file names in different directories. Using this
option will break those builds. It can only be used on the `protoc`
command line.
- it uses common code with the `base_namespace` option for C# in
`protoc`, which unfortunately has a slightly different name mangling
algorithm for converting proto file names to C# camel case names. This
only affects files with punctation or numbers in the name. This should
not matter unless you are expecting specific file names
- See
https://protobuf.dev/reference/csharp/csharp-generated/#compiler_options
for an explanation of this option
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
- Fix broken `bin/run_channelz.py` helper
- Create `bin/run_ping_pong.py` helper that runs the baseline (aka
"ping_pong") test against preconfigured infra
- Setup automatic port forwarding when running `bin/run_channelz.py` and
`bin/run_ping_pong.py`
- Create `bin/cleanup_cluster.sh` helper to wipe xds out resources based
namespaces present on the cluster
Note: this involves a small change to the non-helper code, but it's just
moving a the part that makes XdsTestServer/XdsTestClient instance for a
given pod.
Apply Obsolete attribute to deprecated services and methods in C#
generated code
Fix for https://github.com/grpc/grpc/issues/28597
- Deprecated support for enums and enum values is already fixed by
https://github.com/protocolbuffers/protobuf/pull/10520 but this is not
yet released. It is fixed in Protocol Buffers v22.0-rc1 but the gRPC
repo currently has 21.12 as the protocol buffers submodule.
- Deprecated support for messages and fields already exists in the
protocol buffers compiler.
The fix in this PR adds `Obsolete` attribute to classes and methods for
deprecated services and methods within services. e.g.
```
service Greeter {
option deprecated=true; // service level deprecated
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {
option deprecated=true; // method level deprecated
}
}
```
I couldn't find any protocol buffers plugin tests to update. Tested
locally.
Audit logging APIs for both built-in loggers and third-party logger
implementations.
C++ uses using decls referring to C-Core APIs.
---------
Co-authored-by: rockspore <rockspore@users.noreply.github.com>
Fix: https://github.com/grpc/grpc/issues/18075
From comments in https://github.com/grpc/grpc/issues/18075, `CPython`
reinitialize the `GIL` after `pthread_atfork` child handler, thus we
shouldn't use any `GIL` related functions in child handler which is what
we're currently doing, this PR uses `os.register_at_fork` to replace
`pthread_atfork` to prevent any undesired bevahior.
This also seems to fixes a thread hanging issue cased by changes in
core: https://github.com/grpc/grpc/pull/32869
### Testing:
* Passed existing fork tests. (Note that due to some issues in `Bazel`,
this change was not verified by `Bazel runs_per_test`).
* Tested by patch the core PR, was able to fix Python fork tests:
https://github.com/grpc/grpc/pull/32933
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
---------
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
Third-party loggers will be added in subsequent PRs once the logger
factory APIs are available to validate the configs here.
This registry is used in `xds_http_rbac_filter.cc` to generate service
config json.
These checks have not been needed since way back in #22100, but they
were never removed, and they've even propagated to a bunch of new
policies via copy-paste.
Fix at-head tests (this is a missing piece of
https://github.com/grpc/grpc/pull/32905) with the following error;
```
/var/local/git/grpc/tools/run_tests/helper_scripts/build_python.sh: line 126: python3.8: command not found
```
While a proper fix is on the way, this mitigates the number of
duplicated container logs in the xds test server/client pod logs.
The issue is that we only wait between stream restarts when an exception
is caught, which isn't always the reason the stream gets broken. Another
reason is the main container being shut down by k8s. In this situation,
we essentially do
```py
while True:
try:
restart_stream()
read_all_logs_from_pod_start()
except Exception:
logger.warning('error')
wait_seconds(1)
```
This PR makes it
```py
while True:
try:
restart_stream()
read_all_logs_from_pod_start()
except Exception:
logger.warning('error')
finally:
wait_seconds(5)
```
Valgrind will now only fail the build on definite leaks, not "possible"
leaks. A trivial example that fails the PHP valgrind test as it is
configured today:
```
namespace {
grpc_core::NoDestruct<grpc_core::BackOff> g_backoff{
grpc_core::BackOff::Options()};
} // namespace
```
Valgrind detects a possible leak because BackOff contains an
absl::BitGen, which calls `new` through a chain of ownership
indirection. This is what Valgrind calls an [interior
pointer](https://valgrind.org/docs/manual/mc-manual.html#mc-manual.options:~:text=%22Possibly%20lost%22.%20This,have%20interior%2Dpointers.).
Our CI will no longer fail them
There are some places where the G name was not updated properly in the
previous release
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Oops I missed important changes from
https://github.com/grpc/grpc/pull/32712. And it turned out that there
are two problems that I couldn't fix at this point.
- Windows Bazel RBE Linker Error: This may be caused by how new Bazel 6
invokes build tools chain but it's not clear. I put workaround to use
Bazel 5 by using `OVERRIDE_BAZEL_VERSION=5.4.1`
- Rule `rules_pods` to fetch CronetFramework from CocoaPod has
incompatibility with sort of built-in apple toolchain.
(https://github.com/bazel-xcode/PodToBUILD/issues/232): I couldn't find
a workaround to fix this so I ended up disabling all tests depending
this target.
Fix `python_alpine` test failure with
```
fatal: detected dubious ownership in repository at '/var/local/jenkins/grpc'
To add an exception for this directory, call:
git config --global --add safe.directory /var/local/jenkins/grpc
```
Fix https://github.com/grpc/grpc/issues/32638
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Craig Tiller
Luwei Ge
AJ Heller
Vignesh Babu
Esun Kim
Mark D. Roth
Yijie Ma
Ravi Prakash
Yousuk Seung
Eugene Ostroukhov
Luwei Ge
Jan Tattermusch
Sergii Tkachenko
tony
nanahpang
Michael Jarrett
Yash Tibrewal
Xuan Wang