Long story here:
CallbackAlternativeCQ operates a thread pool which processes a
completion queue and then directly invokes the completion function in
the same thread. This thread pool is initialized on first Ref() and
unallocated on last Unref().
When running an in-process synchronous server (as we do for tests, using
this
https://github.com/google/tensorstore/blob/master/tensorstore/internal/grpc/grpc_mock.h),
called by an async() interface caller, if the async() callback happens
to drop the last reference to the grpc Channel, then the channel
shutdown will attempt to run in one of the the CallbackAlternativeCQ
threads.
This will cause a deadlock/race condition, as `CallbackAlternativeCQ` is
not designed to shutdown itself. When this deadlock happens,
`pthread_join(pthread_id_)` will return `EDEADLK` and the thread will
keep running. However `EDEADLK` is silently ignored by Join() so
`CallbackAlternativeCQ::Unref` will continue to delete the underlying
grpc_completion_queue, leading to a `SIGSEGV` later in the process.
97ba987132/src/cpp/common/completion_queue_cc.cc (L115)
This adds an assert that pthread_join succeeded, which is useful as it
avoids a later SIGSEBV. Alternatively, the thread implementation could
gpr_log the errorcode before asserting.
Example backtrace of crash:
frame #0
gRPC is a modern, open source, high-performance remote procedure call (RPC)
framework that can run anywhere. gRPC enables client and server applications to
communicate transparently, and simplifies the building of connected systems.
To maximize usability, gRPC supports the standard method for adding dependencies
to a user's chosen language (if there is one). In most languages, the gRPC
runtime comes as a package available in a user's language package manager.
For instructions on how to use the language-specific gRPC runtime for a project,
please refer to these documents
C++: follow the instructions under the src/cpp directory
Precompiled bleeding-edge package builds of gRPC master branch's HEAD are
uploaded daily to packages.grpc.io.
To start developing gRPC
Contributions are welcome!
Please read How to contribute which will guide you through
the entire workflow of how to build the source code, how to run the tests, and
how to contribute changes to the gRPC codebase. The "How to contribute" document
also contains info on how the contribution process works and contains best
practices for creating contributions.
Troubleshooting
Sometimes things go wrong. Please check out the
Troubleshooting guide if you are experiencing issues with
gRPC.
Performance
See the
Performance dashboard
for performance numbers of master branch daily builds.
Laramie Leavitt
