From 93f49753accc92d1b64b4c45df318a1bcf990613 Mon Sep 17 00:00:00 2001 From: kpayson64 Date: Thu, 10 May 2018 13:17:30 -0700 Subject: [PATCH] More PR feedback --- doc/fork_support.md | 49 +++++++++++++++++++++------------------------ 1 file changed, 23 insertions(+), 26 deletions(-) diff --git a/doc/fork_support.md b/doc/fork_support.md index 45466434acd..ba8483815d8 100644 --- a/doc/fork_support.md +++ b/doc/fork_support.md @@ -1,22 +1,24 @@ # Background # -In Python multithreading is ineffective at concurrency for CPU bound tasks -due to the GIL. -(global interpreter lock). Users use multiprocessing, subprocess, -and concurrent.futures.ProcessPoolExecutor, to work around the GIL. -These modules call fork() underneath the hood. Various issues have -been reported when using these modules with gRPC Python. -Historically, we didn't support forking in gRPC, but some users seem +In Python, multithreading is ineffective at concurrency for CPU bound tasks +due to the GIL (global interpreter lock). Extension modules can release +the GIL in CPU bound tasks, but that isn't an option in pure Python. +Users use libraries such as multiprocessing, subprocess, concurrent.futures.ProcessPoolExecutor, +etc, to work around the GIL. These modules call fork() underneath the hood. Various issues have +been reported when using these modules with gRPC Python. gRPC Python wraps +gRPC core, which uses multithreading for performance, and hence doesn't support fork(). +Historically, we didn't support forking in gRPC, but some users seemed to be doing fine until their code started to break on version 1.6. This was likely caused by the addition of background c-threads and a background Python thread. # Current Status # -## 1.7 ## -A pthread_atfork() handler was added in 1.7 to automatically shut down -the background c-threads when fork was called. This does not shut down the -background Python thread, so users could not have any open channels when -forking(). + +## 1.11 ## +The background Python thread was removed entirely. This allows forking +after creating a channel. However, the channel must not have issued any +RPCs prior to the fork. Attempting to fork with an active channel that +has been used can result in deadlocks/corrupted wire data. ## 1.9 ## A regression was noted in cases where users are doing fork/exec. This @@ -28,22 +30,17 @@ off the handler using env flag ```GRPC_ENABLE_FORK_SUPPORT=False```. This should be set whenever a user expects to always call exec immediately following fork. It will disable the fork handlers. -## 1.11 ## -The background Python thread was removed entirely. This allows forking -after creating a channel. However, the channel cannot be used by both the -parent and child process after the fork. Additionaly, the process should -not fork if there are any active RPCs on the channel. +## 1.7 ## +A pthread_atfork() handler was added in 1.7 to automatically shut down +the background c-threads when fork was called. This does not shut down the +background Python thread, so users could not have any open channels when +forking(). # Future Work # + ## 1.13 ## The workaround when using fork/exec by setting -```GRPC_ENABLE_FORK_SUPPORT=False``` should no longer be needed. Now the fork -handlers are automatically not run when multiple threads are calling +```GRPC_ENABLE_FORK_SUPPORT=False``` should no longer be needed. Following +[this PR](https://github.com/grpc/grpc/pull/14647), fork +handlers will not automatically run when multiple threads are calling into gRPC. - - -## 1.1x ## -We would like to support forking and using the channel from both the parent -and child process. Additionally, we would like to support servers that -use a prefork model, where the child processes accept the connections -and handle requests.