diff --git a/doc/images/load-balancing.png b/doc/images/load-balancing.png new file mode 100644 index 00000000000..7c70465e653 Binary files /dev/null and b/doc/images/load-balancing.png differ diff --git a/doc/images/load-balancing.svg b/doc/images/load-balancing.svg new file mode 100644 index 00000000000..18d836d345b --- /dev/null +++ b/doc/images/load-balancing.svg @@ -0,0 +1,4 @@ + + + + diff --git a/doc/images/load_balancing_design.png b/doc/images/load_balancing_design.png deleted file mode 100644 index 86183966fb8..00000000000 Binary files a/doc/images/load_balancing_design.png and /dev/null differ diff --git a/doc/load-balancing.md b/doc/load-balancing.md index dfaa7a7f33b..f56d2b0c73a 100644 --- a/doc/load-balancing.md +++ b/doc/load-balancing.md @@ -1,13 +1,21 @@ Load Balancing in gRPC -======================= +====================== -# Objective +# Scope -To design a load balancing API between a gRPC client and a Load Balancer to -instruct the client how to send load to multiple backend servers. +This document explains the design for load balancing within gRPC. # Background +## Per-Call Load Balancing + +It is worth noting that load-balancing within gRPC happens on a per-call +basis, not a per-connection basis. In other words, even if all requests +come from a single client, we still want them to be load-balanced across +all servers. + +## Approaches to Load Balancing + Prior to any gRPC specifics, we explore some usual ways to approach load balancing. @@ -44,30 +52,31 @@ list of servers to the client. ### External Load Balancing Service The client load balancing code is kept simple and portable, implementing -well-known algorithms (ie, Round Robin) for server selection. -Complex load balancing algorithms are instead provided by the load balancer. The -client relies on the load balancer to provide _load balancing configuration_ and -_the list of servers_ to which the client should send requests. The balancer -updates the server list as needed to balance the load as well as handle server +well-known algorithms (e.g., Round Robin) for server selection. +Complex load balancing algorithms are instead provided by the load +balancer. The client relies on the load balancer to provide _load +balancing configuration_ and _the list of servers_ to which the client +should send requests. The balancer updates the server list as needed +to balance the load as well as handle server unavailability or health +issues. The load balancer will make any necessary complex decisions and +inform the client. The load balancer may communicate with the backend +servers to collect load and health information. + +# Requirements + +## Simple API and client + +The gRPC client load balancing code must be simple and portable. The +client should only contain simple algorithms (e.g., Round Robin) for +server selection. For complex algorithms, the client should rely on +a load balancer to provide load balancing configuration and the list of +servers to which the client should send requests. The balancer will update +the server list as needed to balance the load as well as handle server unavailability or health issues. The load balancer will make any necessary -complex decisions and inform the client. The load balancer may communicate with -the backend servers to collect load and health information. - - -## Requirements - -#### Simple API and client - -The gRPC client load balancing code must be simple and portable. The client -should only contain simple algorithms (ie Round Robin) for server selection. For -complex algorithms, the client should rely on a load balancer to provide load -balancing configuration and the list of servers to which the client should send -requests. The balancer will update the server list as needed to balance the load -as well as handle server unavailability or health issues. The load balancer will -make any necessary complex decisions and inform the client. The load balancer -may communicate with the backend servers to collect load and health information. +complex decisions and inform the client. The load balancer may communicate +with the backend servers to collect load and health information. -#### Security +## Security The load balancer may be separate from the actual server backends and a compromise of the load balancer should only lead to a compromise of the @@ -75,70 +84,64 @@ loadbalancing functionality. In other words, a compromised load balancer should not be able to cause a client to trust a (potentially malicious) backend server any more than in a comparable situation without loadbalancing. -# Proposed Architecture - -The gRPC load balancing implements the external load balancing server approach: -an external load balancer provides simple clients with an up-to-date list of -servers. - -![image](images/load_balancing_design.png) - -1. On startup, the gRPC client issues a name resolution request for the service. - The name will resolve to one or more IP addresses to gRPC servers, a hint on - whether the IP address(es) point to a load balancer or not, and also return a - client config. -2. The gRPC client connects to a gRPC Server. - 1. If the name resolution has hinted that the endpoint is a load balancer, - the client's gRPC LB policy will attempt to open a stream to the load - balancer service. The server may respond in only one of the following - ways. - 1. `status::UNIMPLEMENTED`. There is no loadbalancing in use. The client - call will fail. - 2. "I am a Load Balancer and here is the server list." (Goto Step 4.) - 3. "Please contact Load Balancer X" (See Step 3.) The client will close - this connection and cancel the stream. - 4. If the server fails to respond, the client will wait for some timeout - and then re-resolve the name (process to Step 1 above). - 2. If the name resolution has not hinted that the endpoint is a load - balancer, the client connects directly to the service it wants to talk to. -3. The gRPC client's gRPC LB policy opens a separate connection to the Load - Balancer. If this fails, it will go back to step 1 and try another address. - 1. During channel initialization to the Load Balancer, the client will - attempt to open a stream to the Load Balancer service. - 2. The Load Balancer will return a server list to the gRPC client. If the - server list is empty, the call will wait until a non-empty one is - received. Optional: The Load Balancer will also open channels to the gRPC - servers if load reporting is needed. -4. The gRPC client will send RPCs to the gRPC servers contained in the server - list from the Load Balancer. -5. Optional: The gRPC servers may periodically report load to the Load Balancer. - -## Client - -When establishing a gRPC _stream_ to the balancer, the client will send an initial -request to the load balancer (via a regular gRPC message). The load balancer -will respond with client config (including, for example, settings for flow -control, RPC deadlines, etc.) or a redirect to another load balancer. If the -balancer did not redirect the client, it will then send a list of servers to the -client. The client will contain simple load balancing logic for choosing the -next server when it needs to send a request. - -## Load Balancer - -The Load Balancer is responsible for providing the client with a list of servers -and client RPC parameters. The balancer chooses when to update the list of -servers and can decide whether to provide a complete list, a subset, or a -specific list of “picked” servers in a particular order. The balancer can -optionally provide an expiration interval after which the server list should no -longer be trusted and should be updated by the balancer. - -The load balancer may open reporting streams to each server contained in the -server list. These streams are primarily used for load reporting. For example, -Weighted Round Robin requires that the servers report utilization to the load -balancer in order to compute the next list of servers. - -## Server - -The gRPC Server is responsible for answering RPC requests and providing -responses to the client. The server will also report load to the load balancer -if a reporting stream was opened for this purpose. +# Architecture + +## Overview + +The primary mechanism for load-balancing in gRPC is external +load-balancing, where an external load balancer provides simple clients +with an up-to-date list of servers. + +The gRPC client does support an API for built-in load balancing policies. +However, there are only a small number of these (one of which is the +`grpclb` policy, which implements external load balancing), and users +are discouraged from trying to extend gRPC by adding more. Instead, new +load balancing policies should be implemented in external load balancers. + +## Workflow + +Load-balancing policies fit into the gRPC client workflow in between +name resolution and the connection to the server. Here's how it all +works: + +![image](images/load-balancing.png) + +1. On startup, the gRPC client issues a [name resolution](naming.md) request + for the server name. The name will resolve to one or more IP addresses, + each of which will indicate whether it is a server address or + a load balancer address, and a [service config](service_config.md) + that indicates which client-side load-balancing policy to use (e.g., + `round_robin` or `grpclb`). +2. The client instantiates the load balancing policy. + - Note: If all addresses returned by the resolver are balancer + addresses, then the client will use the `grpclb` policy, regardless + of what load-balancing policy was requested by the service config. + Otherwise, the client will use the load-balancing policy requested + by the service config. If no load-balancing policy is requested + by the service config, then the client will default to a policy + that picks the first available server address. +3. The load balancing policy creates a subchannel to each server address. + - For all policies *except* `grpclb`, this means one subchannel for each + address returned by the resolver. Note that these policies + ignore any balancer addresses returned by the resolver. + - In the case of the `grpclb` policy, the workflow is as follows: + 1. The policy opens a stream to one of the balancer addresses returned + by the resolver. It asks the balancer for the server addresses to + use for the server name originally requested by the client (i.e., + the same one originally passed to the name resolver). + - Note: The `grpclb` policy currently ignores any non-balancer + addresses returned by the resolver. However, in the future, it + may be changed to use these addresses as a fallback in case no + balancers can be contacted. + 2. The gRPC servers to which the load balancer is directing the client + may report load to the load balancers, if that information is needed + by the load balancer's configuration. + 3. The load balancer returns a server list to the gRPC client's `grpclb` + policy. The `grpclb` policy will then create a subchannel to each of + server in the list. +4. For each RPC sent, the load balancing policy decides which + subchannel (i.e., which server) the RPC should be sent to. + - In the case of the `grpclb` policy, the client will send requests + to the servers in the order in which they were returned by the load + balancer. If the server list is empty, the call will block until a + non-empty one is received. diff --git a/doc/naming.md b/doc/naming.md index d0c892e8d92..676aa9f2980 100644 --- a/doc/naming.md +++ b/doc/naming.md @@ -1,30 +1,65 @@ -#gRPC Naming and Discovery Support +# gRPC Name Resolution ## Overview -gRPC supports DNS as the default name-system. A number of alternative name-systems are used in various deployments. We propose an API that is general enough to support a range of name-systems and the corresponding syntax for names. The gRPC client library in various languages will provide a plugin mechanism so resolvers for different name-systems can be plugged in. +gRPC supports DNS as the default name-system. A number of alternative +name-systems are used in various deployments. We support an API that is +general enough to support a range of name-systems and the corresponding +syntax for names. The gRPC client library in various languages will +provide a plugin mechanism so resolvers for different name-systems can +be plugged in. -## Detailed Proposal +## Detailed Design - A fully qualified, self contained name used for gRPC channel construction uses the syntax: +### Name Syntax + +A fully qualified, self contained name used for gRPC channel construction +uses the syntax: ``` scheme://authority/endpoint_name ``` -Here, scheme indicates the name-system to be used. Example schemes to be supported include: +Here, `scheme` indicates the name-system to be used. Currently, we +support the following schemes: + +- `dns` + +- `ipv4` (IPv4 address) + +- `ipv6` (IPv6 address) + +- `unix` (path to unix domain socket -- unix systems only) -* `dns` +In the future, additional schemes such as `etcd` could be added. -* `etcd` +The `authority` indicates some scheme-specific bootstrap information, e.g., +for DNS, the authority may include the IP[:port] of the DNS server to +use. Often, a DNS name may be used as the authority, since the ability to +resolve DNS names is already built into all gRPC client libraries. -Authority indicates some scheme-specific bootstrap information, e.g., for DNS, the authority may include the IP[:port] of the DNS server to use. Often, a DNS name may used as the authority, since the ability to resolve DNS names is already built into all gRPC client libraries. +Finally, the `endpoint_name` indicates a concrete name to be looked up +in a given name-system identified by the scheme and the authority. The +syntax of the endpoint name is dictated by the scheme in use. -Finally, the endpoint_name indicates a concrete name to be looked up in a given name-system identified by the scheme and the authority. The syntax of endpoint name is dictated by the scheme in use. +### Resolver Plugins -### Plugins +The gRPC client library will use the specified scheme to pick the right +resolver plugin and pass it the fully qualified name string. -The gRPC client library will switch on the scheme to pick the right resolver plugin and pass it the fully qualified name string. +Resolvers should be able to contact the authority and get a resolution +that they return back to the gRPC client library. The returned contents +include: -Resolvers should be able to contact the authority and get a resolution that they return back to the gRPC client library. The returned contents include a list of IP:port, an optional config and optional auth config data to be used for channel authentication. The plugin API allows the resolvers to continuously watch an endpoint_name and return updated resolutions as needed. +- A list of resolved addresses, each of which has three attributes: + - The address itself, including both IP address and port. + - A boolean indicating whether the address is a backend address (i.e., + the address to use to contact the server directly) or a balancer + address (for cases where [external load balancing](load-balancing.md) + is in use). + - The name of the balancer, if the address is a balancer address. + This will be used to perform peer authorization. +- A [service config](service_config.md). +The plugin API allows the resolvers to continuously watch an endpoint +and return updated resolutions as needed. diff --git a/doc/service_config.md b/doc/service_config.md new file mode 100644 index 00000000000..7318b69f214 --- /dev/null +++ b/doc/service_config.md @@ -0,0 +1,147 @@ +Service Config in gRPC +====================== + +# Objective + +The service config is a mechanism that allows service owners to publish +parameters to be automatically used by all clients of their service. + +# Format + +The service config is a JSON string of the following form: + +``` +{ + # Load balancing policy name. + # Supported values are 'round_robin' and 'grpclb'. + # Optional; if unset, the default behavior is pick the first available + # backend. + # Note that if the resolver returns only balancer addresses and no + # backend addresses, gRPC will always use the 'grpclb' policy, + # regardless of what this field is set to. + 'loadBalancingPolicy': string, + + # Per-method configuration. Optional. + 'methodConfig': [ + { + # The names of the methods to which this method config applies. There + # must be at least one name. Each name entry must be unique across the + # entire service config. If the 'method' field is empty, then this + # method config specifies the defaults for all methods for the specified + # service. + # + # For example, let's say that the service config contains the following + # method config entries: + # + # 'methodConfig': [ + # { 'name': [ { 'service': 'MyService' } ] ... }, + # { 'name': [ { 'service': 'MyService', 'method': 'Foo' } ] ... } + # ] + # + # For a request for MyService/Foo, we will use the second entry, because + # it exactly matches the service and method name. + # For a request for MyService/Bar, we will use the first entry, because + # it provides the default for all methods of MyService. + 'name': [ + { + # RPC service name. Required. + # If using gRPC with protobuf as the IDL, then this will be of + # the form "pkg.service_name", where "pkg" is the package name + # defined in the proto file. + 'service': string, + + # RPC method name. Optional (see above). + 'method': string, + } + ], + + # Whether RPCs sent to this method should wait until the connection is + # ready by default. If false, the RPC will abort immediately if there + # is a transient failure connecting to the server. Otherwise, gRPC will + # attempt to connect until the deadline is exceeded. + # + # The value specified via the gRPC client API will override the value + # set here. However, note that setting the value in the client API will + # also affect transient errors encountered during name resolution, + # which cannot be caught by the value here, since the service config + # is obtained by the gRPC client via name resolution. + 'waitForReady': bool, + + # The default timeout in seconds for RPCs sent to this method. This can + # be overridden in code. If no reply is received in the specified amount + # of time, the request is aborted and a deadline-exceeded error status + # is returned to the caller. + # + # The actual deadline used will be the minimum of the value specified + # here and the value set by the application via the gRPC client API. + # If either one is not set, then the other will be used. + # If neither is set, then the request has no deadline. + # + # The format of the value is that of the 'Duration' type defined here: + # https://developers.google.com/protocol-buffers/docs/proto3#json + 'timeout': string, + + # The maximum allowed payload size for an individual request or object + # in a stream (client->server) in bytes. The size which is measured is + # the serialized, uncompressed payload in bytes. This applies both + # to streaming and non-streaming requests. + # + # The actual value used is the minimum of the value specified here and + # the value set by the application via the gRPC client API. + # If either one is not set, then the other will be used. + # If neither is set, then the built-in default is used. + # + # If a client attempts to send an object larger than this value, it + # will not be sent and the client will see an error. + # Note that 0 is a valid value, meaning that the request message must + # be empty. + # + # The format of the value is that of the 'uint64' type defined here: + # https://developers.google.com/protocol-buffers/docs/proto3#json + 'maxRequestMessageBytes': string, + + # The maximum allowed payload size for an individual response or object + # in a stream (server->client) in bytes. The size which is measured is + # the serialized, uncompressed payload in bytes. This applies both + # to streaming and non-streaming requests. + # + # The actual value used is the minimum of the value specified here and + # the value set by the application via the gRPC client API. + # If either one is not set, then the other will be used. + # If neither is set, then the built-in default is used. + # + # If a server attempts to send an object larger than this value, it + # will not be sent, and the client will see an error. + # Note that 0 is a valid value, meaning that the response message must + # be empty. + # + # The format of the value is that of the 'uint64' type defined here: + # https://developers.google.com/protocol-buffers/docs/proto3#json + 'maxResponseMessageBytes': string + } + ] +} +``` + +Note that new per-method parameters may be added in the future as new +functionality is introduced. + +# Architecture + +A service config is associated with a server name. The [name +resolver](naming.md) plugin, when asked to resolve a particular server +name, will return both the resolved addresses and the service config. + +TODO(roth): Design how the service config will be encoded in DNS. + +# APIs + +The service config is used in the following APIs: + +- In the resolver API, used by resolver plugins to return the service + config to the gRPC client. +- In the gRPC client API, where users can query the channel to obtain + the service config associated with the channel (for debugging + purposes). +- In the gRPC client API, where users can set the service config + explicitly. This is intended for use in unit tests. diff --git a/tools/doxygen/Doxyfile.c++ b/tools/doxygen/Doxyfile.c++ index 84daed1041c..af844d661be 100644 --- a/tools/doxygen/Doxyfile.c++ +++ b/tools/doxygen/Doxyfile.c++ @@ -785,6 +785,7 @@ doc/naming.md \ doc/negative-http2-interop-test-descriptions.md \ doc/server-reflection.md \ doc/server_reflection_tutorial.md \ +doc/service_config.md \ doc/statuscodes.md \ doc/stress_test_framework.md \ doc/wait-for-ready.md \ diff --git a/tools/doxygen/Doxyfile.c++.internal b/tools/doxygen/Doxyfile.c++.internal index 37e4ad3f373..2b2d8a3fe1f 100644 --- a/tools/doxygen/Doxyfile.c++.internal +++ b/tools/doxygen/Doxyfile.c++.internal @@ -785,6 +785,7 @@ doc/naming.md \ doc/negative-http2-interop-test-descriptions.md \ doc/server-reflection.md \ doc/server_reflection_tutorial.md \ +doc/service_config.md \ doc/statuscodes.md \ doc/stress_test_framework.md \ doc/wait-for-ready.md \ diff --git a/tools/doxygen/Doxyfile.core b/tools/doxygen/Doxyfile.core index da990379ce4..2f9f021db90 100644 --- a/tools/doxygen/Doxyfile.core +++ b/tools/doxygen/Doxyfile.core @@ -784,6 +784,7 @@ doc/naming.md \ doc/negative-http2-interop-test-descriptions.md \ doc/server-reflection.md \ doc/server_reflection_tutorial.md \ +doc/service_config.md \ doc/statuscodes.md \ doc/stress_test_framework.md \ doc/wait-for-ready.md \ diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index 4591aa50e3c..ffe83a2ddb3 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -784,6 +784,7 @@ doc/naming.md \ doc/negative-http2-interop-test-descriptions.md \ doc/server-reflection.md \ doc/server_reflection_tutorial.md \ +doc/service_config.md \ doc/statuscodes.md \ doc/stress_test_framework.md \ doc/wait-for-ready.md \