grpc/doc/PROTOCOL-HTTP2.md

gRPC over HTTP2

Introduction

This document serves as a detailed description for an implementation of gRPC carried over HTTP2 draft 17 framing. It assumes familiarity with the HTTP2 specification.

Protocol

Production rules are using ABNF syntax.

Outline

The following is the general sequence of message atoms in a GRPC request & response message stream

• Request → Request-Headers *Length-Prefixed-Message EOS
• Response → (Response-Headers *Length-Prefixed-Message Trailers) / Trailers-Only

Requests

• Request → Request-Headers *Length-Prefixed-Message EOS

Request-Headers are delivered as HTTP2 headers in HEADERS + CONTINUATION frames.

• Request-Headers → Call-Definition *Custom-Metadata
• Call-Definition → Method Scheme Path TE [Authority] [Timeout] Content-Type [Message-Type] [Message-Encoding] [Message-Accept-Encoding] [User-Agent]
• Method → ":method POST"
• Scheme → ":scheme " ("http" / "https")
• Path → ":path" "/" Service-Name "/" {method name}
• Service-Name → {IDL-specific service name}
• Authority → ":authority" {virtual host name of authority}
• TE → "te" "trailers" # Used to detect incompatible proxies
• Timeout → "grpc-timeout" TimeoutValue TimeoutUnit
• TimeoutValue → {positive integer as ASCII string of at most 8 digits}
• TimeoutUnit → Hour / Minute / Second / Millisecond / Microsecond / Nanosecond
• Hour → "H"
• Minute → "M"
• Second → "S"
• Millisecond → "m"
• Microsecond → "u"
• Nanosecond → "n"
• Content-Type → "content-type" "application/grpc" [("+proto" / "+json" / {custom})]
• Content-Coding → "identity" / "gzip" / "deflate" / "snappy" / {custom}
• Message-Encoding → "grpc-encoding" Content-Coding
• Message-Accept-Encoding → "grpc-accept-encoding" Content-Coding *("," Content-Coding)
• User-Agent → "user-agent" {structured user-agent string}
• Message-Type → "grpc-message-type" {type name for message schema}
• Custom-Metadata → Binary-Header / ASCII-Header
• Binary-Header → {Header-Name "-bin" } {base64 encoded value}
• ASCII-Header → Header-Name ASCII-Value
• Header-Name → 1*( %x30-39 / %x61-7A / "_" / "-" / ".") ; 0-9 a-z _ - .
• ASCII-Value → 1*( %x20-%x7E ) ; space and printable ASCII

HTTP2 requires that reserved headers, ones starting with ":" appear before all other headers. Additionally implementations should send Timeout immediately after the reserved headers and they should send the Call-Definition headers before sending Custom-Metadata.

Some gRPC implementations may allow the Path format shown above to be overridden, but this functionality is strongly discouraged. gRPC does not go out of its way to break users that are using this kind of override, but we do not actively support it, and some functionality (e.g., service config support) will not work when the path is not of the form shown above.

If Timeout is omitted a server should assume an infinite timeout. Client implementations are free to send a default minimum timeout based on their deployment requirements.

Custom-Metadata is an arbitrary set of key-value pairs defined by the application layer. Header names starting with "grpc-" but not listed here are reserved for future GRPC use and should not be used by applications as Custom-Metadata.

Note that HTTP2 does not allow arbitrary octet sequences for header values so binary header values must be encoded using Base64 as per https://tools.ietf.org/html/rfc4648#section-4. Implementations MUST accept padded and un-padded values and should emit un-padded values. Applications define binary headers by having their names end with "-bin". Runtime libraries use this suffix to detect binary headers and properly apply base64 encoding & decoding as headers are sent and received.

Custom-Metadata header order is not guaranteed to be preserved except for values with duplicate header names. Duplicate header names may have their values joined with "," as the delimiter and be considered semantically equivalent. Implementations must split Binary-Headers on "," before decoding the Base64-encoded values.

ASCII-Value should not have leading or trailing whitespace. If it contains leading or trailing whitespace, it may be stripped. The ASCII-Value character range defined is more strict than HTTP. Implementations must not error due to receiving an invalid ASCII-Value that's a valid field-value in HTTP, but the precise behavior is not strictly defined: they may throw the value away or accept the value. If accepted, care must be taken to make sure that the application is permitted to echo the value back as metadata. For example, if the metadata is provided to the application as a list in a request, the application should not trigger an error by providing that same list as the metadata in the response.

Servers may limit the size of Request-Headers, with a default of 8 KiB suggested. Implementations are encouraged to compute total header size like HTTP/2's SETTINGS_MAX_HEADER_LIST_SIZE: the sum of all header fields, for each field the sum of the uncompressed field name and value lengths plus 32, with binary values' lengths being post-Base64.

The repeated sequence of Length-Prefixed-Message items is delivered in DATA frames

• Length-Prefixed-Message → Compressed-Flag Message-Length Message
• Compressed-Flag → 0 / 1 # encoded as 1 byte unsigned integer
• Message-Length → {length of Message} # encoded as 4 byte unsigned integer
• Message → *{binary octet}

A Compressed-Flag value of 1 indicates that the binary octet sequence of Message is compressed using the mechanism declared by the Message-Encoding header. A value of 0 indicates that no encoding of Message bytes has occurred. Compression contexts are NOT maintained over message boundaries, implementations must create a new context for each message in the stream. If the Message-Encoding header is omitted then the Compressed-Flag must be 0.

For requests, EOS (end-of-stream) is indicated by the presence of the END_STREAM flag on the last received DATA frame. In scenarios where the Request stream needs to be closed but no data remains to be sent implementations MUST send an empty DATA frame with this flag set.

Responses

• Response → (Response-Headers *Length-Prefixed-Message Trailers) / Trailers-Only
• Response-Headers → HTTP-Status [Message-Encoding] [Message-Accept-Encoding] Content-Type *Custom-Metadata
• Trailers-Only → HTTP-Status Content-Type Trailers
• Trailers → Status [Status-Message] *Custom-Metadata
• HTTP-Status → ":status 200"
• Status → "grpc-status" 1*DIGIT ; 0-9
• Status-Message → "grpc-message" Percent-Encoded
• Percent-Encoded → 1*(Percent-Byte-Unencoded / Percent-Byte-Encoded)
• Percent-Byte-Unencoded → 1*( %x20-%x24 / %x26-%x7E ) ; space and VCHAR, except %
• Percent-Byte-Encoded → "%" 2HEXDIGIT ; 0-9 A-F

Response-Headers & Trailers-Only are each delivered in a single HTTP2 HEADERS frame block. Most responses are expected to have both headers and trailers but Trailers-Only is permitted for calls that produce an immediate error. Status must be sent in Trailers even if the status code is OK.

For responses end-of-stream is indicated by the presence of the END_STREAM flag on the last received HEADERS frame that carries Trailers.

Implementations should expect broken deployments to send non-200 HTTP status codes in responses as well as a variety of non-GRPC content-types and to omit Status & Status-Message. Implementations must synthesize a Status & Status-Message to propagate to the application layer when this occurs.

Clients may limit the size of Response-Headers, Trailers, and Trailers-Only, with a default of 8 KiB each suggested.

The value portion of Status is a decimal-encoded integer as an ASCII string, without any leading zeros.

The value portion of Status-Message is conceptually a Unicode string description of the error, physically encoded as UTF-8 followed by percent-encoding. Percent-encoding is specified in RFC 3986 §2.1, although the form used here has different restricted characters. When decoding invalid values, implementations MUST NOT error or throw away the message. At worst, the implementation can abort decoding the status message altogether such that the user would received the raw percent-encoded form. Alternatively, the implementation can decode valid portions while leaving broken %-encodings as-is or replacing them with a replacement character (e.g., '?' or the Unicode replacement character).

Example

Sample unary-call showing HTTP2 framing sequence

Request

HEADERS (flags = END_HEADERS)
:method = POST
:scheme = http
:path = /google.pubsub.v2.PublisherService/CreateTopic
:authority = pubsub.googleapis.com
grpc-timeout = 1S
content-type = application/grpc+proto
grpc-encoding = gzip
authorization = Bearer y235.wef315yfh138vh31hv93hv8h3v

DATA (flags = END_STREAM)
<Length-Prefixed Message>

Response

HEADERS (flags = END_HEADERS)
:status = 200
grpc-encoding = gzip
content-type = application/grpc+proto

DATA
<Length-Prefixed Message>

HEADERS (flags = END_STREAM, END_HEADERS)
grpc-status = 0 # OK
trace-proto-bin = jher831yy13JHy3hc

User Agents

While the protocol does not require a user-agent to function it is recommended that clients provide a structured user-agent string that provides a basic description of the calling library, version & platform to facilitate issue diagnosis in heterogeneous environments. The following structure is recommended to library developers

User-Agent → "grpc-" Language ?("-" Variant) "/" Version ?( " ("  *(AdditionalProperty ";") ")" )

E.g.

grpc-java/1.2.3
grpc-ruby/1.2.3
grpc-ruby-jruby/1.3.4
grpc-java-android/0.9.1 (gingerbread/1.2.4; nexus5; tmobile)

Idempotency and Retries

Unless explicitly defined to be, gRPC Calls are not assumed to be idempotent. Specifically:

• Calls that cannot be proven to have started will not be retried.
• There is no mechanisim for duplicate suppression as it is not necessary.
• Calls that are marked as idempotent may be sent multiple times.

HTTP2 Transport Mapping

Stream Identification

All GRPC calls need to specify an internal ID. We will use HTTP2 stream-ids as call identifiers in this scheme. NOTE: These id’s are contextual to an open HTTP2 session and will not be unique within a given process that is handling more than one HTTP2 session nor can they be used as GUIDs.

Data Frames

DATA frame boundaries have no relation to Length-Prefixed-Message boundaries and implementations should make no assumptions about their alignment.

Errors

When an application or runtime error occurs during an RPC a Status and Status-Message are delivered in Trailers.

In some cases it is possible that the framing of the message stream has become corrupt and the RPC runtime will choose to use an RST_STREAM frame to indicate this state to its peer. RPC runtime implementations should interpret RST_STREAM as immediate full-closure of the stream and should propagate an error up to the calling application layer.

The following mapping from RST_STREAM error codes to GRPC error codes is applied.

HTTP2 Code	GRPC Code
NO_ERROR(0)	INTERNAL - An explicit GRPC status of OK should have been sent but this might be used to aggressively lameduck in some scenarios.
PROTOCOL_ERROR(1)	INTERNAL
INTERNAL_ERROR(2)	INTERNAL
FLOW_CONTROL_ERROR(3)	INTERNAL
SETTINGS_TIMEOUT(4)	INTERNAL
STREAM_CLOSED	No mapping as there is no open stream to propagate to. Implementations should log.
FRAME_SIZE_ERROR	INTERNAL
REFUSED_STREAM	UNAVAILABLE - Indicates that no processing occurred and the request can be retried, possibly elsewhere.
CANCEL(8)	Mapped to call cancellation when sent by a client.Mapped to CANCELLED when sent by a server. Note that servers should only use this mechanism when they need to cancel a call but the payload byte sequence is incomplete.
COMPRESSION_ERROR	INTERNAL
CONNECT_ERROR	INTERNAL
ENHANCE_YOUR_CALM	RESOURCE_EXHAUSTED ...with additional error detail provided by runtime to indicate that the exhausted resource is bandwidth.
INADEQUATE_SECURITY	PERMISSION_DENIED … with additional detail indicating that permission was denied as protocol is not secure enough for call.

Security

The HTTP2 specification mandates the use of TLS 1.2 or higher when TLS is used with HTTP2. It also places some additional constraints on the allowed ciphers in deployments to avoid known-problems as well as requiring SNI support. It is also expected that HTTP2 will be used in conjunction with proprietary transport security mechanisms about which the specification can make no meaningful recommendations.

Connection Management

GOAWAY Frame

Sent by servers to clients to indicate that they will no longer accept any new streams on the associated connections. This frame includes the id of the last successfully accepted stream by the server. Clients should consider any stream initiated after the last successfully accepted stream as UNAVAILABLE and retry the call elsewhere. Clients are free to continue working with the already accepted streams until they complete or the connection is terminated.

Servers should send GOAWAY before terminating a connection to reliably inform clients which work has been accepted by the server and is being executed.

PING Frame

Both clients and servers can send a PING frame that the peer must respond to by precisely echoing what they received. This is used to assert that the connection is still live as well as providing a means to estimate end-to-end latency. If a server initiated PING does not receive a response within the deadline expected by the runtime all outstanding calls on the server will be closed with a CANCELLED status. An expired client initiated PING will cause all calls to be closed with an UNAVAILABLE status. Note that the frequency of PINGs is highly dependent on the network environment, implementations are free to adjust PING frequency based on network and application requirements.

Connection failure

If a detectable connection failure occurs on the client all calls will be closed with an UNAVAILABLE status. For servers open calls will be closed with a CANCELLED status.

Appendix A - GRPC for Protobuf

The service interfaces declared by protobuf are easily mapped onto GRPC by code generation extensions to protoc. The following defines the mapping to be used.

• Service-Name → ?( {proto package name} "." ) {service name}
• Message-Type → {fully qualified proto message name}
• Content-Type → "application/grpc+proto"