grpc/doc/compression_cookbook.md

gRPC (Core) Compression Cookbook

Introduction

This document describes compression as implemented by the gRPC C core. See the full compression specification for details.

Intended Audience

Wrapped languages developers, for the purposes of supporting compression by interacting with the C core.

Criteria for GA readiness

1. Be able to set compression at channel, call and message level. In principle this API should be based on compression levels as opposed to algorithms. See the discussion below.
2. Have unit tests covering the cases from the spec.
3. Interop tests implemented and passing on Jenkins. The two relevant interop test cases are large_compressed_unary and server_compressed_streaming.

Summary Flowcharts

The following flowcharts depict the evolution of a message, both incoming and outgoing, irrespective of the client/server character of the call. Aspects still not symmetric between clients and servers (e.g. the use of compression levels) are explicitly marked. The in-detail textual description for the different scenarios is described in subsequent sections.

Incoming Messages

Outgoing Messages

Levels vs Algorithms

As mentioned in the relevant discussion on the spec document, compression levels are the primary mechanism for compression selection at the server side. In the future, it'll also be at the client side. The use of levels abstracts away the intricacies of selecting a concrete algorithm supported by a peer, on top of removing the burden of choice from the developer. As of this writing (Q2 2016), clients can only specify compression algorithms. Clients will support levels as soon as an automatic retry/negotiation mechanism is in place.

Per Channel Settings

Compression may be configured at channel creation. This is a convenience to avoid having to repeatedly configure compression for every call. Note that any compression setting on individual calls or messages overrides channel settings.

The following aspects can be configured at channel-creation time via channel arguments:

Disable Compression Algorithms

Use the channel argument key GRPC_COMPRESSION_CHANNEL_ENABLED_ALGORITHMS_BITSET (from grpc/impl/codegen/compression_types.h), takes a 32 bit bitset value. A set bit means the algorithm with that enum value according to grpc_compression_algorithm is enabled. For example, GRPC_COMPRESS_GZIP currently has a numeric value of 2. To enable/disable GZIP for a channel, one would set/clear the 3rd LSB (eg, 0b100 = 0x4). Note that setting/clearing 0th position, that corresponding to GRPC_COMPRESS_NONE, has no effect, as no-compression (a.k.a. identity) is always supported. Incoming messages compressed (ie, encoded) with a disabled algorithm will result in the call being closed with GRPC_STATUS_UNIMPLEMENTED.

Default Compression Level

(currently, Q2 2016, only applicable for server side channels. It's ignored for clients.) Use the channel argument key GRPC_COMPRESSION_CHANNEL_DEFAULT_LEVEL (from grpc/impl/codegen/compression_types.h), valued by an integer corresponding to a value from the grpc_compression_level enum.

Default Compression Algorithm

Use the channel argument key GRPC_COMPRESSION_CHANNEL_DEFAULT_ALGORITHM (from grpc/impl/codegen/compression_types.h), valued by an integer corresponding to a value from the grpc_compression_level enum.

Per Call Settings

Compression Level in Call Responses

The server requests a compression level via initial metadata. The send_initial_metadata grpc_op contains a maybe_compression_level field with two fields, is_set and compression_level. The former must be set when actively choosing a level to disambiguate the default value of zero (no compression) from the proactive selection of no compression.

The core will receive the request for the compression level and automatically choose a compression algorithm based on its knowledge about the peer (communicated by the client via the grpc-accept-encoding header. Note that the absence of this header means no compression is supported by the client/peer).

Compression Algorithm in Call Responses

Server should avoid setting the compression algorithm directly. Prefer setting compression levels unless there's a very compelling reason to choose specific algorithms (benchmarking, testing).

Selection of concrete compression algorithms is performed by adding a (GRPC_COMPRESS_REQUEST_ALGORITHM_KEY, <algorithm-name>) key-value pair to the initial metadata, where GRPC_COMPRESS_REQUEST_ALGORITHM_KEY is defined in grpc/impl/codegen/compression_types.h), and <algorithm-name> is the human readable name of the algorithm as given in the HTTP2 spec for Message-Encoding (e.g. gzip, identity, etc.). See grpc_compression_algorithm_name for the mapping between the grpc_compression_algorithm enum values and their textual representation.

Per Message Settings

To disable compression for a specific message, the flags field of grpc_op instances of type GRPC_OP_SEND_MESSAGE must have its GRPC_WRITE_NO_COMPRESS bit set. Refer to grpc/impl/codegen/compression_types.h),