Chiebot-Mirror
/
grpc
mirror of https://github.com/grpc/grpc.git
8
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #6830 from dgquintas/simplify_compression_interop

Browse Source 
Simplify compression interop
pull/6841/head
Jan Tattermusch 9 years ago
parent e15377484d 20d802db98
commit 129cd65869
4 changed files with 91 additions and 184 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 127
      doc/interop-test-descriptions.md
  2. 19
      src/proto/grpc/testing/messages.proto
  3. 105
      test/cpp/interop/interop_client.cc
  4. 24
      test/cpp/interop/server_main.cc

127
doc/interop-test-descriptions.md
Unescape View File

@ -93,26 +93,24 @@ Client asserts:
### large_compressed_unary
This test verifies compressed unary calls succeed in sending messages. It
sends one unary request for every combination of compression algorithm and
payload type.
sends one unary request for every payload type, with and without requesting a
compressed response from the server.
In all scenarios, whether compression was actually performed is determined by
the compression bit in the response's message flags. The response's compression
value indicates which algorithm was used if said compression bit is set.
the compression bit in the response's message flags.
Server features:
* [UnaryCall][]
* [Compressable Payload][]
* [Uncompressable Payload][]
* [Random Payload][]
Procedure:
1. Client calls UnaryCall with:
```
{
response_compression: <one of {NONE, GZIP, DEFLATE}>
request_compressed_response: bool
response_type: COMPRESSABLE
response_size: 314159
payload:{
@ -123,11 +121,10 @@ Procedure:
Client asserts:
* call was successful
* response payload type is COMPRESSABLE
* response compression is consistent with the requested one.
* if `response_compression == NONE`, the response MUST NOT have the
* if `request_compressed_response` is false, the response MUST NOT have the
compressed message flag set.
* if `request_compressed_response` is true, the response MUST have the
compressed message flag set.
* if `response_compression != NONE`, the response MUST have the compressed
message flag set.
* response payload body is 314159 bytes in size
* clients are free to assert that the response payload body contents are
zero and comparing the entire response message against a golden response
@ -136,7 +133,7 @@ Procedure:
2. Client calls UnaryCall with:
```
{
response_compression: <one of {NONE, GZIP, DEFLATE}>
request_compressed_response: bool
response_type: UNCOMPRESSABLE
response_size: 314159
payload:{
@ -147,29 +144,11 @@ Procedure:
Client asserts:
* call was successful
* response payload type is UNCOMPRESSABLE
* response compression is consistent with the requested one.
* the response MUST NOT have the compressed message flag set.
* the response MAY have the compressed message flag set. Some
implementations will choose to compress the payload even when the output
size if larger than the input.
* response payload body is 314159 bytes in size
* clients are free to assert that the response payload body contents are
identical to the golden uncompressable data at `test/cpp/interop/rnd.dat`.
3. Client calls UnaryCall with:
```
{
response_compression: <one of {NONE, GZIP, DEFLATE}>
response_type: RANDOM
response_size: 314159
payload:{
body: 271828 bytes of zeros
}
}
```
Client asserts:
* call was successful
* response payload type is either COMPRESSABLE or UNCOMPRESSABLE
* the behavior is consistent with the randomly chosen incoming payload type,
as described in their respective sections.
### client_streaming
@ -245,7 +224,7 @@ Procedure:
size: 31415
}
response_parameters:{
size: 9
size: 59
}
response_parameters:{
size: 2653
@ -272,7 +251,6 @@ Server features:
* [StreamingOutputCall][]
* [Compressable Payload][]
* [Uncompressable Payload][]
* [Random Payload][]
Procedure:
@ -280,13 +258,13 @@ Procedure:
```
{
response_compression: <one of {NONE, GZIP, DEFLATE}>
request_compressed_response: bool
response_type:COMPRESSABLE
response_parameters:{
size: 31415
}
response_parameters:{
size: 9
size: 59
}
response_parameters:{
size: 2653
@ -301,12 +279,11 @@ Procedure:
* call was successful
* exactly four responses
* response payloads are COMPRESSABLE
* response compression is consistent with the requested one.
* if `response_compression == NONE`, the response MUST NOT have the
compressed message flag set.
* if `response_compression != NONE`, the response MUST have the compressed
message flag set.
* response payload bodies are sized (in order): 31415, 9, 2653, 58979
* if `request_compressed_response` is false, the response's messages MUST
NOT have the compressed message flag set.
* if `request_compressed_response` is true, the response's messages MUST
have the compressed message flag set.
* response payload bodies are sized (in order): 31415, 59, 2653, 58979
* clients are free to assert that the response payload body contents are
zero and comparing the entire response messages against golden responses
@ -315,13 +292,13 @@ Procedure:
```
{
response_compression: <one of {NONE, GZIP, DEFLATE}>
request_compressed_response: bool
response_type:UNCOMPRESSABLE
response_parameters:{
size: 31415
}
response_parameters:{
size: 9
size: 59
}
response_parameters:{
size: 2653
@ -336,40 +313,14 @@ Procedure:
* call was successful
* exactly four responses
* response payloads are UNCOMPRESSABLE
* response compressions are consistent with the requested one.
* the responses MUST NOT have the compressed message flag set.
* response payload bodies are sized (in order): 31415, 9, 2653, 58979
* the response MAY have the compressed message flag set. Some
implementations will choose to compress the payload even when the output
size if larger than the input.
* response payload bodies are sized (in order): 31415, 59, 2653, 58979
* clients are free to assert that the body of the responses are identical to
the golden uncompressable data at `test/cpp/interop/rnd.dat`.
3. Client calls StreamingOutputCall with:
```
{
response_compression: <one of {NONE, GZIP, DEFLATE}>
response_type:RANDOM
response_parameters:{
size: 31415
}
response_parameters:{
size: 9
}
response_parameters:{
size: 2653
}
response_parameters:{
size: 58979
}
}
```
Client asserts:
* call was successful
* response payload type is either COMPRESSABLE or UNCOMPRESSABLE
* the behavior is consistent with the randomly chosen incoming payload type,
as described in their respective sections.
### ping_pong
This test verifies that full duplex bidi is supported.
@ -399,7 +350,7 @@ Procedure:
{
response_type: COMPRESSABLE
response_parameters:{
size: 9
size: 59
}
payload:{
body: 8 bytes of zeros
@ -932,9 +883,9 @@ Server implements EmptyCall which immediately returns the empty message.
[UnaryCall]: #unarycall
Server implements UnaryCall which immediately returns a SimpleResponse with a
payload body of size SimpleRequest.response_size bytes and type as appropriate
for the SimpleRequest.response_type. If the server does not support the
response_type, then it should fail the RPC with INVALID_ARGUMENT.
payload body of size `SimpleRequest.response_size` bytes and type as appropriate
for the `SimpleRequest.response_type`. If the server does not support the
`response_type`, then it should fail the RPC with `INVALID_ARGUMENT`.
### StreamingInputCall
[StreamingInputCall]: #streaminginputcall
@ -974,15 +925,7 @@ COMPRESSABLE.
When the client requests UNCOMPRESSABLE payload, the response includes a payload
of the size requested containing uncompressable data and the payload type is
UNCOMPRESSABLE. A 512 kB dump from /dev/urandom is the current golden data,
stored at `test/cpp/interop/rnd.dat`
### Random Payload
[Random Payload]: #random-payload
When the client requests RANDOM payload, the response includes either a randomly
chosen COMPRESSABLE or UNCOMPRESSABLE payload. The data and the payload type
will be consistent with this choice.
UNCOMPRESSABLE.
### Echo Status
[Echo Status]: #echo-status
@ -1004,8 +947,8 @@ key and the corresponding value back to the client as trailing metadata.
[Observe ResponseParameters.interval_us]: #observe-responseparametersinterval_us
In StreamingOutputCall and FullDuplexCall, server delays sending a
StreamingOutputCallResponse by the ResponseParameters's interval_us for that
particular response, relative to the last response sent. That is, interval_us
StreamingOutputCallResponse by the ResponseParameters's `interval_us` for that
particular response, relative to the last response sent. That is, `interval_us`
acts like a sleep *before* sending the response and accumulates from one
response to the next.
@ -1027,13 +970,13 @@ an email address.
#### Echo OAuth scope
[Echo OAuth Scope]: #echo-oauth-scope
If a SimpleRequest has fill_oauth_scope=true and that request was successfully
If a SimpleRequest has `fill_oauth_scope=true` and that request was successfully
authenticated via OAuth, then the SimpleResponse should have oauth_scope filled
with the scope of the method being invoked.
Although a general server-side feature, most test servers won't implement this
feature. The TLS server grpc-test.sandbox.googleapis.com:443 supports this feature.
It requires at least the OAuth scope
feature. The TLS server `grpc-test.sandbox.googleapis.com:443` supports this
feature. It requires at least the OAuth scope
`https://www.googleapis.com/auth/xapi.zoo` for authentication to succeed.
Discussion:

19
src/proto/grpc/testing/messages.proto
Unescape View File

@ -41,17 +41,6 @@ enum PayloadType {
// Uncompressable binary format.
UNCOMPRESSABLE = 1;
// Randomly chosen from all other formats defined in this enum.
RANDOM = 2;
}
// Compression algorithms
enum CompressionType {
// No compression
NONE = 0;
GZIP = 1;
DEFLATE = 2;
}
// A block of data, to simply increase gRPC message size.
@ -88,8 +77,8 @@ message SimpleRequest {
// Whether SimpleResponse should include OAuth scope.
bool fill_oauth_scope = 5;
// Compression algorithm to be used by the server for the response (stream)
CompressionType response_compression = 6;
// Whether to request the server to compress the response.
bool request_compressed_response = 6;
// Whether server should return a given status
EchoStatus response_status = 7;
@ -145,8 +134,8 @@ message StreamingOutputCallRequest {
// Optional input payload sent along with the request.
Payload payload = 3;
// Compression algorithm to be used by the server for the response (stream)
CompressionType response_compression = 6;
// Whether to request the server to compress the response.
bool request_compressed_response = 6;
// Whether server should return a given status
EchoStatus response_status = 7;

105
test/cpp/interop/interop_client.cc
Unescape View File

@ -55,8 +55,6 @@
namespace grpc {
namespace testing {
static const char* kRandomFile = "test/cpp/interop/rnd.dat";
namespace {
// The same value is defined by the Java client.
const std::vector<int> request_stream_sizes = {27182, 8, 1828, 45904};
@ -67,30 +65,26 @@ const int kReceiveDelayMilliSeconds = 20;
const int kLargeRequestSize = 271828;
const int kLargeResponseSize = 314159;
CompressionType GetInteropCompressionTypeFromCompressionAlgorithm(
grpc_compression_algorithm algorithm) {
switch (algorithm) {
case GRPC_COMPRESS_NONE:
return CompressionType::NONE;
case GRPC_COMPRESS_GZIP:
return CompressionType::GZIP;
case GRPC_COMPRESS_DEFLATE:
return CompressionType::DEFLATE;
default:
GPR_ASSERT(false);
}
}
void NoopChecks(const InteropClientContextInspector& inspector,
const SimpleRequest* request, const SimpleResponse* response) {}
void CompressionChecks(const InteropClientContextInspector& inspector,
const SimpleRequest* request,
const SimpleResponse* response) {
GPR_ASSERT(request->response_compression() ==
GetInteropCompressionTypeFromCompressionAlgorithm(
inspector.GetCallCompressionAlgorithm()));
if (request->response_compression() == NONE) {
const grpc_compression_algorithm received_compression =
inspector.GetCallCompressionAlgorithm();
if (request->request_compressed_response() &&
received_compression == GRPC_COMPRESS_NONE) {
if (request->request_compressed_response() &&
received_compression == GRPC_COMPRESS_NONE) {
// Requested some compression, got NONE. This is an error.
gpr_log(GPR_ERROR,
"Failure: Requested compression but got uncompressed response "
"from server.");
abort();
}
}
if (!request->request_compressed_response()) {
GPR_ASSERT(!(inspector.GetMessageFlags() & GRPC_WRITE_INTERNAL_COMPRESS));
} else if (request->response_type() == PayloadType::COMPRESSABLE) {
// requested compression and compressable response => results should always
@ -211,20 +205,22 @@ bool InteropClient::PerformLargeUnary(SimpleRequest* request,
custom_checks_fn(inspector, request, response);
// Payload related checks.
if (request->response_type() != PayloadType::RANDOM) {
GPR_ASSERT(response->payload().type() == request->response_type());
}
GPR_ASSERT(response->payload().type() == request->response_type());
switch (response->payload().type()) {
case PayloadType::COMPRESSABLE:
GPR_ASSERT(response->payload().body() ==
grpc::string(kLargeResponseSize, '\0'));
break;
case PayloadType::UNCOMPRESSABLE: {
std::ifstream rnd_file(kRandomFile);
GPR_ASSERT(rnd_file.good());
for (int i = 0; i < kLargeResponseSize; i++) {
GPR_ASSERT(response->payload().body()[i] == (char)rnd_file.get());
}
// We don't really check anything: We can't assert that the payload is
// uncompressed because it's the server's prerogative to decide on that,
// and different implementations decide differently (ie, Java always
// compresses when requested to do so, whereas C core throws away the
// compressed payload if the output is larger than the input).
// In addition, we don't compare the actual random bytes received because
// asserting that data is sent/received properly isn't the purpose of this
// test. Moreover, different implementations are also free to use
// different sets of random bytes.
} break;
default:
GPR_ASSERT(false);
@ -341,13 +337,13 @@ bool InteropClient::DoLargeUnary() {
}
bool InteropClient::DoLargeCompressedUnary() {
const CompressionType compression_types[] = {NONE, GZIP, DEFLATE};
const PayloadType payload_types[] = {COMPRESSABLE, UNCOMPRESSABLE, RANDOM};
const bool request_compression[] = {false, true};
const PayloadType payload_types[] = {COMPRESSABLE, UNCOMPRESSABLE};
for (size_t i = 0; i < GPR_ARRAY_SIZE(payload_types); i++) {
for (size_t j = 0; j < GPR_ARRAY_SIZE(compression_types); j++) {
for (size_t j = 0; j < GPR_ARRAY_SIZE(request_compression); j++) {
char* log_suffix;
gpr_asprintf(&log_suffix, "(compression=%s; payload=%s)",
CompressionType_Name(compression_types[j]).c_str(),
request_compression[j] ? "true" : "false",
PayloadType_Name(payload_types[i]).c_str());
gpr_log(GPR_DEBUG, "Sending a large compressed unary rpc %s.",
@ -355,7 +351,7 @@ bool InteropClient::DoLargeCompressedUnary() {
SimpleRequest request;
SimpleResponse response;
request.set_response_type(payload_types[i]);
request.set_response_compression(compression_types[j]);
request.set_request_compressed_response(request_compression[j]);
if (!PerformLargeUnary(&request, &response, CompressionChecks)) {
gpr_log(GPR_ERROR, "Large compressed unary failed %s", log_suffix);
@ -452,23 +448,23 @@ bool InteropClient::DoResponseStreaming() {
}
bool InteropClient::DoResponseCompressedStreaming() {
const CompressionType compression_types[] = {NONE, GZIP, DEFLATE};
const PayloadType payload_types[] = {COMPRESSABLE, UNCOMPRESSABLE, RANDOM};
const bool request_compression[] = {false, true};
const PayloadType payload_types[] = {COMPRESSABLE, UNCOMPRESSABLE};
for (size_t i = 0; i < GPR_ARRAY_SIZE(payload_types); i++) {
for (size_t j = 0; j < GPR_ARRAY_SIZE(compression_types); j++) {
for (size_t j = 0; j < GPR_ARRAY_SIZE(request_compression); j++) {
ClientContext context;
InteropClientContextInspector inspector(context);
StreamingOutputCallRequest request;
char* log_suffix;
gpr_asprintf(&log_suffix, "(compression=%s; payload=%s)",
CompressionType_Name(compression_types[j]).c_str(),
request_compression[j] ? "true" : "false",
PayloadType_Name(payload_types[i]).c_str());
gpr_log(GPR_DEBUG, "Receiving response streaming rpc %s.", log_suffix);
request.set_response_type(payload_types[i]);
request.set_response_compression(compression_types[j]);
request.set_request_compressed_response(request_compression[j]);
for (size_t k = 0; k < response_stream_sizes.size(); ++k) {
ResponseParameters* response_parameter =
@ -483,37 +479,32 @@ bool InteropClient::DoResponseCompressedStreaming() {
size_t k = 0;
while (stream->Read(&response)) {
// Payload related checks.
if (request.response_type() != PayloadType::RANDOM) {
GPR_ASSERT(response.payload().type() == request.response_type());
}
GPR_ASSERT(response.payload().type() == request.response_type());
switch (response.payload().type()) {
case PayloadType::COMPRESSABLE:
GPR_ASSERT(response.payload().body() ==
grpc::string(response_stream_sizes[k], '\0'));
break;
case PayloadType::UNCOMPRESSABLE: {
std::ifstream rnd_file(kRandomFile);
GPR_ASSERT(rnd_file.good());
for (int n = 0; n < response_stream_sizes[k]; n++) {
GPR_ASSERT(response.payload().body()[n] == (char)rnd_file.get());
}
} break;
case PayloadType::UNCOMPRESSABLE:
break;
default:
GPR_ASSERT(false);
}
// Compression related checks.
GPR_ASSERT(request.response_compression() ==
GetInteropCompressionTypeFromCompressionAlgorithm(
inspector.GetCallCompressionAlgorithm()));
if (request.response_compression() == NONE) {
if (request.request_compressed_response()) {
GPR_ASSERT(inspector.GetCallCompressionAlgorithm() >
GRPC_COMPRESS_NONE);
if (request.response_type() == PayloadType::COMPRESSABLE) {
// requested compression and compressable response => results should
// always be compressed.
GPR_ASSERT(inspector.GetMessageFlags() &
GRPC_WRITE_INTERNAL_COMPRESS);
}
} else {
// requested *no* compression.
GPR_ASSERT(
!(inspector.GetMessageFlags() & GRPC_WRITE_INTERNAL_COMPRESS));
} else if (request.response_type() == PayloadType::COMPRESSABLE) {
// requested compression and compressable response => results should
// always be compressed.
GPR_ASSERT(inspector.GetMessageFlags() &
GRPC_WRITE_INTERNAL_COMPRESS);
}
++k;

24
test/cpp/interop/server_main.cc
Unescape View File

@ -110,14 +110,7 @@ void MaybeEchoMetadata(ServerContext* context) {
}
}
bool SetPayload(PayloadType type, int size, Payload* payload) {
PayloadType response_type;
if (type == PayloadType::RANDOM) {
response_type =
rand() & 0x1 ? PayloadType::COMPRESSABLE : PayloadType::UNCOMPRESSABLE;
} else {
response_type = type;
}
bool SetPayload(PayloadType response_type, int size, Payload* payload) {
payload->set_type(response_type);
switch (response_type) {
case PayloadType::COMPRESSABLE: {
@ -141,18 +134,9 @@ bool SetPayload(PayloadType type, int size, Payload* payload) {
template <typename RequestType>
void SetResponseCompression(ServerContext* context,
const RequestType& request) {
switch (request.response_compression()) {
case grpc::testing::NONE:
context->set_compression_algorithm(GRPC_COMPRESS_NONE);
break;
case grpc::testing::GZIP:
context->set_compression_algorithm(GRPC_COMPRESS_GZIP);
break;
case grpc::testing::DEFLATE:
context->set_compression_algorithm(GRPC_COMPRESS_DEFLATE);
break;
default:
abort();
if (request.request_compressed_response()) {
// Any level would do, let's go for HIGH because we are overachievers.
context->set_compression_level(GRPC_COMPRESS_LEVEL_HIGH);
}
}

Write Preview
Loading…
Cancel
Save