From 3165ff3d95d46e3e8f091bc64ad88e7e52dd8a65 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Tue, 8 Mar 2016 00:53:53 -0800 Subject: [PATCH 0001/1039] Added compression spec --- doc/PROTOCOL-HTTP2.md | 4 +- doc/compression.md | 105 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 107 insertions(+), 2 deletions(-) create mode 100644 doc/compression.md diff --git a/doc/PROTOCOL-HTTP2.md b/doc/PROTOCOL-HTTP2.md index 357ea72571b..31d694b803d 100644 --- a/doc/PROTOCOL-HTTP2.md +++ b/doc/PROTOCOL-HTTP2.md @@ -38,7 +38,7 @@ Request-Headers are delivered as HTTP2 headers in HEADERS + CONTINUATION frames. * **Nanosecond** → "n" * **Content-Type** → "content-type" "application/grpc" [("+proto" / "+json" / {_custom_})] * **Content-Coding** → "identity" / "gzip" / "deflate" / "snappy" / {_custom_} -* **Message-Encoding** → "grpc-encoding" Content-Coding +* **Message-Encoding** → "grpc-encoding" Content-Coding * **Message-Accept-Encoding** → "grpc-accept-encoding" Content-Coding \*("," Content-Coding) * **User-Agent** → "user-agent" {_structured user-agent string_} * **Message-Type** → "grpc-message-type" {_type name for message schema_} @@ -83,7 +83,7 @@ binary values' lengths being post-Base64. The repeated sequence of **Length-Prefixed-Message** items is delivered in DATA frames * **Length-Prefixed-Message** → Compressed-Flag Message-Length Message -* **Compressed-Flag** → 0 / 1 # encoded as 1 byte unsigned integer +* **Compressed-Flag** → 0 / 1 # encoded as 1 byte unsigned integer * **Message-Length** → {_length of Message_} # encoded as 4 byte unsigned integer * **Message** → \*{binary octet} diff --git a/doc/compression.md b/doc/compression.md new file mode 100644 index 00000000000..ddbbff1c541 --- /dev/null +++ b/doc/compression.md @@ -0,0 +1,105 @@ +## **gRPC Compression** + +The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", +"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be +interpreted as described in [RFC 2119](http://www.ietf.org/rfc/rfc2119.txt). + +### Intent + +Compression is used to reduce the amount of bandwidth used between peers. The +compression supported by gRPC acts _at the individual message level_, taking +_message_ [as defined in the wire format +document](PROTOCOL-HTTP2.md). + +The implementation supports different compression algorithms. A _default +compression level_, to be used in the absence of message-specific settings, MAY +be specified for during channel creation. + +The ability to control compression settings per call and to enable/disable +compression on a per message basis MAY be used to prevent CRIME/BEAST attacks. +It also allows for asymmetric compression communication, whereby a response MAY +be compressed differently, if at all. + +### Specification + +Compression MAY be configured by the Client Application by calling the +appropriate API method. There are two scenarios where compression MAY be +configured: + ++ At channel creation time, which sets the channel default compression and + therefore the compression that SHALL be used in the absence of per-RPC + compression configuration. ++ At response time, via: + + For unary RPCs, the {Client,Server}Context instance. + + For streaming RPCs, the {Client,Server}Writer instance. In this case, + configuration is reduced to disabling compression altogether. + +### Compression Method Asymmetry Between Peers + +A gRPC peer MAY choose to respond using a different compression method to that +of the request, including not performing any compression, regardless of channel +and RPC settings (for example, if compression would result in small or negative +gains). + +A compressed message from a client with an algorithm unsupported by a server, +WILL result in an INVALID\_ARGUMENT error, alongside the receiving peer's +`grpc-accept-encoding` header specifying the algorithms it accepts. If an +INTERNAL error is returned from the server despite having used one of the +algorithms from the `grpc-accept-encoding h`eader, the cause MUST NOT be related +to compression. Data sent from a server compressed with an algorithm not +supported by the client will also result in an INTERNAL error. + +Note that a peer MAY choose to not disclose all the encodings it supports. +However, if it receives a message compressed in an undisclosed but supported +encoding, it MUST include said encoding in the response's `grpc-accept-encoding +h`eader. + +For every message a server is requested to compress using an algorithm it knows +the client doesn't support (as indicated by the last `grpc-accept-encoding` +header received from the client), it SHALL send the message uncompressed. + +### Specific Disabling of Compression + +If the user (through the previously described mechanisms) requests to disable +compression the next message MUST be sent uncompressed. This is instrumental in +preventing BEAST/CRIME attacks. This applies to both the the unary and streaming +cases. + +### Compression Levels and Algorithms + +We currently (as of July 2015) support _gzip_ and _deflate_ as algorithms (with +the possible future addition of snappy). In order to simplify the public API, +it's intended to abstract the algorithms as _compression levels_ (such as "low", +"medium", "high") that'd map to concrete algorithms and/or their settings (such +as "low" mapping to "gzip -3" and "high" mapping to "gzip -9"). However, we +can't presently (July 2015) implement said compression levels at the client side +without either a initial negotiation of capabilities or an automatic retry +mechanism. Therefore, compression levels are only supported at the server side, +which is aware of the client's capabilities by virtue of the incoming +Message-Accept-Encoding header. + +### Propagation to child RPCs + +The inheritance of the compression configuration by child RPCs is left up to the +implementation. Note that in the absence of changes to the parent channel, its +configuration will be used. + +### Test cases + +1. When a compression level is not specified for either the channel or the +message, the default channel level _none_ is considered: data MUST NOT be +compressed. +1. When per-RPC compression configuration isn't present for a message, the +channel compression configuration MUST be used. +1. When a compression method (including no compression) is specified for an +outgoing message, the message MUST be compressed accordingly. +1. A message compressed in a way not supported by its endpoint MUST fail with +INVALID\_ARGUMENT status, its associated description indicating the unsupported +condition as well as the supported ones. The returned `grpc-accept-encoding` +header MUST NOT contain the compression method (encoding) used. +1. An ill-constructed message with its [Compressed-Flag +bit](PROTOCOL-HTTP2.md#compressed-flag) +set but lacking a +"[grpc-encoding](PROTOCOL-HTTP2.md#message-encoding)" +entry different from _identity_ in its metadata MUST fail with INTERNAL status, +its associated description indicating the invalid Compressed-Flag condition. From 46e0453b7e63c70af6954f591eac65086b8784e2 Mon Sep 17 00:00:00 2001 From: "Nicolas \"Pixel\" Noble" Date: Fri, 18 Mar 2016 01:45:30 +0100 Subject: [PATCH 0002/1039] Adding a few curly braces. --- src/core/json/json_reader.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/src/core/json/json_reader.c b/src/core/json/json_reader.c index 9a97826287e..7e718939044 100644 --- a/src/core/json/json_reader.c +++ b/src/core/json/json_reader.c @@ -171,8 +171,9 @@ grpc_json_reader_status grpc_json_reader_run(grpc_json_reader *reader) { switch (reader->state) { case GRPC_JSON_STATE_OBJECT_KEY_STRING: case GRPC_JSON_STATE_VALUE_STRING: - if (reader->unicode_high_surrogate != 0) + if (reader->unicode_high_surrogate != 0) { return GRPC_JSON_PARSE_ERROR; + } json_reader_string_add_char(reader, c); break; @@ -280,8 +281,9 @@ grpc_json_reader_status grpc_json_reader_run(grpc_json_reader *reader) { break; case GRPC_JSON_STATE_OBJECT_KEY_STRING: - if (reader->unicode_high_surrogate != 0) + if (reader->unicode_high_surrogate != 0) { return GRPC_JSON_PARSE_ERROR; + } if (c == '"') { reader->state = GRPC_JSON_STATE_OBJECT_KEY_END; json_reader_set_key(reader); @@ -293,8 +295,9 @@ grpc_json_reader_status grpc_json_reader_run(grpc_json_reader *reader) { break; case GRPC_JSON_STATE_VALUE_STRING: - if (reader->unicode_high_surrogate != 0) + if (reader->unicode_high_surrogate != 0) { return GRPC_JSON_PARSE_ERROR; + } if (c == '"') { reader->state = GRPC_JSON_STATE_VALUE_END; json_reader_set_string(reader); @@ -374,8 +377,9 @@ grpc_json_reader_status grpc_json_reader_run(grpc_json_reader *reader) { } else { reader->state = GRPC_JSON_STATE_VALUE_STRING; } - if (reader->unicode_high_surrogate && c != 'u') + if (reader->unicode_high_surrogate && c != 'u') { return GRPC_JSON_PARSE_ERROR; + } switch (c) { case '"': case '/': From 23df75a97054822e1a7f89ab5a0601d6bd4b3aa6 Mon Sep 17 00:00:00 2001 From: "Nicolas \"Pixel\" Noble" Date: Fri, 18 Mar 2016 08:27:13 +0100 Subject: [PATCH 0003/1039] Copyrights. --- src/core/json/json_reader.c | 2 +- test/core/json/json_test.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/core/json/json_reader.c b/src/core/json/json_reader.c index 7e718939044..aa654dfc0e2 100644 --- a/src/core/json/json_reader.c +++ b/src/core/json/json_reader.c @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without diff --git a/test/core/json/json_test.c b/test/core/json/json_test.c index 035265a6be5..cbd96a75fe2 100644 --- a/test/core/json/json_test.c +++ b/test/core/json/json_test.c @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without From 9a4d0893881f6811646695cc73fe9752fed308ae Mon Sep 17 00:00:00 2001 From: Yosuke Ishikawa Date: Fri, 29 Apr 2016 01:05:39 +0900 Subject: [PATCH 0004/1039] Fix build error caused by immutability of value type --- src/objective-c/examples/SwiftSample/ViewController.swift | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/objective-c/examples/SwiftSample/ViewController.swift b/src/objective-c/examples/SwiftSample/ViewController.swift index a21ce07978d..80d7a47917f 100644 --- a/src/objective-c/examples/SwiftSample/ViewController.swift +++ b/src/objective-c/examples/SwiftSample/ViewController.swift @@ -71,7 +71,7 @@ class ViewController: UIViewController { NSLog("2. Response trailers: \(RPC.responseTrailers)") } - RPC.requestHeaders["My-Header"] = "My value" + RPC.requestHeaders.setObject("My value", forKey: "My-Header") RPC.start() @@ -84,7 +84,7 @@ class ViewController: UIViewController { let call = GRPCCall(host: RemoteHost, path: method.HTTPPath, requestsWriter: requestsWriter) - call.requestHeaders["My-Header"] = "My value" + call.requestHeaders.setObject("My value", forKey: "My-Header") call.startWithWriteable(GRXWriteable { response, error in if let response = response as? NSData { From 454432542a6cc6145621003ecd8303c82a4e2b7b Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Fri, 29 Apr 2016 16:19:51 -0700 Subject: [PATCH 0005/1039] Add a maybe-temporary way for apps to clear the channel cache --- .../GRPCClient/GRPCCall+ChannelArg.h | 5 +++++ .../GRPCClient/GRPCCall+ChannelArg.m | 4 ++++ src/objective-c/GRPCClient/private/GRPCHost.h | 2 ++ src/objective-c/GRPCClient/private/GRPCHost.m | 21 ++++++++++++++----- 4 files changed, 27 insertions(+), 5 deletions(-) diff --git a/src/objective-c/GRPCClient/GRPCCall+ChannelArg.h b/src/objective-c/GRPCClient/GRPCCall+ChannelArg.h index bd6b064f166..646bf43b547 100644 --- a/src/objective-c/GRPCClient/GRPCCall+ChannelArg.h +++ b/src/objective-c/GRPCClient/GRPCCall+ChannelArg.h @@ -32,6 +32,8 @@ */ #import "GRPCCall.h" +#include + /** * Methods to configure GRPC channel options. */ @@ -43,4 +45,7 @@ */ + (void)setUserAgentPrefix:(NSString *)userAgentPrefix forHost:(NSString *)host; ++ (void)closeOpenConnections DEPRECATED_MSG_ATTRIBUTE("The API for this feature is experimental, " + "and might be removed or modified at any " + "time."); @end diff --git a/src/objective-c/GRPCClient/GRPCCall+ChannelArg.m b/src/objective-c/GRPCClient/GRPCCall+ChannelArg.m index 5f9932d86dc..bcc3b915075 100644 --- a/src/objective-c/GRPCClient/GRPCCall+ChannelArg.m +++ b/src/objective-c/GRPCClient/GRPCCall+ChannelArg.m @@ -46,4 +46,8 @@ hostConfig.userAgentPrefix = userAgentPrefix; } ++ (void)closeOpenConnections { + [GRPCHost flushChannelCache]; +} + @end diff --git a/src/objective-c/GRPCClient/private/GRPCHost.h b/src/objective-c/GRPCClient/private/GRPCHost.h index 9220e2a33db..350c69bf8e8 100644 --- a/src/objective-c/GRPCClient/private/GRPCHost.h +++ b/src/objective-c/GRPCClient/private/GRPCHost.h @@ -41,6 +41,8 @@ struct grpc_channel_credentials; @interface GRPCHost : NSObject ++ (void)flushChannelCache; + @property(nonatomic, readonly) NSString *address; @property(nonatomic, copy, nullable) NSString *userAgentPrefix; @property(nonatomic, nullable) struct grpc_channel_credentials *channelCreds; diff --git a/src/objective-c/GRPCClient/private/GRPCHost.m b/src/objective-c/GRPCClient/private/GRPCHost.m index 43166cbb527..0358cc62365 100644 --- a/src/objective-c/GRPCClient/private/GRPCHost.m +++ b/src/objective-c/GRPCClient/private/GRPCHost.m @@ -48,6 +48,8 @@ NS_ASSUME_NONNULL_BEGIN // templates/src/core/surface/version.c.template . #define GRPC_OBJC_VERSION_STRING @"0.13.0" +static NSMutableDictionary *kHostCache; + @implementation GRPCHost { // TODO(mlumish): Investigate whether caching channels with strong links is a good idea. GRPCChannel *_channel; @@ -79,13 +81,12 @@ NS_ASSUME_NONNULL_BEGIN } // Look up the GRPCHost in the cache. - static NSMutableDictionary *hostCache; static dispatch_once_t cacheInitialization; dispatch_once(&cacheInitialization, ^{ - hostCache = [NSMutableDictionary dictionary]; + kHostCache = [NSMutableDictionary dictionary]; }); - @synchronized(hostCache) { - GRPCHost *cachedHost = hostCache[address]; + @synchronized(kHostCache) { + GRPCHost *cachedHost = kHostCache[address]; if (cachedHost) { return cachedHost; } @@ -93,12 +94,22 @@ NS_ASSUME_NONNULL_BEGIN if ((self = [super init])) { _address = address; _secure = YES; - hostCache[address] = self; + kHostCache[address] = self; } } return self; } ++ (void)flushChannelCache { + @synchronized(kHostCache) { + [kHostCache enumerateKeysAndObjectsUsingBlock:^(id _Nonnull key, + GRPCHost * _Nonnull host, + BOOL * _Nonnull stop) { + [host disconnect]; + }]; + } +} + - (nullable grpc_call *)unmanagedCallWithPath:(NSString *)path completionQueue:(GRPCCompletionQueue *)queue { GRPCChannel *channel; From fa70dacf95b93486b7dfe0c21ada90d75a5d5bcd Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Fri, 29 Apr 2016 19:58:52 -0700 Subject: [PATCH 0006/1039] =?UTF-8?q?Smoke=20test=20that=20things=20still?= =?UTF-8?q?=20work=20after=20=E2=80=9CcloseOpenConnections=E2=80=9D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- src/objective-c/tests/InteropTests.m | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/src/objective-c/tests/InteropTests.m b/src/objective-c/tests/InteropTests.m index 26877b1ae84..379271a312d 100644 --- a/src/objective-c/tests/InteropTests.m +++ b/src/objective-c/tests/InteropTests.m @@ -35,6 +35,7 @@ #include +#import #import #import #import @@ -312,4 +313,25 @@ [self waitForExpectationsWithTimeout:8 handler:nil]; } +- (void)testRPCAfterClosingOpenConnections { + XCTAssertNotNil(self.class.host); + __weak XCTestExpectation *expectation = + [self expectationWithDescription:@"RPC after closing connection"]; + + RMTEmpty *request = [RMTEmpty message]; + + [_service emptyCallWithRequest:request handler:^(RMTEmpty *response, NSError *error) { + XCTAssertNil(error, @"First RPC finished with unexpected error: %@", error); + + [GRPCCall closeOpenConnections]; + + [_service emptyCallWithRequest:request handler:^(RMTEmpty *response, NSError *error) { + XCTAssertNil(error, @"Second RPC finished with unexpected error: %@", error); + [expectation fulfill]; + }]; + }]; + + [self waitForExpectationsWithTimeout:4 handler:nil]; +} + @end From 2e3c9ad6dd6fbf4f7532eedd9d5d2b52e7a3eb1f Mon Sep 17 00:00:00 2001 From: Julien Boeuf Date: Tue, 19 Jan 2016 17:14:38 -0800 Subject: [PATCH 0007/1039] Starting the work to fix #3803. - We still need a way to bubble up this error. --- src/core/lib/security/client_auth_filter.c | 3 +- src/core/lib/security/credentials.c | 35 +++++++------ src/core/lib/security/credentials.h | 4 +- test/core/security/credentials_test.c | 49 ++++++++++--------- test/core/security/oauth2_utils.c | 3 +- .../print_google_default_creds_token.c | 3 +- 6 files changed, 54 insertions(+), 43 deletions(-) diff --git a/src/core/lib/security/client_auth_filter.c b/src/core/lib/security/client_auth_filter.c index 8b58cb86bf9..f6de877021e 100644 --- a/src/core/lib/security/client_auth_filter.c +++ b/src/core/lib/security/client_auth_filter.c @@ -98,7 +98,8 @@ static void bubble_up_error(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, static void on_credentials_metadata(grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, size_t num_md, - grpc_credentials_status status) { + grpc_credentials_status status, + const char *error_details) { grpc_call_element *elem = (grpc_call_element *)user_data; call_data *calld = elem->call_data; grpc_transport_stream_op *op = &calld->op; diff --git a/src/core/lib/security/credentials.c b/src/core/lib/security/credentials.c index fd5ad3589b7..1c9832333aa 100644 --- a/src/core/lib/security/credentials.c +++ b/src/core/lib/security/credentials.c @@ -122,7 +122,7 @@ void grpc_call_credentials_get_request_metadata( grpc_credentials_metadata_cb cb, void *user_data) { if (creds == NULL || creds->vtable->get_request_metadata == NULL) { if (cb != NULL) { - cb(exec_ctx, user_data, NULL, 0, GRPC_CREDENTIALS_OK); + cb(exec_ctx, user_data, NULL, 0, GRPC_CREDENTIALS_OK, NULL); } return; } @@ -497,10 +497,10 @@ static void jwt_get_request_metadata(grpc_exec_ctx *exec_ctx, if (jwt_md != NULL) { cb(exec_ctx, user_data, jwt_md->entries, jwt_md->num_entries, - GRPC_CREDENTIALS_OK); + GRPC_CREDENTIALS_OK, NULL); grpc_credentials_md_store_unref(jwt_md); } else { - cb(exec_ctx, user_data, NULL, 0, GRPC_CREDENTIALS_ERROR); + cb(exec_ctx, user_data, NULL, 0, GRPC_CREDENTIALS_ERROR, ""); } } @@ -660,10 +660,10 @@ static void on_oauth2_token_fetcher_http_response( c->token_expiration = gpr_time_add(gpr_now(GPR_CLOCK_REALTIME), token_lifetime); r->cb(exec_ctx, r->user_data, c->access_token_md->entries, - c->access_token_md->num_entries, status); + c->access_token_md->num_entries, status, NULL); } else { c->token_expiration = gpr_inf_past(GPR_CLOCK_REALTIME); - r->cb(exec_ctx, r->user_data, NULL, 0, status); + r->cb(exec_ctx, r->user_data, NULL, 0, status, ""); } gpr_mu_unlock(&c->mu); grpc_credentials_metadata_request_destroy(r); @@ -691,7 +691,7 @@ static void oauth2_token_fetcher_get_request_metadata( } if (cached_access_token_md != NULL) { cb(exec_ctx, user_data, cached_access_token_md->entries, - cached_access_token_md->num_entries, GRPC_CREDENTIALS_OK); + cached_access_token_md->num_entries, GRPC_CREDENTIALS_OK, NULL); grpc_credentials_md_store_unref(cached_access_token_md); } else { c->fetch_func( @@ -821,7 +821,7 @@ static void on_simulated_token_fetch_done(grpc_exec_ctx *exec_ctx, (grpc_credentials_metadata_request *)user_data; grpc_md_only_test_credentials *c = (grpc_md_only_test_credentials *)r->creds; r->cb(exec_ctx, r->user_data, c->md_store->entries, c->md_store->num_entries, - GRPC_CREDENTIALS_OK); + GRPC_CREDENTIALS_OK, NULL); grpc_credentials_metadata_request_destroy(r); } @@ -837,7 +837,7 @@ static void md_only_test_get_request_metadata( grpc_executor_enqueue( grpc_closure_create(on_simulated_token_fetch_done, cb_arg), true); } else { - cb(exec_ctx, user_data, c->md_store->entries, 1, GRPC_CREDENTIALS_OK); + cb(exec_ctx, user_data, c->md_store->entries, 1, GRPC_CREDENTIALS_OK, NULL); } } @@ -870,7 +870,8 @@ static void access_token_get_request_metadata( grpc_pollset *pollset, grpc_auth_metadata_context context, grpc_credentials_metadata_cb cb, void *user_data) { grpc_access_token_credentials *c = (grpc_access_token_credentials *)creds; - cb(exec_ctx, user_data, c->access_token_md->entries, 1, GRPC_CREDENTIALS_OK); + cb(exec_ctx, user_data, c->access_token_md->entries, 1, GRPC_CREDENTIALS_OK, + NULL); } static grpc_call_credentials_vtable access_token_vtable = { @@ -973,11 +974,12 @@ static void composite_call_md_context_destroy( static void composite_call_metadata_cb(grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, size_t num_md, - grpc_credentials_status status) { + grpc_credentials_status status, + const char *error_details) { grpc_composite_call_credentials_metadata_context *ctx = (grpc_composite_call_credentials_metadata_context *)user_data; if (status != GRPC_CREDENTIALS_OK) { - ctx->cb(exec_ctx, ctx->user_data, NULL, 0, status); + ctx->cb(exec_ctx, ctx->user_data, NULL, 0, status, NULL); return; } @@ -1002,7 +1004,7 @@ static void composite_call_metadata_cb(grpc_exec_ctx *exec_ctx, void *user_data, /* We're done!. */ ctx->cb(exec_ctx, ctx->user_data, ctx->md_elems->entries, - ctx->md_elems->num_entries, GRPC_CREDENTIALS_OK); + ctx->md_elems->num_entries, GRPC_CREDENTIALS_OK, NULL); composite_call_md_context_destroy(ctx); } @@ -1122,7 +1124,7 @@ static void iam_get_request_metadata(grpc_exec_ctx *exec_ctx, void *user_data) { grpc_google_iam_credentials *c = (grpc_google_iam_credentials *)creds; cb(exec_ctx, user_data, c->iam_md->entries, c->iam_md->num_entries, - GRPC_CREDENTIALS_OK); + GRPC_CREDENTIALS_OK, NULL); } static grpc_call_credentials_vtable iam_vtable = {iam_destruct, @@ -1178,7 +1180,8 @@ static void plugin_md_request_metadata_ready(void *request, gpr_log(GPR_ERROR, "Getting metadata from plugin failed with error: %s", error_details); } - r->cb(&exec_ctx, r->user_data, NULL, 0, GRPC_CREDENTIALS_ERROR); + r->cb(&exec_ctx, r->user_data, NULL, 0, GRPC_CREDENTIALS_ERROR, + error_details); } else { size_t i; grpc_credentials_md *md_array = NULL; @@ -1190,7 +1193,7 @@ static void plugin_md_request_metadata_ready(void *request, gpr_slice_from_copied_buffer(md[i].value, md[i].value_length); } } - r->cb(&exec_ctx, r->user_data, md_array, num_md, GRPC_CREDENTIALS_OK); + r->cb(&exec_ctx, r->user_data, md_array, num_md, GRPC_CREDENTIALS_OK, NULL); if (md_array != NULL) { for (i = 0; i < num_md; i++) { gpr_slice_unref(md_array[i].key); @@ -1218,7 +1221,7 @@ static void plugin_get_request_metadata(grpc_exec_ctx *exec_ctx, c->plugin.get_metadata(c->plugin.state, context, plugin_md_request_metadata_ready, request); } else { - cb(exec_ctx, user_data, NULL, 0, GRPC_CREDENTIALS_OK); + cb(exec_ctx, user_data, NULL, 0, GRPC_CREDENTIALS_OK, NULL); } } diff --git a/src/core/lib/security/credentials.h b/src/core/lib/security/credentials.h index 0373ceaa3fc..412b6e48fc9 100644 --- a/src/core/lib/security/credentials.h +++ b/src/core/lib/security/credentials.h @@ -160,11 +160,13 @@ void grpc_credentials_md_store_unref(grpc_credentials_md_store *store); /* --- grpc_call_credentials. --- */ +/* error_details must be NULL if status is GRPC_CREDENTIALS_OK. */ typedef void (*grpc_credentials_metadata_cb)(grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, size_t num_md, - grpc_credentials_status status); + grpc_credentials_status status, + const char *error_details); typedef struct { void (*destruct)(grpc_call_credentials *c); diff --git a/test/core/security/credentials_test.c b/test/core/security/credentials_test.c index 78672932787..36a4fdae14e 100644 --- a/test/core/security/credentials_test.c +++ b/test/core/security/credentials_test.c @@ -338,13 +338,15 @@ static void check_metadata(expected_md *expected, grpc_credentials_md *md_elems, static void check_google_iam_metadata(grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, size_t num_md, - grpc_credentials_status status) { + grpc_credentials_status status, + const char *error_details) { grpc_call_credentials *c = (grpc_call_credentials *)user_data; expected_md emd[] = {{GRPC_IAM_AUTHORIZATION_TOKEN_METADATA_KEY, test_google_iam_authorization_token}, {GRPC_IAM_AUTHORITY_SELECTOR_METADATA_KEY, test_google_iam_authority_selector}}; GPR_ASSERT(status == GRPC_CREDENTIALS_OK); + GPR_ASSERT(error_details == NULL); GPR_ASSERT(num_md == 2); check_metadata(emd, md_elems, num_md); grpc_call_credentials_unref(c); @@ -362,14 +364,13 @@ static void test_google_iam_creds(void) { grpc_exec_ctx_finish(&exec_ctx); } -static void check_access_token_metadata(grpc_exec_ctx *exec_ctx, - void *user_data, - grpc_credentials_md *md_elems, - size_t num_md, - grpc_credentials_status status) { +static void check_access_token_metadata( + grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, + size_t num_md, grpc_credentials_status status, const char *error_details) { grpc_call_credentials *c = (grpc_call_credentials *)user_data; expected_md emd[] = {{GRPC_AUTHORIZATION_METADATA_KEY, "Bearer blah"}}; GPR_ASSERT(status == GRPC_CREDENTIALS_OK); + GPR_ASSERT(error_details == NULL); GPR_ASSERT(num_md == 1); check_metadata(emd, md_elems, num_md); grpc_call_credentials_unref(c); @@ -418,7 +419,7 @@ static void test_channel_oauth2_composite_creds(void) { static void check_oauth2_google_iam_composite_metadata( grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, - size_t num_md, grpc_credentials_status status) { + size_t num_md, grpc_credentials_status status, const char *error_details) { grpc_call_credentials *c = (grpc_call_credentials *)user_data; expected_md emd[] = { {GRPC_AUTHORIZATION_METADATA_KEY, test_oauth2_bearer_token}, @@ -427,6 +428,7 @@ static void check_oauth2_google_iam_composite_metadata( {GRPC_IAM_AUTHORITY_SELECTOR_METADATA_KEY, test_google_iam_authority_selector}}; GPR_ASSERT(status == GRPC_CREDENTIALS_OK); + GPR_ASSERT(error_details == NULL); GPR_ASSERT(num_md == 3); check_metadata(emd, md_elems, num_md); grpc_call_credentials_unref(c); @@ -511,8 +513,9 @@ static void test_channel_oauth2_google_iam_composite_creds(void) { static void on_oauth2_creds_get_metadata_success( grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, - size_t num_md, grpc_credentials_status status) { + size_t num_md, grpc_credentials_status status, const char *error_details) { GPR_ASSERT(status == GRPC_CREDENTIALS_OK); + GPR_ASSERT(error_details == NULL); GPR_ASSERT(num_md == 1); GPR_ASSERT(gpr_slice_str_cmp(md_elems[0].key, "authorization") == 0); GPR_ASSERT(gpr_slice_str_cmp(md_elems[0].value, @@ -524,7 +527,7 @@ static void on_oauth2_creds_get_metadata_success( static void on_oauth2_creds_get_metadata_failure( grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, - size_t num_md, grpc_credentials_status status) { + size_t num_md, grpc_credentials_status status, const char *error_details) { GPR_ASSERT(status == GRPC_CREDENTIALS_ERROR); GPR_ASSERT(num_md == 0); GPR_ASSERT(user_data != NULL); @@ -760,14 +763,13 @@ static char *encode_and_sign_jwt_should_not_be_called( return NULL; } -static void on_jwt_creds_get_metadata_success(grpc_exec_ctx *exec_ctx, - void *user_data, - grpc_credentials_md *md_elems, - size_t num_md, - grpc_credentials_status status) { +static void on_jwt_creds_get_metadata_success( + grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, + size_t num_md, grpc_credentials_status status, const char *error_details) { char *expected_md_value; gpr_asprintf(&expected_md_value, "Bearer %s", test_signed_jwt); GPR_ASSERT(status == GRPC_CREDENTIALS_OK); + GPR_ASSERT(error_details == NULL); GPR_ASSERT(num_md == 1); GPR_ASSERT(gpr_slice_str_cmp(md_elems[0].key, "authorization") == 0); GPR_ASSERT(gpr_slice_str_cmp(md_elems[0].value, expected_md_value) == 0); @@ -776,11 +778,9 @@ static void on_jwt_creds_get_metadata_success(grpc_exec_ctx *exec_ctx, gpr_free(expected_md_value); } -static void on_jwt_creds_get_metadata_failure(grpc_exec_ctx *exec_ctx, - void *user_data, - grpc_credentials_md *md_elems, - size_t num_md, - grpc_credentials_status status) { +static void on_jwt_creds_get_metadata_failure( + grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, + size_t num_md, grpc_credentials_status status, const char *error_details) { GPR_ASSERT(status == GRPC_CREDENTIALS_ERROR); GPR_ASSERT(num_md == 0); GPR_ASSERT(user_data != NULL); @@ -1024,6 +1024,8 @@ static void plugin_get_metadata_success(void *state, cb(user_data, md, GPR_ARRAY_SIZE(md), GRPC_STATUS_OK, NULL); } +static const char *plugin_error_details = "Could not get metadata for plugin."; + static void plugin_get_metadata_failure(void *state, grpc_auth_metadata_context context, grpc_credentials_plugin_metadata_cb cb, @@ -1034,13 +1036,12 @@ static void plugin_get_metadata_failure(void *state, GPR_ASSERT(context.channel_auth_context == NULL); GPR_ASSERT(context.reserved == NULL); *s = PLUGIN_GET_METADATA_CALLED_STATE; - cb(user_data, NULL, 0, GRPC_STATUS_UNAUTHENTICATED, - "Could not get metadata for plugin."); + cb(user_data, NULL, 0, GRPC_STATUS_UNAUTHENTICATED, plugin_error_details); } static void on_plugin_metadata_received_success( grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, - size_t num_md, grpc_credentials_status status) { + size_t num_md, grpc_credentials_status status, const char *error_details) { size_t i = 0; GPR_ASSERT(user_data == NULL); GPR_ASSERT(md_elems != NULL); @@ -1053,11 +1054,13 @@ static void on_plugin_metadata_received_success( static void on_plugin_metadata_received_failure( grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, - size_t num_md, grpc_credentials_status status) { + size_t num_md, grpc_credentials_status status, const char *error_details) { GPR_ASSERT(user_data == NULL); GPR_ASSERT(md_elems == NULL); GPR_ASSERT(num_md == 0); GPR_ASSERT(status == GRPC_CREDENTIALS_ERROR); + GPR_ASSERT(error_details != NULL); + GPR_ASSERT(strcmp(error_details, plugin_error_details) == 0); } static void plugin_destroy(void *state) { diff --git a/test/core/security/oauth2_utils.c b/test/core/security/oauth2_utils.c index 20815d184cd..10155fccf59 100644 --- a/test/core/security/oauth2_utils.c +++ b/test/core/security/oauth2_utils.c @@ -53,7 +53,8 @@ typedef struct { static void on_oauth2_response(grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, size_t num_md, - grpc_credentials_status status) { + grpc_credentials_status status, + const char *error_details) { oauth2_request *request = user_data; char *token = NULL; gpr_slice token_slice; diff --git a/test/core/security/print_google_default_creds_token.c b/test/core/security/print_google_default_creds_token.c index 99bce4fbdfb..be900d84498 100644 --- a/test/core/security/print_google_default_creds_token.c +++ b/test/core/security/print_google_default_creds_token.c @@ -53,7 +53,8 @@ typedef struct { static void on_metadata_response(grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, size_t num_md, - grpc_credentials_status status) { + grpc_credentials_status status, + const char *error_details) { synchronizer *sync = user_data; if (status == GRPC_CREDENTIALS_ERROR) { fprintf(stderr, "Fetching token failed.\n"); From ef96264d8724a71f4f1fef5ebaf361d28b1a1752 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 18 May 2016 22:57:17 -0700 Subject: [PATCH 0008/1039] Support SO_REUSEPORT --- include/grpc/impl/codegen/grpc_types.h | 2 + .../chttp2/server/insecure/server_chttp2.c | 3 +- .../server/secure/server_secure_chttp2.c | 3 +- .../lib/iomgr/socket_utils_common_posix.c | 22 ++++ src/core/lib/iomgr/socket_utils_posix.h | 3 + src/core/lib/iomgr/tcp_server.h | 3 + src/core/lib/iomgr/tcp_server_posix.c | 122 ++++++++++++++++-- 7 files changed, 143 insertions(+), 15 deletions(-) diff --git a/include/grpc/impl/codegen/grpc_types.h b/include/grpc/impl/codegen/grpc_types.h index af3d0c21916..cd3cae71b6a 100644 --- a/include/grpc/impl/codegen/grpc_types.h +++ b/include/grpc/impl/codegen/grpc_types.h @@ -154,6 +154,8 @@ typedef struct { #define GRPC_SSL_TARGET_NAME_OVERRIDE_ARG "grpc.ssl_target_name_override" /* Maximum metadata size */ #define GRPC_ARG_MAX_METADATA_SIZE "grpc.max_metadata_size" +/** If non-zero, allow the use of SO_REUSEPORT if it's available (default 1) */ +#define GRPC_ARG_ALLOW_REUSEPORT "grpc.so_reuseport" /** Result of a grpc call. If the caller satisfies the prerequisites of a particular operation, the grpc_call_error returned will be GRPC_CALL_OK. diff --git a/src/core/ext/transport/chttp2/server/insecure/server_chttp2.c b/src/core/ext/transport/chttp2/server/insecure/server_chttp2.c index d1a58b66211..08a09a933a6 100644 --- a/src/core/ext/transport/chttp2/server/insecure/server_chttp2.c +++ b/src/core/ext/transport/chttp2/server/insecure/server_chttp2.c @@ -102,7 +102,8 @@ int grpc_server_add_insecure_http2_port(grpc_server *server, const char *addr) { goto error; } - err = grpc_tcp_server_create(NULL, &tcp); + err = + grpc_tcp_server_create(NULL, grpc_server_get_channel_args(server), &tcp); if (err != GRPC_ERROR_NONE) { goto error; } diff --git a/src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c b/src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c index 629959b7c03..64162cc7857 100644 --- a/src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c +++ b/src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c @@ -208,7 +208,8 @@ int grpc_server_add_secure_http2_port(grpc_server *server, const char *addr, state = gpr_malloc(sizeof(*state)); memset(state, 0, sizeof(*state)); grpc_closure_init(&state->destroy_closure, destroy_done, state); - err = grpc_tcp_server_create(&state->destroy_closure, &tcp); + err = grpc_tcp_server_create(&state->destroy_closure, + grpc_server_get_channel_args(server), &tcp); if (err != GRPC_ERROR_NONE) { goto error; } diff --git a/src/core/lib/iomgr/socket_utils_common_posix.c b/src/core/lib/iomgr/socket_utils_common_posix.c index d1721c910c4..bd8ef00c859 100644 --- a/src/core/lib/iomgr/socket_utils_common_posix.c +++ b/src/core/lib/iomgr/socket_utils_common_posix.c @@ -154,6 +154,28 @@ grpc_error *grpc_set_socket_reuse_addr(int fd, int reuse) { return GRPC_ERROR_NONE; } +/* set a socket to reuse old addresses */ +grpc_error *grpc_set_socket_reuse_port(int fd, int reuse) { +#ifndef SO_REUSEPORT + return GRPC_ERROR_CREATE("SO_REUSEPORT unavailable on compiling system"); +#else + int val = (reuse != 0); + int newval; + socklen_t intlen = sizeof(newval); + if (0 != setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &val, sizeof(val))) { + return GRPC_OS_ERROR(errno, "setsockopt(SO_REUSEPORT)"); + } + if (0 != getsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &newval, &intlen)) { + return GRPC_OS_ERROR(errno, "getsockopt(SO_REUSEPORT)"); + } + if ((newval != 0) != val) { + return GRPC_ERROR_CREATE("Failed to set SO_REUSEPORT"); + } + + return GRPC_ERROR_NONE; +#endif +} + /* disable nagle */ grpc_error *grpc_set_socket_low_latency(int fd, int low_latency) { int val = (low_latency != 0); diff --git a/src/core/lib/iomgr/socket_utils_posix.h b/src/core/lib/iomgr/socket_utils_posix.h index 4e66cbc0c58..607fbc7a791 100644 --- a/src/core/lib/iomgr/socket_utils_posix.h +++ b/src/core/lib/iomgr/socket_utils_posix.h @@ -55,6 +55,9 @@ grpc_error *grpc_set_socket_reuse_addr(int fd, int reuse); /* disable nagle */ grpc_error *grpc_set_socket_low_latency(int fd, int low_latency); +/* set SO_REUSEPORT */ +grpc_error *grpc_set_socket_reuse_port(int fd, int reuse); + /* Returns true if this system can create AF_INET6 sockets bound to ::1. The value is probed once, and cached for the life of the process. diff --git a/src/core/lib/iomgr/tcp_server.h b/src/core/lib/iomgr/tcp_server.h index 924121f0c79..79f0242b26d 100644 --- a/src/core/lib/iomgr/tcp_server.h +++ b/src/core/lib/iomgr/tcp_server.h @@ -34,6 +34,8 @@ #ifndef GRPC_CORE_LIB_IOMGR_TCP_SERVER_H #define GRPC_CORE_LIB_IOMGR_TCP_SERVER_H +#include + #include "src/core/lib/iomgr/closure.h" #include "src/core/lib/iomgr/endpoint.h" @@ -58,6 +60,7 @@ typedef void (*grpc_tcp_server_cb)(grpc_exec_ctx *exec_ctx, void *arg, If shutdown_complete is not NULL, it will be used by grpc_tcp_server_unref() when the ref count reaches zero. */ grpc_error *grpc_tcp_server_create(grpc_closure *shutdown_complete, + const grpc_channel_args *args, grpc_tcp_server **server); /* Start listening to bound ports */ diff --git a/src/core/lib/iomgr/tcp_server_posix.c b/src/core/lib/iomgr/tcp_server_posix.c index db70fada96a..7047934780b 100644 --- a/src/core/lib/iomgr/tcp_server_posix.c +++ b/src/core/lib/iomgr/tcp_server_posix.c @@ -112,8 +112,10 @@ struct grpc_tcp_server { /* destroyed port count: how many ports are completely destroyed */ size_t destroyed_ports; - /* is this server shutting down? (boolean) */ - int shutdown; + /* is this server shutting down? */ + bool shutdown; + /* use SO_REUSEPORT */ + bool so_reuseport; /* linked list of server ports */ grpc_tcp_listener *head; @@ -132,14 +134,42 @@ struct grpc_tcp_server { size_t pollset_count; }; +static gpr_once check_init = GPR_ONCE_INIT; +static bool has_so_reuseport; + +static void init(void) { + int s = socket(AF_INET, SOCK_STREAM, 0); + if (s >= 0) { + has_so_reuseport = GRPC_LOG_IF_ERROR("check for SO_REUSEPORT", + grpc_set_socket_reuse_port(s, 1)); + close(s); + } +} + grpc_error *grpc_tcp_server_create(grpc_closure *shutdown_complete, + const grpc_channel_args *args, grpc_tcp_server **server) { + gpr_once_init(&check_init, init); + grpc_tcp_server *s = gpr_malloc(sizeof(grpc_tcp_server)); + s->so_reuseport = has_so_reuseport; + for (size_t i = 0; i < (args == NULL ? 0 : args->num_args); i++) { + if (0 == strcmp(GRPC_ARG_ALLOW_REUSEPORT, args->args[i].key)) { + if (args->args[i].type == GRPC_ARG_INTEGER) { + s->so_reuseport = + has_so_reuseport && (args->args[i].value.integer != 0); + } else { + gpr_free(s); + return GRPC_ERROR_CREATE(GRPC_ARG_ALLOW_REUSEPORT + " must be an integer"); + } + } + } gpr_ref_init(&s->refs, 1); gpr_mu_init(&s->mu); s->active_ports = 0; s->destroyed_ports = 0; - s->shutdown = 0; + s->shutdown = false; s->shutdown_starting.head = NULL; s->shutdown_starting.tail = NULL; s->shutdown_complete = shutdown_complete; @@ -214,7 +244,7 @@ static void tcp_server_destroy(grpc_exec_ctx *exec_ctx, grpc_tcp_server *s) { gpr_mu_lock(&s->mu); GPR_ASSERT(!s->shutdown); - s->shutdown = 1; + s->shutdown = true; /* shutdown all fd's */ if (s->active_ports) { @@ -264,13 +294,19 @@ static int get_max_accept_queue_size(void) { /* Prepare a recently-created socket for listening. */ static grpc_error *prepare_socket(int fd, const struct sockaddr *addr, - size_t addr_len, int *port) { + size_t addr_len, bool so_reuseport, + int *port) { struct sockaddr_storage sockname_temp; socklen_t sockname_len; grpc_error *err = GRPC_ERROR_NONE; GPR_ASSERT(fd >= 0); + if (so_reuseport) { + err = grpc_set_socket_reuse_port(fd, 1); + if (err != GRPC_ERROR_NONE) goto error; + } + err = grpc_set_socket_nonblocking(fd, 1); if (err != GRPC_ERROR_NONE) goto error; err = grpc_set_socket_cloexec(fd, 1); @@ -397,7 +433,7 @@ static grpc_error *add_socket_to_server(grpc_tcp_server *s, int fd, char *addr_str; char *name; - grpc_error *err = prepare_socket(fd, addr, addr_len, &port); + grpc_error *err = prepare_socket(fd, addr, addr_len, s->so_reuseport, &port); if (err == GRPC_ERROR_NONE) { GPR_ASSERT(port > 0); grpc_sockaddr_to_string(&addr_str, (struct sockaddr *)&addr, 1); @@ -433,6 +469,51 @@ static grpc_error *add_socket_to_server(grpc_tcp_server *s, int fd, return err; } +static grpc_error *clone_port(grpc_tcp_listener *listener, unsigned count) { + grpc_tcp_listener *sp = NULL; + char *addr_str; + char *name; + grpc_error *err; + + for (grpc_tcp_listener *l = listener->next; l && l->is_sibling; l = l->next) { + l->fd_index += count; + } + + for (unsigned i = 0; i < count; i++) { + int fd, port; + grpc_dualstack_mode dsmode; + err = grpc_create_dualstack_socket(&listener->addr.sockaddr, SOCK_STREAM, 0, + &dsmode, &fd); + if (err != GRPC_ERROR_NONE) return err; + err = prepare_socket(fd, &listener->addr.sockaddr, listener->addr_len, true, + &port); + if (err != GRPC_ERROR_NONE) return err; + grpc_sockaddr_to_string(&addr_str, &listener->addr.sockaddr, 1); + gpr_asprintf(&name, "tcp-server-listener:%s/clone-%d", addr_str, i); + sp = gpr_malloc(sizeof(grpc_tcp_listener)); + sp->next = listener->next; + listener->next = sp; + sp->server = listener->server; + sp->fd = fd; + sp->emfd = grpc_fd_create(fd, name); + memcpy(sp->addr.untyped, listener->addr.untyped, listener->addr_len); + sp->addr_len = listener->addr_len; + sp->port = port; + sp->port_index = listener->port_index; + sp->fd_index = listener->fd_index + count - i; + sp->is_sibling = 1; + sp->sibling = listener->is_sibling ? listener->sibling : listener; + GPR_ASSERT(sp->emfd); + while (listener->server->tail->next != NULL) { + listener->server->tail = listener->server->tail->next; + } + gpr_free(addr_str); + gpr_free(name); + } + + return GRPC_ERROR_NONE; +} + grpc_error *grpc_tcp_server_add_port(grpc_tcp_server *s, const void *addr, size_t addr_len, int *out_port) { grpc_tcp_listener *sp; @@ -589,14 +670,29 @@ void grpc_tcp_server_start(grpc_exec_ctx *exec_ctx, grpc_tcp_server *s, s->on_accept_cb_arg = on_accept_cb_arg; s->pollsets = pollsets; s->pollset_count = pollset_count; - for (sp = s->head; sp; sp = sp->next) { - for (i = 0; i < pollset_count; i++) { - grpc_pollset_add_fd(exec_ctx, pollsets[i], sp->emfd); + sp = s->head; + while (sp != NULL) { + if (s->so_reuseport && pollset_count > 1) { + GPR_ASSERT(GRPC_LOG_IF_ERROR( + "clone_port", clone_port(sp, (unsigned)(pollset_count - 1)))); + for (i = 0; i < pollset_count; i++) { + grpc_pollset_add_fd(exec_ctx, pollsets[i], sp->emfd); + sp->read_closure.cb = on_read; + sp->read_closure.cb_arg = sp; + grpc_fd_notify_on_read(exec_ctx, sp->emfd, &sp->read_closure); + s->active_ports++; + sp = sp->next; + } + } else { + for (i = 0; i < pollset_count; i++) { + grpc_pollset_add_fd(exec_ctx, pollsets[i], sp->emfd); + } + sp->read_closure.cb = on_read; + sp->read_closure.cb_arg = sp; + grpc_fd_notify_on_read(exec_ctx, sp->emfd, &sp->read_closure); + s->active_ports++; + sp = sp->next; } - sp->read_closure.cb = on_read; - sp->read_closure.cb_arg = sp; - grpc_fd_notify_on_read(exec_ctx, sp->emfd, &sp->read_closure); - s->active_ports++; } gpr_mu_unlock(&s->mu); } From c49464de3ed6108956561128b006c78777bd6db7 Mon Sep 17 00:00:00 2001 From: Julien Boeuf Date: Wed, 18 May 2016 23:11:50 -0700 Subject: [PATCH 0009/1039] replacing cancel op by close op in order to plumb error message. --- src/core/lib/security/client_auth_filter.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/core/lib/security/client_auth_filter.c b/src/core/lib/security/client_auth_filter.c index f6de877021e..3908b734a27 100644 --- a/src/core/lib/security/client_auth_filter.c +++ b/src/core/lib/security/client_auth_filter.c @@ -91,7 +91,8 @@ static void bubble_up_error(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_status_code status, const char *error_msg) { call_data *calld = elem->call_data; gpr_log(GPR_ERROR, "Client side authentication failure: %s", error_msg); - grpc_transport_stream_op_add_cancellation(&calld->op, status); + gpr_slice error_slice = gpr_slice_from_copied_string(error_msg); + grpc_transport_stream_op_add_close(&calld->op, status, &error_slice); grpc_call_next_op(exec_ctx, elem, &calld->op); } @@ -108,7 +109,9 @@ static void on_credentials_metadata(grpc_exec_ctx *exec_ctx, void *user_data, reset_auth_metadata_context(&calld->auth_md_context); if (status != GRPC_CREDENTIALS_OK) { bubble_up_error(exec_ctx, elem, GRPC_STATUS_UNAUTHENTICATED, - "Credentials failed to get metadata."); + (error_details != NULL && strlen(error_details) > 0) + ? error_details + : "Credentials failed to get metadata."); return; } GPR_ASSERT(num_md <= MAX_CREDENTIALS_METADATA_COUNT); From 9565fd717586d01b092dc6eb70dd9485d3c633a9 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 18 May 2016 23:17:09 -0700 Subject: [PATCH 0010/1039] Fix compile --- .../ext/transport/chttp2/server/secure/server_secure_chttp2.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c b/src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c index 9898bf05ae0..3286b220d40 100644 --- a/src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c +++ b/src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c @@ -144,7 +144,7 @@ static void start(grpc_exec_ctx *exec_ctx, grpc_server *server, void *statep, static void destroy_done(grpc_exec_ctx *exec_ctx, void *statep, grpc_error *error) { - grpc_server_secure_state *state = statep; + server_secure_state *state = statep; if (state->destroy_callback != NULL) { state->destroy_callback->cb(exec_ctx, state->destroy_callback->cb_arg, GRPC_ERROR_REF(error)); From 644da9857300151b27a1f297c40d971597a0845f Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 19 May 2016 09:19:28 -0700 Subject: [PATCH 0011/1039] Fixes --- src/core/lib/iomgr/tcp_server_posix.c | 1 + test/core/iomgr/tcp_server_posix_test.c | 10 +++++----- test/core/surface/concurrent_connectivity_test.c | 2 +- test/core/surface/server_chttp2_test.c | 8 +++++++- test/core/surface/server_test.c | 10 ++++++++-- test/core/util/test_tcp_server.c | 2 +- third_party/protobuf | 2 +- 7 files changed, 24 insertions(+), 11 deletions(-) diff --git a/src/core/lib/iomgr/tcp_server_posix.c b/src/core/lib/iomgr/tcp_server_posix.c index 7047934780b..83627e59d9d 100644 --- a/src/core/lib/iomgr/tcp_server_posix.c +++ b/src/core/lib/iomgr/tcp_server_posix.c @@ -488,6 +488,7 @@ static grpc_error *clone_port(grpc_tcp_listener *listener, unsigned count) { err = prepare_socket(fd, &listener->addr.sockaddr, listener->addr_len, true, &port); if (err != GRPC_ERROR_NONE) return err; + listener->server->nports++; grpc_sockaddr_to_string(&addr_str, &listener->addr.sockaddr, 1); gpr_asprintf(&name, "tcp-server-listener:%s/clone-%d", addr_str, i); sp = gpr_malloc(sizeof(grpc_tcp_listener)); diff --git a/test/core/iomgr/tcp_server_posix_test.c b/test/core/iomgr/tcp_server_posix_test.c index a37ff1773b2..54437f6a9bd 100644 --- a/test/core/iomgr/tcp_server_posix_test.c +++ b/test/core/iomgr/tcp_server_posix_test.c @@ -128,7 +128,7 @@ static void on_connect(grpc_exec_ctx *exec_ctx, void *arg, grpc_endpoint *tcp, static void test_no_op(void) { grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; grpc_tcp_server *s; - GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, &s)); + GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, NULL, &s)); grpc_tcp_server_unref(&exec_ctx, s); grpc_exec_ctx_finish(&exec_ctx); } @@ -136,7 +136,7 @@ static void test_no_op(void) { static void test_no_op_with_start(void) { grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; grpc_tcp_server *s; - GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, &s)); + GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, NULL, &s)); LOG_TEST("test_no_op_with_start"); grpc_tcp_server_start(&exec_ctx, s, NULL, 0, on_connect, NULL); grpc_tcp_server_unref(&exec_ctx, s); @@ -147,7 +147,7 @@ static void test_no_op_with_port(void) { grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; struct sockaddr_in addr; grpc_tcp_server *s; - GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, &s)); + GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, NULL, &s)); LOG_TEST("test_no_op_with_port"); memset(&addr, 0, sizeof(addr)); @@ -165,7 +165,7 @@ static void test_no_op_with_port_and_start(void) { grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; struct sockaddr_in addr; grpc_tcp_server *s; - GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, &s)); + GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, NULL, &s)); LOG_TEST("test_no_op_with_port_and_start"); int port; @@ -225,7 +225,7 @@ static void test_connect(unsigned n) { unsigned svr1_fd_count; int svr1_port; grpc_tcp_server *s; - GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, &s)); + GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, NULL, &s)); unsigned i; server_weak_ref weak_ref; server_weak_ref_init(&weak_ref); diff --git a/test/core/surface/concurrent_connectivity_test.c b/test/core/surface/concurrent_connectivity_test.c index 74e3183c26b..6d7d0413b28 100644 --- a/test/core/surface/concurrent_connectivity_test.c +++ b/test/core/surface/concurrent_connectivity_test.c @@ -112,7 +112,7 @@ void bad_server_thread(void *vargs) { socklen_t addr_len = sizeof(addr); int port; grpc_tcp_server *s; - grpc_error *error = grpc_tcp_server_create(NULL, &s); + grpc_error *error = grpc_tcp_server_create(NULL, NULL, &s); GPR_ASSERT(error == GRPC_ERROR_NONE); memset(&addr, 0, sizeof(addr)); addr.ss_family = AF_INET; diff --git a/test/core/surface/server_chttp2_test.c b/test/core/surface/server_chttp2_test.c index f42ca9f9cdf..40cfa6b5989 100644 --- a/test/core/surface/server_chttp2_test.c +++ b/test/core/surface/server_chttp2_test.c @@ -49,10 +49,16 @@ void test_unparsable_target(void) { } void test_add_same_port_twice() { + grpc_arg a; + a.type = GRPC_ARG_INTEGER; + a.key = GRPC_ARG_ALLOW_REUSEPORT; + a.value.integer = 0; + grpc_channel_args args = {1, &a}; + int port = grpc_pick_unused_port_or_die(); char *addr = NULL; grpc_completion_queue *cq = grpc_completion_queue_create(NULL); - grpc_server *server = grpc_server_create(NULL, NULL); + grpc_server *server = grpc_server_create(&args, NULL); grpc_server_credentials *fake_creds = grpc_fake_transport_security_server_credentials_create(); gpr_join_host_port(&addr, "localhost", port); diff --git a/test/core/surface/server_test.c b/test/core/surface/server_test.c index 3d2e25379a0..d1e646a9f15 100644 --- a/test/core/surface/server_test.c +++ b/test/core/surface/server_test.c @@ -76,9 +76,15 @@ void test_request_call_on_no_server_cq(void) { } void test_bind_server_twice(void) { + grpc_arg a; + a.type = GRPC_ARG_INTEGER; + a.key = GRPC_ARG_ALLOW_REUSEPORT; + a.value.integer = 0; + grpc_channel_args args = {1, &a}; + char *addr; - grpc_server *server1 = grpc_server_create(NULL, NULL); - grpc_server *server2 = grpc_server_create(NULL, NULL); + grpc_server *server1 = grpc_server_create(&args, NULL); + grpc_server *server2 = grpc_server_create(&args, NULL); grpc_completion_queue *cq = grpc_completion_queue_create(NULL); int port = grpc_pick_unused_port_or_die(); gpr_asprintf(&addr, "[::]:%d", port); diff --git a/test/core/util/test_tcp_server.c b/test/core/util/test_tcp_server.c index 27c16fc764a..c1d307fc779 100644 --- a/test/core/util/test_tcp_server.c +++ b/test/core/util/test_tcp_server.c @@ -73,7 +73,7 @@ void test_tcp_server_start(test_tcp_server *server, int port) { memset(&addr.sin_addr, 0, sizeof(addr.sin_addr)); grpc_error *error = - grpc_tcp_server_create(&server->shutdown_complete, &server->tcp_server); + grpc_tcp_server_create(&server->shutdown_complete, NULL, &server->tcp_server); GPR_ASSERT(error == GRPC_ERROR_NONE); error = grpc_tcp_server_add_port(server->tcp_server, &addr, sizeof(addr), &port_added); diff --git a/third_party/protobuf b/third_party/protobuf index a1938b2aa9c..d5fb408ddc2 160000 --- a/third_party/protobuf +++ b/third_party/protobuf @@ -1 +1 @@ -Subproject commit a1938b2aa9ca86ce7ce50c27ff9737c1008d2a03 +Subproject commit d5fb408ddc281ffcadeb08699e65bb694656d0bd From 316f9c84702c9db293b97ccb8e40d0b35a827ab7 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 19 May 2016 11:23:17 -0700 Subject: [PATCH 0012/1039] Fix compile --- src/core/lib/surface/server.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/core/lib/surface/server.c b/src/core/lib/surface/server.c index 06005bf12e3..90368f99939 100644 --- a/src/core/lib/surface/server.c +++ b/src/core/lib/surface/server.c @@ -676,8 +676,8 @@ static void kill_pending_work_locked(grpc_exec_ctx *exec_ctx, exec_ctx, &server->unregistered_request_matcher); for (registered_method *rm = server->registered_methods; rm; rm = rm->next) { - request_matcher_kill_requests(exec_ctx, server, &rm->request_matcher); - request_matcher_zombify_all_pending_calls(exec_ctx, &rm->request_matcher, GRPC_ERROR_REF(error)); + request_matcher_kill_requests(exec_ctx, server, &rm->request_matcher, GRPC_ERROR_REF(error)); + request_matcher_zombify_all_pending_calls(exec_ctx, &rm->request_matcher); } } GRPC_ERROR_UNREF(error); From 136698c9ea976c13c05155b6a72e3cb7a6a80545 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 19 May 2016 11:49:44 -0700 Subject: [PATCH 0013/1039] Initialize variables to appease compilers --- src/core/lib/iomgr/tcp_server_posix.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/core/lib/iomgr/tcp_server_posix.c b/src/core/lib/iomgr/tcp_server_posix.c index aac3beb8bd7..b6a0382916d 100644 --- a/src/core/lib/iomgr/tcp_server_posix.c +++ b/src/core/lib/iomgr/tcp_server_posix.c @@ -484,7 +484,8 @@ static grpc_error *clone_port(grpc_tcp_listener *listener, unsigned count) { } for (unsigned i = 0; i < count; i++) { - int fd, port; + int fd = -1; + int port = -1; grpc_dualstack_mode dsmode; err = grpc_create_dualstack_socket(&listener->addr.sockaddr, SOCK_STREAM, 0, &dsmode, &fd); From 13d455e21b7cd0085c657380e527da468d9bde52 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 27 May 2016 12:06:34 -0700 Subject: [PATCH 0014/1039] Initial pass of delayed writing --- .../chttp2/transport/chttp2_transport.c | 144 +++++++++++++----- .../ext/transport/chttp2/transport/internal.h | 31 ++-- .../ext/transport/chttp2/transport/parsing.c | 5 +- .../transport/chttp2/transport/stream_lists.c | 3 +- src/core/lib/iomgr/workqueue.h | 4 + src/core/lib/iomgr/workqueue_posix.c | 6 + 6 files changed, 137 insertions(+), 56 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index be517154e62..b94a112b4a1 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -47,6 +47,7 @@ #include "src/core/ext/transport/chttp2/transport/internal.h" #include "src/core/ext/transport/chttp2/transport/status_conversion.h" #include "src/core/ext/transport/chttp2/transport/timeout_encoding.h" +#include "src/core/lib/iomgr/workqueue.h" #include "src/core/lib/profiling/timers.h" #include "src/core/lib/support/string.h" #include "src/core/lib/transport/static_metadata.h" @@ -87,10 +88,12 @@ static const grpc_transport_vtable vtable; static void writing_action(grpc_exec_ctx *exec_ctx, void *t, grpc_error *error); static void reading_action(grpc_exec_ctx *exec_ctx, void *t, grpc_error *error); static void parsing_action(grpc_exec_ctx *exec_ctx, void *t, grpc_error *error); +static void initiate_writing(grpc_exec_ctx *exec_ctx, void *t, + grpc_error *error); /** Set a transport level setting, and push it to our peer */ -static void push_setting(grpc_chttp2_transport *t, grpc_chttp2_setting_id id, - uint32_t value); +static void push_setting(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, + grpc_chttp2_setting_id id, uint32_t value); /** Start disconnection chain */ static void drop_connection(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, @@ -137,7 +140,7 @@ static void check_read_ops(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global); static void incoming_byte_stream_update_flow_control( - grpc_chttp2_transport_global *transport_global, + grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global, size_t max_size_hint, size_t have_already); static void incoming_byte_stream_destroy_locked(grpc_exec_ctx *exec_ctx, @@ -231,7 +234,7 @@ static void ref_transport(grpc_chttp2_transport *t) { gpr_ref(&t->refs); } static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, const grpc_channel_args *channel_args, - grpc_endpoint *ep, uint8_t is_client) { + grpc_endpoint *ep, bool is_client) { size_t i; int j; @@ -247,6 +250,9 @@ static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, /* ref is dropped at transport close() */ gpr_ref_init(&t->shutdown_ep_refs, 1); gpr_mu_init(&t->executor.mu); + GPR_ASSERT(GRPC_LOG_IF_ERROR( + "workqueue_create", + grpc_workqueue_create(exec_ctx, &t->executor.workqueue))); t->peer_string = grpc_endpoint_get_peer(ep); t->endpoint_reading = 1; t->global.next_stream_id = is_client ? 1 : 2; @@ -272,6 +278,7 @@ static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, grpc_closure_init(&t->writing_action, writing_action, t); grpc_closure_init(&t->reading_action, reading_action, t); grpc_closure_init(&t->parsing_action, parsing_action, t); + grpc_closure_init(&t->initiate_writing, initiate_writing, t); gpr_slice_buffer_init(&t->parsing.qbuf); grpc_chttp2_goaway_parser_init(&t->parsing.goaway_parser); @@ -285,6 +292,7 @@ static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, gpr_slice_buffer_add( &t->global.qbuf, gpr_slice_from_copied_string(GRPC_CHTTP2_CLIENT_CONNECT_STRING)); + grpc_chttp2_initiate_write(exec_ctx, &t->global); } /* 8 is a random stab in the dark as to a good initial size: it's small enough that it shouldn't waste memory for infrequently used connections, yet @@ -310,11 +318,12 @@ static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, /* configure http2 the way we like it */ if (is_client) { - push_setting(t, GRPC_CHTTP2_SETTINGS_ENABLE_PUSH, 0); - push_setting(t, GRPC_CHTTP2_SETTINGS_MAX_CONCURRENT_STREAMS, 0); + push_setting(exec_ctx, t, GRPC_CHTTP2_SETTINGS_ENABLE_PUSH, 0); + push_setting(exec_ctx, t, GRPC_CHTTP2_SETTINGS_MAX_CONCURRENT_STREAMS, 0); } - push_setting(t, GRPC_CHTTP2_SETTINGS_INITIAL_WINDOW_SIZE, DEFAULT_WINDOW); - push_setting(t, GRPC_CHTTP2_SETTINGS_MAX_HEADER_LIST_SIZE, + push_setting(exec_ctx, t, GRPC_CHTTP2_SETTINGS_INITIAL_WINDOW_SIZE, + DEFAULT_WINDOW); + push_setting(exec_ctx, t, GRPC_CHTTP2_SETTINGS_MAX_HEADER_LIST_SIZE, DEFAULT_MAX_HEADER_LIST_SIZE); if (channel_args) { @@ -328,7 +337,7 @@ static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, gpr_log(GPR_ERROR, "%s: must be an integer", GRPC_ARG_MAX_CONCURRENT_STREAMS); } else { - push_setting(t, GRPC_CHTTP2_SETTINGS_MAX_CONCURRENT_STREAMS, + push_setting(exec_ctx, t, GRPC_CHTTP2_SETTINGS_MAX_CONCURRENT_STREAMS, (uint32_t)channel_args->args[i].value.integer); } } else if (0 == strcmp(channel_args->args[i].key, @@ -367,7 +376,7 @@ static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, gpr_log(GPR_ERROR, "%s: must be non-negative", GRPC_ARG_HTTP2_HPACK_TABLE_SIZE_DECODER); } else { - push_setting(t, GRPC_CHTTP2_SETTINGS_HEADER_TABLE_SIZE, + push_setting(exec_ctx, t, GRPC_CHTTP2_SETTINGS_HEADER_TABLE_SIZE, (uint32_t)channel_args->args[i].value.integer); } } else if (0 == strcmp(channel_args->args[i].key, @@ -392,7 +401,7 @@ static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, gpr_log(GPR_ERROR, "%s: must be non-negative", GRPC_ARG_MAX_METADATA_SIZE); } else { - push_setting(t, GRPC_CHTTP2_SETTINGS_MAX_HEADER_LIST_SIZE, + push_setting(exec_ctx, t, GRPC_CHTTP2_SETTINGS_MAX_HEADER_LIST_SIZE, (uint32_t)channel_args->args[i].value.integer); } } @@ -637,14 +646,6 @@ static void finish_global_actions(grpc_exec_ctx *exec_ctx, grpc_chttp2_executor_action_header *next; for (;;) { - if (!t->executor.writing_active && !t->closed && - grpc_chttp2_unlocking_check_writes(exec_ctx, &t->global, &t->writing, - t->executor.parsing_active)) { - t->executor.writing_active = 1; - REF_TRANSPORT(t, "writing"); - prevent_endpoint_shutdown(t); - grpc_exec_ctx_sched(exec_ctx, &t->writing_action, GRPC_ERROR_NONE, NULL); - } check_read_ops(exec_ctx, &t->global); gpr_mu_lock(&t->executor.mu); @@ -727,16 +728,52 @@ void grpc_chttp2_run_with_global_lock(grpc_exec_ctx *exec_ctx, * OUTPUT PROCESSING */ -void grpc_chttp2_become_writable(grpc_chttp2_transport_global *transport_global, +void grpc_chttp2_initiate_write( + grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global) { + grpc_chttp2_transport *t = TRANSPORT_FROM_GLOBAL(transport_global); + t->executor.writing_needed = true; + if (!t->executor.writing_initiated) { + t->executor.writing_initiated = true; + grpc_workqueue_enqueue(exec_ctx, t->executor.workqueue, + &t->initiate_writing, GRPC_ERROR_NONE); + } +} + +static void initiate_writing_locked(grpc_exec_ctx *exec_ctx, + grpc_chttp2_transport *t, + grpc_chttp2_stream *s_unused, + void *arg_ignored) { + GPR_ASSERT(t->executor.writing_needed); + GPR_ASSERT(!t->executor.writing_active); + if (!t->closed && + grpc_chttp2_unlocking_check_writes(exec_ctx, &t->global, &t->writing, + t->executor.parsing_active)) { + t->executor.writing_needed = false; + t->executor.writing_active = true; + REF_TRANSPORT(t, "writing"); + prevent_endpoint_shutdown(t); + grpc_exec_ctx_sched(exec_ctx, &t->writing_action, GRPC_ERROR_NONE, NULL); + } +} + +static void initiate_writing(grpc_exec_ctx *exec_ctx, void *arg, + grpc_error *error) { + grpc_chttp2_run_with_global_lock(exec_ctx, arg, NULL, initiate_writing_locked, + NULL, 0); +} + +void grpc_chttp2_become_writable(grpc_exec_ctx *exec_ctx, + grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global) { if (!TRANSPORT_FROM_GLOBAL(transport_global)->closed && grpc_chttp2_list_add_writable_stream(transport_global, stream_global)) { GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing"); + grpc_chttp2_initiate_write(exec_ctx, transport_global); } } -static void push_setting(grpc_chttp2_transport *t, grpc_chttp2_setting_id id, - uint32_t value) { +static void push_setting(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, + grpc_chttp2_setting_id id, uint32_t value) { const grpc_chttp2_setting_parameters *sp = &grpc_chttp2_settings_parameters[id]; uint32_t use_value = GPR_CLAMP(value, sp->min_value, sp->max_value); @@ -747,6 +784,7 @@ static void push_setting(grpc_chttp2_transport *t, grpc_chttp2_setting_id id, if (use_value != t->global.settings[GRPC_LOCAL_SETTINGS][id]) { t->global.settings[GRPC_LOCAL_SETTINGS][id] = use_value; t->global.dirtied_local_settings = 1; + grpc_chttp2_initiate_write(exec_ctx, &t->global); } } @@ -772,10 +810,12 @@ static void terminate_writing_with_lock(grpc_exec_ctx *exec_ctx, GRPC_CHTTP2_STREAM_UNREF(exec_ctx, stream_global, "finish_writes"); } - /* leave the writing flag up on shutdown to prevent further writes in - unlock() - from starting */ - t->executor.writing_active = 0; + t->executor.writing_active = false; + if (t->executor.writing_needed) { + initiate_writing_locked(exec_ctx, t, NULL, NULL); + } else { + t->executor.writing_initiated = false; + } if (t->ep && !t->endpoint_reading) { destroy_endpoint(exec_ctx, t); } @@ -862,7 +902,7 @@ static void maybe_start_some_streams( stream_global->id, STREAM_FROM_GLOBAL(stream_global)); stream_global->in_stream_map = true; transport_global->concurrent_stream_count++; - grpc_chttp2_become_writable(transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); } /* cancel out streams that will never be started */ while (transport_global->next_stream_id >= MAX_CLIENT_STREAM_ID && @@ -991,7 +1031,8 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, maybe_start_some_streams(exec_ctx, transport_global); } else { GPR_ASSERT(stream_global->id != 0); - grpc_chttp2_become_writable(transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, + stream_global); } } else { grpc_chttp2_complete_closure_step( @@ -1015,7 +1056,7 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, } else { stream_global->send_message = op->send_message; if (stream_global->id != 0) { - grpc_chttp2_become_writable(transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); } } } @@ -1054,7 +1095,7 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, } else if (stream_global->id != 0) { /* TODO(ctiller): check if there's flow control for any outstanding bytes before going writable */ - grpc_chttp2_become_writable(transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); } } } @@ -1075,8 +1116,8 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, (stream_global->incoming_frames.head == NULL || stream_global->incoming_frames.head->is_tail)) { incoming_byte_stream_update_flow_control( - transport_global, stream_global, transport_global->stream_lookahead, - 0); + exec_ctx, transport_global, stream_global, + transport_global->stream_lookahead, 0); } grpc_chttp2_list_add_check_read_ops(transport_global, stream_global); } @@ -1103,7 +1144,8 @@ static void perform_stream_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt, sizeof(*op)); } -static void send_ping_locked(grpc_chttp2_transport *t, grpc_closure *on_recv) { +static void send_ping_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, + grpc_closure *on_recv) { grpc_chttp2_outstanding_ping *p = gpr_malloc(sizeof(*p)); p->next = &t->global.pings; p->prev = p->next->prev; @@ -1118,6 +1160,7 @@ static void send_ping_locked(grpc_chttp2_transport *t, grpc_closure *on_recv) { p->id[7] = (uint8_t)(t->global.ping_counter & 0xff); p->on_recv = on_recv; gpr_slice_buffer_add(&t->global.qbuf, grpc_chttp2_ping_create(0, p->id)); + grpc_chttp2_initiate_write(exec_ctx, &t->global); } static void ack_ping_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, @@ -1177,6 +1220,7 @@ static void perform_transport_op_locked(grpc_exec_ctx *exec_ctx, close_transport = grpc_chttp2_has_streams(t) ? GRPC_ERROR_NONE : GRPC_ERROR_CREATE("GOAWAY sent"); + grpc_chttp2_initiate_write(exec_ctx, &t->global); } if (op->set_accept_stream) { @@ -1194,7 +1238,7 @@ static void perform_transport_op_locked(grpc_exec_ctx *exec_ctx, } if (op->send_ping) { - send_ping_locked(t, op->send_ping); + send_ping_locked(exec_ctx, t, op->send_ping); } if (close_transport != GRPC_ERROR_NONE) { @@ -1343,6 +1387,7 @@ static void cancel_from_api(grpc_exec_ctx *exec_ctx, stream_global->id, (uint32_t)grpc_chttp2_grpc_status_to_http2_error(status), &stream_global->stats.outgoing)); + grpc_chttp2_initiate_write(exec_ctx, transport_global); } grpc_chttp2_fake_status(exec_ctx, transport_global, stream_global, status, NULL); @@ -1564,6 +1609,7 @@ static void close_from_api(grpc_exec_ctx *exec_ctx, } grpc_chttp2_mark_stream_closed(exec_ctx, transport_global, stream_global, 1, 1, err); + grpc_chttp2_initiate_write(exec_ctx, transport_global); } static void cancel_stream_cb(grpc_chttp2_transport_global *transport_global, @@ -1585,8 +1631,14 @@ static void drop_connection(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, } /** update window from a settings change */ +typedef struct { + grpc_chttp2_transport *t; + grpc_exec_ctx *exec_ctx; +} update_global_window_args; + static void update_global_window(void *args, uint32_t id, void *stream) { - grpc_chttp2_transport *t = args; + update_global_window_args *a = args; + grpc_chttp2_transport *t = a->t; grpc_chttp2_stream *s = stream; grpc_chttp2_transport_global *transport_global = &t->global; grpc_chttp2_stream_global *stream_global = &s->global; @@ -1600,7 +1652,7 @@ static void update_global_window(void *args, uint32_t id, void *stream) { is_zero = stream_global->outgoing_window <= 0; if (was_zero && !is_zero) { - grpc_chttp2_become_writable(transport_global, stream_global); + grpc_chttp2_become_writable(a->exec_ctx, transport_global, stream_global); } } @@ -1677,14 +1729,18 @@ static void post_parse_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, grpc_chttp2_transport_global *transport_global = &t->global; grpc_chttp2_transport_parsing *transport_parsing = &t->parsing; /* copy parsing qbuf to global qbuf */ - gpr_slice_buffer_move_into(&t->parsing.qbuf, &t->global.qbuf); + if (t->parsing.qbuf.count > 0) { + gpr_slice_buffer_move_into(&t->parsing.qbuf, &t->global.qbuf); + grpc_chttp2_initiate_write(exec_ctx, transport_global); + } /* merge stream lists */ grpc_chttp2_stream_map_move_into(&t->new_stream_map, &t->parsing_stream_map); transport_global->concurrent_stream_count = (uint32_t)grpc_chttp2_stream_map_size(&t->parsing_stream_map); if (transport_parsing->initial_window_update != 0) { + update_global_window_args args = {t, exec_ctx}; grpc_chttp2_stream_map_for_each(&t->parsing_stream_map, - update_global_window, t); + update_global_window, &args); transport_parsing->initial_window_update = 0; } /* handle higher level things */ @@ -1774,6 +1830,7 @@ static void add_to_pollset_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_stream *s_unused, void *pollset) { if (t->ep) { grpc_endpoint_add_to_pollset(exec_ctx, t->ep, pollset); + grpc_workqueue_add_to_pollset(exec_ctx, t->executor.workqueue, pollset); } } @@ -1783,6 +1840,8 @@ static void add_to_pollset_set_locked(grpc_exec_ctx *exec_ctx, void *pollset_set) { if (t->ep) { grpc_endpoint_add_to_pollset_set(exec_ctx, t->ep, pollset_set); + grpc_workqueue_add_to_pollset_set(exec_ctx, t->executor.workqueue, + pollset_set); } } @@ -1808,7 +1867,7 @@ static void incoming_byte_stream_unref(grpc_exec_ctx *exec_ctx, } static void incoming_byte_stream_update_flow_control( - grpc_chttp2_transport_global *transport_global, + grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global, size_t max_size_hint, size_t have_already) { uint32_t max_recv_bytes; @@ -1843,7 +1902,7 @@ static void incoming_byte_stream_update_flow_control( add_max_recv_bytes); grpc_chttp2_list_add_unannounced_incoming_window_available(transport_global, stream_global); - grpc_chttp2_become_writable(transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); } } @@ -1865,8 +1924,9 @@ static void incoming_byte_stream_next_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_stream_global *stream_global = &bs->stream->global; if (bs->is_tail) { - incoming_byte_stream_update_flow_control( - transport_global, stream_global, arg->max_size_hint, bs->slices.length); + incoming_byte_stream_update_flow_control(exec_ctx, transport_global, + stream_global, arg->max_size_hint, + bs->slices.length); } if (bs->slices.count > 0) { *arg->slice = gpr_slice_buffer_take_first(&bs->slices); diff --git a/src/core/ext/transport/chttp2/transport/internal.h b/src/core/ext/transport/chttp2/transport/internal.h index 7e5d58380e7..bdc4a69f4fd 100644 --- a/src/core/ext/transport/chttp2/transport/internal.h +++ b/src/core/ext/transport/chttp2/transport/internal.h @@ -315,11 +315,16 @@ struct grpc_chttp2_transport { struct { gpr_mu mu; + grpc_workqueue *workqueue; /** is a thread currently in the global lock */ bool global_active; - /** is a thread currently writing */ + /** is a write currently initiated */ + bool writing_initiated; + /** is a write actually going on right now */ bool writing_active; + /** is a write needed */ + bool writing_needed; /** is a thread currently parsing */ bool parsing_active; @@ -362,6 +367,8 @@ struct grpc_chttp2_transport { grpc_closure reading_action; /** closure to actually do parsing */ grpc_closure parsing_action; + /** closure to initiate writing */ + grpc_closure initiate_writing; /** incoming read bytes */ gpr_slice_buffer read_buffer; @@ -507,15 +514,16 @@ struct grpc_chttp2_stream { }; /** Transport writing call flow: - chttp2_transport.c calls grpc_chttp2_unlocking_check_writes to see if writes - are required; - if they are, chttp2_transport.c calls grpc_chttp2_perform_writes to do the - writes. - Once writes have been completed (meaning another write could potentially be - started), - grpc_chttp2_terminate_writing is called. This will call - grpc_chttp2_cleanup_writing, at which - point the write phase is complete. */ + grpc_chttp2_initiate_write() is called anywhere that we know bytes need to + go out on the wire. + If no other write has been started, a task is enqueued onto our workqueue. + When that task executes, it obtains the global lock, and gathers the data + to write. + The global lock is dropped and we do the syscall to write. + After writing, a follow-up check is made to see if another round of writing + should be performed. */ +void grpc_chttp2_initiate_write(grpc_exec_ctx *exec_ctx, + grpc_chttp2_transport_global *transport_global); /** Someone is unlocking the transport mutex: check to see if writes are required, and schedule them if so */ @@ -816,7 +824,8 @@ void grpc_chttp2_ack_ping(grpc_exec_ctx *exec_ctx, /** add a ref to the stream and add it to the writable list; ref will be dropped in writing.c */ -void grpc_chttp2_become_writable(grpc_chttp2_transport_global *transport_global, +void grpc_chttp2_become_writable(grpc_exec_ctx *exec_ctx, + grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global); #endif /* GRPC_CORE_EXT_TRANSPORT_CHTTP2_TRANSPORT_INTERNAL_H */ diff --git a/src/core/ext/transport/chttp2/transport/parsing.c b/src/core/ext/transport/chttp2/transport/parsing.c index 72b3131d7b3..3d3abccd204 100644 --- a/src/core/ext/transport/chttp2/transport/parsing.c +++ b/src/core/ext/transport/chttp2/transport/parsing.c @@ -156,7 +156,7 @@ void grpc_chttp2_publish_reads( if (was_zero && !is_zero) { while (grpc_chttp2_list_pop_stalled_by_transport(transport_global, &stream_global)) { - grpc_chttp2_become_writable(transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); } } @@ -168,6 +168,7 @@ void grpc_chttp2_publish_reads( announce_incoming_window, announce_bytes); GRPC_CHTTP2_FLOW_CREDIT_TRANSPORT("parsed", transport_parsing, incoming_window, announce_bytes); + grpc_chttp2_initiate_write(exec_ctx, transport_global); } /* for each stream that saw an update, fixup global state */ @@ -190,7 +191,7 @@ void grpc_chttp2_publish_reads( outgoing_window); is_zero = stream_global->outgoing_window <= 0; if (was_zero && !is_zero) { - grpc_chttp2_become_writable(transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); } stream_global->max_recv_bytes -= (uint32_t)GPR_MIN( diff --git a/src/core/ext/transport/chttp2/transport/stream_lists.c b/src/core/ext/transport/chttp2/transport/stream_lists.c index 8f3ab00e6df..f3690015eaa 100644 --- a/src/core/ext/transport/chttp2/transport/stream_lists.c +++ b/src/core/ext/transport/chttp2/transport/stream_lists.c @@ -344,7 +344,8 @@ void grpc_chttp2_list_flush_writing_stalled_by_transport( while (stream_list_pop(transport, &stream, GRPC_CHTTP2_LIST_WRITING_STALLED_BY_TRANSPORT)) { if (is_window_available) { - grpc_chttp2_become_writable(&transport->global, &stream->global); + grpc_chttp2_become_writable(exec_ctx, &transport->global, + &stream->global); } else { grpc_chttp2_list_add_stalled_by_transport(transport_writing, &stream->writing); diff --git a/src/core/lib/iomgr/workqueue.h b/src/core/lib/iomgr/workqueue.h index 7b6402e3a42..88e4211c471 100644 --- a/src/core/lib/iomgr/workqueue.h +++ b/src/core/lib/iomgr/workqueue.h @@ -38,6 +38,7 @@ #include "src/core/lib/iomgr/exec_ctx.h" #include "src/core/lib/iomgr/iomgr.h" #include "src/core/lib/iomgr/pollset.h" +#include "src/core/lib/iomgr/pollset_set.h" #ifdef GPR_POSIX_SOCKET #include "src/core/lib/iomgr/workqueue_posix.h" @@ -76,6 +77,9 @@ void grpc_workqueue_unref(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue); void grpc_workqueue_add_to_pollset(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, grpc_pollset *pollset); +void grpc_workqueue_add_to_pollset_set(grpc_exec_ctx *exec_ctx, + grpc_workqueue *workqueue, + grpc_pollset_set *pollset_set); /** Add a work item to a workqueue */ void grpc_workqueue_enqueue(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, diff --git a/src/core/lib/iomgr/workqueue_posix.c b/src/core/lib/iomgr/workqueue_posix.c index 45e0f6063b4..c6323e0594d 100644 --- a/src/core/lib/iomgr/workqueue_posix.c +++ b/src/core/lib/iomgr/workqueue_posix.c @@ -106,6 +106,12 @@ void grpc_workqueue_add_to_pollset(grpc_exec_ctx *exec_ctx, grpc_pollset_add_fd(exec_ctx, pollset, workqueue->wakeup_read_fd); } +void grpc_workqueue_add_to_pollset_set(grpc_exec_ctx *exec_ctx, + grpc_workqueue *workqueue, + grpc_pollset_set *pollset_set) { + grpc_pollset_set_add_fd(exec_ctx, pollset_set, workqueue->wakeup_read_fd); +} + void grpc_workqueue_flush(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue) { gpr_mu_lock(&workqueue->mu); grpc_exec_ctx_enqueue_list(exec_ctx, &workqueue->closure_list, NULL); From e48b1bc011e2b5783ce75f4122dfd5aba01f0d97 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 11 May 2016 15:17:22 -0700 Subject: [PATCH 0015/1039] Base changes. Create ev_epoll_posix.{c,h} files by making a copy of ev_poll_and_epoll.c file --- BUILD | 6 + Makefile | 2 + binding.gyp | 1 + build.yaml | 2 + config.m4 | 1 + gRPC.podspec | 3 + grpc.gemspec | 2 + package.xml | 2 + src/core/lib/iomgr/ev_epoll_posix.c | 1733 +++++++++++++++++ src/core/lib/iomgr/ev_epoll_posix.h | 41 + src/core/lib/iomgr/ev_posix.c | 3 +- src/python/grpcio/grpc_core_dependencies.py | 1 + tools/doxygen/Doxyfile.core.internal | 2 + tools/run_tests/sources_and_headers.json | 3 + vsprojects/vcxproj/grpc/grpc.vcxproj | 3 + vsprojects/vcxproj/grpc/grpc.vcxproj.filters | 6 + .../grpc_unsecure/grpc_unsecure.vcxproj | 3 + .../grpc_unsecure.vcxproj.filters | 6 + 18 files changed, 1819 insertions(+), 1 deletion(-) create mode 100644 src/core/lib/iomgr/ev_epoll_posix.c create mode 100644 src/core/lib/iomgr/ev_epoll_posix.h diff --git a/BUILD b/BUILD index fd03d52e3cb..0be8f27a01b 100644 --- a/BUILD +++ b/BUILD @@ -178,6 +178,7 @@ cc_library( "src/core/lib/iomgr/closure.h", "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", + "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", "src/core/lib/iomgr/exec_ctx.h", @@ -321,6 +322,7 @@ cc_library( "src/core/lib/iomgr/endpoint.c", "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", + "src/core/lib/iomgr/ev_epoll_posix.c", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_posix.c", "src/core/lib/iomgr/exec_ctx.c", @@ -546,6 +548,7 @@ cc_library( "src/core/lib/iomgr/closure.h", "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", + "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", "src/core/lib/iomgr/exec_ctx.h", @@ -666,6 +669,7 @@ cc_library( "src/core/lib/iomgr/endpoint.c", "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", + "src/core/lib/iomgr/ev_epoll_posix.c", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_posix.c", "src/core/lib/iomgr/exec_ctx.c", @@ -1358,6 +1362,7 @@ objc_library( "src/core/lib/iomgr/endpoint.c", "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", + "src/core/lib/iomgr/ev_epoll_posix.c", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_posix.c", "src/core/lib/iomgr/exec_ctx.c", @@ -1562,6 +1567,7 @@ objc_library( "src/core/lib/iomgr/closure.h", "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", + "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", "src/core/lib/iomgr/exec_ctx.h", diff --git a/Makefile b/Makefile index 817fcd072d7..29ebc0e5adf 100644 --- a/Makefile +++ b/Makefile @@ -2486,6 +2486,7 @@ LIBGRPC_SRC = \ src/core/lib/iomgr/endpoint.c \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ + src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ src/core/lib/iomgr/exec_ctx.c \ @@ -2840,6 +2841,7 @@ LIBGRPC_UNSECURE_SRC = \ src/core/lib/iomgr/endpoint.c \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ + src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ src/core/lib/iomgr/exec_ctx.c \ diff --git a/binding.gyp b/binding.gyp index 7b187070005..89774ead4da 100644 --- a/binding.gyp +++ b/binding.gyp @@ -581,6 +581,7 @@ 'src/core/lib/iomgr/endpoint.c', 'src/core/lib/iomgr/endpoint_pair_posix.c', 'src/core/lib/iomgr/endpoint_pair_windows.c', + 'src/core/lib/iomgr/ev_epoll_posix.c', 'src/core/lib/iomgr/ev_poll_posix.c', 'src/core/lib/iomgr/ev_posix.c', 'src/core/lib/iomgr/exec_ctx.c', diff --git a/build.yaml b/build.yaml index 429dbb3351a..7ba65332972 100644 --- a/build.yaml +++ b/build.yaml @@ -165,6 +165,7 @@ filegroups: - src/core/lib/iomgr/closure.h - src/core/lib/iomgr/endpoint.h - src/core/lib/iomgr/endpoint_pair.h + - src/core/lib/iomgr/ev_epoll_posix.h - src/core/lib/iomgr/ev_poll_posix.h - src/core/lib/iomgr/ev_posix.h - src/core/lib/iomgr/exec_ctx.h @@ -239,6 +240,7 @@ filegroups: - src/core/lib/iomgr/endpoint.c - src/core/lib/iomgr/endpoint_pair_posix.c - src/core/lib/iomgr/endpoint_pair_windows.c + - src/core/lib/iomgr/ev_epoll_posix.c - src/core/lib/iomgr/ev_poll_posix.c - src/core/lib/iomgr/ev_posix.c - src/core/lib/iomgr/exec_ctx.c diff --git a/config.m4 b/config.m4 index 29df77ce1db..6987c741541 100644 --- a/config.m4 +++ b/config.m4 @@ -100,6 +100,7 @@ if test "$PHP_GRPC" != "no"; then src/core/lib/iomgr/endpoint.c \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ + src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ src/core/lib/iomgr/exec_ctx.c \ diff --git a/gRPC.podspec b/gRPC.podspec index cc8ba3de94e..3b4dd52380e 100644 --- a/gRPC.podspec +++ b/gRPC.podspec @@ -181,6 +181,7 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/closure.h', 'src/core/lib/iomgr/endpoint.h', 'src/core/lib/iomgr/endpoint_pair.h', + 'src/core/lib/iomgr/ev_epoll_posix.h', 'src/core/lib/iomgr/ev_poll_posix.h', 'src/core/lib/iomgr/ev_posix.h', 'src/core/lib/iomgr/exec_ctx.h', @@ -358,6 +359,7 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/endpoint.c', 'src/core/lib/iomgr/endpoint_pair_posix.c', 'src/core/lib/iomgr/endpoint_pair_windows.c', + 'src/core/lib/iomgr/ev_epoll_posix.c', 'src/core/lib/iomgr/ev_poll_posix.c', 'src/core/lib/iomgr/ev_posix.c', 'src/core/lib/iomgr/exec_ctx.c', @@ -546,6 +548,7 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/closure.h', 'src/core/lib/iomgr/endpoint.h', 'src/core/lib/iomgr/endpoint_pair.h', + 'src/core/lib/iomgr/ev_epoll_posix.h', 'src/core/lib/iomgr/ev_poll_posix.h', 'src/core/lib/iomgr/ev_posix.h', 'src/core/lib/iomgr/exec_ctx.h', diff --git a/grpc.gemspec b/grpc.gemspec index ae7f9b7d2ee..71cccb6ca8d 100755 --- a/grpc.gemspec +++ b/grpc.gemspec @@ -190,6 +190,7 @@ Gem::Specification.new do |s| s.files += %w( src/core/lib/iomgr/closure.h ) s.files += %w( src/core/lib/iomgr/endpoint.h ) s.files += %w( src/core/lib/iomgr/endpoint_pair.h ) + s.files += %w( src/core/lib/iomgr/ev_epoll_posix.h ) s.files += %w( src/core/lib/iomgr/ev_poll_posix.h ) s.files += %w( src/core/lib/iomgr/ev_posix.h ) s.files += %w( src/core/lib/iomgr/exec_ctx.h ) @@ -337,6 +338,7 @@ Gem::Specification.new do |s| s.files += %w( src/core/lib/iomgr/endpoint.c ) s.files += %w( src/core/lib/iomgr/endpoint_pair_posix.c ) s.files += %w( src/core/lib/iomgr/endpoint_pair_windows.c ) + s.files += %w( src/core/lib/iomgr/ev_epoll_posix.c ) s.files += %w( src/core/lib/iomgr/ev_poll_posix.c ) s.files += %w( src/core/lib/iomgr/ev_posix.c ) s.files += %w( src/core/lib/iomgr/exec_ctx.c ) diff --git a/package.xml b/package.xml index 507a2a7ed6c..0fc5d0dee47 100644 --- a/package.xml +++ b/package.xml @@ -197,6 +197,7 @@ + @@ -344,6 +345,7 @@ + diff --git a/src/core/lib/iomgr/ev_epoll_posix.c b/src/core/lib/iomgr/ev_epoll_posix.c new file mode 100644 index 00000000000..ce8d3981b3f --- /dev/null +++ b/src/core/lib/iomgr/ev_epoll_posix.c @@ -0,0 +1,1733 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include + +#ifdef GPR_POSIX_SOCKET + +#include "src/core/lib/iomgr/ev_epoll_posix.h" + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include "src/core/lib/iomgr/iomgr_internal.h" +#include "src/core/lib/iomgr/wakeup_fd_posix.h" +#include "src/core/lib/profiling/timers.h" +#include "src/core/lib/support/block_annotate.h" + +/******************************************************************************* + * FD declarations + */ + +/* TODO(sreek) : Check if grpc_fd_watcher is needed (and if so, check if we can + * share this between ev_poll_posix.h and ev_epoll_posix versions */ + +typedef struct grpc_fd_watcher { + struct grpc_fd_watcher *next; + struct grpc_fd_watcher *prev; + grpc_pollset *pollset; + grpc_pollset_worker *worker; + grpc_fd *fd; +} grpc_fd_watcher; + +struct grpc_fd { + int fd; + /* refst format: + bit0: 1=active/0=orphaned + bit1-n: refcount + meaning that mostly we ref by two to avoid altering the orphaned bit, + and just unref by 1 when we're ready to flag the object as orphaned */ + gpr_atm refst; + + gpr_mu mu; + int shutdown; + int closed; + int released; + + /* The watcher list. + + The following watcher related fields are protected by watcher_mu. + + An fd_watcher is an ephemeral object created when an fd wants to + begin polling, and destroyed after the poll. + + It denotes the fd's interest in whether to read poll or write poll + or both or neither on this fd. + + If a watcher is asked to poll for reads or writes, the read_watcher + or write_watcher fields are set respectively. A watcher may be asked + to poll for both, in which case both fields will be set. + + read_watcher and write_watcher may be NULL if no watcher has been + asked to poll for reads or writes. + + If an fd_watcher is not asked to poll for reads or writes, it's added + to a linked list of inactive watchers, rooted at inactive_watcher_root. + If at a later time there becomes need of a poller to poll, one of + the inactive pollers may be kicked out of their poll loops to take + that responsibility. */ + grpc_fd_watcher inactive_watcher_root; + grpc_fd_watcher *read_watcher; + grpc_fd_watcher *write_watcher; + + grpc_closure *read_closure; + grpc_closure *write_closure; + + struct grpc_fd *freelist_next; + + grpc_closure *on_done_closure; + + grpc_iomgr_object iomgr_object; +}; + +/* Begin polling on an fd. + Registers that the given pollset is interested in this fd - so that if read + or writability interest changes, the pollset can be kicked to pick up that + new interest. + Return value is: + (fd_needs_read? read_mask : 0) | (fd_needs_write? write_mask : 0) + i.e. a combination of read_mask and write_mask determined by the fd's current + interest in said events. + Polling strategies that do not need to alter their behavior depending on the + fd's current interest (such as epoll) do not need to call this function. + MUST NOT be called with a pollset lock taken */ +static uint32_t fd_begin_poll(grpc_fd *fd, grpc_pollset *pollset, + grpc_pollset_worker *worker, uint32_t read_mask, + uint32_t write_mask, grpc_fd_watcher *rec); +/* Complete polling previously started with fd_begin_poll + MUST NOT be called with a pollset lock taken + if got_read or got_write are 1, also does the become_{readable,writable} as + appropriate. */ +static void fd_end_poll(grpc_exec_ctx *exec_ctx, grpc_fd_watcher *rec, + int got_read, int got_write); + +/* Return 1 if this fd is orphaned, 0 otherwise */ +static bool fd_is_orphaned(grpc_fd *fd); + +/* Reference counting for fds */ +/*#define GRPC_FD_REF_COUNT_DEBUG*/ +#ifdef GRPC_FD_REF_COUNT_DEBUG +static void fd_ref(grpc_fd *fd, const char *reason, const char *file, int line); +static void fd_unref(grpc_fd *fd, const char *reason, const char *file, + int line); +#define GRPC_FD_REF(fd, reason) fd_ref(fd, reason, __FILE__, __LINE__) +#define GRPC_FD_UNREF(fd, reason) fd_unref(fd, reason, __FILE__, __LINE__) +#else +static void fd_ref(grpc_fd *fd); +static void fd_unref(grpc_fd *fd); +#define GRPC_FD_REF(fd, reason) fd_ref(fd) +#define GRPC_FD_UNREF(fd, reason) fd_unref(fd) +#endif + +static void fd_global_init(void); +static void fd_global_shutdown(void); + +#define CLOSURE_NOT_READY ((grpc_closure *)0) +#define CLOSURE_READY ((grpc_closure *)1) + +/******************************************************************************* + * pollset declarations + */ + +typedef struct grpc_pollset_vtable grpc_pollset_vtable; + +typedef struct grpc_cached_wakeup_fd { + grpc_wakeup_fd fd; + struct grpc_cached_wakeup_fd *next; +} grpc_cached_wakeup_fd; + +struct grpc_pollset_worker { + grpc_cached_wakeup_fd *wakeup_fd; + int reevaluate_polling_on_wakeup; + int kicked_specifically; + struct grpc_pollset_worker *next; + struct grpc_pollset_worker *prev; +}; + +struct grpc_pollset { + /* pollsets under posix can mutate representation as fds are added and + removed. + For example, we may choose a poll() based implementation on linux for + few fds, and an epoll() based implementation for many fds */ + const grpc_pollset_vtable *vtable; + gpr_mu mu; + grpc_pollset_worker root_worker; + int in_flight_cbs; + int shutting_down; + int called_shutdown; + int kicked_without_pollers; + grpc_closure *shutdown_done; + grpc_closure_list idle_jobs; + union { + int fd; + void *ptr; + } data; + /* Local cache of eventfds for workers */ + grpc_cached_wakeup_fd *local_wakeup_cache; +}; + +struct grpc_pollset_vtable { + void (*add_fd)(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + struct grpc_fd *fd, int and_unlock_pollset); + void (*maybe_work_and_unlock)(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_pollset_worker *worker, + gpr_timespec deadline, gpr_timespec now); + void (*finish_shutdown)(grpc_pollset *pollset); + void (*destroy)(grpc_pollset *pollset); +}; + +/* Add an fd to a pollset */ +static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + struct grpc_fd *fd); + +static void pollset_set_add_fd(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, grpc_fd *fd); + +/* Convert a timespec to milliseconds: + - very small or negative poll times are clamped to zero to do a + non-blocking poll (which becomes spin polling) + - other small values are rounded up to one millisecond + - longer than a millisecond polls are rounded up to the next nearest + millisecond to avoid spinning + - infinite timeouts are converted to -1 */ +static int poll_deadline_to_millis_timeout(gpr_timespec deadline, + gpr_timespec now); + +/* Allow kick to wakeup the currently polling worker */ +#define GRPC_POLLSET_CAN_KICK_SELF 1 +/* Force the wakee to repoll when awoken */ +#define GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP 2 +/* As per pollset_kick, with an extended set of flags (defined above) + -- mostly for fd_posix's use. */ +static void pollset_kick_ext(grpc_pollset *p, + grpc_pollset_worker *specific_worker, + uint32_t flags); + +/* turn a pollset into a multipoller: platform specific */ +typedef void (*platform_become_multipoller_type)(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset, + struct grpc_fd **fds, + size_t fd_count); +static platform_become_multipoller_type platform_become_multipoller; + + +/* Return 1 if the pollset has active threads in pollset_work (pollset must + * be locked) */ +static int pollset_has_workers(grpc_pollset *pollset); + +static void remove_fd_from_all_epoll_sets(int fd); + +/******************************************************************************* + * pollset_set definitions + */ + +struct grpc_pollset_set { + gpr_mu mu; + + size_t pollset_count; + size_t pollset_capacity; + grpc_pollset **pollsets; + + size_t pollset_set_count; + size_t pollset_set_capacity; + struct grpc_pollset_set **pollset_sets; + + size_t fd_count; + size_t fd_capacity; + grpc_fd **fds; +}; + +/******************************************************************************* + * fd_posix.c + */ + +/* We need to keep a freelist not because of any concerns of malloc performance + * but instead so that implementations with multiple threads in (for example) + * epoll_wait deal with the race between pollset removal and incoming poll + * notifications. + * + * The problem is that the poller ultimately holds a reference to this + * object, so it is very difficult to know when is safe to free it, at least + * without some expensive synchronization. + * + * If we keep the object freelisted, in the worst case losing this race just + * becomes a spurious read notification on a reused fd. + */ +/* TODO(klempner): We could use some form of polling generation count to know + * when these are safe to free. */ +/* TODO(klempner): Consider disabling freelisting if we don't have multiple + * threads in poll on the same fd */ +/* TODO(klempner): Batch these allocations to reduce fragmentation */ +static grpc_fd *fd_freelist = NULL; +static gpr_mu fd_freelist_mu; + +static void freelist_fd(grpc_fd *fd) { + gpr_mu_lock(&fd_freelist_mu); + fd->freelist_next = fd_freelist; + fd_freelist = fd; + grpc_iomgr_unregister_object(&fd->iomgr_object); + gpr_mu_unlock(&fd_freelist_mu); +} + +static grpc_fd *alloc_fd(int fd) { + grpc_fd *r = NULL; + gpr_mu_lock(&fd_freelist_mu); + if (fd_freelist != NULL) { + r = fd_freelist; + fd_freelist = fd_freelist->freelist_next; + } + gpr_mu_unlock(&fd_freelist_mu); + if (r == NULL) { + r = gpr_malloc(sizeof(grpc_fd)); + gpr_mu_init(&r->mu); + } + + gpr_mu_lock(&r->mu); + gpr_atm_rel_store(&r->refst, 1); + r->shutdown = 0; + r->read_closure = CLOSURE_NOT_READY; + r->write_closure = CLOSURE_NOT_READY; + r->fd = fd; + r->inactive_watcher_root.next = r->inactive_watcher_root.prev = + &r->inactive_watcher_root; + r->freelist_next = NULL; + r->read_watcher = r->write_watcher = NULL; + r->on_done_closure = NULL; + r->closed = 0; + r->released = 0; + gpr_mu_unlock(&r->mu); + return r; +} + +static void destroy(grpc_fd *fd) { + gpr_mu_destroy(&fd->mu); + gpr_free(fd); +} + +#ifdef GRPC_FD_REF_COUNT_DEBUG +#define REF_BY(fd, n, reason) ref_by(fd, n, reason, __FILE__, __LINE__) +#define UNREF_BY(fd, n, reason) unref_by(fd, n, reason, __FILE__, __LINE__) +static void ref_by(grpc_fd *fd, int n, const char *reason, const char *file, + int line) { + gpr_log(GPR_DEBUG, "FD %d %p ref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, + gpr_atm_no_barrier_load(&fd->refst), + gpr_atm_no_barrier_load(&fd->refst) + n, reason, file, line); +#else +#define REF_BY(fd, n, reason) ref_by(fd, n) +#define UNREF_BY(fd, n, reason) unref_by(fd, n) +static void ref_by(grpc_fd *fd, int n) { +#endif + GPR_ASSERT(gpr_atm_no_barrier_fetch_add(&fd->refst, n) > 0); +} + +#ifdef GRPC_FD_REF_COUNT_DEBUG +static void unref_by(grpc_fd *fd, int n, const char *reason, const char *file, + int line) { + gpr_atm old; + gpr_log(GPR_DEBUG, "FD %d %p unref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, + gpr_atm_no_barrier_load(&fd->refst), + gpr_atm_no_barrier_load(&fd->refst) - n, reason, file, line); +#else +static void unref_by(grpc_fd *fd, int n) { + gpr_atm old; +#endif + old = gpr_atm_full_fetch_add(&fd->refst, -n); + if (old == n) { + freelist_fd(fd); + } else { + GPR_ASSERT(old > n); + } +} + +static void fd_global_init(void) { gpr_mu_init(&fd_freelist_mu); } + +static void fd_global_shutdown(void) { + gpr_mu_lock(&fd_freelist_mu); + gpr_mu_unlock(&fd_freelist_mu); + while (fd_freelist != NULL) { + grpc_fd *fd = fd_freelist; + fd_freelist = fd_freelist->freelist_next; + destroy(fd); + } + gpr_mu_destroy(&fd_freelist_mu); +} + +static grpc_fd *fd_create(int fd, const char *name) { + grpc_fd *r = alloc_fd(fd); + char *name2; + gpr_asprintf(&name2, "%s fd=%d", name, fd); + grpc_iomgr_register_object(&r->iomgr_object, name2); + gpr_free(name2); +#ifdef GRPC_FD_REF_COUNT_DEBUG + gpr_log(GPR_DEBUG, "FD %d %p create %s", fd, r, name); +#endif + return r; +} + +static bool fd_is_orphaned(grpc_fd *fd) { + return (gpr_atm_acq_load(&fd->refst) & 1) == 0; +} + +static void pollset_kick_locked(grpc_fd_watcher *watcher) { + gpr_mu_lock(&watcher->pollset->mu); + GPR_ASSERT(watcher->worker); + pollset_kick_ext(watcher->pollset, watcher->worker, + GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP); + gpr_mu_unlock(&watcher->pollset->mu); +} + +static void maybe_wake_one_watcher_locked(grpc_fd *fd) { + if (fd->inactive_watcher_root.next != &fd->inactive_watcher_root) { + pollset_kick_locked(fd->inactive_watcher_root.next); + } else if (fd->read_watcher) { + pollset_kick_locked(fd->read_watcher); + } else if (fd->write_watcher) { + pollset_kick_locked(fd->write_watcher); + } +} + +static void wake_all_watchers_locked(grpc_fd *fd) { + grpc_fd_watcher *watcher; + for (watcher = fd->inactive_watcher_root.next; + watcher != &fd->inactive_watcher_root; watcher = watcher->next) { + pollset_kick_locked(watcher); + } + if (fd->read_watcher) { + pollset_kick_locked(fd->read_watcher); + } + if (fd->write_watcher && fd->write_watcher != fd->read_watcher) { + pollset_kick_locked(fd->write_watcher); + } +} + +static int has_watchers(grpc_fd *fd) { + return fd->read_watcher != NULL || fd->write_watcher != NULL || + fd->inactive_watcher_root.next != &fd->inactive_watcher_root; +} + +static void close_fd_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { + fd->closed = 1; + if (!fd->released) { + close(fd->fd); + } else { + remove_fd_from_all_epoll_sets(fd->fd); + } + grpc_exec_ctx_enqueue(exec_ctx, fd->on_done_closure, true, NULL); +} + +static int fd_wrapped_fd(grpc_fd *fd) { + if (fd->released || fd->closed) { + return -1; + } else { + return fd->fd; + } +} + +static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure *on_done, int *release_fd, + const char *reason) { + fd->on_done_closure = on_done; + fd->released = release_fd != NULL; + if (!fd->released) { + shutdown(fd->fd, SHUT_RDWR); + } else { + *release_fd = fd->fd; + } + gpr_mu_lock(&fd->mu); + REF_BY(fd, 1, reason); /* remove active status, but keep referenced */ + if (!has_watchers(fd)) { + close_fd_locked(exec_ctx, fd); + } else { + wake_all_watchers_locked(fd); + } + gpr_mu_unlock(&fd->mu); + UNREF_BY(fd, 2, reason); /* drop the reference */ +} + +/* increment refcount by two to avoid changing the orphan bit */ +#ifdef GRPC_FD_REF_COUNT_DEBUG +static void fd_ref(grpc_fd *fd, const char *reason, const char *file, + int line) { + ref_by(fd, 2, reason, file, line); +} + +static void fd_unref(grpc_fd *fd, const char *reason, const char *file, + int line) { + unref_by(fd, 2, reason, file, line); +} +#else +static void fd_ref(grpc_fd *fd) { ref_by(fd, 2); } + +static void fd_unref(grpc_fd *fd) { unref_by(fd, 2); } +#endif + +static void notify_on_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure **st, grpc_closure *closure) { + if (*st == CLOSURE_NOT_READY) { + /* not ready ==> switch to a waiting state by setting the closure */ + *st = closure; + } else if (*st == CLOSURE_READY) { + /* already ready ==> queue the closure to run immediately */ + *st = CLOSURE_NOT_READY; + grpc_exec_ctx_enqueue(exec_ctx, closure, !fd->shutdown, NULL); + maybe_wake_one_watcher_locked(fd); + } else { + /* upcallptr was set to a different closure. This is an error! */ + gpr_log(GPR_ERROR, + "User called a notify_on function with a previous callback still " + "pending"); + abort(); + } +} + +/* returns 1 if state becomes not ready */ +static int set_ready_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure **st) { + if (*st == CLOSURE_READY) { + /* duplicate ready ==> ignore */ + return 0; + } else if (*st == CLOSURE_NOT_READY) { + /* not ready, and not waiting ==> flag ready */ + *st = CLOSURE_READY; + return 0; + } else { + /* waiting ==> queue closure */ + grpc_exec_ctx_enqueue(exec_ctx, *st, !fd->shutdown, NULL); + *st = CLOSURE_NOT_READY; + return 1; + } +} + +static void fd_shutdown(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { + gpr_mu_lock(&fd->mu); + GPR_ASSERT(!fd->shutdown); + fd->shutdown = 1; + set_ready_locked(exec_ctx, fd, &fd->read_closure); + set_ready_locked(exec_ctx, fd, &fd->write_closure); + gpr_mu_unlock(&fd->mu); +} + +static void fd_notify_on_read(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure *closure) { + gpr_mu_lock(&fd->mu); + notify_on_locked(exec_ctx, fd, &fd->read_closure, closure); + gpr_mu_unlock(&fd->mu); +} + +static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure *closure) { + gpr_mu_lock(&fd->mu); + notify_on_locked(exec_ctx, fd, &fd->write_closure, closure); + gpr_mu_unlock(&fd->mu); +} + +static uint32_t fd_begin_poll(grpc_fd *fd, grpc_pollset *pollset, + grpc_pollset_worker *worker, uint32_t read_mask, + uint32_t write_mask, grpc_fd_watcher *watcher) { + uint32_t mask = 0; + grpc_closure *cur; + int requested; + /* keep track of pollers that have requested our events, in case they change + */ + GRPC_FD_REF(fd, "poll"); + + gpr_mu_lock(&fd->mu); + + /* if we are shutdown, then don't add to the watcher set */ + if (fd->shutdown) { + watcher->fd = NULL; + watcher->pollset = NULL; + watcher->worker = NULL; + gpr_mu_unlock(&fd->mu); + GRPC_FD_UNREF(fd, "poll"); + return 0; + } + + /* if there is nobody polling for read, but we need to, then start doing so */ + cur = fd->read_closure; + requested = cur != CLOSURE_READY; + if (read_mask && fd->read_watcher == NULL && requested) { + fd->read_watcher = watcher; + mask |= read_mask; + } + /* if there is nobody polling for write, but we need to, then start doing so + */ + cur = fd->write_closure; + requested = cur != CLOSURE_READY; + if (write_mask && fd->write_watcher == NULL && requested) { + fd->write_watcher = watcher; + mask |= write_mask; + } + /* if not polling, remember this watcher in case we need someone to later */ + if (mask == 0 && worker != NULL) { + watcher->next = &fd->inactive_watcher_root; + watcher->prev = watcher->next->prev; + watcher->next->prev = watcher->prev->next = watcher; + } + watcher->pollset = pollset; + watcher->worker = worker; + watcher->fd = fd; + gpr_mu_unlock(&fd->mu); + + return mask; +} + +static void fd_end_poll(grpc_exec_ctx *exec_ctx, grpc_fd_watcher *watcher, + int got_read, int got_write) { + int was_polling = 0; + int kick = 0; + grpc_fd *fd = watcher->fd; + + if (fd == NULL) { + return; + } + + gpr_mu_lock(&fd->mu); + + if (watcher == fd->read_watcher) { + /* remove read watcher, kick if we still need a read */ + was_polling = 1; + if (!got_read) { + kick = 1; + } + fd->read_watcher = NULL; + } + if (watcher == fd->write_watcher) { + /* remove write watcher, kick if we still need a write */ + was_polling = 1; + if (!got_write) { + kick = 1; + } + fd->write_watcher = NULL; + } + if (!was_polling && watcher->worker != NULL) { + /* remove from inactive list */ + watcher->next->prev = watcher->prev; + watcher->prev->next = watcher->next; + } + if (got_read) { + if (set_ready_locked(exec_ctx, fd, &fd->read_closure)) { + kick = 1; + } + } + if (got_write) { + if (set_ready_locked(exec_ctx, fd, &fd->write_closure)) { + kick = 1; + } + } + if (kick) { + maybe_wake_one_watcher_locked(fd); + } + if (fd_is_orphaned(fd) && !has_watchers(fd) && !fd->closed) { + close_fd_locked(exec_ctx, fd); + } + gpr_mu_unlock(&fd->mu); + + GRPC_FD_UNREF(fd, "poll"); +} + +/******************************************************************************* + * pollset_posix.c + */ + +GPR_TLS_DECL(g_current_thread_poller); +GPR_TLS_DECL(g_current_thread_worker); + +/** The alarm system needs to be able to wakeup 'some poller' sometimes + * (specifically when a new alarm needs to be triggered earlier than the next + * alarm 'epoch'). + * This wakeup_fd gives us something to alert on when such a case occurs. */ +grpc_wakeup_fd grpc_global_wakeup_fd; + +static void remove_worker(grpc_pollset *p, grpc_pollset_worker *worker) { + worker->prev->next = worker->next; + worker->next->prev = worker->prev; +} + +static int pollset_has_workers(grpc_pollset *p) { + return p->root_worker.next != &p->root_worker; +} + +static grpc_pollset_worker *pop_front_worker(grpc_pollset *p) { + if (pollset_has_workers(p)) { + grpc_pollset_worker *w = p->root_worker.next; + remove_worker(p, w); + return w; + } else { + return NULL; + } +} + +static void push_back_worker(grpc_pollset *p, grpc_pollset_worker *worker) { + worker->next = &p->root_worker; + worker->prev = worker->next->prev; + worker->prev->next = worker->next->prev = worker; +} + +static void push_front_worker(grpc_pollset *p, grpc_pollset_worker *worker) { + worker->prev = &p->root_worker; + worker->next = worker->prev->next; + worker->prev->next = worker->next->prev = worker; +} + +static void pollset_kick_ext(grpc_pollset *p, + grpc_pollset_worker *specific_worker, + uint32_t flags) { + GPR_TIMER_BEGIN("pollset_kick_ext", 0); + + /* pollset->mu already held */ + if (specific_worker != NULL) { + if (specific_worker == GRPC_POLLSET_KICK_BROADCAST) { + GPR_TIMER_BEGIN("pollset_kick_ext.broadcast", 0); + GPR_ASSERT((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) == 0); + for (specific_worker = p->root_worker.next; + specific_worker != &p->root_worker; + specific_worker = specific_worker->next) { + grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + } + p->kicked_without_pollers = 1; + GPR_TIMER_END("pollset_kick_ext.broadcast", 0); + } else if (gpr_tls_get(&g_current_thread_worker) != + (intptr_t)specific_worker) { + GPR_TIMER_MARK("different_thread_worker", 0); + if ((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) != 0) { + specific_worker->reevaluate_polling_on_wakeup = 1; + } + specific_worker->kicked_specifically = 1; + grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + } else if ((flags & GRPC_POLLSET_CAN_KICK_SELF) != 0) { + GPR_TIMER_MARK("kick_yoself", 0); + if ((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) != 0) { + specific_worker->reevaluate_polling_on_wakeup = 1; + } + specific_worker->kicked_specifically = 1; + grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + } + } else if (gpr_tls_get(&g_current_thread_poller) != (intptr_t)p) { + GPR_ASSERT((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) == 0); + GPR_TIMER_MARK("kick_anonymous", 0); + specific_worker = pop_front_worker(p); + if (specific_worker != NULL) { + if (gpr_tls_get(&g_current_thread_worker) == (intptr_t)specific_worker) { + GPR_TIMER_MARK("kick_anonymous_not_self", 0); + push_back_worker(p, specific_worker); + specific_worker = pop_front_worker(p); + if ((flags & GRPC_POLLSET_CAN_KICK_SELF) == 0 && + gpr_tls_get(&g_current_thread_worker) == + (intptr_t)specific_worker) { + push_back_worker(p, specific_worker); + specific_worker = NULL; + } + } + if (specific_worker != NULL) { + GPR_TIMER_MARK("finally_kick", 0); + push_back_worker(p, specific_worker); + grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + } + } else { + GPR_TIMER_MARK("kicked_no_pollers", 0); + p->kicked_without_pollers = 1; + } + } + + GPR_TIMER_END("pollset_kick_ext", 0); +} + +static void pollset_kick(grpc_pollset *p, + grpc_pollset_worker *specific_worker) { + pollset_kick_ext(p, specific_worker, 0); +} + +/* global state management */ + +static void pollset_global_init(void) { + gpr_tls_init(&g_current_thread_poller); + gpr_tls_init(&g_current_thread_worker); + grpc_wakeup_fd_init(&grpc_global_wakeup_fd); +} + +static void pollset_global_shutdown(void) { + grpc_wakeup_fd_destroy(&grpc_global_wakeup_fd); + gpr_tls_destroy(&g_current_thread_poller); + gpr_tls_destroy(&g_current_thread_worker); +} + +static void kick_poller(void) { grpc_wakeup_fd_wakeup(&grpc_global_wakeup_fd); } + +/* main interface */ + +static void become_basic_pollset(grpc_pollset *pollset, grpc_fd *fd_or_null); + +static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { + gpr_mu_init(&pollset->mu); + *mu = &pollset->mu; + pollset->root_worker.next = pollset->root_worker.prev = &pollset->root_worker; + pollset->in_flight_cbs = 0; + pollset->shutting_down = 0; + pollset->called_shutdown = 0; + pollset->kicked_without_pollers = 0; + pollset->idle_jobs.head = pollset->idle_jobs.tail = NULL; + pollset->local_wakeup_cache = NULL; + pollset->kicked_without_pollers = 0; + become_basic_pollset(pollset, NULL); +} + +static void pollset_destroy(grpc_pollset *pollset) { + GPR_ASSERT(pollset->in_flight_cbs == 0); + GPR_ASSERT(!pollset_has_workers(pollset)); + GPR_ASSERT(pollset->idle_jobs.head == pollset->idle_jobs.tail); + pollset->vtable->destroy(pollset); + while (pollset->local_wakeup_cache) { + grpc_cached_wakeup_fd *next = pollset->local_wakeup_cache->next; + grpc_wakeup_fd_destroy(&pollset->local_wakeup_cache->fd); + gpr_free(pollset->local_wakeup_cache); + pollset->local_wakeup_cache = next; + } + gpr_mu_destroy(&pollset->mu); +} + +static void pollset_reset(grpc_pollset *pollset) { + GPR_ASSERT(pollset->shutting_down); + GPR_ASSERT(pollset->in_flight_cbs == 0); + GPR_ASSERT(!pollset_has_workers(pollset)); + GPR_ASSERT(pollset->idle_jobs.head == pollset->idle_jobs.tail); + pollset->vtable->destroy(pollset); + pollset->shutting_down = 0; + pollset->called_shutdown = 0; + pollset->kicked_without_pollers = 0; + become_basic_pollset(pollset, NULL); +} + +static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_fd *fd) { + gpr_mu_lock(&pollset->mu); + pollset->vtable->add_fd(exec_ctx, pollset, fd, 1); +/* the following (enabled only in debug) will reacquire and then release + our lock - meaning that if the unlocking flag passed to add_fd above is + not respected, the code will deadlock (in a way that we have a chance of + debugging) */ +#ifndef NDEBUG + gpr_mu_lock(&pollset->mu); + gpr_mu_unlock(&pollset->mu); +#endif +} + +static void finish_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset) { + GPR_ASSERT(grpc_closure_list_empty(pollset->idle_jobs)); + pollset->vtable->finish_shutdown(pollset); + grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); +} + +static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_pollset_worker **worker_hdl, gpr_timespec now, + gpr_timespec deadline) { + grpc_pollset_worker worker; + *worker_hdl = &worker; + + /* pollset->mu already held */ + int added_worker = 0; + int locked = 1; + int queued_work = 0; + int keep_polling = 0; + GPR_TIMER_BEGIN("pollset_work", 0); + /* this must happen before we (potentially) drop pollset->mu */ + worker.next = worker.prev = NULL; + worker.reevaluate_polling_on_wakeup = 0; + if (pollset->local_wakeup_cache != NULL) { + worker.wakeup_fd = pollset->local_wakeup_cache; + pollset->local_wakeup_cache = worker.wakeup_fd->next; + } else { + worker.wakeup_fd = gpr_malloc(sizeof(*worker.wakeup_fd)); + grpc_wakeup_fd_init(&worker.wakeup_fd->fd); + } + worker.kicked_specifically = 0; + /* If there's work waiting for the pollset to be idle, and the + pollset is idle, then do that work */ + if (!pollset_has_workers(pollset) && + !grpc_closure_list_empty(pollset->idle_jobs)) { + GPR_TIMER_MARK("pollset_work.idle_jobs", 0); + grpc_exec_ctx_enqueue_list(exec_ctx, &pollset->idle_jobs, NULL); + goto done; + } + /* If we're shutting down then we don't execute any extended work */ + if (pollset->shutting_down) { + GPR_TIMER_MARK("pollset_work.shutting_down", 0); + goto done; + } + /* Give do_promote priority so we don't starve it out */ + if (pollset->in_flight_cbs) { + GPR_TIMER_MARK("pollset_work.in_flight_cbs", 0); + gpr_mu_unlock(&pollset->mu); + locked = 0; + goto done; + } + /* Start polling, and keep doing so while we're being asked to + re-evaluate our pollers (this allows poll() based pollers to + ensure they don't miss wakeups) */ + keep_polling = 1; + while (keep_polling) { + keep_polling = 0; + if (!pollset->kicked_without_pollers) { + if (!added_worker) { + push_front_worker(pollset, &worker); + added_worker = 1; + gpr_tls_set(&g_current_thread_worker, (intptr_t)&worker); + } + gpr_tls_set(&g_current_thread_poller, (intptr_t)pollset); + GPR_TIMER_BEGIN("maybe_work_and_unlock", 0); + pollset->vtable->maybe_work_and_unlock(exec_ctx, pollset, &worker, + deadline, now); + GPR_TIMER_END("maybe_work_and_unlock", 0); + locked = 0; + gpr_tls_set(&g_current_thread_poller, 0); + } else { + GPR_TIMER_MARK("pollset_work.kicked_without_pollers", 0); + pollset->kicked_without_pollers = 0; + } + /* Finished execution - start cleaning up. + Note that we may arrive here from outside the enclosing while() loop. + In that case we won't loop though as we haven't added worker to the + worker list, which means nobody could ask us to re-evaluate polling). */ + done: + if (!locked) { + queued_work |= grpc_exec_ctx_flush(exec_ctx); + gpr_mu_lock(&pollset->mu); + locked = 1; + } + /* If we're forced to re-evaluate polling (via pollset_kick with + GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) then we land here and force + a loop */ + if (worker.reevaluate_polling_on_wakeup) { + worker.reevaluate_polling_on_wakeup = 0; + pollset->kicked_without_pollers = 0; + if (queued_work || worker.kicked_specifically) { + /* If there's queued work on the list, then set the deadline to be + immediate so we get back out of the polling loop quickly */ + deadline = gpr_inf_past(GPR_CLOCK_MONOTONIC); + } + keep_polling = 1; + } + } + if (added_worker) { + remove_worker(pollset, &worker); + gpr_tls_set(&g_current_thread_worker, 0); + } + /* release wakeup fd to the local pool */ + worker.wakeup_fd->next = pollset->local_wakeup_cache; + pollset->local_wakeup_cache = worker.wakeup_fd; + /* check shutdown conditions */ + if (pollset->shutting_down) { + if (pollset_has_workers(pollset)) { + pollset_kick(pollset, NULL); + } else if (!pollset->called_shutdown && pollset->in_flight_cbs == 0) { + pollset->called_shutdown = 1; + gpr_mu_unlock(&pollset->mu); + finish_shutdown(exec_ctx, pollset); + grpc_exec_ctx_flush(exec_ctx); + /* Continuing to access pollset here is safe -- it is the caller's + * responsibility to not destroy when it has outstanding calls to + * pollset_work. + * TODO(dklempner): Can we refactor the shutdown logic to avoid this? */ + gpr_mu_lock(&pollset->mu); + } else if (!grpc_closure_list_empty(pollset->idle_jobs)) { + grpc_exec_ctx_enqueue_list(exec_ctx, &pollset->idle_jobs, NULL); + gpr_mu_unlock(&pollset->mu); + grpc_exec_ctx_flush(exec_ctx); + gpr_mu_lock(&pollset->mu); + } + } + *worker_hdl = NULL; + GPR_TIMER_END("pollset_work", 0); +} + +static void pollset_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_closure *closure) { + GPR_ASSERT(!pollset->shutting_down); + pollset->shutting_down = 1; + pollset->shutdown_done = closure; + pollset_kick(pollset, GRPC_POLLSET_KICK_BROADCAST); + if (!pollset_has_workers(pollset)) { + grpc_exec_ctx_enqueue_list(exec_ctx, &pollset->idle_jobs, NULL); + } + if (!pollset->called_shutdown && pollset->in_flight_cbs == 0 && + !pollset_has_workers(pollset)) { + pollset->called_shutdown = 1; + finish_shutdown(exec_ctx, pollset); + } +} + +static int poll_deadline_to_millis_timeout(gpr_timespec deadline, + gpr_timespec now) { + gpr_timespec timeout; + static const int64_t max_spin_polling_us = 10; + if (gpr_time_cmp(deadline, gpr_inf_future(deadline.clock_type)) == 0) { + return -1; + } + if (gpr_time_cmp(deadline, gpr_time_add(now, gpr_time_from_micros( + max_spin_polling_us, + GPR_TIMESPAN))) <= 0) { + return 0; + } + timeout = gpr_time_sub(deadline, now); + return gpr_time_to_millis(gpr_time_add( + timeout, gpr_time_from_nanos(GPR_NS_PER_MS - 1, GPR_TIMESPAN))); +} + +/* + * basic_pollset - a vtable that provides polling for zero or one file + * descriptor via poll() + */ + +typedef struct grpc_unary_promote_args { + const grpc_pollset_vtable *original_vtable; + grpc_pollset *pollset; + grpc_fd *fd; + grpc_closure promotion_closure; +} grpc_unary_promote_args; + +static void basic_do_promote(grpc_exec_ctx *exec_ctx, void *args, + bool success) { + grpc_unary_promote_args *up_args = args; + const grpc_pollset_vtable *original_vtable = up_args->original_vtable; + grpc_pollset *pollset = up_args->pollset; + grpc_fd *fd = up_args->fd; + + /* + * This is quite tricky. There are a number of cases to keep in mind here: + * 1. fd may have been orphaned + * 2. The pollset may no longer be a unary poller (and we can't let case #1 + * leak to other pollset types!) + * 3. pollset's fd (which may have changed) may have been orphaned + * 4. The pollset may be shutting down. + */ + + gpr_mu_lock(&pollset->mu); + /* First we need to ensure that nobody is polling concurrently */ + GPR_ASSERT(!pollset_has_workers(pollset)); + + gpr_free(up_args); + /* At this point the pollset may no longer be a unary poller. In that case + * we should just call the right add function and be done. */ + /* TODO(klempner): If we're not careful this could cause infinite recursion. + * That's not a problem for now because empty_pollset has a trivial poller + * and we don't have any mechanism to unbecome multipoller. */ + pollset->in_flight_cbs--; + if (pollset->shutting_down) { + /* We don't care about this pollset anymore. */ + if (pollset->in_flight_cbs == 0 && !pollset->called_shutdown) { + pollset->called_shutdown = 1; + finish_shutdown(exec_ctx, pollset); + } + } else if (fd_is_orphaned(fd)) { + /* Don't try to add it to anything, we'll drop our ref on it below */ + } else if (pollset->vtable != original_vtable) { + pollset->vtable->add_fd(exec_ctx, pollset, fd, 0); + } else if (fd != pollset->data.ptr) { + grpc_fd *fds[2]; + fds[0] = pollset->data.ptr; + fds[1] = fd; + + if (fds[0] && !fd_is_orphaned(fds[0])) { + platform_become_multipoller(exec_ctx, pollset, fds, GPR_ARRAY_SIZE(fds)); + GRPC_FD_UNREF(fds[0], "basicpoll"); + } else { + /* old fd is orphaned and we haven't cleaned it up until now, so remain a + * unary poller */ + /* Note that it is possible that fds[1] is also orphaned at this point. + * That's okay, we'll correct it at the next add or poll. */ + if (fds[0]) GRPC_FD_UNREF(fds[0], "basicpoll"); + pollset->data.ptr = fd; + GRPC_FD_REF(fd, "basicpoll"); + } + } + + gpr_mu_unlock(&pollset->mu); + + /* Matching ref in basic_pollset_add_fd */ + GRPC_FD_UNREF(fd, "basicpoll_add"); +} + +static void basic_pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_fd *fd, int and_unlock_pollset) { + grpc_unary_promote_args *up_args; + GPR_ASSERT(fd); + if (fd == pollset->data.ptr) goto exit; + + if (!pollset_has_workers(pollset)) { + /* Fast path -- no in flight cbs */ + /* TODO(klempner): Comment this out and fix any test failures or establish + * they are due to timing issues */ + grpc_fd *fds[2]; + fds[0] = pollset->data.ptr; + fds[1] = fd; + + if (fds[0] == NULL) { + pollset->data.ptr = fd; + GRPC_FD_REF(fd, "basicpoll"); + } else if (!fd_is_orphaned(fds[0])) { + platform_become_multipoller(exec_ctx, pollset, fds, GPR_ARRAY_SIZE(fds)); + GRPC_FD_UNREF(fds[0], "basicpoll"); + } else { + /* old fd is orphaned and we haven't cleaned it up until now, so remain a + * unary poller */ + GRPC_FD_UNREF(fds[0], "basicpoll"); + pollset->data.ptr = fd; + GRPC_FD_REF(fd, "basicpoll"); + } + goto exit; + } + + /* Now we need to promote. This needs to happen when we're not polling. Since + * this may be called from poll, the wait needs to happen asynchronously. */ + GRPC_FD_REF(fd, "basicpoll_add"); + pollset->in_flight_cbs++; + up_args = gpr_malloc(sizeof(*up_args)); + up_args->fd = fd; + up_args->original_vtable = pollset->vtable; + up_args->pollset = pollset; + up_args->promotion_closure.cb = basic_do_promote; + up_args->promotion_closure.cb_arg = up_args; + + grpc_closure_list_add(&pollset->idle_jobs, &up_args->promotion_closure, 1); + pollset_kick(pollset, GRPC_POLLSET_KICK_BROADCAST); + +exit: + if (and_unlock_pollset) { + gpr_mu_unlock(&pollset->mu); + } +} + +static void basic_pollset_maybe_work_and_unlock(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset, + grpc_pollset_worker *worker, + gpr_timespec deadline, + gpr_timespec now) { +#define POLLOUT_CHECK (POLLOUT | POLLHUP | POLLERR) +#define POLLIN_CHECK (POLLIN | POLLHUP | POLLERR) + + struct pollfd pfd[3]; + grpc_fd *fd; + grpc_fd_watcher fd_watcher; + int timeout; + int r; + nfds_t nfds; + + fd = pollset->data.ptr; + if (fd && fd_is_orphaned(fd)) { + GRPC_FD_UNREF(fd, "basicpoll"); + fd = pollset->data.ptr = NULL; + } + timeout = poll_deadline_to_millis_timeout(deadline, now); + pfd[0].fd = GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd); + pfd[0].events = POLLIN; + pfd[0].revents = 0; + pfd[1].fd = GRPC_WAKEUP_FD_GET_READ_FD(&worker->wakeup_fd->fd); + pfd[1].events = POLLIN; + pfd[1].revents = 0; + nfds = 2; + if (fd) { + pfd[2].fd = fd->fd; + pfd[2].revents = 0; + GRPC_FD_REF(fd, "basicpoll_begin"); + gpr_mu_unlock(&pollset->mu); + pfd[2].events = + (short)fd_begin_poll(fd, pollset, worker, POLLIN, POLLOUT, &fd_watcher); + if (pfd[2].events != 0) { + nfds++; + } + } else { + gpr_mu_unlock(&pollset->mu); + } + + /* TODO(vpai): Consider first doing a 0 timeout poll here to avoid + even going into the blocking annotation if possible */ + /* poll fd count (argument 2) is shortened by one if we have no events + to poll on - such that it only includes the kicker */ + GPR_TIMER_BEGIN("poll", 0); + GRPC_SCHEDULING_START_BLOCKING_REGION; + r = grpc_poll_function(pfd, nfds, timeout); + GRPC_SCHEDULING_END_BLOCKING_REGION; + GPR_TIMER_END("poll", 0); + + if (r < 0) { + if (errno != EINTR) { + gpr_log(GPR_ERROR, "poll() failed: %s", strerror(errno)); + } + if (fd) { + fd_end_poll(exec_ctx, &fd_watcher, 0, 0); + } + } else if (r == 0) { + if (fd) { + fd_end_poll(exec_ctx, &fd_watcher, 0, 0); + } + } else { + if (pfd[0].revents & POLLIN_CHECK) { + grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); + } + if (pfd[1].revents & POLLIN_CHECK) { + grpc_wakeup_fd_consume_wakeup(&worker->wakeup_fd->fd); + } + if (nfds > 2) { + fd_end_poll(exec_ctx, &fd_watcher, pfd[2].revents & POLLIN_CHECK, + pfd[2].revents & POLLOUT_CHECK); + } else if (fd) { + fd_end_poll(exec_ctx, &fd_watcher, 0, 0); + } + } + + if (fd) { + GRPC_FD_UNREF(fd, "basicpoll_begin"); + } +} + +static void basic_pollset_destroy(grpc_pollset *pollset) { + if (pollset->data.ptr != NULL) { + GRPC_FD_UNREF(pollset->data.ptr, "basicpoll"); + pollset->data.ptr = NULL; + } +} + +static const grpc_pollset_vtable basic_pollset = { + basic_pollset_add_fd, basic_pollset_maybe_work_and_unlock, + basic_pollset_destroy, basic_pollset_destroy}; + +static void become_basic_pollset(grpc_pollset *pollset, grpc_fd *fd_or_null) { + pollset->vtable = &basic_pollset; + pollset->data.ptr = fd_or_null; + if (fd_or_null != NULL) { + GRPC_FD_REF(fd_or_null, "basicpoll"); + } +} + + +/******************************************************************************* + * pollset_multipoller_with_epoll_posix.c + */ + +#include +#include +#include +#include +#include + +#include +#include +#include + +#include "src/core/lib/iomgr/ev_posix.h" +#include "src/core/lib/profiling/timers.h" +#include "src/core/lib/support/block_annotate.h" + +static void set_ready(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure **st) { + /* only one set_ready can be active at once (but there may be a racing + notify_on) */ + gpr_mu_lock(&fd->mu); + set_ready_locked(exec_ctx, fd, st); + gpr_mu_unlock(&fd->mu); +} + +static void fd_become_readable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { + set_ready(exec_ctx, fd, &fd->read_closure); +} + +static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { + set_ready(exec_ctx, fd, &fd->write_closure); +} + +struct epoll_fd_list { + int *epoll_fds; + size_t count; + size_t capacity; +}; + +static struct epoll_fd_list epoll_fd_global_list; +static gpr_once init_epoll_fd_list_mu = GPR_ONCE_INIT; +static gpr_mu epoll_fd_list_mu; + +static void init_mu(void) { gpr_mu_init(&epoll_fd_list_mu); } + +static void add_epoll_fd_to_global_list(int epoll_fd) { + gpr_once_init(&init_epoll_fd_list_mu, init_mu); + + gpr_mu_lock(&epoll_fd_list_mu); + if (epoll_fd_global_list.count == epoll_fd_global_list.capacity) { + epoll_fd_global_list.capacity = + GPR_MAX((size_t)8, epoll_fd_global_list.capacity * 2); + epoll_fd_global_list.epoll_fds = + gpr_realloc(epoll_fd_global_list.epoll_fds, + epoll_fd_global_list.capacity * sizeof(int)); + } + epoll_fd_global_list.epoll_fds[epoll_fd_global_list.count++] = epoll_fd; + gpr_mu_unlock(&epoll_fd_list_mu); +} + +static void remove_epoll_fd_from_global_list(int epoll_fd) { + gpr_mu_lock(&epoll_fd_list_mu); + GPR_ASSERT(epoll_fd_global_list.count > 0); + for (size_t i = 0; i < epoll_fd_global_list.count; i++) { + if (epoll_fd == epoll_fd_global_list.epoll_fds[i]) { + epoll_fd_global_list.epoll_fds[i] = + epoll_fd_global_list.epoll_fds[--(epoll_fd_global_list.count)]; + break; + } + } + gpr_mu_unlock(&epoll_fd_list_mu); +} + +static void remove_fd_from_all_epoll_sets(int fd) { + int err; + gpr_once_init(&init_epoll_fd_list_mu, init_mu); + gpr_mu_lock(&epoll_fd_list_mu); + if (epoll_fd_global_list.count == 0) { + gpr_mu_unlock(&epoll_fd_list_mu); + return; + } + for (size_t i = 0; i < epoll_fd_global_list.count; i++) { + err = epoll_ctl(epoll_fd_global_list.epoll_fds[i], EPOLL_CTL_DEL, fd, NULL); + if (err < 0 && errno != ENOENT) { + gpr_log(GPR_ERROR, "epoll_ctl del for %d failed: %s", fd, + strerror(errno)); + } + } + gpr_mu_unlock(&epoll_fd_list_mu); +} + +typedef struct { + grpc_pollset *pollset; + grpc_fd *fd; + grpc_closure closure; +} delayed_add; + +typedef struct { int epoll_fd; } epoll_hdr; + +static void finally_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_fd *fd) { + epoll_hdr *h = pollset->data.ptr; + struct epoll_event ev; + int err; + grpc_fd_watcher watcher; + + /* We pretend to be polling whilst adding an fd to keep the fd from being + closed during the add. This may result in a spurious wakeup being assigned + to this pollset whilst adding, but that should be benign. */ + GPR_ASSERT(fd_begin_poll(fd, pollset, NULL, 0, 0, &watcher) == 0); + if (watcher.fd != NULL) { + ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); + ev.data.ptr = fd; + err = epoll_ctl(h->epoll_fd, EPOLL_CTL_ADD, fd->fd, &ev); + if (err < 0) { + /* FDs may be added to a pollset multiple times, so EEXIST is normal. */ + if (errno != EEXIST) { + gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", fd->fd, + strerror(errno)); + } + } + } + fd_end_poll(exec_ctx, &watcher, 0, 0); +} + +static void perform_delayed_add(grpc_exec_ctx *exec_ctx, void *arg, + bool iomgr_status) { + delayed_add *da = arg; + + if (!fd_is_orphaned(da->fd)) { + finally_add_fd(exec_ctx, da->pollset, da->fd); + } + + gpr_mu_lock(&da->pollset->mu); + da->pollset->in_flight_cbs--; + if (da->pollset->shutting_down) { + /* We don't care about this pollset anymore. */ + if (da->pollset->in_flight_cbs == 0 && !da->pollset->called_shutdown) { + da->pollset->called_shutdown = 1; + grpc_exec_ctx_enqueue(exec_ctx, da->pollset->shutdown_done, true, NULL); + } + } + gpr_mu_unlock(&da->pollset->mu); + + GRPC_FD_UNREF(da->fd, "delayed_add"); + + gpr_free(da); +} + +static void multipoll_with_epoll_pollset_add_fd(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset, + grpc_fd *fd, + int and_unlock_pollset) { + if (and_unlock_pollset) { + gpr_mu_unlock(&pollset->mu); + finally_add_fd(exec_ctx, pollset, fd); + } else { + delayed_add *da = gpr_malloc(sizeof(*da)); + da->pollset = pollset; + da->fd = fd; + GRPC_FD_REF(fd, "delayed_add"); + grpc_closure_init(&da->closure, perform_delayed_add, da); + pollset->in_flight_cbs++; + grpc_exec_ctx_enqueue(exec_ctx, &da->closure, true, NULL); + } +} + +/* TODO(klempner): We probably want to turn this down a bit */ +#define GRPC_EPOLL_MAX_EVENTS 1000 + +static void multipoll_with_epoll_pollset_maybe_work_and_unlock( + grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker *worker, + gpr_timespec deadline, gpr_timespec now) { + struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; + int ep_rv; + int poll_rv; + epoll_hdr *h = pollset->data.ptr; + int timeout_ms; + struct pollfd pfds[2]; + + /* If you want to ignore epoll's ability to sanely handle parallel pollers, + * for a more apples-to-apples performance comparison with poll, add a + * if (pollset->counter != 0) { return 0; } + * here. + */ + + gpr_mu_unlock(&pollset->mu); + + timeout_ms = poll_deadline_to_millis_timeout(deadline, now); + + pfds[0].fd = GRPC_WAKEUP_FD_GET_READ_FD(&worker->wakeup_fd->fd); + pfds[0].events = POLLIN; + pfds[0].revents = 0; + pfds[1].fd = h->epoll_fd; + pfds[1].events = POLLIN; + pfds[1].revents = 0; + + /* TODO(vpai): Consider first doing a 0 timeout poll here to avoid + even going into the blocking annotation if possible */ + GPR_TIMER_BEGIN("poll", 0); + GRPC_SCHEDULING_START_BLOCKING_REGION; + poll_rv = grpc_poll_function(pfds, 2, timeout_ms); + GRPC_SCHEDULING_END_BLOCKING_REGION; + GPR_TIMER_END("poll", 0); + + if (poll_rv < 0) { + if (errno != EINTR) { + gpr_log(GPR_ERROR, "poll() failed: %s", strerror(errno)); + } + } else if (poll_rv == 0) { + /* do nothing */ + } else { + if (pfds[0].revents) { + grpc_wakeup_fd_consume_wakeup(&worker->wakeup_fd->fd); + } + if (pfds[1].revents) { + do { + /* The following epoll_wait never blocks; it has a timeout of 0 */ + ep_rv = epoll_wait(h->epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, 0); + if (ep_rv < 0) { + if (errno != EINTR) { + gpr_log(GPR_ERROR, "epoll_wait() failed: %s", strerror(errno)); + } + } else { + int i; + for (i = 0; i < ep_rv; ++i) { + grpc_fd *fd = ep_ev[i].data.ptr; + /* TODO(klempner): We might want to consider making err and pri + * separate events */ + int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); + int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); + int write_ev = ep_ev[i].events & EPOLLOUT; + if (fd == NULL) { + grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); + } else { + if (read_ev || cancel) { + fd_become_readable(exec_ctx, fd); + } + if (write_ev || cancel) { + fd_become_writable(exec_ctx, fd); + } + } + } + } + } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); + } + } +} + +static void multipoll_with_epoll_pollset_finish_shutdown( + grpc_pollset *pollset) {} + +static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset) { + epoll_hdr *h = pollset->data.ptr; + close(h->epoll_fd); + remove_epoll_fd_from_global_list(h->epoll_fd); + gpr_free(h); +} + +static const grpc_pollset_vtable multipoll_with_epoll_pollset = { + multipoll_with_epoll_pollset_add_fd, + multipoll_with_epoll_pollset_maybe_work_and_unlock, + multipoll_with_epoll_pollset_finish_shutdown, + multipoll_with_epoll_pollset_destroy}; + +static void epoll_become_multipoller(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset, grpc_fd **fds, + size_t nfds) { + size_t i; + epoll_hdr *h = gpr_malloc(sizeof(epoll_hdr)); + struct epoll_event ev; + int err; + + pollset->vtable = &multipoll_with_epoll_pollset; + pollset->data.ptr = h; + h->epoll_fd = epoll_create1(EPOLL_CLOEXEC); + if (h->epoll_fd < 0) { + /* TODO(klempner): Fall back to poll here, especially on ENOSYS */ + gpr_log(GPR_ERROR, "epoll_create1 failed: %s", strerror(errno)); + abort(); + } + add_epoll_fd_to_global_list(h->epoll_fd); + + ev.events = (uint32_t)(EPOLLIN | EPOLLET); + ev.data.ptr = NULL; + err = epoll_ctl(h->epoll_fd, EPOLL_CTL_ADD, + GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), &ev); + if (err < 0) { + gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", + GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), + strerror(errno)); + } + + for (i = 0; i < nfds; i++) { + multipoll_with_epoll_pollset_add_fd(exec_ctx, pollset, fds[i], 0); + } +} + +/******************************************************************************* + * pollset_set_posix.c + */ + +static grpc_pollset_set *pollset_set_create(void) { + grpc_pollset_set *pollset_set = gpr_malloc(sizeof(*pollset_set)); + memset(pollset_set, 0, sizeof(*pollset_set)); + gpr_mu_init(&pollset_set->mu); + return pollset_set; +} + +static void pollset_set_destroy(grpc_pollset_set *pollset_set) { + size_t i; + gpr_mu_destroy(&pollset_set->mu); + for (i = 0; i < pollset_set->fd_count; i++) { + GRPC_FD_UNREF(pollset_set->fds[i], "pollset_set"); + } + gpr_free(pollset_set->pollsets); + gpr_free(pollset_set->pollset_sets); + gpr_free(pollset_set->fds); + gpr_free(pollset_set); +} + +static void pollset_set_add_pollset(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, + grpc_pollset *pollset) { + size_t i, j; + gpr_mu_lock(&pollset_set->mu); + if (pollset_set->pollset_count == pollset_set->pollset_capacity) { + pollset_set->pollset_capacity = + GPR_MAX(8, 2 * pollset_set->pollset_capacity); + pollset_set->pollsets = + gpr_realloc(pollset_set->pollsets, pollset_set->pollset_capacity * + sizeof(*pollset_set->pollsets)); + } + pollset_set->pollsets[pollset_set->pollset_count++] = pollset; + for (i = 0, j = 0; i < pollset_set->fd_count; i++) { + if (fd_is_orphaned(pollset_set->fds[i])) { + GRPC_FD_UNREF(pollset_set->fds[i], "pollset_set"); + } else { + pollset_add_fd(exec_ctx, pollset, pollset_set->fds[i]); + pollset_set->fds[j++] = pollset_set->fds[i]; + } + } + pollset_set->fd_count = j; + gpr_mu_unlock(&pollset_set->mu); +} + +static void pollset_set_del_pollset(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, + grpc_pollset *pollset) { + size_t i; + gpr_mu_lock(&pollset_set->mu); + for (i = 0; i < pollset_set->pollset_count; i++) { + if (pollset_set->pollsets[i] == pollset) { + pollset_set->pollset_count--; + GPR_SWAP(grpc_pollset *, pollset_set->pollsets[i], + pollset_set->pollsets[pollset_set->pollset_count]); + break; + } + } + gpr_mu_unlock(&pollset_set->mu); +} + +static void pollset_set_add_pollset_set(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *bag, + grpc_pollset_set *item) { + size_t i, j; + gpr_mu_lock(&bag->mu); + if (bag->pollset_set_count == bag->pollset_set_capacity) { + bag->pollset_set_capacity = GPR_MAX(8, 2 * bag->pollset_set_capacity); + bag->pollset_sets = + gpr_realloc(bag->pollset_sets, + bag->pollset_set_capacity * sizeof(*bag->pollset_sets)); + } + bag->pollset_sets[bag->pollset_set_count++] = item; + for (i = 0, j = 0; i < bag->fd_count; i++) { + if (fd_is_orphaned(bag->fds[i])) { + GRPC_FD_UNREF(bag->fds[i], "pollset_set"); + } else { + pollset_set_add_fd(exec_ctx, item, bag->fds[i]); + bag->fds[j++] = bag->fds[i]; + } + } + bag->fd_count = j; + gpr_mu_unlock(&bag->mu); +} + +static void pollset_set_del_pollset_set(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *bag, + grpc_pollset_set *item) { + size_t i; + gpr_mu_lock(&bag->mu); + for (i = 0; i < bag->pollset_set_count; i++) { + if (bag->pollset_sets[i] == item) { + bag->pollset_set_count--; + GPR_SWAP(grpc_pollset_set *, bag->pollset_sets[i], + bag->pollset_sets[bag->pollset_set_count]); + break; + } + } + gpr_mu_unlock(&bag->mu); +} + +static void pollset_set_add_fd(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, grpc_fd *fd) { + size_t i; + gpr_mu_lock(&pollset_set->mu); + if (pollset_set->fd_count == pollset_set->fd_capacity) { + pollset_set->fd_capacity = GPR_MAX(8, 2 * pollset_set->fd_capacity); + pollset_set->fds = gpr_realloc( + pollset_set->fds, pollset_set->fd_capacity * sizeof(*pollset_set->fds)); + } + GRPC_FD_REF(fd, "pollset_set"); + pollset_set->fds[pollset_set->fd_count++] = fd; + for (i = 0; i < pollset_set->pollset_count; i++) { + pollset_add_fd(exec_ctx, pollset_set->pollsets[i], fd); + } + for (i = 0; i < pollset_set->pollset_set_count; i++) { + pollset_set_add_fd(exec_ctx, pollset_set->pollset_sets[i], fd); + } + gpr_mu_unlock(&pollset_set->mu); +} + +static void pollset_set_del_fd(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, grpc_fd *fd) { + size_t i; + gpr_mu_lock(&pollset_set->mu); + for (i = 0; i < pollset_set->fd_count; i++) { + if (pollset_set->fds[i] == fd) { + pollset_set->fd_count--; + GPR_SWAP(grpc_fd *, pollset_set->fds[i], + pollset_set->fds[pollset_set->fd_count]); + GRPC_FD_UNREF(fd, "pollset_set"); + break; + } + } + for (i = 0; i < pollset_set->pollset_set_count; i++) { + pollset_set_del_fd(exec_ctx, pollset_set->pollset_sets[i], fd); + } + gpr_mu_unlock(&pollset_set->mu); +} + +/******************************************************************************* + * event engine binding + */ + +static void shutdown_engine(void) { + fd_global_shutdown(); + pollset_global_shutdown(); +} + +static const grpc_event_engine_vtable vtable = { + .pollset_size = sizeof(grpc_pollset), + + .fd_create = fd_create, + .fd_wrapped_fd = fd_wrapped_fd, + .fd_orphan = fd_orphan, + .fd_shutdown = fd_shutdown, + .fd_notify_on_read = fd_notify_on_read, + .fd_notify_on_write = fd_notify_on_write, + + .pollset_init = pollset_init, + .pollset_shutdown = pollset_shutdown, + .pollset_reset = pollset_reset, + .pollset_destroy = pollset_destroy, + .pollset_work = pollset_work, + .pollset_kick = pollset_kick, + .pollset_add_fd = pollset_add_fd, + + .pollset_set_create = pollset_set_create, + .pollset_set_destroy = pollset_set_destroy, + .pollset_set_add_pollset = pollset_set_add_pollset, + .pollset_set_del_pollset = pollset_set_del_pollset, + .pollset_set_add_pollset_set = pollset_set_add_pollset_set, + .pollset_set_del_pollset_set = pollset_set_del_pollset_set, + .pollset_set_add_fd = pollset_set_add_fd, + .pollset_set_del_fd = pollset_set_del_fd, + + .kick_poller = kick_poller, + + .shutdown_engine = shutdown_engine, +}; + +const grpc_event_engine_vtable *grpc_init_epoll_posix(void) { + platform_become_multipoller = epoll_become_multipoller; + fd_global_init(); + pollset_global_init(); + return &vtable; +} + +#endif diff --git a/src/core/lib/iomgr/ev_epoll_posix.h b/src/core/lib/iomgr/ev_epoll_posix.h new file mode 100644 index 00000000000..35319b4fc5d --- /dev/null +++ b/src/core/lib/iomgr/ev_epoll_posix.h @@ -0,0 +1,41 @@ +/* + * + * Copyright 2015, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#ifndef GRPC_CORE_LIB_IOMGR_EV_EPOLL_POSIX_H +#define GRPC_CORE_LIB_IOMGR_EV_EPOLL_POSIX_H + +#include "src/core/lib/iomgr/ev_posix.h" + +const grpc_event_engine_vtable *grpc_init_epoll_posix(void); + +#endif /* GRPC_CORE_LIB_IOMGR_EV_EPOLL_POSIX_H */ diff --git a/src/core/lib/iomgr/ev_posix.c b/src/core/lib/iomgr/ev_posix.c index 95520b01d3a..baa3b9856ad 100644 --- a/src/core/lib/iomgr/ev_posix.c +++ b/src/core/lib/iomgr/ev_posix.c @@ -44,6 +44,7 @@ #include #include +#include "src/core/lib/iomgr/ev_epoll_posix.h" #include "src/core/lib/iomgr/ev_poll_posix.h" #include "src/core/lib/support/env.h" @@ -61,7 +62,7 @@ typedef struct { } event_engine_factory; static const event_engine_factory g_factories[] = { - {"poll", grpc_init_poll_posix}, + {"poll", grpc_init_poll_posix}, {"epoll", grpc_init_epoll_posix}, }; static void add(const char *beg, const char *end, char ***ss, size_t *ns) { diff --git a/src/python/grpcio/grpc_core_dependencies.py b/src/python/grpcio/grpc_core_dependencies.py index be24dc7cf02..49b4ddc457c 100644 --- a/src/python/grpcio/grpc_core_dependencies.py +++ b/src/python/grpcio/grpc_core_dependencies.py @@ -94,6 +94,7 @@ CORE_SOURCE_FILES = [ 'src/core/lib/iomgr/endpoint.c', 'src/core/lib/iomgr/endpoint_pair_posix.c', 'src/core/lib/iomgr/endpoint_pair_windows.c', + 'src/core/lib/iomgr/ev_epoll_posix.c', 'src/core/lib/iomgr/ev_poll_posix.c', 'src/core/lib/iomgr/ev_posix.c', 'src/core/lib/iomgr/exec_ctx.c', diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index c3fd59b3c23..36b25cca161 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -807,6 +807,7 @@ src/core/lib/http/parser.h \ src/core/lib/iomgr/closure.h \ src/core/lib/iomgr/endpoint.h \ src/core/lib/iomgr/endpoint_pair.h \ +src/core/lib/iomgr/ev_epoll_posix.h \ src/core/lib/iomgr/ev_poll_posix.h \ src/core/lib/iomgr/ev_posix.h \ src/core/lib/iomgr/exec_ctx.h \ @@ -954,6 +955,7 @@ src/core/lib/iomgr/closure.c \ src/core/lib/iomgr/endpoint.c \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ +src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ src/core/lib/iomgr/exec_ctx.c \ diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 24d23fe28bd..43940495864 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -5530,6 +5530,7 @@ "src/core/lib/iomgr/closure.h", "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", + "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", "src/core/lib/iomgr/exec_ctx.h", @@ -5629,6 +5630,8 @@ "src/core/lib/iomgr/endpoint_pair.h", "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", + "src/core/lib/iomgr/ev_epoll_posix.c", + "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.c", diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj b/vsprojects/vcxproj/grpc/grpc.vcxproj index e5379dc6a49..55304af5869 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj @@ -316,6 +316,7 @@ + @@ -483,6 +484,8 @@ + + diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters index 95a5a73d268..7d1c90fda7c 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters @@ -55,6 +55,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr @@ -674,6 +677,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj index 90ad80f2fc2..3d0cdfc668b 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj @@ -304,6 +304,7 @@ + @@ -449,6 +450,8 @@ + + diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters index 2b19c0fb341..d2ff4c630fd 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters @@ -58,6 +58,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr @@ -572,6 +575,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr From a7786001a22f511130a4292893cc8e2ab0ccdf75 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 11 May 2016 18:13:31 -0700 Subject: [PATCH 0016/1039] Remove basic_pollset and the promotion related code --- src/core/lib/iomgr/ev_epoll_posix.c | 355 ++++++---------------------- 1 file changed, 71 insertions(+), 284 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_posix.c b/src/core/lib/iomgr/ev_epoll_posix.c index ce8d3981b3f..cb4a00a75c4 100644 --- a/src/core/lib/iomgr/ev_epoll_posix.c +++ b/src/core/lib/iomgr/ev_epoll_posix.c @@ -185,11 +185,6 @@ struct grpc_pollset_worker { }; struct grpc_pollset { - /* pollsets under posix can mutate representation as fds are added and - removed. - For example, we may choose a poll() based implementation on linux for - few fds, and an epoll() based implementation for many fds */ - const grpc_pollset_vtable *vtable; gpr_mu mu; grpc_pollset_worker root_worker; int in_flight_cbs; @@ -248,8 +243,6 @@ typedef void (*platform_become_multipoller_type)(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, struct grpc_fd **fds, size_t fd_count); -static platform_become_multipoller_type platform_become_multipoller; - /* Return 1 if the pollset has active threads in pollset_work (pollset must * be locked) */ @@ -796,8 +789,6 @@ static void kick_poller(void) { grpc_wakeup_fd_wakeup(&grpc_global_wakeup_fd); } /* main interface */ -static void become_basic_pollset(grpc_pollset *pollset, grpc_fd *fd_or_null); - static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { gpr_mu_init(&pollset->mu); *mu = &pollset->mu; @@ -809,14 +800,20 @@ static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { pollset->idle_jobs.head = pollset->idle_jobs.tail = NULL; pollset->local_wakeup_cache = NULL; pollset->kicked_without_pollers = 0; - become_basic_pollset(pollset, NULL); + pollset->data.ptr = NULL; } +/* TODO(sreek): Maybe merge multipoll_*_destroy() with pollset_destroy() + * function */ +static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset); + static void pollset_destroy(grpc_pollset *pollset) { GPR_ASSERT(pollset->in_flight_cbs == 0); GPR_ASSERT(!pollset_has_workers(pollset)); GPR_ASSERT(pollset->idle_jobs.head == pollset->idle_jobs.tail); - pollset->vtable->destroy(pollset); + + multipoll_with_epoll_pollset_destroy(pollset); + while (pollset->local_wakeup_cache) { grpc_cached_wakeup_fd *next = pollset->local_wakeup_cache->next; grpc_wakeup_fd_destroy(&pollset->local_wakeup_cache->fd); @@ -831,17 +828,24 @@ static void pollset_reset(grpc_pollset *pollset) { GPR_ASSERT(pollset->in_flight_cbs == 0); GPR_ASSERT(!pollset_has_workers(pollset)); GPR_ASSERT(pollset->idle_jobs.head == pollset->idle_jobs.tail); - pollset->vtable->destroy(pollset); + + multipoll_with_epoll_pollset_destroy(pollset); + pollset->shutting_down = 0; pollset->called_shutdown = 0; pollset->kicked_without_pollers = 0; - become_basic_pollset(pollset, NULL); } +/* TODO (sreek): Remove multipoll_with_epoll_add_fd declaration*/ +static void multipoll_with_epoll_pollset_add_fd(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset, + grpc_fd *fd, + int and_unlock_pollset); + static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { gpr_mu_lock(&pollset->mu); - pollset->vtable->add_fd(exec_ctx, pollset, fd, 1); + multipoll_with_epoll_pollset_add_fd(exec_ctx, pollset, fd, 1); /* the following (enabled only in debug) will reacquire and then release our lock - meaning that if the unlocking flag passed to add_fd above is not respected, the code will deadlock (in a way that we have a chance of @@ -852,12 +856,21 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, #endif } +/* TODO (sreek): Remove multipoll_with_epoll_finish_shutdown() declaration */ +static void multipoll_with_epoll_pollset_finish_shutdown(grpc_pollset *pollset); + static void finish_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset) { GPR_ASSERT(grpc_closure_list_empty(pollset->idle_jobs)); - pollset->vtable->finish_shutdown(pollset); + multipoll_with_epoll_pollset_finish_shutdown(pollset); grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); } +/* TODO(sreek): Remove multipoll_with_epoll_*_maybe_work_and_unlock declaration + */ +static void multipoll_with_epoll_pollset_maybe_work_and_unlock( + grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker *worker, + gpr_timespec deadline, gpr_timespec now); + static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker **worker_hdl, gpr_timespec now, gpr_timespec deadline) { @@ -915,8 +928,10 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, } gpr_tls_set(&g_current_thread_poller, (intptr_t)pollset); GPR_TIMER_BEGIN("maybe_work_and_unlock", 0); - pollset->vtable->maybe_work_and_unlock(exec_ctx, pollset, &worker, - deadline, now); + + multipoll_with_epoll_pollset_maybe_work_and_unlock( + exec_ctx, pollset, &worker, deadline, now); + GPR_TIMER_END("maybe_work_and_unlock", 0); locked = 0; gpr_tls_set(&g_current_thread_poller, 0); @@ -1013,233 +1028,6 @@ static int poll_deadline_to_millis_timeout(gpr_timespec deadline, timeout, gpr_time_from_nanos(GPR_NS_PER_MS - 1, GPR_TIMESPAN))); } -/* - * basic_pollset - a vtable that provides polling for zero or one file - * descriptor via poll() - */ - -typedef struct grpc_unary_promote_args { - const grpc_pollset_vtable *original_vtable; - grpc_pollset *pollset; - grpc_fd *fd; - grpc_closure promotion_closure; -} grpc_unary_promote_args; - -static void basic_do_promote(grpc_exec_ctx *exec_ctx, void *args, - bool success) { - grpc_unary_promote_args *up_args = args; - const grpc_pollset_vtable *original_vtable = up_args->original_vtable; - grpc_pollset *pollset = up_args->pollset; - grpc_fd *fd = up_args->fd; - - /* - * This is quite tricky. There are a number of cases to keep in mind here: - * 1. fd may have been orphaned - * 2. The pollset may no longer be a unary poller (and we can't let case #1 - * leak to other pollset types!) - * 3. pollset's fd (which may have changed) may have been orphaned - * 4. The pollset may be shutting down. - */ - - gpr_mu_lock(&pollset->mu); - /* First we need to ensure that nobody is polling concurrently */ - GPR_ASSERT(!pollset_has_workers(pollset)); - - gpr_free(up_args); - /* At this point the pollset may no longer be a unary poller. In that case - * we should just call the right add function and be done. */ - /* TODO(klempner): If we're not careful this could cause infinite recursion. - * That's not a problem for now because empty_pollset has a trivial poller - * and we don't have any mechanism to unbecome multipoller. */ - pollset->in_flight_cbs--; - if (pollset->shutting_down) { - /* We don't care about this pollset anymore. */ - if (pollset->in_flight_cbs == 0 && !pollset->called_shutdown) { - pollset->called_shutdown = 1; - finish_shutdown(exec_ctx, pollset); - } - } else if (fd_is_orphaned(fd)) { - /* Don't try to add it to anything, we'll drop our ref on it below */ - } else if (pollset->vtable != original_vtable) { - pollset->vtable->add_fd(exec_ctx, pollset, fd, 0); - } else if (fd != pollset->data.ptr) { - grpc_fd *fds[2]; - fds[0] = pollset->data.ptr; - fds[1] = fd; - - if (fds[0] && !fd_is_orphaned(fds[0])) { - platform_become_multipoller(exec_ctx, pollset, fds, GPR_ARRAY_SIZE(fds)); - GRPC_FD_UNREF(fds[0], "basicpoll"); - } else { - /* old fd is orphaned and we haven't cleaned it up until now, so remain a - * unary poller */ - /* Note that it is possible that fds[1] is also orphaned at this point. - * That's okay, we'll correct it at the next add or poll. */ - if (fds[0]) GRPC_FD_UNREF(fds[0], "basicpoll"); - pollset->data.ptr = fd; - GRPC_FD_REF(fd, "basicpoll"); - } - } - - gpr_mu_unlock(&pollset->mu); - - /* Matching ref in basic_pollset_add_fd */ - GRPC_FD_UNREF(fd, "basicpoll_add"); -} - -static void basic_pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_fd *fd, int and_unlock_pollset) { - grpc_unary_promote_args *up_args; - GPR_ASSERT(fd); - if (fd == pollset->data.ptr) goto exit; - - if (!pollset_has_workers(pollset)) { - /* Fast path -- no in flight cbs */ - /* TODO(klempner): Comment this out and fix any test failures or establish - * they are due to timing issues */ - grpc_fd *fds[2]; - fds[0] = pollset->data.ptr; - fds[1] = fd; - - if (fds[0] == NULL) { - pollset->data.ptr = fd; - GRPC_FD_REF(fd, "basicpoll"); - } else if (!fd_is_orphaned(fds[0])) { - platform_become_multipoller(exec_ctx, pollset, fds, GPR_ARRAY_SIZE(fds)); - GRPC_FD_UNREF(fds[0], "basicpoll"); - } else { - /* old fd is orphaned and we haven't cleaned it up until now, so remain a - * unary poller */ - GRPC_FD_UNREF(fds[0], "basicpoll"); - pollset->data.ptr = fd; - GRPC_FD_REF(fd, "basicpoll"); - } - goto exit; - } - - /* Now we need to promote. This needs to happen when we're not polling. Since - * this may be called from poll, the wait needs to happen asynchronously. */ - GRPC_FD_REF(fd, "basicpoll_add"); - pollset->in_flight_cbs++; - up_args = gpr_malloc(sizeof(*up_args)); - up_args->fd = fd; - up_args->original_vtable = pollset->vtable; - up_args->pollset = pollset; - up_args->promotion_closure.cb = basic_do_promote; - up_args->promotion_closure.cb_arg = up_args; - - grpc_closure_list_add(&pollset->idle_jobs, &up_args->promotion_closure, 1); - pollset_kick(pollset, GRPC_POLLSET_KICK_BROADCAST); - -exit: - if (and_unlock_pollset) { - gpr_mu_unlock(&pollset->mu); - } -} - -static void basic_pollset_maybe_work_and_unlock(grpc_exec_ctx *exec_ctx, - grpc_pollset *pollset, - grpc_pollset_worker *worker, - gpr_timespec deadline, - gpr_timespec now) { -#define POLLOUT_CHECK (POLLOUT | POLLHUP | POLLERR) -#define POLLIN_CHECK (POLLIN | POLLHUP | POLLERR) - - struct pollfd pfd[3]; - grpc_fd *fd; - grpc_fd_watcher fd_watcher; - int timeout; - int r; - nfds_t nfds; - - fd = pollset->data.ptr; - if (fd && fd_is_orphaned(fd)) { - GRPC_FD_UNREF(fd, "basicpoll"); - fd = pollset->data.ptr = NULL; - } - timeout = poll_deadline_to_millis_timeout(deadline, now); - pfd[0].fd = GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd); - pfd[0].events = POLLIN; - pfd[0].revents = 0; - pfd[1].fd = GRPC_WAKEUP_FD_GET_READ_FD(&worker->wakeup_fd->fd); - pfd[1].events = POLLIN; - pfd[1].revents = 0; - nfds = 2; - if (fd) { - pfd[2].fd = fd->fd; - pfd[2].revents = 0; - GRPC_FD_REF(fd, "basicpoll_begin"); - gpr_mu_unlock(&pollset->mu); - pfd[2].events = - (short)fd_begin_poll(fd, pollset, worker, POLLIN, POLLOUT, &fd_watcher); - if (pfd[2].events != 0) { - nfds++; - } - } else { - gpr_mu_unlock(&pollset->mu); - } - - /* TODO(vpai): Consider first doing a 0 timeout poll here to avoid - even going into the blocking annotation if possible */ - /* poll fd count (argument 2) is shortened by one if we have no events - to poll on - such that it only includes the kicker */ - GPR_TIMER_BEGIN("poll", 0); - GRPC_SCHEDULING_START_BLOCKING_REGION; - r = grpc_poll_function(pfd, nfds, timeout); - GRPC_SCHEDULING_END_BLOCKING_REGION; - GPR_TIMER_END("poll", 0); - - if (r < 0) { - if (errno != EINTR) { - gpr_log(GPR_ERROR, "poll() failed: %s", strerror(errno)); - } - if (fd) { - fd_end_poll(exec_ctx, &fd_watcher, 0, 0); - } - } else if (r == 0) { - if (fd) { - fd_end_poll(exec_ctx, &fd_watcher, 0, 0); - } - } else { - if (pfd[0].revents & POLLIN_CHECK) { - grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); - } - if (pfd[1].revents & POLLIN_CHECK) { - grpc_wakeup_fd_consume_wakeup(&worker->wakeup_fd->fd); - } - if (nfds > 2) { - fd_end_poll(exec_ctx, &fd_watcher, pfd[2].revents & POLLIN_CHECK, - pfd[2].revents & POLLOUT_CHECK); - } else if (fd) { - fd_end_poll(exec_ctx, &fd_watcher, 0, 0); - } - } - - if (fd) { - GRPC_FD_UNREF(fd, "basicpoll_begin"); - } -} - -static void basic_pollset_destroy(grpc_pollset *pollset) { - if (pollset->data.ptr != NULL) { - GRPC_FD_UNREF(pollset->data.ptr, "basicpoll"); - pollset->data.ptr = NULL; - } -} - -static const grpc_pollset_vtable basic_pollset = { - basic_pollset_add_fd, basic_pollset_maybe_work_and_unlock, - basic_pollset_destroy, basic_pollset_destroy}; - -static void become_basic_pollset(grpc_pollset *pollset, grpc_fd *fd_or_null) { - pollset->vtable = &basic_pollset; - pollset->data.ptr = fd_or_null; - if (fd_or_null != NULL) { - GRPC_FD_REF(fd_or_null, "basicpoll"); - } -} - - /******************************************************************************* * pollset_multipoller_with_epoll_posix.c */ @@ -1274,6 +1062,7 @@ static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { set_ready(exec_ctx, fd, &fd->write_closure); } +/* TODO (sreek): Maybe this global list is not required. Double check*/ struct epoll_fd_list { int *epoll_fds; size_t count; @@ -1390,10 +1179,48 @@ static void perform_delayed_add(grpc_exec_ctx *exec_ctx, void *arg, gpr_free(da); } +/* Creates an epoll fd and initializes the pollset */ +static void multipoll_with_epoll_pollset_create_efd(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset) { + epoll_hdr *h = gpr_malloc(sizeof(epoll_hdr)); + struct epoll_event ev; + int err; + + /* Ensuring that the pollset is infact empty (with no epoll fd either) */ + GPR_ASSERT(pollset->data.ptr == NULL); + + pollset->data.ptr = h; + h->epoll_fd = epoll_create1(EPOLL_CLOEXEC); + if (h->epoll_fd < 0) { + gpr_log(GPR_ERROR, "epoll_create1 failed: %s", strerror(errno)); + abort(); + } + add_epoll_fd_to_global_list(h->epoll_fd); + + ev.events = (uint32_t)(EPOLLIN | EPOLLET); + ev.data.ptr = NULL; + + /* TODO (sreek): Double-check the use of grpc_global_wakeup_fd here (right now + * I do not know why this is used. I just copied this code from + * epoll_become_mutipoller() function in ev_poll_and_epoll_posix.c file */ + err = epoll_ctl(h->epoll_fd, EPOLL_CTL_ADD, + GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), &ev); + if (err < 0) { + gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", + GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), + strerror(errno)); + } +} + static void multipoll_with_epoll_pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd, int and_unlock_pollset) { + /* If there is no epoll fd on the pollset, create one */ + if (pollset->data.ptr == NULL) { + multipoll_with_epoll_pollset_create_efd(exec_ctx, pollset); + } + if (and_unlock_pollset) { gpr_mu_unlock(&pollset->mu); finally_add_fd(exec_ctx, pollset, fd); @@ -1500,45 +1327,6 @@ static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset) { gpr_free(h); } -static const grpc_pollset_vtable multipoll_with_epoll_pollset = { - multipoll_with_epoll_pollset_add_fd, - multipoll_with_epoll_pollset_maybe_work_and_unlock, - multipoll_with_epoll_pollset_finish_shutdown, - multipoll_with_epoll_pollset_destroy}; - -static void epoll_become_multipoller(grpc_exec_ctx *exec_ctx, - grpc_pollset *pollset, grpc_fd **fds, - size_t nfds) { - size_t i; - epoll_hdr *h = gpr_malloc(sizeof(epoll_hdr)); - struct epoll_event ev; - int err; - - pollset->vtable = &multipoll_with_epoll_pollset; - pollset->data.ptr = h; - h->epoll_fd = epoll_create1(EPOLL_CLOEXEC); - if (h->epoll_fd < 0) { - /* TODO(klempner): Fall back to poll here, especially on ENOSYS */ - gpr_log(GPR_ERROR, "epoll_create1 failed: %s", strerror(errno)); - abort(); - } - add_epoll_fd_to_global_list(h->epoll_fd); - - ev.events = (uint32_t)(EPOLLIN | EPOLLET); - ev.data.ptr = NULL; - err = epoll_ctl(h->epoll_fd, EPOLL_CTL_ADD, - GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), &ev); - if (err < 0) { - gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", - GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), - strerror(errno)); - } - - for (i = 0; i < nfds; i++) { - multipoll_with_epoll_pollset_add_fd(exec_ctx, pollset, fds[i], 0); - } -} - /******************************************************************************* * pollset_set_posix.c */ @@ -1724,7 +1512,6 @@ static const grpc_event_engine_vtable vtable = { }; const grpc_event_engine_vtable *grpc_init_epoll_posix(void) { - platform_become_multipoller = epoll_become_multipoller; fd_global_init(); pollset_global_init(); return &vtable; From ab7f10ed61c7e0d104ea839206d2cd0340f5fed8 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 12 May 2016 20:21:37 -0700 Subject: [PATCH 0017/1039] Remove delayed_add --- src/core/lib/iomgr/ev_epoll_posix.c | 61 ++++++----------------------- 1 file changed, 13 insertions(+), 48 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_posix.c b/src/core/lib/iomgr/ev_epoll_posix.c index cb4a00a75c4..03c544b30c0 100644 --- a/src/core/lib/iomgr/ev_epoll_posix.c +++ b/src/core/lib/iomgr/ev_epoll_posix.c @@ -187,7 +187,7 @@ struct grpc_pollset_worker { struct grpc_pollset { gpr_mu mu; grpc_pollset_worker root_worker; - int in_flight_cbs; + int in_flight_cbs; /* TODO (sreek): Most likely this isn't needed anymore */ int shutting_down; int called_shutdown; int kicked_without_pollers; @@ -839,13 +839,12 @@ static void pollset_reset(grpc_pollset *pollset) { /* TODO (sreek): Remove multipoll_with_epoll_add_fd declaration*/ static void multipoll_with_epoll_pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_fd *fd, - int and_unlock_pollset); + grpc_fd *fd); static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { gpr_mu_lock(&pollset->mu); - multipoll_with_epoll_pollset_add_fd(exec_ctx, pollset, fd, 1); + multipoll_with_epoll_pollset_add_fd(exec_ctx, pollset, fd); /* the following (enabled only in debug) will reacquire and then release our lock - meaning that if the unlocking flag passed to add_fd above is not respected, the code will deadlock (in a way that we have a chance of @@ -1121,12 +1120,6 @@ static void remove_fd_from_all_epoll_sets(int fd) { gpr_mu_unlock(&epoll_fd_list_mu); } -typedef struct { - grpc_pollset *pollset; - grpc_fd *fd; - grpc_closure closure; -} delayed_add; - typedef struct { int epoll_fd; } epoll_hdr; static void finally_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, @@ -1139,6 +1132,13 @@ static void finally_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, /* We pretend to be polling whilst adding an fd to keep the fd from being closed during the add. This may result in a spurious wakeup being assigned to this pollset whilst adding, but that should be benign. */ + /* TODO (sreek). This fd_begin_poll() really seem to accomplish adding + * GRPC_FD_REF() (i.e adding a refcount to the fd) and checking that the + * fd is not shutting down (in which case watcher.fd will be NULL and no + * refcount is added). The ref count is added only durng hte duration of + * adding it to the epoll set (after which fd_end_poll would be called and + * the fd's ref count is decremented by 1. So do we still need fd_begin_poll + * ??? */ GPR_ASSERT(fd_begin_poll(fd, pollset, NULL, 0, 0, &watcher) == 0); if (watcher.fd != NULL) { ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); @@ -1155,30 +1155,6 @@ static void finally_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, fd_end_poll(exec_ctx, &watcher, 0, 0); } -static void perform_delayed_add(grpc_exec_ctx *exec_ctx, void *arg, - bool iomgr_status) { - delayed_add *da = arg; - - if (!fd_is_orphaned(da->fd)) { - finally_add_fd(exec_ctx, da->pollset, da->fd); - } - - gpr_mu_lock(&da->pollset->mu); - da->pollset->in_flight_cbs--; - if (da->pollset->shutting_down) { - /* We don't care about this pollset anymore. */ - if (da->pollset->in_flight_cbs == 0 && !da->pollset->called_shutdown) { - da->pollset->called_shutdown = 1; - grpc_exec_ctx_enqueue(exec_ctx, da->pollset->shutdown_done, true, NULL); - } - } - gpr_mu_unlock(&da->pollset->mu); - - GRPC_FD_UNREF(da->fd, "delayed_add"); - - gpr_free(da); -} - /* Creates an epoll fd and initializes the pollset */ static void multipoll_with_epoll_pollset_create_efd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset) { @@ -1214,25 +1190,14 @@ static void multipoll_with_epoll_pollset_create_efd(grpc_exec_ctx *exec_ctx, static void multipoll_with_epoll_pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_fd *fd, - int and_unlock_pollset) { + grpc_fd *fd) { /* If there is no epoll fd on the pollset, create one */ if (pollset->data.ptr == NULL) { multipoll_with_epoll_pollset_create_efd(exec_ctx, pollset); } - if (and_unlock_pollset) { - gpr_mu_unlock(&pollset->mu); - finally_add_fd(exec_ctx, pollset, fd); - } else { - delayed_add *da = gpr_malloc(sizeof(*da)); - da->pollset = pollset; - da->fd = fd; - GRPC_FD_REF(fd, "delayed_add"); - grpc_closure_init(&da->closure, perform_delayed_add, da); - pollset->in_flight_cbs++; - grpc_exec_ctx_enqueue(exec_ctx, &da->closure, true, NULL); - } + gpr_mu_unlock(&pollset->mu); + finally_add_fd(exec_ctx, pollset, fd); } /* TODO(klempner): We probably want to turn this down a bit */ From f6a2adf0cfa14e473e05b6cb2d6b8fdc7667945b Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 13 May 2016 16:00:20 -0700 Subject: [PATCH 0018/1039] Pollset_reset should not destroy the epoll_fd --- src/core/lib/iomgr/ev_epoll_posix.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_posix.c b/src/core/lib/iomgr/ev_epoll_posix.c index 03c544b30c0..19465280a1b 100644 --- a/src/core/lib/iomgr/ev_epoll_posix.c +++ b/src/core/lib/iomgr/ev_epoll_posix.c @@ -828,9 +828,6 @@ static void pollset_reset(grpc_pollset *pollset) { GPR_ASSERT(pollset->in_flight_cbs == 0); GPR_ASSERT(!pollset_has_workers(pollset)); GPR_ASSERT(pollset->idle_jobs.head == pollset->idle_jobs.tail); - - multipoll_with_epoll_pollset_destroy(pollset); - pollset->shutting_down = 0; pollset->called_shutdown = 0; pollset->kicked_without_pollers = 0; From 24f0f57ea16b77e710f7ff4ae8a048c888ab6b12 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 13 May 2016 19:22:44 -0700 Subject: [PATCH 0019/1039] Moving the creation of epoll_fd to pollset_init() instead of pollset_add_fd() [Verified stable. All tests pass] --- src/core/lib/iomgr/ev_epoll_posix.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_posix.c b/src/core/lib/iomgr/ev_epoll_posix.c index 19465280a1b..0ee0ee58a8b 100644 --- a/src/core/lib/iomgr/ev_epoll_posix.c +++ b/src/core/lib/iomgr/ev_epoll_posix.c @@ -787,6 +787,9 @@ static void pollset_global_shutdown(void) { static void kick_poller(void) { grpc_wakeup_fd_wakeup(&grpc_global_wakeup_fd); } +/* TODO: sreek. Try to Remove this forward declaration*/ +static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset); + /* main interface */ static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { @@ -800,7 +803,9 @@ static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { pollset->idle_jobs.head = pollset->idle_jobs.tail = NULL; pollset->local_wakeup_cache = NULL; pollset->kicked_without_pollers = 0; + pollset->data.ptr = NULL; + multipoll_with_epoll_pollset_create_efd(pollset); } /* TODO(sreek): Maybe merge multipoll_*_destroy() with pollset_destroy() @@ -1153,13 +1158,15 @@ static void finally_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, } /* Creates an epoll fd and initializes the pollset */ -static void multipoll_with_epoll_pollset_create_efd(grpc_exec_ctx *exec_ctx, - grpc_pollset *pollset) { +/* TODO: This has to be called ONLY from pollset_init function. and hence it + * does not acquire any lock */ +static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset) { epoll_hdr *h = gpr_malloc(sizeof(epoll_hdr)); struct epoll_event ev; int err; - /* Ensuring that the pollset is infact empty (with no epoll fd either) */ + /* TODO (sreek). remove this assert. Currently added this just to ensure that + * we do not overwrite h->epoll_fd without freeing the older one*/ GPR_ASSERT(pollset->data.ptr == NULL); pollset->data.ptr = h; @@ -1188,10 +1195,10 @@ static void multipoll_with_epoll_pollset_create_efd(grpc_exec_ctx *exec_ctx, static void multipoll_with_epoll_pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { - /* If there is no epoll fd on the pollset, create one */ - if (pollset->data.ptr == NULL) { - multipoll_with_epoll_pollset_create_efd(exec_ctx, pollset); - } + GPR_ASSERT(pollset->data.ptr != NULL); + + /* TODO(sreek). Remove this unlock code (and also the code that acquires the + * lock before calling multipoll_with_epoll_add_fd() function */ gpr_mu_unlock(&pollset->mu); finally_add_fd(exec_ctx, pollset, fd); From 9ff57f67a0920e8e39baedfec5671fb6e662d257 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Sat, 14 May 2016 15:56:57 -0700 Subject: [PATCH 0020/1039] Remove idle_jobs and in_flight_cbs from pollset --- src/core/lib/iomgr/ev_epoll_posix.c | 50 ++--------------------------- 1 file changed, 3 insertions(+), 47 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_posix.c b/src/core/lib/iomgr/ev_epoll_posix.c index 0ee0ee58a8b..5776b6c1cf1 100644 --- a/src/core/lib/iomgr/ev_epoll_posix.c +++ b/src/core/lib/iomgr/ev_epoll_posix.c @@ -169,8 +169,6 @@ static void fd_global_shutdown(void); * pollset declarations */ -typedef struct grpc_pollset_vtable grpc_pollset_vtable; - typedef struct grpc_cached_wakeup_fd { grpc_wakeup_fd fd; struct grpc_cached_wakeup_fd *next; @@ -187,12 +185,10 @@ struct grpc_pollset_worker { struct grpc_pollset { gpr_mu mu; grpc_pollset_worker root_worker; - int in_flight_cbs; /* TODO (sreek): Most likely this isn't needed anymore */ int shutting_down; int called_shutdown; int kicked_without_pollers; grpc_closure *shutdown_done; - grpc_closure_list idle_jobs; union { int fd; void *ptr; @@ -201,16 +197,6 @@ struct grpc_pollset { grpc_cached_wakeup_fd *local_wakeup_cache; }; -struct grpc_pollset_vtable { - void (*add_fd)(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - struct grpc_fd *fd, int and_unlock_pollset); - void (*maybe_work_and_unlock)(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_pollset_worker *worker, - gpr_timespec deadline, gpr_timespec now); - void (*finish_shutdown)(grpc_pollset *pollset); - void (*destroy)(grpc_pollset *pollset); -}; - /* Add an fd to a pollset */ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, struct grpc_fd *fd); @@ -796,11 +782,9 @@ static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { gpr_mu_init(&pollset->mu); *mu = &pollset->mu; pollset->root_worker.next = pollset->root_worker.prev = &pollset->root_worker; - pollset->in_flight_cbs = 0; pollset->shutting_down = 0; pollset->called_shutdown = 0; pollset->kicked_without_pollers = 0; - pollset->idle_jobs.head = pollset->idle_jobs.tail = NULL; pollset->local_wakeup_cache = NULL; pollset->kicked_without_pollers = 0; @@ -813,9 +797,7 @@ static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset); static void pollset_destroy(grpc_pollset *pollset) { - GPR_ASSERT(pollset->in_flight_cbs == 0); GPR_ASSERT(!pollset_has_workers(pollset)); - GPR_ASSERT(pollset->idle_jobs.head == pollset->idle_jobs.tail); multipoll_with_epoll_pollset_destroy(pollset); @@ -830,9 +812,7 @@ static void pollset_destroy(grpc_pollset *pollset) { static void pollset_reset(grpc_pollset *pollset) { GPR_ASSERT(pollset->shutting_down); - GPR_ASSERT(pollset->in_flight_cbs == 0); GPR_ASSERT(!pollset_has_workers(pollset)); - GPR_ASSERT(pollset->idle_jobs.head == pollset->idle_jobs.tail); pollset->shutting_down = 0; pollset->called_shutdown = 0; pollset->kicked_without_pollers = 0; @@ -861,7 +841,6 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, static void multipoll_with_epoll_pollset_finish_shutdown(grpc_pollset *pollset); static void finish_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset) { - GPR_ASSERT(grpc_closure_list_empty(pollset->idle_jobs)); multipoll_with_epoll_pollset_finish_shutdown(pollset); grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); } @@ -895,26 +874,11 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_wakeup_fd_init(&worker.wakeup_fd->fd); } worker.kicked_specifically = 0; - /* If there's work waiting for the pollset to be idle, and the - pollset is idle, then do that work */ - if (!pollset_has_workers(pollset) && - !grpc_closure_list_empty(pollset->idle_jobs)) { - GPR_TIMER_MARK("pollset_work.idle_jobs", 0); - grpc_exec_ctx_enqueue_list(exec_ctx, &pollset->idle_jobs, NULL); - goto done; - } /* If we're shutting down then we don't execute any extended work */ if (pollset->shutting_down) { GPR_TIMER_MARK("pollset_work.shutting_down", 0); goto done; } - /* Give do_promote priority so we don't starve it out */ - if (pollset->in_flight_cbs) { - GPR_TIMER_MARK("pollset_work.in_flight_cbs", 0); - gpr_mu_unlock(&pollset->mu); - locked = 0; - goto done; - } /* Start polling, and keep doing so while we're being asked to re-evaluate our pollers (this allows poll() based pollers to ensure they don't miss wakeups) */ @@ -975,7 +939,7 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, if (pollset->shutting_down) { if (pollset_has_workers(pollset)) { pollset_kick(pollset, NULL); - } else if (!pollset->called_shutdown && pollset->in_flight_cbs == 0) { + } else if (!pollset->called_shutdown) { pollset->called_shutdown = 1; gpr_mu_unlock(&pollset->mu); finish_shutdown(exec_ctx, pollset); @@ -985,11 +949,6 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, * pollset_work. * TODO(dklempner): Can we refactor the shutdown logic to avoid this? */ gpr_mu_lock(&pollset->mu); - } else if (!grpc_closure_list_empty(pollset->idle_jobs)) { - grpc_exec_ctx_enqueue_list(exec_ctx, &pollset->idle_jobs, NULL); - gpr_mu_unlock(&pollset->mu); - grpc_exec_ctx_flush(exec_ctx); - gpr_mu_lock(&pollset->mu); } } *worker_hdl = NULL; @@ -1002,11 +961,8 @@ static void pollset_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, pollset->shutting_down = 1; pollset->shutdown_done = closure; pollset_kick(pollset, GRPC_POLLSET_KICK_BROADCAST); - if (!pollset_has_workers(pollset)) { - grpc_exec_ctx_enqueue_list(exec_ctx, &pollset->idle_jobs, NULL); - } - if (!pollset->called_shutdown && pollset->in_flight_cbs == 0 && - !pollset_has_workers(pollset)) { + + if (!pollset->called_shutdown && !pollset_has_workers(pollset)) { pollset->called_shutdown = 1; finish_shutdown(exec_ctx, pollset); } From 97c2d6a269472a8a6e56d9e3d88f89bd27aff5ac Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Sat, 14 May 2016 16:33:16 -0700 Subject: [PATCH 0021/1039] Remove grpc_fd_watcher and related code from ev_epoll_posix.c --- src/core/lib/iomgr/ev_epoll_posix.c | 271 ++++------------------------ 1 file changed, 38 insertions(+), 233 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_posix.c b/src/core/lib/iomgr/ev_epoll_posix.c index 5776b6c1cf1..920be1951fa 100644 --- a/src/core/lib/iomgr/ev_epoll_posix.c +++ b/src/core/lib/iomgr/ev_epoll_posix.c @@ -59,17 +59,6 @@ * FD declarations */ -/* TODO(sreek) : Check if grpc_fd_watcher is needed (and if so, check if we can - * share this between ev_poll_posix.h and ev_epoll_posix versions */ - -typedef struct grpc_fd_watcher { - struct grpc_fd_watcher *next; - struct grpc_fd_watcher *prev; - grpc_pollset *pollset; - grpc_pollset_worker *worker; - grpc_fd *fd; -} grpc_fd_watcher; - struct grpc_fd { int fd; /* refst format: @@ -84,32 +73,6 @@ struct grpc_fd { int closed; int released; - /* The watcher list. - - The following watcher related fields are protected by watcher_mu. - - An fd_watcher is an ephemeral object created when an fd wants to - begin polling, and destroyed after the poll. - - It denotes the fd's interest in whether to read poll or write poll - or both or neither on this fd. - - If a watcher is asked to poll for reads or writes, the read_watcher - or write_watcher fields are set respectively. A watcher may be asked - to poll for both, in which case both fields will be set. - - read_watcher and write_watcher may be NULL if no watcher has been - asked to poll for reads or writes. - - If an fd_watcher is not asked to poll for reads or writes, it's added - to a linked list of inactive watchers, rooted at inactive_watcher_root. - If at a later time there becomes need of a poller to poll, one of - the inactive pollers may be kicked out of their poll loops to take - that responsibility. */ - grpc_fd_watcher inactive_watcher_root; - grpc_fd_watcher *read_watcher; - grpc_fd_watcher *write_watcher; - grpc_closure *read_closure; grpc_closure *write_closure; @@ -120,27 +83,6 @@ struct grpc_fd { grpc_iomgr_object iomgr_object; }; -/* Begin polling on an fd. - Registers that the given pollset is interested in this fd - so that if read - or writability interest changes, the pollset can be kicked to pick up that - new interest. - Return value is: - (fd_needs_read? read_mask : 0) | (fd_needs_write? write_mask : 0) - i.e. a combination of read_mask and write_mask determined by the fd's current - interest in said events. - Polling strategies that do not need to alter their behavior depending on the - fd's current interest (such as epoll) do not need to call this function. - MUST NOT be called with a pollset lock taken */ -static uint32_t fd_begin_poll(grpc_fd *fd, grpc_pollset *pollset, - grpc_pollset_worker *worker, uint32_t read_mask, - uint32_t write_mask, grpc_fd_watcher *rec); -/* Complete polling previously started with fd_begin_poll - MUST NOT be called with a pollset lock taken - if got_read or got_write are 1, also does the become_{readable,writable} as - appropriate. */ -static void fd_end_poll(grpc_exec_ctx *exec_ctx, grpc_fd_watcher *rec, - int got_read, int got_write); - /* Return 1 if this fd is orphaned, 0 otherwise */ static bool fd_is_orphaned(grpc_fd *fd); @@ -307,10 +249,7 @@ static grpc_fd *alloc_fd(int fd) { r->read_closure = CLOSURE_NOT_READY; r->write_closure = CLOSURE_NOT_READY; r->fd = fd; - r->inactive_watcher_root.next = r->inactive_watcher_root.prev = - &r->inactive_watcher_root; r->freelist_next = NULL; - r->read_watcher = r->write_watcher = NULL; r->on_done_closure = NULL; r->closed = 0; r->released = 0; @@ -387,43 +326,6 @@ static bool fd_is_orphaned(grpc_fd *fd) { return (gpr_atm_acq_load(&fd->refst) & 1) == 0; } -static void pollset_kick_locked(grpc_fd_watcher *watcher) { - gpr_mu_lock(&watcher->pollset->mu); - GPR_ASSERT(watcher->worker); - pollset_kick_ext(watcher->pollset, watcher->worker, - GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP); - gpr_mu_unlock(&watcher->pollset->mu); -} - -static void maybe_wake_one_watcher_locked(grpc_fd *fd) { - if (fd->inactive_watcher_root.next != &fd->inactive_watcher_root) { - pollset_kick_locked(fd->inactive_watcher_root.next); - } else if (fd->read_watcher) { - pollset_kick_locked(fd->read_watcher); - } else if (fd->write_watcher) { - pollset_kick_locked(fd->write_watcher); - } -} - -static void wake_all_watchers_locked(grpc_fd *fd) { - grpc_fd_watcher *watcher; - for (watcher = fd->inactive_watcher_root.next; - watcher != &fd->inactive_watcher_root; watcher = watcher->next) { - pollset_kick_locked(watcher); - } - if (fd->read_watcher) { - pollset_kick_locked(fd->read_watcher); - } - if (fd->write_watcher && fd->write_watcher != fd->read_watcher) { - pollset_kick_locked(fd->write_watcher); - } -} - -static int has_watchers(grpc_fd *fd) { - return fd->read_watcher != NULL || fd->write_watcher != NULL || - fd->inactive_watcher_root.next != &fd->inactive_watcher_root; -} - static void close_fd_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { fd->closed = 1; if (!fd->released) { @@ -454,11 +356,7 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, } gpr_mu_lock(&fd->mu); REF_BY(fd, 1, reason); /* remove active status, but keep referenced */ - if (!has_watchers(fd)) { - close_fd_locked(exec_ctx, fd); - } else { - wake_all_watchers_locked(fd); - } + close_fd_locked(exec_ctx, fd); gpr_mu_unlock(&fd->mu); UNREF_BY(fd, 2, reason); /* drop the reference */ } @@ -489,7 +387,6 @@ static void notify_on_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, /* already ready ==> queue the closure to run immediately */ *st = CLOSURE_NOT_READY; grpc_exec_ctx_enqueue(exec_ctx, closure, !fd->shutdown, NULL); - maybe_wake_one_watcher_locked(fd); } else { /* upcallptr was set to a different closure. This is an error! */ gpr_log(GPR_ERROR, @@ -540,111 +437,6 @@ static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, gpr_mu_unlock(&fd->mu); } -static uint32_t fd_begin_poll(grpc_fd *fd, grpc_pollset *pollset, - grpc_pollset_worker *worker, uint32_t read_mask, - uint32_t write_mask, grpc_fd_watcher *watcher) { - uint32_t mask = 0; - grpc_closure *cur; - int requested; - /* keep track of pollers that have requested our events, in case they change - */ - GRPC_FD_REF(fd, "poll"); - - gpr_mu_lock(&fd->mu); - - /* if we are shutdown, then don't add to the watcher set */ - if (fd->shutdown) { - watcher->fd = NULL; - watcher->pollset = NULL; - watcher->worker = NULL; - gpr_mu_unlock(&fd->mu); - GRPC_FD_UNREF(fd, "poll"); - return 0; - } - - /* if there is nobody polling for read, but we need to, then start doing so */ - cur = fd->read_closure; - requested = cur != CLOSURE_READY; - if (read_mask && fd->read_watcher == NULL && requested) { - fd->read_watcher = watcher; - mask |= read_mask; - } - /* if there is nobody polling for write, but we need to, then start doing so - */ - cur = fd->write_closure; - requested = cur != CLOSURE_READY; - if (write_mask && fd->write_watcher == NULL && requested) { - fd->write_watcher = watcher; - mask |= write_mask; - } - /* if not polling, remember this watcher in case we need someone to later */ - if (mask == 0 && worker != NULL) { - watcher->next = &fd->inactive_watcher_root; - watcher->prev = watcher->next->prev; - watcher->next->prev = watcher->prev->next = watcher; - } - watcher->pollset = pollset; - watcher->worker = worker; - watcher->fd = fd; - gpr_mu_unlock(&fd->mu); - - return mask; -} - -static void fd_end_poll(grpc_exec_ctx *exec_ctx, grpc_fd_watcher *watcher, - int got_read, int got_write) { - int was_polling = 0; - int kick = 0; - grpc_fd *fd = watcher->fd; - - if (fd == NULL) { - return; - } - - gpr_mu_lock(&fd->mu); - - if (watcher == fd->read_watcher) { - /* remove read watcher, kick if we still need a read */ - was_polling = 1; - if (!got_read) { - kick = 1; - } - fd->read_watcher = NULL; - } - if (watcher == fd->write_watcher) { - /* remove write watcher, kick if we still need a write */ - was_polling = 1; - if (!got_write) { - kick = 1; - } - fd->write_watcher = NULL; - } - if (!was_polling && watcher->worker != NULL) { - /* remove from inactive list */ - watcher->next->prev = watcher->prev; - watcher->prev->next = watcher->next; - } - if (got_read) { - if (set_ready_locked(exec_ctx, fd, &fd->read_closure)) { - kick = 1; - } - } - if (got_write) { - if (set_ready_locked(exec_ctx, fd, &fd->write_closure)) { - kick = 1; - } - } - if (kick) { - maybe_wake_one_watcher_locked(fd); - } - if (fd_is_orphaned(fd) && !has_watchers(fd) && !fd->closed) { - close_fd_locked(exec_ctx, fd); - } - gpr_mu_unlock(&fd->mu); - - GRPC_FD_UNREF(fd, "poll"); -} - /******************************************************************************* * pollset_posix.c */ @@ -1085,32 +877,45 @@ static void finally_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, epoll_hdr *h = pollset->data.ptr; struct epoll_event ev; int err; - grpc_fd_watcher watcher; - - /* We pretend to be polling whilst adding an fd to keep the fd from being - closed during the add. This may result in a spurious wakeup being assigned - to this pollset whilst adding, but that should be benign. */ - /* TODO (sreek). This fd_begin_poll() really seem to accomplish adding - * GRPC_FD_REF() (i.e adding a refcount to the fd) and checking that the - * fd is not shutting down (in which case watcher.fd will be NULL and no - * refcount is added). The ref count is added only durng hte duration of - * adding it to the epoll set (after which fd_end_poll would be called and - * the fd's ref count is decremented by 1. So do we still need fd_begin_poll - * ??? */ - GPR_ASSERT(fd_begin_poll(fd, pollset, NULL, 0, 0, &watcher) == 0); - if (watcher.fd != NULL) { - ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); - ev.data.ptr = fd; - err = epoll_ctl(h->epoll_fd, EPOLL_CTL_ADD, fd->fd, &ev); - if (err < 0) { - /* FDs may be added to a pollset multiple times, so EEXIST is normal. */ - if (errno != EEXIST) { - gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", fd->fd, - strerror(errno)); - } + + /* Hold a ref to the fd to keep it from being closed during the add. This may + result in a spurious wakeup being assigned to this pollset whilst adding, + but that should be benign. */ + /* TODO: (sreek): Understand how a spurious wake up migh be assinged to this + * pollset..and how holding a reference will prevent the fd from being closed + * (and perhaps more importantly, see how can an fd be closed while being + * added to the epollset */ + GRPC_FD_REF(fd, "add fd"); + + gpr_mu_lock(&fd->mu); + if (fd->shutdown) { + gpr_mu_unlock(&fd->mu); + GRPC_FD_UNREF(fd, "add fd"); + return; + } + gpr_mu_unlock(&fd->mu); + + ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); + ev.data.ptr = fd; + err = epoll_ctl(h->epoll_fd, EPOLL_CTL_ADD, fd->fd, &ev); + if (err < 0) { + /* FDs may be added to a pollset multiple times, so EEXIST is normal. */ + if (errno != EEXIST) { + gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", fd->fd, + strerror(errno)); } } - fd_end_poll(exec_ctx, &watcher, 0, 0); + + /* The fd might have been orphaned while we were adding it to the epoll set. + Close the fd in such a case (which will also take care of removing it from + the epoll set */ + gpr_mu_lock(&fd->mu); + if (fd_is_orphaned(fd) && !fd->closed) { + close_fd_locked(exec_ctx, fd); + } + gpr_mu_unlock(&fd->mu); + + GRPC_FD_UNREF(fd, "add fd"); } /* Creates an epoll fd and initializes the pollset */ From e9ee1f34b8efdc47ea12821351e5cc23125d62b2 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Sat, 14 May 2016 17:22:10 -0700 Subject: [PATCH 0022/1039] Minor refactor of add_fd path --- src/core/lib/iomgr/ev_epoll_posix.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_posix.c b/src/core/lib/iomgr/ev_epoll_posix.c index 920be1951fa..15126b3b62a 100644 --- a/src/core/lib/iomgr/ev_epoll_posix.c +++ b/src/core/lib/iomgr/ev_epoll_posix.c @@ -617,16 +617,13 @@ static void multipoll_with_epoll_pollset_add_fd(grpc_exec_ctx *exec_ctx, static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { + /* TODO (sreek) - Does reading pollset->data.ptr need pollset->mu lock ? + * because finally_add_fd() also reads it but without the lock! */ gpr_mu_lock(&pollset->mu); - multipoll_with_epoll_pollset_add_fd(exec_ctx, pollset, fd); -/* the following (enabled only in debug) will reacquire and then release - our lock - meaning that if the unlocking flag passed to add_fd above is - not respected, the code will deadlock (in a way that we have a chance of - debugging) */ -#ifndef NDEBUG - gpr_mu_lock(&pollset->mu); + GPR_ASSERT(pollset->data.ptr != NULL); gpr_mu_unlock(&pollset->mu); -#endif + + multipoll_with_epoll_pollset_add_fd(exec_ctx, pollset, fd); } /* TODO (sreek): Remove multipoll_with_epoll_finish_shutdown() declaration */ @@ -874,6 +871,8 @@ typedef struct { int epoll_fd; } epoll_hdr; static void finally_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { + /*TODO: (sree) Shouldn't this read (pollset->data.ptr) be done under a + pollset lock - i.e pollset->mu ? */ epoll_hdr *h = pollset->data.ptr; struct epoll_event ev; int err; @@ -941,9 +940,6 @@ static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset) { ev.events = (uint32_t)(EPOLLIN | EPOLLET); ev.data.ptr = NULL; - /* TODO (sreek): Double-check the use of grpc_global_wakeup_fd here (right now - * I do not know why this is used. I just copied this code from - * epoll_become_mutipoller() function in ev_poll_and_epoll_posix.c file */ err = epoll_ctl(h->epoll_fd, EPOLL_CTL_ADD, GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), &ev); if (err < 0) { @@ -956,12 +952,6 @@ static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset) { static void multipoll_with_epoll_pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { - GPR_ASSERT(pollset->data.ptr != NULL); - - /* TODO(sreek). Remove this unlock code (and also the code that acquires the - * lock before calling multipoll_with_epoll_add_fd() function */ - - gpr_mu_unlock(&pollset->mu); finally_add_fd(exec_ctx, pollset, fd); } From c22eb5ac4dbf44209f0d431b6d1e9267210e0120 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Sat, 14 May 2016 19:22:36 -0700 Subject: [PATCH 0023/1039] Add epoll polling strategy to run_tests.py --- tools/run_tests/run_tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index 0538dce4198..ddaf96c345a 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -892,7 +892,7 @@ for l in languages: language_make_options=[] if any(language.make_options() for language in languages): - if not 'gcov' in args.config and len(languages) != 1: + if len(languages) != 1: print 'languages with custom make options cannot be built simultaneously with other languages' sys.exit(1) else: From 2ea165911b23099499da0af48088c47c47d659ba Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Tue, 17 May 2016 09:37:48 -0700 Subject: [PATCH 0024/1039] experiment with signals --- src/core/lib/iomgr/ev_epoll_posix.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/core/lib/iomgr/ev_epoll_posix.c b/src/core/lib/iomgr/ev_epoll_posix.c index 15126b3b62a..4481bab4380 100644 --- a/src/core/lib/iomgr/ev_epoll_posix.c +++ b/src/core/lib/iomgr/ev_epoll_posix.c @@ -40,6 +40,7 @@ #include #include #include +#include #include #include #include @@ -120,6 +121,7 @@ struct grpc_pollset_worker { grpc_cached_wakeup_fd *wakeup_fd; int reevaluate_polling_on_wakeup; int kicked_specifically; + pthread_t pt_id; struct grpc_pollset_worker *next; struct grpc_pollset_worker *prev; }; @@ -506,6 +508,8 @@ static void pollset_kick_ext(grpc_pollset *p, } specific_worker->kicked_specifically = 1; grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + /* TODO (sreek): Refactor this into a separate file*/ + pthread_kill(specific_worker->pt_id, SIGUSR1); } else if ((flags & GRPC_POLLSET_CAN_KICK_SELF) != 0) { GPR_TIMER_MARK("kick_yoself", 0); if ((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) != 0) { @@ -551,10 +555,15 @@ static void pollset_kick(grpc_pollset *p, /* global state management */ +static void sig_handler(int sig_num) { + gpr_log(GPR_INFO, "Received signal %d", sig_num); +} + static void pollset_global_init(void) { gpr_tls_init(&g_current_thread_poller); gpr_tls_init(&g_current_thread_worker); grpc_wakeup_fd_init(&grpc_global_wakeup_fd); + signal(SIGUSR1, sig_handler); } static void pollset_global_shutdown(void) { @@ -663,6 +672,9 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_wakeup_fd_init(&worker.wakeup_fd->fd); } worker.kicked_specifically = 0; + + /* TODO(sreek): Abstract this thread id stuff out into a separate file */ + worker.pt_id = pthread_self(); /* If we're shutting down then we don't execute any extended work */ if (pollset->shutting_down) { GPR_TIMER_MARK("pollset_work.shutting_down", 0); From f448c34a6839f75476900a4a2b24b2160fe4d164 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 19 May 2016 10:51:24 -0700 Subject: [PATCH 0025/1039] Remove union { } data and epoll_hdr structures. Added ev_epoll_linux files --- BUILD | 6 + Makefile | 2 + binding.gyp | 1 + build.yaml | 2 + config.m4 | 1 + gRPC.podspec | 3 + grpc.gemspec | 2 + package.xml | 2 + src/core/lib/iomgr/ev_epoll_linux.c | 1335 +++++++++++++++++ src/core/lib/iomgr/ev_epoll_linux.h | 41 + src/core/lib/iomgr/ev_epoll_posix.c | 87 +- src/core/lib/iomgr/ev_posix.c | 7 +- src/python/grpcio/grpc_core_dependencies.py | 1 + tools/doxygen/Doxyfile.core.internal | 2 + tools/run_tests/sources_and_headers.json | 3 + vsprojects/vcxproj/grpc/grpc.vcxproj | 3 + vsprojects/vcxproj/grpc/grpc.vcxproj.filters | 6 + .../grpc_unsecure/grpc_unsecure.vcxproj | 3 + .../grpc_unsecure.vcxproj.filters | 6 + 19 files changed, 1442 insertions(+), 71 deletions(-) create mode 100644 src/core/lib/iomgr/ev_epoll_linux.c create mode 100644 src/core/lib/iomgr/ev_epoll_linux.h diff --git a/BUILD b/BUILD index 0be8f27a01b..a32352ebb30 100644 --- a/BUILD +++ b/BUILD @@ -178,6 +178,7 @@ cc_library( "src/core/lib/iomgr/closure.h", "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", + "src/core/lib/iomgr/ev_epoll_linux.h", "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", @@ -322,6 +323,7 @@ cc_library( "src/core/lib/iomgr/endpoint.c", "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", + "src/core/lib/iomgr/ev_epoll_linux.c", "src/core/lib/iomgr/ev_epoll_posix.c", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_posix.c", @@ -548,6 +550,7 @@ cc_library( "src/core/lib/iomgr/closure.h", "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", + "src/core/lib/iomgr/ev_epoll_linux.h", "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", @@ -669,6 +672,7 @@ cc_library( "src/core/lib/iomgr/endpoint.c", "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", + "src/core/lib/iomgr/ev_epoll_linux.c", "src/core/lib/iomgr/ev_epoll_posix.c", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_posix.c", @@ -1362,6 +1366,7 @@ objc_library( "src/core/lib/iomgr/endpoint.c", "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", + "src/core/lib/iomgr/ev_epoll_linux.c", "src/core/lib/iomgr/ev_epoll_posix.c", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_posix.c", @@ -1567,6 +1572,7 @@ objc_library( "src/core/lib/iomgr/closure.h", "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", + "src/core/lib/iomgr/ev_epoll_linux.h", "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", diff --git a/Makefile b/Makefile index 29ebc0e5adf..063698d943f 100644 --- a/Makefile +++ b/Makefile @@ -2486,6 +2486,7 @@ LIBGRPC_SRC = \ src/core/lib/iomgr/endpoint.c \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ + src/core/lib/iomgr/ev_epoll_linux.c \ src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ @@ -2841,6 +2842,7 @@ LIBGRPC_UNSECURE_SRC = \ src/core/lib/iomgr/endpoint.c \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ + src/core/lib/iomgr/ev_epoll_linux.c \ src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ diff --git a/binding.gyp b/binding.gyp index 89774ead4da..41e1b5bb416 100644 --- a/binding.gyp +++ b/binding.gyp @@ -581,6 +581,7 @@ 'src/core/lib/iomgr/endpoint.c', 'src/core/lib/iomgr/endpoint_pair_posix.c', 'src/core/lib/iomgr/endpoint_pair_windows.c', + 'src/core/lib/iomgr/ev_epoll_linux.c', 'src/core/lib/iomgr/ev_epoll_posix.c', 'src/core/lib/iomgr/ev_poll_posix.c', 'src/core/lib/iomgr/ev_posix.c', diff --git a/build.yaml b/build.yaml index 7ba65332972..2f3d07071da 100644 --- a/build.yaml +++ b/build.yaml @@ -165,6 +165,7 @@ filegroups: - src/core/lib/iomgr/closure.h - src/core/lib/iomgr/endpoint.h - src/core/lib/iomgr/endpoint_pair.h + - src/core/lib/iomgr/ev_epoll_linux.h - src/core/lib/iomgr/ev_epoll_posix.h - src/core/lib/iomgr/ev_poll_posix.h - src/core/lib/iomgr/ev_posix.h @@ -240,6 +241,7 @@ filegroups: - src/core/lib/iomgr/endpoint.c - src/core/lib/iomgr/endpoint_pair_posix.c - src/core/lib/iomgr/endpoint_pair_windows.c + - src/core/lib/iomgr/ev_epoll_linux.c - src/core/lib/iomgr/ev_epoll_posix.c - src/core/lib/iomgr/ev_poll_posix.c - src/core/lib/iomgr/ev_posix.c diff --git a/config.m4 b/config.m4 index 6987c741541..4308295afda 100644 --- a/config.m4 +++ b/config.m4 @@ -100,6 +100,7 @@ if test "$PHP_GRPC" != "no"; then src/core/lib/iomgr/endpoint.c \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ + src/core/lib/iomgr/ev_epoll_linux.c \ src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ diff --git a/gRPC.podspec b/gRPC.podspec index 3b4dd52380e..de55880125a 100644 --- a/gRPC.podspec +++ b/gRPC.podspec @@ -181,6 +181,7 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/closure.h', 'src/core/lib/iomgr/endpoint.h', 'src/core/lib/iomgr/endpoint_pair.h', + 'src/core/lib/iomgr/ev_epoll_linux.h', 'src/core/lib/iomgr/ev_epoll_posix.h', 'src/core/lib/iomgr/ev_poll_posix.h', 'src/core/lib/iomgr/ev_posix.h', @@ -359,6 +360,7 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/endpoint.c', 'src/core/lib/iomgr/endpoint_pair_posix.c', 'src/core/lib/iomgr/endpoint_pair_windows.c', + 'src/core/lib/iomgr/ev_epoll_linux.c', 'src/core/lib/iomgr/ev_epoll_posix.c', 'src/core/lib/iomgr/ev_poll_posix.c', 'src/core/lib/iomgr/ev_posix.c', @@ -548,6 +550,7 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/closure.h', 'src/core/lib/iomgr/endpoint.h', 'src/core/lib/iomgr/endpoint_pair.h', + 'src/core/lib/iomgr/ev_epoll_linux.h', 'src/core/lib/iomgr/ev_epoll_posix.h', 'src/core/lib/iomgr/ev_poll_posix.h', 'src/core/lib/iomgr/ev_posix.h', diff --git a/grpc.gemspec b/grpc.gemspec index 71cccb6ca8d..54ae2eb68dd 100755 --- a/grpc.gemspec +++ b/grpc.gemspec @@ -190,6 +190,7 @@ Gem::Specification.new do |s| s.files += %w( src/core/lib/iomgr/closure.h ) s.files += %w( src/core/lib/iomgr/endpoint.h ) s.files += %w( src/core/lib/iomgr/endpoint_pair.h ) + s.files += %w( src/core/lib/iomgr/ev_epoll_linux.h ) s.files += %w( src/core/lib/iomgr/ev_epoll_posix.h ) s.files += %w( src/core/lib/iomgr/ev_poll_posix.h ) s.files += %w( src/core/lib/iomgr/ev_posix.h ) @@ -338,6 +339,7 @@ Gem::Specification.new do |s| s.files += %w( src/core/lib/iomgr/endpoint.c ) s.files += %w( src/core/lib/iomgr/endpoint_pair_posix.c ) s.files += %w( src/core/lib/iomgr/endpoint_pair_windows.c ) + s.files += %w( src/core/lib/iomgr/ev_epoll_linux.c ) s.files += %w( src/core/lib/iomgr/ev_epoll_posix.c ) s.files += %w( src/core/lib/iomgr/ev_poll_posix.c ) s.files += %w( src/core/lib/iomgr/ev_posix.c ) diff --git a/package.xml b/package.xml index 0fc5d0dee47..d8e82a8bc37 100644 --- a/package.xml +++ b/package.xml @@ -197,6 +197,7 @@ + @@ -345,6 +346,7 @@ + diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c new file mode 100644 index 00000000000..f257ac8a1dd --- /dev/null +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -0,0 +1,1335 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include + +#ifdef GPR_POSIX_SOCKET + +#include "src/core/lib/iomgr/ev_epoll_posix.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include "src/core/lib/iomgr/ev_posix.h" +#include "src/core/lib/iomgr/iomgr_internal.h" +#include "src/core/lib/iomgr/wakeup_fd_posix.h" +#include "src/core/lib/profiling/timers.h" +#include "src/core/lib/support/block_annotate.h" + +struct polling_island; + +/******************************************************************************* + * FD declarations + */ + +struct grpc_fd { + int fd; + /* refst format: + bit0: 1=active/0=orphaned + bit1-n: refcount + meaning that mostly we ref by two to avoid altering the orphaned bit, + and just unref by 1 when we're ready to flag the object as orphaned */ + gpr_atm refst; + + gpr_mu mu; + int shutdown; + int closed; + int released; + + grpc_closure *read_closure; + grpc_closure *write_closure; + + /* Mutex protecting the 'polling_island' field */ + gpr_mu pi_mu; + + /* The polling island to which this fd belongs to. An fd belongs to exactly + one polling island */ + struct polling_island *polling_island; + + struct grpc_fd *freelist_next; + + grpc_closure *on_done_closure; + + grpc_iomgr_object iomgr_object; +}; + +/* Return 1 if this fd is orphaned, 0 otherwise */ +static bool fd_is_orphaned(grpc_fd *fd); + +/* Reference counting for fds */ +/*#define GRPC_FD_REF_COUNT_DEBUG*/ +#ifdef GRPC_FD_REF_COUNT_DEBUG +static void fd_ref(grpc_fd *fd, const char *reason, const char *file, int line); +static void fd_unref(grpc_fd *fd, const char *reason, const char *file, + int line); +#define GRPC_FD_REF(fd, reason) fd_ref(fd, reason, __FILE__, __LINE__) +#define GRPC_FD_UNREF(fd, reason) fd_unref(fd, reason, __FILE__, __LINE__) +#else +static void fd_ref(grpc_fd *fd); +static void fd_unref(grpc_fd *fd); +#define GRPC_FD_REF(fd, reason) fd_ref(fd) +#define GRPC_FD_UNREF(fd, reason) fd_unref(fd) +#endif + +static void fd_global_init(void); +static void fd_global_shutdown(void); + +#define CLOSURE_NOT_READY ((grpc_closure *)0) +#define CLOSURE_READY ((grpc_closure *)1) + +/******************************************************************************* + * Polling Island + */ +typedef struct polling_island { + gpr_mu mu; + int ref_cnt; + + /* Pointer to the polling_island this merged into. If this is not NULL, all + the remaining fields in this pollset (i.e all fields except mu and ref_cnt) + are considered invalid and must be ignored */ + struct polling_island *merged_to; + + /* The fd of the underlying epoll set */ + int epoll_fd; + + /* The file descriptors in the epoll set */ + size_t fd_cnt; + size_t fd_capacity; + grpc_fd **fds; + + /* Polling islands that are no longer needed are kept in a freelist so that + they can be reused. This field points to the next polling island in the + free list. Note that this is only used if the polling island is in the + free list */ + struct polling_island *next_free; +} polling_island; + +/* Polling island freelist */ +static gpr_mu g_pi_freelist_mu; +static polling_island *g_pi_freelist = NULL; + +/* TODO: sreek - Should we hold a lock on fd or add a ref to the fd ? */ +static void add_fd_to_polling_island_locked(polling_island *pi, grpc_fd *fd) { + int err; + struct epoll_event ev; + + ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); + ev.data.ptr = fd; + err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_ADD, fd->fd, &ev); + + if (err < 0 && errno != EEXIST) { + gpr_log(GPR_ERROR, "epoll_ctl add for fd: %d failed with error: %s", fd->fd, + strerror(errno)); + return; + } + + pi->fd_capacity = GPR_MAX(pi->fd_capacity + 8, pi->fd_cnt * 3 / 2); + pi->fds = gpr_realloc(pi->fds, sizeof(grpc_fd *) * pi->fd_capacity); + pi->fds[pi->fd_cnt++] = fd; +} + +static polling_island *polling_island_create(int initial_ref_cnt, + grpc_fd *initial_fd) { + polling_island *pi = NULL; + gpr_mu_lock(&g_pi_freelist_mu); + if (g_pi_freelist != NULL) { + pi = g_pi_freelist; + g_pi_freelist = g_pi_freelist->next_free; + pi->next_free = NULL; + } + gpr_mu_unlock(&g_pi_freelist_mu); + + /* Create new polling island if we could not get one from the free list */ + if (pi == NULL) { + pi = gpr_malloc(sizeof(*pi)); + gpr_mu_init(&pi->mu); + pi->fd_cnt = 0; + pi->fd_capacity = 0; + pi->fds = NULL; + + pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); + if (pi->epoll_fd < 0) { + gpr_log(GPR_ERROR, "epoll_create1() failed with error: %s", + strerror(errno)); + } + GPR_ASSERT(pi->epoll_fd >= 0); + } + + pi->ref_cnt = initial_ref_cnt; + pi->merged_to = NULL; + pi->next_free = NULL; + + if (initial_fd != NULL) { + /* add_fd_to_polling_island_locked() expects the caller to hold a pi->mu + * lock. However, since this is a new polling island (and no one has a + * reference to it yet), it is okay to not acquire pi->mu here */ + add_fd_to_polling_island_locked(pi, initial_fd); + } + + return pi; +} + +static void polling_island_global_init() { + polling_island_create(0, NULL); /* TODO(sreek): Delete this line */ + gpr_mu_init(&g_pi_freelist_mu); + g_pi_freelist = NULL; +} + +/******************************************************************************* + * pollset declarations + */ + +typedef struct grpc_cached_wakeup_fd { + grpc_wakeup_fd fd; + struct grpc_cached_wakeup_fd *next; +} grpc_cached_wakeup_fd; + +struct grpc_pollset_worker { + grpc_cached_wakeup_fd *wakeup_fd; + int reevaluate_polling_on_wakeup; + int kicked_specifically; + pthread_t pt_id; + struct grpc_pollset_worker *next; + struct grpc_pollset_worker *prev; +}; + +struct grpc_pollset { + gpr_mu mu; + grpc_pollset_worker root_worker; + int shutting_down; + int called_shutdown; + int kicked_without_pollers; + grpc_closure *shutdown_done; + + int epoll_fd; + + /* Mutex protecting the 'polling_island' field */ + gpr_mu pi_mu; + + /* The polling island to which this fd belongs to. An fd belongs to exactly + one polling island */ + struct polling_island *polling_island; + + /* Local cache of eventfds for workers */ + grpc_cached_wakeup_fd *local_wakeup_cache; +}; + +/* Add an fd to a pollset */ +static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + struct grpc_fd *fd); + +static void pollset_set_add_fd(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, grpc_fd *fd); + +/* Convert a timespec to milliseconds: + - very small or negative poll times are clamped to zero to do a + non-blocking poll (which becomes spin polling) + - other small values are rounded up to one millisecond + - longer than a millisecond polls are rounded up to the next nearest + millisecond to avoid spinning + - infinite timeouts are converted to -1 */ +static int poll_deadline_to_millis_timeout(gpr_timespec deadline, + gpr_timespec now); + +/* Allow kick to wakeup the currently polling worker */ +#define GRPC_POLLSET_CAN_KICK_SELF 1 +/* Force the wakee to repoll when awoken */ +#define GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP 2 +/* As per pollset_kick, with an extended set of flags (defined above) + -- mostly for fd_posix's use. */ +static void pollset_kick_ext(grpc_pollset *p, + grpc_pollset_worker *specific_worker, + uint32_t flags); + +/* turn a pollset into a multipoller: platform specific */ +typedef void (*platform_become_multipoller_type)(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset, + struct grpc_fd **fds, + size_t fd_count); + +/* Return 1 if the pollset has active threads in pollset_work (pollset must + * be locked) */ +static int pollset_has_workers(grpc_pollset *pollset); + +static void remove_fd_from_all_epoll_sets(int fd); + +/******************************************************************************* + * pollset_set definitions + */ + +struct grpc_pollset_set { + gpr_mu mu; + + size_t pollset_count; + size_t pollset_capacity; + grpc_pollset **pollsets; + + size_t pollset_set_count; + size_t pollset_set_capacity; + struct grpc_pollset_set **pollset_sets; + + size_t fd_count; + size_t fd_capacity; + grpc_fd **fds; +}; + +/******************************************************************************* + * fd_posix.c + */ + +/* We need to keep a freelist not because of any concerns of malloc performance + * but instead so that implementations with multiple threads in (for example) + * epoll_wait deal with the race between pollset removal and incoming poll + * notifications. + * + * The problem is that the poller ultimately holds a reference to this + * object, so it is very difficult to know when is safe to free it, at least + * without some expensive synchronization. + * + * If we keep the object freelisted, in the worst case losing this race just + * becomes a spurious read notification on a reused fd. + */ +/* TODO(klempner): We could use some form of polling generation count to know + * when these are safe to free. */ +/* TODO(klempner): Consider disabling freelisting if we don't have multiple + * threads in poll on the same fd */ +/* TODO(klempner): Batch these allocations to reduce fragmentation */ +static grpc_fd *fd_freelist = NULL; +static gpr_mu fd_freelist_mu; + +static void freelist_fd(grpc_fd *fd) { + gpr_mu_lock(&fd_freelist_mu); + fd->freelist_next = fd_freelist; + fd_freelist = fd; + grpc_iomgr_unregister_object(&fd->iomgr_object); + gpr_mu_unlock(&fd_freelist_mu); +} + +static grpc_fd *alloc_fd(int fd) { + grpc_fd *r = NULL; + + gpr_mu_lock(&fd_freelist_mu); + if (fd_freelist != NULL) { + r = fd_freelist; + fd_freelist = fd_freelist->freelist_next; + } + gpr_mu_unlock(&fd_freelist_mu); + + if (r == NULL) { + r = gpr_malloc(sizeof(grpc_fd)); + gpr_mu_init(&r->mu); + gpr_mu_init(&r->pi_mu); + } + + /* TODO: sreek - check with ctiller on why we need to acquire a lock here */ + gpr_mu_lock(&r->mu); + gpr_atm_rel_store(&r->refst, 1); + r->shutdown = 0; + r->read_closure = CLOSURE_NOT_READY; + r->write_closure = CLOSURE_NOT_READY; + r->fd = fd; + r->polling_island = NULL; + r->freelist_next = NULL; + r->on_done_closure = NULL; + r->closed = 0; + r->released = 0; + gpr_mu_unlock(&r->mu); + return r; +} + +static void destroy(grpc_fd *fd) { + gpr_mu_destroy(&fd->mu); + gpr_free(fd); +} + +#ifdef GRPC_FD_REF_COUNT_DEBUG +#define REF_BY(fd, n, reason) ref_by(fd, n, reason, __FILE__, __LINE__) +#define UNREF_BY(fd, n, reason) unref_by(fd, n, reason, __FILE__, __LINE__) +static void ref_by(grpc_fd *fd, int n, const char *reason, const char *file, + int line) { + gpr_log(GPR_DEBUG, "FD %d %p ref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, + gpr_atm_no_barrier_load(&fd->refst), + gpr_atm_no_barrier_load(&fd->refst) + n, reason, file, line); +#else +#define REF_BY(fd, n, reason) ref_by(fd, n) +#define UNREF_BY(fd, n, reason) unref_by(fd, n) +static void ref_by(grpc_fd *fd, int n) { +#endif + GPR_ASSERT(gpr_atm_no_barrier_fetch_add(&fd->refst, n) > 0); +} + +#ifdef GRPC_FD_REF_COUNT_DEBUG +static void unref_by(grpc_fd *fd, int n, const char *reason, const char *file, + int line) { + gpr_atm old; + gpr_log(GPR_DEBUG, "FD %d %p unref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, + gpr_atm_no_barrier_load(&fd->refst), + gpr_atm_no_barrier_load(&fd->refst) - n, reason, file, line); +#else +static void unref_by(grpc_fd *fd, int n) { + gpr_atm old; +#endif + old = gpr_atm_full_fetch_add(&fd->refst, -n); + if (old == n) { + freelist_fd(fd); + } else { + GPR_ASSERT(old > n); + } +} + +static void fd_global_init(void) { gpr_mu_init(&fd_freelist_mu); } + +static void fd_global_shutdown(void) { + gpr_mu_lock(&fd_freelist_mu); + gpr_mu_unlock(&fd_freelist_mu); + while (fd_freelist != NULL) { + grpc_fd *fd = fd_freelist; + fd_freelist = fd_freelist->freelist_next; + destroy(fd); + } + gpr_mu_destroy(&fd_freelist_mu); +} + +static grpc_fd *fd_create(int fd, const char *name) { + grpc_fd *r = alloc_fd(fd); + char *name2; + gpr_asprintf(&name2, "%s fd=%d", name, fd); + grpc_iomgr_register_object(&r->iomgr_object, name2); + gpr_free(name2); +#ifdef GRPC_FD_REF_COUNT_DEBUG + gpr_log(GPR_DEBUG, "FD %d %p create %s", fd, r, name); +#endif + return r; +} + +static bool fd_is_orphaned(grpc_fd *fd) { + return (gpr_atm_acq_load(&fd->refst) & 1) == 0; +} + +static void close_fd_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { + fd->closed = 1; + if (!fd->released) { + close(fd->fd); + } else { + remove_fd_from_all_epoll_sets(fd->fd); + } + grpc_exec_ctx_enqueue(exec_ctx, fd->on_done_closure, true, NULL); +} + +static int fd_wrapped_fd(grpc_fd *fd) { + if (fd->released || fd->closed) { + return -1; + } else { + return fd->fd; + } +} + +/* TODO: sreek - do something here with the pollset island link */ +static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure *on_done, int *release_fd, + const char *reason) { + fd->on_done_closure = on_done; + fd->released = release_fd != NULL; + if (!fd->released) { + shutdown(fd->fd, SHUT_RDWR); + } else { + *release_fd = fd->fd; + } + gpr_mu_lock(&fd->mu); + REF_BY(fd, 1, reason); /* remove active status, but keep referenced */ + close_fd_locked(exec_ctx, fd); + gpr_mu_unlock(&fd->mu); + UNREF_BY(fd, 2, reason); /* drop the reference */ +} + +/* increment refcount by two to avoid changing the orphan bit */ +#ifdef GRPC_FD_REF_COUNT_DEBUG +static void fd_ref(grpc_fd *fd, const char *reason, const char *file, + int line) { + ref_by(fd, 2, reason, file, line); +} + +static void fd_unref(grpc_fd *fd, const char *reason, const char *file, + int line) { + unref_by(fd, 2, reason, file, line); +} +#else +static void fd_ref(grpc_fd *fd) { ref_by(fd, 2); } + +static void fd_unref(grpc_fd *fd) { unref_by(fd, 2); } +#endif + +static void notify_on_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure **st, grpc_closure *closure) { + if (*st == CLOSURE_NOT_READY) { + /* not ready ==> switch to a waiting state by setting the closure */ + *st = closure; + } else if (*st == CLOSURE_READY) { + /* already ready ==> queue the closure to run immediately */ + *st = CLOSURE_NOT_READY; + grpc_exec_ctx_enqueue(exec_ctx, closure, !fd->shutdown, NULL); + } else { + /* upcallptr was set to a different closure. This is an error! */ + gpr_log(GPR_ERROR, + "User called a notify_on function with a previous callback still " + "pending"); + abort(); + } +} + +/* returns 1 if state becomes not ready */ +static int set_ready_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure **st) { + if (*st == CLOSURE_READY) { + /* duplicate ready ==> ignore */ + return 0; + } else if (*st == CLOSURE_NOT_READY) { + /* not ready, and not waiting ==> flag ready */ + *st = CLOSURE_READY; + return 0; + } else { + /* waiting ==> queue closure */ + grpc_exec_ctx_enqueue(exec_ctx, *st, !fd->shutdown, NULL); + *st = CLOSURE_NOT_READY; + return 1; + } +} + +/* Do something here with the pollset island link (?) */ +static void fd_shutdown(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { + gpr_mu_lock(&fd->mu); + GPR_ASSERT(!fd->shutdown); + fd->shutdown = 1; + set_ready_locked(exec_ctx, fd, &fd->read_closure); + set_ready_locked(exec_ctx, fd, &fd->write_closure); + gpr_mu_unlock(&fd->mu); +} + +static void fd_notify_on_read(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure *closure) { + gpr_mu_lock(&fd->mu); + notify_on_locked(exec_ctx, fd, &fd->read_closure, closure); + gpr_mu_unlock(&fd->mu); +} + +static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure *closure) { + gpr_mu_lock(&fd->mu); + notify_on_locked(exec_ctx, fd, &fd->write_closure, closure); + gpr_mu_unlock(&fd->mu); +} + +/******************************************************************************* + * pollset_posix.c + */ + +GPR_TLS_DECL(g_current_thread_poller); +GPR_TLS_DECL(g_current_thread_worker); + +/** The alarm system needs to be able to wakeup 'some poller' sometimes + * (specifically when a new alarm needs to be triggered earlier than the next + * alarm 'epoch'). + * This wakeup_fd gives us something to alert on when such a case occurs. */ +grpc_wakeup_fd grpc_global_wakeup_fd; + +static void remove_worker(grpc_pollset *p, grpc_pollset_worker *worker) { + worker->prev->next = worker->next; + worker->next->prev = worker->prev; +} + +static int pollset_has_workers(grpc_pollset *p) { + return p->root_worker.next != &p->root_worker; +} + +static grpc_pollset_worker *pop_front_worker(grpc_pollset *p) { + if (pollset_has_workers(p)) { + grpc_pollset_worker *w = p->root_worker.next; + remove_worker(p, w); + return w; + } else { + return NULL; + } +} + +static void push_back_worker(grpc_pollset *p, grpc_pollset_worker *worker) { + worker->next = &p->root_worker; + worker->prev = worker->next->prev; + worker->prev->next = worker->next->prev = worker; +} + +static void push_front_worker(grpc_pollset *p, grpc_pollset_worker *worker) { + worker->prev = &p->root_worker; + worker->next = worker->prev->next; + worker->prev->next = worker->next->prev = worker; +} + +static void pollset_kick_ext(grpc_pollset *p, + grpc_pollset_worker *specific_worker, + uint32_t flags) { + GPR_TIMER_BEGIN("pollset_kick_ext", 0); + + /* pollset->mu already held */ + if (specific_worker != NULL) { + if (specific_worker == GRPC_POLLSET_KICK_BROADCAST) { + GPR_TIMER_BEGIN("pollset_kick_ext.broadcast", 0); + GPR_ASSERT((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) == 0); + for (specific_worker = p->root_worker.next; + specific_worker != &p->root_worker; + specific_worker = specific_worker->next) { + grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + } + p->kicked_without_pollers = 1; + GPR_TIMER_END("pollset_kick_ext.broadcast", 0); + } else if (gpr_tls_get(&g_current_thread_worker) != + (intptr_t)specific_worker) { + GPR_TIMER_MARK("different_thread_worker", 0); + if ((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) != 0) { + specific_worker->reevaluate_polling_on_wakeup = 1; + } + specific_worker->kicked_specifically = 1; + grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + /* TODO (sreek): Refactor this into a separate file*/ + pthread_kill(specific_worker->pt_id, SIGUSR1); + } else if ((flags & GRPC_POLLSET_CAN_KICK_SELF) != 0) { + GPR_TIMER_MARK("kick_yoself", 0); + if ((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) != 0) { + specific_worker->reevaluate_polling_on_wakeup = 1; + } + specific_worker->kicked_specifically = 1; + grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + } + } else if (gpr_tls_get(&g_current_thread_poller) != (intptr_t)p) { + GPR_ASSERT((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) == 0); + GPR_TIMER_MARK("kick_anonymous", 0); + specific_worker = pop_front_worker(p); + if (specific_worker != NULL) { + if (gpr_tls_get(&g_current_thread_worker) == (intptr_t)specific_worker) { + GPR_TIMER_MARK("kick_anonymous_not_self", 0); + push_back_worker(p, specific_worker); + specific_worker = pop_front_worker(p); + if ((flags & GRPC_POLLSET_CAN_KICK_SELF) == 0 && + gpr_tls_get(&g_current_thread_worker) == + (intptr_t)specific_worker) { + push_back_worker(p, specific_worker); + specific_worker = NULL; + } + } + if (specific_worker != NULL) { + GPR_TIMER_MARK("finally_kick", 0); + push_back_worker(p, specific_worker); + grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + } + } else { + GPR_TIMER_MARK("kicked_no_pollers", 0); + p->kicked_without_pollers = 1; + } + } + + GPR_TIMER_END("pollset_kick_ext", 0); +} + +static void pollset_kick(grpc_pollset *p, + grpc_pollset_worker *specific_worker) { + pollset_kick_ext(p, specific_worker, 0); +} + +/* global state management */ + +static void sig_handler(int sig_num) { + gpr_log(GPR_INFO, "Received signal %d", sig_num); +} + +static void pollset_global_init(void) { + gpr_tls_init(&g_current_thread_poller); + gpr_tls_init(&g_current_thread_worker); + grpc_wakeup_fd_init(&grpc_global_wakeup_fd); + signal(SIGUSR1, sig_handler); +} + +static void pollset_global_shutdown(void) { + grpc_wakeup_fd_destroy(&grpc_global_wakeup_fd); + gpr_tls_destroy(&g_current_thread_poller); + gpr_tls_destroy(&g_current_thread_worker); +} + +static void kick_poller(void) { grpc_wakeup_fd_wakeup(&grpc_global_wakeup_fd); } + +/* TODO: sreek. Try to Remove this forward declaration*/ +static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset); + +/* main interface */ + +static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { + gpr_mu_init(&pollset->mu); + *mu = &pollset->mu; + pollset->root_worker.next = pollset->root_worker.prev = &pollset->root_worker; + gpr_mu_init(&pollset->pi_mu); + pollset->polling_island = NULL; + pollset->shutting_down = 0; + pollset->called_shutdown = 0; + pollset->kicked_without_pollers = 0; + pollset->local_wakeup_cache = NULL; + pollset->kicked_without_pollers = 0; + + multipoll_with_epoll_pollset_create_efd(pollset); +} + +/* TODO(sreek): Maybe merge multipoll_*_destroy() with pollset_destroy() + * function */ +static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset); + +static void pollset_destroy(grpc_pollset *pollset) { + GPR_ASSERT(!pollset_has_workers(pollset)); + + multipoll_with_epoll_pollset_destroy(pollset); + + while (pollset->local_wakeup_cache) { + grpc_cached_wakeup_fd *next = pollset->local_wakeup_cache->next; + grpc_wakeup_fd_destroy(&pollset->local_wakeup_cache->fd); + gpr_free(pollset->local_wakeup_cache); + pollset->local_wakeup_cache = next; + } + gpr_mu_destroy(&pollset->pi_mu); + gpr_mu_destroy(&pollset->mu); +} + +/* TODO(sreek) - Do something with the pollset island link (??) */ +static void pollset_reset(grpc_pollset *pollset) { + GPR_ASSERT(pollset->shutting_down); + GPR_ASSERT(!pollset_has_workers(pollset)); + pollset->shutting_down = 0; + pollset->called_shutdown = 0; + pollset->kicked_without_pollers = 0; +} + +/* TODO (sreek): Remove multipoll_with_epoll_finish_shutdown() declaration */ +static void multipoll_with_epoll_pollset_finish_shutdown(grpc_pollset *pollset); + +static void finish_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset) { + multipoll_with_epoll_pollset_finish_shutdown(pollset); + grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); +} + +/* TODO(sreek): Remove multipoll_with_epoll_*_maybe_work_and_unlock declaration + */ +static void multipoll_with_epoll_pollset_maybe_work_and_unlock( + grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker *worker, + gpr_timespec deadline, gpr_timespec now); + +static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_pollset_worker **worker_hdl, gpr_timespec now, + gpr_timespec deadline) { + grpc_pollset_worker worker; + *worker_hdl = &worker; + + /* pollset->mu already held */ + int added_worker = 0; + int locked = 1; + int queued_work = 0; + int keep_polling = 0; + GPR_TIMER_BEGIN("pollset_work", 0); + /* this must happen before we (potentially) drop pollset->mu */ + worker.next = worker.prev = NULL; + worker.reevaluate_polling_on_wakeup = 0; + if (pollset->local_wakeup_cache != NULL) { + worker.wakeup_fd = pollset->local_wakeup_cache; + pollset->local_wakeup_cache = worker.wakeup_fd->next; + } else { + worker.wakeup_fd = gpr_malloc(sizeof(*worker.wakeup_fd)); + grpc_wakeup_fd_init(&worker.wakeup_fd->fd); + } + worker.kicked_specifically = 0; + + /* TODO(sreek): Abstract this thread id stuff out into a separate file */ + worker.pt_id = pthread_self(); + /* If we're shutting down then we don't execute any extended work */ + if (pollset->shutting_down) { + GPR_TIMER_MARK("pollset_work.shutting_down", 0); + goto done; + } + /* Start polling, and keep doing so while we're being asked to + re-evaluate our pollers (this allows poll() based pollers to + ensure they don't miss wakeups) */ + keep_polling = 1; + while (keep_polling) { + keep_polling = 0; + if (!pollset->kicked_without_pollers) { + if (!added_worker) { + push_front_worker(pollset, &worker); + added_worker = 1; + gpr_tls_set(&g_current_thread_worker, (intptr_t)&worker); + } + gpr_tls_set(&g_current_thread_poller, (intptr_t)pollset); + GPR_TIMER_BEGIN("maybe_work_and_unlock", 0); + + multipoll_with_epoll_pollset_maybe_work_and_unlock( + exec_ctx, pollset, &worker, deadline, now); + + GPR_TIMER_END("maybe_work_and_unlock", 0); + locked = 0; + gpr_tls_set(&g_current_thread_poller, 0); + } else { + GPR_TIMER_MARK("pollset_work.kicked_without_pollers", 0); + pollset->kicked_without_pollers = 0; + } + /* Finished execution - start cleaning up. + Note that we may arrive here from outside the enclosing while() loop. + In that case we won't loop though as we haven't added worker to the + worker list, which means nobody could ask us to re-evaluate polling). */ + done: + if (!locked) { + queued_work |= grpc_exec_ctx_flush(exec_ctx); + gpr_mu_lock(&pollset->mu); + locked = 1; + } + /* If we're forced to re-evaluate polling (via pollset_kick with + GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) then we land here and force + a loop */ + if (worker.reevaluate_polling_on_wakeup) { + worker.reevaluate_polling_on_wakeup = 0; + pollset->kicked_without_pollers = 0; + if (queued_work || worker.kicked_specifically) { + /* If there's queued work on the list, then set the deadline to be + immediate so we get back out of the polling loop quickly */ + deadline = gpr_inf_past(GPR_CLOCK_MONOTONIC); + } + keep_polling = 1; + } + } + if (added_worker) { + remove_worker(pollset, &worker); + gpr_tls_set(&g_current_thread_worker, 0); + } + /* release wakeup fd to the local pool */ + worker.wakeup_fd->next = pollset->local_wakeup_cache; + pollset->local_wakeup_cache = worker.wakeup_fd; + /* check shutdown conditions */ + if (pollset->shutting_down) { + if (pollset_has_workers(pollset)) { + pollset_kick(pollset, NULL); + } else if (!pollset->called_shutdown) { + pollset->called_shutdown = 1; + gpr_mu_unlock(&pollset->mu); + finish_shutdown(exec_ctx, pollset); + grpc_exec_ctx_flush(exec_ctx); + /* Continuing to access pollset here is safe -- it is the caller's + * responsibility to not destroy when it has outstanding calls to + * pollset_work. + * TODO(dklempner): Can we refactor the shutdown logic to avoid this? */ + gpr_mu_lock(&pollset->mu); + } + } + *worker_hdl = NULL; + GPR_TIMER_END("pollset_work", 0); +} + +/* TODO: (sreek) Do something with the pollset island link */ +static void pollset_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_closure *closure) { + GPR_ASSERT(!pollset->shutting_down); + pollset->shutting_down = 1; + pollset->shutdown_done = closure; + pollset_kick(pollset, GRPC_POLLSET_KICK_BROADCAST); + + if (!pollset->called_shutdown && !pollset_has_workers(pollset)) { + pollset->called_shutdown = 1; + finish_shutdown(exec_ctx, pollset); + } +} + +static int poll_deadline_to_millis_timeout(gpr_timespec deadline, + gpr_timespec now) { + gpr_timespec timeout; + static const int64_t max_spin_polling_us = 10; + if (gpr_time_cmp(deadline, gpr_inf_future(deadline.clock_type)) == 0) { + return -1; + } + if (gpr_time_cmp(deadline, gpr_time_add(now, gpr_time_from_micros( + max_spin_polling_us, + GPR_TIMESPAN))) <= 0) { + return 0; + } + timeout = gpr_time_sub(deadline, now); + return gpr_time_to_millis(gpr_time_add( + timeout, gpr_time_from_nanos(GPR_NS_PER_MS - 1, GPR_TIMESPAN))); +} + +/******************************************************************************* + * pollset_multipoller_with_epoll_posix.c + */ + +static void set_ready(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure **st) { + /* only one set_ready can be active at once (but there may be a racing + notify_on) */ + gpr_mu_lock(&fd->mu); + set_ready_locked(exec_ctx, fd, st); + gpr_mu_unlock(&fd->mu); +} + +static void fd_become_readable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { + set_ready(exec_ctx, fd, &fd->read_closure); +} + +static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { + set_ready(exec_ctx, fd, &fd->write_closure); +} + +/* TODO (sreek): Maybe this global list is not required. Double check*/ +struct epoll_fd_list { + int *epoll_fds; + size_t count; + size_t capacity; +}; + +static struct epoll_fd_list epoll_fd_global_list; +static gpr_once init_epoll_fd_list_mu = GPR_ONCE_INIT; +static gpr_mu epoll_fd_list_mu; + +static void init_mu(void) { gpr_mu_init(&epoll_fd_list_mu); } + +static void add_epoll_fd_to_global_list(int epoll_fd) { + gpr_once_init(&init_epoll_fd_list_mu, init_mu); + + gpr_mu_lock(&epoll_fd_list_mu); + if (epoll_fd_global_list.count == epoll_fd_global_list.capacity) { + epoll_fd_global_list.capacity = + GPR_MAX((size_t)8, epoll_fd_global_list.capacity * 2); + epoll_fd_global_list.epoll_fds = + gpr_realloc(epoll_fd_global_list.epoll_fds, + epoll_fd_global_list.capacity * sizeof(int)); + } + epoll_fd_global_list.epoll_fds[epoll_fd_global_list.count++] = epoll_fd; + gpr_mu_unlock(&epoll_fd_list_mu); +} + +static void remove_epoll_fd_from_global_list(int epoll_fd) { + gpr_mu_lock(&epoll_fd_list_mu); + GPR_ASSERT(epoll_fd_global_list.count > 0); + for (size_t i = 0; i < epoll_fd_global_list.count; i++) { + if (epoll_fd == epoll_fd_global_list.epoll_fds[i]) { + epoll_fd_global_list.epoll_fds[i] = + epoll_fd_global_list.epoll_fds[--(epoll_fd_global_list.count)]; + break; + } + } + gpr_mu_unlock(&epoll_fd_list_mu); +} + +static void remove_fd_from_all_epoll_sets(int fd) { + int err; + gpr_once_init(&init_epoll_fd_list_mu, init_mu); + gpr_mu_lock(&epoll_fd_list_mu); + if (epoll_fd_global_list.count == 0) { + gpr_mu_unlock(&epoll_fd_list_mu); + return; + } + for (size_t i = 0; i < epoll_fd_global_list.count; i++) { + err = epoll_ctl(epoll_fd_global_list.epoll_fds[i], EPOLL_CTL_DEL, fd, NULL); + if (err < 0 && errno != ENOENT) { + gpr_log(GPR_ERROR, "epoll_ctl del for %d failed: %s", fd, + strerror(errno)); + } + } + gpr_mu_unlock(&epoll_fd_list_mu); +} + +/* TODO: sreek - This function multipoll_with_epoll_pollset_add_fd() and + * finally_add_fd() in ev_poll_and_epoll_posix.c */ +static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_fd *fd) { + + /* TODO sreek - Check if we need to get a pollset->mu lock here */ + + struct epoll_event ev; + int err; + + /* Hold a ref to the fd to keep it from being closed during the add. This may + result in a spurious wakeup being assigned to this pollset whilst adding, + but that should be benign. */ + /* TODO: (sreek): Understand how a spurious wake up migh be assinged to this + * pollset..and how holding a reference will prevent the fd from being closed + * (and perhaps more importantly, see how can an fd be closed while being + * added to the epollset */ + GRPC_FD_REF(fd, "add fd"); + + gpr_mu_lock(&fd->mu); + if (fd->shutdown) { + gpr_mu_unlock(&fd->mu); + GRPC_FD_UNREF(fd, "add fd"); + return; + } + gpr_mu_unlock(&fd->mu); + + ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); + ev.data.ptr = fd; + err = epoll_ctl(pollset->epoll_fd, EPOLL_CTL_ADD, fd->fd, &ev); + if (err < 0) { + /* FDs may be added to a pollset multiple times, so EEXIST is normal. */ + if (errno != EEXIST) { + gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", fd->fd, + strerror(errno)); + } + } + + /* The fd might have been orphaned while we were adding it to the epoll set. + Close the fd in such a case (which will also take care of removing it from + the epoll set */ + gpr_mu_lock(&fd->mu); + if (fd_is_orphaned(fd) && !fd->closed) { + close_fd_locked(exec_ctx, fd); + } + gpr_mu_unlock(&fd->mu); + + GRPC_FD_UNREF(fd, "add fd"); +} + +/* Creates an epoll fd and initializes the pollset */ +/* TODO: This has to be called ONLY from pollset_init function. and hence it + * does not acquire any lock */ +static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset) { + struct epoll_event ev; + int err; + + pollset->epoll_fd = epoll_create1(EPOLL_CLOEXEC); + if (pollset->epoll_fd < 0) { + gpr_log(GPR_ERROR, "epoll_create1 failed: %s", strerror(errno)); + abort(); + } + add_epoll_fd_to_global_list(pollset->epoll_fd); + + ev.events = (uint32_t)(EPOLLIN | EPOLLET); + ev.data.ptr = NULL; + + err = epoll_ctl(pollset->epoll_fd, EPOLL_CTL_ADD, + GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), &ev); + if (err < 0) { + gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", + GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), + strerror(errno)); + } +} + +/* TODO(klempner): We probably want to turn this down a bit */ +#define GRPC_EPOLL_MAX_EVENTS 1000 + +static void multipoll_with_epoll_pollset_maybe_work_and_unlock( + grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker *worker, + gpr_timespec deadline, gpr_timespec now) { + struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; + int epoll_fd = pollset->epoll_fd; + int ep_rv; + int poll_rv; + int timeout_ms; + struct pollfd pfds[2]; + + /* If you want to ignore epoll's ability to sanely handle parallel pollers, + * for a more apples-to-apples performance comparison with poll, add a + * if (pollset->counter != 0) { return 0; } + * here. + */ + + gpr_mu_unlock(&pollset->mu); + + timeout_ms = poll_deadline_to_millis_timeout(deadline, now); + + pfds[0].fd = GRPC_WAKEUP_FD_GET_READ_FD(&worker->wakeup_fd->fd); + pfds[0].events = POLLIN; + pfds[0].revents = 0; + pfds[1].fd = epoll_fd; + pfds[1].events = POLLIN; + pfds[1].revents = 0; + + /* TODO(vpai): Consider first doing a 0 timeout poll here to avoid + even going into the blocking annotation if possible */ + GPR_TIMER_BEGIN("poll", 0); + GRPC_SCHEDULING_START_BLOCKING_REGION; + poll_rv = grpc_poll_function(pfds, 2, timeout_ms); + GRPC_SCHEDULING_END_BLOCKING_REGION; + GPR_TIMER_END("poll", 0); + + if (poll_rv < 0) { + if (errno != EINTR) { + gpr_log(GPR_ERROR, "poll() failed: %s", strerror(errno)); + } + } else if (poll_rv == 0) { + /* do nothing */ + } else { + if (pfds[0].revents) { + grpc_wakeup_fd_consume_wakeup(&worker->wakeup_fd->fd); + } + if (pfds[1].revents) { + do { + /* The following epoll_wait never blocks; it has a timeout of 0 */ + ep_rv = epoll_wait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, 0); + if (ep_rv < 0) { + if (errno != EINTR) { + gpr_log(GPR_ERROR, "epoll_wait() failed: %s", strerror(errno)); + } + } else { + int i; + for (i = 0; i < ep_rv; ++i) { + grpc_fd *fd = ep_ev[i].data.ptr; + /* TODO(klempner): We might want to consider making err and pri + * separate events */ + int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); + int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); + int write_ev = ep_ev[i].events & EPOLLOUT; + if (fd == NULL) { + grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); + } else { + if (read_ev || cancel) { + fd_become_readable(exec_ctx, fd); + } + if (write_ev || cancel) { + fd_become_writable(exec_ctx, fd); + } + } + } + } + } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); + } + } +} + +static void multipoll_with_epoll_pollset_finish_shutdown( + grpc_pollset *pollset) {} + +static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset) { + close(pollset->epoll_fd); + remove_epoll_fd_from_global_list(pollset->epoll_fd); +} + +/******************************************************************************* + * pollset_set_posix.c + */ + +static grpc_pollset_set *pollset_set_create(void) { + grpc_pollset_set *pollset_set = gpr_malloc(sizeof(*pollset_set)); + memset(pollset_set, 0, sizeof(*pollset_set)); + gpr_mu_init(&pollset_set->mu); + return pollset_set; +} + +static void pollset_set_destroy(grpc_pollset_set *pollset_set) { + size_t i; + gpr_mu_destroy(&pollset_set->mu); + for (i = 0; i < pollset_set->fd_count; i++) { + GRPC_FD_UNREF(pollset_set->fds[i], "pollset_set"); + } + gpr_free(pollset_set->pollsets); + gpr_free(pollset_set->pollset_sets); + gpr_free(pollset_set->fds); + gpr_free(pollset_set); +} + +static void pollset_set_add_pollset(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, + grpc_pollset *pollset) { + size_t i, j; + gpr_mu_lock(&pollset_set->mu); + if (pollset_set->pollset_count == pollset_set->pollset_capacity) { + pollset_set->pollset_capacity = + GPR_MAX(8, 2 * pollset_set->pollset_capacity); + pollset_set->pollsets = + gpr_realloc(pollset_set->pollsets, pollset_set->pollset_capacity * + sizeof(*pollset_set->pollsets)); + } + pollset_set->pollsets[pollset_set->pollset_count++] = pollset; + for (i = 0, j = 0; i < pollset_set->fd_count; i++) { + if (fd_is_orphaned(pollset_set->fds[i])) { + GRPC_FD_UNREF(pollset_set->fds[i], "pollset_set"); + } else { + pollset_add_fd(exec_ctx, pollset, pollset_set->fds[i]); + pollset_set->fds[j++] = pollset_set->fds[i]; + } + } + pollset_set->fd_count = j; + gpr_mu_unlock(&pollset_set->mu); +} + +static void pollset_set_del_pollset(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, + grpc_pollset *pollset) { + size_t i; + gpr_mu_lock(&pollset_set->mu); + for (i = 0; i < pollset_set->pollset_count; i++) { + if (pollset_set->pollsets[i] == pollset) { + pollset_set->pollset_count--; + GPR_SWAP(grpc_pollset *, pollset_set->pollsets[i], + pollset_set->pollsets[pollset_set->pollset_count]); + break; + } + } + gpr_mu_unlock(&pollset_set->mu); +} + +static void pollset_set_add_pollset_set(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *bag, + grpc_pollset_set *item) { + size_t i, j; + gpr_mu_lock(&bag->mu); + if (bag->pollset_set_count == bag->pollset_set_capacity) { + bag->pollset_set_capacity = GPR_MAX(8, 2 * bag->pollset_set_capacity); + bag->pollset_sets = + gpr_realloc(bag->pollset_sets, + bag->pollset_set_capacity * sizeof(*bag->pollset_sets)); + } + bag->pollset_sets[bag->pollset_set_count++] = item; + for (i = 0, j = 0; i < bag->fd_count; i++) { + if (fd_is_orphaned(bag->fds[i])) { + GRPC_FD_UNREF(bag->fds[i], "pollset_set"); + } else { + pollset_set_add_fd(exec_ctx, item, bag->fds[i]); + bag->fds[j++] = bag->fds[i]; + } + } + bag->fd_count = j; + gpr_mu_unlock(&bag->mu); +} + +static void pollset_set_del_pollset_set(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *bag, + grpc_pollset_set *item) { + size_t i; + gpr_mu_lock(&bag->mu); + for (i = 0; i < bag->pollset_set_count; i++) { + if (bag->pollset_sets[i] == item) { + bag->pollset_set_count--; + GPR_SWAP(grpc_pollset_set *, bag->pollset_sets[i], + bag->pollset_sets[bag->pollset_set_count]); + break; + } + } + gpr_mu_unlock(&bag->mu); +} + +static void pollset_set_add_fd(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, grpc_fd *fd) { + size_t i; + gpr_mu_lock(&pollset_set->mu); + if (pollset_set->fd_count == pollset_set->fd_capacity) { + pollset_set->fd_capacity = GPR_MAX(8, 2 * pollset_set->fd_capacity); + pollset_set->fds = gpr_realloc( + pollset_set->fds, pollset_set->fd_capacity * sizeof(*pollset_set->fds)); + } + GRPC_FD_REF(fd, "pollset_set"); + pollset_set->fds[pollset_set->fd_count++] = fd; + for (i = 0; i < pollset_set->pollset_count; i++) { + pollset_add_fd(exec_ctx, pollset_set->pollsets[i], fd); + } + for (i = 0; i < pollset_set->pollset_set_count; i++) { + pollset_set_add_fd(exec_ctx, pollset_set->pollset_sets[i], fd); + } + gpr_mu_unlock(&pollset_set->mu); +} + +static void pollset_set_del_fd(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, grpc_fd *fd) { + size_t i; + gpr_mu_lock(&pollset_set->mu); + for (i = 0; i < pollset_set->fd_count; i++) { + if (pollset_set->fds[i] == fd) { + pollset_set->fd_count--; + GPR_SWAP(grpc_fd *, pollset_set->fds[i], + pollset_set->fds[pollset_set->fd_count]); + GRPC_FD_UNREF(fd, "pollset_set"); + break; + } + } + for (i = 0; i < pollset_set->pollset_set_count; i++) { + pollset_set_del_fd(exec_ctx, pollset_set->pollset_sets[i], fd); + } + gpr_mu_unlock(&pollset_set->mu); +} + +/******************************************************************************* + * event engine binding + */ + +static void shutdown_engine(void) { + fd_global_shutdown(); + pollset_global_shutdown(); +} + +static const grpc_event_engine_vtable vtable = { + .pollset_size = sizeof(grpc_pollset), + + .fd_create = fd_create, + .fd_wrapped_fd = fd_wrapped_fd, + .fd_orphan = fd_orphan, + .fd_shutdown = fd_shutdown, + .fd_notify_on_read = fd_notify_on_read, + .fd_notify_on_write = fd_notify_on_write, + + .pollset_init = pollset_init, + .pollset_shutdown = pollset_shutdown, + .pollset_reset = pollset_reset, + .pollset_destroy = pollset_destroy, + .pollset_work = pollset_work, + .pollset_kick = pollset_kick, + .pollset_add_fd = pollset_add_fd, + + .pollset_set_create = pollset_set_create, + .pollset_set_destroy = pollset_set_destroy, + .pollset_set_add_pollset = pollset_set_add_pollset, + .pollset_set_del_pollset = pollset_set_del_pollset, + .pollset_set_add_pollset_set = pollset_set_add_pollset_set, + .pollset_set_del_pollset_set = pollset_set_del_pollset_set, + .pollset_set_add_fd = pollset_set_add_fd, + .pollset_set_del_fd = pollset_set_del_fd, + + .kick_poller = kick_poller, + + .shutdown_engine = shutdown_engine, +}; + +const grpc_event_engine_vtable *grpc_init_epoll_linux(void) { + fd_global_init(); + pollset_global_init(); + polling_island_global_init(); + return &vtable; +} + +#endif diff --git a/src/core/lib/iomgr/ev_epoll_linux.h b/src/core/lib/iomgr/ev_epoll_linux.h new file mode 100644 index 00000000000..8c819975a4c --- /dev/null +++ b/src/core/lib/iomgr/ev_epoll_linux.h @@ -0,0 +1,41 @@ +/* + * + * Copyright 2015, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#ifndef GRPC_CORE_LIB_IOMGR_EV_EPOLL_LINUX_H +#define GRPC_CORE_LIB_IOMGR_EV_EPOLL_LINUX_H + +#include "src/core/lib/iomgr/ev_posix.h" + +const grpc_event_engine_vtable *grpc_init_epoll_linux(void); + +#endif /* GRPC_CORE_LIB_IOMGR_EV_EPOLL_LINUX_H */ diff --git a/src/core/lib/iomgr/ev_epoll_posix.c b/src/core/lib/iomgr/ev_epoll_posix.c index 4481bab4380..5abd5b2a94c 100644 --- a/src/core/lib/iomgr/ev_epoll_posix.c +++ b/src/core/lib/iomgr/ev_epoll_posix.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include @@ -51,11 +52,13 @@ #include #include +#include "src/core/lib/iomgr/ev_posix.h" #include "src/core/lib/iomgr/iomgr_internal.h" #include "src/core/lib/iomgr/wakeup_fd_posix.h" #include "src/core/lib/profiling/timers.h" #include "src/core/lib/support/block_annotate.h" + /******************************************************************************* * FD declarations */ @@ -133,10 +136,9 @@ struct grpc_pollset { int called_shutdown; int kicked_without_pollers; grpc_closure *shutdown_done; - union { - int fd; - void *ptr; - } data; + + int epoll_fd; + /* Local cache of eventfds for workers */ grpc_cached_wakeup_fd *local_wakeup_cache; }; @@ -589,7 +591,6 @@ static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { pollset->local_wakeup_cache = NULL; pollset->kicked_without_pollers = 0; - pollset->data.ptr = NULL; multipoll_with_epoll_pollset_create_efd(pollset); } @@ -619,22 +620,6 @@ static void pollset_reset(grpc_pollset *pollset) { pollset->kicked_without_pollers = 0; } -/* TODO (sreek): Remove multipoll_with_epoll_add_fd declaration*/ -static void multipoll_with_epoll_pollset_add_fd(grpc_exec_ctx *exec_ctx, - grpc_pollset *pollset, - grpc_fd *fd); - -static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_fd *fd) { - /* TODO (sreek) - Does reading pollset->data.ptr need pollset->mu lock ? - * because finally_add_fd() also reads it but without the lock! */ - gpr_mu_lock(&pollset->mu); - GPR_ASSERT(pollset->data.ptr != NULL); - gpr_mu_unlock(&pollset->mu); - - multipoll_with_epoll_pollset_add_fd(exec_ctx, pollset, fd); -} - /* TODO (sreek): Remove multipoll_with_epoll_finish_shutdown() declaration */ static void multipoll_with_epoll_pollset_finish_shutdown(grpc_pollset *pollset); @@ -790,20 +775,6 @@ static int poll_deadline_to_millis_timeout(gpr_timespec deadline, * pollset_multipoller_with_epoll_posix.c */ -#include -#include -#include -#include -#include - -#include -#include -#include - -#include "src/core/lib/iomgr/ev_posix.h" -#include "src/core/lib/profiling/timers.h" -#include "src/core/lib/support/block_annotate.h" - static void set_ready(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure **st) { /* only one set_ready can be active at once (but there may be a racing notify_on) */ @@ -879,13 +850,13 @@ static void remove_fd_from_all_epoll_sets(int fd) { gpr_mu_unlock(&epoll_fd_list_mu); } -typedef struct { int epoll_fd; } epoll_hdr; - -static void finally_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, +/* TODO: sreek - This function multipoll_with_epoll_pollset_add_fd() and + * finally_add_fd() in ev_poll_and_epoll_posix.c */ +static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { - /*TODO: (sree) Shouldn't this read (pollset->data.ptr) be done under a - pollset lock - i.e pollset->mu ? */ - epoll_hdr *h = pollset->data.ptr; + + /* TODO sreek - Check if we need to get a pollset->mu lock here */ + struct epoll_event ev; int err; @@ -908,7 +879,7 @@ static void finally_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); ev.data.ptr = fd; - err = epoll_ctl(h->epoll_fd, EPOLL_CTL_ADD, fd->fd, &ev); + err = epoll_ctl(pollset->epoll_fd, EPOLL_CTL_ADD, fd->fd, &ev); if (err < 0) { /* FDs may be added to a pollset multiple times, so EEXIST is normal. */ if (errno != EEXIST) { @@ -933,26 +904,20 @@ static void finally_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, /* TODO: This has to be called ONLY from pollset_init function. and hence it * does not acquire any lock */ static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset) { - epoll_hdr *h = gpr_malloc(sizeof(epoll_hdr)); struct epoll_event ev; int err; - /* TODO (sreek). remove this assert. Currently added this just to ensure that - * we do not overwrite h->epoll_fd without freeing the older one*/ - GPR_ASSERT(pollset->data.ptr == NULL); - - pollset->data.ptr = h; - h->epoll_fd = epoll_create1(EPOLL_CLOEXEC); - if (h->epoll_fd < 0) { + pollset->epoll_fd = epoll_create1(EPOLL_CLOEXEC); + if (pollset->epoll_fd < 0) { gpr_log(GPR_ERROR, "epoll_create1 failed: %s", strerror(errno)); abort(); } - add_epoll_fd_to_global_list(h->epoll_fd); + add_epoll_fd_to_global_list(pollset->epoll_fd); ev.events = (uint32_t)(EPOLLIN | EPOLLET); ev.data.ptr = NULL; - err = epoll_ctl(h->epoll_fd, EPOLL_CTL_ADD, + err = epoll_ctl(pollset->epoll_fd, EPOLL_CTL_ADD, GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), &ev); if (err < 0) { gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", @@ -961,12 +926,6 @@ static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset) { } } -static void multipoll_with_epoll_pollset_add_fd(grpc_exec_ctx *exec_ctx, - grpc_pollset *pollset, - grpc_fd *fd) { - finally_add_fd(exec_ctx, pollset, fd); -} - /* TODO(klempner): We probably want to turn this down a bit */ #define GRPC_EPOLL_MAX_EVENTS 1000 @@ -974,9 +933,9 @@ static void multipoll_with_epoll_pollset_maybe_work_and_unlock( grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker *worker, gpr_timespec deadline, gpr_timespec now) { struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; + int epoll_fd = pollset->epoll_fd; int ep_rv; int poll_rv; - epoll_hdr *h = pollset->data.ptr; int timeout_ms; struct pollfd pfds[2]; @@ -993,7 +952,7 @@ static void multipoll_with_epoll_pollset_maybe_work_and_unlock( pfds[0].fd = GRPC_WAKEUP_FD_GET_READ_FD(&worker->wakeup_fd->fd); pfds[0].events = POLLIN; pfds[0].revents = 0; - pfds[1].fd = h->epoll_fd; + pfds[1].fd = epoll_fd; pfds[1].events = POLLIN; pfds[1].revents = 0; @@ -1018,7 +977,7 @@ static void multipoll_with_epoll_pollset_maybe_work_and_unlock( if (pfds[1].revents) { do { /* The following epoll_wait never blocks; it has a timeout of 0 */ - ep_rv = epoll_wait(h->epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, 0); + ep_rv = epoll_wait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, 0); if (ep_rv < 0) { if (errno != EINTR) { gpr_log(GPR_ERROR, "epoll_wait() failed: %s", strerror(errno)); @@ -1053,10 +1012,8 @@ static void multipoll_with_epoll_pollset_finish_shutdown( grpc_pollset *pollset) {} static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset) { - epoll_hdr *h = pollset->data.ptr; - close(h->epoll_fd); - remove_epoll_fd_from_global_list(h->epoll_fd); - gpr_free(h); + close(pollset->epoll_fd); + remove_epoll_fd_from_global_list(pollset->epoll_fd); } /******************************************************************************* diff --git a/src/core/lib/iomgr/ev_posix.c b/src/core/lib/iomgr/ev_posix.c index baa3b9856ad..404ef2a64b9 100644 --- a/src/core/lib/iomgr/ev_posix.c +++ b/src/core/lib/iomgr/ev_posix.c @@ -44,7 +44,7 @@ #include #include -#include "src/core/lib/iomgr/ev_epoll_posix.h" +#include "src/core/lib/iomgr/ev_epoll_linux.h" #include "src/core/lib/iomgr/ev_poll_posix.h" #include "src/core/lib/support/env.h" @@ -163,11 +163,6 @@ void grpc_fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, g_event_engine->fd_notify_on_write(exec_ctx, fd, closure); } -grpc_pollset *grpc_fd_get_read_notifier_pollset(grpc_exec_ctx *exec_ctx, - grpc_fd *fd) { - return g_event_engine->fd_get_read_notifier_pollset(exec_ctx, fd); -} - size_t grpc_pollset_size(void) { return g_event_engine->pollset_size; } void grpc_pollset_init(grpc_pollset *pollset, gpr_mu **mu) { diff --git a/src/python/grpcio/grpc_core_dependencies.py b/src/python/grpcio/grpc_core_dependencies.py index 49b4ddc457c..13bc6888d66 100644 --- a/src/python/grpcio/grpc_core_dependencies.py +++ b/src/python/grpcio/grpc_core_dependencies.py @@ -94,6 +94,7 @@ CORE_SOURCE_FILES = [ 'src/core/lib/iomgr/endpoint.c', 'src/core/lib/iomgr/endpoint_pair_posix.c', 'src/core/lib/iomgr/endpoint_pair_windows.c', + 'src/core/lib/iomgr/ev_epoll_linux.c', 'src/core/lib/iomgr/ev_epoll_posix.c', 'src/core/lib/iomgr/ev_poll_posix.c', 'src/core/lib/iomgr/ev_posix.c', diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index 36b25cca161..d968278f2a4 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -807,6 +807,7 @@ src/core/lib/http/parser.h \ src/core/lib/iomgr/closure.h \ src/core/lib/iomgr/endpoint.h \ src/core/lib/iomgr/endpoint_pair.h \ +src/core/lib/iomgr/ev_epoll_linux.h \ src/core/lib/iomgr/ev_epoll_posix.h \ src/core/lib/iomgr/ev_poll_posix.h \ src/core/lib/iomgr/ev_posix.h \ @@ -955,6 +956,7 @@ src/core/lib/iomgr/closure.c \ src/core/lib/iomgr/endpoint.c \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ +src/core/lib/iomgr/ev_epoll_linux.c \ src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 43940495864..97cc55db36e 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -5530,6 +5530,7 @@ "src/core/lib/iomgr/closure.h", "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", + "src/core/lib/iomgr/ev_epoll_linux.h", "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", @@ -5630,6 +5631,8 @@ "src/core/lib/iomgr/endpoint_pair.h", "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", + "src/core/lib/iomgr/ev_epoll_linux.c", + "src/core/lib/iomgr/ev_epoll_linux.h", "src/core/lib/iomgr/ev_epoll_posix.c", "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.c", diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj b/vsprojects/vcxproj/grpc/grpc.vcxproj index 55304af5869..a67e4d16dad 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj @@ -316,6 +316,7 @@ + @@ -484,6 +485,8 @@ + + diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters index 7d1c90fda7c..bf9b7dc7dcf 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters @@ -55,6 +55,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr @@ -677,6 +680,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj index 3d0cdfc668b..afc9a2ca1b2 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj @@ -304,6 +304,7 @@ + @@ -450,6 +451,8 @@ + + diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters index d2ff4c630fd..b7507f9a963 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters @@ -58,6 +58,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr @@ -575,6 +578,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr From 9442bab5d303d7bd33e9406129ad897588d07111 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 20 May 2016 17:54:06 -0700 Subject: [PATCH 0026/1039] Write most of the methods in the new epoll implementation --- src/core/lib/iomgr/ev_epoll_linux.c | 301 ++++++++++++++++++++++------ 1 file changed, 244 insertions(+), 57 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index f257ac8a1dd..0d30bb659b6 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -150,28 +150,84 @@ typedef struct polling_island { static gpr_mu g_pi_freelist_mu; static polling_island *g_pi_freelist = NULL; -/* TODO: sreek - Should we hold a lock on fd or add a ref to the fd ? */ -static void add_fd_to_polling_island_locked(polling_island *pi, grpc_fd *fd) { +/* TODO: sreek - Should we hold a lock on fd or add a ref to the fd ? + * TODO: sreek - Should this add a ref to the grpc_fd ? */ +/* The caller is expected to hold pi->mu lock before calling this function */ +static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, + size_t fd_count) { int err; + size_t i; struct epoll_event ev; - ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); - ev.data.ptr = fd; - err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_ADD, fd->fd, &ev); + for (i = 0; i < fd_count; i++) { + ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); + ev.data.ptr = fds[i]; + err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_ADD, fds[i]->fd, &ev); + + if (err < 0 && errno != EEXIST) { + gpr_log(GPR_ERROR, "epoll_ctl add for fd: %d failed with error: %s", + fds[i]->fd, strerror(errno)); + /* TODO: sreek - Not sure if it is a good idea to continue here. We need a + * better way to bubble up this error instead of doing an abort() */ + continue; + } - if (err < 0 && errno != EEXIST) { - gpr_log(GPR_ERROR, "epoll_ctl add for fd: %d failed with error: %s", fd->fd, - strerror(errno)); - return; + if (pi->fd_cnt == pi->fd_capacity) { + pi->fd_capacity = GPR_MAX(pi->fd_capacity + 8, pi->fd_cnt * 3 / 2); + pi->fds = gpr_realloc(pi->fds, sizeof(grpc_fd *) * pi->fd_capacity); + } + + pi->fds[pi->fd_cnt++] = fds[i]; + } +} + +/* TODO: sreek - Should we hold a lock on fd or add a ref to the fd ? + * TODO: sreek - Might have to unref the fds (assuming whether we add a ref to + * the fd when adding it to the epollset) */ +/* The caller is expected to hold pi->mu lock before calling this function */ +static void polling_island_clear_fds_locked(polling_island *pi) { + int err; + size_t i; + + for (i = 0; i < pi->fd_cnt; i++) { + err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_DEL, pi->fds[i]->fd, NULL); + + if (err < 0 && errno != ENOENT) { + gpr_log(GPR_ERROR, + "epoll_ctl delete for fds[i]: %d failed with error: %s", i, + pi->fds[i]->fd, strerror(errno)); + /* TODO: sreek - Not sure if it is a good idea to continue here. We need a + * better way to bubble up this error instead of doing an abort() */ + continue; + } + } + + pi->fd_cnt = 0; +} + +/* TODO: sreek - Should we hold a lock on fd or add a ref to the fd ? + * TODO: sreek - Might have to unref the fd (assuming whether we add a ref to + * the fd when adding it to the epollset) */ +/* The caller is expected to hold pi->mu lock before calling this function */ +static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd) { + int err; + size_t i; + err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_DEL, fd->fd, NULL); + if (err < 0 && errno != ENOENT) { + gpr_log(GPR_ERROR, "epoll_ctl delete for fd: %d failed with error; %s", + fd->fd, strerror(errno)); } - pi->fd_capacity = GPR_MAX(pi->fd_capacity + 8, pi->fd_cnt * 3 / 2); - pi->fds = gpr_realloc(pi->fds, sizeof(grpc_fd *) * pi->fd_capacity); - pi->fds[pi->fd_cnt++] = fd; + for (i = 0; i < pi->fd_cnt; i++) { + if (pi->fds[i] == fd) { + pi->fds[i] = pi->fds[--pi->fd_cnt]; + break; + } + } } -static polling_island *polling_island_create(int initial_ref_cnt, - grpc_fd *initial_fd) { +static polling_island *polling_island_create(grpc_fd *initial_fd, + int initial_ref_cnt) { polling_island *pi = NULL; gpr_mu_lock(&g_pi_freelist_mu); if (g_pi_freelist != NULL) { @@ -202,17 +258,151 @@ static polling_island *polling_island_create(int initial_ref_cnt, pi->next_free = NULL; if (initial_fd != NULL) { - /* add_fd_to_polling_island_locked() expects the caller to hold a pi->mu + /* polling_island_add_fds_locked() expects the caller to hold a pi->mu * lock. However, since this is a new polling island (and no one has a * reference to it yet), it is okay to not acquire pi->mu here */ - add_fd_to_polling_island_locked(pi, initial_fd); + polling_island_add_fds_locked(pi, &initial_fd, 1); } return pi; } +static void polling_island_delete(polling_island *pi) { + GPR_ASSERT(pi->ref_cnt == 0); + GPR_ASSERT(pi->fd_cnt == 0); + + pi->merged_to = NULL; + + gpr_mu_lock(&g_pi_freelist_mu); + pi->next_free = g_pi_freelist; + g_pi_freelist = pi; + gpr_mu_unlock(&g_pi_freelist_mu); +} + +void polling_island_unref_and_unlock(polling_island *pi, int unref_by) { + pi->ref_cnt -= unref_by; + int ref_cnt = pi->ref_cnt; + GPR_ASSERT(ref_cnt >= 0); + + gpr_mu_unlock(&pi->mu); + + if (ref_cnt == 0) { + polling_island_delete(pi); + } +} + +polling_island *polling_island_update_and_lock(polling_island *pi, int unref_by, + int add_ref_by) { + polling_island *next = NULL; + gpr_mu_lock(&pi->mu); + while (pi->merged_to != NULL) { + next = pi->merged_to; + polling_island_unref_and_unlock(pi, unref_by); + pi = next; + gpr_mu_lock(&pi->mu); + } + + pi->ref_cnt += add_ref_by; + return pi; +} + +void polling_island_pair_update_and_lock(polling_island **p, + polling_island **q) { + polling_island *pi_1 = *p; + polling_island *pi_2 = *q; + polling_island *temp = NULL; + bool pi_1_locked = false; + bool pi_2_locked = false; + int num_swaps = 0; + + while (pi_1 != pi_2 && !(pi_1_locked && pi_2_locked)) { + // pi_1 is NOT equal to pi_2 + // pi_1 MAY be locked + + if (pi_1 > pi_2) { + if (pi_1_locked) { + gpr_mu_unlock(&pi_1->mu); + pi_1_locked = false; + } + + GPR_SWAP(polling_island *, pi_1, pi_2); + num_swaps++; + } + + // p1 < p2 + // p1 MAY BE locked + // p2 is NOT locked + + if (!pi_1_locked) { + gpr_mu_lock(&pi_1->mu); + pi_1_locked = true; + + if (pi_1->merged_to != NULL) { + temp = pi_1->merged_to; + polling_island_unref_and_unlock(pi_1, 1); + pi_1 = temp; + pi_1_locked = false; + + continue; + } + } + + // p1 is LOCKED + // p2 is UNLOCKED + // p1 != p2 + + gpr_mu_lock(&pi_2->mu); + pi_2_locked = true; + + if (pi_2->merged_to != NULL) { + temp = pi_2->merged_to; + polling_island_unref_and_unlock(pi_2, 1); + pi_2 = temp; + pi_2_locked = false; + } + } + + // Either pi_1 == pi_2 OR we got both locks! + if (pi_1 == pi_2) { + GPR_ASSERT(pi_1_locked || (!pi_1_locked && !pi_2_locked)); + if (!pi_1_locked) { + pi_1 = pi_2 = polling_island_update_and_lock(pi_1, 2, 0); + } + } else { + GPR_ASSERT(pi_1_locked && pi_2_locked); + if (num_swaps % 2 > 0) { + GPR_SWAP(polling_island *, pi_1, pi_2); + } + } + + *p = pi_1; + *q = pi_2; +} + +polling_island *polling_island_merge(polling_island *p, polling_island *q) { + polling_island *merged = NULL; + + polling_island_pair_update_and_lock(&p, &q); + + /* TODO: sreek: Think about this scenario some more. Is it possible ?. what + * does it mean, when would this happen */ + if (p == q) { + merged = p; + } + + // Move all the fds from polling_island p to polling_island q + polling_island_add_fds_locked(q, p->fds, p->fd_cnt); + polling_island_clear_fds_locked(p); + + q->ref_cnt += p->ref_cnt; + + gpr_mu_unlock(&p->mu); + gpr_mu_unlock(&q->mu); + + return merged; +} + static void polling_island_global_init() { - polling_island_create(0, NULL); /* TODO(sreek): Delete this line */ gpr_mu_init(&g_pi_freelist_mu); g_pi_freelist = NULL; } @@ -245,7 +435,7 @@ struct grpc_pollset { int epoll_fd; - /* Mutex protecting the 'polling_island' field */ + /* Mutex protecting the 'polling_island' field */ gpr_mu pi_mu; /* The polling island to which this fd belongs to. An fd belongs to exactly @@ -319,7 +509,8 @@ struct grpc_pollset_set { * fd_posix.c */ -/* We need to keep a freelist not because of any concerns of malloc performance +/* We need to keep a freelist not because of any concerns of malloc + * performance * but instead so that implementations with multiple threads in (for example) * epoll_wait deal with the race between pollset removal and incoming poll * notifications. @@ -434,6 +625,7 @@ static void fd_global_shutdown(void) { static grpc_fd *fd_create(int fd, const char *name) { grpc_fd *r = alloc_fd(fd); + char *name2; gpr_asprintf(&name2, "%s fd=%d", name, fd); grpc_iomgr_register_object(&r->iomgr_object, name2); @@ -453,6 +645,20 @@ static void close_fd_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { if (!fd->released) { close(fd->fd); } else { + /* TODO: sreek - Check for deadlocks */ + + gpr_mu_lock(&fd->pi_mu); + fd->polling_island = + polling_island_update_and_lock(fd->polling_island, 1, 0); + + polling_island_remove_fd_locked(fd->polling_island, fd); + polling_island_unref_and_unlock(fd->polling_island, 1); + + fd->polling_island = NULL; + gpr_mu_unlock(&fd->pi_mu); + + + /* TODO: sreek - This should be no longer needed */ remove_fd_from_all_epoll_sets(fd->fd); } grpc_exec_ctx_enqueue(exec_ctx, fd->on_done_closure, true, NULL); @@ -752,7 +958,8 @@ static void finish_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset) { grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); } -/* TODO(sreek): Remove multipoll_with_epoll_*_maybe_work_and_unlock declaration +/* TODO(sreek): Remove multipoll_with_epoll_*_maybe_work_and_unlock + * declaration */ static void multipoll_with_epoll_pollset_maybe_work_and_unlock( grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker *worker, @@ -979,50 +1186,30 @@ static void remove_fd_from_all_epoll_sets(int fd) { * finally_add_fd() in ev_poll_and_epoll_posix.c */ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { - /* TODO sreek - Check if we need to get a pollset->mu lock here */ + gpr_mu_lock(&pollset->pi_mu); + gpr_mu_lock(&fd->pi_mu); - struct epoll_event ev; - int err; - - /* Hold a ref to the fd to keep it from being closed during the add. This may - result in a spurious wakeup being assigned to this pollset whilst adding, - but that should be benign. */ - /* TODO: (sreek): Understand how a spurious wake up migh be assinged to this - * pollset..and how holding a reference will prevent the fd from being closed - * (and perhaps more importantly, see how can an fd be closed while being - * added to the epollset */ - GRPC_FD_REF(fd, "add fd"); + polling_island *pi_new = NULL; - gpr_mu_lock(&fd->mu); - if (fd->shutdown) { - gpr_mu_unlock(&fd->mu); - GRPC_FD_UNREF(fd, "add fd"); - return; - } - gpr_mu_unlock(&fd->mu); - - ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); - ev.data.ptr = fd; - err = epoll_ctl(pollset->epoll_fd, EPOLL_CTL_ADD, fd->fd, &ev); - if (err < 0) { - /* FDs may be added to a pollset multiple times, so EEXIST is normal. */ - if (errno != EEXIST) { - gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", fd->fd, - strerror(errno)); + if (fd->polling_island == pollset->polling_island) { + pi_new = fd->polling_island; + if (pi_new == NULL) { + pi_new = polling_island_create(fd, 2); } - } + } else if (fd->polling_island == NULL) { + pi_new = polling_island_update_and_lock(pollset->polling_island, 1, 1); - /* The fd might have been orphaned while we were adding it to the epoll set. - Close the fd in such a case (which will also take care of removing it from - the epoll set */ - gpr_mu_lock(&fd->mu); - if (fd_is_orphaned(fd) && !fd->closed) { - close_fd_locked(exec_ctx, fd); + } else if (pollset->polling_island == NULL) { + pi_new = polling_island_update_and_lock(fd->polling_island, 1, 1); + } else { // Non null and different + pi_new = polling_island_merge(fd->polling_island, pollset->polling_island); } - gpr_mu_unlock(&fd->mu); - GRPC_FD_UNREF(fd, "add fd"); + fd->polling_island = pollset->polling_island = pi_new; + + gpr_mu_unlock(&fd->pi_mu); + gpr_mu_unlock(&pollset->pi_mu); } /* Creates an epoll fd and initializes the pollset */ From d806145573f8e78a52012b4b8ab94ff46b855d58 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 20 May 2016 18:12:30 -0700 Subject: [PATCH 0027/1039] Removed epoll_fd_global_list --- src/core/lib/iomgr/ev_epoll_linux.c | 68 +---------------------------- 1 file changed, 1 insertion(+), 67 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 0d30bb659b6..7793a952016 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -483,8 +483,6 @@ typedef void (*platform_become_multipoller_type)(grpc_exec_ctx *exec_ctx, * be locked) */ static int pollset_has_workers(grpc_pollset *pollset); -static void remove_fd_from_all_epoll_sets(int fd); - /******************************************************************************* * pollset_set definitions */ @@ -656,11 +654,8 @@ static void close_fd_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { fd->polling_island = NULL; gpr_mu_unlock(&fd->pi_mu); - - - /* TODO: sreek - This should be no longer needed */ - remove_fd_from_all_epoll_sets(fd->fd); } + grpc_exec_ctx_enqueue(exec_ctx, fd->on_done_closure, true, NULL); } @@ -1123,65 +1118,6 @@ static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { set_ready(exec_ctx, fd, &fd->write_closure); } -/* TODO (sreek): Maybe this global list is not required. Double check*/ -struct epoll_fd_list { - int *epoll_fds; - size_t count; - size_t capacity; -}; - -static struct epoll_fd_list epoll_fd_global_list; -static gpr_once init_epoll_fd_list_mu = GPR_ONCE_INIT; -static gpr_mu epoll_fd_list_mu; - -static void init_mu(void) { gpr_mu_init(&epoll_fd_list_mu); } - -static void add_epoll_fd_to_global_list(int epoll_fd) { - gpr_once_init(&init_epoll_fd_list_mu, init_mu); - - gpr_mu_lock(&epoll_fd_list_mu); - if (epoll_fd_global_list.count == epoll_fd_global_list.capacity) { - epoll_fd_global_list.capacity = - GPR_MAX((size_t)8, epoll_fd_global_list.capacity * 2); - epoll_fd_global_list.epoll_fds = - gpr_realloc(epoll_fd_global_list.epoll_fds, - epoll_fd_global_list.capacity * sizeof(int)); - } - epoll_fd_global_list.epoll_fds[epoll_fd_global_list.count++] = epoll_fd; - gpr_mu_unlock(&epoll_fd_list_mu); -} - -static void remove_epoll_fd_from_global_list(int epoll_fd) { - gpr_mu_lock(&epoll_fd_list_mu); - GPR_ASSERT(epoll_fd_global_list.count > 0); - for (size_t i = 0; i < epoll_fd_global_list.count; i++) { - if (epoll_fd == epoll_fd_global_list.epoll_fds[i]) { - epoll_fd_global_list.epoll_fds[i] = - epoll_fd_global_list.epoll_fds[--(epoll_fd_global_list.count)]; - break; - } - } - gpr_mu_unlock(&epoll_fd_list_mu); -} - -static void remove_fd_from_all_epoll_sets(int fd) { - int err; - gpr_once_init(&init_epoll_fd_list_mu, init_mu); - gpr_mu_lock(&epoll_fd_list_mu); - if (epoll_fd_global_list.count == 0) { - gpr_mu_unlock(&epoll_fd_list_mu); - return; - } - for (size_t i = 0; i < epoll_fd_global_list.count; i++) { - err = epoll_ctl(epoll_fd_global_list.epoll_fds[i], EPOLL_CTL_DEL, fd, NULL); - if (err < 0 && errno != ENOENT) { - gpr_log(GPR_ERROR, "epoll_ctl del for %d failed: %s", fd, - strerror(errno)); - } - } - gpr_mu_unlock(&epoll_fd_list_mu); -} - /* TODO: sreek - This function multipoll_with_epoll_pollset_add_fd() and * finally_add_fd() in ev_poll_and_epoll_posix.c */ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, @@ -1224,7 +1160,6 @@ static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset) { gpr_log(GPR_ERROR, "epoll_create1 failed: %s", strerror(errno)); abort(); } - add_epoll_fd_to_global_list(pollset->epoll_fd); ev.events = (uint32_t)(EPOLLIN | EPOLLET); ev.data.ptr = NULL; @@ -1325,7 +1260,6 @@ static void multipoll_with_epoll_pollset_finish_shutdown( static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset) { close(pollset->epoll_fd); - remove_epoll_fd_from_global_list(pollset->epoll_fd); } /******************************************************************************* From 96b2554313120949dd26e3c0968e8aea9b8a650f Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 20 May 2016 18:14:48 -0700 Subject: [PATCH 0028/1039] ctiller's ev_epoll_linux.c file (for reference) --- src/core/lib/iomgr/ctiller_ev_epoll_linux.c | 461 ++++++++++++++++++++ 1 file changed, 461 insertions(+) create mode 100644 src/core/lib/iomgr/ctiller_ev_epoll_linux.c diff --git a/src/core/lib/iomgr/ctiller_ev_epoll_linux.c b/src/core/lib/iomgr/ctiller_ev_epoll_linux.c new file mode 100644 index 00000000000..23c20a77aae --- /dev/null +++ b/src/core/lib/iomgr/ctiller_ev_epoll_linux.c @@ -0,0 +1,461 @@ +/* + * + * Copyright 2015-2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include "src/core/lib/iomgr/ev_epoll_linux.h" + +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "src/core/lib/iomgr/iomgr_internal.h" + +/* TODO(sreek) Remove this file */ + + +//////////////////////////////////////////////////////////////////////////////// +// Definitions + +#define STATE_NOT_READY ((gpr_atm)0) +#define STATE_READY ((gpr_atm)1) + +typedef enum { POLLABLE_FD, POLLABLE_EPOLL_SET } pollable_type; + +typedef struct { + pollable_type type; + int fd; + grpc_iomgr_object iomgr_object; +} pollable_object; + +typedef struct polling_island { + pollable_object pollable; + gpr_mu mu; + int refs; + grpc_fd *only_fd; + struct polling_island *became; + struct polling_island *next; +} polling_island; + +struct grpc_fd { + pollable_object pollable; + + // each event atomic is a tri state: + // STATE_NOT_READY - no event received, nobody waiting for it either + // STATE_READY - event received, nobody waiting for it + // closure pointer - no event received, upper layer is waiting for it + gpr_atm on_readable; + gpr_atm on_writable; + + // mutex guarding set_ready & shutdown state + gpr_mu set_ready_mu; + bool shutdown; + + // mutex protecting polling_island + gpr_mu polling_island_mu; + // current polling island + polling_island *polling_island; + + grpc_fd *next_free; +}; + +struct grpc_pollset_worker {}; + +struct grpc_pollset { + gpr_mu mu; + // current polling island + polling_island *polling_island; +}; + +//////////////////////////////////////////////////////////////////////////////// +// Polling island implementation + +static gpr_mu g_pi_freelist_mu; +static polling_island *g_first_free_pi; + +static void add_pollable_to_epoll_set(pollable_object *pollable, int epoll_set, + uint32_t events) { + struct epoll_event ev; + ev.events = events; + ev.data.ptr = pollable; + int err = epoll_ctl(epoll_set, EPOLL_CTL_ADD, pollable->fd, &ev); + if (err < 0) { + gpr_log(GPR_ERROR, "epoll_ctl add for %d faild: %s", pollable->fd, + strerror(errno)); + } +} + +static void add_fd_to_epoll_set(grpc_fd *fd, int epoll_set) { + add_pollable_to_epoll_set(&fd->pollable, epoll_set, + EPOLLIN | EPOLLOUT | EPOLLET); +} + +static void add_island_to_epoll_set(polling_island *pi, int epoll_set) { + add_pollable_to_epoll_set(&pi->pollable, epoll_set, EPOLLIN | EPOLLET); +} + +static polling_island *polling_island_create(grpc_fd *initial_fd) { + polling_island *r = NULL; + gpr_mu_lock(&g_pi_freelist_mu); + if (g_first_free_pi == NULL) { + r = gpr_malloc(sizeof(*r)); + r->pollable.type = POLLABLE_EPOLL_SET; + gpr_mu_init(&r->mu); + } else { + r = g_first_free_pi; + g_first_free_pi = r->next; + } + gpr_mu_unlock(&g_pi_freelist_mu); + + r->pollable.fd = epoll_create1(EPOLL_CLOEXEC); + GPR_ASSERT(r->pollable.fd >= 0); + + gpr_mu_lock(&r->mu); + r->only_fd = initial_fd; + r->refs = 2; // creation of a polling island => a referencing pollset & fd + gpr_mu_unlock(&r->mu); + + add_fd_to_epoll_set(initial_fd, r->pollable.fd); + return r; +} + +static void polling_island_delete(polling_island *p) { + gpr_mu_lock(&g_pi_freelist_mu); + p->next = g_first_free_pi; + g_first_free_pi = p; + gpr_mu_unlock(&g_pi_freelist_mu); +} + +static polling_island *polling_island_add(polling_island *p, grpc_fd *fd) { + gpr_mu_lock(&p->mu); + p->only_fd = NULL; + p->refs++; // new fd picks up a ref + gpr_mu_unlock(&p->mu); + + add_fd_to_epoll_set(fd, p->pollable.fd); + + return p; +} + +static void add_siblings_to(polling_island *siblings, polling_island *dest) { + polling_island *sibling_tail = dest; + while (sibling_tail->next != NULL) { + sibling_tail = sibling_tail->next; + } + sibling_tail->next = siblings; +} + +static polling_island *polling_island_merge(polling_island *a, + polling_island *b) { + GPR_ASSERT(a != b); + polling_island *out; + + gpr_mu_lock(&GPR_MIN(a, b)->mu); + gpr_mu_lock(&GPR_MAX(a, b)->mu); + + GPR_ASSERT(a->became == NULL); + GPR_ASSERT(b->became == NULL); + + if (a->only_fd == NULL && b->only_fd == NULL) { + b->became = a; + add_siblings_to(b, a); + add_island_to_epoll_set(b, a->pollable.fd); + out = a; + } else if (a->only_fd == NULL) { + GPR_ASSERT(b->only_fd != NULL); + add_fd_to_epoll_set(b->only_fd, a->pollable.fd); + b->became = a; + out = a; + } else if (b->only_fd == NULL) { + GPR_ASSERT(a->only_fd != NULL); + add_fd_to_epoll_set(a->only_fd, b->pollable.fd); + a->became = b; + out = b; + } else { + add_fd_to_epoll_set(b->only_fd, a->pollable.fd); + a->only_fd = NULL; + b->only_fd = NULL; + b->became = a; + out = a; + } + + gpr_mu_unlock(&a->mu); + gpr_mu_unlock(&b->mu); + + return out; +} + +static polling_island *polling_island_update_and_lock(polling_island *p) { + gpr_mu_lock(&p->mu); + if (p->became != NULL) { + do { + polling_island *from = p; + p = p->became; + gpr_mu_lock(&p->mu); + bool delete_from = 0 == --from->refs; + p->refs++; + gpr_mu_unlock(&from->mu); + if (delete_from) { + polling_island_delete(from); + } + } while (p->became != NULL); + } + return p; +} + +static polling_island *polling_island_ref(polling_island *p) { + gpr_mu_lock(&p->mu); + gpr_mu_unlock(&p->mu); + return p; +} + +static void polling_island_drop(polling_island *p) {} + +static polling_island *polling_island_update(polling_island *p, + int updating_owner_count) { + p = polling_island_update_and_lock(p); + GPR_ASSERT(p->refs != 0); + p->refs += updating_owner_count; + gpr_mu_unlock(&p->mu); + return p; +} + +//////////////////////////////////////////////////////////////////////////////// +// FD implementation + +static gpr_mu g_fd_freelist_mu; +static grpc_fd *g_first_free_fd; + +static grpc_fd *fd_create(int fd, const char *name) { + grpc_fd *r = NULL; + gpr_mu_lock(&g_fd_freelist_mu); + if (g_first_free_fd == NULL) { + r = gpr_malloc(sizeof(*r)); + r->pollable.type = POLLABLE_FD; + gpr_atm_rel_store(&r->on_readable, 0); + gpr_atm_rel_store(&r->on_writable, 0); + gpr_mu_init(&r->polling_island_mu); + gpr_mu_init(&r->set_ready_mu); + } else { + r = g_first_free_fd; + g_first_free_fd = r->next_free; + } + gpr_mu_unlock(&g_fd_freelist_mu); + + r->pollable.fd = fd; + grpc_iomgr_register_object(&r->pollable.iomgr_object, name); + r->next_free = NULL; + return r; +} + +static int fd_wrapped_fd(grpc_fd *fd) { return fd->pollable.fd; } + +static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure *on_done, int *release_fd, + const char *reason) { + if (release_fd != NULL) { + *release_fd = fd->pollable.fd; + } else { + close(fd->pollable.fd); + } + + gpr_mu_lock(&fd->polling_island_mu); + if (fd->polling_island != NULL) { + polling_island_drop(fd->polling_island); + } + gpr_mu_unlock(&fd->polling_island_mu); + + gpr_mu_lock(&g_fd_freelist_mu); + fd->next_free = g_first_free_fd; + g_first_free_fd = fd; + grpc_iomgr_unregister_object(&fd->pollable.iomgr_object); + gpr_mu_unlock(&g_fd_freelist_mu); + + grpc_exec_ctx_enqueue(exec_ctx, on_done, true, NULL); +} + +static void notify_on(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure *closure, gpr_atm *state) { + if (gpr_atm_acq_cas(state, STATE_NOT_READY, (gpr_atm)closure)) { + // state was not ready, and is now the closure - we're done */ + } else { + // cas failed - we MUST be in STATE_READY (can't request two notifications + // for the same event) + // flip back to not ready, enqueue the closure directly + GPR_ASSERT(gpr_atm_rel_cas(state, STATE_READY, STATE_NOT_READY)); + grpc_exec_ctx_enqueue(exec_ctx, closure, true, NULL); + } +} + +static void fd_notify_on_read(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure *closure) { + notify_on(exec_ctx, fd, closure, &fd->on_readable); +} + +static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_closure *closure) { + notify_on(exec_ctx, fd, closure, &fd->on_readable); +} + +static void destroy_fd_freelist(void) { + while (g_first_free_fd) { + grpc_fd *next = g_first_free_fd->next_free; + gpr_mu_destroy(&g_first_free_fd->polling_island_mu); + gpr_free(next); + g_first_free_fd = next; + } +} + +static void set_ready_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + gpr_atm *state) { + if (gpr_atm_acq_cas(state, STATE_NOT_READY, STATE_READY)) { + // state was not ready, and is now ready - we're done + } else { + // cas failed - either there's a closure queued which we should consume OR + // the state was already STATE_READY + gpr_atm cur_state = gpr_atm_acq_load(state); + if (cur_state != STATE_READY) { + // state wasn't STATE_READY - it *must* have been a closure + // since it's illegal to ask for notification twice, it's safe to assume + // that we'll resume being the closure + GPR_ASSERT(gpr_atm_rel_cas(state, cur_state, STATE_NOT_READY)); + grpc_exec_ctx_enqueue(exec_ctx, (grpc_closure *)cur_state, !fd->shutdown, + NULL); + } + } +} + +static void fd_shutdown(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { + gpr_mu_lock(&fd->set_ready_mu); + GPR_ASSERT(!fd->shutdown); + fd->shutdown = 1; + set_ready_locked(exec_ctx, fd, &fd->on_readable); + set_ready_locked(exec_ctx, fd, &fd->on_writable); + gpr_mu_unlock(&fd->set_ready_mu); +} + +//////////////////////////////////////////////////////////////////////////////// +// Pollset implementation + +static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { + gpr_mu_init(&pollset->mu); + *mu = &pollset->mu; + pollset->polling_island = NULL; +} + +static void pollset_destroy(grpc_pollset *pollset) { + gpr_mu_destroy(&pollset->mu); + if (pollset->polling_island) { + polling_island_drop(pollset->polling_island); + } +} + +static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + struct grpc_fd *fd) { + gpr_mu_lock(&pollset->mu); + gpr_mu_lock(&fd->polling_island_mu); + + polling_island *new; + + if (fd->polling_island == NULL) { + if (pollset->polling_island == NULL) { + new = polling_island_create(fd); + } else { + new = polling_island_add(pollset->polling_island, fd); + } + } else if (pollset->polling_island == NULL) { + new = polling_island_ref(fd->polling_island); + } else if (pollset->polling_island != fd->polling_island) { + new = polling_island_merge(pollset->polling_island, fd->polling_island); + } else { + new = polling_island_update(pollset->polling_island, 1); + } + + fd->polling_island = pollset->polling_island = new; + + gpr_mu_unlock(&fd->polling_island_mu); + gpr_mu_unlock(&pollset->mu); +} + +//////////////////////////////////////////////////////////////////////////////// +// Engine binding + +static void shutdown_engine(void) { destroy_fd_freelist(); } + +static const grpc_event_engine_vtable vtable = { + .pollset_size = sizeof(grpc_pollset), + + .fd_create = fd_create, + .fd_wrapped_fd = fd_wrapped_fd, + .fd_orphan = fd_orphan, + .fd_shutdown = fd_shutdown, + .fd_notify_on_read = fd_notify_on_read, + .fd_notify_on_write = fd_notify_on_write, + + .pollset_init = pollset_init, + .pollset_shutdown = pollset_shutdown, + .pollset_reset = pollset_reset, + .pollset_destroy = pollset_destroy, + .pollset_work = pollset_work, + .pollset_kick = pollset_kick, + .pollset_add_fd = pollset_add_fd, + + .pollset_set_create = pollset_set_create, + .pollset_set_destroy = pollset_set_destroy, + .pollset_set_add_pollset = pollset_set_add_pollset, + .pollset_set_del_pollset = pollset_set_del_pollset, + .pollset_set_add_pollset_set = pollset_set_add_pollset_set, + .pollset_set_del_pollset_set = pollset_set_del_pollset_set, + .pollset_set_add_fd = pollset_set_add_fd, + .pollset_set_del_fd = pollset_set_del_fd, + + .kick_poller = kick_poller, + + .shutdown_engine = shutdown_engine, +}; + +static bool is_epoll_available(void) { + abort(); + return false; +} + +const grpc_event_engine_vtable *grpc_init_poll_posix(void) { + if (!is_epoll_available()) { + return NULL; + } + return &vtable; +} From d7d6eed78822ed8de14fdf9b39255365946cf8c4 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Tue, 31 May 2016 09:44:08 -0700 Subject: [PATCH 0029/1039] Correct typo --- src/core/lib/iomgr/ev_posix.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/core/lib/iomgr/ev_posix.c b/src/core/lib/iomgr/ev_posix.c index 404ef2a64b9..96399ef8379 100644 --- a/src/core/lib/iomgr/ev_posix.c +++ b/src/core/lib/iomgr/ev_posix.c @@ -44,7 +44,7 @@ #include #include -#include "src/core/lib/iomgr/ev_epoll_linux.h" +#include "src/core/lib/iomgr/ev_epoll_posix.h" #include "src/core/lib/iomgr/ev_poll_posix.h" #include "src/core/lib/support/env.h" From 8c6c9067bc001a7cd94c2689570a6ec27affee82 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Tue, 31 May 2016 10:57:20 -0700 Subject: [PATCH 0030/1039] run epoll tests too --- tools/run_tests/run_tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index ddaf96c345a..f3b191feab7 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -157,7 +157,7 @@ class CLanguage(object): 'windows': ['all'], 'mac': ['all'], 'posix': ['all'], - 'linux': ['poll'], + 'linux': ['poll', 'epoll'], } for target in binaries: polling_strategies = (POLLING_STRATEGIES[self.platform] From dbd03b5e2d97f869089b2165edd2c419101b1c83 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Tue, 31 May 2016 13:00:33 -0700 Subject: [PATCH 0031/1039] Make client_channels be across all clients, not per-client --- test/cpp/qps/driver.cc | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/test/cpp/qps/driver.cc b/test/cpp/qps/driver.cc index 04b2b453f9e..da47a985b8c 100644 --- a/test/cpp/qps/driver.cc +++ b/test/cpp/qps/driver.cc @@ -306,6 +306,7 @@ std::unique_ptr RunScenario( // clients is array rather than std::vector to avoid gcc-4.4 issues // where class contained in std::vector must have a copy constructor auto* clients = new ClientData[num_clients]; + size_t channels_allocated = 0; for (size_t i = 0; i < num_clients; i++) { const auto& worker = workers[i + num_servers]; gpr_log(GPR_INFO, "Starting client on %s (worker #%d)", worker.c_str(), @@ -341,6 +342,13 @@ std::unique_ptr RunScenario( } } + size_t num_channels = + (i == num_clients - 1) + ? channels_allocated - client_config.client_channels() + : client_config.client_channels() / num_clients; + channels_allocated += num_channels; + per_client_config.set_client_channels(num_channels); + ClientArgs args; *args.mutable_setup() = per_client_config; clients[i].stream = From 1adb13143f21a54a3e3d7018d0db4d15d78c1adc Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Tue, 31 May 2016 13:04:32 -0700 Subject: [PATCH 0032/1039] Add comment --- test/cpp/qps/driver.cc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/test/cpp/qps/driver.cc b/test/cpp/qps/driver.cc index da47a985b8c..147c540840e 100644 --- a/test/cpp/qps/driver.cc +++ b/test/cpp/qps/driver.cc @@ -342,6 +342,8 @@ std::unique_ptr RunScenario( } } + // Reduce channel count so that total channels specified is held regardless + // of the number of clients available size_t num_channels = (i == num_clients - 1) ? channels_allocated - client_config.client_channels() From 248fcbf06d9b34f67a8aa2dcc552b6180894d093 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Tue, 31 May 2016 13:56:10 -0700 Subject: [PATCH 0033/1039] Fixes --- src/core/ext/transport/chttp2/transport/chttp2_transport.c | 5 +++++ src/core/lib/iomgr/workqueue_posix.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index b94a112b4a1..03e02d25226 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -200,6 +200,8 @@ static void destruct_transport(grpc_exec_ctx *exec_ctx, gpr_free(ping); } + GRPC_WORKQUEUE_UNREF(exec_ctx, t->executor.workqueue, "transport"); + gpr_free(t->peer_string); gpr_free(t); } @@ -734,6 +736,7 @@ void grpc_chttp2_initiate_write( t->executor.writing_needed = true; if (!t->executor.writing_initiated) { t->executor.writing_initiated = true; + REF_TRANSPORT(t, "initiate_writing"); grpc_workqueue_enqueue(exec_ctx, t->executor.workqueue, &t->initiate_writing, GRPC_ERROR_NONE); } @@ -754,6 +757,7 @@ static void initiate_writing_locked(grpc_exec_ctx *exec_ctx, prevent_endpoint_shutdown(t); grpc_exec_ctx_sched(exec_ctx, &t->writing_action, GRPC_ERROR_NONE, NULL); } + UNREF_TRANSPORT(exec_ctx, t, "initiate_writing"); } static void initiate_writing(grpc_exec_ctx *exec_ctx, void *arg, @@ -812,6 +816,7 @@ static void terminate_writing_with_lock(grpc_exec_ctx *exec_ctx, t->executor.writing_active = false; if (t->executor.writing_needed) { + REF_TRANSPORT(t, "initiate_writing"); initiate_writing_locked(exec_ctx, t, NULL, NULL); } else { t->executor.writing_initiated = false; diff --git a/src/core/lib/iomgr/workqueue_posix.c b/src/core/lib/iomgr/workqueue_posix.c index c6323e0594d..bcbc2699be0 100644 --- a/src/core/lib/iomgr/workqueue_posix.c +++ b/src/core/lib/iomgr/workqueue_posix.c @@ -70,7 +70,7 @@ grpc_error *grpc_workqueue_create(grpc_exec_ctx *exec_ctx, static void workqueue_destroy(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue) { - GPR_ASSERT(grpc_closure_list_empty(workqueue->closure_list)); + grpc_exec_ctx_enqueue_list(exec_ctx, &workqueue->closure_list, NULL); grpc_fd_shutdown(exec_ctx, workqueue->wakeup_read_fd); } From d4aa7cf5a502462debd4a42f0daf4c018c824263 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Tue, 31 May 2016 13:59:25 -0700 Subject: [PATCH 0034/1039] Fix negation --- test/cpp/qps/driver.cc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/test/cpp/qps/driver.cc b/test/cpp/qps/driver.cc index 147c540840e..298087dd36f 100644 --- a/test/cpp/qps/driver.cc +++ b/test/cpp/qps/driver.cc @@ -346,9 +346,10 @@ std::unique_ptr RunScenario( // of the number of clients available size_t num_channels = (i == num_clients - 1) - ? channels_allocated - client_config.client_channels() + ? client_config.client_channels() - channels_allocated : client_config.client_channels() / num_clients; channels_allocated += num_channels; + gpr_log(GPR_DEBUG, "Client %d gets %d channels", i, num_channels); per_client_config.set_client_channels(num_channels); ClientArgs args; From d7906f5d0eaa684175eabc586d1d04b49de53696 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Tue, 31 May 2016 16:07:27 -0700 Subject: [PATCH 0035/1039] Add state machine for writes --- .../chttp2/transport/chttp2_plugin.c | 3 + .../chttp2/transport/chttp2_transport.c | 189 ++++++++++++++---- .../ext/transport/chttp2/transport/internal.h | 30 ++- .../ext/transport/chttp2/transport/parsing.c | 8 +- .../transport/chttp2/transport/stream_lists.c | 4 +- 5 files changed, 180 insertions(+), 54 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_plugin.c b/src/core/ext/transport/chttp2/transport/chttp2_plugin.c index bd87253ed32..7d5279b9da4 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_plugin.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_plugin.c @@ -36,11 +36,14 @@ #include "src/core/lib/debug/trace.h" #include "src/core/lib/transport/metadata.h" +extern int grpc_http_write_state_trace; + void grpc_chttp2_plugin_init(void) { grpc_chttp2_base64_encode_and_huffman_compress = grpc_chttp2_base64_encode_and_huffman_compress_impl; grpc_register_tracer("http", &grpc_http_trace); grpc_register_tracer("flowctl", &grpc_flowctl_trace); + grpc_register_tracer("http_write_state", &grpc_http_write_state_trace); } void grpc_chttp2_plugin_shutdown(void) {} diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index 03e02d25226..752591a6dbd 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -63,6 +63,7 @@ int grpc_http_trace = 0; int grpc_flowctl_trace = 0; +int grpc_http_write_state_trace = 0; #define TRANSPORT_FROM_WRITING(tw) \ ((grpc_chttp2_transport *)((char *)(tw)-offsetof(grpc_chttp2_transport, \ @@ -91,6 +92,8 @@ static void parsing_action(grpc_exec_ctx *exec_ctx, void *t, grpc_error *error); static void initiate_writing(grpc_exec_ctx *exec_ctx, void *t, grpc_error *error); +static void start_writing(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t); + /** Set a transport level setting, and push it to our peer */ static void push_setting(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, grpc_chttp2_setting_id id, uint32_t value); @@ -294,7 +297,7 @@ static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, gpr_slice_buffer_add( &t->global.qbuf, gpr_slice_from_copied_string(GRPC_CHTTP2_CLIENT_CONNECT_STRING)); - grpc_chttp2_initiate_write(exec_ctx, &t->global); + grpc_chttp2_initiate_write(exec_ctx, &t->global, false); } /* 8 is a random stab in the dark as to a good initial size: it's small enough that it shouldn't waste memory for infrequently used connections, yet @@ -642,6 +645,36 @@ grpc_chttp2_stream_parsing *grpc_chttp2_parsing_accept_stream( * LOCK MANAGEMENT */ +static const char *write_state_name(grpc_chttp2_write_state state) { + switch (state) { + case GRPC_CHTTP2_WRITING_INACTIVE: + return "INACTIVE"; + case GRPC_CHTTP2_WRITE_REQUESTED_NO_POLLER: + return "REQUESTED[p=0]"; + case GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER: + return "REQUESTED[p=1]"; + case GRPC_CHTTP2_WRITE_SCHEDULED: + return "SCHEDULED"; + case GRPC_CHTTP2_WRITING: + return "WRITING"; + case GRPC_CHTTP2_WRITING_STALE_WITH_POLLER: + return "WRITING[p=1]"; + case GRPC_CHTTP2_WRITING_STALE_NO_POLLER: + return "WRITING[p=0]"; + } + GPR_UNREACHABLE_CODE(return "UNKNOWN"); +} + +static void set_write_state(grpc_chttp2_transport *t, + grpc_chttp2_write_state state, const char *reason) { + if (grpc_http_write_state_trace) { + gpr_log(GPR_DEBUG, "W:%p %s -> %s because %s", t, + write_state_name(t->executor.write_state), write_state_name(state), + reason); + } + t->executor.write_state = state; +} + static void finish_global_actions(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t) { grpc_chttp2_executor_action_header *hdr; @@ -666,8 +699,27 @@ static void finish_global_actions(grpc_exec_ctx *exec_ctx, continue; } else { t->executor.global_active = false; + switch (t->executor.write_state) { + case GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER: + set_write_state(t, GRPC_CHTTP2_WRITE_SCHEDULED, "unlocking"); + REF_TRANSPORT(t, "initiate_writing"); + gpr_mu_unlock(&t->executor.mu); + grpc_workqueue_enqueue(exec_ctx, t->executor.workqueue, + &t->initiate_writing, GRPC_ERROR_NONE); + break; + case GRPC_CHTTP2_WRITE_REQUESTED_NO_POLLER: + start_writing(exec_ctx, t); + gpr_mu_unlock(&t->executor.mu); + break; + case GRPC_CHTTP2_WRITING_INACTIVE: + case GRPC_CHTTP2_WRITING: + case GRPC_CHTTP2_WRITING_STALE_WITH_POLLER: + case GRPC_CHTTP2_WRITING_STALE_NO_POLLER: + case GRPC_CHTTP2_WRITE_SCHEDULED: + gpr_mu_unlock(&t->executor.mu); + break; + } } - gpr_mu_unlock(&t->executor.mu); break; } } @@ -730,33 +782,69 @@ void grpc_chttp2_run_with_global_lock(grpc_exec_ctx *exec_ctx, * OUTPUT PROCESSING */ -void grpc_chttp2_initiate_write( - grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global) { +void grpc_chttp2_initiate_write(grpc_exec_ctx *exec_ctx, + grpc_chttp2_transport_global *transport_global, + bool covered_by_poller) { grpc_chttp2_transport *t = TRANSPORT_FROM_GLOBAL(transport_global); - t->executor.writing_needed = true; - if (!t->executor.writing_initiated) { - t->executor.writing_initiated = true; - REF_TRANSPORT(t, "initiate_writing"); - grpc_workqueue_enqueue(exec_ctx, t->executor.workqueue, - &t->initiate_writing, GRPC_ERROR_NONE); + switch (t->executor.write_state) { + case GRPC_CHTTP2_WRITING_INACTIVE: + set_write_state(t, covered_by_poller + ? GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER + : GRPC_CHTTP2_WRITE_REQUESTED_NO_POLLER, + "initiate_write"); + break; + case GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER: + /* nothing to do: write already requested */ + break; + case GRPC_CHTTP2_WRITE_REQUESTED_NO_POLLER: + if (covered_by_poller) { + /* upgrade to note poller is available to cover the write */ + set_write_state(t, GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER, + "initiate_write"); + } + break; + case GRPC_CHTTP2_WRITE_SCHEDULED: + /* nothing to do: write already scheduled */ + break; + case GRPC_CHTTP2_WRITING: + set_write_state(t, + covered_by_poller ? GRPC_CHTTP2_WRITING_STALE_WITH_POLLER + : GRPC_CHTTP2_WRITING_STALE_NO_POLLER, + "initiate_write"); + break; + case GRPC_CHTTP2_WRITING_STALE_WITH_POLLER: + /* nothing to do: write already requested */ + break; + case GRPC_CHTTP2_WRITING_STALE_NO_POLLER: + if (covered_by_poller) { + /* upgrade to note poller is available to cover the write */ + set_write_state(t, GRPC_CHTTP2_WRITING_STALE_WITH_POLLER, + "initiate_write"); + } + break; } } -static void initiate_writing_locked(grpc_exec_ctx *exec_ctx, - grpc_chttp2_transport *t, - grpc_chttp2_stream *s_unused, - void *arg_ignored) { - GPR_ASSERT(t->executor.writing_needed); - GPR_ASSERT(!t->executor.writing_active); +static void start_writing(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t) { + GPR_ASSERT(t->executor.write_state == GRPC_CHTTP2_WRITE_SCHEDULED || + t->executor.write_state == GRPC_CHTTP2_WRITE_REQUESTED_NO_POLLER); if (!t->closed && grpc_chttp2_unlocking_check_writes(exec_ctx, &t->global, &t->writing, t->executor.parsing_active)) { - t->executor.writing_needed = false; - t->executor.writing_active = true; + set_write_state(t, GRPC_CHTTP2_WRITING, "start_writing"); REF_TRANSPORT(t, "writing"); prevent_endpoint_shutdown(t); grpc_exec_ctx_sched(exec_ctx, &t->writing_action, GRPC_ERROR_NONE, NULL); + } else { + set_write_state(t, GRPC_CHTTP2_WRITING_INACTIVE, "start_writing"); } +} + +static void initiate_writing_locked(grpc_exec_ctx *exec_ctx, + grpc_chttp2_transport *t, + grpc_chttp2_stream *s_unused, + void *arg_ignored) { + start_writing(exec_ctx, t); UNREF_TRANSPORT(exec_ctx, t, "initiate_writing"); } @@ -768,11 +856,12 @@ static void initiate_writing(grpc_exec_ctx *exec_ctx, void *arg, void grpc_chttp2_become_writable(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, - grpc_chttp2_stream_global *stream_global) { + grpc_chttp2_stream_global *stream_global, + bool covered_by_poller) { if (!TRANSPORT_FROM_GLOBAL(transport_global)->closed && grpc_chttp2_list_add_writable_stream(transport_global, stream_global)) { GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing"); - grpc_chttp2_initiate_write(exec_ctx, transport_global); + grpc_chttp2_initiate_write(exec_ctx, transport_global, covered_by_poller); } } @@ -788,7 +877,7 @@ static void push_setting(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, if (use_value != t->global.settings[GRPC_LOCAL_SETTINGS][id]) { t->global.settings[GRPC_LOCAL_SETTINGS][id] = use_value; t->global.dirtied_local_settings = 1; - grpc_chttp2_initiate_write(exec_ctx, &t->global); + grpc_chttp2_initiate_write(exec_ctx, &t->global, false); } } @@ -814,13 +903,25 @@ static void terminate_writing_with_lock(grpc_exec_ctx *exec_ctx, GRPC_CHTTP2_STREAM_UNREF(exec_ctx, stream_global, "finish_writes"); } - t->executor.writing_active = false; - if (t->executor.writing_needed) { - REF_TRANSPORT(t, "initiate_writing"); - initiate_writing_locked(exec_ctx, t, NULL, NULL); - } else { - t->executor.writing_initiated = false; + switch (t->executor.write_state) { + case GRPC_CHTTP2_WRITING_INACTIVE: + case GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER: + case GRPC_CHTTP2_WRITE_REQUESTED_NO_POLLER: + case GRPC_CHTTP2_WRITE_SCHEDULED: + GPR_UNREACHABLE_CODE(break); + case GRPC_CHTTP2_WRITING: + set_write_state(t, GRPC_CHTTP2_WRITING_INACTIVE, "terminate_writing"); + break; + case GRPC_CHTTP2_WRITING_STALE_WITH_POLLER: + set_write_state(t, GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER, + "terminate_writing"); + break; + case GRPC_CHTTP2_WRITING_STALE_NO_POLLER: + set_write_state(t, GRPC_CHTTP2_WRITE_REQUESTED_NO_POLLER, + "terminate_writing"); + break; } + if (t->ep && !t->endpoint_reading) { destroy_endpoint(exec_ctx, t); } @@ -907,7 +1008,8 @@ static void maybe_start_some_streams( stream_global->id, STREAM_FROM_GLOBAL(stream_global)); stream_global->in_stream_map = true; transport_global->concurrent_stream_count++; - grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, + true); } /* cancel out streams that will never be started */ while (transport_global->next_stream_id >= MAX_CLIENT_STREAM_ID && @@ -1036,8 +1138,8 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, maybe_start_some_streams(exec_ctx, transport_global); } else { GPR_ASSERT(stream_global->id != 0); - grpc_chttp2_become_writable(exec_ctx, transport_global, - stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, + true); } } else { grpc_chttp2_complete_closure_step( @@ -1061,7 +1163,8 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, } else { stream_global->send_message = op->send_message; if (stream_global->id != 0) { - grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, + true); } } } @@ -1100,7 +1203,8 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, } else if (stream_global->id != 0) { /* TODO(ctiller): check if there's flow control for any outstanding bytes before going writable */ - grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, + true); } } } @@ -1165,7 +1269,7 @@ static void send_ping_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, p->id[7] = (uint8_t)(t->global.ping_counter & 0xff); p->on_recv = on_recv; gpr_slice_buffer_add(&t->global.qbuf, grpc_chttp2_ping_create(0, p->id)); - grpc_chttp2_initiate_write(exec_ctx, &t->global); + grpc_chttp2_initiate_write(exec_ctx, &t->global, true); } static void ack_ping_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, @@ -1225,7 +1329,7 @@ static void perform_transport_op_locked(grpc_exec_ctx *exec_ctx, close_transport = grpc_chttp2_has_streams(t) ? GRPC_ERROR_NONE : GRPC_ERROR_CREATE("GOAWAY sent"); - grpc_chttp2_initiate_write(exec_ctx, &t->global); + grpc_chttp2_initiate_write(exec_ctx, &t->global, false); } if (op->set_accept_stream) { @@ -1392,7 +1496,7 @@ static void cancel_from_api(grpc_exec_ctx *exec_ctx, stream_global->id, (uint32_t)grpc_chttp2_grpc_status_to_http2_error(status), &stream_global->stats.outgoing)); - grpc_chttp2_initiate_write(exec_ctx, transport_global); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false); } grpc_chttp2_fake_status(exec_ctx, transport_global, stream_global, status, NULL); @@ -1477,7 +1581,8 @@ void grpc_chttp2_mark_stream_closed( } if (close_writes && !stream_global->write_closed) { stream_global->write_closed = true; - if (TRANSPORT_FROM_GLOBAL(transport_global)->executor.writing_active) { + if (TRANSPORT_FROM_GLOBAL(transport_global)->executor.write_state != + GRPC_CHTTP2_WRITING_INACTIVE) { GRPC_CHTTP2_STREAM_REF(stream_global, "finish_writes"); grpc_chttp2_list_add_closed_waiting_for_writing(transport_global, stream_global); @@ -1614,7 +1719,7 @@ static void close_from_api(grpc_exec_ctx *exec_ctx, } grpc_chttp2_mark_stream_closed(exec_ctx, transport_global, stream_global, 1, 1, err); - grpc_chttp2_initiate_write(exec_ctx, transport_global); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false); } static void cancel_stream_cb(grpc_chttp2_transport_global *transport_global, @@ -1657,7 +1762,8 @@ static void update_global_window(void *args, uint32_t id, void *stream) { is_zero = stream_global->outgoing_window <= 0; if (was_zero && !is_zero) { - grpc_chttp2_become_writable(a->exec_ctx, transport_global, stream_global); + grpc_chttp2_become_writable(a->exec_ctx, transport_global, stream_global, + true); } } @@ -1736,7 +1842,7 @@ static void post_parse_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, /* copy parsing qbuf to global qbuf */ if (t->parsing.qbuf.count > 0) { gpr_slice_buffer_move_into(&t->parsing.qbuf, &t->global.qbuf); - grpc_chttp2_initiate_write(exec_ctx, transport_global); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false); } /* merge stream lists */ grpc_chttp2_stream_map_move_into(&t->new_stream_map, &t->parsing_stream_map); @@ -1787,7 +1893,7 @@ static void post_reading_action_locked(grpc_exec_ctx *exec_ctx, if (error != GRPC_ERROR_NONE) { drop_connection(exec_ctx, t, GRPC_ERROR_REF(error)); t->endpoint_reading = 0; - if (!t->executor.writing_active && t->ep) { + if (t->executor.write_state == GRPC_CHTTP2_WRITING_INACTIVE && t->ep) { grpc_endpoint_destroy(exec_ctx, t->ep); t->ep = NULL; /* safe as we still have a ref for read */ @@ -1907,7 +2013,8 @@ static void incoming_byte_stream_update_flow_control( add_max_recv_bytes); grpc_chttp2_list_add_unannounced_incoming_window_available(transport_global, stream_global); - grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, + false); } } diff --git a/src/core/ext/transport/chttp2/transport/internal.h b/src/core/ext/transport/chttp2/transport/internal.h index bdc4a69f4fd..988e6b6b004 100644 --- a/src/core/ext/transport/chttp2/transport/internal.h +++ b/src/core/ext/transport/chttp2/transport/internal.h @@ -304,6 +304,22 @@ typedef struct grpc_chttp2_executor_action_header { void *arg; } grpc_chttp2_executor_action_header; +typedef enum { + /** no writing activity */ + GRPC_CHTTP2_WRITING_INACTIVE, + /** write has been requested, but not scheduled yet */ + GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER, + GRPC_CHTTP2_WRITE_REQUESTED_NO_POLLER, + /** write has been requested and scheduled against the workqueue */ + GRPC_CHTTP2_WRITE_SCHEDULED, + /** write has been initiated after being reaped from the workqueue */ + GRPC_CHTTP2_WRITING, + /** write has been initiated, AND another write needs to be started once it's + done */ + GRPC_CHTTP2_WRITING_STALE_WITH_POLLER, + GRPC_CHTTP2_WRITING_STALE_NO_POLLER, +} grpc_chttp2_write_state; + struct grpc_chttp2_transport { grpc_transport base; /* must be first */ gpr_refcount refs; @@ -319,14 +335,10 @@ struct grpc_chttp2_transport { /** is a thread currently in the global lock */ bool global_active; - /** is a write currently initiated */ - bool writing_initiated; - /** is a write actually going on right now */ - bool writing_active; - /** is a write needed */ - bool writing_needed; /** is a thread currently parsing */ bool parsing_active; + /** write execution state of the transport */ + grpc_chttp2_write_state write_state; grpc_chttp2_executor_action_header *pending_actions_head; grpc_chttp2_executor_action_header *pending_actions_tail; @@ -523,7 +535,8 @@ struct grpc_chttp2_stream { After writing, a follow-up check is made to see if another round of writing should be performed. */ void grpc_chttp2_initiate_write(grpc_exec_ctx *exec_ctx, - grpc_chttp2_transport_global *transport_global); + grpc_chttp2_transport_global *transport_global, + bool covered_by_poller); /** Someone is unlocking the transport mutex: check to see if writes are required, and schedule them if so */ @@ -826,6 +839,7 @@ void grpc_chttp2_ack_ping(grpc_exec_ctx *exec_ctx, ref will be dropped in writing.c */ void grpc_chttp2_become_writable(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, - grpc_chttp2_stream_global *stream_global); + grpc_chttp2_stream_global *stream_global, + bool covered_by_poller); #endif /* GRPC_CORE_EXT_TRANSPORT_CHTTP2_TRANSPORT_INTERNAL_H */ diff --git a/src/core/ext/transport/chttp2/transport/parsing.c b/src/core/ext/transport/chttp2/transport/parsing.c index 3d3abccd204..8f80cb61b28 100644 --- a/src/core/ext/transport/chttp2/transport/parsing.c +++ b/src/core/ext/transport/chttp2/transport/parsing.c @@ -156,7 +156,8 @@ void grpc_chttp2_publish_reads( if (was_zero && !is_zero) { while (grpc_chttp2_list_pop_stalled_by_transport(transport_global, &stream_global)) { - grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, + false); } } @@ -168,7 +169,7 @@ void grpc_chttp2_publish_reads( announce_incoming_window, announce_bytes); GRPC_CHTTP2_FLOW_CREDIT_TRANSPORT("parsed", transport_parsing, incoming_window, announce_bytes); - grpc_chttp2_initiate_write(exec_ctx, transport_global); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false); } /* for each stream that saw an update, fixup global state */ @@ -191,7 +192,8 @@ void grpc_chttp2_publish_reads( outgoing_window); is_zero = stream_global->outgoing_window <= 0; if (was_zero && !is_zero) { - grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, + false); } stream_global->max_recv_bytes -= (uint32_t)GPR_MIN( diff --git a/src/core/ext/transport/chttp2/transport/stream_lists.c b/src/core/ext/transport/chttp2/transport/stream_lists.c index f3690015eaa..f227a0af982 100644 --- a/src/core/ext/transport/chttp2/transport/stream_lists.c +++ b/src/core/ext/transport/chttp2/transport/stream_lists.c @@ -344,8 +344,8 @@ void grpc_chttp2_list_flush_writing_stalled_by_transport( while (stream_list_pop(transport, &stream, GRPC_CHTTP2_LIST_WRITING_STALLED_BY_TRANSPORT)) { if (is_window_available) { - grpc_chttp2_become_writable(exec_ctx, &transport->global, - &stream->global); + grpc_chttp2_become_writable(exec_ctx, &transport->global, &stream->global, + true); } else { grpc_chttp2_list_add_stalled_by_transport(transport_writing, &stream->writing); From 2cdf0a8b912db8882c5f2a7c93ecde9e5a172073 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Tue, 31 May 2016 21:47:19 -0700 Subject: [PATCH 0036/1039] Fix ref counting --- .../chttp2/transport/chttp2_transport.c | 48 ++++++++++++++----- src/core/lib/iomgr/workqueue.h | 2 +- 2 files changed, 36 insertions(+), 14 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index 752591a6dbd..fba1f0e3140 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -93,6 +93,8 @@ static void initiate_writing(grpc_exec_ctx *exec_ctx, void *t, grpc_error *error); static void start_writing(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t); +static void end_waiting_for_write(grpc_exec_ctx *exec_ctx, + grpc_chttp2_transport *t, grpc_error *error); /** Set a transport level setting, and push it to our peer */ static void push_setting(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, @@ -209,6 +211,7 @@ static void destruct_transport(grpc_exec_ctx *exec_ctx, gpr_free(t); } +/*#define REFCOUNTING_DEBUG 1*/ #ifdef REFCOUNTING_DEBUG #define REF_TRANSPORT(t, r) ref_transport(t, r, __FILE__, __LINE__) #define UNREF_TRANSPORT(cl, t, r) unref_transport(cl, t, r, __FILE__, __LINE__) @@ -457,6 +460,9 @@ static void close_transport_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, grpc_error *error) { if (!t->closed) { + if (grpc_http_write_state_trace) { + gpr_log(GPR_DEBUG, "W:%p close transport", t); + } t->closed = 1; connectivity_state_set(exec_ctx, &t->global, GRPC_CHANNEL_FATAL_FAILURE, GRPC_ERROR_REF(error), "close_transport"); @@ -836,7 +842,17 @@ static void start_writing(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t) { prevent_endpoint_shutdown(t); grpc_exec_ctx_sched(exec_ctx, &t->writing_action, GRPC_ERROR_NONE, NULL); } else { - set_write_state(t, GRPC_CHTTP2_WRITING_INACTIVE, "start_writing"); + if (t->closed) { + set_write_state(t, GRPC_CHTTP2_WRITING_INACTIVE, + "start_writing:transport_closed"); + } else { + set_write_state(t, GRPC_CHTTP2_WRITING_INACTIVE, + "start_writing:nothing_to_write"); + } + end_waiting_for_write(exec_ctx, t, GRPC_ERROR_CREATE("Nothing to write")); + if (!t->endpoint_reading) { + destroy_endpoint(exec_ctx, t); + } } } @@ -881,6 +897,18 @@ static void push_setting(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, } } +static void end_waiting_for_write(grpc_exec_ctx *exec_ctx, + grpc_chttp2_transport *t, grpc_error *error) { + grpc_chttp2_stream_global *stream_global; + while (grpc_chttp2_list_pop_closed_waiting_for_writing(&t->global, + &stream_global)) { + fail_pending_writes(exec_ctx, &t->global, stream_global, + GRPC_ERROR_REF(error)); + GRPC_CHTTP2_STREAM_UNREF(exec_ctx, stream_global, "finish_writes"); + } + GRPC_ERROR_UNREF(error); +} + static void terminate_writing_with_lock(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, grpc_chttp2_stream *s_ignored, @@ -895,13 +923,7 @@ static void terminate_writing_with_lock(grpc_exec_ctx *exec_ctx, grpc_chttp2_cleanup_writing(exec_ctx, &t->global, &t->writing); - grpc_chttp2_stream_global *stream_global; - while (grpc_chttp2_list_pop_closed_waiting_for_writing(&t->global, - &stream_global)) { - fail_pending_writes(exec_ctx, &t->global, stream_global, - GRPC_ERROR_REF(error)); - GRPC_CHTTP2_STREAM_UNREF(exec_ctx, stream_global, "finish_writes"); - } + end_waiting_for_write(exec_ctx, t, error); switch (t->executor.write_state) { case GRPC_CHTTP2_WRITING_INACTIVE: @@ -927,7 +949,6 @@ static void terminate_writing_with_lock(grpc_exec_ctx *exec_ctx, } UNREF_TRANSPORT(exec_ctx, t, "writing"); - GRPC_ERROR_UNREF(error); } void grpc_chttp2_terminate_writing(grpc_exec_ctx *exec_ctx, @@ -1893,11 +1914,12 @@ static void post_reading_action_locked(grpc_exec_ctx *exec_ctx, if (error != GRPC_ERROR_NONE) { drop_connection(exec_ctx, t, GRPC_ERROR_REF(error)); t->endpoint_reading = 0; + if (grpc_http_write_state_trace) { + gpr_log(GPR_DEBUG, "R:%p -> 0 ws=%s", t, + write_state_name(t->executor.write_state)); + } if (t->executor.write_state == GRPC_CHTTP2_WRITING_INACTIVE && t->ep) { - grpc_endpoint_destroy(exec_ctx, t->ep); - t->ep = NULL; - /* safe as we still have a ref for read */ - UNREF_TRANSPORT(exec_ctx, t, "disconnect"); + destroy_endpoint(exec_ctx, t); } } else if (!t->closed) { keep_reading = true; diff --git a/src/core/lib/iomgr/workqueue.h b/src/core/lib/iomgr/workqueue.h index 88e4211c471..f4a2cfa4fd2 100644 --- a/src/core/lib/iomgr/workqueue.h +++ b/src/core/lib/iomgr/workqueue.h @@ -56,7 +56,7 @@ grpc_error *grpc_workqueue_create(grpc_exec_ctx *exec_ctx, void grpc_workqueue_flush(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue); -#define GRPC_WORKQUEUE_REFCOUNT_DEBUG +/*#define GRPC_WORKQUEUE_REFCOUNT_DEBUG*/ #ifdef GRPC_WORKQUEUE_REFCOUNT_DEBUG #define GRPC_WORKQUEUE_REF(p, r) \ grpc_workqueue_ref((p), __FILE__, __LINE__, (r)) From caad18024d93f3c12d3e7f4c3f903e7df5a4c5c8 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Tue, 31 May 2016 22:05:00 -0700 Subject: [PATCH 0037/1039] Fix crash --- src/core/ext/transport/chttp2/transport/chttp2_transport.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index fba1f0e3140..827d65f6105 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -850,7 +850,7 @@ static void start_writing(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t) { "start_writing:nothing_to_write"); } end_waiting_for_write(exec_ctx, t, GRPC_ERROR_CREATE("Nothing to write")); - if (!t->endpoint_reading) { + if (t->ep && !t->endpoint_reading) { destroy_endpoint(exec_ctx, t); } } From 09c464a1424067fc3b5e5f0b1280a8f8c3444342 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 1 Jun 2016 08:16:29 -0700 Subject: [PATCH 0038/1039] Fix too much metadata --- .../transport/chttp2/transport/chttp2_transport.c | 13 +++++++++---- src/core/lib/iomgr/error.c | 2 ++ src/core/lib/iomgr/error.h | 1 + 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index 827d65f6105..31b43fb026a 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -1140,10 +1140,15 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, transport_global->settings[GRPC_PEER_SETTINGS] [GRPC_CHTTP2_SETTINGS_MAX_HEADER_LIST_SIZE]; if (metadata_size > metadata_peer_limit) { - gpr_log(GPR_DEBUG, - "to-be-sent initial metadata size exceeds peer limit " - "(%lu vs. %lu)", - metadata_size, metadata_peer_limit); + grpc_chttp2_complete_closure_step( + exec_ctx, transport_global, stream_global, + &stream_global->send_initial_metadata_finished, + grpc_error_set_int( + grpc_error_set_int( + GRPC_ERROR_CREATE( + "to-be-sent initial metadata size exceeds peer limit"), + GRPC_ERROR_INT_SIZE, (intptr_t)metadata_size), + GRPC_ERROR_INT_LIMIT, (intptr_t)metadata_peer_limit)); cancel_from_api(exec_ctx, transport_global, stream_global, GRPC_STATUS_RESOURCE_EXHAUSTED); } else { diff --git a/src/core/lib/iomgr/error.c b/src/core/lib/iomgr/error.c index b38584100df..35154268f31 100644 --- a/src/core/lib/iomgr/error.c +++ b/src/core/lib/iomgr/error.c @@ -102,6 +102,8 @@ static const char *error_int_name(grpc_error_ints key) { return "grpc_status"; case GRPC_ERROR_INT_OFFSET: return "offset"; + case GRPC_ERROR_INT_LIMIT: + return "limit"; case GRPC_ERROR_INT_INDEX: return "index"; case GRPC_ERROR_INT_SIZE: diff --git a/src/core/lib/iomgr/error.h b/src/core/lib/iomgr/error.h index 695724c0be8..ede159526e1 100644 --- a/src/core/lib/iomgr/error.h +++ b/src/core/lib/iomgr/error.h @@ -62,6 +62,7 @@ typedef enum { GRPC_ERROR_INT_OFFSET, GRPC_ERROR_INT_INDEX, GRPC_ERROR_INT_SIZE, + GRPC_ERROR_INT_LIMIT, GRPC_ERROR_INT_HTTP2_ERROR, GRPC_ERROR_INT_TSI_CODE, GRPC_ERROR_INT_SECURITY_STATUS, From 5098f91159f2a5c0494688b8cfaff4debef5686f Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Tue, 31 May 2016 10:58:17 -0700 Subject: [PATCH 0039/1039] Rewrite all the pollset and fd functions in ev_epoll_linux.c --- src/core/lib/iomgr/ev_epoll_linux.c | 161 +++++++--------------------- 1 file changed, 38 insertions(+), 123 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 7793a952016..1201c10a7e0 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -67,10 +67,10 @@ struct polling_island; struct grpc_fd { int fd; /* refst format: - bit0: 1=active/0=orphaned - bit1-n: refcount - meaning that mostly we ref by two to avoid altering the orphaned bit, - and just unref by 1 when we're ready to flag the object as orphaned */ + bit 0 : 1=Active / 0=Orphaned + bits 1-n : refcount + - ref/unref by two to avoid altering the orphaned bit + - To orphan, unref by 1 */ gpr_atm refst; gpr_mu mu; @@ -84,12 +84,11 @@ struct grpc_fd { /* Mutex protecting the 'polling_island' field */ gpr_mu pi_mu; - /* The polling island to which this fd belongs to. An fd belongs to exactly - one polling island */ + /* The polling island to which this fd belongs to. + * An fd belongs to exactly one polling island */ struct polling_island *polling_island; struct grpc_fd *freelist_next; - grpc_closure *on_done_closure; grpc_iomgr_object iomgr_object; @@ -141,7 +140,6 @@ typedef struct polling_island { /* Polling islands that are no longer needed are kept in a freelist so that they can be reused. This field points to the next polling island in the - free list. Note that this is only used if the polling island is in the free list */ struct polling_island *next_free; } polling_island; @@ -185,7 +183,7 @@ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, * TODO: sreek - Might have to unref the fds (assuming whether we add a ref to * the fd when adding it to the epollset) */ /* The caller is expected to hold pi->mu lock before calling this function */ -static void polling_island_clear_fds_locked(polling_island *pi) { +static void polling_island_remove_all_fds_locked(polling_island *pi) { int err; size_t i; @@ -392,7 +390,7 @@ polling_island *polling_island_merge(polling_island *p, polling_island *q) { // Move all the fds from polling_island p to polling_island q polling_island_add_fds_locked(q, p->fds, p->fd_cnt); - polling_island_clear_fds_locked(p); + polling_island_remove_all_fds_locked(p); q->ref_cnt += p->ref_cnt; @@ -411,14 +409,7 @@ static void polling_island_global_init() { * pollset declarations */ -typedef struct grpc_cached_wakeup_fd { - grpc_wakeup_fd fd; - struct grpc_cached_wakeup_fd *next; -} grpc_cached_wakeup_fd; - struct grpc_pollset_worker { - grpc_cached_wakeup_fd *wakeup_fd; - int reevaluate_polling_on_wakeup; int kicked_specifically; pthread_t pt_id; struct grpc_pollset_worker *next; @@ -441,9 +432,6 @@ struct grpc_pollset { /* The polling island to which this fd belongs to. An fd belongs to exactly one polling island */ struct polling_island *polling_island; - - /* Local cache of eventfds for workers */ - grpc_cached_wakeup_fd *local_wakeup_cache; }; /* Add an fd to a pollset */ @@ -465,8 +453,6 @@ static int poll_deadline_to_millis_timeout(gpr_timespec deadline, /* Allow kick to wakeup the currently polling worker */ #define GRPC_POLLSET_CAN_KICK_SELF 1 -/* Force the wakee to repoll when awoken */ -#define GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP 2 /* As per pollset_kick, with an extended set of flags (defined above) -- mostly for fd_posix's use. */ static void pollset_kick_ext(grpc_pollset *p, @@ -815,34 +801,25 @@ static void pollset_kick_ext(grpc_pollset *p, if (specific_worker != NULL) { if (specific_worker == GRPC_POLLSET_KICK_BROADCAST) { GPR_TIMER_BEGIN("pollset_kick_ext.broadcast", 0); - GPR_ASSERT((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) == 0); for (specific_worker = p->root_worker.next; specific_worker != &p->root_worker; specific_worker = specific_worker->next) { - grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + pthread_kill(specific_worker->pt_id, SIGUSR1); } p->kicked_without_pollers = 1; GPR_TIMER_END("pollset_kick_ext.broadcast", 0); } else if (gpr_tls_get(&g_current_thread_worker) != (intptr_t)specific_worker) { GPR_TIMER_MARK("different_thread_worker", 0); - if ((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) != 0) { - specific_worker->reevaluate_polling_on_wakeup = 1; - } specific_worker->kicked_specifically = 1; - grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); /* TODO (sreek): Refactor this into a separate file*/ pthread_kill(specific_worker->pt_id, SIGUSR1); } else if ((flags & GRPC_POLLSET_CAN_KICK_SELF) != 0) { GPR_TIMER_MARK("kick_yoself", 0); - if ((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) != 0) { - specific_worker->reevaluate_polling_on_wakeup = 1; - } specific_worker->kicked_specifically = 1; - grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + pthread_kill(specific_worker->pt_id, SIGUSR1); } } else if (gpr_tls_get(&g_current_thread_poller) != (intptr_t)p) { - GPR_ASSERT((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) == 0); GPR_TIMER_MARK("kick_anonymous", 0); specific_worker = pop_front_worker(p); if (specific_worker != NULL) { @@ -860,7 +837,7 @@ static void pollset_kick_ext(grpc_pollset *p, if (specific_worker != NULL) { GPR_TIMER_MARK("finally_kick", 0); push_back_worker(p, specific_worker); - grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); + pthread_kill(specific_worker->pt_id, SIGUSR1); } } else { GPR_TIMER_MARK("kicked_no_pollers", 0); @@ -911,8 +888,6 @@ static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { pollset->shutting_down = 0; pollset->called_shutdown = 0; pollset->kicked_without_pollers = 0; - pollset->local_wakeup_cache = NULL; - pollset->kicked_without_pollers = 0; multipoll_with_epoll_pollset_create_efd(pollset); } @@ -926,12 +901,6 @@ static void pollset_destroy(grpc_pollset *pollset) { multipoll_with_epoll_pollset_destroy(pollset); - while (pollset->local_wakeup_cache) { - grpc_cached_wakeup_fd *next = pollset->local_wakeup_cache->next; - grpc_wakeup_fd_destroy(&pollset->local_wakeup_cache->fd); - gpr_free(pollset->local_wakeup_cache); - pollset->local_wakeup_cache = next; - } gpr_mu_destroy(&pollset->pi_mu); gpr_mu_destroy(&pollset->mu); } @@ -974,14 +943,6 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, GPR_TIMER_BEGIN("pollset_work", 0); /* this must happen before we (potentially) drop pollset->mu */ worker.next = worker.prev = NULL; - worker.reevaluate_polling_on_wakeup = 0; - if (pollset->local_wakeup_cache != NULL) { - worker.wakeup_fd = pollset->local_wakeup_cache; - pollset->local_wakeup_cache = worker.wakeup_fd->next; - } else { - worker.wakeup_fd = gpr_malloc(sizeof(*worker.wakeup_fd)); - grpc_wakeup_fd_init(&worker.wakeup_fd->fd); - } worker.kicked_specifically = 0; /* TODO(sreek): Abstract this thread id stuff out into a separate file */ @@ -1026,27 +987,12 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, gpr_mu_lock(&pollset->mu); locked = 1; } - /* If we're forced to re-evaluate polling (via pollset_kick with - GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) then we land here and force - a loop */ - if (worker.reevaluate_polling_on_wakeup) { - worker.reevaluate_polling_on_wakeup = 0; - pollset->kicked_without_pollers = 0; - if (queued_work || worker.kicked_specifically) { - /* If there's queued work on the list, then set the deadline to be - immediate so we get back out of the polling loop quickly */ - deadline = gpr_inf_past(GPR_CLOCK_MONOTONIC); - } - keep_polling = 1; - } } if (added_worker) { remove_worker(pollset, &worker); gpr_tls_set(&g_current_thread_worker, 0); } - /* release wakeup fd to the local pool */ - worker.wakeup_fd->next = pollset->local_wakeup_cache; - pollset->local_wakeup_cache = worker.wakeup_fd; + /* check shutdown conditions */ if (pollset->shutting_down) { if (pollset_has_workers(pollset)) { @@ -1135,10 +1081,9 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, } } else if (fd->polling_island == NULL) { pi_new = polling_island_update_and_lock(pollset->polling_island, 1, 1); - } else if (pollset->polling_island == NULL) { pi_new = polling_island_update_and_lock(fd->polling_island, 1, 1); - } else { // Non null and different + } else { pi_new = polling_island_merge(fd->polling_island, pollset->polling_island); } @@ -1182,9 +1127,7 @@ static void multipoll_with_epoll_pollset_maybe_work_and_unlock( struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; int epoll_fd = pollset->epoll_fd; int ep_rv; - int poll_rv; int timeout_ms; - struct pollfd pfds[2]; /* If you want to ignore epoll's ability to sanely handle parallel pollers, * for a more apples-to-apples performance comparison with poll, add a @@ -1196,63 +1139,35 @@ static void multipoll_with_epoll_pollset_maybe_work_and_unlock( timeout_ms = poll_deadline_to_millis_timeout(deadline, now); - pfds[0].fd = GRPC_WAKEUP_FD_GET_READ_FD(&worker->wakeup_fd->fd); - pfds[0].events = POLLIN; - pfds[0].revents = 0; - pfds[1].fd = epoll_fd; - pfds[1].events = POLLIN; - pfds[1].revents = 0; - - /* TODO(vpai): Consider first doing a 0 timeout poll here to avoid - even going into the blocking annotation if possible */ - GPR_TIMER_BEGIN("poll", 0); - GRPC_SCHEDULING_START_BLOCKING_REGION; - poll_rv = grpc_poll_function(pfds, 2, timeout_ms); - GRPC_SCHEDULING_END_BLOCKING_REGION; - GPR_TIMER_END("poll", 0); - - if (poll_rv < 0) { - if (errno != EINTR) { - gpr_log(GPR_ERROR, "poll() failed: %s", strerror(errno)); - } - } else if (poll_rv == 0) { - /* do nothing */ - } else { - if (pfds[0].revents) { - grpc_wakeup_fd_consume_wakeup(&worker->wakeup_fd->fd); - } - if (pfds[1].revents) { - do { - /* The following epoll_wait never blocks; it has a timeout of 0 */ - ep_rv = epoll_wait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, 0); - if (ep_rv < 0) { - if (errno != EINTR) { - gpr_log(GPR_ERROR, "epoll_wait() failed: %s", strerror(errno)); - } + do { + /* The following epoll_wait never blocks; it has a timeout of 0 */ + ep_rv = epoll_wait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, timeout_ms); + if (ep_rv < 0) { + if (errno != EINTR) { + gpr_log(GPR_ERROR, "epoll_wait() failed: %s", strerror(errno)); + } + } else { + int i; + for (i = 0; i < ep_rv; ++i) { + grpc_fd *fd = ep_ev[i].data.ptr; + /* TODO(klempner): We might want to consider making err and pri + * separate events */ + int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); + int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); + int write_ev = ep_ev[i].events & EPOLLOUT; + if (fd == NULL) { + grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); } else { - int i; - for (i = 0; i < ep_rv; ++i) { - grpc_fd *fd = ep_ev[i].data.ptr; - /* TODO(klempner): We might want to consider making err and pri - * separate events */ - int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); - int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); - int write_ev = ep_ev[i].events & EPOLLOUT; - if (fd == NULL) { - grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); - } else { - if (read_ev || cancel) { - fd_become_readable(exec_ctx, fd); - } - if (write_ev || cancel) { - fd_become_writable(exec_ctx, fd); - } - } + if (read_ev || cancel) { + fd_become_readable(exec_ctx, fd); + } + if (write_ev || cancel) { + fd_become_writable(exec_ctx, fd); } } - } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); + } } - } + } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); } static void multipoll_with_epoll_pollset_finish_shutdown( From 894900734662b7012d7b858db716c3cc1e9f8179 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 1 Jun 2016 08:52:43 -0700 Subject: [PATCH 0040/1039] Add a epoll_test.c file to experiment. REMOVE this from the final commit --- Makefile | 36 +++ build.yaml | 12 + test/core/network_benchmarks/epoll_test.c | 263 ++++++++++++++++++ .../network_benchmarks/low_level_ping_pong.c | 2 + tools/run_tests/sources_and_headers.json | 16 ++ tools/run_tests/tests.json | 15 + 6 files changed, 344 insertions(+) create mode 100644 test/core/network_benchmarks/epoll_test.c diff --git a/Makefile b/Makefile index 063698d943f..235f32d9a30 100644 --- a/Makefile +++ b/Makefile @@ -903,6 +903,7 @@ dns_resolver_connectivity_test: $(BINDIR)/$(CONFIG)/dns_resolver_connectivity_te dns_resolver_test: $(BINDIR)/$(CONFIG)/dns_resolver_test dualstack_socket_test: $(BINDIR)/$(CONFIG)/dualstack_socket_test endpoint_pair_test: $(BINDIR)/$(CONFIG)/endpoint_pair_test +epoll_test: $(BINDIR)/$(CONFIG)/epoll_test fd_conservation_posix_test: $(BINDIR)/$(CONFIG)/fd_conservation_posix_test fd_posix_test: $(BINDIR)/$(CONFIG)/fd_posix_test fling_client: $(BINDIR)/$(CONFIG)/fling_client @@ -1234,6 +1235,7 @@ buildtests_c: privatelibs_c \ $(BINDIR)/$(CONFIG)/dns_resolver_test \ $(BINDIR)/$(CONFIG)/dualstack_socket_test \ $(BINDIR)/$(CONFIG)/endpoint_pair_test \ + $(BINDIR)/$(CONFIG)/epoll_test \ $(BINDIR)/$(CONFIG)/fd_conservation_posix_test \ $(BINDIR)/$(CONFIG)/fd_posix_test \ $(BINDIR)/$(CONFIG)/fling_client \ @@ -1497,6 +1499,8 @@ test_c: buildtests_c $(Q) $(BINDIR)/$(CONFIG)/dualstack_socket_test || ( echo test dualstack_socket_test failed ; exit 1 ) $(E) "[RUN] Testing endpoint_pair_test" $(Q) $(BINDIR)/$(CONFIG)/endpoint_pair_test || ( echo test endpoint_pair_test failed ; exit 1 ) + $(E) "[RUN] Testing epoll_test" + $(Q) $(BINDIR)/$(CONFIG)/epoll_test || ( echo test epoll_test failed ; exit 1 ) $(E) "[RUN] Testing fd_conservation_posix_test" $(Q) $(BINDIR)/$(CONFIG)/fd_conservation_posix_test || ( echo test fd_conservation_posix_test failed ; exit 1 ) $(E) "[RUN] Testing fd_posix_test" @@ -6577,6 +6581,38 @@ endif endif +EPOLL_TEST_SRC = \ + test/core/network_benchmarks/epoll_test.c \ + +EPOLL_TEST_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(EPOLL_TEST_SRC)))) +ifeq ($(NO_SECURE),true) + +# You can't build secure targets if you don't have OpenSSL. + +$(BINDIR)/$(CONFIG)/epoll_test: openssl_dep_error + +else + + + +$(BINDIR)/$(CONFIG)/epoll_test: $(EPOLL_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + $(E) "[LD] Linking $@" + $(Q) mkdir -p `dirname $@` + $(Q) $(LD) $(LDFLAGS) $(EPOLL_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/epoll_test + +endif + +$(OBJDIR)/$(CONFIG)/test/core/network_benchmarks/epoll_test.o: $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + +deps_epoll_test: $(EPOLL_TEST_OBJS:.o=.dep) + +ifneq ($(NO_SECURE),true) +ifneq ($(NO_DEPS),true) +-include $(EPOLL_TEST_OBJS:.o=.dep) +endif +endif + + FD_CONSERVATION_POSIX_TEST_SRC = \ test/core/iomgr/fd_conservation_posix_test.c \ diff --git a/build.yaml b/build.yaml index 2f3d07071da..db9787546ad 100644 --- a/build.yaml +++ b/build.yaml @@ -1321,6 +1321,18 @@ targets: - grpc - gpr_test_util - gpr +- name: epoll_test + build: test + language: c + src: + - test/core/network_benchmarks/epoll_test.c + deps: + - grpc_test_util + - grpc + - gpr_test_util + - gpr + platforms: + - linux - name: fd_conservation_posix_test build: test language: c diff --git a/test/core/network_benchmarks/epoll_test.c b/test/core/network_benchmarks/epoll_test.c new file mode 100644 index 00000000000..a918dd9bb94 --- /dev/null +++ b/test/core/network_benchmarks/epoll_test.c @@ -0,0 +1,263 @@ +/* + * + * Copyright 2015, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +/* TODO: sreek: REMOVE THIS FILE */ + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +int g_signal_num = SIGUSR1; + +int g_timeout_secs = 2; + +int g_eventfd_create = 1; +int g_eventfd_wakeup = 0; +int g_eventfd_teardown = 0; +int g_close_epoll_fd = 1; + +typedef struct thread_args { + gpr_thd_id id; + int epoll_fd; + int thread_num; +} thread_args; + +static int eventfd_create() { + if (!g_eventfd_create) { + return -1; + } + + int efd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + GPR_ASSERT(efd >= 0); + return efd; +} + +static void eventfd_wakeup(int efd) { + if (!g_eventfd_wakeup) { + return; + } + + int err; + do { + err = eventfd_write(efd, 1); + } while (err < 0 && errno == EINTR); +} + +static void epoll_teardown(int epoll_fd, int fd) { + if (!g_eventfd_teardown) { + return; + } + + if (epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, NULL) < 0) { + if (errno != ENOENT) { + gpr_log(GPR_ERROR, "epoll_ctl: %s", strerror(errno)); + GPR_ASSERT(0); + } + } +} + +/* Special case for epoll, where we need to create the fd ahead of time. */ +static int epoll_setup(int fd) { + int epoll_fd; + struct epoll_event ev; + + epoll_fd = epoll_create(1); + if (epoll_fd < 0) { + gpr_log(GPR_ERROR, "epoll_create: %s", strerror(errno)); + return -1; + } + + ev.events = (uint32_t)EPOLLIN; + ev.data.fd = fd; + if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) < 0) { + if (errno != EEXIST) { + gpr_log(GPR_ERROR, "epoll_ctl: %s", strerror(errno)); + return -1; + } + + gpr_log(GPR_ERROR, "epoll_ctl: The fd %d already exists", fd); + } + + return epoll_fd; +} + +#define GRPC_EPOLL_MAX_EVENTS 1000 +static void thread_main(void *args) { + int ep_rv; + struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; + int fd; + int i; + int cancel; + int read; + int write; + thread_args *thd_args = args; + sigset_t new_mask; + sigset_t orig_mask; + int keep_polling = 0; + + gpr_log(GPR_INFO, "Thread: %d Started", thd_args->thread_num); + + do { + keep_polling = 0; + + /* Mask the signal before getting the epoll_fd */ + gpr_log(GPR_INFO, "Thread: %d Blocking signal: %d", thd_args->thread_num, + g_signal_num); + sigemptyset(&new_mask); + sigaddset(&new_mask, g_signal_num); + pthread_sigmask(SIG_BLOCK, &new_mask, &orig_mask); + + gpr_log(GPR_INFO, "Thread: %d Waiting on epoll_wait()", + thd_args->thread_num); + ep_rv = epoll_pwait(thd_args->epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, + g_timeout_secs * 5000, &orig_mask); + gpr_log(GPR_INFO, "Thread: %d out of epoll_wait. ep_rv = %d", + thd_args->thread_num, ep_rv); + + if (ep_rv < 0) { + if (errno != EINTR) { + gpr_log(GPR_ERROR, "Thread: %d. epoll_wait failed with error: %d", + thd_args->thread_num, errno); + } else { + gpr_log(GPR_INFO, + "Thread: %d. epoll_wait was interrupted. Polling again >>>>>>>", + thd_args->thread_num); + keep_polling = 1; + } + } else { + if (ep_rv == 0) { + gpr_log(GPR_INFO, + "Thread: %d - epoll_wait returned 0. Most likely a timeout. " + "Polling again", + thd_args->thread_num); + keep_polling = 1; + } + + for (i = 0; i < ep_rv; i++) { + fd = ep_ev[i].data.fd; + cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); + read = ep_ev[i].events & (EPOLLIN | EPOLLPRI); + write = ep_ev[i].events & EPOLLOUT; + gpr_log(GPR_INFO, + "Thread: %d. epoll_wait returned that fd: %d has event of " + "interest. read: %d, write: %d, cancel: %d", + thd_args->thread_num, fd, read, write, cancel); + } + } + } while (keep_polling); +} + +static void close_fd(int fd) { + if (!g_close_epoll_fd) { + return; + } + + gpr_log(GPR_INFO, "*** Closing fd : %d ****", fd); + close(fd); + gpr_log(GPR_INFO, "*** Closed fd : %d ****", fd); +} + + +static void sig_handler(int sig_num) { + gpr_log(GPR_INFO, "<<<<< Received signal %d", sig_num); +} + +static void set_signal_handler() { + gpr_log(GPR_INFO, "Setting signal handler"); + signal(g_signal_num, sig_handler); +} + +#define NUM_THREADS 2 +int main(int argc, char **argv) { + int efd; + int epoll_fd; + int i; + thread_args thd_args[NUM_THREADS]; + gpr_thd_options options = gpr_thd_options_default(); + + set_signal_handler(); + + gpr_log(GPR_INFO, "Starting.."); + efd = eventfd_create(); + gpr_log(GPR_INFO, "Created event fd: %d", efd); + epoll_fd = epoll_setup(efd); + gpr_log(GPR_INFO, "Created epoll_fd: %d", epoll_fd); + + gpr_thd_options_set_joinable(&options); + for (i = 0; i < NUM_THREADS; i++) { + thd_args[i].thread_num = i; + thd_args[i].epoll_fd = epoll_fd; + gpr_log(GPR_INFO, "Starting thread: %d", i); + gpr_thd_new(&thd_args[i].id, thread_main, &thd_args[i], &options); + } + + sleep((unsigned)g_timeout_secs * 2); + + /* Send signals first */ + for (i = 0; i < NUM_THREADS; i++) { + gpr_log(GPR_INFO, "Sending signal to thread: %d", thd_args->thread_num); + pthread_kill(thd_args[i].id, g_signal_num); + gpr_log(GPR_INFO, "Sent signal to thread: %d >>>>>> ", + thd_args->thread_num); + } + + sleep((unsigned)g_timeout_secs * 2); + + close_fd(epoll_fd); + + sleep((unsigned)g_timeout_secs * 2); + + eventfd_wakeup(efd); + epoll_teardown(epoll_fd, efd); + + for (i = 0; i < NUM_THREADS; i++) { + gpr_thd_join(thd_args[i].id); + gpr_log(GPR_INFO, "Thread: %d joined", i); + } + + return 0; +} diff --git a/test/core/network_benchmarks/low_level_ping_pong.c b/test/core/network_benchmarks/low_level_ping_pong.c index 1b40895a719..b72a07778e0 100644 --- a/test/core/network_benchmarks/low_level_ping_pong.c +++ b/test/core/network_benchmarks/low_level_ping_pong.c @@ -44,6 +44,7 @@ #include #include #include +#include #ifdef __linux__ #include #endif @@ -84,6 +85,7 @@ typedef struct thread_args { static int read_bytes(int fd, char *buf, size_t read_size, int spin) { size_t bytes_read = 0; ssize_t err; + do { err = read(fd, buf + bytes_read, read_size - bytes_read); if (err < 0) { diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 97cc55db36e..85b71a8255a 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -301,6 +301,22 @@ "third_party": false, "type": "target" }, + { + "deps": [ + "gpr", + "gpr_test_util", + "grpc", + "grpc_test_util" + ], + "headers": [], + "language": "c", + "name": "epoll_test", + "src": [ + "test/core/network_benchmarks/epoll_test.c" + ], + "third_party": false, + "type": "target" + }, { "deps": [ "gpr", diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index 850f9474aec..be7b72f61d2 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -356,6 +356,21 @@ "windows" ] }, + { + "args": [], + "ci_platforms": [ + "linux" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "gtest": false, + "language": "c", + "name": "epoll_test", + "platforms": [ + "linux" + ] + }, { "args": [], "ci_platforms": [ From 9b9708a9c484c02ad8ef27060370ae605a83942c Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Wed, 1 Jun 2016 11:42:20 -0700 Subject: [PATCH 0041/1039] Allow Node users to set a custom logger and log verbosity. Defaults to existing core logger --- src/node/ext/node_grpc.cc | 117 ++++++++++++++++++++++++++++++++++++ src/node/index.js | 33 ++++++++++ src/node/src/common.js | 21 +++++++ src/node/src/credentials.js | 4 +- 4 files changed, 174 insertions(+), 1 deletion(-) diff --git a/src/node/ext/node_grpc.cc b/src/node/ext/node_grpc.cc index 6b6e42737b5..45762cad2e6 100644 --- a/src/node/ext/node_grpc.cc +++ b/src/node/ext/node_grpc.cc @@ -31,12 +31,15 @@ * */ +#include + #include #include #include #include "grpc/grpc.h" #include "grpc/grpc_security.h" #include "grpc/support/alloc.h" +#include "grpc/support/log.h" #include "call.h" #include "call_credentials.h" @@ -49,10 +52,22 @@ using v8::FunctionTemplate; using v8::Local; using v8::Value; +using v8::Number; using v8::Object; using v8::Uint32; using v8::String; +typedef struct logger_state { + Nan::Callback *callback; + std::list *pending_args; + uv_mutex_t mutex; + uv_async_t async; + // Indicates that a logger has been set + bool logger_set; +} logger_state; + +logger_state grpc_logger_state; + static char *pem_root_certs = NULL; void InitStatusConstants(Local exports) { @@ -235,6 +250,18 @@ void InitWriteFlags(Local exports) { Nan::Set(write_flags, Nan::New("NO_COMPRESS").ToLocalChecked(), NO_COMPRESS); } +void InitLogConstants(Local exports) { + Nan::HandleScope scope; + Local log_verbosity = Nan::New(); + Nan::Set(exports, Nan::New("logVerbosity").ToLocalChecked(), log_verbosity); + Local DEBUG(Nan::New(GPR_LOG_SEVERITY_DEBUG)); + Nan::Set(log_verbosity, Nan::New("DEBUG").ToLocalChecked(), DEBUG); + Local INFO(Nan::New(GPR_LOG_SEVERITY_INFO)); + Nan::Set(log_verbosity, Nan::New("INFO").ToLocalChecked(), INFO); + Local ERROR(Nan::New(GPR_LOG_SEVERITY_ERROR)); + Nan::Set(log_verbosity, Nan::New("ERROR").ToLocalChecked(), ERROR); +} + NAN_METHOD(MetadataKeyIsLegal) { if (!info[0]->IsString()) { return Nan::ThrowTypeError( @@ -298,16 +325,98 @@ NAN_METHOD(SetDefaultRootsPem) { } } +NAUV_WORK_CB(LogMessagesCallback) { + Nan::HandleScope scope; + std::list args; + uv_mutex_lock(&grpc_logger_state.mutex); + args.splice(args.begin(), *grpc_logger_state.pending_args); + uv_mutex_unlock(&grpc_logger_state.mutex); + /* Call the callback with each log message */ + while (!args.empty()) { + gpr_log_func_args *arg = args.front(); + args.pop_front(); + Local file = Nan::New(arg->file).ToLocalChecked(); + Local line = Nan::New(arg->line); + Local severity = Nan::New( + gpr_log_severity_string(arg->severity)).ToLocalChecked(); + Local message = Nan::New(arg->message).ToLocalChecked(); + const int argc = 4; + Local argv[argc] = {file, line, severity, message}; + grpc_logger_state.callback->Call(argc, argv); + delete[] arg->message; + delete arg; + } +} + +void node_log_func(gpr_log_func_args *args) { + // TODO(mlumish): Use the core's log formatter when it becomes available + gpr_log_func_args *args_copy = new gpr_log_func_args; + size_t message_len = strlen(args->message) + 1; + char *message = new char[message_len]; + memcpy(message, args->message, message_len); + memcpy(args_copy, args, sizeof(gpr_log_func_args)); + args_copy->message = message; + + uv_mutex_lock(&grpc_logger_state.mutex); + grpc_logger_state.pending_args->push_back(args_copy); + uv_mutex_unlock(&grpc_logger_state.mutex); + + uv_async_send(&grpc_logger_state.async); +} + +void init_logger() { + memset(&grpc_logger_state, 0, sizeof(logger_state)); + grpc_logger_state.pending_args = new std::list(); + uv_mutex_init(&grpc_logger_state.mutex); + uv_async_init(uv_default_loop(), + &grpc_logger_state.async, + LogMessagesCallback); + uv_unref((uv_handle_t*)&grpc_logger_state.async); + grpc_logger_state.logger_set = false; + + gpr_log_verbosity_init(); +} + +/* This registers a JavaScript logger for messages from the gRPC core. Because + that handler has to be run in the context of the JavaScript event loop, it + will be run asynchronously. To minimize the problems that could cause for + debugging, we leave core to do its default synchronous logging until a + JavaScript logger is set */ +NAN_METHOD(SetDefaultLoggerCallback) { + if (!info[0]->IsFunction()) { + return Nan::ThrowTypeError( + "setDefaultLoggerCallback's argument must be a function"); + } + if (!grpc_logger_state.logger_set) { + gpr_set_log_function(node_log_func); + grpc_logger_state.logger_set = true; + } + grpc_logger_state.callback = new Nan::Callback(info[0].As()); +} + +NAN_METHOD(SetLogVerbosity) { + if (!info[0]->IsUint32()) { + return Nan::ThrowTypeError( + "setLogVerbosity's argument must be a number"); + } + gpr_log_severity severity = static_cast( + Nan::To(info[0]).FromJust()); + gpr_set_log_verbosity(severity); +} + void init(Local exports) { Nan::HandleScope scope; grpc_init(); grpc_set_ssl_roots_override_callback(get_ssl_roots_override); + init_logger(); + InitStatusConstants(exports); InitCallErrorConstants(exports); InitOpTypeConstants(exports); InitPropagateConstants(exports); InitConnectivityStateConstants(exports); InitWriteFlags(exports); + InitLogConstants(exports); grpc::node::Call::Init(exports); grpc::node::CallCredentials::Init(exports); @@ -333,6 +442,14 @@ void init(Local exports) { Nan::GetFunction( Nan::New(SetDefaultRootsPem) ).ToLocalChecked()); + Nan::Set(exports, Nan::New("setDefaultLoggerCallback").ToLocalChecked(), + Nan::GetFunction( + Nan::New(SetDefaultLoggerCallback) + ).ToLocalChecked()); + Nan::Set(exports, Nan::New("setLogVerbosity").ToLocalChecked(), + Nan::GetFunction( + Nan::New(SetLogVerbosity) + ).ToLocalChecked()); } NODE_MODULE(grpc_node, init) diff --git a/src/node/index.js b/src/node/index.js index 66664d94b5a..b85cec68414 100644 --- a/src/node/index.js +++ b/src/node/index.js @@ -46,6 +46,8 @@ var client = require('./src/client.js'); var server = require('./src/server.js'); +var common = require('./src/common.js'); + var Metadata = require('./src/metadata.js'); var grpc = require('./src/grpc_extension'); @@ -122,6 +124,32 @@ exports.load = function load(filename, format, options) { return loadObject(builder.ns, options); }; +/** + * Sets the logger function for the gRPC module. For debugging purposes, the C + * core will log synchronously directly to stdout unless this function is + * called. Note: the output format here is intended to be informational, and + * is not guaranteed to stay the same in the future. + * Logs will be directed to logger.error. + * @param {Console} logger A Console-like object. + */ +exports.setLogger = function setLogger(logger) { + common.logger = logger; + grpc.setDefaultLoggerCallback(function(file, line, severity, message) { + file = path.basename(file); + logger.error(severity + '\t' + file + ':' + line + ']\t' + message); + }); +}; + +/** + * Sets the logger verbosity for gRPC module logging. The options are members + * of the grpc.logVerbosity map. + * @param {Number} verbosity The minimum severity to log + */ +exports.setLogVerbosity = function setLogVerbosity(verbosity) { + common.logVerbosity = verbosity; + grpc.setLogVerbosity(verbosity); +}; + /** * @see module:src/server.Server */ @@ -152,6 +180,11 @@ exports.callError = grpc.callError; */ exports.writeFlags = grpc.writeFlags; +/** + * Log verbosity setting name to code number mapping + */ +exports.logVerbosity = grpc.logVerbosity; + /** * Credentials factories */ diff --git a/src/node/src/common.js b/src/node/src/common.js index 8cf43b7a84c..e2d1a50df4a 100644 --- a/src/node/src/common.js +++ b/src/node/src/common.js @@ -157,3 +157,24 @@ exports.getProtobufServiceAttrs = function getProtobufServiceAttrs(service, }]; })); }; + +/** + * The logger object for the gRPC module. Defaults to console. + */ +exports.logger = console; + +/** + * The current logging verbosity. UNSET corresponds to logging everything + */ +exports.logVerbosity = -1; + +/** + * Log a message if the severity is at least as high as the current verbosity + * @param {Number} severity A value of the grpc.logVerbosity map + * @param {String} message The message to log + */ +exports.log = function log(severity, message) { + if (severity >= exports.logVerbosity) { + exports.logger.error(message); + } +}; diff --git a/src/node/src/credentials.js b/src/node/src/credentials.js index a12eade4e1f..b746d0625df 100644 --- a/src/node/src/credentials.js +++ b/src/node/src/credentials.js @@ -69,6 +69,8 @@ var ChannelCredentials = grpc.ChannelCredentials; var Metadata = require('./metadata.js'); +var common = require('./common.js'); + /** * Create an SSL Credentials object. If using a client-side certificate, both * the second and third arguments must be passed. @@ -120,7 +122,7 @@ exports.createFromGoogleCredential = function(google_credential) { var service_url = auth_context.service_url; google_credential.getRequestMetadata(service_url, function(err, header) { if (err) { - console.log('Auth error:', err); + common.log(grpc.logVerbosity.INFO, 'Auth error:' + err); callback(err); return; } From 4a43fe4a2fc62ab179e6fff7ca10f2c9c13dd1fc Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Wed, 1 Jun 2016 16:05:48 -0700 Subject: [PATCH 0042/1039] Minor change to common.js --- src/node/src/common.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/node/src/common.js b/src/node/src/common.js index e2d1a50df4a..22159dd39f7 100644 --- a/src/node/src/common.js +++ b/src/node/src/common.js @@ -164,9 +164,9 @@ exports.getProtobufServiceAttrs = function getProtobufServiceAttrs(service, exports.logger = console; /** - * The current logging verbosity. UNSET corresponds to logging everything + * The current logging verbosity. 0 corresponds to logging everything */ -exports.logVerbosity = -1; +exports.logVerbosity = 0; /** * Log a message if the severity is at least as high as the current verbosity From e1ad41a0ff89ed215458441b4ee08e8d4ecd9670 Mon Sep 17 00:00:00 2001 From: Alistair Veitch Date: Thu, 2 Jun 2016 14:16:05 -0700 Subject: [PATCH 0043/1039] change Metric->Resource, simplify Resource definition --- src/core/ext/census/gen/census.pb.c | 38 +++++------ src/core/ext/census/gen/census.pb.h | 98 +++++++++++++---------------- src/proto/census/census.proto | 62 +++++++++--------- 3 files changed, 86 insertions(+), 112 deletions(-) diff --git a/src/core/ext/census/gen/census.pb.c b/src/core/ext/census/gen/census.pb.c index d614636c908..c73c2c9b7cc 100644 --- a/src/core/ext/census/gen/census.pb.c +++ b/src/core/ext/census/gen/census.pb.c @@ -53,23 +53,17 @@ const pb_field_t google_census_Timestamp_fields[3] = { PB_LAST_FIELD }; -const pb_field_t google_census_Metric_fields[5] = { - PB_FIELD( 1, STRING , OPTIONAL, CALLBACK, FIRST, google_census_Metric, name, name, 0), - PB_FIELD( 2, STRING , OPTIONAL, CALLBACK, OTHER, google_census_Metric, description, name, 0), - PB_FIELD( 3, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_Metric, unit, description, &google_census_Metric_MeasurementUnit_fields), - PB_FIELD( 4, INT32 , OPTIONAL, STATIC , OTHER, google_census_Metric, id, unit, 0), +const pb_field_t google_census_Resource_fields[4] = { + PB_FIELD( 1, STRING , OPTIONAL, CALLBACK, FIRST, google_census_Resource, name, name, 0), + PB_FIELD( 2, STRING , OPTIONAL, CALLBACK, OTHER, google_census_Resource, description, name, 0), + PB_FIELD( 3, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_Resource, unit, description, &google_census_Resource_MeasurementUnit_fields), PB_LAST_FIELD }; -const pb_field_t google_census_Metric_BasicUnit_fields[2] = { - PB_FIELD( 1, UENUM , OPTIONAL, STATIC , FIRST, google_census_Metric_BasicUnit, type, type, 0), - PB_LAST_FIELD -}; - -const pb_field_t google_census_Metric_MeasurementUnit_fields[4] = { - PB_FIELD( 1, INT32 , OPTIONAL, STATIC , FIRST, google_census_Metric_MeasurementUnit, prefix, prefix, 0), - PB_FIELD( 2, MESSAGE , REPEATED, CALLBACK, OTHER, google_census_Metric_MeasurementUnit, numerator, prefix, &google_census_Metric_BasicUnit_fields), - PB_FIELD( 3, MESSAGE , REPEATED, CALLBACK, OTHER, google_census_Metric_MeasurementUnit, denominator, numerator, &google_census_Metric_BasicUnit_fields), +const pb_field_t google_census_Resource_MeasurementUnit_fields[4] = { + PB_FIELD( 1, INT32 , OPTIONAL, STATIC , FIRST, google_census_Resource_MeasurementUnit, prefix, prefix, 0), + PB_FIELD( 2, UENUM , REPEATED, CALLBACK, OTHER, google_census_Resource_MeasurementUnit, numerator, prefix, 0), + PB_FIELD( 3, UENUM , REPEATED, CALLBACK, OTHER, google_census_Resource_MeasurementUnit, denominator, numerator, 0), PB_LAST_FIELD }; @@ -124,8 +118,8 @@ const pb_field_t google_census_Tag_fields[3] = { const pb_field_t google_census_View_fields[6] = { PB_FIELD( 1, STRING , OPTIONAL, CALLBACK, FIRST, google_census_View, name, name, 0), PB_FIELD( 2, STRING , OPTIONAL, CALLBACK, OTHER, google_census_View, description, name, 0), - PB_FIELD( 3, INT32 , OPTIONAL, STATIC , OTHER, google_census_View, metric_id, description, 0), - PB_FIELD( 4, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_View, aggregation, metric_id, &google_census_AggregationDescriptor_fields), + PB_FIELD( 3, STRING , OPTIONAL, CALLBACK, OTHER, google_census_View, resource_name, description, 0), + PB_FIELD( 4, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_View, aggregation, resource_name, &google_census_AggregationDescriptor_fields), PB_FIELD( 5, STRING , REPEATED, CALLBACK, OTHER, google_census_View, tag_key, aggregation, 0), PB_LAST_FIELD }; @@ -139,10 +133,10 @@ const pb_field_t google_census_Aggregation_fields[6] = { PB_LAST_FIELD }; -const pb_field_t google_census_ViewAggregations_fields[4] = { - PB_FIELD( 1, MESSAGE , REPEATED, CALLBACK, FIRST, google_census_ViewAggregations, aggregation, aggregation, &google_census_Aggregation_fields), - PB_FIELD( 2, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_ViewAggregations, start, aggregation, &google_census_Timestamp_fields), - PB_FIELD( 3, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_ViewAggregations, end, start, &google_census_Timestamp_fields), +const pb_field_t google_census_Metric_fields[4] = { + PB_FIELD( 1, MESSAGE , REPEATED, CALLBACK, FIRST, google_census_Metric, aggregation, aggregation, &google_census_Aggregation_fields), + PB_FIELD( 2, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_Metric, start, aggregation, &google_census_Timestamp_fields), + PB_FIELD( 3, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_Metric, end, start, &google_census_Timestamp_fields), PB_LAST_FIELD }; @@ -156,7 +150,7 @@ const pb_field_t google_census_ViewAggregations_fields[4] = { * numbers or field sizes that are larger than what can fit in 8 or 16 bit * field descriptors. */ -PB_STATIC_ASSERT((pb_membersize(google_census_Metric, unit) < 65536 && pb_membersize(google_census_Metric_MeasurementUnit, numerator) < 65536 && pb_membersize(google_census_Metric_MeasurementUnit, denominator) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 65536 && pb_membersize(google_census_Metric, unit) < 65536 && pb_membersize(google_census_Metric_MeasurementUnit, numerator) < 65536 && pb_membersize(google_census_Metric_MeasurementUnit, denominator) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 65536 && pb_membersize(google_census_Distribution, range) < 65536 && pb_membersize(google_census_IntervalStats, window) < 65536 && pb_membersize(google_census_IntervalStats_Window, window_size) < 65536 && pb_membersize(google_census_View, aggregation) < 65536 && pb_membersize(google_census_Aggregation, data.distribution) < 65536 && pb_membersize(google_census_Aggregation, data.interval_stats) < 65536 && pb_membersize(google_census_Metric, unit) < 65536 && pb_membersize(google_census_Metric_MeasurementUnit, numerator) < 65536 && pb_membersize(google_census_Metric_MeasurementUnit, denominator) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 65536 && pb_membersize(google_census_Metric, unit) < 65536 && pb_membersize(google_census_Metric_MeasurementUnit, numerator) < 65536 && pb_membersize(google_census_Metric_MeasurementUnit, denominator) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 65536 && pb_membersize(google_census_Distribution, range) < 65536 && pb_membersize(google_census_IntervalStats, window) < 65536 && pb_membersize(google_census_IntervalStats_Window, window_size) < 65536 && pb_membersize(google_census_View, aggregation) < 65536 && pb_membersize(google_census_Aggregation, data.distribution) < 65536 && pb_membersize(google_census_Aggregation, data.interval_stats) < 65536 && pb_membersize(google_census_Aggregation, tag) < 65536 && pb_membersize(google_census_ViewAggregations, aggregation) < 65536 && pb_membersize(google_census_ViewAggregations, start) < 65536 && pb_membersize(google_census_ViewAggregations, end) < 65536), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_google_census_Duration_google_census_Timestamp_google_census_Metric_google_census_Metric_BasicUnit_google_census_Metric_MeasurementUnit_google_census_AggregationDescriptor_google_census_AggregationDescriptor_BucketBoundaries_google_census_AggregationDescriptor_IntervalBoundaries_google_census_Distribution_google_census_Distribution_Range_google_census_IntervalStats_google_census_IntervalStats_Window_google_census_Tag_google_census_View_google_census_Aggregation_google_census_ViewAggregations) +PB_STATIC_ASSERT((pb_membersize(google_census_Resource, unit) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 65536 && pb_membersize(google_census_Resource, unit) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 65536 && pb_membersize(google_census_Distribution, range) < 65536 && pb_membersize(google_census_IntervalStats, window) < 65536 && pb_membersize(google_census_IntervalStats_Window, window_size) < 65536 && pb_membersize(google_census_View, aggregation) < 65536 && pb_membersize(google_census_Aggregation, data.distribution) < 65536 && pb_membersize(google_census_Aggregation, data.interval_stats) < 65536 && pb_membersize(google_census_Resource, unit) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 65536 && pb_membersize(google_census_Resource, unit) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 65536 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 65536 && pb_membersize(google_census_Distribution, range) < 65536 && pb_membersize(google_census_IntervalStats, window) < 65536 && pb_membersize(google_census_IntervalStats_Window, window_size) < 65536 && pb_membersize(google_census_View, aggregation) < 65536 && pb_membersize(google_census_Aggregation, data.distribution) < 65536 && pb_membersize(google_census_Aggregation, data.interval_stats) < 65536 && pb_membersize(google_census_Aggregation, tag) < 65536 && pb_membersize(google_census_Metric, aggregation) < 65536 && pb_membersize(google_census_Metric, start) < 65536 && pb_membersize(google_census_Metric, end) < 65536), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_google_census_Duration_google_census_Timestamp_google_census_Resource_google_census_Resource_MeasurementUnit_google_census_AggregationDescriptor_google_census_AggregationDescriptor_BucketBoundaries_google_census_AggregationDescriptor_IntervalBoundaries_google_census_Distribution_google_census_Distribution_Range_google_census_IntervalStats_google_census_IntervalStats_Window_google_census_Tag_google_census_View_google_census_Aggregation_google_census_Metric) #endif #if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT) @@ -167,7 +161,7 @@ PB_STATIC_ASSERT((pb_membersize(google_census_Metric, unit) < 65536 && pb_member * numbers or field sizes that are larger than what can fit in the default * 8 bit descriptors. */ -PB_STATIC_ASSERT((pb_membersize(google_census_Metric, unit) < 256 && pb_membersize(google_census_Metric_MeasurementUnit, numerator) < 256 && pb_membersize(google_census_Metric_MeasurementUnit, denominator) < 256 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 256 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 256 && pb_membersize(google_census_Metric, unit) < 256 && pb_membersize(google_census_Metric_MeasurementUnit, numerator) < 256 && pb_membersize(google_census_Metric_MeasurementUnit, denominator) < 256 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 256 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 256 && pb_membersize(google_census_Distribution, range) < 256 && pb_membersize(google_census_IntervalStats, window) < 256 && pb_membersize(google_census_IntervalStats_Window, window_size) < 256 && pb_membersize(google_census_View, aggregation) < 256 && pb_membersize(google_census_Aggregation, data.distribution) < 256 && pb_membersize(google_census_Aggregation, data.interval_stats) < 256 && pb_membersize(google_census_Metric, unit) < 256 && pb_membersize(google_census_Metric_MeasurementUnit, numerator) < 256 && pb_membersize(google_census_Metric_MeasurementUnit, denominator) < 256 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 256 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 256 && pb_membersize(google_census_Metric, unit) < 256 && pb_membersize(google_census_Metric_MeasurementUnit, numerator) < 256 && pb_membersize(google_census_Metric_MeasurementUnit, denominator) < 256 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 256 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 256 && pb_membersize(google_census_Distribution, range) < 256 && pb_membersize(google_census_IntervalStats, window) < 256 && pb_membersize(google_census_IntervalStats_Window, window_size) < 256 && pb_membersize(google_census_View, aggregation) < 256 && pb_membersize(google_census_Aggregation, data.distribution) < 256 && pb_membersize(google_census_Aggregation, data.interval_stats) < 256 && pb_membersize(google_census_Aggregation, tag) < 256 && pb_membersize(google_census_ViewAggregations, aggregation) < 256 && pb_membersize(google_census_ViewAggregations, start) < 256 && pb_membersize(google_census_ViewAggregations, end) < 256), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_google_census_Duration_google_census_Timestamp_google_census_Metric_google_census_Metric_BasicUnit_google_census_Metric_MeasurementUnit_google_census_AggregationDescriptor_google_census_AggregationDescriptor_BucketBoundaries_google_census_AggregationDescriptor_IntervalBoundaries_google_census_Distribution_google_census_Distribution_Range_google_census_IntervalStats_google_census_IntervalStats_Window_google_census_Tag_google_census_View_google_census_Aggregation_google_census_ViewAggregations) +PB_STATIC_ASSERT((pb_membersize(google_census_Resource, unit) < 256 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 256 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 256 && pb_membersize(google_census_Resource, unit) < 256 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 256 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 256 && pb_membersize(google_census_Distribution, range) < 256 && pb_membersize(google_census_IntervalStats, window) < 256 && pb_membersize(google_census_IntervalStats_Window, window_size) < 256 && pb_membersize(google_census_View, aggregation) < 256 && pb_membersize(google_census_Aggregation, data.distribution) < 256 && pb_membersize(google_census_Aggregation, data.interval_stats) < 256 && pb_membersize(google_census_Resource, unit) < 256 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 256 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 256 && pb_membersize(google_census_Resource, unit) < 256 && pb_membersize(google_census_AggregationDescriptor, options.bucket_boundaries) < 256 && pb_membersize(google_census_AggregationDescriptor, options.interval_boundaries) < 256 && pb_membersize(google_census_Distribution, range) < 256 && pb_membersize(google_census_IntervalStats, window) < 256 && pb_membersize(google_census_IntervalStats_Window, window_size) < 256 && pb_membersize(google_census_View, aggregation) < 256 && pb_membersize(google_census_Aggregation, data.distribution) < 256 && pb_membersize(google_census_Aggregation, data.interval_stats) < 256 && pb_membersize(google_census_Aggregation, tag) < 256 && pb_membersize(google_census_Metric, aggregation) < 256 && pb_membersize(google_census_Metric, start) < 256 && pb_membersize(google_census_Metric, end) < 256), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_google_census_Duration_google_census_Timestamp_google_census_Resource_google_census_Resource_MeasurementUnit_google_census_AggregationDescriptor_google_census_AggregationDescriptor_BucketBoundaries_google_census_AggregationDescriptor_IntervalBoundaries_google_census_Distribution_google_census_Distribution_Range_google_census_IntervalStats_google_census_IntervalStats_Window_google_census_Tag_google_census_View_google_census_Aggregation_google_census_Metric) #endif diff --git a/src/core/ext/census/gen/census.pb.h b/src/core/ext/census/gen/census.pb.h index d040fe29e74..dfb53948204 100644 --- a/src/core/ext/census/gen/census.pb.h +++ b/src/core/ext/census/gen/census.pb.h @@ -45,14 +45,14 @@ extern "C" { #endif /* Enum definitions */ -typedef enum _google_census_Metric_BasicUnit_Measure { - google_census_Metric_BasicUnit_Measure_UNKNOWN = 0, - google_census_Metric_BasicUnit_Measure_BITS = 1, - google_census_Metric_BasicUnit_Measure_BYTES = 2, - google_census_Metric_BasicUnit_Measure_SECS = 3, - google_census_Metric_BasicUnit_Measure_CORES = 4, - google_census_Metric_BasicUnit_Measure_MAX_UNITS = 5 -} google_census_Metric_BasicUnit_Measure; +typedef enum _google_census_Resource_BasicUnit { + google_census_Resource_BasicUnit_UNKNOWN = 0, + google_census_Resource_BasicUnit_BITS = 1, + google_census_Resource_BasicUnit_BYTES = 2, + google_census_Resource_BasicUnit_SECS = 3, + google_census_Resource_BasicUnit_CORES = 4, + google_census_Resource_BasicUnit_MAX_UNITS = 5 +} google_census_Resource_BasicUnit; /* Struct definitions */ typedef struct _google_census_AggregationDescriptor_BucketBoundaries { @@ -89,17 +89,12 @@ typedef struct _google_census_Duration { int32_t nanos; } google_census_Duration; -typedef struct _google_census_Metric_BasicUnit { - bool has_type; - google_census_Metric_BasicUnit_Measure type; -} google_census_Metric_BasicUnit; - -typedef struct _google_census_Metric_MeasurementUnit { +typedef struct _google_census_Resource_MeasurementUnit { bool has_prefix; int32_t prefix; pb_callback_t numerator; pb_callback_t denominator; -} google_census_Metric_MeasurementUnit; +} google_census_Resource_MeasurementUnit; typedef struct _google_census_Tag { bool has_key; @@ -135,32 +130,29 @@ typedef struct _google_census_IntervalStats_Window { } google_census_IntervalStats_Window; typedef struct _google_census_Metric { + pb_callback_t aggregation; + bool has_start; + google_census_Timestamp start; + bool has_end; + google_census_Timestamp end; +} google_census_Metric; + +typedef struct _google_census_Resource { pb_callback_t name; pb_callback_t description; bool has_unit; - google_census_Metric_MeasurementUnit unit; - bool has_id; - int32_t id; -} google_census_Metric; + google_census_Resource_MeasurementUnit unit; +} google_census_Resource; typedef struct _google_census_View { pb_callback_t name; pb_callback_t description; - bool has_metric_id; - int32_t metric_id; + pb_callback_t resource_name; bool has_aggregation; google_census_AggregationDescriptor aggregation; pb_callback_t tag_key; } google_census_View; -typedef struct _google_census_ViewAggregations { - pb_callback_t aggregation; - bool has_start; - google_census_Timestamp start; - bool has_end; - google_census_Timestamp end; -} google_census_ViewAggregations; - typedef struct _google_census_Aggregation { pb_callback_t name; pb_callback_t description; @@ -177,9 +169,8 @@ typedef struct _google_census_Aggregation { /* Initializer values for message structs */ #define google_census_Duration_init_default {false, 0, false, 0} #define google_census_Timestamp_init_default {false, 0, false, 0} -#define google_census_Metric_init_default {{{NULL}, NULL}, {{NULL}, NULL}, false, google_census_Metric_MeasurementUnit_init_default, false, 0} -#define google_census_Metric_BasicUnit_init_default {false, (google_census_Metric_BasicUnit_Measure)0} -#define google_census_Metric_MeasurementUnit_init_default {false, 0, {{NULL}, NULL}, {{NULL}, NULL}} +#define google_census_Resource_init_default {{{NULL}, NULL}, {{NULL}, NULL}, false, google_census_Resource_MeasurementUnit_init_default} +#define google_census_Resource_MeasurementUnit_init_default {false, 0, {{NULL}, NULL}, {{NULL}, NULL}} #define google_census_AggregationDescriptor_init_default {0, {google_census_AggregationDescriptor_BucketBoundaries_init_default}} #define google_census_AggregationDescriptor_BucketBoundaries_init_default {{{NULL}, NULL}} #define google_census_AggregationDescriptor_IntervalBoundaries_init_default {{{NULL}, NULL}} @@ -188,14 +179,13 @@ typedef struct _google_census_Aggregation { #define google_census_IntervalStats_init_default {{{NULL}, NULL}} #define google_census_IntervalStats_Window_init_default {false, google_census_Duration_init_default, false, 0, false, 0} #define google_census_Tag_init_default {false, "", false, ""} -#define google_census_View_init_default {{{NULL}, NULL}, {{NULL}, NULL}, false, 0, false, google_census_AggregationDescriptor_init_default, {{NULL}, NULL}} +#define google_census_View_init_default {{{NULL}, NULL}, {{NULL}, NULL}, {{NULL}, NULL}, false, google_census_AggregationDescriptor_init_default, {{NULL}, NULL}} #define google_census_Aggregation_init_default {{{NULL}, NULL}, {{NULL}, NULL}, 0, {google_census_Distribution_init_default}, {{NULL}, NULL}} -#define google_census_ViewAggregations_init_default {{{NULL}, NULL}, false, google_census_Timestamp_init_default, false, google_census_Timestamp_init_default} +#define google_census_Metric_init_default {{{NULL}, NULL}, false, google_census_Timestamp_init_default, false, google_census_Timestamp_init_default} #define google_census_Duration_init_zero {false, 0, false, 0} #define google_census_Timestamp_init_zero {false, 0, false, 0} -#define google_census_Metric_init_zero {{{NULL}, NULL}, {{NULL}, NULL}, false, google_census_Metric_MeasurementUnit_init_zero, false, 0} -#define google_census_Metric_BasicUnit_init_zero {false, (google_census_Metric_BasicUnit_Measure)0} -#define google_census_Metric_MeasurementUnit_init_zero {false, 0, {{NULL}, NULL}, {{NULL}, NULL}} +#define google_census_Resource_init_zero {{{NULL}, NULL}, {{NULL}, NULL}, false, google_census_Resource_MeasurementUnit_init_zero} +#define google_census_Resource_MeasurementUnit_init_zero {false, 0, {{NULL}, NULL}, {{NULL}, NULL}} #define google_census_AggregationDescriptor_init_zero {0, {google_census_AggregationDescriptor_BucketBoundaries_init_zero}} #define google_census_AggregationDescriptor_BucketBoundaries_init_zero {{{NULL}, NULL}} #define google_census_AggregationDescriptor_IntervalBoundaries_init_zero {{{NULL}, NULL}} @@ -204,9 +194,9 @@ typedef struct _google_census_Aggregation { #define google_census_IntervalStats_init_zero {{{NULL}, NULL}} #define google_census_IntervalStats_Window_init_zero {false, google_census_Duration_init_zero, false, 0, false, 0} #define google_census_Tag_init_zero {false, "", false, ""} -#define google_census_View_init_zero {{{NULL}, NULL}, {{NULL}, NULL}, false, 0, false, google_census_AggregationDescriptor_init_zero, {{NULL}, NULL}} +#define google_census_View_init_zero {{{NULL}, NULL}, {{NULL}, NULL}, {{NULL}, NULL}, false, google_census_AggregationDescriptor_init_zero, {{NULL}, NULL}} #define google_census_Aggregation_init_zero {{{NULL}, NULL}, {{NULL}, NULL}, 0, {google_census_Distribution_init_zero}, {{NULL}, NULL}} -#define google_census_ViewAggregations_init_zero {{{NULL}, NULL}, false, google_census_Timestamp_init_zero, false, google_census_Timestamp_init_zero} +#define google_census_Metric_init_zero {{{NULL}, NULL}, false, google_census_Timestamp_init_zero, false, google_census_Timestamp_init_zero} /* Field tags (for use in manual encoding/decoding) */ #define google_census_AggregationDescriptor_BucketBoundaries_bounds_tag 1 @@ -219,10 +209,9 @@ typedef struct _google_census_Aggregation { #define google_census_Distribution_Range_max_tag 2 #define google_census_Duration_seconds_tag 1 #define google_census_Duration_nanos_tag 2 -#define google_census_Metric_BasicUnit_type_tag 1 -#define google_census_Metric_MeasurementUnit_prefix_tag 1 -#define google_census_Metric_MeasurementUnit_numerator_tag 2 -#define google_census_Metric_MeasurementUnit_denominator_tag 3 +#define google_census_Resource_MeasurementUnit_prefix_tag 1 +#define google_census_Resource_MeasurementUnit_numerator_tag 2 +#define google_census_Resource_MeasurementUnit_denominator_tag 3 #define google_census_Tag_key_tag 1 #define google_census_Tag_value_tag 2 #define google_census_Timestamp_seconds_tag 1 @@ -234,18 +223,17 @@ typedef struct _google_census_Aggregation { #define google_census_IntervalStats_Window_window_size_tag 1 #define google_census_IntervalStats_Window_count_tag 2 #define google_census_IntervalStats_Window_mean_tag 3 -#define google_census_Metric_name_tag 1 -#define google_census_Metric_description_tag 2 -#define google_census_Metric_unit_tag 3 -#define google_census_Metric_id_tag 4 +#define google_census_Metric_aggregation_tag 1 +#define google_census_Metric_start_tag 2 +#define google_census_Metric_end_tag 3 +#define google_census_Resource_name_tag 1 +#define google_census_Resource_description_tag 2 +#define google_census_Resource_unit_tag 3 #define google_census_View_name_tag 1 #define google_census_View_description_tag 2 -#define google_census_View_metric_id_tag 3 +#define google_census_View_resource_name_tag 3 #define google_census_View_aggregation_tag 4 #define google_census_View_tag_key_tag 5 -#define google_census_ViewAggregations_aggregation_tag 1 -#define google_census_ViewAggregations_start_tag 2 -#define google_census_ViewAggregations_end_tag 3 #define google_census_Aggregation_distribution_tag 3 #define google_census_Aggregation_interval_stats_tag 4 @@ -256,9 +244,8 @@ typedef struct _google_census_Aggregation { /* Struct field encoding specification for nanopb */ extern const pb_field_t google_census_Duration_fields[3]; extern const pb_field_t google_census_Timestamp_fields[3]; -extern const pb_field_t google_census_Metric_fields[5]; -extern const pb_field_t google_census_Metric_BasicUnit_fields[2]; -extern const pb_field_t google_census_Metric_MeasurementUnit_fields[4]; +extern const pb_field_t google_census_Resource_fields[4]; +extern const pb_field_t google_census_Resource_MeasurementUnit_fields[4]; extern const pb_field_t google_census_AggregationDescriptor_fields[3]; extern const pb_field_t google_census_AggregationDescriptor_BucketBoundaries_fields[2]; extern const pb_field_t google_census_AggregationDescriptor_IntervalBoundaries_fields[2]; @@ -269,12 +256,11 @@ extern const pb_field_t google_census_IntervalStats_Window_fields[4]; extern const pb_field_t google_census_Tag_fields[3]; extern const pb_field_t google_census_View_fields[6]; extern const pb_field_t google_census_Aggregation_fields[6]; -extern const pb_field_t google_census_ViewAggregations_fields[4]; +extern const pb_field_t google_census_Metric_fields[4]; /* Maximum encoded size of messages (where known) */ #define google_census_Duration_size 22 #define google_census_Timestamp_size 22 -#define google_census_Metric_BasicUnit_size 2 #define google_census_Distribution_Range_size 18 #define google_census_IntervalStats_Window_size 44 #define google_census_Tag_size 516 diff --git a/src/proto/census/census.proto b/src/proto/census/census.proto index c869d851ff1..56c291ff17c 100644 --- a/src/proto/census/census.proto +++ b/src/proto/census/census.proto @@ -33,12 +33,12 @@ package google.census; // All the census protos. // -// Nomenclature note: capitalized names below (like Metric) are protos. +// Nomenclature note: capitalized names below (like Resource) are protos. // -// Census lets you define a Metric - something which can be measured, like the +// Census lets you define a Resource - something which can be measured, like the // latency of an RPC, the number of CPU cycles spent on an operation, or // anything else you care to measure. You can record individual instances of -// measurements (a double value) for every metric of interest. These +// measurements (a double value) for every Resource of interest. These // individual measurements are aggregated together into an Aggregation. There // are two Aggregation types available: Distribution (describes the // distribution of all measurements, possibly with a histogram) and @@ -47,8 +47,8 @@ package google.census; // // You can define how your stats are broken down by Tag values and which // Aggregations to use through a View. The corresponding combination of -// Metric/View/Aggregation which is available to census clients is called a -// ViewAggregation. +// Resource/View/Aggregation which is available to census clients is called a +// Metric. // The following two types are copied from @@ -85,26 +85,23 @@ message Timestamp { int32 nanos = 2; } -// Describes a metric -message Metric { - // name of metric, e.g. rpc_latency, cpu. +// Describes a Resource. +message Resource { + // name of resource, e.g. rpc_latency, cpu. Must be unique. string name = 1; - // More detailed description of the metric, used in documentation. + // More detailed description of the resource, used in documentation. string description = 2; // Fundamental units of measurement supported by Census // TODO(aveitch): expand this to include other S.I. units? - message BasicUnit { - enum Measure { - UNKNOWN = 0; - BITS = 1; - BYTES = 2; - SECS = 3; - CORES = 4; - MAX_UNITS = 5; - } - Measure type = 1; + enum BasicUnit { + UNKNOWN = 0; + BITS = 1; + BYTES = 2; + SECS = 3; + CORES = 4; + MAX_UNITS = 5; } // MeasurementUnit lets you build compound units of the form @@ -141,14 +138,11 @@ message Metric { repeated BasicUnit denominator = 3; } - // The units in which the Metric value is reported. + // The units in which Resource values are measured. MeasurementUnit unit = 3; - - // Metrics will be assigned an ID when registered. Invalid if <= 0. - int32 id = 4; } -// An Aggregation summarizes a series of individual Metric measurements, an +// An Aggregation summarizes a series of individual Resource measurements, an // AggregationDescriptor describes an Aggregation. message AggregationDescriptor { // At most one set of options. If neither option is set, a default type @@ -251,7 +245,7 @@ message IntervalStats { double mean = 3; } - // Full set of windows for this metric. + // Full set of windows for this aggregation. repeated Window window = 1; } @@ -264,24 +258,24 @@ message Tag { // A View specifies an Aggregation and a set of tag keys. The Aggregation will // be broken down by the unique set of matching tag values for each measurement. message View { - // Name of view. + // Name of view. Must be unique. string name = 1; // More detailed description, for documentation purposes. string description = 2; - // ID of Metric to associate with this View. - int32 metric_id = 3; + // Name of Resource to be broken down for this view. + string resource_name = 3; // Aggregation type to associate with this View. AggregationDescriptor aggregation = 4; - // Tag keys to match with a given Metric. If no keys are specified, then all - // stats for the Metric are recorded. Keys must be unique. + // Tag keys to match with a given Resource measurement. If no keys are + // specified, then all stats are recorded. Keys must be unique. repeated string tag_key = 5; } -// An Aggregation summarizes a series of individual Metric measures. +// An Aggregation summarizes a series of individual Resource measurements. message Aggregation { // Name of this aggregation. string name = 1; @@ -299,13 +293,13 @@ message Aggregation { repeated Tag tag = 5; } -// A ViewAggregations represents all the Aggregations for a particular view. -message ViewAggregations { +// A Metric represents all the Aggregations for a particular view. +message Metric { // Aggregations - each will have a unique set of tag values for the tag_keys // associated with the corresponding View. repeated Aggregation aggregation = 1; - // Start and end timestamps over which the value was accumulated. These + // Start and end timestamps over which the metric was accumulated. These // values are not relevant/defined for IntervalStats aggregations, which are // always accumulated over a fixed time period. Timestamp start = 2; From 1d2f28913e93481cc4b425c608dc7c59ebe026e4 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Thu, 2 Jun 2016 14:33:22 -0700 Subject: [PATCH 0044/1039] Add timestamps to custom log output --- src/node/ext/node_grpc.cc | 38 ++++++++++++++++++++++++-------------- src/node/index.js | 16 +++++++++++++--- 2 files changed, 37 insertions(+), 17 deletions(-) diff --git a/src/node/ext/node_grpc.cc b/src/node/ext/node_grpc.cc index 45762cad2e6..6daa1039342 100644 --- a/src/node/ext/node_grpc.cc +++ b/src/node/ext/node_grpc.cc @@ -40,6 +40,7 @@ #include "grpc/grpc_security.h" #include "grpc/support/alloc.h" #include "grpc/support/log.h" +#include "grpc/support/time.h" #include "call.h" #include "call_credentials.h" @@ -48,6 +49,7 @@ #include "server.h" #include "completion_queue_async_worker.h" #include "server_credentials.h" +#include "timeval.h" using v8::FunctionTemplate; using v8::Local; @@ -57,9 +59,14 @@ using v8::Object; using v8::Uint32; using v8::String; +typedef struct log_args { + gpr_log_func_args core_args; + gpr_timespec timestamp; +} log_args; + typedef struct logger_state { Nan::Callback *callback; - std::list *pending_args; + std::list *pending_args; uv_mutex_t mutex; uv_async_t async; // Indicates that a logger has been set @@ -327,35 +334,38 @@ NAN_METHOD(SetDefaultRootsPem) { NAUV_WORK_CB(LogMessagesCallback) { Nan::HandleScope scope; - std::list args; + std::list args; uv_mutex_lock(&grpc_logger_state.mutex); args.splice(args.begin(), *grpc_logger_state.pending_args); uv_mutex_unlock(&grpc_logger_state.mutex); /* Call the callback with each log message */ while (!args.empty()) { - gpr_log_func_args *arg = args.front(); + log_args *arg = args.front(); args.pop_front(); - Local file = Nan::New(arg->file).ToLocalChecked(); - Local line = Nan::New(arg->line); + Local file = Nan::New(arg->core_args.file).ToLocalChecked(); + Local line = Nan::New(arg->core_args.line); Local severity = Nan::New( - gpr_log_severity_string(arg->severity)).ToLocalChecked(); - Local message = Nan::New(arg->message).ToLocalChecked(); - const int argc = 4; - Local argv[argc] = {file, line, severity, message}; + gpr_log_severity_string(arg->core_args.severity)).ToLocalChecked(); + Local message = Nan::New(arg->core_args.message).ToLocalChecked(); + Local timestamp = Nan::New( + grpc::node::TimespecToMilliseconds(arg->timestamp)).ToLocalChecked(); + const int argc = 5; + Local argv[argc] = {file, line, severity, message, timestamp}; grpc_logger_state.callback->Call(argc, argv); - delete[] arg->message; + delete[] arg->core_args.message; delete arg; } } void node_log_func(gpr_log_func_args *args) { // TODO(mlumish): Use the core's log formatter when it becomes available - gpr_log_func_args *args_copy = new gpr_log_func_args; + log_args *args_copy = new log_args; size_t message_len = strlen(args->message) + 1; char *message = new char[message_len]; memcpy(message, args->message, message_len); - memcpy(args_copy, args, sizeof(gpr_log_func_args)); - args_copy->message = message; + memcpy(&args_copy->core_args, args, sizeof(gpr_log_func_args)); + args_copy->core_args.message = message; + args_copy->timestamp = gpr_now(GPR_CLOCK_REALTIME); uv_mutex_lock(&grpc_logger_state.mutex); grpc_logger_state.pending_args->push_back(args_copy); @@ -366,7 +376,7 @@ void node_log_func(gpr_log_func_args *args) { void init_logger() { memset(&grpc_logger_state, 0, sizeof(logger_state)); - grpc_logger_state.pending_args = new std::list(); + grpc_logger_state.pending_args = new std::list(); uv_mutex_init(&grpc_logger_state.mutex); uv_async_init(uv_default_loop(), &grpc_logger_state.async, diff --git a/src/node/index.js b/src/node/index.js index b85cec68414..9fb6faa5d7c 100644 --- a/src/node/index.js +++ b/src/node/index.js @@ -124,6 +124,10 @@ exports.load = function load(filename, format, options) { return loadObject(builder.ns, options); }; +var log_template = _.template( + '{severity} {timestamp}\t{file}:{line}]\t{message}', + {interpolate: /{([\s\S]+?)}/g}); + /** * Sets the logger function for the gRPC module. For debugging purposes, the C * core will log synchronously directly to stdout unless this function is @@ -134,9 +138,15 @@ exports.load = function load(filename, format, options) { */ exports.setLogger = function setLogger(logger) { common.logger = logger; - grpc.setDefaultLoggerCallback(function(file, line, severity, message) { - file = path.basename(file); - logger.error(severity + '\t' + file + ':' + line + ']\t' + message); + grpc.setDefaultLoggerCallback(function(file, line, severity, + message, timestamp) { + logger.error(log_template({ + file: path.basename(file), + line: line, + severity: severity, + message: message, + timestamp: timestamp.toISOString() + })); }); }; From cc0f4e1c807090c97aa64f83649013e0e6ab879b Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Thu, 2 Jun 2016 16:00:54 -0700 Subject: [PATCH 0045/1039] Remove Ruby copy initializers. They don't make sense with wrapped structs, and we don't use them anyway --- src/ruby/ext/grpc/rb_call_credentials.c | 31 +-------------------- src/ruby/ext/grpc/rb_channel.c | 30 +------------------- src/ruby/ext/grpc/rb_channel_credentials.c | 32 +--------------------- src/ruby/ext/grpc/rb_grpc.c | 2 +- src/ruby/ext/grpc/rb_server.c | 31 +-------------------- src/ruby/ext/grpc/rb_server_credentials.c | 32 +--------------------- 6 files changed, 6 insertions(+), 152 deletions(-) diff --git a/src/ruby/ext/grpc/rb_call_credentials.c b/src/ruby/ext/grpc/rb_call_credentials.c index 79ca5b32ced..9b6675da846 100644 --- a/src/ruby/ext/grpc/rb_call_credentials.c +++ b/src/ruby/ext/grpc/rb_call_credentials.c @@ -211,35 +211,6 @@ VALUE grpc_rb_wrap_call_credentials(grpc_call_credentials *c, VALUE mark) { return rb_wrapper; } -/* Clones CallCredentials instances. - Gives CallCredentials a consistent implementation of Ruby's object copy/dup - protocol. */ -static VALUE grpc_rb_call_credentials_init_copy(VALUE copy, VALUE orig) { - grpc_rb_call_credentials *orig_cred = NULL; - grpc_rb_call_credentials *copy_cred = NULL; - - if (copy == orig) { - return copy; - } - - /* Raise an error if orig is not a credentials object or a subclass. */ - if (TYPE(orig) != T_DATA || - RDATA(orig)->dfree != (RUBY_DATA_FUNC)grpc_rb_call_credentials_free) { - rb_raise(rb_eTypeError, "not a %s", - rb_obj_classname(grpc_rb_cCallCredentials)); - } - - TypedData_Get_Struct(orig, grpc_rb_call_credentials, - &grpc_rb_call_credentials_data_type, orig_cred); - TypedData_Get_Struct(copy, grpc_rb_call_credentials, - &grpc_rb_call_credentials_data_type, copy_cred); - - /* use ruby's MEMCPY to make a byte-for-byte copy of the credentials - * wrapper object. */ - MEMCPY(copy_cred, orig_cred, grpc_rb_call_credentials, 1); - return copy; -} - /* The attribute used on the mark object to hold the callback */ static ID id_callback; @@ -308,7 +279,7 @@ void Init_grpc_call_credentials() { rb_define_method(grpc_rb_cCallCredentials, "initialize", grpc_rb_call_credentials_init, 1); rb_define_method(grpc_rb_cCallCredentials, "initialize_copy", - grpc_rb_call_credentials_init_copy, 1); + grpc_rb_cannot_init_copy, 1); rb_define_method(grpc_rb_cCallCredentials, "compose", grpc_rb_call_credentials_compose, -1); diff --git a/src/ruby/ext/grpc/rb_channel.c b/src/ruby/ext/grpc/rb_channel.c index 013321ffc8a..9d29b96a35b 100644 --- a/src/ruby/ext/grpc/rb_channel.c +++ b/src/ruby/ext/grpc/rb_channel.c @@ -231,34 +231,6 @@ static VALUE grpc_rb_channel_watch_connectivity_state(VALUE self, return Qnil; } -/* Clones Channel instances. - - Gives Channel a consistent implementation of Ruby's object copy/dup - protocol. */ -static VALUE grpc_rb_channel_init_copy(VALUE copy, VALUE orig) { - grpc_rb_channel *orig_ch = NULL; - grpc_rb_channel *copy_ch = NULL; - - if (copy == orig) { - return copy; - } - - /* Raise an error if orig is not a channel object or a subclass. */ - if (TYPE(orig) != T_DATA || - RDATA(orig)->dfree != (RUBY_DATA_FUNC)grpc_rb_channel_free) { - rb_raise(rb_eTypeError, "not a %s", rb_obj_classname(grpc_rb_cChannel)); - return Qnil; - } - - TypedData_Get_Struct(orig, grpc_rb_channel, &grpc_channel_data_type, orig_ch); - TypedData_Get_Struct(copy, grpc_rb_channel, &grpc_channel_data_type, copy_ch); - - /* use ruby's MEMCPY to make a byte-for-byte copy of the channel wrapper - * object. */ - MEMCPY(copy_ch, orig_ch, grpc_rb_channel, 1); - return copy; -} - /* Create a call given a grpc_channel, in order to call method. The request is not sent until grpc_call_invoke is called. */ static VALUE grpc_rb_channel_create_call(VALUE self, VALUE cqueue, @@ -387,7 +359,7 @@ void Init_grpc_channel() { /* Provides a ruby constructor and support for dup/clone. */ rb_define_method(grpc_rb_cChannel, "initialize", grpc_rb_channel_init, -1); rb_define_method(grpc_rb_cChannel, "initialize_copy", - grpc_rb_channel_init_copy, 1); + grpc_rb_cannot_init_copy, 1); /* Add ruby analogues of the Channel methods. */ rb_define_method(grpc_rb_cChannel, "connectivity_state", diff --git a/src/ruby/ext/grpc/rb_channel_credentials.c b/src/ruby/ext/grpc/rb_channel_credentials.c index cbb23885aa6..5b7aa3417e6 100644 --- a/src/ruby/ext/grpc/rb_channel_credentials.c +++ b/src/ruby/ext/grpc/rb_channel_credentials.c @@ -126,36 +126,6 @@ VALUE grpc_rb_wrap_channel_credentials(grpc_channel_credentials *c, VALUE mark) return rb_wrapper; } -/* Clones ChannelCredentials instances. - Gives ChannelCredentials a consistent implementation of Ruby's object copy/dup - protocol. */ -static VALUE grpc_rb_channel_credentials_init_copy(VALUE copy, VALUE orig) { - grpc_rb_channel_credentials *orig_cred = NULL; - grpc_rb_channel_credentials *copy_cred = NULL; - - if (copy == orig) { - return copy; - } - - /* Raise an error if orig is not a credentials object or a subclass. */ - if (TYPE(orig) != T_DATA || - RDATA(orig)->dfree != (RUBY_DATA_FUNC)grpc_rb_channel_credentials_free) { - rb_raise(rb_eTypeError, "not a %s", - rb_obj_classname(grpc_rb_cChannelCredentials)); - } - - TypedData_Get_Struct(orig, grpc_rb_channel_credentials, - &grpc_rb_channel_credentials_data_type, orig_cred); - TypedData_Get_Struct(copy, grpc_rb_channel_credentials, - &grpc_rb_channel_credentials_data_type, copy_cred); - - /* use ruby's MEMCPY to make a byte-for-byte copy of the credentials - * wrapper object. */ - MEMCPY(copy_cred, orig_cred, grpc_rb_channel_credentials, 1); - return copy; -} - - /* The attribute used on the mark object to hold the pem_root_certs. */ static ID id_pem_root_certs; @@ -271,7 +241,7 @@ void Init_grpc_channel_credentials() { rb_define_method(grpc_rb_cChannelCredentials, "initialize", grpc_rb_channel_credentials_init, -1); rb_define_method(grpc_rb_cChannelCredentials, "initialize_copy", - grpc_rb_channel_credentials_init_copy, 1); + grpc_rb_cannot_init_copy, 1); rb_define_method(grpc_rb_cChannelCredentials, "compose", grpc_rb_channel_credentials_compose, -1); rb_define_module_function(grpc_rb_cChannelCredentials, diff --git a/src/ruby/ext/grpc/rb_grpc.c b/src/ruby/ext/grpc/rb_grpc.c index 9246893f9fb..8527da52d27 100644 --- a/src/ruby/ext/grpc/rb_grpc.c +++ b/src/ruby/ext/grpc/rb_grpc.c @@ -85,7 +85,7 @@ VALUE grpc_rb_cannot_init(VALUE self) { VALUE grpc_rb_cannot_init_copy(VALUE copy, VALUE self) { (void)self; rb_raise(rb_eTypeError, - "initialization of %s only allowed from the gRPC native layer", + "Copy initialization of %s is not supported", rb_obj_classname(copy)); return Qnil; } diff --git a/src/ruby/ext/grpc/rb_server.c b/src/ruby/ext/grpc/rb_server.c index f108b8acfcd..c92059d12cc 100644 --- a/src/ruby/ext/grpc/rb_server.c +++ b/src/ruby/ext/grpc/rb_server.c @@ -154,35 +154,6 @@ static VALUE grpc_rb_server_init(VALUE self, VALUE cqueue, VALUE channel_args) { return self; } -/* Clones Server instances. - - Gives Server a consistent implementation of Ruby's object copy/dup - protocol. */ -static VALUE grpc_rb_server_init_copy(VALUE copy, VALUE orig) { - grpc_rb_server *orig_srv = NULL; - grpc_rb_server *copy_srv = NULL; - - if (copy == orig) { - return copy; - } - - /* Raise an error if orig is not a server object or a subclass. */ - if (TYPE(orig) != T_DATA || - RDATA(orig)->dfree != (RUBY_DATA_FUNC)grpc_rb_server_free) { - rb_raise(rb_eTypeError, "not a %s", rb_obj_classname(grpc_rb_cServer)); - } - - TypedData_Get_Struct(orig, grpc_rb_server, &grpc_rb_server_data_type, - orig_srv); - TypedData_Get_Struct(copy, grpc_rb_server, &grpc_rb_server_data_type, - copy_srv); - - /* use ruby's MEMCPY to make a byte-for-byte copy of the server wrapper - object. */ - MEMCPY(copy_srv, orig_srv, grpc_rb_server, 1); - return copy; -} - /* request_call_stack holds various values used by the * grpc_rb_server_request_call function */ typedef struct request_call_stack { @@ -378,7 +349,7 @@ void Init_grpc_server() { /* Provides a ruby constructor and support for dup/clone. */ rb_define_method(grpc_rb_cServer, "initialize", grpc_rb_server_init, 2); rb_define_method(grpc_rb_cServer, "initialize_copy", - grpc_rb_server_init_copy, 1); + grpc_rb_cannot_init_copy, 1); /* Add the server methods. */ rb_define_method(grpc_rb_cServer, "request_call", diff --git a/src/ruby/ext/grpc/rb_server_credentials.c b/src/ruby/ext/grpc/rb_server_credentials.c index 3b0fb6c910d..58ec852505c 100644 --- a/src/ruby/ext/grpc/rb_server_credentials.c +++ b/src/ruby/ext/grpc/rb_server_credentials.c @@ -111,36 +111,6 @@ static VALUE grpc_rb_server_credentials_alloc(VALUE cls) { wrapper); } -/* Clones ServerCredentials instances. - - Gives ServerCredentials a consistent implementation of Ruby's object copy/dup - protocol. */ -static VALUE grpc_rb_server_credentials_init_copy(VALUE copy, VALUE orig) { - grpc_rb_server_credentials *orig_ch = NULL; - grpc_rb_server_credentials *copy_ch = NULL; - - if (copy == orig) { - return copy; - } - - /* Raise an error if orig is not a server_credentials object or a subclass. */ - if (TYPE(orig) != T_DATA || - RDATA(orig)->dfree != (RUBY_DATA_FUNC)grpc_rb_server_credentials_free) { - rb_raise(rb_eTypeError, "not a %s", - rb_obj_classname(grpc_rb_cServerCredentials)); - } - - TypedData_Get_Struct(orig, grpc_rb_server_credentials, - &grpc_rb_server_credentials_data_type, orig_ch); - TypedData_Get_Struct(copy, grpc_rb_server_credentials, - &grpc_rb_server_credentials_data_type, copy_ch); - - /* use ruby's MEMCPY to make a byte-for-byte copy of the server_credentials - wrapper object. */ - MEMCPY(copy_ch, orig_ch, grpc_rb_server_credentials, 1); - return copy; -} - /* The attribute used on the mark object to preserve the pem_root_certs. */ static ID id_pem_root_certs; @@ -270,7 +240,7 @@ void Init_grpc_server_credentials() { rb_define_method(grpc_rb_cServerCredentials, "initialize", grpc_rb_server_credentials_init, 3); rb_define_method(grpc_rb_cServerCredentials, "initialize_copy", - grpc_rb_server_credentials_init_copy, 1); + grpc_rb_cannot_init_copy, 1); id_pem_key_certs = rb_intern("__pem_key_certs"); id_pem_root_certs = rb_intern("__pem_root_certs"); From 4aaba75a822c6dde718d7fe646f7671b4f64d8a0 Mon Sep 17 00:00:00 2001 From: Alistair Veitch Date: Thu, 2 Jun 2016 17:11:46 -0700 Subject: [PATCH 0046/1039] initial implementation of resource handling --- BUILD | 12 + Makefile | 40 +++ binding.gyp | 2 + build.yaml | 14 + config.m4 | 2 + gRPC.podspec | 6 + grpc.def | 11 +- grpc.gemspec | 4 + include/grpc/census.h | 136 +++------- package.xml | 4 + src/core/ext/census/base_resources.c | 148 ++++++++++ src/core/ext/census/base_resources.h | 39 +++ src/core/ext/census/initialize.c | 20 +- src/core/ext/census/placeholders.c | 45 ---- src/core/ext/census/resource.c | 255 ++++++++++++++++++ src/core/ext/census/resource.h | 42 +++ .../grpcio/grpc/_cython/imports.generated.c | 22 +- .../grpcio/grpc/_cython/imports.generated.h | 33 +-- src/python/grpcio/grpc_core_dependencies.py | 2 + src/ruby/ext/grpc/rb_grpc_imports.generated.c | 22 +- src/ruby/ext/grpc/rb_grpc_imports.generated.h | 33 +-- test/core/census/README | 7 + test/core/census/data/resource_empty_name.pb | 1 + test/core/census/data/resource_empty_name.txt | 5 + test/core/census/data/resource_full.pb | 2 + test/core/census/data/resource_full.txt | 9 + .../core/census/data/resource_minimal_good.pb | 2 + .../census/data/resource_minimal_good.txt | 5 + test/core/census/data/resource_no_name.pb | 1 + test/core/census/data/resource_no_name.txt | 4 + .../core/census/data/resource_no_numerator.pb | 2 + .../census/data/resource_no_numerator.txt | 6 + test/core/census/data/resource_no_unit.pb | 2 + test/core/census/data/resource_no_unit.txt | 2 + test/core/census/resource_test.c | 157 +++++++++++ tools/doxygen/Doxyfile.core.internal | 4 + tools/run_tests/sources_and_headers.json | 22 ++ tools/run_tests/tests.json | 21 ++ vsprojects/buildtests_c.sln | 27 ++ vsprojects/vcxproj/grpc/grpc.vcxproj | 6 + vsprojects/vcxproj/grpc/grpc.vcxproj.filters | 12 + .../grpc_unsecure/grpc_unsecure.vcxproj | 6 + .../grpc_unsecure.vcxproj.filters | 12 + .../census_resource_test.vcxproj | 199 ++++++++++++++ .../census_resource_test.vcxproj.filters | 21 ++ 45 files changed, 1194 insertions(+), 233 deletions(-) create mode 100644 src/core/ext/census/base_resources.c create mode 100644 src/core/ext/census/base_resources.h create mode 100644 src/core/ext/census/resource.c create mode 100644 src/core/ext/census/resource.h create mode 100644 test/core/census/README create mode 100644 test/core/census/data/resource_empty_name.pb create mode 100644 test/core/census/data/resource_empty_name.txt create mode 100644 test/core/census/data/resource_full.pb create mode 100644 test/core/census/data/resource_full.txt create mode 100644 test/core/census/data/resource_minimal_good.pb create mode 100644 test/core/census/data/resource_minimal_good.txt create mode 100644 test/core/census/data/resource_no_name.pb create mode 100644 test/core/census/data/resource_no_name.txt create mode 100644 test/core/census/data/resource_no_numerator.pb create mode 100644 test/core/census/data/resource_no_numerator.txt create mode 100644 test/core/census/data/resource_no_unit.pb create mode 100644 test/core/census/data/resource_no_unit.txt create mode 100644 test/core/census/resource_test.c create mode 100644 vsprojects/vcxproj/test/census_resource_test/census_resource_test.vcxproj create mode 100644 vsprojects/vcxproj/test/census_resource_test/census_resource_test.vcxproj.filters diff --git a/BUILD b/BUILD index 024a5182ce5..b3b75ed0bb8 100644 --- a/BUILD +++ b/BUILD @@ -299,11 +299,13 @@ cc_library( "src/core/ext/lb_policy/grpclb/load_balancer_api.h", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h", "src/core/ext/census/aggregation.h", + "src/core/ext/census/base_resources.h", "src/core/ext/census/census_interface.h", "src/core/ext/census/census_rpc_stats.h", "src/core/ext/census/gen/census.pb.h", "src/core/ext/census/grpc_filter.h", "src/core/ext/census/mlog.h", + "src/core/ext/census/resource.h", "src/core/ext/census/rpc_metric_id.h", "src/core/lib/surface/init.c", "src/core/lib/channel/channel_args.c", @@ -469,6 +471,7 @@ cc_library( "src/core/ext/lb_policy/round_robin/round_robin.c", "src/core/ext/resolver/dns/native/dns_resolver.c", "src/core/ext/resolver/sockaddr/sockaddr_resolver.c", + "src/core/ext/census/base_resources.c", "src/core/ext/census/context.c", "src/core/ext/census/gen/census.pb.c", "src/core/ext/census/grpc_context.c", @@ -478,6 +481,7 @@ cc_library( "src/core/ext/census/mlog.c", "src/core/ext/census/operation.c", "src/core/ext/census/placeholders.c", + "src/core/ext/census/resource.c", "src/core/ext/census/tracing.c", "src/core/plugin_registry/grpc_plugin_registry.c", ], @@ -647,11 +651,13 @@ cc_library( "src/core/ext/lb_policy/grpclb/load_balancer_api.h", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h", "src/core/ext/census/aggregation.h", + "src/core/ext/census/base_resources.h", "src/core/ext/census/census_interface.h", "src/core/ext/census/census_rpc_stats.h", "src/core/ext/census/gen/census.pb.h", "src/core/ext/census/grpc_filter.h", "src/core/ext/census/mlog.h", + "src/core/ext/census/resource.h", "src/core/ext/census/rpc_metric_id.h", "src/core/lib/surface/init.c", "src/core/lib/surface/init_unsecure.c", @@ -786,6 +792,7 @@ cc_library( "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c", "src/core/ext/lb_policy/pick_first/pick_first.c", "src/core/ext/lb_policy/round_robin/round_robin.c", + "src/core/ext/census/base_resources.c", "src/core/ext/census/context.c", "src/core/ext/census/gen/census.pb.c", "src/core/ext/census/grpc_context.c", @@ -795,6 +802,7 @@ cc_library( "src/core/ext/census/mlog.c", "src/core/ext/census/operation.c", "src/core/ext/census/placeholders.c", + "src/core/ext/census/resource.c", "src/core/ext/census/tracing.c", "src/core/plugin_registry/grpc_unsecure_plugin_registry.c", ], @@ -1512,6 +1520,7 @@ objc_library( "src/core/ext/lb_policy/round_robin/round_robin.c", "src/core/ext/resolver/dns/native/dns_resolver.c", "src/core/ext/resolver/sockaddr/sockaddr_resolver.c", + "src/core/ext/census/base_resources.c", "src/core/ext/census/context.c", "src/core/ext/census/gen/census.pb.c", "src/core/ext/census/grpc_context.c", @@ -1521,6 +1530,7 @@ objc_library( "src/core/ext/census/mlog.c", "src/core/ext/census/operation.c", "src/core/ext/census/placeholders.c", + "src/core/ext/census/resource.c", "src/core/ext/census/tracing.c", "src/core/plugin_registry/grpc_plugin_registry.c", ], @@ -1693,11 +1703,13 @@ objc_library( "src/core/ext/lb_policy/grpclb/load_balancer_api.h", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h", "src/core/ext/census/aggregation.h", + "src/core/ext/census/base_resources.h", "src/core/ext/census/census_interface.h", "src/core/ext/census/census_rpc_stats.h", "src/core/ext/census/gen/census.pb.h", "src/core/ext/census/grpc_filter.h", "src/core/ext/census/mlog.h", + "src/core/ext/census/resource.h", "src/core/ext/census/rpc_metric_id.h", ], includes = [ diff --git a/Makefile b/Makefile index 98f6e562c69..34daedc6d27 100644 --- a/Makefile +++ b/Makefile @@ -891,6 +891,7 @@ alpn_test: $(BINDIR)/$(CONFIG)/alpn_test api_fuzzer: $(BINDIR)/$(CONFIG)/api_fuzzer bin_encoder_test: $(BINDIR)/$(CONFIG)/bin_encoder_test census_context_test: $(BINDIR)/$(CONFIG)/census_context_test +census_resource_test: $(BINDIR)/$(CONFIG)/census_resource_test channel_create_test: $(BINDIR)/$(CONFIG)/channel_create_test chttp2_hpack_encoder_test: $(BINDIR)/$(CONFIG)/chttp2_hpack_encoder_test chttp2_status_conversion_test: $(BINDIR)/$(CONFIG)/chttp2_status_conversion_test @@ -1223,6 +1224,7 @@ buildtests_c: privatelibs_c \ $(BINDIR)/$(CONFIG)/alpn_test \ $(BINDIR)/$(CONFIG)/bin_encoder_test \ $(BINDIR)/$(CONFIG)/census_context_test \ + $(BINDIR)/$(CONFIG)/census_resource_test \ $(BINDIR)/$(CONFIG)/channel_create_test \ $(BINDIR)/$(CONFIG)/chttp2_hpack_encoder_test \ $(BINDIR)/$(CONFIG)/chttp2_status_conversion_test \ @@ -1475,6 +1477,8 @@ test_c: buildtests_c $(Q) $(BINDIR)/$(CONFIG)/bin_encoder_test || ( echo test bin_encoder_test failed ; exit 1 ) $(E) "[RUN] Testing census_context_test" $(Q) $(BINDIR)/$(CONFIG)/census_context_test || ( echo test census_context_test failed ; exit 1 ) + $(E) "[RUN] Testing census_resource_test" + $(Q) $(BINDIR)/$(CONFIG)/census_resource_test || ( echo test census_resource_test failed ; exit 1 ) $(E) "[RUN] Testing channel_create_test" $(Q) $(BINDIR)/$(CONFIG)/channel_create_test || ( echo test channel_create_test failed ; exit 1 ) $(E) "[RUN] Testing chttp2_hpack_encoder_test" @@ -2637,6 +2641,7 @@ LIBGRPC_SRC = \ src/core/ext/lb_policy/round_robin/round_robin.c \ src/core/ext/resolver/dns/native/dns_resolver.c \ src/core/ext/resolver/sockaddr/sockaddr_resolver.c \ + src/core/ext/census/base_resources.c \ src/core/ext/census/context.c \ src/core/ext/census/gen/census.pb.c \ src/core/ext/census/grpc_context.c \ @@ -2646,6 +2651,7 @@ LIBGRPC_SRC = \ src/core/ext/census/mlog.c \ src/core/ext/census/operation.c \ src/core/ext/census/placeholders.c \ + src/core/ext/census/resource.c \ src/core/ext/census/tracing.c \ src/core/plugin_registry/grpc_plugin_registry.c \ @@ -2961,6 +2967,7 @@ LIBGRPC_UNSECURE_SRC = \ third_party/nanopb/pb_encode.c \ src/core/ext/lb_policy/pick_first/pick_first.c \ src/core/ext/lb_policy/round_robin/round_robin.c \ + src/core/ext/census/base_resources.c \ src/core/ext/census/context.c \ src/core/ext/census/gen/census.pb.c \ src/core/ext/census/grpc_context.c \ @@ -2970,6 +2977,7 @@ LIBGRPC_UNSECURE_SRC = \ src/core/ext/census/mlog.c \ src/core/ext/census/operation.c \ src/core/ext/census/placeholders.c \ + src/core/ext/census/resource.c \ src/core/ext/census/tracing.c \ src/core/plugin_registry/grpc_unsecure_plugin_registry.c \ @@ -6195,6 +6203,38 @@ endif endif +CENSUS_RESOURCE_TEST_SRC = \ + test/core/census/resource_test.c \ + +CENSUS_RESOURCE_TEST_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(CENSUS_RESOURCE_TEST_SRC)))) +ifeq ($(NO_SECURE),true) + +# You can't build secure targets if you don't have OpenSSL. + +$(BINDIR)/$(CONFIG)/census_resource_test: openssl_dep_error + +else + + + +$(BINDIR)/$(CONFIG)/census_resource_test: $(CENSUS_RESOURCE_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + $(E) "[LD] Linking $@" + $(Q) mkdir -p `dirname $@` + $(Q) $(LD) $(LDFLAGS) $(CENSUS_RESOURCE_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/census_resource_test + +endif + +$(OBJDIR)/$(CONFIG)/test/core/census/resource_test.o: $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + +deps_census_resource_test: $(CENSUS_RESOURCE_TEST_OBJS:.o=.dep) + +ifneq ($(NO_SECURE),true) +ifneq ($(NO_DEPS),true) +-include $(CENSUS_RESOURCE_TEST_OBJS:.o=.dep) +endif +endif + + CHANNEL_CREATE_TEST_SRC = \ test/core/surface/channel_create_test.c \ diff --git a/binding.gyp b/binding.gyp index 1e8b5e294f8..cc8077d5662 100644 --- a/binding.gyp +++ b/binding.gyp @@ -730,6 +730,7 @@ 'src/core/ext/lb_policy/round_robin/round_robin.c', 'src/core/ext/resolver/dns/native/dns_resolver.c', 'src/core/ext/resolver/sockaddr/sockaddr_resolver.c', + 'src/core/ext/census/base_resources.c', 'src/core/ext/census/context.c', 'src/core/ext/census/gen/census.pb.c', 'src/core/ext/census/grpc_context.c', @@ -739,6 +740,7 @@ 'src/core/ext/census/mlog.c', 'src/core/ext/census/operation.c', 'src/core/ext/census/placeholders.c', + 'src/core/ext/census/resource.c', 'src/core/ext/census/tracing.c', 'src/core/plugin_registry/grpc_plugin_registry.c', ], diff --git a/build.yaml b/build.yaml index f211575e925..0d192eee1d8 100644 --- a/build.yaml +++ b/build.yaml @@ -14,13 +14,16 @@ filegroups: - include/grpc/census.h headers: - src/core/ext/census/aggregation.h + - src/core/ext/census/base_resources.h - src/core/ext/census/census_interface.h - src/core/ext/census/census_rpc_stats.h - src/core/ext/census/gen/census.pb.h - src/core/ext/census/grpc_filter.h - src/core/ext/census/mlog.h + - src/core/ext/census/resource.h - src/core/ext/census/rpc_metric_id.h src: + - src/core/ext/census/base_resources.c - src/core/ext/census/context.c - src/core/ext/census/gen/census.pb.c - src/core/ext/census/grpc_context.c @@ -30,6 +33,7 @@ filegroups: - src/core/ext/census/mlog.c - src/core/ext/census/operation.c - src/core/ext/census/placeholders.c + - src/core/ext/census/resource.c - src/core/ext/census/tracing.c plugin: census_grpc_plugin uses: @@ -1192,6 +1196,16 @@ targets: - grpc - gpr_test_util - gpr +- name: census_resource_test + build: test + language: c + src: + - test/core/census/resource_test.c + deps: + - grpc_test_util + - grpc + - gpr_test_util + - gpr - name: channel_create_test build: test language: c diff --git a/config.m4 b/config.m4 index e8506bb269c..b8a84291745 100644 --- a/config.m4 +++ b/config.m4 @@ -249,6 +249,7 @@ if test "$PHP_GRPC" != "no"; then src/core/ext/lb_policy/round_robin/round_robin.c \ src/core/ext/resolver/dns/native/dns_resolver.c \ src/core/ext/resolver/sockaddr/sockaddr_resolver.c \ + src/core/ext/census/base_resources.c \ src/core/ext/census/context.c \ src/core/ext/census/gen/census.pb.c \ src/core/ext/census/grpc_context.c \ @@ -258,6 +259,7 @@ if test "$PHP_GRPC" != "no"; then src/core/ext/census/mlog.c \ src/core/ext/census/operation.c \ src/core/ext/census/placeholders.c \ + src/core/ext/census/resource.c \ src/core/ext/census/tracing.c \ src/core/plugin_registry/grpc_plugin_registry.c \ src/boringssl/err_data.c \ diff --git a/gRPC.podspec b/gRPC.podspec index 9c049f03b19..8187b40a29d 100644 --- a/gRPC.podspec +++ b/gRPC.podspec @@ -306,11 +306,13 @@ Pod::Spec.new do |s| 'third_party/nanopb/pb_decode.h', 'third_party/nanopb/pb_encode.h', 'src/core/ext/census/aggregation.h', + 'src/core/ext/census/base_resources.h', 'src/core/ext/census/census_interface.h', 'src/core/ext/census/census_rpc_stats.h', 'src/core/ext/census/gen/census.pb.h', 'src/core/ext/census/grpc_filter.h', 'src/core/ext/census/mlog.h', + 'src/core/ext/census/resource.h', 'src/core/ext/census/rpc_metric_id.h', 'include/grpc/byte_buffer.h', 'include/grpc/byte_buffer_reader.h', @@ -509,6 +511,7 @@ Pod::Spec.new do |s| 'src/core/ext/lb_policy/round_robin/round_robin.c', 'src/core/ext/resolver/dns/native/dns_resolver.c', 'src/core/ext/resolver/sockaddr/sockaddr_resolver.c', + 'src/core/ext/census/base_resources.c', 'src/core/ext/census/context.c', 'src/core/ext/census/gen/census.pb.c', 'src/core/ext/census/grpc_context.c', @@ -518,6 +521,7 @@ Pod::Spec.new do |s| 'src/core/ext/census/mlog.c', 'src/core/ext/census/operation.c', 'src/core/ext/census/placeholders.c', + 'src/core/ext/census/resource.c', 'src/core/ext/census/tracing.c', 'src/core/plugin_registry/grpc_plugin_registry.c' @@ -675,11 +679,13 @@ Pod::Spec.new do |s| 'third_party/nanopb/pb_decode.h', 'third_party/nanopb/pb_encode.h', 'src/core/ext/census/aggregation.h', + 'src/core/ext/census/base_resources.h', 'src/core/ext/census/census_interface.h', 'src/core/ext/census/census_rpc_stats.h', 'src/core/ext/census/gen/census.pb.h', 'src/core/ext/census/grpc_filter.h', 'src/core/ext/census/mlog.h', + 'src/core/ext/census/resource.h', 'src/core/ext/census/rpc_metric_id.h' ss.header_mappings_dir = '.' diff --git a/grpc.def b/grpc.def index b811f0ff129..f69370ae8ab 100644 --- a/grpc.def +++ b/grpc.def @@ -23,15 +23,10 @@ EXPORTS census_trace_scan_start census_get_trace_record census_trace_scan_end + census_define_resource + census_delete_resource + census_resource_id census_record_values - census_view_create - census_view_delete - census_view_metric - census_view_naggregations - census_view_tags - census_view_aggregrations - census_view_get_data - census_view_reset grpc_compression_algorithm_parse grpc_compression_algorithm_name grpc_compression_algorithm_for_level diff --git a/grpc.gemspec b/grpc.gemspec index 047845cd843..acb23bd043d 100755 --- a/grpc.gemspec +++ b/grpc.gemspec @@ -315,11 +315,13 @@ Gem::Specification.new do |s| s.files += %w( third_party/nanopb/pb_decode.h ) s.files += %w( third_party/nanopb/pb_encode.h ) s.files += %w( src/core/ext/census/aggregation.h ) + s.files += %w( src/core/ext/census/base_resources.h ) s.files += %w( src/core/ext/census/census_interface.h ) s.files += %w( src/core/ext/census/census_rpc_stats.h ) s.files += %w( src/core/ext/census/gen/census.pb.h ) s.files += %w( src/core/ext/census/grpc_filter.h ) s.files += %w( src/core/ext/census/mlog.h ) + s.files += %w( src/core/ext/census/resource.h ) s.files += %w( src/core/ext/census/rpc_metric_id.h ) s.files += %w( src/core/lib/surface/init.c ) s.files += %w( src/core/lib/channel/channel_args.c ) @@ -488,6 +490,7 @@ Gem::Specification.new do |s| s.files += %w( src/core/ext/lb_policy/round_robin/round_robin.c ) s.files += %w( src/core/ext/resolver/dns/native/dns_resolver.c ) s.files += %w( src/core/ext/resolver/sockaddr/sockaddr_resolver.c ) + s.files += %w( src/core/ext/census/base_resources.c ) s.files += %w( src/core/ext/census/context.c ) s.files += %w( src/core/ext/census/gen/census.pb.c ) s.files += %w( src/core/ext/census/grpc_context.c ) @@ -497,6 +500,7 @@ Gem::Specification.new do |s| s.files += %w( src/core/ext/census/mlog.c ) s.files += %w( src/core/ext/census/operation.c ) s.files += %w( src/core/ext/census/placeholders.c ) + s.files += %w( src/core/ext/census/resource.c ) s.files += %w( src/core/ext/census/tracing.c ) s.files += %w( src/core/plugin_registry/grpc_plugin_registry.c ) s.files += %w( third_party/boringssl/crypto/aes/internal.h ) diff --git a/include/grpc/census.h b/include/grpc/census.h index 39d87ba119c..62ff45d8941 100644 --- a/include/grpc/census.h +++ b/include/grpc/census.h @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -56,9 +56,11 @@ enum census_features { /** Shutdown and startup census subsystem. The 'features' argument should be * the OR (|) of census_features values. If census fails to initialize, then - * census_initialize() will return a non-zero value. It is an error to call - * census_initialize() more than once (without an intervening - * census_shutdown()). */ + * census_initialize() will return -1, otherwise the set of enabled features + * (which may be smaller than that provided in the `features` argument, see + * census_supported()) is returned. It is an error to call census_initialize() + * more than once (without an intervening census_shutdown()). These functions + * are not thread-safe. */ CENSUSAPI int census_initialize(int features); CENSUSAPI void census_shutdown(void); @@ -430,30 +432,44 @@ CENSUSAPI int census_get_trace_record(census_trace_record *trace_record); CENSUSAPI void census_trace_scan_end(); /* Core stats collection API's. The following concepts are used: - * Aggregation: A collection of values. Census supports the following - aggregation types: - Sum - a single summation type. Typically used for keeping (e.g.) - counts of events. - Distribution - statistical distribution information, used for - recording average, standard deviation etc. - Histogram - a histogram of measurements falling in defined bucket - boundaries. - Window - a count of events that happen in reolling time window. - New aggregation types can be added by the user, if desired (see - census_register_aggregation()). - * Metric: Each measurement is for a single metric. Examples include RPC - latency, CPU seconds consumed, and bytes transmitted. - * View: A view is a combination of a metric, a tag set (in which the tag - values are regular expressions) and a set of aggregations. When a - measurement for a metric matches the view tags, it is recorded (for each - unique set of tags) against each aggregation. Each metric can have an - arbitrary number of views by which it will be broken down. + * Resource: Users record measurements for a single resource. Examples + include RPC latency, CPU seconds consumed, and bytes transmitted. + * Aggregation: An aggregation of a set of measurements. Census supports the + following aggregation types: + * Distribution - statistical distribution information, used for + recording average, standard deviation etc. Can include a histogram. + * Interval - a count of events that happen in a rolling time window. + * View: A view is a combination of a Resource, a set of tag keys and an + Aggregation. When a measurement for a Resource matches the View tags, it is + recorded (for each unique set of tag values) using the Aggregation type. + Each resource can have an arbitrary number of views by which it will be + broken down. + + Census uses protos to define each of the above, and output results. This + ensures unification across the different language and runtime + implementations. The proto definitions can be found in src/proto/census. */ +/* Define a new resource. `resource_pb` should contain an encoded Resource + protobuf, `resource_pb_size` being the size of the buffer. Returns a -ve + value on error, or a positive (>= 0) resource id (for use in + census_delete_resource() and census_record_values()). In order to be valid, a + resource must have a name, and at least one numerator in it's unit type. The + resource name must be unique, and an error will be returned if it is not. */ +CENSUSAPI int32_t census_define_resource(const uint8_t *resource_pb, + size_t resource_pb_size); + +/* Delete a resource created by census_define_resource(). */ +CENSUSAPI void census_delete_resource(int32_t resource_id); + +/* Determine the id of a resource, given it's name. returns -1 if the resource + does not exist. */ +CENSUSAPI int32_t census_resource_id(const char *name); + /* A single value to be recorded comprises two parts: an ID for the particular - * metric and the value to be recorded against it. */ + * resource and the value to be recorded against it. */ typedef struct { - uint32_t metric_id; + int32_t resource_id; double value; } census_value; @@ -461,78 +477,6 @@ typedef struct { CENSUSAPI void census_record_values(census_context *context, census_value *values, size_t nvalues); -/** Type representing a particular aggregation */ -typedef struct census_aggregation_ops census_aggregation_ops; - -/* Predefined aggregation types, for use with census_view_create(). */ -extern census_aggregation_ops census_agg_sum; -extern census_aggregation_ops census_agg_distribution; -extern census_aggregation_ops census_agg_histogram; -extern census_aggregation_ops census_agg_window; - -/** Information needed to instantiate a new aggregation. Used in view - construction via census_define_view(). */ -typedef struct { - const census_aggregation_ops *ops; - const void *create_arg; /* Aaggregation initialization argument. */ -} census_aggregation; - -/** A census view type. Opaque. */ -typedef struct census_view census_view; - -/** Create a new view. - @param metric_id Metric with which this view is associated. - @param tags tags that define the view. - @param aggregations aggregations to associate with the view - @param naggregations number of aggregations - - @return A new census view -*/ - -/* TODO(aveitch): consider if context is the right argument type to pass in - tags. */ -CENSUSAPI census_view *census_view_create( - uint32_t metric_id, const census_context *tags, - const census_aggregation *aggregations, size_t naggregations); - -/** Destroy a previously created view. */ -CENSUSAPI void census_view_delete(census_view *view); - -/** Metric ID associated with a view */ -CENSUSAPI size_t census_view_metric(const census_view *view); - -/** Number of aggregations associated with view. */ -CENSUSAPI size_t census_view_naggregations(const census_view *view); - -/** Get tags associated with view. */ -CENSUSAPI const census_context *census_view_tags(const census_view *view); - -/** Get aggregation descriptors associated with a view. */ -CENSUSAPI const census_aggregation *census_view_aggregrations( - const census_view *view); - -/** Holds all the aggregation data for a particular view instantiation. Forms - part of the data returned by census_view_data(). */ -typedef struct { - const census_context *tags; /* Tags for this set of aggregations. */ - const void **data; /* One data set for every aggregation in the view. */ -} census_view_aggregation_data; - -/** Census view data as returned by census_view_get_data(). */ -typedef struct { - size_t n_tag_sets; /* Number of unique tag sets that matched view. */ - const census_view_aggregation_data *data; /* n_tag_sets entries */ -} census_view_data; - -/** Get data from aggregations associated with a view. - @param view View from which to get data. - @return Full set of data for all aggregations for the view. -*/ -CENSUSAPI const census_view_data *census_view_get_data(const census_view *view); - -/** Reset all view data to zero for the specified view */ -CENSUSAPI void census_view_reset(census_view *view); - #ifdef __cplusplus } #endif diff --git a/package.xml b/package.xml index 6f6b8dd4bdb..575b7438fae 100644 --- a/package.xml +++ b/package.xml @@ -322,11 +322,13 @@ + + @@ -495,6 +497,7 @@ + @@ -504,6 +507,7 @@ + diff --git a/src/core/ext/census/base_resources.c b/src/core/ext/census/base_resources.c new file mode 100644 index 00000000000..8f0329a72bd --- /dev/null +++ b/src/core/ext/census/base_resources.c @@ -0,0 +1,148 @@ +/* + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include +#include + +#include +#include + +#include "base_resources.h" +#include "gen/census.pb.h" +#include "third_party/nanopb/pb_encode.h" + +// Add base RPC resource definitions for use by RPC runtime. +// +// TODO(aveitch): All of these are currently hardwired definitions encoded in +// the code in this file. These should be converted to use an external +// configuration mechanism, in which these resources are defined in a text +// file, which is compiled to .pb format and read by still-to-be-written +// configuration functions. + +// Structure representing a MeasurementUnit proto. +typedef struct { + int32_t prefix; + int n_numerators; + const google_census_Resource_BasicUnit *numerators; + int n_denominators; + const google_census_Resource_BasicUnit *denominators; +} measurement_unit; + +// Encode a nanopb string. Expects the string argument to be passed in as `arg`. +static bool encode_string(pb_ostream_t *stream, const pb_field_t *field, + void *const *arg) { + if (!pb_encode_tag_for_field(stream, field)) { + return false; + } + return pb_encode_string(stream, (uint8_t *)*arg, strlen((const char *)*arg)); +} + +// Encode the numerators part of a measurement_unit (passed in as `arg`). +static bool encode_numerators(pb_ostream_t *stream, const pb_field_t *field, + void *const *arg) { + const measurement_unit *mu = (const measurement_unit *)*arg; + for (int i = 0; i < mu->n_numerators; i++) { + if (!pb_encode_tag_for_field(stream, field)) { + return false; + } + if (!pb_encode_varint(stream, mu->numerators[i])) { + return false; + } + } + return true; +} + +// Encode the denominators part of a measurement_unit (passed in as `arg`). +static bool encode_denominators(pb_ostream_t *stream, const pb_field_t *field, + void *const *arg) { + const measurement_unit *mu = (const measurement_unit *)*arg; + for (int i = 0; i < mu->n_denominators; i++) { + if (!pb_encode_tag_for_field(stream, field)) { + return false; + } + if (!pb_encode_varint(stream, mu->numerators[i])) { + return false; + } + } + return true; +} + +// Define a Resource, given the important details. Encodes a protobuf, which +// is then passed to census_define_resource. +static void define_resource(const char *name, const char *description, + const measurement_unit *unit) { + // nanopb generated type for Resource. Initialize encoding functions to NULL + // since we can't directly initialize them due to embedded union in struct. + google_census_Resource resource = { + {{NULL}, (void *)name}, + {{NULL}, (void *)description}, + true, // has_unit + {true, unit->prefix, {{NULL}, (void *)unit}, {{NULL}, (void *)unit}}}; + resource.name.funcs.encode = &encode_string; + resource.description.funcs.encode = &encode_string; + resource.unit.numerator.funcs.encode = &encode_numerators; + resource.unit.denominator.funcs.encode = &encode_denominators; + + // Buffer for storing encoded proto. + uint8_t buffer[512]; + pb_ostream_t stream = pb_ostream_from_buffer(buffer, 512); + if (!pb_encode(&stream, google_census_Resource_fields, &resource)) { + gpr_log(GPR_ERROR, "Error encoding resource %s.", name); + return; + } + int32_t mid = census_define_resource(buffer, stream.bytes_written); + if (mid < 0) { + gpr_log(GPR_ERROR, "Error defining resource %s.", name); + } +} + +// Define a resource for client RPC latency. +static void define_client_rpc_latency_resource() { + google_census_Resource_BasicUnit numerator = + google_census_Resource_BasicUnit_SECS; + measurement_unit unit = {0, 1, &numerator, 0, NULL}; + define_resource("client_rpc_latency", "Client RPC latency in seconds", &unit); +} + +// Define a resource for server RPC latency. +static void define_server_rpc_latency_resource() { + google_census_Resource_BasicUnit numerator = + google_census_Resource_BasicUnit_SECS; + measurement_unit unit = {0, 1, &numerator, 0, NULL}; + define_resource("server_rpc_latency", "Server RPC latency in seconds", &unit); +} + +// Define all base resources. This should be called by census initialization. +void define_base_resources() { + define_client_rpc_latency_resource(); + define_server_rpc_latency_resource(); +} diff --git a/src/core/ext/census/base_resources.h b/src/core/ext/census/base_resources.h new file mode 100644 index 00000000000..992e5f6ebea --- /dev/null +++ b/src/core/ext/census/base_resources.h @@ -0,0 +1,39 @@ +/* + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#ifndef GRPC_CORE_EXT_CENSUS_BASE_RESOURCES_H +#define GRPC_CORE_EXT_CENSUS_BASE_RESOURCES_H + +// Define all base resources. This should be called by census initialization. +void define_base_resources(); + +#endif // GRPC_CORE_EXT_CENSUS_BASE_RESOURCES_H diff --git a/src/core/ext/census/initialize.c b/src/core/ext/census/initialize.c index 896276e44a1..118163512ad 100644 --- a/src/core/ext/census/initialize.c +++ b/src/core/ext/census/initialize.c @@ -32,19 +32,31 @@ */ #include +#include "base_resources.h" +#include "resource.h" static int features_enabled = CENSUS_FEATURE_NONE; int census_initialize(int features) { if (features_enabled != CENSUS_FEATURE_NONE) { // Must have been a previous call to census_initialize; return error - return 1; + return -1; } - features_enabled = features; - return 0; + features_enabled = features & CENSUS_FEATURE_ALL; + if (features & CENSUS_FEATURE_STATS) { + initialize_resources(); + define_base_resources(); + } + + return features_enabled; } -void census_shutdown(void) { features_enabled = CENSUS_FEATURE_NONE; } +void census_shutdown(void) { + if (features_enabled & CENSUS_FEATURE_STATS) { + shutdown_resources(); + } + features_enabled = CENSUS_FEATURE_NONE; +} int census_supported(void) { /* TODO(aveitch): improve this as we implement features... */ diff --git a/src/core/ext/census/placeholders.c b/src/core/ext/census/placeholders.c index fe23d13971a..9f99c5bdcfb 100644 --- a/src/core/ext/census/placeholders.c +++ b/src/core/ext/census/placeholders.c @@ -62,48 +62,3 @@ int census_trace_scan_start(int consume) { (void)consume; abort(); } - -const census_aggregation *census_view_aggregrations(const census_view *view) { - (void)view; - abort(); -} - -census_view *census_view_create(uint32_t metric_id, const census_context *tags, - const census_aggregation *aggregations, - size_t naggregations) { - (void)metric_id; - (void)tags; - (void)aggregations; - (void)naggregations; - abort(); -} - -const census_context *census_view_tags(const census_view *view) { - (void)view; - abort(); -} - -void census_view_delete(census_view *view) { - (void)view; - abort(); -} - -const census_view_data *census_view_get_data(const census_view *view) { - (void)view; - abort(); -} - -size_t census_view_metric(const census_view *view) { - (void)view; - abort(); -} - -size_t census_view_naggregations(const census_view *view) { - (void)view; - abort(); -} - -void census_view_reset(census_view *view) { - (void)view; - abort(); -} diff --git a/src/core/ext/census/resource.c b/src/core/ext/census/resource.c new file mode 100644 index 00000000000..632b899e415 --- /dev/null +++ b/src/core/ext/census/resource.c @@ -0,0 +1,255 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include "resource.h" +#include "gen/census.pb.h" +#include "third_party/nanopb/pb_decode.h" + +#include +#include +#include +#include + +#include +#include + +// Internal representation of a resource. +typedef struct { + char *name; + // Pointer to raw protobuf used in resource definition. + uint8_t *raw_pb; +} resource; + +// Protect local resource data structures. +static gpr_mu resource_lock; + +// Deleteing and creating resources are relatively rare events, and should not +// be done in the critical path of performance sensitive code. We record +// current resource id's used in a simple array, and just search it each time +// we need to assign a new id, or look up a resource. +static resource **resources = NULL; + +// Number of entries in *resources +static size_t n_resources = 0; + +// Number of defined resources +static size_t n_defined_resources = 0; + +void initialize_resources() { + gpr_mu_init(&resource_lock); + gpr_mu_lock(&resource_lock); + GPR_ASSERT(resources == NULL && n_resources == 0 && n_defined_resources == 0); + // 8 seems like a reasonable size for initial number of resources. + n_resources = 8; + resources = gpr_malloc(n_resources * sizeof(resource *)); + memset(resources, 0, n_resources * sizeof(resource *)); + gpr_mu_unlock(&resource_lock); +} + +// Delete a resource given it's ID. Must be called with resource_lock held. +static void delete_resource_locked(size_t rid) { + GPR_ASSERT(resources[rid] != NULL && resources[rid]->raw_pb != NULL && + resources[rid]->name != NULL && n_defined_resources > 0); + gpr_free(resources[rid]->name); + gpr_free(resources[rid]->raw_pb); + gpr_free(resources[rid]); + resources[rid] = NULL; + n_defined_resources--; +} + +void shutdown_resources() { + gpr_mu_lock(&resource_lock); + for (size_t i = 0; i < n_resources; i++) { + if (resources[i] != NULL) { + delete_resource_locked(i); + } + } + GPR_ASSERT(n_defined_resources == 0); + gpr_free(resources); + resources = NULL; + n_resources = 0; + gpr_mu_unlock(&resource_lock); +} + +// Check the contents of string fields in a resource proto. +static bool validate_string(pb_istream_t *stream, const pb_field_t *field, + void **arg) { + resource *vresource = (resource *)*arg; + switch (field->tag) { + case google_census_Resource_name_tag: + // Name must have at least one character + if (stream->bytes_left == 0) { + gpr_log(GPR_INFO, "Zero-length Resource name."); + return false; + } + vresource->name = gpr_malloc(stream->bytes_left + 1); + vresource->name[stream->bytes_left] = '\0'; + if (!pb_read(stream, (uint8_t *)vresource->name, stream->bytes_left)) { + return false; + } + // Can't have same name as an existing resource. + for (size_t i = 0; i < n_resources; i++) { + resource *compare = resources[i]; + if (compare == vresource || compare == NULL) continue; + if (strcmp(compare->name, vresource->name) == 0) { + gpr_log(GPR_INFO, "Duplicate Resource name %s.", vresource->name); + return false; + } + } + break; + case google_census_Resource_description_tag: + // Description is optional, does not need validating, just skip. + if (!pb_read(stream, NULL, stream->bytes_left)) { + return false; + } + break; + default: + // No other string fields in Resource. Print warning and skip. + gpr_log(GPR_INFO, "Unknown string field type in Resource protobuf."); + if (!pb_read(stream, NULL, stream->bytes_left)) { + return false; + } + break; + } + return true; +} + +// Validate units field of a Resource proto. +static bool validate_units(pb_istream_t *stream, const pb_field_t *field, + void **arg) { + if (field->tag == google_census_Resource_MeasurementUnit_numerator_tag) { + *(bool *)*arg = true; // has_numerator = true. + } + while (stream->bytes_left) { + uint64_t value; + if (!pb_decode_varint(stream, &value)) { + return false; + } + } + return true; +} + +// Vlaidate the contents of a Resource proto. `id` is the intended resource id. +static bool validate_resource_pb(const uint8_t *resource_pb, + size_t resource_pb_size, size_t id) { + GPR_ASSERT(id < n_resources); + if (resource_pb == NULL) { + return false; + } + google_census_Resource vresource; + vresource.name.funcs.decode = &validate_string; + vresource.name.arg = resources[id]; + vresource.description.funcs.decode = &validate_string; + vresource.unit.numerator.funcs.decode = &validate_units; + bool has_numerator = false; + vresource.unit.numerator.arg = &has_numerator; + vresource.unit.denominator.funcs.decode = &validate_units; + + pb_istream_t stream = + pb_istream_from_buffer((uint8_t *)resource_pb, resource_pb_size); + if (!pb_decode(&stream, google_census_Resource_fields, &vresource)) { + return false; + } + // A Resource must have a name, a unit, with at least one numerator. + return (resources[id]->name != NULL && vresource.has_unit && has_numerator); +} + +int32_t census_define_resource(const uint8_t *resource_pb, + size_t resource_pb_size) { + // use next_id to optimize expected placement of next new resource. + static size_t next_id = 0; + if (resource_pb == NULL) { + return -1; + } + gpr_mu_lock(&resource_lock); + size_t id = n_resources; // resource ID - initialize to invalid value. + // Expand resources if needed. + if (n_resources == n_defined_resources) { + resource **new_resources = gpr_malloc(n_resources * 2 * sizeof(resource *)); + memcpy(new_resources, resources, n_resources * sizeof(resource *)); + memset(new_resources + n_resources, 0, n_resources * sizeof(resource *)); + gpr_free(resources); + resources = new_resources; + n_resources *= 2; + id = n_defined_resources; + } else { + GPR_ASSERT(n_defined_resources < n_resources); + // Find a free id. + for (size_t base = 0; base < n_resources; base++) { + id = (next_id + base) % n_resources; + if (resources[id] == NULL) break; + } + } + GPR_ASSERT(id < n_resources && resources[id] == NULL); + resources[id] = gpr_malloc(sizeof(resource)); + resources[id]->name = NULL; + // Validate pb, extract name. + if (!validate_resource_pb(resource_pb, resource_pb_size, id)) { + if (resources[id]->name != NULL) { + gpr_free(resources[id]->name); + } + gpr_free(resources[id]); + resources[id] = NULL; + gpr_mu_unlock(&resource_lock); + return -1; + } + next_id = (id + 1) % n_resources; + // Make copy of raw proto, and return. + resources[id]->raw_pb = gpr_malloc(resource_pb_size); + memcpy(resources[id]->raw_pb, resource_pb, resource_pb_size); + n_defined_resources++; + gpr_mu_unlock(&resource_lock); + return (int32_t)id; +} + +void census_delete_resource(int32_t rid) { + gpr_mu_lock(&resource_lock); + if (rid >= 0 && (size_t)rid < n_resources && resources[rid] != NULL) { + delete_resource_locked((size_t)rid); + } + gpr_mu_unlock(&resource_lock); +} + +int32_t census_resource_id(const char *name) { + gpr_mu_lock(&resource_lock); + for (int32_t id = 0; (size_t)id < n_resources; id++) { + if (resources[id] != NULL) { + if (strcmp(resources[id]->name, name) == 0) { + gpr_mu_unlock(&resource_lock); + return id; + } + } + } + gpr_mu_unlock(&resource_lock); + return -1; +} diff --git a/src/core/ext/census/resource.h b/src/core/ext/census/resource.h new file mode 100644 index 00000000000..5ad5a0f7a0c --- /dev/null +++ b/src/core/ext/census/resource.h @@ -0,0 +1,42 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +// Census-internal resource definition and manipluation functions. +#ifndef GRPC_CORE_EXT_CENSUS_RESOURCE_H +#define GRPC_CORE_EXT_CENSUS_RESOURCE_H + +// Initialize and shutdown the resources subsystem. +void initialize_resources(); +void shutdown_resources(); + +#endif /* GRPC_CORE_EXT_CENSUS_RESOURCE_H */ diff --git a/src/python/grpcio/grpc/_cython/imports.generated.c b/src/python/grpcio/grpc/_cython/imports.generated.c index f71cf128445..29ada16afdd 100644 --- a/src/python/grpcio/grpc/_cython/imports.generated.c +++ b/src/python/grpcio/grpc/_cython/imports.generated.c @@ -61,15 +61,10 @@ census_trace_print_type census_trace_print_import; census_trace_scan_start_type census_trace_scan_start_import; census_get_trace_record_type census_get_trace_record_import; census_trace_scan_end_type census_trace_scan_end_import; +census_define_resource_type census_define_resource_import; +census_delete_resource_type census_delete_resource_import; +census_resource_id_type census_resource_id_import; census_record_values_type census_record_values_import; -census_view_create_type census_view_create_import; -census_view_delete_type census_view_delete_import; -census_view_metric_type census_view_metric_import; -census_view_naggregations_type census_view_naggregations_import; -census_view_tags_type census_view_tags_import; -census_view_aggregrations_type census_view_aggregrations_import; -census_view_get_data_type census_view_get_data_import; -census_view_reset_type census_view_reset_import; grpc_compression_algorithm_parse_type grpc_compression_algorithm_parse_import; grpc_compression_algorithm_name_type grpc_compression_algorithm_name_import; grpc_compression_algorithm_for_level_type grpc_compression_algorithm_for_level_import; @@ -333,15 +328,10 @@ void pygrpc_load_imports(HMODULE library) { census_trace_scan_start_import = (census_trace_scan_start_type) GetProcAddress(library, "census_trace_scan_start"); census_get_trace_record_import = (census_get_trace_record_type) GetProcAddress(library, "census_get_trace_record"); census_trace_scan_end_import = (census_trace_scan_end_type) GetProcAddress(library, "census_trace_scan_end"); + census_define_resource_import = (census_define_resource_type) GetProcAddress(library, "census_define_resource"); + census_delete_resource_import = (census_delete_resource_type) GetProcAddress(library, "census_delete_resource"); + census_resource_id_import = (census_resource_id_type) GetProcAddress(library, "census_resource_id"); census_record_values_import = (census_record_values_type) GetProcAddress(library, "census_record_values"); - census_view_create_import = (census_view_create_type) GetProcAddress(library, "census_view_create"); - census_view_delete_import = (census_view_delete_type) GetProcAddress(library, "census_view_delete"); - census_view_metric_import = (census_view_metric_type) GetProcAddress(library, "census_view_metric"); - census_view_naggregations_import = (census_view_naggregations_type) GetProcAddress(library, "census_view_naggregations"); - census_view_tags_import = (census_view_tags_type) GetProcAddress(library, "census_view_tags"); - census_view_aggregrations_import = (census_view_aggregrations_type) GetProcAddress(library, "census_view_aggregrations"); - census_view_get_data_import = (census_view_get_data_type) GetProcAddress(library, "census_view_get_data"); - census_view_reset_import = (census_view_reset_type) GetProcAddress(library, "census_view_reset"); grpc_compression_algorithm_parse_import = (grpc_compression_algorithm_parse_type) GetProcAddress(library, "grpc_compression_algorithm_parse"); grpc_compression_algorithm_name_import = (grpc_compression_algorithm_name_type) GetProcAddress(library, "grpc_compression_algorithm_name"); grpc_compression_algorithm_for_level_import = (grpc_compression_algorithm_for_level_type) GetProcAddress(library, "grpc_compression_algorithm_for_level"); diff --git a/src/python/grpcio/grpc/_cython/imports.generated.h b/src/python/grpcio/grpc/_cython/imports.generated.h index a364075e9e3..f6dd79cadc2 100644 --- a/src/python/grpcio/grpc/_cython/imports.generated.h +++ b/src/python/grpcio/grpc/_cython/imports.generated.h @@ -134,33 +134,18 @@ extern census_get_trace_record_type census_get_trace_record_import; typedef void(*census_trace_scan_end_type)(); extern census_trace_scan_end_type census_trace_scan_end_import; #define census_trace_scan_end census_trace_scan_end_import +typedef int32_t(*census_define_resource_type)(const uint8_t *resource_pb, size_t resource_pb_size); +extern census_define_resource_type census_define_resource_import; +#define census_define_resource census_define_resource_import +typedef void(*census_delete_resource_type)(int32_t resource_id); +extern census_delete_resource_type census_delete_resource_import; +#define census_delete_resource census_delete_resource_import +typedef int32_t(*census_resource_id_type)(const char *name); +extern census_resource_id_type census_resource_id_import; +#define census_resource_id census_resource_id_import typedef void(*census_record_values_type)(census_context *context, census_value *values, size_t nvalues); extern census_record_values_type census_record_values_import; #define census_record_values census_record_values_import -typedef census_view *(*census_view_create_type)(uint32_t metric_id, const census_context *tags, const census_aggregation *aggregations, size_t naggregations); -extern census_view_create_type census_view_create_import; -#define census_view_create census_view_create_import -typedef void(*census_view_delete_type)(census_view *view); -extern census_view_delete_type census_view_delete_import; -#define census_view_delete census_view_delete_import -typedef size_t(*census_view_metric_type)(const census_view *view); -extern census_view_metric_type census_view_metric_import; -#define census_view_metric census_view_metric_import -typedef size_t(*census_view_naggregations_type)(const census_view *view); -extern census_view_naggregations_type census_view_naggregations_import; -#define census_view_naggregations census_view_naggregations_import -typedef const census_context *(*census_view_tags_type)(const census_view *view); -extern census_view_tags_type census_view_tags_import; -#define census_view_tags census_view_tags_import -typedef const census_aggregation *(*census_view_aggregrations_type)(const census_view *view); -extern census_view_aggregrations_type census_view_aggregrations_import; -#define census_view_aggregrations census_view_aggregrations_import -typedef const census_view_data *(*census_view_get_data_type)(const census_view *view); -extern census_view_get_data_type census_view_get_data_import; -#define census_view_get_data census_view_get_data_import -typedef void(*census_view_reset_type)(census_view *view); -extern census_view_reset_type census_view_reset_import; -#define census_view_reset census_view_reset_import typedef int(*grpc_compression_algorithm_parse_type)(const char *name, size_t name_length, grpc_compression_algorithm *algorithm); extern grpc_compression_algorithm_parse_type grpc_compression_algorithm_parse_import; #define grpc_compression_algorithm_parse grpc_compression_algorithm_parse_import diff --git a/src/python/grpcio/grpc_core_dependencies.py b/src/python/grpcio/grpc_core_dependencies.py index d0f23f42ccc..c7d042d5657 100644 --- a/src/python/grpcio/grpc_core_dependencies.py +++ b/src/python/grpcio/grpc_core_dependencies.py @@ -243,6 +243,7 @@ CORE_SOURCE_FILES = [ 'src/core/ext/lb_policy/round_robin/round_robin.c', 'src/core/ext/resolver/dns/native/dns_resolver.c', 'src/core/ext/resolver/sockaddr/sockaddr_resolver.c', + 'src/core/ext/census/base_resources.c', 'src/core/ext/census/context.c', 'src/core/ext/census/gen/census.pb.c', 'src/core/ext/census/grpc_context.c', @@ -252,6 +253,7 @@ CORE_SOURCE_FILES = [ 'src/core/ext/census/mlog.c', 'src/core/ext/census/operation.c', 'src/core/ext/census/placeholders.c', + 'src/core/ext/census/resource.c', 'src/core/ext/census/tracing.c', 'src/core/plugin_registry/grpc_plugin_registry.c', 'src/boringssl/err_data.c', diff --git a/src/ruby/ext/grpc/rb_grpc_imports.generated.c b/src/ruby/ext/grpc/rb_grpc_imports.generated.c index 3b62984defa..5d0f565ea83 100644 --- a/src/ruby/ext/grpc/rb_grpc_imports.generated.c +++ b/src/ruby/ext/grpc/rb_grpc_imports.generated.c @@ -61,15 +61,10 @@ census_trace_print_type census_trace_print_import; census_trace_scan_start_type census_trace_scan_start_import; census_get_trace_record_type census_get_trace_record_import; census_trace_scan_end_type census_trace_scan_end_import; +census_define_resource_type census_define_resource_import; +census_delete_resource_type census_delete_resource_import; +census_resource_id_type census_resource_id_import; census_record_values_type census_record_values_import; -census_view_create_type census_view_create_import; -census_view_delete_type census_view_delete_import; -census_view_metric_type census_view_metric_import; -census_view_naggregations_type census_view_naggregations_import; -census_view_tags_type census_view_tags_import; -census_view_aggregrations_type census_view_aggregrations_import; -census_view_get_data_type census_view_get_data_import; -census_view_reset_type census_view_reset_import; grpc_compression_algorithm_parse_type grpc_compression_algorithm_parse_import; grpc_compression_algorithm_name_type grpc_compression_algorithm_name_import; grpc_compression_algorithm_for_level_type grpc_compression_algorithm_for_level_import; @@ -329,15 +324,10 @@ void grpc_rb_load_imports(HMODULE library) { census_trace_scan_start_import = (census_trace_scan_start_type) GetProcAddress(library, "census_trace_scan_start"); census_get_trace_record_import = (census_get_trace_record_type) GetProcAddress(library, "census_get_trace_record"); census_trace_scan_end_import = (census_trace_scan_end_type) GetProcAddress(library, "census_trace_scan_end"); + census_define_resource_import = (census_define_resource_type) GetProcAddress(library, "census_define_resource"); + census_delete_resource_import = (census_delete_resource_type) GetProcAddress(library, "census_delete_resource"); + census_resource_id_import = (census_resource_id_type) GetProcAddress(library, "census_resource_id"); census_record_values_import = (census_record_values_type) GetProcAddress(library, "census_record_values"); - census_view_create_import = (census_view_create_type) GetProcAddress(library, "census_view_create"); - census_view_delete_import = (census_view_delete_type) GetProcAddress(library, "census_view_delete"); - census_view_metric_import = (census_view_metric_type) GetProcAddress(library, "census_view_metric"); - census_view_naggregations_import = (census_view_naggregations_type) GetProcAddress(library, "census_view_naggregations"); - census_view_tags_import = (census_view_tags_type) GetProcAddress(library, "census_view_tags"); - census_view_aggregrations_import = (census_view_aggregrations_type) GetProcAddress(library, "census_view_aggregrations"); - census_view_get_data_import = (census_view_get_data_type) GetProcAddress(library, "census_view_get_data"); - census_view_reset_import = (census_view_reset_type) GetProcAddress(library, "census_view_reset"); grpc_compression_algorithm_parse_import = (grpc_compression_algorithm_parse_type) GetProcAddress(library, "grpc_compression_algorithm_parse"); grpc_compression_algorithm_name_import = (grpc_compression_algorithm_name_type) GetProcAddress(library, "grpc_compression_algorithm_name"); grpc_compression_algorithm_for_level_import = (grpc_compression_algorithm_for_level_type) GetProcAddress(library, "grpc_compression_algorithm_for_level"); diff --git a/src/ruby/ext/grpc/rb_grpc_imports.generated.h b/src/ruby/ext/grpc/rb_grpc_imports.generated.h index 1428e6d71c3..4033e5e6263 100644 --- a/src/ruby/ext/grpc/rb_grpc_imports.generated.h +++ b/src/ruby/ext/grpc/rb_grpc_imports.generated.h @@ -134,33 +134,18 @@ extern census_get_trace_record_type census_get_trace_record_import; typedef void(*census_trace_scan_end_type)(); extern census_trace_scan_end_type census_trace_scan_end_import; #define census_trace_scan_end census_trace_scan_end_import +typedef int32_t(*census_define_resource_type)(const uint8_t *resource_pb, size_t resource_pb_size); +extern census_define_resource_type census_define_resource_import; +#define census_define_resource census_define_resource_import +typedef void(*census_delete_resource_type)(int32_t resource_id); +extern census_delete_resource_type census_delete_resource_import; +#define census_delete_resource census_delete_resource_import +typedef int32_t(*census_resource_id_type)(const char *name); +extern census_resource_id_type census_resource_id_import; +#define census_resource_id census_resource_id_import typedef void(*census_record_values_type)(census_context *context, census_value *values, size_t nvalues); extern census_record_values_type census_record_values_import; #define census_record_values census_record_values_import -typedef census_view *(*census_view_create_type)(uint32_t metric_id, const census_context *tags, const census_aggregation *aggregations, size_t naggregations); -extern census_view_create_type census_view_create_import; -#define census_view_create census_view_create_import -typedef void(*census_view_delete_type)(census_view *view); -extern census_view_delete_type census_view_delete_import; -#define census_view_delete census_view_delete_import -typedef size_t(*census_view_metric_type)(const census_view *view); -extern census_view_metric_type census_view_metric_import; -#define census_view_metric census_view_metric_import -typedef size_t(*census_view_naggregations_type)(const census_view *view); -extern census_view_naggregations_type census_view_naggregations_import; -#define census_view_naggregations census_view_naggregations_import -typedef const census_context *(*census_view_tags_type)(const census_view *view); -extern census_view_tags_type census_view_tags_import; -#define census_view_tags census_view_tags_import -typedef const census_aggregation *(*census_view_aggregrations_type)(const census_view *view); -extern census_view_aggregrations_type census_view_aggregrations_import; -#define census_view_aggregrations census_view_aggregrations_import -typedef const census_view_data *(*census_view_get_data_type)(const census_view *view); -extern census_view_get_data_type census_view_get_data_import; -#define census_view_get_data census_view_get_data_import -typedef void(*census_view_reset_type)(census_view *view); -extern census_view_reset_type census_view_reset_import; -#define census_view_reset census_view_reset_import typedef int(*grpc_compression_algorithm_parse_type)(const char *name, size_t name_length, grpc_compression_algorithm *algorithm); extern grpc_compression_algorithm_parse_type grpc_compression_algorithm_parse_import; #define grpc_compression_algorithm_parse grpc_compression_algorithm_parse_import diff --git a/test/core/census/README b/test/core/census/README new file mode 100644 index 00000000000..d5363b72333 --- /dev/null +++ b/test/core/census/README @@ -0,0 +1,7 @@ +Test source and data files for Census. + +binary proto files (*.pb) in data directory are generated from the *.txt file, +via: + +BASE="filename" +cat $BASE.txt | protoc --encode=google.census.Resource census.proto > $BASE.pb diff --git a/test/core/census/data/resource_empty_name.pb b/test/core/census/data/resource_empty_name.pb new file mode 100644 index 00000000000..4d547445fae --- /dev/null +++ b/test/core/census/data/resource_empty_name.pb @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/test/core/census/data/resource_empty_name.txt b/test/core/census/data/resource_empty_name.txt new file mode 100644 index 00000000000..271fd3274c2 --- /dev/null +++ b/test/core/census/data/resource_empty_name.txt @@ -0,0 +1,5 @@ +# Name is present, but empty. +name : '' +unit { + numerator : SECS +} diff --git a/test/core/census/data/resource_full.pb b/test/core/census/data/resource_full.pb new file mode 100644 index 00000000000..e4c6a2aef53 --- /dev/null +++ b/test/core/census/data/resource_full.pb @@ -0,0 +1,2 @@ + + full_resource"A resource with everything defined \ No newline at end of file diff --git a/test/core/census/data/resource_full.txt b/test/core/census/data/resource_full.txt new file mode 100644 index 00000000000..1aa2fafe3a4 --- /dev/null +++ b/test/core/census/data/resource_full.txt @@ -0,0 +1,9 @@ +# A full resource definition - all fields filled out. +name : 'full_resource' +description : 'A resource with everything defined' +unit { + # Megabits per second. + prefix : 6 + numerator : BITS + denominator : SECS +} diff --git a/test/core/census/data/resource_minimal_good.pb b/test/core/census/data/resource_minimal_good.pb new file mode 100644 index 00000000000..7100c462bf1 --- /dev/null +++ b/test/core/census/data/resource_minimal_good.pb @@ -0,0 +1,2 @@ + + minimal_good \ No newline at end of file diff --git a/test/core/census/data/resource_minimal_good.txt b/test/core/census/data/resource_minimal_good.txt new file mode 100644 index 00000000000..a7a7e71dd62 --- /dev/null +++ b/test/core/census/data/resource_minimal_good.txt @@ -0,0 +1,5 @@ +# A minimal "good" Resource definition: has a name and numerator/unit. +name : 'minimal_good' +unit { + numerator : SECS +} diff --git a/test/core/census/data/resource_no_name.pb b/test/core/census/data/resource_no_name.pb new file mode 100644 index 00000000000..4d547445fae --- /dev/null +++ b/test/core/census/data/resource_no_name.pb @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/test/core/census/data/resource_no_name.txt b/test/core/census/data/resource_no_name.txt new file mode 100644 index 00000000000..8f12a91d35e --- /dev/null +++ b/test/core/census/data/resource_no_name.txt @@ -0,0 +1,4 @@ +# The minimal good Resource without a name. +unit { + numerator : SECS +} diff --git a/test/core/census/data/resource_no_numerator.pb b/test/core/census/data/resource_no_numerator.pb new file mode 100644 index 00000000000..2a5cceee70c --- /dev/null +++ b/test/core/census/data/resource_no_numerator.pb @@ -0,0 +1,2 @@ + +resource_no_numerator \ No newline at end of file diff --git a/test/core/census/data/resource_no_numerator.txt b/test/core/census/data/resource_no_numerator.txt new file mode 100644 index 00000000000..fc1fec74a24 --- /dev/null +++ b/test/core/census/data/resource_no_numerator.txt @@ -0,0 +1,6 @@ +# Resource without a numerator +name : 'resource_no_numerator' +unit { + prefix : -3 + denominator : SECS +} diff --git a/test/core/census/data/resource_no_unit.pb b/test/core/census/data/resource_no_unit.pb new file mode 100644 index 00000000000..9dca2620e0a --- /dev/null +++ b/test/core/census/data/resource_no_unit.pb @@ -0,0 +1,2 @@ + +resource_no_unit \ No newline at end of file diff --git a/test/core/census/data/resource_no_unit.txt b/test/core/census/data/resource_no_unit.txt new file mode 100644 index 00000000000..c5d5115cebf --- /dev/null +++ b/test/core/census/data/resource_no_unit.txt @@ -0,0 +1,2 @@ +# The minimal good resource without a unit +name : 'resource_no_unit' diff --git a/test/core/census/resource_test.c b/test/core/census/resource_test.c new file mode 100644 index 00000000000..bfb76bcfb5e --- /dev/null +++ b/test/core/census/resource_test.c @@ -0,0 +1,157 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include "src/core/ext/census/resource.h" +#include +#include +#include +#include +#include +#include +#include +#include "src/core/ext/census/base_resources.h" +#include "test/core/util/test_config.h" + +// Test all the functionality for dealing with Resources. + +// Just startup and shutdown resources subsystem. +static void test_enable_disable() { + initialize_resources(); + shutdown_resources(); +} + +// A blank/empty initialization should not work. +static void test_empty_definition() { + initialize_resources(); + int32_t rid = census_define_resource(NULL, 0); + GPR_ASSERT(rid == -1); + uint8_t buffer[50] = {0}; + rid = census_define_resource(buffer, 50); + GPR_ASSERT(rid == -1); + shutdown_resources(); +} + +// Given a file name, read raw proto and define the resource included within. +// Returns resource id from census_define_resource(). +static int32_t define_resource_from_file(const char *file) { + const size_t buf_size = 512; + uint8_t buffer[buf_size]; + FILE *input = fopen(file, "r"); + GPR_ASSERT(input != NULL); + size_t nbytes = fread(buffer, 1, buf_size, input); + GPR_ASSERT(nbytes != 0 && nbytes < buf_size); + int32_t rid = census_define_resource(buffer, nbytes); + GPR_ASSERT(fclose(input) == 0); + return rid; +} + +// Test definition of a single resource, using a proto read from a file. The +// `succeed` parameter indicates whether we expect the definition to succeed or +// fail. `name` is used to check that the returned resource can be looked up by +// name. +static void test_define_single_resource(const char *file, const char *name, + bool succeed) { + gpr_log(GPR_INFO, "Test defining resource \"%s\"\n", name); + initialize_resources(); + int32_t rid = define_resource_from_file(file); + if (succeed) { + GPR_ASSERT(rid >= 0); + int32_t rid2 = census_resource_id(name); + GPR_ASSERT(rid == rid2); + } else { + GPR_ASSERT(rid < 0); + } + shutdown_resources(); +} + +// Try deleting various resources (both those that exist and those that don't). +static void test_delete_resource() { + initialize_resources(); + // Try deleting resource before any are defined. + census_delete_resource(0); + // Create and check a couple of resources. + int32_t rid1 = define_resource_from_file( + "test/core/census/data/resource_minimal_good.pb"); + int32_t rid2 = + define_resource_from_file("test/core/census/data/resource_full.pb"); + GPR_ASSERT(rid1 >= 0 && rid2 >= 0 && rid1 != rid2); + int32_t rid3 = census_resource_id("minimal_good"); + int32_t rid4 = census_resource_id("full_resource"); + GPR_ASSERT(rid1 == rid3 && rid2 == rid4); + // Try deleting non-existant resources. + census_delete_resource(-1); + census_delete_resource(rid1 + rid2 + 1); + census_delete_resource(10000000); + // Delete one of the previously defined resources and check for deletion. + census_delete_resource(rid1); + rid3 = census_resource_id("minimal_good"); + GPR_ASSERT(rid3 < 0); + // Check that re-adding works. + rid1 = define_resource_from_file( + "test/core/census/data/resource_minimal_good.pb"); + GPR_ASSERT(rid1 >= 0); + rid3 = census_resource_id("minimal_good"); + GPR_ASSERT(rid1 == rid3); + shutdown_resources(); +} + +// Test define base resources. +static void test_base_resources() { + initialize_resources(); + define_base_resources(); + int32_t rid1 = census_resource_id("client_rpc_latency"); + int32_t rid2 = census_resource_id("server_rpc_latency"); + GPR_ASSERT(rid1 >= 0 && rid2 >= 0 && rid1 != rid2); + shutdown_resources(); +} + +int main(int argc, char **argv) { + grpc_test_init(argc, argv); + test_enable_disable(); + test_empty_definition(); + test_define_single_resource("test/core/census/data/resource_minimal_good.pb", + "minimal_good", true); + test_define_single_resource("test/core/census/data/resource_full.pb", + "full_resource", true); + test_define_single_resource("test/core/census/data/resource_no_name.pb", + "resource_no_name", false); + test_define_single_resource("test/core/census/data/resource_no_numerator.pb", + "resource_no_numerator", false); + test_define_single_resource("test/core/census/data/resource_no_unit.pb", + "resource_no_unit", false); + test_define_single_resource("test/core/census/data/resource_empty_name.pb", + "resource_empty_name", false); + test_delete_resource(); + test_base_resources(); + return 0; +} diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index 7446fec8244..51007a5da96 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -932,11 +932,13 @@ third_party/nanopb/pb_common.h \ third_party/nanopb/pb_decode.h \ third_party/nanopb/pb_encode.h \ src/core/ext/census/aggregation.h \ +src/core/ext/census/base_resources.h \ src/core/ext/census/census_interface.h \ src/core/ext/census/census_rpc_stats.h \ src/core/ext/census/gen/census.pb.h \ src/core/ext/census/grpc_filter.h \ src/core/ext/census/mlog.h \ +src/core/ext/census/resource.h \ src/core/ext/census/rpc_metric_id.h \ src/core/lib/surface/init.c \ src/core/lib/channel/channel_args.c \ @@ -1105,6 +1107,7 @@ src/core/ext/lb_policy/pick_first/pick_first.c \ src/core/ext/lb_policy/round_robin/round_robin.c \ src/core/ext/resolver/dns/native/dns_resolver.c \ src/core/ext/resolver/sockaddr/sockaddr_resolver.c \ +src/core/ext/census/base_resources.c \ src/core/ext/census/context.c \ src/core/ext/census/gen/census.pb.c \ src/core/ext/census/grpc_context.c \ @@ -1114,6 +1117,7 @@ src/core/ext/census/initialize.c \ src/core/ext/census/mlog.c \ src/core/ext/census/operation.c \ src/core/ext/census/placeholders.c \ +src/core/ext/census/resource.c \ src/core/ext/census/tracing.c \ src/core/plugin_registry/grpc_plugin_registry.c \ include/grpc/support/alloc.h \ diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index e0471234954..28f54e7c9e2 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -109,6 +109,22 @@ "third_party": false, "type": "target" }, + { + "deps": [ + "gpr", + "gpr_test_util", + "grpc", + "grpc_test_util" + ], + "headers": [], + "language": "c", + "name": "census_resource_test", + "src": [ + "test/core/census/resource_test.c" + ], + "third_party": false, + "type": "target" + }, { "deps": [ "gpr", @@ -5296,11 +5312,13 @@ "headers": [ "include/grpc/census.h", "src/core/ext/census/aggregation.h", + "src/core/ext/census/base_resources.h", "src/core/ext/census/census_interface.h", "src/core/ext/census/census_rpc_stats.h", "src/core/ext/census/gen/census.pb.h", "src/core/ext/census/grpc_filter.h", "src/core/ext/census/mlog.h", + "src/core/ext/census/resource.h", "src/core/ext/census/rpc_metric_id.h" ], "language": "c", @@ -5308,6 +5326,8 @@ "src": [ "include/grpc/census.h", "src/core/ext/census/aggregation.h", + "src/core/ext/census/base_resources.c", + "src/core/ext/census/base_resources.h", "src/core/ext/census/census_interface.h", "src/core/ext/census/census_rpc_stats.h", "src/core/ext/census/context.c", @@ -5322,6 +5342,8 @@ "src/core/ext/census/mlog.h", "src/core/ext/census/operation.c", "src/core/ext/census/placeholders.c", + "src/core/ext/census/resource.c", + "src/core/ext/census/resource.h", "src/core/ext/census/rpc_metric_id.h", "src/core/ext/census/tracing.c" ], diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index 850f9474aec..a1946d27cf8 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -127,6 +127,27 @@ "windows" ] }, + { + "args": [], + "ci_platforms": [ + "linux", + "mac", + "posix", + "windows" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "gtest": false, + "language": "c", + "name": "census_resource_test", + "platforms": [ + "linux", + "mac", + "posix", + "windows" + ] + }, { "args": [], "ci_platforms": [ diff --git a/vsprojects/buildtests_c.sln b/vsprojects/buildtests_c.sln index be8b5d40ace..d94a43501e0 100644 --- a/vsprojects/buildtests_c.sln +++ b/vsprojects/buildtests_c.sln @@ -180,6 +180,17 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "census_context_test", "vcxp {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "census_resource_test", "vcxproj\test\census_resource_test\census_resource_test.vcxproj", "{18CF99B5-3C61-EC3D-9509-3C95334C3B88}" + ProjectSection(myProperties) = preProject + lib = "False" + EndProjectSection + ProjectSection(ProjectDependencies) = postProject + {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} + {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} + {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + EndProjectSection +EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "channel_create_test", "vcxproj\test\channel_create_test\channel_create_test.vcxproj", "{AFC88484-3A2E-32BC-25B2-23DF741D4F3D}" ProjectSection(myProperties) = preProject lib = "False" @@ -1707,6 +1718,22 @@ Global {5C1CFC2D-AF3C-D7CB-BA74-D267E91CBC73}.Release-DLL|Win32.Build.0 = Release|Win32 {5C1CFC2D-AF3C-D7CB-BA74-D267E91CBC73}.Release-DLL|x64.ActiveCfg = Release|x64 {5C1CFC2D-AF3C-D7CB-BA74-D267E91CBC73}.Release-DLL|x64.Build.0 = Release|x64 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Debug|Win32.ActiveCfg = Debug|Win32 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Debug|x64.ActiveCfg = Debug|x64 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Release|Win32.ActiveCfg = Release|Win32 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Release|x64.ActiveCfg = Release|x64 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Debug|Win32.Build.0 = Debug|Win32 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Debug|x64.Build.0 = Debug|x64 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Release|Win32.Build.0 = Release|Win32 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Release|x64.Build.0 = Release|x64 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Debug-DLL|Win32.ActiveCfg = Debug|Win32 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Debug-DLL|Win32.Build.0 = Debug|Win32 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Debug-DLL|x64.ActiveCfg = Debug|x64 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Debug-DLL|x64.Build.0 = Debug|x64 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Release-DLL|Win32.ActiveCfg = Release|Win32 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Release-DLL|Win32.Build.0 = Release|Win32 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Release-DLL|x64.ActiveCfg = Release|x64 + {18CF99B5-3C61-EC3D-9509-3C95334C3B88}.Release-DLL|x64.Build.0 = Release|x64 {AFC88484-3A2E-32BC-25B2-23DF741D4F3D}.Debug|Win32.ActiveCfg = Debug|Win32 {AFC88484-3A2E-32BC-25B2-23DF741D4F3D}.Debug|x64.ActiveCfg = Debug|x64 {AFC88484-3A2E-32BC-25B2-23DF741D4F3D}.Release|Win32.ActiveCfg = Release|Win32 diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj b/vsprojects/vcxproj/grpc/grpc.vcxproj index 61a59e7c3f0..d5ac625eb22 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj @@ -441,11 +441,13 @@ + + @@ -783,6 +785,8 @@ + + @@ -801,6 +805,8 @@ + + diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters index 4bd436aa167..37b1f0993a8 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters @@ -502,6 +502,9 @@ src\core\ext\resolver\sockaddr + + src\core\ext\census + src\core\ext\census @@ -529,6 +532,9 @@ src\core\ext\census + + src\core\ext\census + src\core\ext\census @@ -1055,6 +1061,9 @@ src\core\ext\census + + src\core\ext\census + src\core\ext\census @@ -1070,6 +1079,9 @@ src\core\ext\census + + src\core\ext\census + src\core\ext\census diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj index e29a275d5ab..5bb261486e9 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj @@ -405,11 +405,13 @@ + + @@ -685,6 +687,8 @@ + + @@ -703,6 +707,8 @@ + + diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters index e5e4acc9a5d..3616227ac3e 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters @@ -409,6 +409,9 @@ src\core\ext\lb_policy\round_robin + + src\core\ext\census + src\core\ext\census @@ -436,6 +439,9 @@ src\core\ext\census + + src\core\ext\census + src\core\ext\census @@ -881,6 +887,9 @@ src\core\ext\census + + src\core\ext\census + src\core\ext\census @@ -896,6 +905,9 @@ src\core\ext\census + + src\core\ext\census + src\core\ext\census diff --git a/vsprojects/vcxproj/test/census_resource_test/census_resource_test.vcxproj b/vsprojects/vcxproj/test/census_resource_test/census_resource_test.vcxproj new file mode 100644 index 00000000000..c4fbd54e222 --- /dev/null +++ b/vsprojects/vcxproj/test/census_resource_test/census_resource_test.vcxproj @@ -0,0 +1,199 @@ + + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {18CF99B5-3C61-EC3D-9509-3C95334C3B88} + true + $(SolutionDir)IntDir\$(MSBuildProjectName)\ + + + + v100 + + + v110 + + + v120 + + + v140 + + + Application + true + Unicode + + + Application + false + true + Unicode + + + + + + + + + + + + + + census_resource_test + static + Debug + static + Debug + + + census_resource_test + static + Release + static + Release + + + + NotUsing + Level3 + Disabled + WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) + true + MultiThreadedDebug + true + None + false + + + Console + true + false + + + + + + NotUsing + Level3 + Disabled + WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) + true + MultiThreadedDebug + true + None + false + + + Console + true + false + + + + + + NotUsing + Level3 + MaxSpeed + WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) + true + true + true + MultiThreaded + true + None + false + + + Console + true + false + true + true + + + + + + NotUsing + Level3 + MaxSpeed + WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) + true + true + true + MultiThreaded + true + None + false + + + Console + true + false + true + true + + + + + + + + + + {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} + + + {29D16885-7228-4C31-81ED-5F9187C7F2A9} + + + {EAB0A629-17A9-44DB-B5FF-E91A721FE037} + + + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + diff --git a/vsprojects/vcxproj/test/census_resource_test/census_resource_test.vcxproj.filters b/vsprojects/vcxproj/test/census_resource_test/census_resource_test.vcxproj.filters new file mode 100644 index 00000000000..93faac55d07 --- /dev/null +++ b/vsprojects/vcxproj/test/census_resource_test/census_resource_test.vcxproj.filters @@ -0,0 +1,21 @@ + + + + + test\core\census + + + + + + {313aad4e-d33b-88c5-7d94-e04a4cb0c912} + + + {ff2d74ef-228a-1739-7fa7-7dccbec5b0c5} + + + {4f529bd9-396f-027c-bc49-f9a6bbc97c72} + + + + From 7a97fe2c10e5b55c5f28f7031c2c67fc47581e09 Mon Sep 17 00:00:00 2001 From: Alistair Veitch Date: Fri, 3 Jun 2016 12:10:45 -0700 Subject: [PATCH 0047/1039] use macro rather than constant in array size --- test/core/census/resource_test.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/test/core/census/resource_test.c b/test/core/census/resource_test.c index bfb76bcfb5e..1375d56a6bc 100644 --- a/test/core/census/resource_test.c +++ b/test/core/census/resource_test.c @@ -64,12 +64,12 @@ static void test_empty_definition() { // Given a file name, read raw proto and define the resource included within. // Returns resource id from census_define_resource(). static int32_t define_resource_from_file(const char *file) { - const size_t buf_size = 512; - uint8_t buffer[buf_size]; +#define BUF_SIZE 512 + uint8_t buffer[BUF_SIZE]; FILE *input = fopen(file, "r"); GPR_ASSERT(input != NULL); - size_t nbytes = fread(buffer, 1, buf_size, input); - GPR_ASSERT(nbytes != 0 && nbytes < buf_size); + size_t nbytes = fread(buffer, 1, BUF_SIZE, input); + GPR_ASSERT(nbytes != 0 && nbytes < BUF_SIZE); int32_t rid = census_define_resource(buffer, nbytes); GPR_ASSERT(fclose(input) == 0); return rid; From 0bcbd79baa57c16d4ab64a070b5fbfc93293f543 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 1 Jun 2016 15:43:03 -0700 Subject: [PATCH 0048/1039] Functionality complete in ev_epoll_linux.c --- src/core/lib/iomgr/ev_epoll_linux.c | 1202 +++++++++++++-------------- 1 file changed, 591 insertions(+), 611 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 1201c10a7e0..ce42a9e7ce6 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -61,31 +61,27 @@ struct polling_island; /******************************************************************************* - * FD declarations + * Fd Declarations */ - struct grpc_fd { int fd; /* refst format: bit 0 : 1=Active / 0=Orphaned bits 1-n : refcount - - ref/unref by two to avoid altering the orphaned bit - - To orphan, unref by 1 */ + Ref/Unref by two to avoid altering the orphaned bit */ gpr_atm refst; gpr_mu mu; - int shutdown; + bool shutdown; int closed; - int released; + bool released; grpc_closure *read_closure; grpc_closure *write_closure; - /* Mutex protecting the 'polling_island' field */ + /* The polling island to which this fd belongs to and the mutex protecting the + the field */ gpr_mu pi_mu; - - /* The polling island to which this fd belongs to. - * An fd belongs to exactly one polling island */ struct polling_island *polling_island; struct grpc_fd *freelist_next; @@ -94,11 +90,7 @@ struct grpc_fd { grpc_iomgr_object iomgr_object; }; -/* Return 1 if this fd is orphaned, 0 otherwise */ -static bool fd_is_orphaned(grpc_fd *fd); - /* Reference counting for fds */ -/*#define GRPC_FD_REF_COUNT_DEBUG*/ #ifdef GRPC_FD_REF_COUNT_DEBUG static void fd_ref(grpc_fd *fd, const char *reason, const char *file, int line); static void fd_unref(grpc_fd *fd, const char *reason, const char *file, @@ -119,15 +111,15 @@ static void fd_global_shutdown(void); #define CLOSURE_READY ((grpc_closure *)1) /******************************************************************************* - * Polling Island + * Polling-island Declarations */ typedef struct polling_island { gpr_mu mu; int ref_cnt; - /* Pointer to the polling_island this merged into. If this is not NULL, all - the remaining fields in this pollset (i.e all fields except mu and ref_cnt) - are considered invalid and must be ignored */ + /* Points to the polling_island this merged into. + * If merged_to is not NULL, all the remaining fields (except mu and ref_cnt) + * are invalid and must be ignored */ struct polling_island *merged_to; /* The fd of the underlying epoll set */ @@ -144,15 +136,62 @@ typedef struct polling_island { struct polling_island *next_free; } polling_island; +/******************************************************************************* + * Pollset Declarations + */ + +struct grpc_pollset_worker { + int kicked_specifically; + pthread_t pt_id; /* TODO (sreek) - Add an abstraction here */ + struct grpc_pollset_worker *next; + struct grpc_pollset_worker *prev; +}; + +struct grpc_pollset { + gpr_mu mu; + grpc_pollset_worker root_worker; + bool kicked_without_pollers; + + bool shutting_down; /* Is the pollset shutting down ? */ + bool finish_shutdown_called; /* Is the 'finish_shutdown_locked()' called ? */ + grpc_closure *shutdown_done; /* Called after after shutdown is complete */ + + /* The polling island to which this pollset belongs to and the mutex + protecting the field */ + gpr_mu pi_mu; + struct polling_island *polling_island; +}; + +/******************************************************************************* + * Pollset-set Declarations + */ +struct grpc_pollset_set { + gpr_mu mu; + + size_t pollset_count; + size_t pollset_capacity; + grpc_pollset **pollsets; + + size_t pollset_set_count; + size_t pollset_set_capacity; + struct grpc_pollset_set **pollset_sets; + + size_t fd_count; + size_t fd_capacity; + grpc_fd **fds; +}; + +/******************************************************************************* + * Polling-island Definitions + */ + /* Polling island freelist */ static gpr_mu g_pi_freelist_mu; static polling_island *g_pi_freelist = NULL; -/* TODO: sreek - Should we hold a lock on fd or add a ref to the fd ? - * TODO: sreek - Should this add a ref to the grpc_fd ? */ /* The caller is expected to hold pi->mu lock before calling this function */ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, - size_t fd_count) { + size_t fd_count, bool add_fd_refs) { int err; size_t i; struct epoll_event ev; @@ -162,11 +201,14 @@ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, ev.data.ptr = fds[i]; err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_ADD, fds[i]->fd, &ev); - if (err < 0 && errno != EEXIST) { - gpr_log(GPR_ERROR, "epoll_ctl add for fd: %d failed with error: %s", - fds[i]->fd, strerror(errno)); - /* TODO: sreek - Not sure if it is a good idea to continue here. We need a - * better way to bubble up this error instead of doing an abort() */ + if (err < 0) { + if (errno != EEXIST) { + /* TODO: sreek - We need a better way to bubble up this error instead of + just logging a message */ + gpr_log(GPR_ERROR, "epoll_ctl add for fd: %d failed with error: %s", + fds[i]->fd, strerror(errno)); + } + continue; } @@ -176,26 +218,30 @@ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, } pi->fds[pi->fd_cnt++] = fds[i]; + if (add_fd_refs) { + GRPC_FD_REF(fds[i], "polling_island"); + } } } -/* TODO: sreek - Should we hold a lock on fd or add a ref to the fd ? - * TODO: sreek - Might have to unref the fds (assuming whether we add a ref to - * the fd when adding it to the epollset) */ /* The caller is expected to hold pi->mu lock before calling this function */ -static void polling_island_remove_all_fds_locked(polling_island *pi) { +static void polling_island_remove_all_fds_locked(polling_island *pi, + bool remove_fd_refs) { int err; size_t i; for (i = 0; i < pi->fd_cnt; i++) { - err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_DEL, pi->fds[i]->fd, NULL); + if (remove_fd_refs) { + GRPC_FD_UNREF(pi->fds[i], "polling_island"); + } + err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_DEL, pi->fds[i]->fd, NULL); if (err < 0 && errno != ENOENT) { gpr_log(GPR_ERROR, "epoll_ctl delete for fds[i]: %d failed with error: %s", i, pi->fds[i]->fd, strerror(errno)); - /* TODO: sreek - Not sure if it is a good idea to continue here. We need a - * better way to bubble up this error instead of doing an abort() */ + /* TODO: sreek - We need a better way to bubble up this error instead of + * just logging a message */ continue; } } @@ -203,22 +249,31 @@ static void polling_island_remove_all_fds_locked(polling_island *pi) { pi->fd_cnt = 0; } -/* TODO: sreek - Should we hold a lock on fd or add a ref to the fd ? - * TODO: sreek - Might have to unref the fd (assuming whether we add a ref to - * the fd when adding it to the epollset) */ /* The caller is expected to hold pi->mu lock before calling this function */ -static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd) { +static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd, + bool close_fd, bool remove_fd_ref) { int err; size_t i; - err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_DEL, fd->fd, NULL); - if (err < 0 && errno != ENOENT) { - gpr_log(GPR_ERROR, "epoll_ctl delete for fd: %d failed with error; %s", - fd->fd, strerror(errno)); + + /* Calling close() on the fd will automatically remove it from the epoll set. + If not calling close(), the fd must be explicitly removed from the epoll + set */ + if (close_fd) { + close(fd->fd); + } else { + err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_DEL, fd->fd, NULL); + if (err < 0 && errno != ENOENT) { + gpr_log(GPR_ERROR, "epoll_ctl delete for fd: %d failed with error; %s", + fd->fd, strerror(errno)); + } } for (i = 0; i < pi->fd_cnt; i++) { if (pi->fds[i] == fd) { pi->fds[i] = pi->fds[--pi->fd_cnt]; + if (remove_fd_ref) { + GRPC_FD_UNREF(fd, "polling_island"); + } break; } } @@ -227,6 +282,10 @@ static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd) { static polling_island *polling_island_create(grpc_fd *initial_fd, int initial_ref_cnt) { polling_island *pi = NULL; + struct epoll_event ev; + int err; + + /* Try to get one from the polling island freelist */ gpr_mu_lock(&g_pi_freelist_mu); if (g_pi_freelist != NULL) { pi = g_pi_freelist; @@ -242,13 +301,25 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, pi->fd_cnt = 0; pi->fd_capacity = 0; pi->fds = NULL; + } - pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); - if (pi->epoll_fd < 0) { - gpr_log(GPR_ERROR, "epoll_create1() failed with error: %s", - strerror(errno)); - } - GPR_ASSERT(pi->epoll_fd >= 0); + pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); + if (pi->epoll_fd < 0) { + gpr_log(GPR_ERROR, "epoll_create1() failed with error: %s", + strerror(errno)); + } + GPR_ASSERT(pi->epoll_fd >= 0); + + ev.events = (uint32_t)(EPOLLIN | EPOLLET); + ev.data.ptr = NULL; + err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_ADD, + GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), &ev); + if (err < 0) { + gpr_log(GPR_ERROR, + "Failed to add grpc_global_wake_up_fd (%d) to the epoll set " + "(epoll_fd: %d) with error: %s", + GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), pi->epoll_fd, + strerror(errno)); } pi->ref_cnt = initial_ref_cnt; @@ -256,10 +327,12 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, pi->next_free = NULL; if (initial_fd != NULL) { - /* polling_island_add_fds_locked() expects the caller to hold a pi->mu - * lock. However, since this is a new polling island (and no one has a - * reference to it yet), it is okay to not acquire pi->mu here */ - polling_island_add_fds_locked(pi, &initial_fd, 1); + /* It is not really needed to get the pi->mu lock here. If this is a newly + created polling island (or one that we got from the freelist), no one + else would be holding a lock to it anyway */ + gpr_mu_lock(&pi->mu); + polling_island_add_fds_locked(pi, &initial_fd, 1, true); + gpr_mu_unlock(&pi->mu); } return pi; @@ -269,6 +342,9 @@ static void polling_island_delete(polling_island *pi) { GPR_ASSERT(pi->ref_cnt == 0); GPR_ASSERT(pi->fd_cnt == 0); + close(pi->epoll_fd); + pi->epoll_fd = -1; + pi->merged_to = NULL; gpr_mu_lock(&g_pi_freelist_mu); @@ -313,10 +389,20 @@ void polling_island_pair_update_and_lock(polling_island **p, bool pi_2_locked = false; int num_swaps = 0; + /* Loop until either pi_1 == pi_2 or until we acquired locks on both pi_1 + and pi_2 */ while (pi_1 != pi_2 && !(pi_1_locked && pi_2_locked)) { - // pi_1 is NOT equal to pi_2 - // pi_1 MAY be locked - + /* The following assertions are true at this point: + - pi_1 != pi_2 (else, the while loop would have exited) + - pi_1 MAY be locked + - pi_2 is NOT locked */ + + /* To maintain lock order consistency, always lock polling_island node with + lower address first. + First, make sure pi_1 < pi_2 before proceeding any further. If it turns + out that pi_1 > pi_2, unlock pi_1 if locked (because pi_2 is not locked + at this point and having pi_1 locked would violate the lock order) and + swap pi_1 and pi_2 so that pi_1 becomes less than pi_2 */ if (pi_1 > pi_2) { if (pi_1_locked) { gpr_mu_unlock(&pi_1->mu); @@ -327,14 +413,22 @@ void polling_island_pair_update_and_lock(polling_island **p, num_swaps++; } - // p1 < p2 - // p1 MAY BE locked - // p2 is NOT locked + /* The following assertions are true at this point: + - pi_1 != pi_2 + - pi_1 < pi_2 (address of pi_1 is less than that of pi_2) + - pi_1 MAYBE locked + - pi_2 is NOT locked */ + /* Lock pi_1 (if pi_1 is pointing to the terminal node in the list) */ if (!pi_1_locked) { gpr_mu_lock(&pi_1->mu); pi_1_locked = true; + /* If pi_1 is not terminal node (i.e pi_1->merged_to != NULL), we are not + done locking this polling_island yet. Release the lock on this node and + advance pi_1 to the next node in the list; and go to the beginning of + the loop (we can't proceed to locking pi_2 unless we locked pi_1 first) + */ if (pi_1->merged_to != NULL) { temp = pi_1->merged_to; polling_island_unref_and_unlock(pi_1, 1); @@ -345,13 +439,16 @@ void polling_island_pair_update_and_lock(polling_island **p, } } - // p1 is LOCKED - // p2 is UNLOCKED - // p1 != p2 + /* The following assertions are true at this point: + - pi_1 is locked + - pi_2 is unlocked + - pi_1 != pi_2 */ gpr_mu_lock(&pi_2->mu); pi_2_locked = true; + /* If pi_2 is not terminal node, we are not done locking this polling_island + yet. Release the lock and update pi_2 to the next node in the list */ if (pi_2->merged_to != NULL) { temp = pi_2->merged_to; polling_island_unref_and_unlock(pi_2, 1); @@ -360,14 +457,19 @@ void polling_island_pair_update_and_lock(polling_island **p, } } - // Either pi_1 == pi_2 OR we got both locks! + /* At this point, either pi_1 == pi_2 AND/OR we got both locks */ if (pi_1 == pi_2) { + /* We may or may not have gotten the lock. If we didn't, walk the rest of + the polling_island list and get the lock */ GPR_ASSERT(pi_1_locked || (!pi_1_locked && !pi_2_locked)); if (!pi_1_locked) { pi_1 = pi_2 = polling_island_update_and_lock(pi_1, 2, 0); } } else { GPR_ASSERT(pi_1_locked && pi_2_locked); + /* If we swapped pi_1 and pi_2 odd number of times, do one more swap so that + pi_1 and pi_2 point to the same polling_island lists they started off + with at the beginning of this function (i.e *p and *q respectively) */ if (num_swaps % 2 > 0) { GPR_SWAP(polling_island *, pi_1, pi_2); } @@ -378,26 +480,37 @@ void polling_island_pair_update_and_lock(polling_island **p, } polling_island *polling_island_merge(polling_island *p, polling_island *q) { - polling_island *merged = NULL; - + /* Get locks on both the polling islands */ polling_island_pair_update_and_lock(&p, &q); /* TODO: sreek: Think about this scenario some more. Is it possible ?. what * does it mean, when would this happen */ if (p == q) { - merged = p; + /* Nothing needs to be done here */ + gpr_mu_unlock(&p->mu); + return p; } - // Move all the fds from polling_island p to polling_island q - polling_island_add_fds_locked(q, p->fds, p->fd_cnt); - polling_island_remove_all_fds_locked(p); + /* Make sure that p points to the polling island with fewer fds than q */ + if (p->fd_cnt > q->fd_cnt) { + GPR_SWAP(polling_island *, p, q); + } + + /* "Merge" p with q i.e move all the fds from p (the polling_island with fewer + fds) to q. + Note: Not altering the ref counts on the affected fds here because they + would effectively remain unchanged */ + polling_island_add_fds_locked(q, p->fds, p->fd_cnt, false); + polling_island_remove_all_fds_locked(p, false); + /* The merged polling island inherits all the ref counts of the island merging + with it */ q->ref_cnt += p->ref_cnt; gpr_mu_unlock(&p->mu); gpr_mu_unlock(&q->mu); - return merged; + return q; } static void polling_island_global_init() { @@ -406,95 +519,10 @@ static void polling_island_global_init() { } /******************************************************************************* - * pollset declarations - */ - -struct grpc_pollset_worker { - int kicked_specifically; - pthread_t pt_id; - struct grpc_pollset_worker *next; - struct grpc_pollset_worker *prev; -}; - -struct grpc_pollset { - gpr_mu mu; - grpc_pollset_worker root_worker; - int shutting_down; - int called_shutdown; - int kicked_without_pollers; - grpc_closure *shutdown_done; - - int epoll_fd; - - /* Mutex protecting the 'polling_island' field */ - gpr_mu pi_mu; - - /* The polling island to which this fd belongs to. An fd belongs to exactly - one polling island */ - struct polling_island *polling_island; -}; - -/* Add an fd to a pollset */ -static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - struct grpc_fd *fd); - -static void pollset_set_add_fd(grpc_exec_ctx *exec_ctx, - grpc_pollset_set *pollset_set, grpc_fd *fd); - -/* Convert a timespec to milliseconds: - - very small or negative poll times are clamped to zero to do a - non-blocking poll (which becomes spin polling) - - other small values are rounded up to one millisecond - - longer than a millisecond polls are rounded up to the next nearest - millisecond to avoid spinning - - infinite timeouts are converted to -1 */ -static int poll_deadline_to_millis_timeout(gpr_timespec deadline, - gpr_timespec now); - -/* Allow kick to wakeup the currently polling worker */ -#define GRPC_POLLSET_CAN_KICK_SELF 1 -/* As per pollset_kick, with an extended set of flags (defined above) - -- mostly for fd_posix's use. */ -static void pollset_kick_ext(grpc_pollset *p, - grpc_pollset_worker *specific_worker, - uint32_t flags); - -/* turn a pollset into a multipoller: platform specific */ -typedef void (*platform_become_multipoller_type)(grpc_exec_ctx *exec_ctx, - grpc_pollset *pollset, - struct grpc_fd **fds, - size_t fd_count); - -/* Return 1 if the pollset has active threads in pollset_work (pollset must - * be locked) */ -static int pollset_has_workers(grpc_pollset *pollset); - -/******************************************************************************* - * pollset_set definitions - */ - -struct grpc_pollset_set { - gpr_mu mu; - - size_t pollset_count; - size_t pollset_capacity; - grpc_pollset **pollsets; - - size_t pollset_set_count; - size_t pollset_set_capacity; - struct grpc_pollset_set **pollset_sets; - - size_t fd_count; - size_t fd_capacity; - grpc_fd **fds; -}; - -/******************************************************************************* - * fd_posix.c + * Fd Definitions */ -/* We need to keep a freelist not because of any concerns of malloc - * performance +/* We need to keep a freelist not because of any concerns of malloc performance * but instead so that implementations with multiple threads in (for example) * epoll_wait deal with the race between pollset removal and incoming poll * notifications. @@ -506,58 +534,16 @@ struct grpc_pollset_set { * If we keep the object freelisted, in the worst case losing this race just * becomes a spurious read notification on a reused fd. */ -/* TODO(klempner): We could use some form of polling generation count to know - * when these are safe to free. */ -/* TODO(klempner): Consider disabling freelisting if we don't have multiple - * threads in poll on the same fd */ -/* TODO(klempner): Batch these allocations to reduce fragmentation */ -static grpc_fd *fd_freelist = NULL; -static gpr_mu fd_freelist_mu; - -static void freelist_fd(grpc_fd *fd) { - gpr_mu_lock(&fd_freelist_mu); - fd->freelist_next = fd_freelist; - fd_freelist = fd; - grpc_iomgr_unregister_object(&fd->iomgr_object); - gpr_mu_unlock(&fd_freelist_mu); -} - -static grpc_fd *alloc_fd(int fd) { - grpc_fd *r = NULL; - - gpr_mu_lock(&fd_freelist_mu); - if (fd_freelist != NULL) { - r = fd_freelist; - fd_freelist = fd_freelist->freelist_next; - } - gpr_mu_unlock(&fd_freelist_mu); - - if (r == NULL) { - r = gpr_malloc(sizeof(grpc_fd)); - gpr_mu_init(&r->mu); - gpr_mu_init(&r->pi_mu); - } - /* TODO: sreek - check with ctiller on why we need to acquire a lock here */ - gpr_mu_lock(&r->mu); - gpr_atm_rel_store(&r->refst, 1); - r->shutdown = 0; - r->read_closure = CLOSURE_NOT_READY; - r->write_closure = CLOSURE_NOT_READY; - r->fd = fd; - r->polling_island = NULL; - r->freelist_next = NULL; - r->on_done_closure = NULL; - r->closed = 0; - r->released = 0; - gpr_mu_unlock(&r->mu); - return r; -} +/* The alarm system needs to be able to wakeup 'some poller' sometimes + * (specifically when a new alarm needs to be triggered earlier than the next + * alarm 'epoch'). This wakeup_fd gives us something to alert on when such a + * case occurs. */ +/* TODO: sreek: Right now, this wakes up all pollers */ +grpc_wakeup_fd grpc_global_wakeup_fd; -static void destroy(grpc_fd *fd) { - gpr_mu_destroy(&fd->mu); - gpr_free(fd); -} +static grpc_fd *fd_freelist = NULL; +static gpr_mu fd_freelist_mu; #ifdef GRPC_FD_REF_COUNT_DEBUG #define REF_BY(fd, n, reason) ref_by(fd, n, reason, __FILE__, __LINE__) @@ -588,12 +574,33 @@ static void unref_by(grpc_fd *fd, int n) { #endif old = gpr_atm_full_fetch_add(&fd->refst, -n); if (old == n) { - freelist_fd(fd); + /* Add the fd to the freelist */ + gpr_mu_lock(&fd_freelist_mu); + fd->freelist_next = fd_freelist; + fd_freelist = fd; + grpc_iomgr_unregister_object(&fd->iomgr_object); + gpr_mu_unlock(&fd_freelist_mu); } else { GPR_ASSERT(old > n); } } +/* Increment refcount by two to avoid changing the orphan bit */ +#ifdef GRPC_FD_REF_COUNT_DEBUG +static void fd_ref(grpc_fd *fd, const char *reason, const char *file, + int line) { + ref_by(fd, 2, reason, file, line); +} + +static void fd_unref(grpc_fd *fd, const char *reason, const char *file, + int line) { + unref_by(fd, 2, reason, file, line); +} +#else +static void fd_ref(grpc_fd *fd) { ref_by(fd, 2); } +static void fd_unref(grpc_fd *fd) { unref_by(fd, 2); } +#endif + static void fd_global_init(void) { gpr_mu_init(&fd_freelist_mu); } static void fd_global_shutdown(void) { @@ -602,91 +609,111 @@ static void fd_global_shutdown(void) { while (fd_freelist != NULL) { grpc_fd *fd = fd_freelist; fd_freelist = fd_freelist->freelist_next; - destroy(fd); + gpr_mu_destroy(&fd->mu); + gpr_free(fd); } gpr_mu_destroy(&fd_freelist_mu); } static grpc_fd *fd_create(int fd, const char *name) { - grpc_fd *r = alloc_fd(fd); + grpc_fd *new_fd = NULL; + + gpr_mu_lock(&fd_freelist_mu); + if (fd_freelist != NULL) { + new_fd = fd_freelist; + fd_freelist = fd_freelist->freelist_next; + } + gpr_mu_unlock(&fd_freelist_mu); + + if (new_fd == NULL) { + new_fd = gpr_malloc(sizeof(grpc_fd)); + gpr_mu_init(&new_fd->mu); + gpr_mu_init(&new_fd->pi_mu); + } - char *name2; - gpr_asprintf(&name2, "%s fd=%d", name, fd); - grpc_iomgr_register_object(&r->iomgr_object, name2); - gpr_free(name2); + /* Note: It is not really needed to get the new_fd->mu lock here. If this is a + newly created fd (or an fd we got from the freelist), no one else would be + holding a lock to it anyway. */ + gpr_mu_lock(&new_fd->mu); + + gpr_atm_rel_store(&new_fd->refst, 1); + new_fd->shutdown = false; + new_fd->read_closure = CLOSURE_NOT_READY; + new_fd->write_closure = CLOSURE_NOT_READY; + new_fd->fd = fd; + new_fd->polling_island = NULL; + new_fd->freelist_next = NULL; + new_fd->on_done_closure = NULL; + new_fd->closed = 0; + new_fd->released = false; + + gpr_mu_unlock(&new_fd->mu); + + char *fd_name; + gpr_asprintf(&fd_name, "%s fd=%d", name, fd); + grpc_iomgr_register_object(&new_fd->iomgr_object, fd_name); + gpr_free(fd_name); #ifdef GRPC_FD_REF_COUNT_DEBUG - gpr_log(GPR_DEBUG, "FD %d %p create %s", fd, r, name); + gpr_log(GPR_DEBUG, "FD %d %p create %s", fd, r, fd_name); #endif - return r; + return new_fd; } static bool fd_is_orphaned(grpc_fd *fd) { return (gpr_atm_acq_load(&fd->refst) & 1) == 0; } -static void close_fd_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { - fd->closed = 1; - if (!fd->released) { - close(fd->fd); - } else { - /* TODO: sreek - Check for deadlocks */ - - gpr_mu_lock(&fd->pi_mu); - fd->polling_island = - polling_island_update_and_lock(fd->polling_island, 1, 0); - - polling_island_remove_fd_locked(fd->polling_island, fd); - polling_island_unref_and_unlock(fd->polling_island, 1); - - fd->polling_island = NULL; - gpr_mu_unlock(&fd->pi_mu); - } - - grpc_exec_ctx_enqueue(exec_ctx, fd->on_done_closure, true, NULL); -} - static int fd_wrapped_fd(grpc_fd *fd) { - if (fd->released || fd->closed) { - return -1; - } else { - return fd->fd; + int ret_fd = -1; + gpr_mu_lock(&fd->mu); + if (!fd->released && !fd->closed) { + ret_fd = fd->fd; } + gpr_mu_unlock(&fd->mu); + + return ret_fd; } -/* TODO: sreek - do something here with the pollset island link */ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure *on_done, int *release_fd, const char *reason) { + /* TODO(sreek) In ev_poll_posix.c,the lock is acquired a little later. Why? */ + gpr_mu_lock(&fd->mu); fd->on_done_closure = on_done; + + /* If release_fd is not NULL, we should be relinquishing control of the file + descriptor fd->fd (but we still own the grpc_fd structure). */ fd->released = release_fd != NULL; if (!fd->released) { shutdown(fd->fd, SHUT_RDWR); } else { *release_fd = fd->fd; } - gpr_mu_lock(&fd->mu); - REF_BY(fd, 1, reason); /* remove active status, but keep referenced */ - close_fd_locked(exec_ctx, fd); - gpr_mu_unlock(&fd->mu); - UNREF_BY(fd, 2, reason); /* drop the reference */ -} -/* increment refcount by two to avoid changing the orphan bit */ -#ifdef GRPC_FD_REF_COUNT_DEBUG -static void fd_ref(grpc_fd *fd, const char *reason, const char *file, - int line) { - ref_by(fd, 2, reason, file, line); -} + REF_BY(fd, 1, reason); /* Remove active status, but keep referenced */ + fd->closed = 1; -static void fd_unref(grpc_fd *fd, const char *reason, const char *file, - int line) { - unref_by(fd, 2, reason, file, line); -} -#else -static void fd_ref(grpc_fd *fd) { ref_by(fd, 2); } + /* Remove the fd from the polling island: + - Update the fd->polling_island to point to the latest polling island + - Remove the fd from the polling island. Also, call close() on the file + descriptor fd->fd ONLY if we haven't relinquised control (i.e + fd->released is 'false') + - Decrement the ref count on the polling island and det fd->polling_island + to NULL */ + gpr_mu_lock(&fd->pi_mu); -static void fd_unref(grpc_fd *fd) { unref_by(fd, 2); } -#endif + fd->polling_island = polling_island_update_and_lock(fd->polling_island, 1, 0); + polling_island_remove_fd_locked(fd->polling_island, fd, !fd->released, true); + polling_island_unref_and_unlock(fd->polling_island, 1); + fd->polling_island = NULL; + + gpr_mu_unlock(&fd->pi_mu); + + grpc_exec_ctx_enqueue(exec_ctx, fd->on_done_closure, true, NULL); + + gpr_mu_unlock(&fd->mu); + UNREF_BY(fd, 2, reason); /* Drop the reference */ +} static void notify_on_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure **st, grpc_closure *closure) { @@ -724,11 +751,13 @@ static int set_ready_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, } } -/* Do something here with the pollset island link (?) */ static void fd_shutdown(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { gpr_mu_lock(&fd->mu); GPR_ASSERT(!fd->shutdown); - fd->shutdown = 1; + fd->shutdown = true; + + /* Flush any pending read and write closures. Since fd->shutdown is 'true' at + this point, the closures would be called with 'success = false' */ set_ready_locked(exec_ctx, fd, &fd->read_closure); set_ready_locked(exec_ctx, fd, &fd->write_closure); gpr_mu_unlock(&fd->mu); @@ -749,27 +778,39 @@ static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, } /******************************************************************************* - * pollset_posix.c + * Pollset Definitions */ -GPR_TLS_DECL(g_current_thread_poller); -GPR_TLS_DECL(g_current_thread_worker); +static void sig_handler(int sig_num) { + /* TODO: sreek - Remove this expensive log line */ + gpr_log(GPR_INFO, "Received signal %d", sig_num); +} -/** The alarm system needs to be able to wakeup 'some poller' sometimes - * (specifically when a new alarm needs to be triggered earlier than the next - * alarm 'epoch'). - * This wakeup_fd gives us something to alert on when such a case occurs. */ -grpc_wakeup_fd grpc_global_wakeup_fd; +/* Global state management */ +static void pollset_global_init(void) { + gpr_tls_init(&g_current_thread_poller); + gpr_tls_init(&g_current_thread_worker); + grpc_wakeup_fd_init(&grpc_global_wakeup_fd); + signal(SIGUSR1, sig_handler); +} -static void remove_worker(grpc_pollset *p, grpc_pollset_worker *worker) { - worker->prev->next = worker->next; - worker->next->prev = worker->prev; +static void pollset_global_shutdown(void) { + grpc_wakeup_fd_destroy(&grpc_global_wakeup_fd); + gpr_tls_destroy(&g_current_thread_poller); + gpr_tls_destroy(&g_current_thread_worker); } +/* Return 1 if the pollset has active threads in pollset_work (pollset must + * be locked) */ static int pollset_has_workers(grpc_pollset *p) { return p->root_worker.next != &p->root_worker; } +static void remove_worker(grpc_pollset *p, grpc_pollset_worker *worker) { + worker->prev->next = worker->next; + worker->next->prev = worker->prev; +} + static grpc_pollset_worker *pop_front_worker(grpc_pollset *p) { if (pollset_has_workers(p)) { grpc_pollset_worker *w = p->root_worker.next; @@ -792,241 +833,69 @@ static void push_front_worker(grpc_pollset *p, grpc_pollset_worker *worker) { worker->prev->next = worker->next->prev = worker; } -static void pollset_kick_ext(grpc_pollset *p, - grpc_pollset_worker *specific_worker, - uint32_t flags) { - GPR_TIMER_BEGIN("pollset_kick_ext", 0); - - /* pollset->mu already held */ - if (specific_worker != NULL) { - if (specific_worker == GRPC_POLLSET_KICK_BROADCAST) { - GPR_TIMER_BEGIN("pollset_kick_ext.broadcast", 0); - for (specific_worker = p->root_worker.next; - specific_worker != &p->root_worker; - specific_worker = specific_worker->next) { - pthread_kill(specific_worker->pt_id, SIGUSR1); +/* p->mu must be held before calling this function */ +static void pollset_kick(grpc_pollset *p, + grpc_pollset_worker *specific_worker) { + GPR_TIMER_BEGIN("pollset_kick", 0); + + grpc_pollset_worker *worker = specific_worker; + if (worker != NULL) { + if (worker == GRPC_POLLSET_KICK_BROADCAST) { + GPR_TIMER_BEGIN("pollset_kick.broadcast", 0); + if (pollset_has_workers(p)) { + for (worker = p->root_worker.next; worker != &p->root_worker; + worker = worker->next) { + pthread_kill(worker->pt_id, SIGUSR1); + } + } else { + p->kicked_without_pollers = true; } - p->kicked_without_pollers = 1; - GPR_TIMER_END("pollset_kick_ext.broadcast", 0); - } else if (gpr_tls_get(&g_current_thread_worker) != - (intptr_t)specific_worker) { - GPR_TIMER_MARK("different_thread_worker", 0); - specific_worker->kicked_specifically = 1; - /* TODO (sreek): Refactor this into a separate file*/ - pthread_kill(specific_worker->pt_id, SIGUSR1); - } else if ((flags & GRPC_POLLSET_CAN_KICK_SELF) != 0) { - GPR_TIMER_MARK("kick_yoself", 0); - specific_worker->kicked_specifically = 1; - pthread_kill(specific_worker->pt_id, SIGUSR1); + GPR_TIMER_END("pollset_kick.broadcast", 0); + } else { + GPR_TIMER_MARK("kicked_specifically", 0); + worker->kicked_specifically = true; + pthread_kill(worker->pt_id, SIGUSR1); } - } else if (gpr_tls_get(&g_current_thread_poller) != (intptr_t)p) { + } else { GPR_TIMER_MARK("kick_anonymous", 0); - specific_worker = pop_front_worker(p); - if (specific_worker != NULL) { - if (gpr_tls_get(&g_current_thread_worker) == (intptr_t)specific_worker) { - GPR_TIMER_MARK("kick_anonymous_not_self", 0); - push_back_worker(p, specific_worker); - specific_worker = pop_front_worker(p); - if ((flags & GRPC_POLLSET_CAN_KICK_SELF) == 0 && - gpr_tls_get(&g_current_thread_worker) == - (intptr_t)specific_worker) { - push_back_worker(p, specific_worker); - specific_worker = NULL; - } - } - if (specific_worker != NULL) { - GPR_TIMER_MARK("finally_kick", 0); - push_back_worker(p, specific_worker); - pthread_kill(specific_worker->pt_id, SIGUSR1); - } + worker = pop_front_worker(p); + if (worker != NULL) { + GPR_TIMER_MARK("finally_kick", 0); + push_back_worker(p, worker); + pthread_kill(worker->pt_id, SIGUSR1); } else { GPR_TIMER_MARK("kicked_no_pollers", 0); - p->kicked_without_pollers = 1; + p->kicked_without_pollers = true; } } - GPR_TIMER_END("pollset_kick_ext", 0); + GPR_TIMER_END("pollset_kick", 0); } -static void pollset_kick(grpc_pollset *p, - grpc_pollset_worker *specific_worker) { - pollset_kick_ext(p, specific_worker, 0); -} +static void kick_poller(void) { grpc_wakeup_fd_wakeup(&grpc_global_wakeup_fd); } -/* global state management */ +static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { + gpr_mu_init(&pollset->mu); + *mu = &pollset->mu; -static void sig_handler(int sig_num) { - gpr_log(GPR_INFO, "Received signal %d", sig_num); -} + pollset->root_worker.next = pollset->root_worker.prev = &pollset->root_worker; + pollset->kicked_without_pollers = false; -static void pollset_global_init(void) { - gpr_tls_init(&g_current_thread_poller); - gpr_tls_init(&g_current_thread_worker); - grpc_wakeup_fd_init(&grpc_global_wakeup_fd); - signal(SIGUSR1, sig_handler); -} + pollset->shutting_down = false; + pollset->finish_shutdown_called = false; + pollset->shutdown_done = NULL; -static void pollset_global_shutdown(void) { - grpc_wakeup_fd_destroy(&grpc_global_wakeup_fd); - gpr_tls_destroy(&g_current_thread_poller); - gpr_tls_destroy(&g_current_thread_worker); -} - -static void kick_poller(void) { grpc_wakeup_fd_wakeup(&grpc_global_wakeup_fd); } - -/* TODO: sreek. Try to Remove this forward declaration*/ -static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset); - -/* main interface */ - -static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { - gpr_mu_init(&pollset->mu); - *mu = &pollset->mu; - pollset->root_worker.next = pollset->root_worker.prev = &pollset->root_worker; gpr_mu_init(&pollset->pi_mu); pollset->polling_island = NULL; - pollset->shutting_down = 0; - pollset->called_shutdown = 0; - pollset->kicked_without_pollers = 0; - - multipoll_with_epoll_pollset_create_efd(pollset); -} - -/* TODO(sreek): Maybe merge multipoll_*_destroy() with pollset_destroy() - * function */ -static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset); - -static void pollset_destroy(grpc_pollset *pollset) { - GPR_ASSERT(!pollset_has_workers(pollset)); - - multipoll_with_epoll_pollset_destroy(pollset); - - gpr_mu_destroy(&pollset->pi_mu); - gpr_mu_destroy(&pollset->mu); -} - -/* TODO(sreek) - Do something with the pollset island link (??) */ -static void pollset_reset(grpc_pollset *pollset) { - GPR_ASSERT(pollset->shutting_down); - GPR_ASSERT(!pollset_has_workers(pollset)); - pollset->shutting_down = 0; - pollset->called_shutdown = 0; - pollset->kicked_without_pollers = 0; -} - -/* TODO (sreek): Remove multipoll_with_epoll_finish_shutdown() declaration */ -static void multipoll_with_epoll_pollset_finish_shutdown(grpc_pollset *pollset); - -static void finish_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset) { - multipoll_with_epoll_pollset_finish_shutdown(pollset); - grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); -} - -/* TODO(sreek): Remove multipoll_with_epoll_*_maybe_work_and_unlock - * declaration - */ -static void multipoll_with_epoll_pollset_maybe_work_and_unlock( - grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker *worker, - gpr_timespec deadline, gpr_timespec now); - -static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_pollset_worker **worker_hdl, gpr_timespec now, - gpr_timespec deadline) { - grpc_pollset_worker worker; - *worker_hdl = &worker; - - /* pollset->mu already held */ - int added_worker = 0; - int locked = 1; - int queued_work = 0; - int keep_polling = 0; - GPR_TIMER_BEGIN("pollset_work", 0); - /* this must happen before we (potentially) drop pollset->mu */ - worker.next = worker.prev = NULL; - worker.kicked_specifically = 0; - - /* TODO(sreek): Abstract this thread id stuff out into a separate file */ - worker.pt_id = pthread_self(); - /* If we're shutting down then we don't execute any extended work */ - if (pollset->shutting_down) { - GPR_TIMER_MARK("pollset_work.shutting_down", 0); - goto done; - } - /* Start polling, and keep doing so while we're being asked to - re-evaluate our pollers (this allows poll() based pollers to - ensure they don't miss wakeups) */ - keep_polling = 1; - while (keep_polling) { - keep_polling = 0; - if (!pollset->kicked_without_pollers) { - if (!added_worker) { - push_front_worker(pollset, &worker); - added_worker = 1; - gpr_tls_set(&g_current_thread_worker, (intptr_t)&worker); - } - gpr_tls_set(&g_current_thread_poller, (intptr_t)pollset); - GPR_TIMER_BEGIN("maybe_work_and_unlock", 0); - - multipoll_with_epoll_pollset_maybe_work_and_unlock( - exec_ctx, pollset, &worker, deadline, now); - - GPR_TIMER_END("maybe_work_and_unlock", 0); - locked = 0; - gpr_tls_set(&g_current_thread_poller, 0); - } else { - GPR_TIMER_MARK("pollset_work.kicked_without_pollers", 0); - pollset->kicked_without_pollers = 0; - } - /* Finished execution - start cleaning up. - Note that we may arrive here from outside the enclosing while() loop. - In that case we won't loop though as we haven't added worker to the - worker list, which means nobody could ask us to re-evaluate polling). */ - done: - if (!locked) { - queued_work |= grpc_exec_ctx_flush(exec_ctx); - gpr_mu_lock(&pollset->mu); - locked = 1; - } - } - if (added_worker) { - remove_worker(pollset, &worker); - gpr_tls_set(&g_current_thread_worker, 0); - } - - /* check shutdown conditions */ - if (pollset->shutting_down) { - if (pollset_has_workers(pollset)) { - pollset_kick(pollset, NULL); - } else if (!pollset->called_shutdown) { - pollset->called_shutdown = 1; - gpr_mu_unlock(&pollset->mu); - finish_shutdown(exec_ctx, pollset); - grpc_exec_ctx_flush(exec_ctx); - /* Continuing to access pollset here is safe -- it is the caller's - * responsibility to not destroy when it has outstanding calls to - * pollset_work. - * TODO(dklempner): Can we refactor the shutdown logic to avoid this? */ - gpr_mu_lock(&pollset->mu); - } - } - *worker_hdl = NULL; - GPR_TIMER_END("pollset_work", 0); -} - -/* TODO: (sreek) Do something with the pollset island link */ -static void pollset_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_closure *closure) { - GPR_ASSERT(!pollset->shutting_down); - pollset->shutting_down = 1; - pollset->shutdown_done = closure; - pollset_kick(pollset, GRPC_POLLSET_KICK_BROADCAST); - - if (!pollset->called_shutdown && !pollset_has_workers(pollset)) { - pollset->called_shutdown = 1; - finish_shutdown(exec_ctx, pollset); - } } +/* Convert a timespec to milliseconds: + - Very small or negative poll times are clamped to zero to do a non-blocking + poll (which becomes spin polling) + - Other small values are rounded up to one millisecond + - Longer than a millisecond polls are rounded up to the next nearest + millisecond to avoid spinning + - Infinite timeouts are converted to -1 */ static int poll_deadline_to_millis_timeout(gpr_timespec deadline, gpr_timespec now) { gpr_timespec timeout; @@ -1034,6 +903,7 @@ static int poll_deadline_to_millis_timeout(gpr_timespec deadline, if (gpr_time_cmp(deadline, gpr_inf_future(deadline.clock_type)) == 0) { return -1; } + if (gpr_time_cmp(deadline, gpr_time_add(now, gpr_time_from_micros( max_spin_polling_us, GPR_TIMESPAN))) <= 0) { @@ -1044,10 +914,6 @@ static int poll_deadline_to_millis_timeout(gpr_timespec deadline, timeout, gpr_time_from_nanos(GPR_NS_PER_MS - 1, GPR_TIMESPAN))); } -/******************************************************************************* - * pollset_multipoller_with_epoll_posix.c - */ - static void set_ready(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure **st) { /* only one set_ready can be active at once (but there may be a racing notify_on) */ @@ -1064,94 +930,46 @@ static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { set_ready(exec_ctx, fd, &fd->write_closure); } -/* TODO: sreek - This function multipoll_with_epoll_pollset_add_fd() and - * finally_add_fd() in ev_poll_and_epoll_posix.c */ -static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_fd *fd) { - /* TODO sreek - Check if we need to get a pollset->mu lock here */ - gpr_mu_lock(&pollset->pi_mu); - gpr_mu_lock(&fd->pi_mu); - - polling_island *pi_new = NULL; - - if (fd->polling_island == pollset->polling_island) { - pi_new = fd->polling_island; - if (pi_new == NULL) { - pi_new = polling_island_create(fd, 2); - } - } else if (fd->polling_island == NULL) { - pi_new = polling_island_update_and_lock(pollset->polling_island, 1, 1); - } else if (pollset->polling_island == NULL) { - pi_new = polling_island_update_and_lock(fd->polling_island, 1, 1); - } else { - pi_new = polling_island_merge(fd->polling_island, pollset->polling_island); - } - - fd->polling_island = pollset->polling_island = pi_new; - - gpr_mu_unlock(&fd->pi_mu); - gpr_mu_unlock(&pollset->pi_mu); -} - -/* Creates an epoll fd and initializes the pollset */ -/* TODO: This has to be called ONLY from pollset_init function. and hence it - * does not acquire any lock */ -static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset) { - struct epoll_event ev; - int err; - - pollset->epoll_fd = epoll_create1(EPOLL_CLOEXEC); - if (pollset->epoll_fd < 0) { - gpr_log(GPR_ERROR, "epoll_create1 failed: %s", strerror(errno)); - abort(); - } - - ev.events = (uint32_t)(EPOLLIN | EPOLLET); - ev.data.ptr = NULL; - - err = epoll_ctl(pollset->epoll_fd, EPOLL_CTL_ADD, - GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), &ev); - if (err < 0) { - gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", - GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), - strerror(errno)); - } -} - /* TODO(klempner): We probably want to turn this down a bit */ #define GRPC_EPOLL_MAX_EVENTS 1000 - -static void multipoll_with_epoll_pollset_maybe_work_and_unlock( - grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker *worker, - gpr_timespec deadline, gpr_timespec now) { +static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset, int timeout_ms, + sigset_t *sig_mask) { struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; - int epoll_fd = pollset->epoll_fd; + int epoll_fd; int ep_rv; - int timeout_ms; + GPR_TIMER_BEGIN("pollset_work_and_unlock", 0); + + /* We need to get the epoll_fd to wait on. The epoll_fd is in inside the + polling island pointed by pollset->polling_island. + Acquire the following locks: + - pollset->mu (which we already have) + - pollset->pi_mu + - pollset->polling_island->mu */ + gpr_mu_lock(&pollset->pi_mu); + pollset->polling_island = + polling_island_update_and_lock(pollset->polling_island, 1, 0); - /* If you want to ignore epoll's ability to sanely handle parallel pollers, - * for a more apples-to-apples performance comparison with poll, add a - * if (pollset->counter != 0) { return 0; } - * here. - */ + epoll_fd = pollset->polling_island->epoll_fd; + /* Release the locks */ + polling_island_unref_and_unlock(pollset->polling_island, 0); /* Keep the ref*/ + gpr_mu_unlock(&pollset->pi_mu); gpr_mu_unlock(&pollset->mu); - timeout_ms = poll_deadline_to_millis_timeout(deadline, now); - do { - /* The following epoll_wait never blocks; it has a timeout of 0 */ - ep_rv = epoll_wait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, timeout_ms); + ep_rv = epoll_pwait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, timeout_ms, + sig_mask); + if (ep_rv < 0) { if (errno != EINTR) { - gpr_log(GPR_ERROR, "epoll_wait() failed: %s", strerror(errno)); + /* TODO (sreek) - Check for bad file descriptor error */ + gpr_log(GPR_ERROR, "epoll_pwait() failed: %s", strerror(errno)); } } else { int i; for (i = 0; i < ep_rv; ++i) { grpc_fd *fd = ep_ev[i].data.ptr; - /* TODO(klempner): We might want to consider making err and pri - * separate events */ int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); int write_ev = ep_ev[i].events & EPOLLOUT; @@ -1168,17 +986,179 @@ static void multipoll_with_epoll_pollset_maybe_work_and_unlock( } } } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); + + GPR_TIMER_END("pollset_work_and_unlock", 0); +} + +/* Release the reference to pollset->polling_island and set it to NULL. + pollset->mu must be held */ +static void pollset_release_polling_island_locked(grpc_pollset *pollset) { + gpr_mu_lock(&pollset->pi_mu); + if (pollset->polling_island) { + pollset->polling_island = + polling_island_update_and_lock(pollset->polling_island, 1, 0); + polling_island_unref_and_unlock(pollset->polling_island, 1); + pollset->polling_island = NULL; + } + gpr_mu_unlock(&pollset->pi_mu); +} + +static void finish_shutdown_locked(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset) { + /* The pollset cannot have any workers if we are at this stage */ + GPR_ASSERT(!pollset_has_workers(pollset)); + + pollset->finish_shutdown_called = true; + pollset_release_polling_island_locked(pollset); + + grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); +} + +/* pollset->mu lock must be held by the caller before calling this */ +static void pollset_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_closure *closure) { + GPR_TIMER_BEGIN("pollset_shutdown", 0); + GPR_ASSERT(!pollset->shutting_down); + pollset->shutting_down = true; + pollset->shutdown_done = closure; + pollset_kick(pollset, GRPC_POLLSET_KICK_BROADCAST); + + /* If the pollset has any workers, we cannot call finish_shutdown_locked() + because it would release the underlying polling island. In such a case, we + let the last worker call finish_shutdown_locked() from pollset_work() */ + if (!pollset_has_workers(pollset)) { + GPR_ASSERT(!pollset->finish_shutdown_called); + GPR_TIMER_MARK("pollset_shutdown.finish_shutdown_locked", 0); + finish_shutdown_locked(exec_ctx, pollset); + } + GPR_TIMER_END("pollset_shutdown", 0); } -static void multipoll_with_epoll_pollset_finish_shutdown( - grpc_pollset *pollset) {} +/* TODO(sreek) Is pollset_shutdown() guranteed to be called before this? */ +static void pollset_destroy(grpc_pollset *pollset) { + GPR_ASSERT(!pollset_has_workers(pollset)); + gpr_mu_destroy(&pollset->pi_mu); + gpr_mu_destroy(&pollset->mu); +} -static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset) { - close(pollset->epoll_fd); +static void pollset_reset(grpc_pollset *pollset) { + GPR_ASSERT(pollset->shutting_down); + GPR_ASSERT(!pollset_has_workers(pollset)); + pollset->shutting_down = false; + pollset->finish_shutdown_called = false; + pollset->kicked_without_pollers = false; + /* TODO(sreek) - Should pollset->shutdown closure be set to NULL here? */ + pollset_release_polling_island_locked(pollset); +} + +/* pollset->mu lock must be held by the caller before calling this. + The function pollset_work() may temporarily release the lock (pollset->mu) + during the course of its execution but it will always re-acquire the lock and + ensure that it is held by the time the function returns */ +static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_pollset_worker **worker_hdl, gpr_timespec now, + gpr_timespec deadline) { + GPR_TIMER_BEGIN("pollset_work", 0); + + int timeout_ms = poll_deadline_to_millis_timeout(deadline, now); + + sigset_t new_mask; + sigset_t orig_mask; + + grpc_pollset_worker worker; + worker.next = worker.prev = NULL; + worker.kicked_specifically = 0; + worker.pt_id = pthread_self(); + + *worker_hdl = &worker; + + if (pollset->kicked_without_pollers) { + /* If the pollset was kicked without pollers, pretend that the current + worker got the kick and skip polling. A kick indicates that there is some + work that needs attention like an event on the completion queue or an + alarm */ + GPR_TIMER_MARK("pollset_work.kicked_without_pollers", 0); + pollset->kicked_without_pollers = 0; + } else if (!pollset->shutting_down) { + sigemptyset(&new_mask); + sigaddset(&new_mask, SIGUSR1); + pthread_sigmask(SIG_BLOCK, &new_mask, &orig_mask); + sigdelset(&orig_mask, SIGUSR1); + + push_front_worker(pollset, &worker); + + pollset_work_and_unlock(exec_ctx, pollset, timeout_ms, &orig_mask); + grpc_exec_ctx_flush(exec_ctx); + + gpr_mu_lock(&pollset->mu); + remove_worker(pollset, &worker); + } + + /* If we are the last worker on the pollset (i.e pollset_has_workers() is + false at this point) and the pollset is shutting down, we may have to + finish the shutdown process by calling finish_shutdown_locked(). + See pollset_shutdown() for more details. + + Note: Continuing to access pollset here is safe; it is the caller's + responsibility to not destroy a pollset when it has outstanding calls to + pollset_work() */ + if (pollset->shutting_down && !pollset_has_workers(pollset) && + !pollset->finish_shutdown_called) { + GPR_TIMER_MARK("pollset_work.finish_shutdown_locked", 0); + finish_shutdown_locked(exec_ctx, pollset); + + gpr_mu_unlock(&pollset->mu); + grpc_exec_ctx_flush(exec_ctx); + gpr_mu_lock(&pollset->mu); + } + + *worker_hdl = NULL; + GPR_TIMER_END("pollset_work", 0); +} + +static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_fd *fd) { + /* TODO sreek - Check if we need to get a pollset->mu lock here */ + gpr_mu_lock(&pollset->pi_mu); + gpr_mu_lock(&fd->pi_mu); + + polling_island *pi_new = NULL; + + /* 1) If fd->polling_island and pollset->polling_island are both non-NULL and + * equal, do nothing. + * 2) If fd->polling_island and pollset->polling_island are both NULL, create + * a new polling island (with a refcount of 2) and make the polling_island + * fields in both fd and pollset to point to the new island + * 3) If one of fd->polling_island or pollset->polling_island is NULL, update + * the NULL polling_island field to point to the non-NULL polling_island + * field (ensure that the refcount on the polling island is incremented by + * 1 to account for the newly added reference) + * 4) Finally, if fd->polling_island and pollset->polling_island are non-NULL + * and different, merge both the polling islands and update the + * polling_island fields in both fd and pollset to point to the merged + * polling island. + */ + if (fd->polling_island == pollset->polling_island) { + pi_new = fd->polling_island; + if (pi_new == NULL) { + pi_new = polling_island_create(fd, 2); + } + } else if (fd->polling_island == NULL) { + pi_new = polling_island_update_and_lock(pollset->polling_island, 1, 1); + } else if (pollset->polling_island == NULL) { + pi_new = polling_island_update_and_lock(fd->polling_island, 1, 1); + } else { + pi_new = polling_island_merge(fd->polling_island, pollset->polling_island); + } + + fd->polling_island = pollset->polling_island = pi_new; + + gpr_mu_unlock(&fd->pi_mu); + gpr_mu_unlock(&pollset->pi_mu); } /******************************************************************************* - * pollset_set_posix.c + * Pollset-set Definitions */ static grpc_pollset_set *pollset_set_create(void) { @@ -1200,6 +1180,45 @@ static void pollset_set_destroy(grpc_pollset_set *pollset_set) { gpr_free(pollset_set); } +static void pollset_set_add_fd(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, grpc_fd *fd) { + size_t i; + gpr_mu_lock(&pollset_set->mu); + if (pollset_set->fd_count == pollset_set->fd_capacity) { + pollset_set->fd_capacity = GPR_MAX(8, 2 * pollset_set->fd_capacity); + pollset_set->fds = gpr_realloc( + pollset_set->fds, pollset_set->fd_capacity * sizeof(*pollset_set->fds)); + } + GRPC_FD_REF(fd, "pollset_set"); + pollset_set->fds[pollset_set->fd_count++] = fd; + for (i = 0; i < pollset_set->pollset_count; i++) { + pollset_add_fd(exec_ctx, pollset_set->pollsets[i], fd); + } + for (i = 0; i < pollset_set->pollset_set_count; i++) { + pollset_set_add_fd(exec_ctx, pollset_set->pollset_sets[i], fd); + } + gpr_mu_unlock(&pollset_set->mu); +} + +static void pollset_set_del_fd(grpc_exec_ctx *exec_ctx, + grpc_pollset_set *pollset_set, grpc_fd *fd) { + size_t i; + gpr_mu_lock(&pollset_set->mu); + for (i = 0; i < pollset_set->fd_count; i++) { + if (pollset_set->fds[i] == fd) { + pollset_set->fd_count--; + GPR_SWAP(grpc_fd *, pollset_set->fds[i], + pollset_set->fds[pollset_set->fd_count]); + GRPC_FD_UNREF(fd, "pollset_set"); + break; + } + } + for (i = 0; i < pollset_set->pollset_set_count; i++) { + pollset_set_del_fd(exec_ctx, pollset_set->pollset_sets[i], fd); + } + gpr_mu_unlock(&pollset_set->mu); +} + static void pollset_set_add_pollset(grpc_exec_ctx *exec_ctx, grpc_pollset_set *pollset_set, grpc_pollset *pollset) { @@ -1281,47 +1300,8 @@ static void pollset_set_del_pollset_set(grpc_exec_ctx *exec_ctx, gpr_mu_unlock(&bag->mu); } -static void pollset_set_add_fd(grpc_exec_ctx *exec_ctx, - grpc_pollset_set *pollset_set, grpc_fd *fd) { - size_t i; - gpr_mu_lock(&pollset_set->mu); - if (pollset_set->fd_count == pollset_set->fd_capacity) { - pollset_set->fd_capacity = GPR_MAX(8, 2 * pollset_set->fd_capacity); - pollset_set->fds = gpr_realloc( - pollset_set->fds, pollset_set->fd_capacity * sizeof(*pollset_set->fds)); - } - GRPC_FD_REF(fd, "pollset_set"); - pollset_set->fds[pollset_set->fd_count++] = fd; - for (i = 0; i < pollset_set->pollset_count; i++) { - pollset_add_fd(exec_ctx, pollset_set->pollsets[i], fd); - } - for (i = 0; i < pollset_set->pollset_set_count; i++) { - pollset_set_add_fd(exec_ctx, pollset_set->pollset_sets[i], fd); - } - gpr_mu_unlock(&pollset_set->mu); -} - -static void pollset_set_del_fd(grpc_exec_ctx *exec_ctx, - grpc_pollset_set *pollset_set, grpc_fd *fd) { - size_t i; - gpr_mu_lock(&pollset_set->mu); - for (i = 0; i < pollset_set->fd_count; i++) { - if (pollset_set->fds[i] == fd) { - pollset_set->fd_count--; - GPR_SWAP(grpc_fd *, pollset_set->fds[i], - pollset_set->fds[pollset_set->fd_count]); - GRPC_FD_UNREF(fd, "pollset_set"); - break; - } - } - for (i = 0; i < pollset_set->pollset_set_count; i++) { - pollset_set_del_fd(exec_ctx, pollset_set->pollset_sets[i], fd); - } - gpr_mu_unlock(&pollset_set->mu); -} - /******************************************************************************* - * event engine binding + * Event engine binding */ static void shutdown_engine(void) { From 4eb67b47a0918f5633585cfc3335476139d37685 Mon Sep 17 00:00:00 2001 From: Alistair Veitch Date: Fri, 3 Jun 2016 15:45:04 -0700 Subject: [PATCH 0049/1039] Proto tweaks --- src/core/ext/census/gen/census.pb.c | 25 +++++++------ src/core/ext/census/gen/census.pb.h | 49 ++++++++++++++++--------- src/proto/census/census.proto | 57 ++++++++++++++++++----------- 3 files changed, 82 insertions(+), 49 deletions(-) diff --git a/src/core/ext/census/gen/census.pb.c b/src/core/ext/census/gen/census.pb.c index c73c2c9b7cc..647f0635b74 100644 --- a/src/core/ext/census/gen/census.pb.c +++ b/src/core/ext/census/gen/census.pb.c @@ -67,9 +67,10 @@ const pb_field_t google_census_Resource_MeasurementUnit_fields[4] = { PB_LAST_FIELD }; -const pb_field_t google_census_AggregationDescriptor_fields[3] = { - PB_ONEOF_FIELD(options, 1, MESSAGE , ONEOF, STATIC , FIRST, google_census_AggregationDescriptor, bucket_boundaries, bucket_boundaries, &google_census_AggregationDescriptor_BucketBoundaries_fields), - PB_ONEOF_FIELD(options, 2, MESSAGE , ONEOF, STATIC , FIRST, google_census_AggregationDescriptor, interval_boundaries, interval_boundaries, &google_census_AggregationDescriptor_IntervalBoundaries_fields), +const pb_field_t google_census_AggregationDescriptor_fields[4] = { + PB_FIELD( 1, UENUM , OPTIONAL, STATIC , FIRST, google_census_AggregationDescriptor, type, type, 0), + PB_ONEOF_FIELD(options, 2, MESSAGE , ONEOF, STATIC , OTHER, google_census_AggregationDescriptor, bucket_boundaries, type, &google_census_AggregationDescriptor_BucketBoundaries_fields), + PB_ONEOF_FIELD(options, 3, MESSAGE , ONEOF, STATIC , OTHER, google_census_AggregationDescriptor, interval_boundaries, type, &google_census_AggregationDescriptor_IntervalBoundaries_fields), PB_LAST_FIELD }; @@ -124,19 +125,21 @@ const pb_field_t google_census_View_fields[6] = { PB_LAST_FIELD }; -const pb_field_t google_census_Aggregation_fields[6] = { +const pb_field_t google_census_Aggregation_fields[7] = { PB_FIELD( 1, STRING , OPTIONAL, CALLBACK, FIRST, google_census_Aggregation, name, name, 0), PB_FIELD( 2, STRING , OPTIONAL, CALLBACK, OTHER, google_census_Aggregation, description, name, 0), - PB_ONEOF_FIELD(data, 3, MESSAGE , ONEOF, STATIC , OTHER, google_census_Aggregation, distribution, description, &google_census_Distribution_fields), - PB_ONEOF_FIELD(data, 4, MESSAGE , ONEOF, STATIC , OTHER, google_census_Aggregation, interval_stats, description, &google_census_IntervalStats_fields), - PB_FIELD( 5, MESSAGE , REPEATED, CALLBACK, OTHER, google_census_Aggregation, tag, data.interval_stats, &google_census_Tag_fields), + PB_ONEOF_FIELD(data, 3, UINT64 , ONEOF, STATIC , OTHER, google_census_Aggregation, count, description, 0), + PB_ONEOF_FIELD(data, 4, MESSAGE , ONEOF, STATIC , OTHER, google_census_Aggregation, distribution, description, &google_census_Distribution_fields), + PB_ONEOF_FIELD(data, 5, MESSAGE , ONEOF, STATIC , OTHER, google_census_Aggregation, interval_stats, description, &google_census_IntervalStats_fields), + PB_FIELD( 6, MESSAGE , REPEATED, CALLBACK, OTHER, google_census_Aggregation, tag, data.interval_stats, &google_census_Tag_fields), PB_LAST_FIELD }; -const pb_field_t google_census_Metric_fields[4] = { - PB_FIELD( 1, MESSAGE , REPEATED, CALLBACK, FIRST, google_census_Metric, aggregation, aggregation, &google_census_Aggregation_fields), - PB_FIELD( 2, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_Metric, start, aggregation, &google_census_Timestamp_fields), - PB_FIELD( 3, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_Metric, end, start, &google_census_Timestamp_fields), +const pb_field_t google_census_Metric_fields[5] = { + PB_FIELD( 1, STRING , OPTIONAL, CALLBACK, FIRST, google_census_Metric, view_name, view_name, 0), + PB_FIELD( 2, MESSAGE , REPEATED, CALLBACK, OTHER, google_census_Metric, aggregation, view_name, &google_census_Aggregation_fields), + PB_FIELD( 3, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_Metric, start, aggregation, &google_census_Timestamp_fields), + PB_FIELD( 4, MESSAGE , OPTIONAL, STATIC , OTHER, google_census_Metric, end, start, &google_census_Timestamp_fields), PB_LAST_FIELD }; diff --git a/src/core/ext/census/gen/census.pb.h b/src/core/ext/census/gen/census.pb.h index dfb53948204..dae583f33d3 100644 --- a/src/core/ext/census/gen/census.pb.h +++ b/src/core/ext/census/gen/census.pb.h @@ -54,6 +54,13 @@ typedef enum _google_census_Resource_BasicUnit { google_census_Resource_BasicUnit_MAX_UNITS = 5 } google_census_Resource_BasicUnit; +typedef enum _google_census_AggregationDescriptor_AggregationType { + google_census_AggregationDescriptor_AggregationType_UNKNOWN = 0, + google_census_AggregationDescriptor_AggregationType_COUNT = 1, + google_census_AggregationDescriptor_AggregationType_DISTRIBUTION = 2, + google_census_AggregationDescriptor_AggregationType_INTERVAL = 3 +} google_census_AggregationDescriptor_AggregationType; + /* Struct definitions */ typedef struct _google_census_AggregationDescriptor_BucketBoundaries { pb_callback_t bounds; @@ -68,6 +75,8 @@ typedef struct _google_census_IntervalStats { } google_census_IntervalStats; typedef struct _google_census_AggregationDescriptor { + bool has_type; + google_census_AggregationDescriptor_AggregationType type; pb_size_t which_options; union { google_census_AggregationDescriptor_BucketBoundaries bucket_boundaries; @@ -130,6 +139,7 @@ typedef struct _google_census_IntervalStats_Window { } google_census_IntervalStats_Window; typedef struct _google_census_Metric { + pb_callback_t view_name; pb_callback_t aggregation; bool has_start; google_census_Timestamp start; @@ -158,6 +168,7 @@ typedef struct _google_census_Aggregation { pb_callback_t description; pb_size_t which_data; union { + uint64_t count; google_census_Distribution distribution; google_census_IntervalStats interval_stats; } data; @@ -171,7 +182,7 @@ typedef struct _google_census_Aggregation { #define google_census_Timestamp_init_default {false, 0, false, 0} #define google_census_Resource_init_default {{{NULL}, NULL}, {{NULL}, NULL}, false, google_census_Resource_MeasurementUnit_init_default} #define google_census_Resource_MeasurementUnit_init_default {false, 0, {{NULL}, NULL}, {{NULL}, NULL}} -#define google_census_AggregationDescriptor_init_default {0, {google_census_AggregationDescriptor_BucketBoundaries_init_default}} +#define google_census_AggregationDescriptor_init_default {false, (google_census_AggregationDescriptor_AggregationType)0, 0, {google_census_AggregationDescriptor_BucketBoundaries_init_default}} #define google_census_AggregationDescriptor_BucketBoundaries_init_default {{{NULL}, NULL}} #define google_census_AggregationDescriptor_IntervalBoundaries_init_default {{{NULL}, NULL}} #define google_census_Distribution_init_default {false, 0, false, 0, false, google_census_Distribution_Range_init_default, {{NULL}, NULL}} @@ -180,13 +191,13 @@ typedef struct _google_census_Aggregation { #define google_census_IntervalStats_Window_init_default {false, google_census_Duration_init_default, false, 0, false, 0} #define google_census_Tag_init_default {false, "", false, ""} #define google_census_View_init_default {{{NULL}, NULL}, {{NULL}, NULL}, {{NULL}, NULL}, false, google_census_AggregationDescriptor_init_default, {{NULL}, NULL}} -#define google_census_Aggregation_init_default {{{NULL}, NULL}, {{NULL}, NULL}, 0, {google_census_Distribution_init_default}, {{NULL}, NULL}} -#define google_census_Metric_init_default {{{NULL}, NULL}, false, google_census_Timestamp_init_default, false, google_census_Timestamp_init_default} +#define google_census_Aggregation_init_default {{{NULL}, NULL}, {{NULL}, NULL}, 0, {0}, {{NULL}, NULL}} +#define google_census_Metric_init_default {{{NULL}, NULL}, {{NULL}, NULL}, false, google_census_Timestamp_init_default, false, google_census_Timestamp_init_default} #define google_census_Duration_init_zero {false, 0, false, 0} #define google_census_Timestamp_init_zero {false, 0, false, 0} #define google_census_Resource_init_zero {{{NULL}, NULL}, {{NULL}, NULL}, false, google_census_Resource_MeasurementUnit_init_zero} #define google_census_Resource_MeasurementUnit_init_zero {false, 0, {{NULL}, NULL}, {{NULL}, NULL}} -#define google_census_AggregationDescriptor_init_zero {0, {google_census_AggregationDescriptor_BucketBoundaries_init_zero}} +#define google_census_AggregationDescriptor_init_zero {false, (google_census_AggregationDescriptor_AggregationType)0, 0, {google_census_AggregationDescriptor_BucketBoundaries_init_zero}} #define google_census_AggregationDescriptor_BucketBoundaries_init_zero {{{NULL}, NULL}} #define google_census_AggregationDescriptor_IntervalBoundaries_init_zero {{{NULL}, NULL}} #define google_census_Distribution_init_zero {false, 0, false, 0, false, google_census_Distribution_Range_init_zero, {{NULL}, NULL}} @@ -195,16 +206,17 @@ typedef struct _google_census_Aggregation { #define google_census_IntervalStats_Window_init_zero {false, google_census_Duration_init_zero, false, 0, false, 0} #define google_census_Tag_init_zero {false, "", false, ""} #define google_census_View_init_zero {{{NULL}, NULL}, {{NULL}, NULL}, {{NULL}, NULL}, false, google_census_AggregationDescriptor_init_zero, {{NULL}, NULL}} -#define google_census_Aggregation_init_zero {{{NULL}, NULL}, {{NULL}, NULL}, 0, {google_census_Distribution_init_zero}, {{NULL}, NULL}} -#define google_census_Metric_init_zero {{{NULL}, NULL}, false, google_census_Timestamp_init_zero, false, google_census_Timestamp_init_zero} +#define google_census_Aggregation_init_zero {{{NULL}, NULL}, {{NULL}, NULL}, 0, {0}, {{NULL}, NULL}} +#define google_census_Metric_init_zero {{{NULL}, NULL}, {{NULL}, NULL}, false, google_census_Timestamp_init_zero, false, google_census_Timestamp_init_zero} /* Field tags (for use in manual encoding/decoding) */ #define google_census_AggregationDescriptor_BucketBoundaries_bounds_tag 1 #define google_census_AggregationDescriptor_IntervalBoundaries_window_size_tag 1 #define google_census_IntervalStats_window_tag 1 -#define google_census_AggregationDescriptor_bucket_boundaries_tag 1 +#define google_census_AggregationDescriptor_bucket_boundaries_tag 2 -#define google_census_AggregationDescriptor_interval_boundaries_tag 2 +#define google_census_AggregationDescriptor_interval_boundaries_tag 3 +#define google_census_AggregationDescriptor_type_tag 1 #define google_census_Distribution_Range_min_tag 1 #define google_census_Distribution_Range_max_tag 2 #define google_census_Duration_seconds_tag 1 @@ -223,9 +235,10 @@ typedef struct _google_census_Aggregation { #define google_census_IntervalStats_Window_window_size_tag 1 #define google_census_IntervalStats_Window_count_tag 2 #define google_census_IntervalStats_Window_mean_tag 3 -#define google_census_Metric_aggregation_tag 1 -#define google_census_Metric_start_tag 2 -#define google_census_Metric_end_tag 3 +#define google_census_Metric_view_name_tag 1 +#define google_census_Metric_aggregation_tag 2 +#define google_census_Metric_start_tag 3 +#define google_census_Metric_end_tag 4 #define google_census_Resource_name_tag 1 #define google_census_Resource_description_tag 2 #define google_census_Resource_unit_tag 3 @@ -234,19 +247,21 @@ typedef struct _google_census_Aggregation { #define google_census_View_resource_name_tag 3 #define google_census_View_aggregation_tag 4 #define google_census_View_tag_key_tag 5 -#define google_census_Aggregation_distribution_tag 3 +#define google_census_Aggregation_count_tag 3 + +#define google_census_Aggregation_distribution_tag 4 -#define google_census_Aggregation_interval_stats_tag 4 +#define google_census_Aggregation_interval_stats_tag 5 #define google_census_Aggregation_name_tag 1 #define google_census_Aggregation_description_tag 2 -#define google_census_Aggregation_tag_tag 5 +#define google_census_Aggregation_tag_tag 6 /* Struct field encoding specification for nanopb */ extern const pb_field_t google_census_Duration_fields[3]; extern const pb_field_t google_census_Timestamp_fields[3]; extern const pb_field_t google_census_Resource_fields[4]; extern const pb_field_t google_census_Resource_MeasurementUnit_fields[4]; -extern const pb_field_t google_census_AggregationDescriptor_fields[3]; +extern const pb_field_t google_census_AggregationDescriptor_fields[4]; extern const pb_field_t google_census_AggregationDescriptor_BucketBoundaries_fields[2]; extern const pb_field_t google_census_AggregationDescriptor_IntervalBoundaries_fields[2]; extern const pb_field_t google_census_Distribution_fields[5]; @@ -255,8 +270,8 @@ extern const pb_field_t google_census_IntervalStats_fields[2]; extern const pb_field_t google_census_IntervalStats_Window_fields[4]; extern const pb_field_t google_census_Tag_fields[3]; extern const pb_field_t google_census_View_fields[6]; -extern const pb_field_t google_census_Aggregation_fields[6]; -extern const pb_field_t google_census_Metric_fields[4]; +extern const pb_field_t google_census_Aggregation_fields[7]; +extern const pb_field_t google_census_Metric_fields[5]; /* Maximum encoded size of messages (where known) */ #define google_census_Duration_size 22 diff --git a/src/proto/census/census.proto b/src/proto/census/census.proto index 56c291ff17c..c2a594b6415 100644 --- a/src/proto/census/census.proto +++ b/src/proto/census/census.proto @@ -121,7 +121,7 @@ message Resource { // denominator: SECS // denominator: SECS // - // To specify multiples (in power of 10) units, specify a non-zero prefix + // To specify multiples (in power of 10) of units, specify a non-zero prefix // value, for example: // // - MB/s (i.e. megabytes / s): @@ -145,22 +145,33 @@ message Resource { // An Aggregation summarizes a series of individual Resource measurements, an // AggregationDescriptor describes an Aggregation. message AggregationDescriptor { - // At most one set of options. If neither option is set, a default type - // of Distribution (without a histogram component) will be used. + enum AggregationType { + // Unspecified. Should not be used. + UNKNOWN = 0; + // A count of measurements made. + COUNT = 1; + // A Distribution. + DISTRIBUTION = 2; + // Counts over fixed time intervals. + INTERVAL = 3; + } + // The type of Aggregation. + AggregationType type = 1; + + // At most one set of options. It is illegal to specifiy an option for + // COUNT Aggregations. interval_boundaries must be set for INTERVAL types. + // bucket_boundaries are optional for DISTRIBUTION types. oneof options { - // Defines the histogram bucket boundaries for Distributions. - BucketBoundaries bucket_boundaries = 1; + // Defines histogram bucket boundaries for Distributions. + BucketBoundaries bucket_boundaries = 2; // Defines the time windows to record for IntervalStats. - IntervalBoundaries interval_boundaries = 2; + IntervalBoundaries interval_boundaries = 3; } // A Distribution may optionally contain a histogram of the values in the - // population. The bucket boundaries for that histogram is described by - // `bucket_boundaries`. - // - // Describes histogram bucket boundaries. Defines `size(bounds) + 1` (= N) - // buckets (for size(bounds) >= 1; if size(bounds) == 0, then no histogram - // will be defined. The boundaries for bucket index i are: + // population. The bucket boundaries for that histogram are described by + // `bucket_boundaries`. This defines `size(bounds) + 1` (= N) buckets. The + // boundaries for bucket index i are: // // [-infinity, bounds[i]) for i == 0 // [bounds[i-1], bounds[i]) for 0 < i < N-2 @@ -190,8 +201,8 @@ message AggregationDescriptor { // a specified set of histogram buckets, as defined in // Aggregation.bucket_options. // -// The summary statistics are the count, mean, sum of the squared deviation from -// the mean, the minimum, and the maximum of the set of population of values. +// The summary statistics are the count, mean, minimum, and the maximum of the +// set of population of values. // // Although it is not forbidden, it is generally a bad idea to include // non-finite values (infinities or NaNs) in the population of values, as this @@ -237,7 +248,7 @@ message Distribution { message IntervalStats { // Summary statistic over a single time window. message Window { - // The window duration. + // The window duration. Must be positive. Duration window_size = 1; // The number of measurements in this window. int64 count = 2; @@ -285,23 +296,27 @@ message Aggregation { // The data for this Aggregation. oneof data { - Distribution distribution = 3; - IntervalStats interval_stats = 4; + uint64 count = 3; + Distribution distribution = 4; + IntervalStats interval_stats = 5; } // Tags associated with this Aggregation. - repeated Tag tag = 5; + repeated Tag tag = 6; } // A Metric represents all the Aggregations for a particular view. message Metric { + // View associated with this Metric. + string view_name = 1; + // Aggregations - each will have a unique set of tag values for the tag_keys // associated with the corresponding View. - repeated Aggregation aggregation = 1; + repeated Aggregation aggregation = 2; // Start and end timestamps over which the metric was accumulated. These // values are not relevant/defined for IntervalStats aggregations, which are // always accumulated over a fixed time period. - Timestamp start = 2; - Timestamp end = 3; + Timestamp start = 3; + Timestamp end = 4; } From 73ef9154024290e86a3566a574c04992afc93d00 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 3 Jun 2016 15:25:05 -0700 Subject: [PATCH 0050/1039] epoll polling strategy now points to the new code --- src/core/lib/iomgr/ev_epoll_linux.c | 4 ---- src/core/lib/iomgr/ev_posix.c | 4 ++-- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index ce42a9e7ce6..ab4224b2d56 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -788,16 +788,12 @@ static void sig_handler(int sig_num) { /* Global state management */ static void pollset_global_init(void) { - gpr_tls_init(&g_current_thread_poller); - gpr_tls_init(&g_current_thread_worker); grpc_wakeup_fd_init(&grpc_global_wakeup_fd); signal(SIGUSR1, sig_handler); } static void pollset_global_shutdown(void) { grpc_wakeup_fd_destroy(&grpc_global_wakeup_fd); - gpr_tls_destroy(&g_current_thread_poller); - gpr_tls_destroy(&g_current_thread_worker); } /* Return 1 if the pollset has active threads in pollset_work (pollset must diff --git a/src/core/lib/iomgr/ev_posix.c b/src/core/lib/iomgr/ev_posix.c index 96399ef8379..b6b113aed3f 100644 --- a/src/core/lib/iomgr/ev_posix.c +++ b/src/core/lib/iomgr/ev_posix.c @@ -44,7 +44,7 @@ #include #include -#include "src/core/lib/iomgr/ev_epoll_posix.h" +#include "src/core/lib/iomgr/ev_epoll_linux.h" #include "src/core/lib/iomgr/ev_poll_posix.h" #include "src/core/lib/support/env.h" @@ -62,7 +62,7 @@ typedef struct { } event_engine_factory; static const event_engine_factory g_factories[] = { - {"poll", grpc_init_poll_posix}, {"epoll", grpc_init_epoll_posix}, + {"poll", grpc_init_poll_posix}, {"epoll", grpc_init_epoll_linux}, }; static void add(const char *beg, const char *end, char ***ss, size_t *ns) { From 88ee12fbe98685e736366d6a151a10ed103f8979 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 3 Jun 2016 19:26:48 -0700 Subject: [PATCH 0051/1039] Handle pollsets and fds witn no polling islands and fix locking bug in pollset_add_fd --- src/core/lib/iomgr/ev_epoll_linux.c | 83 ++++++++++++++++------------- 1 file changed, 45 insertions(+), 38 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index ab4224b2d56..0fb1ccfa0fe 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -701,12 +701,14 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - Decrement the ref count on the polling island and det fd->polling_island to NULL */ gpr_mu_lock(&fd->pi_mu); - - fd->polling_island = polling_island_update_and_lock(fd->polling_island, 1, 0); - polling_island_remove_fd_locked(fd->polling_island, fd, !fd->released, true); - polling_island_unref_and_unlock(fd->polling_island, 1); - fd->polling_island = NULL; - + if (fd->polling_island != NULL) { + fd->polling_island = + polling_island_update_and_lock(fd->polling_island, 1, 0); + polling_island_remove_fd_locked(fd->polling_island, fd, !fd->released, + true); + polling_island_unref_and_unlock(fd->polling_island, 1); + fd->polling_island = NULL; + } gpr_mu_unlock(&fd->pi_mu); grpc_exec_ctx_enqueue(exec_ctx, fd->on_done_closure, true, NULL); @@ -926,13 +928,12 @@ static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { set_ready(exec_ctx, fd, &fd->write_closure); } -/* TODO(klempner): We probably want to turn this down a bit */ #define GRPC_EPOLL_MAX_EVENTS 1000 static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, int timeout_ms, sigset_t *sig_mask) { struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; - int epoll_fd; + int epoll_fd = -1; int ep_rv; GPR_TIMER_BEGIN("pollset_work_and_unlock", 0); @@ -943,45 +944,49 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, - pollset->pi_mu - pollset->polling_island->mu */ gpr_mu_lock(&pollset->pi_mu); - pollset->polling_island = - polling_island_update_and_lock(pollset->polling_island, 1, 0); - epoll_fd = pollset->polling_island->epoll_fd; + if (pollset->polling_island != NULL) { + pollset->polling_island = + polling_island_update_and_lock(pollset->polling_island, 1, 0); + epoll_fd = pollset->polling_island->epoll_fd; + gpr_mu_unlock(&pollset->polling_island->mu); + } - /* Release the locks */ - polling_island_unref_and_unlock(pollset->polling_island, 0); /* Keep the ref*/ gpr_mu_unlock(&pollset->pi_mu); gpr_mu_unlock(&pollset->mu); - do { - ep_rv = epoll_pwait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, timeout_ms, - sig_mask); + /* If epoll_fd == -1, this is a blank pollset and does not have any fds yet */ + if (epoll_fd != -1) { + do { + ep_rv = epoll_pwait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, timeout_ms, + sig_mask); - if (ep_rv < 0) { - if (errno != EINTR) { - /* TODO (sreek) - Check for bad file descriptor error */ - gpr_log(GPR_ERROR, "epoll_pwait() failed: %s", strerror(errno)); - } - } else { - int i; - for (i = 0; i < ep_rv; ++i) { - grpc_fd *fd = ep_ev[i].data.ptr; - int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); - int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); - int write_ev = ep_ev[i].events & EPOLLOUT; - if (fd == NULL) { - grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); - } else { - if (read_ev || cancel) { - fd_become_readable(exec_ctx, fd); - } - if (write_ev || cancel) { - fd_become_writable(exec_ctx, fd); + if (ep_rv < 0) { + if (errno != EINTR) { + /* TODO (sreek) - Check for bad file descriptor error */ + gpr_log(GPR_ERROR, "epoll_pwait() failed: %s", strerror(errno)); + } + } else { + int i; + for (i = 0; i < ep_rv; ++i) { + grpc_fd *fd = ep_ev[i].data.ptr; + int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); + int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); + int write_ev = ep_ev[i].events & EPOLLOUT; + if (fd == NULL) { + grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); + } else { + if (read_ev || cancel) { + fd_become_readable(exec_ctx, fd); + } + if (write_ev || cancel) { + fd_become_writable(exec_ctx, fd); + } } } } - } - } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); + } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); + } GPR_TIMER_END("pollset_work_and_unlock", 0); } @@ -1141,8 +1146,10 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, } } else if (fd->polling_island == NULL) { pi_new = polling_island_update_and_lock(pollset->polling_island, 1, 1); + gpr_mu_unlock(&pi_new->mu); } else if (pollset->polling_island == NULL) { pi_new = polling_island_update_and_lock(fd->polling_island, 1, 1); + gpr_mu_unlock(&pi_new->mu); } else { pi_new = polling_island_merge(fd->polling_island, pollset->polling_island); } From 79a6233bef501e1f51250351a55abc61cc024827 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Sat, 4 Jun 2016 14:01:03 -0700 Subject: [PATCH 0052/1039] Fix a few bugs in ev_epoll_linux.c 1. pollset_add_fd: Add fd to epoll set if fd->polling_island == NULL 2. close(fd) in fd_orphan instead of polling_island_remove_fd_locked() since fd->polling_island may be NULL 3. If pollset work() is interrupted, do a zero timeout epoll_wait(). pollset_work may be called without a polling island --- src/core/lib/iomgr/ev_epoll_linux.c | 125 +++++++++++++++++----------- 1 file changed, 78 insertions(+), 47 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 0fb1ccfa0fe..3aa26109f22 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -72,9 +72,14 @@ struct grpc_fd { gpr_atm refst; gpr_mu mu; + + /* Indicates that the fd is shutdown and that any pending read/write closures + should fail */ bool shutdown; - int closed; - bool released; + + /* The fd is either closed or we relinquished control of it. In either cases, + this indicates that the 'fd' on this structure is no longer valid */ + bool orphaned; grpc_closure *read_closure; grpc_closure *write_closure; @@ -251,16 +256,13 @@ static void polling_island_remove_all_fds_locked(polling_island *pi, /* The caller is expected to hold pi->mu lock before calling this function */ static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd, - bool close_fd, bool remove_fd_ref) { + bool is_fd_closed) { int err; size_t i; - /* Calling close() on the fd will automatically remove it from the epoll set. - If not calling close(), the fd must be explicitly removed from the epoll - set */ - if (close_fd) { - close(fd->fd); - } else { + /* If fd is already closed, then it would have been automatically been removed + from the epoll set */ + if (!is_fd_closed) { err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_DEL, fd->fd, NULL); if (err < 0 && errno != ENOENT) { gpr_log(GPR_ERROR, "epoll_ctl delete for fd: %d failed with error; %s", @@ -271,9 +273,7 @@ static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd, for (i = 0; i < pi->fd_cnt; i++) { if (pi->fds[i] == fd) { pi->fds[i] = pi->fds[--pi->fd_cnt]; - if (remove_fd_ref) { - GRPC_FD_UNREF(fd, "polling_island"); - } + GRPC_FD_UNREF(fd, "polling_island"); break; } } @@ -644,8 +644,7 @@ static grpc_fd *fd_create(int fd, const char *name) { new_fd->polling_island = NULL; new_fd->freelist_next = NULL; new_fd->on_done_closure = NULL; - new_fd->closed = 0; - new_fd->released = false; + new_fd->orphaned = false; gpr_mu_unlock(&new_fd->mu); @@ -666,7 +665,7 @@ static bool fd_is_orphaned(grpc_fd *fd) { static int fd_wrapped_fd(grpc_fd *fd) { int ret_fd = -1; gpr_mu_lock(&fd->mu); - if (!fd->released && !fd->closed) { + if (!fd->orphaned) { ret_fd = fd->fd; } gpr_mu_unlock(&fd->mu); @@ -678,34 +677,35 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure *on_done, int *release_fd, const char *reason) { /* TODO(sreek) In ev_poll_posix.c,the lock is acquired a little later. Why? */ + bool is_fd_closed = false; gpr_mu_lock(&fd->mu); fd->on_done_closure = on_done; /* If release_fd is not NULL, we should be relinquishing control of the file descriptor fd->fd (but we still own the grpc_fd structure). */ - fd->released = release_fd != NULL; - if (!fd->released) { - shutdown(fd->fd, SHUT_RDWR); - } else { + if (release_fd != NULL) { *release_fd = fd->fd; + } else { + close(fd->fd); + is_fd_closed = true; } - REF_BY(fd, 1, reason); /* Remove active status, but keep referenced */ - fd->closed = 1; + fd->orphaned = true; + + /* Remove the active status but keep referenced. We want this grpc_fd struct + to be alive (and not added to freelist) until the end of this function */ + REF_BY(fd, 1, reason); /* Remove the fd from the polling island: - Update the fd->polling_island to point to the latest polling island - - Remove the fd from the polling island. Also, call close() on the file - descriptor fd->fd ONLY if we haven't relinquised control (i.e - fd->released is 'false') - - Decrement the ref count on the polling island and det fd->polling_island - to NULL */ + - Remove the fd from the polling island. + - Remove a ref to the polling island and set fd->polling_island to NULL */ gpr_mu_lock(&fd->pi_mu); if (fd->polling_island != NULL) { fd->polling_island = polling_island_update_and_lock(fd->polling_island, 1, 0); - polling_island_remove_fd_locked(fd->polling_island, fd, !fd->released, - true); + polling_island_remove_fd_locked(fd->polling_island, fd, is_fd_closed); + polling_island_unref_and_unlock(fd->polling_island, 1); fd->polling_island = NULL; } @@ -839,17 +839,20 @@ static void pollset_kick(grpc_pollset *p, grpc_pollset_worker *worker = specific_worker; if (worker != NULL) { if (worker == GRPC_POLLSET_KICK_BROADCAST) { - GPR_TIMER_BEGIN("pollset_kick.broadcast", 0); + gpr_log(GPR_DEBUG, "pollset_kick: broadcast!"); if (pollset_has_workers(p)) { + GPR_TIMER_BEGIN("pollset_kick.broadcast", 0); for (worker = p->root_worker.next; worker != &p->root_worker; worker = worker->next) { pthread_kill(worker->pt_id, SIGUSR1); } } else { + gpr_log(GPR_DEBUG, "pollset_kick: (broadcast) Kicked without pollers"); p->kicked_without_pollers = true; } GPR_TIMER_END("pollset_kick.broadcast", 0); } else { + gpr_log(GPR_DEBUG, "pollset_kick: kicked kicked_specifically"); GPR_TIMER_MARK("kicked_specifically", 0); worker->kicked_specifically = true; pthread_kill(worker->pt_id, SIGUSR1); @@ -860,9 +863,11 @@ static void pollset_kick(grpc_pollset *p, if (worker != NULL) { GPR_TIMER_MARK("finally_kick", 0); push_back_worker(p, worker); + gpr_log(GPR_DEBUG, "pollset_kick: anonymous kick"); pthread_kill(worker->pt_id, SIGUSR1); } else { GPR_TIMER_MARK("kicked_no_pollers", 0); + gpr_log(GPR_DEBUG, "pollset_kick: kicked without pollers"); p->kicked_without_pollers = true; } } @@ -935,6 +940,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; int epoll_fd = -1; int ep_rv; + gpr_log(GPR_DEBUG, "pollset_work_and_unlock: Entering.."); GPR_TIMER_BEGIN("pollset_work_and_unlock", 0); /* We need to get the epoll_fd to wait on. The epoll_fd is in inside the @@ -949,6 +955,16 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, pollset->polling_island = polling_island_update_and_lock(pollset->polling_island, 1, 0); epoll_fd = pollset->polling_island->epoll_fd; + if (pollset->polling_island->fd_cnt == 0) { + gpr_log(GPR_DEBUG, "pollset_work_and_unlock: epoll_fd: %d, No other fds", + epoll_fd); + } + for (size_t i = 0; i < pollset->polling_island->fd_cnt; i++) { + gpr_log(GPR_DEBUG, + "pollset_work_and_unlock: epoll_fd: %d, fd_count: %d, fd[%d]: %d", + epoll_fd, pollset->polling_island->fd_cnt, i, + pollset->polling_island->fds[i]->fd); + } gpr_mu_unlock(&pollset->polling_island->mu); } @@ -958,36 +974,47 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, /* If epoll_fd == -1, this is a blank pollset and does not have any fds yet */ if (epoll_fd != -1) { do { + gpr_timespec before_epoll = gpr_now(GPR_CLOCK_PRECISE); + gpr_log(GPR_DEBUG, "pollset_work_and_unlock: epoll_wait()...."); ep_rv = epoll_pwait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, timeout_ms, sig_mask); + gpr_timespec after_epoll = gpr_now(GPR_CLOCK_PRECISE); + int dur = gpr_time_to_millis(gpr_time_sub(after_epoll, before_epoll)); + gpr_log(GPR_DEBUG, + "pollset_work_and_unlock: DONE epoll_wait() : %d ms, ep_rv: %d", + dur, ep_rv); if (ep_rv < 0) { if (errno != EINTR) { /* TODO (sreek) - Check for bad file descriptor error */ gpr_log(GPR_ERROR, "epoll_pwait() failed: %s", strerror(errno)); + } else { + gpr_log(GPR_DEBUG, "pollset_work_and_unlock: 0-timeout epoll_wait()"); + ep_rv = epoll_wait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, 0); + gpr_log(GPR_DEBUG, "pollset_work_and_unlock: ep_rv: %d", ep_rv); } - } else { - int i; - for (i = 0; i < ep_rv; ++i) { - grpc_fd *fd = ep_ev[i].data.ptr; - int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); - int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); - int write_ev = ep_ev[i].events & EPOLLOUT; - if (fd == NULL) { - grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); - } else { - if (read_ev || cancel) { - fd_become_readable(exec_ctx, fd); - } - if (write_ev || cancel) { - fd_become_writable(exec_ctx, fd); - } + } + + int i; + for (i = 0; i < ep_rv; ++i) { + grpc_fd *fd = ep_ev[i].data.ptr; + int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); + int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); + int write_ev = ep_ev[i].events & EPOLLOUT; + if (fd == NULL) { + grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); + } else { + if (read_ev || cancel) { + fd_become_readable(exec_ctx, fd); + } + if (write_ev || cancel) { + fd_become_writable(exec_ctx, fd); } } } } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); } - + gpr_log(GPR_DEBUG, "pollset_work_and_unlock: Leaving.."); GPR_TIMER_END("pollset_work_and_unlock", 0); } @@ -1060,7 +1087,7 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker **worker_hdl, gpr_timespec now, gpr_timespec deadline) { GPR_TIMER_BEGIN("pollset_work", 0); - + gpr_log(GPR_DEBUG, "pollset_work: enter"); int timeout_ms = poll_deadline_to_millis_timeout(deadline, now); sigset_t new_mask; @@ -1079,6 +1106,7 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, work that needs attention like an event on the completion queue or an alarm */ GPR_TIMER_MARK("pollset_work.kicked_without_pollers", 0); + gpr_log(GPR_INFO, "pollset_work: kicked without pollers.."); pollset->kicked_without_pollers = 0; } else if (!pollset->shutting_down) { sigemptyset(&new_mask); @@ -1113,12 +1141,14 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, gpr_mu_lock(&pollset->mu); } + gpr_log(GPR_DEBUG, "pollset_work(): leaving"); *worker_hdl = NULL; GPR_TIMER_END("pollset_work", 0); } static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { + gpr_log(GPR_DEBUG, "pollset_add_fd: pollset: %p, fd: %d", pollset, fd->fd); /* TODO sreek - Check if we need to get a pollset->mu lock here */ gpr_mu_lock(&pollset->pi_mu); gpr_mu_lock(&fd->pi_mu); @@ -1146,6 +1176,7 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, } } else if (fd->polling_island == NULL) { pi_new = polling_island_update_and_lock(pollset->polling_island, 1, 1); + polling_island_add_fds_locked(pollset->polling_island, &fd, 1, true); gpr_mu_unlock(&pi_new->mu); } else if (pollset->polling_island == NULL) { pi_new = polling_island_update_and_lock(fd->polling_island, 1, 1); From b5fe44ea0f7e5ef6f013d6c97725f93a74fed5ef Mon Sep 17 00:00:00 2001 From: Julien Boeuf Date: Mon, 6 Jun 2016 09:50:38 +0200 Subject: [PATCH 0053/1039] Adding error details for JWT and oauth2. --- src/core/lib/security/credentials/jwt/jwt_credentials.c | 3 ++- .../lib/security/credentials/oauth2/oauth2_credentials.c | 5 +++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/src/core/lib/security/credentials/jwt/jwt_credentials.c b/src/core/lib/security/credentials/jwt/jwt_credentials.c index 0d29729da17..7f96e5247b0 100644 --- a/src/core/lib/security/credentials/jwt/jwt_credentials.c +++ b/src/core/lib/security/credentials/jwt/jwt_credentials.c @@ -116,7 +116,8 @@ static void jwt_get_request_metadata(grpc_exec_ctx *exec_ctx, GRPC_CREDENTIALS_OK, NULL); grpc_credentials_md_store_unref(jwt_md); } else { - cb(exec_ctx, user_data, NULL, 0, GRPC_CREDENTIALS_ERROR, NULL); + cb(exec_ctx, user_data, NULL, 0, GRPC_CREDENTIALS_ERROR, + "Could not generate JWT."); } } diff --git a/src/core/lib/security/credentials/oauth2/oauth2_credentials.c b/src/core/lib/security/credentials/oauth2/oauth2_credentials.c index 81457183d2d..a0e4b6fe8e9 100644 --- a/src/core/lib/security/credentials/oauth2/oauth2_credentials.c +++ b/src/core/lib/security/credentials/oauth2/oauth2_credentials.c @@ -233,10 +233,11 @@ static void on_oauth2_token_fetcher_http_response( c->token_expiration = gpr_time_add(gpr_now(GPR_CLOCK_REALTIME), token_lifetime); r->cb(exec_ctx, r->user_data, c->access_token_md->entries, - c->access_token_md->num_entries, status, NULL); + c->access_token_md->num_entries, GRPC_CREDENTIALS_OK, NULL); } else { c->token_expiration = gpr_inf_past(GPR_CLOCK_REALTIME); - r->cb(exec_ctx, r->user_data, NULL, 0, status, NULL); + r->cb(exec_ctx, r->user_data, NULL, 0, status, + "Error occured when fetching oauth2 token."); } gpr_mu_unlock(&c->mu); grpc_credentials_metadata_request_destroy(r); From 38c0cdee34697bacb95f34d1a4d31131d0df3af5 Mon Sep 17 00:00:00 2001 From: Julien Boeuf Date: Mon, 6 Jun 2016 14:46:08 +0200 Subject: [PATCH 0054/1039] Fix double-free issue for optional_message. Also better end to end testing of plumbing of plugin error message. --- .../ext/transport/chttp2/transport/chttp2_transport.c | 8 +++----- test/cpp/end2end/end2end_test.cc | 6 +++++- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index 1abc49f435e..b0da0578f5e 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -1504,17 +1504,15 @@ static void close_from_api(grpc_exec_ctx *exec_ctx, gpr_slice_buffer_add(&transport_global->qbuf, gpr_slice_ref(*optional_message)); } - gpr_slice_buffer_add( &transport_global->qbuf, grpc_chttp2_rst_stream_create(stream_global->id, GRPC_CHTTP2_NO_ERROR, &stream_global->stats.outgoing)); - - if (optional_message) { - gpr_slice_ref(*optional_message); - } } + if (optional_message) { + gpr_slice_ref(*optional_message); + } grpc_chttp2_fake_status(exec_ctx, transport_global, stream_global, status, optional_message); grpc_chttp2_mark_stream_closed(exec_ctx, transport_global, stream_global, 1, diff --git a/test/cpp/end2end/end2end_test.cc b/test/cpp/end2end/end2end_test.cc index f52aa52f39f..4602145f54b 100644 --- a/test/cpp/end2end/end2end_test.cc +++ b/test/cpp/end2end/end2end_test.cc @@ -75,6 +75,8 @@ bool CheckIsLocalhost(const grpc::string& addr) { addr.substr(0, kIpv6.size()) == kIpv6; } +const char kTestCredsPluginErrorMsg[] = "Could not find plugin metadata."; + class TestMetadataCredentialsPlugin : public MetadataCredentialsPlugin { public: static const char kMetadataKey[]; @@ -99,7 +101,7 @@ class TestMetadataCredentialsPlugin : public MetadataCredentialsPlugin { metadata->insert(std::make_pair(kMetadataKey, metadata_value_)); return Status::OK; } else { - return Status(StatusCode::NOT_FOUND, "Could not find plugin metadata."); + return Status(StatusCode::NOT_FOUND, kTestCredsPluginErrorMsg); } } @@ -1318,6 +1320,7 @@ TEST_P(SecureEnd2endTest, NonBlockingAuthMetadataPluginFailure) { Status s = stub_->Echo(&context, request, &response); EXPECT_FALSE(s.ok()); EXPECT_EQ(s.error_code(), StatusCode::UNAUTHENTICATED); + EXPECT_EQ(s.error_message(), kTestCredsPluginErrorMsg); } TEST_P(SecureEnd2endTest, NonBlockingAuthMetadataPluginAndProcessorSuccess) { @@ -1375,6 +1378,7 @@ TEST_P(SecureEnd2endTest, BlockingAuthMetadataPluginFailure) { Status s = stub_->Echo(&context, request, &response); EXPECT_FALSE(s.ok()); EXPECT_EQ(s.error_code(), StatusCode::UNAUTHENTICATED); + EXPECT_EQ(s.error_message(), kTestCredsPluginErrorMsg); } TEST_P(SecureEnd2endTest, ClientAuthContext) { From 7d3d4d48cd3a0b916d7f561f6fa525886fb34d8a Mon Sep 17 00:00:00 2001 From: Julien Boeuf Date: Mon, 6 Jun 2016 14:48:41 +0200 Subject: [PATCH 0055/1039] Fixing clang-format. --- src/core/lib/security/credentials/credentials.h | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/src/core/lib/security/credentials/credentials.h b/src/core/lib/security/credentials/credentials.h index c6df6cdf2f7..b097233bee4 100644 --- a/src/core/lib/security/credentials/credentials.h +++ b/src/core/lib/security/credentials/credentials.h @@ -156,12 +156,9 @@ void grpc_credentials_md_store_unref(grpc_credentials_md_store *store); /* --- grpc_call_credentials. --- */ /* error_details must be NULL if status is GRPC_CREDENTIALS_OK. */ -typedef void (*grpc_credentials_metadata_cb)(grpc_exec_ctx *exec_ctx, - void *user_data, - grpc_credentials_md *md_elems, - size_t num_md, - grpc_credentials_status status, - const char *error_details); +typedef void (*grpc_credentials_metadata_cb)( + grpc_exec_ctx *exec_ctx, void *user_data, grpc_credentials_md *md_elems, + size_t num_md, grpc_credentials_status status, const char *error_details); typedef struct { void (*destruct)(grpc_call_credentials *c); From 4c11a20bf0b6b1d64a2800bcb06f76404294aa84 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Mon, 6 Jun 2016 09:23:25 -0700 Subject: [PATCH 0056/1039] Remove unused files --- BUILD | 6 - Makefile | 38 - binding.gyp | 1 - build.yaml | 14 - config.m4 | 1 - gRPC.podspec | 3 - grpc.gemspec | 2 - package.xml | 2 - src/core/lib/iomgr/ctiller_ev_epoll_linux.c | 461 ------- src/core/lib/iomgr/ev_epoll_linux.c | 2 +- src/core/lib/iomgr/ev_epoll_posix.c | 1209 ----------------- src/core/lib/iomgr/ev_epoll_posix.h | 41 - src/python/grpcio/grpc_core_dependencies.py | 1 - test/core/network_benchmarks/epoll_test.c | 263 ---- .../network_benchmarks/low_level_ping_pong.c | 2 - tools/doxygen/Doxyfile.core.internal | 2 - tools/run_tests/sources_and_headers.json | 19 - tools/run_tests/tests.json | 15 - vsprojects/vcxproj/grpc/grpc.vcxproj | 3 - vsprojects/vcxproj/grpc/grpc.vcxproj.filters | 6 - .../grpc_unsecure/grpc_unsecure.vcxproj | 3 - .../grpc_unsecure.vcxproj.filters | 6 - 22 files changed, 1 insertion(+), 2099 deletions(-) delete mode 100644 src/core/lib/iomgr/ctiller_ev_epoll_linux.c delete mode 100644 src/core/lib/iomgr/ev_epoll_posix.c delete mode 100644 src/core/lib/iomgr/ev_epoll_posix.h delete mode 100644 test/core/network_benchmarks/epoll_test.c diff --git a/BUILD b/BUILD index a32352ebb30..70354a88101 100644 --- a/BUILD +++ b/BUILD @@ -179,7 +179,6 @@ cc_library( "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", "src/core/lib/iomgr/ev_epoll_linux.h", - "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", "src/core/lib/iomgr/exec_ctx.h", @@ -324,7 +323,6 @@ cc_library( "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", "src/core/lib/iomgr/ev_epoll_linux.c", - "src/core/lib/iomgr/ev_epoll_posix.c", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_posix.c", "src/core/lib/iomgr/exec_ctx.c", @@ -551,7 +549,6 @@ cc_library( "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", "src/core/lib/iomgr/ev_epoll_linux.h", - "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", "src/core/lib/iomgr/exec_ctx.h", @@ -673,7 +670,6 @@ cc_library( "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", "src/core/lib/iomgr/ev_epoll_linux.c", - "src/core/lib/iomgr/ev_epoll_posix.c", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_posix.c", "src/core/lib/iomgr/exec_ctx.c", @@ -1367,7 +1363,6 @@ objc_library( "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", "src/core/lib/iomgr/ev_epoll_linux.c", - "src/core/lib/iomgr/ev_epoll_posix.c", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_posix.c", "src/core/lib/iomgr/exec_ctx.c", @@ -1573,7 +1568,6 @@ objc_library( "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", "src/core/lib/iomgr/ev_epoll_linux.h", - "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", "src/core/lib/iomgr/exec_ctx.h", diff --git a/Makefile b/Makefile index 235f32d9a30..1c83aec21e7 100644 --- a/Makefile +++ b/Makefile @@ -903,7 +903,6 @@ dns_resolver_connectivity_test: $(BINDIR)/$(CONFIG)/dns_resolver_connectivity_te dns_resolver_test: $(BINDIR)/$(CONFIG)/dns_resolver_test dualstack_socket_test: $(BINDIR)/$(CONFIG)/dualstack_socket_test endpoint_pair_test: $(BINDIR)/$(CONFIG)/endpoint_pair_test -epoll_test: $(BINDIR)/$(CONFIG)/epoll_test fd_conservation_posix_test: $(BINDIR)/$(CONFIG)/fd_conservation_posix_test fd_posix_test: $(BINDIR)/$(CONFIG)/fd_posix_test fling_client: $(BINDIR)/$(CONFIG)/fling_client @@ -1235,7 +1234,6 @@ buildtests_c: privatelibs_c \ $(BINDIR)/$(CONFIG)/dns_resolver_test \ $(BINDIR)/$(CONFIG)/dualstack_socket_test \ $(BINDIR)/$(CONFIG)/endpoint_pair_test \ - $(BINDIR)/$(CONFIG)/epoll_test \ $(BINDIR)/$(CONFIG)/fd_conservation_posix_test \ $(BINDIR)/$(CONFIG)/fd_posix_test \ $(BINDIR)/$(CONFIG)/fling_client \ @@ -1499,8 +1497,6 @@ test_c: buildtests_c $(Q) $(BINDIR)/$(CONFIG)/dualstack_socket_test || ( echo test dualstack_socket_test failed ; exit 1 ) $(E) "[RUN] Testing endpoint_pair_test" $(Q) $(BINDIR)/$(CONFIG)/endpoint_pair_test || ( echo test endpoint_pair_test failed ; exit 1 ) - $(E) "[RUN] Testing epoll_test" - $(Q) $(BINDIR)/$(CONFIG)/epoll_test || ( echo test epoll_test failed ; exit 1 ) $(E) "[RUN] Testing fd_conservation_posix_test" $(Q) $(BINDIR)/$(CONFIG)/fd_conservation_posix_test || ( echo test fd_conservation_posix_test failed ; exit 1 ) $(E) "[RUN] Testing fd_posix_test" @@ -2491,7 +2487,6 @@ LIBGRPC_SRC = \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ src/core/lib/iomgr/ev_epoll_linux.c \ - src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ src/core/lib/iomgr/exec_ctx.c \ @@ -2847,7 +2842,6 @@ LIBGRPC_UNSECURE_SRC = \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ src/core/lib/iomgr/ev_epoll_linux.c \ - src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ src/core/lib/iomgr/exec_ctx.c \ @@ -6581,38 +6575,6 @@ endif endif -EPOLL_TEST_SRC = \ - test/core/network_benchmarks/epoll_test.c \ - -EPOLL_TEST_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(EPOLL_TEST_SRC)))) -ifeq ($(NO_SECURE),true) - -# You can't build secure targets if you don't have OpenSSL. - -$(BINDIR)/$(CONFIG)/epoll_test: openssl_dep_error - -else - - - -$(BINDIR)/$(CONFIG)/epoll_test: $(EPOLL_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a - $(E) "[LD] Linking $@" - $(Q) mkdir -p `dirname $@` - $(Q) $(LD) $(LDFLAGS) $(EPOLL_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/epoll_test - -endif - -$(OBJDIR)/$(CONFIG)/test/core/network_benchmarks/epoll_test.o: $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a - -deps_epoll_test: $(EPOLL_TEST_OBJS:.o=.dep) - -ifneq ($(NO_SECURE),true) -ifneq ($(NO_DEPS),true) --include $(EPOLL_TEST_OBJS:.o=.dep) -endif -endif - - FD_CONSERVATION_POSIX_TEST_SRC = \ test/core/iomgr/fd_conservation_posix_test.c \ diff --git a/binding.gyp b/binding.gyp index 41e1b5bb416..255998aafd8 100644 --- a/binding.gyp +++ b/binding.gyp @@ -582,7 +582,6 @@ 'src/core/lib/iomgr/endpoint_pair_posix.c', 'src/core/lib/iomgr/endpoint_pair_windows.c', 'src/core/lib/iomgr/ev_epoll_linux.c', - 'src/core/lib/iomgr/ev_epoll_posix.c', 'src/core/lib/iomgr/ev_poll_posix.c', 'src/core/lib/iomgr/ev_posix.c', 'src/core/lib/iomgr/exec_ctx.c', diff --git a/build.yaml b/build.yaml index db9787546ad..75c7a76bdb3 100644 --- a/build.yaml +++ b/build.yaml @@ -166,7 +166,6 @@ filegroups: - src/core/lib/iomgr/endpoint.h - src/core/lib/iomgr/endpoint_pair.h - src/core/lib/iomgr/ev_epoll_linux.h - - src/core/lib/iomgr/ev_epoll_posix.h - src/core/lib/iomgr/ev_poll_posix.h - src/core/lib/iomgr/ev_posix.h - src/core/lib/iomgr/exec_ctx.h @@ -242,7 +241,6 @@ filegroups: - src/core/lib/iomgr/endpoint_pair_posix.c - src/core/lib/iomgr/endpoint_pair_windows.c - src/core/lib/iomgr/ev_epoll_linux.c - - src/core/lib/iomgr/ev_epoll_posix.c - src/core/lib/iomgr/ev_poll_posix.c - src/core/lib/iomgr/ev_posix.c - src/core/lib/iomgr/exec_ctx.c @@ -1321,18 +1319,6 @@ targets: - grpc - gpr_test_util - gpr -- name: epoll_test - build: test - language: c - src: - - test/core/network_benchmarks/epoll_test.c - deps: - - grpc_test_util - - grpc - - gpr_test_util - - gpr - platforms: - - linux - name: fd_conservation_posix_test build: test language: c diff --git a/config.m4 b/config.m4 index 4308295afda..e2d1c00b6e4 100644 --- a/config.m4 +++ b/config.m4 @@ -101,7 +101,6 @@ if test "$PHP_GRPC" != "no"; then src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ src/core/lib/iomgr/ev_epoll_linux.c \ - src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ src/core/lib/iomgr/exec_ctx.c \ diff --git a/gRPC.podspec b/gRPC.podspec index de55880125a..736ae98b543 100644 --- a/gRPC.podspec +++ b/gRPC.podspec @@ -182,7 +182,6 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/endpoint.h', 'src/core/lib/iomgr/endpoint_pair.h', 'src/core/lib/iomgr/ev_epoll_linux.h', - 'src/core/lib/iomgr/ev_epoll_posix.h', 'src/core/lib/iomgr/ev_poll_posix.h', 'src/core/lib/iomgr/ev_posix.h', 'src/core/lib/iomgr/exec_ctx.h', @@ -361,7 +360,6 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/endpoint_pair_posix.c', 'src/core/lib/iomgr/endpoint_pair_windows.c', 'src/core/lib/iomgr/ev_epoll_linux.c', - 'src/core/lib/iomgr/ev_epoll_posix.c', 'src/core/lib/iomgr/ev_poll_posix.c', 'src/core/lib/iomgr/ev_posix.c', 'src/core/lib/iomgr/exec_ctx.c', @@ -551,7 +549,6 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/endpoint.h', 'src/core/lib/iomgr/endpoint_pair.h', 'src/core/lib/iomgr/ev_epoll_linux.h', - 'src/core/lib/iomgr/ev_epoll_posix.h', 'src/core/lib/iomgr/ev_poll_posix.h', 'src/core/lib/iomgr/ev_posix.h', 'src/core/lib/iomgr/exec_ctx.h', diff --git a/grpc.gemspec b/grpc.gemspec index 54ae2eb68dd..01b2890493f 100755 --- a/grpc.gemspec +++ b/grpc.gemspec @@ -191,7 +191,6 @@ Gem::Specification.new do |s| s.files += %w( src/core/lib/iomgr/endpoint.h ) s.files += %w( src/core/lib/iomgr/endpoint_pair.h ) s.files += %w( src/core/lib/iomgr/ev_epoll_linux.h ) - s.files += %w( src/core/lib/iomgr/ev_epoll_posix.h ) s.files += %w( src/core/lib/iomgr/ev_poll_posix.h ) s.files += %w( src/core/lib/iomgr/ev_posix.h ) s.files += %w( src/core/lib/iomgr/exec_ctx.h ) @@ -340,7 +339,6 @@ Gem::Specification.new do |s| s.files += %w( src/core/lib/iomgr/endpoint_pair_posix.c ) s.files += %w( src/core/lib/iomgr/endpoint_pair_windows.c ) s.files += %w( src/core/lib/iomgr/ev_epoll_linux.c ) - s.files += %w( src/core/lib/iomgr/ev_epoll_posix.c ) s.files += %w( src/core/lib/iomgr/ev_poll_posix.c ) s.files += %w( src/core/lib/iomgr/ev_posix.c ) s.files += %w( src/core/lib/iomgr/exec_ctx.c ) diff --git a/package.xml b/package.xml index d8e82a8bc37..ba6e11fadc7 100644 --- a/package.xml +++ b/package.xml @@ -198,7 +198,6 @@ - @@ -347,7 +346,6 @@ - diff --git a/src/core/lib/iomgr/ctiller_ev_epoll_linux.c b/src/core/lib/iomgr/ctiller_ev_epoll_linux.c deleted file mode 100644 index 23c20a77aae..00000000000 --- a/src/core/lib/iomgr/ctiller_ev_epoll_linux.c +++ /dev/null @@ -1,461 +0,0 @@ -/* - * - * Copyright 2015-2016, Google Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following disclaimer - * in the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Google Inc. nor the names of its - * contributors may be used to endorse or promote products derived from - * this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - */ - -#include "src/core/lib/iomgr/ev_epoll_linux.h" - -#include -#include -#include -#include - -#include -#include -#include -#include - -#include "src/core/lib/iomgr/iomgr_internal.h" - -/* TODO(sreek) Remove this file */ - - -//////////////////////////////////////////////////////////////////////////////// -// Definitions - -#define STATE_NOT_READY ((gpr_atm)0) -#define STATE_READY ((gpr_atm)1) - -typedef enum { POLLABLE_FD, POLLABLE_EPOLL_SET } pollable_type; - -typedef struct { - pollable_type type; - int fd; - grpc_iomgr_object iomgr_object; -} pollable_object; - -typedef struct polling_island { - pollable_object pollable; - gpr_mu mu; - int refs; - grpc_fd *only_fd; - struct polling_island *became; - struct polling_island *next; -} polling_island; - -struct grpc_fd { - pollable_object pollable; - - // each event atomic is a tri state: - // STATE_NOT_READY - no event received, nobody waiting for it either - // STATE_READY - event received, nobody waiting for it - // closure pointer - no event received, upper layer is waiting for it - gpr_atm on_readable; - gpr_atm on_writable; - - // mutex guarding set_ready & shutdown state - gpr_mu set_ready_mu; - bool shutdown; - - // mutex protecting polling_island - gpr_mu polling_island_mu; - // current polling island - polling_island *polling_island; - - grpc_fd *next_free; -}; - -struct grpc_pollset_worker {}; - -struct grpc_pollset { - gpr_mu mu; - // current polling island - polling_island *polling_island; -}; - -//////////////////////////////////////////////////////////////////////////////// -// Polling island implementation - -static gpr_mu g_pi_freelist_mu; -static polling_island *g_first_free_pi; - -static void add_pollable_to_epoll_set(pollable_object *pollable, int epoll_set, - uint32_t events) { - struct epoll_event ev; - ev.events = events; - ev.data.ptr = pollable; - int err = epoll_ctl(epoll_set, EPOLL_CTL_ADD, pollable->fd, &ev); - if (err < 0) { - gpr_log(GPR_ERROR, "epoll_ctl add for %d faild: %s", pollable->fd, - strerror(errno)); - } -} - -static void add_fd_to_epoll_set(grpc_fd *fd, int epoll_set) { - add_pollable_to_epoll_set(&fd->pollable, epoll_set, - EPOLLIN | EPOLLOUT | EPOLLET); -} - -static void add_island_to_epoll_set(polling_island *pi, int epoll_set) { - add_pollable_to_epoll_set(&pi->pollable, epoll_set, EPOLLIN | EPOLLET); -} - -static polling_island *polling_island_create(grpc_fd *initial_fd) { - polling_island *r = NULL; - gpr_mu_lock(&g_pi_freelist_mu); - if (g_first_free_pi == NULL) { - r = gpr_malloc(sizeof(*r)); - r->pollable.type = POLLABLE_EPOLL_SET; - gpr_mu_init(&r->mu); - } else { - r = g_first_free_pi; - g_first_free_pi = r->next; - } - gpr_mu_unlock(&g_pi_freelist_mu); - - r->pollable.fd = epoll_create1(EPOLL_CLOEXEC); - GPR_ASSERT(r->pollable.fd >= 0); - - gpr_mu_lock(&r->mu); - r->only_fd = initial_fd; - r->refs = 2; // creation of a polling island => a referencing pollset & fd - gpr_mu_unlock(&r->mu); - - add_fd_to_epoll_set(initial_fd, r->pollable.fd); - return r; -} - -static void polling_island_delete(polling_island *p) { - gpr_mu_lock(&g_pi_freelist_mu); - p->next = g_first_free_pi; - g_first_free_pi = p; - gpr_mu_unlock(&g_pi_freelist_mu); -} - -static polling_island *polling_island_add(polling_island *p, grpc_fd *fd) { - gpr_mu_lock(&p->mu); - p->only_fd = NULL; - p->refs++; // new fd picks up a ref - gpr_mu_unlock(&p->mu); - - add_fd_to_epoll_set(fd, p->pollable.fd); - - return p; -} - -static void add_siblings_to(polling_island *siblings, polling_island *dest) { - polling_island *sibling_tail = dest; - while (sibling_tail->next != NULL) { - sibling_tail = sibling_tail->next; - } - sibling_tail->next = siblings; -} - -static polling_island *polling_island_merge(polling_island *a, - polling_island *b) { - GPR_ASSERT(a != b); - polling_island *out; - - gpr_mu_lock(&GPR_MIN(a, b)->mu); - gpr_mu_lock(&GPR_MAX(a, b)->mu); - - GPR_ASSERT(a->became == NULL); - GPR_ASSERT(b->became == NULL); - - if (a->only_fd == NULL && b->only_fd == NULL) { - b->became = a; - add_siblings_to(b, a); - add_island_to_epoll_set(b, a->pollable.fd); - out = a; - } else if (a->only_fd == NULL) { - GPR_ASSERT(b->only_fd != NULL); - add_fd_to_epoll_set(b->only_fd, a->pollable.fd); - b->became = a; - out = a; - } else if (b->only_fd == NULL) { - GPR_ASSERT(a->only_fd != NULL); - add_fd_to_epoll_set(a->only_fd, b->pollable.fd); - a->became = b; - out = b; - } else { - add_fd_to_epoll_set(b->only_fd, a->pollable.fd); - a->only_fd = NULL; - b->only_fd = NULL; - b->became = a; - out = a; - } - - gpr_mu_unlock(&a->mu); - gpr_mu_unlock(&b->mu); - - return out; -} - -static polling_island *polling_island_update_and_lock(polling_island *p) { - gpr_mu_lock(&p->mu); - if (p->became != NULL) { - do { - polling_island *from = p; - p = p->became; - gpr_mu_lock(&p->mu); - bool delete_from = 0 == --from->refs; - p->refs++; - gpr_mu_unlock(&from->mu); - if (delete_from) { - polling_island_delete(from); - } - } while (p->became != NULL); - } - return p; -} - -static polling_island *polling_island_ref(polling_island *p) { - gpr_mu_lock(&p->mu); - gpr_mu_unlock(&p->mu); - return p; -} - -static void polling_island_drop(polling_island *p) {} - -static polling_island *polling_island_update(polling_island *p, - int updating_owner_count) { - p = polling_island_update_and_lock(p); - GPR_ASSERT(p->refs != 0); - p->refs += updating_owner_count; - gpr_mu_unlock(&p->mu); - return p; -} - -//////////////////////////////////////////////////////////////////////////////// -// FD implementation - -static gpr_mu g_fd_freelist_mu; -static grpc_fd *g_first_free_fd; - -static grpc_fd *fd_create(int fd, const char *name) { - grpc_fd *r = NULL; - gpr_mu_lock(&g_fd_freelist_mu); - if (g_first_free_fd == NULL) { - r = gpr_malloc(sizeof(*r)); - r->pollable.type = POLLABLE_FD; - gpr_atm_rel_store(&r->on_readable, 0); - gpr_atm_rel_store(&r->on_writable, 0); - gpr_mu_init(&r->polling_island_mu); - gpr_mu_init(&r->set_ready_mu); - } else { - r = g_first_free_fd; - g_first_free_fd = r->next_free; - } - gpr_mu_unlock(&g_fd_freelist_mu); - - r->pollable.fd = fd; - grpc_iomgr_register_object(&r->pollable.iomgr_object, name); - r->next_free = NULL; - return r; -} - -static int fd_wrapped_fd(grpc_fd *fd) { return fd->pollable.fd; } - -static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - grpc_closure *on_done, int *release_fd, - const char *reason) { - if (release_fd != NULL) { - *release_fd = fd->pollable.fd; - } else { - close(fd->pollable.fd); - } - - gpr_mu_lock(&fd->polling_island_mu); - if (fd->polling_island != NULL) { - polling_island_drop(fd->polling_island); - } - gpr_mu_unlock(&fd->polling_island_mu); - - gpr_mu_lock(&g_fd_freelist_mu); - fd->next_free = g_first_free_fd; - g_first_free_fd = fd; - grpc_iomgr_unregister_object(&fd->pollable.iomgr_object); - gpr_mu_unlock(&g_fd_freelist_mu); - - grpc_exec_ctx_enqueue(exec_ctx, on_done, true, NULL); -} - -static void notify_on(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - grpc_closure *closure, gpr_atm *state) { - if (gpr_atm_acq_cas(state, STATE_NOT_READY, (gpr_atm)closure)) { - // state was not ready, and is now the closure - we're done */ - } else { - // cas failed - we MUST be in STATE_READY (can't request two notifications - // for the same event) - // flip back to not ready, enqueue the closure directly - GPR_ASSERT(gpr_atm_rel_cas(state, STATE_READY, STATE_NOT_READY)); - grpc_exec_ctx_enqueue(exec_ctx, closure, true, NULL); - } -} - -static void fd_notify_on_read(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - grpc_closure *closure) { - notify_on(exec_ctx, fd, closure, &fd->on_readable); -} - -static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - grpc_closure *closure) { - notify_on(exec_ctx, fd, closure, &fd->on_readable); -} - -static void destroy_fd_freelist(void) { - while (g_first_free_fd) { - grpc_fd *next = g_first_free_fd->next_free; - gpr_mu_destroy(&g_first_free_fd->polling_island_mu); - gpr_free(next); - g_first_free_fd = next; - } -} - -static void set_ready_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - gpr_atm *state) { - if (gpr_atm_acq_cas(state, STATE_NOT_READY, STATE_READY)) { - // state was not ready, and is now ready - we're done - } else { - // cas failed - either there's a closure queued which we should consume OR - // the state was already STATE_READY - gpr_atm cur_state = gpr_atm_acq_load(state); - if (cur_state != STATE_READY) { - // state wasn't STATE_READY - it *must* have been a closure - // since it's illegal to ask for notification twice, it's safe to assume - // that we'll resume being the closure - GPR_ASSERT(gpr_atm_rel_cas(state, cur_state, STATE_NOT_READY)); - grpc_exec_ctx_enqueue(exec_ctx, (grpc_closure *)cur_state, !fd->shutdown, - NULL); - } - } -} - -static void fd_shutdown(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { - gpr_mu_lock(&fd->set_ready_mu); - GPR_ASSERT(!fd->shutdown); - fd->shutdown = 1; - set_ready_locked(exec_ctx, fd, &fd->on_readable); - set_ready_locked(exec_ctx, fd, &fd->on_writable); - gpr_mu_unlock(&fd->set_ready_mu); -} - -//////////////////////////////////////////////////////////////////////////////// -// Pollset implementation - -static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { - gpr_mu_init(&pollset->mu); - *mu = &pollset->mu; - pollset->polling_island = NULL; -} - -static void pollset_destroy(grpc_pollset *pollset) { - gpr_mu_destroy(&pollset->mu); - if (pollset->polling_island) { - polling_island_drop(pollset->polling_island); - } -} - -static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - struct grpc_fd *fd) { - gpr_mu_lock(&pollset->mu); - gpr_mu_lock(&fd->polling_island_mu); - - polling_island *new; - - if (fd->polling_island == NULL) { - if (pollset->polling_island == NULL) { - new = polling_island_create(fd); - } else { - new = polling_island_add(pollset->polling_island, fd); - } - } else if (pollset->polling_island == NULL) { - new = polling_island_ref(fd->polling_island); - } else if (pollset->polling_island != fd->polling_island) { - new = polling_island_merge(pollset->polling_island, fd->polling_island); - } else { - new = polling_island_update(pollset->polling_island, 1); - } - - fd->polling_island = pollset->polling_island = new; - - gpr_mu_unlock(&fd->polling_island_mu); - gpr_mu_unlock(&pollset->mu); -} - -//////////////////////////////////////////////////////////////////////////////// -// Engine binding - -static void shutdown_engine(void) { destroy_fd_freelist(); } - -static const grpc_event_engine_vtable vtable = { - .pollset_size = sizeof(grpc_pollset), - - .fd_create = fd_create, - .fd_wrapped_fd = fd_wrapped_fd, - .fd_orphan = fd_orphan, - .fd_shutdown = fd_shutdown, - .fd_notify_on_read = fd_notify_on_read, - .fd_notify_on_write = fd_notify_on_write, - - .pollset_init = pollset_init, - .pollset_shutdown = pollset_shutdown, - .pollset_reset = pollset_reset, - .pollset_destroy = pollset_destroy, - .pollset_work = pollset_work, - .pollset_kick = pollset_kick, - .pollset_add_fd = pollset_add_fd, - - .pollset_set_create = pollset_set_create, - .pollset_set_destroy = pollset_set_destroy, - .pollset_set_add_pollset = pollset_set_add_pollset, - .pollset_set_del_pollset = pollset_set_del_pollset, - .pollset_set_add_pollset_set = pollset_set_add_pollset_set, - .pollset_set_del_pollset_set = pollset_set_del_pollset_set, - .pollset_set_add_fd = pollset_set_add_fd, - .pollset_set_del_fd = pollset_set_del_fd, - - .kick_poller = kick_poller, - - .shutdown_engine = shutdown_engine, -}; - -static bool is_epoll_available(void) { - abort(); - return false; -} - -const grpc_event_engine_vtable *grpc_init_poll_posix(void) { - if (!is_epoll_available()) { - return NULL; - } - return &vtable; -} diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 3aa26109f22..61106faef90 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -35,7 +35,7 @@ #ifdef GPR_POSIX_SOCKET -#include "src/core/lib/iomgr/ev_epoll_posix.h" +#include "src/core/lib/iomgr/ev_epoll_linux.h" #include #include diff --git a/src/core/lib/iomgr/ev_epoll_posix.c b/src/core/lib/iomgr/ev_epoll_posix.c deleted file mode 100644 index 5abd5b2a94c..00000000000 --- a/src/core/lib/iomgr/ev_epoll_posix.c +++ /dev/null @@ -1,1209 +0,0 @@ -/* - * - * Copyright 2016, Google Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following disclaimer - * in the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Google Inc. nor the names of its - * contributors may be used to endorse or promote products derived from - * this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - */ - -#include - -#ifdef GPR_POSIX_SOCKET - -#include "src/core/lib/iomgr/ev_epoll_posix.h" - -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include -#include - -#include "src/core/lib/iomgr/ev_posix.h" -#include "src/core/lib/iomgr/iomgr_internal.h" -#include "src/core/lib/iomgr/wakeup_fd_posix.h" -#include "src/core/lib/profiling/timers.h" -#include "src/core/lib/support/block_annotate.h" - - -/******************************************************************************* - * FD declarations - */ - -struct grpc_fd { - int fd; - /* refst format: - bit0: 1=active/0=orphaned - bit1-n: refcount - meaning that mostly we ref by two to avoid altering the orphaned bit, - and just unref by 1 when we're ready to flag the object as orphaned */ - gpr_atm refst; - - gpr_mu mu; - int shutdown; - int closed; - int released; - - grpc_closure *read_closure; - grpc_closure *write_closure; - - struct grpc_fd *freelist_next; - - grpc_closure *on_done_closure; - - grpc_iomgr_object iomgr_object; -}; - -/* Return 1 if this fd is orphaned, 0 otherwise */ -static bool fd_is_orphaned(grpc_fd *fd); - -/* Reference counting for fds */ -/*#define GRPC_FD_REF_COUNT_DEBUG*/ -#ifdef GRPC_FD_REF_COUNT_DEBUG -static void fd_ref(grpc_fd *fd, const char *reason, const char *file, int line); -static void fd_unref(grpc_fd *fd, const char *reason, const char *file, - int line); -#define GRPC_FD_REF(fd, reason) fd_ref(fd, reason, __FILE__, __LINE__) -#define GRPC_FD_UNREF(fd, reason) fd_unref(fd, reason, __FILE__, __LINE__) -#else -static void fd_ref(grpc_fd *fd); -static void fd_unref(grpc_fd *fd); -#define GRPC_FD_REF(fd, reason) fd_ref(fd) -#define GRPC_FD_UNREF(fd, reason) fd_unref(fd) -#endif - -static void fd_global_init(void); -static void fd_global_shutdown(void); - -#define CLOSURE_NOT_READY ((grpc_closure *)0) -#define CLOSURE_READY ((grpc_closure *)1) - -/******************************************************************************* - * pollset declarations - */ - -typedef struct grpc_cached_wakeup_fd { - grpc_wakeup_fd fd; - struct grpc_cached_wakeup_fd *next; -} grpc_cached_wakeup_fd; - -struct grpc_pollset_worker { - grpc_cached_wakeup_fd *wakeup_fd; - int reevaluate_polling_on_wakeup; - int kicked_specifically; - pthread_t pt_id; - struct grpc_pollset_worker *next; - struct grpc_pollset_worker *prev; -}; - -struct grpc_pollset { - gpr_mu mu; - grpc_pollset_worker root_worker; - int shutting_down; - int called_shutdown; - int kicked_without_pollers; - grpc_closure *shutdown_done; - - int epoll_fd; - - /* Local cache of eventfds for workers */ - grpc_cached_wakeup_fd *local_wakeup_cache; -}; - -/* Add an fd to a pollset */ -static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - struct grpc_fd *fd); - -static void pollset_set_add_fd(grpc_exec_ctx *exec_ctx, - grpc_pollset_set *pollset_set, grpc_fd *fd); - -/* Convert a timespec to milliseconds: - - very small or negative poll times are clamped to zero to do a - non-blocking poll (which becomes spin polling) - - other small values are rounded up to one millisecond - - longer than a millisecond polls are rounded up to the next nearest - millisecond to avoid spinning - - infinite timeouts are converted to -1 */ -static int poll_deadline_to_millis_timeout(gpr_timespec deadline, - gpr_timespec now); - -/* Allow kick to wakeup the currently polling worker */ -#define GRPC_POLLSET_CAN_KICK_SELF 1 -/* Force the wakee to repoll when awoken */ -#define GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP 2 -/* As per pollset_kick, with an extended set of flags (defined above) - -- mostly for fd_posix's use. */ -static void pollset_kick_ext(grpc_pollset *p, - grpc_pollset_worker *specific_worker, - uint32_t flags); - -/* turn a pollset into a multipoller: platform specific */ -typedef void (*platform_become_multipoller_type)(grpc_exec_ctx *exec_ctx, - grpc_pollset *pollset, - struct grpc_fd **fds, - size_t fd_count); - -/* Return 1 if the pollset has active threads in pollset_work (pollset must - * be locked) */ -static int pollset_has_workers(grpc_pollset *pollset); - -static void remove_fd_from_all_epoll_sets(int fd); - -/******************************************************************************* - * pollset_set definitions - */ - -struct grpc_pollset_set { - gpr_mu mu; - - size_t pollset_count; - size_t pollset_capacity; - grpc_pollset **pollsets; - - size_t pollset_set_count; - size_t pollset_set_capacity; - struct grpc_pollset_set **pollset_sets; - - size_t fd_count; - size_t fd_capacity; - grpc_fd **fds; -}; - -/******************************************************************************* - * fd_posix.c - */ - -/* We need to keep a freelist not because of any concerns of malloc performance - * but instead so that implementations with multiple threads in (for example) - * epoll_wait deal with the race between pollset removal and incoming poll - * notifications. - * - * The problem is that the poller ultimately holds a reference to this - * object, so it is very difficult to know when is safe to free it, at least - * without some expensive synchronization. - * - * If we keep the object freelisted, in the worst case losing this race just - * becomes a spurious read notification on a reused fd. - */ -/* TODO(klempner): We could use some form of polling generation count to know - * when these are safe to free. */ -/* TODO(klempner): Consider disabling freelisting if we don't have multiple - * threads in poll on the same fd */ -/* TODO(klempner): Batch these allocations to reduce fragmentation */ -static grpc_fd *fd_freelist = NULL; -static gpr_mu fd_freelist_mu; - -static void freelist_fd(grpc_fd *fd) { - gpr_mu_lock(&fd_freelist_mu); - fd->freelist_next = fd_freelist; - fd_freelist = fd; - grpc_iomgr_unregister_object(&fd->iomgr_object); - gpr_mu_unlock(&fd_freelist_mu); -} - -static grpc_fd *alloc_fd(int fd) { - grpc_fd *r = NULL; - gpr_mu_lock(&fd_freelist_mu); - if (fd_freelist != NULL) { - r = fd_freelist; - fd_freelist = fd_freelist->freelist_next; - } - gpr_mu_unlock(&fd_freelist_mu); - if (r == NULL) { - r = gpr_malloc(sizeof(grpc_fd)); - gpr_mu_init(&r->mu); - } - - gpr_mu_lock(&r->mu); - gpr_atm_rel_store(&r->refst, 1); - r->shutdown = 0; - r->read_closure = CLOSURE_NOT_READY; - r->write_closure = CLOSURE_NOT_READY; - r->fd = fd; - r->freelist_next = NULL; - r->on_done_closure = NULL; - r->closed = 0; - r->released = 0; - gpr_mu_unlock(&r->mu); - return r; -} - -static void destroy(grpc_fd *fd) { - gpr_mu_destroy(&fd->mu); - gpr_free(fd); -} - -#ifdef GRPC_FD_REF_COUNT_DEBUG -#define REF_BY(fd, n, reason) ref_by(fd, n, reason, __FILE__, __LINE__) -#define UNREF_BY(fd, n, reason) unref_by(fd, n, reason, __FILE__, __LINE__) -static void ref_by(grpc_fd *fd, int n, const char *reason, const char *file, - int line) { - gpr_log(GPR_DEBUG, "FD %d %p ref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, - gpr_atm_no_barrier_load(&fd->refst), - gpr_atm_no_barrier_load(&fd->refst) + n, reason, file, line); -#else -#define REF_BY(fd, n, reason) ref_by(fd, n) -#define UNREF_BY(fd, n, reason) unref_by(fd, n) -static void ref_by(grpc_fd *fd, int n) { -#endif - GPR_ASSERT(gpr_atm_no_barrier_fetch_add(&fd->refst, n) > 0); -} - -#ifdef GRPC_FD_REF_COUNT_DEBUG -static void unref_by(grpc_fd *fd, int n, const char *reason, const char *file, - int line) { - gpr_atm old; - gpr_log(GPR_DEBUG, "FD %d %p unref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, - gpr_atm_no_barrier_load(&fd->refst), - gpr_atm_no_barrier_load(&fd->refst) - n, reason, file, line); -#else -static void unref_by(grpc_fd *fd, int n) { - gpr_atm old; -#endif - old = gpr_atm_full_fetch_add(&fd->refst, -n); - if (old == n) { - freelist_fd(fd); - } else { - GPR_ASSERT(old > n); - } -} - -static void fd_global_init(void) { gpr_mu_init(&fd_freelist_mu); } - -static void fd_global_shutdown(void) { - gpr_mu_lock(&fd_freelist_mu); - gpr_mu_unlock(&fd_freelist_mu); - while (fd_freelist != NULL) { - grpc_fd *fd = fd_freelist; - fd_freelist = fd_freelist->freelist_next; - destroy(fd); - } - gpr_mu_destroy(&fd_freelist_mu); -} - -static grpc_fd *fd_create(int fd, const char *name) { - grpc_fd *r = alloc_fd(fd); - char *name2; - gpr_asprintf(&name2, "%s fd=%d", name, fd); - grpc_iomgr_register_object(&r->iomgr_object, name2); - gpr_free(name2); -#ifdef GRPC_FD_REF_COUNT_DEBUG - gpr_log(GPR_DEBUG, "FD %d %p create %s", fd, r, name); -#endif - return r; -} - -static bool fd_is_orphaned(grpc_fd *fd) { - return (gpr_atm_acq_load(&fd->refst) & 1) == 0; -} - -static void close_fd_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { - fd->closed = 1; - if (!fd->released) { - close(fd->fd); - } else { - remove_fd_from_all_epoll_sets(fd->fd); - } - grpc_exec_ctx_enqueue(exec_ctx, fd->on_done_closure, true, NULL); -} - -static int fd_wrapped_fd(grpc_fd *fd) { - if (fd->released || fd->closed) { - return -1; - } else { - return fd->fd; - } -} - -static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - grpc_closure *on_done, int *release_fd, - const char *reason) { - fd->on_done_closure = on_done; - fd->released = release_fd != NULL; - if (!fd->released) { - shutdown(fd->fd, SHUT_RDWR); - } else { - *release_fd = fd->fd; - } - gpr_mu_lock(&fd->mu); - REF_BY(fd, 1, reason); /* remove active status, but keep referenced */ - close_fd_locked(exec_ctx, fd); - gpr_mu_unlock(&fd->mu); - UNREF_BY(fd, 2, reason); /* drop the reference */ -} - -/* increment refcount by two to avoid changing the orphan bit */ -#ifdef GRPC_FD_REF_COUNT_DEBUG -static void fd_ref(grpc_fd *fd, const char *reason, const char *file, - int line) { - ref_by(fd, 2, reason, file, line); -} - -static void fd_unref(grpc_fd *fd, const char *reason, const char *file, - int line) { - unref_by(fd, 2, reason, file, line); -} -#else -static void fd_ref(grpc_fd *fd) { ref_by(fd, 2); } - -static void fd_unref(grpc_fd *fd) { unref_by(fd, 2); } -#endif - -static void notify_on_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - grpc_closure **st, grpc_closure *closure) { - if (*st == CLOSURE_NOT_READY) { - /* not ready ==> switch to a waiting state by setting the closure */ - *st = closure; - } else if (*st == CLOSURE_READY) { - /* already ready ==> queue the closure to run immediately */ - *st = CLOSURE_NOT_READY; - grpc_exec_ctx_enqueue(exec_ctx, closure, !fd->shutdown, NULL); - } else { - /* upcallptr was set to a different closure. This is an error! */ - gpr_log(GPR_ERROR, - "User called a notify_on function with a previous callback still " - "pending"); - abort(); - } -} - -/* returns 1 if state becomes not ready */ -static int set_ready_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - grpc_closure **st) { - if (*st == CLOSURE_READY) { - /* duplicate ready ==> ignore */ - return 0; - } else if (*st == CLOSURE_NOT_READY) { - /* not ready, and not waiting ==> flag ready */ - *st = CLOSURE_READY; - return 0; - } else { - /* waiting ==> queue closure */ - grpc_exec_ctx_enqueue(exec_ctx, *st, !fd->shutdown, NULL); - *st = CLOSURE_NOT_READY; - return 1; - } -} - -static void fd_shutdown(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { - gpr_mu_lock(&fd->mu); - GPR_ASSERT(!fd->shutdown); - fd->shutdown = 1; - set_ready_locked(exec_ctx, fd, &fd->read_closure); - set_ready_locked(exec_ctx, fd, &fd->write_closure); - gpr_mu_unlock(&fd->mu); -} - -static void fd_notify_on_read(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - grpc_closure *closure) { - gpr_mu_lock(&fd->mu); - notify_on_locked(exec_ctx, fd, &fd->read_closure, closure); - gpr_mu_unlock(&fd->mu); -} - -static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - grpc_closure *closure) { - gpr_mu_lock(&fd->mu); - notify_on_locked(exec_ctx, fd, &fd->write_closure, closure); - gpr_mu_unlock(&fd->mu); -} - -/******************************************************************************* - * pollset_posix.c - */ - -GPR_TLS_DECL(g_current_thread_poller); -GPR_TLS_DECL(g_current_thread_worker); - -/** The alarm system needs to be able to wakeup 'some poller' sometimes - * (specifically when a new alarm needs to be triggered earlier than the next - * alarm 'epoch'). - * This wakeup_fd gives us something to alert on when such a case occurs. */ -grpc_wakeup_fd grpc_global_wakeup_fd; - -static void remove_worker(grpc_pollset *p, grpc_pollset_worker *worker) { - worker->prev->next = worker->next; - worker->next->prev = worker->prev; -} - -static int pollset_has_workers(grpc_pollset *p) { - return p->root_worker.next != &p->root_worker; -} - -static grpc_pollset_worker *pop_front_worker(grpc_pollset *p) { - if (pollset_has_workers(p)) { - grpc_pollset_worker *w = p->root_worker.next; - remove_worker(p, w); - return w; - } else { - return NULL; - } -} - -static void push_back_worker(grpc_pollset *p, grpc_pollset_worker *worker) { - worker->next = &p->root_worker; - worker->prev = worker->next->prev; - worker->prev->next = worker->next->prev = worker; -} - -static void push_front_worker(grpc_pollset *p, grpc_pollset_worker *worker) { - worker->prev = &p->root_worker; - worker->next = worker->prev->next; - worker->prev->next = worker->next->prev = worker; -} - -static void pollset_kick_ext(grpc_pollset *p, - grpc_pollset_worker *specific_worker, - uint32_t flags) { - GPR_TIMER_BEGIN("pollset_kick_ext", 0); - - /* pollset->mu already held */ - if (specific_worker != NULL) { - if (specific_worker == GRPC_POLLSET_KICK_BROADCAST) { - GPR_TIMER_BEGIN("pollset_kick_ext.broadcast", 0); - GPR_ASSERT((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) == 0); - for (specific_worker = p->root_worker.next; - specific_worker != &p->root_worker; - specific_worker = specific_worker->next) { - grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); - } - p->kicked_without_pollers = 1; - GPR_TIMER_END("pollset_kick_ext.broadcast", 0); - } else if (gpr_tls_get(&g_current_thread_worker) != - (intptr_t)specific_worker) { - GPR_TIMER_MARK("different_thread_worker", 0); - if ((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) != 0) { - specific_worker->reevaluate_polling_on_wakeup = 1; - } - specific_worker->kicked_specifically = 1; - grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); - /* TODO (sreek): Refactor this into a separate file*/ - pthread_kill(specific_worker->pt_id, SIGUSR1); - } else if ((flags & GRPC_POLLSET_CAN_KICK_SELF) != 0) { - GPR_TIMER_MARK("kick_yoself", 0); - if ((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) != 0) { - specific_worker->reevaluate_polling_on_wakeup = 1; - } - specific_worker->kicked_specifically = 1; - grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); - } - } else if (gpr_tls_get(&g_current_thread_poller) != (intptr_t)p) { - GPR_ASSERT((flags & GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) == 0); - GPR_TIMER_MARK("kick_anonymous", 0); - specific_worker = pop_front_worker(p); - if (specific_worker != NULL) { - if (gpr_tls_get(&g_current_thread_worker) == (intptr_t)specific_worker) { - GPR_TIMER_MARK("kick_anonymous_not_self", 0); - push_back_worker(p, specific_worker); - specific_worker = pop_front_worker(p); - if ((flags & GRPC_POLLSET_CAN_KICK_SELF) == 0 && - gpr_tls_get(&g_current_thread_worker) == - (intptr_t)specific_worker) { - push_back_worker(p, specific_worker); - specific_worker = NULL; - } - } - if (specific_worker != NULL) { - GPR_TIMER_MARK("finally_kick", 0); - push_back_worker(p, specific_worker); - grpc_wakeup_fd_wakeup(&specific_worker->wakeup_fd->fd); - } - } else { - GPR_TIMER_MARK("kicked_no_pollers", 0); - p->kicked_without_pollers = 1; - } - } - - GPR_TIMER_END("pollset_kick_ext", 0); -} - -static void pollset_kick(grpc_pollset *p, - grpc_pollset_worker *specific_worker) { - pollset_kick_ext(p, specific_worker, 0); -} - -/* global state management */ - -static void sig_handler(int sig_num) { - gpr_log(GPR_INFO, "Received signal %d", sig_num); -} - -static void pollset_global_init(void) { - gpr_tls_init(&g_current_thread_poller); - gpr_tls_init(&g_current_thread_worker); - grpc_wakeup_fd_init(&grpc_global_wakeup_fd); - signal(SIGUSR1, sig_handler); -} - -static void pollset_global_shutdown(void) { - grpc_wakeup_fd_destroy(&grpc_global_wakeup_fd); - gpr_tls_destroy(&g_current_thread_poller); - gpr_tls_destroy(&g_current_thread_worker); -} - -static void kick_poller(void) { grpc_wakeup_fd_wakeup(&grpc_global_wakeup_fd); } - -/* TODO: sreek. Try to Remove this forward declaration*/ -static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset); - -/* main interface */ - -static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { - gpr_mu_init(&pollset->mu); - *mu = &pollset->mu; - pollset->root_worker.next = pollset->root_worker.prev = &pollset->root_worker; - pollset->shutting_down = 0; - pollset->called_shutdown = 0; - pollset->kicked_without_pollers = 0; - pollset->local_wakeup_cache = NULL; - pollset->kicked_without_pollers = 0; - - multipoll_with_epoll_pollset_create_efd(pollset); -} - -/* TODO(sreek): Maybe merge multipoll_*_destroy() with pollset_destroy() - * function */ -static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset); - -static void pollset_destroy(grpc_pollset *pollset) { - GPR_ASSERT(!pollset_has_workers(pollset)); - - multipoll_with_epoll_pollset_destroy(pollset); - - while (pollset->local_wakeup_cache) { - grpc_cached_wakeup_fd *next = pollset->local_wakeup_cache->next; - grpc_wakeup_fd_destroy(&pollset->local_wakeup_cache->fd); - gpr_free(pollset->local_wakeup_cache); - pollset->local_wakeup_cache = next; - } - gpr_mu_destroy(&pollset->mu); -} - -static void pollset_reset(grpc_pollset *pollset) { - GPR_ASSERT(pollset->shutting_down); - GPR_ASSERT(!pollset_has_workers(pollset)); - pollset->shutting_down = 0; - pollset->called_shutdown = 0; - pollset->kicked_without_pollers = 0; -} - -/* TODO (sreek): Remove multipoll_with_epoll_finish_shutdown() declaration */ -static void multipoll_with_epoll_pollset_finish_shutdown(grpc_pollset *pollset); - -static void finish_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset) { - multipoll_with_epoll_pollset_finish_shutdown(pollset); - grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); -} - -/* TODO(sreek): Remove multipoll_with_epoll_*_maybe_work_and_unlock declaration - */ -static void multipoll_with_epoll_pollset_maybe_work_and_unlock( - grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker *worker, - gpr_timespec deadline, gpr_timespec now); - -static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_pollset_worker **worker_hdl, gpr_timespec now, - gpr_timespec deadline) { - grpc_pollset_worker worker; - *worker_hdl = &worker; - - /* pollset->mu already held */ - int added_worker = 0; - int locked = 1; - int queued_work = 0; - int keep_polling = 0; - GPR_TIMER_BEGIN("pollset_work", 0); - /* this must happen before we (potentially) drop pollset->mu */ - worker.next = worker.prev = NULL; - worker.reevaluate_polling_on_wakeup = 0; - if (pollset->local_wakeup_cache != NULL) { - worker.wakeup_fd = pollset->local_wakeup_cache; - pollset->local_wakeup_cache = worker.wakeup_fd->next; - } else { - worker.wakeup_fd = gpr_malloc(sizeof(*worker.wakeup_fd)); - grpc_wakeup_fd_init(&worker.wakeup_fd->fd); - } - worker.kicked_specifically = 0; - - /* TODO(sreek): Abstract this thread id stuff out into a separate file */ - worker.pt_id = pthread_self(); - /* If we're shutting down then we don't execute any extended work */ - if (pollset->shutting_down) { - GPR_TIMER_MARK("pollset_work.shutting_down", 0); - goto done; - } - /* Start polling, and keep doing so while we're being asked to - re-evaluate our pollers (this allows poll() based pollers to - ensure they don't miss wakeups) */ - keep_polling = 1; - while (keep_polling) { - keep_polling = 0; - if (!pollset->kicked_without_pollers) { - if (!added_worker) { - push_front_worker(pollset, &worker); - added_worker = 1; - gpr_tls_set(&g_current_thread_worker, (intptr_t)&worker); - } - gpr_tls_set(&g_current_thread_poller, (intptr_t)pollset); - GPR_TIMER_BEGIN("maybe_work_and_unlock", 0); - - multipoll_with_epoll_pollset_maybe_work_and_unlock( - exec_ctx, pollset, &worker, deadline, now); - - GPR_TIMER_END("maybe_work_and_unlock", 0); - locked = 0; - gpr_tls_set(&g_current_thread_poller, 0); - } else { - GPR_TIMER_MARK("pollset_work.kicked_without_pollers", 0); - pollset->kicked_without_pollers = 0; - } - /* Finished execution - start cleaning up. - Note that we may arrive here from outside the enclosing while() loop. - In that case we won't loop though as we haven't added worker to the - worker list, which means nobody could ask us to re-evaluate polling). */ - done: - if (!locked) { - queued_work |= grpc_exec_ctx_flush(exec_ctx); - gpr_mu_lock(&pollset->mu); - locked = 1; - } - /* If we're forced to re-evaluate polling (via pollset_kick with - GRPC_POLLSET_REEVALUATE_POLLING_ON_WAKEUP) then we land here and force - a loop */ - if (worker.reevaluate_polling_on_wakeup) { - worker.reevaluate_polling_on_wakeup = 0; - pollset->kicked_without_pollers = 0; - if (queued_work || worker.kicked_specifically) { - /* If there's queued work on the list, then set the deadline to be - immediate so we get back out of the polling loop quickly */ - deadline = gpr_inf_past(GPR_CLOCK_MONOTONIC); - } - keep_polling = 1; - } - } - if (added_worker) { - remove_worker(pollset, &worker); - gpr_tls_set(&g_current_thread_worker, 0); - } - /* release wakeup fd to the local pool */ - worker.wakeup_fd->next = pollset->local_wakeup_cache; - pollset->local_wakeup_cache = worker.wakeup_fd; - /* check shutdown conditions */ - if (pollset->shutting_down) { - if (pollset_has_workers(pollset)) { - pollset_kick(pollset, NULL); - } else if (!pollset->called_shutdown) { - pollset->called_shutdown = 1; - gpr_mu_unlock(&pollset->mu); - finish_shutdown(exec_ctx, pollset); - grpc_exec_ctx_flush(exec_ctx); - /* Continuing to access pollset here is safe -- it is the caller's - * responsibility to not destroy when it has outstanding calls to - * pollset_work. - * TODO(dklempner): Can we refactor the shutdown logic to avoid this? */ - gpr_mu_lock(&pollset->mu); - } - } - *worker_hdl = NULL; - GPR_TIMER_END("pollset_work", 0); -} - -static void pollset_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_closure *closure) { - GPR_ASSERT(!pollset->shutting_down); - pollset->shutting_down = 1; - pollset->shutdown_done = closure; - pollset_kick(pollset, GRPC_POLLSET_KICK_BROADCAST); - - if (!pollset->called_shutdown && !pollset_has_workers(pollset)) { - pollset->called_shutdown = 1; - finish_shutdown(exec_ctx, pollset); - } -} - -static int poll_deadline_to_millis_timeout(gpr_timespec deadline, - gpr_timespec now) { - gpr_timespec timeout; - static const int64_t max_spin_polling_us = 10; - if (gpr_time_cmp(deadline, gpr_inf_future(deadline.clock_type)) == 0) { - return -1; - } - if (gpr_time_cmp(deadline, gpr_time_add(now, gpr_time_from_micros( - max_spin_polling_us, - GPR_TIMESPAN))) <= 0) { - return 0; - } - timeout = gpr_time_sub(deadline, now); - return gpr_time_to_millis(gpr_time_add( - timeout, gpr_time_from_nanos(GPR_NS_PER_MS - 1, GPR_TIMESPAN))); -} - -/******************************************************************************* - * pollset_multipoller_with_epoll_posix.c - */ - -static void set_ready(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure **st) { - /* only one set_ready can be active at once (but there may be a racing - notify_on) */ - gpr_mu_lock(&fd->mu); - set_ready_locked(exec_ctx, fd, st); - gpr_mu_unlock(&fd->mu); -} - -static void fd_become_readable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { - set_ready(exec_ctx, fd, &fd->read_closure); -} - -static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { - set_ready(exec_ctx, fd, &fd->write_closure); -} - -/* TODO (sreek): Maybe this global list is not required. Double check*/ -struct epoll_fd_list { - int *epoll_fds; - size_t count; - size_t capacity; -}; - -static struct epoll_fd_list epoll_fd_global_list; -static gpr_once init_epoll_fd_list_mu = GPR_ONCE_INIT; -static gpr_mu epoll_fd_list_mu; - -static void init_mu(void) { gpr_mu_init(&epoll_fd_list_mu); } - -static void add_epoll_fd_to_global_list(int epoll_fd) { - gpr_once_init(&init_epoll_fd_list_mu, init_mu); - - gpr_mu_lock(&epoll_fd_list_mu); - if (epoll_fd_global_list.count == epoll_fd_global_list.capacity) { - epoll_fd_global_list.capacity = - GPR_MAX((size_t)8, epoll_fd_global_list.capacity * 2); - epoll_fd_global_list.epoll_fds = - gpr_realloc(epoll_fd_global_list.epoll_fds, - epoll_fd_global_list.capacity * sizeof(int)); - } - epoll_fd_global_list.epoll_fds[epoll_fd_global_list.count++] = epoll_fd; - gpr_mu_unlock(&epoll_fd_list_mu); -} - -static void remove_epoll_fd_from_global_list(int epoll_fd) { - gpr_mu_lock(&epoll_fd_list_mu); - GPR_ASSERT(epoll_fd_global_list.count > 0); - for (size_t i = 0; i < epoll_fd_global_list.count; i++) { - if (epoll_fd == epoll_fd_global_list.epoll_fds[i]) { - epoll_fd_global_list.epoll_fds[i] = - epoll_fd_global_list.epoll_fds[--(epoll_fd_global_list.count)]; - break; - } - } - gpr_mu_unlock(&epoll_fd_list_mu); -} - -static void remove_fd_from_all_epoll_sets(int fd) { - int err; - gpr_once_init(&init_epoll_fd_list_mu, init_mu); - gpr_mu_lock(&epoll_fd_list_mu); - if (epoll_fd_global_list.count == 0) { - gpr_mu_unlock(&epoll_fd_list_mu); - return; - } - for (size_t i = 0; i < epoll_fd_global_list.count; i++) { - err = epoll_ctl(epoll_fd_global_list.epoll_fds[i], EPOLL_CTL_DEL, fd, NULL); - if (err < 0 && errno != ENOENT) { - gpr_log(GPR_ERROR, "epoll_ctl del for %d failed: %s", fd, - strerror(errno)); - } - } - gpr_mu_unlock(&epoll_fd_list_mu); -} - -/* TODO: sreek - This function multipoll_with_epoll_pollset_add_fd() and - * finally_add_fd() in ev_poll_and_epoll_posix.c */ -static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_fd *fd) { - - /* TODO sreek - Check if we need to get a pollset->mu lock here */ - - struct epoll_event ev; - int err; - - /* Hold a ref to the fd to keep it from being closed during the add. This may - result in a spurious wakeup being assigned to this pollset whilst adding, - but that should be benign. */ - /* TODO: (sreek): Understand how a spurious wake up migh be assinged to this - * pollset..and how holding a reference will prevent the fd from being closed - * (and perhaps more importantly, see how can an fd be closed while being - * added to the epollset */ - GRPC_FD_REF(fd, "add fd"); - - gpr_mu_lock(&fd->mu); - if (fd->shutdown) { - gpr_mu_unlock(&fd->mu); - GRPC_FD_UNREF(fd, "add fd"); - return; - } - gpr_mu_unlock(&fd->mu); - - ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); - ev.data.ptr = fd; - err = epoll_ctl(pollset->epoll_fd, EPOLL_CTL_ADD, fd->fd, &ev); - if (err < 0) { - /* FDs may be added to a pollset multiple times, so EEXIST is normal. */ - if (errno != EEXIST) { - gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", fd->fd, - strerror(errno)); - } - } - - /* The fd might have been orphaned while we were adding it to the epoll set. - Close the fd in such a case (which will also take care of removing it from - the epoll set */ - gpr_mu_lock(&fd->mu); - if (fd_is_orphaned(fd) && !fd->closed) { - close_fd_locked(exec_ctx, fd); - } - gpr_mu_unlock(&fd->mu); - - GRPC_FD_UNREF(fd, "add fd"); -} - -/* Creates an epoll fd and initializes the pollset */ -/* TODO: This has to be called ONLY from pollset_init function. and hence it - * does not acquire any lock */ -static void multipoll_with_epoll_pollset_create_efd(grpc_pollset *pollset) { - struct epoll_event ev; - int err; - - pollset->epoll_fd = epoll_create1(EPOLL_CLOEXEC); - if (pollset->epoll_fd < 0) { - gpr_log(GPR_ERROR, "epoll_create1 failed: %s", strerror(errno)); - abort(); - } - add_epoll_fd_to_global_list(pollset->epoll_fd); - - ev.events = (uint32_t)(EPOLLIN | EPOLLET); - ev.data.ptr = NULL; - - err = epoll_ctl(pollset->epoll_fd, EPOLL_CTL_ADD, - GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), &ev); - if (err < 0) { - gpr_log(GPR_ERROR, "epoll_ctl add for %d failed: %s", - GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), - strerror(errno)); - } -} - -/* TODO(klempner): We probably want to turn this down a bit */ -#define GRPC_EPOLL_MAX_EVENTS 1000 - -static void multipoll_with_epoll_pollset_maybe_work_and_unlock( - grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker *worker, - gpr_timespec deadline, gpr_timespec now) { - struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; - int epoll_fd = pollset->epoll_fd; - int ep_rv; - int poll_rv; - int timeout_ms; - struct pollfd pfds[2]; - - /* If you want to ignore epoll's ability to sanely handle parallel pollers, - * for a more apples-to-apples performance comparison with poll, add a - * if (pollset->counter != 0) { return 0; } - * here. - */ - - gpr_mu_unlock(&pollset->mu); - - timeout_ms = poll_deadline_to_millis_timeout(deadline, now); - - pfds[0].fd = GRPC_WAKEUP_FD_GET_READ_FD(&worker->wakeup_fd->fd); - pfds[0].events = POLLIN; - pfds[0].revents = 0; - pfds[1].fd = epoll_fd; - pfds[1].events = POLLIN; - pfds[1].revents = 0; - - /* TODO(vpai): Consider first doing a 0 timeout poll here to avoid - even going into the blocking annotation if possible */ - GPR_TIMER_BEGIN("poll", 0); - GRPC_SCHEDULING_START_BLOCKING_REGION; - poll_rv = grpc_poll_function(pfds, 2, timeout_ms); - GRPC_SCHEDULING_END_BLOCKING_REGION; - GPR_TIMER_END("poll", 0); - - if (poll_rv < 0) { - if (errno != EINTR) { - gpr_log(GPR_ERROR, "poll() failed: %s", strerror(errno)); - } - } else if (poll_rv == 0) { - /* do nothing */ - } else { - if (pfds[0].revents) { - grpc_wakeup_fd_consume_wakeup(&worker->wakeup_fd->fd); - } - if (pfds[1].revents) { - do { - /* The following epoll_wait never blocks; it has a timeout of 0 */ - ep_rv = epoll_wait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, 0); - if (ep_rv < 0) { - if (errno != EINTR) { - gpr_log(GPR_ERROR, "epoll_wait() failed: %s", strerror(errno)); - } - } else { - int i; - for (i = 0; i < ep_rv; ++i) { - grpc_fd *fd = ep_ev[i].data.ptr; - /* TODO(klempner): We might want to consider making err and pri - * separate events */ - int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); - int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); - int write_ev = ep_ev[i].events & EPOLLOUT; - if (fd == NULL) { - grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); - } else { - if (read_ev || cancel) { - fd_become_readable(exec_ctx, fd); - } - if (write_ev || cancel) { - fd_become_writable(exec_ctx, fd); - } - } - } - } - } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); - } - } -} - -static void multipoll_with_epoll_pollset_finish_shutdown( - grpc_pollset *pollset) {} - -static void multipoll_with_epoll_pollset_destroy(grpc_pollset *pollset) { - close(pollset->epoll_fd); - remove_epoll_fd_from_global_list(pollset->epoll_fd); -} - -/******************************************************************************* - * pollset_set_posix.c - */ - -static grpc_pollset_set *pollset_set_create(void) { - grpc_pollset_set *pollset_set = gpr_malloc(sizeof(*pollset_set)); - memset(pollset_set, 0, sizeof(*pollset_set)); - gpr_mu_init(&pollset_set->mu); - return pollset_set; -} - -static void pollset_set_destroy(grpc_pollset_set *pollset_set) { - size_t i; - gpr_mu_destroy(&pollset_set->mu); - for (i = 0; i < pollset_set->fd_count; i++) { - GRPC_FD_UNREF(pollset_set->fds[i], "pollset_set"); - } - gpr_free(pollset_set->pollsets); - gpr_free(pollset_set->pollset_sets); - gpr_free(pollset_set->fds); - gpr_free(pollset_set); -} - -static void pollset_set_add_pollset(grpc_exec_ctx *exec_ctx, - grpc_pollset_set *pollset_set, - grpc_pollset *pollset) { - size_t i, j; - gpr_mu_lock(&pollset_set->mu); - if (pollset_set->pollset_count == pollset_set->pollset_capacity) { - pollset_set->pollset_capacity = - GPR_MAX(8, 2 * pollset_set->pollset_capacity); - pollset_set->pollsets = - gpr_realloc(pollset_set->pollsets, pollset_set->pollset_capacity * - sizeof(*pollset_set->pollsets)); - } - pollset_set->pollsets[pollset_set->pollset_count++] = pollset; - for (i = 0, j = 0; i < pollset_set->fd_count; i++) { - if (fd_is_orphaned(pollset_set->fds[i])) { - GRPC_FD_UNREF(pollset_set->fds[i], "pollset_set"); - } else { - pollset_add_fd(exec_ctx, pollset, pollset_set->fds[i]); - pollset_set->fds[j++] = pollset_set->fds[i]; - } - } - pollset_set->fd_count = j; - gpr_mu_unlock(&pollset_set->mu); -} - -static void pollset_set_del_pollset(grpc_exec_ctx *exec_ctx, - grpc_pollset_set *pollset_set, - grpc_pollset *pollset) { - size_t i; - gpr_mu_lock(&pollset_set->mu); - for (i = 0; i < pollset_set->pollset_count; i++) { - if (pollset_set->pollsets[i] == pollset) { - pollset_set->pollset_count--; - GPR_SWAP(grpc_pollset *, pollset_set->pollsets[i], - pollset_set->pollsets[pollset_set->pollset_count]); - break; - } - } - gpr_mu_unlock(&pollset_set->mu); -} - -static void pollset_set_add_pollset_set(grpc_exec_ctx *exec_ctx, - grpc_pollset_set *bag, - grpc_pollset_set *item) { - size_t i, j; - gpr_mu_lock(&bag->mu); - if (bag->pollset_set_count == bag->pollset_set_capacity) { - bag->pollset_set_capacity = GPR_MAX(8, 2 * bag->pollset_set_capacity); - bag->pollset_sets = - gpr_realloc(bag->pollset_sets, - bag->pollset_set_capacity * sizeof(*bag->pollset_sets)); - } - bag->pollset_sets[bag->pollset_set_count++] = item; - for (i = 0, j = 0; i < bag->fd_count; i++) { - if (fd_is_orphaned(bag->fds[i])) { - GRPC_FD_UNREF(bag->fds[i], "pollset_set"); - } else { - pollset_set_add_fd(exec_ctx, item, bag->fds[i]); - bag->fds[j++] = bag->fds[i]; - } - } - bag->fd_count = j; - gpr_mu_unlock(&bag->mu); -} - -static void pollset_set_del_pollset_set(grpc_exec_ctx *exec_ctx, - grpc_pollset_set *bag, - grpc_pollset_set *item) { - size_t i; - gpr_mu_lock(&bag->mu); - for (i = 0; i < bag->pollset_set_count; i++) { - if (bag->pollset_sets[i] == item) { - bag->pollset_set_count--; - GPR_SWAP(grpc_pollset_set *, bag->pollset_sets[i], - bag->pollset_sets[bag->pollset_set_count]); - break; - } - } - gpr_mu_unlock(&bag->mu); -} - -static void pollset_set_add_fd(grpc_exec_ctx *exec_ctx, - grpc_pollset_set *pollset_set, grpc_fd *fd) { - size_t i; - gpr_mu_lock(&pollset_set->mu); - if (pollset_set->fd_count == pollset_set->fd_capacity) { - pollset_set->fd_capacity = GPR_MAX(8, 2 * pollset_set->fd_capacity); - pollset_set->fds = gpr_realloc( - pollset_set->fds, pollset_set->fd_capacity * sizeof(*pollset_set->fds)); - } - GRPC_FD_REF(fd, "pollset_set"); - pollset_set->fds[pollset_set->fd_count++] = fd; - for (i = 0; i < pollset_set->pollset_count; i++) { - pollset_add_fd(exec_ctx, pollset_set->pollsets[i], fd); - } - for (i = 0; i < pollset_set->pollset_set_count; i++) { - pollset_set_add_fd(exec_ctx, pollset_set->pollset_sets[i], fd); - } - gpr_mu_unlock(&pollset_set->mu); -} - -static void pollset_set_del_fd(grpc_exec_ctx *exec_ctx, - grpc_pollset_set *pollset_set, grpc_fd *fd) { - size_t i; - gpr_mu_lock(&pollset_set->mu); - for (i = 0; i < pollset_set->fd_count; i++) { - if (pollset_set->fds[i] == fd) { - pollset_set->fd_count--; - GPR_SWAP(grpc_fd *, pollset_set->fds[i], - pollset_set->fds[pollset_set->fd_count]); - GRPC_FD_UNREF(fd, "pollset_set"); - break; - } - } - for (i = 0; i < pollset_set->pollset_set_count; i++) { - pollset_set_del_fd(exec_ctx, pollset_set->pollset_sets[i], fd); - } - gpr_mu_unlock(&pollset_set->mu); -} - -/******************************************************************************* - * event engine binding - */ - -static void shutdown_engine(void) { - fd_global_shutdown(); - pollset_global_shutdown(); -} - -static const grpc_event_engine_vtable vtable = { - .pollset_size = sizeof(grpc_pollset), - - .fd_create = fd_create, - .fd_wrapped_fd = fd_wrapped_fd, - .fd_orphan = fd_orphan, - .fd_shutdown = fd_shutdown, - .fd_notify_on_read = fd_notify_on_read, - .fd_notify_on_write = fd_notify_on_write, - - .pollset_init = pollset_init, - .pollset_shutdown = pollset_shutdown, - .pollset_reset = pollset_reset, - .pollset_destroy = pollset_destroy, - .pollset_work = pollset_work, - .pollset_kick = pollset_kick, - .pollset_add_fd = pollset_add_fd, - - .pollset_set_create = pollset_set_create, - .pollset_set_destroy = pollset_set_destroy, - .pollset_set_add_pollset = pollset_set_add_pollset, - .pollset_set_del_pollset = pollset_set_del_pollset, - .pollset_set_add_pollset_set = pollset_set_add_pollset_set, - .pollset_set_del_pollset_set = pollset_set_del_pollset_set, - .pollset_set_add_fd = pollset_set_add_fd, - .pollset_set_del_fd = pollset_set_del_fd, - - .kick_poller = kick_poller, - - .shutdown_engine = shutdown_engine, -}; - -const grpc_event_engine_vtable *grpc_init_epoll_posix(void) { - fd_global_init(); - pollset_global_init(); - return &vtable; -} - -#endif diff --git a/src/core/lib/iomgr/ev_epoll_posix.h b/src/core/lib/iomgr/ev_epoll_posix.h deleted file mode 100644 index 35319b4fc5d..00000000000 --- a/src/core/lib/iomgr/ev_epoll_posix.h +++ /dev/null @@ -1,41 +0,0 @@ -/* - * - * Copyright 2015, Google Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following disclaimer - * in the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Google Inc. nor the names of its - * contributors may be used to endorse or promote products derived from - * this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - */ - -#ifndef GRPC_CORE_LIB_IOMGR_EV_EPOLL_POSIX_H -#define GRPC_CORE_LIB_IOMGR_EV_EPOLL_POSIX_H - -#include "src/core/lib/iomgr/ev_posix.h" - -const grpc_event_engine_vtable *grpc_init_epoll_posix(void); - -#endif /* GRPC_CORE_LIB_IOMGR_EV_EPOLL_POSIX_H */ diff --git a/src/python/grpcio/grpc_core_dependencies.py b/src/python/grpcio/grpc_core_dependencies.py index 13bc6888d66..50a6f196d8d 100644 --- a/src/python/grpcio/grpc_core_dependencies.py +++ b/src/python/grpcio/grpc_core_dependencies.py @@ -95,7 +95,6 @@ CORE_SOURCE_FILES = [ 'src/core/lib/iomgr/endpoint_pair_posix.c', 'src/core/lib/iomgr/endpoint_pair_windows.c', 'src/core/lib/iomgr/ev_epoll_linux.c', - 'src/core/lib/iomgr/ev_epoll_posix.c', 'src/core/lib/iomgr/ev_poll_posix.c', 'src/core/lib/iomgr/ev_posix.c', 'src/core/lib/iomgr/exec_ctx.c', diff --git a/test/core/network_benchmarks/epoll_test.c b/test/core/network_benchmarks/epoll_test.c deleted file mode 100644 index a918dd9bb94..00000000000 --- a/test/core/network_benchmarks/epoll_test.c +++ /dev/null @@ -1,263 +0,0 @@ -/* - * - * Copyright 2015, Google Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following disclaimer - * in the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Google Inc. nor the names of its - * contributors may be used to endorse or promote products derived from - * this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - */ - -/* TODO: sreek: REMOVE THIS FILE */ - -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include - -int g_signal_num = SIGUSR1; - -int g_timeout_secs = 2; - -int g_eventfd_create = 1; -int g_eventfd_wakeup = 0; -int g_eventfd_teardown = 0; -int g_close_epoll_fd = 1; - -typedef struct thread_args { - gpr_thd_id id; - int epoll_fd; - int thread_num; -} thread_args; - -static int eventfd_create() { - if (!g_eventfd_create) { - return -1; - } - - int efd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); - GPR_ASSERT(efd >= 0); - return efd; -} - -static void eventfd_wakeup(int efd) { - if (!g_eventfd_wakeup) { - return; - } - - int err; - do { - err = eventfd_write(efd, 1); - } while (err < 0 && errno == EINTR); -} - -static void epoll_teardown(int epoll_fd, int fd) { - if (!g_eventfd_teardown) { - return; - } - - if (epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, NULL) < 0) { - if (errno != ENOENT) { - gpr_log(GPR_ERROR, "epoll_ctl: %s", strerror(errno)); - GPR_ASSERT(0); - } - } -} - -/* Special case for epoll, where we need to create the fd ahead of time. */ -static int epoll_setup(int fd) { - int epoll_fd; - struct epoll_event ev; - - epoll_fd = epoll_create(1); - if (epoll_fd < 0) { - gpr_log(GPR_ERROR, "epoll_create: %s", strerror(errno)); - return -1; - } - - ev.events = (uint32_t)EPOLLIN; - ev.data.fd = fd; - if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) < 0) { - if (errno != EEXIST) { - gpr_log(GPR_ERROR, "epoll_ctl: %s", strerror(errno)); - return -1; - } - - gpr_log(GPR_ERROR, "epoll_ctl: The fd %d already exists", fd); - } - - return epoll_fd; -} - -#define GRPC_EPOLL_MAX_EVENTS 1000 -static void thread_main(void *args) { - int ep_rv; - struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; - int fd; - int i; - int cancel; - int read; - int write; - thread_args *thd_args = args; - sigset_t new_mask; - sigset_t orig_mask; - int keep_polling = 0; - - gpr_log(GPR_INFO, "Thread: %d Started", thd_args->thread_num); - - do { - keep_polling = 0; - - /* Mask the signal before getting the epoll_fd */ - gpr_log(GPR_INFO, "Thread: %d Blocking signal: %d", thd_args->thread_num, - g_signal_num); - sigemptyset(&new_mask); - sigaddset(&new_mask, g_signal_num); - pthread_sigmask(SIG_BLOCK, &new_mask, &orig_mask); - - gpr_log(GPR_INFO, "Thread: %d Waiting on epoll_wait()", - thd_args->thread_num); - ep_rv = epoll_pwait(thd_args->epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, - g_timeout_secs * 5000, &orig_mask); - gpr_log(GPR_INFO, "Thread: %d out of epoll_wait. ep_rv = %d", - thd_args->thread_num, ep_rv); - - if (ep_rv < 0) { - if (errno != EINTR) { - gpr_log(GPR_ERROR, "Thread: %d. epoll_wait failed with error: %d", - thd_args->thread_num, errno); - } else { - gpr_log(GPR_INFO, - "Thread: %d. epoll_wait was interrupted. Polling again >>>>>>>", - thd_args->thread_num); - keep_polling = 1; - } - } else { - if (ep_rv == 0) { - gpr_log(GPR_INFO, - "Thread: %d - epoll_wait returned 0. Most likely a timeout. " - "Polling again", - thd_args->thread_num); - keep_polling = 1; - } - - for (i = 0; i < ep_rv; i++) { - fd = ep_ev[i].data.fd; - cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); - read = ep_ev[i].events & (EPOLLIN | EPOLLPRI); - write = ep_ev[i].events & EPOLLOUT; - gpr_log(GPR_INFO, - "Thread: %d. epoll_wait returned that fd: %d has event of " - "interest. read: %d, write: %d, cancel: %d", - thd_args->thread_num, fd, read, write, cancel); - } - } - } while (keep_polling); -} - -static void close_fd(int fd) { - if (!g_close_epoll_fd) { - return; - } - - gpr_log(GPR_INFO, "*** Closing fd : %d ****", fd); - close(fd); - gpr_log(GPR_INFO, "*** Closed fd : %d ****", fd); -} - - -static void sig_handler(int sig_num) { - gpr_log(GPR_INFO, "<<<<< Received signal %d", sig_num); -} - -static void set_signal_handler() { - gpr_log(GPR_INFO, "Setting signal handler"); - signal(g_signal_num, sig_handler); -} - -#define NUM_THREADS 2 -int main(int argc, char **argv) { - int efd; - int epoll_fd; - int i; - thread_args thd_args[NUM_THREADS]; - gpr_thd_options options = gpr_thd_options_default(); - - set_signal_handler(); - - gpr_log(GPR_INFO, "Starting.."); - efd = eventfd_create(); - gpr_log(GPR_INFO, "Created event fd: %d", efd); - epoll_fd = epoll_setup(efd); - gpr_log(GPR_INFO, "Created epoll_fd: %d", epoll_fd); - - gpr_thd_options_set_joinable(&options); - for (i = 0; i < NUM_THREADS; i++) { - thd_args[i].thread_num = i; - thd_args[i].epoll_fd = epoll_fd; - gpr_log(GPR_INFO, "Starting thread: %d", i); - gpr_thd_new(&thd_args[i].id, thread_main, &thd_args[i], &options); - } - - sleep((unsigned)g_timeout_secs * 2); - - /* Send signals first */ - for (i = 0; i < NUM_THREADS; i++) { - gpr_log(GPR_INFO, "Sending signal to thread: %d", thd_args->thread_num); - pthread_kill(thd_args[i].id, g_signal_num); - gpr_log(GPR_INFO, "Sent signal to thread: %d >>>>>> ", - thd_args->thread_num); - } - - sleep((unsigned)g_timeout_secs * 2); - - close_fd(epoll_fd); - - sleep((unsigned)g_timeout_secs * 2); - - eventfd_wakeup(efd); - epoll_teardown(epoll_fd, efd); - - for (i = 0; i < NUM_THREADS; i++) { - gpr_thd_join(thd_args[i].id); - gpr_log(GPR_INFO, "Thread: %d joined", i); - } - - return 0; -} diff --git a/test/core/network_benchmarks/low_level_ping_pong.c b/test/core/network_benchmarks/low_level_ping_pong.c index b72a07778e0..1b40895a719 100644 --- a/test/core/network_benchmarks/low_level_ping_pong.c +++ b/test/core/network_benchmarks/low_level_ping_pong.c @@ -44,7 +44,6 @@ #include #include #include -#include #ifdef __linux__ #include #endif @@ -85,7 +84,6 @@ typedef struct thread_args { static int read_bytes(int fd, char *buf, size_t read_size, int spin) { size_t bytes_read = 0; ssize_t err; - do { err = read(fd, buf + bytes_read, read_size - bytes_read); if (err < 0) { diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index d968278f2a4..21ee1e6ff8a 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -808,7 +808,6 @@ src/core/lib/iomgr/closure.h \ src/core/lib/iomgr/endpoint.h \ src/core/lib/iomgr/endpoint_pair.h \ src/core/lib/iomgr/ev_epoll_linux.h \ -src/core/lib/iomgr/ev_epoll_posix.h \ src/core/lib/iomgr/ev_poll_posix.h \ src/core/lib/iomgr/ev_posix.h \ src/core/lib/iomgr/exec_ctx.h \ @@ -957,7 +956,6 @@ src/core/lib/iomgr/endpoint.c \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ src/core/lib/iomgr/ev_epoll_linux.c \ -src/core/lib/iomgr/ev_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ src/core/lib/iomgr/exec_ctx.c \ diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 85b71a8255a..304a0e1e3a9 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -301,22 +301,6 @@ "third_party": false, "type": "target" }, - { - "deps": [ - "gpr", - "gpr_test_util", - "grpc", - "grpc_test_util" - ], - "headers": [], - "language": "c", - "name": "epoll_test", - "src": [ - "test/core/network_benchmarks/epoll_test.c" - ], - "third_party": false, - "type": "target" - }, { "deps": [ "gpr", @@ -5547,7 +5531,6 @@ "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", "src/core/lib/iomgr/ev_epoll_linux.h", - "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", "src/core/lib/iomgr/exec_ctx.h", @@ -5649,8 +5632,6 @@ "src/core/lib/iomgr/endpoint_pair_windows.c", "src/core/lib/iomgr/ev_epoll_linux.c", "src/core/lib/iomgr/ev_epoll_linux.h", - "src/core/lib/iomgr/ev_epoll_posix.c", - "src/core/lib/iomgr/ev_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.c", diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index be7b72f61d2..850f9474aec 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -356,21 +356,6 @@ "windows" ] }, - { - "args": [], - "ci_platforms": [ - "linux" - ], - "cpu_cost": 1.0, - "exclude_configs": [], - "flaky": false, - "gtest": false, - "language": "c", - "name": "epoll_test", - "platforms": [ - "linux" - ] - }, { "args": [], "ci_platforms": [ diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj b/vsprojects/vcxproj/grpc/grpc.vcxproj index a67e4d16dad..ce523725e8e 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj @@ -317,7 +317,6 @@ - @@ -487,8 +486,6 @@ - - diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters index bf9b7dc7dcf..d46676f2296 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters @@ -58,9 +58,6 @@ src\core\lib\iomgr - - src\core\lib\iomgr - src\core\lib\iomgr @@ -683,9 +680,6 @@ src\core\lib\iomgr - - src\core\lib\iomgr - src\core\lib\iomgr diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj index afc9a2ca1b2..d4dd428c2db 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj @@ -305,7 +305,6 @@ - @@ -453,8 +452,6 @@ - - diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters index b7507f9a963..d14e7e7ab41 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters @@ -61,9 +61,6 @@ src\core\lib\iomgr - - src\core\lib\iomgr - src\core\lib\iomgr @@ -581,9 +578,6 @@ src\core\lib\iomgr - - src\core\lib\iomgr - src\core\lib\iomgr From 9bc3d2d67f32f4dad8cd1319dd4f3fce48c1abee Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Mon, 6 Jun 2016 10:27:56 -0700 Subject: [PATCH 0057/1039] Minor comments --- src/core/lib/iomgr/ev_epoll_linux.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 61106faef90..d3abf3bd84f 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -144,10 +144,9 @@ typedef struct polling_island { /******************************************************************************* * Pollset Declarations */ - struct grpc_pollset_worker { int kicked_specifically; - pthread_t pt_id; /* TODO (sreek) - Add an abstraction here */ + pthread_t pt_id; /* Thread id of this worker */ struct grpc_pollset_worker *next; struct grpc_pollset_worker *prev; }; @@ -483,8 +482,7 @@ polling_island *polling_island_merge(polling_island *p, polling_island *q) { /* Get locks on both the polling islands */ polling_island_pair_update_and_lock(&p, &q); - /* TODO: sreek: Think about this scenario some more. Is it possible ?. what - * does it mean, when would this happen */ + /* TODO: sreek: Think about this scenario some more */ if (p == q) { /* Nothing needs to be done here */ gpr_mu_unlock(&p->mu); @@ -539,7 +537,10 @@ static void polling_island_global_init() { * (specifically when a new alarm needs to be triggered earlier than the next * alarm 'epoch'). This wakeup_fd gives us something to alert on when such a * case occurs. */ -/* TODO: sreek: Right now, this wakes up all pollers */ + +/* TODO: sreek: Right now, this wakes up all pollers. In future we should make + * sure to wake up one polling thread (which can wake up other threads if + * needed) */ grpc_wakeup_fd grpc_global_wakeup_fd; static grpc_fd *fd_freelist = NULL; @@ -676,7 +677,6 @@ static int fd_wrapped_fd(grpc_fd *fd) { static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure *on_done, int *release_fd, const char *reason) { - /* TODO(sreek) In ev_poll_posix.c,the lock is acquired a little later. Why? */ bool is_fd_closed = false; gpr_mu_lock(&fd->mu); fd->on_done_closure = on_done; @@ -784,8 +784,9 @@ static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, */ static void sig_handler(int sig_num) { - /* TODO: sreek - Remove this expensive log line */ +#ifdef GPRC_EPOLL_DEBUG gpr_log(GPR_INFO, "Received signal %d", sig_num); +#endif } /* Global state management */ @@ -986,7 +987,10 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, if (ep_rv < 0) { if (errno != EINTR) { - /* TODO (sreek) - Check for bad file descriptor error */ + /* TODO (sreek) - Do not log an error in case of bad file descriptor + * (A bad file descriptor here would just mean that the epoll set was + * merged with another epoll set and that the current epoll_fd is + * closed) */ gpr_log(GPR_ERROR, "epoll_pwait() failed: %s", strerror(errno)); } else { gpr_log(GPR_DEBUG, "pollset_work_and_unlock: 0-timeout epoll_wait()"); @@ -1062,7 +1066,9 @@ static void pollset_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, GPR_TIMER_END("pollset_shutdown", 0); } -/* TODO(sreek) Is pollset_shutdown() guranteed to be called before this? */ +/* pollset_shutdown is guaranteed to be called before pollset_destroy. So other + * than destroying the mutexes, there is nothing special that needs to be done + * here */ static void pollset_destroy(grpc_pollset *pollset) { GPR_ASSERT(!pollset_has_workers(pollset)); gpr_mu_destroy(&pollset->pi_mu); @@ -1075,7 +1081,7 @@ static void pollset_reset(grpc_pollset *pollset) { pollset->shutting_down = false; pollset->finish_shutdown_called = false; pollset->kicked_without_pollers = false; - /* TODO(sreek) - Should pollset->shutdown closure be set to NULL here? */ + pollset->shutdown_done = NULL; pollset_release_polling_island_locked(pollset); } @@ -1149,7 +1155,7 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { gpr_log(GPR_DEBUG, "pollset_add_fd: pollset: %p, fd: %d", pollset, fd->fd); - /* TODO sreek - Check if we need to get a pollset->mu lock here */ + /* TODO sreek - Double check if we need to get a pollset->mu lock here */ gpr_mu_lock(&pollset->pi_mu); gpr_mu_lock(&fd->pi_mu); From ec1588ba87d665cf629a8893d6070688a73ea7ef Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Mon, 6 Jun 2016 15:37:45 -0700 Subject: [PATCH 0058/1039] Ruby: Moved completion queue entirely into extension code --- src/ruby/ext/grpc/rb_call.c | 100 ++++++++--------- src/ruby/ext/grpc/rb_completion_queue.c | 83 ++------------ src/ruby/ext/grpc/rb_completion_queue.h | 9 +- src/ruby/ext/grpc/rb_grpc.c | 4 +- src/ruby/ext/grpc/rb_server.c | 138 ++++++++++++------------ 5 files changed, 126 insertions(+), 208 deletions(-) diff --git a/src/ruby/ext/grpc/rb_call.c b/src/ruby/ext/grpc/rb_call.c index b436057c163..f2c567c7da7 100644 --- a/src/ruby/ext/grpc/rb_call.c +++ b/src/ruby/ext/grpc/rb_call.c @@ -63,23 +63,10 @@ static VALUE grpc_rb_sBatchResult; * grpc_metadata_array. */ static VALUE grpc_rb_cMdAry; -/* id_cq is the name of the hidden ivar that preserves a reference to a - * completion queue */ -static ID id_cq; - -/* id_flags is the name of the hidden ivar that preserves the value of - * the flags used to create metadata from a Hash */ -static ID id_flags; - /* id_credentials is the name of the hidden ivar that preserves the value * of the credentials added to the call */ static ID id_credentials; -/* id_input_md is the name of the hidden ivar that preserves the hash used to - * create metadata, so that references to the strings it contains last as long - * as the call the metadata is added to. */ -static ID id_input_md; - /* id_metadata is name of the attribute used to access the metadata hash * received by the call and subsequently saved on it. */ static ID id_metadata; @@ -101,14 +88,23 @@ static VALUE sym_message; static VALUE sym_status; static VALUE sym_cancelled; +typedef struct grpc_rb_call { + grpc_call *wrapped; + grpc_completion_queue *queue; +} grpc_rb_call; + +static void destroy_call(grpc_rb_call *call) { + call = (grpc_rb_call *)p; + grpc_call_destroy(call->wrapped); + grpc_rb_completion_queue_destroy(call->queue); +} + /* Destroys a Call. */ static void grpc_rb_call_destroy(void *p) { - grpc_call* call = NULL; if (p == NULL) { return; } - call = (grpc_call *)p; - grpc_call_destroy(call); + destroy_call((grpc_rb_call*)p); } static size_t md_ary_datasize(const void *p) { @@ -167,15 +163,15 @@ const char *grpc_call_error_detail_of(grpc_call_error err) { /* Called by clients to cancel an RPC on the server. Can be called multiple times, from any thread. */ static VALUE grpc_rb_call_cancel(VALUE self) { - grpc_call *call = NULL; + grpc_rb_call *call = NULL; grpc_call_error err; if (RTYPEDDATA_DATA(self) == NULL) { //This call has been closed return Qnil; } - TypedData_Get_Struct(self, grpc_call, &grpc_call_data_type, call); - err = grpc_call_cancel(call, NULL); + TypedData_Get_Struct(self, grpc_rb_call, &grpc_call_data_type, call); + err = grpc_call_cancel(call->wrapped, NULL); if (err != GRPC_CALL_OK) { rb_raise(grpc_rb_eCallError, "cancel failed: %s (code=%d)", grpc_call_error_detail_of(err), err); @@ -189,10 +185,10 @@ static VALUE grpc_rb_call_cancel(VALUE self) { processed. */ static VALUE grpc_rb_call_close(VALUE self) { - grpc_call *call = NULL; - TypedData_Get_Struct(self, grpc_call, &grpc_call_data_type, call); + grpc_rb_call *call = NULL; + TypedData_Get_Struct(self, grpc_rb_call, &grpc_call_data_type, call); if(call != NULL) { - grpc_call_destroy(call); + destroy_call(call); RTYPEDDATA_DATA(self) = NULL; } return Qnil; @@ -201,14 +197,14 @@ static VALUE grpc_rb_call_close(VALUE self) { /* Called to obtain the peer that this call is connected to. */ static VALUE grpc_rb_call_get_peer(VALUE self) { VALUE res = Qnil; - grpc_call *call = NULL; + grpc_rb_call *call = NULL; char *peer = NULL; if (RTYPEDDATA_DATA(self) == NULL) { rb_raise(grpc_rb_eCallError, "Cannot get peer value on closed call"); return Qnil; } - TypedData_Get_Struct(self, grpc_call, &grpc_call_data_type, call); - peer = grpc_call_get_peer(call); + TypedData_Get_Struct(self, grpc_rb_call, &grpc_call_data_type, call); + peer = grpc_call_get_peer(call->wrapped); res = rb_str_new2(peer); gpr_free(peer); @@ -217,16 +213,16 @@ static VALUE grpc_rb_call_get_peer(VALUE self) { /* Called to obtain the x509 cert of an authenticated peer. */ static VALUE grpc_rb_call_get_peer_cert(VALUE self) { - grpc_call *call = NULL; + grpc_rb_call *call = NULL; VALUE res = Qnil; grpc_auth_context *ctx = NULL; if (RTYPEDDATA_DATA(self) == NULL) { rb_raise(grpc_rb_eCallError, "Cannot get peer cert on closed call"); return Qnil; } - TypedData_Get_Struct(self, grpc_call, &grpc_call_data_type, call); + TypedData_Get_Struct(self, grpc_rb_call, &grpc_call_data_type, call); - ctx = grpc_call_auth_context(call); + ctx = grpc_call_auth_context(call->wrapped); if (!ctx || !grpc_auth_context_peer_is_authenticated(ctx)) { return Qnil; @@ -326,21 +322,23 @@ static VALUE grpc_rb_call_set_write_flag(VALUE self, VALUE write_flag) { Sets credentials on a call */ static VALUE grpc_rb_call_set_credentials(VALUE self, VALUE credentials) { - grpc_call *call = NULL; + grpc_rb_call *call = NULL; grpc_call_credentials *creds; grpc_call_error err; if (RTYPEDDATA_DATA(self) == NULL) { rb_raise(grpc_rb_eCallError, "Cannot set credentials of closed call"); return Qnil; } - TypedData_Get_Struct(self, grpc_call, &grpc_call_data_type, call); + TypedData_Get_Struct(self, grpc_rb_call, &grpc_call_data_type, call); creds = grpc_rb_get_wrapped_call_credentials(credentials); - err = grpc_call_set_credentials(call, creds); + err = grpc_call_set_credentials(call->wrapped, creds); if (err != GRPC_CALL_OK) { rb_raise(grpc_rb_eCallError, "grpc_call_set_credentials failed with %s (code=%d)", grpc_call_error_detail_of(err), err); } + /* We need the credentials to be alive for as long as the call is alive, + but we don't care about destruction order. */ rb_ivar_set(self, id_credentials, credentials); return Qnil; } @@ -733,7 +731,6 @@ static VALUE grpc_run_batch_stack_build_result(run_batch_stack *st) { } /* call-seq: - cq = CompletionQueue.new ops = { GRPC::Core::CallOps::SEND_INITIAL_METADATA => , GRPC::Core::CallOps::SEND_MESSAGE => , @@ -741,7 +738,7 @@ static VALUE grpc_run_batch_stack_build_result(run_batch_stack *st) { } tag = Object.new timeout = 10 - call.start_batch(cq, tag, timeout, ops) + call.start_batch(tag, timeout, ops) Start a batch of operations defined in the array ops; when complete, post a completion of type 'tag' to the completion queue bound to the call. @@ -750,20 +747,20 @@ static VALUE grpc_run_batch_stack_build_result(run_batch_stack *st) { The order of ops specified in the batch has no significance. Only one operation of each type can be active at once in any given batch */ -static VALUE grpc_rb_call_run_batch(VALUE self, VALUE cqueue, VALUE tag, - VALUE timeout, VALUE ops_hash) { +static VALUE grpc_rb_call_run_batch(VALUE self, VALUE ops_hash) { run_batch_stack st; - grpc_call *call = NULL; + grpc_rb_call *call = NULL; grpc_event ev; grpc_call_error err; VALUE result = Qnil; VALUE rb_write_flag = rb_ivar_get(self, id_write_flag); unsigned write_flag = 0; + void *tag = (void*)&st; if (RTYPEDDATA_DATA(self) == NULL) { rb_raise(grpc_rb_eCallError, "Cannot run batch on closed call"); return Qnil; } - TypedData_Get_Struct(self, grpc_call, &grpc_call_data_type, call); + TypedData_Get_Struct(self, grpc_rb_call, &grpc_call_data_type, call); /* Validate the ops args, adding them to a ruby array */ if (TYPE(ops_hash) != T_HASH) { @@ -778,7 +775,7 @@ static VALUE grpc_rb_call_run_batch(VALUE self, VALUE cqueue, VALUE tag, /* call grpc_call_start_batch, then wait for it to complete using * pluck_event */ - err = grpc_call_start_batch(call, st.ops, st.op_num, ROBJECT(tag), NULL); + err = grpc_call_start_batch(call->wrapped, st.ops, st.op_num, tag, NULL); if (err != GRPC_CALL_OK) { grpc_run_batch_stack_cleanup(&st); rb_raise(grpc_rb_eCallError, @@ -786,12 +783,7 @@ static VALUE grpc_rb_call_run_batch(VALUE self, VALUE cqueue, VALUE tag, grpc_call_error_detail_of(err), err); return Qnil; } - ev = grpc_rb_completion_queue_pluck_event(cqueue, tag, timeout); - if (ev.type == GRPC_QUEUE_TIMEOUT) { - grpc_run_batch_stack_cleanup(&st); - rb_raise(grpc_rb_eOutOfTime, "grpc_call_start_batch timed out"); - return Qnil; - } + ev = grpc_rb_completion_queue_pluck(call->queue, tag, gpr_inf_future, NULL); /* Build and return the BatchResult struct result, if there is an error, it's reflected in the status */ @@ -900,7 +892,7 @@ void Init_grpc_call() { 1); /* Add ruby analogues of the Call methods. */ - rb_define_method(grpc_rb_cCall, "run_batch", grpc_rb_call_run_batch, 4); + rb_define_method(grpc_rb_cCall, "run_batch", grpc_rb_call_run_batch, 1); rb_define_method(grpc_rb_cCall, "cancel", grpc_rb_call_cancel, 0); rb_define_method(grpc_rb_cCall, "close", grpc_rb_call_close, 0); rb_define_method(grpc_rb_cCall, "peer", grpc_rb_call_get_peer, 0); @@ -921,9 +913,6 @@ void Init_grpc_call() { id_write_flag = rb_intern("write_flag"); /* Ids used by the c wrapping internals. */ - id_cq = rb_intern("__cq"); - id_flags = rb_intern("__flags"); - id_input_md = rb_intern("__input_md"); id_credentials = rb_intern("__credentials"); /* Ids used in constructing the batch result. */ @@ -947,15 +936,18 @@ void Init_grpc_call() { /* Gets the call from the ruby object */ grpc_call *grpc_rb_get_wrapped_call(VALUE v) { - grpc_call *c = NULL; - TypedData_Get_Struct(v, grpc_call, &grpc_call_data_type, c); - return c; + grpc_rb_call *call = NULL; + TypedData_Get_Struct(v, grpc_rb_call, &grpc_call_data_type, call); + return call->wrapped; } /* Obtains the wrapped object for a given call */ -VALUE grpc_rb_wrap_call(grpc_call *c) { - if (c == NULL) { +VALUE grpc_rb_wrap_call(grpc_call *c, grpc_completion_queue *q) { + if (c == NULL || q == NULL) { return Qnil; } - return TypedData_Wrap_Struct(grpc_rb_cCall, &grpc_call_data_type, c); + grpc_rb_call *wrapper = ALLOC(grpc_rb_call); + wrapper->wrapped = c; + wrapper->queue = q; + return TypedData_Wrap_Struct(grpc_rb_cCall, &grpc_call_data_type, wrapper); } diff --git a/src/ruby/ext/grpc/rb_completion_queue.c b/src/ruby/ext/grpc/rb_completion_queue.c index 9466402db0c..1ac2ef2f335 100644 --- a/src/ruby/ext/grpc/rb_completion_queue.c +++ b/src/ruby/ext/grpc/rb_completion_queue.c @@ -42,10 +42,6 @@ #include #include "rb_grpc.h" -/* grpc_rb_cCompletionQueue is the ruby class that proxies - * grpc_completion_queue. */ -static VALUE grpc_rb_cCompletionQueue = Qnil; - /* Used to allow grpc_completion_queue_next call to release the GIL */ typedef struct next_call_stack { grpc_completion_queue *cq; @@ -128,7 +124,7 @@ static void grpc_rb_completion_queue_shutdown_drain(grpc_completion_queue *cq) { } /* Helper function to free a completion queue. */ -static void grpc_rb_completion_queue_destroy(void *p) { +void grpc_rb_completion_queue_destroy(grpc_completion_queue *cq) { grpc_completion_queue *cq = NULL; if (p == NULL) { return; @@ -138,59 +134,22 @@ static void grpc_rb_completion_queue_destroy(void *p) { grpc_completion_queue_destroy(cq); } -static rb_data_type_t grpc_rb_completion_queue_data_type = { - "grpc_completion_queue", - {GRPC_RB_GC_NOT_MARKED, grpc_rb_completion_queue_destroy, - GRPC_RB_MEMSIZE_UNAVAILABLE, {NULL, NULL}}, - NULL, NULL, -#ifdef RUBY_TYPED_FREE_IMMEDIATELY - /* cannot immediately free because grpc_rb_completion_queue_shutdown_drain - * calls rb_thread_call_without_gvl. */ - 0, -#endif -}; - -/* Releases the c-level resources associated with a completion queue */ -static VALUE grpc_rb_completion_queue_close(VALUE self) { - grpc_completion_queue* cq = grpc_rb_get_wrapped_completion_queue(self); - grpc_rb_completion_queue_destroy(cq); - RTYPEDDATA_DATA(self) = NULL; - return Qnil; -} - -/* Allocates a completion queue. */ -static VALUE grpc_rb_completion_queue_alloc(VALUE cls) { - grpc_completion_queue *cq = grpc_completion_queue_create(NULL); - if (cq == NULL) { - rb_raise(rb_eArgError, "could not create a completion queue: not sure why"); - } - return TypedData_Wrap_Struct(cls, &grpc_rb_completion_queue_data_type, cq); -} - static void unblock_func(void *param) { next_call_stack *const next_call = (next_call_stack*)param; next_call->interrupted = 1; } -/* Blocks until the next event for given tag is available, and returns the - * event. */ -grpc_event grpc_rb_completion_queue_pluck_event(VALUE self, VALUE tag, - VALUE timeout) { +/* Does the same thing as grpc_completion_queue_pluck, while properly releasing + the GVL and handling interrupts */ +grpc_event rb_completion_queue_pluck(grpc_completion_queue queue, void *tag, + gpr_timespec deadline, void *reserved) { next_call_stack next_call; MEMZERO(&next_call, next_call_stack, 1); - TypedData_Get_Struct(self, grpc_completion_queue, - &grpc_rb_completion_queue_data_type, next_call.cq); - if (TYPE(timeout) == T_NIL) { - next_call.timeout = gpr_inf_future(GPR_CLOCK_REALTIME); - } else { - next_call.timeout = grpc_rb_time_timeval(timeout, /* absolute time*/ 0); - } - if (TYPE(tag) == T_NIL) { - next_call.tag = NULL; - } else { - next_call.tag = ROBJECT(tag); - } + next_call.cq = queue; + next_call.timeout = deadline; + next_call.tag = tag; next_call.event.type = GRPC_QUEUE_TIMEOUT; + (void)reserved; /* Loop until we finish a pluck without an interruption. The internal pluck function runs either until it is interrupted or it gets an event, or time runs out. @@ -210,27 +169,3 @@ grpc_event grpc_rb_completion_queue_pluck_event(VALUE self, VALUE tag, next_call.event.type == GRPC_QUEUE_TIMEOUT); return next_call.event; } - -void Init_grpc_completion_queue() { - grpc_rb_cCompletionQueue = - rb_define_class_under(grpc_rb_mGrpcCore, "CompletionQueue", rb_cObject); - - /* constructor: uses an alloc func without an initializer. Using a simple - alloc func works here as the grpc header does not specify any args for - this func, so no separate initialization step is necessary. */ - rb_define_alloc_func(grpc_rb_cCompletionQueue, - grpc_rb_completion_queue_alloc); - - /* close: Provides a way to close the underlying file descriptor without - waiting for ruby garbage collection. */ - rb_define_method(grpc_rb_cCompletionQueue, "close", - grpc_rb_completion_queue_close, 0); -} - -/* Gets the wrapped completion queue from the ruby wrapper */ -grpc_completion_queue *grpc_rb_get_wrapped_completion_queue(VALUE v) { - grpc_completion_queue *cq = NULL; - TypedData_Get_Struct(v, grpc_completion_queue, - &grpc_rb_completion_queue_data_type, cq); - return cq; -} diff --git a/src/ruby/ext/grpc/rb_completion_queue.h b/src/ruby/ext/grpc/rb_completion_queue.h index 42de43c3fbb..3543e86275c 100644 --- a/src/ruby/ext/grpc/rb_completion_queue.h +++ b/src/ruby/ext/grpc/rb_completion_queue.h @@ -41,15 +41,14 @@ /* Gets the wrapped completion queue from the ruby wrapper */ grpc_completion_queue *grpc_rb_get_wrapped_completion_queue(VALUE v); +void grpc_rb_completion_queue_destroy(grpc_completion_queue *cq); + /** * Makes the implementation of CompletionQueue#pluck available in other files * * This avoids having code that holds the GIL repeated at multiple sites. */ -grpc_event grpc_rb_completion_queue_pluck_event(VALUE self, VALUE tag, - VALUE timeout); - -/* Initializes the CompletionQueue class. */ -void Init_grpc_completion_queue(); +grpc_event rb_completion_queue_pluck(grpc_completion_queue queue, void *tag, + gpr_timespec deadline, void *reserved); #endif /* GRPC_RB_COMPLETION_QUEUE_H_ */ diff --git a/src/ruby/ext/grpc/rb_grpc.c b/src/ruby/ext/grpc/rb_grpc.c index 8527da52d27..188a62475d2 100644 --- a/src/ruby/ext/grpc/rb_grpc.c +++ b/src/ruby/ext/grpc/rb_grpc.c @@ -46,7 +46,6 @@ #include "rb_call_credentials.h" #include "rb_channel.h" #include "rb_channel_credentials.h" -#include "rb_completion_queue.h" #include "rb_loader.h" #include "rb_server.h" #include "rb_server_credentials.h" @@ -318,7 +317,7 @@ void Init_grpc_c() { grpc_rb_mGrpcCore = rb_define_module_under(grpc_rb_mGRPC, "Core"); grpc_rb_sNewServerRpc = rb_struct_define("NewServerRpc", "method", "host", - "deadline", "metadata", "call", "cq", NULL); + "deadline", "metadata", "call", NULL); grpc_rb_sStatus = rb_struct_define("Status", "code", "details", "metadata", NULL); sym_code = ID2SYM(rb_intern("code")); @@ -326,7 +325,6 @@ void Init_grpc_c() { sym_metadata = ID2SYM(rb_intern("metadata")); Init_grpc_channel(); - Init_grpc_completion_queue(); Init_grpc_call(); Init_grpc_call_credentials(); Init_grpc_channel_credentials(); diff --git a/src/ruby/ext/grpc/rb_server.c b/src/ruby/ext/grpc/rb_server.c index c92059d12cc..cf430495c8b 100644 --- a/src/ruby/ext/grpc/rb_server.c +++ b/src/ruby/ext/grpc/rb_server.c @@ -63,6 +63,22 @@ typedef struct grpc_rb_server { grpc_completion_queue *queue; } grpc_rb_server; +static void destroy_server(grpc_rb_server *server, gpr_timespec deadline) { + grpc_event ev; + if (server->wrapped != NULL) { + grpc_server_shutdown_and_notify(server->wrapped, server->queue, NULL); + ev = rb_completion_queue_pluck(server->queue, NULL, deadline, NULL); + if (ev.type == GRPC_QUEUE_TIMEOUT) { + grpc_server_cancel_all_calls(server->wrapped); + rb_completion_queue_pluck(server->queue, NULL, gpr_inf_future, NULL); + } + grpc_server_destroy(server->wrapped); + grpc_rb_completion_queue_destroy(server->queue); + server->wrapped = NULL; + server->queue = NULL; + } +} + /* Destroys server instances. */ static void grpc_rb_server_free(void *p) { grpc_rb_server *svr = NULL; @@ -71,16 +87,8 @@ static void grpc_rb_server_free(void *p) { }; svr = (grpc_rb_server *)p; - /* Deletes the wrapped object if the mark object is Qnil, which indicates - that no other object is the actual owner. */ - /* grpc_server_shutdown does not exist. Change this to something that does - or delete it */ - if (svr->wrapped != NULL && svr->mark == Qnil) { - // grpc_server_shutdown(svr->wrapped); - // Aborting to indicate a bug - abort(); - grpc_server_destroy(svr->wrapped); - } + // TODO(murgatroid99): Maybe don't wait forever for the server to shutdown + destroy_server(svr, gpr_inf_future); xfree(p); } @@ -122,17 +130,15 @@ static VALUE grpc_rb_server_alloc(VALUE cls) { /* call-seq: - cq = CompletionQueue.new - server = Server.new(cq, {'arg1': 'value1'}) + server = Server.new({'arg1': 'value1'}) Initializes server instances. */ -static VALUE grpc_rb_server_init(VALUE self, VALUE cqueue, VALUE channel_args) { - grpc_completion_queue *cq = NULL; +static VALUE grpc_rb_server_init(VALUE self, VALUE channel_args) { + grpc_completion_queue *cq = grpc_completion_queue_create(NULL); grpc_rb_server *wrapper = NULL; grpc_server *srv = NULL; grpc_channel_args args; MEMZERO(&args, grpc_channel_args, 1); - cq = grpc_rb_get_wrapped_completion_queue(cqueue); TypedData_Get_Struct(self, grpc_rb_server, &grpc_rb_server_data_type, wrapper); grpc_rb_hash_convert_to_channel_args(channel_args, &args); @@ -179,65 +185,57 @@ static void grpc_request_call_stack_cleanup(request_call_stack* st) { } /* call-seq: - cq = CompletionQueue.new - tag = Object.new - timeout = 10 - server.request_call(cqueue, tag, timeout) + server.request_call Requests notification of a new call on a server. */ -static VALUE grpc_rb_server_request_call(VALUE self, VALUE cqueue, - VALUE tag_new, VALUE timeout) { +static VALUE grpc_rb_server_request_call(VALUE self) { grpc_rb_server *s = NULL; grpc_call *call = NULL; grpc_event ev; grpc_call_error err; request_call_stack st; VALUE result; + void *tag = (void*)&st; + grpc_completion_queue *call_queue = grpc_completion_queue_create(NULL); gpr_timespec deadline; TypedData_Get_Struct(self, grpc_rb_server, &grpc_rb_server_data_type, s); if (s->wrapped == NULL) { rb_raise(rb_eRuntimeError, "destroyed!"); return Qnil; - } else { - grpc_request_call_stack_init(&st); - /* call grpc_server_request_call, then wait for it to complete using - * pluck_event */ - err = grpc_server_request_call( - s->wrapped, &call, &st.details, &st.md_ary, - grpc_rb_get_wrapped_completion_queue(cqueue), - grpc_rb_get_wrapped_completion_queue(s->mark), - ROBJECT(tag_new)); - if (err != GRPC_CALL_OK) { - grpc_request_call_stack_cleanup(&st); - rb_raise(grpc_rb_eCallError, - "grpc_server_request_call failed: %s (code=%d)", - grpc_call_error_detail_of(err), err); - return Qnil; - } - - ev = grpc_rb_completion_queue_pluck_event(s->mark, tag_new, timeout); - if (ev.type == GRPC_QUEUE_TIMEOUT) { - grpc_request_call_stack_cleanup(&st); - return Qnil; - } - if (!ev.success) { - grpc_request_call_stack_cleanup(&st); - rb_raise(grpc_rb_eCallError, "request_call completion failed"); - return Qnil; - } + } + grpc_request_call_stack_init(&st); + /* call grpc_server_request_call, then wait for it to complete using + * pluck_event */ + err = grpc_server_request_call( + s->wrapped, &call, &st.details, &st.md_ary, + call_queue, s->queue, tag); + if (err != GRPC_CALL_OK) { + grpc_request_call_stack_cleanup(&st); + rb_raise(grpc_rb_eCallError, + "grpc_server_request_call failed: %s (code=%d)", + grpc_call_error_detail_of(err), err); + return Qnil; + } - /* build the NewServerRpc struct result */ - deadline = gpr_convert_clock_type(st.details.deadline, GPR_CLOCK_REALTIME); - result = rb_struct_new( - grpc_rb_sNewServerRpc, rb_str_new2(st.details.method), - rb_str_new2(st.details.host), - rb_funcall(rb_cTime, id_at, 2, INT2NUM(deadline.tv_sec), - INT2NUM(deadline.tv_nsec)), - grpc_rb_md_ary_to_h(&st.md_ary), grpc_rb_wrap_call(call), cqueue, NULL); + ev = rb_completion_queue_pluck(s->queue, tag, + gpr_inf_future, NULL); + if (!ev.success) { grpc_request_call_stack_cleanup(&st); - return result; + rb_raise(grpc_rb_eCallError, "request_call completion failed"); + return Qnil; } - return Qnil; + + /* build the NewServerRpc struct result */ + deadline = gpr_convert_clock_type(st.details.deadline, GPR_CLOCK_REALTIME); + result = rb_struct_new( + grpc_rb_sNewServerRpc, rb_str_new2(st.details.method), + rb_str_new2(st.details.host), + rb_funcall(rb_cTime, id_at, 2, INT2NUM(deadline.tv_sec), + INT2NUM(deadline.tv_nsec)), + grpc_rb_md_ary_to_h(&st.md_ary), grpc_rb_wrap_call(call, call_queue), + NULL); + grpc_request_call_stack_cleanup(&st); + return result; } static VALUE grpc_rb_server_start(VALUE self) { @@ -275,19 +273,15 @@ static VALUE grpc_rb_server_destroy(int argc, VALUE *argv, VALUE self) { rb_scan_args(argc, argv, "11", &cqueue, &timeout); cq = grpc_rb_get_wrapped_completion_queue(cqueue); TypedData_Get_Struct(self, grpc_rb_server, &grpc_rb_server_data_type, s); - - if (s->wrapped != NULL) { - grpc_server_shutdown_and_notify(s->wrapped, cq, NULL); - ev = grpc_rb_completion_queue_pluck_event(cqueue, Qnil, timeout); - if (!ev.success) { - rb_warn("server shutdown failed, cancelling the calls, objects may leak"); - grpc_server_cancel_all_calls(s->wrapped); - return Qfalse; - } - grpc_server_destroy(s->wrapped); - s->wrapped = NULL; + if (TYPE(timeout) == T_NIL) { + deadline = gpr_inf_future(GPR_CLOCK_REALTIME); + } else { + deadline = grpc_rb_time_timeval(timeout, /* absolute time*/ 0); } - return Qtrue; + + destroy_server(s, deadline); + + return Qnil; } /* @@ -347,13 +341,13 @@ void Init_grpc_server() { rb_define_alloc_func(grpc_rb_cServer, grpc_rb_server_alloc); /* Provides a ruby constructor and support for dup/clone. */ - rb_define_method(grpc_rb_cServer, "initialize", grpc_rb_server_init, 2); + rb_define_method(grpc_rb_cServer, "initialize", grpc_rb_server_init, 1); rb_define_method(grpc_rb_cServer, "initialize_copy", grpc_rb_cannot_init_copy, 1); /* Add the server methods. */ rb_define_method(grpc_rb_cServer, "request_call", - grpc_rb_server_request_call, 3); + grpc_rb_server_request_call, 0); rb_define_method(grpc_rb_cServer, "start", grpc_rb_server_start, 0); rb_define_method(grpc_rb_cServer, "destroy", grpc_rb_server_destroy, -1); rb_define_alias(grpc_rb_cServer, "close", "destroy"); From d627c105847f0d262ce3886fb5b463dc914ddbd7 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Mon, 6 Jun 2016 15:49:32 -0700 Subject: [PATCH 0059/1039] Fix asan failures (i.e add pollset_global_shutdown), remove debug log lines --- src/core/lib/iomgr/ev_epoll_linux.c | 76 ++++++++++++++--------------- 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index d3abf3bd84f..0e00d4d216a 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -516,6 +516,21 @@ static void polling_island_global_init() { g_pi_freelist = NULL; } +static void polling_island_global_shutdown() { + polling_island *next; + gpr_mu_lock(&g_pi_freelist_mu); + gpr_mu_unlock(&g_pi_freelist_mu); + while (g_pi_freelist != NULL) { + next = g_pi_freelist->next_free; + gpr_mu_destroy(&g_pi_freelist->mu); + gpr_free(g_pi_freelist->fds); + gpr_free(g_pi_freelist); + g_pi_freelist = next; + } + + gpr_mu_destroy(&g_pi_freelist_mu); +} + /******************************************************************************* * Fd Definitions */ @@ -784,7 +799,7 @@ static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, */ static void sig_handler(int sig_num) { -#ifdef GPRC_EPOLL_DEBUG +#ifdef GRPC_EPOLL_DEBUG gpr_log(GPR_INFO, "Received signal %d", sig_num); #endif } @@ -792,7 +807,7 @@ static void sig_handler(int sig_num) { /* Global state management */ static void pollset_global_init(void) { grpc_wakeup_fd_init(&grpc_global_wakeup_fd); - signal(SIGUSR1, sig_handler); + signal(SIGUSR1, sig_handler); /* TODO: sreek - Do not hardcode SIGUSR1 */ } static void pollset_global_shutdown(void) { @@ -840,7 +855,6 @@ static void pollset_kick(grpc_pollset *p, grpc_pollset_worker *worker = specific_worker; if (worker != NULL) { if (worker == GRPC_POLLSET_KICK_BROADCAST) { - gpr_log(GPR_DEBUG, "pollset_kick: broadcast!"); if (pollset_has_workers(p)) { GPR_TIMER_BEGIN("pollset_kick.broadcast", 0); for (worker = p->root_worker.next; worker != &p->root_worker; @@ -848,12 +862,10 @@ static void pollset_kick(grpc_pollset *p, pthread_kill(worker->pt_id, SIGUSR1); } } else { - gpr_log(GPR_DEBUG, "pollset_kick: (broadcast) Kicked without pollers"); p->kicked_without_pollers = true; } GPR_TIMER_END("pollset_kick.broadcast", 0); } else { - gpr_log(GPR_DEBUG, "pollset_kick: kicked kicked_specifically"); GPR_TIMER_MARK("kicked_specifically", 0); worker->kicked_specifically = true; pthread_kill(worker->pt_id, SIGUSR1); @@ -864,11 +876,9 @@ static void pollset_kick(grpc_pollset *p, if (worker != NULL) { GPR_TIMER_MARK("finally_kick", 0); push_back_worker(p, worker); - gpr_log(GPR_DEBUG, "pollset_kick: anonymous kick"); pthread_kill(worker->pt_id, SIGUSR1); } else { GPR_TIMER_MARK("kicked_no_pollers", 0); - gpr_log(GPR_DEBUG, "pollset_kick: kicked without pollers"); p->kicked_without_pollers = true; } } @@ -941,7 +951,6 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; int epoll_fd = -1; int ep_rv; - gpr_log(GPR_DEBUG, "pollset_work_and_unlock: Entering.."); GPR_TIMER_BEGIN("pollset_work_and_unlock", 0); /* We need to get the epoll_fd to wait on. The epoll_fd is in inside the @@ -952,22 +961,27 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, - pollset->polling_island->mu */ gpr_mu_lock(&pollset->pi_mu); - if (pollset->polling_island != NULL) { - pollset->polling_island = - polling_island_update_and_lock(pollset->polling_island, 1, 0); - epoll_fd = pollset->polling_island->epoll_fd; - if (pollset->polling_island->fd_cnt == 0) { - gpr_log(GPR_DEBUG, "pollset_work_and_unlock: epoll_fd: %d, No other fds", - epoll_fd); - } - for (size_t i = 0; i < pollset->polling_island->fd_cnt; i++) { - gpr_log(GPR_DEBUG, - "pollset_work_and_unlock: epoll_fd: %d, fd_count: %d, fd[%d]: %d", - epoll_fd, pollset->polling_island->fd_cnt, i, - pollset->polling_island->fds[i]->fd); - } - gpr_mu_unlock(&pollset->polling_island->mu); + if (pollset->polling_island == NULL) { + pollset->polling_island = polling_island_create(NULL, 1); + } + + pollset->polling_island = + polling_island_update_and_lock(pollset->polling_island, 1, 0); + epoll_fd = pollset->polling_island->epoll_fd; + +#ifdef GRPC_EPOLL_DEBUG + if (pollset->polling_island->fd_cnt == 0) { + gpr_log(GPR_DEBUG, "pollset_work_and_unlock: epoll_fd: %d, No other fds", + epoll_fd); + } + for (size_t i = 0; i < pollset->polling_island->fd_cnt; i++) { + gpr_log(GPR_DEBUG, + "pollset_work_and_unlock: epoll_fd: %d, fd_count: %d, fd[%d]: %d", + epoll_fd, pollset->polling_island->fd_cnt, i, + pollset->polling_island->fds[i]->fd); } +#endif + gpr_mu_unlock(&pollset->polling_island->mu); gpr_mu_unlock(&pollset->pi_mu); gpr_mu_unlock(&pollset->mu); @@ -975,16 +989,8 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, /* If epoll_fd == -1, this is a blank pollset and does not have any fds yet */ if (epoll_fd != -1) { do { - gpr_timespec before_epoll = gpr_now(GPR_CLOCK_PRECISE); - gpr_log(GPR_DEBUG, "pollset_work_and_unlock: epoll_wait()...."); ep_rv = epoll_pwait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, timeout_ms, sig_mask); - gpr_timespec after_epoll = gpr_now(GPR_CLOCK_PRECISE); - int dur = gpr_time_to_millis(gpr_time_sub(after_epoll, before_epoll)); - gpr_log(GPR_DEBUG, - "pollset_work_and_unlock: DONE epoll_wait() : %d ms, ep_rv: %d", - dur, ep_rv); - if (ep_rv < 0) { if (errno != EINTR) { /* TODO (sreek) - Do not log an error in case of bad file descriptor @@ -993,9 +999,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, * closed) */ gpr_log(GPR_ERROR, "epoll_pwait() failed: %s", strerror(errno)); } else { - gpr_log(GPR_DEBUG, "pollset_work_and_unlock: 0-timeout epoll_wait()"); ep_rv = epoll_wait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, 0); - gpr_log(GPR_DEBUG, "pollset_work_and_unlock: ep_rv: %d", ep_rv); } } @@ -1018,7 +1022,6 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, } } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); } - gpr_log(GPR_DEBUG, "pollset_work_and_unlock: Leaving.."); GPR_TIMER_END("pollset_work_and_unlock", 0); } @@ -1093,7 +1096,6 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker **worker_hdl, gpr_timespec now, gpr_timespec deadline) { GPR_TIMER_BEGIN("pollset_work", 0); - gpr_log(GPR_DEBUG, "pollset_work: enter"); int timeout_ms = poll_deadline_to_millis_timeout(deadline, now); sigset_t new_mask; @@ -1112,7 +1114,6 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, work that needs attention like an event on the completion queue or an alarm */ GPR_TIMER_MARK("pollset_work.kicked_without_pollers", 0); - gpr_log(GPR_INFO, "pollset_work: kicked without pollers.."); pollset->kicked_without_pollers = 0; } else if (!pollset->shutting_down) { sigemptyset(&new_mask); @@ -1147,14 +1148,12 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, gpr_mu_lock(&pollset->mu); } - gpr_log(GPR_DEBUG, "pollset_work(): leaving"); *worker_hdl = NULL; GPR_TIMER_END("pollset_work", 0); } static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { - gpr_log(GPR_DEBUG, "pollset_add_fd: pollset: %p, fd: %d", pollset, fd->fd); /* TODO sreek - Double check if we need to get a pollset->mu lock here */ gpr_mu_lock(&pollset->pi_mu); gpr_mu_lock(&fd->pi_mu); @@ -1347,6 +1346,7 @@ static void pollset_set_del_pollset_set(grpc_exec_ctx *exec_ctx, static void shutdown_engine(void) { fd_global_shutdown(); pollset_global_shutdown(); + polling_island_global_shutdown(); } static const grpc_event_engine_vtable vtable = { From e5012bac7a57df6b1993a1eaa9b8b3c4d7671975 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Mon, 6 Jun 2016 16:01:45 -0700 Subject: [PATCH 0060/1039] Remove redundant code --- src/core/lib/iomgr/ev_epoll_linux.c | 61 ++++++++++++++--------------- 1 file changed, 29 insertions(+), 32 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 0e00d4d216a..f7ac4ae1ff4 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -986,42 +986,39 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, gpr_mu_unlock(&pollset->pi_mu); gpr_mu_unlock(&pollset->mu); - /* If epoll_fd == -1, this is a blank pollset and does not have any fds yet */ - if (epoll_fd != -1) { - do { - ep_rv = epoll_pwait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, timeout_ms, - sig_mask); - if (ep_rv < 0) { - if (errno != EINTR) { - /* TODO (sreek) - Do not log an error in case of bad file descriptor - * (A bad file descriptor here would just mean that the epoll set was - * merged with another epoll set and that the current epoll_fd is - * closed) */ - gpr_log(GPR_ERROR, "epoll_pwait() failed: %s", strerror(errno)); - } else { - ep_rv = epoll_wait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, 0); - } + do { + ep_rv = epoll_pwait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, timeout_ms, + sig_mask); + if (ep_rv < 0) { + if (errno != EINTR) { + /* TODO (sreek) - Do not log an error in case of bad file descriptor + * (A bad file descriptor here would just mean that the epoll set was + * merged with another epoll set and that the current epoll_fd is + * closed) */ + gpr_log(GPR_ERROR, "epoll_pwait() failed: %s", strerror(errno)); + } else { + ep_rv = epoll_wait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, 0); } + } - int i; - for (i = 0; i < ep_rv; ++i) { - grpc_fd *fd = ep_ev[i].data.ptr; - int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); - int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); - int write_ev = ep_ev[i].events & EPOLLOUT; - if (fd == NULL) { - grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); - } else { - if (read_ev || cancel) { - fd_become_readable(exec_ctx, fd); - } - if (write_ev || cancel) { - fd_become_writable(exec_ctx, fd); - } + int i; + for (i = 0; i < ep_rv; ++i) { + grpc_fd *fd = ep_ev[i].data.ptr; + int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); + int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); + int write_ev = ep_ev[i].events & EPOLLOUT; + if (fd == NULL) { + grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); + } else { + if (read_ev || cancel) { + fd_become_readable(exec_ctx, fd); + } + if (write_ev || cancel) { + fd_become_writable(exec_ctx, fd); } } - } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); - } + } + } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); GPR_TIMER_END("pollset_work_and_unlock", 0); } From ad162ba5a9f91481b71f233d1f4f8c15b01de644 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Mon, 6 Jun 2016 16:23:37 -0700 Subject: [PATCH 0061/1039] Core review comments and remove 'kicked_specifically' field as its not needed --- src/core/lib/iomgr/ev_epoll_linux.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index f7ac4ae1ff4..d5aac96fa46 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -145,7 +145,6 @@ typedef struct polling_island { * Pollset Declarations */ struct grpc_pollset_worker { - int kicked_specifically; pthread_t pt_id; /* Thread id of this worker */ struct grpc_pollset_worker *next; struct grpc_pollset_worker *prev; @@ -235,18 +234,16 @@ static void polling_island_remove_all_fds_locked(polling_island *pi, size_t i; for (i = 0; i < pi->fd_cnt; i++) { - if (remove_fd_refs) { - GRPC_FD_UNREF(pi->fds[i], "polling_island"); - } - err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_DEL, pi->fds[i]->fd, NULL); if (err < 0 && errno != ENOENT) { - gpr_log(GPR_ERROR, - "epoll_ctl delete for fds[i]: %d failed with error: %s", i, - pi->fds[i]->fd, strerror(errno)); /* TODO: sreek - We need a better way to bubble up this error instead of - * just logging a message */ - continue; + * just logging a message */ + gpr_log(GPR_ERROR, "epoll_ctl deleting fds[%d]: %d failed with error: %s", + i, pi->fds[i]->fd, strerror(errno)); + } + + if (remove_fd_refs) { + GRPC_FD_UNREF(pi->fds[i], "polling_island"); } } @@ -264,7 +261,7 @@ static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd, if (!is_fd_closed) { err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_DEL, fd->fd, NULL); if (err < 0 && errno != ENOENT) { - gpr_log(GPR_ERROR, "epoll_ctl delete for fd: %d failed with error; %s", + gpr_log(GPR_ERROR, "epoll_ctl deleting fd: %d failed with error; %s", fd->fd, strerror(errno)); } } @@ -867,7 +864,6 @@ static void pollset_kick(grpc_pollset *p, GPR_TIMER_END("pollset_kick.broadcast", 0); } else { GPR_TIMER_MARK("kicked_specifically", 0); - worker->kicked_specifically = true; pthread_kill(worker->pt_id, SIGUSR1); } } else { @@ -1100,7 +1096,6 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_pollset_worker worker; worker.next = worker.prev = NULL; - worker.kicked_specifically = 0; worker.pt_id = pthread_self(); *worker_hdl = &worker; From 0cf4defa42dfd0e1f09ea3e1fae7c111a673244f Mon Sep 17 00:00:00 2001 From: Alistair Veitch Date: Tue, 7 Jun 2016 08:59:30 -0700 Subject: [PATCH 0062/1039] simplify base resource definition; misc comment updates --- src/core/ext/census/base_resources.c | 119 ++++------------------ src/core/ext/census/base_resources.h | 4 +- src/core/ext/census/resource.c | 142 +++++++++++++++++++-------- src/core/ext/census/resource.h | 29 +++++- 4 files changed, 148 insertions(+), 146 deletions(-) diff --git a/src/core/ext/census/base_resources.c b/src/core/ext/census/base_resources.c index 8f0329a72bd..f1ee41c5d97 100644 --- a/src/core/ext/census/base_resources.c +++ b/src/core/ext/census/base_resources.c @@ -30,15 +30,15 @@ * */ +#include "base_resources.h" + #include #include #include #include -#include "base_resources.h" -#include "gen/census.pb.h" -#include "third_party/nanopb/pb_encode.h" +#include "resource.h" // Add base RPC resource definitions for use by RPC runtime. // @@ -48,101 +48,24 @@ // file, which is compiled to .pb format and read by still-to-be-written // configuration functions. -// Structure representing a MeasurementUnit proto. -typedef struct { - int32_t prefix; - int n_numerators; - const google_census_Resource_BasicUnit *numerators; - int n_denominators; - const google_census_Resource_BasicUnit *denominators; -} measurement_unit; - -// Encode a nanopb string. Expects the string argument to be passed in as `arg`. -static bool encode_string(pb_ostream_t *stream, const pb_field_t *field, - void *const *arg) { - if (!pb_encode_tag_for_field(stream, field)) { - return false; - } - return pb_encode_string(stream, (uint8_t *)*arg, strlen((const char *)*arg)); -} - -// Encode the numerators part of a measurement_unit (passed in as `arg`). -static bool encode_numerators(pb_ostream_t *stream, const pb_field_t *field, - void *const *arg) { - const measurement_unit *mu = (const measurement_unit *)*arg; - for (int i = 0; i < mu->n_numerators; i++) { - if (!pb_encode_tag_for_field(stream, field)) { - return false; - } - if (!pb_encode_varint(stream, mu->numerators[i])) { - return false; - } - } - return true; -} - -// Encode the denominators part of a measurement_unit (passed in as `arg`). -static bool encode_denominators(pb_ostream_t *stream, const pb_field_t *field, - void *const *arg) { - const measurement_unit *mu = (const measurement_unit *)*arg; - for (int i = 0; i < mu->n_denominators; i++) { - if (!pb_encode_tag_for_field(stream, field)) { - return false; - } - if (!pb_encode_varint(stream, mu->numerators[i])) { - return false; - } - } - return true; -} - -// Define a Resource, given the important details. Encodes a protobuf, which -// is then passed to census_define_resource. -static void define_resource(const char *name, const char *description, - const measurement_unit *unit) { - // nanopb generated type for Resource. Initialize encoding functions to NULL - // since we can't directly initialize them due to embedded union in struct. - google_census_Resource resource = { - {{NULL}, (void *)name}, - {{NULL}, (void *)description}, - true, // has_unit - {true, unit->prefix, {{NULL}, (void *)unit}, {{NULL}, (void *)unit}}}; - resource.name.funcs.encode = &encode_string; - resource.description.funcs.encode = &encode_string; - resource.unit.numerator.funcs.encode = &encode_numerators; - resource.unit.denominator.funcs.encode = &encode_denominators; - - // Buffer for storing encoded proto. - uint8_t buffer[512]; - pb_ostream_t stream = pb_ostream_from_buffer(buffer, 512); - if (!pb_encode(&stream, google_census_Resource_fields, &resource)) { - gpr_log(GPR_ERROR, "Error encoding resource %s.", name); - return; - } - int32_t mid = census_define_resource(buffer, stream.bytes_written); - if (mid < 0) { - gpr_log(GPR_ERROR, "Error defining resource %s.", name); - } -} - -// Define a resource for client RPC latency. -static void define_client_rpc_latency_resource() { - google_census_Resource_BasicUnit numerator = - google_census_Resource_BasicUnit_SECS; - measurement_unit unit = {0, 1, &numerator, 0, NULL}; - define_resource("client_rpc_latency", "Client RPC latency in seconds", &unit); -} - -// Define a resource for server RPC latency. -static void define_server_rpc_latency_resource() { - google_census_Resource_BasicUnit numerator = - google_census_Resource_BasicUnit_SECS; - measurement_unit unit = {0, 1, &numerator, 0, NULL}; - define_resource("server_rpc_latency", "Server RPC latency in seconds", &unit); -} - // Define all base resources. This should be called by census initialization. void define_base_resources() { - define_client_rpc_latency_resource(); - define_server_rpc_latency_resource(); + google_census_Resource_BasicUnit numerator = + google_census_Resource_BasicUnit_SECS; + resource r = {"client_rpc_latency", // name + "Client RPC latency in seconds", // description + 0, // prefix + 1, // n_numerators + &numerator, // numerators + 0, // n_denominators + NULL}; // denominators + define_resource(&r); + r = (resource){"server_rpc_latency", // name + "Server RPC latency in seconds", // description + 0, // prefix + 1, // n_numerators + &numerator, // numerators + 0, // n_denominators + NULL}; // denominators + define_resource(&r); } diff --git a/src/core/ext/census/base_resources.h b/src/core/ext/census/base_resources.h index 992e5f6ebea..e5a7696db43 100644 --- a/src/core/ext/census/base_resources.h +++ b/src/core/ext/census/base_resources.h @@ -33,7 +33,7 @@ #ifndef GRPC_CORE_EXT_CENSUS_BASE_RESOURCES_H #define GRPC_CORE_EXT_CENSUS_BASE_RESOURCES_H -// Define all base resources. This should be called by census initialization. +/* Define all base resources. This should be called by census initialization. */ void define_base_resources(); -#endif // GRPC_CORE_EXT_CENSUS_BASE_RESOURCES_H +#endif /* GRPC_CORE_EXT_CENSUS_BASE_RESOURCES_H */ diff --git a/src/core/ext/census/resource.c b/src/core/ext/census/resource.c index 632b899e415..0ca0db94bad 100644 --- a/src/core/ext/census/resource.c +++ b/src/core/ext/census/resource.c @@ -32,7 +32,6 @@ */ #include "resource.h" -#include "gen/census.pb.h" #include "third_party/nanopb/pb_decode.h" #include @@ -43,13 +42,6 @@ #include #include -// Internal representation of a resource. -typedef struct { - char *name; - // Pointer to raw protobuf used in resource definition. - uint8_t *raw_pb; -} resource; - // Protect local resource data structures. static gpr_mu resource_lock; @@ -65,7 +57,7 @@ static size_t n_resources = 0; // Number of defined resources static size_t n_defined_resources = 0; -void initialize_resources() { +void initialize_resources(void) { gpr_mu_init(&resource_lock); gpr_mu_lock(&resource_lock); GPR_ASSERT(resources == NULL && n_resources == 0 && n_defined_resources == 0); @@ -78,16 +70,17 @@ void initialize_resources() { // Delete a resource given it's ID. Must be called with resource_lock held. static void delete_resource_locked(size_t rid) { - GPR_ASSERT(resources[rid] != NULL && resources[rid]->raw_pb != NULL && - resources[rid]->name != NULL && n_defined_resources > 0); + GPR_ASSERT(resources[rid] != NULL); gpr_free(resources[rid]->name); - gpr_free(resources[rid]->raw_pb); + gpr_free(resources[rid]->description); + gpr_free(resources[rid]->numerators); + gpr_free(resources[rid]->denominators); gpr_free(resources[rid]); resources[rid] = NULL; n_defined_resources--; } -void shutdown_resources() { +void shutdown_resources(void) { gpr_mu_lock(&resource_lock); for (size_t i = 0; i < n_resources; i++) { if (resources[i] != NULL) { @@ -128,8 +121,13 @@ static bool validate_string(pb_istream_t *stream, const pb_field_t *field, } break; case google_census_Resource_description_tag: - // Description is optional, does not need validating, just skip. - if (!pb_read(stream, NULL, stream->bytes_left)) { + if (stream->bytes_left == 0) { + return true; + } + vresource->description = gpr_malloc(stream->bytes_left + 1); + vresource->description[stream->bytes_left] = '\0'; + if (!pb_read(stream, (uint8_t *)vresource->description, + stream->bytes_left)) { return false; } break; @@ -144,17 +142,47 @@ static bool validate_string(pb_istream_t *stream, const pb_field_t *field, return true; } +// Decode numerators/denominators in a stream. The `count` and `bup` +// (BasicUnit pointer) are pointers to the approriate fields in a resource +// struct. +static bool validate_units_helper(pb_istream_t *stream, int *count, + google_census_Resource_BasicUnit **bup) { + while (stream->bytes_left) { + (*count)++; + // Have to allocate a new array of values. Normal case is 0 or 1, so + // this should normally not be an issue. + google_census_Resource_BasicUnit *new_bup = + gpr_malloc((size_t)*count * sizeof(google_census_Resource_BasicUnit)); + if (*count != 1) { + memcpy(new_bup, *bup, + (size_t)(*count - 1) * sizeof(google_census_Resource_BasicUnit)); + gpr_free(*bup); + } + *bup = new_bup; + if (!pb_decode_varint(stream, (uint64_t *)(*bup + *count - 1))) { + return false; + } + } + return true; +} + // Validate units field of a Resource proto. static bool validate_units(pb_istream_t *stream, const pb_field_t *field, void **arg) { - if (field->tag == google_census_Resource_MeasurementUnit_numerator_tag) { - *(bool *)*arg = true; // has_numerator = true. - } - while (stream->bytes_left) { - uint64_t value; - if (!pb_decode_varint(stream, &value)) { + resource *vresource = (resource *)(*arg); + switch (field->tag) { + case google_census_Resource_MeasurementUnit_numerator_tag: + return validate_units_helper(stream, &vresource->n_numerators, + &vresource->numerators); + break; + case google_census_Resource_MeasurementUnit_denominator_tag: + return validate_units_helper(stream, &vresource->n_denominators, + &vresource->denominators); + break; + default: + gpr_log(GPR_ERROR, "Unknown field type."); return false; - } + break; } return true; } @@ -170,10 +198,11 @@ static bool validate_resource_pb(const uint8_t *resource_pb, vresource.name.funcs.decode = &validate_string; vresource.name.arg = resources[id]; vresource.description.funcs.decode = &validate_string; + vresource.description.arg = resources[id]; vresource.unit.numerator.funcs.decode = &validate_units; - bool has_numerator = false; - vresource.unit.numerator.arg = &has_numerator; + vresource.unit.numerator.arg = resources[id]; vresource.unit.denominator.funcs.decode = &validate_units; + vresource.unit.denominator.arg = resources[id]; pb_istream_t stream = pb_istream_from_buffer((uint8_t *)resource_pb, resource_pb_size); @@ -181,17 +210,15 @@ static bool validate_resource_pb(const uint8_t *resource_pb, return false; } // A Resource must have a name, a unit, with at least one numerator. - return (resources[id]->name != NULL && vresource.has_unit && has_numerator); + return (resources[id]->name != NULL && vresource.has_unit && + resources[id]->n_numerators > 0); } -int32_t census_define_resource(const uint8_t *resource_pb, - size_t resource_pb_size) { +// Allocate a blank resource, and return associated ID. Must be called with +// resource_lock held. +size_t allocate_resource(void) { // use next_id to optimize expected placement of next new resource. static size_t next_id = 0; - if (resource_pb == NULL) { - return -1; - } - gpr_mu_lock(&resource_lock); size_t id = n_resources; // resource ID - initialize to invalid value. // Expand resources if needed. if (n_resources == n_defined_resources) { @@ -212,22 +239,25 @@ int32_t census_define_resource(const uint8_t *resource_pb, } GPR_ASSERT(id < n_resources && resources[id] == NULL); resources[id] = gpr_malloc(sizeof(resource)); - resources[id]->name = NULL; + memset(resources[id], 0, sizeof(resource)); + n_defined_resources++; + next_id = (id + 1) % n_resources; + return id; +} + +int32_t census_define_resource(const uint8_t *resource_pb, + size_t resource_pb_size) { + if (resource_pb == NULL) { + return -1; + } + gpr_mu_lock(&resource_lock); + size_t id = allocate_resource(); // Validate pb, extract name. if (!validate_resource_pb(resource_pb, resource_pb_size, id)) { - if (resources[id]->name != NULL) { - gpr_free(resources[id]->name); - } - gpr_free(resources[id]); - resources[id] = NULL; + delete_resource_locked(id); gpr_mu_unlock(&resource_lock); return -1; } - next_id = (id + 1) % n_resources; - // Make copy of raw proto, and return. - resources[id]->raw_pb = gpr_malloc(resource_pb_size); - memcpy(resources[id]->raw_pb, resource_pb, resource_pb_size); - n_defined_resources++; gpr_mu_unlock(&resource_lock); return (int32_t)id; } @@ -253,3 +283,31 @@ int32_t census_resource_id(const char *name) { gpr_mu_unlock(&resource_lock); return -1; } + +int32_t define_resource(const resource *base) { + GPR_ASSERT(base != NULL && base->name != NULL && base->n_numerators > 0 && + base->numerators != NULL); + gpr_mu_lock(&resource_lock); + size_t id = allocate_resource(); + size_t len = strlen(base->name) + 1; + resources[id]->name = gpr_malloc(len); + memcpy(resources[id]->name, base->name, len); + if (base->description) { + len = strlen(base->description); + resources[id]->description = gpr_malloc(len); + memcpy(resources[id]->description, base->description, len); + } + resources[id]->prefix = base->prefix; + resources[id]->n_numerators = base->n_numerators; + len = (size_t)base->n_numerators * sizeof(*base->numerators); + resources[id]->numerators = gpr_malloc(len); + memcpy(resources[id]->numerators, base->numerators, len); + resources[id]->n_denominators = base->n_denominators; + if (base->n_denominators != 0) { + len = (size_t)base->n_denominators * sizeof(*base->denominators); + resources[id]->denominators = gpr_malloc(len); + memcpy(resources[id]->denominators, base->denominators, len); + } + gpr_mu_unlock(&resource_lock); + return (int32_t)id; +} diff --git a/src/core/ext/census/resource.h b/src/core/ext/census/resource.h index 5ad5a0f7a0c..a0669f3a39b 100644 --- a/src/core/ext/census/resource.h +++ b/src/core/ext/census/resource.h @@ -31,12 +31,33 @@ * */ -// Census-internal resource definition and manipluation functions. +/* Census-internal resource definition and manipluation functions. */ #ifndef GRPC_CORE_EXT_CENSUS_RESOURCE_H #define GRPC_CORE_EXT_CENSUS_RESOURCE_H -// Initialize and shutdown the resources subsystem. -void initialize_resources(); -void shutdown_resources(); +#include +#include "gen/census.pb.h" + +/* Internal representation of a resource. */ +typedef struct { + char *name; + char *description; + int32_t prefix; + int n_numerators; + google_census_Resource_BasicUnit *numerators; + int n_denominators; + google_census_Resource_BasicUnit *denominators; +} resource; + +/* Initialize and shutdown the resources subsystem. */ +void initialize_resources(void); +void shutdown_resources(void); + +/* Add a new resource, given a proposed resource structure. Returns the + resource ID, or -ve on failure. + TODO(aveitch): this function exists to support addition of the base + resources. It should be removed when we have the ability to add resources + from configuration files. */ +int32_t define_resource(const resource *base); #endif /* GRPC_CORE_EXT_CENSUS_RESOURCE_H */ From 5855c478c69508f000baa4878f515d72b5f5a1e9 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 8 Jun 2016 12:56:56 -0700 Subject: [PATCH 0063/1039] Use poll if not linux, add read notifier pollset support and some groundwork for adding API that allows users to register custom kick signal number --- include/grpc/impl/codegen/port_platform.h | 1 + src/core/lib/iomgr/ev_epoll_linux.c | 72 ++++++++++++++++------- src/core/lib/iomgr/ev_posix.c | 2 +- 3 files changed, 54 insertions(+), 21 deletions(-) diff --git a/include/grpc/impl/codegen/port_platform.h b/include/grpc/impl/codegen/port_platform.h index be4215a54bf..7a6ec53fb4c 100644 --- a/include/grpc/impl/codegen/port_platform.h +++ b/include/grpc/impl/codegen/port_platform.h @@ -189,6 +189,7 @@ #define GPR_GCC_ATOMIC 1 #define GPR_GCC_TLS 1 #define GPR_LINUX 1 +#define GPR_LINUX_EPOLL 1 #define GPR_LINUX_LOG #define GPR_LINUX_MULTIPOLL_WITH_EPOLL 1 #define GPR_POSIX_WAKEUP_FD 1 diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index d5aac96fa46..69ab665e155 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -33,7 +33,7 @@ #include -#ifdef GPR_POSIX_SOCKET +#ifdef GPR_LINUX_EPOLL #include "src/core/lib/iomgr/ev_epoll_linux.h" @@ -60,6 +60,8 @@ struct polling_island; +static int grpc_poller_kick_signum; + /******************************************************************************* * Fd Declarations */ @@ -92,6 +94,9 @@ struct grpc_fd { struct grpc_fd *freelist_next; grpc_closure *on_done_closure; + /* The pollset that last noticed that the fd is readable */ + grpc_pollset *read_notifier_pollset; + grpc_iomgr_object iomgr_object; }; @@ -650,14 +655,15 @@ static grpc_fd *fd_create(int fd, const char *name) { gpr_mu_lock(&new_fd->mu); gpr_atm_rel_store(&new_fd->refst, 1); + new_fd->fd = fd; new_fd->shutdown = false; + new_fd->orphaned = false; new_fd->read_closure = CLOSURE_NOT_READY; new_fd->write_closure = CLOSURE_NOT_READY; - new_fd->fd = fd; new_fd->polling_island = NULL; new_fd->freelist_next = NULL; new_fd->on_done_closure = NULL; - new_fd->orphaned = false; + new_fd->read_notifier_pollset = NULL; gpr_mu_unlock(&new_fd->mu); @@ -765,6 +771,17 @@ static int set_ready_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, } } +static grpc_pollset *fd_get_read_notifier_pollset(grpc_exec_ctx *exec_ctx, + grpc_fd *fd) { + grpc_pollset *notifier = NULL; + + gpr_mu_lock(&fd->mu); + notifier = fd->read_notifier_pollset; + gpr_mu_unlock(&fd->mu); + + return notifier; +} + static void fd_shutdown(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { gpr_mu_lock(&fd->mu); GPR_ASSERT(!fd->shutdown); @@ -801,16 +818,25 @@ static void sig_handler(int sig_num) { #endif } +static void poller_kick_init() { + grpc_poller_kick_signum = SIGRTMIN + 2; + signal(grpc_poller_kick_signum, sig_handler); +} + /* Global state management */ static void pollset_global_init(void) { grpc_wakeup_fd_init(&grpc_global_wakeup_fd); - signal(SIGUSR1, sig_handler); /* TODO: sreek - Do not hardcode SIGUSR1 */ + poller_kick_init(); } static void pollset_global_shutdown(void) { grpc_wakeup_fd_destroy(&grpc_global_wakeup_fd); } +static void pollset_worker_kick(grpc_pollset_worker *worker) { + pthread_kill(worker->pt_id, grpc_poller_kick_signum); +} + /* Return 1 if the pollset has active threads in pollset_work (pollset must * be locked) */ static int pollset_has_workers(grpc_pollset *p) { @@ -856,7 +882,7 @@ static void pollset_kick(grpc_pollset *p, GPR_TIMER_BEGIN("pollset_kick.broadcast", 0); for (worker = p->root_worker.next; worker != &p->root_worker; worker = worker->next) { - pthread_kill(worker->pt_id, SIGUSR1); + pollset_worker_kick(worker); } } else { p->kicked_without_pollers = true; @@ -864,7 +890,7 @@ static void pollset_kick(grpc_pollset *p, GPR_TIMER_END("pollset_kick.broadcast", 0); } else { GPR_TIMER_MARK("kicked_specifically", 0); - pthread_kill(worker->pt_id, SIGUSR1); + pollset_worker_kick(worker); } } else { GPR_TIMER_MARK("kick_anonymous", 0); @@ -872,7 +898,7 @@ static void pollset_kick(grpc_pollset *p, if (worker != NULL) { GPR_TIMER_MARK("finally_kick", 0); push_back_worker(p, worker); - pthread_kill(worker->pt_id, SIGUSR1); + pollset_worker_kick(worker); } else { GPR_TIMER_MARK("kicked_no_pollers", 0); p->kicked_without_pollers = true; @@ -924,20 +950,20 @@ static int poll_deadline_to_millis_timeout(gpr_timespec deadline, timeout, gpr_time_from_nanos(GPR_NS_PER_MS - 1, GPR_TIMESPAN))); } -static void set_ready(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure **st) { - /* only one set_ready can be active at once (but there may be a racing - notify_on) */ +static void fd_become_readable(grpc_exec_ctx *exec_ctx, grpc_fd *fd, + grpc_pollset *notifier) { + /* Need the fd->mu since we might be racing with fd_notify_on_read */ gpr_mu_lock(&fd->mu); - set_ready_locked(exec_ctx, fd, st); + set_ready_locked(exec_ctx, fd, &fd->read_closure); + fd->read_notifier_pollset = notifier; gpr_mu_unlock(&fd->mu); } -static void fd_become_readable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { - set_ready(exec_ctx, fd, &fd->read_closure); -} - static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { - set_ready(exec_ctx, fd, &fd->write_closure); + /* Need the fd->mu since we might be racing with fd_notify_on_write */ + gpr_mu_lock(&fd->mu); + set_ready_locked(exec_ctx, fd, &fd->write_closure); + gpr_mu_unlock(&fd->mu); } #define GRPC_EPOLL_MAX_EVENTS 1000 @@ -1007,7 +1033,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); } else { if (read_ev || cancel) { - fd_become_readable(exec_ctx, fd); + fd_become_readable(exec_ctx, fd, pollset); } if (write_ev || cancel) { fd_become_writable(exec_ctx, fd); @@ -1109,9 +1135,9 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, pollset->kicked_without_pollers = 0; } else if (!pollset->shutting_down) { sigemptyset(&new_mask); - sigaddset(&new_mask, SIGUSR1); + sigaddset(&new_mask, grpc_poller_kick_signum); pthread_sigmask(SIG_BLOCK, &new_mask, &orig_mask); - sigdelset(&orig_mask, SIGUSR1); + sigdelset(&orig_mask, grpc_poller_kick_signum); push_front_worker(pollset, &worker); @@ -1350,6 +1376,7 @@ static const grpc_event_engine_vtable vtable = { .fd_shutdown = fd_shutdown, .fd_notify_on_read = fd_notify_on_read, .fd_notify_on_write = fd_notify_on_write, + .fd_get_read_notifier_pollset = fd_get_read_notifier_pollset, .pollset_init = pollset_init, .pollset_shutdown = pollset_shutdown, @@ -1380,4 +1407,9 @@ const grpc_event_engine_vtable *grpc_init_epoll_linux(void) { return &vtable; } -#endif +#else /* defined(GPR_LINUX_EPOLL) */ +/* If GPR_LINUX_EPOLL is not defined, it means epoll is not available. Return + * NULL */ +const grpc_event_engine_vtable *grpc_init_epoll_linux(void) { return NULL; } + +#endif /* !defined(GPR_LINUX_EPOLL) */ diff --git a/src/core/lib/iomgr/ev_posix.c b/src/core/lib/iomgr/ev_posix.c index e0c3558a51e..2b15967adcc 100644 --- a/src/core/lib/iomgr/ev_posix.c +++ b/src/core/lib/iomgr/ev_posix.c @@ -63,8 +63,8 @@ typedef struct { } event_engine_factory; static const event_engine_factory g_factories[] = { - {"poll", grpc_init_poll_posix}, {"epoll", grpc_init_epoll_linux}, + {"poll", grpc_init_poll_posix}, {"legacy", grpc_init_poll_and_epoll_posix}, }; From 24b1062f42ef01bd47a458e94423f068ec1765f0 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 8 Jun 2016 15:20:17 -0700 Subject: [PATCH 0064/1039] Do not close epoll_fd while there are any pollers and add the ability to wake up all pollers when an island is merged --- src/core/lib/iomgr/ev_epoll_linux.c | 113 ++++++++++++++++++++-------- 1 file changed, 80 insertions(+), 33 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 69ab665e155..3a3c136a5aa 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -190,9 +190,18 @@ struct grpc_pollset_set { }; /******************************************************************************* - * Polling-island Definitions + * Polling island Definitions */ +/* The wakeup fd that is used to wake up all threads in a Polling island. This + is useful in the polling island merge operation where we need to wakeup all + the threads currently polling the smaller polling island (so that they can + start polling the new/merged polling island) + + NOTE: This fd is initialized to be readable and MUST NOT be consumed i.e the + threads that woke up MUST NOT call grpc_wakeup_fd_consume_wakeup() */ +static grpc_wakeup_fd polling_island_wakeup_fd; + /* Polling island freelist */ static gpr_mu g_pi_freelist_mu; static polling_island *g_pi_freelist = NULL; @@ -232,6 +241,25 @@ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, } } +/* The caller is expected to hold pi->mu before calling this */ +static void polling_island_add_wakeup_fd_locked(polling_island *pi, + grpc_wakeup_fd *wakeup_fd) { + struct epoll_event ev; + int err; + + ev.events = (uint32_t)(EPOLLIN | EPOLLET); + ev.data.ptr = wakeup_fd; + err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_ADD, + GRPC_WAKEUP_FD_GET_READ_FD(wakeup_fd), &ev); + if (err < 0) { + gpr_log(GPR_ERROR, + "Failed to add grpc_wake_up_fd (%d) to the epoll set (epoll_fd: %d)" + ". Error: %s", + GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), pi->epoll_fd, + strerror(errno)); + } +} + /* The caller is expected to hold pi->mu lock before calling this function */ static void polling_island_remove_all_fds_locked(polling_island *pi, bool remove_fd_refs) { @@ -283,8 +311,6 @@ static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd, static polling_island *polling_island_create(grpc_fd *initial_fd, int initial_ref_cnt) { polling_island *pi = NULL; - struct epoll_event ev; - int err; /* Try to get one from the polling island freelist */ gpr_mu_lock(&g_pi_freelist_mu); @@ -311,17 +337,7 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, } GPR_ASSERT(pi->epoll_fd >= 0); - ev.events = (uint32_t)(EPOLLIN | EPOLLET); - ev.data.ptr = NULL; - err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_ADD, - GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), &ev); - if (err < 0) { - gpr_log(GPR_ERROR, - "Failed to add grpc_global_wake_up_fd (%d) to the epoll set " - "(epoll_fd: %d) with error: %s", - GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), pi->epoll_fd, - strerror(errno)); - } + polling_island_add_wakeup_fd_locked(pi, &grpc_global_wakeup_fd); pi->ref_cnt = initial_ref_cnt; pi->merged_to = NULL; @@ -496,13 +512,15 @@ polling_island *polling_island_merge(polling_island *p, polling_island *q) { GPR_SWAP(polling_island *, p, q); } - /* "Merge" p with q i.e move all the fds from p (the polling_island with fewer - fds) to q. - Note: Not altering the ref counts on the affected fds here because they - would effectively remain unchanged */ + /* "Merge" p with q i.e move all the fds from p (The one with fewer fds) to q + )Note that the refcounts on the fds being moved will not change here. This + is why the last parameter in the following two functions is 'false') */ polling_island_add_fds_locked(q, p->fds, p->fd_cnt, false); polling_island_remove_all_fds_locked(p, false); + /* Wakeup all the pollers (if any) on p so that they can pickup this change */ + polling_island_add_wakeup_fd_locked(p, &polling_island_wakeup_fd); + /* The merged polling island inherits all the ref counts of the island merging with it */ q->ref_cnt += p->ref_cnt; @@ -516,6 +534,8 @@ polling_island *polling_island_merge(polling_island *p, polling_island *q) { static void polling_island_global_init() { gpr_mu_init(&g_pi_freelist_mu); g_pi_freelist = NULL; + grpc_wakeup_fd_init(&polling_island_wakeup_fd); + grpc_wakeup_fd_wakeup(&polling_island_wakeup_fd); } static void polling_island_global_shutdown() { @@ -529,8 +549,9 @@ static void polling_island_global_shutdown() { gpr_free(g_pi_freelist); g_pi_freelist = next; } - gpr_mu_destroy(&g_pi_freelist_mu); + + grpc_wakeup_fd_destroy(&polling_island_wakeup_fd); } /******************************************************************************* @@ -973,6 +994,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; int epoll_fd = -1; int ep_rv; + polling_island *pi = NULL; GPR_TIMER_BEGIN("pollset_work_and_unlock", 0); /* We need to get the epoll_fd to wait on. The epoll_fd is in inside the @@ -983,13 +1005,19 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, - pollset->polling_island->mu */ gpr_mu_lock(&pollset->pi_mu); - if (pollset->polling_island == NULL) { - pollset->polling_island = polling_island_create(NULL, 1); + pi = pollset->polling_island; + if (pi == NULL) { + pi = polling_island_create(NULL, 1); } - pollset->polling_island = - polling_island_update_and_lock(pollset->polling_island, 1, 0); - epoll_fd = pollset->polling_island->epoll_fd; + /* In addition to locking the polling island, add a ref so that the island + does not get destroyed (which means the epoll_fd won't be closed) while + we are are doing an epoll_wait() on the epoll_fd */ + pi = polling_island_update_and_lock(pi, 1, 1); + epoll_fd = pi->epoll_fd; + + /* Update the pollset->polling_island */ + pollset->polling_island = pi; #ifdef GRPC_EPOLL_DEBUG if (pollset->polling_island->fd_cnt == 0) { @@ -1013,25 +1041,29 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, sig_mask); if (ep_rv < 0) { if (errno != EINTR) { - /* TODO (sreek) - Do not log an error in case of bad file descriptor - * (A bad file descriptor here would just mean that the epoll set was - * merged with another epoll set and that the current epoll_fd is - * closed) */ gpr_log(GPR_ERROR, "epoll_pwait() failed: %s", strerror(errno)); } else { + /* We were interrupted. Save an interation by doing a zero timeout + epoll_wait to see if there are any other events of interest */ ep_rv = epoll_wait(epoll_fd, ep_ev, GRPC_EPOLL_MAX_EVENTS, 0); } } int i; for (i = 0; i < ep_rv; ++i) { - grpc_fd *fd = ep_ev[i].data.ptr; - int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); - int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); - int write_ev = ep_ev[i].events & EPOLLOUT; - if (fd == NULL) { + void *data_ptr = ep_ev[i].data.ptr; + if (data_ptr == &grpc_global_wakeup_fd) { grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); + } else if (data_ptr == &polling_island_wakeup_fd) { + /* This means that our polling island is merged with a different + island. We do not have to do anything here since the subsequent call + to the function pollset_work_and_unlock() will pick up the correct + epoll_fd */ } else { + grpc_fd *fd = data_ptr; + int cancel = ep_ev[i].events & (EPOLLERR | EPOLLHUP); + int read_ev = ep_ev[i].events & (EPOLLIN | EPOLLPRI); + int write_ev = ep_ev[i].events & EPOLLOUT; if (read_ev || cancel) { fd_become_readable(exec_ctx, fd, pollset); } @@ -1041,6 +1073,21 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, } } } while (ep_rv == GRPC_EPOLL_MAX_EVENTS); + + GPR_ASSERT(pi != NULL); + + /* Before leaving, release the extra ref we added to the polling island */ + /* It is important to note that at this point 'pi' may not be the same as + * pollset->polling_island. This is because pollset->polling_island pointer + * gets updated whenever the underlying polling island is merged with another + * island and while we are doing epoll_wait() above, the polling island may + * have been merged */ + + /* TODO (sreek) - Change the ref count on polling island to gpr_atm so that + * we do not have to do this here */ + gpr_mu_lock(&pi->mu); + polling_island_unref_and_unlock(pi, 1); + GPR_TIMER_END("pollset_work_and_unlock", 0); } From e682e46a9e451a94cfcb1cf9c927185abd81fceb Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 8 Jun 2016 15:40:21 -0700 Subject: [PATCH 0065/1039] Add TODOs --- src/core/lib/iomgr/ev_epoll_linux.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 3a3c136a5aa..2e871a4f1ba 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -83,6 +83,7 @@ struct grpc_fd { this indicates that the 'fd' on this structure is no longer valid */ bool orphaned; + /* TODO: sreek - Move this a lockfree implementation */ grpc_closure *read_closure; grpc_closure *write_closure; @@ -166,6 +167,9 @@ struct grpc_pollset { /* The polling island to which this pollset belongs to and the mutex protecting the field */ + /* TODO: sreek: This lock might actually be adding more overhead to the + critical path (i.e pollset_work() function). Consider removing this lock + and just using the overall pollset lock */ gpr_mu pi_mu; struct polling_island *polling_island; }; From 3dbf4d61b26e2364a974e47f16f3a655d3eda908 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 8 Jun 2016 16:26:45 -0700 Subject: [PATCH 0066/1039] More TODOs --- src/core/lib/iomgr/ev_epoll_linux.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 2e871a4f1ba..046ec5e7407 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -83,7 +83,7 @@ struct grpc_fd { this indicates that the 'fd' on this structure is no longer valid */ bool orphaned; - /* TODO: sreek - Move this a lockfree implementation */ + /* TODO: sreek - Move this to a lockfree implementation */ grpc_closure *read_closure; grpc_closure *write_closure; @@ -124,6 +124,8 @@ static void fd_global_shutdown(void); /******************************************************************************* * Polling-island Declarations */ +/* TODO: sree: Consider making ref_cnt and merged_to to gpr_atm - This would + * significantly reduce the number of mutex acquisition calls. */ typedef struct polling_island { gpr_mu mu; int ref_cnt; @@ -177,6 +179,12 @@ struct grpc_pollset { /******************************************************************************* * Pollset-set Declarations */ +/* TODO: sreek - Change the pollset_set implementation such that a pollset_set + * directly points to a polling_island (and adding an fd/pollset/pollset_set to + * the current pollset_set would result in polling island merges. This would + * remove the need to maintain fd_count here. This will also significantly + * simplify the grpc_fd structure since we would no longer need to explicitly + * maintain the orphaned state */ struct grpc_pollset_set { gpr_mu mu; From 5842277ecbbbe694a828bd51934a28673f2f0a7f Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Mon, 25 Apr 2016 11:08:19 -0700 Subject: [PATCH 0067/1039] Make BoringSSL work with frameworks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cocoapods 1.0 keeps the directory structure of the public headers when creating a dynamic framework, so if we keep the header_mappings_dir as it was, includes would need to be of the form #include This means our header_mappings_dir has to be ‘include/openssl’ instead of ‘include’. Which in turn means that, for static libraries, we have to tell Cocoapods to prepend an ‘openssl’ directory to the headers. We do that with the ‘header_dir’ attribute of the podspec. --- src/objective-c/BoringSSL.podspec | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/src/objective-c/BoringSSL.podspec b/src/objective-c/BoringSSL.podspec index 4a6df910a75..3cd067e5d19 100644 --- a/src/objective-c/BoringSSL.podspec +++ b/src/objective-c/BoringSSL.podspec @@ -31,7 +31,7 @@ Pod::Spec.new do |s| s.name = 'BoringSSL' - s.version = '2.0' + s.version = '3.0' s.summary = 'BoringSSL is a fork of OpenSSL that is designed to meet Google’s needs.' # Adapted from the homepage: s.description = <<-DESC @@ -69,15 +69,19 @@ Pod::Spec.new do |s| s.source = { :git => 'https://boringssl.googlesource.com/boringssl', :tag => 'version_for_cocoapods_2.0' } + name = 'openssl' + s.module_name = name + s.header_dir = name + s.source_files = 'ssl/*.{h,c}', 'ssl/**/*.{h,c}', '*.{h,c}', 'crypto/*.{h,c}', 'crypto/**/*.{h,c}', 'include/openssl/*.h' - s.public_header_files = 'include/openssl/*.h' - s.header_mappings_dir = 'include' + s.header_mappings_dir = 'include/openssl' + s.module_map = 'include/openssl/module.modulemap' s.exclude_files = "**/*_test.*" @@ -92,6 +96,25 @@ Pod::Spec.new do |s| # included it in every bridged header). sed -E -i '.back' 's/\\*I,/*i,/g' include/openssl/rsa.h + # Replace `#include "../crypto/internal.h"` in e_tls.c with `#include "../internal.h"`. The + # former assumes crypto/ is in the headers search path, which is hard to enforce when using + # dynamic frameworks. The latters always works, being relative to the current file. + sed -E -i '.back' 's/crypto\\///g' crypto/cipher/e_tls.c + + # Add a module map and an umbrella header + cat > include/openssl/umbrella.h < include/openssl/module.modulemap < Date: Fri, 3 Jun 2016 17:33:18 -0700 Subject: [PATCH 0068/1039] Remove #include from BoringSSL MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Apple hasn’t created a module map for that system header, which means it can’t be used from frameworks. --- src/objective-c/BoringSSL.podspec | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/objective-c/BoringSSL.podspec b/src/objective-c/BoringSSL.podspec index 3cd067e5d19..1954a5c089b 100644 --- a/src/objective-c/BoringSSL.podspec +++ b/src/objective-c/BoringSSL.podspec @@ -115,6 +115,16 @@ Pod::Spec.new do |s| } EOF + # #include fails to compile when building a dynamic framework. libgit2 in + # https://github.com/libgit2/libgit2/commit/1ddada422caf8e72ba97dca2568d2bf879fed5f2 and libvpx + # in https://chromium.googlesource.com/webm/libvpx/+/1bec0c5a7e885ec792f6bb658eb3f34ad8f37b15 + # work around it by removing the include. We need four of its macros, so we expand them here. + sed -E -i '.back' '//d' include/openssl/bn.h + sed -E -i '.back' 's/PRIu32/"u"/g' include/openssl/bn.h + sed -E -i '.back' 's/PRIx32/"x"/g' include/openssl/bn.h + sed -E -i '.back' 's/PRIu64/"llu"/g' include/openssl/bn.h + sed -E -i '.back' 's/PRIx64/"llx"/g' include/openssl/bn.h + # This is a bit ridiculous, but requiring people to install Go in order to build is slightly # more ridiculous IMO. To save you from scrolling, this is the last part of the podspec. # TODO(jcanizales): Translate err_data_generate.go into a Bash or Ruby script. From e487a7271d21f60f1472a9c24ad3f026e048653c Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Sat, 30 Apr 2016 12:05:26 -0700 Subject: [PATCH 0069/1039] Separate Core, ProtoRuntime, and RxLibrary subspecs --- gRPC-Core.podspec | 658 ++++++++++++++++++ gRPC-ProtoRPC.podspec | 69 ++ gRPC-RxLibrary.podspec | 62 ++ gRPC.podspec | 651 +---------------- src/objective-c/tests/Podfile | 3 + ...ec.template => gRPC-Core.podspec.template} | 89 +-- 6 files changed, 838 insertions(+), 694 deletions(-) create mode 100644 gRPC-Core.podspec create mode 100644 gRPC-ProtoRPC.podspec create mode 100644 gRPC-RxLibrary.podspec rename templates/{gRPC.podspec.template => gRPC-Core.podspec.template} (51%) diff --git a/gRPC-Core.podspec b/gRPC-Core.podspec new file mode 100644 index 00000000000..c924ab3a5df --- /dev/null +++ b/gRPC-Core.podspec @@ -0,0 +1,658 @@ +# GRPC CocoaPods podspec +# This file has been automatically generated from a template file. +# Please look at the templates directory instead. +# This file can be regenerated from the template by running +# tools/buildgen/generate_projects.sh + +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Pod::Spec.new do |s| + s.name = 'gRPC-Core' + version = '0.14.0' + s.version = version + s.summary = 'Core cross-platform gRPC library, written in C' + s.homepage = 'http://www.grpc.io' + s.license = 'New BSD' + s.authors = { 'The gRPC contributors' => 'grpc-packages@google.com' } + + s.source = { + :git => 'https://github.com/grpc/grpc.git', + :tag => "release-#{version.gsub(/\./, '_')}-objectivec-#{version}", + } + + s.ios.deployment_target = '7.1' + s.osx.deployment_target = '10.9' + s.requires_arc = false + + name = 'grpc' + s.module_name = name + s.header_dir = name + + s.source_files = 'src/core/lib/profiling/timers.h', + 'src/core/lib/support/backoff.h', + 'src/core/lib/support/block_annotate.h', + 'src/core/lib/support/env.h', + 'src/core/lib/support/load_file.h', + 'src/core/lib/support/murmur_hash.h', + 'src/core/lib/support/stack_lockfree.h', + 'src/core/lib/support/string.h', + 'src/core/lib/support/string_win32.h', + 'src/core/lib/support/thd_internal.h', + 'src/core/lib/support/time_precise.h', + 'src/core/lib/support/tmpfile.h', + 'include/grpc/support/alloc.h', + 'include/grpc/support/atm.h', + 'include/grpc/support/atm_gcc_atomic.h', + 'include/grpc/support/atm_gcc_sync.h', + 'include/grpc/support/atm_win32.h', + 'include/grpc/support/avl.h', + 'include/grpc/support/cmdline.h', + 'include/grpc/support/cpu.h', + 'include/grpc/support/histogram.h', + 'include/grpc/support/host_port.h', + 'include/grpc/support/log.h', + 'include/grpc/support/log_win32.h', + 'include/grpc/support/port_platform.h', + 'include/grpc/support/slice.h', + 'include/grpc/support/slice_buffer.h', + 'include/grpc/support/string_util.h', + 'include/grpc/support/subprocess.h', + 'include/grpc/support/sync.h', + 'include/grpc/support/sync_generic.h', + 'include/grpc/support/sync_posix.h', + 'include/grpc/support/sync_win32.h', + 'include/grpc/support/thd.h', + 'include/grpc/support/time.h', + 'include/grpc/support/tls.h', + 'include/grpc/support/tls_gcc.h', + 'include/grpc/support/tls_msvc.h', + 'include/grpc/support/tls_pthread.h', + 'include/grpc/support/useful.h', + 'include/grpc/impl/codegen/alloc.h', + 'include/grpc/impl/codegen/atm.h', + 'include/grpc/impl/codegen/atm_gcc_atomic.h', + 'include/grpc/impl/codegen/atm_gcc_sync.h', + 'include/grpc/impl/codegen/atm_win32.h', + 'include/grpc/impl/codegen/log.h', + 'include/grpc/impl/codegen/port_platform.h', + 'include/grpc/impl/codegen/slice.h', + 'include/grpc/impl/codegen/slice_buffer.h', + 'include/grpc/impl/codegen/sync.h', + 'include/grpc/impl/codegen/sync_generic.h', + 'include/grpc/impl/codegen/sync_posix.h', + 'include/grpc/impl/codegen/sync_win32.h', + 'include/grpc/impl/codegen/time.h', + 'src/core/lib/profiling/basic_timers.c', + 'src/core/lib/profiling/stap_timers.c', + 'src/core/lib/support/alloc.c', + 'src/core/lib/support/avl.c', + 'src/core/lib/support/backoff.c', + 'src/core/lib/support/cmdline.c', + 'src/core/lib/support/cpu_iphone.c', + 'src/core/lib/support/cpu_linux.c', + 'src/core/lib/support/cpu_posix.c', + 'src/core/lib/support/cpu_windows.c', + 'src/core/lib/support/env_linux.c', + 'src/core/lib/support/env_posix.c', + 'src/core/lib/support/env_win32.c', + 'src/core/lib/support/histogram.c', + 'src/core/lib/support/host_port.c', + 'src/core/lib/support/load_file.c', + 'src/core/lib/support/log.c', + 'src/core/lib/support/log_android.c', + 'src/core/lib/support/log_linux.c', + 'src/core/lib/support/log_posix.c', + 'src/core/lib/support/log_win32.c', + 'src/core/lib/support/murmur_hash.c', + 'src/core/lib/support/slice.c', + 'src/core/lib/support/slice_buffer.c', + 'src/core/lib/support/stack_lockfree.c', + 'src/core/lib/support/string.c', + 'src/core/lib/support/string_posix.c', + 'src/core/lib/support/string_util_win32.c', + 'src/core/lib/support/string_win32.c', + 'src/core/lib/support/subprocess_posix.c', + 'src/core/lib/support/subprocess_windows.c', + 'src/core/lib/support/sync.c', + 'src/core/lib/support/sync_posix.c', + 'src/core/lib/support/sync_win32.c', + 'src/core/lib/support/thd.c', + 'src/core/lib/support/thd_posix.c', + 'src/core/lib/support/thd_win32.c', + 'src/core/lib/support/time.c', + 'src/core/lib/support/time_posix.c', + 'src/core/lib/support/time_precise.c', + 'src/core/lib/support/time_win32.c', + 'src/core/lib/support/tls_pthread.c', + 'src/core/lib/support/tmpfile_msys.c', + 'src/core/lib/support/tmpfile_posix.c', + 'src/core/lib/support/tmpfile_win32.c', + 'src/core/lib/support/wrap_memcpy.c', + 'src/core/lib/channel/channel_args.h', + 'src/core/lib/channel/channel_stack.h', + 'src/core/lib/channel/channel_stack_builder.h', + 'src/core/lib/channel/compress_filter.h', + 'src/core/lib/channel/connected_channel.h', + 'src/core/lib/channel/context.h', + 'src/core/lib/channel/http_client_filter.h', + 'src/core/lib/channel/http_server_filter.h', + 'src/core/lib/compression/algorithm_metadata.h', + 'src/core/lib/compression/message_compress.h', + 'src/core/lib/debug/trace.h', + 'src/core/lib/http/format_request.h', + 'src/core/lib/http/httpcli.h', + 'src/core/lib/http/parser.h', + 'src/core/lib/iomgr/closure.h', + 'src/core/lib/iomgr/endpoint.h', + 'src/core/lib/iomgr/endpoint_pair.h', + 'src/core/lib/iomgr/ev_poll_and_epoll_posix.h', + 'src/core/lib/iomgr/ev_posix.h', + 'src/core/lib/iomgr/exec_ctx.h', + 'src/core/lib/iomgr/executor.h', + 'src/core/lib/iomgr/iocp_windows.h', + 'src/core/lib/iomgr/iomgr.h', + 'src/core/lib/iomgr/iomgr_internal.h', + 'src/core/lib/iomgr/iomgr_posix.h', + 'src/core/lib/iomgr/pollset.h', + 'src/core/lib/iomgr/pollset_set.h', + 'src/core/lib/iomgr/pollset_set_windows.h', + 'src/core/lib/iomgr/pollset_windows.h', + 'src/core/lib/iomgr/resolve_address.h', + 'src/core/lib/iomgr/sockaddr.h', + 'src/core/lib/iomgr/sockaddr_posix.h', + 'src/core/lib/iomgr/sockaddr_utils.h', + 'src/core/lib/iomgr/sockaddr_win32.h', + 'src/core/lib/iomgr/socket_utils_posix.h', + 'src/core/lib/iomgr/socket_windows.h', + 'src/core/lib/iomgr/tcp_client.h', + 'src/core/lib/iomgr/tcp_posix.h', + 'src/core/lib/iomgr/tcp_server.h', + 'src/core/lib/iomgr/tcp_windows.h', + 'src/core/lib/iomgr/time_averaged_stats.h', + 'src/core/lib/iomgr/timer.h', + 'src/core/lib/iomgr/timer_heap.h', + 'src/core/lib/iomgr/udp_server.h', + 'src/core/lib/iomgr/unix_sockets_posix.h', + 'src/core/lib/iomgr/wakeup_fd_pipe.h', + 'src/core/lib/iomgr/wakeup_fd_posix.h', + 'src/core/lib/iomgr/workqueue.h', + 'src/core/lib/iomgr/workqueue_posix.h', + 'src/core/lib/iomgr/workqueue_windows.h', + 'src/core/lib/json/json.h', + 'src/core/lib/json/json_common.h', + 'src/core/lib/json/json_reader.h', + 'src/core/lib/json/json_writer.h', + 'src/core/lib/surface/api_trace.h', + 'src/core/lib/surface/call.h', + 'src/core/lib/surface/call_test_only.h', + 'src/core/lib/surface/channel.h', + 'src/core/lib/surface/channel_init.h', + 'src/core/lib/surface/channel_stack_type.h', + 'src/core/lib/surface/completion_queue.h', + 'src/core/lib/surface/event_string.h', + 'src/core/lib/surface/init.h', + 'src/core/lib/surface/lame_client.h', + 'src/core/lib/surface/server.h', + 'src/core/lib/surface/surface_trace.h', + 'src/core/lib/transport/byte_stream.h', + 'src/core/lib/transport/connectivity_state.h', + 'src/core/lib/transport/metadata.h', + 'src/core/lib/transport/metadata_batch.h', + 'src/core/lib/transport/static_metadata.h', + 'src/core/lib/transport/transport.h', + 'src/core/lib/transport/transport_impl.h', + 'src/core/ext/transport/chttp2/transport/bin_encoder.h', + 'src/core/ext/transport/chttp2/transport/chttp2_transport.h', + 'src/core/ext/transport/chttp2/transport/frame.h', + 'src/core/ext/transport/chttp2/transport/frame_data.h', + 'src/core/ext/transport/chttp2/transport/frame_goaway.h', + 'src/core/ext/transport/chttp2/transport/frame_ping.h', + 'src/core/ext/transport/chttp2/transport/frame_rst_stream.h', + 'src/core/ext/transport/chttp2/transport/frame_settings.h', + 'src/core/ext/transport/chttp2/transport/frame_window_update.h', + 'src/core/ext/transport/chttp2/transport/hpack_encoder.h', + 'src/core/ext/transport/chttp2/transport/hpack_parser.h', + 'src/core/ext/transport/chttp2/transport/hpack_table.h', + 'src/core/ext/transport/chttp2/transport/http2_errors.h', + 'src/core/ext/transport/chttp2/transport/huffsyms.h', + 'src/core/ext/transport/chttp2/transport/incoming_metadata.h', + 'src/core/ext/transport/chttp2/transport/internal.h', + 'src/core/ext/transport/chttp2/transport/status_conversion.h', + 'src/core/ext/transport/chttp2/transport/stream_map.h', + 'src/core/ext/transport/chttp2/transport/timeout_encoding.h', + 'src/core/ext/transport/chttp2/transport/varint.h', + 'src/core/ext/transport/chttp2/alpn/alpn.h', + 'src/core/lib/security/auth_filters.h', + 'src/core/lib/security/b64.h', + 'src/core/lib/security/credentials.h', + 'src/core/lib/security/handshake.h', + 'src/core/lib/security/json_token.h', + 'src/core/lib/security/jwt_verifier.h', + 'src/core/lib/security/secure_endpoint.h', + 'src/core/lib/security/security_connector.h', + 'src/core/lib/security/security_context.h', + 'src/core/lib/tsi/fake_transport_security.h', + 'src/core/lib/tsi/ssl_transport_security.h', + 'src/core/lib/tsi/ssl_types.h', + 'src/core/lib/tsi/transport_security.h', + 'src/core/lib/tsi/transport_security_interface.h', + 'src/core/ext/client_config/client_channel.h', + 'src/core/ext/client_config/client_channel_factory.h', + 'src/core/ext/client_config/client_config.h', + 'src/core/ext/client_config/connector.h', + 'src/core/ext/client_config/initial_connect_string.h', + 'src/core/ext/client_config/lb_policy.h', + 'src/core/ext/client_config/lb_policy_factory.h', + 'src/core/ext/client_config/lb_policy_registry.h', + 'src/core/ext/client_config/parse_address.h', + 'src/core/ext/client_config/resolver.h', + 'src/core/ext/client_config/resolver_factory.h', + 'src/core/ext/client_config/resolver_registry.h', + 'src/core/ext/client_config/subchannel.h', + 'src/core/ext/client_config/subchannel_call_holder.h', + 'src/core/ext/client_config/subchannel_index.h', + 'src/core/ext/client_config/uri_parser.h', + 'src/core/ext/lb_policy/grpclb/load_balancer_api.h', + 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v0/load_balancer.pb.h', + 'third_party/nanopb/pb.h', + 'third_party/nanopb/pb_common.h', + 'third_party/nanopb/pb_decode.h', + 'third_party/nanopb/pb_encode.h', + 'src/core/ext/census/aggregation.h', + 'src/core/ext/census/census_interface.h', + 'src/core/ext/census/census_rpc_stats.h', + 'src/core/ext/census/grpc_filter.h', + 'src/core/ext/census/mlog.h', + 'src/core/ext/census/rpc_metric_id.h', + 'include/grpc/byte_buffer.h', + 'include/grpc/byte_buffer_reader.h', + 'include/grpc/compression.h', + 'include/grpc/grpc.h', + 'include/grpc/status.h', + 'include/grpc/impl/codegen/byte_buffer.h', + 'include/grpc/impl/codegen/byte_buffer_reader.h', + 'include/grpc/impl/codegen/compression_types.h', + 'include/grpc/impl/codegen/connectivity_state.h', + 'include/grpc/impl/codegen/grpc_types.h', + 'include/grpc/impl/codegen/propagation_bits.h', + 'include/grpc/impl/codegen/status.h', + 'include/grpc/impl/codegen/alloc.h', + 'include/grpc/impl/codegen/atm.h', + 'include/grpc/impl/codegen/atm_gcc_atomic.h', + 'include/grpc/impl/codegen/atm_gcc_sync.h', + 'include/grpc/impl/codegen/atm_win32.h', + 'include/grpc/impl/codegen/log.h', + 'include/grpc/impl/codegen/port_platform.h', + 'include/grpc/impl/codegen/slice.h', + 'include/grpc/impl/codegen/slice_buffer.h', + 'include/grpc/impl/codegen/sync.h', + 'include/grpc/impl/codegen/sync_generic.h', + 'include/grpc/impl/codegen/sync_posix.h', + 'include/grpc/impl/codegen/sync_win32.h', + 'include/grpc/impl/codegen/time.h', + 'include/grpc/grpc_security.h', + 'include/grpc/grpc_security_constants.h', + 'include/grpc/census.h', + 'src/core/lib/surface/init.c', + 'src/core/lib/channel/channel_args.c', + 'src/core/lib/channel/channel_stack.c', + 'src/core/lib/channel/channel_stack_builder.c', + 'src/core/lib/channel/compress_filter.c', + 'src/core/lib/channel/connected_channel.c', + 'src/core/lib/channel/http_client_filter.c', + 'src/core/lib/channel/http_server_filter.c', + 'src/core/lib/compression/compression_algorithm.c', + 'src/core/lib/compression/message_compress.c', + 'src/core/lib/debug/trace.c', + 'src/core/lib/http/format_request.c', + 'src/core/lib/http/httpcli.c', + 'src/core/lib/http/parser.c', + 'src/core/lib/iomgr/closure.c', + 'src/core/lib/iomgr/endpoint.c', + 'src/core/lib/iomgr/endpoint_pair_posix.c', + 'src/core/lib/iomgr/endpoint_pair_windows.c', + 'src/core/lib/iomgr/ev_poll_and_epoll_posix.c', + 'src/core/lib/iomgr/ev_posix.c', + 'src/core/lib/iomgr/exec_ctx.c', + 'src/core/lib/iomgr/executor.c', + 'src/core/lib/iomgr/iocp_windows.c', + 'src/core/lib/iomgr/iomgr.c', + 'src/core/lib/iomgr/iomgr_posix.c', + 'src/core/lib/iomgr/iomgr_windows.c', + 'src/core/lib/iomgr/pollset_set_windows.c', + 'src/core/lib/iomgr/pollset_windows.c', + 'src/core/lib/iomgr/resolve_address_posix.c', + 'src/core/lib/iomgr/resolve_address_windows.c', + 'src/core/lib/iomgr/sockaddr_utils.c', + 'src/core/lib/iomgr/socket_utils_common_posix.c', + 'src/core/lib/iomgr/socket_utils_linux.c', + 'src/core/lib/iomgr/socket_utils_posix.c', + 'src/core/lib/iomgr/socket_windows.c', + 'src/core/lib/iomgr/tcp_client_posix.c', + 'src/core/lib/iomgr/tcp_client_windows.c', + 'src/core/lib/iomgr/tcp_posix.c', + 'src/core/lib/iomgr/tcp_server_posix.c', + 'src/core/lib/iomgr/tcp_server_windows.c', + 'src/core/lib/iomgr/tcp_windows.c', + 'src/core/lib/iomgr/time_averaged_stats.c', + 'src/core/lib/iomgr/timer.c', + 'src/core/lib/iomgr/timer_heap.c', + 'src/core/lib/iomgr/udp_server.c', + 'src/core/lib/iomgr/unix_sockets_posix.c', + 'src/core/lib/iomgr/unix_sockets_posix_noop.c', + 'src/core/lib/iomgr/wakeup_fd_eventfd.c', + 'src/core/lib/iomgr/wakeup_fd_nospecial.c', + 'src/core/lib/iomgr/wakeup_fd_pipe.c', + 'src/core/lib/iomgr/wakeup_fd_posix.c', + 'src/core/lib/iomgr/workqueue_posix.c', + 'src/core/lib/iomgr/workqueue_windows.c', + 'src/core/lib/json/json.c', + 'src/core/lib/json/json_reader.c', + 'src/core/lib/json/json_string.c', + 'src/core/lib/json/json_writer.c', + 'src/core/lib/surface/alarm.c', + 'src/core/lib/surface/api_trace.c', + 'src/core/lib/surface/byte_buffer.c', + 'src/core/lib/surface/byte_buffer_reader.c', + 'src/core/lib/surface/call.c', + 'src/core/lib/surface/call_details.c', + 'src/core/lib/surface/call_log_batch.c', + 'src/core/lib/surface/channel.c', + 'src/core/lib/surface/channel_init.c', + 'src/core/lib/surface/channel_ping.c', + 'src/core/lib/surface/channel_stack_type.c', + 'src/core/lib/surface/completion_queue.c', + 'src/core/lib/surface/event_string.c', + 'src/core/lib/surface/lame_client.c', + 'src/core/lib/surface/metadata_array.c', + 'src/core/lib/surface/server.c', + 'src/core/lib/surface/validate_metadata.c', + 'src/core/lib/surface/version.c', + 'src/core/lib/transport/byte_stream.c', + 'src/core/lib/transport/connectivity_state.c', + 'src/core/lib/transport/metadata.c', + 'src/core/lib/transport/metadata_batch.c', + 'src/core/lib/transport/static_metadata.c', + 'src/core/lib/transport/transport.c', + 'src/core/lib/transport/transport_op_string.c', + 'src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c', + 'src/core/ext/transport/chttp2/transport/bin_encoder.c', + 'src/core/ext/transport/chttp2/transport/chttp2_plugin.c', + 'src/core/ext/transport/chttp2/transport/chttp2_transport.c', + 'src/core/ext/transport/chttp2/transport/frame_data.c', + 'src/core/ext/transport/chttp2/transport/frame_goaway.c', + 'src/core/ext/transport/chttp2/transport/frame_ping.c', + 'src/core/ext/transport/chttp2/transport/frame_rst_stream.c', + 'src/core/ext/transport/chttp2/transport/frame_settings.c', + 'src/core/ext/transport/chttp2/transport/frame_window_update.c', + 'src/core/ext/transport/chttp2/transport/hpack_encoder.c', + 'src/core/ext/transport/chttp2/transport/hpack_parser.c', + 'src/core/ext/transport/chttp2/transport/hpack_table.c', + 'src/core/ext/transport/chttp2/transport/huffsyms.c', + 'src/core/ext/transport/chttp2/transport/incoming_metadata.c', + 'src/core/ext/transport/chttp2/transport/parsing.c', + 'src/core/ext/transport/chttp2/transport/status_conversion.c', + 'src/core/ext/transport/chttp2/transport/stream_lists.c', + 'src/core/ext/transport/chttp2/transport/stream_map.c', + 'src/core/ext/transport/chttp2/transport/timeout_encoding.c', + 'src/core/ext/transport/chttp2/transport/varint.c', + 'src/core/ext/transport/chttp2/transport/writing.c', + 'src/core/ext/transport/chttp2/alpn/alpn.c', + 'src/core/lib/http/httpcli_security_connector.c', + 'src/core/lib/security/b64.c', + 'src/core/lib/security/client_auth_filter.c', + 'src/core/lib/security/credentials.c', + 'src/core/lib/security/credentials_metadata.c', + 'src/core/lib/security/credentials_posix.c', + 'src/core/lib/security/credentials_win32.c', + 'src/core/lib/security/google_default_credentials.c', + 'src/core/lib/security/handshake.c', + 'src/core/lib/security/json_token.c', + 'src/core/lib/security/jwt_verifier.c', + 'src/core/lib/security/secure_endpoint.c', + 'src/core/lib/security/security_connector.c', + 'src/core/lib/security/security_context.c', + 'src/core/lib/security/server_auth_filter.c', + 'src/core/lib/surface/init_secure.c', + 'src/core/lib/tsi/fake_transport_security.c', + 'src/core/lib/tsi/ssl_transport_security.c', + 'src/core/lib/tsi/transport_security.c', + 'src/core/ext/transport/chttp2/client/secure/secure_channel_create.c', + 'src/core/ext/client_config/channel_connectivity.c', + 'src/core/ext/client_config/client_channel.c', + 'src/core/ext/client_config/client_channel_factory.c', + 'src/core/ext/client_config/client_config.c', + 'src/core/ext/client_config/client_config_plugin.c', + 'src/core/ext/client_config/connector.c', + 'src/core/ext/client_config/default_initial_connect_string.c', + 'src/core/ext/client_config/initial_connect_string.c', + 'src/core/ext/client_config/lb_policy.c', + 'src/core/ext/client_config/lb_policy_factory.c', + 'src/core/ext/client_config/lb_policy_registry.c', + 'src/core/ext/client_config/parse_address.c', + 'src/core/ext/client_config/resolver.c', + 'src/core/ext/client_config/resolver_factory.c', + 'src/core/ext/client_config/resolver_registry.c', + 'src/core/ext/client_config/subchannel.c', + 'src/core/ext/client_config/subchannel_call_holder.c', + 'src/core/ext/client_config/subchannel_index.c', + 'src/core/ext/client_config/uri_parser.c', + 'src/core/ext/transport/chttp2/server/insecure/server_chttp2.c', + 'src/core/ext/transport/chttp2/client/insecure/channel_create.c', + 'src/core/ext/lb_policy/grpclb/load_balancer_api.c', + 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v0/load_balancer.pb.c', + 'third_party/nanopb/pb_common.c', + 'third_party/nanopb/pb_decode.c', + 'third_party/nanopb/pb_encode.c', + 'src/core/ext/lb_policy/pick_first/pick_first.c', + 'src/core/ext/lb_policy/round_robin/round_robin.c', + 'src/core/ext/resolver/dns/native/dns_resolver.c', + 'src/core/ext/resolver/sockaddr/sockaddr_resolver.c', + 'src/core/ext/census/context.c', + 'src/core/ext/census/grpc_context.c', + 'src/core/ext/census/grpc_filter.c', + 'src/core/ext/census/grpc_plugin.c', + 'src/core/ext/census/initialize.c', + 'src/core/ext/census/mlog.c', + 'src/core/ext/census/operation.c', + 'src/core/ext/census/placeholders.c', + 'src/core/ext/census/tracing.c', + 'src/core/plugin_registry/grpc_plugin_registry.c' + + s.private_header_files = 'src/core/lib/profiling/timers.h', + 'src/core/lib/support/backoff.h', + 'src/core/lib/support/block_annotate.h', + 'src/core/lib/support/env.h', + 'src/core/lib/support/load_file.h', + 'src/core/lib/support/murmur_hash.h', + 'src/core/lib/support/stack_lockfree.h', + 'src/core/lib/support/string.h', + 'src/core/lib/support/string_win32.h', + 'src/core/lib/support/thd_internal.h', + 'src/core/lib/support/time_precise.h', + 'src/core/lib/support/tmpfile.h', + 'src/core/lib/channel/channel_args.h', + 'src/core/lib/channel/channel_stack.h', + 'src/core/lib/channel/channel_stack_builder.h', + 'src/core/lib/channel/compress_filter.h', + 'src/core/lib/channel/connected_channel.h', + 'src/core/lib/channel/context.h', + 'src/core/lib/channel/http_client_filter.h', + 'src/core/lib/channel/http_server_filter.h', + 'src/core/lib/compression/algorithm_metadata.h', + 'src/core/lib/compression/message_compress.h', + 'src/core/lib/debug/trace.h', + 'src/core/lib/http/format_request.h', + 'src/core/lib/http/httpcli.h', + 'src/core/lib/http/parser.h', + 'src/core/lib/iomgr/closure.h', + 'src/core/lib/iomgr/endpoint.h', + 'src/core/lib/iomgr/endpoint_pair.h', + 'src/core/lib/iomgr/ev_poll_and_epoll_posix.h', + 'src/core/lib/iomgr/ev_posix.h', + 'src/core/lib/iomgr/exec_ctx.h', + 'src/core/lib/iomgr/executor.h', + 'src/core/lib/iomgr/iocp_windows.h', + 'src/core/lib/iomgr/iomgr.h', + 'src/core/lib/iomgr/iomgr_internal.h', + 'src/core/lib/iomgr/iomgr_posix.h', + 'src/core/lib/iomgr/pollset.h', + 'src/core/lib/iomgr/pollset_set.h', + 'src/core/lib/iomgr/pollset_set_windows.h', + 'src/core/lib/iomgr/pollset_windows.h', + 'src/core/lib/iomgr/resolve_address.h', + 'src/core/lib/iomgr/sockaddr.h', + 'src/core/lib/iomgr/sockaddr_posix.h', + 'src/core/lib/iomgr/sockaddr_utils.h', + 'src/core/lib/iomgr/sockaddr_win32.h', + 'src/core/lib/iomgr/socket_utils_posix.h', + 'src/core/lib/iomgr/socket_windows.h', + 'src/core/lib/iomgr/tcp_client.h', + 'src/core/lib/iomgr/tcp_posix.h', + 'src/core/lib/iomgr/tcp_server.h', + 'src/core/lib/iomgr/tcp_windows.h', + 'src/core/lib/iomgr/time_averaged_stats.h', + 'src/core/lib/iomgr/timer.h', + 'src/core/lib/iomgr/timer_heap.h', + 'src/core/lib/iomgr/udp_server.h', + 'src/core/lib/iomgr/unix_sockets_posix.h', + 'src/core/lib/iomgr/wakeup_fd_pipe.h', + 'src/core/lib/iomgr/wakeup_fd_posix.h', + 'src/core/lib/iomgr/workqueue.h', + 'src/core/lib/iomgr/workqueue_posix.h', + 'src/core/lib/iomgr/workqueue_windows.h', + 'src/core/lib/json/json.h', + 'src/core/lib/json/json_common.h', + 'src/core/lib/json/json_reader.h', + 'src/core/lib/json/json_writer.h', + 'src/core/lib/surface/api_trace.h', + 'src/core/lib/surface/call.h', + 'src/core/lib/surface/call_test_only.h', + 'src/core/lib/surface/channel.h', + 'src/core/lib/surface/channel_init.h', + 'src/core/lib/surface/channel_stack_type.h', + 'src/core/lib/surface/completion_queue.h', + 'src/core/lib/surface/event_string.h', + 'src/core/lib/surface/init.h', + 'src/core/lib/surface/lame_client.h', + 'src/core/lib/surface/server.h', + 'src/core/lib/surface/surface_trace.h', + 'src/core/lib/transport/byte_stream.h', + 'src/core/lib/transport/connectivity_state.h', + 'src/core/lib/transport/metadata.h', + 'src/core/lib/transport/metadata_batch.h', + 'src/core/lib/transport/static_metadata.h', + 'src/core/lib/transport/transport.h', + 'src/core/lib/transport/transport_impl.h', + 'src/core/ext/transport/chttp2/transport/bin_encoder.h', + 'src/core/ext/transport/chttp2/transport/chttp2_transport.h', + 'src/core/ext/transport/chttp2/transport/frame.h', + 'src/core/ext/transport/chttp2/transport/frame_data.h', + 'src/core/ext/transport/chttp2/transport/frame_goaway.h', + 'src/core/ext/transport/chttp2/transport/frame_ping.h', + 'src/core/ext/transport/chttp2/transport/frame_rst_stream.h', + 'src/core/ext/transport/chttp2/transport/frame_settings.h', + 'src/core/ext/transport/chttp2/transport/frame_window_update.h', + 'src/core/ext/transport/chttp2/transport/hpack_encoder.h', + 'src/core/ext/transport/chttp2/transport/hpack_parser.h', + 'src/core/ext/transport/chttp2/transport/hpack_table.h', + 'src/core/ext/transport/chttp2/transport/http2_errors.h', + 'src/core/ext/transport/chttp2/transport/huffsyms.h', + 'src/core/ext/transport/chttp2/transport/incoming_metadata.h', + 'src/core/ext/transport/chttp2/transport/internal.h', + 'src/core/ext/transport/chttp2/transport/status_conversion.h', + 'src/core/ext/transport/chttp2/transport/stream_map.h', + 'src/core/ext/transport/chttp2/transport/timeout_encoding.h', + 'src/core/ext/transport/chttp2/transport/varint.h', + 'src/core/ext/transport/chttp2/alpn/alpn.h', + 'src/core/lib/security/auth_filters.h', + 'src/core/lib/security/b64.h', + 'src/core/lib/security/credentials.h', + 'src/core/lib/security/handshake.h', + 'src/core/lib/security/json_token.h', + 'src/core/lib/security/jwt_verifier.h', + 'src/core/lib/security/secure_endpoint.h', + 'src/core/lib/security/security_connector.h', + 'src/core/lib/security/security_context.h', + 'src/core/lib/tsi/fake_transport_security.h', + 'src/core/lib/tsi/ssl_transport_security.h', + 'src/core/lib/tsi/ssl_types.h', + 'src/core/lib/tsi/transport_security.h', + 'src/core/lib/tsi/transport_security_interface.h', + 'src/core/ext/client_config/client_channel.h', + 'src/core/ext/client_config/client_channel_factory.h', + 'src/core/ext/client_config/client_config.h', + 'src/core/ext/client_config/connector.h', + 'src/core/ext/client_config/initial_connect_string.h', + 'src/core/ext/client_config/lb_policy.h', + 'src/core/ext/client_config/lb_policy_factory.h', + 'src/core/ext/client_config/lb_policy_registry.h', + 'src/core/ext/client_config/parse_address.h', + 'src/core/ext/client_config/resolver.h', + 'src/core/ext/client_config/resolver_factory.h', + 'src/core/ext/client_config/resolver_registry.h', + 'src/core/ext/client_config/subchannel.h', + 'src/core/ext/client_config/subchannel_call_holder.h', + 'src/core/ext/client_config/subchannel_index.h', + 'src/core/ext/client_config/uri_parser.h', + 'src/core/ext/lb_policy/grpclb/load_balancer_api.h', + 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v0/load_balancer.pb.h', + 'third_party/nanopb/pb.h', + 'third_party/nanopb/pb_common.h', + 'third_party/nanopb/pb_decode.h', + 'third_party/nanopb/pb_encode.h', + 'src/core/ext/census/aggregation.h', + 'src/core/ext/census/census_interface.h', + 'src/core/ext/census/census_rpc_stats.h', + 'src/core/ext/census/grpc_filter.h', + 'src/core/ext/census/mlog.h', + 'src/core/ext/census/rpc_metric_id.h' + + s.header_mappings_dir = 'include/grpc' + + src_root = '$(PODS_ROOT)/gRPC-Core' + # This isn't officially supported in Cocoapods. We've asked for an alternative: + # https://github.com/CocoaPods/CocoaPods/issues/4386 + public_build_settings = { + 'GRPC_SRC_ROOT' => src_root, + 'HEADER_SEARCH_PATHS' => '"$(inherited)" "$(GRPC_SRC_ROOT)/include"', + } + private_build_settings = public_build_settings.merge({ + 'USE_HEADERMAP' => 'NO', + 'ALWAYS_SEARCH_USER_PATHS' => 'NO', + 'USER_HEADER_SEARCH_PATHS' => '"$(GRPC_SRC_ROOT)"', + }) + s.user_target_xcconfig = public_build_settings + s.pod_target_xcconfig = private_build_settings + + s.libraries = 'z' + s.dependency 'BoringSSL', '~> 3.0' +end diff --git a/gRPC-ProtoRPC.podspec b/gRPC-ProtoRPC.podspec new file mode 100644 index 00000000000..1ff539d5ee6 --- /dev/null +++ b/gRPC-ProtoRPC.podspec @@ -0,0 +1,69 @@ +# GRPC CocoaPods podspec +# This file has been automatically generated from a template file. +# Please look at the templates directory instead. +# This file can be regenerated from the template by running +# tools/buildgen/generate_projects.sh + +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Pod::Spec.new do |s| + s.name = 'gRPC-ProtoRPC' + version = '0.14.0' + s.version = version + s.summary = 'RPC library for Protocol Buffers, based on gRPC' + s.homepage = 'http://www.grpc.io' + s.license = 'New BSD' + s.authors = { 'The gRPC contributors' => 'grpc-packages@google.com' } + + s.source = { + :git => 'https://github.com/grpc/grpc.git', + :tag => "release-#{version.gsub(/\./, '_')}-objectivec-#{version}", + } + + s.ios.deployment_target = '7.1' + s.osx.deployment_target = '10.9' + + name = 'ProtoRPC' + s.module_name = name + s.header_dir = name + + src_dir = 'src/objective-c/ProtoRPC' + s.source_files = "#{src_dir}/*.{h,m}" + s.header_mappings_dir = "#{src_dir}" + + s.dependency 'gRPC', version + s.dependency 'gRPC-RxLibrary', version + s.dependency 'Protobuf', '~> 3.0.0-alpha-4' + # This is needed by all pods that depend on Protobuf: + s.pod_target_xcconfig = { + 'GCC_PREPROCESSOR_DEFINITIONS' => '$(inherited) GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1', + } +end diff --git a/gRPC-RxLibrary.podspec b/gRPC-RxLibrary.podspec new file mode 100644 index 00000000000..6263878213e --- /dev/null +++ b/gRPC-RxLibrary.podspec @@ -0,0 +1,62 @@ +# GRPC CocoaPods podspec +# This file has been automatically generated from a template file. +# Please look at the templates directory instead. +# This file can be regenerated from the template by running +# tools/buildgen/generate_projects.sh + +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Pod::Spec.new do |s| + s.name = 'gRPC-RxLibrary' + version = '0.14.0' + s.version = version + s.summary = 'Reactive Extensions library for iOS/OSX.' + s.homepage = 'http://www.grpc.io' + s.license = 'New BSD' + s.authors = { 'The gRPC contributors' => 'grpc-packages@google.com' } + + s.source = { + :git => 'https://github.com/grpc/grpc.git', + :tag => "release-#{version.gsub(/\./, '_')}-objectivec-#{version}", + } + + s.ios.deployment_target = '7.1' + s.osx.deployment_target = '10.9' + + name = 'RxLibrary' + s.module_name = name + s.header_dir = name + + src_dir = 'src/objective-c/RxLibrary' + s.source_files = "#{src_dir}/*.{h,m}", "#{src_dir}/**/*.{h,m}" + s.private_header_files = "#{src_dir}/private/*.h" + s.header_mappings_dir = "#{src_dir}" +end diff --git a/gRPC.podspec b/gRPC.podspec index 569f89bf7c0..e5556cc5448 100644 --- a/gRPC.podspec +++ b/gRPC.podspec @@ -36,652 +36,33 @@ Pod::Spec.new do |s| s.name = 'gRPC' - version = '0.12.0' + version = '0.14.0' s.version = version s.summary = 'gRPC client library for iOS/OSX' s.homepage = 'http://www.grpc.io' s.license = 'New BSD' s.authors = { 'The gRPC contributors' => 'grpc-packages@google.com' } - s.source = { :git => 'https://github.com/grpc/grpc.git', - :tag => "release-#{version.gsub(/\./, '_')}-objectivec-#{version}" } - + s.source = { + :git => 'https://github.com/grpc/grpc.git', + :tag => "release-#{version.gsub(/\./, '_')}-objectivec-#{version}", + } s.ios.deployment_target = '7.1' s.osx.deployment_target = '10.9' - s.requires_arc = true - - objc_dir = 'src/objective-c' - - # Reactive Extensions library for iOS. - s.subspec 'RxLibrary' do |ss| - src_dir = "#{objc_dir}/RxLibrary" - ss.source_files = "#{src_dir}/*.{h,m}", "#{src_dir}/**/*.{h,m}" - ss.private_header_files = "#{src_dir}/private/*.h" - ss.header_mappings_dir = "#{objc_dir}" - end - - # Core cross-platform gRPC library, written in C. - s.subspec 'C-Core' do |ss| - ss.source_files = 'src/core/lib/profiling/timers.h', - 'src/core/lib/support/backoff.h', - 'src/core/lib/support/block_annotate.h', - 'src/core/lib/support/env.h', - 'src/core/lib/support/load_file.h', - 'src/core/lib/support/murmur_hash.h', - 'src/core/lib/support/stack_lockfree.h', - 'src/core/lib/support/string.h', - 'src/core/lib/support/string_win32.h', - 'src/core/lib/support/thd_internal.h', - 'src/core/lib/support/time_precise.h', - 'src/core/lib/support/tmpfile.h', - 'include/grpc/support/alloc.h', - 'include/grpc/support/atm.h', - 'include/grpc/support/atm_gcc_atomic.h', - 'include/grpc/support/atm_gcc_sync.h', - 'include/grpc/support/atm_win32.h', - 'include/grpc/support/avl.h', - 'include/grpc/support/cmdline.h', - 'include/grpc/support/cpu.h', - 'include/grpc/support/histogram.h', - 'include/grpc/support/host_port.h', - 'include/grpc/support/log.h', - 'include/grpc/support/log_win32.h', - 'include/grpc/support/port_platform.h', - 'include/grpc/support/slice.h', - 'include/grpc/support/slice_buffer.h', - 'include/grpc/support/string_util.h', - 'include/grpc/support/subprocess.h', - 'include/grpc/support/sync.h', - 'include/grpc/support/sync_generic.h', - 'include/grpc/support/sync_posix.h', - 'include/grpc/support/sync_win32.h', - 'include/grpc/support/thd.h', - 'include/grpc/support/time.h', - 'include/grpc/support/tls.h', - 'include/grpc/support/tls_gcc.h', - 'include/grpc/support/tls_msvc.h', - 'include/grpc/support/tls_pthread.h', - 'include/grpc/support/useful.h', - 'include/grpc/impl/codegen/alloc.h', - 'include/grpc/impl/codegen/atm.h', - 'include/grpc/impl/codegen/atm_gcc_atomic.h', - 'include/grpc/impl/codegen/atm_gcc_sync.h', - 'include/grpc/impl/codegen/atm_win32.h', - 'include/grpc/impl/codegen/log.h', - 'include/grpc/impl/codegen/port_platform.h', - 'include/grpc/impl/codegen/slice.h', - 'include/grpc/impl/codegen/slice_buffer.h', - 'include/grpc/impl/codegen/sync.h', - 'include/grpc/impl/codegen/sync_generic.h', - 'include/grpc/impl/codegen/sync_posix.h', - 'include/grpc/impl/codegen/sync_win32.h', - 'include/grpc/impl/codegen/time.h', - 'src/core/lib/profiling/basic_timers.c', - 'src/core/lib/profiling/stap_timers.c', - 'src/core/lib/support/alloc.c', - 'src/core/lib/support/avl.c', - 'src/core/lib/support/backoff.c', - 'src/core/lib/support/cmdline.c', - 'src/core/lib/support/cpu_iphone.c', - 'src/core/lib/support/cpu_linux.c', - 'src/core/lib/support/cpu_posix.c', - 'src/core/lib/support/cpu_windows.c', - 'src/core/lib/support/env_linux.c', - 'src/core/lib/support/env_posix.c', - 'src/core/lib/support/env_win32.c', - 'src/core/lib/support/histogram.c', - 'src/core/lib/support/host_port.c', - 'src/core/lib/support/load_file.c', - 'src/core/lib/support/log.c', - 'src/core/lib/support/log_android.c', - 'src/core/lib/support/log_linux.c', - 'src/core/lib/support/log_posix.c', - 'src/core/lib/support/log_win32.c', - 'src/core/lib/support/murmur_hash.c', - 'src/core/lib/support/slice.c', - 'src/core/lib/support/slice_buffer.c', - 'src/core/lib/support/stack_lockfree.c', - 'src/core/lib/support/string.c', - 'src/core/lib/support/string_posix.c', - 'src/core/lib/support/string_util_win32.c', - 'src/core/lib/support/string_win32.c', - 'src/core/lib/support/subprocess_posix.c', - 'src/core/lib/support/subprocess_windows.c', - 'src/core/lib/support/sync.c', - 'src/core/lib/support/sync_posix.c', - 'src/core/lib/support/sync_win32.c', - 'src/core/lib/support/thd.c', - 'src/core/lib/support/thd_posix.c', - 'src/core/lib/support/thd_win32.c', - 'src/core/lib/support/time.c', - 'src/core/lib/support/time_posix.c', - 'src/core/lib/support/time_precise.c', - 'src/core/lib/support/time_win32.c', - 'src/core/lib/support/tls_pthread.c', - 'src/core/lib/support/tmpfile_msys.c', - 'src/core/lib/support/tmpfile_posix.c', - 'src/core/lib/support/tmpfile_win32.c', - 'src/core/lib/support/wrap_memcpy.c', - 'src/core/lib/channel/channel_args.h', - 'src/core/lib/channel/channel_stack.h', - 'src/core/lib/channel/channel_stack_builder.h', - 'src/core/lib/channel/compress_filter.h', - 'src/core/lib/channel/connected_channel.h', - 'src/core/lib/channel/context.h', - 'src/core/lib/channel/http_client_filter.h', - 'src/core/lib/channel/http_server_filter.h', - 'src/core/lib/compression/algorithm_metadata.h', - 'src/core/lib/compression/message_compress.h', - 'src/core/lib/debug/trace.h', - 'src/core/lib/http/format_request.h', - 'src/core/lib/http/httpcli.h', - 'src/core/lib/http/parser.h', - 'src/core/lib/iomgr/closure.h', - 'src/core/lib/iomgr/endpoint.h', - 'src/core/lib/iomgr/endpoint_pair.h', - 'src/core/lib/iomgr/ev_poll_and_epoll_posix.h', - 'src/core/lib/iomgr/ev_posix.h', - 'src/core/lib/iomgr/exec_ctx.h', - 'src/core/lib/iomgr/executor.h', - 'src/core/lib/iomgr/iocp_windows.h', - 'src/core/lib/iomgr/iomgr.h', - 'src/core/lib/iomgr/iomgr_internal.h', - 'src/core/lib/iomgr/iomgr_posix.h', - 'src/core/lib/iomgr/pollset.h', - 'src/core/lib/iomgr/pollset_set.h', - 'src/core/lib/iomgr/pollset_set_windows.h', - 'src/core/lib/iomgr/pollset_windows.h', - 'src/core/lib/iomgr/resolve_address.h', - 'src/core/lib/iomgr/sockaddr.h', - 'src/core/lib/iomgr/sockaddr_posix.h', - 'src/core/lib/iomgr/sockaddr_utils.h', - 'src/core/lib/iomgr/sockaddr_win32.h', - 'src/core/lib/iomgr/socket_utils_posix.h', - 'src/core/lib/iomgr/socket_windows.h', - 'src/core/lib/iomgr/tcp_client.h', - 'src/core/lib/iomgr/tcp_posix.h', - 'src/core/lib/iomgr/tcp_server.h', - 'src/core/lib/iomgr/tcp_windows.h', - 'src/core/lib/iomgr/time_averaged_stats.h', - 'src/core/lib/iomgr/timer.h', - 'src/core/lib/iomgr/timer_heap.h', - 'src/core/lib/iomgr/udp_server.h', - 'src/core/lib/iomgr/unix_sockets_posix.h', - 'src/core/lib/iomgr/wakeup_fd_pipe.h', - 'src/core/lib/iomgr/wakeup_fd_posix.h', - 'src/core/lib/iomgr/workqueue.h', - 'src/core/lib/iomgr/workqueue_posix.h', - 'src/core/lib/iomgr/workqueue_windows.h', - 'src/core/lib/json/json.h', - 'src/core/lib/json/json_common.h', - 'src/core/lib/json/json_reader.h', - 'src/core/lib/json/json_writer.h', - 'src/core/lib/surface/api_trace.h', - 'src/core/lib/surface/call.h', - 'src/core/lib/surface/call_test_only.h', - 'src/core/lib/surface/channel.h', - 'src/core/lib/surface/channel_init.h', - 'src/core/lib/surface/channel_stack_type.h', - 'src/core/lib/surface/completion_queue.h', - 'src/core/lib/surface/event_string.h', - 'src/core/lib/surface/init.h', - 'src/core/lib/surface/lame_client.h', - 'src/core/lib/surface/server.h', - 'src/core/lib/surface/surface_trace.h', - 'src/core/lib/transport/byte_stream.h', - 'src/core/lib/transport/connectivity_state.h', - 'src/core/lib/transport/metadata.h', - 'src/core/lib/transport/metadata_batch.h', - 'src/core/lib/transport/static_metadata.h', - 'src/core/lib/transport/transport.h', - 'src/core/lib/transport/transport_impl.h', - 'src/core/ext/transport/chttp2/transport/bin_encoder.h', - 'src/core/ext/transport/chttp2/transport/chttp2_transport.h', - 'src/core/ext/transport/chttp2/transport/frame.h', - 'src/core/ext/transport/chttp2/transport/frame_data.h', - 'src/core/ext/transport/chttp2/transport/frame_goaway.h', - 'src/core/ext/transport/chttp2/transport/frame_ping.h', - 'src/core/ext/transport/chttp2/transport/frame_rst_stream.h', - 'src/core/ext/transport/chttp2/transport/frame_settings.h', - 'src/core/ext/transport/chttp2/transport/frame_window_update.h', - 'src/core/ext/transport/chttp2/transport/hpack_encoder.h', - 'src/core/ext/transport/chttp2/transport/hpack_parser.h', - 'src/core/ext/transport/chttp2/transport/hpack_table.h', - 'src/core/ext/transport/chttp2/transport/http2_errors.h', - 'src/core/ext/transport/chttp2/transport/huffsyms.h', - 'src/core/ext/transport/chttp2/transport/incoming_metadata.h', - 'src/core/ext/transport/chttp2/transport/internal.h', - 'src/core/ext/transport/chttp2/transport/status_conversion.h', - 'src/core/ext/transport/chttp2/transport/stream_map.h', - 'src/core/ext/transport/chttp2/transport/timeout_encoding.h', - 'src/core/ext/transport/chttp2/transport/varint.h', - 'src/core/ext/transport/chttp2/alpn/alpn.h', - 'src/core/lib/security/auth_filters.h', - 'src/core/lib/security/b64.h', - 'src/core/lib/security/credentials.h', - 'src/core/lib/security/handshake.h', - 'src/core/lib/security/json_token.h', - 'src/core/lib/security/jwt_verifier.h', - 'src/core/lib/security/secure_endpoint.h', - 'src/core/lib/security/security_connector.h', - 'src/core/lib/security/security_context.h', - 'src/core/lib/tsi/fake_transport_security.h', - 'src/core/lib/tsi/ssl_transport_security.h', - 'src/core/lib/tsi/ssl_types.h', - 'src/core/lib/tsi/transport_security.h', - 'src/core/lib/tsi/transport_security_interface.h', - 'src/core/ext/client_config/client_channel.h', - 'src/core/ext/client_config/client_channel_factory.h', - 'src/core/ext/client_config/client_config.h', - 'src/core/ext/client_config/connector.h', - 'src/core/ext/client_config/initial_connect_string.h', - 'src/core/ext/client_config/lb_policy.h', - 'src/core/ext/client_config/lb_policy_factory.h', - 'src/core/ext/client_config/lb_policy_registry.h', - 'src/core/ext/client_config/parse_address.h', - 'src/core/ext/client_config/resolver.h', - 'src/core/ext/client_config/resolver_factory.h', - 'src/core/ext/client_config/resolver_registry.h', - 'src/core/ext/client_config/subchannel.h', - 'src/core/ext/client_config/subchannel_call_holder.h', - 'src/core/ext/client_config/subchannel_index.h', - 'src/core/ext/client_config/uri_parser.h', - 'src/core/ext/lb_policy/grpclb/load_balancer_api.h', - 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v0/load_balancer.pb.h', - 'third_party/nanopb/pb.h', - 'third_party/nanopb/pb_common.h', - 'third_party/nanopb/pb_decode.h', - 'third_party/nanopb/pb_encode.h', - 'src/core/ext/census/aggregation.h', - 'src/core/ext/census/census_interface.h', - 'src/core/ext/census/census_rpc_stats.h', - 'src/core/ext/census/grpc_filter.h', - 'src/core/ext/census/mlog.h', - 'src/core/ext/census/rpc_metric_id.h', - 'include/grpc/byte_buffer.h', - 'include/grpc/byte_buffer_reader.h', - 'include/grpc/compression.h', - 'include/grpc/grpc.h', - 'include/grpc/status.h', - 'include/grpc/impl/codegen/byte_buffer.h', - 'include/grpc/impl/codegen/byte_buffer_reader.h', - 'include/grpc/impl/codegen/compression_types.h', - 'include/grpc/impl/codegen/connectivity_state.h', - 'include/grpc/impl/codegen/grpc_types.h', - 'include/grpc/impl/codegen/propagation_bits.h', - 'include/grpc/impl/codegen/status.h', - 'include/grpc/impl/codegen/alloc.h', - 'include/grpc/impl/codegen/atm.h', - 'include/grpc/impl/codegen/atm_gcc_atomic.h', - 'include/grpc/impl/codegen/atm_gcc_sync.h', - 'include/grpc/impl/codegen/atm_win32.h', - 'include/grpc/impl/codegen/log.h', - 'include/grpc/impl/codegen/port_platform.h', - 'include/grpc/impl/codegen/slice.h', - 'include/grpc/impl/codegen/slice_buffer.h', - 'include/grpc/impl/codegen/sync.h', - 'include/grpc/impl/codegen/sync_generic.h', - 'include/grpc/impl/codegen/sync_posix.h', - 'include/grpc/impl/codegen/sync_win32.h', - 'include/grpc/impl/codegen/time.h', - 'include/grpc/grpc_security.h', - 'include/grpc/grpc_security_constants.h', - 'include/grpc/census.h', - 'src/core/lib/surface/init.c', - 'src/core/lib/channel/channel_args.c', - 'src/core/lib/channel/channel_stack.c', - 'src/core/lib/channel/channel_stack_builder.c', - 'src/core/lib/channel/compress_filter.c', - 'src/core/lib/channel/connected_channel.c', - 'src/core/lib/channel/http_client_filter.c', - 'src/core/lib/channel/http_server_filter.c', - 'src/core/lib/compression/compression_algorithm.c', - 'src/core/lib/compression/message_compress.c', - 'src/core/lib/debug/trace.c', - 'src/core/lib/http/format_request.c', - 'src/core/lib/http/httpcli.c', - 'src/core/lib/http/parser.c', - 'src/core/lib/iomgr/closure.c', - 'src/core/lib/iomgr/endpoint.c', - 'src/core/lib/iomgr/endpoint_pair_posix.c', - 'src/core/lib/iomgr/endpoint_pair_windows.c', - 'src/core/lib/iomgr/ev_poll_and_epoll_posix.c', - 'src/core/lib/iomgr/ev_posix.c', - 'src/core/lib/iomgr/exec_ctx.c', - 'src/core/lib/iomgr/executor.c', - 'src/core/lib/iomgr/iocp_windows.c', - 'src/core/lib/iomgr/iomgr.c', - 'src/core/lib/iomgr/iomgr_posix.c', - 'src/core/lib/iomgr/iomgr_windows.c', - 'src/core/lib/iomgr/pollset_set_windows.c', - 'src/core/lib/iomgr/pollset_windows.c', - 'src/core/lib/iomgr/resolve_address_posix.c', - 'src/core/lib/iomgr/resolve_address_windows.c', - 'src/core/lib/iomgr/sockaddr_utils.c', - 'src/core/lib/iomgr/socket_utils_common_posix.c', - 'src/core/lib/iomgr/socket_utils_linux.c', - 'src/core/lib/iomgr/socket_utils_posix.c', - 'src/core/lib/iomgr/socket_windows.c', - 'src/core/lib/iomgr/tcp_client_posix.c', - 'src/core/lib/iomgr/tcp_client_windows.c', - 'src/core/lib/iomgr/tcp_posix.c', - 'src/core/lib/iomgr/tcp_server_posix.c', - 'src/core/lib/iomgr/tcp_server_windows.c', - 'src/core/lib/iomgr/tcp_windows.c', - 'src/core/lib/iomgr/time_averaged_stats.c', - 'src/core/lib/iomgr/timer.c', - 'src/core/lib/iomgr/timer_heap.c', - 'src/core/lib/iomgr/udp_server.c', - 'src/core/lib/iomgr/unix_sockets_posix.c', - 'src/core/lib/iomgr/unix_sockets_posix_noop.c', - 'src/core/lib/iomgr/wakeup_fd_eventfd.c', - 'src/core/lib/iomgr/wakeup_fd_nospecial.c', - 'src/core/lib/iomgr/wakeup_fd_pipe.c', - 'src/core/lib/iomgr/wakeup_fd_posix.c', - 'src/core/lib/iomgr/workqueue_posix.c', - 'src/core/lib/iomgr/workqueue_windows.c', - 'src/core/lib/json/json.c', - 'src/core/lib/json/json_reader.c', - 'src/core/lib/json/json_string.c', - 'src/core/lib/json/json_writer.c', - 'src/core/lib/surface/alarm.c', - 'src/core/lib/surface/api_trace.c', - 'src/core/lib/surface/byte_buffer.c', - 'src/core/lib/surface/byte_buffer_reader.c', - 'src/core/lib/surface/call.c', - 'src/core/lib/surface/call_details.c', - 'src/core/lib/surface/call_log_batch.c', - 'src/core/lib/surface/channel.c', - 'src/core/lib/surface/channel_init.c', - 'src/core/lib/surface/channel_ping.c', - 'src/core/lib/surface/channel_stack_type.c', - 'src/core/lib/surface/completion_queue.c', - 'src/core/lib/surface/event_string.c', - 'src/core/lib/surface/lame_client.c', - 'src/core/lib/surface/metadata_array.c', - 'src/core/lib/surface/server.c', - 'src/core/lib/surface/validate_metadata.c', - 'src/core/lib/surface/version.c', - 'src/core/lib/transport/byte_stream.c', - 'src/core/lib/transport/connectivity_state.c', - 'src/core/lib/transport/metadata.c', - 'src/core/lib/transport/metadata_batch.c', - 'src/core/lib/transport/static_metadata.c', - 'src/core/lib/transport/transport.c', - 'src/core/lib/transport/transport_op_string.c', - 'src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c', - 'src/core/ext/transport/chttp2/transport/bin_encoder.c', - 'src/core/ext/transport/chttp2/transport/chttp2_plugin.c', - 'src/core/ext/transport/chttp2/transport/chttp2_transport.c', - 'src/core/ext/transport/chttp2/transport/frame_data.c', - 'src/core/ext/transport/chttp2/transport/frame_goaway.c', - 'src/core/ext/transport/chttp2/transport/frame_ping.c', - 'src/core/ext/transport/chttp2/transport/frame_rst_stream.c', - 'src/core/ext/transport/chttp2/transport/frame_settings.c', - 'src/core/ext/transport/chttp2/transport/frame_window_update.c', - 'src/core/ext/transport/chttp2/transport/hpack_encoder.c', - 'src/core/ext/transport/chttp2/transport/hpack_parser.c', - 'src/core/ext/transport/chttp2/transport/hpack_table.c', - 'src/core/ext/transport/chttp2/transport/huffsyms.c', - 'src/core/ext/transport/chttp2/transport/incoming_metadata.c', - 'src/core/ext/transport/chttp2/transport/parsing.c', - 'src/core/ext/transport/chttp2/transport/status_conversion.c', - 'src/core/ext/transport/chttp2/transport/stream_lists.c', - 'src/core/ext/transport/chttp2/transport/stream_map.c', - 'src/core/ext/transport/chttp2/transport/timeout_encoding.c', - 'src/core/ext/transport/chttp2/transport/varint.c', - 'src/core/ext/transport/chttp2/transport/writing.c', - 'src/core/ext/transport/chttp2/alpn/alpn.c', - 'src/core/lib/http/httpcli_security_connector.c', - 'src/core/lib/security/b64.c', - 'src/core/lib/security/client_auth_filter.c', - 'src/core/lib/security/credentials.c', - 'src/core/lib/security/credentials_metadata.c', - 'src/core/lib/security/credentials_posix.c', - 'src/core/lib/security/credentials_win32.c', - 'src/core/lib/security/google_default_credentials.c', - 'src/core/lib/security/handshake.c', - 'src/core/lib/security/json_token.c', - 'src/core/lib/security/jwt_verifier.c', - 'src/core/lib/security/secure_endpoint.c', - 'src/core/lib/security/security_connector.c', - 'src/core/lib/security/security_context.c', - 'src/core/lib/security/server_auth_filter.c', - 'src/core/lib/surface/init_secure.c', - 'src/core/lib/tsi/fake_transport_security.c', - 'src/core/lib/tsi/ssl_transport_security.c', - 'src/core/lib/tsi/transport_security.c', - 'src/core/ext/transport/chttp2/client/secure/secure_channel_create.c', - 'src/core/ext/client_config/channel_connectivity.c', - 'src/core/ext/client_config/client_channel.c', - 'src/core/ext/client_config/client_channel_factory.c', - 'src/core/ext/client_config/client_config.c', - 'src/core/ext/client_config/client_config_plugin.c', - 'src/core/ext/client_config/connector.c', - 'src/core/ext/client_config/default_initial_connect_string.c', - 'src/core/ext/client_config/initial_connect_string.c', - 'src/core/ext/client_config/lb_policy.c', - 'src/core/ext/client_config/lb_policy_factory.c', - 'src/core/ext/client_config/lb_policy_registry.c', - 'src/core/ext/client_config/parse_address.c', - 'src/core/ext/client_config/resolver.c', - 'src/core/ext/client_config/resolver_factory.c', - 'src/core/ext/client_config/resolver_registry.c', - 'src/core/ext/client_config/subchannel.c', - 'src/core/ext/client_config/subchannel_call_holder.c', - 'src/core/ext/client_config/subchannel_index.c', - 'src/core/ext/client_config/uri_parser.c', - 'src/core/ext/transport/chttp2/server/insecure/server_chttp2.c', - 'src/core/ext/transport/chttp2/client/insecure/channel_create.c', - 'src/core/ext/lb_policy/grpclb/load_balancer_api.c', - 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v0/load_balancer.pb.c', - 'third_party/nanopb/pb_common.c', - 'third_party/nanopb/pb_decode.c', - 'third_party/nanopb/pb_encode.c', - 'src/core/ext/lb_policy/pick_first/pick_first.c', - 'src/core/ext/lb_policy/round_robin/round_robin.c', - 'src/core/ext/resolver/dns/native/dns_resolver.c', - 'src/core/ext/resolver/sockaddr/sockaddr_resolver.c', - 'src/core/ext/census/context.c', - 'src/core/ext/census/grpc_context.c', - 'src/core/ext/census/grpc_filter.c', - 'src/core/ext/census/grpc_plugin.c', - 'src/core/ext/census/initialize.c', - 'src/core/ext/census/mlog.c', - 'src/core/ext/census/operation.c', - 'src/core/ext/census/placeholders.c', - 'src/core/ext/census/tracing.c', - 'src/core/plugin_registry/grpc_plugin_registry.c' - - ss.private_header_files = 'src/core/lib/profiling/timers.h', - 'src/core/lib/support/backoff.h', - 'src/core/lib/support/block_annotate.h', - 'src/core/lib/support/env.h', - 'src/core/lib/support/load_file.h', - 'src/core/lib/support/murmur_hash.h', - 'src/core/lib/support/stack_lockfree.h', - 'src/core/lib/support/string.h', - 'src/core/lib/support/string_win32.h', - 'src/core/lib/support/thd_internal.h', - 'src/core/lib/support/time_precise.h', - 'src/core/lib/support/tmpfile.h', - 'src/core/lib/channel/channel_args.h', - 'src/core/lib/channel/channel_stack.h', - 'src/core/lib/channel/channel_stack_builder.h', - 'src/core/lib/channel/compress_filter.h', - 'src/core/lib/channel/connected_channel.h', - 'src/core/lib/channel/context.h', - 'src/core/lib/channel/http_client_filter.h', - 'src/core/lib/channel/http_server_filter.h', - 'src/core/lib/compression/algorithm_metadata.h', - 'src/core/lib/compression/message_compress.h', - 'src/core/lib/debug/trace.h', - 'src/core/lib/http/format_request.h', - 'src/core/lib/http/httpcli.h', - 'src/core/lib/http/parser.h', - 'src/core/lib/iomgr/closure.h', - 'src/core/lib/iomgr/endpoint.h', - 'src/core/lib/iomgr/endpoint_pair.h', - 'src/core/lib/iomgr/ev_poll_and_epoll_posix.h', - 'src/core/lib/iomgr/ev_posix.h', - 'src/core/lib/iomgr/exec_ctx.h', - 'src/core/lib/iomgr/executor.h', - 'src/core/lib/iomgr/iocp_windows.h', - 'src/core/lib/iomgr/iomgr.h', - 'src/core/lib/iomgr/iomgr_internal.h', - 'src/core/lib/iomgr/iomgr_posix.h', - 'src/core/lib/iomgr/pollset.h', - 'src/core/lib/iomgr/pollset_set.h', - 'src/core/lib/iomgr/pollset_set_windows.h', - 'src/core/lib/iomgr/pollset_windows.h', - 'src/core/lib/iomgr/resolve_address.h', - 'src/core/lib/iomgr/sockaddr.h', - 'src/core/lib/iomgr/sockaddr_posix.h', - 'src/core/lib/iomgr/sockaddr_utils.h', - 'src/core/lib/iomgr/sockaddr_win32.h', - 'src/core/lib/iomgr/socket_utils_posix.h', - 'src/core/lib/iomgr/socket_windows.h', - 'src/core/lib/iomgr/tcp_client.h', - 'src/core/lib/iomgr/tcp_posix.h', - 'src/core/lib/iomgr/tcp_server.h', - 'src/core/lib/iomgr/tcp_windows.h', - 'src/core/lib/iomgr/time_averaged_stats.h', - 'src/core/lib/iomgr/timer.h', - 'src/core/lib/iomgr/timer_heap.h', - 'src/core/lib/iomgr/udp_server.h', - 'src/core/lib/iomgr/unix_sockets_posix.h', - 'src/core/lib/iomgr/wakeup_fd_pipe.h', - 'src/core/lib/iomgr/wakeup_fd_posix.h', - 'src/core/lib/iomgr/workqueue.h', - 'src/core/lib/iomgr/workqueue_posix.h', - 'src/core/lib/iomgr/workqueue_windows.h', - 'src/core/lib/json/json.h', - 'src/core/lib/json/json_common.h', - 'src/core/lib/json/json_reader.h', - 'src/core/lib/json/json_writer.h', - 'src/core/lib/surface/api_trace.h', - 'src/core/lib/surface/call.h', - 'src/core/lib/surface/call_test_only.h', - 'src/core/lib/surface/channel.h', - 'src/core/lib/surface/channel_init.h', - 'src/core/lib/surface/channel_stack_type.h', - 'src/core/lib/surface/completion_queue.h', - 'src/core/lib/surface/event_string.h', - 'src/core/lib/surface/init.h', - 'src/core/lib/surface/lame_client.h', - 'src/core/lib/surface/server.h', - 'src/core/lib/surface/surface_trace.h', - 'src/core/lib/transport/byte_stream.h', - 'src/core/lib/transport/connectivity_state.h', - 'src/core/lib/transport/metadata.h', - 'src/core/lib/transport/metadata_batch.h', - 'src/core/lib/transport/static_metadata.h', - 'src/core/lib/transport/transport.h', - 'src/core/lib/transport/transport_impl.h', - 'src/core/ext/transport/chttp2/transport/bin_encoder.h', - 'src/core/ext/transport/chttp2/transport/chttp2_transport.h', - 'src/core/ext/transport/chttp2/transport/frame.h', - 'src/core/ext/transport/chttp2/transport/frame_data.h', - 'src/core/ext/transport/chttp2/transport/frame_goaway.h', - 'src/core/ext/transport/chttp2/transport/frame_ping.h', - 'src/core/ext/transport/chttp2/transport/frame_rst_stream.h', - 'src/core/ext/transport/chttp2/transport/frame_settings.h', - 'src/core/ext/transport/chttp2/transport/frame_window_update.h', - 'src/core/ext/transport/chttp2/transport/hpack_encoder.h', - 'src/core/ext/transport/chttp2/transport/hpack_parser.h', - 'src/core/ext/transport/chttp2/transport/hpack_table.h', - 'src/core/ext/transport/chttp2/transport/http2_errors.h', - 'src/core/ext/transport/chttp2/transport/huffsyms.h', - 'src/core/ext/transport/chttp2/transport/incoming_metadata.h', - 'src/core/ext/transport/chttp2/transport/internal.h', - 'src/core/ext/transport/chttp2/transport/status_conversion.h', - 'src/core/ext/transport/chttp2/transport/stream_map.h', - 'src/core/ext/transport/chttp2/transport/timeout_encoding.h', - 'src/core/ext/transport/chttp2/transport/varint.h', - 'src/core/ext/transport/chttp2/alpn/alpn.h', - 'src/core/lib/security/auth_filters.h', - 'src/core/lib/security/b64.h', - 'src/core/lib/security/credentials.h', - 'src/core/lib/security/handshake.h', - 'src/core/lib/security/json_token.h', - 'src/core/lib/security/jwt_verifier.h', - 'src/core/lib/security/secure_endpoint.h', - 'src/core/lib/security/security_connector.h', - 'src/core/lib/security/security_context.h', - 'src/core/lib/tsi/fake_transport_security.h', - 'src/core/lib/tsi/ssl_transport_security.h', - 'src/core/lib/tsi/ssl_types.h', - 'src/core/lib/tsi/transport_security.h', - 'src/core/lib/tsi/transport_security_interface.h', - 'src/core/ext/client_config/client_channel.h', - 'src/core/ext/client_config/client_channel_factory.h', - 'src/core/ext/client_config/client_config.h', - 'src/core/ext/client_config/connector.h', - 'src/core/ext/client_config/initial_connect_string.h', - 'src/core/ext/client_config/lb_policy.h', - 'src/core/ext/client_config/lb_policy_factory.h', - 'src/core/ext/client_config/lb_policy_registry.h', - 'src/core/ext/client_config/parse_address.h', - 'src/core/ext/client_config/resolver.h', - 'src/core/ext/client_config/resolver_factory.h', - 'src/core/ext/client_config/resolver_registry.h', - 'src/core/ext/client_config/subchannel.h', - 'src/core/ext/client_config/subchannel_call_holder.h', - 'src/core/ext/client_config/subchannel_index.h', - 'src/core/ext/client_config/uri_parser.h', - 'src/core/ext/lb_policy/grpclb/load_balancer_api.h', - 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v0/load_balancer.pb.h', - 'third_party/nanopb/pb.h', - 'third_party/nanopb/pb_common.h', - 'third_party/nanopb/pb_decode.h', - 'third_party/nanopb/pb_encode.h', - 'src/core/ext/census/aggregation.h', - 'src/core/ext/census/census_interface.h', - 'src/core/ext/census/census_rpc_stats.h', - 'src/core/ext/census/grpc_filter.h', - 'src/core/ext/census/mlog.h', - 'src/core/ext/census/rpc_metric_id.h' - - ss.header_mappings_dir = '.' - # This isn't officially supported in Cocoapods. We've asked for an alternative: - # https://github.com/CocoaPods/CocoaPods/issues/4386 - ss.xcconfig = { - 'USE_HEADERMAP' => 'NO', - 'ALWAYS_SEARCH_USER_PATHS' => 'NO', - 'USER_HEADER_SEARCH_PATHS' => '"$(PODS_ROOT)/Headers/Private/gRPC"', - 'HEADER_SEARCH_PATHS' => '"$(PODS_ROOT)/Headers/Private/gRPC/include"' - } - - ss.requires_arc = false - ss.libraries = 'z' - ss.dependency 'BoringSSL', '~> 2.0' - - # ss.compiler_flags = '-GCC_WARN_INHIBIT_ALL_WARNINGS', '-w' - end - - # Objective-C wrapper around the core gRPC library. - s.subspec 'GRPCClient' do |ss| - src_dir = "#{objc_dir}/GRPCClient" - ss.source_files = "#{src_dir}/*.{h,m}", "#{src_dir}/**/*.{h,m}" - ss.private_header_files = "#{src_dir}/private/*.h" - ss.header_mappings_dir = "#{objc_dir}" - ss.dependency 'gRPC/C-Core' - ss.dependency 'gRPC/RxLibrary' + name = 'GRPCClient' + s.module_name = name + s.header_dir = name - # Certificates, to be able to establish TLS connections: - ss.resource_bundles = { 'gRPCCertificates' => ['etc/roots.pem'] } - end + src_dir = 'src/objective-c/GRPCClient' + s.source_files = "#{src_dir}/*.{h,m}", "#{src_dir}/**/*.{h,m}" + s.private_header_files = "#{src_dir}/private/*.h" + s.header_mappings_dir = "#{src_dir}" - # RPC library for ProtocolBuffers, based on gRPC - s.subspec 'ProtoRPC' do |ss| - src_dir = "#{objc_dir}/ProtoRPC" - ss.source_files = "#{src_dir}/*.{h,m}" - ss.header_mappings_dir = "#{objc_dir}" + s.dependency 'gRPC-Core', version + s.dependency 'gRPC-RxLibrary', version - ss.dependency 'gRPC/GRPCClient' - ss.dependency 'gRPC/RxLibrary' - ss.dependency 'Protobuf', '~> 3.0.0-alpha-4' - end + # Certificates, to be able to establish TLS connections: + s.resource_bundles = { 'gRPCCertificates' => ['etc/roots.pem'] } end diff --git a/src/objective-c/tests/Podfile b/src/objective-c/tests/Podfile index 7fe047aa21f..4cb94715709 100644 --- a/src/objective-c/tests/Podfile +++ b/src/objective-c/tests/Podfile @@ -7,6 +7,9 @@ def shared_pods pod 'Protobuf', :path => "../../../third_party/protobuf" pod 'BoringSSL', :podspec => ".." pod 'gRPC', :path => "../../.." + pod 'gRPC-Core', :path => "../../.." + pod 'gRPC-RxLibrary', :path => "../../.." + pod 'gRPC-ProtoRPC', :path => "../../.." pod 'RemoteTest', :path => "RemoteTestClient" end diff --git a/templates/gRPC.podspec.template b/templates/gRPC-Core.podspec.template similarity index 51% rename from templates/gRPC.podspec.template rename to templates/gRPC-Core.podspec.template index a9948a41df4..ebc23e14904 100644 --- a/templates/gRPC.podspec.template +++ b/templates/gRPC-Core.podspec.template @@ -53,77 +53,48 @@ return out %> Pod::Spec.new do |s| - s.name = 'gRPC' - version = '0.12.0' + s.name = 'gRPC-Core' + version = '0.14.0' s.version = version - s.summary = 'gRPC client library for iOS/OSX' + s.summary = 'Core cross-platform gRPC library, written in C' s.homepage = 'http://www.grpc.io' s.license = 'New BSD' s.authors = { 'The gRPC contributors' => 'grpc-packages@google.com' } - s.source = { :git => 'https://github.com/grpc/grpc.git', - :tag => "release-#{version.gsub(/\./, '_')}-objectivec-#{version}" } + s.source = { + :git => 'https://github.com/grpc/grpc.git', + :tag => "release-#{version.gsub(/\./, '_')}-objectivec-#{version}", + } - s.ios.deployment_target = '7.1' s.osx.deployment_target = '10.9' - s.requires_arc = true - - objc_dir = 'src/objective-c' - - # Reactive Extensions library for iOS. - s.subspec 'RxLibrary' do |ss| - src_dir = "#{objc_dir}/RxLibrary" - ss.source_files = "#{src_dir}/*.{h,m}", "#{src_dir}/**/*.{h,m}" - ss.private_header_files = "#{src_dir}/private/*.h" - ss.header_mappings_dir = "#{objc_dir}" - end - - # Core cross-platform gRPC library, written in C. - s.subspec 'C-Core' do |ss| - ss.source_files = ${(',\n' + 22*' ').join('\'%s\'' % f for f in grpc_files(libs))} - - ss.private_header_files = ${(',\n' + 30*' ').join('\'%s\'' % f for f in grpc_private_headers(libs))} - - ss.header_mappings_dir = '.' - # This isn't officially supported in Cocoapods. We've asked for an alternative: - # https://github.com/CocoaPods/CocoaPods/issues/4386 - ss.xcconfig = { - 'USE_HEADERMAP' => 'NO', - 'ALWAYS_SEARCH_USER_PATHS' => 'NO', - 'USER_HEADER_SEARCH_PATHS' => '"$(PODS_ROOT)/Headers/Private/gRPC"', - 'HEADER_SEARCH_PATHS' => '"$(PODS_ROOT)/Headers/Private/gRPC/include"' - } - - ss.requires_arc = false - ss.libraries = 'z' - ss.dependency 'BoringSSL', '~> 2.0' + s.requires_arc = false - # ss.compiler_flags = '-GCC_WARN_INHIBIT_ALL_WARNINGS', '-w' - end + name = 'grpc' + s.module_name = name + s.header_dir = name - # Objective-C wrapper around the core gRPC library. - s.subspec 'GRPCClient' do |ss| - src_dir = "#{objc_dir}/GRPCClient" - ss.source_files = "#{src_dir}/*.{h,m}", "#{src_dir}/**/*.{h,m}" - ss.private_header_files = "#{src_dir}/private/*.h" - ss.header_mappings_dir = "#{objc_dir}" + s.source_files = ${(',\n' + 19*' ').join('\'%s\'' % f for f in grpc_files(libs))} - ss.dependency 'gRPC/C-Core' - ss.dependency 'gRPC/RxLibrary' + s.private_header_files = ${(',\n' + 27*' ').join('\'%s\'' % f for f in grpc_private_headers(libs))} - # Certificates, to be able to establish TLS connections: - ss.resource_bundles = { 'gRPCCertificates' => ['etc/roots.pem'] } - end + s.header_mappings_dir = 'include/grpc' - # RPC library for ProtocolBuffers, based on gRPC - s.subspec 'ProtoRPC' do |ss| - src_dir = "#{objc_dir}/ProtoRPC" - ss.source_files = "#{src_dir}/*.{h,m}" - ss.header_mappings_dir = "#{objc_dir}" + src_root = '$(PODS_ROOT)/gRPC-Core' + # This isn't officially supported in Cocoapods. We've asked for an alternative: + # https://github.com/CocoaPods/CocoaPods/issues/4386 + public_build_settings = { + 'GRPC_SRC_ROOT' => src_root, + 'HEADER_SEARCH_PATHS' => '"$(inherited)" "$(GRPC_SRC_ROOT)/include"', + } + private_build_settings = public_build_settings.merge({ + 'USE_HEADERMAP' => 'NO', + 'ALWAYS_SEARCH_USER_PATHS' => 'NO', + 'USER_HEADER_SEARCH_PATHS' => '"$(GRPC_SRC_ROOT)"', + }) + s.user_target_xcconfig = public_build_settings + s.pod_target_xcconfig = private_build_settings - ss.dependency 'gRPC/GRPCClient' - ss.dependency 'gRPC/RxLibrary' - ss.dependency 'Protobuf', '~> 3.0.0-alpha-4' - end + s.libraries = 'z' + s.dependency 'BoringSSL', '~> 3.0' end From 547f0656fc8ce1c97bf5311ef571a1ea86076945 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Sun, 1 May 2016 19:52:21 -0700 Subject: [PATCH 0070/1039] Add modulemap for gRPC core --- gRPC-Core.podspec | 2 ++ include/grpc/module.modulemap | 5 +++++ templates/gRPC-Core.podspec.template | 2 ++ 3 files changed, 9 insertions(+) create mode 100644 include/grpc/module.modulemap diff --git a/gRPC-Core.podspec b/gRPC-Core.podspec index c924ab3a5df..54e652eb1a2 100644 --- a/gRPC-Core.podspec +++ b/gRPC-Core.podspec @@ -56,6 +56,8 @@ Pod::Spec.new do |s| s.module_name = name s.header_dir = name + s.module_map = 'include/grpc/module.modulemap' + s.source_files = 'src/core/lib/profiling/timers.h', 'src/core/lib/support/backoff.h', 'src/core/lib/support/block_annotate.h', diff --git a/include/grpc/module.modulemap b/include/grpc/module.modulemap new file mode 100644 index 00000000000..ae11a78b74a --- /dev/null +++ b/include/grpc/module.modulemap @@ -0,0 +1,5 @@ +framework module grpc { + umbrella header "grpc.h" + export * + module * { export * } +} diff --git a/templates/gRPC-Core.podspec.template b/templates/gRPC-Core.podspec.template index ebc23e14904..efe3738c420 100644 --- a/templates/gRPC-Core.podspec.template +++ b/templates/gRPC-Core.podspec.template @@ -74,6 +74,8 @@ s.module_name = name s.header_dir = name + s.module_map = 'include/grpc/module.modulemap' + s.source_files = ${(',\n' + 19*' ').join('\'%s\'' % f for f in grpc_files(libs))} s.private_header_files = ${(',\n' + 27*' ').join('\'%s\'' % f for f in grpc_private_headers(libs))} From fb4a54541a6e00c43e1671cda0a25e2eda4792df Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Fri, 3 Jun 2016 17:34:24 -0700 Subject: [PATCH 0071/1039] ProtoRPC: Framework-like import of proto runtime --- src/objective-c/ProtoRPC/ProtoRPC.m | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/objective-c/ProtoRPC/ProtoRPC.m b/src/objective-c/ProtoRPC/ProtoRPC.m index 9bf66f347ac..b526708c580 100644 --- a/src/objective-c/ProtoRPC/ProtoRPC.m +++ b/src/objective-c/ProtoRPC/ProtoRPC.m @@ -33,7 +33,7 @@ #import "ProtoRPC.h" -#import +#import #import #import From 12e03fe30a97c8d4b8d7dcd0c8690b19cc4b35f7 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Wed, 8 Jun 2016 12:53:04 -0700 Subject: [PATCH 0072/1039] Tests Podfile: remove redundancy --- src/objective-c/tests/Podfile | 51 +++++++++++++++-------------------- 1 file changed, 21 insertions(+), 30 deletions(-) diff --git a/src/objective-c/tests/Podfile b/src/objective-c/tests/Podfile index 4cb94715709..f36c60c9970 100644 --- a/src/objective-c/tests/Podfile +++ b/src/objective-c/tests/Podfile @@ -3,36 +3,27 @@ platform :ios, '8.0' install! 'cocoapods', :deterministic_uuids => false -def shared_pods - pod 'Protobuf', :path => "../../../third_party/protobuf" - pod 'BoringSSL', :podspec => ".." - pod 'gRPC', :path => "../../.." - pod 'gRPC-Core', :path => "../../.." - pod 'gRPC-RxLibrary', :path => "../../.." - pod 'gRPC-ProtoRPC', :path => "../../.." - pod 'RemoteTest', :path => "RemoteTestClient" +# Location of gRPC's repo root relative to this file. +GRPC_LOCAL_SRC = '../../..' + +# Install the dependencies in the main target plus all test targets. +%w( + Tests + AllTests + RxLibraryUnitTests + InteropTestsRemote + InteropTestsLocalSSL + InteropTestsLocalCleartext +).each do |target_name| + target target_name do + pod 'Protobuf', :path => "#{GRPC_LOCAL_SRC}/third_party/protobuf" + pod 'BoringSSL', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c" + pod 'gRPC', :path => GRPC_LOCAL_SRC + pod 'gRPC-Core', :path => GRPC_LOCAL_SRC + pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC + pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC + pod 'RemoteTest', :path => "RemoteTestClient" + end end -target 'Tests' do - shared_pods -end - -target 'AllTests' do - shared_pods -end - -target 'RxLibraryUnitTests' do - shared_pods -end - -target 'InteropTestsRemote' do - shared_pods -end - -target 'InteropTestsLocalSSL' do - shared_pods -end - -target 'InteropTestsLocalCleartext' do - shared_pods end From eb71417df91abe8515f522b242003630e664e7df Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Wed, 8 Jun 2016 14:43:37 -0700 Subject: [PATCH 0073/1039] Podspec changes for generated code. --- src/objective-c/tests/RemoteTestClient/RemoteTest.podspec | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec b/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec index e1fd9910389..7bf81c94e49 100644 --- a/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec +++ b/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec @@ -23,13 +23,17 @@ Pod::Spec.new do |s| ms.header_mappings_dir = "." ms.requires_arc = false ms.dependency "Protobuf", "~> 3.0.0-alpha-4" + # This is needed by all pods that depend on Protobuf: + ms.pod_target_xcconfig = { + 'GCC_PREPROCESSOR_DEFINITIONS' => '$(inherited) GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1', + } end s.subspec "Services" do |ss| ss.source_files = "*.pbrpc.{h,m}" ss.header_mappings_dir = "." ss.requires_arc = true - ss.dependency "gRPC", "~> 0.12" + ss.dependency "gRPC-ProtoRPC", "~> 0.14" ss.dependency "#{s.name}/Messages" end end From 2f46326213a466b24137b60d9a8660cb07ccfe31 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Wed, 8 Jun 2016 14:51:41 -0700 Subject: [PATCH 0074/1039] Pre-install hook in the Tests podspec for local dev of gRPC-Core --- src/objective-c/tests/Podfile | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/src/objective-c/tests/Podfile b/src/objective-c/tests/Podfile index f36c60c9970..41814f18afb 100644 --- a/src/objective-c/tests/Podfile +++ b/src/objective-c/tests/Podfile @@ -26,4 +26,36 @@ GRPC_LOCAL_SRC = '../../..' end end +# gRPC-Core.podspec needs to be modified to be successfully used for local development. A Podfile's +# pre_install hook lets us do that. The block passed to it runs after the podspecs are downloaded +# and before they are installed in the user project. +# +# This podspec searches for the gRPC core library headers under "$(PODS_ROOT)/gRPC-Core", where +# Cocoapods normally places the downloaded sources. When doing local development of the libraries, +# though, Cocoapods just takes the sources from whatever directory was specified using `:path`, and +# doesn't copy them under $(PODS_ROOT). When using static libraries, one can sometimes rely on the +# symbolic links to the pods headers that Cocoapods creates under "$(PODS_ROOT)/Headers". But those +# aren't created when using dynamic frameworks. So our solution is to modify the podspec on the fly +# to point at the local directory where the sources are. +# +# TODO(jcanizales): Send a PR to Cocoapods to get rid of this need. +pre_install do |installer| + # This is the gRPC-Core podspec object, as initialized by its podspec file. + grpc_core_spec = installer.pod_targets.find{|t| t.name == 'gRPC-Core'}.root_spec + + # Copied from gRPC-Core.podspec, except for the adjusted src_root: + + src_root = "$(PODS_ROOT)/../#{GRPC_LOCAL_SRC}" + + public_build_settings = { + 'GRPC_SRC_ROOT' => src_root, + 'HEADER_SEARCH_PATHS' => '"$(inherited)" "$(GRPC_SRC_ROOT)/include"', + } + private_build_settings = public_build_settings.merge({ + 'USE_HEADERMAP' => 'NO', + 'ALWAYS_SEARCH_USER_PATHS' => 'NO', + 'USER_HEADER_SEARCH_PATHS' => '"$(GRPC_SRC_ROOT)"', + }) + grpc_core_spec.user_target_xcconfig = public_build_settings + grpc_core_spec.pod_target_xcconfig = private_build_settings end From 66c8ddd9201f1da0476f5f85ce27c34c867d27d0 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Wed, 8 Jun 2016 17:39:40 -0700 Subject: [PATCH 0075/1039] gRPC-Core.podspec: init submodules, for nanopb --- gRPC-Core.podspec | 1 + templates/gRPC-Core.podspec.template | 1 + 2 files changed, 2 insertions(+) diff --git a/gRPC-Core.podspec b/gRPC-Core.podspec index 54e652eb1a2..5d21ca94ded 100644 --- a/gRPC-Core.podspec +++ b/gRPC-Core.podspec @@ -46,6 +46,7 @@ Pod::Spec.new do |s| s.source = { :git => 'https://github.com/grpc/grpc.git', :tag => "release-#{version.gsub(/\./, '_')}-objectivec-#{version}", + :submodules => true, } s.ios.deployment_target = '7.1' diff --git a/templates/gRPC-Core.podspec.template b/templates/gRPC-Core.podspec.template index efe3738c420..a634d46a482 100644 --- a/templates/gRPC-Core.podspec.template +++ b/templates/gRPC-Core.podspec.template @@ -64,6 +64,7 @@ s.source = { :git => 'https://github.com/grpc/grpc.git', :tag => "release-#{version.gsub(/\./, '_')}-objectivec-#{version}", + :submodules => true, } s.ios.deployment_target = '7.1' From db502a79862c1db33da355e7c2ebaf97019132b6 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Wed, 8 Jun 2016 19:20:16 -0700 Subject: [PATCH 0076/1039] Update documentation and samples. --- gRPC-ProtoRPC.podspec | 2 +- src/objective-c/README.md | 12 ++++- .../RemoteTestClient/RemoteTest.podspec | 30 ++++++++----- src/objective-c/examples/Sample/Podfile | 18 ++++++-- .../Sample/Sample.xcodeproj/project.pbxproj | 44 ++++++++++++++----- src/objective-c/examples/SwiftSample/Podfile | 18 ++++++-- .../SwiftSample.xcodeproj/project.pbxproj | 36 ++++++++------- .../tests/RemoteTestClient/RemoteTest.podspec | 2 +- 8 files changed, 113 insertions(+), 49 deletions(-) diff --git a/gRPC-ProtoRPC.podspec b/gRPC-ProtoRPC.podspec index 1ff539d5ee6..6f66f928e6b 100644 --- a/gRPC-ProtoRPC.podspec +++ b/gRPC-ProtoRPC.podspec @@ -61,7 +61,7 @@ Pod::Spec.new do |s| s.dependency 'gRPC', version s.dependency 'gRPC-RxLibrary', version - s.dependency 'Protobuf', '~> 3.0.0-alpha-4' + s.dependency 'Protobuf', '~> 3.0.0-beta-2' # This is needed by all pods that depend on Protobuf: s.pod_target_xcconfig = { 'GCC_PREPROCESSOR_DEFINITIONS' => '$(inherited) GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1', diff --git a/src/objective-c/README.md b/src/objective-c/README.md index 30d9aad64cd..736c324ca95 100644 --- a/src/objective-c/README.md +++ b/src/objective-c/README.md @@ -47,6 +47,10 @@ Pod::Spec.new do |s| s.name = '' s.version = '0.0.1' s.license = '...' + s.authors = { '' => '' } + s.homepage = '...' + s.summary = '...' + s.source = { :git => 'https://github.com/...' } s.ios.deployment_target = '7.1' s.osx.deployment_target = '10.9' @@ -60,7 +64,11 @@ Pod::Spec.new do |s| ms.source_files = "*.pbobjc.{h,m}" ms.header_mappings_dir = "." ms.requires_arc = false - ms.dependency "Protobuf", "~> 3.0.0-alpha-4" + ms.dependency "Protobuf", "~> 3.0.0-beta-2" + # This is needed by all pods that depend on Protobuf: + ms.pod_target_xcconfig = { + 'GCC_PREPROCESSOR_DEFINITIONS' => '$(inherited) GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1', + } end # The --objcgrpc_out plugin generates a pair of .pbrpc.h/.pbrpc.m files for each .proto file with @@ -69,7 +77,7 @@ Pod::Spec.new do |s| ss.source_files = "*.pbrpc.{h,m}" ss.header_mappings_dir = "." ss.requires_arc = true - ss.dependency "gRPC", "~> 0.12" + ss.dependency "gRPC-ProtoRPC", "~> 0.14" ss.dependency "#{s.name}/Messages" end end diff --git a/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec b/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec index 5addf26fc46..18a9443944a 100644 --- a/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec +++ b/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec @@ -1,7 +1,11 @@ Pod::Spec.new do |s| - s.name = "RemoteTest" - s.version = "0.0.1" - s.license = "New BSD" + s.name = 'RemoteTest' + s.version = '0.0.1' + s.license = 'New BSD' + s.authors = { 'gRPC contributors' => 'grpc-io@googlegroups.com' } + s.homepage = 'http://www.grpc.io/' + s.summary = 'RemoteTest example' + s.source = { :git => 'https://github.com/grpc/grpc.git' } s.ios.deployment_target = '7.1' s.osx.deployment_target = '10.9' @@ -11,18 +15,22 @@ Pod::Spec.new do |s| protoc --objc_out=. --objcgrpc_out=. *.proto CMD - s.subspec "Messages" do |ms| - ms.source_files = "*.pbobjc.{h,m}" - ms.header_mappings_dir = "." + s.subspec 'Messages' do |ms| + ms.source_files = '*.pbobjc.{h,m}' + ms.header_mappings_dir = '.' ms.requires_arc = false - ms.dependency "Protobuf", "~> 3.0.0-alpha-4" + ms.dependency 'Protobuf', '~> 3.0.0-beta-2' + # This is needed by all pods that depend on Protobuf: + ms.pod_target_xcconfig = { + 'GCC_PREPROCESSOR_DEFINITIONS' => '$(inherited) GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1', + } end - s.subspec "Services" do |ss| - ss.source_files = "*.pbrpc.{h,m}" - ss.header_mappings_dir = "." + s.subspec 'Services' do |ss| + ss.source_files = '*.pbrpc.{h,m}' + ss.header_mappings_dir = '.' ss.requires_arc = true - ss.dependency "gRPC", "~> 0.12" + ss.dependency 'gRPC-ProtoRPC', '~> 0.14' ss.dependency "#{s.name}/Messages" end end diff --git a/src/objective-c/examples/Sample/Podfile b/src/objective-c/examples/Sample/Podfile index 93859fb7340..77e37e98afd 100644 --- a/src/objective-c/examples/Sample/Podfile +++ b/src/objective-c/examples/Sample/Podfile @@ -1,10 +1,20 @@ source 'https://github.com/CocoaPods/Specs.git' platform :ios, '8.0' -pod 'Protobuf', :path => "../../../../third_party/protobuf" -pod 'BoringSSL', :podspec => "../.." -pod 'gRPC', :path => "../../../.." -pod 'RemoteTest', :path => "../RemoteTestClient" +install! 'cocoapods', :deterministic_uuids => false + +# Location of gRPC's repo root relative to this file. +GRPC_LOCAL_SRC = '../../../..' target 'Sample' do + pod 'Protobuf', :path => "#{GRPC_LOCAL_SRC}/third_party/protobuf" + + pod 'BoringSSL', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c" + + pod 'gRPC', :path => GRPC_LOCAL_SRC + pod 'gRPC-Core', :path => GRPC_LOCAL_SRC + pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC + pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC + + pod 'RemoteTest', :path => "../RemoteTestClient" end diff --git a/src/objective-c/examples/Sample/Sample.xcodeproj/project.pbxproj b/src/objective-c/examples/Sample/Sample.xcodeproj/project.pbxproj index 611eb6032d5..067ff3a13b9 100644 --- a/src/objective-c/examples/Sample/Sample.xcodeproj/project.pbxproj +++ b/src/objective-c/examples/Sample/Sample.xcodeproj/project.pbxproj @@ -12,10 +12,11 @@ 6369A2761A9322E20015FC5C /* ViewController.m in Sources */ = {isa = PBXBuildFile; fileRef = 6369A2751A9322E20015FC5C /* ViewController.m */; }; 6369A2791A9322E20015FC5C /* Main.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 6369A2771A9322E20015FC5C /* Main.storyboard */; }; 6369A27B1A9322E20015FC5C /* Images.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 6369A27A1A9322E20015FC5C /* Images.xcassets */; }; - FC81FE63CA655031F3524EC0 /* libPods.a in Frameworks */ = {isa = PBXBuildFile; fileRef = 2DC7B7C4C0410F43B9621631 /* libPods.a */; }; + AF4A3AE4267AAFB18431B287 /* libPods-Sample.a in Frameworks */ = {isa = PBXBuildFile; fileRef = 0E797A3F68012317421BA87E /* libPods-Sample.a */; }; /* End PBXBuildFile section */ /* Begin PBXFileReference section */ + 0E797A3F68012317421BA87E /* libPods-Sample.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = "libPods-Sample.a"; sourceTree = BUILT_PRODUCTS_DIR; }; 2DC7B7C4C0410F43B9621631 /* libPods.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = libPods.a; sourceTree = BUILT_PRODUCTS_DIR; }; 6369A26A1A9322E20015FC5C /* Sample.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = Sample.app; sourceTree = BUILT_PRODUCTS_DIR; }; 6369A26E1A9322E20015FC5C /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = ""; }; @@ -26,6 +27,8 @@ 6369A2751A9322E20015FC5C /* ViewController.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = ViewController.m; sourceTree = ""; }; 6369A2781A9322E20015FC5C /* Base */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; name = Base; path = Base.lproj/Main.storyboard; sourceTree = ""; }; 6369A27A1A9322E20015FC5C /* Images.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Images.xcassets; sourceTree = ""; }; + 7F4A2C3C7ABEB427FB51922C /* Pods-Sample.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-Sample.release.xcconfig"; path = "Pods/Target Support Files/Pods-Sample/Pods-Sample.release.xcconfig"; sourceTree = ""; }; + 7FB00EC776A4A5F68401AE2B /* Pods-Sample.debug.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-Sample.debug.xcconfig"; path = "Pods/Target Support Files/Pods-Sample/Pods-Sample.debug.xcconfig"; sourceTree = ""; }; AC29DD6FCDF962F519FEBB0D /* Pods.debug.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = Pods.debug.xcconfig; path = "Pods/Target Support Files/Pods/Pods.debug.xcconfig"; sourceTree = ""; }; C68330F8D451CC6ACEABA09F /* Pods.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = Pods.release.xcconfig; path = "Pods/Target Support Files/Pods/Pods.release.xcconfig"; sourceTree = ""; }; /* End PBXFileReference section */ @@ -35,7 +38,7 @@ isa = PBXFrameworksBuildPhase; buildActionMask = 2147483647; files = ( - FC81FE63CA655031F3524EC0 /* libPods.a in Frameworks */, + AF4A3AE4267AAFB18431B287 /* libPods-Sample.a in Frameworks */, ); runOnlyForDeploymentPostprocessing = 0; }; @@ -88,6 +91,8 @@ children = ( AC29DD6FCDF962F519FEBB0D /* Pods.debug.xcconfig */, C68330F8D451CC6ACEABA09F /* Pods.release.xcconfig */, + 7FB00EC776A4A5F68401AE2B /* Pods-Sample.debug.xcconfig */, + 7F4A2C3C7ABEB427FB51922C /* Pods-Sample.release.xcconfig */, ); name = Pods; sourceTree = ""; @@ -96,6 +101,7 @@ isa = PBXGroup; children = ( 2DC7B7C4C0410F43B9621631 /* libPods.a */, + 0E797A3F68012317421BA87E /* libPods-Sample.a */, ); name = Frameworks; sourceTree = ""; @@ -107,11 +113,12 @@ isa = PBXNativeTarget; buildConfigurationList = 6369A28D1A9322E20015FC5C /* Build configuration list for PBXNativeTarget "Sample" */; buildPhases = ( - 41F7486D8F66994B0BFB84AF /* Check Pods Manifest.lock */, + 41F7486D8F66994B0BFB84AF /* 📦 Check Pods Manifest.lock */, 6369A2661A9322E20015FC5C /* Sources */, 6369A2671A9322E20015FC5C /* Frameworks */, 6369A2681A9322E20015FC5C /* Resources */, - 04554623324BE4A838846086 /* Copy Pods Resources */, + 04554623324BE4A838846086 /* 📦 Copy Pods Resources */, + D2E32169A6ED4D88151F0046 /* 📦 Embed Pods Frameworks */, ); buildRules = ( ); @@ -167,29 +174,29 @@ /* End PBXResourcesBuildPhase section */ /* Begin PBXShellScriptBuildPhase section */ - 04554623324BE4A838846086 /* Copy Pods Resources */ = { + 04554623324BE4A838846086 /* 📦 Copy Pods Resources */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Copy Pods Resources"; + name = "📦 Copy Pods Resources"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods/Pods-resources.sh\"\n"; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-Sample/Pods-Sample-resources.sh\"\n"; showEnvVarsInLog = 0; }; - 41F7486D8F66994B0BFB84AF /* Check Pods Manifest.lock */ = { + 41F7486D8F66994B0BFB84AF /* 📦 Check Pods Manifest.lock */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Check Pods Manifest.lock"; + name = "📦 Check Pods Manifest.lock"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; @@ -197,6 +204,21 @@ shellScript = "diff \"${PODS_ROOT}/../Podfile.lock\" \"${PODS_ROOT}/Manifest.lock\" > /dev/null\nif [[ $? != 0 ]] ; then\n cat << EOM\nerror: The sandbox is not in sync with the Podfile.lock. Run 'pod install' or update your CocoaPods installation.\nEOM\n exit 1\nfi\n"; showEnvVarsInLog = 0; }; + D2E32169A6ED4D88151F0046 /* 📦 Embed Pods Frameworks */ = { + isa = PBXShellScriptBuildPhase; + buildActionMask = 2147483647; + files = ( + ); + inputPaths = ( + ); + name = "📦 Embed Pods Frameworks"; + outputPaths = ( + ); + runOnlyForDeploymentPostprocessing = 0; + shellPath = /bin/sh; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-Sample/Pods-Sample-frameworks.sh\"\n"; + showEnvVarsInLog = 0; + }; /* End PBXShellScriptBuildPhase section */ /* Begin PBXSourcesBuildPhase section */ @@ -304,7 +326,7 @@ }; 6369A28E1A9322E20015FC5C /* Debug */ = { isa = XCBuildConfiguration; - baseConfigurationReference = AC29DD6FCDF962F519FEBB0D /* Pods.debug.xcconfig */; + baseConfigurationReference = 7FB00EC776A4A5F68401AE2B /* Pods-Sample.debug.xcconfig */; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; INFOPLIST_FILE = Sample/Info.plist; @@ -315,7 +337,7 @@ }; 6369A28F1A9322E20015FC5C /* Release */ = { isa = XCBuildConfiguration; - baseConfigurationReference = C68330F8D451CC6ACEABA09F /* Pods.release.xcconfig */; + baseConfigurationReference = 7F4A2C3C7ABEB427FB51922C /* Pods-Sample.release.xcconfig */; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; INFOPLIST_FILE = Sample/Info.plist; diff --git a/src/objective-c/examples/SwiftSample/Podfile b/src/objective-c/examples/SwiftSample/Podfile index f2df4a34a34..2e789c3ef1c 100644 --- a/src/objective-c/examples/SwiftSample/Podfile +++ b/src/objective-c/examples/SwiftSample/Podfile @@ -1,10 +1,20 @@ source 'https://github.com/CocoaPods/Specs.git' platform :ios, '8.0' -pod 'Protobuf', :path => "../../../../third_party/protobuf" -pod 'BoringSSL', :podspec => "../.." -pod 'gRPC', :path => "../../../.." -pod 'RemoteTest', :path => "../RemoteTestClient" +install! 'cocoapods', :deterministic_uuids => false + +# Location of gRPC's repo root relative to this file. +GRPC_LOCAL_SRC = '../../../..' target 'SwiftSample' do + pod 'Protobuf', :path => "#{GRPC_LOCAL_SRC}/third_party/protobuf" + + pod 'BoringSSL', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c" + + pod 'gRPC', :path => GRPC_LOCAL_SRC + pod 'gRPC-Core', :path => GRPC_LOCAL_SRC + pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC + pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC + + pod 'RemoteTest', :path => "../RemoteTestClient" end diff --git a/src/objective-c/examples/SwiftSample/SwiftSample.xcodeproj/project.pbxproj b/src/objective-c/examples/SwiftSample/SwiftSample.xcodeproj/project.pbxproj index 2f5716082bf..2f9a41875d1 100644 --- a/src/objective-c/examples/SwiftSample/SwiftSample.xcodeproj/project.pbxproj +++ b/src/objective-c/examples/SwiftSample/SwiftSample.xcodeproj/project.pbxproj @@ -7,15 +7,16 @@ objects = { /* Begin PBXBuildFile section */ - 253D3A297105CA46DA960A11 /* libPods.a in Frameworks */ = {isa = PBXBuildFile; fileRef = DC58ACA18DCCB1553531B885 /* libPods.a */; }; 633BFFC81B950B210007E424 /* AppDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = 633BFFC71B950B210007E424 /* AppDelegate.swift */; }; 633BFFCA1B950B210007E424 /* ViewController.swift in Sources */ = {isa = PBXBuildFile; fileRef = 633BFFC91B950B210007E424 /* ViewController.swift */; }; 633BFFCD1B950B210007E424 /* Main.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 633BFFCB1B950B210007E424 /* Main.storyboard */; }; 633BFFCF1B950B210007E424 /* Images.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 633BFFCE1B950B210007E424 /* Images.xcassets */; }; + AE76C196CEB7CF33421129CD /* libPods-SwiftSample.a in Frameworks */ = {isa = PBXBuildFile; fileRef = FBDB042A015267195A988FFB /* libPods-SwiftSample.a */; }; /* End PBXBuildFile section */ /* Begin PBXFileReference section */ 12C7B447AA80E624D93B5C54 /* Pods.debug.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = Pods.debug.xcconfig; path = "Pods/Target Support Files/Pods/Pods.debug.xcconfig"; sourceTree = ""; }; + 5ABC04686A90EA11D0BD35A1 /* Pods-SwiftSample.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-SwiftSample.release.xcconfig"; path = "Pods/Target Support Files/Pods-SwiftSample/Pods-SwiftSample.release.xcconfig"; sourceTree = ""; }; 633BFFC21B950B210007E424 /* SwiftSample.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = SwiftSample.app; sourceTree = BUILT_PRODUCTS_DIR; }; 633BFFC61B950B210007E424 /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = ""; }; 633BFFC71B950B210007E424 /* AppDelegate.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AppDelegate.swift; sourceTree = ""; }; @@ -24,7 +25,9 @@ 633BFFCE1B950B210007E424 /* Images.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Images.xcassets; sourceTree = ""; }; 6367AD231B951655007FD3A4 /* Bridging-Header.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = "Bridging-Header.h"; sourceTree = ""; }; C335CBC4C160E0D9EDEE646B /* Pods.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = Pods.release.xcconfig; path = "Pods/Target Support Files/Pods/Pods.release.xcconfig"; sourceTree = ""; }; + C4278D3EA95326A34DF5D15F /* Pods-SwiftSample.debug.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-SwiftSample.debug.xcconfig"; path = "Pods/Target Support Files/Pods-SwiftSample/Pods-SwiftSample.debug.xcconfig"; sourceTree = ""; }; DC58ACA18DCCB1553531B885 /* libPods.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = libPods.a; sourceTree = BUILT_PRODUCTS_DIR; }; + FBDB042A015267195A988FFB /* libPods-SwiftSample.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = "libPods-SwiftSample.a"; sourceTree = BUILT_PRODUCTS_DIR; }; /* End PBXFileReference section */ /* Begin PBXFrameworksBuildPhase section */ @@ -32,7 +35,7 @@ isa = PBXFrameworksBuildPhase; buildActionMask = 2147483647; files = ( - 253D3A297105CA46DA960A11 /* libPods.a in Frameworks */, + AE76C196CEB7CF33421129CD /* libPods-SwiftSample.a in Frameworks */, ); runOnlyForDeploymentPostprocessing = 0; }; @@ -44,6 +47,8 @@ children = ( 12C7B447AA80E624D93B5C54 /* Pods.debug.xcconfig */, C335CBC4C160E0D9EDEE646B /* Pods.release.xcconfig */, + C4278D3EA95326A34DF5D15F /* Pods-SwiftSample.debug.xcconfig */, + 5ABC04686A90EA11D0BD35A1 /* Pods-SwiftSample.release.xcconfig */, ); name = Pods; sourceTree = ""; @@ -91,6 +96,7 @@ isa = PBXGroup; children = ( DC58ACA18DCCB1553531B885 /* libPods.a */, + FBDB042A015267195A988FFB /* libPods-SwiftSample.a */, ); name = Frameworks; sourceTree = ""; @@ -102,12 +108,12 @@ isa = PBXNativeTarget; buildConfigurationList = 633BFFE11B950B210007E424 /* Build configuration list for PBXNativeTarget "SwiftSample" */; buildPhases = ( - 6BEEB33CA2705D7D2F2210E6 /* Check Pods Manifest.lock */, + 6BEEB33CA2705D7D2F2210E6 /* 📦 Check Pods Manifest.lock */, 633BFFBE1B950B210007E424 /* Sources */, 633BFFBF1B950B210007E424 /* Frameworks */, 633BFFC01B950B210007E424 /* Resources */, - AC2F6F9AB1C090BB0BEE6E4D /* Copy Pods Resources */, - A1738A987353B0BF2C64F0F7 /* Embed Pods Frameworks */, + AC2F6F9AB1C090BB0BEE6E4D /* 📦 Copy Pods Resources */, + A1738A987353B0BF2C64F0F7 /* 📦 Embed Pods Frameworks */, ); buildRules = ( ); @@ -164,14 +170,14 @@ /* End PBXResourcesBuildPhase section */ /* Begin PBXShellScriptBuildPhase section */ - 6BEEB33CA2705D7D2F2210E6 /* Check Pods Manifest.lock */ = { + 6BEEB33CA2705D7D2F2210E6 /* 📦 Check Pods Manifest.lock */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Check Pods Manifest.lock"; + name = "📦 Check Pods Manifest.lock"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; @@ -179,34 +185,34 @@ shellScript = "diff \"${PODS_ROOT}/../Podfile.lock\" \"${PODS_ROOT}/Manifest.lock\" > /dev/null\nif [[ $? != 0 ]] ; then\n cat << EOM\nerror: The sandbox is not in sync with the Podfile.lock. Run 'pod install' or update your CocoaPods installation.\nEOM\n exit 1\nfi\n"; showEnvVarsInLog = 0; }; - A1738A987353B0BF2C64F0F7 /* Embed Pods Frameworks */ = { + A1738A987353B0BF2C64F0F7 /* 📦 Embed Pods Frameworks */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Embed Pods Frameworks"; + name = "📦 Embed Pods Frameworks"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods/Pods-frameworks.sh\"\n"; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-SwiftSample/Pods-SwiftSample-frameworks.sh\"\n"; showEnvVarsInLog = 0; }; - AC2F6F9AB1C090BB0BEE6E4D /* Copy Pods Resources */ = { + AC2F6F9AB1C090BB0BEE6E4D /* 📦 Copy Pods Resources */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Copy Pods Resources"; + name = "📦 Copy Pods Resources"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods/Pods-resources.sh\"\n"; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-SwiftSample/Pods-SwiftSample-resources.sh\"\n"; showEnvVarsInLog = 0; }; /* End PBXShellScriptBuildPhase section */ @@ -320,7 +326,7 @@ }; 633BFFE21B950B210007E424 /* Debug */ = { isa = XCBuildConfiguration; - baseConfigurationReference = 12C7B447AA80E624D93B5C54 /* Pods.debug.xcconfig */; + baseConfigurationReference = C4278D3EA95326A34DF5D15F /* Pods-SwiftSample.debug.xcconfig */; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; INFOPLIST_FILE = Info.plist; @@ -333,7 +339,7 @@ }; 633BFFE31B950B210007E424 /* Release */ = { isa = XCBuildConfiguration; - baseConfigurationReference = C335CBC4C160E0D9EDEE646B /* Pods.release.xcconfig */; + baseConfigurationReference = 5ABC04686A90EA11D0BD35A1 /* Pods-SwiftSample.release.xcconfig */; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; INFOPLIST_FILE = Info.plist; diff --git a/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec b/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec index 7bf81c94e49..8e72b2c8fe3 100644 --- a/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec +++ b/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec @@ -22,7 +22,7 @@ Pod::Spec.new do |s| ms.source_files = "*.pbobjc.{h,m}" ms.header_mappings_dir = "." ms.requires_arc = false - ms.dependency "Protobuf", "~> 3.0.0-alpha-4" + ms.dependency "Protobuf", "~> 3.0.0-beta-2" # This is needed by all pods that depend on Protobuf: ms.pod_target_xcconfig = { 'GCC_PREPROCESSOR_DEFINITIONS' => '$(inherited) GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1', From 3775074c8606f728d1d58949aaa13852701d6471 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Wed, 8 Jun 2016 20:10:20 -0700 Subject: [PATCH 0077/1039] Clean up and document gRPC-Core.podspec better --- gRPC-Core.podspec | 49 ++++++++++++++++------------ templates/gRPC-Core.podspec.template | 21 ++++++++---- 2 files changed, 44 insertions(+), 26 deletions(-) diff --git a/gRPC-Core.podspec b/gRPC-Core.podspec index 5d21ca94ded..6411e348be8 100644 --- a/gRPC-Core.podspec +++ b/gRPC-Core.podspec @@ -57,8 +57,37 @@ Pod::Spec.new do |s| s.module_name = name s.header_dir = name + s.header_mappings_dir = 'include/grpc' + + src_root = '$(PODS_ROOT)/gRPC-Core' + # This isn't officially supported in Cocoapods. We've asked for an alternative: + # https://github.com/CocoaPods/CocoaPods/issues/4386 + # + # The src_root value of $(PODS_ROOT)/gRPC-Core assumes Cocoapods is installing this pod from its + # remote repo. For local development of this library, enabled by using ":path" in the Podfile, + # that assumption is wrong. In such case, the following settings need to be reset with the + # appropriate value of src_root. This can be accomplished in the pre_install hook of the Podfile; + # see src/objective-c/tests/Podfile for an example. + public_build_settings = { + 'GRPC_SRC_ROOT' => src_root, + 'HEADER_SEARCH_PATHS' => '"$(inherited)" "$(GRPC_SRC_ROOT)/include"', + } + private_build_settings = public_build_settings.merge({ + 'USE_HEADERMAP' => 'NO', + 'ALWAYS_SEARCH_USER_PATHS' => 'NO', + 'USER_HEADER_SEARCH_PATHS' => '"$(GRPC_SRC_ROOT)"', + }) + s.user_target_xcconfig = public_build_settings + s.pod_target_xcconfig = private_build_settings + + s.libraries = 'z' + s.dependency 'BoringSSL', '~> 3.0' + + # A module map is necessary for a dynamic framework to be correctly created by Cocoapods. s.module_map = 'include/grpc/module.modulemap' + # List of source files generated by a template. To save you from scrolling, this is the last part + # of the podspec. s.source_files = 'src/core/lib/profiling/timers.h', 'src/core/lib/support/backoff.h', 'src/core/lib/support/block_annotate.h', @@ -638,24 +667,4 @@ Pod::Spec.new do |s| 'src/core/ext/census/grpc_filter.h', 'src/core/ext/census/mlog.h', 'src/core/ext/census/rpc_metric_id.h' - - s.header_mappings_dir = 'include/grpc' - - src_root = '$(PODS_ROOT)/gRPC-Core' - # This isn't officially supported in Cocoapods. We've asked for an alternative: - # https://github.com/CocoaPods/CocoaPods/issues/4386 - public_build_settings = { - 'GRPC_SRC_ROOT' => src_root, - 'HEADER_SEARCH_PATHS' => '"$(inherited)" "$(GRPC_SRC_ROOT)/include"', - } - private_build_settings = public_build_settings.merge({ - 'USE_HEADERMAP' => 'NO', - 'ALWAYS_SEARCH_USER_PATHS' => 'NO', - 'USER_HEADER_SEARCH_PATHS' => '"$(GRPC_SRC_ROOT)"', - }) - s.user_target_xcconfig = public_build_settings - s.pod_target_xcconfig = private_build_settings - - s.libraries = 'z' - s.dependency 'BoringSSL', '~> 3.0' end diff --git a/templates/gRPC-Core.podspec.template b/templates/gRPC-Core.podspec.template index a634d46a482..4572a6151f6 100644 --- a/templates/gRPC-Core.podspec.template +++ b/templates/gRPC-Core.podspec.template @@ -75,17 +75,17 @@ s.module_name = name s.header_dir = name - s.module_map = 'include/grpc/module.modulemap' - - s.source_files = ${(',\n' + 19*' ').join('\'%s\'' % f for f in grpc_files(libs))} - - s.private_header_files = ${(',\n' + 27*' ').join('\'%s\'' % f for f in grpc_private_headers(libs))} - s.header_mappings_dir = 'include/grpc' src_root = '$(PODS_ROOT)/gRPC-Core' # This isn't officially supported in Cocoapods. We've asked for an alternative: # https://github.com/CocoaPods/CocoaPods/issues/4386 + # + # The src_root value of $(PODS_ROOT)/gRPC-Core assumes Cocoapods is installing this pod from its + # remote repo. For local development of this library, enabled by using ":path" in the Podfile, + # that assumption is wrong. In such case, the following settings need to be reset with the + # appropriate value of src_root. This can be accomplished in the pre_install hook of the Podfile; + # see src/objective-c/tests/Podfile for an example. public_build_settings = { 'GRPC_SRC_ROOT' => src_root, 'HEADER_SEARCH_PATHS' => '"$(inherited)" "$(GRPC_SRC_ROOT)/include"', @@ -100,4 +100,13 @@ s.libraries = 'z' s.dependency 'BoringSSL', '~> 3.0' + + # A module map is necessary for a dynamic framework to be correctly created by Cocoapods. + s.module_map = 'include/grpc/module.modulemap' + + # List of source files generated by a template. To save you from scrolling, this is the last part + # of the podspec. + s.source_files = ${(',\n' + 19*' ').join('\'%s\'' % f for f in grpc_files(libs))} + + s.private_header_files = ${(',\n' + 27*' ').join('\'%s\'' % f for f in grpc_private_headers(libs))} end From 8e4926c0eeb0df9e5c8029136e39f6c8700f0814 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 8 Jun 2016 20:33:19 -0700 Subject: [PATCH 0078/1039] pollset_kick optimization (do not kick any other thread if the current thread can be kicked) --- src/core/lib/iomgr/ev_epoll_linux.c | 153 ++++++++++++++++------------ 1 file changed, 87 insertions(+), 66 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 046ec5e7407..d45f87c2f8e 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -844,6 +844,8 @@ static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, /******************************************************************************* * Pollset Definitions */ +GPR_TLS_DECL(g_current_thread_pollset); +GPR_TLS_DECL(g_current_thread_worker); static void sig_handler(int sig_num) { #ifdef GRPC_EPOLL_DEBUG @@ -859,11 +861,15 @@ static void poller_kick_init() { /* Global state management */ static void pollset_global_init(void) { grpc_wakeup_fd_init(&grpc_global_wakeup_fd); + gpr_tls_init(&g_current_thread_pollset); + gpr_tls_init(&g_current_thread_worker); poller_kick_init(); } static void pollset_global_shutdown(void) { grpc_wakeup_fd_destroy(&grpc_global_wakeup_fd); + gpr_tls_destroy(&g_current_thread_pollset); + gpr_tls_destroy(&g_current_thread_worker); } static void pollset_worker_kick(grpc_pollset_worker *worker) { @@ -915,7 +921,9 @@ static void pollset_kick(grpc_pollset *p, GPR_TIMER_BEGIN("pollset_kick.broadcast", 0); for (worker = p->root_worker.next; worker != &p->root_worker; worker = worker->next) { - pollset_worker_kick(worker); + if (gpr_tls_get(&g_current_thread_worker) != (intptr_t)worker) { + pollset_worker_kick(worker); + } } } else { p->kicked_without_pollers = true; @@ -923,9 +931,18 @@ static void pollset_kick(grpc_pollset *p, GPR_TIMER_END("pollset_kick.broadcast", 0); } else { GPR_TIMER_MARK("kicked_specifically", 0); - pollset_worker_kick(worker); + if (gpr_tls_get(&g_current_thread_worker) != (intptr_t)worker) { + pollset_worker_kick(worker); + } } - } else { + } else if (gpr_tls_get(&g_current_thread_pollset) != (intptr_t)p) { + /* Since worker == NULL, it means that we can kick "any" worker on this + pollset 'p'. If 'p' happens to be the same pollset this thread is + currently polling (i.e in pollset_work() function), then there is no need + to kick any other worker since the current thread can just absorb the + kick. This is the reason why we enter this case only when + g_current_thread_pollset is != p */ + GPR_TIMER_MARK("kick_anonymous", 0); worker = pop_front_worker(p); if (worker != NULL) { @@ -999,6 +1016,69 @@ static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { gpr_mu_unlock(&fd->mu); } +/* Release the reference to pollset->polling_island and set it to NULL. + pollset->mu must be held */ +static void pollset_release_polling_island_locked(grpc_pollset *pollset) { + gpr_mu_lock(&pollset->pi_mu); + if (pollset->polling_island) { + pollset->polling_island = + polling_island_update_and_lock(pollset->polling_island, 1, 0); + polling_island_unref_and_unlock(pollset->polling_island, 1); + pollset->polling_island = NULL; + } + gpr_mu_unlock(&pollset->pi_mu); +} + +static void finish_shutdown_locked(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset) { + /* The pollset cannot have any workers if we are at this stage */ + GPR_ASSERT(!pollset_has_workers(pollset)); + + pollset->finish_shutdown_called = true; + pollset_release_polling_island_locked(pollset); + + grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); +} + +/* pollset->mu lock must be held by the caller before calling this */ +static void pollset_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_closure *closure) { + GPR_TIMER_BEGIN("pollset_shutdown", 0); + GPR_ASSERT(!pollset->shutting_down); + pollset->shutting_down = true; + pollset->shutdown_done = closure; + pollset_kick(pollset, GRPC_POLLSET_KICK_BROADCAST); + + /* If the pollset has any workers, we cannot call finish_shutdown_locked() + because it would release the underlying polling island. In such a case, we + let the last worker call finish_shutdown_locked() from pollset_work() */ + if (!pollset_has_workers(pollset)) { + GPR_ASSERT(!pollset->finish_shutdown_called); + GPR_TIMER_MARK("pollset_shutdown.finish_shutdown_locked", 0); + finish_shutdown_locked(exec_ctx, pollset); + } + GPR_TIMER_END("pollset_shutdown", 0); +} + +/* pollset_shutdown is guaranteed to be called before pollset_destroy. So other + * than destroying the mutexes, there is nothing special that needs to be done + * here */ +static void pollset_destroy(grpc_pollset *pollset) { + GPR_ASSERT(!pollset_has_workers(pollset)); + gpr_mu_destroy(&pollset->pi_mu); + gpr_mu_destroy(&pollset->mu); +} + +static void pollset_reset(grpc_pollset *pollset) { + GPR_ASSERT(pollset->shutting_down); + GPR_ASSERT(!pollset_has_workers(pollset)); + pollset->shutting_down = false; + pollset->finish_shutdown_called = false; + pollset->kicked_without_pollers = false; + pollset->shutdown_done = NULL; + pollset_release_polling_island_locked(pollset); +} + #define GRPC_EPOLL_MAX_EVENTS 1000 static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, int timeout_ms, @@ -1103,69 +1183,6 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, GPR_TIMER_END("pollset_work_and_unlock", 0); } -/* Release the reference to pollset->polling_island and set it to NULL. - pollset->mu must be held */ -static void pollset_release_polling_island_locked(grpc_pollset *pollset) { - gpr_mu_lock(&pollset->pi_mu); - if (pollset->polling_island) { - pollset->polling_island = - polling_island_update_and_lock(pollset->polling_island, 1, 0); - polling_island_unref_and_unlock(pollset->polling_island, 1); - pollset->polling_island = NULL; - } - gpr_mu_unlock(&pollset->pi_mu); -} - -static void finish_shutdown_locked(grpc_exec_ctx *exec_ctx, - grpc_pollset *pollset) { - /* The pollset cannot have any workers if we are at this stage */ - GPR_ASSERT(!pollset_has_workers(pollset)); - - pollset->finish_shutdown_called = true; - pollset_release_polling_island_locked(pollset); - - grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); -} - -/* pollset->mu lock must be held by the caller before calling this */ -static void pollset_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_closure *closure) { - GPR_TIMER_BEGIN("pollset_shutdown", 0); - GPR_ASSERT(!pollset->shutting_down); - pollset->shutting_down = true; - pollset->shutdown_done = closure; - pollset_kick(pollset, GRPC_POLLSET_KICK_BROADCAST); - - /* If the pollset has any workers, we cannot call finish_shutdown_locked() - because it would release the underlying polling island. In such a case, we - let the last worker call finish_shutdown_locked() from pollset_work() */ - if (!pollset_has_workers(pollset)) { - GPR_ASSERT(!pollset->finish_shutdown_called); - GPR_TIMER_MARK("pollset_shutdown.finish_shutdown_locked", 0); - finish_shutdown_locked(exec_ctx, pollset); - } - GPR_TIMER_END("pollset_shutdown", 0); -} - -/* pollset_shutdown is guaranteed to be called before pollset_destroy. So other - * than destroying the mutexes, there is nothing special that needs to be done - * here */ -static void pollset_destroy(grpc_pollset *pollset) { - GPR_ASSERT(!pollset_has_workers(pollset)); - gpr_mu_destroy(&pollset->pi_mu); - gpr_mu_destroy(&pollset->mu); -} - -static void pollset_reset(grpc_pollset *pollset) { - GPR_ASSERT(pollset->shutting_down); - GPR_ASSERT(!pollset_has_workers(pollset)); - pollset->shutting_down = false; - pollset->finish_shutdown_called = false; - pollset->kicked_without_pollers = false; - pollset->shutdown_done = NULL; - pollset_release_polling_island_locked(pollset); -} - /* pollset->mu lock must be held by the caller before calling this. The function pollset_work() may temporarily release the lock (pollset->mu) during the course of its execution but it will always re-acquire the lock and @@ -1184,6 +1201,8 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, worker.pt_id = pthread_self(); *worker_hdl = &worker; + gpr_tls_set(&g_current_thread_pollset, (intptr_t)pollset); + gpr_tls_set(&g_current_thread_worker, (intptr_t)&worker); if (pollset->kicked_without_pollers) { /* If the pollset was kicked without pollers, pretend that the current @@ -1226,6 +1245,8 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, } *worker_hdl = NULL; + gpr_tls_set(&g_current_thread_pollset, (intptr_t)0); + gpr_tls_set(&g_current_thread_worker, (intptr_t)0); GPR_TIMER_END("pollset_work", 0); } From 0553a436610201b252cfce0ed5d2cea69da15e85 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 9 Jun 2016 00:42:41 -0700 Subject: [PATCH 0079/1039] Fix refcounting bug in polling_island_merge --- src/core/lib/iomgr/ev_epoll_linux.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index d45f87c2f8e..66bbae52b28 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -512,7 +512,6 @@ polling_island *polling_island_merge(polling_island *p, polling_island *q) { /* Get locks on both the polling islands */ polling_island_pair_update_and_lock(&p, &q); - /* TODO: sreek: Think about this scenario some more */ if (p == q) { /* Nothing needs to be done here */ gpr_mu_unlock(&p->mu); @@ -525,7 +524,7 @@ polling_island *polling_island_merge(polling_island *p, polling_island *q) { } /* "Merge" p with q i.e move all the fds from p (The one with fewer fds) to q - )Note that the refcounts on the fds being moved will not change here. This + Note that the refcounts on the fds being moved will not change here. This is why the last parameter in the following two functions is 'false') */ polling_island_add_fds_locked(q, p->fds, p->fd_cnt, false); polling_island_remove_all_fds_locked(p, false); @@ -533,9 +532,11 @@ polling_island *polling_island_merge(polling_island *p, polling_island *q) { /* Wakeup all the pollers (if any) on p so that they can pickup this change */ polling_island_add_wakeup_fd_locked(p, &polling_island_wakeup_fd); - /* The merged polling island inherits all the ref counts of the island merging - with it */ + /* - The merged polling island (i.e q) inherits all the ref counts of the + island merging with it (i.e p) + - The island p will lose a ref count */ q->ref_cnt += p->ref_cnt; + p->ref_cnt--; gpr_mu_unlock(&p->mu); gpr_mu_unlock(&q->mu); From 667557f22b1cc601c60d3227f209b8cf848738b5 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Thu, 9 Jun 2016 09:15:28 -0700 Subject: [PATCH 0080/1039] gRPC-Core.podspec: Clarify where the template is --- gRPC-Core.podspec | 12 ++++++------ templates/gRPC-Core.podspec.template | 12 ++++++------ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/gRPC-Core.podspec b/gRPC-Core.podspec index 6411e348be8..68275e444b4 100644 --- a/gRPC-Core.podspec +++ b/gRPC-Core.podspec @@ -1,8 +1,7 @@ # GRPC CocoaPods podspec -# This file has been automatically generated from a template file. -# Please look at the templates directory instead. -# This file can be regenerated from the template by running -# tools/buildgen/generate_projects.sh +# This file has been automatically generated from a template file. Please make modifications to +# `templates/gRPC-Core.podspec.template` instead. This file can be regenerated from the template by +# running `tools/buildgen/generate_projects.sh`. # Copyright 2015, Google Inc. # All rights reserved. @@ -86,8 +85,9 @@ Pod::Spec.new do |s| # A module map is necessary for a dynamic framework to be correctly created by Cocoapods. s.module_map = 'include/grpc/module.modulemap' - # List of source files generated by a template. To save you from scrolling, this is the last part - # of the podspec. + # List of source files generated by a template: `templates/gRPC-Core.podspec.template`. It can be + # regenerated from the template by running `tools/buildgen/generate_projects.sh`. + # To save you from scrolling, this is the last part of the podspec. s.source_files = 'src/core/lib/profiling/timers.h', 'src/core/lib/support/backoff.h', 'src/core/lib/support/block_annotate.h', diff --git a/templates/gRPC-Core.podspec.template b/templates/gRPC-Core.podspec.template index 4572a6151f6..ee0ca4cafe0 100644 --- a/templates/gRPC-Core.podspec.template +++ b/templates/gRPC-Core.podspec.template @@ -1,10 +1,9 @@ %YAML 1.2 --- | # GRPC CocoaPods podspec - # This file has been automatically generated from a template file. - # Please look at the templates directory instead. - # This file can be regenerated from the template by running - # tools/buildgen/generate_projects.sh + # This file has been automatically generated from a template file. Please make modifications to + # `templates/gRPC-Core.podspec.template` instead. This file can be regenerated from the template by + # running `tools/buildgen/generate_projects.sh`. # Copyright 2015, Google Inc. # All rights reserved. @@ -104,8 +103,9 @@ # A module map is necessary for a dynamic framework to be correctly created by Cocoapods. s.module_map = 'include/grpc/module.modulemap' - # List of source files generated by a template. To save you from scrolling, this is the last part - # of the podspec. + # List of source files generated by a template: `templates/gRPC-Core.podspec.template`. It can be + # regenerated from the template by running `tools/buildgen/generate_projects.sh`. + # To save you from scrolling, this is the last part of the podspec. s.source_files = ${(',\n' + 19*' ').join('\'%s\'' % f for f in grpc_files(libs))} s.private_header_files = ${(',\n' + 27*' ').join('\'%s\'' % f for f in grpc_private_headers(libs))} From 727440216553c01925c0f5bc88293108bb3f051f Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 9 Jun 2016 09:42:06 -0700 Subject: [PATCH 0081/1039] Check epoll is actually available. set GPR_LINUX_EPOLL only in GLIBC ver 2.9 and above --- include/grpc/impl/codegen/port_platform.h | 2 +- src/core/lib/iomgr/ev_epoll_linux.c | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/include/grpc/impl/codegen/port_platform.h b/include/grpc/impl/codegen/port_platform.h index 7a6ec53fb4c..affef9e66b6 100644 --- a/include/grpc/impl/codegen/port_platform.h +++ b/include/grpc/impl/codegen/port_platform.h @@ -189,7 +189,6 @@ #define GPR_GCC_ATOMIC 1 #define GPR_GCC_TLS 1 #define GPR_LINUX 1 -#define GPR_LINUX_EPOLL 1 #define GPR_LINUX_LOG #define GPR_LINUX_MULTIPOLL_WITH_EPOLL 1 #define GPR_POSIX_WAKEUP_FD 1 @@ -201,6 +200,7 @@ #ifdef __GLIBC_PREREQ #if __GLIBC_PREREQ(2, 9) #define GPR_LINUX_EVENTFD 1 +#define GPR_LINUX_EPOLL 1 #endif #if __GLIBC_PREREQ(2, 10) #define GPR_LINUX_SOCKETUTILS 1 diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 66bbae52b28..d2d5d2852b9 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -1481,7 +1481,26 @@ static const grpc_event_engine_vtable vtable = { .shutdown_engine = shutdown_engine, }; +/* It is possible that GLIBC has epoll but the underlying kernel doesn't. + * Create a dummy epoll_fd to make sure epoll support is available */ +static bool is_epoll_available() { + int fd = epoll_create1(EPOLL_CLOEXEC); + if (fd < 0) { + gpr_log( + GPR_ERROR, + "epoll_create1 failed with error: %d. Not using epoll polling engine", + fd); + return false; + } + close(fd); + return true; +} + const grpc_event_engine_vtable *grpc_init_epoll_linux(void) { + if (!is_epoll_available()) { + return NULL; + } + fd_global_init(); pollset_global_init(); polling_island_global_init(); From c7be7c688829281b543428ec22029d4d09bd2a9c Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 9 Jun 2016 17:08:50 -0700 Subject: [PATCH 0082/1039] Add an API at the core level to disable signals or use a different signal number --- include/grpc/grpc_posix.h | 8 +++++ src/core/lib/iomgr/ev_epoll_linux.c | 45 ++++++++++++++++++++++------- 2 files changed, 43 insertions(+), 10 deletions(-) diff --git a/include/grpc/grpc_posix.h b/include/grpc/grpc_posix.h index 9742b83374a..5e89ae3b1ee 100644 --- a/include/grpc/grpc_posix.h +++ b/include/grpc/grpc_posix.h @@ -63,6 +63,14 @@ GRPCAPI void grpc_server_add_insecure_channel_from_fd(grpc_server *server, grpc_completion_queue *cq, int fd); +/** GRPC Core POSIX library may internally use signals to optimize some work. + The library uses (SIGRTMIN + 2) signal by default. Use this API to instruct + the library to use a different signal i.e 'signum' instead. + Note: + - To prevent GRPC library from using any signals, pass a 'signum' of -1 + - This API is optional but if called, it MUST be called before grpc_init() */ +GRPCAPI void grpc_use_signal(int signum); + #ifdef __cplusplus } #endif diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index d2d5d2852b9..7e01ac144f2 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -31,6 +31,7 @@ * */ +#include #include #ifdef GPR_LINUX_EPOLL @@ -58,9 +59,26 @@ #include "src/core/lib/profiling/timers.h" #include "src/core/lib/support/block_annotate.h" -struct polling_island; +static int grpc_wakeup_signal = -1; +static bool is_grpc_wakeup_signal_initialized = false; + +/* Implements the function defined in grpc_posix.h. This function might be + * called before even calling grpc_init() to set either a different signal to + * use. If signum == -1, then the use of signals is disabled */ +void grpc_use_signal(int signum) { + grpc_wakeup_signal = signum; + is_grpc_wakeup_signal_initialized = true; -static int grpc_poller_kick_signum; + if (grpc_wakeup_signal < 0) { + gpr_log(GPR_INFO, + "Use of signals is disabled. Epoll engine will not be used"); + } else { + gpr_log(GPR_INFO, "epoll engine will be using signal: %d", + grpc_wakeup_signal); + } +} + +struct polling_island; /******************************************************************************* * Fd Declarations @@ -854,10 +872,7 @@ static void sig_handler(int sig_num) { #endif } -static void poller_kick_init() { - grpc_poller_kick_signum = SIGRTMIN + 2; - signal(grpc_poller_kick_signum, sig_handler); -} +static void poller_kick_init() { signal(grpc_wakeup_signal, sig_handler); } /* Global state management */ static void pollset_global_init(void) { @@ -874,7 +889,7 @@ static void pollset_global_shutdown(void) { } static void pollset_worker_kick(grpc_pollset_worker *worker) { - pthread_kill(worker->pt_id, grpc_poller_kick_signum); + pthread_kill(worker->pt_id, grpc_wakeup_signal); } /* Return 1 if the pollset has active threads in pollset_work (pollset must @@ -1214,9 +1229,9 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, pollset->kicked_without_pollers = 0; } else if (!pollset->shutting_down) { sigemptyset(&new_mask); - sigaddset(&new_mask, grpc_poller_kick_signum); + sigaddset(&new_mask, grpc_wakeup_signal); pthread_sigmask(SIG_BLOCK, &new_mask, &orig_mask); - sigdelset(&orig_mask, grpc_poller_kick_signum); + sigdelset(&orig_mask, grpc_wakeup_signal); push_front_worker(pollset, &worker); @@ -1497,19 +1512,29 @@ static bool is_epoll_available() { } const grpc_event_engine_vtable *grpc_init_epoll_linux(void) { + /* If use of signals is disabled, we cannot use epoll engine*/ + if (is_grpc_wakeup_signal_initialized && grpc_wakeup_signal < 0) { + return NULL; + } + if (!is_epoll_available()) { return NULL; } + if (!is_grpc_wakeup_signal_initialized) { + grpc_use_signal(SIGRTMIN + 2); + } + fd_global_init(); pollset_global_init(); polling_island_global_init(); return &vtable; } -#else /* defined(GPR_LINUX_EPOLL) */ +#else /* defined(GPR_LINUX_EPOLL) */ /* If GPR_LINUX_EPOLL is not defined, it means epoll is not available. Return * NULL */ const grpc_event_engine_vtable *grpc_init_epoll_linux(void) { return NULL; } +void grpc_use_signal(int signum) {} #endif /* !defined(GPR_LINUX_EPOLL) */ From 492fd961824447e7f12974893dd9bbf37291e5cc Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 10 Jun 2016 09:03:34 -0700 Subject: [PATCH 0083/1039] generate_projects.sh after adding grpc_use_signal() API --- BUILD | 2 ++ Makefile | 1 + grpc.def | 1 + src/python/grpcio/grpc/_cython/imports.generated.c | 2 ++ src/python/grpcio/grpc/_cython/imports.generated.h | 3 +++ src/ruby/ext/grpc/rb_grpc_imports.generated.c | 2 ++ src/ruby/ext/grpc/rb_grpc_imports.generated.h | 3 +++ 7 files changed, 14 insertions(+) diff --git a/BUILD b/BUILD index 68b066d5593..43855751cbc 100644 --- a/BUILD +++ b/BUILD @@ -556,6 +556,7 @@ cc_library( "src/core/lib/iomgr/closure.h", "src/core/lib/iomgr/endpoint.h", "src/core/lib/iomgr/endpoint_pair.h", + "src/core/lib/iomgr/ev_epoll_linux.h", "src/core/lib/iomgr/ev_poll_and_epoll_posix.h", "src/core/lib/iomgr/ev_poll_posix.h", "src/core/lib/iomgr/ev_posix.h", @@ -693,6 +694,7 @@ cc_library( "src/core/lib/iomgr/endpoint.c", "src/core/lib/iomgr/endpoint_pair_posix.c", "src/core/lib/iomgr/endpoint_pair_windows.c", + "src/core/lib/iomgr/ev_epoll_linux.c", "src/core/lib/iomgr/ev_poll_and_epoll_posix.c", "src/core/lib/iomgr/ev_poll_posix.c", "src/core/lib/iomgr/ev_posix.c", diff --git a/Makefile b/Makefile index 620299ed15b..4d8b060760a 100644 --- a/Makefile +++ b/Makefile @@ -2764,6 +2764,7 @@ LIBGRPC_CRONET_SRC = \ src/core/lib/iomgr/endpoint.c \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ + src/core/lib/iomgr/ev_epoll_linux.c \ src/core/lib/iomgr/ev_poll_and_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ diff --git a/grpc.def b/grpc.def index 0046028949e..9f70ce61a30 100644 --- a/grpc.def +++ b/grpc.def @@ -90,6 +90,7 @@ EXPORTS grpc_call_error_to_string grpc_insecure_channel_create_from_fd grpc_server_add_insecure_channel_from_fd + grpc_use_signal grpc_auth_property_iterator_next grpc_auth_context_property_iterator grpc_auth_context_peer_identity diff --git a/src/python/grpcio/grpc/_cython/imports.generated.c b/src/python/grpcio/grpc/_cython/imports.generated.c index 5c49f6cf3e9..a025b99047d 100644 --- a/src/python/grpcio/grpc/_cython/imports.generated.c +++ b/src/python/grpcio/grpc/_cython/imports.generated.c @@ -128,6 +128,7 @@ grpc_is_binary_header_type grpc_is_binary_header_import; grpc_call_error_to_string_type grpc_call_error_to_string_import; grpc_insecure_channel_create_from_fd_type grpc_insecure_channel_create_from_fd_import; grpc_server_add_insecure_channel_from_fd_type grpc_server_add_insecure_channel_from_fd_import; +grpc_use_signal_type grpc_use_signal_import; grpc_auth_property_iterator_next_type grpc_auth_property_iterator_next_import; grpc_auth_context_property_iterator_type grpc_auth_context_property_iterator_import; grpc_auth_context_peer_identity_type grpc_auth_context_peer_identity_import; @@ -401,6 +402,7 @@ void pygrpc_load_imports(HMODULE library) { grpc_call_error_to_string_import = (grpc_call_error_to_string_type) GetProcAddress(library, "grpc_call_error_to_string"); grpc_insecure_channel_create_from_fd_import = (grpc_insecure_channel_create_from_fd_type) GetProcAddress(library, "grpc_insecure_channel_create_from_fd"); grpc_server_add_insecure_channel_from_fd_import = (grpc_server_add_insecure_channel_from_fd_type) GetProcAddress(library, "grpc_server_add_insecure_channel_from_fd"); + grpc_use_signal_import = (grpc_use_signal_type) GetProcAddress(library, "grpc_use_signal"); grpc_auth_property_iterator_next_import = (grpc_auth_property_iterator_next_type) GetProcAddress(library, "grpc_auth_property_iterator_next"); grpc_auth_context_property_iterator_import = (grpc_auth_context_property_iterator_type) GetProcAddress(library, "grpc_auth_context_property_iterator"); grpc_auth_context_peer_identity_import = (grpc_auth_context_peer_identity_type) GetProcAddress(library, "grpc_auth_context_peer_identity"); diff --git a/src/python/grpcio/grpc/_cython/imports.generated.h b/src/python/grpcio/grpc/_cython/imports.generated.h index 16bb5cdfab6..5bdbbce89b5 100644 --- a/src/python/grpcio/grpc/_cython/imports.generated.h +++ b/src/python/grpcio/grpc/_cython/imports.generated.h @@ -335,6 +335,9 @@ extern grpc_insecure_channel_create_from_fd_type grpc_insecure_channel_create_fr typedef void(*grpc_server_add_insecure_channel_from_fd_type)(grpc_server *server, grpc_completion_queue *cq, int fd); extern grpc_server_add_insecure_channel_from_fd_type grpc_server_add_insecure_channel_from_fd_import; #define grpc_server_add_insecure_channel_from_fd grpc_server_add_insecure_channel_from_fd_import +typedef void(*grpc_use_signal_type)(int signum); +extern grpc_use_signal_type grpc_use_signal_import; +#define grpc_use_signal grpc_use_signal_import typedef const grpc_auth_property *(*grpc_auth_property_iterator_next_type)(grpc_auth_property_iterator *it); extern grpc_auth_property_iterator_next_type grpc_auth_property_iterator_next_import; #define grpc_auth_property_iterator_next grpc_auth_property_iterator_next_import diff --git a/src/ruby/ext/grpc/rb_grpc_imports.generated.c b/src/ruby/ext/grpc/rb_grpc_imports.generated.c index c13d1a00d74..8e24ef38ed5 100644 --- a/src/ruby/ext/grpc/rb_grpc_imports.generated.c +++ b/src/ruby/ext/grpc/rb_grpc_imports.generated.c @@ -128,6 +128,7 @@ grpc_is_binary_header_type grpc_is_binary_header_import; grpc_call_error_to_string_type grpc_call_error_to_string_import; grpc_insecure_channel_create_from_fd_type grpc_insecure_channel_create_from_fd_import; grpc_server_add_insecure_channel_from_fd_type grpc_server_add_insecure_channel_from_fd_import; +grpc_use_signal_type grpc_use_signal_import; grpc_auth_property_iterator_next_type grpc_auth_property_iterator_next_import; grpc_auth_context_property_iterator_type grpc_auth_context_property_iterator_import; grpc_auth_context_peer_identity_type grpc_auth_context_peer_identity_import; @@ -397,6 +398,7 @@ void grpc_rb_load_imports(HMODULE library) { grpc_call_error_to_string_import = (grpc_call_error_to_string_type) GetProcAddress(library, "grpc_call_error_to_string"); grpc_insecure_channel_create_from_fd_import = (grpc_insecure_channel_create_from_fd_type) GetProcAddress(library, "grpc_insecure_channel_create_from_fd"); grpc_server_add_insecure_channel_from_fd_import = (grpc_server_add_insecure_channel_from_fd_type) GetProcAddress(library, "grpc_server_add_insecure_channel_from_fd"); + grpc_use_signal_import = (grpc_use_signal_type) GetProcAddress(library, "grpc_use_signal"); grpc_auth_property_iterator_next_import = (grpc_auth_property_iterator_next_type) GetProcAddress(library, "grpc_auth_property_iterator_next"); grpc_auth_context_property_iterator_import = (grpc_auth_context_property_iterator_type) GetProcAddress(library, "grpc_auth_context_property_iterator"); grpc_auth_context_peer_identity_import = (grpc_auth_context_peer_identity_type) GetProcAddress(library, "grpc_auth_context_peer_identity"); diff --git a/src/ruby/ext/grpc/rb_grpc_imports.generated.h b/src/ruby/ext/grpc/rb_grpc_imports.generated.h index 9c86a3690c5..d6e2a367999 100644 --- a/src/ruby/ext/grpc/rb_grpc_imports.generated.h +++ b/src/ruby/ext/grpc/rb_grpc_imports.generated.h @@ -335,6 +335,9 @@ extern grpc_insecure_channel_create_from_fd_type grpc_insecure_channel_create_fr typedef void(*grpc_server_add_insecure_channel_from_fd_type)(grpc_server *server, grpc_completion_queue *cq, int fd); extern grpc_server_add_insecure_channel_from_fd_type grpc_server_add_insecure_channel_from_fd_import; #define grpc_server_add_insecure_channel_from_fd grpc_server_add_insecure_channel_from_fd_import +typedef void(*grpc_use_signal_type)(int signum); +extern grpc_use_signal_type grpc_use_signal_import; +#define grpc_use_signal grpc_use_signal_import typedef const grpc_auth_property *(*grpc_auth_property_iterator_next_type)(grpc_auth_property_iterator *it); extern grpc_auth_property_iterator_next_type grpc_auth_property_iterator_next_import; #define grpc_auth_property_iterator_next grpc_auth_property_iterator_next_import From eb16b3dc3cd579931d730ba3fef1f7008f649003 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 10 Jun 2016 23:06:25 -0700 Subject: [PATCH 0084/1039] Fix ref counting bug --- src/core/lib/iomgr/ev_epoll_linux.c | 20 ++------------------ 1 file changed, 2 insertions(+), 18 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 7e01ac144f2..617afad1975 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -1127,20 +1127,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, /* Update the pollset->polling_island */ pollset->polling_island = pi; -#ifdef GRPC_EPOLL_DEBUG - if (pollset->polling_island->fd_cnt == 0) { - gpr_log(GPR_DEBUG, "pollset_work_and_unlock: epoll_fd: %d, No other fds", - epoll_fd); - } - for (size_t i = 0; i < pollset->polling_island->fd_cnt; i++) { - gpr_log(GPR_DEBUG, - "pollset_work_and_unlock: epoll_fd: %d, fd_count: %d, fd[%d]: %d", - epoll_fd, pollset->polling_island->fd_cnt, i, - pollset->polling_island->fds[i]->fd); - } -#endif - gpr_mu_unlock(&pollset->polling_island->mu); - + polling_island_unref_and_unlock(pollset->polling_island, 0); /* Keep the ref*/ gpr_mu_unlock(&pollset->pi_mu); gpr_mu_unlock(&pollset->mu); @@ -1190,10 +1177,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, * gets updated whenever the underlying polling island is merged with another * island and while we are doing epoll_wait() above, the polling island may * have been merged */ - - /* TODO (sreek) - Change the ref count on polling island to gpr_atm so that - * we do not have to do this here */ - gpr_mu_lock(&pi->mu); + polling_island_update_and_lock(pi, 1, 0); /* No new ref added */ polling_island_unref_and_unlock(pi, 1); GPR_TIMER_END("pollset_work_and_unlock", 0); From 58e589644403b10afb31ffd45befabe13b652db8 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Mon, 13 Jun 2016 00:52:56 -0700 Subject: [PATCH 0085/1039] Fix bad merge --- src/core/lib/iomgr/ev_epoll_linux.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 617afad1975..a8a874cd4b0 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -550,14 +550,14 @@ polling_island *polling_island_merge(polling_island *p, polling_island *q) { /* Wakeup all the pollers (if any) on p so that they can pickup this change */ polling_island_add_wakeup_fd_locked(p, &polling_island_wakeup_fd); + p->merged_to = q; + /* - The merged polling island (i.e q) inherits all the ref counts of the island merging with it (i.e p) - The island p will lose a ref count */ q->ref_cnt += p->ref_cnt; - p->ref_cnt--; - - gpr_mu_unlock(&p->mu); - gpr_mu_unlock(&q->mu); + polling_island_unref_and_unlock(p, 1); /* Decrement refcount */ + polling_island_unref_and_unlock(q, 0); /* Just Unlock. Don't decrement ref */ return q; } @@ -1110,7 +1110,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, Acquire the following locks: - pollset->mu (which we already have) - pollset->pi_mu - - pollset->polling_island->mu */ + - pollset->polling_island->mu (call polling_island_update_and_lock())*/ gpr_mu_lock(&pollset->pi_mu); pi = pollset->polling_island; @@ -1144,8 +1144,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, } } - int i; - for (i = 0; i < ep_rv; ++i) { + for (int i = 0; i < ep_rv; ++i) { void *data_ptr = ep_ev[i].data.ptr; if (data_ptr == &grpc_global_wakeup_fd) { grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); @@ -1177,7 +1176,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, * gets updated whenever the underlying polling island is merged with another * island and while we are doing epoll_wait() above, the polling island may * have been merged */ - polling_island_update_and_lock(pi, 1, 0); /* No new ref added */ + pi = polling_island_update_and_lock(pi, 1, 0); /* No new ref added */ polling_island_unref_and_unlock(pi, 1); GPR_TIMER_END("pollset_work_and_unlock", 0); From 5ea4a99d8ccb7f1bc19b78e8cd86b770f1ddcc8e Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Mon, 13 Jun 2016 10:36:41 -0700 Subject: [PATCH 0086/1039] Finished removing CompletionQueue from Ruby API, made some changes for clarity --- src/ruby/ext/grpc/rb_call.c | 11 +- src/ruby/ext/grpc/rb_call.h | 2 +- src/ruby/ext/grpc/rb_channel.c | 23 +--- src/ruby/ext/grpc/rb_completion_queue.c | 7 +- src/ruby/ext/grpc/rb_completion_queue.h | 2 +- src/ruby/ext/grpc/rb_server.c | 32 +++-- src/ruby/lib/grpc/generic/active_call.rb | 70 ++++------ src/ruby/lib/grpc/generic/bidi_call.rb | 34 ++--- src/ruby/lib/grpc/generic/client_stub.rb | 11 +- src/ruby/lib/grpc/generic/rpc_server.rb | 40 ++---- src/ruby/lib/grpc/generic/service.rb | 2 +- src/ruby/spec/call_spec.rb | 3 +- src/ruby/spec/channel_spec.rb | 5 +- src/ruby/spec/client_server_spec.rb | 90 +++++-------- src/ruby/spec/completion_queue_spec.rb | 42 ------ src/ruby/spec/generic/active_call_spec.rb | 148 ++++++++++------------ src/ruby/spec/generic/client_stub_spec.rb | 75 ++++------- src/ruby/spec/generic/rpc_server_spec.rb | 34 +---- src/ruby/spec/pb/health/checker_spec.rb | 2 - src/ruby/spec/server_spec.rb | 44 +++---- 20 files changed, 235 insertions(+), 442 deletions(-) delete mode 100644 src/ruby/spec/completion_queue_spec.rb diff --git a/src/ruby/ext/grpc/rb_call.c b/src/ruby/ext/grpc/rb_call.c index f2c567c7da7..c1f5075b52a 100644 --- a/src/ruby/ext/grpc/rb_call.c +++ b/src/ruby/ext/grpc/rb_call.c @@ -94,7 +94,6 @@ typedef struct grpc_rb_call { } grpc_rb_call; static void destroy_call(grpc_rb_call *call) { - call = (grpc_rb_call *)p; grpc_call_destroy(call->wrapped); grpc_rb_completion_queue_destroy(call->queue); } @@ -783,8 +782,11 @@ static VALUE grpc_rb_call_run_batch(VALUE self, VALUE ops_hash) { grpc_call_error_detail_of(err), err); return Qnil; } - ev = grpc_rb_completion_queue_pluck(call->queue, tag, gpr_inf_future, NULL); - + ev = rb_completion_queue_pluck(call->queue, tag, + gpr_inf_future(GPR_CLOCK_REALTIME), NULL); + if (!ev.success) { + rb_raise(grpc_rb_eCallError, "call#run_batch failed somehow"); + } /* Build and return the BatchResult struct result, if there is an error, it's reflected in the status */ result = grpc_run_batch_stack_build_result(&st); @@ -943,10 +945,11 @@ grpc_call *grpc_rb_get_wrapped_call(VALUE v) { /* Obtains the wrapped object for a given call */ VALUE grpc_rb_wrap_call(grpc_call *c, grpc_completion_queue *q) { + grpc_rb_call *wrapper; if (c == NULL || q == NULL) { return Qnil; } - grpc_rb_call *wrapper = ALLOC(grpc_rb_call); + wrapper = ALLOC(grpc_rb_call); wrapper->wrapped = c; wrapper->queue = q; return TypedData_Wrap_Struct(grpc_rb_cCall, &grpc_call_data_type, wrapper); diff --git a/src/ruby/ext/grpc/rb_call.h b/src/ruby/ext/grpc/rb_call.h index 24adb3477ba..56becdc5a4e 100644 --- a/src/ruby/ext/grpc/rb_call.h +++ b/src/ruby/ext/grpc/rb_call.h @@ -42,7 +42,7 @@ grpc_call* grpc_rb_get_wrapped_call(VALUE v); /* Gets the VALUE corresponding to given grpc_call. */ -VALUE grpc_rb_wrap_call(grpc_call* c); +VALUE grpc_rb_wrap_call(grpc_call *c, grpc_completion_queue *q); /* Provides the details of an call error */ const char* grpc_call_error_detail_of(grpc_call_error err); diff --git a/src/ruby/ext/grpc/rb_channel.c b/src/ruby/ext/grpc/rb_channel.c index 9d29b96a35b..f5190b85f74 100644 --- a/src/ruby/ext/grpc/rb_channel.c +++ b/src/ruby/ext/grpc/rb_channel.c @@ -55,11 +55,6 @@ static ID id_channel; * GCed before the channel */ static ID id_target; -/* id_cqueue is the name of the hidden ivar that preserves a reference to the - * completion queue used to create the call, preserved so that it does not get - * GCed before the channel */ -static ID id_cqueue; - /* id_insecure_channel is used to indicate that a channel is insecure */ static VALUE id_insecure_channel; @@ -233,10 +228,9 @@ static VALUE grpc_rb_channel_watch_connectivity_state(VALUE self, /* Create a call given a grpc_channel, in order to call method. The request is not sent until grpc_call_invoke is called. */ -static VALUE grpc_rb_channel_create_call(VALUE self, VALUE cqueue, - VALUE parent, VALUE mask, - VALUE method, VALUE host, - VALUE deadline) { +static VALUE grpc_rb_channel_create_call(VALUE self, VALUE parent, + VALUE mask, VALUE method, + VALUE host, VALUE deadline) { VALUE res = Qnil; grpc_rb_channel *wrapper = NULL; grpc_call *call = NULL; @@ -256,7 +250,7 @@ static VALUE grpc_rb_channel_create_call(VALUE self, VALUE cqueue, parent_call = grpc_rb_get_wrapped_call(parent); } - cq = grpc_rb_get_wrapped_completion_queue(cqueue); + cq = grpc_completion_queue_create(NULL); TypedData_Get_Struct(self, grpc_rb_channel, &grpc_channel_data_type, wrapper); ch = wrapper->wrapped; if (ch == NULL) { @@ -273,15 +267,11 @@ static VALUE grpc_rb_channel_create_call(VALUE self, VALUE cqueue, method_chars); return Qnil; } - res = grpc_rb_wrap_call(call); + res = grpc_rb_wrap_call(call, cq); /* Make this channel an instance attribute of the call so that it is not GCed * before the call. */ rb_ivar_set(res, id_channel, self); - - /* Make the completion queue an instance attribute of the call so that it is - * not GCed before the call. */ - rb_ivar_set(res, id_cqueue, cqueue); return res; } @@ -368,13 +358,12 @@ void Init_grpc_channel() { rb_define_method(grpc_rb_cChannel, "watch_connectivity_state", grpc_rb_channel_watch_connectivity_state, 4); rb_define_method(grpc_rb_cChannel, "create_call", - grpc_rb_channel_create_call, 6); + grpc_rb_channel_create_call, 5); rb_define_method(grpc_rb_cChannel, "target", grpc_rb_channel_get_target, 0); rb_define_method(grpc_rb_cChannel, "destroy", grpc_rb_channel_destroy, 0); rb_define_alias(grpc_rb_cChannel, "close", "destroy"); id_channel = rb_intern("__channel"); - id_cqueue = rb_intern("__cqueue"); id_target = rb_intern("__target"); rb_define_const(grpc_rb_cChannel, "SSL_TARGET", ID2SYM(rb_intern(GRPC_SSL_TARGET_NAME_OVERRIDE_ARG))); diff --git a/src/ruby/ext/grpc/rb_completion_queue.c b/src/ruby/ext/grpc/rb_completion_queue.c index 1ac2ef2f335..1a69a175c4c 100644 --- a/src/ruby/ext/grpc/rb_completion_queue.c +++ b/src/ruby/ext/grpc/rb_completion_queue.c @@ -125,11 +125,6 @@ static void grpc_rb_completion_queue_shutdown_drain(grpc_completion_queue *cq) { /* Helper function to free a completion queue. */ void grpc_rb_completion_queue_destroy(grpc_completion_queue *cq) { - grpc_completion_queue *cq = NULL; - if (p == NULL) { - return; - } - cq = (grpc_completion_queue *)p; grpc_rb_completion_queue_shutdown_drain(cq); grpc_completion_queue_destroy(cq); } @@ -141,7 +136,7 @@ static void unblock_func(void *param) { /* Does the same thing as grpc_completion_queue_pluck, while properly releasing the GVL and handling interrupts */ -grpc_event rb_completion_queue_pluck(grpc_completion_queue queue, void *tag, +grpc_event rb_completion_queue_pluck(grpc_completion_queue *queue, void *tag, gpr_timespec deadline, void *reserved) { next_call_stack next_call; MEMZERO(&next_call, next_call_stack, 1); diff --git a/src/ruby/ext/grpc/rb_completion_queue.h b/src/ruby/ext/grpc/rb_completion_queue.h index 3543e86275c..9f8f6aa5fff 100644 --- a/src/ruby/ext/grpc/rb_completion_queue.h +++ b/src/ruby/ext/grpc/rb_completion_queue.h @@ -48,7 +48,7 @@ void grpc_rb_completion_queue_destroy(grpc_completion_queue *cq); * * This avoids having code that holds the GIL repeated at multiple sites. */ -grpc_event rb_completion_queue_pluck(grpc_completion_queue queue, void *tag, +grpc_event rb_completion_queue_pluck(grpc_completion_queue *queue, void *tag, gpr_timespec deadline, void *reserved); #endif /* GRPC_RB_COMPLETION_QUEUE_H_ */ diff --git a/src/ruby/ext/grpc/rb_server.c b/src/ruby/ext/grpc/rb_server.c index cf430495c8b..1ed2c35da34 100644 --- a/src/ruby/ext/grpc/rb_server.c +++ b/src/ruby/ext/grpc/rb_server.c @@ -70,7 +70,8 @@ static void destroy_server(grpc_rb_server *server, gpr_timespec deadline) { ev = rb_completion_queue_pluck(server->queue, NULL, deadline, NULL); if (ev.type == GRPC_QUEUE_TIMEOUT) { grpc_server_cancel_all_calls(server->wrapped); - rb_completion_queue_pluck(server->queue, NULL, gpr_inf_future, NULL); + rb_completion_queue_pluck(server->queue, NULL, + gpr_inf_future(GPR_CLOCK_REALTIME), NULL); } grpc_server_destroy(server->wrapped); grpc_rb_completion_queue_destroy(server->queue); @@ -82,13 +83,17 @@ static void destroy_server(grpc_rb_server *server, gpr_timespec deadline) { /* Destroys server instances. */ static void grpc_rb_server_free(void *p) { grpc_rb_server *svr = NULL; + gpr_timespec deadline; if (p == NULL) { return; }; svr = (grpc_rb_server *)p; - // TODO(murgatroid99): Maybe don't wait forever for the server to shutdown - destroy_server(svr, gpr_inf_future); + deadline = gpr_time_add( + gpr_now(GPR_CLOCK_REALTIME), + gpr_time_from_seconds(2, GPR_TIMESPAN)); + + destroy_server(svr, deadline); xfree(p); } @@ -154,9 +159,6 @@ static VALUE grpc_rb_server_init(VALUE self, VALUE channel_args) { wrapper->wrapped = srv; wrapper->queue = cq; - /* Add the cq as the server's mark object. This ensures the ruby cq can't be - GCed before the server */ - wrapper->mark = cqueue; return self; } @@ -218,7 +220,7 @@ static VALUE grpc_rb_server_request_call(VALUE self) { } ev = rb_completion_queue_pluck(s->queue, tag, - gpr_inf_future, NULL); + gpr_inf_future(GPR_CLOCK_REALTIME), NULL); if (!ev.success) { grpc_request_call_stack_cleanup(&st); rb_raise(grpc_rb_eCallError, "request_call completion failed"); @@ -251,27 +253,23 @@ static VALUE grpc_rb_server_start(VALUE self) { /* call-seq: - cq = CompletionQueue.new - server = Server.new(cq, {'arg1': 'value1'}) + server = Server.new({'arg1': 'value1'}) ... // do stuff with server ... ... // to shutdown the server - server.destroy(cq) + server.destroy() ... // to shutdown the server with a timeout - server.destroy(cq, timeout) + server.destroy(timeout) Destroys server instances. */ static VALUE grpc_rb_server_destroy(int argc, VALUE *argv, VALUE self) { - VALUE cqueue = Qnil; VALUE timeout = Qnil; - grpc_completion_queue *cq = NULL; - grpc_event ev; + gpr_timespec deadline; grpc_rb_server *s = NULL; - /* "11" == 1 mandatory args, 1 (timeout) is optional */ - rb_scan_args(argc, argv, "11", &cqueue, &timeout); - cq = grpc_rb_get_wrapped_completion_queue(cqueue); + /* "01" == 0 mandatory args, 1 (timeout) is optional */ + rb_scan_args(argc, argv, "01", &timeout); TypedData_Get_Struct(self, grpc_rb_server, &grpc_rb_server_data_type, s); if (TYPE(timeout) == T_NIL) { deadline = gpr_inf_future(GPR_CLOCK_REALTIME); diff --git a/src/ruby/lib/grpc/generic/active_call.rb b/src/ruby/lib/grpc/generic/active_call.rb index b03ddbc193c..18126c44323 100644 --- a/src/ruby/lib/grpc/generic/active_call.rb +++ b/src/ruby/lib/grpc/generic/active_call.rb @@ -75,17 +75,10 @@ module GRPC # if a keyword value is a list, multiple metadata for it's key are sent # # @param call [Call] a call on which to start and invocation - # @param q [CompletionQueue] the completion queue # @param metadata [Hash] the metadata - def self.client_invoke(call, q, metadata = {}) + def self.client_invoke(call, metadata = {}) fail(TypeError, '!Core::Call') unless call.is_a? Core::Call - unless q.is_a? Core::CompletionQueue - fail(TypeError, '!Core::CompletionQueue') - end - metadata_tag = Object.new - call.run_batch(q, metadata_tag, INFINITE_FUTURE, - SEND_INITIAL_METADATA => metadata) - metadata_tag + call.run_batch(SEND_INITIAL_METADATA => metadata) end # Creates an ActiveCall. @@ -102,26 +95,21 @@ module GRPC # deadline is the absolute deadline for the call. # # @param call [Call] the call used by the ActiveCall - # @param q [CompletionQueue] the completion queue used to accept - # the call. This queue will be closed on call completion. # @param marshal [Function] f(obj)->string that marshal requests # @param unmarshal [Function] f(string)->obj that unmarshals responses # @param deadline [Fixnum] the deadline for the call to complete - # @param metadata_tag [Object] the object use obtain metadata for clients - # @param started [true|false] indicates if the call has begun - def initialize(call, q, marshal, unmarshal, deadline, started: true, - metadata_tag: nil) + # @param started [true|false] indicates that metadata was sent + # @param metadata_received [true|false] indicates if metadata has already + # been received. Should always be true for server calls + def initialize(call, marshal, unmarshal, deadline, started: true, + metadata_received: false) fail(TypeError, '!Core::Call') unless call.is_a? Core::Call - unless q.is_a? Core::CompletionQueue - fail(TypeError, '!Core::CompletionQueue') - end @call = call - @cq = q @deadline = deadline @marshal = marshal - @started = started @unmarshal = unmarshal - @metadata_tag = metadata_tag + @metadata_received = metadata_received + @metadata_sent = started @op_notifier = nil end @@ -168,7 +156,7 @@ module GRPC SEND_CLOSE_FROM_CLIENT => nil } ops[RECV_STATUS_ON_CLIENT] = nil if assert_finished - batch_result = @call.run_batch(@cq, self, INFINITE_FUTURE, ops) + batch_result = @call.run_batch(ops) return unless assert_finished @call.status = batch_result.status op_is_done @@ -179,8 +167,7 @@ module GRPC # # It blocks until the remote endpoint acknowledges by sending a status. def finished - batch_result = @call.run_batch(@cq, self, INFINITE_FUTURE, - RECV_STATUS_ON_CLIENT => nil) + batch_result = @call.run_batch(RECV_STATUS_ON_CLIENT => nil) unless batch_result.status.nil? if @call.metadata.nil? @call.metadata = batch_result.status.metadata @@ -192,7 +179,6 @@ module GRPC op_is_done batch_result.check_status @call.close - @cq.close end # remote_send sends a request to the remote endpoint. @@ -203,9 +189,10 @@ module GRPC # @param marshalled [false, true] indicates if the object is already # marshalled. def remote_send(req, marshalled = false) + # TODO(murgatroid99): ensure metadata was sent GRPC.logger.debug("sending #{req}, marshalled? #{marshalled}") payload = marshalled ? req : @marshal.call(req) - @call.run_batch(@cq, self, INFINITE_FUTURE, SEND_MESSAGE => payload) + @call.run_batch(SEND_MESSAGE => payload) end # send_status sends a status to the remote endpoint. @@ -222,7 +209,7 @@ module GRPC SEND_STATUS_FROM_SERVER => Struct::Status.new(code, details, metadata) } ops[RECV_CLOSE_ON_SERVER] = nil if assert_finished - @call.run_batch(@cq, self, INFINITE_FUTURE, ops) + @call.run_batch(ops) nil end @@ -234,11 +221,11 @@ module GRPC # raising BadStatus def remote_read ops = { RECV_MESSAGE => nil } - ops[RECV_INITIAL_METADATA] = nil unless @metadata_tag.nil? - batch_result = @call.run_batch(@cq, self, INFINITE_FUTURE, ops) - unless @metadata_tag.nil? + ops[RECV_INITIAL_METADATA] = nil unless @metadata_received + batch_result = @call.run_batch(ops) + unless @metadata_received @call.metadata = batch_result.metadata - @metadata_tag = nil + @metadata_received = true end GRPC.logger.debug("received req: #{batch_result}") unless batch_result.nil? || batch_result.message.nil? @@ -318,7 +305,7 @@ module GRPC # a list, multiple metadata for its key are sent # @return [Object] the response received from the server def request_response(req, metadata: {}) - start_call(metadata) unless @started + start_call(metadata) remote_send(req) writes_done(false) response = remote_read @@ -342,7 +329,7 @@ module GRPC # a list, multiple metadata for its key are sent # @return [Object] the response received from the server def client_streamer(requests, metadata: {}) - start_call(metadata) unless @started + start_call(metadata) requests.each { |r| remote_send(r) } writes_done(false) response = remote_read @@ -368,7 +355,7 @@ module GRPC # a list, multiple metadata for its key are sent # @return [Enumerator|nil] a response Enumerator def server_streamer(req, metadata: {}) - start_call(metadata) unless @started + start_call(metadata) remote_send(req) writes_done(false) replies = enum_for(:each_remote_read_then_finish) @@ -407,10 +394,9 @@ module GRPC # a list, multiple metadata for its key are sent # @return [Enumerator, nil] a response Enumerator def bidi_streamer(requests, metadata: {}, &blk) - start_call(metadata) unless @started - bd = BidiCall.new(@call, @cq, @marshal, @unmarshal, - metadata_tag: @metadata_tag) - @metadata_tag = nil # run_on_client ensures metadata is read + start_call(metadata) + bd = BidiCall.new(@call, @marshal, @unmarshal, + metadata_received: @metadata_received) bd.run_on_client(requests, @op_notifier, &blk) end @@ -426,7 +412,7 @@ module GRPC # # @param gen_each_reply [Proc] generates the BiDi stream replies def run_server_bidi(gen_each_reply) - bd = BidiCall.new(@call, @cq, @marshal, @unmarshal) + bd = BidiCall.new(@call, @marshal, @unmarshal) bd.run_on_server(gen_each_reply) end @@ -449,9 +435,9 @@ module GRPC # @param metadata [Hash] metadata to be sent to the server. If a value is # a list, multiple metadata for its key are sent def start_call(metadata = {}) - return if @started - @metadata_tag = ActiveCall.client_invoke(@call, @cq, metadata) - @started = true + return if @metadata_sent + @metadata_tag = ActiveCall.client_invoke(@call, metadata) + @metadata_sent = true end def self.view_class(*visible_methods) diff --git a/src/ruby/lib/grpc/generic/bidi_call.rb b/src/ruby/lib/grpc/generic/bidi_call.rb index 238f409a1d6..f4f9d1b3ddc 100644 --- a/src/ruby/lib/grpc/generic/bidi_call.rb +++ b/src/ruby/lib/grpc/generic/bidi_call.rb @@ -52,23 +52,18 @@ module GRPC # deadline is the absolute deadline for the call. # # @param call [Call] the call used by the ActiveCall - # @param q [CompletionQueue] the completion queue used to accept - # the call # @param marshal [Function] f(obj)->string that marshal requests # @param unmarshal [Function] f(string)->obj that unmarshals responses - # @param metadata_tag [Object] tag object used to collect metadata - def initialize(call, q, marshal, unmarshal, metadata_tag: nil) + # @param metadata_received [true|false] indicates if metadata has already + # been received. Should always be true for server calls + def initialize(call, marshal, unmarshal, metadata_received: false) fail(ArgumentError, 'not a call') unless call.is_a? Core::Call - unless q.is_a? Core::CompletionQueue - fail(ArgumentError, 'not a CompletionQueue') - end @call = call - @cq = q @marshal = marshal @op_notifier = nil # signals completion on clients @readq = Queue.new @unmarshal = unmarshal - @metadata_tag = metadata_tag + @metadata_received = metadata_received @reads_complete = false @writes_complete = false @complete = false @@ -124,7 +119,6 @@ module GRPC @done_mutex.synchronize do return unless @reads_complete && @writes_complete && !@complete @call.close - @cq.close @complete = true end end @@ -132,11 +126,11 @@ module GRPC # performs a read using @call.run_batch, ensures metadata is set up def read_using_run_batch ops = { RECV_MESSAGE => nil } - ops[RECV_INITIAL_METADATA] = nil unless @metadata_tag.nil? - batch_result = @call.run_batch(@cq, self, INFINITE_FUTURE, ops) - unless @metadata_tag.nil? + ops[RECV_INITIAL_METADATA] = nil unless @metadata_received + batch_result = @call.run_batch(ops) + unless @metadata_received @call.metadata = batch_result.metadata - @metadata_tag = nil + @metadata_received = true end batch_result end @@ -161,20 +155,18 @@ module GRPC def write_loop(requests, is_client: true) GRPC.logger.debug('bidi-write-loop: starting') - write_tag = Object.new count = 0 requests.each do |req| GRPC.logger.debug("bidi-write-loop: #{count}") count += 1 payload = @marshal.call(req) - @call.run_batch(@cq, write_tag, INFINITE_FUTURE, - SEND_MESSAGE => payload) + # Fails if status already received + @call.run_batch(SEND_MESSAGE => payload) end GRPC.logger.debug("bidi-write-loop: #{count} writes done") if is_client GRPC.logger.debug("bidi-write-loop: client sent #{count}, waiting") - @call.run_batch(@cq, write_tag, INFINITE_FUTURE, - SEND_CLOSE_FROM_CLIENT => nil) + @call.run_batch(SEND_CLOSE_FROM_CLIENT => nil) GRPC.logger.debug('bidi-write-loop: done') notify_done @writes_complete = true @@ -195,7 +187,6 @@ module GRPC Thread.new do GRPC.logger.debug('bidi-read-loop: starting') begin - read_tag = Object.new count = 0 # queue the initial read before beginning the loop loop do @@ -208,8 +199,7 @@ module GRPC GRPC.logger.debug("bidi-read-loop: null batch #{batch_result}") if is_client - batch_result = @call.run_batch(@cq, read_tag, INFINITE_FUTURE, - RECV_STATUS_ON_CLIENT => nil) + batch_result = @call.run_batch(RECV_STATUS_ON_CLIENT => nil) @call.status = batch_result.status batch_result.check_status GRPC.logger.debug("bidi-read-loop: done status #{@call.status}") diff --git a/src/ruby/lib/grpc/generic/client_stub.rb b/src/ruby/lib/grpc/generic/client_stub.rb index cddca13d173..9d6bd3bf590 100644 --- a/src/ruby/lib/grpc/generic/client_stub.rb +++ b/src/ruby/lib/grpc/generic/client_stub.rb @@ -90,19 +90,16 @@ module GRPC # when present, this is the default timeout used for calls # # @param host [String] the host the stub connects to - # @param q [Core::CompletionQueue] used to wait for events - now deprecated - # since each new active call gets its own separately # @param creds [Core::ChannelCredentials|Symbol] the channel credentials, or # :this_channel_is_insecure # @param channel_override [Core::Channel] a pre-created channel # @param timeout [Number] the default timeout to use in requests # @param channel_args [Hash] the channel arguments - def initialize(host, q, creds, + def initialize(host, creds, channel_override: nil, timeout: nil, propagate_mask: nil, channel_args: {}) - fail(TypeError, '!CompletionQueue') unless q.is_a?(Core::CompletionQueue) @ch = ClientStub.setup_channel(channel_override, host, creds, channel_args) alt_host = channel_args[Core::Channel::SSL_TARGET] @@ -441,15 +438,13 @@ module GRPC deadline = from_relative_time(@timeout) if deadline.nil? # Provide each new client call with its own completion queue - call_queue = Core::CompletionQueue.new - call = @ch.create_call(call_queue, - parent, # parent call + call = @ch.create_call(parent, # parent call @propagate_mask, # propagation options method, nil, # host use nil, deadline) call.set_credentials! credentials unless credentials.nil? - ActiveCall.new(call, call_queue, marshal, unmarshal, deadline, + ActiveCall.new(call, marshal, unmarshal, deadline, started: false) end end diff --git a/src/ruby/lib/grpc/generic/rpc_server.rb b/src/ruby/lib/grpc/generic/rpc_server.rb index ab7333d1337..c92a532a500 100644 --- a/src/ruby/lib/grpc/generic/rpc_server.rb +++ b/src/ruby/lib/grpc/generic/rpc_server.rb @@ -159,16 +159,6 @@ module GRPC # Signal check period is 0.25s SIGNAL_CHECK_PERIOD = 0.25 - # setup_cq is used by #initialize to constuct a Core::CompletionQueue from - # its arguments. - def self.setup_cq(alt_cq) - return Core::CompletionQueue.new if alt_cq.nil? - unless alt_cq.is_a? Core::CompletionQueue - fail(TypeError, '!CompletionQueue') - end - alt_cq - end - # setup_connect_md_proc is used by #initialize to validate the # connect_md_proc. def self.setup_connect_md_proc(a_proc) @@ -191,10 +181,6 @@ module GRPC # * pool_size: the size of the thread pool the server uses to run its # threads # - # * completion_queue_override: when supplied, this will be used as the - # completion_queue that the server uses to receive network events, - # otherwise its creates a new instance itself - # # * creds: [GRPC::Core::ServerCredentials] # the credentials used to secure the server # @@ -212,11 +198,9 @@ module GRPC def initialize(pool_size:DEFAULT_POOL_SIZE, max_waiting_requests:DEFAULT_MAX_WAITING_REQUESTS, poll_period:DEFAULT_POLL_PERIOD, - completion_queue_override:nil, connect_md_proc:nil, server_args:{}) @connect_md_proc = RpcServer.setup_connect_md_proc(connect_md_proc) - @cq = RpcServer.setup_cq(completion_queue_override) @max_waiting_requests = max_waiting_requests @poll_period = poll_period @pool_size = pool_size @@ -226,7 +210,7 @@ module GRPC # running_state can take 4 values: :not_started, :running, :stopping, and # :stopped. State transitions can only proceed in that order. @running_state = :not_started - @server = Core::Server.new(@cq, server_args) + @server = Core::Server.new(server_args) end # stops a running server @@ -240,7 +224,7 @@ module GRPC transition_running_state(:stopping) end deadline = from_relative_time(@poll_period) - @server.close(@cq, deadline) + @server.close(deadline) @pool.stop end @@ -355,7 +339,8 @@ module GRPC return an_rpc if @pool.jobs_waiting <= @max_waiting_requests GRPC.logger.warn("NOT AVAILABLE: too many jobs_waiting: #{an_rpc}") noop = proc { |x| x } - c = ActiveCall.new(an_rpc.call, an_rpc.cq, noop, noop, an_rpc.deadline) + c = ActiveCall.new(an_rpc.call, noop, noop, an_rpc.deadline, + metadata_received: true) c.send_status(GRPC::Core::StatusCodes::RESOURCE_EXHAUSTED, '') nil end @@ -366,7 +351,8 @@ module GRPC return an_rpc if rpc_descs.key?(mth) GRPC.logger.warn("UNIMPLEMENTED: #{an_rpc}") noop = proc { |x| x } - c = ActiveCall.new(an_rpc.call, an_rpc.cq, noop, noop, an_rpc.deadline) + c = ActiveCall.new(an_rpc.call, noop, noop, an_rpc.deadline, + metadata_received: true) c.send_status(GRPC::Core::StatusCodes::UNIMPLEMENTED, '') nil end @@ -374,11 +360,9 @@ module GRPC # handles calls to the server def loop_handle_server_calls fail 'not started' if running_state == :not_started - loop_tag = Object.new while running_state == :running begin - comp_queue = Core::CompletionQueue.new - an_rpc = @server.request_call(comp_queue, loop_tag, INFINITE_FUTURE) + an_rpc = @server.request_call break if (!an_rpc.nil?) && an_rpc.call.nil? active_call = new_active_server_call(an_rpc) unless active_call.nil? @@ -410,15 +394,13 @@ module GRPC return nil if an_rpc.nil? || an_rpc.call.nil? # allow the metadata to be accessed from the call - handle_call_tag = Object.new an_rpc.call.metadata = an_rpc.metadata # attaches md to call for handlers GRPC.logger.debug("call md is #{an_rpc.metadata}") connect_md = nil unless @connect_md_proc.nil? connect_md = @connect_md_proc.call(an_rpc.method, an_rpc.metadata) end - an_rpc.call.run_batch(an_rpc.cq, handle_call_tag, INFINITE_FUTURE, - SEND_INITIAL_METADATA => connect_md) + an_rpc.call.run_batch(SEND_INITIAL_METADATA => connect_md) return nil unless available?(an_rpc) return nil unless implemented?(an_rpc) @@ -426,9 +408,9 @@ module GRPC # Create the ActiveCall GRPC.logger.info("deadline is #{an_rpc.deadline}; (now=#{Time.now})") rpc_desc = rpc_descs[an_rpc.method.to_sym] - c = ActiveCall.new(an_rpc.call, an_rpc.cq, - rpc_desc.marshal_proc, rpc_desc.unmarshal_proc(:input), - an_rpc.deadline) + c = ActiveCall.new(an_rpc.call, rpc_desc.marshal_proc, + rpc_desc.unmarshal_proc(:input), an_rpc.deadline, + metadata_received: true) mth = an_rpc.method.to_sym [c, mth] end diff --git a/src/ruby/lib/grpc/generic/service.rb b/src/ruby/lib/grpc/generic/service.rb index f30242ee801..7cb9f1cc99d 100644 --- a/src/ruby/lib/grpc/generic/service.rb +++ b/src/ruby/lib/grpc/generic/service.rb @@ -168,7 +168,7 @@ module GRPC # @param kw [KeywordArgs] the channel arguments, plus any optional # args for configuring the client's channel def initialize(host, creds, **kw) - super(host, Core::CompletionQueue.new, creds, **kw) + super(host, creds, **kw) end # Used define_method to add a method for each rpc_desc. Each method diff --git a/src/ruby/spec/call_spec.rb b/src/ruby/spec/call_spec.rb index ae3ce0748a2..1c44b333de7 100644 --- a/src/ruby/spec/call_spec.rb +++ b/src/ruby/spec/call_spec.rb @@ -96,7 +96,6 @@ describe GRPC::Core::CallOps do end describe GRPC::Core::Call do - let(:client_queue) { GRPC::Core::CompletionQueue.new } let(:test_tag) { Object.new } let(:fake_host) { 'localhost:10101' } @@ -154,7 +153,7 @@ describe GRPC::Core::Call do end def make_test_call - @ch.create_call(client_queue, nil, nil, 'dummy_method', nil, deadline) + @ch.create_call(nil, nil, 'dummy_method', nil, deadline) end def deadline diff --git a/src/ruby/spec/channel_spec.rb b/src/ruby/spec/channel_spec.rb index 355f95c9d79..740eac631a3 100644 --- a/src/ruby/spec/channel_spec.rb +++ b/src/ruby/spec/channel_spec.rb @@ -37,7 +37,6 @@ end describe GRPC::Core::Channel do let(:fake_host) { 'localhost:0' } - let(:cq) { GRPC::Core::CompletionQueue.new } def create_test_cert GRPC::Core::ChannelCredentials.new(load_test_certs[0]) @@ -122,7 +121,7 @@ describe GRPC::Core::Channel do deadline = Time.now + 5 blk = proc do - ch.create_call(cq, nil, nil, 'dummy_method', nil, deadline) + ch.create_call(nil, nil, 'dummy_method', nil, deadline) end expect(&blk).to_not raise_error end @@ -133,7 +132,7 @@ describe GRPC::Core::Channel do deadline = Time.now + 5 blk = proc do - ch.create_call(cq, nil, nil, 'dummy_method', nil, deadline) + ch.create_call(nil, nil, 'dummy_method', nil, deadline) end expect(&blk).to raise_error(RuntimeError) end diff --git a/src/ruby/spec/client_server_spec.rb b/src/ruby/spec/client_server_spec.rb index aedeca272d3..5440e6d0233 100644 --- a/src/ruby/spec/client_server_spec.rb +++ b/src/ruby/spec/client_server_spec.rb @@ -34,27 +34,23 @@ include GRPC::Core shared_context 'setup: tags' do let(:sent_message) { 'sent message' } let(:reply_text) { 'the reply' } - before(:example) do - @client_tag = Object.new - @server_tag = Object.new - end def deadline Time.now + 5 end def server_allows_client_to_proceed - recvd_rpc = @server.request_call(@server_queue, @server_tag, deadline) + recvd_rpc = @server.request_call expect(recvd_rpc).to_not eq nil server_call = recvd_rpc.call ops = { CallOps::SEND_INITIAL_METADATA => {} } - svr_batch = server_call.run_batch(@server_queue, @server_tag, deadline, ops) + svr_batch = server_call.run_batch(ops) expect(svr_batch.send_metadata).to be true server_call end def new_client_call - @ch.create_call(@client_queue, nil, nil, '/method', nil, deadline) + @ch.create_call(nil, nil, '/method', nil, deadline) end end @@ -91,8 +87,7 @@ shared_examples 'basic GRPC message delivery is OK' do CallOps::SEND_INITIAL_METADATA => {}, CallOps::SEND_MESSAGE => sent_message } - batch_result = call.run_batch(@client_queue, @client_tag, deadline, - client_ops) + batch_result = call.run_batch(client_ops) expect(batch_result.send_metadata).to be true expect(batch_result.send_message).to be true @@ -101,8 +96,7 @@ shared_examples 'basic GRPC message delivery is OK' do server_ops = { CallOps::RECV_MESSAGE => nil } - svr_batch = server_call.run_batch(@server_queue, @server_tag, deadline, - server_ops) + svr_batch = server_call.run_batch(server_ops) expect(svr_batch.message).to eq(sent_message) end @@ -118,8 +112,7 @@ shared_examples 'basic GRPC message delivery is OK' do CallOps::SEND_INITIAL_METADATA => {}, CallOps::SEND_MESSAGE => sent_message } - batch_result = call.run_batch(@client_queue, @client_tag, deadline, - client_ops) + batch_result = call.run_batch(client_ops) expect(batch_result.send_metadata).to be true expect(batch_result.send_message).to be true @@ -129,8 +122,7 @@ shared_examples 'basic GRPC message delivery is OK' do CallOps::RECV_MESSAGE => nil, CallOps::SEND_MESSAGE => reply_text } - svr_batch = server_call.run_batch(@server_queue, @server_tag, deadline, - server_ops) + svr_batch = server_call.run_batch(server_ops) expect(svr_batch.message).to eq(sent_message) expect(svr_batch.send_message).to be true end @@ -147,8 +139,7 @@ shared_examples 'basic GRPC message delivery is OK' do CallOps::SEND_INITIAL_METADATA => {}, CallOps::SEND_MESSAGE => sent_message } - batch_result = call.run_batch(@client_queue, @client_tag, deadline, - client_ops) + batch_result = call.run_batch(client_ops) expect(batch_result.send_metadata).to be true expect(batch_result.send_message).to be true @@ -158,8 +149,7 @@ shared_examples 'basic GRPC message delivery is OK' do server_ops = { CallOps::SEND_STATUS_FROM_SERVER => the_status } - svr_batch = server_call.run_batch(@server_queue, @server_tag, deadline, - server_ops) + svr_batch = server_call.run_batch(server_ops) expect(svr_batch.message).to eq nil expect(svr_batch.send_status).to be true end @@ -176,8 +166,7 @@ shared_examples 'basic GRPC message delivery is OK' do CallOps::SEND_INITIAL_METADATA => {}, CallOps::SEND_MESSAGE => sent_message } - batch_result = call.run_batch(@client_queue, @client_tag, deadline, - client_ops) + batch_result = call.run_batch(client_ops) expect(batch_result.send_metadata).to be true expect(batch_result.send_message).to be true @@ -189,8 +178,7 @@ shared_examples 'basic GRPC message delivery is OK' do CallOps::SEND_MESSAGE => reply_text, CallOps::SEND_STATUS_FROM_SERVER => the_status } - svr_batch = server_call.run_batch(@server_queue, @server_tag, deadline, - server_ops) + svr_batch = server_call.run_batch(server_ops) expect(svr_batch.message).to eq sent_message expect(svr_batch.send_status).to be true expect(svr_batch.send_message).to be true @@ -202,8 +190,7 @@ shared_examples 'basic GRPC message delivery is OK' do CallOps::RECV_MESSAGE => nil, CallOps::RECV_STATUS_ON_CLIENT => nil } - batch_result = call.run_batch(@client_queue, @client_tag, deadline, - client_ops) + batch_result = call.run_batch(client_ops) expect(batch_result.send_close).to be true expect(batch_result.message).to eq reply_text expect(batch_result.status).to eq the_status @@ -212,8 +199,7 @@ shared_examples 'basic GRPC message delivery is OK' do server_ops = { CallOps::RECV_CLOSE_ON_SERVER => nil } - svr_batch = server_call.run_batch(@server_queue, @server_tag, deadline, - server_ops) + svr_batch = server_call.run_batch(server_ops) expect(svr_batch.send_close).to be true end end @@ -244,8 +230,7 @@ shared_examples 'GRPC metadata delivery works OK' do CallOps::SEND_INITIAL_METADATA => md } blk = proc do - call.run_batch(@client_queue, @client_tag, deadline, - client_ops) + call.run_batch(client_ops) end expect(&blk).to raise_error end @@ -255,15 +240,14 @@ shared_examples 'GRPC metadata delivery works OK' do @valid_metadata.each do |md| recvd_rpc = nil rcv_thread = Thread.new do - recvd_rpc = @server.request_call(@server_queue, @server_tag, deadline) + recvd_rpc = @server.request_call end call = new_client_call client_ops = { CallOps::SEND_INITIAL_METADATA => md } - batch_result = call.run_batch(@client_queue, @client_tag, deadline, - client_ops) + batch_result = call.run_batch(client_ops) expect(batch_result.send_metadata).to be true # confirm the server can receive the client metadata @@ -296,7 +280,7 @@ shared_examples 'GRPC metadata delivery works OK' do @bad_keys.each do |md| recvd_rpc = nil rcv_thread = Thread.new do - recvd_rpc = @server.request_call(@server_queue, @server_tag, deadline) + recvd_rpc = @server.request_call end call = new_client_call @@ -305,7 +289,7 @@ shared_examples 'GRPC metadata delivery works OK' do client_ops = { CallOps::SEND_INITIAL_METADATA => nil } - call.run_batch(@client_queue, @client_tag, deadline, client_ops) + call.run_batch(client_ops) # server gets the invocation rcv_thread.join @@ -314,8 +298,7 @@ shared_examples 'GRPC metadata delivery works OK' do CallOps::SEND_INITIAL_METADATA => md } blk = proc do - recvd_rpc.call.run_batch(@server_queue, @server_tag, deadline, - server_ops) + recvd_rpc.call.run_batch(server_ops) end expect(&blk).to raise_error end @@ -324,7 +307,7 @@ shared_examples 'GRPC metadata delivery works OK' do it 'sends an empty hash if no metadata is added' do recvd_rpc = nil rcv_thread = Thread.new do - recvd_rpc = @server.request_call(@server_queue, @server_tag, deadline) + recvd_rpc = @server.request_call end call = new_client_call @@ -333,7 +316,7 @@ shared_examples 'GRPC metadata delivery works OK' do client_ops = { CallOps::SEND_INITIAL_METADATA => nil } - call.run_batch(@client_queue, @client_tag, deadline, client_ops) + call.run_batch(client_ops) # server gets the invocation but sends no metadata back rcv_thread.join @@ -342,14 +325,13 @@ shared_examples 'GRPC metadata delivery works OK' do server_ops = { CallOps::SEND_INITIAL_METADATA => nil } - server_call.run_batch(@server_queue, @server_tag, deadline, server_ops) + server_call.run_batch(server_ops) # client receives nothing as expected client_ops = { CallOps::RECV_INITIAL_METADATA => nil } - batch_result = call.run_batch(@client_queue, @client_tag, deadline, - client_ops) + batch_result = call.run_batch(client_ops) expect(batch_result.metadata).to eq({}) end @@ -357,7 +339,7 @@ shared_examples 'GRPC metadata delivery works OK' do @valid_metadata.each do |md| recvd_rpc = nil rcv_thread = Thread.new do - recvd_rpc = @server.request_call(@server_queue, @server_tag, deadline) + recvd_rpc = @server.request_call end call = new_client_call @@ -366,7 +348,7 @@ shared_examples 'GRPC metadata delivery works OK' do client_ops = { CallOps::SEND_INITIAL_METADATA => nil } - call.run_batch(@client_queue, @client_tag, deadline, client_ops) + call.run_batch(client_ops) # server gets the invocation but sends no metadata back rcv_thread.join @@ -375,14 +357,13 @@ shared_examples 'GRPC metadata delivery works OK' do server_ops = { CallOps::SEND_INITIAL_METADATA => md } - server_call.run_batch(@server_queue, @server_tag, deadline, server_ops) + server_call.run_batch(server_ops) # client receives nothing as expected client_ops = { CallOps::RECV_INITIAL_METADATA => nil } - batch_result = call.run_batch(@client_queue, @client_tag, deadline, - client_ops) + batch_result = call.run_batch(client_ops) replace_symbols = Hash[md.each_pair.collect { |x, y| [x.to_s, y] }] expect(batch_result.metadata).to eq(replace_symbols) end @@ -393,9 +374,7 @@ end describe 'the http client/server' do before(:example) do server_host = '0.0.0.0:0' - @client_queue = GRPC::Core::CompletionQueue.new - @server_queue = GRPC::Core::CompletionQueue.new - @server = GRPC::Core::Server.new(@server_queue, nil) + @server = GRPC::Core::Server.new(nil) server_port = @server.add_http2_port(server_host, :this_port_is_insecure) @server.start @ch = Channel.new("0.0.0.0:#{server_port}", nil, :this_channel_is_insecure) @@ -403,7 +382,7 @@ describe 'the http client/server' do after(:example) do @ch.close - @server.close(@server_queue, deadline) + @server.close(deadline) end it_behaves_like 'basic GRPC message delivery is OK' do @@ -425,11 +404,9 @@ describe 'the secure http client/server' do before(:example) do certs = load_test_certs server_host = '0.0.0.0:0' - @client_queue = GRPC::Core::CompletionQueue.new - @server_queue = GRPC::Core::CompletionQueue.new server_creds = GRPC::Core::ServerCredentials.new( nil, [{ private_key: certs[1], cert_chain: certs[2] }], false) - @server = GRPC::Core::Server.new(@server_queue, nil) + @server = GRPC::Core::Server.new(nil) server_port = @server.add_http2_port(server_host, server_creds) @server.start args = { Channel::SSL_TARGET => 'foo.test.google.fr' } @@ -438,7 +415,7 @@ describe 'the secure http client/server' do end after(:example) do - @server.close(@server_queue, deadline) + @server.close(deadline) end it_behaves_like 'basic GRPC message delivery is OK' do @@ -454,7 +431,7 @@ describe 'the secure http client/server' do expected_md = { 'k1' => 'updated-v1', 'k2' => 'v2' } recvd_rpc = nil rcv_thread = Thread.new do - recvd_rpc = @server.request_call(@server_queue, @server_tag, deadline) + recvd_rpc = @server.request_call end call = new_client_call @@ -462,8 +439,7 @@ describe 'the secure http client/server' do client_ops = { CallOps::SEND_INITIAL_METADATA => md } - batch_result = call.run_batch(@client_queue, @client_tag, deadline, - client_ops) + batch_result = call.run_batch(client_ops) expect(batch_result.send_metadata).to be true # confirm the server can receive the client metadata diff --git a/src/ruby/spec/completion_queue_spec.rb b/src/ruby/spec/completion_queue_spec.rb deleted file mode 100644 index 886a7f263ba..00000000000 --- a/src/ruby/spec/completion_queue_spec.rb +++ /dev/null @@ -1,42 +0,0 @@ -# Copyright 2015, Google Inc. -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are -# met: -# -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above -# copyright notice, this list of conditions and the following disclaimer -# in the documentation and/or other materials provided with the -# distribution. -# * Neither the name of Google Inc. nor the names of its -# contributors may be used to endorse or promote products derived from -# this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -require 'grpc' - -describe GRPC::Core::CompletionQueue do - before(:example) do - @cq = GRPC::Core::CompletionQueue.new - end - - describe '#new' do - it 'is constructed successufully' do - expect { GRPC::Core::CompletionQueue.new }.not_to raise_error - end - end -end diff --git a/src/ruby/spec/generic/active_call_spec.rb b/src/ruby/spec/generic/active_call_spec.rb index d9c9780c93b..b4e6f9ee028 100644 --- a/src/ruby/spec/generic/active_call_spec.rb +++ b/src/ruby/spec/generic/active_call_spec.rb @@ -39,13 +39,8 @@ describe GRPC::ActiveCall do before(:each) do @pass_through = proc { |x| x } - @server_tag = Object.new - @tag = Object.new - - @client_queue = GRPC::Core::CompletionQueue.new - @server_queue = GRPC::Core::CompletionQueue.new host = '0.0.0.0:0' - @server = GRPC::Core::Server.new(@server_queue, nil) + @server = GRPC::Core::Server.new(nil) server_port = @server.add_http2_port(host, :this_port_is_insecure) @server.start @ch = GRPC::Core::Channel.new("0.0.0.0:#{server_port}", nil, @@ -53,16 +48,15 @@ describe GRPC::ActiveCall do end after(:each) do - @server.close(@server_queue, deadline) + @server.close(deadline) end describe 'restricted view methods' do before(:each) do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) - @client_call = ActiveCall.new(call, @client_queue, @pass_through, - @pass_through, deadline, - metadata_tag: md_tag) + ActiveCall.client_invoke(call) + @client_call = ActiveCall.new(call, @pass_through, + @pass_through, deadline) end describe '#multi_req_view' do @@ -89,46 +83,42 @@ describe GRPC::ActiveCall do describe '#remote_send' do it 'allows a client to send a payload to the server' do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) - @client_call = ActiveCall.new(call, @client_queue, @pass_through, - @pass_through, deadline, - metadata_tag: md_tag) + ActiveCall.client_invoke(call) + @client_call = ActiveCall.new(call, @pass_through, + @pass_through, deadline) msg = 'message is a string' @client_call.remote_send(msg) # check that server rpc new was received - recvd_rpc = @server.request_call(@server_queue, @server_tag, deadline) + recvd_rpc = @server.request_call expect(recvd_rpc).to_not eq nil recvd_call = recvd_rpc.call # Accept the call, and verify that the server reads the response ok. - server_ops = { - CallOps::SEND_INITIAL_METADATA => {} - } - recvd_call.run_batch(@server_queue, @server_tag, deadline, server_ops) - server_call = ActiveCall.new(recvd_call, @server_queue, @pass_through, - @pass_through, deadline) + server_call = ActiveCall.new(recvd_call, @pass_through, + @pass_through, deadline, + metadata_received: true) expect(server_call.remote_read).to eq(msg) end it 'marshals the payload using the marshal func' do call = make_test_call - ActiveCall.client_invoke(call, @client_queue) + ActiveCall.client_invoke(call) marshal = proc { |x| 'marshalled:' + x } - client_call = ActiveCall.new(call, @client_queue, marshal, - @pass_through, deadline) + client_call = ActiveCall.new(call, marshal, @pass_through, deadline) msg = 'message is a string' client_call.remote_send(msg) # confirm that the message was marshalled - recvd_rpc = @server.request_call(@server_queue, @server_tag, deadline) + recvd_rpc = @server.request_call recvd_call = recvd_rpc.call server_ops = { CallOps::SEND_INITIAL_METADATA => nil } - recvd_call.run_batch(@server_queue, @server_tag, deadline, server_ops) - server_call = ActiveCall.new(recvd_call, @server_queue, @pass_through, - @pass_through, deadline) + recvd_call.run_batch(server_ops) + server_call = ActiveCall.new(recvd_call, @pass_through, + @pass_through, deadline, + metadata_received: true) expect(server_call.remote_read).to eq('marshalled:' + msg) end @@ -136,23 +126,24 @@ describe GRPC::ActiveCall do TEST_WRITE_FLAGS.each do |f| it "successfully makes calls with write_flag set to #{f}" do call = make_test_call - ActiveCall.client_invoke(call, @client_queue) + ActiveCall.client_invoke(call) marshal = proc { |x| 'marshalled:' + x } - client_call = ActiveCall.new(call, @client_queue, marshal, + client_call = ActiveCall.new(call, marshal, @pass_through, deadline) msg = 'message is a string' client_call.write_flag = f client_call.remote_send(msg) # confirm that the message was marshalled - recvd_rpc = @server.request_call(@server_queue, @server_tag, deadline) + recvd_rpc = @server.request_call recvd_call = recvd_rpc.call server_ops = { CallOps::SEND_INITIAL_METADATA => nil } - recvd_call.run_batch(@server_queue, @server_tag, deadline, server_ops) - server_call = ActiveCall.new(recvd_call, @server_queue, @pass_through, - @pass_through, deadline) + recvd_call.run_batch(server_ops) + server_call = ActiveCall.new(recvd_call, @pass_through, + @pass_through, deadline, + metadata_received: true) expect(server_call.remote_read).to eq('marshalled:' + msg) end end @@ -162,8 +153,8 @@ describe GRPC::ActiveCall do it 'sends metadata to the server when present' do call = make_test_call metadata = { k1: 'v1', k2: 'v2' } - ActiveCall.client_invoke(call, @client_queue, metadata) - recvd_rpc = @server.request_call(@server_queue, @server_tag, deadline) + ActiveCall.client_invoke(call, metadata) + recvd_rpc = @server.request_call recvd_call = recvd_rpc.call expect(recvd_call).to_not be_nil expect(recvd_rpc.metadata).to_not be_nil @@ -175,10 +166,9 @@ describe GRPC::ActiveCall do describe '#remote_read' do it 'reads the response sent by a server' do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) - client_call = ActiveCall.new(call, @client_queue, @pass_through, - @pass_through, deadline, - metadata_tag: md_tag) + ActiveCall.client_invoke(call) + client_call = ActiveCall.new(call, @pass_through, + @pass_through, deadline) msg = 'message is a string' client_call.remote_send(msg) server_call = expect_server_to_receive(msg) @@ -188,10 +178,9 @@ describe GRPC::ActiveCall do it 'saves no metadata when the server adds no metadata' do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) - client_call = ActiveCall.new(call, @client_queue, @pass_through, - @pass_through, deadline, - metadata_tag: md_tag) + ActiveCall.client_invoke(call) + client_call = ActiveCall.new(call, @pass_through, + @pass_through, deadline) msg = 'message is a string' client_call.remote_send(msg) server_call = expect_server_to_receive(msg) @@ -203,10 +192,9 @@ describe GRPC::ActiveCall do it 'saves metadata add by the server' do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) - client_call = ActiveCall.new(call, @client_queue, @pass_through, - @pass_through, deadline, - metadata_tag: md_tag) + ActiveCall.client_invoke(call) + client_call = ActiveCall.new(call, @pass_through, + @pass_through, deadline) msg = 'message is a string' client_call.remote_send(msg) server_call = expect_server_to_receive(msg, k1: 'v1', k2: 'v2') @@ -219,10 +207,9 @@ describe GRPC::ActiveCall do it 'get a nil msg before a status when an OK status is sent' do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) - client_call = ActiveCall.new(call, @client_queue, @pass_through, - @pass_through, deadline, - metadata_tag: md_tag) + ActiveCall.client_invoke(call) + client_call = ActiveCall.new(call, @pass_through, + @pass_through, deadline) msg = 'message is a string' client_call.remote_send(msg) client_call.writes_done(false) @@ -236,11 +223,10 @@ describe GRPC::ActiveCall do it 'unmarshals the response using the unmarshal func' do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) + ActiveCall.client_invoke(call) unmarshal = proc { |x| 'unmarshalled:' + x } - client_call = ActiveCall.new(call, @client_queue, @pass_through, - unmarshal, deadline, - metadata_tag: md_tag) + client_call = ActiveCall.new(call, @pass_through, + unmarshal, deadline) # confirm the client receives the unmarshalled message msg = 'message is a string' @@ -254,17 +240,16 @@ describe GRPC::ActiveCall do describe '#each_remote_read' do it 'creates an Enumerator' do call = make_test_call - client_call = ActiveCall.new(call, @client_queue, @pass_through, + client_call = ActiveCall.new(call, @pass_through, @pass_through, deadline) expect(client_call.each_remote_read).to be_a(Enumerator) end it 'the returns an enumerator that can read n responses' do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) - client_call = ActiveCall.new(call, @client_queue, @pass_through, - @pass_through, deadline, - metadata_tag: md_tag) + ActiveCall.client_invoke(call) + client_call = ActiveCall.new(call, @pass_through, + @pass_through, deadline) msg = 'message is a string' reply = 'server_response' client_call.remote_send(msg) @@ -279,10 +264,9 @@ describe GRPC::ActiveCall do it 'the returns an enumerator that stops after an OK Status' do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) - client_call = ActiveCall.new(call, @client_queue, @pass_through, - @pass_through, deadline, - metadata_tag: md_tag) + ActiveCall.client_invoke(call) + client_call = ActiveCall.new(call, @pass_through, + @pass_through, deadline) msg = 'message is a string' reply = 'server_response' client_call.remote_send(msg) @@ -302,10 +286,9 @@ describe GRPC::ActiveCall do describe '#writes_done' do it 'finishes ok if the server sends a status response' do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) - client_call = ActiveCall.new(call, @client_queue, @pass_through, - @pass_through, deadline, - metadata_tag: md_tag) + ActiveCall.client_invoke(call) + client_call = ActiveCall.new(call, @pass_through, + @pass_through, deadline) msg = 'message is a string' client_call.remote_send(msg) expect { client_call.writes_done(false) }.to_not raise_error @@ -318,10 +301,9 @@ describe GRPC::ActiveCall do it 'finishes ok if the server sends an early status response' do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) - client_call = ActiveCall.new(call, @client_queue, @pass_through, - @pass_through, deadline, - metadata_tag: md_tag) + ActiveCall.client_invoke(call) + client_call = ActiveCall.new(call, @pass_through, + @pass_through, deadline) msg = 'message is a string' client_call.remote_send(msg) server_call = expect_server_to_receive(msg) @@ -334,10 +316,9 @@ describe GRPC::ActiveCall do it 'finishes ok if writes_done is true' do call = make_test_call - md_tag = ActiveCall.client_invoke(call, @client_queue) - client_call = ActiveCall.new(call, @client_queue, @pass_through, - @pass_through, deadline, - metadata_tag: md_tag) + ActiveCall.client_invoke(call) + client_call = ActiveCall.new(call, @pass_through, + @pass_through, deadline) msg = 'message is a string' client_call.remote_send(msg) server_call = expect_server_to_receive(msg) @@ -355,17 +336,16 @@ describe GRPC::ActiveCall do end def expect_server_to_be_invoked(**kw) - recvd_rpc = @server.request_call(@server_queue, @server_tag, deadline) + recvd_rpc = @server.request_call expect(recvd_rpc).to_not eq nil recvd_call = recvd_rpc.call - recvd_call.run_batch(@server_queue, @server_tag, deadline, - CallOps::SEND_INITIAL_METADATA => kw) - ActiveCall.new(recvd_call, @server_queue, @pass_through, - @pass_through, deadline) + recvd_call.run_batch(CallOps::SEND_INITIAL_METADATA => kw) + ActiveCall.new(recvd_call, @pass_through, @pass_through, deadline, + metadata_received: true, started: true) end def make_test_call - @ch.create_call(@client_queue, nil, nil, '/method', nil, deadline) + @ch.create_call(nil, nil, '/method', nil, deadline) end def deadline diff --git a/src/ruby/spec/generic/client_stub_spec.rb b/src/ruby/spec/generic/client_stub_spec.rb index 168e7fb7919..6034b5419c9 100644 --- a/src/ruby/spec/generic/client_stub_spec.rb +++ b/src/ruby/spec/generic/client_stub_spec.rb @@ -29,11 +29,14 @@ require 'grpc' +Thread.abort_on_exception = true + def wakey_thread(&blk) n = GRPC::Notifier.new t = Thread.new do blk.call(n) end + t.abort_on_exception = true n.wait t end @@ -54,15 +57,13 @@ describe 'ClientStub' do before(:each) do Thread.abort_on_exception = true @server = nil - @server_queue = nil @method = 'an_rpc_method' @pass = OK @fail = INTERNAL - @cq = GRPC::Core::CompletionQueue.new end after(:each) do - @server.close(@server_queue) unless @server_queue.nil? + @server.close(from_relative_time(2)) unless @server.nil? end describe '#new' do @@ -70,7 +71,7 @@ describe 'ClientStub' do it 'can be created from a host and args' do opts = { channel_args: { a_channel_arg: 'an_arg' } } blk = proc do - GRPC::ClientStub.new(fake_host, @cq, :this_channel_is_insecure, **opts) + GRPC::ClientStub.new(fake_host, :this_channel_is_insecure, **opts) end expect(&blk).not_to raise_error end @@ -81,7 +82,7 @@ describe 'ClientStub' do channel_override: @ch } blk = proc do - GRPC::ClientStub.new(fake_host, @cq, :this_channel_is_insecure, **opts) + GRPC::ClientStub.new(fake_host, :this_channel_is_insecure, **opts) end expect(&blk).not_to raise_error end @@ -92,7 +93,7 @@ describe 'ClientStub' do channel_args: { a_channel_arg: 'an_arg' }, channel_override: Object.new } - GRPC::ClientStub.new(fake_host, @cq, :this_channel_is_insecure, **opts) + GRPC::ClientStub.new(fake_host, :this_channel_is_insecure, **opts) end expect(&blk).to raise_error end @@ -100,7 +101,7 @@ describe 'ClientStub' do it 'cannot be created with bad credentials' do blk = proc do opts = { channel_args: { a_channel_arg: 'an_arg' } } - GRPC::ClientStub.new(fake_host, @cq, Object.new, **opts) + GRPC::ClientStub.new(fake_host, Object.new, **opts) end expect(&blk).to raise_error end @@ -115,7 +116,7 @@ describe 'ClientStub' do } } creds = GRPC::Core::ChannelCredentials.new(certs[0], nil, nil) - GRPC::ClientStub.new(fake_host, @cq, creds, **opts) + GRPC::ClientStub.new(fake_host, creds, **opts) end expect(&blk).to_not raise_error end @@ -130,7 +131,7 @@ describe 'ClientStub' do it 'should send a request to/receive a reply from a server' do server_port = create_test_server th = run_request_response(@sent_msg, @resp, @pass) - stub = GRPC::ClientStub.new("localhost:#{server_port}", @cq, + stub = GRPC::ClientStub.new("localhost:#{server_port}", :this_channel_is_insecure) expect(get_response(stub)).to eq(@resp) th.join @@ -141,7 +142,7 @@ describe 'ClientStub' do host = "localhost:#{server_port}" th = run_request_response(@sent_msg, @resp, @pass, k1: 'v1', k2: 'v2') - stub = GRPC::ClientStub.new(host, @cq, :this_channel_is_insecure) + stub = GRPC::ClientStub.new(host, :this_channel_is_insecure) expect(get_response(stub)).to eq(@resp) th.join end @@ -151,7 +152,7 @@ describe 'ClientStub' do alt_host = "localhost:#{server_port}" th = run_request_response(@sent_msg, @resp, @pass) ch = GRPC::Core::Channel.new(alt_host, nil, :this_channel_is_insecure) - stub = GRPC::ClientStub.new('ignored-host', @cq, + stub = GRPC::ClientStub.new('ignored-host', :this_channel_is_insecure, channel_override: ch) expect(get_response(stub)).to eq(@resp) @@ -162,7 +163,7 @@ describe 'ClientStub' do server_port = create_test_server host = "localhost:#{server_port}" th = run_request_response(@sent_msg, @resp, @fail) - stub = GRPC::ClientStub.new(host, @cq, :this_channel_is_insecure) + stub = GRPC::ClientStub.new(host, :this_channel_is_insecure) blk = proc { get_response(stub) } expect(&blk).to raise_error(GRPC::BadStatus) th.join @@ -182,7 +183,8 @@ describe 'ClientStub' do def get_response(stub) op = stub.request_response(@method, @sent_msg, noop, noop, return_op: true, - metadata: { k1: 'v1', k2: 'v2' }) + metadata: { k1: 'v1', k2: 'v2' }, + deadline: from_relative_time(2)) expect(op).to be_a(GRPC::ActiveCall::Operation) op.execute end @@ -196,7 +198,7 @@ describe 'ClientStub' do before(:each) do server_port = create_test_server host = "localhost:#{server_port}" - @stub = GRPC::ClientStub.new(host, @cq, :this_channel_is_insecure) + @stub = GRPC::ClientStub.new(host, :this_channel_is_insecure) @metadata = { k1: 'v1', k2: 'v2' } @sent_msgs = Array.new(3) { |i| 'msg_' + (i + 1).to_s } @resp = 'a_reply' @@ -262,7 +264,7 @@ describe 'ClientStub' do server_port = create_test_server host = "localhost:#{server_port}" th = run_server_streamer(@sent_msg, @replys, @pass) - stub = GRPC::ClientStub.new(host, @cq, :this_channel_is_insecure) + stub = GRPC::ClientStub.new(host, :this_channel_is_insecure) expect(get_responses(stub).collect { |r| r }).to eq(@replys) th.join end @@ -271,7 +273,7 @@ describe 'ClientStub' do server_port = create_test_server host = "localhost:#{server_port}" th = run_server_streamer(@sent_msg, @replys, @fail) - stub = GRPC::ClientStub.new(host, @cq, :this_channel_is_insecure) + stub = GRPC::ClientStub.new(host, :this_channel_is_insecure) e = get_responses(stub) expect { e.collect { |r| r } }.to raise_error(GRPC::BadStatus) th.join @@ -282,7 +284,7 @@ describe 'ClientStub' do host = "localhost:#{server_port}" th = run_server_streamer(@sent_msg, @replys, @fail, k1: 'v1', k2: 'v2') - stub = GRPC::ClientStub.new(host, @cq, :this_channel_is_insecure) + stub = GRPC::ClientStub.new(host, :this_channel_is_insecure) e = get_responses(stub) expect { e.collect { |r| r } }.to raise_error(GRPC::BadStatus) th.join @@ -327,7 +329,7 @@ describe 'ClientStub' do it 'supports sending all the requests first', bidi: true do th = run_bidi_streamer_handle_inputs_first(@sent_msgs, @replys, @pass) - stub = GRPC::ClientStub.new(@host, @cq, :this_channel_is_insecure) + stub = GRPC::ClientStub.new(@host, :this_channel_is_insecure) e = get_responses(stub) expect(e.collect { |r| r }).to eq(@replys) th.join @@ -335,7 +337,7 @@ describe 'ClientStub' do it 'supports client-initiated ping pong', bidi: true do th = run_bidi_streamer_echo_ping_pong(@sent_msgs, @pass, true) - stub = GRPC::ClientStub.new(@host, @cq, :this_channel_is_insecure) + stub = GRPC::ClientStub.new(@host, :this_channel_is_insecure) e = get_responses(stub) expect(e.collect { |r| r }).to eq(@sent_msgs) th.join @@ -343,7 +345,7 @@ describe 'ClientStub' do it 'supports a server-initiated ping pong', bidi: true do th = run_bidi_streamer_echo_ping_pong(@sent_msgs, @pass, false) - stub = GRPC::ClientStub.new(@host, @cq, :this_channel_is_insecure) + stub = GRPC::ClientStub.new(@host, :this_channel_is_insecure) e = get_responses(stub) expect(e.collect { |r| r }).to eq(@sent_msgs) th.join @@ -372,26 +374,6 @@ describe 'ClientStub' do it_behaves_like 'bidi streaming' end - - describe 'without enough time to run' do - before(:each) do - @sent_msgs = Array.new(3) { |i| 'msg_' + (i + 1).to_s } - @replys = Array.new(3) { |i| 'reply_' + (i + 1).to_s } - server_port = create_test_server - @host = "localhost:#{server_port}" - end - - it 'should fail with DeadlineExceeded', bidi: true do - @server.start - stub = GRPC::ClientStub.new(@host, @cq, :this_channel_is_insecure) - blk = proc do - e = stub.bidi_streamer(@method, @sent_msgs, noop, noop, - deadline: from_relative_time(0.001)) - e.collect { |r| r } - end - expect(&blk).to raise_error GRPC::BadStatus, /Deadline Exceeded/ - end - end end def run_server_streamer(expected_input, replys, status, **kw) @@ -460,21 +442,18 @@ describe 'ClientStub' do end def create_test_server - @server_queue = GRPC::Core::CompletionQueue.new - @server = GRPC::Core::Server.new(@server_queue, nil) + @server = GRPC::Core::Server.new(nil) @server.add_http2_port('0.0.0.0:0', :this_port_is_insecure) end def expect_server_to_be_invoked(notifier) @server.start notifier.notify(nil) - server_tag = Object.new - recvd_rpc = @server.request_call(@server_queue, server_tag, - INFINITE_FUTURE) + recvd_rpc = @server.request_call recvd_call = recvd_rpc.call recvd_call.metadata = recvd_rpc.metadata - recvd_call.run_batch(@server_queue, server_tag, Time.now + 2, - SEND_INITIAL_METADATA => nil) - GRPC::ActiveCall.new(recvd_call, @server_queue, noop, noop, INFINITE_FUTURE) + recvd_call.run_batch(SEND_INITIAL_METADATA => nil) + GRPC::ActiveCall.new(recvd_call, noop, noop, INFINITE_FUTURE, + metadata_received: true) end end diff --git a/src/ruby/spec/generic/rpc_server_spec.rb b/src/ruby/spec/generic/rpc_server_spec.rb index 943502cea23..901c84fc783 100644 --- a/src/ruby/spec/generic/rpc_server_spec.rb +++ b/src/ruby/spec/generic/rpc_server_spec.rb @@ -135,8 +135,6 @@ describe GRPC::RpcServer do @pass = 0 @fail = 1 @noop = proc { |x| x } - - @server_queue = GRPC::Core::CompletionQueue.new end describe '#new' do @@ -148,28 +146,6 @@ describe GRPC::RpcServer do expect(&blk).not_to raise_error end - it 'can be created with a completion queue override' do - opts = { - server_args: { a_channel_arg: 'an_arg' }, - completion_queue_override: @server_queue - } - blk = proc do - RpcServer.new(**opts) - end - expect(&blk).not_to raise_error - end - - it 'cannot be created with a bad completion queue override' do - blk = proc do - opts = { - server_args: { a_channel_arg: 'an_arg' }, - completion_queue_override: Object.new - } - RpcServer.new(**opts) - end - expect(&blk).to raise_error - end - it 'cannot be created with invalid ServerCredentials' do blk = proc do opts = { @@ -294,7 +270,6 @@ describe GRPC::RpcServer do context 'with no connect_metadata' do before(:each) do server_opts = { - completion_queue_override: @server_queue, poll_period: 1 } @srv = RpcServer.new(**server_opts) @@ -309,8 +284,7 @@ describe GRPC::RpcServer do @srv.wait_till_running req = EchoMsg.new blk = proc do - cq = GRPC::Core::CompletionQueue.new - stub = GRPC::ClientStub.new(@host, cq, :this_channel_is_insecure, + stub = GRPC::ClientStub.new(@host, :this_channel_is_insecure, **client_opts) stub.request_response('/unknown', req, marshal, unmarshal) end @@ -325,8 +299,7 @@ describe GRPC::RpcServer do @srv.wait_till_running req = EchoMsg.new blk = proc do - cq = GRPC::Core::CompletionQueue.new - stub = GRPC::ClientStub.new(@host, cq, :this_channel_is_insecure, + stub = GRPC::ClientStub.new(@host, :this_channel_is_insecure, **client_opts) stub.request_response('/an_rpc', req, marshal, unmarshal) end @@ -422,7 +395,6 @@ describe GRPC::RpcServer do it 'should return RESOURCE_EXHAUSTED on too many jobs', server: true do opts = { server_args: { a_channel_arg: 'an_arg' }, - completion_queue_override: @server_queue, pool_size: 1, poll_period: 1, max_waiting_requests: 0 @@ -466,7 +438,6 @@ describe GRPC::RpcServer do end before(:each) do server_opts = { - completion_queue_override: @server_queue, poll_period: 1, connect_md_proc: test_md_proc } @@ -502,7 +473,6 @@ describe GRPC::RpcServer do context 'with trailing metadata' do before(:each) do server_opts = { - completion_queue_override: @server_queue, poll_period: 1 } @srv = RpcServer.new(**server_opts) diff --git a/src/ruby/spec/pb/health/checker_spec.rb b/src/ruby/spec/pb/health/checker_spec.rb index f3d121a31ef..de11c9fedf7 100644 --- a/src/ruby/spec/pb/health/checker_spec.rb +++ b/src/ruby/spec/pb/health/checker_spec.rb @@ -168,11 +168,9 @@ describe Grpc::Health::Checker do CheckerStub = Grpc::Health::Checker.rpc_stub_class before(:each) do - @server_queue = GRPC::Core::CompletionQueue.new server_host = '0.0.0.0:0' @client_opts = { channel_override: @ch } server_opts = { - completion_queue_override: @server_queue, poll_period: 1 } @srv = RpcServer.new(**server_opts) diff --git a/src/ruby/spec/server_spec.rb b/src/ruby/spec/server_spec.rb index 439b19fb8db..003d8f69d57 100644 --- a/src/ruby/spec/server_spec.rb +++ b/src/ruby/spec/server_spec.rb @@ -43,19 +43,15 @@ describe Server do GRPC::Core::ServerCredentials.new(*load_test_certs) end - before(:each) do - @cq = GRPC::Core::CompletionQueue.new - end - describe '#start' do it 'runs without failing' do - blk = proc { Server.new(@cq, nil).start } + blk = proc { Server.new(nil).start } expect(&blk).to_not raise_error end it 'fails if the server is closed' do - s = Server.new(@cq, nil) - s.close(@cq) + s = Server.new(nil) + s.close expect { s.start }.to raise_error(RuntimeError) end end @@ -63,19 +59,19 @@ describe Server do describe '#destroy' do it 'destroys a server ok' do s = start_a_server - blk = proc { s.destroy(@cq) } + blk = proc { s.destroy } expect(&blk).to_not raise_error end it 'can be called more than once without error' do s = start_a_server begin - blk = proc { s.destroy(@cq) } + blk = proc { s.destroy } expect(&blk).to_not raise_error blk.call expect(&blk).to_not raise_error ensure - s.close(@cq) + s.close end end end @@ -84,7 +80,7 @@ describe Server do it 'closes a server ok' do s = start_a_server begin - blk = proc { s.close(@cq) } + blk = proc { s.close } expect(&blk).to_not raise_error ensure s.close(@cq) @@ -93,7 +89,7 @@ describe Server do it 'can be called more than once without error' do s = start_a_server - blk = proc { s.close(@cq) } + blk = proc { s.close } expect(&blk).to_not raise_error blk.call expect(&blk).to_not raise_error @@ -104,16 +100,16 @@ describe Server do describe 'for insecure servers' do it 'runs without failing' do blk = proc do - s = Server.new(@cq, nil) + s = Server.new(nil) s.add_http2_port('localhost:0', :this_port_is_insecure) - s.close(@cq) + s.close end expect(&blk).to_not raise_error end it 'fails if the server is closed' do - s = Server.new(@cq, nil) - s.close(@cq) + s = Server.new(nil) + s.close blk = proc do s.add_http2_port('localhost:0', :this_port_is_insecure) end @@ -125,16 +121,16 @@ describe Server do let(:cert) { create_test_cert } it 'runs without failing' do blk = proc do - s = Server.new(@cq, nil) + s = Server.new(nil) s.add_http2_port('localhost:0', cert) - s.close(@cq) + s.close end expect(&blk).to_not raise_error end it 'fails if the server is closed' do - s = Server.new(@cq, nil) - s.close(@cq) + s = Server.new(nil) + s.close blk = proc { s.add_http2_port('localhost:0', cert) } expect(&blk).to raise_error(RuntimeError) end @@ -142,8 +138,8 @@ describe Server do end shared_examples '#new' do - it 'takes a completion queue with nil channel args' do - expect { Server.new(@cq, nil) }.to_not raise_error + it 'takes nil channel args' do + expect { Server.new(nil) }.to_not raise_error end it 'does not take a hash with bad keys as channel args' do @@ -194,14 +190,14 @@ describe Server do describe '#new with an insecure channel' do def construct_with_args(a) - proc { Server.new(@cq, a) } + proc { Server.new(a) } end it_behaves_like '#new' end def start_a_server - s = Server.new(@cq, nil) + s = Server.new(nil) s.add_http2_port('0.0.0.0:0', :this_port_is_insecure) s.start s From ddc3ebb6ea694b252c27e149507bec4df427a592 Mon Sep 17 00:00:00 2001 From: yang-g Date: Mon, 13 Jun 2016 10:40:32 -0700 Subject: [PATCH 0087/1039] add missing language tag --- build.yaml | 11 +++++----- tools/run_tests/sources_and_headers.json | 26 ++++++++++++------------ 2 files changed, 19 insertions(+), 18 deletions(-) diff --git a/build.yaml b/build.yaml index a94fb052a5d..8589ba28f8e 100644 --- a/build.yaml +++ b/build.yaml @@ -143,11 +143,6 @@ filegroups: - include/grpc/impl/codegen/sync_posix.h - include/grpc/impl/codegen/sync_windows.h - include/grpc/impl/codegen/time.h -- name: grpc++_codegen_base_src - src: - - src/cpp/codegen/codegen_init.cc - uses: - - grpc++_codegen_base - name: grpc_base public_headers: - include/grpc/byte_buffer.h @@ -747,6 +742,12 @@ filegroups: - include/grpc++/impl/codegen/time.h uses: - grpc_codegen +- name: grpc++_codegen_base_src + language: c++ + src: + - src/cpp/codegen/codegen_init.cc + uses: + - grpc++_codegen_base - name: grpc++_codegen_proto language: c++ public_headers: diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index fd522ee1739..1d73434f3dc 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -5607,19 +5607,6 @@ "third_party": false, "type": "filegroup" }, - { - "deps": [ - "grpc++_codegen_base" - ], - "headers": [], - "language": "c", - "name": "grpc++_codegen_base_src", - "src": [ - "src/cpp/codegen/codegen_init.cc" - ], - "third_party": false, - "type": "filegroup" - }, { "deps": [ "gpr", @@ -6636,6 +6623,19 @@ "third_party": false, "type": "filegroup" }, + { + "deps": [ + "grpc++_codegen_base" + ], + "headers": [], + "language": "c++", + "name": "grpc++_codegen_base_src", + "src": [ + "src/cpp/codegen/codegen_init.cc" + ], + "third_party": false, + "type": "filegroup" + }, { "deps": [ "grpc++_codegen_base", From 1fa96c5c49b4fda6ebd2be0cfe353297e3e92e0b Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Mon, 13 Jun 2016 10:48:22 -0700 Subject: [PATCH 0088/1039] Removed an extra mark object, fixed some documentation --- src/ruby/ext/grpc/rb_server.c | 20 ++------------------ src/ruby/ext/grpc/rb_server_credentials.c | 4 ++-- 2 files changed, 4 insertions(+), 20 deletions(-) diff --git a/src/ruby/ext/grpc/rb_server.c b/src/ruby/ext/grpc/rb_server.c index 1ed2c35da34..a7d53350920 100644 --- a/src/ruby/ext/grpc/rb_server.c +++ b/src/ruby/ext/grpc/rb_server.c @@ -53,11 +53,8 @@ static ID id_at; /* id_insecure_server is used to indicate that a server is insecure */ static VALUE id_insecure_server; -/* grpc_rb_server wraps a grpc_server. It provides a peer ruby object, - 'mark' to minimize copying when a server is created from ruby. */ +/* grpc_rb_server wraps a grpc_server. */ typedef struct grpc_rb_server { - /* Holder of ruby objects involved in constructing the server */ - VALUE mark; /* The actual server */ grpc_server *wrapped; grpc_completion_queue *queue; @@ -98,21 +95,9 @@ static void grpc_rb_server_free(void *p) { xfree(p); } -/* Protects the mark object from GC */ -static void grpc_rb_server_mark(void *p) { - grpc_rb_server *server = NULL; - if (p == NULL) { - return; - } - server = (grpc_rb_server *)p; - if (server->mark != Qnil) { - rb_gc_mark(server->mark); - } -} - static const rb_data_type_t grpc_rb_server_data_type = { "grpc_server", - {grpc_rb_server_mark, grpc_rb_server_free, GRPC_RB_MEMSIZE_UNAVAILABLE, + {GRPC_RB_GC_NOT_MARKED, grpc_rb_server_free, GRPC_RB_MEMSIZE_UNAVAILABLE, {NULL, NULL}}, NULL, NULL, @@ -129,7 +114,6 @@ static const rb_data_type_t grpc_rb_server_data_type = { static VALUE grpc_rb_server_alloc(VALUE cls) { grpc_rb_server *wrapper = ALLOC(grpc_rb_server); wrapper->wrapped = NULL; - wrapper->mark = Qnil; return TypedData_Wrap_Struct(cls, &grpc_rb_server_data_type, wrapper); } diff --git a/src/ruby/ext/grpc/rb_server_credentials.c b/src/ruby/ext/grpc/rb_server_credentials.c index 58ec852505c..cb7545b3645 100644 --- a/src/ruby/ext/grpc/rb_server_credentials.c +++ b/src/ruby/ext/grpc/rb_server_credentials.c @@ -46,8 +46,8 @@ static VALUE grpc_rb_cServerCredentials = Qnil; /* grpc_rb_server_credentials wraps a grpc_server_credentials. It provides a - peer ruby object, 'mark' to minimize copying when a server credential is - created from ruby. */ + peer ruby object, 'mark' to hold references to objects involved in + constructing the server credentials. */ typedef struct grpc_rb_server_credentials { /* Holder of ruby objects involved in constructing the server credentials */ VALUE mark; From 773a8825bcb9cae06af18263b112b8f8fd96f10d Mon Sep 17 00:00:00 2001 From: yang-g Date: Mon, 13 Jun 2016 11:09:29 -0700 Subject: [PATCH 0089/1039] Minor fixes --- src/core/lib/iomgr/udp_server.c | 4 ++-- src/cpp/server/server_posix.cc | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/core/lib/iomgr/udp_server.c b/src/core/lib/iomgr/udp_server.c index 16150687d3d..1ebccf2ee2e 100644 --- a/src/core/lib/iomgr/udp_server.c +++ b/src/core/lib/iomgr/udp_server.c @@ -243,13 +243,13 @@ static int prepare_socket(int fd, const struct sockaddr *addr, if (!grpc_set_socket_sndbuf(fd, buffer_size_bytes)) { gpr_log(GPR_ERROR, "Failed to set send buffer size to %d bytes", - buf_size_bytes); + buffer_size_bytes); goto error; } if (!grpc_set_socket_rcvbuf(fd, buffer_size_bytes)) { gpr_log(GPR_ERROR, "Failed to set receive buffer size to %d bytes", - buf_size_bytes); + buffer_size_bytes); goto error; } diff --git a/src/cpp/server/server_posix.cc b/src/cpp/server/server_posix.cc index 8cb9753a125..c3aa2adc60e 100644 --- a/src/cpp/server/server_posix.cc +++ b/src/cpp/server/server_posix.cc @@ -42,8 +42,8 @@ namespace grpc { void AddInsecureChannelFromFd(Server* server, int fd) { grpc_server_add_insecure_channel_from_fd( server->c_server(), server->completion_queue()->cq(), fd); +} #endif // GPR_SUPPORT_CHANNELS_FROM_FD -} } // namespace grpc From 41622a8e389e8eda38d6d3bfbf34cbf35f437156 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Mon, 13 Jun 2016 16:43:14 -0700 Subject: [PATCH 0090/1039] Fix tsan failures --- Makefile | 1 + build.yaml | 1 + src/core/lib/iomgr/ev_epoll_linux.c | 27 ++++++++++++++++++++++++++- 3 files changed, 28 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 4d8b060760a..28d6842c76c 100644 --- a/Makefile +++ b/Makefile @@ -200,6 +200,7 @@ LD_tsan = clang LDXX_tsan = clang++ CPPFLAGS_tsan = -O0 -fsanitize=thread -fno-omit-frame-pointer -Wno-unused-command-line-argument -DGPR_NO_DIRECT_SYSCALLS LDFLAGS_tsan = -fsanitize=thread +DEFINES_tsan = _GRPC_TSAN DEFINES_tsan += GRPC_TEST_SLOWDOWN_BUILD_FACTOR=5 VALID_CONFIG_stapprof = 1 diff --git a/build.yaml b/build.yaml index 85b66d985bb..139ab3e8bca 100644 --- a/build.yaml +++ b/build.yaml @@ -3231,6 +3231,7 @@ configs: CPPFLAGS: -O0 -fsanitize=thread -fno-omit-frame-pointer -Wno-unused-command-line-argument -DGPR_NO_DIRECT_SYSCALLS CXX: clang++ + DEFINES: _GRPC_TSAN LD: clang LDFLAGS: -fsanitize=thread LDXX: clang++ diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index a8a874cd4b0..35a15e00c96 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -236,6 +236,17 @@ static grpc_wakeup_fd polling_island_wakeup_fd; static gpr_mu g_pi_freelist_mu; static polling_island *g_pi_freelist = NULL; +#ifdef _GRPC_TSAN +/* Currently TSAN may incorrectly flag data races between epoll_ctl and + epoll_wait for any grpc_fd structs that are added to the epoll set via + epoll_ctl and are returned (within a very short window) via epoll_wait(). + + To work-around this race, we establish a happens-before relation between + the code just-before epoll_ctl() and the code after epoll_wait() by using + this atomic */ +gpr_atm g_epoll_sync; +#endif + /* The caller is expected to hold pi->mu lock before calling this function */ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, size_t fd_count, bool add_fd_refs) { @@ -243,6 +254,11 @@ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, size_t i; struct epoll_event ev; +#ifdef _GRPC_TSAN + /* See the definition of g_epoll_sync for more context */ + gpr_atm_rel_store(&g_epoll_sync, 0); +#endif + for (i = 0; i < fd_count; i++) { ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); ev.data.ptr = fds[i]; @@ -361,6 +377,7 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, } pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); + if (pi->epoll_fd < 0) { gpr_log(GPR_ERROR, "epoll_create1() failed with error: %s", strerror(errno)); @@ -1144,6 +1161,11 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, } } +#ifdef _GRPC_TSAN + /* See the definition of g_poll_sync for more details */ + gpr_atm_acq_load(&g_epoll_sync); +#endif + for (int i = 0; i < ep_rv; ++i) { void *data_ptr = ep_ev[i].data.ptr; if (data_ptr == &grpc_global_wakeup_fd) { @@ -1514,10 +1536,13 @@ const grpc_event_engine_vtable *grpc_init_epoll_linux(void) { return &vtable; } -#else /* defined(GPR_LINUX_EPOLL) */ +#else /* defined(GPR_LINUX_EPOLL) */ +#if defined(GPR_POSIX_SOCKET) +#include "src/core/lib/iomgr/ev_posix.h" /* If GPR_LINUX_EPOLL is not defined, it means epoll is not available. Return * NULL */ const grpc_event_engine_vtable *grpc_init_epoll_linux(void) { return NULL; } +#endif /* defined(GPR_POSIX_SOCKET) */ void grpc_use_signal(int signum) {} #endif /* !defined(GPR_LINUX_EPOLL) */ From ec066b3d32ed484961cd73e84822455da4f65ca3 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Mon, 13 Jun 2016 18:10:23 -0700 Subject: [PATCH 0091/1039] Add http2 status code in error_message if it's not 200 --- .../chttp2/transport/chttp2_transport.c | 2 +- src/core/lib/channel/channel_stack.c | 11 +++++++ src/core/lib/channel/channel_stack.h | 5 ++++ src/core/lib/channel/http_client_filter.c | 8 ++++- src/core/lib/transport/transport.c | 30 +++++++++++++++++++ src/core/lib/transport/transport.h | 5 ++++ 6 files changed, 59 insertions(+), 2 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index 2375a4587fe..38da36c192c 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -943,7 +943,7 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, if (op->cancel_with_status != GRPC_STATUS_OK) { cancel_from_api(exec_ctx, transport_global, stream_global, - op->cancel_with_status, op->optional_close_message); + op->cancel_with_status, op->optional_cancel_message); } if (op->close_with_status != GRPC_STATUS_OK) { diff --git a/src/core/lib/channel/channel_stack.c b/src/core/lib/channel/channel_stack.c index bbba85d80ba..b52b1af83aa 100644 --- a/src/core/lib/channel/channel_stack.c +++ b/src/core/lib/channel/channel_stack.c @@ -266,3 +266,14 @@ void grpc_call_element_send_cancel(grpc_exec_ctx *exec_ctx, op.cancel_with_status = GRPC_STATUS_CANCELLED; grpc_call_next_op(exec_ctx, cur_elem, &op); } + +void grpc_call_element_send_cancel_with_message(grpc_exec_ctx *exec_ctx, + grpc_call_element *cur_elem, + grpc_status_code status, + gpr_slice *optional_message) { + grpc_transport_stream_op op; + memset(&op, 0, sizeof(op)); + grpc_transport_stream_op_add_cancellation_with_message(&op, status, + optional_message); + grpc_call_next_op(exec_ctx, cur_elem, &op); +} diff --git a/src/core/lib/channel/channel_stack.h b/src/core/lib/channel/channel_stack.h index 41dd4a0d8a1..d72c015b677 100644 --- a/src/core/lib/channel/channel_stack.h +++ b/src/core/lib/channel/channel_stack.h @@ -273,6 +273,11 @@ void grpc_call_log_op(char *file, int line, gpr_log_severity severity, void grpc_call_element_send_cancel(grpc_exec_ctx *exec_ctx, grpc_call_element *cur_elem); +void grpc_call_element_send_cancel_with_message(grpc_exec_ctx *exec_ctx, + grpc_call_element *cur_elem, + grpc_status_code status, + gpr_slice *optional_message); + extern int grpc_trace_channel; #define GRPC_CALL_LOG_OP(sev, elem, op) \ diff --git a/src/core/lib/channel/http_client_filter.c b/src/core/lib/channel/http_client_filter.c index 792f0d8ae68..94645bebb24 100644 --- a/src/core/lib/channel/http_client_filter.c +++ b/src/core/lib/channel/http_client_filter.c @@ -76,7 +76,13 @@ static grpc_mdelem *client_recv_filter(void *user_data, grpc_mdelem *md) { if (md == GRPC_MDELEM_STATUS_200) { return NULL; } else if (md->key == GRPC_MDSTR_STATUS) { - grpc_call_element_send_cancel(a->exec_ctx, a->elem); + char *message_string; + gpr_asprintf(&message_string, "Received http2 header with status: %s", + grpc_mdstr_as_c_string(md->value)); + gpr_slice message = gpr_slice_from_copied_string(message_string); + gpr_free(message_string); + grpc_call_element_send_cancel_with_message(a->exec_ctx, a->elem, + GRPC_STATUS_CANCELLED, &message); return NULL; } else if (md == GRPC_MDELEM_CONTENT_TYPE_APPLICATION_SLASH_GRPC) { return NULL; diff --git a/src/core/lib/transport/transport.c b/src/core/lib/transport/transport.c index b65e157a02a..3ed72adab89 100644 --- a/src/core/lib/transport/transport.c +++ b/src/core/lib/transport/transport.c @@ -164,6 +164,7 @@ void grpc_transport_stream_op_add_cancellation(grpc_transport_stream_op *op, GPR_ASSERT(status != GRPC_STATUS_OK); if (op->cancel_with_status == GRPC_STATUS_OK) { op->cancel_with_status = status; + op->optional_cancel_message = NULL; } if (op->close_with_status != GRPC_STATUS_OK) { op->close_with_status = GRPC_STATUS_OK; @@ -189,6 +190,35 @@ static void free_message(grpc_exec_ctx *exec_ctx, void *p, bool iomgr_success) { gpr_free(cmd); } +void grpc_transport_stream_op_add_cancellation_with_message( + grpc_transport_stream_op *op, grpc_status_code status, + gpr_slice *optional_message) { + close_message_data *cmd; + GPR_ASSERT(status != GRPC_STATUS_OK); + if (op->cancel_with_status != GRPC_STATUS_OK) { + if (optional_message) { + gpr_slice_unref(*optional_message); + } + return; + } + if (optional_message) { + cmd = gpr_malloc(sizeof(*cmd)); + cmd->message = *optional_message; + cmd->then_call = op->on_complete; + grpc_closure_init(&cmd->closure, free_message, cmd); + op->on_complete = &cmd->closure; + op->optional_cancel_message = &cmd->message; + } + op->cancel_with_status = status; + if (op->close_with_status != GRPC_STATUS_OK) { + op->close_with_status = GRPC_STATUS_OK; + if (op->optional_close_message != NULL) { + gpr_slice_unref(*op->optional_close_message); + op->optional_close_message = NULL; + } + } +} + void grpc_transport_stream_op_add_close(grpc_transport_stream_op *op, grpc_status_code status, gpr_slice *optional_message) { diff --git a/src/core/lib/transport/transport.h b/src/core/lib/transport/transport.h index ed06fc3ed21..9e528ce8cbf 100644 --- a/src/core/lib/transport/transport.h +++ b/src/core/lib/transport/transport.h @@ -137,6 +137,7 @@ typedef struct grpc_transport_stream_op { /** If != GRPC_STATUS_OK, cancel this stream */ grpc_status_code cancel_with_status; + gpr_slice *optional_cancel_message; /** If != GRPC_STATUS_OK, send grpc-status, grpc-message, and close this stream for both reading and writing */ @@ -221,6 +222,10 @@ void grpc_transport_stream_op_finish_with_failure(grpc_exec_ctx *exec_ctx, void grpc_transport_stream_op_add_cancellation(grpc_transport_stream_op *op, grpc_status_code status); +void grpc_transport_stream_op_add_cancellation_with_message( + grpc_transport_stream_op *op, grpc_status_code status, + gpr_slice *optional_message); + void grpc_transport_stream_op_add_close(grpc_transport_stream_op *op, grpc_status_code status, gpr_slice *optional_message); From ad2c4778fc560f10f38550428189c97c9e2bc5a1 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Mon, 13 Jun 2016 19:06:54 -0700 Subject: [PATCH 0092/1039] Rename _GRPC_TSAN to GRPC_TSAN --- Makefile | 2 +- build.yaml | 2 +- src/core/lib/iomgr/ev_epoll_linux.c | 12 ++++++------ 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Makefile b/Makefile index 040ebd4102b..e35e360785b 100644 --- a/Makefile +++ b/Makefile @@ -200,7 +200,7 @@ LD_tsan = clang LDXX_tsan = clang++ CPPFLAGS_tsan = -O0 -fsanitize=thread -fno-omit-frame-pointer -Wno-unused-command-line-argument -DGPR_NO_DIRECT_SYSCALLS LDFLAGS_tsan = -fsanitize=thread -DEFINES_tsan = _GRPC_TSAN +DEFINES_tsan = GRPC_TSAN DEFINES_tsan += GRPC_TEST_SLOWDOWN_BUILD_FACTOR=5 VALID_CONFIG_stapprof = 1 diff --git a/build.yaml b/build.yaml index 0847232b504..3d327f8bffd 100644 --- a/build.yaml +++ b/build.yaml @@ -3266,7 +3266,7 @@ configs: CPPFLAGS: -O0 -fsanitize=thread -fno-omit-frame-pointer -Wno-unused-command-line-argument -DGPR_NO_DIRECT_SYSCALLS CXX: clang++ - DEFINES: _GRPC_TSAN + DEFINES: GRPC_TSAN LD: clang LDFLAGS: -fsanitize=thread LDXX: clang++ diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 35a15e00c96..006c2a8ee7f 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -236,7 +236,7 @@ static grpc_wakeup_fd polling_island_wakeup_fd; static gpr_mu g_pi_freelist_mu; static polling_island *g_pi_freelist = NULL; -#ifdef _GRPC_TSAN +#ifdef GRPC_TSAN /* Currently TSAN may incorrectly flag data races between epoll_ctl and epoll_wait for any grpc_fd structs that are added to the epoll set via epoll_ctl and are returned (within a very short window) via epoll_wait(). @@ -245,7 +245,7 @@ static polling_island *g_pi_freelist = NULL; the code just-before epoll_ctl() and the code after epoll_wait() by using this atomic */ gpr_atm g_epoll_sync; -#endif +#endif /* defined(GRPC_TSAN) */ /* The caller is expected to hold pi->mu lock before calling this function */ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, @@ -254,10 +254,10 @@ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, size_t i; struct epoll_event ev; -#ifdef _GRPC_TSAN +#ifdef GRPC_TSAN /* See the definition of g_epoll_sync for more context */ gpr_atm_rel_store(&g_epoll_sync, 0); -#endif +#endif /* defined(GRPC_TSAN) */ for (i = 0; i < fd_count; i++) { ev.events = (uint32_t)(EPOLLIN | EPOLLOUT | EPOLLET); @@ -1161,10 +1161,10 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, } } -#ifdef _GRPC_TSAN +#ifdef GRPC_TSAN /* See the definition of g_poll_sync for more details */ gpr_atm_acq_load(&g_epoll_sync); -#endif +#endif /* defined(GRPC_TSAN) */ for (int i = 0; i < ep_rv; ++i) { void *data_ptr = ep_ev[i].data.ptr; From 88651de8a713b068eaf499d36bf0b67c0cc9e8e3 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Tue, 14 Jun 2016 16:30:58 -0700 Subject: [PATCH 0093/1039] Fixes #2646 Pass NULL in the host parameter of grpc_channel_create_call --- src/objective-c/GRPCClient/private/GRPCChannel.m | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/objective-c/GRPCClient/private/GRPCChannel.m b/src/objective-c/GRPCClient/private/GRPCChannel.m index d3192c983d1..6cd4cb0ca30 100644 --- a/src/objective-c/GRPCClient/private/GRPCChannel.m +++ b/src/objective-c/GRPCClient/private/GRPCChannel.m @@ -199,9 +199,7 @@ grpc_channel_args * buildChannelArgs(NSDictionary *dictionary) { NULL, GRPC_PROPAGATE_DEFAULTS, queue.unmanagedQueue, path.UTF8String, - // Get "host" from "host:port" - // TODO(jcanizales): Use NSURLs throughout, to clarify these. - [_host componentsSeparatedByString:@":"][0].UTF8String, + NULL, gpr_inf_future(GPR_CLOCK_REALTIME), NULL); } From cf4205dff54d8e3920f98dac28049770b5f8f044 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 15 Jun 2016 11:22:20 -0700 Subject: [PATCH 0094/1039] Compilation error --- src/core/lib/iomgr/ev_epoll_linux.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 006c2a8ee7f..1fb59474640 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -317,7 +317,7 @@ static void polling_island_remove_all_fds_locked(polling_island *pi, if (err < 0 && errno != ENOENT) { /* TODO: sreek - We need a better way to bubble up this error instead of * just logging a message */ - gpr_log(GPR_ERROR, "epoll_ctl deleting fds[%d]: %d failed with error: %s", + gpr_log(GPR_ERROR, "epoll_ctl deleting fds[%zu]: %d failed with error: %s", i, pi->fds[i]->fd, strerror(errno)); } From ec0bc8b4ed4760ff0ab1e51d505f1b235fc9d60d Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Wed, 15 Jun 2016 14:02:57 -0700 Subject: [PATCH 0095/1039] Initial attempt at a C++ API for defining channel filters. --- BUILD | 4 + Makefile | 4 + build.yaml | 2 + include/grpc++/channel_filter.h | 211 ++++++++++++++++++ src/cpp/common/channel_filter.cc | 98 ++++++++ tools/doxygen/Doxyfile.c++ | 1 + tools/doxygen/Doxyfile.c++.internal | 2 + tools/run_tests/sources_and_headers.json | 3 + vsprojects/vcxproj/grpc++/grpc++.vcxproj | 3 + .../vcxproj/grpc++/grpc++.vcxproj.filters | 6 + .../grpc++_unsecure/grpc++_unsecure.vcxproj | 3 + .../grpc++_unsecure.vcxproj.filters | 6 + 12 files changed, 343 insertions(+) create mode 100644 include/grpc++/channel_filter.h create mode 100644 src/cpp/common/channel_filter.cc diff --git a/BUILD b/BUILD index f049e3c4056..2940fed33f2 100644 --- a/BUILD +++ b/BUILD @@ -1247,6 +1247,7 @@ cc_library( "src/cpp/client/generic_stub.cc", "src/cpp/client/insecure_credentials.cc", "src/cpp/common/channel_arguments.cc", + "src/cpp/common/channel_filter.cc", "src/cpp/common/completion_queue.cc", "src/cpp/common/core_codegen.cc", "src/cpp/common/rpc_method.cc", @@ -1269,6 +1270,7 @@ cc_library( hdrs = [ "include/grpc++/alarm.h", "include/grpc++/channel.h", + "include/grpc++/channel_filter.h", "include/grpc++/client_context.h", "include/grpc++/completion_queue.h", "include/grpc++/create_channel.h", @@ -1473,6 +1475,7 @@ cc_library( "src/cpp/client/generic_stub.cc", "src/cpp/client/insecure_credentials.cc", "src/cpp/common/channel_arguments.cc", + "src/cpp/common/channel_filter.cc", "src/cpp/common/completion_queue.cc", "src/cpp/common/core_codegen.cc", "src/cpp/common/rpc_method.cc", @@ -1495,6 +1498,7 @@ cc_library( hdrs = [ "include/grpc++/alarm.h", "include/grpc++/channel.h", + "include/grpc++/channel_filter.h", "include/grpc++/client_context.h", "include/grpc++/completion_queue.h", "include/grpc++/create_channel.h", diff --git a/Makefile b/Makefile index 6eccd06952c..2f88411c95b 100644 --- a/Makefile +++ b/Makefile @@ -3463,6 +3463,7 @@ LIBGRPC++_SRC = \ src/cpp/client/generic_stub.cc \ src/cpp/client/insecure_credentials.cc \ src/cpp/common/channel_arguments.cc \ + src/cpp/common/channel_filter.cc \ src/cpp/common/completion_queue.cc \ src/cpp/common/core_codegen.cc \ src/cpp/common/rpc_method.cc \ @@ -3485,6 +3486,7 @@ LIBGRPC++_SRC = \ PUBLIC_HEADERS_CXX += \ include/grpc++/alarm.h \ include/grpc++/channel.h \ + include/grpc++/channel_filter.h \ include/grpc++/client_context.h \ include/grpc++/completion_queue.h \ include/grpc++/create_channel.h \ @@ -3950,6 +3952,7 @@ LIBGRPC++_UNSECURE_SRC = \ src/cpp/client/generic_stub.cc \ src/cpp/client/insecure_credentials.cc \ src/cpp/common/channel_arguments.cc \ + src/cpp/common/channel_filter.cc \ src/cpp/common/completion_queue.cc \ src/cpp/common/core_codegen.cc \ src/cpp/common/rpc_method.cc \ @@ -3972,6 +3975,7 @@ LIBGRPC++_UNSECURE_SRC = \ PUBLIC_HEADERS_CXX += \ include/grpc++/alarm.h \ include/grpc++/channel.h \ + include/grpc++/channel_filter.h \ include/grpc++/client_context.h \ include/grpc++/completion_queue.h \ include/grpc++/create_channel.h \ diff --git a/build.yaml b/build.yaml index a83ebc595ad..b6b1f758532 100644 --- a/build.yaml +++ b/build.yaml @@ -634,6 +634,7 @@ filegroups: public_headers: - include/grpc++/alarm.h - include/grpc++/channel.h + - include/grpc++/channel_filter.h - include/grpc++/client_context.h - include/grpc++/completion_queue.h - include/grpc++/create_channel.h @@ -693,6 +694,7 @@ filegroups: - src/cpp/client/generic_stub.cc - src/cpp/client/insecure_credentials.cc - src/cpp/common/channel_arguments.cc + - src/cpp/common/channel_filter.cc - src/cpp/common/completion_queue.cc - src/cpp/common/core_codegen.cc - src/cpp/common/rpc_method.cc diff --git a/include/grpc++/channel_filter.h b/include/grpc++/channel_filter.h new file mode 100644 index 00000000000..8067ca9c603 --- /dev/null +++ b/include/grpc++/channel_filter.h @@ -0,0 +1,211 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#ifndef GRPCXX_CHANNEL_FILTER_H +#define GRPCXX_CHANNEL_FILTER_H + +#include + +#include + +#include "src/core/lib/channel/channel_stack.h" +#include "src/core/lib/surface/channel_init.h" + +// +// An interface to define filters. +// +// To define a filter, implement a subclass of each of CallData and +// ChannelData. Then register the filter like this: +// RegisterChannelFilter( +// "name-of-filter", GRPC_SERVER_CHANNEL, INT_MAX); +// + +namespace grpc { + +// Represents call data. +// Note: Must be copyable. +class CallData { + public: + // Do not override the destructor. Instead, put clean-up code in the + // Destroy() method. + virtual ~CallData() {} + + virtual void Destroy() {} + + virtual void StartTransportStreamOp( + grpc_exec_ctx *exec_ctx, grpc_call_element *elem, + grpc_transport_stream_op *op); + + virtual void SetPollsetOrPollsetSet( + grpc_exec_ctx *exec_ctx, grpc_call_element *elem, + grpc_polling_entity *pollent); + + virtual char* GetPeer(grpc_exec_ctx *exec_ctx, grpc_call_element *elem); + + protected: + CallData() {} +}; + +// Represents channel data. +// Note: Must be copyable. +class ChannelData { + public: + // Do not override the destructor. Instead, put clean-up code in the + // Destroy() method. + virtual ~ChannelData() {} + + virtual void Destroy() {} + + virtual void StartTransportOp( + grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, + grpc_transport_op *op); + + protected: + ChannelData() {} +}; + +namespace internal { + +// Defines static members for passing to C core. +template +class ChannelFilter { + static const size_t call_data_size = sizeof(CallDataType); + + static void InitCallElement( + grpc_exec_ctx *exec_ctx, grpc_call_element *elem, + grpc_call_element_args *args) { + CallDataType* call_data = elem->call_data; + *call_data = CallDataType(); + } + + static void DestroyCallElement( + grpc_exec_ctx *exec_ctx, grpc_call_element *elem, + const grpc_call_stats *stats, void *and_free_memory) { + CallDataType* call_data = elem->call_data; + // Can't destroy the object here, since it's not allocated by + // itself; instead, it's part of a larger allocation managed by + // C-core. So instead, we call the Destroy() method. + call_data->Destroy(); + } + + static void StartTransportStreamOp( + grpc_exec_ctx *exec_ctx, grpc_call_element *elem, + grpc_transport_stream_op *op) { + CallDataType* call_data = elem->call_data; + call_data->StartTransportStreamOp(exec_ctx, op); + } + + static void SetPollsetOrPollsetSet( + grpc_exec_ctx *exec_ctx, grpc_call_element *elem, + grpc_polling_entity *pollent) { + CallDataType* call_data = elem->call_data; + call_data->SetPollsetOrPollsetSet(exec_ctx, pollent); + } + + static char* GetPeer( + grpc_exec_ctx *exec_ctx, grpc_call_element *elem) { + CallDataType* call_data = elem->call_data; + return call_data->GetPeer(exec_ctx); + } + + static const size_t channel_data_size = sizeof(ChannelDataType); + + static void InitChannelElement( + grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, + grpc_channel_element_args *args) { + ChannelDataType* channel_data = elem->channel_data; + *channel_data = ChannelDataType(); + } + + static void DestroyChannelElement( + grpc_exec_ctx *exec_ctx, grpc_channel_element *elem) { + ChannelDataType* channel_data = elem->channel_data; + // Can't destroy the object here, since it's not allocated by + // itself; instead, it's part of a larger allocation managed by + // C-core. So instead, we call the Destroy() method. + channel_data->Destroy(); + } + + static void StartTransportOp( + grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, + grpc_transport_op *op) { + ChannelDataType* channel_data = elem->channel_data; + channel_data->StartTransportOp(exec_ctx, op); + } +}; + +struct FilterRecord { + grpc_channel_stack_type stack_type; + int priority; + grpc_channel_filter filter; +}; +extern std::vector* channel_filters; + +void ChannelFilterPluginInit(); +void ChannelFilterPluginShutdown() {} + +} // namespace internal + +// Registers a new filter. +// Must be called by only one thread at a time. +template +void RegisterChannelFilter(const char* name, + grpc_channel_stack_type stack_type, int priority) { + // If we haven't been called before, initialize channel_filters and + // call grpc_register_plugin(). + if (internal::channel_filters == nullptr) { + grpc_register_plugin(internal::ChannelFilterPluginInit, + internal::ChannelFilterPluginShutdown); + internal::channel_filters = new std::vector(); + } + // Add an entry to channel_filters. The filter will be added when the + // C-core initialization code calls ChannelFilterPluginInit(). + typedef internal::ChannelFilter FilterType; + internal::channel_filters->emplace_back({ + stack_type, priority, { + FilterType::StartTransportStreamOp, + FilterType::StartTransportOp, + FilterType::call_data_size, + FilterType::InitCallElement, + FilterType::SetPollsetOrPollsetSet, + FilterType::DestroyCallElement, + FilterType::channel_data_size, + FilterType::InitChannelElement, + FilterType::DestroyChannelElement, + FilterType::GetPeer, + name}}); +} + +} // namespace grpc + +#endif // GRPCXX_CHANNEL_FILTER_H diff --git a/src/cpp/common/channel_filter.cc b/src/cpp/common/channel_filter.cc new file mode 100644 index 00000000000..9a80195e867 --- /dev/null +++ b/src/cpp/common/channel_filter.cc @@ -0,0 +1,98 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include + +#include "src/core/lib/channel/channel_stack.h" + +namespace grpc { + +// +// CallData +// + +void CallData::StartTransportStreamOp( + grpc_exec_ctx *exec_ctx, grpc_call_element *elem, + grpc_transport_stream_op *op) { + grpc_call_next_op(exec_ctx, elem, op); +} + +void CallData::SetPollsetOrPollsetSet( + grpc_exec_ctx *exec_ctx, grpc_call_element *elem, + grpc_polling_entity *pollent) { + grpc_call_stack_ignore_set_pollset_or_pollset_set(exec_ctx, elem, pollent); +} + +char* CallData::GetPeer( + grpc_exec_ctx *exec_ctx, grpc_call_element *elem) { + return grpc_call_next_get_peer(exec_ctx, elem); +} + +// +// ChannelData +// + +void ChannelData::StartTransportOp( + grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, + grpc_transport_op *op) { + grpc_channel_next_op(exec_ctx, elem, op); +} + +// +// RegisterChannelFilter() +// + +namespace internal { + +std::vector* channel_filters = nullptr; + +namespace { + +bool AppendFilter(grpc_channel_stack_builder* builder, void* arg) { + return grpc_channel_stack_builder_append_filter( + builder, (const grpc_channel_filter *)arg, nullptr, nullptr); +} + +} // namespace + +void ChannelFilterPluginInit() { + for (size_t i = 0; i < channel_filters->size(); ++i) { + FilterRecord& filter = (*channel_filters)[i]; + grpc_channel_init_register_stage(filter.stack_type, filter.priority, + AppendFilter, (void*)&filter.filter); + } +} + +} // namespace internal + +} // namespace grpc diff --git a/tools/doxygen/Doxyfile.c++ b/tools/doxygen/Doxyfile.c++ index 7f9d2df6f6c..e770574cb1f 100644 --- a/tools/doxygen/Doxyfile.c++ +++ b/tools/doxygen/Doxyfile.c++ @@ -762,6 +762,7 @@ WARN_LOGFILE = INPUT = include/grpc++/alarm.h \ include/grpc++/channel.h \ +include/grpc++/channel_filter.h \ include/grpc++/client_context.h \ include/grpc++/completion_queue.h \ include/grpc++/create_channel.h \ diff --git a/tools/doxygen/Doxyfile.c++.internal b/tools/doxygen/Doxyfile.c++.internal index dcf1a4c8c40..a3c4a109264 100644 --- a/tools/doxygen/Doxyfile.c++.internal +++ b/tools/doxygen/Doxyfile.c++.internal @@ -762,6 +762,7 @@ WARN_LOGFILE = INPUT = include/grpc++/alarm.h \ include/grpc++/channel.h \ +include/grpc++/channel_filter.h \ include/grpc++/client_context.h \ include/grpc++/completion_queue.h \ include/grpc++/create_channel.h \ @@ -880,6 +881,7 @@ src/cpp/client/credentials.cc \ src/cpp/client/generic_stub.cc \ src/cpp/client/insecure_credentials.cc \ src/cpp/common/channel_arguments.cc \ +src/cpp/common/channel_filter.cc \ src/cpp/common/completion_queue.cc \ src/cpp/common/core_codegen.cc \ src/cpp/common/rpc_method.cc \ diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 4aad52c69d0..00ea2fdc058 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -6497,6 +6497,7 @@ "headers": [ "include/grpc++/alarm.h", "include/grpc++/channel.h", + "include/grpc++/channel_filter.h", "include/grpc++/client_context.h", "include/grpc++/completion_queue.h", "include/grpc++/create_channel.h", @@ -6551,6 +6552,7 @@ "src": [ "include/grpc++/alarm.h", "include/grpc++/channel.h", + "include/grpc++/channel_filter.h", "include/grpc++/client_context.h", "include/grpc++/completion_queue.h", "include/grpc++/create_channel.h", @@ -6606,6 +6608,7 @@ "src/cpp/client/generic_stub.cc", "src/cpp/client/insecure_credentials.cc", "src/cpp/common/channel_arguments.cc", + "src/cpp/common/channel_filter.cc", "src/cpp/common/completion_queue.cc", "src/cpp/common/core_codegen.cc", "src/cpp/common/rpc_method.cc", diff --git a/vsprojects/vcxproj/grpc++/grpc++.vcxproj b/vsprojects/vcxproj/grpc++/grpc++.vcxproj index cb9e41ea22f..b882c302bbb 100644 --- a/vsprojects/vcxproj/grpc++/grpc++.vcxproj +++ b/vsprojects/vcxproj/grpc++/grpc++.vcxproj @@ -260,6 +260,7 @@ + @@ -397,6 +398,8 @@ + + diff --git a/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters b/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters index a9051182b3c..08fffb74b2f 100644 --- a/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters +++ b/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters @@ -46,6 +46,9 @@ src\cpp\common + + src\cpp\common + src\cpp\common @@ -108,6 +111,9 @@ include\grpc++ + + include\grpc++ + include\grpc++ diff --git a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj index 03be485b297..b5a27f624d1 100644 --- a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj @@ -260,6 +260,7 @@ + @@ -383,6 +384,8 @@ + + diff --git a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters index ba99bc53c8c..68d9a47973d 100644 --- a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters +++ b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters @@ -31,6 +31,9 @@ src\cpp\common + + src\cpp\common + src\cpp\common @@ -93,6 +96,9 @@ include\grpc++ + + include\grpc++ + include\grpc++ From 635c1caf3b943d634f43a5bec4c16b937c595782 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Wed, 15 Jun 2016 14:30:03 -0700 Subject: [PATCH 0096/1039] Fixed ruby timeout_on_sleeping_server implementation --- src/ruby/pb/test/client.rb | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/src/ruby/pb/test/client.rb b/src/ruby/pb/test/client.rb index b6695482a28..146623e0abf 100755 --- a/src/ruby/pb/test/client.rb +++ b/src/ruby/pb/test/client.rb @@ -197,6 +197,25 @@ class PingPongPlayer end end +class BlockingEnumerator + include Grpc::Testing + include Grpc::Testing::PayloadType + + def initialize(req_size, sleep_time) + @req_size = req_size + @sleep_time = sleep_time + end + + def each_item + return enum_for(:each_item) unless block_given? + req_cls = StreamingOutputCallRequest + req = req_cls.new(payload: Payload.new(body: nulls(@req_size))) + yield req + # Sleep until after the deadline should have passed + sleep(@sleep_time) + end +end + # defines methods corresponding to each interop test case. class NamedTests include Grpc::Testing @@ -315,11 +334,10 @@ class NamedTests end def timeout_on_sleeping_server - msg_sizes = [[27_182, 31_415]] - ppp = PingPongPlayer.new(msg_sizes) - deadline = GRPC::Core::TimeConsts::from_relative_time(0.001) - resps = @stub.full_duplex_call(ppp.each_item, deadline: deadline) - resps.each { |r| ppp.queue.push(r) } + enum = BlockingEnumerator.new(27_182, 2) + deadline = GRPC::Core::TimeConsts::from_relative_time(1) + resps = @stub.full_duplex_call(enum.each_item, deadline: deadline) + resps.each { } # wait to receive each request (or timeout) fail 'Should have raised GRPC::BadStatus(DEADLINE_EXCEEDED)' rescue GRPC::BadStatus => e assert("#{__callee__}: status was wrong") do From 5fe21a11bb8c3b5af62094ae9932ebb0d45b3f2b Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Wed, 15 Jun 2016 16:00:00 -0700 Subject: [PATCH 0097/1039] Fix run_interop_tests override_server option --- tools/run_tests/run_interop_tests.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tools/run_tests/run_interop_tests.py b/tools/run_tests/run_interop_tests.py index edbdf05e2a2..81ba729d315 100755 --- a/tools/run_tests/run_interop_tests.py +++ b/tools/run_tests/run_interop_tests.py @@ -465,7 +465,8 @@ def cloud_to_prod_jobspec(language, test_case, server_host_name, flake_retries=5 if args.allow_flakes else 0, timeout_retries=2 if args.allow_flakes else 0, kill_handler=_job_kill_handler) - test_job.container_name = container_name + if docker_image: + test_job.container_name = container_name return test_job @@ -501,7 +502,8 @@ def cloud_to_cloud_jobspec(language, test_case, server_name, server_host, flake_retries=5 if args.allow_flakes else 0, timeout_retries=2 if args.allow_flakes else 0, kill_handler=_job_kill_handler) - test_job.container_name = container_name + if docker_image: + test_job.container_name = container_name return test_job From 9f6a80597771eae74760b23a66737eb8960f8fd7 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Wed, 15 Jun 2016 17:39:03 -0700 Subject: [PATCH 0098/1039] Allow disabling traces, add trace variables for pluck and timeout events --- BUILD | 4 -- build.yaml | 1 - gRPC.podspec | 2 - grpc.gemspec | 1 - package.xml | 1 - src/core/lib/debug/trace.c | 8 +++- src/core/lib/surface/call.h | 1 - src/core/lib/surface/completion_queue.c | 29 +++++++---- src/core/lib/surface/completion_queue.h | 3 ++ src/core/lib/surface/init.c | 7 ++- src/core/lib/surface/surface_trace.h | 48 ------------------- tools/doxygen/Doxyfile.core.internal | 1 - tools/run_tests/sources_and_headers.json | 2 - vsprojects/vcxproj/grpc/grpc.vcxproj | 1 - vsprojects/vcxproj/grpc/grpc.vcxproj.filters | 3 -- .../grpc_unsecure/grpc_unsecure.vcxproj | 1 - .../grpc_unsecure.vcxproj.filters | 3 -- 17 files changed, 36 insertions(+), 80 deletions(-) delete mode 100644 src/core/lib/surface/surface_trace.h diff --git a/BUILD b/BUILD index f049e3c4056..bf001ac7273 100644 --- a/BUILD +++ b/BUILD @@ -228,7 +228,6 @@ cc_library( "src/core/lib/surface/init.h", "src/core/lib/surface/lame_client.h", "src/core/lib/surface/server.h", - "src/core/lib/surface/surface_trace.h", "src/core/lib/transport/byte_stream.h", "src/core/lib/transport/connectivity_state.h", "src/core/lib/transport/metadata.h", @@ -606,7 +605,6 @@ cc_library( "src/core/lib/surface/init.h", "src/core/lib/surface/lame_client.h", "src/core/lib/surface/server.h", - "src/core/lib/surface/surface_trace.h", "src/core/lib/transport/byte_stream.h", "src/core/lib/transport/connectivity_state.h", "src/core/lib/transport/metadata.h", @@ -949,7 +947,6 @@ cc_library( "src/core/lib/surface/init.h", "src/core/lib/surface/lame_client.h", "src/core/lib/surface/server.h", - "src/core/lib/surface/surface_trace.h", "src/core/lib/transport/byte_stream.h", "src/core/lib/transport/connectivity_state.h", "src/core/lib/transport/metadata.h", @@ -2061,7 +2058,6 @@ objc_library( "src/core/lib/surface/init.h", "src/core/lib/surface/lame_client.h", "src/core/lib/surface/server.h", - "src/core/lib/surface/surface_trace.h", "src/core/lib/transport/byte_stream.h", "src/core/lib/transport/connectivity_state.h", "src/core/lib/transport/metadata.h", diff --git a/build.yaml b/build.yaml index a83ebc595ad..11cc9347cb0 100644 --- a/build.yaml +++ b/build.yaml @@ -224,7 +224,6 @@ filegroups: - src/core/lib/surface/init.h - src/core/lib/surface/lame_client.h - src/core/lib/surface/server.h - - src/core/lib/surface/surface_trace.h - src/core/lib/transport/byte_stream.h - src/core/lib/transport/connectivity_state.h - src/core/lib/transport/metadata.h diff --git a/gRPC.podspec b/gRPC.podspec index d3665d51744..622be89ca74 100644 --- a/gRPC.podspec +++ b/gRPC.podspec @@ -231,7 +231,6 @@ Pod::Spec.new do |s| 'src/core/lib/surface/init.h', 'src/core/lib/surface/lame_client.h', 'src/core/lib/surface/server.h', - 'src/core/lib/surface/surface_trace.h', 'src/core/lib/transport/byte_stream.h', 'src/core/lib/transport/connectivity_state.h', 'src/core/lib/transport/metadata.h', @@ -606,7 +605,6 @@ Pod::Spec.new do |s| 'src/core/lib/surface/init.h', 'src/core/lib/surface/lame_client.h', 'src/core/lib/surface/server.h', - 'src/core/lib/surface/surface_trace.h', 'src/core/lib/transport/byte_stream.h', 'src/core/lib/transport/connectivity_state.h', 'src/core/lib/transport/metadata.h', diff --git a/grpc.gemspec b/grpc.gemspec index 9f3ae048a70..e0a8ff00a42 100755 --- a/grpc.gemspec +++ b/grpc.gemspec @@ -240,7 +240,6 @@ Gem::Specification.new do |s| s.files += %w( src/core/lib/surface/init.h ) s.files += %w( src/core/lib/surface/lame_client.h ) s.files += %w( src/core/lib/surface/server.h ) - s.files += %w( src/core/lib/surface/surface_trace.h ) s.files += %w( src/core/lib/transport/byte_stream.h ) s.files += %w( src/core/lib/transport/connectivity_state.h ) s.files += %w( src/core/lib/transport/metadata.h ) diff --git a/package.xml b/package.xml index b25e42c1c01..fef06ef9eed 100644 --- a/package.xml +++ b/package.xml @@ -247,7 +247,6 @@ - diff --git a/src/core/lib/debug/trace.c b/src/core/lib/debug/trace.c index 555f497b784..c56046785b3 100644 --- a/src/core/lib/debug/trace.c +++ b/src/core/lib/debug/trace.c @@ -88,7 +88,11 @@ static void parse(const char *s) { split(s, &strings, &nstrings); for (i = 0; i < nstrings; i++) { - grpc_tracer_set_enabled(strings[i], 1); + if (strings[i][0] == '-') { + grpc_tracer_set_enabled(strings[i] + 1, 0); + } else { + grpc_tracer_set_enabled(strings[i], 1); + } } for (i = 0; i < nstrings; i++) { @@ -117,7 +121,7 @@ int grpc_tracer_set_enabled(const char *name, int enabled) { tracer *t; if (0 == strcmp(name, "all")) { for (t = tracers; t; t = t->next) { - *t->flag = 1; + *t->flag = enabled; } } else { int found = 0; diff --git a/src/core/lib/surface/call.h b/src/core/lib/surface/call.h index b640345c21d..3a78fe3aa36 100644 --- a/src/core/lib/surface/call.h +++ b/src/core/lib/surface/call.h @@ -37,7 +37,6 @@ #include "src/core/lib/channel/channel_stack.h" #include "src/core/lib/channel/context.h" #include "src/core/lib/surface/api_trace.h" -#include "src/core/lib/surface/surface_trace.h" #include #include diff --git a/src/core/lib/surface/completion_queue.c b/src/core/lib/surface/completion_queue.c index 5eb7cf1bf4f..de2b674d0e0 100644 --- a/src/core/lib/surface/completion_queue.c +++ b/src/core/lib/surface/completion_queue.c @@ -48,7 +48,6 @@ #include "src/core/lib/surface/api_trace.h" #include "src/core/lib/surface/call.h" #include "src/core/lib/surface/event_string.h" -#include "src/core/lib/surface/surface_trace.h" typedef struct { grpc_pollset_worker **worker; @@ -91,6 +90,18 @@ struct grpc_completion_queue { static gpr_mu g_freelist_mu; static grpc_completion_queue *g_freelist; +int grpc_cq_pluck_trace; +int grpc_cq_event_timeout_trace; + +#define GRPC_SURFACE_TRACE_RETURNED_EVENT(cq, event) \ + if (grpc_api_trace && \ + (grpc_cq_pluck_trace || (event)->type != GRPC_QUEUE_TIMEOUT)) { \ + char *_ev = grpc_event_string(event); \ + gpr_log(GPR_INFO, "RETURN_EVENT[%p]: %s", cq, _ev); \ + gpr_free(_ev); \ + } + + static void on_pollset_shutdown_done(grpc_exec_ctx *exec_ctx, void *cc, bool success); @@ -396,13 +407,15 @@ grpc_event grpc_completion_queue_pluck(grpc_completion_queue *cc, void *tag, GPR_TIMER_BEGIN("grpc_completion_queue_pluck", 0); - GRPC_API_TRACE( - "grpc_completion_queue_pluck(" - "cc=%p, tag=%p, " - "deadline=gpr_timespec { tv_sec: %lld, tv_nsec: %d, clock_type: %d }, " - "reserved=%p)", - 6, (cc, tag, (long long)deadline.tv_sec, (int)deadline.tv_nsec, - (int)deadline.clock_type, reserved)); + if (grpc_cq_pluck_trace) { + GRPC_API_TRACE( + "grpc_completion_queue_pluck(" + "cc=%p, tag=%p, " + "deadline=gpr_timespec { tv_sec: %lld, tv_nsec: %d, clock_type: %d }, " + "reserved=%p)", + 6, (cc, tag, (long long)deadline.tv_sec, (int)deadline.tv_nsec, + (int)deadline.clock_type, reserved)); + } GPR_ASSERT(!reserved); deadline = gpr_convert_clock_type(deadline, GPR_CLOCK_MONOTONIC); diff --git a/src/core/lib/surface/completion_queue.h b/src/core/lib/surface/completion_queue.h index 3d0dd13c53b..108445b28f1 100644 --- a/src/core/lib/surface/completion_queue.h +++ b/src/core/lib/surface/completion_queue.h @@ -39,6 +39,9 @@ #include #include "src/core/lib/iomgr/pollset.h" +extern int grpc_cq_pluck_trace; +extern int grpc_cq_event_timeout_trace; + typedef struct grpc_cq_completion { /** user supplied tag */ void *tag; diff --git a/src/core/lib/surface/init.c b/src/core/lib/surface/init.c index 1c8b7090156..499ecffc2ae 100644 --- a/src/core/lib/surface/init.c +++ b/src/core/lib/surface/init.c @@ -57,7 +57,6 @@ #include "src/core/lib/surface/init.h" #include "src/core/lib/surface/lame_client.h" #include "src/core/lib/surface/server.h" -#include "src/core/lib/surface/surface_trace.h" #include "src/core/lib/transport/connectivity_state.h" #include "src/core/lib/transport/transport_impl.h" @@ -165,6 +164,12 @@ void grpc_init(void) { &grpc_trace_channel_stack_builder); grpc_register_tracer("http1", &grpc_http1_trace); grpc_register_tracer("compression", &grpc_compression_trace); + grpc_register_tracer("queue_pluck", &grpc_cq_pluck_trace); + // Default pluck trace to 1 + grpc_cq_pluck_trace = 1; + grpc_register_tracer("queue_timeout", &grpc_cq_event_timeout_trace); + // Default timeout trace to 1 + grpc_cq_event_timeout_trace = 1; grpc_security_pre_init(); grpc_iomgr_init(); grpc_executor_init(); diff --git a/src/core/lib/surface/surface_trace.h b/src/core/lib/surface/surface_trace.h deleted file mode 100644 index a69a0fff577..00000000000 --- a/src/core/lib/surface/surface_trace.h +++ /dev/null @@ -1,48 +0,0 @@ -/* - * - * Copyright 2015, Google Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following disclaimer - * in the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Google Inc. nor the names of its - * contributors may be used to endorse or promote products derived from - * this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - */ - -#ifndef GRPC_CORE_LIB_SURFACE_SURFACE_TRACE_H -#define GRPC_CORE_LIB_SURFACE_SURFACE_TRACE_H - -#include -#include "src/core/lib/debug/trace.h" -#include "src/core/lib/surface/api_trace.h" - -#define GRPC_SURFACE_TRACE_RETURNED_EVENT(cq, event) \ - if (grpc_api_trace) { \ - char *_ev = grpc_event_string(event); \ - gpr_log(GPR_INFO, "RETURN_EVENT[%p]: %s", cq, _ev); \ - gpr_free(_ev); \ - } - -#endif /* GRPC_CORE_LIB_SURFACE_SURFACE_TRACE_H */ diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index aee866adf46..337ce16551a 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -857,7 +857,6 @@ src/core/lib/surface/event_string.h \ src/core/lib/surface/init.h \ src/core/lib/surface/lame_client.h \ src/core/lib/surface/server.h \ -src/core/lib/surface/surface_trace.h \ src/core/lib/transport/byte_stream.h \ src/core/lib/transport/connectivity_state.h \ src/core/lib/transport/metadata.h \ diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 4aad52c69d0..f9c497afcdd 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -5762,7 +5762,6 @@ "src/core/lib/surface/init.h", "src/core/lib/surface/lame_client.h", "src/core/lib/surface/server.h", - "src/core/lib/surface/surface_trace.h", "src/core/lib/transport/byte_stream.h", "src/core/lib/transport/connectivity_state.h", "src/core/lib/transport/metadata.h", @@ -5921,7 +5920,6 @@ "src/core/lib/surface/metadata_array.c", "src/core/lib/surface/server.c", "src/core/lib/surface/server.h", - "src/core/lib/surface/surface_trace.h", "src/core/lib/surface/validate_metadata.c", "src/core/lib/surface/version.c", "src/core/lib/transport/byte_stream.c", diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj b/vsprojects/vcxproj/grpc/grpc.vcxproj index ec0c984f2e7..c69623bf736 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj @@ -366,7 +366,6 @@ - diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters index 273f73b2a30..593024993fa 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters @@ -839,9 +839,6 @@ src\core\lib\surface - - src\core\lib\surface - src\core\lib\transport diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj index bba099d803c..28dcd8269eb 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj @@ -355,7 +355,6 @@ - diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters index 78a8777f942..058ae653cad 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters @@ -749,9 +749,6 @@ src\core\lib\surface - - src\core\lib\surface - src\core\lib\transport From dbe2b9e976d50ce7971df32dc9a3122525dad3ce Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Wed, 15 Jun 2016 20:23:04 -0700 Subject: [PATCH 0099/1039] Enable treating warnings as errors in objc tests --- src/objective-c/tests/GRPCClientTests.m | 16 ++++++++-------- src/objective-c/tests/InteropTests.m | 10 ++++++---- .../tests/Tests.xcodeproj/project.pbxproj | 4 ++++ 3 files changed, 18 insertions(+), 12 deletions(-) diff --git a/src/objective-c/tests/GRPCClientTests.m b/src/objective-c/tests/GRPCClientTests.m index 9a8d4253241..2eca7bf5498 100644 --- a/src/objective-c/tests/GRPCClientTests.m +++ b/src/objective-c/tests/GRPCClientTests.m @@ -48,9 +48,9 @@ static NSString * const kPackage = @"grpc.testing"; static NSString * const kService = @"TestService"; static NSString * const kRemoteSSLHost = @"grpc-test.sandbox.googleapis.com"; -static ProtoMethod *kInexistentMethod; -static ProtoMethod *kEmptyCallMethod; -static ProtoMethod *kUnaryCallMethod; +static GRPCProtoMethod *kInexistentMethod; +static GRPCProtoMethod *kEmptyCallMethod; +static GRPCProtoMethod *kUnaryCallMethod; /** Observer class for testing that responseMetadata is KVO-compliant */ @interface PassthroughObserver : NSObject @@ -109,13 +109,13 @@ static ProtoMethod *kUnaryCallMethod; [GRPCCall useInsecureConnectionsForHost:kHostAddress]; // This method isn't implemented by the remote server. - kInexistentMethod = [[ProtoMethod alloc] initWithPackage:kPackage + kInexistentMethod = [[GRPCProtoMethod alloc] initWithPackage:kPackage service:kService method:@"Inexistent"]; - kEmptyCallMethod = [[ProtoMethod alloc] initWithPackage:kPackage + kEmptyCallMethod = [[GRPCProtoMethod alloc] initWithPackage:kPackage service:kService method:@"EmptyCall"]; - kUnaryCallMethod = [[ProtoMethod alloc] initWithPackage:kPackage + kUnaryCallMethod = [[GRPCProtoMethod alloc] initWithPackage:kPackage service:kService method:@"UnaryCall"]; } @@ -303,7 +303,7 @@ static ProtoMethod *kUnaryCallMethod; // Try to set parameters to nil for GRPCCall. This should cause an exception @try { - GRPCCall *call = [[GRPCCall alloc] initWithHost:nil + GRPCCall *call __unused = [[GRPCCall alloc] initWithHost:nil path:nil requestsWriter:nil]; XCTFail(@"Did not receive an exception when parameters are nil"); @@ -316,7 +316,7 @@ static ProtoMethod *kUnaryCallMethod; GRXWriter *requestsWriter = [GRXWriter emptyWriter]; [requestsWriter finishWithError:nil]; @try { - GRPCCall *call = [[GRPCCall alloc] initWithHost:kHostAddress + GRPCCall *call __unused = [[GRPCCall alloc] initWithHost:kHostAddress path:kUnaryCallMethod.HTTPPath requestsWriter:requestsWriter]; XCTFail(@"Did not receive an exception when GRXWriter has incorrect state."); diff --git a/src/objective-c/tests/InteropTests.m b/src/objective-c/tests/InteropTests.m index 781c500f73e..60a83259fa6 100644 --- a/src/objective-c/tests/InteropTests.m +++ b/src/objective-c/tests/InteropTests.m @@ -58,7 +58,7 @@ requestedResponseSize:(NSNumber *)responseSize { RMTStreamingOutputCallRequest *request = [self message]; RMTResponseParameters *parameters = [RMTResponseParameters message]; - parameters.size = responseSize.integerValue; + parameters.size = (int)responseSize.integerValue; [request.responseParametersArray addObject:parameters]; request.payload.body = [NSMutableData dataWithLength:payloadSize.unsignedIntegerValue]; return request; @@ -80,7 +80,9 @@ #pragma mark Tests +#ifdef GRPC_COMPILE_WITH_CRONET static cronet_engine *cronetEngine = NULL; +#endif @implementation InteropTests { RMTTestService *_service; @@ -186,7 +188,7 @@ static cronet_engine *cronetEngine = NULL; RMTStreamingOutputCallRequest *request = [RMTStreamingOutputCallRequest message]; for (NSNumber *size in expectedSizes) { RMTResponseParameters *parameters = [RMTResponseParameters message]; - parameters.size = [size integerValue]; + parameters.size = (int)[size integerValue]; [request.responseParametersArray addObject:parameters]; } @@ -282,7 +284,7 @@ static cronet_engine *cronetEngine = NULL; // A buffered pipe to which we never write any value acts as a writer that just hangs. GRXBufferedPipe *requestsBuffer = [[GRXBufferedPipe alloc] init]; - ProtoRPC *call = [_service RPCToStreamingInputCallWithRequestsWriter:requestsBuffer + GRPCProtoCall *call = [_service RPCToStreamingInputCallWithRequestsWriter:requestsBuffer handler:^(RMTStreamingInputCallResponse *response, NSError *error) { XCTAssertEqual(error.code, GRPC_STATUS_CANCELLED); @@ -313,7 +315,7 @@ static cronet_engine *cronetEngine = NULL; [requestsBuffer writeValue:request]; - __block ProtoRPC *call = + __block GRPCProtoCall *call = [_service RPCToFullDuplexCallWithRequestsWriter:requestsBuffer eventHandler:^(BOOL done, RMTStreamingOutputCallResponse *response, diff --git a/src/objective-c/tests/Tests.xcodeproj/project.pbxproj b/src/objective-c/tests/Tests.xcodeproj/project.pbxproj index b0429617c01..59a6af2e11e 100644 --- a/src/objective-c/tests/Tests.xcodeproj/project.pbxproj +++ b/src/objective-c/tests/Tests.xcodeproj/project.pbxproj @@ -823,6 +823,7 @@ "$(inherited)", ); GCC_SYMBOLS_PRIVATE_EXTERN = NO; + GCC_TREAT_WARNINGS_AS_ERRORS = YES; GCC_WARN_64_TO_32_BIT_CONVERSION = YES; GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; GCC_WARN_UNDECLARED_SELECTOR = YES; @@ -859,6 +860,7 @@ ENABLE_STRICT_OBJC_MSGSEND = YES; GCC_C_LANGUAGE_STANDARD = gnu99; GCC_NO_COMMON_BLOCKS = YES; + GCC_TREAT_WARNINGS_AS_ERRORS = YES; GCC_WARN_64_TO_32_BIT_CONVERSION = YES; GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; GCC_WARN_UNDECLARED_SELECTOR = YES; @@ -875,6 +877,7 @@ 635697DC1B14FC11007A7283 /* Debug */ = { isa = XCBuildConfiguration; buildSettings = { + GCC_TREAT_WARNINGS_AS_ERRORS = YES; PRODUCT_NAME = "$(TARGET_NAME)"; SKIP_INSTALL = YES; }; @@ -883,6 +886,7 @@ 635697DD1B14FC11007A7283 /* Release */ = { isa = XCBuildConfiguration; buildSettings = { + GCC_TREAT_WARNINGS_AS_ERRORS = YES; PRODUCT_NAME = "$(TARGET_NAME)"; SKIP_INSTALL = YES; }; From 3fb8f7360b033874978e517043c7eec59e295e42 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 15 Jun 2016 22:53:08 -0700 Subject: [PATCH 0100/1039] gRPC LB policy --- BUILD | 3 + Makefile | 38 + binding.gyp | 1 + build.yaml | 14 + config.m4 | 1 + gRPC.podspec | 1 + grpc.gemspec | 1 + package.xml | 1 + src/core/ext/client_config/lb_policy.c | 5 +- src/core/ext/client_config/lb_policy.h | 13 +- src/core/ext/lb_policy/grpclb/grpclb.c | 836 ++++++++++++++++++ src/core/ext/lb_policy/grpclb/grpclb.h | 44 + .../ext/lb_policy/grpclb/load_balancer_api.c | 77 ++ .../ext/lb_policy/grpclb/load_balancer_api.h | 16 + .../plugin_registry/grpc_plugin_registry.c | 4 + .../grpc_unsecure_plugin_registry.c | 4 + src/python/grpcio/grpc_core_dependencies.py | 1 + test/core/client_config/grpclb_test.c | 670 ++++++++++++++ test/core/end2end/cq_verifier.c | 9 +- test/core/end2end/cq_verifier.h | 3 + .../codegen/core/gen_grpclb_test_response.py | 97 ++ tools/doxygen/Doxyfile.core.internal | 1 + tools/run_tests/sources_and_headers.json | 19 + tools/run_tests/tests.json | 21 + vsprojects/buildtests_c.sln | 27 + vsprojects/vcxproj/grpc/grpc.vcxproj | 2 + vsprojects/vcxproj/grpc/grpc.vcxproj.filters | 3 + .../grpc_unsecure/grpc_unsecure.vcxproj | 2 + .../grpc_unsecure.vcxproj.filters | 3 + .../test/grpclb_test/grpclb_test.vcxproj | 199 +++++ .../grpclb_test/grpclb_test.vcxproj.filters | 21 + 31 files changed, 2128 insertions(+), 9 deletions(-) create mode 100644 src/core/ext/lb_policy/grpclb/grpclb.c create mode 100644 src/core/ext/lb_policy/grpclb/grpclb.h create mode 100644 test/core/client_config/grpclb_test.c create mode 100755 tools/codegen/core/gen_grpclb_test_response.py create mode 100644 vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj create mode 100644 vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj.filters diff --git a/BUILD b/BUILD index f049e3c4056..42a35a310d6 100644 --- a/BUILD +++ b/BUILD @@ -467,6 +467,7 @@ cc_library( "src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c", "src/core/ext/transport/chttp2/client/insecure/channel_create.c", "src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c", + "src/core/ext/lb_policy/grpclb/grpclb.c", "src/core/ext/lb_policy/grpclb/load_balancer_api.c", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c", "src/core/ext/lb_policy/pick_first/pick_first.c", @@ -1141,6 +1142,7 @@ cc_library( "src/core/ext/resolver/sockaddr/sockaddr_resolver.c", "src/core/ext/load_reporting/load_reporting.c", "src/core/ext/load_reporting/load_reporting_filter.c", + "src/core/ext/lb_policy/grpclb/grpclb.c", "src/core/ext/lb_policy/grpclb/load_balancer_api.c", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c", "src/core/ext/lb_policy/pick_first/pick_first.c", @@ -1943,6 +1945,7 @@ objc_library( "src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c", "src/core/ext/transport/chttp2/client/insecure/channel_create.c", "src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c", + "src/core/ext/lb_policy/grpclb/grpclb.c", "src/core/ext/lb_policy/grpclb/load_balancer_api.c", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c", "src/core/ext/lb_policy/pick_first/pick_first.c", diff --git a/Makefile b/Makefile index 6eccd06952c..e316d4b1533 100644 --- a/Makefile +++ b/Makefile @@ -946,6 +946,7 @@ grpc_jwt_verifier_test: $(BINDIR)/$(CONFIG)/grpc_jwt_verifier_test grpc_print_google_default_creds_token: $(BINDIR)/$(CONFIG)/grpc_print_google_default_creds_token grpc_security_connector_test: $(BINDIR)/$(CONFIG)/grpc_security_connector_test grpc_verify_jwt: $(BINDIR)/$(CONFIG)/grpc_verify_jwt +grpclb_test: $(BINDIR)/$(CONFIG)/grpclb_test hpack_parser_fuzzer_test: $(BINDIR)/$(CONFIG)/hpack_parser_fuzzer_test hpack_parser_test: $(BINDIR)/$(CONFIG)/hpack_parser_test hpack_table_test: $(BINDIR)/$(CONFIG)/hpack_table_test @@ -1277,6 +1278,7 @@ buildtests_c: privatelibs_c \ $(BINDIR)/$(CONFIG)/grpc_json_token_test \ $(BINDIR)/$(CONFIG)/grpc_jwt_verifier_test \ $(BINDIR)/$(CONFIG)/grpc_security_connector_test \ + $(BINDIR)/$(CONFIG)/grpclb_test \ $(BINDIR)/$(CONFIG)/hpack_parser_test \ $(BINDIR)/$(CONFIG)/hpack_table_test \ $(BINDIR)/$(CONFIG)/http_parser_test \ @@ -1579,6 +1581,8 @@ test_c: buildtests_c $(Q) $(BINDIR)/$(CONFIG)/grpc_jwt_verifier_test || ( echo test grpc_jwt_verifier_test failed ; exit 1 ) $(E) "[RUN] Testing grpc_security_connector_test" $(Q) $(BINDIR)/$(CONFIG)/grpc_security_connector_test || ( echo test grpc_security_connector_test failed ; exit 1 ) + $(E) "[RUN] Testing grpclb_test" + $(Q) $(BINDIR)/$(CONFIG)/grpclb_test || ( echo test grpclb_test failed ; exit 1 ) $(E) "[RUN] Testing hpack_parser_test" $(Q) $(BINDIR)/$(CONFIG)/hpack_parser_test || ( echo test hpack_parser_test failed ; exit 1 ) $(E) "[RUN] Testing hpack_table_test" @@ -2662,6 +2666,7 @@ LIBGRPC_SRC = \ src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c \ src/core/ext/transport/chttp2/client/insecure/channel_create.c \ src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c \ + src/core/ext/lb_policy/grpclb/grpclb.c \ src/core/ext/lb_policy/grpclb/load_balancer_api.c \ src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c \ third_party/nanopb/pb_common.c \ @@ -3240,6 +3245,7 @@ LIBGRPC_UNSECURE_SRC = \ src/core/ext/resolver/sockaddr/sockaddr_resolver.c \ src/core/ext/load_reporting/load_reporting.c \ src/core/ext/load_reporting/load_reporting_filter.c \ + src/core/ext/lb_policy/grpclb/grpclb.c \ src/core/ext/lb_policy/grpclb/load_balancer_api.c \ src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c \ third_party/nanopb/pb_common.c \ @@ -8470,6 +8476,38 @@ endif endif +GRPCLB_TEST_SRC = \ + test/core/client_config/grpclb_test.c \ + +GRPCLB_TEST_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(GRPCLB_TEST_SRC)))) +ifeq ($(NO_SECURE),true) + +# You can't build secure targets if you don't have OpenSSL. + +$(BINDIR)/$(CONFIG)/grpclb_test: openssl_dep_error + +else + + + +$(BINDIR)/$(CONFIG)/grpclb_test: $(GRPCLB_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + $(E) "[LD] Linking $@" + $(Q) mkdir -p `dirname $@` + $(Q) $(LD) $(LDFLAGS) $(GRPCLB_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/grpclb_test + +endif + +$(OBJDIR)/$(CONFIG)/test/core/client_config/grpclb_test.o: $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + +deps_grpclb_test: $(GRPCLB_TEST_OBJS:.o=.dep) + +ifneq ($(NO_SECURE),true) +ifneq ($(NO_DEPS),true) +-include $(GRPCLB_TEST_OBJS:.o=.dep) +endif +endif + + HPACK_PARSER_FUZZER_TEST_SRC = \ test/core/transport/chttp2/hpack_parser_fuzzer_test.c \ diff --git a/binding.gyp b/binding.gyp index fc61a9d01fc..f6cb59b4115 100644 --- a/binding.gyp +++ b/binding.gyp @@ -722,6 +722,7 @@ 'src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c', 'src/core/ext/transport/chttp2/client/insecure/channel_create.c', 'src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c', + 'src/core/ext/lb_policy/grpclb/grpclb.c', 'src/core/ext/lb_policy/grpclb/load_balancer_api.c', 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c', 'third_party/nanopb/pb_common.c', diff --git a/build.yaml b/build.yaml index a83ebc595ad..d5d1b816a68 100644 --- a/build.yaml +++ b/build.yaml @@ -377,8 +377,10 @@ filegroups: - src/core/ext/lb_policy/grpclb/load_balancer_api.h - src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h src: + - src/core/ext/lb_policy/grpclb/grpclb.c - src/core/ext/lb_policy/grpclb/load_balancer_api.c - src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c + plugin: grpc_lb_policy_grpclb uses: - grpc_base - grpc_client_config @@ -796,6 +798,7 @@ libs: - grpc_lb_policy_grpclb - grpc_lb_policy_pick_first - grpc_lb_policy_round_robin + - grpc_lb_policy_grpclb - grpc_resolver_dns_native - grpc_resolver_sockaddr - grpc_load_reporting @@ -893,6 +896,7 @@ libs: - grpc_lb_policy_grpclb - grpc_lb_policy_pick_first - grpc_lb_policy_round_robin + - grpc_lb_policy_grpclb - census generate_plugin_registry: true secure: false @@ -1815,6 +1819,16 @@ targets: - grpc - gpr_test_util - gpr +- name: grpclb_test + build: test + language: c + src: + - test/core/client_config/grpclb_test.c + deps: + - grpc_test_util + - grpc + - gpr_test_util + - gpr - name: hpack_parser_fuzzer_test build: fuzzer language: c diff --git a/config.m4 b/config.m4 index bf5aa244ff2..17dc3c86ec9 100644 --- a/config.m4 +++ b/config.m4 @@ -241,6 +241,7 @@ if test "$PHP_GRPC" != "no"; then src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c \ src/core/ext/transport/chttp2/client/insecure/channel_create.c \ src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c \ + src/core/ext/lb_policy/grpclb/grpclb.c \ src/core/ext/lb_policy/grpclb/load_balancer_api.c \ src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c \ third_party/nanopb/pb_common.c \ diff --git a/gRPC.podspec b/gRPC.podspec index d3665d51744..ab04e494a31 100644 --- a/gRPC.podspec +++ b/gRPC.podspec @@ -504,6 +504,7 @@ Pod::Spec.new do |s| 'src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c', 'src/core/ext/transport/chttp2/client/insecure/channel_create.c', 'src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c', + 'src/core/ext/lb_policy/grpclb/grpclb.c', 'src/core/ext/lb_policy/grpclb/load_balancer_api.c', 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c', 'third_party/nanopb/pb_common.c', diff --git a/grpc.gemspec b/grpc.gemspec index 9f3ae048a70..d30d7ae30e9 100755 --- a/grpc.gemspec +++ b/grpc.gemspec @@ -483,6 +483,7 @@ Gem::Specification.new do |s| s.files += %w( src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c ) s.files += %w( src/core/ext/transport/chttp2/client/insecure/channel_create.c ) s.files += %w( src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c ) + s.files += %w( src/core/ext/lb_policy/grpclb/grpclb.c ) s.files += %w( src/core/ext/lb_policy/grpclb/load_balancer_api.c ) s.files += %w( src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c ) s.files += %w( third_party/nanopb/pb_common.c ) diff --git a/package.xml b/package.xml index b25e42c1c01..2c718608647 100644 --- a/package.xml +++ b/package.xml @@ -490,6 +490,7 @@ + diff --git a/src/core/ext/client_config/lb_policy.c b/src/core/ext/client_config/lb_policy.c index 20535398d69..dc1612428ec 100644 --- a/src/core/ext/client_config/lb_policy.c +++ b/src/core/ext/client_config/lb_policy.c @@ -60,8 +60,9 @@ static gpr_atm ref_mutate(grpc_lb_policy *c, gpr_atm delta, : gpr_atm_no_barrier_fetch_add(&c->ref_pair, delta); #ifdef GRPC_LB_POLICY_REFCOUNT_DEBUG gpr_log(file, line, GPR_LOG_SEVERITY_DEBUG, - "LB_POLICY: %p % 12s 0x%08x -> 0x%08x [%s]", c, purpose, old_val, - old_val + delta, reason); + "LB_POLICY: 0x%" PRIxPTR " %12s 0x%" PRIxPTR " -> 0x%" PRIxPTR + " [%s]", + (intptr_t)c, purpose, old_val, old_val + delta, reason); #endif return old_val; } diff --git a/src/core/ext/client_config/lb_policy.h b/src/core/ext/client_config/lb_policy.h index 56fa11198b7..b07824ae1ba 100644 --- a/src/core/ext/client_config/lb_policy.h +++ b/src/core/ext/client_config/lb_policy.h @@ -73,15 +73,17 @@ struct grpc_lb_policy_vtable { void (*ping_one)(grpc_exec_ctx *exec_ctx, grpc_lb_policy *policy, grpc_closure *closure); - /** try to enter a READY connectivity state */ + /** Try to enter a READY connectivity state */ void (*exit_idle)(grpc_exec_ctx *exec_ctx, grpc_lb_policy *policy); - /** check the current connectivity of the lb_policy */ + /** Check the current connectivity */ grpc_connectivity_state (*check_connectivity)(grpc_exec_ctx *exec_ctx, grpc_lb_policy *policy); /** call notify when the connectivity state of a channel changes from *state. - Updates *state with the new state of the policy */ + Updates *state with the new state of the policy. Calling with a NULL \a + state cancels the subscription. + */ void (*notify_on_state_change)(grpc_exec_ctx *exec_ctx, grpc_lb_policy *policy, grpc_connectivity_state *state, @@ -124,7 +126,7 @@ void grpc_lb_policy_init(grpc_lb_policy *policy, /** Given initial metadata in \a initial_metadata, find an appropriate target for this rpc, and 'return' it by calling \a on_complete after setting \a target. - Picking can be asynchronous. Any IO should be done under \a pollset. */ + Picking can be asynchronous. Any IO should be done under \a pollent. */ int grpc_lb_policy_pick(grpc_exec_ctx *exec_ctx, grpc_lb_policy *policy, grpc_polling_entity *pollent, grpc_metadata_batch *initial_metadata, @@ -146,8 +148,11 @@ void grpc_lb_policy_cancel_picks(grpc_exec_ctx *exec_ctx, uint32_t initial_metadata_flags_mask, uint32_t initial_metadata_flags_eq); +/** Try to enter a READY connectivity state */ void grpc_lb_policy_exit_idle(grpc_exec_ctx *exec_ctx, grpc_lb_policy *policy); +/* Call notify when the connectivity state of a channel changes from \a *state. + * Updates \a *state with the new state of the policy */ void grpc_lb_policy_notify_on_state_change(grpc_exec_ctx *exec_ctx, grpc_lb_policy *policy, grpc_connectivity_state *state, diff --git a/src/core/ext/lb_policy/grpclb/grpclb.c b/src/core/ext/lb_policy/grpclb/grpclb.c new file mode 100644 index 00000000000..cfa9669e0a2 --- /dev/null +++ b/src/core/ext/lb_policy/grpclb/grpclb.c @@ -0,0 +1,836 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include "src/core/ext/lb_policy/grpclb/grpclb.h" +#include "src/core/ext/client_config/client_channel_factory.h" +#include "src/core/ext/client_config/lb_policy_registry.h" +#include "src/core/ext/client_config/parse_address.h" +#include "src/core/ext/lb_policy/grpclb/load_balancer_api.h" +#include "src/core/lib/iomgr/sockaddr_utils.h" +#include "src/core/lib/support/string.h" +#include "src/core/lib/surface/call.h" +#include "src/core/lib/surface/channel.h" + +#include + +#include +#include +#include +#include +#include + +int grpc_lb_glb_trace = 0; + +typedef struct wrapped_rr_closure_arg { + grpc_closure *wrapped_closure; + grpc_lb_policy *rr_policy; +} wrapped_rr_closure_arg; + +/* The \a on_complete closure passed as part of the pick requires keeping a + * reference to its associated round robin instance. We wrap this closure in + * order to unref the round robin instance upon its invocation */ +static void wrapped_rr_closure(grpc_exec_ctx *exec_ctx, void *arg, + bool success) { + wrapped_rr_closure_arg *wc = arg; + + if (wc->rr_policy != NULL) { + if (grpc_lb_glb_trace) { + gpr_log(GPR_INFO, "Unreffing RR %p", wc->rr_policy); + } + GRPC_LB_POLICY_UNREF(exec_ctx, wc->rr_policy, "wrapped_rr_closure"); + } + + if (wc->wrapped_closure != NULL) { + grpc_exec_ctx_enqueue(exec_ctx, wc->wrapped_closure, success, NULL); + } + gpr_free(wc); +} + +typedef struct pending_pick { + struct pending_pick *next; + grpc_polling_entity *pollent; + grpc_metadata_batch *initial_metadata; + uint32_t initial_metadata_flags; + grpc_connected_subchannel **target; + grpc_closure *wrapped_on_complete; + wrapped_rr_closure_arg *wrapped_on_complete_arg; +} pending_pick; + +typedef struct pending_ping { + struct pending_ping *next; + grpc_closure *wrapped_notify; + wrapped_rr_closure_arg *wrapped_notify_arg; +} pending_ping; + +typedef struct glb_lb_policy glb_lb_policy; + +#define MAX_LBCD_OPS_LEN 6 +typedef struct lb_client_data { + gpr_mu mu; + grpc_closure md_sent; + grpc_closure md_rcvd; + grpc_closure req_sent; + grpc_closure res_rcvd; + grpc_closure close_sent; + grpc_closure srv_status_rcvd; + + grpc_call *c; + gpr_timespec deadline; + + grpc_metadata_array initial_metadata_recv; + grpc_metadata_array trailing_metadata_recv; + + grpc_byte_buffer *request_payload; + grpc_byte_buffer *response_payload; + + grpc_status_code status; + char *status_details; + size_t status_details_capacity; + + glb_lb_policy *p; +} lb_client_data; + +/* Keeps track and reacts to changes in connectivity of the RR instance */ +typedef struct rr_connectivity_data { + grpc_closure on_change; + grpc_connectivity_state state; + glb_lb_policy *p; +} rr_connectivity_data; + +struct glb_lb_policy { + /** base policy: must be first */ + grpc_lb_policy base; + + /** mutex protecting remaining members */ + gpr_mu mu; + + grpc_client_channel_factory *cc_factory; + + /** for communicating with the LB server */ + grpc_channel *lb_server_channel; + + /** the RR policy to use of the backend servers returned by the LB server */ + grpc_lb_policy *rr_policy; + + bool started_picking; + + /** our connectivity state tracker */ + grpc_connectivity_state_tracker state_tracker; + + grpc_grpclb_serverlist *serverlist; + + /** list of picks that are waiting on connectivity */ + pending_pick *pending_picks; + + /** list of pings that are waiting on connectivity */ + pending_ping *pending_pings; + + /** data associated with the communication with the LB server */ + lb_client_data *lbcd; + + /** for tracking of the RR connectivity */ + rr_connectivity_data *rr_connectivity; +}; + +static void rr_handover(grpc_exec_ctx *exec_ctx, glb_lb_policy *p); +static void rr_connectivity_changed(grpc_exec_ctx *exec_ctx, void *arg, + bool iomgr_success) { + rr_connectivity_data *rrcd = arg; + if (!iomgr_success) { + gpr_free(rrcd); + return; + } + glb_lb_policy *p = rrcd->p; + const grpc_connectivity_state new_state = p->rr_connectivity->state; + if (new_state == GRPC_CHANNEL_SHUTDOWN && p->serverlist != NULL) { + /* a RR policy is shutting down but there's a serverlist available -> + * perform a handover */ + rr_handover(exec_ctx, p); + } else { + grpc_connectivity_state_set(exec_ctx, &p->state_tracker, new_state, + "rr_connectivity_changed"); + /* resubscribe */ + grpc_lb_policy_notify_on_state_change(exec_ctx, p->rr_policy, + &p->rr_connectivity->state, + &p->rr_connectivity->on_change); + } +} + +static void add_pending_pick(pending_pick **root, grpc_polling_entity *pollent, + grpc_metadata_batch *initial_metadata, + uint32_t initial_metadata_flags, + grpc_connected_subchannel **target, + grpc_closure *on_complete) { + pending_pick *pp = gpr_malloc(sizeof(*pp)); + memset(pp, 0, sizeof(pending_pick)); + pp->wrapped_on_complete_arg = gpr_malloc(sizeof(wrapped_rr_closure_arg)); + memset(pp->wrapped_on_complete_arg, 0, sizeof(wrapped_rr_closure_arg)); + pp->next = *root; + pp->pollent = pollent; + pp->target = target; + pp->initial_metadata = initial_metadata; + pp->initial_metadata_flags = initial_metadata_flags; + pp->wrapped_on_complete = + grpc_closure_create(wrapped_rr_closure, pp->wrapped_on_complete_arg); + pp->wrapped_on_complete_arg->wrapped_closure = on_complete; + *root = pp; +} + +static void add_pending_ping(pending_ping **root, grpc_closure *notify) { + pending_ping *pping = gpr_malloc(sizeof(*pping)); + memset(pping, 0, sizeof(pending_ping)); + pping->wrapped_notify_arg = gpr_malloc(sizeof(wrapped_rr_closure_arg)); + memset(pping->wrapped_notify_arg, 0, sizeof(wrapped_rr_closure_arg)); + pping->next = *root; + pping->wrapped_notify = + grpc_closure_create(wrapped_rr_closure, pping->wrapped_notify_arg); + pping->wrapped_notify_arg->wrapped_closure = notify; + *root = pping; +} + +static void lb_client_data_destroy(lb_client_data *lbcd); + +static void md_sent_cb(grpc_exec_ctx *exec_ctx, void *arg, bool success) { + lb_client_data *lbcd = arg; + GPR_ASSERT(lbcd->c); + grpc_call_error error; + grpc_op ops[1]; + memset(ops, 0, sizeof(ops)); + grpc_op *op = ops; + op->op = GRPC_OP_RECV_INITIAL_METADATA; + op->data.recv_initial_metadata = &lbcd->initial_metadata_recv; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch_and_execute(exec_ctx, lbcd->c, ops, + (size_t)(op - ops), &lbcd->md_rcvd); + GPR_ASSERT(GRPC_CALL_OK == error); +} + +static void md_recv_cb(grpc_exec_ctx *exec_ctx, void *arg, bool success) { + lb_client_data *lbcd = arg; + GPR_ASSERT(lbcd->c); + grpc_call_error error; + grpc_op ops[1]; + memset(ops, 0, sizeof(ops)); + grpc_op *op = ops; + + op->op = GRPC_OP_SEND_MESSAGE; + op->data.send_message = lbcd->request_payload; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch_and_execute( + exec_ctx, lbcd->c, ops, (size_t)(op - ops), &lbcd->req_sent); + GPR_ASSERT(GRPC_CALL_OK == error); +} + +static void req_sent_cb(grpc_exec_ctx *exec_ctx, void *arg, bool success) { + lb_client_data *lbcd = arg; + grpc_call_error error; + + grpc_op ops[1]; + memset(ops, 0, sizeof(ops)); + grpc_op *op = ops; + + op->op = GRPC_OP_RECV_MESSAGE; + op->data.recv_message = &lbcd->response_payload; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch_and_execute( + exec_ctx, lbcd->c, ops, (size_t)(op - ops), &lbcd->res_rcvd); + GPR_ASSERT(GRPC_CALL_OK == error); +} + +static void res_rcvd_cb(grpc_exec_ctx *exec_ctx, void *arg, bool success) { + /* look inside lbcd->response_payload, ideally to send it back as the + * serverlist. */ + lb_client_data *lbcd = arg; + grpc_op ops[2]; + memset(ops, 0, sizeof(ops)); + grpc_op *op = ops; + if (lbcd->response_payload) { + grpc_byte_buffer_reader bbr; + grpc_byte_buffer_reader_init(&bbr, lbcd->response_payload); + gpr_slice response_slice = grpc_byte_buffer_reader_readall(&bbr); + grpc_byte_buffer_destroy(lbcd->response_payload); + grpc_grpclb_serverlist *serverlist = + grpc_grpclb_response_parse_serverlist(response_slice); + if (serverlist) { + gpr_slice_unref(response_slice); + if (grpc_lb_glb_trace) { + gpr_log(GPR_INFO, "Serverlist with %zu servers received", + serverlist->num_servers); + } + /* update serverlist */ + if (serverlist->num_servers > 0) { + if (grpc_grpclb_serverlist_equals(lbcd->p->serverlist, serverlist)) { + gpr_log(GPR_INFO, + "Incoming server list identical to current, ignoring."); + } else { + if (lbcd->p->serverlist != NULL) { + grpc_grpclb_destroy_serverlist(lbcd->p->serverlist); + } + lbcd->p->serverlist = serverlist; + } + } + if (lbcd->p->rr_policy == NULL) { + /* initial "handover", in this case from a null RR policy, meaning it'll + * just create the first one */ + rr_handover(exec_ctx, lbcd->p); + } else { + /* unref the RR policy, eventually leading to its substitution with a + * new one constructed from the received serverlist (see + * rr_connectivity_changed) */ + GRPC_LB_POLICY_UNREF(exec_ctx, lbcd->p->rr_policy, + "serverlist_received"); + } + + /* listen for a potential serverlist update */ + op->op = GRPC_OP_RECV_MESSAGE; + op->data.recv_message = &lbcd->response_payload; + op->flags = 0; + op->reserved = NULL; + op++; + const grpc_call_error error = grpc_call_start_batch_and_execute( + exec_ctx, lbcd->c, ops, (size_t)(op - ops), + &lbcd->res_rcvd); /* loop */ + GPR_ASSERT(GRPC_CALL_OK == error); + return; + } else { + gpr_log(GPR_ERROR, "Invalid LB response received: '%s'", + gpr_dump_slice(response_slice, GPR_DUMP_ASCII)); + gpr_slice_unref(response_slice); + + /* Disconnect from server returning invalid response. */ + op->op = GRPC_OP_SEND_CLOSE_FROM_CLIENT; + op->flags = 0; + op->reserved = NULL; + op++; + grpc_call_error error = grpc_call_start_batch_and_execute( + exec_ctx, lbcd->c, ops, (size_t)(op - ops), &lbcd->close_sent); + GPR_ASSERT(GRPC_CALL_OK == error); + } + } + /* empty payload: call cancelled by server. Cleanups happening in + * srv_status_rcvd_cb */ +} +static void close_sent_cb(grpc_exec_ctx *exec_ctx, void *arg, bool success) { + if (grpc_lb_glb_trace) { + gpr_log(GPR_INFO, + "Close from LB client sent. Waiting from server status now"); + } +} +static void srv_status_rcvd_cb(grpc_exec_ctx *exec_ctx, void *arg, + bool success) { + lb_client_data *lbcd = arg; + glb_lb_policy *p = lbcd->p; + if (grpc_lb_glb_trace) { + gpr_log( + GPR_INFO, + "status from lb server received. Status = %d, Details = '%s', Capaticy " + "= %zu", + lbcd->status, lbcd->status_details, lbcd->status_details_capacity); + } + + grpc_call_destroy(lbcd->c); + grpc_channel_destroy(lbcd->p->lb_server_channel); + lbcd->p->lb_server_channel = NULL; + lb_client_data_destroy(lbcd); + p->lbcd = NULL; +} + +static lb_client_data *lb_client_data_create(glb_lb_policy *p) { + lb_client_data *lbcd = gpr_malloc(sizeof(lb_client_data)); + memset(lbcd, 0, sizeof(lb_client_data)); + + gpr_mu_init(&lbcd->mu); + grpc_closure_init(&lbcd->md_sent, md_sent_cb, lbcd); + + grpc_closure_init(&lbcd->md_rcvd, md_recv_cb, lbcd); + grpc_closure_init(&lbcd->req_sent, req_sent_cb, lbcd); + grpc_closure_init(&lbcd->res_rcvd, res_rcvd_cb, lbcd); + grpc_closure_init(&lbcd->close_sent, close_sent_cb, lbcd); + grpc_closure_init(&lbcd->srv_status_rcvd, srv_status_rcvd_cb, lbcd); + + /* TODO(dgq): get the deadline from the client/user instead of fabricating + * one + * here. Make it a policy arg? */ + lbcd->deadline = gpr_time_add(gpr_now(GPR_CLOCK_MONOTONIC), + gpr_time_from_seconds(3, GPR_TIMESPAN)); + + lbcd->c = grpc_channel_create_pollset_set_call( + p->lb_server_channel, NULL, GRPC_PROPAGATE_DEFAULTS, + p->base.interested_parties, "/BalanceLoad", + NULL, /* FIXME(dgq): which "host" value to use? */ + lbcd->deadline, NULL); + + grpc_metadata_array_init(&lbcd->initial_metadata_recv); + grpc_metadata_array_init(&lbcd->trailing_metadata_recv); + + grpc_grpclb_request *request = grpc_grpclb_request_create( + "load.balanced.service.name"); /* FIXME(dgq): get the name of the load + balanced service from above. */ + gpr_slice request_payload_slice = grpc_grpclb_request_encode(request); + lbcd->request_payload = + grpc_raw_byte_buffer_create(&request_payload_slice, 1); + gpr_slice_unref(request_payload_slice); + grpc_grpclb_request_destroy(request); + + lbcd->status_details = NULL; + lbcd->status_details_capacity = 0; + lbcd->p = p; + return lbcd; +} + +static void lb_client_data_destroy(lb_client_data *lbcd) { + grpc_metadata_array_destroy(&lbcd->initial_metadata_recv); + grpc_metadata_array_destroy(&lbcd->trailing_metadata_recv); + + grpc_byte_buffer_destroy(lbcd->request_payload); + + gpr_free(lbcd->status_details); + gpr_mu_destroy(&lbcd->mu); + gpr_free(lbcd); +} + +static void glb_destroy(grpc_exec_ctx *exec_ctx, grpc_lb_policy *pol) { + glb_lb_policy *p = (glb_lb_policy *)pol; + GPR_ASSERT(p->pending_picks == NULL); + GPR_ASSERT(p->pending_pings == NULL); + grpc_connectivity_state_destroy(exec_ctx, &p->state_tracker); + if (p->serverlist != NULL) { + grpc_grpclb_destroy_serverlist(p->serverlist); + } + gpr_mu_destroy(&p->mu); + gpr_free(p); +} + +static void glb_shutdown(grpc_exec_ctx *exec_ctx, grpc_lb_policy *pol) { + glb_lb_policy *p = (glb_lb_policy *)pol; + gpr_mu_lock(&p->mu); + + pending_pick *pp = p->pending_picks; + p->pending_picks = NULL; + pending_ping *pping = p->pending_pings; + p->pending_pings = NULL; + gpr_mu_unlock(&p->mu); + + while (pp != NULL) { + pending_pick *next = pp->next; + *pp->target = NULL; + grpc_exec_ctx_enqueue(exec_ctx, pp->wrapped_on_complete, true, NULL); + gpr_free(pp); + pp = next; + } + + while (pping != NULL) { + pending_ping *next = pping->next; + grpc_exec_ctx_enqueue(exec_ctx, pping->wrapped_notify, true, NULL); + pping = next; + } + + if (p->rr_policy) { + /* unsubscribe */ + grpc_lb_policy_notify_on_state_change(exec_ctx, p->rr_policy, NULL, + &p->rr_connectivity->on_change); + GRPC_LB_POLICY_UNREF(exec_ctx, p->rr_policy, "glb_shutdown"); + } + + grpc_connectivity_state_set(exec_ctx, &p->state_tracker, + GRPC_CHANNEL_SHUTDOWN, "glb_shutdown"); +} + +static void glb_cancel_pick(grpc_exec_ctx *exec_ctx, grpc_lb_policy *pol, + grpc_connected_subchannel **target) { + glb_lb_policy *p = (glb_lb_policy *)pol; + gpr_mu_lock(&p->mu); + pending_pick *pp = p->pending_picks; + p->pending_picks = NULL; + while (pp != NULL) { + pending_pick *next = pp->next; + if (pp->target == target) { + grpc_polling_entity_del_from_pollset_set(exec_ctx, pp->pollent, + p->base.interested_parties); + *target = NULL; + grpc_exec_ctx_enqueue(exec_ctx, pp->wrapped_on_complete, false, NULL); + gpr_free(pp); + } else { + pp->next = p->pending_picks; + p->pending_picks = pp; + } + pp = next; + } + gpr_mu_unlock(&p->mu); +} + +static void glb_cancel_picks(grpc_exec_ctx *exec_ctx, grpc_lb_policy *pol, + uint32_t initial_metadata_flags_mask, + uint32_t initial_metadata_flags_eq) { + glb_lb_policy *p = (glb_lb_policy *)pol; + gpr_mu_lock(&p->mu); + if (p->lbcd != NULL) { + /* cancel the call to the load balancer service, if any */ + grpc_call_cancel(p->lbcd->c, NULL); + } + pending_pick *pp = p->pending_picks; + p->pending_picks = NULL; + while (pp != NULL) { + pending_pick *next = pp->next; + if ((pp->initial_metadata_flags & initial_metadata_flags_mask) == + initial_metadata_flags_eq) { + grpc_polling_entity_del_from_pollset_set(exec_ctx, pp->pollent, + p->base.interested_parties); + grpc_exec_ctx_enqueue(exec_ctx, pp->wrapped_on_complete, false, NULL); + gpr_free(pp); + } else { + pp->next = p->pending_picks; + p->pending_picks = pp; + } + pp = next; + } + gpr_mu_unlock(&p->mu); +} + +static void query_for_backends(grpc_exec_ctx *exec_ctx, glb_lb_policy *p) { + GPR_ASSERT(p->lb_server_channel != NULL); + + p->lbcd = lb_client_data_create(p); + grpc_call_error error; + grpc_op ops[1]; + memset(ops, 0, sizeof(ops)); + grpc_op *op = ops; + op->op = GRPC_OP_SEND_INITIAL_METADATA; + op->data.send_initial_metadata.count = 0; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch_and_execute( + exec_ctx, p->lbcd->c, ops, (size_t)(op - ops), &p->lbcd->md_sent); + GPR_ASSERT(GRPC_CALL_OK == error); + + op = ops; + op->op = GRPC_OP_RECV_STATUS_ON_CLIENT; + op->data.recv_status_on_client.trailing_metadata = + &p->lbcd->trailing_metadata_recv; + op->data.recv_status_on_client.status = &p->lbcd->status; + op->data.recv_status_on_client.status_details = &p->lbcd->status_details; + op->data.recv_status_on_client.status_details_capacity = + &p->lbcd->status_details_capacity; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch_and_execute( + exec_ctx, p->lbcd->c, ops, (size_t)(op - ops), &p->lbcd->srv_status_rcvd); + GPR_ASSERT(GRPC_CALL_OK == error); +} + +static grpc_lb_policy *create_rr(grpc_exec_ctx *exec_ctx, + const grpc_grpclb_serverlist *serverlist, + glb_lb_policy *p) { + /* TODO(dgq): support mixed ip version */ + GPR_ASSERT(serverlist != NULL && serverlist->num_servers > 0); + char **host_ports = gpr_malloc(sizeof(char *) * serverlist->num_servers); + for (size_t i = 0; i < serverlist->num_servers; ++i) { + gpr_join_host_port(&host_ports[i], serverlist->servers[i]->ip_address, + serverlist->servers[i]->port); + } + + size_t uri_path_len; + char *concat_ipports = gpr_strjoin_sep( + (const char **)host_ports, serverlist->num_servers, ",", &uri_path_len); + + grpc_lb_policy_args args; + args.client_channel_factory = p->cc_factory; + args.addresses = gpr_malloc(sizeof(grpc_resolved_addresses)); + args.addresses->naddrs = serverlist->num_servers; + args.addresses->addrs = + gpr_malloc(sizeof(grpc_resolved_address) * args.addresses->naddrs); + size_t out_addrs_idx = 0; + for (size_t i = 0; i < serverlist->num_servers; ++i) { + grpc_uri uri; + struct sockaddr_storage sa; + size_t sa_len; + uri.path = host_ports[i]; + if (parse_ipv4(&uri, &sa, &sa_len)) { /* TODO(dgq): add support for ipv6 */ + memcpy(args.addresses->addrs[out_addrs_idx].addr, &sa, sa_len); + args.addresses->addrs[out_addrs_idx].len = sa_len; + ++out_addrs_idx; + } else { + gpr_log(GPR_ERROR, "Invalid LB service address '%s', ignoring.", + host_ports[i]); + } + } + + grpc_lb_policy *rr = grpc_lb_policy_create(exec_ctx, "round_robin", &args); + + gpr_free(concat_ipports); + for (size_t i = 0; i < serverlist->num_servers; i++) { + gpr_free(host_ports[i]); + } + gpr_free(host_ports); + + gpr_free(args.addresses->addrs); + gpr_free(args.addresses); + + return rr; +} + +static void start_picking(grpc_exec_ctx *exec_ctx, glb_lb_policy *p) { + p->started_picking = true; + query_for_backends(exec_ctx, p); +} + +static void rr_handover(grpc_exec_ctx *exec_ctx, glb_lb_policy *p) { + p->rr_policy = create_rr(exec_ctx, p->serverlist, p); + if (grpc_lb_glb_trace) { + gpr_log(GPR_INFO, "Created RR policy (0x%" PRIxPTR ")", + (intptr_t)p->rr_policy); + } + GPR_ASSERT(p->rr_policy != NULL); + p->rr_connectivity->state = + grpc_lb_policy_check_connectivity(exec_ctx, p->rr_policy); + grpc_lb_policy_notify_on_state_change(exec_ctx, p->rr_policy, + &p->rr_connectivity->state, + &p->rr_connectivity->on_change); + grpc_connectivity_state_set(exec_ctx, &p->state_tracker, + p->rr_connectivity->state, "rr_handover"); + grpc_lb_policy_exit_idle(exec_ctx, p->rr_policy); + + /* flush pending ops */ + pending_pick *pp; + while ((pp = p->pending_picks)) { + p->pending_picks = pp->next; + GRPC_LB_POLICY_REF(p->rr_policy, "rr_handover_pending_pick"); + pp->wrapped_on_complete_arg->rr_policy = p->rr_policy; + if (grpc_lb_glb_trace) { + gpr_log(GPR_INFO, "Pending pick about to PICK from 0x%"PRIxPTR"", (intptr_t)p->rr_policy); + } + grpc_lb_policy_pick(exec_ctx, p->rr_policy, pp->pollent, + pp->initial_metadata, pp->initial_metadata_flags, + pp->target, pp->wrapped_on_complete); + gpr_free(pp); + } + + pending_ping *pping; + while ((pping = p->pending_pings)) { + p->pending_pings = pping->next; + GRPC_LB_POLICY_REF(p->rr_policy, "rr_handover_pending_ping"); + pping->wrapped_notify_arg->rr_policy = p->rr_policy; + if (grpc_lb_glb_trace) { + gpr_log(GPR_INFO, "Pending ping about to PING from 0x%"PRIxPTR"", (intptr_t)p->rr_policy); + } + grpc_lb_policy_ping_one(exec_ctx, p->rr_policy, pping->wrapped_notify); + gpr_free(pping); + } +} + +static void glb_exit_idle(grpc_exec_ctx *exec_ctx, grpc_lb_policy *pol) { + glb_lb_policy *p = (glb_lb_policy *)pol; + gpr_mu_lock(&p->mu); + if (!p->started_picking) { + start_picking(exec_ctx, p); + } + gpr_mu_unlock(&p->mu); +} + +static int glb_pick(grpc_exec_ctx *exec_ctx, grpc_lb_policy *pol, + grpc_polling_entity *pollent, + grpc_metadata_batch *initial_metadata, + uint32_t initial_metadata_flags, + grpc_connected_subchannel **target, + grpc_closure *on_complete) { + glb_lb_policy *p = (glb_lb_policy *)pol; + gpr_mu_lock(&p->mu); + int r; + + if (p->rr_policy != NULL) { + if (grpc_lb_glb_trace) { + gpr_log(GPR_INFO, "about to PICK from 0x%"PRIxPTR"", (intptr_t)p->rr_policy); + } + GRPC_LB_POLICY_REF(p->rr_policy, "rr_pick"); + wrapped_rr_closure_arg *warg = gpr_malloc(sizeof(wrapped_rr_closure_arg)); + warg->rr_policy = p->rr_policy; + warg->wrapped_closure = on_complete; + grpc_closure *wrapped_on_complete = + grpc_closure_create(wrapped_rr_closure, warg); + r = grpc_lb_policy_pick(exec_ctx, p->rr_policy, pollent, initial_metadata, + initial_metadata_flags, target, + wrapped_on_complete); + if (r != 0) { + /* the call to grpc_lb_policy_pick has been sychronous. Invoke a neutered + * wrapped closure */ + warg->wrapped_closure = NULL; + grpc_exec_ctx_enqueue(exec_ctx, wrapped_on_complete, false, NULL); + } + } else { + grpc_polling_entity_add_to_pollset_set(exec_ctx, pollent, + p->base.interested_parties); + add_pending_pick(&p->pending_picks, pollent, initial_metadata, + initial_metadata_flags, target, on_complete); + + if (!p->started_picking) { + start_picking(exec_ctx, p); + } + r = 0; + } + gpr_mu_unlock(&p->mu); + return r; +} + +static grpc_connectivity_state glb_check_connectivity(grpc_exec_ctx *exec_ctx, + grpc_lb_policy *pol) { + glb_lb_policy *p = (glb_lb_policy *)pol; + grpc_connectivity_state st; + gpr_mu_lock(&p->mu); + st = grpc_connectivity_state_check(&p->state_tracker); + gpr_mu_unlock(&p->mu); + return st; +} + +static void glb_ping_one(grpc_exec_ctx *exec_ctx, grpc_lb_policy *pol, + grpc_closure *closure) { + glb_lb_policy *p = (glb_lb_policy *)pol; + gpr_mu_lock(&p->mu); + if (p->rr_policy) { + grpc_lb_policy_ping_one(exec_ctx, p->rr_policy, closure); + } else { + add_pending_ping(&p->pending_pings, closure); + if (!p->started_picking) { + start_picking(exec_ctx, p); + } + } + gpr_mu_unlock(&p->mu); +} + +static void glb_notify_on_state_change(grpc_exec_ctx *exec_ctx, + grpc_lb_policy *pol, + grpc_connectivity_state *current, + grpc_closure *notify) { + glb_lb_policy *p = (glb_lb_policy *)pol; + gpr_mu_lock(&p->mu); + grpc_connectivity_state_notify_on_state_change(exec_ctx, &p->state_tracker, + current, notify); + + gpr_mu_unlock(&p->mu); +} + +static const grpc_lb_policy_vtable glb_lb_policy_vtable = { + glb_destroy, glb_shutdown, glb_pick, + glb_cancel_pick, glb_cancel_picks, glb_ping_one, + glb_exit_idle, glb_check_connectivity, glb_notify_on_state_change}; + +static void glb_factory_ref(grpc_lb_policy_factory *factory) {} + +static void glb_factory_unref(grpc_lb_policy_factory *factory) {} + +static grpc_lb_policy *glb_create(grpc_exec_ctx *exec_ctx, + grpc_lb_policy_factory *factory, + grpc_lb_policy_args *args) { + glb_lb_policy *p = gpr_malloc(sizeof(*p)); + memset(p, 0, sizeof(*p)); + + /* all input addresses in args->addresses come from a resolver that claims + * they are LB services. + * + * Create a client channel over them to communicate with a LB service */ + p->cc_factory = args->client_channel_factory; + GPR_ASSERT(p->cc_factory != NULL); + if (args->addresses->naddrs == 0) { + return NULL; + } + + /* construct a target from the args->addresses, in the form + * ipvX://ip1:port1,ip2:port2,... + * TODO(dgq): support mixed ip version */ + char **addr_strs = gpr_malloc(sizeof(char *) * args->addresses->naddrs); + addr_strs[0] = + grpc_sockaddr_to_uri((const struct sockaddr *)&args->addresses->addrs[0]); + for (size_t i = 1; i < args->addresses->naddrs; i++) { + GPR_ASSERT(grpc_sockaddr_to_string( + &addr_strs[i], + (const struct sockaddr *)&args->addresses->addrs[i], + true) == 0); + } + size_t uri_path_len; + char *target_uri_str = gpr_strjoin_sep( + (const char **)addr_strs, args->addresses->naddrs, ",", &uri_path_len); + + /* will pick using pick_first */ + p->lb_server_channel = grpc_client_channel_factory_create_channel( + exec_ctx, p->cc_factory, target_uri_str, + GRPC_CLIENT_CHANNEL_TYPE_LOAD_BALANCING, NULL); + + gpr_free(target_uri_str); + for (size_t i = 0; i < args->addresses->naddrs; i++) { + gpr_free(addr_strs[i]); + } + gpr_free(addr_strs); + + if (p->lb_server_channel == NULL) { + gpr_free(p); + return NULL; + } + + rr_connectivity_data *rr_connectivity = + gpr_malloc(sizeof(rr_connectivity_data)); + memset(rr_connectivity, 0, sizeof(rr_connectivity_data)); + grpc_closure_init(&rr_connectivity->on_change, rr_connectivity_changed, + rr_connectivity); + rr_connectivity->p = p; + p->rr_connectivity = rr_connectivity; + + grpc_lb_policy_init(&p->base, &glb_lb_policy_vtable); + gpr_mu_init(&p->mu); + grpc_connectivity_state_init(&p->state_tracker, GRPC_CHANNEL_IDLE, "grpclb"); + return &p->base; +} + +static const grpc_lb_policy_factory_vtable glb_factory_vtable = { + glb_factory_ref, glb_factory_unref, glb_create, "grpclb"}; + +static grpc_lb_policy_factory glb_lb_policy_factory = {&glb_factory_vtable}; + +grpc_lb_policy_factory *grpc_glb_lb_factory_create() { + return &glb_lb_policy_factory; +} + +/* Plugin registration */ + +void grpc_lb_policy_grpclb_init() { + grpc_register_lb_policy(grpc_glb_lb_factory_create()); + grpc_register_tracer("glb", &grpc_lb_glb_trace); +} + +void grpc_lb_policy_grpclb_shutdown() {} diff --git a/src/core/ext/lb_policy/grpclb/grpclb.h b/src/core/ext/lb_policy/grpclb/grpclb.h new file mode 100644 index 00000000000..83552b4fa02 --- /dev/null +++ b/src/core/ext/lb_policy/grpclb/grpclb.h @@ -0,0 +1,44 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#ifndef GRPC_CORE_EXT_LB_POLICY_GRPCLB_GRPCLB_H +#define GRPC_CORE_EXT_LB_POLICY_GRPCLB_GRPCLB_H + +#include "src/core/ext/client_config/lb_policy_factory.h" + +/** Returns a load balancing factory for the glb policy, which tries to connect + * to a load balancing server to decide the next successfully connected + * subchannel to pick. */ +grpc_lb_policy_factory *grpc_glb_lb_factory_create(); + +#endif /* GRPC_CORE_EXT_LB_POLICY_GRPCLB_GRPCLB_H */ diff --git a/src/core/ext/lb_policy/grpclb/load_balancer_api.c b/src/core/ext/lb_policy/grpclb/load_balancer_api.c index 59b89997dd9..c3b3888e90b 100644 --- a/src/core/ext/lb_policy/grpclb/load_balancer_api.c +++ b/src/core/ext/lb_policy/grpclb/load_balancer_api.c @@ -56,6 +56,7 @@ static bool decode_serverlist(pb_istream_t *stream, const pb_field_t *field, dec_arg->num_servers++; } else { /* second pass */ grpc_grpclb_server *server = gpr_malloc(sizeof(grpc_grpclb_server)); + memset(server, 0, sizeof(grpc_grpclb_server)); GPR_ASSERT(dec_arg->num_servers > 0); if (dec_arg->i == 0) { /* first iteration of second pass */ dec_arg->servers = @@ -147,6 +148,7 @@ grpc_grpclb_serverlist *grpc_grpclb_response_parse_serverlist( } grpc_grpclb_serverlist *sl = gpr_malloc(sizeof(grpc_grpclb_serverlist)); + memset(sl, 0, sizeof(*sl)); sl->num_servers = arg.num_servers; sl->servers = arg.servers; if (res->server_list.has_expiration_interval) { @@ -167,6 +169,81 @@ void grpc_grpclb_destroy_serverlist(grpc_grpclb_serverlist *serverlist) { gpr_free(serverlist); } +grpc_grpclb_serverlist *grpc_grpclb_serverlist_copy( + const grpc_grpclb_serverlist *sl) { + grpc_grpclb_serverlist *copy = gpr_malloc(sizeof(grpc_grpclb_serverlist)); + memset(copy, 0, sizeof(grpc_grpclb_serverlist)); + copy->num_servers = sl->num_servers; + memcpy(©->expiration_interval, &sl->expiration_interval, + sizeof(grpc_grpclb_duration)); + copy->servers = gpr_malloc(sizeof(grpc_grpclb_server*) * sl->num_servers); + for (size_t i = 0; i < sl->num_servers; i++) { + copy->servers[i] = gpr_malloc(sizeof(grpc_grpclb_server)); + memcpy(copy->servers[i], sl->servers[i], sizeof(grpc_grpclb_server)); + } + return copy; +} + +bool grpc_grpclb_serverlist_equals(const grpc_grpclb_serverlist *lhs, + const grpc_grpclb_serverlist *rhs) { + if ((lhs == NULL) || (rhs == NULL)) { + return false; + } + if (lhs->num_servers != rhs->num_servers) { + return false; + } + if (grpc_grpclb_duration_compare(&lhs->expiration_interval, + &rhs->expiration_interval) != 0) { + return false; + } + for (size_t i = 0; i < lhs->num_servers; i++) { + if (!grpc_grpclb_server_equals(lhs->servers[i], rhs->servers[i])) { + return false; + } + } + return true; +} + +bool grpc_grpclb_server_equals(const grpc_grpclb_server *lhs, + const grpc_grpclb_server *rhs) { + return memcmp(lhs, rhs, sizeof(grpc_grpclb_server)) == 0; +} + +int grpc_grpclb_duration_compare(const grpc_grpclb_duration *lhs, + const grpc_grpclb_duration *rhs) { + GPR_ASSERT(lhs && rhs); + if (lhs->has_seconds && rhs->has_seconds) { + if (lhs->seconds < rhs->seconds) { + return -1; + } + if (lhs->seconds > rhs->seconds) { + return 1; + } + goto compare_nanos; + } + if (lhs->has_seconds + rhs->has_seconds == 0) { + goto compare_nanos; + } + // only lhs or rhs have seconds + return rhs->has_seconds ? 1 : -1; + +compare_nanos: + if (lhs->has_nanos && rhs->has_nanos) { + if (lhs->nanos < rhs->nanos) { + return -1; + } + if (lhs->nanos > rhs->nanos) { + return 1; + } + return 0; + } + if (lhs->has_nanos + rhs->has_nanos == 0) { + return 0; + } + // only lhs or rhs have nanos + return rhs->has_nanos ? 1 : -1; +} + void grpc_grpclb_response_destroy(grpc_grpclb_response *response) { gpr_free(response); } diff --git a/src/core/ext/lb_policy/grpclb/load_balancer_api.h b/src/core/ext/lb_policy/grpclb/load_balancer_api.h index 71b5616d0c8..cc5c4e93439 100644 --- a/src/core/ext/lb_policy/grpclb/load_balancer_api.h +++ b/src/core/ext/lb_policy/grpclb/load_balancer_api.h @@ -68,6 +68,11 @@ void grpc_grpclb_request_destroy(grpc_grpclb_request *request); * grpc_grpclb_response */ grpc_grpclb_response *grpc_grpclb_response_parse(gpr_slice encoded_response); +/** Return a copy of \a sl. The caller is responsible for calling \a + * grpc_grpclb_destroy_serverlist on the returned copy. */ +grpc_grpclb_serverlist *grpc_grpclb_serverlist_copy( + const grpc_grpclb_serverlist *sl); + /** Destroy \a serverlist */ void grpc_grpclb_destroy_serverlist(grpc_grpclb_serverlist *serverlist); @@ -75,6 +80,17 @@ void grpc_grpclb_destroy_serverlist(grpc_grpclb_serverlist *serverlist); grpc_grpclb_serverlist *grpc_grpclb_response_parse_serverlist( gpr_slice encoded_response); +bool grpc_grpclb_serverlist_equals(const grpc_grpclb_serverlist *lhs, + const grpc_grpclb_serverlist *rhs); + +bool grpc_grpclb_server_equals(const grpc_grpclb_server *lhs, + const grpc_grpclb_server *rhs); + +/** Compare \a lhs against \a rhs and return 0 if \a lhs and \a rhs are equal, + * < 0 if \a lhs represents a duration shorter than \a rhs and > 0 otherwise */ +int grpc_grpclb_duration_compare(const grpc_grpclb_duration *lhs, + const grpc_grpclb_duration *rhs); + /** Destroy \a response */ void grpc_grpclb_response_destroy(grpc_grpclb_response *response); diff --git a/src/core/plugin_registry/grpc_plugin_registry.c b/src/core/plugin_registry/grpc_plugin_registry.c index 905cd59e23d..7a7a9ce477a 100644 --- a/src/core/plugin_registry/grpc_plugin_registry.c +++ b/src/core/plugin_registry/grpc_plugin_registry.c @@ -37,6 +37,8 @@ extern void grpc_chttp2_plugin_init(void); extern void grpc_chttp2_plugin_shutdown(void); extern void grpc_client_config_init(void); extern void grpc_client_config_shutdown(void); +extern void grpc_lb_policy_grpclb_init(void); +extern void grpc_lb_policy_grpclb_shutdown(void); extern void grpc_lb_policy_pick_first_init(void); extern void grpc_lb_policy_pick_first_shutdown(void); extern void grpc_lb_policy_round_robin_init(void); @@ -55,6 +57,8 @@ void grpc_register_built_in_plugins(void) { grpc_chttp2_plugin_shutdown); grpc_register_plugin(grpc_client_config_init, grpc_client_config_shutdown); + grpc_register_plugin(grpc_lb_policy_grpclb_init, + grpc_lb_policy_grpclb_shutdown); grpc_register_plugin(grpc_lb_policy_pick_first_init, grpc_lb_policy_pick_first_shutdown); grpc_register_plugin(grpc_lb_policy_round_robin_init, diff --git a/src/core/plugin_registry/grpc_unsecure_plugin_registry.c b/src/core/plugin_registry/grpc_unsecure_plugin_registry.c index 79950787258..ad4ddf0ff48 100644 --- a/src/core/plugin_registry/grpc_unsecure_plugin_registry.c +++ b/src/core/plugin_registry/grpc_unsecure_plugin_registry.c @@ -43,6 +43,8 @@ extern void grpc_resolver_sockaddr_init(void); extern void grpc_resolver_sockaddr_shutdown(void); extern void grpc_load_reporting_plugin_init(void); extern void grpc_load_reporting_plugin_shutdown(void); +extern void grpc_lb_policy_grpclb_init(void); +extern void grpc_lb_policy_grpclb_shutdown(void); extern void grpc_lb_policy_pick_first_init(void); extern void grpc_lb_policy_pick_first_shutdown(void); extern void grpc_lb_policy_round_robin_init(void); @@ -61,6 +63,8 @@ void grpc_register_built_in_plugins(void) { grpc_resolver_sockaddr_shutdown); grpc_register_plugin(grpc_load_reporting_plugin_init, grpc_load_reporting_plugin_shutdown); + grpc_register_plugin(grpc_lb_policy_grpclb_init, + grpc_lb_policy_grpclb_shutdown); grpc_register_plugin(grpc_lb_policy_pick_first_init, grpc_lb_policy_pick_first_shutdown); grpc_register_plugin(grpc_lb_policy_round_robin_init, diff --git a/src/python/grpcio/grpc_core_dependencies.py b/src/python/grpcio/grpc_core_dependencies.py index 2a5229434c9..5768403a040 100644 --- a/src/python/grpcio/grpc_core_dependencies.py +++ b/src/python/grpcio/grpc_core_dependencies.py @@ -235,6 +235,7 @@ CORE_SOURCE_FILES = [ 'src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c', 'src/core/ext/transport/chttp2/client/insecure/channel_create.c', 'src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c', + 'src/core/ext/lb_policy/grpclb/grpclb.c', 'src/core/ext/lb_policy/grpclb/load_balancer_api.c', 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c', 'third_party/nanopb/pb_common.c', diff --git a/test/core/client_config/grpclb_test.c b/test/core/client_config/grpclb_test.c new file mode 100644 index 00000000000..2080fefd1ae --- /dev/null +++ b/test/core/client_config/grpclb_test.c @@ -0,0 +1,670 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "src/core/ext/client_config/client_channel.h" +#include "src/core/lib/channel/channel_stack.h" +#include "src/core/lib/support/string.h" +#include "src/core/lib/support/tmpfile.h" +#include "src/core/lib/surface/channel.h" +#include "src/core/lib/surface/server.h" +#include "test/core/end2end/cq_verifier.h" +#include "test/core/util/port.h" +#include "test/core/util/test_config.h" + +#define NUM_BACKENDS 4 + +typedef struct client_fixture { + grpc_channel *client; + char *server_uri; + grpc_completion_queue *cq; +} client_fixture; + +typedef struct server_fixture { + grpc_server *server; + grpc_call *server_call; + grpc_completion_queue *cq; + char *servers_hostport; + int port; + gpr_thd_id tid; + int num_calls_serviced; +} server_fixture; + +typedef struct test_fixture { + server_fixture lb_server; + server_fixture lb_backends[NUM_BACKENDS]; + client_fixture client; + int lb_server_update_delay_ms; +} test_fixture; + +static gpr_timespec n_seconds_time(int n) { + return GRPC_TIMEOUT_SECONDS_TO_DEADLINE(n); +} + +static void *tag(intptr_t t) { return (void *)t; } + +static gpr_slice build_response_payload_slice(const char *host, int *ports, + size_t nports) { + /* + server_list { + servers { + ip_address: "127.0.0.1" + port: ... + load_balance_token: "token..." + } + ... + } */ + char **hostports_vec = gpr_malloc(sizeof(char *) * nports); + for (size_t i = 0; i < nports; i++) { + gpr_join_host_port(&hostports_vec[i], "127.0.0.1", ports[i]); + } + char *hostports_str = + gpr_strjoin_sep((const char **)hostports_vec, nports, " ", NULL); + gpr_log(GPR_INFO, "generating response for %s", hostports_str); + + char *output_fname; + FILE *tmpfd = gpr_tmpfile("grpclb_test", &output_fname); + fclose(tmpfd); + char *cmdline; + gpr_asprintf(&cmdline, + "./tools/codegen/core/gen_grpclb_test_response.py --lb_proto " + "src/proto/grpc/lb/v1/load_balancer.proto %s " + "--output %s --quiet", + hostports_str, output_fname); + GPR_ASSERT(system(cmdline) == 0); + FILE *f = fopen(output_fname, "rb"); + fseek(f, 0, SEEK_END); + const size_t fsize = (size_t)ftell(f); + rewind(f); + + char *serialized_response = gpr_malloc(fsize); + GPR_ASSERT(fread(serialized_response, fsize, 1, f) == 1); + fclose(f); + gpr_free(output_fname); + gpr_free(cmdline); + + for (size_t i = 0; i < nports; i++) { + gpr_free(hostports_vec[i]); + } + gpr_free(hostports_vec); + gpr_free(hostports_str); + + const gpr_slice response_slice = + gpr_slice_from_copied_buffer(serialized_response, fsize); + gpr_free(serialized_response); + return response_slice; +} + +static void drain_cq(grpc_completion_queue *cq) { + grpc_event ev; + do { + ev = grpc_completion_queue_next(cq, n_seconds_time(5), NULL); + } while (ev.type != GRPC_QUEUE_SHUTDOWN); +} + +static void sleep_ms(int delay_ms) { + gpr_sleep_until(gpr_time_add(gpr_now(GPR_CLOCK_REALTIME), + gpr_time_from_millis(delay_ms, GPR_TIMESPAN))); +} + +static void start_lb_server(server_fixture *sf, int *ports, size_t nports, + int update_delay_ms) { + grpc_call *s; + cq_verifier *cqv = cq_verifier_create(sf->cq); + grpc_op ops[6]; + grpc_op *op; + grpc_metadata_array request_metadata_recv; + grpc_call_details call_details; + grpc_call_error error; + int was_cancelled = 2; + grpc_byte_buffer *request_payload_recv; + grpc_byte_buffer *response_payload; + + memset(ops, 0, sizeof(ops)); + grpc_metadata_array_init(&request_metadata_recv); + grpc_call_details_init(&call_details); + + error = grpc_server_request_call(sf->server, &s, &call_details, + &request_metadata_recv, sf->cq, sf->cq, + tag(200)); + GPR_ASSERT(GRPC_CALL_OK == error); + gpr_log(GPR_INFO, "LB Server[%s] up", sf->servers_hostport); + cq_expect_completion(cqv, tag(200), 1); + cq_verify(cqv); + gpr_log(GPR_INFO, "LB Server[%s] after tag 200", sf->servers_hostport); + + op = ops; + op->op = GRPC_OP_SEND_INITIAL_METADATA; + op->data.send_initial_metadata.count = 0; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_CLOSE_ON_SERVER; + op->data.recv_close_on_server.cancelled = &was_cancelled; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(s, ops, (size_t)(op - ops), tag(201), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + gpr_log(GPR_INFO, "LB Server[%s] after tag 201", sf->servers_hostport); + + /* receive request for backends */ + op = ops; + op->op = GRPC_OP_RECV_MESSAGE; + op->data.recv_message = &request_payload_recv; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(s, ops, (size_t)(op - ops), tag(202), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + cq_expect_completion(cqv, tag(202), 1); + cq_verify(cqv); + gpr_log(GPR_INFO, "LB Server[%s] after RECV_MSG", sf->servers_hostport); + // TODO(dgq): validate request. + grpc_byte_buffer_destroy(request_payload_recv); + gpr_slice response_payload_slice; + for (int i = 0; i < 2; i++) { + if (i == 0) { + // First half of the ports. + response_payload_slice = + build_response_payload_slice("127.0.0.1", ports, nports / 2); + } else { + // Second half of the ports. + sleep_ms(update_delay_ms); + response_payload_slice = build_response_payload_slice( + "127.0.0.1", ports + (nports / 2), (nports + 1) / 2 /* ceil */); + } + + response_payload = grpc_raw_byte_buffer_create(&response_payload_slice, 1); + op = ops; + op->op = GRPC_OP_SEND_MESSAGE; + op->data.send_message = response_payload; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(s, ops, (size_t)(op - ops), tag(203), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + cq_expect_completion(cqv, tag(203), 1); + cq_verify(cqv); + gpr_log(GPR_INFO, "LB Server[%s] after SEND_MESSAGE, iter %d", + sf->servers_hostport, i); + + grpc_byte_buffer_destroy(response_payload); + gpr_slice_unref(response_payload_slice); + } + gpr_log(GPR_INFO, "LB Server[%s] shutting down", sf->servers_hostport); + + op = ops; + op->op = GRPC_OP_SEND_STATUS_FROM_SERVER; + op->data.send_status_from_server.trailing_metadata_count = 0; + op->data.send_status_from_server.status = GRPC_STATUS_OK; + op->data.send_status_from_server.status_details = "xyz"; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(s, ops, (size_t)(op - ops), tag(204), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + + cq_expect_completion(cqv, tag(201), 1); + cq_expect_completion(cqv, tag(204), 1); + cq_verify(cqv); + gpr_log(GPR_INFO, "LB Server[%s] after tag 204. All done. LB server out", + sf->servers_hostport); + + grpc_call_destroy(s); + + cq_verifier_destroy(cqv); + + grpc_metadata_array_destroy(&request_metadata_recv); + grpc_call_details_destroy(&call_details); +} + +static void start_backend_server(server_fixture *sf) { + grpc_call *s; + cq_verifier *cqv; + grpc_op ops[6]; + grpc_op *op; + grpc_metadata_array request_metadata_recv; + grpc_call_details call_details; + grpc_call_error error; + int was_cancelled; + grpc_byte_buffer *request_payload_recv; + grpc_byte_buffer *response_payload; + grpc_event ev; + + while (true) { + memset(ops, 0, sizeof(ops)); + cqv = cq_verifier_create(sf->cq); + was_cancelled = 2; + grpc_metadata_array_init(&request_metadata_recv); + grpc_call_details_init(&call_details); + + error = grpc_server_request_call(sf->server, &s, &call_details, + &request_metadata_recv, sf->cq, sf->cq, + tag(100)); + GPR_ASSERT(GRPC_CALL_OK == error); + gpr_log(GPR_INFO, "Server[%s] up", sf->servers_hostport); + ev = grpc_completion_queue_next(sf->cq, n_seconds_time(60), NULL); + if (!ev.success) { + gpr_log(GPR_INFO, "Server[%s] being torn down", sf->servers_hostport); + cq_verifier_destroy(cqv); + grpc_metadata_array_destroy(&request_metadata_recv); + grpc_call_details_destroy(&call_details); + return; + } + GPR_ASSERT(ev.type == GRPC_OP_COMPLETE); + gpr_log(GPR_INFO, "Server[%s] after tag 100", sf->servers_hostport); + + op = ops; + op->op = GRPC_OP_SEND_INITIAL_METADATA; + op->data.send_initial_metadata.count = 0; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_CLOSE_ON_SERVER; + op->data.recv_close_on_server.cancelled = &was_cancelled; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(s, ops, (size_t)(op - ops), tag(101), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + gpr_log(GPR_INFO, "Server[%s] after tag 101", sf->servers_hostport); + + bool exit = false; + gpr_slice response_payload_slice = + gpr_slice_from_copied_string("hello you"); + while (!exit) { + op = ops; + op->op = GRPC_OP_RECV_MESSAGE; + op->data.recv_message = &request_payload_recv; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(s, ops, (size_t)(op - ops), tag(102), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + ev = grpc_completion_queue_next(sf->cq, n_seconds_time(3), NULL); + if (ev.type == GRPC_OP_COMPLETE && ev.success) { + GPR_ASSERT(ev.tag = tag(102)); + if (request_payload_recv == NULL) { + exit = true; + gpr_log(GPR_INFO, + "Server[%s] recv \"close\" from client, exiting. Call #%d", + sf->servers_hostport, sf->num_calls_serviced); + } + } else { + gpr_log(GPR_INFO, "Server[%s] forced to shutdown. Call #%d", + sf->servers_hostport, sf->num_calls_serviced); + exit = true; + } + gpr_log(GPR_INFO, "Server[%s] after tag 102. Call #%d", + sf->servers_hostport, sf->num_calls_serviced); + + if (!exit) { + response_payload = + grpc_raw_byte_buffer_create(&response_payload_slice, 1); + op = ops; + op->op = GRPC_OP_SEND_MESSAGE; + op->data.send_message = response_payload; + op->flags = 0; + op->reserved = NULL; + op++; + error = + grpc_call_start_batch(s, ops, (size_t)(op - ops), tag(103), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + ev = grpc_completion_queue_next(sf->cq, n_seconds_time(3), NULL); + if (ev.type == GRPC_OP_COMPLETE && ev.success) { + GPR_ASSERT(ev.tag = tag(103)); + } else { + gpr_log(GPR_INFO, "Server[%s] forced to shutdown. Call #%d", + sf->servers_hostport, sf->num_calls_serviced); + exit = true; + } + gpr_log(GPR_INFO, "Server[%s] after tag 103. Call #%d", + sf->servers_hostport, sf->num_calls_serviced); + grpc_byte_buffer_destroy(response_payload); + } + + grpc_byte_buffer_destroy(request_payload_recv); + } + ++sf->num_calls_serviced; + + gpr_log(GPR_INFO, "Server[%s] OUT OF THE LOOP", sf->servers_hostport); + gpr_slice_unref(response_payload_slice); + + op = ops; + op->op = GRPC_OP_SEND_STATUS_FROM_SERVER; + op->data.send_status_from_server.trailing_metadata_count = 0; + op->data.send_status_from_server.status = GRPC_STATUS_OK; + op->data.send_status_from_server.status_details = "Backend server out a-ok"; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(s, ops, (size_t)(op - ops), tag(104), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + + cq_expect_completion(cqv, tag(101), 1); + cq_expect_completion(cqv, tag(104), 1); + cq_verify(cqv); + gpr_log(GPR_INFO, "Server[%s] DONE. After servicing %d calls", + sf->servers_hostport, sf->num_calls_serviced); + + grpc_call_destroy(s); + cq_verifier_destroy(cqv); + grpc_metadata_array_destroy(&request_metadata_recv); + grpc_call_details_destroy(&call_details); + } +} + +static void perform_request(client_fixture *cf) { + grpc_call *c; + cq_verifier *cqv = cq_verifier_create(cf->cq); + grpc_op ops[6]; + grpc_op *op; + grpc_metadata_array initial_metadata_recv; + grpc_metadata_array trailing_metadata_recv; + grpc_status_code status; + grpc_call_error error; + char *details = NULL; + size_t details_capacity = 0; + grpc_byte_buffer *request_payload; + grpc_byte_buffer *response_payload_recv; + int i; + + memset(ops, 0, sizeof(ops)); + gpr_slice request_payload_slice = gpr_slice_from_copied_string("hello world"); + + c = grpc_channel_create_call(cf->client, NULL, GRPC_PROPAGATE_DEFAULTS, + cf->cq, "/foo", "foo.test.google.fr:1234", + n_seconds_time(1000), NULL); + gpr_log(GPR_INFO, "Call 0x%" PRIxPTR " created", (intptr_t)c); + GPR_ASSERT(c); + char *peer; + + grpc_metadata_array_init(&initial_metadata_recv); + grpc_metadata_array_init(&trailing_metadata_recv); + + op = ops; + op->op = GRPC_OP_SEND_INITIAL_METADATA; + op->data.send_initial_metadata.count = 0; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_INITIAL_METADATA; + op->data.recv_initial_metadata = &initial_metadata_recv; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_STATUS_ON_CLIENT; + op->data.recv_status_on_client.trailing_metadata = &trailing_metadata_recv; + op->data.recv_status_on_client.status = &status; + op->data.recv_status_on_client.status_details = &details; + op->data.recv_status_on_client.status_details_capacity = &details_capacity; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(c, ops, (size_t)(op - ops), tag(1), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + + for (i = 0; i < 4; i++) { + request_payload = grpc_raw_byte_buffer_create(&request_payload_slice, 1); + + op = ops; + op->op = GRPC_OP_SEND_MESSAGE; + op->data.send_message = request_payload; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_MESSAGE; + op->data.recv_message = &response_payload_recv; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(c, ops, (size_t)(op - ops), tag(2), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + + peer = grpc_call_get_peer(c); + cq_expect_completion(cqv, tag(2), 1); + cq_verify(cqv); + gpr_free(peer); + + grpc_byte_buffer_destroy(request_payload); + grpc_byte_buffer_destroy(response_payload_recv); + } + + gpr_slice_unref(request_payload_slice); + + op = ops; + op->op = GRPC_OP_SEND_CLOSE_FROM_CLIENT; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(c, ops, (size_t)(op - ops), tag(3), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + + cq_expect_completion(cqv, tag(1), 1); + cq_expect_completion(cqv, tag(3), 1); + cq_verify(cqv); + peer = grpc_call_get_peer(c); + gpr_log(GPR_INFO, "Client DONE WITH SERVER %s ", peer); + gpr_free(peer); + + grpc_call_destroy(c); + + cq_verify_empty_timeout(cqv, 1); + cq_verifier_destroy(cqv); + + grpc_metadata_array_destroy(&initial_metadata_recv); + grpc_metadata_array_destroy(&trailing_metadata_recv); + gpr_free(details); +} + +static void setup_client(const char *server_hostport, client_fixture *cf) { + cf->cq = grpc_completion_queue_create(NULL); + cf->server_uri = gpr_strdup(server_hostport); + cf->client = grpc_insecure_channel_create(cf->server_uri, NULL, NULL); +} + +static void teardown_client(client_fixture *cf) { + grpc_completion_queue_shutdown(cf->cq); + drain_cq(cf->cq); + grpc_completion_queue_destroy(cf->cq); + cf->cq = NULL; + grpc_channel_destroy(cf->client); + cf->client = NULL; + gpr_free(cf->server_uri); +} + +static void setup_server(const char *host, server_fixture *sf) { + int assigned_port; + + sf->cq = grpc_completion_queue_create(NULL); + char *colon_idx = strchr(host, ':'); + if (colon_idx) { + char *port_str = colon_idx + 1; + sf->port = atoi(port_str); + sf->servers_hostport = gpr_strdup(host); + } else { + sf->port = grpc_pick_unused_port_or_die(); + gpr_join_host_port(&sf->servers_hostport, host, sf->port); + } + + sf->server = grpc_server_create(NULL, NULL); + grpc_server_register_completion_queue(sf->server, sf->cq, NULL); + GPR_ASSERT((assigned_port = grpc_server_add_insecure_http2_port( + sf->server, sf->servers_hostport)) > 0); + GPR_ASSERT(sf->port == assigned_port); + grpc_server_start(sf->server); +} + +static void teardown_server(server_fixture *sf) { + if (!sf->server) return; + + gpr_log(GPR_INFO, "Server[%s] shutting down", sf->servers_hostport); + grpc_server_shutdown_and_notify(sf->server, sf->cq, tag(1000)); + GPR_ASSERT(grpc_completion_queue_pluck( + sf->cq, tag(1000), GRPC_TIMEOUT_SECONDS_TO_DEADLINE(5), NULL) + .type == GRPC_OP_COMPLETE); + grpc_server_destroy(sf->server); + gpr_thd_join(sf->tid); + + sf->server = NULL; + grpc_completion_queue_shutdown(sf->cq); + drain_cq(sf->cq); + grpc_completion_queue_destroy(sf->cq); + + gpr_log(GPR_INFO, "Server[%s] bye bye", sf->servers_hostport); + gpr_free(sf->servers_hostport); +} + +static void fork_backend_server(void *arg) { + server_fixture *sf = arg; + start_backend_server(sf); +} + +static void fork_lb_server(void *arg) { + test_fixture *tf = arg; + int ports[NUM_BACKENDS]; + for (int i = 0; i < NUM_BACKENDS; i++) { + ports[i] = tf->lb_backends[i].port; + } + start_lb_server(&tf->lb_server, ports, NUM_BACKENDS, + tf->lb_server_update_delay_ms); +} + +static void setup_test_fixture(test_fixture *tf, + int lb_server_update_delay_ms) { + tf->lb_server_update_delay_ms = lb_server_update_delay_ms; + + gpr_thd_options options = gpr_thd_options_default(); + gpr_thd_options_set_joinable(&options); + + for (int i = 0; i < NUM_BACKENDS; ++i) { + setup_server("127.0.0.1", &tf->lb_backends[i]); + gpr_thd_new(&tf->lb_backends[i].tid, fork_backend_server, + &tf->lb_backends[i], &options); + } + + setup_server("127.0.0.1", &tf->lb_server); + gpr_thd_new(&tf->lb_server.tid, fork_lb_server, &tf->lb_server, &options); + + char *server_uri; + gpr_asprintf(&server_uri, "ipv4:%s?lb_policy=grpclb&lb_enabled=1", + tf->lb_server.servers_hostport); + setup_client(server_uri, &tf->client); + gpr_free(server_uri); +} + +static void teardown_test_fixture(test_fixture *tf) { + teardown_client(&tf->client); + for (int i = 0; i < NUM_BACKENDS; ++i) { + teardown_server(&tf->lb_backends[i]); + } + teardown_server(&tf->lb_server); +} + +// The LB server will send two updates: batch 1 and batch 2. Each batch contains +// two addresses, both of a valid and running backend server. Batch 1 is readily +// available and provided as soon as the client establishes the streaming call. +// Batch 2 is sent after a delay of \a lb_server_update_delay_ms milliseconds. +static test_fixture test_update(int lb_server_update_delay_ms) { + test_fixture tf; + memset(&tf, 0, sizeof(tf)); + setup_test_fixture(&tf, lb_server_update_delay_ms); + perform_request( + &tf.client); // "consumes" 1st backend server of 1st serverlist + perform_request( + &tf.client); // "consumes" 2nd backend server of 1st serverlist + + perform_request( + &tf.client); // "consumes" 1st backend server of 2nd serverlist + perform_request( + &tf.client); // "consumes" 2nd backend server of 2nd serverlist + + teardown_test_fixture(&tf); + return tf; +} + +int main(int argc, char **argv) { + grpc_test_init(argc, argv); + grpc_init(); + + test_fixture tf_result; + // Clients take a bit over one second to complete a call (the last part of the + // call sleeps for 1 second while verifying the client's completion queue is + // empty). Therefore: + // + // If the LB server waits 800ms before sending an update, it will arrive + // before the first client request is done, skipping the second server from + // batch 1 altogether: the 2nd client request will go to the 1st server of + // batch 2 (ie, the third one out of the four total servers). + tf_result = test_update(800); + GPR_ASSERT(tf_result.lb_backends[0].num_calls_serviced == 1); + GPR_ASSERT(tf_result.lb_backends[1].num_calls_serviced == 0); + GPR_ASSERT(tf_result.lb_backends[2].num_calls_serviced == 2); + GPR_ASSERT(tf_result.lb_backends[3].num_calls_serviced == 1); + + // If the LB server waits 1500ms, the update arrives after having picked the + // 2nd server from batch 1 but before the next pick for the first server of + // batch 2. All server are used. + tf_result = test_update(1500); + GPR_ASSERT(tf_result.lb_backends[0].num_calls_serviced == 1); + GPR_ASSERT(tf_result.lb_backends[1].num_calls_serviced == 1); + GPR_ASSERT(tf_result.lb_backends[2].num_calls_serviced == 1); + GPR_ASSERT(tf_result.lb_backends[3].num_calls_serviced == 1); + + // If the LB server waits >= 2000ms, the update arrives after the first two + // request are done and the third pick is performed, which returns, in RR + // fashion, the 1st server of the 1st update. Therefore, the second server of + // batch 1 is hit twice, whereas the first server of batch 2 is never hit. + tf_result = test_update(2000); + GPR_ASSERT(tf_result.lb_backends[0].num_calls_serviced == 2); + GPR_ASSERT(tf_result.lb_backends[1].num_calls_serviced == 1); + GPR_ASSERT(tf_result.lb_backends[2].num_calls_serviced == 1); + GPR_ASSERT(tf_result.lb_backends[3].num_calls_serviced == 0); + + grpc_shutdown(); + return 0; +} diff --git a/test/core/end2end/cq_verifier.c b/test/core/end2end/cq_verifier.c index 8e9fa70b0ec..060e166e6eb 100644 --- a/test/core/end2end/cq_verifier.c +++ b/test/core/end2end/cq_verifier.c @@ -258,9 +258,10 @@ void cq_verify(cq_verifier *v) { gpr_strvec_destroy(&have_tags); } -void cq_verify_empty(cq_verifier *v) { - gpr_timespec deadline = gpr_time_add(gpr_now(GPR_CLOCK_REALTIME), - gpr_time_from_seconds(1, GPR_TIMESPAN)); +void cq_verify_empty_timeout(cq_verifier *v, int timeout_sec) { + gpr_timespec deadline = + gpr_time_add(gpr_now(GPR_CLOCK_REALTIME), + gpr_time_from_seconds(timeout_sec, GPR_TIMESPAN)); grpc_event ev; GPR_ASSERT(v->expect.next == &v->expect && "expectation queue must be empty"); @@ -274,6 +275,8 @@ void cq_verify_empty(cq_verifier *v) { } } +void cq_verify_empty(cq_verifier *v) { cq_verify_empty_timeout(v, 1); } + static expectation *add(cq_verifier *v, grpc_completion_type type, void *tag) { expectation *e = gpr_malloc(sizeof(expectation)); e->type = type; diff --git a/test/core/end2end/cq_verifier.h b/test/core/end2end/cq_verifier.h index 8c9a85c2187..bf82468c9a2 100644 --- a/test/core/end2end/cq_verifier.h +++ b/test/core/end2end/cq_verifier.h @@ -55,6 +55,9 @@ void cq_verify(cq_verifier *v); /* ensure that the completion queue is empty */ void cq_verify_empty(cq_verifier *v); +/* ensure that the completion queue is empty, waiting up to \a timeout secs. */ +void cq_verify_empty_timeout(cq_verifier *v, int timeout_sec); + /* Various expectation matchers Any functions taking ... expect a NULL terminated list of key/value pairs (each pair using two parameter slots) of metadata that MUST be present in diff --git a/tools/codegen/core/gen_grpclb_test_response.py b/tools/codegen/core/gen_grpclb_test_response.py new file mode 100755 index 00000000000..4535efc738c --- /dev/null +++ b/tools/codegen/core/gen_grpclb_test_response.py @@ -0,0 +1,97 @@ +#!/usr/bin/env python2.7 + +# Copyright 2015-2016, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from __future__ import print_function +import argparse +import subprocess +import sys +import os.path +import sys +import tempfile +import importlib + +# Example: tools/codegen/core/gen_grpclb_test_response.py \ +# --lb_proto src/proto/grpc/lb/v1/load_balancer.proto \ +# 127.0.0.1:1234 10.0.0.1:4321 + +# 1) Compile src/proto/grpc/lb/v1/load_balancer.proto to a temp location +parser = argparse.ArgumentParser() +parser.add_argument('--lb_proto', required=True) +parser.add_argument('-e', '--expiration_interval_secs', type=int) +parser.add_argument('-o', '--output') +parser.add_argument('-q', '--quiet', default=False, action='store_true') +parser.add_argument('ipports', nargs='+') +args = parser.parse_args() + +if not os.path.isfile(args.lb_proto): + print("ERROR: file '{}' cannot be accessed (not found, no permissions, etc.)" + .format(args.lb_proto), file=sys.stderr) + sys.exit(1) + +proto_dirname = os.path.dirname(args.lb_proto) +output_dir = tempfile.mkdtemp() + +protoc_cmd = 'protoc -I{} --python_out={} {}'.format( + proto_dirname, output_dir, args.lb_proto) + +with tempfile.TemporaryFile() as stderr_tmpfile: + if subprocess.call(protoc_cmd, stderr=stderr_tmpfile, shell=True) != 0: + stderr_tmpfile.seek(0) + print("ERROR: while running '{}': {}". + format(protoc_cmd, stderr_tmpfile.read())) + sys.exit(2) + +# 2) import the output .py file. +module_name = os.path.splitext(os.path.basename(args.lb_proto))[0] + '_pb2' +sys.path.append(output_dir) +pb_module = importlib.import_module(module_name) + +# 3) Generate! +lb_response = pb_module.LoadBalanceResponse() +if args.expiration_interval_secs: + lb_response.server_list.expiration_interval.seconds = \ + args.expiration_interval_secs + +for ipport in args.ipports: + ip, port = ipport.split(':') + server = lb_response.server_list.servers.add() + server.ip_address = ip + server.port = int(port) + server.load_balance_token = b'token{}'.format(port) + +serialized_bytes = lb_response.SerializeToString() +serialized_hex = ''.join('\\x{:02x}'.format(ord(c)) for c in serialized_bytes) +if args.output: + with open(args.output, 'w') as f: + f.write(serialized_bytes) +if not args.quiet: + print(str(lb_response)) + print(serialized_hex) diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index aee866adf46..62ae97a9a3c 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -1100,6 +1100,7 @@ src/core/ext/transport/chttp2/server/insecure/server_chttp2.c \ src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c \ src/core/ext/transport/chttp2/client/insecure/channel_create.c \ src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c \ +src/core/ext/lb_policy/grpclb/grpclb.c \ src/core/ext/lb_policy/grpclb/load_balancer_api.c \ src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c \ third_party/nanopb/pb_common.c \ diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 4aad52c69d0..b2b0d5bf997 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -944,6 +944,22 @@ "third_party": false, "type": "target" }, + { + "deps": [ + "gpr", + "gpr_test_util", + "grpc", + "grpc_test_util" + ], + "headers": [], + "language": "c", + "name": "grpclb_test", + "src": [ + "test/core/client_config/grpclb_test.c" + ], + "third_party": false, + "type": "target" + }, { "deps": [ "gpr", @@ -4152,6 +4168,7 @@ "gpr", "grpc_base", "grpc_lb_policy_grpclb", + "grpc_lb_policy_grpclb", "grpc_lb_policy_pick_first", "grpc_lb_policy_round_robin", "grpc_load_reporting", @@ -4246,6 +4263,7 @@ "gpr", "grpc_base", "grpc_lb_policy_grpclb", + "grpc_lb_policy_grpclb", "grpc_lb_policy_pick_first", "grpc_lb_policy_round_robin", "grpc_load_reporting", @@ -6048,6 +6066,7 @@ "language": "c", "name": "grpc_lb_policy_grpclb", "src": [ + "src/core/ext/lb_policy/grpclb/grpclb.c", "src/core/ext/lb_policy/grpclb/load_balancer_api.c", "src/core/ext/lb_policy/grpclb/load_balancer_api.h", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c", diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index 5a84a41b638..bf439b43e9c 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -1079,6 +1079,27 @@ "windows" ] }, + { + "args": [], + "ci_platforms": [ + "linux", + "mac", + "posix", + "windows" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "gtest": false, + "language": "c", + "name": "grpclb_test", + "platforms": [ + "linux", + "mac", + "posix", + "windows" + ] + }, { "args": [], "ci_platforms": [ diff --git a/vsprojects/buildtests_c.sln b/vsprojects/buildtests_c.sln index 3a214a4ad41..070f6559b7b 100644 --- a/vsprojects/buildtests_c.sln +++ b/vsprojects/buildtests_c.sln @@ -643,6 +643,17 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_verify_jwt", "vcxproj\ {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpclb_test", "vcxproj\test\grpclb_test\grpclb_test.vcxproj", "{9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}" + ProjectSection(myProperties) = preProject + lib = "False" + EndProjectSection + ProjectSection(ProjectDependencies) = postProject + {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} + {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} + {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + EndProjectSection +EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "h2_census_nosec_test", "vcxproj\test/end2end/fixtures\h2_census_nosec_test\h2_census_nosec_test.vcxproj", "{A8039D43-910E-4248-2A22-74366E8C4DCD}" ProjectSection(myProperties) = preProject lib = "False" @@ -2455,6 +2466,22 @@ Global {02FAC25F-5FF6-34A0-00AE-B82BFBA851A9}.Release-DLL|Win32.Build.0 = Release|Win32 {02FAC25F-5FF6-34A0-00AE-B82BFBA851A9}.Release-DLL|x64.ActiveCfg = Release|x64 {02FAC25F-5FF6-34A0-00AE-B82BFBA851A9}.Release-DLL|x64.Build.0 = Release|x64 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug|Win32.ActiveCfg = Debug|Win32 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug|x64.ActiveCfg = Debug|x64 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release|Win32.ActiveCfg = Release|Win32 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release|x64.ActiveCfg = Release|x64 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug|Win32.Build.0 = Debug|Win32 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug|x64.Build.0 = Debug|x64 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release|Win32.Build.0 = Release|Win32 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release|x64.Build.0 = Release|x64 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug-DLL|Win32.ActiveCfg = Debug|Win32 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug-DLL|Win32.Build.0 = Debug|Win32 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug-DLL|x64.ActiveCfg = Debug|x64 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug-DLL|x64.Build.0 = Debug|x64 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release-DLL|Win32.ActiveCfg = Release|Win32 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release-DLL|Win32.Build.0 = Release|Win32 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release-DLL|x64.ActiveCfg = Release|x64 + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release-DLL|x64.Build.0 = Release|x64 {A8039D43-910E-4248-2A22-74366E8C4DCD}.Debug|Win32.ActiveCfg = Debug|Win32 {A8039D43-910E-4248-2A22-74366E8C4DCD}.Debug|x64.ActiveCfg = Debug|x64 {A8039D43-910E-4248-2A22-74366E8C4DCD}.Release|Win32.ActiveCfg = Release|Win32 diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj b/vsprojects/vcxproj/grpc/grpc.vcxproj index ec0c984f2e7..2473525a97f 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj @@ -770,6 +770,8 @@ + + diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters index 273f73b2a30..28c6483cf2a 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters @@ -478,6 +478,9 @@ src\core\ext\transport\chttp2\client\insecure + + src\core\ext\lb_policy\grpclb + src\core\ext\lb_policy\grpclb diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj index bba099d803c..6e1b3ad05aa 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj @@ -688,6 +688,8 @@ + + diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters index 78a8777f942..ce599a6e66f 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters @@ -406,6 +406,9 @@ src\core\ext\load_reporting + + src\core\ext\lb_policy\grpclb + src\core\ext\lb_policy\grpclb diff --git a/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj b/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj new file mode 100644 index 00000000000..7ea8e06d74a --- /dev/null +++ b/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj @@ -0,0 +1,199 @@ + + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB} + true + $(SolutionDir)IntDir\$(MSBuildProjectName)\ + + + + v100 + + + v110 + + + v120 + + + v140 + + + Application + true + Unicode + + + Application + false + true + Unicode + + + + + + + + + + + + + + grpclb_test + static + Debug + static + Debug + + + grpclb_test + static + Release + static + Release + + + + NotUsing + Level3 + Disabled + WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) + true + MultiThreadedDebug + true + None + false + + + Console + true + false + + + + + + NotUsing + Level3 + Disabled + WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) + true + MultiThreadedDebug + true + None + false + + + Console + true + false + + + + + + NotUsing + Level3 + MaxSpeed + WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) + true + true + true + MultiThreaded + true + None + false + + + Console + true + false + true + true + + + + + + NotUsing + Level3 + MaxSpeed + WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) + true + true + true + MultiThreaded + true + None + false + + + Console + true + false + true + true + + + + + + + + + + {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} + + + {29D16885-7228-4C31-81ED-5F9187C7F2A9} + + + {EAB0A629-17A9-44DB-B5FF-E91A721FE037} + + + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + diff --git a/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj.filters b/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj.filters new file mode 100644 index 00000000000..7a70c0b5e1c --- /dev/null +++ b/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj.filters @@ -0,0 +1,21 @@ + + + + + test\core\client_config + + + + + + {e0ba55a2-37d9-5029-6f4e-64f097307340} + + + {6d1d02a2-d635-142d-16d4-45bd59f5c83a} + + + {356b8663-f1cf-f0df-39d0-d2de958699d0} + + + + From eaef69b572265d3b282cadf33f5e452f31e164ae Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 15 Jun 2016 23:52:25 -0700 Subject: [PATCH 0101/1039] removed unused includes --- test/core/client_config/grpclb_test.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/test/core/client_config/grpclb_test.c b/test/core/client_config/grpclb_test.c index 2080fefd1ae..63f32c241d3 100644 --- a/test/core/client_config/grpclb_test.c +++ b/test/core/client_config/grpclb_test.c @@ -32,9 +32,7 @@ */ #include -#include #include -#include #include #include From 22e8f1db07889afc85f935ad3f5a64b876b73339 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 15 Jun 2016 23:53:00 -0700 Subject: [PATCH 0102/1039] reordered includes --- src/core/ext/lb_policy/grpclb/grpclb.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/core/ext/lb_policy/grpclb/grpclb.c b/src/core/ext/lb_policy/grpclb/grpclb.c index cfa9669e0a2..852e78f0202 100644 --- a/src/core/ext/lb_policy/grpclb/grpclb.c +++ b/src/core/ext/lb_policy/grpclb/grpclb.c @@ -31,6 +31,14 @@ * */ +#include + +#include +#include +#include +#include +#include + #include "src/core/ext/lb_policy/grpclb/grpclb.h" #include "src/core/ext/client_config/client_channel_factory.h" #include "src/core/ext/client_config/lb_policy_registry.h" @@ -41,14 +49,6 @@ #include "src/core/lib/surface/call.h" #include "src/core/lib/surface/channel.h" -#include - -#include -#include -#include -#include -#include - int grpc_lb_glb_trace = 0; typedef struct wrapped_rr_closure_arg { From b2e986b3fa333135533b3e050229c259aac7376c Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 15 Jun 2016 23:55:50 -0700 Subject: [PATCH 0103/1039] added missing grpclb.h dep in build.yaml --- BUILD | 3 +++ build.yaml | 1 + gRPC.podspec | 2 ++ grpc.gemspec | 1 + package.xml | 1 + tools/doxygen/Doxyfile.core.internal | 1 + tools/run_tests/sources_and_headers.json | 2 ++ vsprojects/vcxproj/grpc/grpc.vcxproj | 1 + vsprojects/vcxproj/grpc/grpc.vcxproj.filters | 3 +++ vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj | 1 + vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters | 3 +++ 11 files changed, 19 insertions(+) diff --git a/BUILD b/BUILD index 42a35a310d6..e0de092d674 100644 --- a/BUILD +++ b/BUILD @@ -297,6 +297,7 @@ cc_library( "src/core/ext/client_config/subchannel_call_holder.h", "src/core/ext/client_config/subchannel_index.h", "src/core/ext/client_config/uri_parser.h", + "src/core/ext/lb_policy/grpclb/grpclb.h", "src/core/ext/lb_policy/grpclb/load_balancer_api.h", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h", "src/core/ext/load_reporting/load_reporting.h", @@ -998,6 +999,7 @@ cc_library( "src/core/ext/client_config/uri_parser.h", "src/core/ext/load_reporting/load_reporting.h", "src/core/ext/load_reporting/load_reporting_filter.h", + "src/core/ext/lb_policy/grpclb/grpclb.h", "src/core/ext/lb_policy/grpclb/load_balancer_api.h", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h", "src/core/ext/census/aggregation.h", @@ -2133,6 +2135,7 @@ objc_library( "src/core/ext/client_config/subchannel_call_holder.h", "src/core/ext/client_config/subchannel_index.h", "src/core/ext/client_config/uri_parser.h", + "src/core/ext/lb_policy/grpclb/grpclb.h", "src/core/ext/lb_policy/grpclb/load_balancer_api.h", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h", "src/core/ext/load_reporting/load_reporting.h", diff --git a/build.yaml b/build.yaml index d5d1b816a68..ee854361a19 100644 --- a/build.yaml +++ b/build.yaml @@ -374,6 +374,7 @@ filegroups: - gpr_codegen - name: grpc_lb_policy_grpclb headers: + - src/core/ext/lb_policy/grpclb/grpclb.h - src/core/ext/lb_policy/grpclb/load_balancer_api.h - src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h src: diff --git a/gRPC.podspec b/gRPC.podspec index ab04e494a31..bbf18ee0e59 100644 --- a/gRPC.podspec +++ b/gRPC.podspec @@ -300,6 +300,7 @@ Pod::Spec.new do |s| 'src/core/ext/client_config/subchannel_call_holder.h', 'src/core/ext/client_config/subchannel_index.h', 'src/core/ext/client_config/uri_parser.h', + 'src/core/ext/lb_policy/grpclb/grpclb.h', 'src/core/ext/lb_policy/grpclb/load_balancer_api.h', 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h', 'third_party/nanopb/pb.h', @@ -676,6 +677,7 @@ Pod::Spec.new do |s| 'src/core/ext/client_config/subchannel_call_holder.h', 'src/core/ext/client_config/subchannel_index.h', 'src/core/ext/client_config/uri_parser.h', + 'src/core/ext/lb_policy/grpclb/grpclb.h', 'src/core/ext/lb_policy/grpclb/load_balancer_api.h', 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h', 'third_party/nanopb/pb.h', diff --git a/grpc.gemspec b/grpc.gemspec index d30d7ae30e9..dea6d4f5189 100755 --- a/grpc.gemspec +++ b/grpc.gemspec @@ -309,6 +309,7 @@ Gem::Specification.new do |s| s.files += %w( src/core/ext/client_config/subchannel_call_holder.h ) s.files += %w( src/core/ext/client_config/subchannel_index.h ) s.files += %w( src/core/ext/client_config/uri_parser.h ) + s.files += %w( src/core/ext/lb_policy/grpclb/grpclb.h ) s.files += %w( src/core/ext/lb_policy/grpclb/load_balancer_api.h ) s.files += %w( src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h ) s.files += %w( third_party/nanopb/pb.h ) diff --git a/package.xml b/package.xml index 2c718608647..ef52eba3fed 100644 --- a/package.xml +++ b/package.xml @@ -316,6 +316,7 @@ + diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index 62ae97a9a3c..057c385dd04 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -926,6 +926,7 @@ src/core/ext/client_config/subchannel.h \ src/core/ext/client_config/subchannel_call_holder.h \ src/core/ext/client_config/subchannel_index.h \ src/core/ext/client_config/uri_parser.h \ +src/core/ext/lb_policy/grpclb/grpclb.h \ src/core/ext/lb_policy/grpclb/load_balancer_api.h \ src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h \ third_party/nanopb/pb.h \ diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index b2b0d5bf997..492be7322f6 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -6060,6 +6060,7 @@ "nanopb" ], "headers": [ + "src/core/ext/lb_policy/grpclb/grpclb.h", "src/core/ext/lb_policy/grpclb/load_balancer_api.h", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h" ], @@ -6067,6 +6068,7 @@ "name": "grpc_lb_policy_grpclb", "src": [ "src/core/ext/lb_policy/grpclb/grpclb.c", + "src/core/ext/lb_policy/grpclb/grpclb.h", "src/core/ext/lb_policy/grpclb/load_balancer_api.c", "src/core/ext/lb_policy/grpclb/load_balancer_api.h", "src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c", diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj b/vsprojects/vcxproj/grpc/grpc.vcxproj index 2473525a97f..ee1a6a979fb 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj @@ -435,6 +435,7 @@ + diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters index 28c6483cf2a..ea349a3ed69 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters @@ -1049,6 +1049,9 @@ src\core\ext\client_config + + src\core\ext\lb_policy\grpclb + src\core\ext\lb_policy\grpclb diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj index 6e1b3ad05aa..2ad6dd7d664 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj @@ -403,6 +403,7 @@ + diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters index ce599a6e66f..1612cef4392 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters @@ -896,6 +896,9 @@ src\core\ext\load_reporting + + src\core\ext\lb_policy\grpclb + src\core\ext\lb_policy\grpclb From 8782d1b7e104b74e0bf1ecba386420088140c76c Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 15 Jun 2016 23:58:44 -0700 Subject: [PATCH 0104/1039] clang-format --- src/core/ext/lb_policy/grpclb/grpclb.c | 11 +++++++---- src/core/ext/lb_policy/grpclb/load_balancer_api.c | 2 +- test/core/client_config/grpclb_test.c | 2 +- 3 files changed, 9 insertions(+), 6 deletions(-) diff --git a/src/core/ext/lb_policy/grpclb/grpclb.c b/src/core/ext/lb_policy/grpclb/grpclb.c index 852e78f0202..26cead51b0e 100644 --- a/src/core/ext/lb_policy/grpclb/grpclb.c +++ b/src/core/ext/lb_policy/grpclb/grpclb.c @@ -39,10 +39,10 @@ #include #include -#include "src/core/ext/lb_policy/grpclb/grpclb.h" #include "src/core/ext/client_config/client_channel_factory.h" #include "src/core/ext/client_config/lb_policy_registry.h" #include "src/core/ext/client_config/parse_address.h" +#include "src/core/ext/lb_policy/grpclb/grpclb.h" #include "src/core/ext/lb_policy/grpclb/load_balancer_api.h" #include "src/core/lib/iomgr/sockaddr_utils.h" #include "src/core/lib/support/string.h" @@ -635,7 +635,8 @@ static void rr_handover(grpc_exec_ctx *exec_ctx, glb_lb_policy *p) { GRPC_LB_POLICY_REF(p->rr_policy, "rr_handover_pending_pick"); pp->wrapped_on_complete_arg->rr_policy = p->rr_policy; if (grpc_lb_glb_trace) { - gpr_log(GPR_INFO, "Pending pick about to PICK from 0x%"PRIxPTR"", (intptr_t)p->rr_policy); + gpr_log(GPR_INFO, "Pending pick about to PICK from 0x%" PRIxPTR "", + (intptr_t)p->rr_policy); } grpc_lb_policy_pick(exec_ctx, p->rr_policy, pp->pollent, pp->initial_metadata, pp->initial_metadata_flags, @@ -649,7 +650,8 @@ static void rr_handover(grpc_exec_ctx *exec_ctx, glb_lb_policy *p) { GRPC_LB_POLICY_REF(p->rr_policy, "rr_handover_pending_ping"); pping->wrapped_notify_arg->rr_policy = p->rr_policy; if (grpc_lb_glb_trace) { - gpr_log(GPR_INFO, "Pending ping about to PING from 0x%"PRIxPTR"", (intptr_t)p->rr_policy); + gpr_log(GPR_INFO, "Pending ping about to PING from 0x%" PRIxPTR "", + (intptr_t)p->rr_policy); } grpc_lb_policy_ping_one(exec_ctx, p->rr_policy, pping->wrapped_notify); gpr_free(pping); @@ -677,7 +679,8 @@ static int glb_pick(grpc_exec_ctx *exec_ctx, grpc_lb_policy *pol, if (p->rr_policy != NULL) { if (grpc_lb_glb_trace) { - gpr_log(GPR_INFO, "about to PICK from 0x%"PRIxPTR"", (intptr_t)p->rr_policy); + gpr_log(GPR_INFO, "about to PICK from 0x%" PRIxPTR "", + (intptr_t)p->rr_policy); } GRPC_LB_POLICY_REF(p->rr_policy, "rr_pick"); wrapped_rr_closure_arg *warg = gpr_malloc(sizeof(wrapped_rr_closure_arg)); diff --git a/src/core/ext/lb_policy/grpclb/load_balancer_api.c b/src/core/ext/lb_policy/grpclb/load_balancer_api.c index c3b3888e90b..6c77612e0d4 100644 --- a/src/core/ext/lb_policy/grpclb/load_balancer_api.c +++ b/src/core/ext/lb_policy/grpclb/load_balancer_api.c @@ -176,7 +176,7 @@ grpc_grpclb_serverlist *grpc_grpclb_serverlist_copy( copy->num_servers = sl->num_servers; memcpy(©->expiration_interval, &sl->expiration_interval, sizeof(grpc_grpclb_duration)); - copy->servers = gpr_malloc(sizeof(grpc_grpclb_server*) * sl->num_servers); + copy->servers = gpr_malloc(sizeof(grpc_grpclb_server *) * sl->num_servers); for (size_t i = 0; i < sl->num_servers; i++) { copy->servers[i] = gpr_malloc(sizeof(grpc_grpclb_server)); memcpy(copy->servers[i], sl->servers[i], sizeof(grpc_grpclb_server)); diff --git a/test/core/client_config/grpclb_test.c b/test/core/client_config/grpclb_test.c index 63f32c241d3..85cfe80748f 100644 --- a/test/core/client_config/grpclb_test.c +++ b/test/core/client_config/grpclb_test.c @@ -387,7 +387,7 @@ static void start_backend_server(server_fixture *sf) { cq_expect_completion(cqv, tag(104), 1); cq_verify(cqv); gpr_log(GPR_INFO, "Server[%s] DONE. After servicing %d calls", - sf->servers_hostport, sf->num_calls_serviced); + sf->servers_hostport, sf->num_calls_serviced); grpc_call_destroy(s); cq_verifier_destroy(cqv); From c459ecf7c9beac5e861ade0dd4686348c1bbfdc6 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 16 Jun 2016 09:17:49 -0700 Subject: [PATCH 0105/1039] - fix build problems - make changes suggested by reviewer - add test (not working yet) --- Makefile | 47 +++ build.yaml | 13 + include/grpc++/channel_filter.h | 68 ++-- src/core/lib/channel/channel_stack.h | 8 + src/core/lib/channel/channel_stack_builder.h | 8 + src/core/lib/surface/channel_init.h | 8 + src/cpp/common/channel_filter.cc | 17 +- test/cpp/end2end/filter_end2end_test.cc | 327 ++++++++++++++++++ tools/run_tests/sources_and_headers.json | 18 + tools/run_tests/tests.json | 21 ++ .../filter_end2end_test.vcxproj | 207 +++++++++++ .../filter_end2end_test.vcxproj.filters | 21 ++ 12 files changed, 721 insertions(+), 42 deletions(-) create mode 100644 test/cpp/end2end/filter_end2end_test.cc create mode 100644 vsprojects/vcxproj/test/filter_end2end_test/filter_end2end_test.vcxproj create mode 100644 vsprojects/vcxproj/test/filter_end2end_test/filter_end2end_test.vcxproj.filters diff --git a/Makefile b/Makefile index 2f88411c95b..2556b749207 100644 --- a/Makefile +++ b/Makefile @@ -1014,6 +1014,7 @@ cxx_slice_test: $(BINDIR)/$(CONFIG)/cxx_slice_test cxx_string_ref_test: $(BINDIR)/$(CONFIG)/cxx_string_ref_test cxx_time_test: $(BINDIR)/$(CONFIG)/cxx_time_test end2end_test: $(BINDIR)/$(CONFIG)/end2end_test +filter_end2end_test: $(BINDIR)/$(CONFIG)/filter_end2end_test generic_end2end_test: $(BINDIR)/$(CONFIG)/generic_end2end_test golden_file_test: $(BINDIR)/$(CONFIG)/golden_file_test grpc_cli: $(BINDIR)/$(CONFIG)/grpc_cli @@ -1393,6 +1394,7 @@ buildtests_cxx: buildtests_zookeeper privatelibs_cxx \ $(BINDIR)/$(CONFIG)/cxx_string_ref_test \ $(BINDIR)/$(CONFIG)/cxx_time_test \ $(BINDIR)/$(CONFIG)/end2end_test \ + $(BINDIR)/$(CONFIG)/filter_end2end_test \ $(BINDIR)/$(CONFIG)/generic_end2end_test \ $(BINDIR)/$(CONFIG)/golden_file_test \ $(BINDIR)/$(CONFIG)/grpc_cli \ @@ -1721,6 +1723,8 @@ test_cxx: test_zookeeper buildtests_cxx $(Q) $(BINDIR)/$(CONFIG)/cxx_time_test || ( echo test cxx_time_test failed ; exit 1 ) $(E) "[RUN] Testing end2end_test" $(Q) $(BINDIR)/$(CONFIG)/end2end_test || ( echo test end2end_test failed ; exit 1 ) + $(E) "[RUN] Testing filter_end2end_test" + $(Q) $(BINDIR)/$(CONFIG)/filter_end2end_test || ( echo test filter_end2end_test failed ; exit 1 ) $(E) "[RUN] Testing generic_end2end_test" $(Q) $(BINDIR)/$(CONFIG)/generic_end2end_test || ( echo test generic_end2end_test failed ; exit 1 ) $(E) "[RUN] Testing golden_file_test" @@ -10851,6 +10855,49 @@ endif endif +FILTER_END2END_TEST_SRC = \ + test/cpp/end2end/filter_end2end_test.cc \ + +FILTER_END2END_TEST_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(FILTER_END2END_TEST_SRC)))) +ifeq ($(NO_SECURE),true) + +# You can't build secure targets if you don't have OpenSSL. + +$(BINDIR)/$(CONFIG)/filter_end2end_test: openssl_dep_error + +else + + + + +ifeq ($(NO_PROTOBUF),true) + +# You can't build the protoc plugins or protobuf-enabled targets if you don't have protobuf 3.0.0+. + +$(BINDIR)/$(CONFIG)/filter_end2end_test: protobuf_dep_error + +else + +$(BINDIR)/$(CONFIG)/filter_end2end_test: $(PROTOBUF_DEP) $(FILTER_END2END_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + $(E) "[LD] Linking $@" + $(Q) mkdir -p `dirname $@` + $(Q) $(LDXX) $(LDFLAGS) $(FILTER_END2END_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBSXX) $(LDLIBS_PROTOBUF) $(LDLIBS) $(LDLIBS_SECURE) $(GTEST_LIB) -o $(BINDIR)/$(CONFIG)/filter_end2end_test + +endif + +endif + +$(OBJDIR)/$(CONFIG)/test/cpp/end2end/filter_end2end_test.o: $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + +deps_filter_end2end_test: $(FILTER_END2END_TEST_OBJS:.o=.dep) + +ifneq ($(NO_SECURE),true) +ifneq ($(NO_DEPS),true) +-include $(FILTER_END2END_TEST_OBJS:.o=.dep) +endif +endif + + GENERIC_END2END_TEST_SRC = \ test/cpp/end2end/generic_end2end_test.cc \ diff --git a/build.yaml b/build.yaml index b6b1f758532..ad1211f2192 100644 --- a/build.yaml +++ b/build.yaml @@ -2611,6 +2611,19 @@ targets: - grpc - gpr_test_util - gpr +- name: filter_end2end_test + gtest: true + build: test + language: c++ + src: + - test/cpp/end2end/filter_end2end_test.cc + deps: + - grpc++_test_util + - grpc_test_util + - grpc++ + - grpc + - gpr_test_util + - gpr - name: generic_end2end_test gtest: true build: test diff --git a/include/grpc++/channel_filter.h b/include/grpc++/channel_filter.h index 8067ca9c603..b37d986a5ac 100644 --- a/include/grpc++/channel_filter.h +++ b/include/grpc++/channel_filter.h @@ -36,6 +36,7 @@ #include +#include #include #include "src/core/lib/channel/channel_stack.h" @@ -45,9 +46,9 @@ // An interface to define filters. // // To define a filter, implement a subclass of each of CallData and -// ChannelData. Then register the filter like this: +// ChannelData. Then register the filter using something like this: // RegisterChannelFilter( -// "name-of-filter", GRPC_SERVER_CHANNEL, INT_MAX); +// "name-of-filter", GRPC_SERVER_CHANNEL, INT_MAX, nullptr); // namespace grpc { @@ -56,12 +57,8 @@ namespace grpc { // Note: Must be copyable. class CallData { public: - // Do not override the destructor. Instead, put clean-up code in the - // Destroy() method. virtual ~CallData() {} - virtual void Destroy() {} - virtual void StartTransportStreamOp( grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_transport_stream_op *op); @@ -80,12 +77,8 @@ class CallData { // Note: Must be copyable. class ChannelData { public: - // Do not override the destructor. Instead, put clean-up code in the - // Destroy() method. virtual ~ChannelData() {} - virtual void Destroy() {} - virtual void StartTransportOp( grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, grpc_transport_op *op); @@ -99,43 +92,40 @@ namespace internal { // Defines static members for passing to C core. template class ChannelFilter { + public: static const size_t call_data_size = sizeof(CallDataType); static void InitCallElement( grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { - CallDataType* call_data = elem->call_data; - *call_data = CallDataType(); + // Construct the object in the already-allocated memory. + new (elem->call_data) CallDataType(); } static void DestroyCallElement( grpc_exec_ctx *exec_ctx, grpc_call_element *elem, const grpc_call_stats *stats, void *and_free_memory) { - CallDataType* call_data = elem->call_data; - // Can't destroy the object here, since it's not allocated by - // itself; instead, it's part of a larger allocation managed by - // C-core. So instead, we call the Destroy() method. - call_data->Destroy(); + reinterpret_cast(elem->call_data)->~CallDataType(); } static void StartTransportStreamOp( grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_transport_stream_op *op) { - CallDataType* call_data = elem->call_data; - call_data->StartTransportStreamOp(exec_ctx, op); + CallDataType* call_data = (CallDataType*)elem->call_data; + call_data->StartTransportStreamOp(exec_ctx, elem, op); } static void SetPollsetOrPollsetSet( grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_polling_entity *pollent) { - CallDataType* call_data = elem->call_data; - call_data->SetPollsetOrPollsetSet(exec_ctx, pollent); + CallDataType* call_data = (CallDataType*)elem->call_data; + call_data->SetPollsetOrPollsetSet(exec_ctx, elem, pollent); } static char* GetPeer( grpc_exec_ctx *exec_ctx, grpc_call_element *elem) { - CallDataType* call_data = elem->call_data; - return call_data->GetPeer(exec_ctx); + CallDataType* call_data = (CallDataType*)elem->call_data; + return call_data->GetPeer(exec_ctx, elem); } static const size_t channel_data_size = sizeof(ChannelDataType); @@ -143,44 +133,45 @@ class ChannelFilter { static void InitChannelElement( grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, grpc_channel_element_args *args) { - ChannelDataType* channel_data = elem->channel_data; - *channel_data = ChannelDataType(); + // Construct the object in the already-allocated memory. + new (elem->channel_data) ChannelDataType(); } static void DestroyChannelElement( grpc_exec_ctx *exec_ctx, grpc_channel_element *elem) { - ChannelDataType* channel_data = elem->channel_data; - // Can't destroy the object here, since it's not allocated by - // itself; instead, it's part of a larger allocation managed by - // C-core. So instead, we call the Destroy() method. - channel_data->Destroy(); + reinterpret_cast(elem->channel_data)->~ChannelDataType(); } static void StartTransportOp( grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, grpc_transport_op *op) { - ChannelDataType* channel_data = elem->channel_data; - channel_data->StartTransportOp(exec_ctx, op); + ChannelDataType* channel_data = (ChannelDataType*)elem->channel_data; + channel_data->StartTransportOp(exec_ctx, elem, op); } }; struct FilterRecord { grpc_channel_stack_type stack_type; int priority; + std::function include_filter; grpc_channel_filter filter; }; extern std::vector* channel_filters; void ChannelFilterPluginInit(); -void ChannelFilterPluginShutdown() {} +void ChannelFilterPluginShutdown(); } // namespace internal // Registers a new filter. // Must be called by only one thread at a time. +// The include_filter argument specifies a function that will be called +// to determine at run-time whether or not to add the filter. If the +// value is nullptr, the filter will be added unconditionally. template -void RegisterChannelFilter(const char* name, - grpc_channel_stack_type stack_type, int priority) { +void RegisterChannelFilter( + const char* name, grpc_channel_stack_type stack_type, int priority, + std::function include_filter) { // If we haven't been called before, initialize channel_filters and // call grpc_register_plugin(). if (internal::channel_filters == nullptr) { @@ -191,8 +182,8 @@ void RegisterChannelFilter(const char* name, // Add an entry to channel_filters. The filter will be added when the // C-core initialization code calls ChannelFilterPluginInit(). typedef internal::ChannelFilter FilterType; - internal::channel_filters->emplace_back({ - stack_type, priority, { + internal::FilterRecord filter_record = { + stack_type, priority, include_filter, { FilterType::StartTransportStreamOp, FilterType::StartTransportOp, FilterType::call_data_size, @@ -203,7 +194,8 @@ void RegisterChannelFilter(const char* name, FilterType::InitChannelElement, FilterType::DestroyChannelElement, FilterType::GetPeer, - name}}); + name}}; + internal::channel_filters->push_back(filter_record); } } // namespace grpc diff --git a/src/core/lib/channel/channel_stack.h b/src/core/lib/channel/channel_stack.h index 41dd4a0d8a1..535acde3934 100644 --- a/src/core/lib/channel/channel_stack.h +++ b/src/core/lib/channel/channel_stack.h @@ -51,6 +51,10 @@ #include "src/core/lib/iomgr/polling_entity.h" #include "src/core/lib/transport/transport.h" +#ifdef __cplusplus +extern "C" { +#endif + typedef struct grpc_channel_element grpc_channel_element; typedef struct grpc_call_element grpc_call_element; @@ -278,4 +282,8 @@ extern int grpc_trace_channel; #define GRPC_CALL_LOG_OP(sev, elem, op) \ if (grpc_trace_channel) grpc_call_log_op(sev, elem, op) +#ifdef __cplusplus +} +#endif + #endif /* GRPC_CORE_LIB_CHANNEL_CHANNEL_STACK_H */ diff --git a/src/core/lib/channel/channel_stack_builder.h b/src/core/lib/channel/channel_stack_builder.h index 0e6bfd9aa69..4a00f7bfdbd 100644 --- a/src/core/lib/channel/channel_stack_builder.h +++ b/src/core/lib/channel/channel_stack_builder.h @@ -39,6 +39,10 @@ #include "src/core/lib/channel/channel_args.h" #include "src/core/lib/channel/channel_stack.h" +#ifdef __cplusplus +extern "C" { +#endif + /// grpc_channel_stack_builder offers a programmatic interface to selected /// and order channel filters typedef struct grpc_channel_stack_builder grpc_channel_stack_builder; @@ -158,4 +162,8 @@ void grpc_channel_stack_builder_destroy(grpc_channel_stack_builder *builder); extern int grpc_trace_channel_stack_builder; +#ifdef __cplusplus +} +#endif + #endif /* GRPC_CORE_LIB_CHANNEL_CHANNEL_STACK_BUILDER_H */ diff --git a/src/core/lib/surface/channel_init.h b/src/core/lib/surface/channel_init.h index 3a18a61ddb8..b53f2aefb92 100644 --- a/src/core/lib/surface/channel_init.h +++ b/src/core/lib/surface/channel_init.h @@ -40,6 +40,10 @@ #define GRPC_CHANNEL_INIT_BUILTIN_PRIORITY 10000 +#ifdef __cplusplus +extern "C" { +#endif + /// This module provides a way for plugins (and the grpc core library itself) /// to register mutators for channel stacks. /// It also provides a universal entry path to run those mutators to build @@ -84,4 +88,8 @@ bool grpc_channel_init_create_stack(grpc_exec_ctx *exec_ctx, grpc_channel_stack_builder *builder, grpc_channel_stack_type type); +#ifdef __cplusplus +} +#endif + #endif /* GRPC_CORE_LIB_SURFACE_CHANNEL_INIT_H */ diff --git a/src/cpp/common/channel_filter.cc b/src/cpp/common/channel_filter.cc index 9a80195e867..b5e5e08976b 100644 --- a/src/cpp/common/channel_filter.cc +++ b/src/cpp/common/channel_filter.cc @@ -78,9 +78,16 @@ std::vector* channel_filters = nullptr; namespace { -bool AppendFilter(grpc_channel_stack_builder* builder, void* arg) { - return grpc_channel_stack_builder_append_filter( - builder, (const grpc_channel_filter *)arg, nullptr, nullptr); +bool MaybeAddFilter(grpc_channel_stack_builder* builder, void* arg) { + const FilterRecord& filter = *(FilterRecord*)arg; + if (filter.include_filter != nullptr) { + const grpc_channel_args *args = + grpc_channel_stack_builder_get_channel_arguments(builder); + if (!filter.include_filter(args)) + return true; + } + return grpc_channel_stack_builder_prepend_filter( + builder, &filter.filter, nullptr, nullptr); } } // namespace @@ -89,10 +96,12 @@ void ChannelFilterPluginInit() { for (size_t i = 0; i < channel_filters->size(); ++i) { FilterRecord& filter = (*channel_filters)[i]; grpc_channel_init_register_stage(filter.stack_type, filter.priority, - AppendFilter, (void*)&filter.filter); + MaybeAddFilter, (void*)&filter); } } +void ChannelFilterPluginShutdown() {} + } // namespace internal } // namespace grpc diff --git a/test/cpp/end2end/filter_end2end_test.cc b/test/cpp/end2end/filter_end2end_test.cc new file mode 100644 index 00000000000..79a5287fbcf --- /dev/null +++ b/test/cpp/end2end/filter_end2end_test.cc @@ -0,0 +1,327 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "src/proto/grpc/testing/echo.grpc.pb.h" +#include "test/core/util/port.h" +#include "test/core/util/test_config.h" +#include "test/cpp/util/byte_buffer_proto_helper.h" + +using grpc::testing::EchoRequest; +using grpc::testing::EchoResponse; +using std::chrono::system_clock; + +namespace grpc { +namespace testing { +namespace { + +void* tag(int i) { return (void*)(intptr_t)i; } + +void verify_ok(CompletionQueue* cq, int i, bool expect_ok) { + bool ok; + void* got_tag; + EXPECT_TRUE(cq->Next(&got_tag, &ok)); + EXPECT_EQ(expect_ok, ok); + EXPECT_EQ(tag(i), got_tag); +} + +namespace { + +int global_num_calls = 0; +mutex global_mu; + +void IncrementCounter() { + unique_lock lock(global_mu); + ++global_num_calls; +} + +void ResetCounter() { + unique_lock lock(global_mu); + global_num_calls = 0; +} + +int GetCounterValue() { + unique_lock lock(global_mu); + return global_num_calls; +} + +} // namespace + +class CallDataImpl : public CallData { + public: + CallDataImpl() {} + virtual ~CallDataImpl() {} + + void StartTransportStreamOp( + grpc_exec_ctx *exec_ctx, grpc_call_element *elem, + grpc_transport_stream_op *op) { + gpr_log(GPR_ERROR, "INCREMENTING"); + IncrementCounter(); + grpc_call_next_op(exec_ctx, elem, op); + } +}; + +class ChannelDataImpl : public ChannelData { + public: + ChannelDataImpl() {} + virtual ~ChannelDataImpl() {} +}; + +class FilterEnd2endTest : public ::testing::Test { + protected: + FilterEnd2endTest() : server_host_("localhost") {} + + void SetUp() GRPC_OVERRIDE { + RegisterChannelFilter( + "test-filter", GRPC_SERVER_CHANNEL, INT_MAX, nullptr); + int port = grpc_pick_unused_port_or_die(); + server_address_ << server_host_ << ":" << port; + // Setup server + ServerBuilder builder; + builder.AddListeningPort(server_address_.str(), + InsecureServerCredentials()); + builder.RegisterAsyncGenericService(&generic_service_); + // Include a second call to RegisterAsyncGenericService to make sure that + // we get an error in the log, since it is not allowed to have 2 async + // generic services + builder.RegisterAsyncGenericService(&generic_service_); + srv_cq_ = builder.AddCompletionQueue(); + server_ = builder.BuildAndStart(); + } + + void TearDown() GRPC_OVERRIDE { + server_->Shutdown(); + void* ignored_tag; + bool ignored_ok; + cli_cq_.Shutdown(); + srv_cq_->Shutdown(); + while (cli_cq_.Next(&ignored_tag, &ignored_ok)) + ; + while (srv_cq_->Next(&ignored_tag, &ignored_ok)) + ; + } + + void ResetStub() { + std::shared_ptr channel = + CreateChannel(server_address_.str(), InsecureChannelCredentials()); + generic_stub_.reset(new GenericStub(channel)); + ResetCounter(); + } + + void server_ok(int i) { verify_ok(srv_cq_.get(), i, true); } + void client_ok(int i) { verify_ok(&cli_cq_, i, true); } + void server_fail(int i) { verify_ok(srv_cq_.get(), i, false); } + void client_fail(int i) { verify_ok(&cli_cq_, i, false); } + + void SendRpc(int num_rpcs) { + const grpc::string kMethodName("/grpc.cpp.test.util.EchoTestService/Echo"); + for (int i = 0; i < num_rpcs; i++) { + EchoRequest send_request; + EchoRequest recv_request; + EchoResponse send_response; + EchoResponse recv_response; + Status recv_status; + + ClientContext cli_ctx; + GenericServerContext srv_ctx; + GenericServerAsyncReaderWriter stream(&srv_ctx); + + // The string needs to be long enough to test heap-based slice. + send_request.set_message("Hello world. Hello world. Hello world."); + std::unique_ptr call = + generic_stub_->Call(&cli_ctx, kMethodName, &cli_cq_, tag(1)); + client_ok(1); + std::unique_ptr send_buffer = + SerializeToByteBuffer(&send_request); + call->Write(*send_buffer, tag(2)); + // Send ByteBuffer can be destroyed after calling Write. + send_buffer.reset(); + client_ok(2); + call->WritesDone(tag(3)); + client_ok(3); + + generic_service_.RequestCall(&srv_ctx, &stream, srv_cq_.get(), + srv_cq_.get(), tag(4)); + + verify_ok(srv_cq_.get(), 4, true); + EXPECT_EQ(server_host_, srv_ctx.host().substr(0, server_host_.length())); + EXPECT_EQ(kMethodName, srv_ctx.method()); + ByteBuffer recv_buffer; + stream.Read(&recv_buffer, tag(5)); + server_ok(5); + EXPECT_TRUE(ParseFromByteBuffer(&recv_buffer, &recv_request)); + EXPECT_EQ(send_request.message(), recv_request.message()); + + send_response.set_message(recv_request.message()); + send_buffer = SerializeToByteBuffer(&send_response); + stream.Write(*send_buffer, tag(6)); + send_buffer.reset(); + server_ok(6); + + stream.Finish(Status::OK, tag(7)); + server_ok(7); + + recv_buffer.Clear(); + call->Read(&recv_buffer, tag(8)); + client_ok(8); + EXPECT_TRUE(ParseFromByteBuffer(&recv_buffer, &recv_response)); + + call->Finish(&recv_status, tag(9)); + client_ok(9); + + EXPECT_EQ(send_response.message(), recv_response.message()); + EXPECT_TRUE(recv_status.ok()); + } + } + + CompletionQueue cli_cq_; + std::unique_ptr srv_cq_; + std::unique_ptr stub_; + std::unique_ptr generic_stub_; + std::unique_ptr server_; + AsyncGenericService generic_service_; + const grpc::string server_host_; + std::ostringstream server_address_; +}; + +TEST_F(FilterEnd2endTest, SimpleRpc) { + ResetStub(); + EXPECT_EQ(0, GetCounterValue()); + SendRpc(1); + EXPECT_EQ(1, GetCounterValue()); +} + +TEST_F(FilterEnd2endTest, SequentialRpcs) { + ResetStub(); + EXPECT_EQ(0, GetCounterValue()); + SendRpc(10); + EXPECT_EQ(10, GetCounterValue()); +} + +// One ping, one pong. +TEST_F(FilterEnd2endTest, SimpleBidiStreaming) { + ResetStub(); + EXPECT_EQ(0, GetCounterValue()); + + const grpc::string kMethodName( + "/grpc.cpp.test.util.EchoTestService/BidiStream"); + EchoRequest send_request; + EchoRequest recv_request; + EchoResponse send_response; + EchoResponse recv_response; + Status recv_status; + ClientContext cli_ctx; + GenericServerContext srv_ctx; + GenericServerAsyncReaderWriter srv_stream(&srv_ctx); + + cli_ctx.set_compression_algorithm(GRPC_COMPRESS_GZIP); + send_request.set_message("Hello"); + std::unique_ptr cli_stream = + generic_stub_->Call(&cli_ctx, kMethodName, &cli_cq_, tag(1)); + client_ok(1); + + generic_service_.RequestCall(&srv_ctx, &srv_stream, srv_cq_.get(), + srv_cq_.get(), tag(2)); + + verify_ok(srv_cq_.get(), 2, true); + EXPECT_EQ(server_host_, srv_ctx.host().substr(0, server_host_.length())); + EXPECT_EQ(kMethodName, srv_ctx.method()); + + std::unique_ptr send_buffer = + SerializeToByteBuffer(&send_request); + cli_stream->Write(*send_buffer, tag(3)); + send_buffer.reset(); + client_ok(3); + + ByteBuffer recv_buffer; + srv_stream.Read(&recv_buffer, tag(4)); + server_ok(4); + EXPECT_TRUE(ParseFromByteBuffer(&recv_buffer, &recv_request)); + EXPECT_EQ(send_request.message(), recv_request.message()); + + send_response.set_message(recv_request.message()); + send_buffer = SerializeToByteBuffer(&send_response); + srv_stream.Write(*send_buffer, tag(5)); + send_buffer.reset(); + server_ok(5); + + cli_stream->Read(&recv_buffer, tag(6)); + client_ok(6); + EXPECT_TRUE(ParseFromByteBuffer(&recv_buffer, &recv_response)); + EXPECT_EQ(send_response.message(), recv_response.message()); + + cli_stream->WritesDone(tag(7)); + client_ok(7); + + srv_stream.Read(&recv_buffer, tag(8)); + server_fail(8); + + srv_stream.Finish(Status::OK, tag(9)); + server_ok(9); + + cli_stream->Finish(&recv_status, tag(10)); + client_ok(10); + + EXPECT_EQ(send_response.message(), recv_response.message()); + EXPECT_TRUE(recv_status.ok()); + + EXPECT_EQ(1, GetCounterValue()); +} + +} // namespace +} // namespace testing +} // namespace grpc + +int main(int argc, char** argv) { + grpc_test_init(argc, argv); + ::testing::InitGoogleTest(&argc, argv); + return RUN_ALL_TESTS(); +} diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 00ea2fdc058..ee6cda804a7 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -2062,6 +2062,24 @@ "third_party": false, "type": "target" }, + { + "deps": [ + "gpr", + "gpr_test_util", + "grpc", + "grpc++", + "grpc++_test_util", + "grpc_test_util" + ], + "headers": [], + "language": "c++", + "name": "filter_end2end_test", + "src": [ + "test/cpp/end2end/filter_end2end_test.cc" + ], + "third_party": false, + "type": "target" + }, { "deps": [ "gpr", diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index 5a84a41b638..5744261c99b 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -2210,6 +2210,27 @@ "windows" ] }, + { + "args": [], + "ci_platforms": [ + "linux", + "mac", + "posix", + "windows" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "gtest": true, + "language": "c++", + "name": "filter_end2end_test", + "platforms": [ + "linux", + "mac", + "posix", + "windows" + ] + }, { "args": [], "ci_platforms": [ diff --git a/vsprojects/vcxproj/test/filter_end2end_test/filter_end2end_test.vcxproj b/vsprojects/vcxproj/test/filter_end2end_test/filter_end2end_test.vcxproj new file mode 100644 index 00000000000..99c000e4fa9 --- /dev/null +++ b/vsprojects/vcxproj/test/filter_end2end_test/filter_end2end_test.vcxproj @@ -0,0 +1,207 @@ + + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {1D42975A-18A5-09D0-30B1-2AE6A97FB9DE} + true + $(SolutionDir)IntDir\$(MSBuildProjectName)\ + + + + v100 + + + v110 + + + v120 + + + v140 + + + Application + true + Unicode + + + Application + false + true + Unicode + + + + + + + + + + + + + + + + filter_end2end_test + static + Debug + static + Debug + + + filter_end2end_test + static + Release + static + Release + + + + NotUsing + Level3 + Disabled + WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) + true + MultiThreadedDebug + true + None + false + + + Console + true + false + + + + + + NotUsing + Level3 + Disabled + WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) + true + MultiThreadedDebug + true + None + false + + + Console + true + false + + + + + + NotUsing + Level3 + MaxSpeed + WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) + true + true + true + MultiThreaded + true + None + false + + + Console + true + false + true + true + + + + + + NotUsing + Level3 + MaxSpeed + WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) + true + true + true + MultiThreaded + true + None + false + + + Console + true + false + true + true + + + + + + + + + + {0BE77741-552A-929B-A497-4EF7ECE17A64} + + + {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} + + + {C187A093-A0FE-489D-A40A-6E33DE0F9FEB} + + + {29D16885-7228-4C31-81ED-5F9187C7F2A9} + + + {EAB0A629-17A9-44DB-B5FF-E91A721FE037} + + + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + diff --git a/vsprojects/vcxproj/test/filter_end2end_test/filter_end2end_test.vcxproj.filters b/vsprojects/vcxproj/test/filter_end2end_test/filter_end2end_test.vcxproj.filters new file mode 100644 index 00000000000..c68442365c5 --- /dev/null +++ b/vsprojects/vcxproj/test/filter_end2end_test/filter_end2end_test.vcxproj.filters @@ -0,0 +1,21 @@ + + + + + test\cpp\end2end + + + + + + {f7581160-220d-1118-f29e-6b37e45671c9} + + + {63784d35-03cc-1a7e-0c95-a9451bc1daf4} + + + {d2a01682-970e-d23d-1eb8-ef020e3a1ca3} + + + + From bb2bd6553938ca1a192414084e5800178f67c4a3 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Thu, 16 Jun 2016 09:29:53 -0700 Subject: [PATCH 0106/1039] Clang format --- src/core/lib/surface/completion_queue.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/core/lib/surface/completion_queue.c b/src/core/lib/surface/completion_queue.c index de2b674d0e0..03de97bc2b5 100644 --- a/src/core/lib/surface/completion_queue.c +++ b/src/core/lib/surface/completion_queue.c @@ -93,12 +93,12 @@ static grpc_completion_queue *g_freelist; int grpc_cq_pluck_trace; int grpc_cq_event_timeout_trace; -#define GRPC_SURFACE_TRACE_RETURNED_EVENT(cq, event) \ - if (grpc_api_trace && \ - (grpc_cq_pluck_trace || (event)->type != GRPC_QUEUE_TIMEOUT)) { \ - char *_ev = grpc_event_string(event); \ - gpr_log(GPR_INFO, "RETURN_EVENT[%p]: %s", cq, _ev); \ - gpr_free(_ev); \ +#define GRPC_SURFACE_TRACE_RETURNED_EVENT(cq, event) \ + if (grpc_api_trace && \ + (grpc_cq_pluck_trace || (event)->type != GRPC_QUEUE_TIMEOUT)) { \ + char *_ev = grpc_event_string(event); \ + gpr_log(GPR_INFO, "RETURN_EVENT[%p]: %s", cq, _ev); \ + gpr_free(_ev); \ } From a7cc90055e91b1bda2e916c5884b5305c7f73b53 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Thu, 16 Jun 2016 09:59:51 -0700 Subject: [PATCH 0107/1039] Modified route_guide sample app to show RPC log Modified the screens to show meaningful information about RPC progress. When you click on each tab the screen now shows log messages that previously only went to the debug console. --- .../Misc/Base.lproj/Main.storyboard | 108 ++++++++++-------- .../objective-c/route_guide/ViewControllers.m | 41 +++++++ 2 files changed, 102 insertions(+), 47 deletions(-) diff --git a/examples/objective-c/route_guide/Misc/Base.lproj/Main.storyboard b/examples/objective-c/route_guide/Misc/Base.lproj/Main.storyboard index 9bf9498d62c..306320f7d89 100644 --- a/examples/objective-c/route_guide/Misc/Base.lproj/Main.storyboard +++ b/examples/objective-c/route_guide/Misc/Base.lproj/Main.storyboard @@ -1,7 +1,8 @@ - + - + + @@ -16,33 +17,35 @@ - - - - - + + + + - + @@ -56,29 +59,35 @@ - + + + - - - - + + + + + @@ -117,29 +126,32 @@ - - - - - + + + + + @@ -157,29 +169,31 @@ - - - - - + + + + diff --git a/examples/objective-c/route_guide/ViewControllers.m b/examples/objective-c/route_guide/ViewControllers.m index e32978240b6..b916a4ee0c4 100644 --- a/examples/objective-c/route_guide/ViewControllers.m +++ b/examples/objective-c/route_guide/ViewControllers.m @@ -83,6 +83,7 @@ static NSString * const kHostAddress = @"localhost:50051"; @interface GetFeatureViewController : UIViewController { RTGRouteGuide *service; } +@property (weak, nonatomic) IBOutlet UILabel *output_label; @end @implementation GetFeatureViewController @@ -90,10 +91,16 @@ static NSString * const kHostAddress = @"localhost:50051"; - (void)execRequest { void (^handler)(RTGFeature *response, NSError *error) = ^(RTGFeature *response, NSError *error) { if (response.name.length) { + NSString *str =[NSString stringWithFormat:@"%@\nFound feature called %@ at %@.", self.output_label.text, response.location, response.name]; + self.output_label.text = str; NSLog(@"Found feature called %@ at %@.", response.name, response.location); } else if (response) { + NSString *str =[NSString stringWithFormat:@"%@\nFound no features at %@", self.output_label.text,response.location]; + self.output_label.text = str; NSLog(@"Found no features at %@", response.location); } else { + NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.output_label.text, error]; + self.output_label.text = str; NSLog(@"RPC error: %@", error); } }; @@ -116,6 +123,9 @@ static NSString * const kHostAddress = @"localhost:50051"; } - (void)viewDidAppear:(BOOL)animated { + self.output_label.text = @"RPC log:"; + self.output_label.numberOfLines = 0; + self.output_label.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; [self execRequest]; } @@ -131,6 +141,7 @@ static NSString * const kHostAddress = @"localhost:50051"; @interface ListFeaturesViewController : UIViewController { RTGRouteGuide *service; } +@property (weak, nonatomic) IBOutlet UILabel *output_label; @end @@ -147,8 +158,12 @@ static NSString * const kHostAddress = @"localhost:50051"; [service listFeaturesWithRequest:rectangle eventHandler:^(BOOL done, RTGFeature *response, NSError *error) { if (response) { + NSString *str =[NSString stringWithFormat:@"%@\nFound feature at %@ called %@.", self.output_label.text, response.location, response.name]; + self.output_label.text = str; NSLog(@"Found feature at %@ called %@.", response.location, response.name); } else if (error) { + NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.output_label.text, error]; + self.output_label.text = str; NSLog(@"RPC error: %@", error); } }]; @@ -161,6 +176,9 @@ static NSString * const kHostAddress = @"localhost:50051"; } - (void)viewDidAppear:(BOOL)animated { + self.output_label.text = @"RPC log:"; + self.output_label.numberOfLines = 0; + self.output_label.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; [self execRequest]; } @@ -177,6 +195,7 @@ static NSString * const kHostAddress = @"localhost:50051"; @interface RecordRouteViewController : UIViewController { RTGRouteGuide *service; } +@property (weak, nonatomic) IBOutlet UILabel *output_label; @end @@ -192,6 +211,8 @@ static NSString * const kHostAddress = @"localhost:50051"; RTGPoint *location = [RTGPoint message]; location.longitude = [((NSNumber *) feature[@"location"][@"longitude"]) intValue]; location.latitude = [((NSNumber *) feature[@"location"][@"latitude"]) intValue]; + NSString *str =[NSString stringWithFormat:@"%@\nVisiting point %@", self.output_label.text, location]; + self.output_label.text = str; NSLog(@"Visiting point %@", location); return location; }]; @@ -199,11 +220,19 @@ static NSString * const kHostAddress = @"localhost:50051"; [service recordRouteWithRequestsWriter:locations handler:^(RTGRouteSummary *response, NSError *error) { if (response) { + NSString *str =[NSString stringWithFormat: + @"%@\nFinished trip with %i points\nPassed %i features\n" + "Travelled %i meters\nIt took %i seconds", + self.output_label.text, response.pointCount, response.featureCount, + response.distance, response.elapsedTime]; + self.output_label.text = str; NSLog(@"Finished trip with %i points", response.pointCount); NSLog(@"Passed %i features", response.featureCount); NSLog(@"Travelled %i meters", response.distance); NSLog(@"It took %i seconds", response.elapsedTime); } else { + NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.output_label.text, error]; + self.output_label.text = str; NSLog(@"RPC error: %@", error); } }]; @@ -216,6 +245,9 @@ static NSString * const kHostAddress = @"localhost:50051"; } - (void)viewDidAppear:(BOOL)animated { + self.output_label.text = @"RPC log:"; + self.output_label.numberOfLines = 0; + self.output_label.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; [self execRequest]; } @@ -231,6 +263,7 @@ static NSString * const kHostAddress = @"localhost:50051"; @interface RouteChatViewController : UIViewController { RTGRouteGuide *service; } +@property (weak, nonatomic) IBOutlet UILabel *output_label; @end @@ -249,8 +282,13 @@ static NSString * const kHostAddress = @"localhost:50051"; [service routeChatWithRequestsWriter:notesWriter eventHandler:^(BOOL done, RTGRouteNote *note, NSError *error) { if (note) { + NSString *str =[NSString stringWithFormat:@"%@\nGot message %@ at %@", + self.output_label.text, note.message, note.location]; + self.output_label.text = str; NSLog(@"Got message %@ at %@", note.message, note.location); } else if (error) { + NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.output_label.text, error]; + self.output_label.text = str; NSLog(@"RPC error: %@", error); } if (done) { @@ -266,6 +304,9 @@ static NSString * const kHostAddress = @"localhost:50051"; } - (void)viewDidAppear:(BOOL)animated { + self.output_label.text = @"RPC log:"; + self.output_label.numberOfLines = 0; + self.output_label.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; [self execRequest]; } From 3eba9075ef612f83c363868c24d106580ab25223 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Thu, 16 Jun 2016 10:15:32 -0700 Subject: [PATCH 0108/1039] using camelCase for output_label now --- .../Misc/Base.lproj/Main.storyboard | 8 +- .../objective-c/route_guide/ViewControllers.m | 74 +++++++++---------- 2 files changed, 41 insertions(+), 41 deletions(-) diff --git a/examples/objective-c/route_guide/Misc/Base.lproj/Main.storyboard b/examples/objective-c/route_guide/Misc/Base.lproj/Main.storyboard index 306320f7d89..5ca9f4642b9 100644 --- a/examples/objective-c/route_guide/Misc/Base.lproj/Main.storyboard +++ b/examples/objective-c/route_guide/Misc/Base.lproj/Main.storyboard @@ -40,7 +40,7 @@ - + @@ -86,7 +86,7 @@ - + @@ -150,7 +150,7 @@ - + @@ -192,7 +192,7 @@ - + diff --git a/examples/objective-c/route_guide/ViewControllers.m b/examples/objective-c/route_guide/ViewControllers.m index b916a4ee0c4..1e84c334728 100644 --- a/examples/objective-c/route_guide/ViewControllers.m +++ b/examples/objective-c/route_guide/ViewControllers.m @@ -37,7 +37,7 @@ #import #import -static NSString * const kHostAddress = @"localhost:50051"; +static NSString * const kHostAddress = @"192.168.1.120:50051"; /** Category to override RTGPoint's description. */ @interface RTGPoint (Description) @@ -83,7 +83,7 @@ static NSString * const kHostAddress = @"localhost:50051"; @interface GetFeatureViewController : UIViewController { RTGRouteGuide *service; } -@property (weak, nonatomic) IBOutlet UILabel *output_label; +@property (weak, nonatomic) IBOutlet UILabel *outputLabel; @end @implementation GetFeatureViewController @@ -91,16 +91,16 @@ static NSString * const kHostAddress = @"localhost:50051"; - (void)execRequest { void (^handler)(RTGFeature *response, NSError *error) = ^(RTGFeature *response, NSError *error) { if (response.name.length) { - NSString *str =[NSString stringWithFormat:@"%@\nFound feature called %@ at %@.", self.output_label.text, response.location, response.name]; - self.output_label.text = str; + NSString *str =[NSString stringWithFormat:@"%@\nFound feature called %@ at %@.", self.outputLabel.text, response.location, response.name]; + self.outputLabel.text = str; NSLog(@"Found feature called %@ at %@.", response.name, response.location); } else if (response) { - NSString *str =[NSString stringWithFormat:@"%@\nFound no features at %@", self.output_label.text,response.location]; - self.output_label.text = str; + NSString *str =[NSString stringWithFormat:@"%@\nFound no features at %@", self.outputLabel.text,response.location]; + self.outputLabel.text = str; NSLog(@"Found no features at %@", response.location); } else { - NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.output_label.text, error]; - self.output_label.text = str; + NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.outputLabel.text, error]; + self.outputLabel.text = str; NSLog(@"RPC error: %@", error); } }; @@ -123,9 +123,9 @@ static NSString * const kHostAddress = @"localhost:50051"; } - (void)viewDidAppear:(BOOL)animated { - self.output_label.text = @"RPC log:"; - self.output_label.numberOfLines = 0; - self.output_label.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; + self.outputLabel.text = @"RPC log:"; + self.outputLabel.numberOfLines = 0; + self.outputLabel.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; [self execRequest]; } @@ -141,7 +141,7 @@ static NSString * const kHostAddress = @"localhost:50051"; @interface ListFeaturesViewController : UIViewController { RTGRouteGuide *service; } -@property (weak, nonatomic) IBOutlet UILabel *output_label; +@property (weak, nonatomic) IBOutlet UILabel *outputLabel; @end @@ -158,12 +158,12 @@ static NSString * const kHostAddress = @"localhost:50051"; [service listFeaturesWithRequest:rectangle eventHandler:^(BOOL done, RTGFeature *response, NSError *error) { if (response) { - NSString *str =[NSString stringWithFormat:@"%@\nFound feature at %@ called %@.", self.output_label.text, response.location, response.name]; - self.output_label.text = str; + NSString *str =[NSString stringWithFormat:@"%@\nFound feature at %@ called %@.", self.outputLabel.text, response.location, response.name]; + self.outputLabel.text = str; NSLog(@"Found feature at %@ called %@.", response.location, response.name); } else if (error) { - NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.output_label.text, error]; - self.output_label.text = str; + NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.outputLabel.text, error]; + self.outputLabel.text = str; NSLog(@"RPC error: %@", error); } }]; @@ -176,9 +176,9 @@ static NSString * const kHostAddress = @"localhost:50051"; } - (void)viewDidAppear:(BOOL)animated { - self.output_label.text = @"RPC log:"; - self.output_label.numberOfLines = 0; - self.output_label.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; + self.outputLabel.text = @"RPC log:"; + self.outputLabel.numberOfLines = 0; + self.outputLabel.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; [self execRequest]; } @@ -195,7 +195,7 @@ static NSString * const kHostAddress = @"localhost:50051"; @interface RecordRouteViewController : UIViewController { RTGRouteGuide *service; } -@property (weak, nonatomic) IBOutlet UILabel *output_label; +@property (weak, nonatomic) IBOutlet UILabel *outputLabel; @end @@ -211,8 +211,8 @@ static NSString * const kHostAddress = @"localhost:50051"; RTGPoint *location = [RTGPoint message]; location.longitude = [((NSNumber *) feature[@"location"][@"longitude"]) intValue]; location.latitude = [((NSNumber *) feature[@"location"][@"latitude"]) intValue]; - NSString *str =[NSString stringWithFormat:@"%@\nVisiting point %@", self.output_label.text, location]; - self.output_label.text = str; + NSString *str =[NSString stringWithFormat:@"%@\nVisiting point %@", self.outputLabel.text, location]; + self.outputLabel.text = str; NSLog(@"Visiting point %@", location); return location; }]; @@ -223,16 +223,16 @@ static NSString * const kHostAddress = @"localhost:50051"; NSString *str =[NSString stringWithFormat: @"%@\nFinished trip with %i points\nPassed %i features\n" "Travelled %i meters\nIt took %i seconds", - self.output_label.text, response.pointCount, response.featureCount, + self.outputLabel.text, response.pointCount, response.featureCount, response.distance, response.elapsedTime]; - self.output_label.text = str; + self.outputLabel.text = str; NSLog(@"Finished trip with %i points", response.pointCount); NSLog(@"Passed %i features", response.featureCount); NSLog(@"Travelled %i meters", response.distance); NSLog(@"It took %i seconds", response.elapsedTime); } else { - NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.output_label.text, error]; - self.output_label.text = str; + NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.outputLabel.text, error]; + self.outputLabel.text = str; NSLog(@"RPC error: %@", error); } }]; @@ -245,9 +245,9 @@ static NSString * const kHostAddress = @"localhost:50051"; } - (void)viewDidAppear:(BOOL)animated { - self.output_label.text = @"RPC log:"; - self.output_label.numberOfLines = 0; - self.output_label.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; + self.outputLabel.text = @"RPC log:"; + self.outputLabel.numberOfLines = 0; + self.outputLabel.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; [self execRequest]; } @@ -263,7 +263,7 @@ static NSString * const kHostAddress = @"localhost:50051"; @interface RouteChatViewController : UIViewController { RTGRouteGuide *service; } -@property (weak, nonatomic) IBOutlet UILabel *output_label; +@property (weak, nonatomic) IBOutlet UILabel *outputLabel; @end @@ -283,12 +283,12 @@ static NSString * const kHostAddress = @"localhost:50051"; eventHandler:^(BOOL done, RTGRouteNote *note, NSError *error) { if (note) { NSString *str =[NSString stringWithFormat:@"%@\nGot message %@ at %@", - self.output_label.text, note.message, note.location]; - self.output_label.text = str; + self.outputLabel.text, note.message, note.location]; + self.outputLabel.text = str; NSLog(@"Got message %@ at %@", note.message, note.location); } else if (error) { - NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.output_label.text, error]; - self.output_label.text = str; + NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.outputLabel.text, error]; + self.outputLabel.text = str; NSLog(@"RPC error: %@", error); } if (done) { @@ -304,9 +304,9 @@ static NSString * const kHostAddress = @"localhost:50051"; } - (void)viewDidAppear:(BOOL)animated { - self.output_label.text = @"RPC log:"; - self.output_label.numberOfLines = 0; - self.output_label.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; + self.outputLabel.text = @"RPC log:"; + self.outputLabel.numberOfLines = 0; + self.outputLabel.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; [self execRequest]; } From 05998dc6939bb7ad4230775bb5ac87696ae9e29b Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Thu, 16 Jun 2016 10:16:44 -0700 Subject: [PATCH 0109/1039] More formatting fixes --- src/core/lib/surface/completion_queue.c | 1 - 1 file changed, 1 deletion(-) diff --git a/src/core/lib/surface/completion_queue.c b/src/core/lib/surface/completion_queue.c index 03de97bc2b5..d70f1721463 100644 --- a/src/core/lib/surface/completion_queue.c +++ b/src/core/lib/surface/completion_queue.c @@ -101,7 +101,6 @@ int grpc_cq_event_timeout_trace; gpr_free(_ev); \ } - static void on_pollset_shutdown_done(grpc_exec_ctx *exec_ctx, void *cc, bool success); From 8a1a5976c7eced718e2385b7f36046d6e73017af Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 16 Jun 2016 10:22:26 -0700 Subject: [PATCH 0110/1039] Fixed test. --- test/cpp/end2end/filter_end2end_test.cc | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/test/cpp/end2end/filter_end2end_test.cc b/test/cpp/end2end/filter_end2end_test.cc index 79a5287fbcf..be6988c6baf 100644 --- a/test/cpp/end2end/filter_end2end_test.cc +++ b/test/cpp/end2end/filter_end2end_test.cc @@ -103,8 +103,8 @@ class CallDataImpl : public CallData { void StartTransportStreamOp( grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_transport_stream_op *op) { - gpr_log(GPR_ERROR, "INCREMENTING"); - IncrementCounter(); + if (op->recv_initial_metadata != nullptr) + IncrementCounter(); grpc_call_next_op(exec_ctx, elem, op); } }; @@ -120,8 +120,6 @@ class FilterEnd2endTest : public ::testing::Test { FilterEnd2endTest() : server_host_("localhost") {} void SetUp() GRPC_OVERRIDE { - RegisterChannelFilter( - "test-filter", GRPC_SERVER_CHANNEL, INT_MAX, nullptr); int port = grpc_pick_unused_port_or_die(); server_address_ << server_host_ << ":" << port; // Setup server @@ -323,5 +321,8 @@ TEST_F(FilterEnd2endTest, SimpleBidiStreaming) { int main(int argc, char** argv) { grpc_test_init(argc, argv); ::testing::InitGoogleTest(&argc, argv); + grpc::RegisterChannelFilter( + "test-filter", GRPC_SERVER_CHANNEL, INT_MAX, nullptr); return RUN_ALL_TESTS(); } From 4412c85a1079f2ac55a9a27641076ca34c1a2c62 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Thu, 16 Jun 2016 10:37:51 -0700 Subject: [PATCH 0111/1039] crud removal --- examples/objective-c/route_guide/ViewControllers.m | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/objective-c/route_guide/ViewControllers.m b/examples/objective-c/route_guide/ViewControllers.m index 1e84c334728..5fd411e3ba6 100644 --- a/examples/objective-c/route_guide/ViewControllers.m +++ b/examples/objective-c/route_guide/ViewControllers.m @@ -37,7 +37,7 @@ #import #import -static NSString * const kHostAddress = @"192.168.1.120:50051"; +static NSString * const kHostAddress = @"localhost:50051"; /** Category to override RTGPoint's description. */ @interface RTGPoint (Description) From c008b33c18f1d2c49ee1b6683c2d13a85e2c2432 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 16 Jun 2016 10:39:39 -0700 Subject: [PATCH 0112/1039] Pass channel args to ChannelData ctor and ChannelData to CallData ctor. --- include/grpc++/channel_filter.h | 79 +++++++++++++------------ src/cpp/common/channel_filter.cc | 2 +- test/cpp/end2end/filter_end2end_test.cc | 15 ++--- 3 files changed, 49 insertions(+), 47 deletions(-) diff --git a/include/grpc++/channel_filter.h b/include/grpc++/channel_filter.h index b37d986a5ac..8731a5e9651 100644 --- a/include/grpc++/channel_filter.h +++ b/include/grpc++/channel_filter.h @@ -53,6 +53,20 @@ namespace grpc { +// Represents channel data. +// Note: Must be copyable. +class ChannelData { + public: + virtual ~ChannelData() {} + + virtual void StartTransportOp( + grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, + grpc_transport_op *op); + + protected: + explicit ChannelData(const grpc_channel_args&) {} +}; + // Represents call data. // Note: Must be copyable. class CallData { @@ -70,21 +84,7 @@ class CallData { virtual char* GetPeer(grpc_exec_ctx *exec_ctx, grpc_call_element *elem); protected: - CallData() {} -}; - -// Represents channel data. -// Note: Must be copyable. -class ChannelData { - public: - virtual ~ChannelData() {} - - virtual void StartTransportOp( - grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, - grpc_transport_op *op); - - protected: - ChannelData() {} + explicit CallData(const ChannelData&) {} }; namespace internal { @@ -93,13 +93,35 @@ namespace internal { template class ChannelFilter { public: + static const size_t channel_data_size = sizeof(ChannelDataType); + + static void InitChannelElement( + grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, + grpc_channel_element_args *args) { + // Construct the object in the already-allocated memory. + new (elem->channel_data) ChannelDataType(*args->channel_args); + } + + static void DestroyChannelElement( + grpc_exec_ctx *exec_ctx, grpc_channel_element *elem) { + reinterpret_cast(elem->channel_data)->~ChannelDataType(); + } + + static void StartTransportOp( + grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, + grpc_transport_op *op) { + ChannelDataType* channel_data = (ChannelDataType*)elem->channel_data; + channel_data->StartTransportOp(exec_ctx, elem, op); + } + static const size_t call_data_size = sizeof(CallDataType); static void InitCallElement( grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { + const ChannelDataType& channel_data = *(ChannelDataType*)elem->channel_data; // Construct the object in the already-allocated memory. - new (elem->call_data) CallDataType(); + new (elem->call_data) CallDataType(channel_data); } static void DestroyCallElement( @@ -127,33 +149,12 @@ class ChannelFilter { CallDataType* call_data = (CallDataType*)elem->call_data; return call_data->GetPeer(exec_ctx, elem); } - - static const size_t channel_data_size = sizeof(ChannelDataType); - - static void InitChannelElement( - grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, - grpc_channel_element_args *args) { - // Construct the object in the already-allocated memory. - new (elem->channel_data) ChannelDataType(); - } - - static void DestroyChannelElement( - grpc_exec_ctx *exec_ctx, grpc_channel_element *elem) { - reinterpret_cast(elem->channel_data)->~ChannelDataType(); - } - - static void StartTransportOp( - grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, - grpc_transport_op *op) { - ChannelDataType* channel_data = (ChannelDataType*)elem->channel_data; - channel_data->StartTransportOp(exec_ctx, elem, op); - } }; struct FilterRecord { grpc_channel_stack_type stack_type; int priority; - std::function include_filter; + std::function include_filter; grpc_channel_filter filter; }; extern std::vector* channel_filters; @@ -171,7 +172,7 @@ void ChannelFilterPluginShutdown(); template void RegisterChannelFilter( const char* name, grpc_channel_stack_type stack_type, int priority, - std::function include_filter) { + std::function include_filter) { // If we haven't been called before, initialize channel_filters and // call grpc_register_plugin(). if (internal::channel_filters == nullptr) { diff --git a/src/cpp/common/channel_filter.cc b/src/cpp/common/channel_filter.cc index b5e5e08976b..77b2a26e8cc 100644 --- a/src/cpp/common/channel_filter.cc +++ b/src/cpp/common/channel_filter.cc @@ -83,7 +83,7 @@ bool MaybeAddFilter(grpc_channel_stack_builder* builder, void* arg) { if (filter.include_filter != nullptr) { const grpc_channel_args *args = grpc_channel_stack_builder_get_channel_arguments(builder); - if (!filter.include_filter(args)) + if (!filter.include_filter(*args)) return true; } return grpc_channel_stack_builder_prepend_filter( diff --git a/test/cpp/end2end/filter_end2end_test.cc b/test/cpp/end2end/filter_end2end_test.cc index be6988c6baf..16151c21b84 100644 --- a/test/cpp/end2end/filter_end2end_test.cc +++ b/test/cpp/end2end/filter_end2end_test.cc @@ -95,9 +95,16 @@ int GetCounterValue() { } // namespace +class ChannelDataImpl : public ChannelData { + public: + explicit ChannelDataImpl(const grpc_channel_args& args) : ChannelData(args) {} + virtual ~ChannelDataImpl() {} +}; + class CallDataImpl : public CallData { public: - CallDataImpl() {} + explicit CallDataImpl(const ChannelDataImpl& channel_data) + : CallData(channel_data) {} virtual ~CallDataImpl() {} void StartTransportStreamOp( @@ -109,12 +116,6 @@ class CallDataImpl : public CallData { } }; -class ChannelDataImpl : public ChannelData { - public: - ChannelDataImpl() {} - virtual ~ChannelDataImpl() {} -}; - class FilterEnd2endTest : public ::testing::Test { protected: FilterEnd2endTest() : server_host_("localhost") {} From f9c1f7a412619e78a93a116068fe0fe373e172e1 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 16 Jun 2016 10:57:28 -0700 Subject: [PATCH 0113/1039] clang-format --- include/grpc++/channel_filter.h | 114 ++++++++++++------------ src/cpp/common/channel_filter.cc | 38 ++++---- test/cpp/end2end/filter_end2end_test.cc | 8 +- 3 files changed, 76 insertions(+), 84 deletions(-) diff --git a/include/grpc++/channel_filter.h b/include/grpc++/channel_filter.h index 8731a5e9651..f73abe8e6c1 100644 --- a/include/grpc++/channel_filter.h +++ b/include/grpc++/channel_filter.h @@ -59,12 +59,12 @@ class ChannelData { public: virtual ~ChannelData() {} - virtual void StartTransportOp( - grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, - grpc_transport_op *op); + virtual void StartTransportOp(grpc_exec_ctx *exec_ctx, + grpc_channel_element *elem, + grpc_transport_op *op); protected: - explicit ChannelData(const grpc_channel_args&) {} + explicit ChannelData(const grpc_channel_args &) {} }; // Represents call data. @@ -73,80 +73,80 @@ class CallData { public: virtual ~CallData() {} - virtual void StartTransportStreamOp( - grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_transport_stream_op *op); + virtual void StartTransportStreamOp(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_transport_stream_op *op); - virtual void SetPollsetOrPollsetSet( - grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_polling_entity *pollent); + virtual void SetPollsetOrPollsetSet(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_polling_entity *pollent); - virtual char* GetPeer(grpc_exec_ctx *exec_ctx, grpc_call_element *elem); + virtual char *GetPeer(grpc_exec_ctx *exec_ctx, grpc_call_element *elem); protected: - explicit CallData(const ChannelData&) {} + explicit CallData(const ChannelData &) {} }; namespace internal { // Defines static members for passing to C core. -template +template class ChannelFilter { public: static const size_t channel_data_size = sizeof(ChannelDataType); - static void InitChannelElement( - grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, - grpc_channel_element_args *args) { + static void InitChannelElement(grpc_exec_ctx *exec_ctx, + grpc_channel_element *elem, + grpc_channel_element_args *args) { // Construct the object in the already-allocated memory. new (elem->channel_data) ChannelDataType(*args->channel_args); } - static void DestroyChannelElement( - grpc_exec_ctx *exec_ctx, grpc_channel_element *elem) { - reinterpret_cast(elem->channel_data)->~ChannelDataType(); + static void DestroyChannelElement(grpc_exec_ctx *exec_ctx, + grpc_channel_element *elem) { + reinterpret_cast(elem->channel_data)->~ChannelDataType(); } - static void StartTransportOp( - grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, - grpc_transport_op *op) { - ChannelDataType* channel_data = (ChannelDataType*)elem->channel_data; + static void StartTransportOp(grpc_exec_ctx *exec_ctx, + grpc_channel_element *elem, + grpc_transport_op *op) { + ChannelDataType *channel_data = (ChannelDataType *)elem->channel_data; channel_data->StartTransportOp(exec_ctx, elem, op); } static const size_t call_data_size = sizeof(CallDataType); - static void InitCallElement( - grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) { - const ChannelDataType& channel_data = *(ChannelDataType*)elem->channel_data; + static void InitCallElement(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, + grpc_call_element_args *args) { + const ChannelDataType &channel_data = + *(ChannelDataType *)elem->channel_data; // Construct the object in the already-allocated memory. new (elem->call_data) CallDataType(channel_data); } - static void DestroyCallElement( - grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - const grpc_call_stats *stats, void *and_free_memory) { - reinterpret_cast(elem->call_data)->~CallDataType(); + static void DestroyCallElement(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + const grpc_call_stats *stats, + void *and_free_memory) { + reinterpret_cast(elem->call_data)->~CallDataType(); } - static void StartTransportStreamOp( - grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_transport_stream_op *op) { - CallDataType* call_data = (CallDataType*)elem->call_data; + static void StartTransportStreamOp(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_transport_stream_op *op) { + CallDataType *call_data = (CallDataType *)elem->call_data; call_data->StartTransportStreamOp(exec_ctx, elem, op); } - static void SetPollsetOrPollsetSet( - grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_polling_entity *pollent) { - CallDataType* call_data = (CallDataType*)elem->call_data; + static void SetPollsetOrPollsetSet(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_polling_entity *pollent) { + CallDataType *call_data = (CallDataType *)elem->call_data; call_data->SetPollsetOrPollsetSet(exec_ctx, elem, pollent); } - static char* GetPeer( - grpc_exec_ctx *exec_ctx, grpc_call_element *elem) { - CallDataType* call_data = (CallDataType*)elem->call_data; + static char *GetPeer(grpc_exec_ctx *exec_ctx, grpc_call_element *elem) { + CallDataType *call_data = (CallDataType *)elem->call_data; return call_data->GetPeer(exec_ctx, elem); } }; @@ -154,10 +154,10 @@ class ChannelFilter { struct FilterRecord { grpc_channel_stack_type stack_type; int priority; - std::function include_filter; + std::function include_filter; grpc_channel_filter filter; }; -extern std::vector* channel_filters; +extern std::vector *channel_filters; void ChannelFilterPluginInit(); void ChannelFilterPluginShutdown(); @@ -169,10 +169,10 @@ void ChannelFilterPluginShutdown(); // The include_filter argument specifies a function that will be called // to determine at run-time whether or not to add the filter. If the // value is nullptr, the filter will be added unconditionally. -template +template void RegisterChannelFilter( - const char* name, grpc_channel_stack_type stack_type, int priority, - std::function include_filter) { + const char *name, grpc_channel_stack_type stack_type, int priority, + std::function include_filter) { // If we haven't been called before, initialize channel_filters and // call grpc_register_plugin(). if (internal::channel_filters == nullptr) { @@ -184,18 +184,14 @@ void RegisterChannelFilter( // C-core initialization code calls ChannelFilterPluginInit(). typedef internal::ChannelFilter FilterType; internal::FilterRecord filter_record = { - stack_type, priority, include_filter, { - FilterType::StartTransportStreamOp, - FilterType::StartTransportOp, - FilterType::call_data_size, - FilterType::InitCallElement, - FilterType::SetPollsetOrPollsetSet, - FilterType::DestroyCallElement, - FilterType::channel_data_size, - FilterType::InitChannelElement, - FilterType::DestroyChannelElement, - FilterType::GetPeer, - name}}; + stack_type, + priority, + include_filter, + {FilterType::StartTransportStreamOp, FilterType::StartTransportOp, + FilterType::call_data_size, FilterType::InitCallElement, + FilterType::SetPollsetOrPollsetSet, FilterType::DestroyCallElement, + FilterType::channel_data_size, FilterType::InitChannelElement, + FilterType::DestroyChannelElement, FilterType::GetPeer, name}}; internal::channel_filters->push_back(filter_record); } diff --git a/src/cpp/common/channel_filter.cc b/src/cpp/common/channel_filter.cc index 77b2a26e8cc..86df47903fd 100644 --- a/src/cpp/common/channel_filter.cc +++ b/src/cpp/common/channel_filter.cc @@ -41,20 +41,19 @@ namespace grpc { // CallData // -void CallData::StartTransportStreamOp( - grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_transport_stream_op *op) { +void CallData::StartTransportStreamOp(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_transport_stream_op *op) { grpc_call_next_op(exec_ctx, elem, op); } -void CallData::SetPollsetOrPollsetSet( - grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_polling_entity *pollent) { +void CallData::SetPollsetOrPollsetSet(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_polling_entity *pollent) { grpc_call_stack_ignore_set_pollset_or_pollset_set(exec_ctx, elem, pollent); } -char* CallData::GetPeer( - grpc_exec_ctx *exec_ctx, grpc_call_element *elem) { +char *CallData::GetPeer(grpc_exec_ctx *exec_ctx, grpc_call_element *elem) { return grpc_call_next_get_peer(exec_ctx, elem); } @@ -62,9 +61,9 @@ char* CallData::GetPeer( // ChannelData // -void ChannelData::StartTransportOp( - grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, - grpc_transport_op *op) { +void ChannelData::StartTransportOp(grpc_exec_ctx *exec_ctx, + grpc_channel_element *elem, + grpc_transport_op *op) { grpc_channel_next_op(exec_ctx, elem, op); } @@ -74,29 +73,28 @@ void ChannelData::StartTransportOp( namespace internal { -std::vector* channel_filters = nullptr; +std::vector *channel_filters = nullptr; namespace { -bool MaybeAddFilter(grpc_channel_stack_builder* builder, void* arg) { - const FilterRecord& filter = *(FilterRecord*)arg; +bool MaybeAddFilter(grpc_channel_stack_builder *builder, void *arg) { + const FilterRecord &filter = *(FilterRecord *)arg; if (filter.include_filter != nullptr) { const grpc_channel_args *args = grpc_channel_stack_builder_get_channel_arguments(builder); - if (!filter.include_filter(*args)) - return true; + if (!filter.include_filter(*args)) return true; } - return grpc_channel_stack_builder_prepend_filter( - builder, &filter.filter, nullptr, nullptr); + return grpc_channel_stack_builder_prepend_filter(builder, &filter.filter, + nullptr, nullptr); } } // namespace void ChannelFilterPluginInit() { for (size_t i = 0; i < channel_filters->size(); ++i) { - FilterRecord& filter = (*channel_filters)[i]; + FilterRecord &filter = (*channel_filters)[i]; grpc_channel_init_register_stage(filter.stack_type, filter.priority, - MaybeAddFilter, (void*)&filter); + MaybeAddFilter, (void *)&filter); } } diff --git a/test/cpp/end2end/filter_end2end_test.cc b/test/cpp/end2end/filter_end2end_test.cc index 16151c21b84..d72e8100d74 100644 --- a/test/cpp/end2end/filter_end2end_test.cc +++ b/test/cpp/end2end/filter_end2end_test.cc @@ -107,11 +107,9 @@ class CallDataImpl : public CallData { : CallData(channel_data) {} virtual ~CallDataImpl() {} - void StartTransportStreamOp( - grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_transport_stream_op *op) { - if (op->recv_initial_metadata != nullptr) - IncrementCounter(); + void StartTransportStreamOp(grpc_exec_ctx* exec_ctx, grpc_call_element* elem, + grpc_transport_stream_op* op) { + if (op->recv_initial_metadata != nullptr) IncrementCounter(); grpc_call_next_op(exec_ctx, elem, op); } }; From 905227049b494d6ecd3e8bf4ae0f00e3f499f0cf Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 16 Jun 2016 11:40:25 -0700 Subject: [PATCH 0114/1039] Remove unnecessary code from test. --- test/cpp/end2end/filter_end2end_test.cc | 4 ---- 1 file changed, 4 deletions(-) diff --git a/test/cpp/end2end/filter_end2end_test.cc b/test/cpp/end2end/filter_end2end_test.cc index d72e8100d74..7cb77be42d4 100644 --- a/test/cpp/end2end/filter_end2end_test.cc +++ b/test/cpp/end2end/filter_end2end_test.cc @@ -126,10 +126,6 @@ class FilterEnd2endTest : public ::testing::Test { builder.AddListeningPort(server_address_.str(), InsecureServerCredentials()); builder.RegisterAsyncGenericService(&generic_service_); - // Include a second call to RegisterAsyncGenericService to make sure that - // we get an error in the log, since it is not allowed to have 2 async - // generic services - builder.RegisterAsyncGenericService(&generic_service_); srv_cq_ = builder.AddCompletionQueue(); server_ = builder.BuildAndStart(); } From 00f41c1eed57f84a2efdfbd96a59dd00d523cde0 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Thu, 16 Jun 2016 12:46:52 -0700 Subject: [PATCH 0115/1039] Add comment about new trace flags --- src/core/lib/surface/completion_queue.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/core/lib/surface/completion_queue.h b/src/core/lib/surface/completion_queue.h index 108445b28f1..e4018695e27 100644 --- a/src/core/lib/surface/completion_queue.h +++ b/src/core/lib/surface/completion_queue.h @@ -39,6 +39,8 @@ #include #include "src/core/lib/iomgr/pollset.h" +/* These trace flags default to 1. The corresponding lines are only traced + if grpc_api_trace is also truthy */ extern int grpc_cq_pluck_trace; extern int grpc_cq_event_timeout_trace; From f5bbff9c8e314e29e12e9891c75908f60d941a5f Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 16 Jun 2016 12:56:07 -0700 Subject: [PATCH 0116/1039] Fix portability issues. --- src/cpp/common/channel_filter.cc | 5 +++-- test/cpp/end2end/filter_end2end_test.cc | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/src/cpp/common/channel_filter.cc b/src/cpp/common/channel_filter.cc index 86df47903fd..f81a01c62d2 100644 --- a/src/cpp/common/channel_filter.cc +++ b/src/cpp/common/channel_filter.cc @@ -73,13 +73,14 @@ void ChannelData::StartTransportOp(grpc_exec_ctx *exec_ctx, namespace internal { -std::vector *channel_filters = nullptr; +// Note: Implicitly initialized to nullptr due to static lifetime. +std::vector *channel_filters; namespace { bool MaybeAddFilter(grpc_channel_stack_builder *builder, void *arg) { const FilterRecord &filter = *(FilterRecord *)arg; - if (filter.include_filter != nullptr) { + if (filter.include_filter) { const grpc_channel_args *args = grpc_channel_stack_builder_get_channel_arguments(builder); if (!filter.include_filter(*args)) return true; diff --git a/test/cpp/end2end/filter_end2end_test.cc b/test/cpp/end2end/filter_end2end_test.cc index 7cb77be42d4..4ba3b9e0d1c 100644 --- a/test/cpp/end2end/filter_end2end_test.cc +++ b/test/cpp/end2end/filter_end2end_test.cc @@ -108,7 +108,7 @@ class CallDataImpl : public CallData { virtual ~CallDataImpl() {} void StartTransportStreamOp(grpc_exec_ctx* exec_ctx, grpc_call_element* elem, - grpc_transport_stream_op* op) { + grpc_transport_stream_op* op) GRPC_OVERRIDE { if (op->recv_initial_metadata != nullptr) IncrementCounter(); grpc_call_next_op(exec_ctx, elem, op); } From 9fe5ef5f9d17289f57ee1d5436314c3153b80975 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 16 Jun 2016 14:26:04 -0700 Subject: [PATCH 0117/1039] Removed unnecessary comments. Added connection counter to test. --- include/grpc++/channel_filter.h | 2 - src/cpp/common/channel_filter.cc | 20 ++++----- test/cpp/end2end/filter_end2end_test.cc | 54 ++++++++++++++++++------- 3 files changed, 50 insertions(+), 26 deletions(-) diff --git a/include/grpc++/channel_filter.h b/include/grpc++/channel_filter.h index f73abe8e6c1..542e6a0f35d 100644 --- a/include/grpc++/channel_filter.h +++ b/include/grpc++/channel_filter.h @@ -54,7 +54,6 @@ namespace grpc { // Represents channel data. -// Note: Must be copyable. class ChannelData { public: virtual ~ChannelData() {} @@ -68,7 +67,6 @@ class ChannelData { }; // Represents call data. -// Note: Must be copyable. class CallData { public: virtual ~CallData() {} diff --git a/src/cpp/common/channel_filter.cc b/src/cpp/common/channel_filter.cc index f81a01c62d2..f473d63a257 100644 --- a/src/cpp/common/channel_filter.cc +++ b/src/cpp/common/channel_filter.cc @@ -37,6 +37,16 @@ namespace grpc { +// +// ChannelData +// + +void ChannelData::StartTransportOp(grpc_exec_ctx *exec_ctx, + grpc_channel_element *elem, + grpc_transport_op *op) { + grpc_channel_next_op(exec_ctx, elem, op); +} + // // CallData // @@ -57,16 +67,6 @@ char *CallData::GetPeer(grpc_exec_ctx *exec_ctx, grpc_call_element *elem) { return grpc_call_next_get_peer(exec_ctx, elem); } -// -// ChannelData -// - -void ChannelData::StartTransportOp(grpc_exec_ctx *exec_ctx, - grpc_channel_element *elem, - grpc_transport_op *op) { - grpc_channel_next_op(exec_ctx, elem, op); -} - // // RegisterChannelFilter() // diff --git a/test/cpp/end2end/filter_end2end_test.cc b/test/cpp/end2end/filter_end2end_test.cc index 4ba3b9e0d1c..b21d377d5dd 100644 --- a/test/cpp/end2end/filter_end2end_test.cc +++ b/test/cpp/end2end/filter_end2end_test.cc @@ -75,20 +75,36 @@ void verify_ok(CompletionQueue* cq, int i, bool expect_ok) { namespace { +int global_num_connections = 0; int global_num_calls = 0; mutex global_mu; -void IncrementCounter() { +void IncrementConnectionCounter() { + unique_lock lock(global_mu); + ++global_num_connections; +} + +void ResetConnectionCounter() { + unique_lock lock(global_mu); + global_num_connections = 0; +} + +int GetConnectionCounterValue() { + unique_lock lock(global_mu); + return global_num_connections; +} + +void IncrementCallCounter() { unique_lock lock(global_mu); ++global_num_calls; } -void ResetCounter() { +void ResetCallCounter() { unique_lock lock(global_mu); global_num_calls = 0; } -int GetCounterValue() { +int GetCallCounterValue() { unique_lock lock(global_mu); return global_num_calls; } @@ -97,19 +113,22 @@ int GetCounterValue() { class ChannelDataImpl : public ChannelData { public: - explicit ChannelDataImpl(const grpc_channel_args& args) : ChannelData(args) {} - virtual ~ChannelDataImpl() {} + ChannelDataImpl(const grpc_channel_args& args, const char* peer) + : ChannelData(args, peer) { + IncrementConnectionCounter(); + } }; class CallDataImpl : public CallData { public: explicit CallDataImpl(const ChannelDataImpl& channel_data) : CallData(channel_data) {} - virtual ~CallDataImpl() {} void StartTransportStreamOp(grpc_exec_ctx* exec_ctx, grpc_call_element* elem, grpc_transport_stream_op* op) GRPC_OVERRIDE { - if (op->recv_initial_metadata != nullptr) IncrementCounter(); + // Incrementing the counter could be done from the ctor, but we want + // to test that the individual methods are actually called correctly. + if (op->recv_initial_metadata != nullptr) IncrementCallCounter(); grpc_call_next_op(exec_ctx, elem, op); } }; @@ -146,7 +165,8 @@ class FilterEnd2endTest : public ::testing::Test { std::shared_ptr channel = CreateChannel(server_address_.str(), InsecureChannelCredentials()); generic_stub_.reset(new GenericStub(channel)); - ResetCounter(); + ResetConnectionCounter(); + ResetCallCounter(); } void server_ok(int i) { verify_ok(srv_cq_.get(), i, true); } @@ -227,22 +247,27 @@ class FilterEnd2endTest : public ::testing::Test { TEST_F(FilterEnd2endTest, SimpleRpc) { ResetStub(); - EXPECT_EQ(0, GetCounterValue()); + EXPECT_EQ(0, GetConnectionCounterValue()); + EXPECT_EQ(0, GetCallCounterValue()); SendRpc(1); - EXPECT_EQ(1, GetCounterValue()); + EXPECT_EQ(1, GetConnectionCounterValue()); + EXPECT_EQ(1, GetCallCounterValue()); } TEST_F(FilterEnd2endTest, SequentialRpcs) { ResetStub(); - EXPECT_EQ(0, GetCounterValue()); + EXPECT_EQ(0, GetConnectionCounterValue()); + EXPECT_EQ(0, GetCallCounterValue()); SendRpc(10); - EXPECT_EQ(10, GetCounterValue()); + EXPECT_EQ(1, GetConnectionCounterValue()); + EXPECT_EQ(10, GetCallCounterValue()); } // One ping, one pong. TEST_F(FilterEnd2endTest, SimpleBidiStreaming) { ResetStub(); - EXPECT_EQ(0, GetCounterValue()); + EXPECT_EQ(0, GetConnectionCounterValue()); + EXPECT_EQ(0, GetCallCounterValue()); const grpc::string kMethodName( "/grpc.cpp.test.util.EchoTestService/BidiStream"); @@ -306,7 +331,8 @@ TEST_F(FilterEnd2endTest, SimpleBidiStreaming) { EXPECT_EQ(send_response.message(), recv_response.message()); EXPECT_TRUE(recv_status.ok()); - EXPECT_EQ(1, GetCounterValue()); + EXPECT_EQ(1, GetCallCounterValue()); + EXPECT_EQ(1, GetConnectionCounterValue()); } } // namespace From 035cb3a3d4fe9634834b5814827e49adc9cf6b69 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 16 Jun 2016 14:52:41 -0700 Subject: [PATCH 0118/1039] Pass peer string to ChannelData ctor when available. --- include/grpc++/channel_filter.h | 13 +++++++++++-- src/core/lib/transport/transport.h | 8 ++++++++ 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/include/grpc++/channel_filter.h b/include/grpc++/channel_filter.h index 542e6a0f35d..53e42a7cb81 100644 --- a/include/grpc++/channel_filter.h +++ b/include/grpc++/channel_filter.h @@ -62,8 +62,13 @@ class ChannelData { grpc_channel_element *elem, grpc_transport_op *op); + const char* peer() const { return peer_; } + protected: - explicit ChannelData(const grpc_channel_args &) {} + ChannelData(const grpc_channel_args &args, const char *peer) : peer_(peer) {} + + private: + const char *peer_; // Do not own. }; // Represents call data. @@ -96,8 +101,12 @@ class ChannelFilter { static void InitChannelElement(grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, grpc_channel_element_args *args) { + const char* peer = args->optional_transport + ? grpc_transport_get_peer(exec_ctx, + args->optional_transport) + : nullptr; // Construct the object in the already-allocated memory. - new (elem->channel_data) ChannelDataType(*args->channel_args); + new (elem->channel_data) ChannelDataType(*args->channel_args, peer); } static void DestroyChannelElement(grpc_exec_ctx *exec_ctx, diff --git a/src/core/lib/transport/transport.h b/src/core/lib/transport/transport.h index ed06fc3ed21..107bb802f66 100644 --- a/src/core/lib/transport/transport.h +++ b/src/core/lib/transport/transport.h @@ -43,6 +43,10 @@ #include "src/core/lib/transport/byte_stream.h" #include "src/core/lib/transport/metadata_batch.h" +#ifdef __cplusplus +extern "C" { +#endif + /* forward declarations */ typedef struct grpc_transport grpc_transport; @@ -264,4 +268,8 @@ void grpc_transport_destroy(grpc_exec_ctx *exec_ctx, grpc_transport *transport); char *grpc_transport_get_peer(grpc_exec_ctx *exec_ctx, grpc_transport *transport); +#ifdef __cplusplus +} +#endif + #endif /* GRPC_CORE_LIB_TRANSPORT_TRANSPORT_H */ From 2e12db9c319bcbdbb2fa570149f88e4b496b558c Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 16 Jun 2016 16:53:59 -0700 Subject: [PATCH 0119/1039] Test polling island merges --- Makefile | 36 ++++ build.yaml | 12 ++ src/core/lib/iomgr/ev_epoll_linux.c | 51 +++++- src/core/lib/iomgr/ev_epoll_linux.h | 6 + src/core/lib/iomgr/ev_posix.c | 7 + src/core/lib/iomgr/ev_posix.h | 3 + test/core/iomgr/ev_epoll_linux_test.c | 222 +++++++++++++++++++++++ tools/run_tests/sources_and_headers.json | 16 ++ tools/run_tests/tests.json | 15 ++ 9 files changed, 366 insertions(+), 2 deletions(-) create mode 100644 test/core/iomgr/ev_epoll_linux_test.c diff --git a/Makefile b/Makefile index e615704395b..825684cc2dd 100644 --- a/Makefile +++ b/Makefile @@ -905,6 +905,7 @@ dns_resolver_connectivity_test: $(BINDIR)/$(CONFIG)/dns_resolver_connectivity_te dns_resolver_test: $(BINDIR)/$(CONFIG)/dns_resolver_test dualstack_socket_test: $(BINDIR)/$(CONFIG)/dualstack_socket_test endpoint_pair_test: $(BINDIR)/$(CONFIG)/endpoint_pair_test +ev_epoll_linux_test: $(BINDIR)/$(CONFIG)/ev_epoll_linux_test fd_conservation_posix_test: $(BINDIR)/$(CONFIG)/fd_conservation_posix_test fd_posix_test: $(BINDIR)/$(CONFIG)/fd_posix_test fling_client: $(BINDIR)/$(CONFIG)/fling_client @@ -1242,6 +1243,7 @@ buildtests_c: privatelibs_c \ $(BINDIR)/$(CONFIG)/dns_resolver_test \ $(BINDIR)/$(CONFIG)/dualstack_socket_test \ $(BINDIR)/$(CONFIG)/endpoint_pair_test \ + $(BINDIR)/$(CONFIG)/ev_epoll_linux_test \ $(BINDIR)/$(CONFIG)/fd_conservation_posix_test \ $(BINDIR)/$(CONFIG)/fd_posix_test \ $(BINDIR)/$(CONFIG)/fling_client \ @@ -1512,6 +1514,8 @@ test_c: buildtests_c $(Q) $(BINDIR)/$(CONFIG)/dualstack_socket_test || ( echo test dualstack_socket_test failed ; exit 1 ) $(E) "[RUN] Testing endpoint_pair_test" $(Q) $(BINDIR)/$(CONFIG)/endpoint_pair_test || ( echo test endpoint_pair_test failed ; exit 1 ) + $(E) "[RUN] Testing ev_epoll_linux_test" + $(Q) $(BINDIR)/$(CONFIG)/ev_epoll_linux_test || ( echo test ev_epoll_linux_test failed ; exit 1 ) $(E) "[RUN] Testing fd_conservation_posix_test" $(Q) $(BINDIR)/$(CONFIG)/fd_conservation_posix_test || ( echo test fd_conservation_posix_test failed ; exit 1 ) $(E) "[RUN] Testing fd_posix_test" @@ -7130,6 +7134,38 @@ endif endif +EV_EPOLL_LINUX_TEST_SRC = \ + test/core/iomgr/ev_epoll_linux_test.c \ + +EV_EPOLL_LINUX_TEST_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(EV_EPOLL_LINUX_TEST_SRC)))) +ifeq ($(NO_SECURE),true) + +# You can't build secure targets if you don't have OpenSSL. + +$(BINDIR)/$(CONFIG)/ev_epoll_linux_test: openssl_dep_error + +else + + + +$(BINDIR)/$(CONFIG)/ev_epoll_linux_test: $(EV_EPOLL_LINUX_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + $(E) "[LD] Linking $@" + $(Q) mkdir -p `dirname $@` + $(Q) $(LD) $(LDFLAGS) $(EV_EPOLL_LINUX_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/ev_epoll_linux_test + +endif + +$(OBJDIR)/$(CONFIG)/test/core/iomgr/ev_epoll_linux_test.o: $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + +deps_ev_epoll_linux_test: $(EV_EPOLL_LINUX_TEST_OBJS:.o=.dep) + +ifneq ($(NO_SECURE),true) +ifneq ($(NO_DEPS),true) +-include $(EV_EPOLL_LINUX_TEST_OBJS:.o=.dep) +endif +endif + + FD_CONSERVATION_POSIX_TEST_SRC = \ test/core/iomgr/fd_conservation_posix_test.c \ diff --git a/build.yaml b/build.yaml index 7790e0c5174..84f4ea521be 100644 --- a/build.yaml +++ b/build.yaml @@ -1407,6 +1407,18 @@ targets: - grpc - gpr_test_util - gpr +- name: ev_epoll_linux_test + build: test + language: c + src: + - test/core/iomgr/ev_epoll_linux_test.c + deps: + - grpc_test_util + - grpc + - gpr_test_util + - gpr + platforms: + - linux - name: fd_conservation_posix_test build: test language: c diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 1fb59474640..ed2c494b783 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -317,8 +317,9 @@ static void polling_island_remove_all_fds_locked(polling_island *pi, if (err < 0 && errno != ENOENT) { /* TODO: sreek - We need a better way to bubble up this error instead of * just logging a message */ - gpr_log(GPR_ERROR, "epoll_ctl deleting fds[%zu]: %d failed with error: %s", - i, pi->fds[i]->fd, strerror(errno)); + gpr_log(GPR_ERROR, + "epoll_ctl deleting fds[%zu]: %d failed with error: %s", i, + pi->fds[i]->fd, strerror(errno)); } if (remove_fd_refs) { @@ -1458,6 +1459,52 @@ static void pollset_set_del_pollset_set(grpc_exec_ctx *exec_ctx, gpr_mu_unlock(&bag->mu); } +/* Test helper functions + * */ +void *grpc_fd_get_polling_island(grpc_fd *fd) { + polling_island *pi; + + gpr_mu_lock(&fd->pi_mu); + pi = fd->polling_island; + gpr_mu_unlock(&fd->pi_mu); + + return pi; +} + +void *grpc_pollset_get_polling_island(grpc_pollset *ps) { + polling_island *pi; + + gpr_mu_lock(&ps->pi_mu); + pi = ps->polling_island; + gpr_mu_unlock(&ps->pi_mu); + + return pi; +} + +static polling_island *get_polling_island(polling_island *p) { + if (p == NULL) { + return NULL; + } + + polling_island *next; + gpr_mu_lock(&p->mu); + while (p->merged_to != NULL) { + next = p->merged_to; + gpr_mu_unlock(&p->mu); + p = next; + gpr_mu_lock(&p->mu); + } + gpr_mu_unlock(&p->mu); + + return p; +} + +bool grpc_are_polling_islands_equal(void *p, void *q) { + p = get_polling_island(p); + q = get_polling_island(q); + return p == q; +} + /******************************************************************************* * Event engine binding */ diff --git a/src/core/lib/iomgr/ev_epoll_linux.h b/src/core/lib/iomgr/ev_epoll_linux.h index 8c819975a4c..7a494aba198 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.h +++ b/src/core/lib/iomgr/ev_epoll_linux.h @@ -38,4 +38,10 @@ const grpc_event_engine_vtable *grpc_init_epoll_linux(void); +#ifdef GPR_LINUX_EPOLL +void *grpc_fd_get_polling_island(grpc_fd *fd); +void *grpc_pollset_get_polling_island(grpc_pollset *ps); +bool grpc_are_polling_islands_equal(void *p, void *q); +#endif /* defined(GPR_LINUX_EPOLL) */ + #endif /* GRPC_CORE_LIB_IOMGR_EV_EPOLL_LINUX_H */ diff --git a/src/core/lib/iomgr/ev_posix.c b/src/core/lib/iomgr/ev_posix.c index 2b15967adcc..5b20600a6f7 100644 --- a/src/core/lib/iomgr/ev_posix.c +++ b/src/core/lib/iomgr/ev_posix.c @@ -54,6 +54,7 @@ grpc_poll_function_type grpc_poll_function = poll; static const grpc_event_engine_vtable *g_event_engine; +static const char* g_poll_strategy_name = NULL; typedef const grpc_event_engine_vtable *(*event_engine_factory_fn)(void); @@ -101,6 +102,7 @@ static void try_engine(const char *engine) { for (size_t i = 0; i < GPR_ARRAY_SIZE(g_factories); i++) { if (is(engine, g_factories[i].name)) { if ((g_event_engine = g_factories[i].factory())) { + g_poll_strategy_name = g_factories[i].name; gpr_log(GPR_DEBUG, "Using polling engine: %s", g_factories[i].name); return; } @@ -108,6 +110,11 @@ static void try_engine(const char *engine) { } } +/* Call this only after calling grpc_event_engine_init() */ +const char *grpc_get_poll_strategy_name() { + return g_poll_strategy_name; +} + void grpc_event_engine_init(void) { char *s = gpr_getenv("GRPC_POLL_STRATEGY"); if (s == NULL) { diff --git a/src/core/lib/iomgr/ev_posix.h b/src/core/lib/iomgr/ev_posix.h index 344bf63438a..3ed5a5f9562 100644 --- a/src/core/lib/iomgr/ev_posix.h +++ b/src/core/lib/iomgr/ev_posix.h @@ -98,6 +98,9 @@ typedef struct grpc_event_engine_vtable { void grpc_event_engine_init(void); void grpc_event_engine_shutdown(void); +/* Return the name of the poll strategy */ +const char* grpc_get_poll_strategy_name(); + /* Create a wrapped file descriptor. Requires fd is a non-blocking file descriptor. This takes ownership of closing fd. */ diff --git a/test/core/iomgr/ev_epoll_linux_test.c b/test/core/iomgr/ev_epoll_linux_test.c new file mode 100644 index 00000000000..51da15faa7a --- /dev/null +++ b/test/core/iomgr/ev_epoll_linux_test.c @@ -0,0 +1,222 @@ +/* + * + * Copyright 2015, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include "src/core/lib/iomgr/ev_epoll_linux.h" +#include "src/core/lib/iomgr/ev_posix.h" + +#include +#include +#include +#include + +#include +#include + +#include "src/core/lib/iomgr/iomgr.h" +#include "test/core/util/test_config.h" + +typedef struct test_pollset { + grpc_pollset *pollset; + gpr_mu *mu; +} test_pollset; + +typedef struct test_fd { + int inner_fd; + grpc_fd *fd; +} test_fd; + +static void test_fd_init(test_fd *fds, int num_fds) { + int i; + for (i = 0; i < num_fds; i++) { + fds[i].inner_fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + fds[i].fd = grpc_fd_create(fds[i].inner_fd, "test_fd"); + } +} + +static void test_fd_cleanup(grpc_exec_ctx *exec_ctx, test_fd *fds, + int num_fds) { + int release_fd; + int i; + + for (i = 0; i < num_fds; i++) { + grpc_fd_shutdown(exec_ctx, fds[i].fd); + grpc_exec_ctx_flush(exec_ctx); + + grpc_fd_orphan(exec_ctx, fds[i].fd, NULL, &release_fd, "test_fd_cleanup"); + grpc_exec_ctx_flush(exec_ctx); + + GPR_ASSERT(release_fd == fds[i].inner_fd); + close(fds[i].inner_fd); + } +} + +static void test_pollset_init(test_pollset *pollsets, int num_pollsets) { + int i; + for (i = 0; i < num_pollsets; i++) { + pollsets[i].pollset = gpr_malloc(grpc_pollset_size()); + grpc_pollset_init(pollsets[i].pollset, &pollsets[i].mu); + } +} + +static void destroy_pollset(grpc_exec_ctx *exec_ctx, void *p, bool success) { + grpc_pollset_destroy(p); +} + +static void test_pollset_cleanup(grpc_exec_ctx *exec_ctx, + test_pollset *pollsets, int num_pollsets) { + grpc_closure destroyed; + int i; + + for (i = 0; i < num_pollsets; i++) { + grpc_closure_init(&destroyed, destroy_pollset, pollsets[i].pollset); + grpc_pollset_shutdown(exec_ctx, pollsets[i].pollset, &destroyed); + + grpc_exec_ctx_flush(exec_ctx); + gpr_free(pollsets[i].pollset); + } +} + +#define NUM_FDS 8 +#define NUM_POLLSETS 4 +/* + * Cases to test: + * case 1) Polling islands of both fd and pollset are NULL + * case 2) Polling island of fd is NULL but that of pollset is not-NULL + * case 3) Polling island of fd is not-NULL but that of pollset is NULL + * case 4) Polling islands of both fd and pollset are not-NULL and: + * case 4.1) Polling islands of fd and pollset are equal + * case 4.2) Polling islands of fd and pollset are NOT-equal (This results + * in a merge) + * */ +static void test_add_fd_to_pollset() { + grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; + test_fd fds[NUM_FDS]; + test_pollset pollsets[NUM_POLLSETS]; + void *expected_pi = NULL; + int i; + + test_fd_init(fds, NUM_FDS); + test_pollset_init(pollsets, NUM_POLLSETS); + + /*Step 1. + * Create three polling islands (This will exercise test case 1 and 2) with + * the following configuration: + * polling island 0 = { fds:0,1,2, pollsets:0} + * polling island 1 = { fds:3,4, pollsets:1} + * polling island 2 = { fds:5,6,7 pollsets:2} + * + *Step 2. + * Add pollset 3 to polling island 0 (by adding fds 0 and 1 to pollset 3) + * (This will exercise test cases 3 and 4.1). The configuration becomes: + * polling island 0 = { fds:0,1,2, pollsets:0,3} <<< pollset 3 added here + * polling island 1 = { fds:3,4, pollsets:1} + * polling island 2 = { fds:5,6,7 pollsets:2} + * + *Step 3. + * Merge polling islands 0 and 1 by adding fd 0 to pollset 1 (This will + * exercise test case 4.2). The configuration becomes: + * polling island (merged) = {fds: 0,1,2,3,4, pollsets: 0,1,3} + * polling island 2 = {fds: 5,6,7 pollsets: 2} + * + *Step 4. + * Finally do one more merge by adding fd 3 to pollset 2. + * polling island (merged) = {fds: 0,1,2,3,4,5,6,7, pollsets: 0,1,2,3} + */ + + /* == Step 1 == */ + for (i = 0; i <= 2; i++) { + grpc_pollset_add_fd(&exec_ctx, pollsets[0].pollset, fds[i].fd); + grpc_exec_ctx_flush(&exec_ctx); + } + + for (i = 3; i <= 4; i++) { + grpc_pollset_add_fd(&exec_ctx, pollsets[1].pollset, fds[i].fd); + grpc_exec_ctx_flush(&exec_ctx); + } + + for (i = 5; i <= 7; i++) { + grpc_pollset_add_fd(&exec_ctx, pollsets[2].pollset, fds[i].fd); + grpc_exec_ctx_flush(&exec_ctx); + } + + /* == Step 2 == */ + for (i = 0; i <= 1; i++) { + grpc_pollset_add_fd(&exec_ctx, pollsets[3].pollset, fds[i].fd); + grpc_exec_ctx_flush(&exec_ctx); + } + + /* == Step 3 == */ + grpc_pollset_add_fd(&exec_ctx, pollsets[1].pollset, fds[0].fd); + grpc_exec_ctx_flush(&exec_ctx); + + /* == Step 4 == */ + grpc_pollset_add_fd(&exec_ctx, pollsets[2].pollset, fds[3].fd); + grpc_exec_ctx_flush(&exec_ctx); + + /* All polling islands are merged at this point */ + + /* Compare Fd:0's polling island with that of all other Fds */ + expected_pi = grpc_fd_get_polling_island(fds[0].fd); + for (i = 1; i < NUM_FDS; i++) { + GPR_ASSERT(grpc_are_polling_islands_equal( + expected_pi, grpc_fd_get_polling_island(fds[i].fd))); + } + + /* Compare Fd:0's polling island with that of all other pollsets */ + for (i = 0; i < NUM_POLLSETS; i++) { + GPR_ASSERT(grpc_are_polling_islands_equal( + expected_pi, grpc_pollset_get_polling_island(pollsets[i].pollset))); + } + + test_fd_cleanup(&exec_ctx, fds, NUM_FDS); + test_pollset_cleanup(&exec_ctx, pollsets, NUM_POLLSETS); + grpc_exec_ctx_finish(&exec_ctx); +} + +int main(int argc, char **argv) { + const char *poll_strategy = NULL; + grpc_test_init(argc, argv); + grpc_iomgr_init(); + + poll_strategy = grpc_get_poll_strategy_name(); + if (poll_strategy != NULL && strcmp(poll_strategy, "epoll") == 0) { + test_add_fd_to_pollset(); + } else { + gpr_log(GPR_INFO, + "Skipping the test. The test is only relevant for 'epoll' " + "strategy. and the current strategy is: '%s'", + poll_strategy); + } + grpc_iomgr_shutdown(); + return 0; +} diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index e8ff61dc3fb..e9df72e43a1 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -315,6 +315,22 @@ "third_party": false, "type": "target" }, + { + "deps": [ + "gpr", + "gpr_test_util", + "grpc", + "grpc_test_util" + ], + "headers": [], + "language": "c", + "name": "ev_epoll_linux_test", + "src": [ + "test/core/iomgr/ev_epoll_linux_test.c" + ], + "third_party": false, + "type": "target" + }, { "deps": [ "gpr", diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index 5a84a41b638..ba661840dad 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -377,6 +377,21 @@ "windows" ] }, + { + "args": [], + "ci_platforms": [ + "linux" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "gtest": false, + "language": "c", + "name": "ev_epoll_linux_test", + "platforms": [ + "linux" + ] + }, { "args": [], "ci_platforms": [ From f09ab0c5add9289d40429df63b9f85b6545e9558 Mon Sep 17 00:00:00 2001 From: thinkerou Date: Fri, 3 Jun 2016 18:42:48 +0800 Subject: [PATCH 0120/1039] Make PHP work correctly when receiving a compressed message --- src/php/ext/grpc/byte_buffer.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/src/php/ext/grpc/byte_buffer.c b/src/php/ext/grpc/byte_buffer.c index 7a726de5db4..92ec39952c8 100644 --- a/src/php/ext/grpc/byte_buffer.c +++ b/src/php/ext/grpc/byte_buffer.c @@ -63,17 +63,15 @@ void byte_buffer_to_string(grpc_byte_buffer *buffer, char **out_string, *out_length = 0; return; } - size_t length = grpc_byte_buffer_length(buffer); - char *string = ecalloc(length + 1, sizeof(char)); - size_t offset = 0; + grpc_byte_buffer_reader reader; grpc_byte_buffer_reader_init(&reader, buffer); - gpr_slice next; - while (grpc_byte_buffer_reader_next(&reader, &next) != 0) { - memcpy(string + offset, GPR_SLICE_START_PTR(next), GPR_SLICE_LENGTH(next)); - offset += GPR_SLICE_LENGTH(next); - gpr_slice_unref(next); - } + gpr_slice slice = grpc_byte_buffer_reader_readall(&reader); + size_t length = GPR_SLICE_LENGTH(slice); + char *string = ecalloc(length + 1, sizeof(char)); + memcpy(string, GPR_SLICE_START_PTR(slice), length); + gpr_slice_unref(slice); + *out_string = string; *out_length = length; } From 87e6770a1819d44f352e7ba5782ca1f7dd8cffca Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Thu, 16 Jun 2016 22:04:08 -0700 Subject: [PATCH 0121/1039] pr comments --- doc/compression.md | 40 +++++++++++++++++++++++----------------- 1 file changed, 23 insertions(+), 17 deletions(-) diff --git a/doc/compression.md b/doc/compression.md index ddbbff1c541..6e455636021 100644 --- a/doc/compression.md +++ b/doc/compression.md @@ -41,13 +41,14 @@ of the request, including not performing any compression, regardless of channel and RPC settings (for example, if compression would result in small or negative gains). -A compressed message from a client with an algorithm unsupported by a server, -WILL result in an INVALID\_ARGUMENT error, alongside the receiving peer's -`grpc-accept-encoding` header specifying the algorithms it accepts. If an -INTERNAL error is returned from the server despite having used one of the -algorithms from the `grpc-accept-encoding h`eader, the cause MUST NOT be related -to compression. Data sent from a server compressed with an algorithm not -supported by the client will also result in an INTERNAL error. +When a message from a client compressed with an unsupported algorithm is +processed by a server, it WILL result in an INVALID\_ARGUMENT error. The server +will include in its response a `grpc-accept-encoding` header specifying the +algorithms it does accept. If an INTERNAL error is returned from the server +despite having used one of the algorithms from the `grpc-accept-encoding` +header, the cause MUST NOT be related to compression. Data sent from a server +compressed with an algorithm not supported by the client WILL result in an +INTERNAL error on the client side. Note that a peer MAY choose to not disclose all the encodings it supports. However, if it receives a message compressed in an undisclosed but supported @@ -67,16 +68,21 @@ cases. ### Compression Levels and Algorithms -We currently (as of July 2015) support _gzip_ and _deflate_ as algorithms (with -the possible future addition of snappy). In order to simplify the public API, -it's intended to abstract the algorithms as _compression levels_ (such as "low", -"medium", "high") that'd map to concrete algorithms and/or their settings (such -as "low" mapping to "gzip -3" and "high" mapping to "gzip -9"). However, we -can't presently (July 2015) implement said compression levels at the client side -without either a initial negotiation of capabilities or an automatic retry -mechanism. Therefore, compression levels are only supported at the server side, -which is aware of the client's capabilities by virtue of the incoming -Message-Accept-Encoding header. +The set of supported algorithm is implementation dependent. In order to simplify +the public API and to operate seamlessly across implementations (both in terms +of languages but also different version of the same one), we introduce the idea +of _compression levels_ (such as "low", "medium", "high"). + +Levels map to concrete algorithms and/or their settings (such as "low" mapping +to "gzip -3" and "high" mapping to "gzip -9") automatically depending on what a +peer is known to support. A server is always aware of what its clients support, +as clients disclose it in their Message-Accept-Encoding header as part of their +initial call. A client doesn't a priori (presently) know which algorithms a +server supports. This issue can be addressed with an initial negotiation of +capabilities or an automatic retry mechanism. These features will be implemented +in the future. Currently however, compression levels are only supported at the +server side, which is aware of the client's capabilities by virtue of the +incoming. ### Propagation to child RPCs From b9df2760ed15930429305f01407914db106510d5 Mon Sep 17 00:00:00 2001 From: vjpai Date: Fri, 17 Jun 2016 00:27:00 -0700 Subject: [PATCH 0122/1039] Add proper documentation of ordering and thread-safety for streaming (sync and async) APIs --- include/grpc++/impl/codegen/async_stream.h | 10 +++++++++- include/grpc++/impl/codegen/completion_queue.h | 14 ++++++++++++-- include/grpc++/impl/codegen/sync_stream.h | 10 ++++++++-- 3 files changed, 29 insertions(+), 5 deletions(-) diff --git a/include/grpc++/impl/codegen/async_stream.h b/include/grpc++/impl/codegen/async_stream.h index 0606441549e..c74737ce5f8 100644 --- a/include/grpc++/impl/codegen/async_stream.h +++ b/include/grpc++/impl/codegen/async_stream.h @@ -52,11 +52,14 @@ class ClientAsyncStreamingInterface { /// Request notification of the reading of the initial metadata. Completion /// will be notified by \a tag on the associated completion queue. + /// This call is optional, but if it is used, it cannot be used concurrently + /// with Read /// /// \param[in] tag Tag identifying this request. virtual void ReadInitialMetadata(void* tag) = 0; - /// Request notification completion. + /// Indicate that the stream is to be finished and request notification + /// Should not be used concurrently with other operations /// /// \param[out] status To be updated with the operation status. /// \param[in] tag Tag identifying this request. @@ -71,6 +74,8 @@ class AsyncReaderInterface { /// Read a message of type \a R into \a msg. Completion will be notified by \a /// tag on the associated completion queue. + /// This is thread-safe with respect to other streaming APIs except for Finish + /// on the same stream. /// /// \param[out] msg Where to eventually store the read message. /// \param[in] tag The tag identifying the operation. @@ -88,6 +93,7 @@ class AsyncWriterInterface { /// Only one write may be outstanding at any given time. This means that /// after calling Write, one must wait to receive \a tag from the completion /// queue BEFORE calling Write again. + /// This is thread-safe with respect to Read /// /// \param[in] msg The message to be written. /// \param[in] tag The tag identifying the operation. @@ -158,6 +164,7 @@ class ClientAsyncWriterInterface : public ClientAsyncStreamingInterface, public AsyncWriterInterface { public: /// Signal the client is done with the writes. + /// Thread-safe with respect to Read /// /// \param[in] tag The tag identifying the operation. virtual void WritesDone(void* tag) = 0; @@ -229,6 +236,7 @@ class ClientAsyncReaderWriterInterface : public ClientAsyncStreamingInterface, public AsyncReaderInterface { public: /// Signal the client is done with the writes. + /// Thread-safe with respect to Read /// /// \param[in] tag The tag identifying the operation. virtual void WritesDone(void* tag) = 0; diff --git a/include/grpc++/impl/codegen/completion_queue.h b/include/grpc++/impl/codegen/completion_queue.h index 1b84b447050..f138ebe7de2 100644 --- a/include/grpc++/impl/codegen/completion_queue.h +++ b/include/grpc++/impl/codegen/completion_queue.h @@ -31,8 +31,18 @@ * */ -/// A completion queue implements a concurrent producer-consumer queue, with two -/// main methods, \a Next and \a AsyncNext. +/// A completion queue implements a concurrent producer-consumer queue, with +/// two main API-exposed methods: \a Next and \a AsyncNext. These +/// methods are the essential component of the gRPC C++ asynchronous API. +/// There is also a Shutdown method to indicate that a given completion queue +/// will no longer have regular events. This must be called before the +/// completion queue is destroyed. +/// All completion queue APIs are thread-safe and may be used concurrently with +/// any other completion queue API invocation; it is acceptable to have +/// multiple threads calling Next or AsyncNext on the same or different +/// completion queues, or to call these methods concurrently with a Shutdown +/// elsewhere. All of these should be completed, though, before a completion +/// queue destructor is called. #ifndef GRPCXX_IMPL_CODEGEN_COMPLETION_QUEUE_H #define GRPCXX_IMPL_CODEGEN_COMPLETION_QUEUE_H diff --git a/include/grpc++/impl/codegen/sync_stream.h b/include/grpc++/impl/codegen/sync_stream.h index e94ffe58422..9d7966ba04b 100644 --- a/include/grpc++/impl/codegen/sync_stream.h +++ b/include/grpc++/impl/codegen/sync_stream.h @@ -71,6 +71,8 @@ class ReaderInterface { virtual ~ReaderInterface() {} /// Blocking read a message and parse to \a msg. Returns \a true on success. + /// This is thread-safe with respect to other streaming APIs except for Finish + /// on the same stream. (Finish must be called as described above.) /// /// \param[out] msg The read message. /// @@ -87,6 +89,7 @@ class WriterInterface { virtual ~WriterInterface() {} /// Blocking write \a msg to the stream with options. + /// This is thread-safe with respect to Read /// /// \param msg The message to be written to the stream. /// \param options Options affecting the write operation. @@ -95,6 +98,7 @@ class WriterInterface { virtual bool Write(const W& msg, const WriteOptions& options) = 0; /// Blocking write \a msg to the stream with default options. + /// This is thread-safe with respect to Read /// /// \param msg The message to be written to the stream. /// @@ -174,7 +178,8 @@ class ClientWriterInterface : public ClientStreamingInterface, public WriterInterface { public: /// Half close writing from the client. - /// Block until writes are completed. + /// Block until currently-pending writes are completed. + /// Thread safe with respect to Read operations only /// /// \return Whether the writes were successful. virtual bool WritesDone() = 0; @@ -257,7 +262,8 @@ class ClientReaderWriterInterface : public ClientStreamingInterface, /// the metadata will be available in ClientContext after the first read. virtual void WaitForInitialMetadata() = 0; - /// Block until writes are completed. + /// Block until currently-pending writes are completed. + /// Thread-safe with respect to Read /// /// \return Whether the writes were successful. virtual bool WritesDone() = 0; From 814b80e8e91df31a29ca6e7653e4c4e084145c4e Mon Sep 17 00:00:00 2001 From: thinkerou Date: Thu, 26 May 2016 11:28:50 +0800 Subject: [PATCH 0123/1039] change README for node dir changed --- examples/php/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/examples/php/README.md b/examples/php/README.md index ea9ccb67908..ba7f8c060f0 100644 --- a/examples/php/README.md +++ b/examples/php/README.md @@ -37,7 +37,8 @@ TRY IT! ``` $ cd examples/node $ npm install - $ nodejs greeter_server.js + $ cd dynamic_codegen or cd static_codegen + $ node greeter_server.js ``` - Run the client From 955a364e340cc1954938ae2269693b0a57bec389 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Fri, 17 Jun 2016 13:48:03 -0700 Subject: [PATCH 0124/1039] Add bad_server_response_test --- Makefile | 36 ++ build.yaml | 11 + examples/cpp/helloworld/greeter_client.cc | 2 + test/core/end2end/bad_server_response_test.c | 335 ++++++++++++++++++ tools/run_tests/sources_and_headers.json | 17 + tools/run_tests/tests.json | 21 ++ vsprojects/buildtests_c.sln | 28 ++ .../bad_server_response_test.vcxproj | 202 +++++++++++ .../bad_server_response_test.vcxproj.filters | 21 ++ 9 files changed, 673 insertions(+) create mode 100644 test/core/end2end/bad_server_response_test.c create mode 100644 vsprojects/vcxproj/test/bad_server_response_test/bad_server_response_test.vcxproj create mode 100644 vsprojects/vcxproj/test/bad_server_response_test/bad_server_response_test.vcxproj.filters diff --git a/Makefile b/Makefile index da841af6b5d..5d768357783 100644 --- a/Makefile +++ b/Makefile @@ -889,6 +889,7 @@ algorithm_test: $(BINDIR)/$(CONFIG)/algorithm_test alloc_test: $(BINDIR)/$(CONFIG)/alloc_test alpn_test: $(BINDIR)/$(CONFIG)/alpn_test api_fuzzer: $(BINDIR)/$(CONFIG)/api_fuzzer +bad_server_response_test: $(BINDIR)/$(CONFIG)/bad_server_response_test bin_encoder_test: $(BINDIR)/$(CONFIG)/bin_encoder_test census_context_test: $(BINDIR)/$(CONFIG)/census_context_test channel_create_test: $(BINDIR)/$(CONFIG)/channel_create_test @@ -1226,6 +1227,7 @@ buildtests_c: privatelibs_c \ $(BINDIR)/$(CONFIG)/algorithm_test \ $(BINDIR)/$(CONFIG)/alloc_test \ $(BINDIR)/$(CONFIG)/alpn_test \ + $(BINDIR)/$(CONFIG)/bad_server_response_test \ $(BINDIR)/$(CONFIG)/bin_encoder_test \ $(BINDIR)/$(CONFIG)/census_context_test \ $(BINDIR)/$(CONFIG)/channel_create_test \ @@ -1481,6 +1483,8 @@ test_c: buildtests_c $(Q) $(BINDIR)/$(CONFIG)/alloc_test || ( echo test alloc_test failed ; exit 1 ) $(E) "[RUN] Testing alpn_test" $(Q) $(BINDIR)/$(CONFIG)/alpn_test || ( echo test alpn_test failed ; exit 1 ) + $(E) "[RUN] Testing bad_server_response_test" + $(Q) $(BINDIR)/$(CONFIG)/bad_server_response_test || ( echo test bad_server_response_test failed ; exit 1 ) $(E) "[RUN] Testing bin_encoder_test" $(Q) $(BINDIR)/$(CONFIG)/bin_encoder_test || ( echo test bin_encoder_test failed ; exit 1 ) $(E) "[RUN] Testing census_context_test" @@ -6639,6 +6643,38 @@ endif endif +BAD_SERVER_RESPONSE_TEST_SRC = \ + test/core/end2end/bad_server_response_test.c \ + +BAD_SERVER_RESPONSE_TEST_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(BAD_SERVER_RESPONSE_TEST_SRC)))) +ifeq ($(NO_SECURE),true) + +# You can't build secure targets if you don't have OpenSSL. + +$(BINDIR)/$(CONFIG)/bad_server_response_test: openssl_dep_error + +else + + + +$(BINDIR)/$(CONFIG)/bad_server_response_test: $(BAD_SERVER_RESPONSE_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libtest_tcp_server.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + $(E) "[LD] Linking $@" + $(Q) mkdir -p `dirname $@` + $(Q) $(LD) $(LDFLAGS) $(BAD_SERVER_RESPONSE_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libtest_tcp_server.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/bad_server_response_test + +endif + +$(OBJDIR)/$(CONFIG)/test/core/end2end/bad_server_response_test.o: $(LIBDIR)/$(CONFIG)/libtest_tcp_server.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + +deps_bad_server_response_test: $(BAD_SERVER_RESPONSE_TEST_OBJS:.o=.dep) + +ifneq ($(NO_SECURE),true) +ifneq ($(NO_DEPS),true) +-include $(BAD_SERVER_RESPONSE_TEST_OBJS:.o=.dep) +endif +endif + + BIN_ENCODER_TEST_SRC = \ test/core/transport/chttp2/bin_encoder_test.c \ diff --git a/build.yaml b/build.yaml index 3d1e9b18835..5fa8ad75a1f 100644 --- a/build.yaml +++ b/build.yaml @@ -1247,6 +1247,17 @@ targets: - test/core/end2end/fuzzers/api_fuzzer_corpus dict: test/core/end2end/fuzzers/api_fuzzer.dictionary maxlen: 2048 +- name: bad_server_response_test + build: test + language: c + src: + - test/core/end2end/bad_server_response_test.c + deps: + - test_tcp_server + - grpc_test_util + - grpc + - gpr_test_util + - gpr - name: bin_encoder_test build: test language: c diff --git a/examples/cpp/helloworld/greeter_client.cc b/examples/cpp/helloworld/greeter_client.cc index bf3b63cb577..12209f37dfd 100644 --- a/examples/cpp/helloworld/greeter_client.cc +++ b/examples/cpp/helloworld/greeter_client.cc @@ -72,6 +72,8 @@ class GreeterClient { if (status.ok()) { return reply.message(); } else { + std::cout << status.error_code() << ": " << status.error_message() + << std::endl; return "RPC failed"; } } diff --git a/test/core/end2end/bad_server_response_test.c b/test/core/end2end/bad_server_response_test.c new file mode 100644 index 00000000000..6c00942fb75 --- /dev/null +++ b/test/core/end2end/bad_server_response_test.c @@ -0,0 +1,335 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ +#include + +#include +#include +#include +#include +#include +#include + +#include "src/core/ext/transport/chttp2/transport/internal.h" +#include "src/core/lib/iomgr/sockaddr.h" +#include "src/core/lib/support/string.h" +#include "test/core/end2end/cq_verifier.h" +#include "test/core/util/port.h" +#include "test/core/util/test_config.h" +#include "test/core/util/test_tcp_server.h" + +#define HTTP1_RESP \ + "HTTP/1.0 400 Bad Request\n" \ + "Content-Type: text/html; charset=UTF-8\n" \ + "Content-Length: 0\n" \ + "Date: Tue, 07 Jun 2016 17:43:20 GMT\n\n" + +#define HTTP2_RESP(STATUS_CODE) \ + "\x00\x00\x00\x04\x00\x00\x00\x00\x00" \ + "\x00\x00" \ + "7\x01\x04\x00\x00\x00\x01" \ + "\x10\x0e" \ + "content-length\x01" \ + "0" \ + "\x10\x0c" \ + "content-type\x09text/html" \ + "\x10\x07:status\x03" #STATUS_CODE + +#define UNPARSEABLE_RESP "Bad Request\n" + +#define HTTP1_DETAIL_MSG "Connection dropped: received http1.x response" + +#define HTTP2_DETAIL_MSG(STATUS_CODE) \ + "Received http2 header with status: " #STATUS_CODE + +#define UNPARSEABLE_DETAIL_MSG \ + "Connection dropped: received unparseable response" + +struct rpc_state { + char *target; + grpc_completion_queue *cq; + grpc_channel *channel; + grpc_call *call; + size_t incoming_data_length; + gpr_slice_buffer temp_incoming_buffer; + gpr_slice_buffer outgoing_buffer; + grpc_endpoint *tcp; + gpr_atm done_atm; + bool write_done; + const char *response_payload; + size_t response_payload_length; +}; + +static int server_port; +static struct rpc_state state; +static grpc_closure on_read; +static grpc_closure on_write; + +static void *tag(intptr_t t) { return (void *)t; } + +static void done_write(grpc_exec_ctx *exec_ctx, void *arg, bool success) { + GPR_ASSERT(success); + + gpr_atm_rel_store(&state.done_atm, 1); +} + +static void handle_write(grpc_exec_ctx *exec_ctx) { + gpr_slice slice = gpr_slice_from_copied_buffer(state.response_payload, + state.response_payload_length); + + gpr_slice_buffer_reset_and_unref(&state.outgoing_buffer); + gpr_slice_buffer_add(&state.outgoing_buffer, slice); + grpc_endpoint_write(exec_ctx, state.tcp, &state.outgoing_buffer, &on_write); +} + +static void handle_read(grpc_exec_ctx *exec_ctx, void *arg, bool success) { + GPR_ASSERT(success); + state.incoming_data_length += state.temp_incoming_buffer.length; + + size_t i; + gpr_log(GPR_DEBUG, "read: success=%d", success); + for (i = 0; i < state.temp_incoming_buffer.count; i++) { + char *dump = gpr_dump_slice(state.temp_incoming_buffer.slices[i], + GPR_DUMP_HEX | GPR_DUMP_ASCII); + gpr_log(GPR_DEBUG, "%s", dump); + gpr_free(dump); + } + + gpr_log(GPR_DEBUG, "got %d bytes, http2 connect string is %d bytes", + state.incoming_data_length, GRPC_CHTTP2_CLIENT_CONNECT_STRLEN); + if (state.incoming_data_length > GRPC_CHTTP2_CLIENT_CONNECT_STRLEN) { + handle_write(exec_ctx); + } else { + grpc_endpoint_read(exec_ctx, state.tcp, &state.temp_incoming_buffer, + &on_read); + } +} + +static void on_connect(grpc_exec_ctx *exec_ctx, void *arg, grpc_endpoint *tcp, + grpc_pollset *accepting_pollset, + grpc_tcp_server_acceptor *acceptor) { + test_tcp_server *server = arg; + grpc_closure_init(&on_read, handle_read, NULL); + grpc_closure_init(&on_write, done_write, NULL); + gpr_slice_buffer_init(&state.temp_incoming_buffer); + gpr_slice_buffer_init(&state.outgoing_buffer); + state.tcp = tcp; + grpc_endpoint_add_to_pollset(exec_ctx, tcp, server->pollset); + grpc_endpoint_read(exec_ctx, tcp, &state.temp_incoming_buffer, &on_read); +} + +static gpr_timespec n_sec_deadline(int seconds) { + return gpr_time_add(gpr_now(GPR_CLOCK_REALTIME), + gpr_time_from_seconds(seconds, GPR_TIMESPAN)); +} + +static void start_rpc(int target_port, grpc_status_code expected_status, + const char *expected_detail) { + grpc_op ops[6]; + grpc_op *op; + grpc_metadata_array initial_metadata_recv; + grpc_metadata_array trailing_metadata_recv; + grpc_status_code status; + grpc_call_error error; + cq_verifier *cqv; + char *details = NULL; + size_t details_capacity = 0; + + state.cq = grpc_completion_queue_create(NULL); + cqv = cq_verifier_create(state.cq); + gpr_join_host_port(&state.target, "127.0.0.1", target_port); + state.channel = grpc_insecure_channel_create(state.target, NULL, NULL); + state.call = grpc_channel_create_call( + state.channel, NULL, GRPC_PROPAGATE_DEFAULTS, state.cq, "/Service/Method", + "localhost", gpr_inf_future(GPR_CLOCK_REALTIME), NULL); + + grpc_metadata_array_init(&initial_metadata_recv); + grpc_metadata_array_init(&trailing_metadata_recv); + + op = ops; + op->op = GRPC_OP_SEND_INITIAL_METADATA; + op->data.send_initial_metadata.count = 0; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_SEND_CLOSE_FROM_CLIENT; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_INITIAL_METADATA; + op->data.recv_initial_metadata = &initial_metadata_recv; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_STATUS_ON_CLIENT; + op->data.recv_status_on_client.trailing_metadata = &trailing_metadata_recv; + op->data.recv_status_on_client.status = &status; + op->data.recv_status_on_client.status_details = &details; + op->data.recv_status_on_client.status_details_capacity = &details_capacity; + op->flags = 0; + op->reserved = NULL; + op++; + error = + grpc_call_start_batch(state.call, ops, (size_t)(op - ops), tag(1), NULL); + + GPR_ASSERT(GRPC_CALL_OK == error); + + cq_expect_completion(cqv, tag(1), 1); + cq_verify(cqv); + + gpr_log(GPR_DEBUG, "Rpc status: %d, details: %s", status, details); + GPR_ASSERT(status == expected_status); + GPR_ASSERT(0 == strcmp(details, expected_detail)); + gpr_free(details); + cq_verifier_destroy(cqv); +} + +static void cleanup_rpc(void) { + grpc_event ev; + gpr_slice_buffer_destroy(&state.temp_incoming_buffer); + gpr_slice_buffer_destroy(&state.outgoing_buffer); + grpc_call_destroy(state.call); + grpc_completion_queue_shutdown(state.cq); + do { + ev = grpc_completion_queue_next(state.cq, n_sec_deadline(1), NULL); + } while (ev.type != GRPC_QUEUE_SHUTDOWN); + grpc_completion_queue_destroy(state.cq); + grpc_channel_destroy(state.channel); + gpr_free(state.target); +} + +typedef struct { + test_tcp_server *server; + gpr_event *signal_when_done; +} poll_args; + +static void actually_poll_server(void *arg) { + poll_args *pa = arg; + gpr_timespec deadline = n_sec_deadline(10); + while (true) { + bool done = gpr_atm_acq_load(&state.done_atm) != 0; + gpr_timespec time_left = + gpr_time_sub(deadline, gpr_now(GPR_CLOCK_REALTIME)); + gpr_log(GPR_DEBUG, "done=%d, time_left=%d.%09d", done, time_left.tv_sec, + time_left.tv_nsec); + if (done || gpr_time_cmp(time_left, gpr_time_0(GPR_TIMESPAN)) < 0) { + break; + } + test_tcp_server_poll(pa->server, 1); + } + gpr_event_set(pa->signal_when_done, (void *)1); + gpr_free(pa); +} + +static void poll_server_until_read_done(test_tcp_server *server, + gpr_event *signal_when_done) { + gpr_atm_rel_store(&state.done_atm, 0); + state.write_done = 0; + gpr_thd_id id; + poll_args *pa = gpr_malloc(sizeof(*pa)); + pa->server = server; + pa->signal_when_done = signal_when_done; + gpr_thd_new(&id, actually_poll_server, pa, NULL); +} + +static void run_test(const char *response_payload, + size_t response_payload_length, + grpc_status_code expected_status, + const char *expected_detail) { + test_tcp_server test_server; + server_port = grpc_pick_unused_port_or_die(); + grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; + gpr_event ev; + gpr_event_init(&ev); + + test_tcp_server_init(&test_server, on_connect, &test_server); + test_tcp_server_start(&test_server, server_port); + state.response_payload = response_payload; + state.response_payload_length = response_payload_length; + + /* poll server until sending out the response */ + poll_server_until_read_done(&test_server, &ev); + start_rpc(server_port, expected_status, expected_detail); + gpr_event_wait(&ev, gpr_inf_future(GPR_CLOCK_REALTIME)); + + /* clean up */ + grpc_endpoint_shutdown(&exec_ctx, state.tcp); + grpc_endpoint_destroy(&exec_ctx, state.tcp); + grpc_exec_ctx_finish(&exec_ctx); + cleanup_rpc(); + test_tcp_server_destroy(&test_server); +} + +int main(int argc, char **argv) { + grpc_test_init(argc, argv); + grpc_init(); + + /* status defined in hpack static table */ + run_test(HTTP2_RESP(204), sizeof(HTTP2_RESP(204)) - 1, GRPC_STATUS_CANCELLED, + HTTP2_DETAIL_MSG(204)); + + run_test(HTTP2_RESP(206), sizeof(HTTP2_RESP(206)) - 1, GRPC_STATUS_CANCELLED, + HTTP2_DETAIL_MSG(206)); + + run_test(HTTP2_RESP(304), sizeof(HTTP2_RESP(304)) - 1, GRPC_STATUS_CANCELLED, + HTTP2_DETAIL_MSG(304)); + + run_test(HTTP2_RESP(400), sizeof(HTTP2_RESP(400)) - 1, GRPC_STATUS_CANCELLED, + HTTP2_DETAIL_MSG(400)); + + run_test(HTTP2_RESP(404), sizeof(HTTP2_RESP(404)) - 1, GRPC_STATUS_CANCELLED, + HTTP2_DETAIL_MSG(404)); + + run_test(HTTP2_RESP(500), sizeof(HTTP2_RESP(500)) - 1, GRPC_STATUS_CANCELLED, + HTTP2_DETAIL_MSG(500)); + + /* status not defined in hpack static table */ + run_test(HTTP2_RESP(401), sizeof(HTTP2_RESP(401)) - 1, GRPC_STATUS_CANCELLED, + HTTP2_DETAIL_MSG(401)); + + run_test(HTTP2_RESP(403), sizeof(HTTP2_RESP(403)) - 1, GRPC_STATUS_CANCELLED, + HTTP2_DETAIL_MSG(403)); + + run_test(HTTP2_RESP(502), sizeof(HTTP2_RESP(502)) - 1, GRPC_STATUS_CANCELLED, + HTTP2_DETAIL_MSG(502)); + + /* unparseable response */ + run_test(UNPARSEABLE_RESP, sizeof(UNPARSEABLE_RESP) - 1, + GRPC_STATUS_UNAVAILABLE, UNPARSEABLE_DETAIL_MSG); + + /* http1 response */ + run_test(HTTP1_RESP, sizeof(HTTP1_RESP) - 1, GRPC_STATUS_UNAVAILABLE, + HTTP1_DETAIL_MSG); + + grpc_shutdown(); + return 0; +} diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 4cd5a816fb2..c1f98ddfc39 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -79,6 +79,23 @@ "third_party": false, "type": "target" }, + { + "deps": [ + "gpr", + "gpr_test_util", + "grpc", + "grpc_test_util", + "test_tcp_server" + ], + "headers": [], + "language": "c", + "name": "bad_server_response_test", + "src": [ + "test/core/end2end/bad_server_response_test.c" + ], + "third_party": false, + "type": "target" + }, { "deps": [ "grpc", diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index 2304ce72df8..08259ed5e7c 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -85,6 +85,27 @@ "windows" ] }, + { + "args": [], + "ci_platforms": [ + "linux", + "mac", + "posix", + "windows" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "gtest": false, + "language": "c", + "name": "bad_server_response_test", + "platforms": [ + "linux", + "mac", + "posix", + "windows" + ] + }, { "args": [], "ci_platforms": [ diff --git a/vsprojects/buildtests_c.sln b/vsprojects/buildtests_c.sln index ca43dad1e68..76cf9a59ab7 100644 --- a/vsprojects/buildtests_c.sln +++ b/vsprojects/buildtests_c.sln @@ -56,6 +56,18 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "bad_client_test", "vcxproj\ {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "bad_server_response_test", "vcxproj\test\bad_server_response_test\bad_server_response_test.vcxproj", "{2B73DA77-EF66-362C-24AD-317E3B8B28C1}" + ProjectSection(myProperties) = preProject + lib = "False" + EndProjectSection + ProjectSection(ProjectDependencies) = postProject + {E3110C46-A148-FF65-08FD-3324829BE7FE} = {E3110C46-A148-FF65-08FD-3324829BE7FE} + {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} + {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} + {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + EndProjectSection +EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "badreq_bad_client_test", "vcxproj\test\badreq_bad_client_test\badreq_bad_client_test.vcxproj", "{8A811C28-E04E-A444-E4C1-7588DF5B90AE}" ProjectSection(myProperties) = preProject lib = "False" @@ -1502,6 +1514,22 @@ Global {BA67B418-B699-E41A-9CC4-0279C49481A5}.Release-DLL|Win32.Build.0 = Release|Win32 {BA67B418-B699-E41A-9CC4-0279C49481A5}.Release-DLL|x64.ActiveCfg = Release|x64 {BA67B418-B699-E41A-9CC4-0279C49481A5}.Release-DLL|x64.Build.0 = Release|x64 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Debug|Win32.ActiveCfg = Debug|Win32 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Debug|x64.ActiveCfg = Debug|x64 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Release|Win32.ActiveCfg = Release|Win32 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Release|x64.ActiveCfg = Release|x64 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Debug|Win32.Build.0 = Debug|Win32 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Debug|x64.Build.0 = Debug|x64 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Release|Win32.Build.0 = Release|Win32 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Release|x64.Build.0 = Release|x64 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Debug-DLL|Win32.ActiveCfg = Debug|Win32 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Debug-DLL|Win32.Build.0 = Debug|Win32 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Debug-DLL|x64.ActiveCfg = Debug|x64 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Debug-DLL|x64.Build.0 = Debug|x64 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Release-DLL|Win32.ActiveCfg = Release|Win32 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Release-DLL|Win32.Build.0 = Release|Win32 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Release-DLL|x64.ActiveCfg = Release|x64 + {2B73DA77-EF66-362C-24AD-317E3B8B28C1}.Release-DLL|x64.Build.0 = Release|x64 {8A811C28-E04E-A444-E4C1-7588DF5B90AE}.Debug|Win32.ActiveCfg = Debug|Win32 {8A811C28-E04E-A444-E4C1-7588DF5B90AE}.Debug|x64.ActiveCfg = Debug|x64 {8A811C28-E04E-A444-E4C1-7588DF5B90AE}.Release|Win32.ActiveCfg = Release|Win32 diff --git a/vsprojects/vcxproj/test/bad_server_response_test/bad_server_response_test.vcxproj b/vsprojects/vcxproj/test/bad_server_response_test/bad_server_response_test.vcxproj new file mode 100644 index 00000000000..4676f3f6b6d --- /dev/null +++ b/vsprojects/vcxproj/test/bad_server_response_test/bad_server_response_test.vcxproj @@ -0,0 +1,202 @@ + + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {2B73DA77-EF66-362C-24AD-317E3B8B28C1} + true + $(SolutionDir)IntDir\$(MSBuildProjectName)\ + + + + v100 + + + v110 + + + v120 + + + v140 + + + Application + true + Unicode + + + Application + false + true + Unicode + + + + + + + + + + + + + + bad_server_response_test + static + Debug + static + Debug + + + bad_server_response_test + static + Release + static + Release + + + + NotUsing + Level3 + Disabled + WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) + true + MultiThreadedDebug + true + None + false + + + Console + true + false + + + + + + NotUsing + Level3 + Disabled + WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) + true + MultiThreadedDebug + true + None + false + + + Console + true + false + + + + + + NotUsing + Level3 + MaxSpeed + WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) + true + true + true + MultiThreaded + true + None + false + + + Console + true + false + true + true + + + + + + NotUsing + Level3 + MaxSpeed + WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) + true + true + true + MultiThreaded + true + None + false + + + Console + true + false + true + true + + + + + + + + + + {E3110C46-A148-FF65-08FD-3324829BE7FE} + + + {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} + + + {29D16885-7228-4C31-81ED-5F9187C7F2A9} + + + {EAB0A629-17A9-44DB-B5FF-E91A721FE037} + + + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + diff --git a/vsprojects/vcxproj/test/bad_server_response_test/bad_server_response_test.vcxproj.filters b/vsprojects/vcxproj/test/bad_server_response_test/bad_server_response_test.vcxproj.filters new file mode 100644 index 00000000000..13b11ec9472 --- /dev/null +++ b/vsprojects/vcxproj/test/bad_server_response_test/bad_server_response_test.vcxproj.filters @@ -0,0 +1,21 @@ + + + + + test\core\end2end + + + + + + {d29396a6-e5cf-3f1f-a33d-d1e9f2fa1b38} + + + {332f26c8-dd3f-091d-9e10-5b704377e991} + + + {158709cc-74ed-274f-fe50-b8e64cc9830e} + + + + From b187895fc43a84d75b64aa58d21d5caf9b80ba61 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Mon, 20 Jun 2016 09:11:54 -0700 Subject: [PATCH 0125/1039] Fix windows build error --- src/core/lib/iomgr/tcp_server_windows.c | 1 + 1 file changed, 1 insertion(+) diff --git a/src/core/lib/iomgr/tcp_server_windows.c b/src/core/lib/iomgr/tcp_server_windows.c index 2a51671ec71..4604f56dac7 100644 --- a/src/core/lib/iomgr/tcp_server_windows.c +++ b/src/core/lib/iomgr/tcp_server_windows.c @@ -103,6 +103,7 @@ struct grpc_tcp_server { /* Public function. Allocates the proper data structures to hold a grpc_tcp_server. */ grpc_error *grpc_tcp_server_create(grpc_closure *shutdown_complete, + const grpc_channel_args *args, grpc_tcp_server **server) { grpc_tcp_server *s = gpr_malloc(sizeof(grpc_tcp_server)); gpr_ref_init(&s->refs, 1); From 4fd2134dafc61af764b18dacee0acd8ef307dcc4 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 20 Jun 2016 09:23:55 -0700 Subject: [PATCH 0126/1039] Make another .h file accessible from C++. --- src/core/lib/security/context/security_context.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/core/lib/security/context/security_context.h b/src/core/lib/security/context/security_context.h index ef0c06b1fb6..4e7666dfe31 100644 --- a/src/core/lib/security/context/security_context.h +++ b/src/core/lib/security/context/security_context.h @@ -37,6 +37,10 @@ #include "src/core/lib/iomgr/pollset.h" #include "src/core/lib/security/credentials/credentials.h" +#ifdef __cplusplus +extern "C" { +#endif + /* --- grpc_auth_context --- High level authentication context object. Can optionally be chained. */ @@ -111,4 +115,8 @@ grpc_auth_context *grpc_auth_context_from_arg(const grpc_arg *arg); grpc_auth_context *grpc_find_auth_context_in_args( const grpc_channel_args *args); +#ifdef __cplusplus +} +#endif + #endif /* GRPC_CORE_LIB_SECURITY_CONTEXT_SECURITY_CONTEXT_H */ From 04e47f308ec57c42ec9a0dcdc25e72c03b66bc9e Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Mon, 20 Jun 2016 11:03:15 -0700 Subject: [PATCH 0127/1039] Direct additional log statement through custom logger --- src/node/src/server.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/node/src/server.js b/src/node/src/server.js index fd5153f1244..b3b414969ab 100644 --- a/src/node/src/server.js +++ b/src/node/src/server.js @@ -735,8 +735,8 @@ Server.prototype.addService = function(service, implementation) { } var impl; if (implementation[name] === undefined) { - console.warn('Method handler for %s expected but not provided', - attrs.path); + common.log(grpc.logVerbosity.ERROR, 'Method handler for ' + + attrs.path + ' expected but not provided'); impl = defaultHandler[method_type]; } else { impl = _.bind(implementation[name], implementation); From 3da684dcacac1b5d523897b76a00d5e62e1163bd Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 20 Jun 2016 13:27:12 -0700 Subject: [PATCH 0128/1039] Add "final" to ChannelFilter class, so that compiler can un-indirect the filter method calls. --- include/grpc++/channel_filter.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/grpc++/channel_filter.h b/include/grpc++/channel_filter.h index 53e42a7cb81..262739929a0 100644 --- a/include/grpc++/channel_filter.h +++ b/include/grpc++/channel_filter.h @@ -35,6 +35,7 @@ #define GRPCXX_CHANNEL_FILTER_H #include +#include #include #include @@ -94,7 +95,7 @@ namespace internal { // Defines static members for passing to C core. template -class ChannelFilter { +class ChannelFilter GRPC_FINAL { public: static const size_t channel_data_size = sizeof(ChannelDataType); From 083afc225d298db97b37c28034dad1d024bfb0f5 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 20 Jun 2016 13:33:11 -0700 Subject: [PATCH 0129/1039] Add comments indicating what the grpc_call_context_element values are for each array index. --- src/core/lib/channel/context.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/src/core/lib/channel/context.h b/src/core/lib/channel/context.h index c50e84279d4..22f4cb62f3b 100644 --- a/src/core/lib/channel/context.h +++ b/src/core/lib/channel/context.h @@ -34,10 +34,19 @@ #ifndef GRPC_CORE_LIB_CHANNEL_CONTEXT_H #define GRPC_CORE_LIB_CHANNEL_CONTEXT_H -/* Call object context pointers */ +// Call object context pointers. + +// Call context is represented as an array of grpc_call_context_elements. +// This enum represents the indexes into the array, where each index +// contains a different type of value. typedef enum { + // Value is either a grpc_client_security_context or a + // grpc_server_security_context. GRPC_CONTEXT_SECURITY = 0, + + // Value is a census_context. GRPC_CONTEXT_TRACING, + GRPC_CONTEXT_COUNT } grpc_context_index; From 36a58a7928e6a10b44b6cffdc118337ef7e0b8c1 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Mon, 20 Jun 2016 14:01:07 -0700 Subject: [PATCH 0130/1039] Enable treating warnings as errors for target gRPC --- .../GRPCClient/GRPCCall+ChannelCredentials.h | 4 ++-- src/objective-c/ProtoRPC/ProtoMethod.h | 3 +++ src/objective-c/ProtoRPC/ProtoRPC.h | 3 +++ src/objective-c/ProtoRPC/ProtoService.h | 3 +++ src/objective-c/ProtoRPC/ProtoService.m | 6 +++--- src/objective-c/tests/Podfile | 21 ++++++++++++++++--- 6 files changed, 32 insertions(+), 8 deletions(-) diff --git a/src/objective-c/GRPCClient/GRPCCall+ChannelCredentials.h b/src/objective-c/GRPCClient/GRPCCall+ChannelCredentials.h index 343dd48a14e..ac2a37d75f2 100644 --- a/src/objective-c/GRPCClient/GRPCCall+ChannelCredentials.h +++ b/src/objective-c/GRPCClient/GRPCCall+ChannelCredentials.h @@ -41,7 +41,7 @@ */ + (BOOL)setTLSPEMRootCerts:(nullable NSString *)pemRootCert forHost:(nonnull NSString *)host - error:(NSError **)errorPtr; + error:(NSError * _Nullable * _Nullable)errorPtr; /** * Configures @c host with TLS/SSL Client Credentials and optionally trusted root Certificate * Authorities. If @c pemRootCerts is nil, the default CA Certificates bundled with gRPC will be @@ -51,6 +51,6 @@ withPrivateKey:(nullable NSString *)pemPrivateKey withCertChain:(nullable NSString *)pemCertChain forHost:(nonnull NSString *)host - error:(NSError **)errorPtr; + error:(NSError * _Nullable * _Nullable)errorPtr; @end diff --git a/src/objective-c/ProtoRPC/ProtoMethod.h b/src/objective-c/ProtoRPC/ProtoMethod.h index f9fdbb35ffd..ae3a2723fc4 100644 --- a/src/objective-c/ProtoRPC/ProtoMethod.h +++ b/src/objective-c/ProtoRPC/ProtoMethod.h @@ -54,6 +54,9 @@ __attribute__((deprecated("Please use GRPCProtoMethod."))) * This subclass is empty now. Eventually we'll remove ProtoMethod class * to avoid potential naming conflict */ +#pragma clang diagnostic push +#pragma clang diagnostic ignored "-Wdeprecated-declarations" @interface GRPCProtoMethod : ProtoMethod +#pragma clang diagnostic pop @end diff --git a/src/objective-c/ProtoRPC/ProtoRPC.h b/src/objective-c/ProtoRPC/ProtoRPC.h index 5f91f6bce1e..04fc1e45f16 100644 --- a/src/objective-c/ProtoRPC/ProtoRPC.h +++ b/src/objective-c/ProtoRPC/ProtoRPC.h @@ -56,6 +56,9 @@ __attribute__((deprecated("Please use GRPCProtoCall."))) * This subclass is empty now. Eventually we'll remove ProtoRPC class * to avoid potential naming conflict */ +#pragma clang diagnostic push +#pragma clang diagnostic ignored "-Wdeprecated-declarations" @interface GRPCProtoCall : ProtoRPC +#pragma clang diagnostic pop @end diff --git a/src/objective-c/ProtoRPC/ProtoService.h b/src/objective-c/ProtoRPC/ProtoService.h index 87d06e1ae59..7faae1b49c9 100644 --- a/src/objective-c/ProtoRPC/ProtoService.h +++ b/src/objective-c/ProtoRPC/ProtoService.h @@ -55,6 +55,9 @@ __attribute__((deprecated("Please use GRPCProtoService."))) * This subclass is empty now. Eventually we'll remove ProtoService class * to avoid potential naming conflict */ +#pragma clang diagnostic push +#pragma clang diagnostic ignored "-Wdeprecated-declarations" @interface GRPCProtoService : ProtoService +#pragma clang diagnostic pop @end diff --git a/src/objective-c/ProtoRPC/ProtoService.m b/src/objective-c/ProtoRPC/ProtoService.m index 597c3cf0fed..4a14570d818 100644 --- a/src/objective-c/ProtoRPC/ProtoService.m +++ b/src/objective-c/ProtoRPC/ProtoService.m @@ -65,14 +65,14 @@ return self; } -- (ProtoRPC *)RPCToMethod:(NSString *)method +- (GRPCProtoCall *)RPCToMethod:(NSString *)method requestsWriter:(GRXWriter *)requestsWriter responseClass:(Class)responseClass responsesWriteable:(id)responsesWriteable { - ProtoMethod *methodName = [[ProtoMethod alloc] initWithPackage:_packageName + GRPCProtoMethod *methodName = [[GRPCProtoMethod alloc] initWithPackage:_packageName service:_serviceName method:method]; - return [[ProtoRPC alloc] initWithHost:_host + return [[GRPCProtoCall alloc] initWithHost:_host method:methodName requestsWriter:requestsWriter responseClass:responseClass diff --git a/src/objective-c/tests/Podfile b/src/objective-c/tests/Podfile index 508641d681a..53edf8c890a 100644 --- a/src/objective-c/tests/Podfile +++ b/src/objective-c/tests/Podfile @@ -1,9 +1,9 @@ source 'https://github.com/CocoaPods/Specs.git' platform :ios, '8.0' -pod 'Protobuf', :path => "../../../third_party/protobuf" -pod 'BoringSSL', :podspec => ".." -pod 'CronetFramework', :podspec => ".." +pod 'Protobuf', :path => "../../../third_party/protobuf", :inhibit_warnings => true +pod 'BoringSSL', :podspec => "..", :inhibit_warnings => true +pod 'CronetFramework', :podspec => "..", :inhibit_warnings => true pod 'gRPC', :path => "../../.." pod 'RemoteTest', :path => "RemoteTestClient" @@ -30,3 +30,18 @@ end target 'InteropTestsLocalCleartext' do end + +post_install do |installer| + installer.pods_project.targets.each do |target| + target.build_configurations.each do |config| + config.build_settings['GCC_TREAT_WARNINGS_AS_ERRORS'] = 'YES' + end + if target.name == 'gRPC' + target.build_configurations.each do |config| + # GPR_UNREACHABLE_CODE causes "Control may reach end of non-void + # function" warning + config.build_settings['GCC_WARN_ABOUT_RETURN_TYPE'] = 'NO' + end + end + end +end From abc7427a4d113aeb7fde6d732c63ffea1f615998 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Mon, 20 Jun 2016 14:40:57 -0700 Subject: [PATCH 0131/1039] Update podfiles for objc examples --- examples/objective-c/auth_sample/AuthTestService.podspec | 4 ++++ examples/objective-c/helloworld/HelloWorld.podspec | 4 ++++ examples/objective-c/route_guide/RouteGuide.podspec | 4 ++++ src/objective-c/examples/RemoteTestClient/RemoteTest.podspec | 4 ++++ 4 files changed, 16 insertions(+) diff --git a/examples/objective-c/auth_sample/AuthTestService.podspec b/examples/objective-c/auth_sample/AuthTestService.podspec index e9356260534..d246653ea79 100644 --- a/examples/objective-c/auth_sample/AuthTestService.podspec +++ b/examples/objective-c/auth_sample/AuthTestService.podspec @@ -2,6 +2,10 @@ Pod::Spec.new do |s| s.name = "AuthTestService" s.version = "0.0.1" s.license = "New BSD" + s.authors = { 'gRPC contributors' => 'grpc-io@googlegroups.com' } + s.homepage = "http://www.grpc.io/" + s.summary = "AuthTestService example" + s.source = { :git => 'https://github.com/grpc/grpc.git' } s.ios.deployment_target = "7.1" s.osx.deployment_target = "10.9" diff --git a/examples/objective-c/helloworld/HelloWorld.podspec b/examples/objective-c/helloworld/HelloWorld.podspec index bdf782f6eaf..17b016b31a1 100644 --- a/examples/objective-c/helloworld/HelloWorld.podspec +++ b/examples/objective-c/helloworld/HelloWorld.podspec @@ -2,6 +2,10 @@ Pod::Spec.new do |s| s.name = "HelloWorld" s.version = "0.0.1" s.license = "New BSD" + s.authors = { 'gRPC contributors' => 'grpc-io@googlegroups.com' } + s.homepage = "http://www.grpc.io/" + s.summary = "HelloWorld example" + s.source = { :git => 'https://github.com/grpc/grpc.git' } s.ios.deployment_target = "7.1" s.osx.deployment_target = "10.9" diff --git a/examples/objective-c/route_guide/RouteGuide.podspec b/examples/objective-c/route_guide/RouteGuide.podspec index 4bc2c42cbb8..97a61ff51a5 100644 --- a/examples/objective-c/route_guide/RouteGuide.podspec +++ b/examples/objective-c/route_guide/RouteGuide.podspec @@ -2,6 +2,10 @@ Pod::Spec.new do |s| s.name = "RouteGuide" s.version = "0.0.1" s.license = "New BSD" + s.authors = { 'gRPC contributors' => 'grpc-io@googlegroups.com' } + s.homepage = "http://www.grpc.io/" + s.summary = "RouteGuide example" + s.source = { :git => 'https://github.com/grpc/grpc.git' } s.ios.deployment_target = "7.1" s.osx.deployment_target = "10.9" diff --git a/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec b/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec index 5addf26fc46..107e6de4e21 100644 --- a/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec +++ b/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec @@ -2,6 +2,10 @@ Pod::Spec.new do |s| s.name = "RemoteTest" s.version = "0.0.1" s.license = "New BSD" + s.authors = { 'gRPC contributors' => 'grpc-io@googlegroups.com' } + s.homepage = "http://www.grpc.io/" + s.summary = "RemoteTest example" + s.source = { :git => 'https://github.com/grpc/grpc.git' } s.ios.deployment_target = '7.1' s.osx.deployment_target = '10.9' From 7981e1905f61471f8caece6eea6051b2522b9235 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Mon, 20 Jun 2016 15:38:15 -0700 Subject: [PATCH 0132/1039] Network status tracking Recreated the old PR (https://github.com/grpc/grpc/pull/6283) in a new repo since the old repo was destroyed. Removed changes to tcp_posix.c and tpc_windows.c, instead depending on the idempotent endpoint shutdown code from https://github.com/grpc/grpc/pull/6911. --- src/core/lib/iomgr/network_status_tracker.c | 82 ++++++ src/core/lib/iomgr/network_status_tracker.h | 40 +++ .../end2end/tests/network_status_change.c | 243 ++++++++++++++++++ 3 files changed, 365 insertions(+) create mode 100644 src/core/lib/iomgr/network_status_tracker.c create mode 100644 src/core/lib/iomgr/network_status_tracker.h create mode 100644 test/core/end2end/tests/network_status_change.c diff --git a/src/core/lib/iomgr/network_status_tracker.c b/src/core/lib/iomgr/network_status_tracker.c new file mode 100644 index 00000000000..6e432368cf1 --- /dev/null +++ b/src/core/lib/iomgr/network_status_tracker.c @@ -0,0 +1,82 @@ +/* + * + * Copyright 2015, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include "src/core/lib/iomgr/endpoint.h" +#include + +typedef struct endpoint_ll_node { + grpc_endpoint *ep; + struct endpoint_ll_node *next; +} endpoint_ll_node; + +static endpoint_ll_node *head = NULL; + +// TODO(makarandd): Install callback with OS to monitor network status. +void grpc_initialize_network_status_monitor() { +} + +void grpc_destroy_network_status_monitor() { + for (endpoint_ll_node *curr = head; curr != NULL; ) { + endpoint_ll_node *next = curr->next; + gpr_free(curr); + curr = next; + } +} + +void grpc_network_status_register_endpoint(grpc_endpoint *ep) { + if (head == NULL) { + head = (endpoint_ll_node *)gpr_malloc(sizeof(endpoint_ll_node)); + head->ep = ep; + head->next = NULL; + } else { + endpoint_ll_node *prev_head = head; + head = (endpoint_ll_node *)gpr_malloc(sizeof(endpoint_ll_node)); + head->ep = ep; + head->next = prev_head; + } +} + +// Walk the linked-list from head and execute shutdown. It is possible that +// other threads might be in the process of shutdown as well, but that has +// no side effect. +void grpc_network_status_shutdown_all_endpoints() { + if (head == NULL) { + return; + } + grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; + + for (endpoint_ll_node *curr = head; curr != NULL; curr = curr->next) { + curr->ep->vtable->shutdown(&exec_ctx, curr->ep); + } + grpc_exec_ctx_finish(&exec_ctx); +} diff --git a/src/core/lib/iomgr/network_status_tracker.h b/src/core/lib/iomgr/network_status_tracker.h new file mode 100644 index 00000000000..4154151927f --- /dev/null +++ b/src/core/lib/iomgr/network_status_tracker.h @@ -0,0 +1,40 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#ifndef GRPC_CORE_LIB_IOMGR_NETWORK_STATUS_TRACKER_H +#define GRPC_CORE_LIB_IOMGR_NETWORK_STATUS_TRACKER_H +#include "src/core/lib/iomgr/endpoint.h" + +void grpc_network_status_register_endpoint(grpc_endpoint *ep); +void grpc_network_status_shutdown_all_endpoints(); +#endif /* GRPC_CORE_LIB_IOMGR_NETWORK_STATUS_TRACKER_H */ diff --git a/test/core/end2end/tests/network_status_change.c b/test/core/end2end/tests/network_status_change.c new file mode 100644 index 00000000000..bbc85007022 --- /dev/null +++ b/test/core/end2end/tests/network_status_change.c @@ -0,0 +1,243 @@ +/* + * + * Copyright 2015, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include "test/core/end2end/end2end_tests.h" + +#include +#include + +#include +#include +#include +#include +#include +#include "test/core/end2end/cq_verifier.h" + +/* this is a private API but exposed here for testing*/ +extern void grpc_network_status_shutdown_all_endpoints(); + +enum { TIMEOUT = 200000 }; + +static void *tag(intptr_t t) { return (void *)t; } + +static grpc_end2end_test_fixture begin_test(grpc_end2end_test_config config, + const char *test_name, + grpc_channel_args *client_args, + grpc_channel_args *server_args) { + grpc_end2end_test_fixture f; + gpr_log(GPR_INFO, "%s/%s", test_name, config.name); + f = config.create_fixture(client_args, server_args); + config.init_server(&f, server_args); + config.init_client(&f, client_args); + return f; +} + +static gpr_timespec n_seconds_time(int n) { + return GRPC_TIMEOUT_SECONDS_TO_DEADLINE(n); +} + +static gpr_timespec five_seconds_time(void) { return n_seconds_time(500); } + +static void drain_cq(grpc_completion_queue *cq) { + grpc_event ev; + do { + ev = grpc_completion_queue_next(cq, five_seconds_time(), NULL); + } while (ev.type != GRPC_QUEUE_SHUTDOWN); +} + +static void shutdown_server(grpc_end2end_test_fixture *f) { + if (!f->server) return; + grpc_server_shutdown_and_notify(f->server, f->cq, tag(1000)); + GPR_ASSERT(grpc_completion_queue_pluck( + f->cq, tag(1000), GRPC_TIMEOUT_SECONDS_TO_DEADLINE(5), NULL) + .type == GRPC_OP_COMPLETE); + grpc_server_destroy(f->server); + f->server = NULL; +} + +static void shutdown_client(grpc_end2end_test_fixture *f) { + if (!f->client) return; + grpc_channel_destroy(f->client); + f->client = NULL; +} + +static void end_test(grpc_end2end_test_fixture *f) { + shutdown_server(f); + shutdown_client(f); + + grpc_completion_queue_shutdown(f->cq); + drain_cq(f->cq); + grpc_completion_queue_destroy(f->cq); +} + +/* Client sends a request with payload, server reads then returns status. */ +static void test_invoke_network_status_change(grpc_end2end_test_config config) { + grpc_call *c; + grpc_call *s; + gpr_slice request_payload_slice = gpr_slice_from_copied_string("hello world"); + grpc_byte_buffer *request_payload = + grpc_raw_byte_buffer_create(&request_payload_slice, 1); + gpr_timespec deadline = five_seconds_time(); + grpc_end2end_test_fixture f = + begin_test(config, "test_invoke_request_with_payload", NULL, NULL); + cq_verifier *cqv = cq_verifier_create(f.cq); + grpc_op ops[6]; + grpc_op *op; + grpc_metadata_array initial_metadata_recv; + grpc_metadata_array trailing_metadata_recv; + grpc_metadata_array request_metadata_recv; + grpc_byte_buffer *request_payload_recv = NULL; + grpc_call_details call_details; + grpc_status_code status; + grpc_call_error error; + char *details = NULL; + size_t details_capacity = 0; + int was_cancelled = 2; + + c = grpc_channel_create_call(f.client, NULL, GRPC_PROPAGATE_DEFAULTS, f.cq, + "/foo", "foo.test.google.fr", deadline, NULL); + GPR_ASSERT(c); + + grpc_metadata_array_init(&initial_metadata_recv); + grpc_metadata_array_init(&trailing_metadata_recv); + grpc_metadata_array_init(&request_metadata_recv); + grpc_call_details_init(&call_details); + + memset(ops, 0, sizeof(ops)); + op = ops; + op->op = GRPC_OP_SEND_INITIAL_METADATA; + op->data.send_initial_metadata.count = 0; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_SEND_MESSAGE; + op->data.send_message = request_payload; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_SEND_CLOSE_FROM_CLIENT; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_INITIAL_METADATA; + op->data.recv_initial_metadata = &initial_metadata_recv; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_STATUS_ON_CLIENT; + op->data.recv_status_on_client.trailing_metadata = &trailing_metadata_recv; + op->data.recv_status_on_client.status = &status; + op->data.recv_status_on_client.status_details = &details; + op->data.recv_status_on_client.status_details_capacity = &details_capacity; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(c, ops, (size_t)(op - ops), tag(1), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + + GPR_ASSERT(GRPC_CALL_OK == grpc_server_request_call( + f.server, &s, &call_details, + &request_metadata_recv, f.cq, f.cq, tag(101))); + cq_expect_completion(cqv, tag(101), 1); + cq_verify(cqv); + + op = ops; + op->op = GRPC_OP_SEND_INITIAL_METADATA; + op->data.send_initial_metadata.count = 0; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_MESSAGE; + op->data.recv_message = &request_payload_recv; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(s, ops, (size_t)(op - ops), tag(102), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + + cq_expect_completion(cqv, tag(102), 1); + // Simulate the network loss event + grpc_network_status_shutdown_all_endpoints(); + cq_verify(cqv); + + op = ops; + op->op = GRPC_OP_RECV_CLOSE_ON_SERVER; + op->data.recv_close_on_server.cancelled = &was_cancelled; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_SEND_STATUS_FROM_SERVER; + op->data.send_status_from_server.trailing_metadata_count = 0; + op->data.send_status_from_server.status = GRPC_STATUS_OK; + op->data.send_status_from_server.status_details = "xyz"; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(s, ops, (size_t)(op - ops), tag(103), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + void shutdown_all_endpoints(); + cq_expect_completion(cqv, tag(103), 1); + cq_expect_completion(cqv, tag(1), 1); + cq_verify(cqv); + + // Expected behavior of a RPC when network is lost. + GPR_ASSERT(status == GRPC_STATUS_UNAVAILABLE); + GPR_ASSERT(0 == strcmp(details, "")); + GPR_ASSERT(0 == strcmp(call_details.method, "/foo")); + GPR_ASSERT(0 == strcmp(call_details.host, "foo.test.google.fr")); + GPR_ASSERT(was_cancelled == 0); + + + gpr_free(details); + grpc_metadata_array_destroy(&initial_metadata_recv); + grpc_metadata_array_destroy(&trailing_metadata_recv); + grpc_metadata_array_destroy(&request_metadata_recv); + grpc_call_details_destroy(&call_details); + + grpc_call_destroy(c); + grpc_call_destroy(s); + + cq_verifier_destroy(cqv); + + grpc_byte_buffer_destroy(request_payload); + grpc_byte_buffer_destroy(request_payload_recv); + + end_test(&f); + config.tear_down_data(&f); +} + +void network_status_change(grpc_end2end_test_config config) { + test_invoke_network_status_change(config); +} + +void network_status_change_pre_init(void) {} From 0579cfc334794a32e243e971f77e01b5fd8f3c55 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Mon, 20 Jun 2016 15:45:24 -0700 Subject: [PATCH 0133/1039] more files after running build.yaml changes through --- BUILD | 8 + Makefile | 5 + binding.gyp | 1 + build.yaml | 2 + config.m4 | 1 + gRPC.podspec | 3 + grpc.gemspec | 2 + package.xml | 2 + src/core/lib/iomgr/tcp_posix.c | 3 + src/core/lib/iomgr/tcp_windows.c | 4 + src/python/grpcio/grpc_core_dependencies.py | 1 + test/core/end2end/end2end_nosec_tests.c | 8 + test/core/end2end/end2end_tests.c | 8 + test/core/end2end/gen_build_yaml.py | 1 + tools/doxygen/Doxyfile.core.internal | 2 + tools/run_tests/sources_and_headers.json | 5 + tools/run_tests/tests.json | 627 +++++++++++++++++- vsprojects/vcxproj/grpc/grpc.vcxproj | 3 + vsprojects/vcxproj/grpc/grpc.vcxproj.filters | 6 + .../grpc_unsecure/grpc_unsecure.vcxproj | 3 + .../grpc_unsecure.vcxproj.filters | 6 + .../end2end_nosec_tests.vcxproj | 2 + .../end2end_nosec_tests.vcxproj.filters | 3 + .../tests/end2end_tests/end2end_tests.vcxproj | 2 + .../end2end_tests.vcxproj.filters | 3 + 25 files changed, 702 insertions(+), 9 deletions(-) diff --git a/BUILD b/BUILD index ac0eb8e403d..55c2ea87c65 100644 --- a/BUILD +++ b/BUILD @@ -187,6 +187,7 @@ cc_library( "src/core/lib/iomgr/iomgr_internal.h", "src/core/lib/iomgr/iomgr_posix.h", "src/core/lib/iomgr/load_file.h", + "src/core/lib/iomgr/network_status_tracker.h", "src/core/lib/iomgr/polling_entity.h", "src/core/lib/iomgr/pollset.h", "src/core/lib/iomgr/pollset_set.h", @@ -338,6 +339,7 @@ cc_library( "src/core/lib/iomgr/iomgr_posix.c", "src/core/lib/iomgr/iomgr_windows.c", "src/core/lib/iomgr/load_file.c", + "src/core/lib/iomgr/network_status_tracker.c", "src/core/lib/iomgr/polling_entity.c", "src/core/lib/iomgr/pollset_set_windows.c", "src/core/lib/iomgr/pollset_windows.c", @@ -571,6 +573,7 @@ cc_library( "src/core/lib/iomgr/iomgr_internal.h", "src/core/lib/iomgr/iomgr_posix.h", "src/core/lib/iomgr/load_file.h", + "src/core/lib/iomgr/network_status_tracker.h", "src/core/lib/iomgr/polling_entity.h", "src/core/lib/iomgr/pollset.h", "src/core/lib/iomgr/pollset_set.h", @@ -712,6 +715,7 @@ cc_library( "src/core/lib/iomgr/iomgr_posix.c", "src/core/lib/iomgr/iomgr_windows.c", "src/core/lib/iomgr/load_file.c", + "src/core/lib/iomgr/network_status_tracker.c", "src/core/lib/iomgr/polling_entity.c", "src/core/lib/iomgr/pollset_set_windows.c", "src/core/lib/iomgr/pollset_windows.c", @@ -920,6 +924,7 @@ cc_library( "src/core/lib/iomgr/iomgr_internal.h", "src/core/lib/iomgr/iomgr_posix.h", "src/core/lib/iomgr/load_file.h", + "src/core/lib/iomgr/network_status_tracker.h", "src/core/lib/iomgr/polling_entity.h", "src/core/lib/iomgr/pollset.h", "src/core/lib/iomgr/pollset_set.h", @@ -1048,6 +1053,7 @@ cc_library( "src/core/lib/iomgr/iomgr_posix.c", "src/core/lib/iomgr/iomgr_windows.c", "src/core/lib/iomgr/load_file.c", + "src/core/lib/iomgr/network_status_tracker.c", "src/core/lib/iomgr/polling_entity.c", "src/core/lib/iomgr/pollset_set_windows.c", "src/core/lib/iomgr/pollset_windows.c", @@ -1805,6 +1811,7 @@ objc_library( "src/core/lib/iomgr/iomgr_posix.c", "src/core/lib/iomgr/iomgr_windows.c", "src/core/lib/iomgr/load_file.c", + "src/core/lib/iomgr/network_status_tracker.c", "src/core/lib/iomgr/polling_entity.c", "src/core/lib/iomgr/pollset_set_windows.c", "src/core/lib/iomgr/pollset_windows.c", @@ -2017,6 +2024,7 @@ objc_library( "src/core/lib/iomgr/iomgr_internal.h", "src/core/lib/iomgr/iomgr_posix.h", "src/core/lib/iomgr/load_file.h", + "src/core/lib/iomgr/network_status_tracker.h", "src/core/lib/iomgr/polling_entity.h", "src/core/lib/iomgr/pollset.h", "src/core/lib/iomgr/pollset_set.h", diff --git a/Makefile b/Makefile index d93ce337632..961485b3b80 100644 --- a/Makefile +++ b/Makefile @@ -2574,6 +2574,7 @@ LIBGRPC_SRC = \ src/core/lib/iomgr/iomgr_posix.c \ src/core/lib/iomgr/iomgr_windows.c \ src/core/lib/iomgr/load_file.c \ + src/core/lib/iomgr/network_status_tracker.c \ src/core/lib/iomgr/polling_entity.c \ src/core/lib/iomgr/pollset_set_windows.c \ src/core/lib/iomgr/pollset_windows.c \ @@ -2844,6 +2845,7 @@ LIBGRPC_CRONET_SRC = \ src/core/lib/iomgr/iomgr_posix.c \ src/core/lib/iomgr/iomgr_windows.c \ src/core/lib/iomgr/load_file.c \ + src/core/lib/iomgr/network_status_tracker.c \ src/core/lib/iomgr/polling_entity.c \ src/core/lib/iomgr/pollset_set_windows.c \ src/core/lib/iomgr/pollset_windows.c \ @@ -3183,6 +3185,7 @@ LIBGRPC_UNSECURE_SRC = \ src/core/lib/iomgr/iomgr_posix.c \ src/core/lib/iomgr/iomgr_windows.c \ src/core/lib/iomgr/load_file.c \ + src/core/lib/iomgr/network_status_tracker.c \ src/core/lib/iomgr/polling_entity.c \ src/core/lib/iomgr/pollset_set_windows.c \ src/core/lib/iomgr/pollset_windows.c \ @@ -6376,6 +6379,7 @@ LIBEND2END_TESTS_SRC = \ test/core/end2end/tests/max_concurrent_streams.c \ test/core/end2end/tests/max_message_length.c \ test/core/end2end/tests/negative_deadline.c \ + test/core/end2end/tests/network_status_change.c \ test/core/end2end/tests/no_op.c \ test/core/end2end/tests/payload.c \ test/core/end2end/tests/ping.c \ @@ -6452,6 +6456,7 @@ LIBEND2END_NOSEC_TESTS_SRC = \ test/core/end2end/tests/max_concurrent_streams.c \ test/core/end2end/tests/max_message_length.c \ test/core/end2end/tests/negative_deadline.c \ + test/core/end2end/tests/network_status_change.c \ test/core/end2end/tests/no_op.c \ test/core/end2end/tests/payload.c \ test/core/end2end/tests/ping.c \ diff --git a/binding.gyp b/binding.gyp index 3150d1083b0..9a4b46141ff 100644 --- a/binding.gyp +++ b/binding.gyp @@ -591,6 +591,7 @@ 'src/core/lib/iomgr/iomgr_posix.c', 'src/core/lib/iomgr/iomgr_windows.c', 'src/core/lib/iomgr/load_file.c', + 'src/core/lib/iomgr/network_status_tracker.c', 'src/core/lib/iomgr/polling_entity.c', 'src/core/lib/iomgr/pollset_set_windows.c', 'src/core/lib/iomgr/pollset_windows.c', diff --git a/build.yaml b/build.yaml index d41f4195947..54dae5b6ac2 100644 --- a/build.yaml +++ b/build.yaml @@ -183,6 +183,7 @@ filegroups: - src/core/lib/iomgr/iomgr_internal.h - src/core/lib/iomgr/iomgr_posix.h - src/core/lib/iomgr/load_file.h + - src/core/lib/iomgr/network_status_tracker.h - src/core/lib/iomgr/polling_entity.h - src/core/lib/iomgr/pollset.h - src/core/lib/iomgr/pollset_set.h @@ -261,6 +262,7 @@ filegroups: - src/core/lib/iomgr/iomgr_posix.c - src/core/lib/iomgr/iomgr_windows.c - src/core/lib/iomgr/load_file.c + - src/core/lib/iomgr/network_status_tracker.c - src/core/lib/iomgr/polling_entity.c - src/core/lib/iomgr/pollset_set_windows.c - src/core/lib/iomgr/pollset_windows.c diff --git a/config.m4 b/config.m4 index 716eb9841fb..6cf8b3a33fd 100644 --- a/config.m4 +++ b/config.m4 @@ -110,6 +110,7 @@ if test "$PHP_GRPC" != "no"; then src/core/lib/iomgr/iomgr_posix.c \ src/core/lib/iomgr/iomgr_windows.c \ src/core/lib/iomgr/load_file.c \ + src/core/lib/iomgr/network_status_tracker.c \ src/core/lib/iomgr/polling_entity.c \ src/core/lib/iomgr/pollset_set_windows.c \ src/core/lib/iomgr/pollset_windows.c \ diff --git a/gRPC.podspec b/gRPC.podspec index ff8373637a6..48da2322e73 100644 --- a/gRPC.podspec +++ b/gRPC.podspec @@ -190,6 +190,7 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/iomgr_internal.h', 'src/core/lib/iomgr/iomgr_posix.h', 'src/core/lib/iomgr/load_file.h', + 'src/core/lib/iomgr/network_status_tracker.h', 'src/core/lib/iomgr/polling_entity.h', 'src/core/lib/iomgr/pollset.h', 'src/core/lib/iomgr/pollset_set.h', @@ -375,6 +376,7 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/iomgr_posix.c', 'src/core/lib/iomgr/iomgr_windows.c', 'src/core/lib/iomgr/load_file.c', + 'src/core/lib/iomgr/network_status_tracker.c', 'src/core/lib/iomgr/polling_entity.c', 'src/core/lib/iomgr/pollset_set_windows.c', 'src/core/lib/iomgr/pollset_windows.c', @@ -570,6 +572,7 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/iomgr_internal.h', 'src/core/lib/iomgr/iomgr_posix.h', 'src/core/lib/iomgr/load_file.h', + 'src/core/lib/iomgr/network_status_tracker.h', 'src/core/lib/iomgr/polling_entity.h', 'src/core/lib/iomgr/pollset.h', 'src/core/lib/iomgr/pollset_set.h', diff --git a/grpc.gemspec b/grpc.gemspec index 184a5485917..5796cd68bc2 100755 --- a/grpc.gemspec +++ b/grpc.gemspec @@ -199,6 +199,7 @@ Gem::Specification.new do |s| s.files += %w( src/core/lib/iomgr/iomgr_internal.h ) s.files += %w( src/core/lib/iomgr/iomgr_posix.h ) s.files += %w( src/core/lib/iomgr/load_file.h ) + s.files += %w( src/core/lib/iomgr/network_status_tracker.h ) s.files += %w( src/core/lib/iomgr/polling_entity.h ) s.files += %w( src/core/lib/iomgr/pollset.h ) s.files += %w( src/core/lib/iomgr/pollset_set.h ) @@ -354,6 +355,7 @@ Gem::Specification.new do |s| s.files += %w( src/core/lib/iomgr/iomgr_posix.c ) s.files += %w( src/core/lib/iomgr/iomgr_windows.c ) s.files += %w( src/core/lib/iomgr/load_file.c ) + s.files += %w( src/core/lib/iomgr/network_status_tracker.c ) s.files += %w( src/core/lib/iomgr/polling_entity.c ) s.files += %w( src/core/lib/iomgr/pollset_set_windows.c ) s.files += %w( src/core/lib/iomgr/pollset_windows.c ) diff --git a/package.xml b/package.xml index 67e9bb2c282..623174ad016 100644 --- a/package.xml +++ b/package.xml @@ -206,6 +206,7 @@ + @@ -361,6 +362,7 @@ + diff --git a/src/core/lib/iomgr/tcp_posix.c b/src/core/lib/iomgr/tcp_posix.c index 017f52c3677..f7818353b0a 100644 --- a/src/core/lib/iomgr/tcp_posix.c +++ b/src/core/lib/iomgr/tcp_posix.c @@ -35,6 +35,7 @@ #ifdef GPR_POSIX_SOCKET +#include "src/core/lib/iomgr/network_status_tracker.h" #include "src/core/lib/iomgr/tcp_posix.h" #include @@ -474,6 +475,8 @@ grpc_endpoint *grpc_tcp_create(grpc_fd *em_fd, size_t slice_size, tcp->write_closure.cb = tcp_handle_write; tcp->write_closure.cb_arg = tcp; gpr_slice_buffer_init(&tcp->last_read_buffer); + /* Tell network status tracker about new endpoint */ + grpc_network_status_register_endpoint(&tcp->base); return &tcp->base; } diff --git a/src/core/lib/iomgr/tcp_windows.c b/src/core/lib/iomgr/tcp_windows.c index b2af8030aaa..8140d1d8cd2 100644 --- a/src/core/lib/iomgr/tcp_windows.c +++ b/src/core/lib/iomgr/tcp_windows.c @@ -37,6 +37,7 @@ #include +#include "src/core/lib/iomgr/network_status_tracker.h" #include "src/core/lib/iomgr/sockaddr_windows.h" #include @@ -401,6 +402,9 @@ grpc_endpoint *grpc_tcp_create(grpc_winsocket *socket, char *peer_string) { grpc_closure_init(&tcp->on_read, on_read, tcp); grpc_closure_init(&tcp->on_write, on_write, tcp); tcp->peer_string = gpr_strdup(peer_string); + /* Tell network status tracking code about the new endpoint */ + grpc_network_status_register_endpoint(&tcp->base); + return &tcp->base; } diff --git a/src/python/grpcio/grpc_core_dependencies.py b/src/python/grpcio/grpc_core_dependencies.py index 839c555f055..575dfa9dbde 100644 --- a/src/python/grpcio/grpc_core_dependencies.py +++ b/src/python/grpcio/grpc_core_dependencies.py @@ -104,6 +104,7 @@ CORE_SOURCE_FILES = [ 'src/core/lib/iomgr/iomgr_posix.c', 'src/core/lib/iomgr/iomgr_windows.c', 'src/core/lib/iomgr/load_file.c', + 'src/core/lib/iomgr/network_status_tracker.c', 'src/core/lib/iomgr/polling_entity.c', 'src/core/lib/iomgr/pollset_set_windows.c', 'src/core/lib/iomgr/pollset_windows.c', diff --git a/test/core/end2end/end2end_nosec_tests.c b/test/core/end2end/end2end_nosec_tests.c index b71299c09e0..09ed6e6d688 100644 --- a/test/core/end2end/end2end_nosec_tests.c +++ b/test/core/end2end/end2end_nosec_tests.c @@ -89,6 +89,8 @@ extern void max_message_length(grpc_end2end_test_config config); extern void max_message_length_pre_init(void); extern void negative_deadline(grpc_end2end_test_config config); extern void negative_deadline_pre_init(void); +extern void network_status_change(grpc_end2end_test_config config); +extern void network_status_change_pre_init(void); extern void no_op(grpc_end2end_test_config config); extern void no_op_pre_init(void); extern void payload(grpc_end2end_test_config config); @@ -144,6 +146,7 @@ void grpc_end2end_tests_pre_init(void) { max_concurrent_streams_pre_init(); max_message_length_pre_init(); negative_deadline_pre_init(); + network_status_change_pre_init(); no_op_pre_init(); payload_pre_init(); ping_pre_init(); @@ -190,6 +193,7 @@ void grpc_end2end_tests(int argc, char **argv, max_concurrent_streams(config); max_message_length(config); negative_deadline(config); + network_status_change(config); no_op(config); payload(config); ping(config); @@ -300,6 +304,10 @@ void grpc_end2end_tests(int argc, char **argv, negative_deadline(config); continue; } + if (0 == strcmp("network_status_change", argv[i])) { + network_status_change(config); + continue; + } if (0 == strcmp("no_op", argv[i])) { no_op(config); continue; diff --git a/test/core/end2end/end2end_tests.c b/test/core/end2end/end2end_tests.c index 00c9c44a78c..312700646cd 100644 --- a/test/core/end2end/end2end_tests.c +++ b/test/core/end2end/end2end_tests.c @@ -91,6 +91,8 @@ extern void max_message_length(grpc_end2end_test_config config); extern void max_message_length_pre_init(void); extern void negative_deadline(grpc_end2end_test_config config); extern void negative_deadline_pre_init(void); +extern void network_status_change(grpc_end2end_test_config config); +extern void network_status_change_pre_init(void); extern void no_op(grpc_end2end_test_config config); extern void no_op_pre_init(void); extern void payload(grpc_end2end_test_config config); @@ -147,6 +149,7 @@ void grpc_end2end_tests_pre_init(void) { max_concurrent_streams_pre_init(); max_message_length_pre_init(); negative_deadline_pre_init(); + network_status_change_pre_init(); no_op_pre_init(); payload_pre_init(); ping_pre_init(); @@ -194,6 +197,7 @@ void grpc_end2end_tests(int argc, char **argv, max_concurrent_streams(config); max_message_length(config); negative_deadline(config); + network_status_change(config); no_op(config); payload(config); ping(config); @@ -308,6 +312,10 @@ void grpc_end2end_tests(int argc, char **argv, negative_deadline(config); continue; } + if (0 == strcmp("network_status_change", argv[i])) { + network_status_change(config); + continue; + } if (0 == strcmp("no_op", argv[i])) { no_op(config); continue; diff --git a/test/core/end2end/gen_build_yaml.py b/test/core/end2end/gen_build_yaml.py index 325d9b3cad0..716dca20bb1 100755 --- a/test/core/end2end/gen_build_yaml.py +++ b/test/core/end2end/gen_build_yaml.py @@ -112,6 +112,7 @@ END2END_TESTS = { 'max_concurrent_streams': default_test_options._replace(proxyable=False), 'max_message_length': default_test_options, 'negative_deadline': default_test_options, + 'network_status_change': default_test_options, 'no_op': default_test_options, 'payload': default_test_options, 'ping_pong_streaming': default_test_options, diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index e1555930e91..545536c97c4 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -818,6 +818,7 @@ src/core/lib/iomgr/iomgr.h \ src/core/lib/iomgr/iomgr_internal.h \ src/core/lib/iomgr/iomgr_posix.h \ src/core/lib/iomgr/load_file.h \ +src/core/lib/iomgr/network_status_tracker.h \ src/core/lib/iomgr/polling_entity.h \ src/core/lib/iomgr/pollset.h \ src/core/lib/iomgr/pollset_set.h \ @@ -973,6 +974,7 @@ src/core/lib/iomgr/iomgr.c \ src/core/lib/iomgr/iomgr_posix.c \ src/core/lib/iomgr/iomgr_windows.c \ src/core/lib/iomgr/load_file.c \ +src/core/lib/iomgr/network_status_tracker.c \ src/core/lib/iomgr/polling_entity.c \ src/core/lib/iomgr/pollset_set_windows.c \ src/core/lib/iomgr/pollset_windows.c \ diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 57e3bd63f42..7f8cc8ad436 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -5387,6 +5387,7 @@ "test/core/end2end/tests/max_concurrent_streams.c", "test/core/end2end/tests/max_message_length.c", "test/core/end2end/tests/negative_deadline.c", + "test/core/end2end/tests/network_status_change.c", "test/core/end2end/tests/no_op.c", "test/core/end2end/tests/payload.c", "test/core/end2end/tests/ping.c", @@ -5445,6 +5446,7 @@ "test/core/end2end/tests/max_concurrent_streams.c", "test/core/end2end/tests/max_message_length.c", "test/core/end2end/tests/negative_deadline.c", + "test/core/end2end/tests/network_status_change.c", "test/core/end2end/tests/no_op.c", "test/core/end2end/tests/payload.c", "test/core/end2end/tests/ping.c", @@ -5732,6 +5734,7 @@ "src/core/lib/iomgr/iomgr_internal.h", "src/core/lib/iomgr/iomgr_posix.h", "src/core/lib/iomgr/load_file.h", + "src/core/lib/iomgr/network_status_tracker.h", "src/core/lib/iomgr/polling_entity.h", "src/core/lib/iomgr/pollset.h", "src/core/lib/iomgr/pollset_set.h", @@ -5847,6 +5850,8 @@ "src/core/lib/iomgr/iomgr_windows.c", "src/core/lib/iomgr/load_file.c", "src/core/lib/iomgr/load_file.h", + "src/core/lib/iomgr/network_status_tracker.c", + "src/core/lib/iomgr/network_status_tracker.h", "src/core/lib/iomgr/polling_entity.c", "src/core/lib/iomgr/polling_entity.h", "src/core/lib/iomgr/pollset.h", diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index 3ed7a6bc476..bf5ff5be191 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -4900,6 +4900,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_census_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -5736,6 +5758,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_compress_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -6548,6 +6592,27 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_fakesec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -7262,6 +7327,26 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_fd_test", + "platforms": [ + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -8030,6 +8115,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -8722,6 +8829,22 @@ "linux" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "linux" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full+pipe_test", + "platforms": [ + "linux" + ] + }, { "args": [ "no_op" @@ -9452,6 +9575,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full+trace_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -10288,6 +10433,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_loadreporting_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -11100,6 +11267,27 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_oauth2_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -11814,6 +12002,27 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_proxy_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -12507,6 +12716,27 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_sockpair_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -13179,6 +13409,27 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_sockpair+trace_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -13874,7 +14125,7 @@ }, { "args": [ - "no_op" + "network_status_change" ], "ci_platforms": [ "windows", @@ -13895,7 +14146,7 @@ }, { "args": [ - "payload" + "no_op" ], "ci_platforms": [ "windows", @@ -13916,7 +14167,7 @@ }, { "args": [ - "ping_pong_streaming" + "payload" ], "ci_platforms": [ "windows", @@ -13937,7 +14188,7 @@ }, { "args": [ - "registered_call" + "ping_pong_streaming" ], "ci_platforms": [ "windows", @@ -13958,14 +14209,14 @@ }, { "args": [ - "request_with_flags" + "registered_call" ], "ci_platforms": [ "windows", "linux", "posix" ], - "cpu_cost": 0.1, + "cpu_cost": 1.0, "exclude_configs": [], "flaky": false, "language": "c", @@ -13979,14 +14230,14 @@ }, { "args": [ - "request_with_payload" + "request_with_flags" ], "ci_platforms": [ "windows", "linux", "posix" ], - "cpu_cost": 1.0, + "cpu_cost": 0.1, "exclude_configs": [], "flaky": false, "language": "c", @@ -14000,7 +14251,28 @@ }, { "args": [ - "server_finishes_request" + "request_with_payload" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_sockpair_1byte_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, + { + "args": [ + "server_finishes_request" ], "ci_platforms": [ "windows", @@ -14652,6 +14924,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_ssl_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -15488,6 +15782,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_ssl_cert_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -16216,6 +16532,27 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_ssl_proxy_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -16928,6 +17265,26 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_uds_test", + "platforms": [ + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -17714,6 +18071,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_census_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -18528,6 +18907,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_compress_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -19236,6 +19637,26 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_fd_nosec_test", + "platforms": [ + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -19982,6 +20403,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -20658,6 +21101,22 @@ "linux" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "linux" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full+pipe_nosec_test", + "platforms": [ + "linux" + ] + }, { "args": [ "no_op" @@ -21366,6 +21825,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full+trace_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -22180,6 +22661,28 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_loadreporting_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -22887,6 +23390,27 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_proxy_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -23559,6 +24083,27 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_sockpair_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -24210,6 +24755,27 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_sockpair+trace_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -24922,6 +25488,29 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [ + "msan" + ], + "flaky": false, + "language": "c", + "name": "h2_sockpair_1byte_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" @@ -25638,6 +26227,26 @@ "posix" ] }, + { + "args": [ + "network_status_change" + ], + "ci_platforms": [ + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_uds_nosec_test", + "platforms": [ + "linux", + "mac", + "posix" + ] + }, { "args": [ "no_op" diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj b/vsprojects/vcxproj/grpc/grpc.vcxproj index 84e03f70561..be4a28d5ecf 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj @@ -327,6 +327,7 @@ + @@ -513,6 +514,8 @@ + + diff --git a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters index 0f817e0562d..086b62d572c 100644 --- a/vsprojects/vcxproj/grpc/grpc.vcxproj.filters +++ b/vsprojects/vcxproj/grpc/grpc.vcxproj.filters @@ -88,6 +88,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr @@ -731,6 +734,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj index ac9bb186b71..d88fc0eef0a 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj @@ -316,6 +316,7 @@ + @@ -480,6 +481,8 @@ + + diff --git a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters index 1341474142c..e1bc1801f2f 100644 --- a/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters +++ b/vsprojects/vcxproj/grpc_unsecure/grpc_unsecure.vcxproj.filters @@ -91,6 +91,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr @@ -638,6 +641,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr diff --git a/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj b/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj index 22cd102d116..5297d79e263 100644 --- a/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj +++ b/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj @@ -199,6 +199,8 @@ + + diff --git a/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj.filters b/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj.filters index 1bb208bba8e..897a42386f4 100644 --- a/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj.filters +++ b/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj.filters @@ -73,6 +73,9 @@ test\core\end2end\tests + + test\core\end2end\tests + test\core\end2end\tests diff --git a/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj b/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj index bfd437e8717..3bc61fc581d 100644 --- a/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj +++ b/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj @@ -201,6 +201,8 @@ + + diff --git a/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj.filters b/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj.filters index 61c065f77ca..b896ff8bf1f 100644 --- a/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj.filters +++ b/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj.filters @@ -76,6 +76,9 @@ test\core\end2end\tests + + test\core\end2end\tests + test\core\end2end\tests From adb65a6a4905ee8a74e6f6cb222c0b95ed1b7b1b Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Mon, 20 Jun 2016 15:56:43 -0700 Subject: [PATCH 0134/1039] Fixed gpr_log format issues --- .../chttp2/transport/chttp2_transport.c | 2 +- test/core/end2end/bad_server_response_test.c | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index aad655effde..2a5b503c0c4 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -1705,7 +1705,7 @@ static void parsing_action(grpc_exec_ctx *exec_ctx, void *arg, errors[1] = grpc_chttp2_perform_read(exec_ctx, &t->parsing, t->read_buffer.slices[i]); }; - if (i != t->read_buffer.count) { + if (i != t->read_buffer.count || errors[1] != GRPC_ERROR_NONE) { gpr_slice_unref(t->optional_drop_message); errors[2] = try_http_parsing(exec_ctx, t); if (errors[2] != GRPC_ERROR_NONE) { diff --git a/test/core/end2end/bad_server_response_test.c b/test/core/end2end/bad_server_response_test.c index 6c00942fb75..c2882b62434 100644 --- a/test/core/end2end/bad_server_response_test.c +++ b/test/core/end2end/bad_server_response_test.c @@ -96,8 +96,8 @@ static grpc_closure on_write; static void *tag(intptr_t t) { return (void *)t; } -static void done_write(grpc_exec_ctx *exec_ctx, void *arg, bool success) { - GPR_ASSERT(success); +static void done_write(grpc_exec_ctx *exec_ctx, void *arg, grpc_error *error) { + GPR_ASSERT(error == GRPC_ERROR_NONE); gpr_atm_rel_store(&state.done_atm, 1); } @@ -111,12 +111,11 @@ static void handle_write(grpc_exec_ctx *exec_ctx) { grpc_endpoint_write(exec_ctx, state.tcp, &state.outgoing_buffer, &on_write); } -static void handle_read(grpc_exec_ctx *exec_ctx, void *arg, bool success) { - GPR_ASSERT(success); +static void handle_read(grpc_exec_ctx *exec_ctx, void *arg, grpc_error *error) { + GPR_ASSERT(error == GRPC_ERROR_NONE); state.incoming_data_length += state.temp_incoming_buffer.length; size_t i; - gpr_log(GPR_DEBUG, "read: success=%d", success); for (i = 0; i < state.temp_incoming_buffer.count; i++) { char *dump = gpr_dump_slice(state.temp_incoming_buffer.slices[i], GPR_DUMP_HEX | GPR_DUMP_ASCII); @@ -124,7 +123,8 @@ static void handle_read(grpc_exec_ctx *exec_ctx, void *arg, bool success) { gpr_free(dump); } - gpr_log(GPR_DEBUG, "got %d bytes, http2 connect string is %d bytes", + gpr_log(GPR_DEBUG, + "got %" PRIuPTR " bytes, http2 connect string is %" PRIuMAX " bytes", state.incoming_data_length, GRPC_CHTTP2_CLIENT_CONNECT_STRLEN); if (state.incoming_data_length > GRPC_CHTTP2_CLIENT_CONNECT_STRLEN) { handle_write(exec_ctx); @@ -239,8 +239,8 @@ static void actually_poll_server(void *arg) { bool done = gpr_atm_acq_load(&state.done_atm) != 0; gpr_timespec time_left = gpr_time_sub(deadline, gpr_now(GPR_CLOCK_REALTIME)); - gpr_log(GPR_DEBUG, "done=%d, time_left=%d.%09d", done, time_left.tv_sec, - time_left.tv_nsec); + gpr_log(GPR_DEBUG, "done=%d, time_left=%" PRId64 ".%09d", done, + time_left.tv_sec, time_left.tv_nsec); if (done || gpr_time_cmp(time_left, gpr_time_0(GPR_TIMESPAN)) < 0) { break; } From 2f8ade0b9df48990e3617a302a5da946f032d4f6 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 17 Jun 2016 13:28:38 -0700 Subject: [PATCH 0135/1039] Significantly refactor the polling island locking and refcounting code --- src/core/lib/iomgr/ev_epoll_linux.c | 462 ++++++++++++++++------------ 1 file changed, 270 insertions(+), 192 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index ed2c494b783..72288889c02 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -140,18 +140,40 @@ static void fd_global_shutdown(void); #define CLOSURE_READY ((grpc_closure *)1) /******************************************************************************* - * Polling-island Declarations + * Polling island Declarations */ -/* TODO: sree: Consider making ref_cnt and merged_to to gpr_atm - This would - * significantly reduce the number of mutex acquisition calls. */ + +// #define GRPC_PI_REF_COUNT_DEBUG +#ifdef GRPC_PI_REF_COUNT_DEBUG + +#define PI_ADD_REF(p, r) pi_add_ref_dbg((p), 1, (r), __FILE__, __LINE__) +#define PI_UNREF(p, r) pi_unref_dbg((p), 1, (r), __FILE__, __LINE__) + +#else /* defined(GRPC_PI_REF_COUNT_DEBUG) */ + +#define PI_ADD_REF(p, r) pi_add_ref((p), 1) +#define PI_UNREF(p, r) pi_unref((p), 1) + +#endif /* !defined(GPRC_PI_REF_COUNT_DEBUG) */ + typedef struct polling_island { gpr_mu mu; - int ref_cnt; - - /* Points to the polling_island this merged into. - * If merged_to is not NULL, all the remaining fields (except mu and ref_cnt) - * are invalid and must be ignored */ - struct polling_island *merged_to; + /* Ref count. Use PI_ADD_REF() and PI_UNREF() macros to increment/decrement + the refcount. + Once the ref count becomes zero, this structure is destroyed which means + we should ensure that there is never a scenario where a PI_ADD_REF() is + racing with a PI_UNREF() that just made the ref_count zero. */ + gpr_atm ref_count; + + /* Pointer to the polling_island this merged into. + * merged_to value is only set once in polling_island's lifetime (and that too + * only if the island is merged with another island). Because of this, we can + * use gpr_atm type here so that we can do atomic access on this and reduce + * lock contention on 'mu' mutex. + * + * Note that if this field is not NULL (i.e not 0), all the remaining fields + * (except mu and ref_count) are invalid and must be ignored. */ + gpr_atm merged_to; /* The fd of the underlying epoll set */ int epoll_fd; @@ -236,6 +258,8 @@ static grpc_wakeup_fd polling_island_wakeup_fd; static gpr_mu g_pi_freelist_mu; static polling_island *g_pi_freelist = NULL; +static void polling_island_delete(); /* Forward declaration */ + #ifdef GRPC_TSAN /* Currently TSAN may incorrectly flag data races between epoll_ctl and epoll_wait for any grpc_fd structs that are added to the epoll set via @@ -247,6 +271,51 @@ static polling_island *g_pi_freelist = NULL; gpr_atm g_epoll_sync; #endif /* defined(GRPC_TSAN) */ +#ifdef GRPC_PI_REF_COUNT_DEBUG +long pi_add_ref(polling_island *pi, int ref_cnt); +long pi_unref(polling_island *pi, int ref_cnt); + +void pi_add_ref_dbg(polling_island *pi, int ref_cnt, char *reason, char *file, + int line) { + long old_cnt = pi_add_ref(pi, ref_cnt); + gpr_log(GPR_DEBUG, "Add ref pi: %p, old:%ld -> new:%ld (%s) - (%s, %d)", + (void *)pi, old_cnt, (old_cnt + ref_cnt), reason, file, line); +} + +void pi_unref_dbg(polling_island *pi, int ref_cnt, char *reason, char *file, + int line) { + long old_cnt = pi_unref(pi, ref_cnt); + gpr_log(GPR_DEBUG, "Unref pi: %p, old:%ld -> new:%ld (%s) - (%s, %d)", + (void *)pi, old_cnt, (old_cnt - ref_cnt), reason, file, line); +} +#endif + +long pi_add_ref(polling_island *pi, int ref_cnt) { + return gpr_atm_no_barrier_fetch_add(&pi->ref_count, ref_cnt); +} + +long pi_unref(polling_island *pi, int ref_cnt) { + long old_cnt = gpr_atm_no_barrier_fetch_add(&pi->ref_count, -ref_cnt); + + /* If ref count went to zero, delete the polling island. Note that this need + not be done under a lock. Once the ref count goes to zero, we are + guaranteed that no one else holds a reference to the polling island (and + that there is no racing pi_add_ref() call either. + + Also, if we are deleting the polling island and the merged_to field is + non-empty, we should remove a ref to the merged_to polling island + */ + if (old_cnt == ref_cnt) { + polling_island *next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); + polling_island_delete(pi); + if (next != NULL) { + PI_UNREF(next, "pi_delete"); /* Recursive call */ + } + } + + return old_cnt; +} + /* The caller is expected to hold pi->mu lock before calling this function */ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, size_t fd_count, bool add_fd_refs) { @@ -355,8 +424,7 @@ static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd, } } -static polling_island *polling_island_create(grpc_fd *initial_fd, - int initial_ref_cnt) { +static polling_island *polling_island_create(grpc_fd *initial_fd) { polling_island *pi = NULL; /* Try to get one from the polling island freelist */ @@ -377,6 +445,9 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, pi->fds = NULL; } + gpr_atm_no_barrier_store(&pi->ref_count, 0); + gpr_atm_no_barrier_store(&pi->merged_to, NULL); + pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); if (pi->epoll_fd < 0) { @@ -387,14 +458,12 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, polling_island_add_wakeup_fd_locked(pi, &grpc_global_wakeup_fd); - pi->ref_cnt = initial_ref_cnt; - pi->merged_to = NULL; pi->next_free = NULL; if (initial_fd != NULL) { - /* It is not really needed to get the pi->mu lock here. If this is a newly - created polling island (or one that we got from the freelist), no one - else would be holding a lock to it anyway */ + /* Lock the polling island here just in case we got this structure from the + freelist and the polling island lock was not released yet (by the code + that adds the polling island to the freelist) */ gpr_mu_lock(&pi->mu); polling_island_add_fds_locked(pi, &initial_fd, 1, true); gpr_mu_unlock(&pi->mu); @@ -404,140 +473,136 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, } static void polling_island_delete(polling_island *pi) { - GPR_ASSERT(pi->ref_cnt == 0); GPR_ASSERT(pi->fd_cnt == 0); + gpr_atm_rel_store(&pi->merged_to, NULL); + close(pi->epoll_fd); pi->epoll_fd = -1; - pi->merged_to = NULL; - gpr_mu_lock(&g_pi_freelist_mu); pi->next_free = g_pi_freelist; g_pi_freelist = pi; gpr_mu_unlock(&g_pi_freelist_mu); } -void polling_island_unref_and_unlock(polling_island *pi, int unref_by) { - pi->ref_cnt -= unref_by; - int ref_cnt = pi->ref_cnt; - GPR_ASSERT(ref_cnt >= 0); - - gpr_mu_unlock(&pi->mu); - - if (ref_cnt == 0) { - polling_island_delete(pi); - } -} - -polling_island *polling_island_update_and_lock(polling_island *pi, int unref_by, - int add_ref_by) { +/* Gets the lock on the *latest* polling island i.e the last polling island in + the linked list (linked by 'merged_to' link). Call gpr_mu_unlock on the + returned polling island's mu. + Usage: To lock/unlock polling island "pi", do the following: + polling_island *pi_latest = polling_island_lock(pi); + ... + ... critical section .. + ... + gpr_mu_unlock(&pi_latest->mu); //NOTE: use pi_latest->mu. NOT pi->mu */ +polling_island *polling_island_lock(polling_island *pi) { polling_island *next = NULL; - gpr_mu_lock(&pi->mu); - while (pi->merged_to != NULL) { - next = pi->merged_to; - polling_island_unref_and_unlock(pi, unref_by); + while (true) { + next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); + if (next == NULL) { + /* pi is the last node in the linked list. Get the lock and check again + (under the pi->mu lock) that pi is still the last node (because a merge + may have happend after the (next == NULL) check above and before + getting the pi->mu lock. + If pi is the last node, we are done. If not, unlock and continue + traversing the list */ + gpr_mu_lock(&pi->mu); + next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); + if (next == NULL) { + break; + } + gpr_mu_unlock(&pi->mu); + } + pi = next; - gpr_mu_lock(&pi->mu); } - pi->ref_cnt += add_ref_by; return pi; } -void polling_island_pair_update_and_lock(polling_island **p, - polling_island **q) { +/* Gets the lock on the *latest* polling islands pointed by *p and *q. + This function is needed because calling the following block of code to obtain + locks on polling islands (*p and *q) is prone to deadlocks. + { + polling_island_lock(*p); + polling_island_lock(*q); + } + + Usage/exmaple: + polling_island *p1; + polling_island *p2; + .. + polling_island_lock_pair(&p1, &p2); + .. + .. Critical section with both p1 and p2 locked + .. + // Release locks + // **IMPORTANT**: Make sure you check p1 == p2 AFTER the function + // polling_island_lock_pair() was called and if so, release the lock only + // once. Note: Even if p1 != p2 beforec calling polling_island_lock_pair(), + // they might be after the function returns: + if (p1 == p2) { + gpr_mu_unlock(&p1->mu) + } else { + gpr_mu_unlock(&p1->mu); + gpr_mu_unlock(&p2->mu); + } + +*/ +void polling_island_lock_pair(polling_island **p, polling_island **q) { polling_island *pi_1 = *p; polling_island *pi_2 = *q; - polling_island *temp = NULL; - bool pi_1_locked = false; - bool pi_2_locked = false; - int num_swaps = 0; - - /* Loop until either pi_1 == pi_2 or until we acquired locks on both pi_1 - and pi_2 */ - while (pi_1 != pi_2 && !(pi_1_locked && pi_2_locked)) { - /* The following assertions are true at this point: - - pi_1 != pi_2 (else, the while loop would have exited) - - pi_1 MAY be locked - - pi_2 is NOT locked */ - - /* To maintain lock order consistency, always lock polling_island node with - lower address first. - First, make sure pi_1 < pi_2 before proceeding any further. If it turns - out that pi_1 > pi_2, unlock pi_1 if locked (because pi_2 is not locked - at this point and having pi_1 locked would violate the lock order) and - swap pi_1 and pi_2 so that pi_1 becomes less than pi_2 */ - if (pi_1 > pi_2) { - if (pi_1_locked) { - gpr_mu_unlock(&pi_1->mu); - pi_1_locked = false; - } + polling_island *next_1 = NULL; + polling_island *next_2 = NULL; + + /* The algorithm is simple: + - Go to the last polling islands in the linked lists *pi_1 and *pi_2 (and + keep updating pi_1 and pi_2) + - Then obtain locks on the islands by following a lock order rule of + locking polling_island with lower address first + Special case: Before obtaining the locks, check if pi_1 and pi_2 are + pointing to the same island. If that is the case, we can just call + polling_island_lock() + - After obtaining both the locks, double check that the polling islands + are still the last polling islands in their respective linked lists + (this is because there might have been polling island merges before + we got the lock) + - If the polling islands are the last islands, we are done. If not, + release the locks and continue the process from the first step */ + while (true) { + next_1 = (polling_island *)gpr_atm_acq_load(&pi_1->merged_to); + while (next_1 != NULL) { + pi_1 = next_1; + next_1 = (polling_island *)gpr_atm_acq_load(&pi_1->merged_to); + } - GPR_SWAP(polling_island *, pi_1, pi_2); - num_swaps++; + next_2 = (polling_island *)gpr_atm_acq_load(&pi_2->merged_to); + while (next_2 != NULL) { + pi_2 = next_2; + next_2 = (polling_island *)gpr_atm_acq_load(&pi_2->merged_to); } - /* The following assertions are true at this point: - - pi_1 != pi_2 - - pi_1 < pi_2 (address of pi_1 is less than that of pi_2) - - pi_1 MAYBE locked - - pi_2 is NOT locked */ + if (pi_1 == pi_2) { + pi_1 = pi_2 = polling_island_lock(pi_1); + break; + } - /* Lock pi_1 (if pi_1 is pointing to the terminal node in the list) */ - if (!pi_1_locked) { + if (pi_1 < pi_2) { + gpr_mu_lock(&pi_1->mu); + gpr_mu_lock(&pi_2->mu); + } else { + gpr_mu_lock(&pi_2->mu); gpr_mu_lock(&pi_1->mu); - pi_1_locked = true; - - /* If pi_1 is not terminal node (i.e pi_1->merged_to != NULL), we are not - done locking this polling_island yet. Release the lock on this node and - advance pi_1 to the next node in the list; and go to the beginning of - the loop (we can't proceed to locking pi_2 unless we locked pi_1 first) - */ - if (pi_1->merged_to != NULL) { - temp = pi_1->merged_to; - polling_island_unref_and_unlock(pi_1, 1); - pi_1 = temp; - pi_1_locked = false; - - continue; - } } - /* The following assertions are true at this point: - - pi_1 is locked - - pi_2 is unlocked - - pi_1 != pi_2 */ - - gpr_mu_lock(&pi_2->mu); - pi_2_locked = true; - - /* If pi_2 is not terminal node, we are not done locking this polling_island - yet. Release the lock and update pi_2 to the next node in the list */ - if (pi_2->merged_to != NULL) { - temp = pi_2->merged_to; - polling_island_unref_and_unlock(pi_2, 1); - pi_2 = temp; - pi_2_locked = false; + next_1 = (polling_island *)gpr_atm_acq_load(&pi_1->merged_to); + next_2 = (polling_island *)gpr_atm_acq_load(&pi_2->merged_to); + if (next_1 == NULL && next_2 == NULL) { + break; } - } - /* At this point, either pi_1 == pi_2 AND/OR we got both locks */ - if (pi_1 == pi_2) { - /* We may or may not have gotten the lock. If we didn't, walk the rest of - the polling_island list and get the lock */ - GPR_ASSERT(pi_1_locked || (!pi_1_locked && !pi_2_locked)); - if (!pi_1_locked) { - pi_1 = pi_2 = polling_island_update_and_lock(pi_1, 2, 0); - } - } else { - GPR_ASSERT(pi_1_locked && pi_2_locked); - /* If we swapped pi_1 and pi_2 odd number of times, do one more swap so that - pi_1 and pi_2 point to the same polling_island lists they started off - with at the beginning of this function (i.e *p and *q respectively) */ - if (num_swaps % 2 > 0) { - GPR_SWAP(polling_island *, pi_1, pi_2); - } + gpr_mu_unlock(&pi_1->mu); + gpr_mu_unlock(&pi_2->mu); } *p = pi_1; @@ -546,7 +611,7 @@ void polling_island_pair_update_and_lock(polling_island **p, polling_island *polling_island_merge(polling_island *p, polling_island *q) { /* Get locks on both the polling islands */ - polling_island_pair_update_and_lock(&p, &q); + polling_island_lock_pair(&p, &q); if (p == q) { /* Nothing needs to be done here */ @@ -568,15 +633,14 @@ polling_island *polling_island_merge(polling_island *p, polling_island *q) { /* Wakeup all the pollers (if any) on p so that they can pickup this change */ polling_island_add_wakeup_fd_locked(p, &polling_island_wakeup_fd); - p->merged_to = q; + /* Add the 'merged_to' link from p --> q */ + gpr_atm_rel_store(&p->merged_to, q); + PI_ADD_REF(q, "pi_merge"); /* To account for the new incoming ref from p */ - /* - The merged polling island (i.e q) inherits all the ref counts of the - island merging with it (i.e p) - - The island p will lose a ref count */ - q->ref_cnt += p->ref_cnt; - polling_island_unref_and_unlock(p, 1); /* Decrement refcount */ - polling_island_unref_and_unlock(q, 0); /* Just Unlock. Don't decrement ref */ + gpr_mu_unlock(&p->mu); + gpr_mu_unlock(&q->mu); + /* Return the merged polling island */ return q; } @@ -667,6 +731,7 @@ static void unref_by(grpc_fd *fd, int n) { fd->freelist_next = fd_freelist; fd_freelist = fd; grpc_iomgr_unregister_object(&fd->iomgr_object); + gpr_mu_unlock(&fd_freelist_mu); } else { GPR_ASSERT(old > n); @@ -785,16 +850,20 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, REF_BY(fd, 1, reason); /* Remove the fd from the polling island: - - Update the fd->polling_island to point to the latest polling island - - Remove the fd from the polling island. - - Remove a ref to the polling island and set fd->polling_island to NULL */ + - Get a lock on the latest polling island (i.e the last island in the + linked list pointed by fd->polling_island). This is the island that + would actually contain the fd + - Remove the fd from the latest polling island + - Unlock the latest polling island + - Set fd->polling_island to NULL (but remove the ref on the polling island + before doing this.) */ gpr_mu_lock(&fd->pi_mu); if (fd->polling_island != NULL) { - fd->polling_island = - polling_island_update_and_lock(fd->polling_island, 1, 0); - polling_island_remove_fd_locked(fd->polling_island, fd, is_fd_closed); + polling_island *pi_latest = polling_island_lock(fd->polling_island); + polling_island_remove_fd_locked(pi_latest, fd, is_fd_closed); + gpr_mu_unlock(&pi_latest->mu); - polling_island_unref_and_unlock(fd->polling_island, 1); + PI_UNREF(fd->polling_island, "fd_orphan"); fd->polling_island = NULL; } gpr_mu_unlock(&fd->pi_mu); @@ -1050,17 +1119,13 @@ static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { gpr_mu_unlock(&fd->mu); } -/* Release the reference to pollset->polling_island and set it to NULL. - pollset->mu must be held */ -static void pollset_release_polling_island_locked(grpc_pollset *pollset) { - gpr_mu_lock(&pollset->pi_mu); - if (pollset->polling_island) { - pollset->polling_island = - polling_island_update_and_lock(pollset->polling_island, 1, 0); - polling_island_unref_and_unlock(pollset->polling_island, 1); - pollset->polling_island = NULL; +static void pollset_release_polling_island(grpc_pollset *ps, char *reason) { + gpr_mu_lock(&ps->pi_mu); + if (ps->polling_island != NULL) { + PI_UNREF(ps->polling_island, reason); } - gpr_mu_unlock(&pollset->pi_mu); + ps->polling_island = NULL; + gpr_mu_unlock(&ps->pi_mu); } static void finish_shutdown_locked(grpc_exec_ctx *exec_ctx, @@ -1069,8 +1134,9 @@ static void finish_shutdown_locked(grpc_exec_ctx *exec_ctx, GPR_ASSERT(!pollset_has_workers(pollset)); pollset->finish_shutdown_called = true; - pollset_release_polling_island_locked(pollset); + /* Release the ref and set pollset->polling_island to NULL */ + pollset_release_polling_island(pollset, "ps_shutdown"); grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); } @@ -1110,7 +1176,7 @@ static void pollset_reset(grpc_pollset *pollset) { pollset->finish_shutdown_called = false; pollset->kicked_without_pollers = false; pollset->shutdown_done = NULL; - pollset_release_polling_island_locked(pollset); + pollset_release_polling_island(pollset, "ps_reset"); } #define GRPC_EPOLL_MAX_EVENTS 1000 @@ -1124,28 +1190,37 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, GPR_TIMER_BEGIN("pollset_work_and_unlock", 0); /* We need to get the epoll_fd to wait on. The epoll_fd is in inside the - polling island pointed by pollset->polling_island. + latest polling island pointed by pollset->polling_island. Acquire the following locks: - pollset->mu (which we already have) - pollset->pi_mu - - pollset->polling_island->mu (call polling_island_update_and_lock())*/ + - pollset->polling_island lock */ gpr_mu_lock(&pollset->pi_mu); - pi = pollset->polling_island; - if (pi == NULL) { - pi = polling_island_create(NULL, 1); + if (pollset->polling_island == NULL) { + pollset->polling_island = polling_island_create(NULL); + PI_ADD_REF(pollset->polling_island, "ps"); } - /* In addition to locking the polling island, add a ref so that the island - does not get destroyed (which means the epoll_fd won't be closed) while - we are are doing an epoll_wait() on the epoll_fd */ - pi = polling_island_update_and_lock(pi, 1, 1); + pi = polling_island_lock(pollset->polling_island); epoll_fd = pi->epoll_fd; - /* Update the pollset->polling_island */ - pollset->polling_island = pi; + /* Update the pollset->polling_island since the island being pointed by + pollset->polling_island may not be the latest (i.e pi) */ + if (pollset->polling_island != pi) { + /* Always do PI_ADD_REF before PI_UNREF because PI_UNREF may cause the + polling island to be deleted */ + PI_ADD_REF(pi, "ps"); + PI_UNREF(pollset->polling_island, "ps"); + pollset->polling_island = pi; + } + + /* Add an extra ref so that the island does not get destroyed (which means + the epoll_fd won't be closed) while we are are doing an epoll_wait() on the + epoll_fd */ + PI_ADD_REF(pi, "ps_work"); - polling_island_unref_and_unlock(pollset->polling_island, 0); /* Keep the ref*/ + gpr_mu_unlock(&pi->mu); gpr_mu_unlock(&pollset->pi_mu); gpr_mu_unlock(&pollset->mu); @@ -1193,14 +1268,12 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, GPR_ASSERT(pi != NULL); - /* Before leaving, release the extra ref we added to the polling island */ - /* It is important to note that at this point 'pi' may not be the same as - * pollset->polling_island. This is because pollset->polling_island pointer - * gets updated whenever the underlying polling island is merged with another - * island and while we are doing epoll_wait() above, the polling island may - * have been merged */ - pi = polling_island_update_and_lock(pi, 1, 0); /* No new ref added */ - polling_island_unref_and_unlock(pi, 1); + /* Before leaving, release the extra ref we added to the polling island. It + is important to use "pi" here (i.e our old copy of pollset->polling_island + that we got before releasing the polling island lock). This is because + pollset->polling_island pointer might get udpated in other parts of the + code when there is an island merge while we are doing epoll_wait() above */ + PI_UNREF(pi, "ps_work"); GPR_TIMER_END("pollset_work_and_unlock", 0); } @@ -1297,20 +1370,34 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, if (fd->polling_island == pollset->polling_island) { pi_new = fd->polling_island; if (pi_new == NULL) { - pi_new = polling_island_create(fd, 2); + pi_new = polling_island_create(fd); } } else if (fd->polling_island == NULL) { - pi_new = polling_island_update_and_lock(pollset->polling_island, 1, 1); - polling_island_add_fds_locked(pollset->polling_island, &fd, 1, true); + pi_new = polling_island_lock(pollset->polling_island); + polling_island_add_fds_locked(pi_new, &fd, 1, true); gpr_mu_unlock(&pi_new->mu); } else if (pollset->polling_island == NULL) { - pi_new = polling_island_update_and_lock(fd->polling_island, 1, 1); + pi_new = polling_island_lock(fd->polling_island); gpr_mu_unlock(&pi_new->mu); } else { pi_new = polling_island_merge(fd->polling_island, pollset->polling_island); } - fd->polling_island = pollset->polling_island = pi_new; + if (fd->polling_island != pi_new) { + PI_ADD_REF(pi_new, "fd"); + if (fd->polling_island != NULL) { + PI_UNREF(fd->polling_island, "fd"); + } + fd->polling_island = pi_new; + } + + if (pollset->polling_island != pi_new) { + PI_ADD_REF(pi_new, "ps"); + if (pollset->polling_island != NULL) { + PI_UNREF(pollset->polling_island, "ps"); + } + pollset->polling_island = pi_new; + } gpr_mu_unlock(&fd->pi_mu); gpr_mu_unlock(&pollset->pi_mu); @@ -1481,28 +1568,19 @@ void *grpc_pollset_get_polling_island(grpc_pollset *ps) { return pi; } -static polling_island *get_polling_island(polling_island *p) { - if (p == NULL) { - return NULL; - } +bool grpc_are_polling_islands_equal(void *p, void *q) { + polling_island *p1 = p; + polling_island *p2 = q; - polling_island *next; - gpr_mu_lock(&p->mu); - while (p->merged_to != NULL) { - next = p->merged_to; - gpr_mu_unlock(&p->mu); - p = next; - gpr_mu_lock(&p->mu); + polling_island_lock_pair(&p1, &p2); + if (p1 == p2) { + gpr_mu_unlock(&p1->mu); + } else { + gpr_mu_unlock(&p1->mu); + gpr_mu_unlock(&p2->mu); } - gpr_mu_unlock(&p->mu); - - return p; -} -bool grpc_are_polling_islands_equal(void *p, void *q) { - p = get_polling_island(p); - q = get_polling_island(q); - return p == q; + return p1 == p2; } /******************************************************************************* From aa338326ed399832394078282664c6bcf66d11e7 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Mon, 20 Jun 2016 16:47:40 -0700 Subject: [PATCH 0136/1039] Revert ProtoService.m, add an exception for its incompatible-pointer-types warning --- src/objective-c/ProtoRPC/ProtoService.m | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/objective-c/ProtoRPC/ProtoService.m b/src/objective-c/ProtoRPC/ProtoService.m index 4a14570d818..cd9bc7aeac4 100644 --- a/src/objective-c/ProtoRPC/ProtoService.m +++ b/src/objective-c/ProtoRPC/ProtoService.m @@ -65,19 +65,22 @@ return self; } -- (GRPCProtoCall *)RPCToMethod:(NSString *)method +#pragma clang diagnostic push +#pragma clang diagnostic ignored "-Wincompatible-pointer-types" +- (ProtoRPC *)RPCToMethod:(NSString *)method requestsWriter:(GRXWriter *)requestsWriter responseClass:(Class)responseClass responsesWriteable:(id)responsesWriteable { - GRPCProtoMethod *methodName = [[GRPCProtoMethod alloc] initWithPackage:_packageName + ProtoMethod *methodName = [[ProtoMethod alloc] initWithPackage:_packageName service:_serviceName method:method]; - return [[GRPCProtoCall alloc] initWithHost:_host + return [[ProtoRPC alloc] initWithHost:_host method:methodName requestsWriter:requestsWriter responseClass:responseClass responsesWriteable:responsesWriteable]; } +#pragma clang diagnostic pop @end @implementation GRPCProtoService From 113cdb4fc3a9c35f5493fd3a5ba53814a6470b9b Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Mon, 20 Jun 2016 17:08:29 -0700 Subject: [PATCH 0137/1039] Fixed gpr_log format issues in linux_x86_default_bo --- test/core/end2end/bad_server_response_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/core/end2end/bad_server_response_test.c b/test/core/end2end/bad_server_response_test.c index c2882b62434..4a2355b5f6d 100644 --- a/test/core/end2end/bad_server_response_test.c +++ b/test/core/end2end/bad_server_response_test.c @@ -124,7 +124,7 @@ static void handle_read(grpc_exec_ctx *exec_ctx, void *arg, grpc_error *error) { } gpr_log(GPR_DEBUG, - "got %" PRIuPTR " bytes, http2 connect string is %" PRIuMAX " bytes", + "got %" PRIuPTR " bytes, http2 connect string is %" PRIuPTR " bytes", state.incoming_data_length, GRPC_CHTTP2_CLIENT_CONNECT_STRLEN); if (state.incoming_data_length > GRPC_CHTTP2_CLIENT_CONNECT_STRLEN) { handle_write(exec_ctx); From ea74f915771c0665daea12004f99cc50a11cbfbd Mon Sep 17 00:00:00 2001 From: Tamas Berghammer Date: Tue, 21 Jun 2016 13:27:58 +0100 Subject: [PATCH 0138/1039] Remove an unneccessary dependency from grpc++_base Previously grpc++_base was dependning on grpc what caused a transitive dependency on ssl for grpc++_unsecure. Removing the grpc dependency shouldn't cause any issue as grpc++_base is a filegroup and all library using it already depends on grpc directly. This fixes https://github.com/grpc/grpc/issues/6784 --- BUILD | 1 - Makefile | 10 +++++----- build.yaml | 2 -- tools/run_tests/sources_and_headers.json | 2 -- vsprojects/grpc.sln | 1 - .../vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj | 3 --- 6 files changed, 5 insertions(+), 14 deletions(-) diff --git a/BUILD b/BUILD index ac0eb8e403d..1627f7233f8 100644 --- a/BUILD +++ b/BUILD @@ -1594,7 +1594,6 @@ cc_library( "//external:protobuf_clib", ":gpr", ":grpc_unsecure", - ":grpc", ], ) diff --git a/Makefile b/Makefile index d93ce337632..ca642429519 100644 --- a/Makefile +++ b/Makefile @@ -4101,18 +4101,18 @@ endif ifeq ($(SYSTEM),MINGW32) -$(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_UNSECURE_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/gpr.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/grpc_unsecure.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/grpc.$(SHARED_EXT) +$(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_UNSECURE_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/gpr.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/grpc_unsecure.$(SHARED_EXT) $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared grpc++_unsecure.def -Wl,--output-def=$(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).def -Wl,--out-implib=$(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION)-dll.a -o $(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr-imp -lgrpc_unsecure-imp -lgrpc-imp + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared grpc++_unsecure.def -Wl,--output-def=$(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).def -Wl,--out-implib=$(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION)-dll.a -o $(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr-imp -lgrpc_unsecure-imp else -$(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_UNSECURE_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/libgpr.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc_unsecure.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc.$(SHARED_EXT) +$(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_UNSECURE_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/libgpr.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc_unsecure.$(SHARED_EXT) $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` ifeq ($(SYSTEM),Darwin) - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -install_name $(SHARED_PREFIX)grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) -dynamiclib -o $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr -lgrpc_unsecure -lgrpc + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -install_name $(SHARED_PREFIX)grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) -dynamiclib -o $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr -lgrpc_unsecure else - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared -Wl,-soname,libgrpc++_unsecure.so.0 -o $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr -lgrpc_unsecure -lgrpc + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared -Wl,-soname,libgrpc++_unsecure.so.0 -o $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr -lgrpc_unsecure $(Q) ln -sf $(SHARED_PREFIX)grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).so.0 $(Q) ln -sf $(SHARED_PREFIX)grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).so endif diff --git a/build.yaml b/build.yaml index d41f4195947..59f7d19dfcc 100644 --- a/build.yaml +++ b/build.yaml @@ -714,8 +714,6 @@ filegroups: - src/cpp/util/status.cc - src/cpp/util/string_ref.cc - src/cpp/util/time.cc - deps: - - grpc uses: - grpc++_codegen_base - name: grpc++_codegen_base diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 57e3bd63f42..4cfd1c51f8f 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -4447,7 +4447,6 @@ { "deps": [ "gpr", - "grpc", "grpc++_base", "grpc++_codegen_base", "grpc++_codegen_base_src", @@ -6509,7 +6508,6 @@ }, { "deps": [ - "grpc", "grpc++_codegen_base" ], "headers": [ diff --git a/vsprojects/grpc.sln b/vsprojects/grpc.sln index a43daaf294c..771fefc56bd 100644 --- a/vsprojects/grpc.sln +++ b/vsprojects/grpc.sln @@ -66,7 +66,6 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc++_unsecure", "vcxproj\ ProjectSection(ProjectDependencies) = postProject {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} {46CEDFFF-9692-456A-AA24-38B5D6BCF4C5} = {46CEDFFF-9692-456A-AA24-38B5D6BCF4C5} - {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_create_jwt", "vcxproj\.\grpc_create_jwt\grpc_create_jwt.vcxproj", "{77971F8D-F583-3E77-0E3C-6C1FB6B1749C}" diff --git a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj index 03be485b297..f37fefc65a4 100644 --- a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj @@ -427,9 +427,6 @@ {46CEDFFF-9692-456A-AA24-38B5D6BCF4C5} - - {29D16885-7228-4C31-81ED-5F9187C7F2A9} - From d798a7d80e80ad1e57b08baec81b860c7f136c47 Mon Sep 17 00:00:00 2001 From: Tamas Berghammer Date: Tue, 21 Jun 2016 13:58:18 +0100 Subject: [PATCH 0139/1039] Generate a simple cmake build file This cmake file only builds the C and C++ code and all the third party dependencies with very few configuration options but it supports building on Linux and OSX as well as cross compiling. (tested {Linux, OSX} -> Android { arm, aarch64, x86 }) --- CMakeLists.txt | 950 ++++++++++++++++++++++++++++++ templates/CMakeLists.txt.template | 133 +++++ 2 files changed, 1083 insertions(+) create mode 100644 CMakeLists.txt create mode 100644 templates/CMakeLists.txt.template diff --git a/CMakeLists.txt b/CMakeLists.txt new file mode 100644 index 00000000000..5bd48b78ea9 --- /dev/null +++ b/CMakeLists.txt @@ -0,0 +1,950 @@ +# GRPC global cmake file +# This currently builds C and C++ code. +# This file has been automatically generated from a template file. +# Please look at the templates directory instead. +# This file can be regenerated from the template by running +# tools/buildgen/generate_projects.sh + +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + +cmake_minimum_required(VERSION 2.8) + +if (NOT BORINGSSL_ROOT_DIR) + set(BORINGSSL_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/third_party/boringssl) +endif() +if (NOT PROTOBUF_ROOT_DIR) + set(PROTOBUF_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/third_party/protobuf) +endif() +if (NOT ZLIB_ROOT_DIR) + set(ZLIB_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/third_party/zlib) +endif() + +add_subdirectory(${BORINGSSL_ROOT_DIR} third_party/boringssl) +add_subdirectory(${PROTOBUF_ROOT_DIR}/cmake third_party/protobuf) +add_subdirectory(${ZLIB_ROOT_DIR} third_party/zlib) + +set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=c11") +set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11") + + +add_library(gpr + src/core/lib/profiling/basic_timers.c + src/core/lib/profiling/stap_timers.c + src/core/lib/support/alloc.c + src/core/lib/support/avl.c + src/core/lib/support/backoff.c + src/core/lib/support/cmdline.c + src/core/lib/support/cpu_iphone.c + src/core/lib/support/cpu_linux.c + src/core/lib/support/cpu_posix.c + src/core/lib/support/cpu_windows.c + src/core/lib/support/env_linux.c + src/core/lib/support/env_posix.c + src/core/lib/support/env_windows.c + src/core/lib/support/histogram.c + src/core/lib/support/host_port.c + src/core/lib/support/log.c + src/core/lib/support/log_android.c + src/core/lib/support/log_linux.c + src/core/lib/support/log_posix.c + src/core/lib/support/log_windows.c + src/core/lib/support/murmur_hash.c + src/core/lib/support/slice.c + src/core/lib/support/slice_buffer.c + src/core/lib/support/stack_lockfree.c + src/core/lib/support/string.c + src/core/lib/support/string_posix.c + src/core/lib/support/string_util_windows.c + src/core/lib/support/string_windows.c + src/core/lib/support/subprocess_posix.c + src/core/lib/support/subprocess_windows.c + src/core/lib/support/sync.c + src/core/lib/support/sync_posix.c + src/core/lib/support/sync_windows.c + src/core/lib/support/thd.c + src/core/lib/support/thd_posix.c + src/core/lib/support/thd_windows.c + src/core/lib/support/time.c + src/core/lib/support/time_posix.c + src/core/lib/support/time_precise.c + src/core/lib/support/time_windows.c + src/core/lib/support/tls_pthread.c + src/core/lib/support/tmpfile_msys.c + src/core/lib/support/tmpfile_posix.c + src/core/lib/support/tmpfile_windows.c + src/core/lib/support/wrap_memcpy.c +) + +target_include_directories(gpr + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + + + +add_library(grpc + src/core/lib/surface/init.c + src/core/lib/channel/channel_args.c + src/core/lib/channel/channel_stack.c + src/core/lib/channel/channel_stack_builder.c + src/core/lib/channel/compress_filter.c + src/core/lib/channel/connected_channel.c + src/core/lib/channel/http_client_filter.c + src/core/lib/channel/http_server_filter.c + src/core/lib/compression/compression.c + src/core/lib/compression/message_compress.c + src/core/lib/debug/trace.c + src/core/lib/http/format_request.c + src/core/lib/http/httpcli.c + src/core/lib/http/parser.c + src/core/lib/iomgr/closure.c + src/core/lib/iomgr/endpoint.c + src/core/lib/iomgr/endpoint_pair_posix.c + src/core/lib/iomgr/endpoint_pair_windows.c + src/core/lib/iomgr/error.c + src/core/lib/iomgr/ev_poll_and_epoll_posix.c + src/core/lib/iomgr/ev_poll_posix.c + src/core/lib/iomgr/ev_posix.c + src/core/lib/iomgr/exec_ctx.c + src/core/lib/iomgr/executor.c + src/core/lib/iomgr/iocp_windows.c + src/core/lib/iomgr/iomgr.c + src/core/lib/iomgr/iomgr_posix.c + src/core/lib/iomgr/iomgr_windows.c + src/core/lib/iomgr/load_file.c + src/core/lib/iomgr/polling_entity.c + src/core/lib/iomgr/pollset_set_windows.c + src/core/lib/iomgr/pollset_windows.c + src/core/lib/iomgr/resolve_address_posix.c + src/core/lib/iomgr/resolve_address_windows.c + src/core/lib/iomgr/sockaddr_utils.c + src/core/lib/iomgr/socket_utils_common_posix.c + src/core/lib/iomgr/socket_utils_linux.c + src/core/lib/iomgr/socket_utils_posix.c + src/core/lib/iomgr/socket_windows.c + src/core/lib/iomgr/tcp_client_posix.c + src/core/lib/iomgr/tcp_client_windows.c + src/core/lib/iomgr/tcp_posix.c + src/core/lib/iomgr/tcp_server_posix.c + src/core/lib/iomgr/tcp_server_windows.c + src/core/lib/iomgr/tcp_windows.c + src/core/lib/iomgr/time_averaged_stats.c + src/core/lib/iomgr/timer.c + src/core/lib/iomgr/timer_heap.c + src/core/lib/iomgr/udp_server.c + src/core/lib/iomgr/unix_sockets_posix.c + src/core/lib/iomgr/unix_sockets_posix_noop.c + src/core/lib/iomgr/wakeup_fd_eventfd.c + src/core/lib/iomgr/wakeup_fd_nospecial.c + src/core/lib/iomgr/wakeup_fd_pipe.c + src/core/lib/iomgr/wakeup_fd_posix.c + src/core/lib/iomgr/workqueue_posix.c + src/core/lib/iomgr/workqueue_windows.c + src/core/lib/json/json.c + src/core/lib/json/json_reader.c + src/core/lib/json/json_string.c + src/core/lib/json/json_writer.c + src/core/lib/surface/alarm.c + src/core/lib/surface/api_trace.c + src/core/lib/surface/byte_buffer.c + src/core/lib/surface/byte_buffer_reader.c + src/core/lib/surface/call.c + src/core/lib/surface/call_details.c + src/core/lib/surface/call_log_batch.c + src/core/lib/surface/channel.c + src/core/lib/surface/channel_init.c + src/core/lib/surface/channel_ping.c + src/core/lib/surface/channel_stack_type.c + src/core/lib/surface/completion_queue.c + src/core/lib/surface/event_string.c + src/core/lib/surface/lame_client.c + src/core/lib/surface/metadata_array.c + src/core/lib/surface/server.c + src/core/lib/surface/validate_metadata.c + src/core/lib/surface/version.c + src/core/lib/transport/byte_stream.c + src/core/lib/transport/connectivity_state.c + src/core/lib/transport/metadata.c + src/core/lib/transport/metadata_batch.c + src/core/lib/transport/static_metadata.c + src/core/lib/transport/transport.c + src/core/lib/transport/transport_op_string.c + src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c + src/core/ext/transport/chttp2/transport/bin_decoder.c + src/core/ext/transport/chttp2/transport/bin_encoder.c + src/core/ext/transport/chttp2/transport/chttp2_plugin.c + src/core/ext/transport/chttp2/transport/chttp2_transport.c + src/core/ext/transport/chttp2/transport/frame_data.c + src/core/ext/transport/chttp2/transport/frame_goaway.c + src/core/ext/transport/chttp2/transport/frame_ping.c + src/core/ext/transport/chttp2/transport/frame_rst_stream.c + src/core/ext/transport/chttp2/transport/frame_settings.c + src/core/ext/transport/chttp2/transport/frame_window_update.c + src/core/ext/transport/chttp2/transport/hpack_encoder.c + src/core/ext/transport/chttp2/transport/hpack_parser.c + src/core/ext/transport/chttp2/transport/hpack_table.c + src/core/ext/transport/chttp2/transport/huffsyms.c + src/core/ext/transport/chttp2/transport/incoming_metadata.c + src/core/ext/transport/chttp2/transport/parsing.c + src/core/ext/transport/chttp2/transport/status_conversion.c + src/core/ext/transport/chttp2/transport/stream_lists.c + src/core/ext/transport/chttp2/transport/stream_map.c + src/core/ext/transport/chttp2/transport/timeout_encoding.c + src/core/ext/transport/chttp2/transport/varint.c + src/core/ext/transport/chttp2/transport/writing.c + src/core/ext/transport/chttp2/alpn/alpn.c + src/core/lib/http/httpcli_security_connector.c + src/core/lib/security/context/security_context.c + src/core/lib/security/credentials/composite/composite_credentials.c + src/core/lib/security/credentials/credentials.c + src/core/lib/security/credentials/credentials_metadata.c + src/core/lib/security/credentials/fake/fake_credentials.c + src/core/lib/security/credentials/google_default/credentials_posix.c + src/core/lib/security/credentials/google_default/credentials_windows.c + src/core/lib/security/credentials/google_default/google_default_credentials.c + src/core/lib/security/credentials/iam/iam_credentials.c + src/core/lib/security/credentials/jwt/json_token.c + src/core/lib/security/credentials/jwt/jwt_credentials.c + src/core/lib/security/credentials/jwt/jwt_verifier.c + src/core/lib/security/credentials/oauth2/oauth2_credentials.c + src/core/lib/security/credentials/plugin/plugin_credentials.c + src/core/lib/security/credentials/ssl/ssl_credentials.c + src/core/lib/security/transport/client_auth_filter.c + src/core/lib/security/transport/handshake.c + src/core/lib/security/transport/secure_endpoint.c + src/core/lib/security/transport/security_connector.c + src/core/lib/security/transport/server_auth_filter.c + src/core/lib/security/transport/tsi_error.c + src/core/lib/security/util/b64.c + src/core/lib/security/util/json_util.c + src/core/lib/surface/init_secure.c + src/core/lib/tsi/fake_transport_security.c + src/core/lib/tsi/ssl_transport_security.c + src/core/lib/tsi/transport_security.c + src/core/ext/transport/chttp2/client/secure/secure_channel_create.c + src/core/ext/client_config/channel_connectivity.c + src/core/ext/client_config/client_channel.c + src/core/ext/client_config/client_channel_factory.c + src/core/ext/client_config/client_config.c + src/core/ext/client_config/client_config_plugin.c + src/core/ext/client_config/connector.c + src/core/ext/client_config/default_initial_connect_string.c + src/core/ext/client_config/initial_connect_string.c + src/core/ext/client_config/lb_policy.c + src/core/ext/client_config/lb_policy_factory.c + src/core/ext/client_config/lb_policy_registry.c + src/core/ext/client_config/parse_address.c + src/core/ext/client_config/resolver.c + src/core/ext/client_config/resolver_factory.c + src/core/ext/client_config/resolver_registry.c + src/core/ext/client_config/subchannel.c + src/core/ext/client_config/subchannel_call_holder.c + src/core/ext/client_config/subchannel_index.c + src/core/ext/client_config/uri_parser.c + src/core/ext/transport/chttp2/server/insecure/server_chttp2.c + src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c + src/core/ext/transport/chttp2/client/insecure/channel_create.c + src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c + src/core/ext/lb_policy/grpclb/load_balancer_api.c + src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c + third_party/nanopb/pb_common.c + third_party/nanopb/pb_decode.c + third_party/nanopb/pb_encode.c + src/core/ext/lb_policy/pick_first/pick_first.c + src/core/ext/lb_policy/round_robin/round_robin.c + src/core/ext/resolver/dns/native/dns_resolver.c + src/core/ext/resolver/sockaddr/sockaddr_resolver.c + src/core/ext/load_reporting/load_reporting.c + src/core/ext/load_reporting/load_reporting_filter.c + src/core/ext/census/context.c + src/core/ext/census/gen/census.pb.c + src/core/ext/census/grpc_context.c + src/core/ext/census/grpc_filter.c + src/core/ext/census/grpc_plugin.c + src/core/ext/census/initialize.c + src/core/ext/census/mlog.c + src/core/ext/census/operation.c + src/core/ext/census/placeholders.c + src/core/ext/census/tracing.c + src/core/plugin_registry/grpc_plugin_registry.c +) + +target_include_directories(grpc + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc + ssl + gpr +) + + +add_library(grpc_cronet + src/core/lib/surface/init.c + src/core/lib/channel/channel_args.c + src/core/lib/channel/channel_stack.c + src/core/lib/channel/channel_stack_builder.c + src/core/lib/channel/compress_filter.c + src/core/lib/channel/connected_channel.c + src/core/lib/channel/http_client_filter.c + src/core/lib/channel/http_server_filter.c + src/core/lib/compression/compression.c + src/core/lib/compression/message_compress.c + src/core/lib/debug/trace.c + src/core/lib/http/format_request.c + src/core/lib/http/httpcli.c + src/core/lib/http/parser.c + src/core/lib/iomgr/closure.c + src/core/lib/iomgr/endpoint.c + src/core/lib/iomgr/endpoint_pair_posix.c + src/core/lib/iomgr/endpoint_pair_windows.c + src/core/lib/iomgr/error.c + src/core/lib/iomgr/ev_poll_and_epoll_posix.c + src/core/lib/iomgr/ev_poll_posix.c + src/core/lib/iomgr/ev_posix.c + src/core/lib/iomgr/exec_ctx.c + src/core/lib/iomgr/executor.c + src/core/lib/iomgr/iocp_windows.c + src/core/lib/iomgr/iomgr.c + src/core/lib/iomgr/iomgr_posix.c + src/core/lib/iomgr/iomgr_windows.c + src/core/lib/iomgr/load_file.c + src/core/lib/iomgr/polling_entity.c + src/core/lib/iomgr/pollset_set_windows.c + src/core/lib/iomgr/pollset_windows.c + src/core/lib/iomgr/resolve_address_posix.c + src/core/lib/iomgr/resolve_address_windows.c + src/core/lib/iomgr/sockaddr_utils.c + src/core/lib/iomgr/socket_utils_common_posix.c + src/core/lib/iomgr/socket_utils_linux.c + src/core/lib/iomgr/socket_utils_posix.c + src/core/lib/iomgr/socket_windows.c + src/core/lib/iomgr/tcp_client_posix.c + src/core/lib/iomgr/tcp_client_windows.c + src/core/lib/iomgr/tcp_posix.c + src/core/lib/iomgr/tcp_server_posix.c + src/core/lib/iomgr/tcp_server_windows.c + src/core/lib/iomgr/tcp_windows.c + src/core/lib/iomgr/time_averaged_stats.c + src/core/lib/iomgr/timer.c + src/core/lib/iomgr/timer_heap.c + src/core/lib/iomgr/udp_server.c + src/core/lib/iomgr/unix_sockets_posix.c + src/core/lib/iomgr/unix_sockets_posix_noop.c + src/core/lib/iomgr/wakeup_fd_eventfd.c + src/core/lib/iomgr/wakeup_fd_nospecial.c + src/core/lib/iomgr/wakeup_fd_pipe.c + src/core/lib/iomgr/wakeup_fd_posix.c + src/core/lib/iomgr/workqueue_posix.c + src/core/lib/iomgr/workqueue_windows.c + src/core/lib/json/json.c + src/core/lib/json/json_reader.c + src/core/lib/json/json_string.c + src/core/lib/json/json_writer.c + src/core/lib/surface/alarm.c + src/core/lib/surface/api_trace.c + src/core/lib/surface/byte_buffer.c + src/core/lib/surface/byte_buffer_reader.c + src/core/lib/surface/call.c + src/core/lib/surface/call_details.c + src/core/lib/surface/call_log_batch.c + src/core/lib/surface/channel.c + src/core/lib/surface/channel_init.c + src/core/lib/surface/channel_ping.c + src/core/lib/surface/channel_stack_type.c + src/core/lib/surface/completion_queue.c + src/core/lib/surface/event_string.c + src/core/lib/surface/lame_client.c + src/core/lib/surface/metadata_array.c + src/core/lib/surface/server.c + src/core/lib/surface/validate_metadata.c + src/core/lib/surface/version.c + src/core/lib/transport/byte_stream.c + src/core/lib/transport/connectivity_state.c + src/core/lib/transport/metadata.c + src/core/lib/transport/metadata_batch.c + src/core/lib/transport/static_metadata.c + src/core/lib/transport/transport.c + src/core/lib/transport/transport_op_string.c + src/core/ext/transport/cronet/client/secure/cronet_channel_create.c + src/core/ext/transport/cronet/transport/cronet_api_dummy.c + src/core/ext/transport/cronet/transport/cronet_transport.c + src/core/ext/transport/chttp2/client/secure/secure_channel_create.c + src/core/ext/transport/chttp2/transport/bin_decoder.c + src/core/ext/transport/chttp2/transport/bin_encoder.c + src/core/ext/transport/chttp2/transport/chttp2_plugin.c + src/core/ext/transport/chttp2/transport/chttp2_transport.c + src/core/ext/transport/chttp2/transport/frame_data.c + src/core/ext/transport/chttp2/transport/frame_goaway.c + src/core/ext/transport/chttp2/transport/frame_ping.c + src/core/ext/transport/chttp2/transport/frame_rst_stream.c + src/core/ext/transport/chttp2/transport/frame_settings.c + src/core/ext/transport/chttp2/transport/frame_window_update.c + src/core/ext/transport/chttp2/transport/hpack_encoder.c + src/core/ext/transport/chttp2/transport/hpack_parser.c + src/core/ext/transport/chttp2/transport/hpack_table.c + src/core/ext/transport/chttp2/transport/huffsyms.c + src/core/ext/transport/chttp2/transport/incoming_metadata.c + src/core/ext/transport/chttp2/transport/parsing.c + src/core/ext/transport/chttp2/transport/status_conversion.c + src/core/ext/transport/chttp2/transport/stream_lists.c + src/core/ext/transport/chttp2/transport/stream_map.c + src/core/ext/transport/chttp2/transport/timeout_encoding.c + src/core/ext/transport/chttp2/transport/varint.c + src/core/ext/transport/chttp2/transport/writing.c + src/core/ext/transport/chttp2/alpn/alpn.c + src/core/ext/client_config/channel_connectivity.c + src/core/ext/client_config/client_channel.c + src/core/ext/client_config/client_channel_factory.c + src/core/ext/client_config/client_config.c + src/core/ext/client_config/client_config_plugin.c + src/core/ext/client_config/connector.c + src/core/ext/client_config/default_initial_connect_string.c + src/core/ext/client_config/initial_connect_string.c + src/core/ext/client_config/lb_policy.c + src/core/ext/client_config/lb_policy_factory.c + src/core/ext/client_config/lb_policy_registry.c + src/core/ext/client_config/parse_address.c + src/core/ext/client_config/resolver.c + src/core/ext/client_config/resolver_factory.c + src/core/ext/client_config/resolver_registry.c + src/core/ext/client_config/subchannel.c + src/core/ext/client_config/subchannel_call_holder.c + src/core/ext/client_config/subchannel_index.c + src/core/ext/client_config/uri_parser.c + src/core/lib/http/httpcli_security_connector.c + src/core/lib/security/context/security_context.c + src/core/lib/security/credentials/composite/composite_credentials.c + src/core/lib/security/credentials/credentials.c + src/core/lib/security/credentials/credentials_metadata.c + src/core/lib/security/credentials/fake/fake_credentials.c + src/core/lib/security/credentials/google_default/credentials_posix.c + src/core/lib/security/credentials/google_default/credentials_windows.c + src/core/lib/security/credentials/google_default/google_default_credentials.c + src/core/lib/security/credentials/iam/iam_credentials.c + src/core/lib/security/credentials/jwt/json_token.c + src/core/lib/security/credentials/jwt/jwt_credentials.c + src/core/lib/security/credentials/jwt/jwt_verifier.c + src/core/lib/security/credentials/oauth2/oauth2_credentials.c + src/core/lib/security/credentials/plugin/plugin_credentials.c + src/core/lib/security/credentials/ssl/ssl_credentials.c + src/core/lib/security/transport/client_auth_filter.c + src/core/lib/security/transport/handshake.c + src/core/lib/security/transport/secure_endpoint.c + src/core/lib/security/transport/security_connector.c + src/core/lib/security/transport/server_auth_filter.c + src/core/lib/security/transport/tsi_error.c + src/core/lib/security/util/b64.c + src/core/lib/security/util/json_util.c + src/core/lib/surface/init_secure.c + src/core/lib/tsi/fake_transport_security.c + src/core/lib/tsi/ssl_transport_security.c + src/core/lib/tsi/transport_security.c + src/core/plugin_registry/grpc_cronet_plugin_registry.c +) + +target_include_directories(grpc_cronet + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_cronet + ssl + gpr +) + + +add_library(grpc_unsecure + src/core/lib/surface/init.c + src/core/lib/surface/init_unsecure.c + src/core/lib/channel/channel_args.c + src/core/lib/channel/channel_stack.c + src/core/lib/channel/channel_stack_builder.c + src/core/lib/channel/compress_filter.c + src/core/lib/channel/connected_channel.c + src/core/lib/channel/http_client_filter.c + src/core/lib/channel/http_server_filter.c + src/core/lib/compression/compression.c + src/core/lib/compression/message_compress.c + src/core/lib/debug/trace.c + src/core/lib/http/format_request.c + src/core/lib/http/httpcli.c + src/core/lib/http/parser.c + src/core/lib/iomgr/closure.c + src/core/lib/iomgr/endpoint.c + src/core/lib/iomgr/endpoint_pair_posix.c + src/core/lib/iomgr/endpoint_pair_windows.c + src/core/lib/iomgr/error.c + src/core/lib/iomgr/ev_poll_and_epoll_posix.c + src/core/lib/iomgr/ev_poll_posix.c + src/core/lib/iomgr/ev_posix.c + src/core/lib/iomgr/exec_ctx.c + src/core/lib/iomgr/executor.c + src/core/lib/iomgr/iocp_windows.c + src/core/lib/iomgr/iomgr.c + src/core/lib/iomgr/iomgr_posix.c + src/core/lib/iomgr/iomgr_windows.c + src/core/lib/iomgr/load_file.c + src/core/lib/iomgr/polling_entity.c + src/core/lib/iomgr/pollset_set_windows.c + src/core/lib/iomgr/pollset_windows.c + src/core/lib/iomgr/resolve_address_posix.c + src/core/lib/iomgr/resolve_address_windows.c + src/core/lib/iomgr/sockaddr_utils.c + src/core/lib/iomgr/socket_utils_common_posix.c + src/core/lib/iomgr/socket_utils_linux.c + src/core/lib/iomgr/socket_utils_posix.c + src/core/lib/iomgr/socket_windows.c + src/core/lib/iomgr/tcp_client_posix.c + src/core/lib/iomgr/tcp_client_windows.c + src/core/lib/iomgr/tcp_posix.c + src/core/lib/iomgr/tcp_server_posix.c + src/core/lib/iomgr/tcp_server_windows.c + src/core/lib/iomgr/tcp_windows.c + src/core/lib/iomgr/time_averaged_stats.c + src/core/lib/iomgr/timer.c + src/core/lib/iomgr/timer_heap.c + src/core/lib/iomgr/udp_server.c + src/core/lib/iomgr/unix_sockets_posix.c + src/core/lib/iomgr/unix_sockets_posix_noop.c + src/core/lib/iomgr/wakeup_fd_eventfd.c + src/core/lib/iomgr/wakeup_fd_nospecial.c + src/core/lib/iomgr/wakeup_fd_pipe.c + src/core/lib/iomgr/wakeup_fd_posix.c + src/core/lib/iomgr/workqueue_posix.c + src/core/lib/iomgr/workqueue_windows.c + src/core/lib/json/json.c + src/core/lib/json/json_reader.c + src/core/lib/json/json_string.c + src/core/lib/json/json_writer.c + src/core/lib/surface/alarm.c + src/core/lib/surface/api_trace.c + src/core/lib/surface/byte_buffer.c + src/core/lib/surface/byte_buffer_reader.c + src/core/lib/surface/call.c + src/core/lib/surface/call_details.c + src/core/lib/surface/call_log_batch.c + src/core/lib/surface/channel.c + src/core/lib/surface/channel_init.c + src/core/lib/surface/channel_ping.c + src/core/lib/surface/channel_stack_type.c + src/core/lib/surface/completion_queue.c + src/core/lib/surface/event_string.c + src/core/lib/surface/lame_client.c + src/core/lib/surface/metadata_array.c + src/core/lib/surface/server.c + src/core/lib/surface/validate_metadata.c + src/core/lib/surface/version.c + src/core/lib/transport/byte_stream.c + src/core/lib/transport/connectivity_state.c + src/core/lib/transport/metadata.c + src/core/lib/transport/metadata_batch.c + src/core/lib/transport/static_metadata.c + src/core/lib/transport/transport.c + src/core/lib/transport/transport_op_string.c + src/core/ext/transport/chttp2/server/insecure/server_chttp2.c + src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c + src/core/ext/transport/chttp2/transport/bin_decoder.c + src/core/ext/transport/chttp2/transport/bin_encoder.c + src/core/ext/transport/chttp2/transport/chttp2_plugin.c + src/core/ext/transport/chttp2/transport/chttp2_transport.c + src/core/ext/transport/chttp2/transport/frame_data.c + src/core/ext/transport/chttp2/transport/frame_goaway.c + src/core/ext/transport/chttp2/transport/frame_ping.c + src/core/ext/transport/chttp2/transport/frame_rst_stream.c + src/core/ext/transport/chttp2/transport/frame_settings.c + src/core/ext/transport/chttp2/transport/frame_window_update.c + src/core/ext/transport/chttp2/transport/hpack_encoder.c + src/core/ext/transport/chttp2/transport/hpack_parser.c + src/core/ext/transport/chttp2/transport/hpack_table.c + src/core/ext/transport/chttp2/transport/huffsyms.c + src/core/ext/transport/chttp2/transport/incoming_metadata.c + src/core/ext/transport/chttp2/transport/parsing.c + src/core/ext/transport/chttp2/transport/status_conversion.c + src/core/ext/transport/chttp2/transport/stream_lists.c + src/core/ext/transport/chttp2/transport/stream_map.c + src/core/ext/transport/chttp2/transport/timeout_encoding.c + src/core/ext/transport/chttp2/transport/varint.c + src/core/ext/transport/chttp2/transport/writing.c + src/core/ext/transport/chttp2/alpn/alpn.c + src/core/ext/transport/chttp2/client/insecure/channel_create.c + src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c + src/core/ext/client_config/channel_connectivity.c + src/core/ext/client_config/client_channel.c + src/core/ext/client_config/client_channel_factory.c + src/core/ext/client_config/client_config.c + src/core/ext/client_config/client_config_plugin.c + src/core/ext/client_config/connector.c + src/core/ext/client_config/default_initial_connect_string.c + src/core/ext/client_config/initial_connect_string.c + src/core/ext/client_config/lb_policy.c + src/core/ext/client_config/lb_policy_factory.c + src/core/ext/client_config/lb_policy_registry.c + src/core/ext/client_config/parse_address.c + src/core/ext/client_config/resolver.c + src/core/ext/client_config/resolver_factory.c + src/core/ext/client_config/resolver_registry.c + src/core/ext/client_config/subchannel.c + src/core/ext/client_config/subchannel_call_holder.c + src/core/ext/client_config/subchannel_index.c + src/core/ext/client_config/uri_parser.c + src/core/ext/resolver/dns/native/dns_resolver.c + src/core/ext/resolver/sockaddr/sockaddr_resolver.c + src/core/ext/load_reporting/load_reporting.c + src/core/ext/load_reporting/load_reporting_filter.c + src/core/ext/lb_policy/grpclb/load_balancer_api.c + src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c + third_party/nanopb/pb_common.c + third_party/nanopb/pb_decode.c + third_party/nanopb/pb_encode.c + src/core/ext/lb_policy/pick_first/pick_first.c + src/core/ext/lb_policy/round_robin/round_robin.c + src/core/ext/census/context.c + src/core/ext/census/gen/census.pb.c + src/core/ext/census/grpc_context.c + src/core/ext/census/grpc_filter.c + src/core/ext/census/grpc_plugin.c + src/core/ext/census/initialize.c + src/core/ext/census/mlog.c + src/core/ext/census/operation.c + src/core/ext/census/placeholders.c + src/core/ext/census/tracing.c + src/core/plugin_registry/grpc_unsecure_plugin_registry.c +) + +target_include_directories(grpc_unsecure + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_unsecure + gpr +) + + +add_library(grpc++ + src/cpp/client/secure_credentials.cc + src/cpp/common/auth_property_iterator.cc + src/cpp/common/secure_auth_context.cc + src/cpp/common/secure_channel_arguments.cc + src/cpp/common/secure_create_auth_context.cc + src/cpp/server/secure_server_credentials.cc + src/cpp/client/channel.cc + src/cpp/client/client_context.cc + src/cpp/client/create_channel.cc + src/cpp/client/create_channel_internal.cc + src/cpp/client/create_channel_posix.cc + src/cpp/client/credentials.cc + src/cpp/client/generic_stub.cc + src/cpp/client/insecure_credentials.cc + src/cpp/common/channel_arguments.cc + src/cpp/common/completion_queue.cc + src/cpp/common/core_codegen.cc + src/cpp/common/rpc_method.cc + src/cpp/server/async_generic_service.cc + src/cpp/server/create_default_thread_pool.cc + src/cpp/server/dynamic_thread_pool.cc + src/cpp/server/insecure_server_credentials.cc + src/cpp/server/server.cc + src/cpp/server/server_builder.cc + src/cpp/server/server_context.cc + src/cpp/server/server_credentials.cc + src/cpp/server/server_posix.cc + src/cpp/util/byte_buffer.cc + src/cpp/util/slice.cc + src/cpp/util/status.cc + src/cpp/util/string_ref.cc + src/cpp/util/time.cc + src/cpp/codegen/codegen_init.cc +) + +target_include_directories(grpc++ + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc++ + ssl + libprotobuf + grpc +) + + +add_library(grpc++_reflection + src/cpp/ext/proto_server_reflection.cc + src/cpp/ext/proto_server_reflection_plugin.cc + src/cpp/ext/reflection.grpc.pb.cc + src/cpp/ext/reflection.pb.cc +) + +target_include_directories(grpc++_reflection + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc++_reflection + grpc++ +) + + +add_library(grpc++_unsecure + src/cpp/common/insecure_create_auth_context.cc + src/cpp/client/channel.cc + src/cpp/client/client_context.cc + src/cpp/client/create_channel.cc + src/cpp/client/create_channel_internal.cc + src/cpp/client/create_channel_posix.cc + src/cpp/client/credentials.cc + src/cpp/client/generic_stub.cc + src/cpp/client/insecure_credentials.cc + src/cpp/common/channel_arguments.cc + src/cpp/common/completion_queue.cc + src/cpp/common/core_codegen.cc + src/cpp/common/rpc_method.cc + src/cpp/server/async_generic_service.cc + src/cpp/server/create_default_thread_pool.cc + src/cpp/server/dynamic_thread_pool.cc + src/cpp/server/insecure_server_credentials.cc + src/cpp/server/server.cc + src/cpp/server/server_builder.cc + src/cpp/server/server_context.cc + src/cpp/server/server_credentials.cc + src/cpp/server/server_posix.cc + src/cpp/util/byte_buffer.cc + src/cpp/util/slice.cc + src/cpp/util/status.cc + src/cpp/util/string_ref.cc + src/cpp/util/time.cc + src/cpp/codegen/codegen_init.cc +) + +target_include_directories(grpc++_unsecure + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc++_unsecure + libprotobuf + gpr + grpc_unsecure +) + + +add_library(grpc_plugin_support + src/compiler/cpp_generator.cc + src/compiler/csharp_generator.cc + src/compiler/node_generator.cc + src/compiler/objective_c_generator.cc + src/compiler/python_generator.cc + src/compiler/ruby_generator.cc +) + +target_include_directories(grpc_plugin_support + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_plugin_support + libprotoc +) + + +add_library(grpc_csharp_ext + src/csharp/ext/grpc_csharp_ext.c +) + +target_include_directories(grpc_csharp_ext + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_csharp_ext + grpc + gpr +) + + + +add_executable(grpc_cpp_plugin + src/compiler/cpp_plugin.cc +) + +target_include_directories(grpc_cpp_plugin + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_cpp_plugin + libprotoc + grpc_plugin_support +) + + +add_executable(grpc_csharp_plugin + src/compiler/csharp_plugin.cc +) + +target_include_directories(grpc_csharp_plugin + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_csharp_plugin + libprotoc + grpc_plugin_support +) + + +add_executable(grpc_node_plugin + src/compiler/node_plugin.cc +) + +target_include_directories(grpc_node_plugin + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_node_plugin + libprotoc + grpc_plugin_support +) + + +add_executable(grpc_objective_c_plugin + src/compiler/objective_c_plugin.cc +) + +target_include_directories(grpc_objective_c_plugin + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_objective_c_plugin + libprotoc + grpc_plugin_support +) + + +add_executable(grpc_python_plugin + src/compiler/python_plugin.cc +) + +target_include_directories(grpc_python_plugin + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_python_plugin + libprotoc + grpc_plugin_support +) + + +add_executable(grpc_ruby_plugin + src/compiler/ruby_plugin.cc +) + +target_include_directories(grpc_ruby_plugin + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_ruby_plugin + libprotoc + grpc_plugin_support +) + + + + + diff --git a/templates/CMakeLists.txt.template b/templates/CMakeLists.txt.template new file mode 100644 index 00000000000..00a37de3cb9 --- /dev/null +++ b/templates/CMakeLists.txt.template @@ -0,0 +1,133 @@ +%YAML 1.2 +--- | + # GRPC global cmake file + # This currently builds C and C++ code. + # This file has been automatically generated from a template file. + # Please look at the templates directory instead. + # This file can be regenerated from the template by running + # tools/buildgen/generate_projects.sh + + # Copyright 2015, Google Inc. + # All rights reserved. + # + # Redistribution and use in source and binary forms, with or without + # modification, are permitted provided that the following conditions are + # met: + # + # * Redistributions of source code must retain the above copyright + # notice, this list of conditions and the following disclaimer. + # * Redistributions in binary form must reproduce the above + # copyright notice, this list of conditions and the following disclaimer + # in the documentation and/or other materials provided with the + # distribution. + # * Neither the name of Google Inc. nor the names of its + # contributors may be used to endorse or promote products derived from + # this software without specific prior written permission. + # + # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + <%! + def get_deps(target_dict): + deps = [] + if target_dict.get('build', None) in ['protoc']: + deps.append("libprotoc") + if target_dict.get('secure', False): + deps = ["ssl"] + if target_dict['name'] in ['grpc++', 'grpc++_unsecure', 'grpc++_codegen_lib']: + deps.append("libprotobuf") + for d in target_dict.get('deps', []): + deps.append(d) + return deps + %> + + cmake_minimum_required(VERSION 2.8) + + if (NOT BORINGSSL_ROOT_DIR) + set(BORINGSSL_ROOT_DIR <%text>${CMAKE_CURRENT_SOURCE_DIR}/third_party/boringssl) + endif() + if (NOT PROTOBUF_ROOT_DIR) + set(PROTOBUF_ROOT_DIR <%text>${CMAKE_CURRENT_SOURCE_DIR}/third_party/protobuf) + endif() + if (NOT ZLIB_ROOT_DIR) + set(ZLIB_ROOT_DIR <%text>${CMAKE_CURRENT_SOURCE_DIR}/third_party/zlib) + endif() + + add_subdirectory(<%text>${BORINGSSL_ROOT_DIR} third_party/boringssl) + add_subdirectory(<%text>${PROTOBUF_ROOT_DIR}/cmake third_party/protobuf) + add_subdirectory(<%text>${ZLIB_ROOT_DIR} third_party/zlib) + + set(CMAKE_C_FLAGS "<%text>${CMAKE_C_FLAGS} -std=c11") + set(CMAKE_CXX_FLAGS "<%text>${CMAKE_CXX_FLAGS} -std=c++11") + + % for lib in libs: + % if lib.build in ["all", "protoc"]: + ${cc_library(lib)} + % endif + % endfor + + % for tgt in targets: + % if tgt.build == 'protoc': + ${cc_binary(tgt)} + % endif + % endfor + + <%def name="cc_library(lib)"> + add_library(${lib.name} + % for src in lib.src: + ${src} + % endfor + ) + + target_include_directories(${lib.name} + PRIVATE <%text>${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE <%text>${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE <%text>${BORINGSSL_ROOT_DIR}/include + PRIVATE <%text>${PROTOBUF_ROOT_DIR}/src + PRIVATE <%text>${ZLIB_ROOT_DIR} + PRIVATE <%text>${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib + ) + + % if len(get_deps(lib)) > 0: + target_link_libraries(${lib.name} + % for dep in get_deps(lib): + ${dep} + % endfor + ) + % endif + + + <%def name="cc_binary(tgt)"> + add_executable(${tgt.name} + % for src in tgt.src: + ${src} + % endfor + ) + + target_include_directories(${tgt.name} + PRIVATE <%text>${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE <%text>${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE <%text>${BORINGSSL_ROOT_DIR}/include + PRIVATE <%text>${PROTOBUF_ROOT_DIR}/src + PRIVATE <%text>${ZLIB_ROOT_DIR} + PRIVATE <%text>${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib + ) + + % if len(get_deps(tgt)) > 0: + target_link_libraries(${tgt.name} + % for dep in get_deps(tgt): + ${dep} + % endfor + ) + % endif + + From cddf697ab44a7bab1821915e1e3f6a0f08ca1706 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Tue, 21 Jun 2016 08:27:07 -0700 Subject: [PATCH 0140/1039] Fix refcounting tsan failures and grab pollset lock in the function pollset_add_fd --- src/core/lib/iomgr/ev_epoll_linux.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 72288889c02..7cc69c876db 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -291,11 +291,11 @@ void pi_unref_dbg(polling_island *pi, int ref_cnt, char *reason, char *file, #endif long pi_add_ref(polling_island *pi, int ref_cnt) { - return gpr_atm_no_barrier_fetch_add(&pi->ref_count, ref_cnt); + return gpr_atm_full_fetch_add(&pi->ref_count, ref_cnt); } long pi_unref(polling_island *pi, int ref_cnt) { - long old_cnt = gpr_atm_no_barrier_fetch_add(&pi->ref_count, -ref_cnt); + long old_cnt = gpr_atm_full_fetch_add(&pi->ref_count, -ref_cnt); /* If ref count went to zero, delete the polling island. Note that this need not be done under a lock. Once the ref count goes to zero, we are @@ -311,6 +311,8 @@ long pi_unref(polling_island *pi, int ref_cnt) { if (next != NULL) { PI_UNREF(next, "pi_delete"); /* Recursive call */ } + } else { + GPR_ASSERT(old_cnt > ref_cnt); } return old_cnt; @@ -445,8 +447,8 @@ static polling_island *polling_island_create(grpc_fd *initial_fd) { pi->fds = NULL; } - gpr_atm_no_barrier_store(&pi->ref_count, 0); - gpr_atm_no_barrier_store(&pi->merged_to, NULL); + gpr_atm_rel_store(&pi->ref_count, 0); + gpr_atm_rel_store(&pi->merged_to, NULL); pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); @@ -1347,7 +1349,7 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { - /* TODO sreek - Double check if we need to get a pollset->mu lock here */ + gpr_mu_lock(&pollset->mu); gpr_mu_lock(&pollset->pi_mu); gpr_mu_lock(&fd->pi_mu); @@ -1401,6 +1403,7 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, gpr_mu_unlock(&fd->pi_mu); gpr_mu_unlock(&pollset->pi_mu); + gpr_mu_unlock(&pollset->mu); } /******************************************************************************* From e63246d1007516ca5c74043139304a371c0ce672 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Tue, 21 Jun 2016 09:03:11 -0700 Subject: [PATCH 0141/1039] clang-format --- test/core/iomgr/tcp_server_posix_test.c | 2 +- test/core/surface/server_chttp2_test.c | 2 +- test/core/util/test_tcp_server.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/test/core/iomgr/tcp_server_posix_test.c b/test/core/iomgr/tcp_server_posix_test.c index 6361e7afcbc..6e2d1d0fc9e 100644 --- a/test/core/iomgr/tcp_server_posix_test.c +++ b/test/core/iomgr/tcp_server_posix_test.c @@ -166,7 +166,7 @@ static void test_no_op_with_port_and_start(void) { grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; struct sockaddr_in addr; grpc_tcp_server *s; - GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, NULL, &s)); + GPR_ASSERT(GRPC_ERROR_NONE == grpc_tcp_server_create(NULL, NULL, &s)); LOG_TEST("test_no_op_with_port_and_start"); int port; diff --git a/test/core/surface/server_chttp2_test.c b/test/core/surface/server_chttp2_test.c index 40cfa6b5989..6310b6f00b0 100644 --- a/test/core/surface/server_chttp2_test.c +++ b/test/core/surface/server_chttp2_test.c @@ -54,7 +54,7 @@ void test_add_same_port_twice() { a.key = GRPC_ARG_ALLOW_REUSEPORT; a.value.integer = 0; grpc_channel_args args = {1, &a}; - + int port = grpc_pick_unused_port_or_die(); char *addr = NULL; grpc_completion_queue *cq = grpc_completion_queue_create(NULL); diff --git a/test/core/util/test_tcp_server.c b/test/core/util/test_tcp_server.c index c1d307fc779..8a0b3932d86 100644 --- a/test/core/util/test_tcp_server.c +++ b/test/core/util/test_tcp_server.c @@ -72,8 +72,8 @@ void test_tcp_server_start(test_tcp_server *server, int port) { addr.sin_port = htons((uint16_t)port); memset(&addr.sin_addr, 0, sizeof(addr.sin_addr)); - grpc_error *error = - grpc_tcp_server_create(&server->shutdown_complete, NULL, &server->tcp_server); + grpc_error *error = grpc_tcp_server_create(&server->shutdown_complete, NULL, + &server->tcp_server); GPR_ASSERT(error == GRPC_ERROR_NONE); error = grpc_tcp_server_add_port(server->tcp_server, &addr, sizeof(addr), &port_added); From 39f9ac9b2a51fdcdbd3693d8a04eec3473eb9438 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 16 Jun 2016 16:53:59 -0700 Subject: [PATCH 0142/1039] Test polling island merges --- Makefile | 36 ++++ build.yaml | 12 ++ src/core/lib/iomgr/ev_epoll_linux.c | 51 +++++- src/core/lib/iomgr/ev_epoll_linux.h | 6 + src/core/lib/iomgr/ev_posix.c | 7 + src/core/lib/iomgr/ev_posix.h | 3 + test/core/iomgr/ev_epoll_linux_test.c | 222 +++++++++++++++++++++++ tools/run_tests/sources_and_headers.json | 16 ++ tools/run_tests/tests.json | 15 ++ 9 files changed, 366 insertions(+), 2 deletions(-) create mode 100644 test/core/iomgr/ev_epoll_linux_test.c diff --git a/Makefile b/Makefile index e615704395b..825684cc2dd 100644 --- a/Makefile +++ b/Makefile @@ -905,6 +905,7 @@ dns_resolver_connectivity_test: $(BINDIR)/$(CONFIG)/dns_resolver_connectivity_te dns_resolver_test: $(BINDIR)/$(CONFIG)/dns_resolver_test dualstack_socket_test: $(BINDIR)/$(CONFIG)/dualstack_socket_test endpoint_pair_test: $(BINDIR)/$(CONFIG)/endpoint_pair_test +ev_epoll_linux_test: $(BINDIR)/$(CONFIG)/ev_epoll_linux_test fd_conservation_posix_test: $(BINDIR)/$(CONFIG)/fd_conservation_posix_test fd_posix_test: $(BINDIR)/$(CONFIG)/fd_posix_test fling_client: $(BINDIR)/$(CONFIG)/fling_client @@ -1242,6 +1243,7 @@ buildtests_c: privatelibs_c \ $(BINDIR)/$(CONFIG)/dns_resolver_test \ $(BINDIR)/$(CONFIG)/dualstack_socket_test \ $(BINDIR)/$(CONFIG)/endpoint_pair_test \ + $(BINDIR)/$(CONFIG)/ev_epoll_linux_test \ $(BINDIR)/$(CONFIG)/fd_conservation_posix_test \ $(BINDIR)/$(CONFIG)/fd_posix_test \ $(BINDIR)/$(CONFIG)/fling_client \ @@ -1512,6 +1514,8 @@ test_c: buildtests_c $(Q) $(BINDIR)/$(CONFIG)/dualstack_socket_test || ( echo test dualstack_socket_test failed ; exit 1 ) $(E) "[RUN] Testing endpoint_pair_test" $(Q) $(BINDIR)/$(CONFIG)/endpoint_pair_test || ( echo test endpoint_pair_test failed ; exit 1 ) + $(E) "[RUN] Testing ev_epoll_linux_test" + $(Q) $(BINDIR)/$(CONFIG)/ev_epoll_linux_test || ( echo test ev_epoll_linux_test failed ; exit 1 ) $(E) "[RUN] Testing fd_conservation_posix_test" $(Q) $(BINDIR)/$(CONFIG)/fd_conservation_posix_test || ( echo test fd_conservation_posix_test failed ; exit 1 ) $(E) "[RUN] Testing fd_posix_test" @@ -7130,6 +7134,38 @@ endif endif +EV_EPOLL_LINUX_TEST_SRC = \ + test/core/iomgr/ev_epoll_linux_test.c \ + +EV_EPOLL_LINUX_TEST_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(EV_EPOLL_LINUX_TEST_SRC)))) +ifeq ($(NO_SECURE),true) + +# You can't build secure targets if you don't have OpenSSL. + +$(BINDIR)/$(CONFIG)/ev_epoll_linux_test: openssl_dep_error + +else + + + +$(BINDIR)/$(CONFIG)/ev_epoll_linux_test: $(EV_EPOLL_LINUX_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + $(E) "[LD] Linking $@" + $(Q) mkdir -p `dirname $@` + $(Q) $(LD) $(LDFLAGS) $(EV_EPOLL_LINUX_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/ev_epoll_linux_test + +endif + +$(OBJDIR)/$(CONFIG)/test/core/iomgr/ev_epoll_linux_test.o: $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a + +deps_ev_epoll_linux_test: $(EV_EPOLL_LINUX_TEST_OBJS:.o=.dep) + +ifneq ($(NO_SECURE),true) +ifneq ($(NO_DEPS),true) +-include $(EV_EPOLL_LINUX_TEST_OBJS:.o=.dep) +endif +endif + + FD_CONSERVATION_POSIX_TEST_SRC = \ test/core/iomgr/fd_conservation_posix_test.c \ diff --git a/build.yaml b/build.yaml index 7790e0c5174..84f4ea521be 100644 --- a/build.yaml +++ b/build.yaml @@ -1407,6 +1407,18 @@ targets: - grpc - gpr_test_util - gpr +- name: ev_epoll_linux_test + build: test + language: c + src: + - test/core/iomgr/ev_epoll_linux_test.c + deps: + - grpc_test_util + - grpc + - gpr_test_util + - gpr + platforms: + - linux - name: fd_conservation_posix_test build: test language: c diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 1fb59474640..ed2c494b783 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -317,8 +317,9 @@ static void polling_island_remove_all_fds_locked(polling_island *pi, if (err < 0 && errno != ENOENT) { /* TODO: sreek - We need a better way to bubble up this error instead of * just logging a message */ - gpr_log(GPR_ERROR, "epoll_ctl deleting fds[%zu]: %d failed with error: %s", - i, pi->fds[i]->fd, strerror(errno)); + gpr_log(GPR_ERROR, + "epoll_ctl deleting fds[%zu]: %d failed with error: %s", i, + pi->fds[i]->fd, strerror(errno)); } if (remove_fd_refs) { @@ -1458,6 +1459,52 @@ static void pollset_set_del_pollset_set(grpc_exec_ctx *exec_ctx, gpr_mu_unlock(&bag->mu); } +/* Test helper functions + * */ +void *grpc_fd_get_polling_island(grpc_fd *fd) { + polling_island *pi; + + gpr_mu_lock(&fd->pi_mu); + pi = fd->polling_island; + gpr_mu_unlock(&fd->pi_mu); + + return pi; +} + +void *grpc_pollset_get_polling_island(grpc_pollset *ps) { + polling_island *pi; + + gpr_mu_lock(&ps->pi_mu); + pi = ps->polling_island; + gpr_mu_unlock(&ps->pi_mu); + + return pi; +} + +static polling_island *get_polling_island(polling_island *p) { + if (p == NULL) { + return NULL; + } + + polling_island *next; + gpr_mu_lock(&p->mu); + while (p->merged_to != NULL) { + next = p->merged_to; + gpr_mu_unlock(&p->mu); + p = next; + gpr_mu_lock(&p->mu); + } + gpr_mu_unlock(&p->mu); + + return p; +} + +bool grpc_are_polling_islands_equal(void *p, void *q) { + p = get_polling_island(p); + q = get_polling_island(q); + return p == q; +} + /******************************************************************************* * Event engine binding */ diff --git a/src/core/lib/iomgr/ev_epoll_linux.h b/src/core/lib/iomgr/ev_epoll_linux.h index 8c819975a4c..7a494aba198 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.h +++ b/src/core/lib/iomgr/ev_epoll_linux.h @@ -38,4 +38,10 @@ const grpc_event_engine_vtable *grpc_init_epoll_linux(void); +#ifdef GPR_LINUX_EPOLL +void *grpc_fd_get_polling_island(grpc_fd *fd); +void *grpc_pollset_get_polling_island(grpc_pollset *ps); +bool grpc_are_polling_islands_equal(void *p, void *q); +#endif /* defined(GPR_LINUX_EPOLL) */ + #endif /* GRPC_CORE_LIB_IOMGR_EV_EPOLL_LINUX_H */ diff --git a/src/core/lib/iomgr/ev_posix.c b/src/core/lib/iomgr/ev_posix.c index 2b15967adcc..5b20600a6f7 100644 --- a/src/core/lib/iomgr/ev_posix.c +++ b/src/core/lib/iomgr/ev_posix.c @@ -54,6 +54,7 @@ grpc_poll_function_type grpc_poll_function = poll; static const grpc_event_engine_vtable *g_event_engine; +static const char* g_poll_strategy_name = NULL; typedef const grpc_event_engine_vtable *(*event_engine_factory_fn)(void); @@ -101,6 +102,7 @@ static void try_engine(const char *engine) { for (size_t i = 0; i < GPR_ARRAY_SIZE(g_factories); i++) { if (is(engine, g_factories[i].name)) { if ((g_event_engine = g_factories[i].factory())) { + g_poll_strategy_name = g_factories[i].name; gpr_log(GPR_DEBUG, "Using polling engine: %s", g_factories[i].name); return; } @@ -108,6 +110,11 @@ static void try_engine(const char *engine) { } } +/* Call this only after calling grpc_event_engine_init() */ +const char *grpc_get_poll_strategy_name() { + return g_poll_strategy_name; +} + void grpc_event_engine_init(void) { char *s = gpr_getenv("GRPC_POLL_STRATEGY"); if (s == NULL) { diff --git a/src/core/lib/iomgr/ev_posix.h b/src/core/lib/iomgr/ev_posix.h index 344bf63438a..3ed5a5f9562 100644 --- a/src/core/lib/iomgr/ev_posix.h +++ b/src/core/lib/iomgr/ev_posix.h @@ -98,6 +98,9 @@ typedef struct grpc_event_engine_vtable { void grpc_event_engine_init(void); void grpc_event_engine_shutdown(void); +/* Return the name of the poll strategy */ +const char* grpc_get_poll_strategy_name(); + /* Create a wrapped file descriptor. Requires fd is a non-blocking file descriptor. This takes ownership of closing fd. */ diff --git a/test/core/iomgr/ev_epoll_linux_test.c b/test/core/iomgr/ev_epoll_linux_test.c new file mode 100644 index 00000000000..51da15faa7a --- /dev/null +++ b/test/core/iomgr/ev_epoll_linux_test.c @@ -0,0 +1,222 @@ +/* + * + * Copyright 2015, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include "src/core/lib/iomgr/ev_epoll_linux.h" +#include "src/core/lib/iomgr/ev_posix.h" + +#include +#include +#include +#include + +#include +#include + +#include "src/core/lib/iomgr/iomgr.h" +#include "test/core/util/test_config.h" + +typedef struct test_pollset { + grpc_pollset *pollset; + gpr_mu *mu; +} test_pollset; + +typedef struct test_fd { + int inner_fd; + grpc_fd *fd; +} test_fd; + +static void test_fd_init(test_fd *fds, int num_fds) { + int i; + for (i = 0; i < num_fds; i++) { + fds[i].inner_fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + fds[i].fd = grpc_fd_create(fds[i].inner_fd, "test_fd"); + } +} + +static void test_fd_cleanup(grpc_exec_ctx *exec_ctx, test_fd *fds, + int num_fds) { + int release_fd; + int i; + + for (i = 0; i < num_fds; i++) { + grpc_fd_shutdown(exec_ctx, fds[i].fd); + grpc_exec_ctx_flush(exec_ctx); + + grpc_fd_orphan(exec_ctx, fds[i].fd, NULL, &release_fd, "test_fd_cleanup"); + grpc_exec_ctx_flush(exec_ctx); + + GPR_ASSERT(release_fd == fds[i].inner_fd); + close(fds[i].inner_fd); + } +} + +static void test_pollset_init(test_pollset *pollsets, int num_pollsets) { + int i; + for (i = 0; i < num_pollsets; i++) { + pollsets[i].pollset = gpr_malloc(grpc_pollset_size()); + grpc_pollset_init(pollsets[i].pollset, &pollsets[i].mu); + } +} + +static void destroy_pollset(grpc_exec_ctx *exec_ctx, void *p, bool success) { + grpc_pollset_destroy(p); +} + +static void test_pollset_cleanup(grpc_exec_ctx *exec_ctx, + test_pollset *pollsets, int num_pollsets) { + grpc_closure destroyed; + int i; + + for (i = 0; i < num_pollsets; i++) { + grpc_closure_init(&destroyed, destroy_pollset, pollsets[i].pollset); + grpc_pollset_shutdown(exec_ctx, pollsets[i].pollset, &destroyed); + + grpc_exec_ctx_flush(exec_ctx); + gpr_free(pollsets[i].pollset); + } +} + +#define NUM_FDS 8 +#define NUM_POLLSETS 4 +/* + * Cases to test: + * case 1) Polling islands of both fd and pollset are NULL + * case 2) Polling island of fd is NULL but that of pollset is not-NULL + * case 3) Polling island of fd is not-NULL but that of pollset is NULL + * case 4) Polling islands of both fd and pollset are not-NULL and: + * case 4.1) Polling islands of fd and pollset are equal + * case 4.2) Polling islands of fd and pollset are NOT-equal (This results + * in a merge) + * */ +static void test_add_fd_to_pollset() { + grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; + test_fd fds[NUM_FDS]; + test_pollset pollsets[NUM_POLLSETS]; + void *expected_pi = NULL; + int i; + + test_fd_init(fds, NUM_FDS); + test_pollset_init(pollsets, NUM_POLLSETS); + + /*Step 1. + * Create three polling islands (This will exercise test case 1 and 2) with + * the following configuration: + * polling island 0 = { fds:0,1,2, pollsets:0} + * polling island 1 = { fds:3,4, pollsets:1} + * polling island 2 = { fds:5,6,7 pollsets:2} + * + *Step 2. + * Add pollset 3 to polling island 0 (by adding fds 0 and 1 to pollset 3) + * (This will exercise test cases 3 and 4.1). The configuration becomes: + * polling island 0 = { fds:0,1,2, pollsets:0,3} <<< pollset 3 added here + * polling island 1 = { fds:3,4, pollsets:1} + * polling island 2 = { fds:5,6,7 pollsets:2} + * + *Step 3. + * Merge polling islands 0 and 1 by adding fd 0 to pollset 1 (This will + * exercise test case 4.2). The configuration becomes: + * polling island (merged) = {fds: 0,1,2,3,4, pollsets: 0,1,3} + * polling island 2 = {fds: 5,6,7 pollsets: 2} + * + *Step 4. + * Finally do one more merge by adding fd 3 to pollset 2. + * polling island (merged) = {fds: 0,1,2,3,4,5,6,7, pollsets: 0,1,2,3} + */ + + /* == Step 1 == */ + for (i = 0; i <= 2; i++) { + grpc_pollset_add_fd(&exec_ctx, pollsets[0].pollset, fds[i].fd); + grpc_exec_ctx_flush(&exec_ctx); + } + + for (i = 3; i <= 4; i++) { + grpc_pollset_add_fd(&exec_ctx, pollsets[1].pollset, fds[i].fd); + grpc_exec_ctx_flush(&exec_ctx); + } + + for (i = 5; i <= 7; i++) { + grpc_pollset_add_fd(&exec_ctx, pollsets[2].pollset, fds[i].fd); + grpc_exec_ctx_flush(&exec_ctx); + } + + /* == Step 2 == */ + for (i = 0; i <= 1; i++) { + grpc_pollset_add_fd(&exec_ctx, pollsets[3].pollset, fds[i].fd); + grpc_exec_ctx_flush(&exec_ctx); + } + + /* == Step 3 == */ + grpc_pollset_add_fd(&exec_ctx, pollsets[1].pollset, fds[0].fd); + grpc_exec_ctx_flush(&exec_ctx); + + /* == Step 4 == */ + grpc_pollset_add_fd(&exec_ctx, pollsets[2].pollset, fds[3].fd); + grpc_exec_ctx_flush(&exec_ctx); + + /* All polling islands are merged at this point */ + + /* Compare Fd:0's polling island with that of all other Fds */ + expected_pi = grpc_fd_get_polling_island(fds[0].fd); + for (i = 1; i < NUM_FDS; i++) { + GPR_ASSERT(grpc_are_polling_islands_equal( + expected_pi, grpc_fd_get_polling_island(fds[i].fd))); + } + + /* Compare Fd:0's polling island with that of all other pollsets */ + for (i = 0; i < NUM_POLLSETS; i++) { + GPR_ASSERT(grpc_are_polling_islands_equal( + expected_pi, grpc_pollset_get_polling_island(pollsets[i].pollset))); + } + + test_fd_cleanup(&exec_ctx, fds, NUM_FDS); + test_pollset_cleanup(&exec_ctx, pollsets, NUM_POLLSETS); + grpc_exec_ctx_finish(&exec_ctx); +} + +int main(int argc, char **argv) { + const char *poll_strategy = NULL; + grpc_test_init(argc, argv); + grpc_iomgr_init(); + + poll_strategy = grpc_get_poll_strategy_name(); + if (poll_strategy != NULL && strcmp(poll_strategy, "epoll") == 0) { + test_add_fd_to_pollset(); + } else { + gpr_log(GPR_INFO, + "Skipping the test. The test is only relevant for 'epoll' " + "strategy. and the current strategy is: '%s'", + poll_strategy); + } + grpc_iomgr_shutdown(); + return 0; +} diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index e8ff61dc3fb..e9df72e43a1 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -315,6 +315,22 @@ "third_party": false, "type": "target" }, + { + "deps": [ + "gpr", + "gpr_test_util", + "grpc", + "grpc_test_util" + ], + "headers": [], + "language": "c", + "name": "ev_epoll_linux_test", + "src": [ + "test/core/iomgr/ev_epoll_linux_test.c" + ], + "third_party": false, + "type": "target" + }, { "deps": [ "gpr", diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index 5a84a41b638..ba661840dad 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -377,6 +377,21 @@ "windows" ] }, + { + "args": [], + "ci_platforms": [ + "linux" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "gtest": false, + "language": "c", + "name": "ev_epoll_linux_test", + "platforms": [ + "linux" + ] + }, { "args": [], "ci_platforms": [ From 94cda1a9c6ae86ab176d357b8822332d70283cde Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 17 Jun 2016 13:28:38 -0700 Subject: [PATCH 0143/1039] Significantly refactor the polling island locking and refcounting code --- src/core/lib/iomgr/ev_epoll_linux.c | 462 ++++++++++++++++------------ 1 file changed, 270 insertions(+), 192 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index ed2c494b783..72288889c02 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -140,18 +140,40 @@ static void fd_global_shutdown(void); #define CLOSURE_READY ((grpc_closure *)1) /******************************************************************************* - * Polling-island Declarations + * Polling island Declarations */ -/* TODO: sree: Consider making ref_cnt and merged_to to gpr_atm - This would - * significantly reduce the number of mutex acquisition calls. */ + +// #define GRPC_PI_REF_COUNT_DEBUG +#ifdef GRPC_PI_REF_COUNT_DEBUG + +#define PI_ADD_REF(p, r) pi_add_ref_dbg((p), 1, (r), __FILE__, __LINE__) +#define PI_UNREF(p, r) pi_unref_dbg((p), 1, (r), __FILE__, __LINE__) + +#else /* defined(GRPC_PI_REF_COUNT_DEBUG) */ + +#define PI_ADD_REF(p, r) pi_add_ref((p), 1) +#define PI_UNREF(p, r) pi_unref((p), 1) + +#endif /* !defined(GPRC_PI_REF_COUNT_DEBUG) */ + typedef struct polling_island { gpr_mu mu; - int ref_cnt; - - /* Points to the polling_island this merged into. - * If merged_to is not NULL, all the remaining fields (except mu and ref_cnt) - * are invalid and must be ignored */ - struct polling_island *merged_to; + /* Ref count. Use PI_ADD_REF() and PI_UNREF() macros to increment/decrement + the refcount. + Once the ref count becomes zero, this structure is destroyed which means + we should ensure that there is never a scenario where a PI_ADD_REF() is + racing with a PI_UNREF() that just made the ref_count zero. */ + gpr_atm ref_count; + + /* Pointer to the polling_island this merged into. + * merged_to value is only set once in polling_island's lifetime (and that too + * only if the island is merged with another island). Because of this, we can + * use gpr_atm type here so that we can do atomic access on this and reduce + * lock contention on 'mu' mutex. + * + * Note that if this field is not NULL (i.e not 0), all the remaining fields + * (except mu and ref_count) are invalid and must be ignored. */ + gpr_atm merged_to; /* The fd of the underlying epoll set */ int epoll_fd; @@ -236,6 +258,8 @@ static grpc_wakeup_fd polling_island_wakeup_fd; static gpr_mu g_pi_freelist_mu; static polling_island *g_pi_freelist = NULL; +static void polling_island_delete(); /* Forward declaration */ + #ifdef GRPC_TSAN /* Currently TSAN may incorrectly flag data races between epoll_ctl and epoll_wait for any grpc_fd structs that are added to the epoll set via @@ -247,6 +271,51 @@ static polling_island *g_pi_freelist = NULL; gpr_atm g_epoll_sync; #endif /* defined(GRPC_TSAN) */ +#ifdef GRPC_PI_REF_COUNT_DEBUG +long pi_add_ref(polling_island *pi, int ref_cnt); +long pi_unref(polling_island *pi, int ref_cnt); + +void pi_add_ref_dbg(polling_island *pi, int ref_cnt, char *reason, char *file, + int line) { + long old_cnt = pi_add_ref(pi, ref_cnt); + gpr_log(GPR_DEBUG, "Add ref pi: %p, old:%ld -> new:%ld (%s) - (%s, %d)", + (void *)pi, old_cnt, (old_cnt + ref_cnt), reason, file, line); +} + +void pi_unref_dbg(polling_island *pi, int ref_cnt, char *reason, char *file, + int line) { + long old_cnt = pi_unref(pi, ref_cnt); + gpr_log(GPR_DEBUG, "Unref pi: %p, old:%ld -> new:%ld (%s) - (%s, %d)", + (void *)pi, old_cnt, (old_cnt - ref_cnt), reason, file, line); +} +#endif + +long pi_add_ref(polling_island *pi, int ref_cnt) { + return gpr_atm_no_barrier_fetch_add(&pi->ref_count, ref_cnt); +} + +long pi_unref(polling_island *pi, int ref_cnt) { + long old_cnt = gpr_atm_no_barrier_fetch_add(&pi->ref_count, -ref_cnt); + + /* If ref count went to zero, delete the polling island. Note that this need + not be done under a lock. Once the ref count goes to zero, we are + guaranteed that no one else holds a reference to the polling island (and + that there is no racing pi_add_ref() call either. + + Also, if we are deleting the polling island and the merged_to field is + non-empty, we should remove a ref to the merged_to polling island + */ + if (old_cnt == ref_cnt) { + polling_island *next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); + polling_island_delete(pi); + if (next != NULL) { + PI_UNREF(next, "pi_delete"); /* Recursive call */ + } + } + + return old_cnt; +} + /* The caller is expected to hold pi->mu lock before calling this function */ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, size_t fd_count, bool add_fd_refs) { @@ -355,8 +424,7 @@ static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd, } } -static polling_island *polling_island_create(grpc_fd *initial_fd, - int initial_ref_cnt) { +static polling_island *polling_island_create(grpc_fd *initial_fd) { polling_island *pi = NULL; /* Try to get one from the polling island freelist */ @@ -377,6 +445,9 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, pi->fds = NULL; } + gpr_atm_no_barrier_store(&pi->ref_count, 0); + gpr_atm_no_barrier_store(&pi->merged_to, NULL); + pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); if (pi->epoll_fd < 0) { @@ -387,14 +458,12 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, polling_island_add_wakeup_fd_locked(pi, &grpc_global_wakeup_fd); - pi->ref_cnt = initial_ref_cnt; - pi->merged_to = NULL; pi->next_free = NULL; if (initial_fd != NULL) { - /* It is not really needed to get the pi->mu lock here. If this is a newly - created polling island (or one that we got from the freelist), no one - else would be holding a lock to it anyway */ + /* Lock the polling island here just in case we got this structure from the + freelist and the polling island lock was not released yet (by the code + that adds the polling island to the freelist) */ gpr_mu_lock(&pi->mu); polling_island_add_fds_locked(pi, &initial_fd, 1, true); gpr_mu_unlock(&pi->mu); @@ -404,140 +473,136 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, } static void polling_island_delete(polling_island *pi) { - GPR_ASSERT(pi->ref_cnt == 0); GPR_ASSERT(pi->fd_cnt == 0); + gpr_atm_rel_store(&pi->merged_to, NULL); + close(pi->epoll_fd); pi->epoll_fd = -1; - pi->merged_to = NULL; - gpr_mu_lock(&g_pi_freelist_mu); pi->next_free = g_pi_freelist; g_pi_freelist = pi; gpr_mu_unlock(&g_pi_freelist_mu); } -void polling_island_unref_and_unlock(polling_island *pi, int unref_by) { - pi->ref_cnt -= unref_by; - int ref_cnt = pi->ref_cnt; - GPR_ASSERT(ref_cnt >= 0); - - gpr_mu_unlock(&pi->mu); - - if (ref_cnt == 0) { - polling_island_delete(pi); - } -} - -polling_island *polling_island_update_and_lock(polling_island *pi, int unref_by, - int add_ref_by) { +/* Gets the lock on the *latest* polling island i.e the last polling island in + the linked list (linked by 'merged_to' link). Call gpr_mu_unlock on the + returned polling island's mu. + Usage: To lock/unlock polling island "pi", do the following: + polling_island *pi_latest = polling_island_lock(pi); + ... + ... critical section .. + ... + gpr_mu_unlock(&pi_latest->mu); //NOTE: use pi_latest->mu. NOT pi->mu */ +polling_island *polling_island_lock(polling_island *pi) { polling_island *next = NULL; - gpr_mu_lock(&pi->mu); - while (pi->merged_to != NULL) { - next = pi->merged_to; - polling_island_unref_and_unlock(pi, unref_by); + while (true) { + next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); + if (next == NULL) { + /* pi is the last node in the linked list. Get the lock and check again + (under the pi->mu lock) that pi is still the last node (because a merge + may have happend after the (next == NULL) check above and before + getting the pi->mu lock. + If pi is the last node, we are done. If not, unlock and continue + traversing the list */ + gpr_mu_lock(&pi->mu); + next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); + if (next == NULL) { + break; + } + gpr_mu_unlock(&pi->mu); + } + pi = next; - gpr_mu_lock(&pi->mu); } - pi->ref_cnt += add_ref_by; return pi; } -void polling_island_pair_update_and_lock(polling_island **p, - polling_island **q) { +/* Gets the lock on the *latest* polling islands pointed by *p and *q. + This function is needed because calling the following block of code to obtain + locks on polling islands (*p and *q) is prone to deadlocks. + { + polling_island_lock(*p); + polling_island_lock(*q); + } + + Usage/exmaple: + polling_island *p1; + polling_island *p2; + .. + polling_island_lock_pair(&p1, &p2); + .. + .. Critical section with both p1 and p2 locked + .. + // Release locks + // **IMPORTANT**: Make sure you check p1 == p2 AFTER the function + // polling_island_lock_pair() was called and if so, release the lock only + // once. Note: Even if p1 != p2 beforec calling polling_island_lock_pair(), + // they might be after the function returns: + if (p1 == p2) { + gpr_mu_unlock(&p1->mu) + } else { + gpr_mu_unlock(&p1->mu); + gpr_mu_unlock(&p2->mu); + } + +*/ +void polling_island_lock_pair(polling_island **p, polling_island **q) { polling_island *pi_1 = *p; polling_island *pi_2 = *q; - polling_island *temp = NULL; - bool pi_1_locked = false; - bool pi_2_locked = false; - int num_swaps = 0; - - /* Loop until either pi_1 == pi_2 or until we acquired locks on both pi_1 - and pi_2 */ - while (pi_1 != pi_2 && !(pi_1_locked && pi_2_locked)) { - /* The following assertions are true at this point: - - pi_1 != pi_2 (else, the while loop would have exited) - - pi_1 MAY be locked - - pi_2 is NOT locked */ - - /* To maintain lock order consistency, always lock polling_island node with - lower address first. - First, make sure pi_1 < pi_2 before proceeding any further. If it turns - out that pi_1 > pi_2, unlock pi_1 if locked (because pi_2 is not locked - at this point and having pi_1 locked would violate the lock order) and - swap pi_1 and pi_2 so that pi_1 becomes less than pi_2 */ - if (pi_1 > pi_2) { - if (pi_1_locked) { - gpr_mu_unlock(&pi_1->mu); - pi_1_locked = false; - } + polling_island *next_1 = NULL; + polling_island *next_2 = NULL; + + /* The algorithm is simple: + - Go to the last polling islands in the linked lists *pi_1 and *pi_2 (and + keep updating pi_1 and pi_2) + - Then obtain locks on the islands by following a lock order rule of + locking polling_island with lower address first + Special case: Before obtaining the locks, check if pi_1 and pi_2 are + pointing to the same island. If that is the case, we can just call + polling_island_lock() + - After obtaining both the locks, double check that the polling islands + are still the last polling islands in their respective linked lists + (this is because there might have been polling island merges before + we got the lock) + - If the polling islands are the last islands, we are done. If not, + release the locks and continue the process from the first step */ + while (true) { + next_1 = (polling_island *)gpr_atm_acq_load(&pi_1->merged_to); + while (next_1 != NULL) { + pi_1 = next_1; + next_1 = (polling_island *)gpr_atm_acq_load(&pi_1->merged_to); + } - GPR_SWAP(polling_island *, pi_1, pi_2); - num_swaps++; + next_2 = (polling_island *)gpr_atm_acq_load(&pi_2->merged_to); + while (next_2 != NULL) { + pi_2 = next_2; + next_2 = (polling_island *)gpr_atm_acq_load(&pi_2->merged_to); } - /* The following assertions are true at this point: - - pi_1 != pi_2 - - pi_1 < pi_2 (address of pi_1 is less than that of pi_2) - - pi_1 MAYBE locked - - pi_2 is NOT locked */ + if (pi_1 == pi_2) { + pi_1 = pi_2 = polling_island_lock(pi_1); + break; + } - /* Lock pi_1 (if pi_1 is pointing to the terminal node in the list) */ - if (!pi_1_locked) { + if (pi_1 < pi_2) { + gpr_mu_lock(&pi_1->mu); + gpr_mu_lock(&pi_2->mu); + } else { + gpr_mu_lock(&pi_2->mu); gpr_mu_lock(&pi_1->mu); - pi_1_locked = true; - - /* If pi_1 is not terminal node (i.e pi_1->merged_to != NULL), we are not - done locking this polling_island yet. Release the lock on this node and - advance pi_1 to the next node in the list; and go to the beginning of - the loop (we can't proceed to locking pi_2 unless we locked pi_1 first) - */ - if (pi_1->merged_to != NULL) { - temp = pi_1->merged_to; - polling_island_unref_and_unlock(pi_1, 1); - pi_1 = temp; - pi_1_locked = false; - - continue; - } } - /* The following assertions are true at this point: - - pi_1 is locked - - pi_2 is unlocked - - pi_1 != pi_2 */ - - gpr_mu_lock(&pi_2->mu); - pi_2_locked = true; - - /* If pi_2 is not terminal node, we are not done locking this polling_island - yet. Release the lock and update pi_2 to the next node in the list */ - if (pi_2->merged_to != NULL) { - temp = pi_2->merged_to; - polling_island_unref_and_unlock(pi_2, 1); - pi_2 = temp; - pi_2_locked = false; + next_1 = (polling_island *)gpr_atm_acq_load(&pi_1->merged_to); + next_2 = (polling_island *)gpr_atm_acq_load(&pi_2->merged_to); + if (next_1 == NULL && next_2 == NULL) { + break; } - } - /* At this point, either pi_1 == pi_2 AND/OR we got both locks */ - if (pi_1 == pi_2) { - /* We may or may not have gotten the lock. If we didn't, walk the rest of - the polling_island list and get the lock */ - GPR_ASSERT(pi_1_locked || (!pi_1_locked && !pi_2_locked)); - if (!pi_1_locked) { - pi_1 = pi_2 = polling_island_update_and_lock(pi_1, 2, 0); - } - } else { - GPR_ASSERT(pi_1_locked && pi_2_locked); - /* If we swapped pi_1 and pi_2 odd number of times, do one more swap so that - pi_1 and pi_2 point to the same polling_island lists they started off - with at the beginning of this function (i.e *p and *q respectively) */ - if (num_swaps % 2 > 0) { - GPR_SWAP(polling_island *, pi_1, pi_2); - } + gpr_mu_unlock(&pi_1->mu); + gpr_mu_unlock(&pi_2->mu); } *p = pi_1; @@ -546,7 +611,7 @@ void polling_island_pair_update_and_lock(polling_island **p, polling_island *polling_island_merge(polling_island *p, polling_island *q) { /* Get locks on both the polling islands */ - polling_island_pair_update_and_lock(&p, &q); + polling_island_lock_pair(&p, &q); if (p == q) { /* Nothing needs to be done here */ @@ -568,15 +633,14 @@ polling_island *polling_island_merge(polling_island *p, polling_island *q) { /* Wakeup all the pollers (if any) on p so that they can pickup this change */ polling_island_add_wakeup_fd_locked(p, &polling_island_wakeup_fd); - p->merged_to = q; + /* Add the 'merged_to' link from p --> q */ + gpr_atm_rel_store(&p->merged_to, q); + PI_ADD_REF(q, "pi_merge"); /* To account for the new incoming ref from p */ - /* - The merged polling island (i.e q) inherits all the ref counts of the - island merging with it (i.e p) - - The island p will lose a ref count */ - q->ref_cnt += p->ref_cnt; - polling_island_unref_and_unlock(p, 1); /* Decrement refcount */ - polling_island_unref_and_unlock(q, 0); /* Just Unlock. Don't decrement ref */ + gpr_mu_unlock(&p->mu); + gpr_mu_unlock(&q->mu); + /* Return the merged polling island */ return q; } @@ -667,6 +731,7 @@ static void unref_by(grpc_fd *fd, int n) { fd->freelist_next = fd_freelist; fd_freelist = fd; grpc_iomgr_unregister_object(&fd->iomgr_object); + gpr_mu_unlock(&fd_freelist_mu); } else { GPR_ASSERT(old > n); @@ -785,16 +850,20 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, REF_BY(fd, 1, reason); /* Remove the fd from the polling island: - - Update the fd->polling_island to point to the latest polling island - - Remove the fd from the polling island. - - Remove a ref to the polling island and set fd->polling_island to NULL */ + - Get a lock on the latest polling island (i.e the last island in the + linked list pointed by fd->polling_island). This is the island that + would actually contain the fd + - Remove the fd from the latest polling island + - Unlock the latest polling island + - Set fd->polling_island to NULL (but remove the ref on the polling island + before doing this.) */ gpr_mu_lock(&fd->pi_mu); if (fd->polling_island != NULL) { - fd->polling_island = - polling_island_update_and_lock(fd->polling_island, 1, 0); - polling_island_remove_fd_locked(fd->polling_island, fd, is_fd_closed); + polling_island *pi_latest = polling_island_lock(fd->polling_island); + polling_island_remove_fd_locked(pi_latest, fd, is_fd_closed); + gpr_mu_unlock(&pi_latest->mu); - polling_island_unref_and_unlock(fd->polling_island, 1); + PI_UNREF(fd->polling_island, "fd_orphan"); fd->polling_island = NULL; } gpr_mu_unlock(&fd->pi_mu); @@ -1050,17 +1119,13 @@ static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { gpr_mu_unlock(&fd->mu); } -/* Release the reference to pollset->polling_island and set it to NULL. - pollset->mu must be held */ -static void pollset_release_polling_island_locked(grpc_pollset *pollset) { - gpr_mu_lock(&pollset->pi_mu); - if (pollset->polling_island) { - pollset->polling_island = - polling_island_update_and_lock(pollset->polling_island, 1, 0); - polling_island_unref_and_unlock(pollset->polling_island, 1); - pollset->polling_island = NULL; +static void pollset_release_polling_island(grpc_pollset *ps, char *reason) { + gpr_mu_lock(&ps->pi_mu); + if (ps->polling_island != NULL) { + PI_UNREF(ps->polling_island, reason); } - gpr_mu_unlock(&pollset->pi_mu); + ps->polling_island = NULL; + gpr_mu_unlock(&ps->pi_mu); } static void finish_shutdown_locked(grpc_exec_ctx *exec_ctx, @@ -1069,8 +1134,9 @@ static void finish_shutdown_locked(grpc_exec_ctx *exec_ctx, GPR_ASSERT(!pollset_has_workers(pollset)); pollset->finish_shutdown_called = true; - pollset_release_polling_island_locked(pollset); + /* Release the ref and set pollset->polling_island to NULL */ + pollset_release_polling_island(pollset, "ps_shutdown"); grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); } @@ -1110,7 +1176,7 @@ static void pollset_reset(grpc_pollset *pollset) { pollset->finish_shutdown_called = false; pollset->kicked_without_pollers = false; pollset->shutdown_done = NULL; - pollset_release_polling_island_locked(pollset); + pollset_release_polling_island(pollset, "ps_reset"); } #define GRPC_EPOLL_MAX_EVENTS 1000 @@ -1124,28 +1190,37 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, GPR_TIMER_BEGIN("pollset_work_and_unlock", 0); /* We need to get the epoll_fd to wait on. The epoll_fd is in inside the - polling island pointed by pollset->polling_island. + latest polling island pointed by pollset->polling_island. Acquire the following locks: - pollset->mu (which we already have) - pollset->pi_mu - - pollset->polling_island->mu (call polling_island_update_and_lock())*/ + - pollset->polling_island lock */ gpr_mu_lock(&pollset->pi_mu); - pi = pollset->polling_island; - if (pi == NULL) { - pi = polling_island_create(NULL, 1); + if (pollset->polling_island == NULL) { + pollset->polling_island = polling_island_create(NULL); + PI_ADD_REF(pollset->polling_island, "ps"); } - /* In addition to locking the polling island, add a ref so that the island - does not get destroyed (which means the epoll_fd won't be closed) while - we are are doing an epoll_wait() on the epoll_fd */ - pi = polling_island_update_and_lock(pi, 1, 1); + pi = polling_island_lock(pollset->polling_island); epoll_fd = pi->epoll_fd; - /* Update the pollset->polling_island */ - pollset->polling_island = pi; + /* Update the pollset->polling_island since the island being pointed by + pollset->polling_island may not be the latest (i.e pi) */ + if (pollset->polling_island != pi) { + /* Always do PI_ADD_REF before PI_UNREF because PI_UNREF may cause the + polling island to be deleted */ + PI_ADD_REF(pi, "ps"); + PI_UNREF(pollset->polling_island, "ps"); + pollset->polling_island = pi; + } + + /* Add an extra ref so that the island does not get destroyed (which means + the epoll_fd won't be closed) while we are are doing an epoll_wait() on the + epoll_fd */ + PI_ADD_REF(pi, "ps_work"); - polling_island_unref_and_unlock(pollset->polling_island, 0); /* Keep the ref*/ + gpr_mu_unlock(&pi->mu); gpr_mu_unlock(&pollset->pi_mu); gpr_mu_unlock(&pollset->mu); @@ -1193,14 +1268,12 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, GPR_ASSERT(pi != NULL); - /* Before leaving, release the extra ref we added to the polling island */ - /* It is important to note that at this point 'pi' may not be the same as - * pollset->polling_island. This is because pollset->polling_island pointer - * gets updated whenever the underlying polling island is merged with another - * island and while we are doing epoll_wait() above, the polling island may - * have been merged */ - pi = polling_island_update_and_lock(pi, 1, 0); /* No new ref added */ - polling_island_unref_and_unlock(pi, 1); + /* Before leaving, release the extra ref we added to the polling island. It + is important to use "pi" here (i.e our old copy of pollset->polling_island + that we got before releasing the polling island lock). This is because + pollset->polling_island pointer might get udpated in other parts of the + code when there is an island merge while we are doing epoll_wait() above */ + PI_UNREF(pi, "ps_work"); GPR_TIMER_END("pollset_work_and_unlock", 0); } @@ -1297,20 +1370,34 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, if (fd->polling_island == pollset->polling_island) { pi_new = fd->polling_island; if (pi_new == NULL) { - pi_new = polling_island_create(fd, 2); + pi_new = polling_island_create(fd); } } else if (fd->polling_island == NULL) { - pi_new = polling_island_update_and_lock(pollset->polling_island, 1, 1); - polling_island_add_fds_locked(pollset->polling_island, &fd, 1, true); + pi_new = polling_island_lock(pollset->polling_island); + polling_island_add_fds_locked(pi_new, &fd, 1, true); gpr_mu_unlock(&pi_new->mu); } else if (pollset->polling_island == NULL) { - pi_new = polling_island_update_and_lock(fd->polling_island, 1, 1); + pi_new = polling_island_lock(fd->polling_island); gpr_mu_unlock(&pi_new->mu); } else { pi_new = polling_island_merge(fd->polling_island, pollset->polling_island); } - fd->polling_island = pollset->polling_island = pi_new; + if (fd->polling_island != pi_new) { + PI_ADD_REF(pi_new, "fd"); + if (fd->polling_island != NULL) { + PI_UNREF(fd->polling_island, "fd"); + } + fd->polling_island = pi_new; + } + + if (pollset->polling_island != pi_new) { + PI_ADD_REF(pi_new, "ps"); + if (pollset->polling_island != NULL) { + PI_UNREF(pollset->polling_island, "ps"); + } + pollset->polling_island = pi_new; + } gpr_mu_unlock(&fd->pi_mu); gpr_mu_unlock(&pollset->pi_mu); @@ -1481,28 +1568,19 @@ void *grpc_pollset_get_polling_island(grpc_pollset *ps) { return pi; } -static polling_island *get_polling_island(polling_island *p) { - if (p == NULL) { - return NULL; - } +bool grpc_are_polling_islands_equal(void *p, void *q) { + polling_island *p1 = p; + polling_island *p2 = q; - polling_island *next; - gpr_mu_lock(&p->mu); - while (p->merged_to != NULL) { - next = p->merged_to; - gpr_mu_unlock(&p->mu); - p = next; - gpr_mu_lock(&p->mu); + polling_island_lock_pair(&p1, &p2); + if (p1 == p2) { + gpr_mu_unlock(&p1->mu); + } else { + gpr_mu_unlock(&p1->mu); + gpr_mu_unlock(&p2->mu); } - gpr_mu_unlock(&p->mu); - - return p; -} -bool grpc_are_polling_islands_equal(void *p, void *q) { - p = get_polling_island(p); - q = get_polling_island(q); - return p == q; + return p1 == p2; } /******************************************************************************* From 65c6c59bcddd2847eb26eb7518747ebeea839d0b Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Tue, 21 Jun 2016 08:27:07 -0700 Subject: [PATCH 0144/1039] Fix refcounting tsan failures and grab pollset lock in the function pollset_add_fd --- src/core/lib/iomgr/ev_epoll_linux.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 72288889c02..7cc69c876db 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -291,11 +291,11 @@ void pi_unref_dbg(polling_island *pi, int ref_cnt, char *reason, char *file, #endif long pi_add_ref(polling_island *pi, int ref_cnt) { - return gpr_atm_no_barrier_fetch_add(&pi->ref_count, ref_cnt); + return gpr_atm_full_fetch_add(&pi->ref_count, ref_cnt); } long pi_unref(polling_island *pi, int ref_cnt) { - long old_cnt = gpr_atm_no_barrier_fetch_add(&pi->ref_count, -ref_cnt); + long old_cnt = gpr_atm_full_fetch_add(&pi->ref_count, -ref_cnt); /* If ref count went to zero, delete the polling island. Note that this need not be done under a lock. Once the ref count goes to zero, we are @@ -311,6 +311,8 @@ long pi_unref(polling_island *pi, int ref_cnt) { if (next != NULL) { PI_UNREF(next, "pi_delete"); /* Recursive call */ } + } else { + GPR_ASSERT(old_cnt > ref_cnt); } return old_cnt; @@ -445,8 +447,8 @@ static polling_island *polling_island_create(grpc_fd *initial_fd) { pi->fds = NULL; } - gpr_atm_no_barrier_store(&pi->ref_count, 0); - gpr_atm_no_barrier_store(&pi->merged_to, NULL); + gpr_atm_rel_store(&pi->ref_count, 0); + gpr_atm_rel_store(&pi->merged_to, NULL); pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); @@ -1347,7 +1349,7 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { - /* TODO sreek - Double check if we need to get a pollset->mu lock here */ + gpr_mu_lock(&pollset->mu); gpr_mu_lock(&pollset->pi_mu); gpr_mu_lock(&fd->pi_mu); @@ -1401,6 +1403,7 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, gpr_mu_unlock(&fd->pi_mu); gpr_mu_unlock(&pollset->pi_mu); + gpr_mu_unlock(&pollset->mu); } /******************************************************************************* From d263b925e8b8ce36042f34eda34715696b6eef3e Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Tue, 21 Jun 2016 09:15:57 -0700 Subject: [PATCH 0145/1039] Make sure to poll cq --- test/cpp/end2end/server_builder_plugin_test.cc | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/test/cpp/end2end/server_builder_plugin_test.cc b/test/cpp/end2end/server_builder_plugin_test.cc index 75f23b64a73..7d0b467d816 100644 --- a/test/cpp/end2end/server_builder_plugin_test.cc +++ b/test/cpp/end2end/server_builder_plugin_test.cc @@ -37,6 +37,7 @@ #include #include #include +#include #include #include #include @@ -188,6 +189,7 @@ class ServerBuilderPluginTest : public ::testing::TestWithParam { grpc::string server_address = "localhost:" + to_string(port_); builder_->AddListeningPort(server_address, InsecureServerCredentials()); cq_ = builder_->AddCompletionQueue(); + cq_thread_ = grpc::thread(std::bind(&ServerBuilderPluginTest::RunCQ, this)); server_ = builder_->BuildAndStart(); EXPECT_TRUE(CheckPresent()); } @@ -204,11 +206,8 @@ class ServerBuilderPluginTest : public ::testing::TestWithParam { EXPECT_TRUE(plugin->init_server_is_called()); EXPECT_TRUE(plugin->finish_is_called()); server_->Shutdown(); - void* tag; - bool ok; cq_->Shutdown(); - while (cq_->Next(&tag, &ok)) - ; + cq_thread_.join(); } string to_string(const int number) { @@ -223,6 +222,7 @@ class ServerBuilderPluginTest : public ::testing::TestWithParam { std::unique_ptr stub_; std::unique_ptr cq_; std::unique_ptr server_; + grpc::thread cq_thread_; TestServiceImpl service_; int port_; @@ -238,6 +238,13 @@ class ServerBuilderPluginTest : public ::testing::TestWithParam { return nullptr; } } + + void RunCQ() { + void* tag; + bool ok; + while (cq_->Next(&tag, &ok)) + ; + } }; TEST_P(ServerBuilderPluginTest, PluginWithoutServiceTest) { From 04a468122f0072cbd8a67d66dfd8243ce3ddface Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Tue, 21 Jun 2016 09:16:57 -0700 Subject: [PATCH 0146/1039] Add comment --- test/cpp/end2end/server_builder_plugin_test.cc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/test/cpp/end2end/server_builder_plugin_test.cc b/test/cpp/end2end/server_builder_plugin_test.cc index 7d0b467d816..778a2be573e 100644 --- a/test/cpp/end2end/server_builder_plugin_test.cc +++ b/test/cpp/end2end/server_builder_plugin_test.cc @@ -188,6 +188,8 @@ class ServerBuilderPluginTest : public ::testing::TestWithParam { void StartServer() { grpc::string server_address = "localhost:" + to_string(port_); builder_->AddListeningPort(server_address, InsecureServerCredentials()); + // we run some tests without a service, and for those we need to supply a + // frequently polled completion queue cq_ = builder_->AddCompletionQueue(); cq_thread_ = grpc::thread(std::bind(&ServerBuilderPluginTest::RunCQ, this)); server_ = builder_->BuildAndStart(); From 9f26a149a5da48364ec62dd607e34104bb851c6d Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Tue, 21 Jun 2016 10:15:01 -0700 Subject: [PATCH 0147/1039] fixed incomplete sentence --- doc/compression.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/compression.md b/doc/compression.md index 6e455636021..a116ac50a16 100644 --- a/doc/compression.md +++ b/doc/compression.md @@ -81,8 +81,8 @@ initial call. A client doesn't a priori (presently) know which algorithms a server supports. This issue can be addressed with an initial negotiation of capabilities or an automatic retry mechanism. These features will be implemented in the future. Currently however, compression levels are only supported at the -server side, which is aware of the client's capabilities by virtue of the -incoming. +server side, which is aware of the client's capabilities through the incoming +Message-Accept-Encoding header. ### Propagation to child RPCs From d5fd7ddc70e658a0364bd2e43c8a486a25db267c Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Tue, 21 Jun 2016 11:13:23 -0700 Subject: [PATCH 0148/1039] Addressed review comments Removed the silencing for incompatible-pointer-types Removed unused objects Fixed format issues --- src/objective-c/ProtoRPC/ProtoService.m | 9 +++------ src/objective-c/tests/GRPCClientTests.m | 24 ++++++++++++------------ src/objective-c/tests/Podfile | 3 ++- 3 files changed, 17 insertions(+), 19 deletions(-) diff --git a/src/objective-c/ProtoRPC/ProtoService.m b/src/objective-c/ProtoRPC/ProtoService.m index cd9bc7aeac4..97401908519 100644 --- a/src/objective-c/ProtoRPC/ProtoService.m +++ b/src/objective-c/ProtoRPC/ProtoService.m @@ -65,22 +65,19 @@ return self; } -#pragma clang diagnostic push -#pragma clang diagnostic ignored "-Wincompatible-pointer-types" - (ProtoRPC *)RPCToMethod:(NSString *)method requestsWriter:(GRXWriter *)requestsWriter responseClass:(Class)responseClass responsesWriteable:(id)responsesWriteable { - ProtoMethod *methodName = [[ProtoMethod alloc] initWithPackage:_packageName - service:_serviceName - method:method]; + GRPCProtoMethod *methodName = [[GRPCProtoMethod alloc] initWithPackage:_packageName + service:_serviceName + method:method]; return [[ProtoRPC alloc] initWithHost:_host method:methodName requestsWriter:requestsWriter responseClass:responseClass responsesWriteable:responsesWriteable]; } -#pragma clang diagnostic pop @end @implementation GRPCProtoService diff --git a/src/objective-c/tests/GRPCClientTests.m b/src/objective-c/tests/GRPCClientTests.m index 2eca7bf5498..1167a715bb9 100644 --- a/src/objective-c/tests/GRPCClientTests.m +++ b/src/objective-c/tests/GRPCClientTests.m @@ -110,14 +110,14 @@ static GRPCProtoMethod *kUnaryCallMethod; // This method isn't implemented by the remote server. kInexistentMethod = [[GRPCProtoMethod alloc] initWithPackage:kPackage - service:kService - method:@"Inexistent"]; + service:kService + method:@"Inexistent"]; kEmptyCallMethod = [[GRPCProtoMethod alloc] initWithPackage:kPackage - service:kService - method:@"EmptyCall"]; + service:kService + method:@"EmptyCall"]; kUnaryCallMethod = [[GRPCProtoMethod alloc] initWithPackage:kPackage - service:kService - method:@"UnaryCall"]; + service:kService + method:@"UnaryCall"]; } - (void)testConnectionToRemoteServer { @@ -303,9 +303,9 @@ static GRPCProtoMethod *kUnaryCallMethod; // Try to set parameters to nil for GRPCCall. This should cause an exception @try { - GRPCCall *call __unused = [[GRPCCall alloc] initWithHost:nil - path:nil - requestsWriter:nil]; + (void)[[GRPCCall alloc] initWithHost:nil + path:nil + requestsWriter:nil]; XCTFail(@"Did not receive an exception when parameters are nil"); } @catch(NSException *theException) { NSLog(@"Received exception as expected: %@", theException.name); @@ -316,9 +316,9 @@ static GRPCProtoMethod *kUnaryCallMethod; GRXWriter *requestsWriter = [GRXWriter emptyWriter]; [requestsWriter finishWithError:nil]; @try { - GRPCCall *call __unused = [[GRPCCall alloc] initWithHost:kHostAddress - path:kUnaryCallMethod.HTTPPath - requestsWriter:requestsWriter]; + (void)[[GRPCCall alloc] initWithHost:kHostAddress + path:kUnaryCallMethod.HTTPPath + requestsWriter:requestsWriter]; XCTFail(@"Did not receive an exception when GRXWriter has incorrect state."); } @catch(NSException *theException) { NSLog(@"Received exception as expected: %@", theException.name); diff --git a/src/objective-c/tests/Podfile b/src/objective-c/tests/Podfile index d51b18cc34d..6d5f94cbda1 100644 --- a/src/objective-c/tests/Podfile +++ b/src/objective-c/tests/Podfile @@ -6,7 +6,7 @@ install! 'cocoapods', :deterministic_uuids => false def shared_pods pod 'Protobuf', :path => "../../../third_party/protobuf", :inhibit_warnings => true pod 'BoringSSL', :podspec => "..", :inhibit_warnings => true - pod 'CronetFramework', :podspec => "..", :inhibit_warnings => true + pod 'CronetFramework', :podspec => ".." pod 'gRPC', :path => "../../.." pod 'RemoteTest', :path => "RemoteTestClient" end @@ -42,6 +42,7 @@ post_install do |installer| end if target.name == 'gRPC' target.build_configurations.each do |config| + # TODO(zyc) Remove this setting after the issue is resolved # GPR_UNREACHABLE_CODE causes "Control may reach end of non-void # function" warning config.build_settings['GCC_WARN_ABOUT_RETURN_TYPE'] = 'NO' From 8d9e83806d24ad5e1a47bb705d88816bfc8b4864 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Tue, 21 Jun 2016 11:21:48 -0700 Subject: [PATCH 0149/1039] Fixed format issues --- src/objective-c/tests/InteropTests.m | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/objective-c/tests/InteropTests.m b/src/objective-c/tests/InteropTests.m index 60a83259fa6..15ce120c551 100644 --- a/src/objective-c/tests/InteropTests.m +++ b/src/objective-c/tests/InteropTests.m @@ -285,8 +285,8 @@ static cronet_engine *cronetEngine = NULL; GRXBufferedPipe *requestsBuffer = [[GRXBufferedPipe alloc] init]; GRPCProtoCall *call = [_service RPCToStreamingInputCallWithRequestsWriter:requestsBuffer - handler:^(RMTStreamingInputCallResponse *response, - NSError *error) { + handler:^(RMTStreamingInputCallResponse *response, + NSError *error) { XCTAssertEqual(error.code, GRPC_STATUS_CANCELLED); [expectation fulfill]; }]; From b665dd2722371e5c4a28bbf6cb5f791f8ab5cca9 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Tue, 21 Jun 2016 11:52:29 -0700 Subject: [PATCH 0150/1039] reworded some --- doc/compression.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/compression.md b/doc/compression.md index a116ac50a16..15fae4d29bf 100644 --- a/doc/compression.md +++ b/doc/compression.md @@ -42,13 +42,13 @@ and RPC settings (for example, if compression would result in small or negative gains). When a message from a client compressed with an unsupported algorithm is -processed by a server, it WILL result in an INVALID\_ARGUMENT error. The server -will include in its response a `grpc-accept-encoding` header specifying the -algorithms it does accept. If an INTERNAL error is returned from the server -despite having used one of the algorithms from the `grpc-accept-encoding` -header, the cause MUST NOT be related to compression. Data sent from a server -compressed with an algorithm not supported by the client WILL result in an -INTERNAL error on the client side. +processed by a server, it WILL result in an INVALID\_ARGUMENT error on the +server. The server will then include in its response a `grpc-accept-encoding` +header specifying the algorithms it does accept. If an INTERNAL error is +returned from the server despite having used one of the algorithms from the +`grpc-accept-encoding` header, the cause MUST NOT be related to compression. +Data sent from a server compressed with an algorithm not supported by the client +WILL result in an INTERNAL error on the client side. Note that a peer MAY choose to not disclose all the encodings it supports. However, if it receives a message compressed in an undisclosed but supported From 3d399cb102510e46ca252213ad2799b40704d33d Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Tue, 21 Jun 2016 11:57:46 -0700 Subject: [PATCH 0151/1039] Fix format issues --- src/objective-c/tests/InteropTests.m | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/objective-c/tests/InteropTests.m b/src/objective-c/tests/InteropTests.m index 15ce120c551..3102cf597a0 100644 --- a/src/objective-c/tests/InteropTests.m +++ b/src/objective-c/tests/InteropTests.m @@ -284,9 +284,10 @@ static cronet_engine *cronetEngine = NULL; // A buffered pipe to which we never write any value acts as a writer that just hangs. GRXBufferedPipe *requestsBuffer = [[GRXBufferedPipe alloc] init]; - GRPCProtoCall *call = [_service RPCToStreamingInputCallWithRequestsWriter:requestsBuffer - handler:^(RMTStreamingInputCallResponse *response, - NSError *error) { + GRPCProtoCall *call = + [_service RPCToStreamingInputCallWithRequestsWriter:requestsBuffer + handler:^(RMTStreamingInputCallResponse *response, + NSError *error) { XCTAssertEqual(error.code, GRPC_STATUS_CANCELLED); [expectation fulfill]; }]; From 55145c03d2a164d207e4a112963c91036cae7b28 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Tue, 21 Jun 2016 14:51:54 -0700 Subject: [PATCH 0152/1039] first step in transforming grpclb_test to C++ --- Makefile | 197 ++++-- build.yaml | 25 +- .../grpclb/grpclb_test.cc} | 15 +- tools/run_tests/sources_and_headers.json | 38 +- tools/run_tests/tests.json | 42 +- vsprojects/buildtests_c.sln | 27 - .../grpc_test_util/grpc_test_util.vcxproj | 270 ++++++++ .../grpc_test_util.vcxproj.filters | 602 ++++++++++++++++++ .../test/grpclb_test/grpclb_test.vcxproj | 30 +- .../grpclb_test/grpclb_test.vcxproj.filters | 30 +- 10 files changed, 1147 insertions(+), 129 deletions(-) rename test/{core/client_config/grpclb_test.c => cpp/grpclb/grpclb_test.cc} (98%) diff --git a/Makefile b/Makefile index e316d4b1533..7c88917f11e 100644 --- a/Makefile +++ b/Makefile @@ -946,7 +946,6 @@ grpc_jwt_verifier_test: $(BINDIR)/$(CONFIG)/grpc_jwt_verifier_test grpc_print_google_default_creds_token: $(BINDIR)/$(CONFIG)/grpc_print_google_default_creds_token grpc_security_connector_test: $(BINDIR)/$(CONFIG)/grpc_security_connector_test grpc_verify_jwt: $(BINDIR)/$(CONFIG)/grpc_verify_jwt -grpclb_test: $(BINDIR)/$(CONFIG)/grpclb_test hpack_parser_fuzzer_test: $(BINDIR)/$(CONFIG)/hpack_parser_fuzzer_test hpack_parser_test: $(BINDIR)/$(CONFIG)/hpack_parser_test hpack_table_test: $(BINDIR)/$(CONFIG)/hpack_table_test @@ -1025,6 +1024,7 @@ grpc_objective_c_plugin: $(BINDIR)/$(CONFIG)/grpc_objective_c_plugin grpc_python_plugin: $(BINDIR)/$(CONFIG)/grpc_python_plugin grpc_ruby_plugin: $(BINDIR)/$(CONFIG)/grpc_ruby_plugin grpclb_api_test: $(BINDIR)/$(CONFIG)/grpclb_api_test +grpclb_test: $(BINDIR)/$(CONFIG)/grpclb_test hybrid_end2end_test: $(BINDIR)/$(CONFIG)/hybrid_end2end_test interop_client: $(BINDIR)/$(CONFIG)/interop_client interop_server: $(BINDIR)/$(CONFIG)/interop_server @@ -1278,7 +1278,6 @@ buildtests_c: privatelibs_c \ $(BINDIR)/$(CONFIG)/grpc_json_token_test \ $(BINDIR)/$(CONFIG)/grpc_jwt_verifier_test \ $(BINDIR)/$(CONFIG)/grpc_security_connector_test \ - $(BINDIR)/$(CONFIG)/grpclb_test \ $(BINDIR)/$(CONFIG)/hpack_parser_test \ $(BINDIR)/$(CONFIG)/hpack_table_test \ $(BINDIR)/$(CONFIG)/http_parser_test \ @@ -1399,6 +1398,7 @@ buildtests_cxx: buildtests_zookeeper privatelibs_cxx \ $(BINDIR)/$(CONFIG)/golden_file_test \ $(BINDIR)/$(CONFIG)/grpc_cli \ $(BINDIR)/$(CONFIG)/grpclb_api_test \ + $(BINDIR)/$(CONFIG)/grpclb_test \ $(BINDIR)/$(CONFIG)/hybrid_end2end_test \ $(BINDIR)/$(CONFIG)/interop_client \ $(BINDIR)/$(CONFIG)/interop_server \ @@ -1581,8 +1581,6 @@ test_c: buildtests_c $(Q) $(BINDIR)/$(CONFIG)/grpc_jwt_verifier_test || ( echo test grpc_jwt_verifier_test failed ; exit 1 ) $(E) "[RUN] Testing grpc_security_connector_test" $(Q) $(BINDIR)/$(CONFIG)/grpc_security_connector_test || ( echo test grpc_security_connector_test failed ; exit 1 ) - $(E) "[RUN] Testing grpclb_test" - $(Q) $(BINDIR)/$(CONFIG)/grpclb_test || ( echo test grpclb_test failed ; exit 1 ) $(E) "[RUN] Testing hpack_parser_test" $(Q) $(BINDIR)/$(CONFIG)/hpack_parser_test || ( echo test hpack_parser_test failed ; exit 1 ) $(E) "[RUN] Testing hpack_table_test" @@ -1731,6 +1729,8 @@ test_cxx: test_zookeeper buildtests_cxx $(Q) $(BINDIR)/$(CONFIG)/golden_file_test || ( echo test golden_file_test failed ; exit 1 ) $(E) "[RUN] Testing grpclb_api_test" $(Q) $(BINDIR)/$(CONFIG)/grpclb_api_test || ( echo test grpclb_api_test failed ; exit 1 ) + $(E) "[RUN] Testing grpclb_test" + $(Q) $(BINDIR)/$(CONFIG)/grpclb_test || ( echo test grpclb_test failed ; exit 1 ) $(E) "[RUN] Testing hybrid_end2end_test" $(Q) $(BINDIR)/$(CONFIG)/hybrid_end2end_test || ( echo test hybrid_end2end_test failed ; exit 1 ) $(E) "[RUN] Testing interop_test" @@ -3036,8 +3036,118 @@ LIBGRPC_TEST_UTIL_SRC = \ test/core/util/port_server_client.c \ test/core/util/port_windows.c \ test/core/util/slice_splitter.c \ + src/core/lib/channel/channel_args.c \ + src/core/lib/channel/channel_stack.c \ + src/core/lib/channel/channel_stack_builder.c \ + src/core/lib/channel/compress_filter.c \ + src/core/lib/channel/connected_channel.c \ + src/core/lib/channel/http_client_filter.c \ + src/core/lib/channel/http_server_filter.c \ + src/core/lib/compression/compression.c \ + src/core/lib/compression/message_compress.c \ + src/core/lib/debug/trace.c \ + src/core/lib/http/format_request.c \ + src/core/lib/http/httpcli.c \ + src/core/lib/http/parser.c \ + src/core/lib/iomgr/closure.c \ + src/core/lib/iomgr/endpoint.c \ + src/core/lib/iomgr/endpoint_pair_posix.c \ + src/core/lib/iomgr/endpoint_pair_windows.c \ + src/core/lib/iomgr/ev_poll_and_epoll_posix.c \ + src/core/lib/iomgr/ev_poll_posix.c \ + src/core/lib/iomgr/ev_posix.c \ + src/core/lib/iomgr/exec_ctx.c \ + src/core/lib/iomgr/executor.c \ + src/core/lib/iomgr/iocp_windows.c \ + src/core/lib/iomgr/iomgr.c \ + src/core/lib/iomgr/iomgr_posix.c \ + src/core/lib/iomgr/iomgr_windows.c \ + src/core/lib/iomgr/polling_entity.c \ + src/core/lib/iomgr/pollset_set_windows.c \ + src/core/lib/iomgr/pollset_windows.c \ + src/core/lib/iomgr/resolve_address_posix.c \ + src/core/lib/iomgr/resolve_address_windows.c \ + src/core/lib/iomgr/sockaddr_utils.c \ + src/core/lib/iomgr/socket_utils_common_posix.c \ + src/core/lib/iomgr/socket_utils_linux.c \ + src/core/lib/iomgr/socket_utils_posix.c \ + src/core/lib/iomgr/socket_windows.c \ + src/core/lib/iomgr/tcp_client_posix.c \ + src/core/lib/iomgr/tcp_client_windows.c \ + src/core/lib/iomgr/tcp_posix.c \ + src/core/lib/iomgr/tcp_server_posix.c \ + src/core/lib/iomgr/tcp_server_windows.c \ + src/core/lib/iomgr/tcp_windows.c \ + src/core/lib/iomgr/time_averaged_stats.c \ + src/core/lib/iomgr/timer.c \ + src/core/lib/iomgr/timer_heap.c \ + src/core/lib/iomgr/udp_server.c \ + src/core/lib/iomgr/unix_sockets_posix.c \ + src/core/lib/iomgr/unix_sockets_posix_noop.c \ + src/core/lib/iomgr/wakeup_fd_eventfd.c \ + src/core/lib/iomgr/wakeup_fd_nospecial.c \ + src/core/lib/iomgr/wakeup_fd_pipe.c \ + src/core/lib/iomgr/wakeup_fd_posix.c \ + src/core/lib/iomgr/workqueue_posix.c \ + src/core/lib/iomgr/workqueue_windows.c \ + src/core/lib/json/json.c \ + src/core/lib/json/json_reader.c \ + src/core/lib/json/json_string.c \ + src/core/lib/json/json_writer.c \ + src/core/lib/surface/alarm.c \ + src/core/lib/surface/api_trace.c \ + src/core/lib/surface/byte_buffer.c \ + src/core/lib/surface/byte_buffer_reader.c \ + src/core/lib/surface/call.c \ + src/core/lib/surface/call_details.c \ + src/core/lib/surface/call_log_batch.c \ + src/core/lib/surface/channel.c \ + src/core/lib/surface/channel_init.c \ + src/core/lib/surface/channel_ping.c \ + src/core/lib/surface/channel_stack_type.c \ + src/core/lib/surface/completion_queue.c \ + src/core/lib/surface/event_string.c \ + src/core/lib/surface/lame_client.c \ + src/core/lib/surface/metadata_array.c \ + src/core/lib/surface/server.c \ + src/core/lib/surface/validate_metadata.c \ + src/core/lib/surface/version.c \ + src/core/lib/transport/byte_stream.c \ + src/core/lib/transport/connectivity_state.c \ + src/core/lib/transport/metadata.c \ + src/core/lib/transport/metadata_batch.c \ + src/core/lib/transport/static_metadata.c \ + src/core/lib/transport/transport.c \ + src/core/lib/transport/transport_op_string.c \ PUBLIC_HEADERS_C += \ + include/grpc/byte_buffer.h \ + include/grpc/byte_buffer_reader.h \ + include/grpc/compression.h \ + include/grpc/grpc.h \ + include/grpc/grpc_posix.h \ + include/grpc/status.h \ + include/grpc/impl/codegen/byte_buffer.h \ + include/grpc/impl/codegen/byte_buffer_reader.h \ + include/grpc/impl/codegen/compression_types.h \ + include/grpc/impl/codegen/connectivity_state.h \ + include/grpc/impl/codegen/grpc_types.h \ + include/grpc/impl/codegen/propagation_bits.h \ + include/grpc/impl/codegen/status.h \ + include/grpc/impl/codegen/alloc.h \ + include/grpc/impl/codegen/atm.h \ + include/grpc/impl/codegen/atm_gcc_atomic.h \ + include/grpc/impl/codegen/atm_gcc_sync.h \ + include/grpc/impl/codegen/atm_windows.h \ + include/grpc/impl/codegen/log.h \ + include/grpc/impl/codegen/port_platform.h \ + include/grpc/impl/codegen/slice.h \ + include/grpc/impl/codegen/slice_buffer.h \ + include/grpc/impl/codegen/sync.h \ + include/grpc/impl/codegen/sync_generic.h \ + include/grpc/impl/codegen/sync_posix.h \ + include/grpc/impl/codegen/sync_windows.h \ + include/grpc/impl/codegen/time.h \ LIBGRPC_TEST_UTIL_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(LIBGRPC_TEST_UTIL_SRC)))) @@ -8476,38 +8586,6 @@ endif endif -GRPCLB_TEST_SRC = \ - test/core/client_config/grpclb_test.c \ - -GRPCLB_TEST_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(GRPCLB_TEST_SRC)))) -ifeq ($(NO_SECURE),true) - -# You can't build secure targets if you don't have OpenSSL. - -$(BINDIR)/$(CONFIG)/grpclb_test: openssl_dep_error - -else - - - -$(BINDIR)/$(CONFIG)/grpclb_test: $(GRPCLB_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a - $(E) "[LD] Linking $@" - $(Q) mkdir -p `dirname $@` - $(Q) $(LD) $(LDFLAGS) $(GRPCLB_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/grpclb_test - -endif - -$(OBJDIR)/$(CONFIG)/test/core/client_config/grpclb_test.o: $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a - -deps_grpclb_test: $(GRPCLB_TEST_OBJS:.o=.dep) - -ifneq ($(NO_SECURE),true) -ifneq ($(NO_DEPS),true) --include $(GRPCLB_TEST_OBJS:.o=.dep) -endif -endif - - HPACK_PARSER_FUZZER_TEST_SRC = \ test/core/transport/chttp2/hpack_parser_fuzzer_test.c \ @@ -11251,6 +11329,53 @@ endif $(OBJDIR)/$(CONFIG)/test/cpp/grpclb/grpclb_api_test.o: $(GENDIR)/src/proto/grpc/lb/v1/load_balancer.pb.cc $(GENDIR)/src/proto/grpc/lb/v1/load_balancer.grpc.pb.cc +GRPCLB_TEST_SRC = \ + $(GENDIR)/src/proto/grpc/lb/v1/load_balancer.pb.cc $(GENDIR)/src/proto/grpc/lb/v1/load_balancer.grpc.pb.cc \ + test/cpp/grpclb/grpclb_test.cc \ + +GRPCLB_TEST_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(GRPCLB_TEST_SRC)))) +ifeq ($(NO_SECURE),true) + +# You can't build secure targets if you don't have OpenSSL. + +$(BINDIR)/$(CONFIG)/grpclb_test: openssl_dep_error + +else + + + + +ifeq ($(NO_PROTOBUF),true) + +# You can't build the protoc plugins or protobuf-enabled targets if you don't have protobuf 3.0.0+. + +$(BINDIR)/$(CONFIG)/grpclb_test: protobuf_dep_error + +else + +$(BINDIR)/$(CONFIG)/grpclb_test: $(PROTOBUF_DEP) $(GRPCLB_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a + $(E) "[LD] Linking $@" + $(Q) mkdir -p `dirname $@` + $(Q) $(LDXX) $(LDFLAGS) $(GRPCLB_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LDLIBSXX) $(LDLIBS_PROTOBUF) $(LDLIBS) $(LDLIBS_SECURE) $(GTEST_LIB) -o $(BINDIR)/$(CONFIG)/grpclb_test + +endif + +endif + +$(OBJDIR)/$(CONFIG)/src/proto/grpc/lb/v1/load_balancer.o: $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a + +$(OBJDIR)/$(CONFIG)/test/cpp/grpclb/grpclb_test.o: $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a + +deps_grpclb_test: $(GRPCLB_TEST_OBJS:.o=.dep) + +ifneq ($(NO_SECURE),true) +ifneq ($(NO_DEPS),true) +-include $(GRPCLB_TEST_OBJS:.o=.dep) +endif +endif +$(OBJDIR)/$(CONFIG)/test/cpp/grpclb/grpclb_test.o: $(GENDIR)/src/proto/grpc/lb/v1/load_balancer.pb.cc $(GENDIR)/src/proto/grpc/lb/v1/load_balancer.grpc.pb.cc + + HYBRID_END2END_TEST_SRC = \ test/cpp/end2end/hybrid_end2end_test.cc \ diff --git a/build.yaml b/build.yaml index ee854361a19..d62b0a03f70 100644 --- a/build.yaml +++ b/build.yaml @@ -866,6 +866,7 @@ libs: - grpc filegroups: - grpc_test_util_base + - grpc_base vs_project_guid: '{17BCAFC0-5FDC-4C94-AEB9-95F3E220614B}' - name: grpc_test_util_unsecure build: private @@ -1820,16 +1821,6 @@ targets: - grpc - gpr_test_util - gpr -- name: grpclb_test - build: test - language: c - src: - - test/core/client_config/grpclb_test.c - deps: - - grpc_test_util - - grpc - - gpr_test_util - - gpr - name: hpack_parser_fuzzer_test build: fuzzer language: c @@ -2734,6 +2725,20 @@ targets: - grpc_test_util - grpc++ - grpc +- name: grpclb_test + gtest: false + build: test + language: c++ + src: + - src/proto/grpc/lb/v1/load_balancer.proto + - test/cpp/grpclb/grpclb_test.cc + deps: + - gpr + - gpr_test_util + - grpc + - grpc++ + - grpc++_test_util + - grpc_test_util - name: hybrid_end2end_test gtest: true build: test diff --git a/test/core/client_config/grpclb_test.c b/test/cpp/grpclb/grpclb_test.cc similarity index 98% rename from test/core/client_config/grpclb_test.c rename to test/cpp/grpclb/grpclb_test.cc index 85cfe80748f..4805bf024d5 100644 --- a/test/core/client_config/grpclb_test.c +++ b/test/cpp/grpclb/grpclb_test.cc @@ -34,6 +34,7 @@ #include #include +extern "C" { #include #include #include @@ -52,6 +53,7 @@ #include "test/core/end2end/cq_verifier.h" #include "test/core/util/port.h" #include "test/core/util/test_config.h" +} #define NUM_BACKENDS 4 @@ -95,7 +97,8 @@ static gpr_slice build_response_payload_slice(const char *host, int *ports, } ... } */ - char **hostports_vec = gpr_malloc(sizeof(char *) * nports); + char **hostports_vec = + static_cast(gpr_malloc(sizeof(char *) * nports)); for (size_t i = 0; i < nports; i++) { gpr_join_host_port(&hostports_vec[i], "127.0.0.1", ports[i]); } @@ -118,7 +121,7 @@ static gpr_slice build_response_payload_slice(const char *host, int *ports, const size_t fsize = (size_t)ftell(f); rewind(f); - char *serialized_response = gpr_malloc(fsize); + char *serialized_response = static_cast(gpr_malloc(fsize)); GPR_ASSERT(fread(serialized_response, fsize, 1, f) == 1); fclose(f); gpr_free(output_fname); @@ -519,9 +522,9 @@ static void setup_server(const char *host, server_fixture *sf) { int assigned_port; sf->cq = grpc_completion_queue_create(NULL); - char *colon_idx = strchr(host, ':'); + const char *colon_idx = strchr(host, ':'); if (colon_idx) { - char *port_str = colon_idx + 1; + const char *port_str = colon_idx + 1; sf->port = atoi(port_str); sf->servers_hostport = gpr_strdup(host); } else { @@ -558,12 +561,12 @@ static void teardown_server(server_fixture *sf) { } static void fork_backend_server(void *arg) { - server_fixture *sf = arg; + server_fixture *sf = static_cast(arg); start_backend_server(sf); } static void fork_lb_server(void *arg) { - test_fixture *tf = arg; + test_fixture *tf = static_cast(arg); int ports[NUM_BACKENDS]; for (int i = 0; i < NUM_BACKENDS; i++) { ports[i] = tf->lb_backends[i].port; diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 492be7322f6..6e292e5983e 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -944,22 +944,6 @@ "third_party": false, "type": "target" }, - { - "deps": [ - "gpr", - "gpr_test_util", - "grpc", - "grpc_test_util" - ], - "headers": [], - "language": "c", - "name": "grpclb_test", - "src": [ - "test/core/client_config/grpclb_test.c" - ], - "third_party": false, - "type": "target" - }, { "deps": [ "gpr", @@ -2231,6 +2215,27 @@ "third_party": false, "type": "target" }, + { + "deps": [ + "gpr", + "gpr_test_util", + "grpc", + "grpc++", + "grpc++_test_util", + "grpc_test_util" + ], + "headers": [ + "src/proto/grpc/lb/v1/load_balancer.grpc.pb.h", + "src/proto/grpc/lb/v1/load_balancer.pb.h" + ], + "language": "c++", + "name": "grpclb_test", + "src": [ + "test/cpp/grpclb/grpclb_test.cc" + ], + "third_party": false, + "type": "target" + }, { "deps": [ "gpr", @@ -4222,6 +4227,7 @@ "gpr", "gpr_test_util", "grpc", + "grpc_base", "grpc_test_util_base" ], "headers": [ diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index bf439b43e9c..b69bea4242f 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -1079,27 +1079,6 @@ "windows" ] }, - { - "args": [], - "ci_platforms": [ - "linux", - "mac", - "posix", - "windows" - ], - "cpu_cost": 1.0, - "exclude_configs": [], - "flaky": false, - "gtest": false, - "language": "c", - "name": "grpclb_test", - "platforms": [ - "linux", - "mac", - "posix", - "windows" - ] - }, { "args": [], "ci_platforms": [ @@ -2294,6 +2273,27 @@ "windows" ] }, + { + "args": [], + "ci_platforms": [ + "linux", + "mac", + "posix", + "windows" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "gtest": false, + "language": "c++", + "name": "grpclb_test", + "platforms": [ + "linux", + "mac", + "posix", + "windows" + ] + }, { "args": [], "ci_platforms": [ diff --git a/vsprojects/buildtests_c.sln b/vsprojects/buildtests_c.sln index 070f6559b7b..3a214a4ad41 100644 --- a/vsprojects/buildtests_c.sln +++ b/vsprojects/buildtests_c.sln @@ -643,17 +643,6 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_verify_jwt", "vcxproj\ {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpclb_test", "vcxproj\test\grpclb_test\grpclb_test.vcxproj", "{9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}" - ProjectSection(myProperties) = preProject - lib = "False" - EndProjectSection - ProjectSection(ProjectDependencies) = postProject - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} - {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} - {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} - EndProjectSection -EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "h2_census_nosec_test", "vcxproj\test/end2end/fixtures\h2_census_nosec_test\h2_census_nosec_test.vcxproj", "{A8039D43-910E-4248-2A22-74366E8C4DCD}" ProjectSection(myProperties) = preProject lib = "False" @@ -2466,22 +2455,6 @@ Global {02FAC25F-5FF6-34A0-00AE-B82BFBA851A9}.Release-DLL|Win32.Build.0 = Release|Win32 {02FAC25F-5FF6-34A0-00AE-B82BFBA851A9}.Release-DLL|x64.ActiveCfg = Release|x64 {02FAC25F-5FF6-34A0-00AE-B82BFBA851A9}.Release-DLL|x64.Build.0 = Release|x64 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug|Win32.ActiveCfg = Debug|Win32 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug|x64.ActiveCfg = Debug|x64 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release|Win32.ActiveCfg = Release|Win32 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release|x64.ActiveCfg = Release|x64 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug|Win32.Build.0 = Debug|Win32 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug|x64.Build.0 = Debug|x64 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release|Win32.Build.0 = Release|Win32 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release|x64.Build.0 = Release|x64 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug-DLL|Win32.ActiveCfg = Debug|Win32 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug-DLL|Win32.Build.0 = Debug|Win32 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug-DLL|x64.ActiveCfg = Debug|x64 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Debug-DLL|x64.Build.0 = Debug|x64 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release-DLL|Win32.ActiveCfg = Release|Win32 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release-DLL|Win32.Build.0 = Release|Win32 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release-DLL|x64.ActiveCfg = Release|x64 - {9D6FFE17-ABF0-A851-268E-9E3E8C573CBB}.Release-DLL|x64.Build.0 = Release|x64 {A8039D43-910E-4248-2A22-74366E8C4DCD}.Debug|Win32.ActiveCfg = Debug|Win32 {A8039D43-910E-4248-2A22-74366E8C4DCD}.Debug|x64.ActiveCfg = Debug|x64 {A8039D43-910E-4248-2A22-74366E8C4DCD}.Release|Win32.ActiveCfg = Release|Win32 diff --git a/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj b/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj index f0a8f7b6b9e..fd37c566921 100644 --- a/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj +++ b/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj @@ -146,6 +146,35 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -160,6 +189,81 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -196,6 +300,172 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj.filters b/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj.filters index a1d31eb54e0..f04c6ea773c 100644 --- a/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj.filters +++ b/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj.filters @@ -52,6 +52,338 @@ test\core\util + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\compression + + + src\core\lib\compression + + + src\core\lib\debug + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + + + include\grpc\impl\codegen + @@ -93,9 +425,279 @@ test\core\util + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\compression + + + src\core\lib\compression + + + src\core\lib\debug + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + {50129440-aff7-7df7-682c-b9671be19a6f} + + + {d448b078-95a6-6fca-fe4a-8b44dd71f359} + + + {314a6801-6fe3-9211-33d8-ecf3332c1151} + + + {8e97f1e1-f4d1-a56e-0837-7901778fb3b9} + + + {7d107d7c-1da3-9525-3ba1-3a411b552ea8} + + + {f7bfac91-5eb2-dea7-4601-6c63edbbf997} + + + {f4e8c61e-1ca6-0fdd-7b5e-b7f9a30c9a21} + + + {1cd1503c-bec0-5ade-c75f-aa25c80975ec} + + + {09632582-2cc3-5618-d673-65d3884f8ce5} + + + {2c1a72e9-886e-8082-9d2f-0fc9cb3ab996} + + + {4862ecce-fa07-eb5e-5c05-bfa753c8bfe5} + + + {fc7f488e-08b4-8366-3720-1f7ffaa0b0b3} + + + {89bc8f83-e29a-ddab-8f6b-22df11cdc867} + + + {7f2b7dca-395f-94dd-c9ad-9a286bd9751e} + + + {5249e884-ea07-6782-531d-ec622c54b9af} + {a2783de3-4fcf-718d-a859-c2108350ff33} diff --git a/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj b/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj index 7ea8e06d74a..91b9a6eaccc 100644 --- a/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj +++ b/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj @@ -53,8 +53,10 @@ + + @@ -158,21 +160,35 @@ - + + + + + + + + + - - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} + + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + + + {EAB0A629-17A9-44DB-B5FF-E91A721FE037} {29D16885-7228-4C31-81ED-5F9187C7F2A9} - - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} + + {C187A093-A0FE-489D-A40A-6E33DE0F9FEB} - - {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + + {0BE77741-552A-929B-A497-4EF7ECE17A64} + + + {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} diff --git a/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj.filters b/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj.filters index 7a70c0b5e1c..5493df2e7b2 100644 --- a/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj.filters +++ b/vsprojects/vcxproj/test/grpclb_test/grpclb_test.vcxproj.filters @@ -1,20 +1,38 @@ - - test\core\client_config + + src\proto\grpc\lb\v1 + + + test\cpp\grpclb + + {082bbcbc-bbd8-db7d-afe9-fa0fec76863d} + + + {7c61995a-80c2-ef3d-0a4b-baf2221c12f8} + + + {fc2920d4-902c-fdf0-d4b8-6b22e1361105} + + + {4778352a-559f-b008-e3d2-4ae9181260df} + + + {36a4326b-4f2a-4720-b730-e40fe078fd40} + {e0ba55a2-37d9-5029-6f4e-64f097307340} - - {6d1d02a2-d635-142d-16d4-45bd59f5c83a} + + {ad67d6ba-d08b-7879-2929-20f0f01c9918} - - {356b8663-f1cf-f0df-39d0-d2de958699d0} + + {4498fa30-2dab-584c-5711-a1055cec30f6} From ef01edf5b7c6eb008c7b0581354678aeeabd7680 Mon Sep 17 00:00:00 2001 From: vjpai Date: Tue, 21 Jun 2016 16:42:08 -0700 Subject: [PATCH 0153/1039] Fix review comments --- include/grpc++/impl/codegen/async_stream.h | 12 ++++++------ include/grpc++/impl/codegen/completion_queue.h | 11 ++++++----- include/grpc++/impl/codegen/sync_stream.h | 12 ++++++------ 3 files changed, 18 insertions(+), 17 deletions(-) diff --git a/include/grpc++/impl/codegen/async_stream.h b/include/grpc++/impl/codegen/async_stream.h index c74737ce5f8..cbc5d944295 100644 --- a/include/grpc++/impl/codegen/async_stream.h +++ b/include/grpc++/impl/codegen/async_stream.h @@ -53,7 +53,7 @@ class ClientAsyncStreamingInterface { /// Request notification of the reading of the initial metadata. Completion /// will be notified by \a tag on the associated completion queue. /// This call is optional, but if it is used, it cannot be used concurrently - /// with Read + /// with or after the \a Read method. /// /// \param[in] tag Tag identifying this request. virtual void ReadInitialMetadata(void* tag) = 0; @@ -74,8 +74,8 @@ class AsyncReaderInterface { /// Read a message of type \a R into \a msg. Completion will be notified by \a /// tag on the associated completion queue. - /// This is thread-safe with respect to other streaming APIs except for Finish - /// on the same stream. + /// This is thread-safe with respect to other streaming APIs except for \a + /// Finish on the same stream. /// /// \param[out] msg Where to eventually store the read message. /// \param[in] tag The tag identifying the operation. @@ -93,7 +93,7 @@ class AsyncWriterInterface { /// Only one write may be outstanding at any given time. This means that /// after calling Write, one must wait to receive \a tag from the completion /// queue BEFORE calling Write again. - /// This is thread-safe with respect to Read + /// This is thread-safe with respect to \a Read /// /// \param[in] msg The message to be written. /// \param[in] tag The tag identifying the operation. @@ -164,7 +164,7 @@ class ClientAsyncWriterInterface : public ClientAsyncStreamingInterface, public AsyncWriterInterface { public: /// Signal the client is done with the writes. - /// Thread-safe with respect to Read + /// Thread-safe with respect to \a Read /// /// \param[in] tag The tag identifying the operation. virtual void WritesDone(void* tag) = 0; @@ -236,7 +236,7 @@ class ClientAsyncReaderWriterInterface : public ClientAsyncStreamingInterface, public AsyncReaderInterface { public: /// Signal the client is done with the writes. - /// Thread-safe with respect to Read + /// Thread-safe with respect to \a Read /// /// \param[in] tag The tag identifying the operation. virtual void WritesDone(void* tag) = 0; diff --git a/include/grpc++/impl/codegen/completion_queue.h b/include/grpc++/impl/codegen/completion_queue.h index f138ebe7de2..03009e0561d 100644 --- a/include/grpc++/impl/codegen/completion_queue.h +++ b/include/grpc++/impl/codegen/completion_queue.h @@ -34,15 +34,16 @@ /// A completion queue implements a concurrent producer-consumer queue, with /// two main API-exposed methods: \a Next and \a AsyncNext. These /// methods are the essential component of the gRPC C++ asynchronous API. -/// There is also a Shutdown method to indicate that a given completion queue +/// There is also a \a Shutdown method to indicate that a given completion queue /// will no longer have regular events. This must be called before the /// completion queue is destroyed. /// All completion queue APIs are thread-safe and may be used concurrently with /// any other completion queue API invocation; it is acceptable to have -/// multiple threads calling Next or AsyncNext on the same or different -/// completion queues, or to call these methods concurrently with a Shutdown -/// elsewhere. All of these should be completed, though, before a completion -/// queue destructor is called. +/// multiple threads calling \a Next or \a AsyncNext on the same or different +/// completion queues, or to call these methods concurrently with a \a Shutdown +/// elsewhere. +/// \remark{All other API calls on completion queue should be completed before +/// a completion queue destructor is called.} #ifndef GRPCXX_IMPL_CODEGEN_COMPLETION_QUEUE_H #define GRPCXX_IMPL_CODEGEN_COMPLETION_QUEUE_H diff --git a/include/grpc++/impl/codegen/sync_stream.h b/include/grpc++/impl/codegen/sync_stream.h index 9d7966ba04b..e53717cabfc 100644 --- a/include/grpc++/impl/codegen/sync_stream.h +++ b/include/grpc++/impl/codegen/sync_stream.h @@ -71,8 +71,8 @@ class ReaderInterface { virtual ~ReaderInterface() {} /// Blocking read a message and parse to \a msg. Returns \a true on success. - /// This is thread-safe with respect to other streaming APIs except for Finish - /// on the same stream. (Finish must be called as described above.) + /// This is thread-safe with respect to other streaming APIs except for \a + /// Finish on the same stream. (\a Finish must be called as described above.) /// /// \param[out] msg The read message. /// @@ -89,7 +89,7 @@ class WriterInterface { virtual ~WriterInterface() {} /// Blocking write \a msg to the stream with options. - /// This is thread-safe with respect to Read + /// This is thread-safe with respect to \a Read /// /// \param msg The message to be written to the stream. /// \param options Options affecting the write operation. @@ -98,7 +98,7 @@ class WriterInterface { virtual bool Write(const W& msg, const WriteOptions& options) = 0; /// Blocking write \a msg to the stream with default options. - /// This is thread-safe with respect to Read + /// This is thread-safe with respect to \a Read /// /// \param msg The message to be written to the stream. /// @@ -179,7 +179,7 @@ class ClientWriterInterface : public ClientStreamingInterface, public: /// Half close writing from the client. /// Block until currently-pending writes are completed. - /// Thread safe with respect to Read operations only + /// Thread safe with respect to \a Read operations only /// /// \return Whether the writes were successful. virtual bool WritesDone() = 0; @@ -263,7 +263,7 @@ class ClientReaderWriterInterface : public ClientStreamingInterface, virtual void WaitForInitialMetadata() = 0; /// Block until currently-pending writes are completed. - /// Thread-safe with respect to Read + /// Thread-safe with respect to \a Read /// /// \return Whether the writes were successful. virtual bool WritesDone() = 0; From 3131c269c14f97294ebf8b6e3d1a235d4acf3317 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Tue, 21 Jun 2016 17:28:28 -0700 Subject: [PATCH 0154/1039] Integrate with unified error reporting --- src/core/lib/iomgr/ev_epoll_linux.c | 116 +++++++++++++++++++------- test/core/iomgr/ev_epoll_linux_test.c | 3 +- 2 files changed, 90 insertions(+), 29 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 7cc69c876db..d625b096a10 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -646,11 +646,18 @@ polling_island *polling_island_merge(polling_island *p, polling_island *q) { return q; } -static void polling_island_global_init() { +static grpc_error *polling_island_global_init() { + grpc_error *error = GRPC_ERROR_NONE; + gpr_mu_init(&g_pi_freelist_mu); g_pi_freelist = NULL; - grpc_wakeup_fd_init(&polling_island_wakeup_fd); - grpc_wakeup_fd_wakeup(&polling_island_wakeup_fd); + + error = grpc_wakeup_fd_init(&polling_island_wakeup_fd); + if (error == GRPC_ERROR_NONE) { + error = grpc_wakeup_fd_wakeup(&polling_island_wakeup_fd); + } + + return error; } static void polling_island_global_shutdown() { @@ -870,21 +877,33 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, } gpr_mu_unlock(&fd->pi_mu); - grpc_exec_ctx_enqueue(exec_ctx, fd->on_done_closure, true, NULL); + grpc_exec_ctx_sched(exec_ctx, fd->on_done_closure, GRPC_ERROR_NONE, NULL); gpr_mu_unlock(&fd->mu); UNREF_BY(fd, 2, reason); /* Drop the reference */ } +static grpc_error *fd_shutdown_error(bool shutdown) { + if (!shutdown) { + return GRPC_ERROR_NONE; + } else { + return GRPC_ERROR_CREATE("FD shutdown"); + } +} + static void notify_on_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure **st, grpc_closure *closure) { - if (*st == CLOSURE_NOT_READY) { + if (fd->shutdown) { + grpc_exec_ctx_sched(exec_ctx, closure, GRPC_ERROR_CREATE("FD shutdown"), + NULL); + } else if (*st == CLOSURE_NOT_READY) { /* not ready ==> switch to a waiting state by setting the closure */ *st = closure; } else if (*st == CLOSURE_READY) { /* already ready ==> queue the closure to run immediately */ *st = CLOSURE_NOT_READY; - grpc_exec_ctx_enqueue(exec_ctx, closure, !fd->shutdown, NULL); + grpc_exec_ctx_sched(exec_ctx, closure, fd_shutdown_error(fd->shutdown), + NULL); } else { /* upcallptr was set to a different closure. This is an error! */ gpr_log(GPR_ERROR, @@ -906,7 +925,7 @@ static int set_ready_locked(grpc_exec_ctx *exec_ctx, grpc_fd *fd, return 0; } else { /* waiting ==> queue closure */ - grpc_exec_ctx_enqueue(exec_ctx, *st, !fd->shutdown, NULL); + grpc_exec_ctx_sched(exec_ctx, *st, fd_shutdown_error(fd->shutdown), NULL); *st = CLOSURE_NOT_READY; return 1; } @@ -964,11 +983,11 @@ static void sig_handler(int sig_num) { static void poller_kick_init() { signal(grpc_wakeup_signal, sig_handler); } /* Global state management */ -static void pollset_global_init(void) { - grpc_wakeup_fd_init(&grpc_global_wakeup_fd); +static grpc_error *pollset_global_init(void) { gpr_tls_init(&g_current_thread_pollset); gpr_tls_init(&g_current_thread_worker); poller_kick_init(); + return grpc_wakeup_fd_init(&grpc_global_wakeup_fd); } static void pollset_global_shutdown(void) { @@ -977,8 +996,13 @@ static void pollset_global_shutdown(void) { gpr_tls_destroy(&g_current_thread_worker); } -static void pollset_worker_kick(grpc_pollset_worker *worker) { - pthread_kill(worker->pt_id, grpc_wakeup_signal); +static grpc_error *pollset_worker_kick(grpc_pollset_worker *worker) { + grpc_error *err = GRPC_ERROR_NONE; + int err_num = pthread_kill(worker->pt_id, grpc_wakeup_signal); + if (err_num != 0) { + err = GRPC_OS_ERROR(err_num, "pthread_kill"); + } + return err; } /* Return 1 if the pollset has active threads in pollset_work (pollset must @@ -1014,10 +1038,19 @@ static void push_front_worker(grpc_pollset *p, grpc_pollset_worker *worker) { worker->prev->next = worker->next->prev = worker; } +static void kick_append_error(grpc_error **composite, grpc_error *error) { + if (error == GRPC_ERROR_NONE) return; + if (*composite == GRPC_ERROR_NONE) { + *composite = GRPC_ERROR_CREATE("Kick Failure"); + } + *composite = grpc_error_add_child(*composite, error); +} + /* p->mu must be held before calling this function */ -static void pollset_kick(grpc_pollset *p, - grpc_pollset_worker *specific_worker) { +static grpc_error *pollset_kick(grpc_pollset *p, + grpc_pollset_worker *specific_worker) { GPR_TIMER_BEGIN("pollset_kick", 0); + grpc_error *error = GRPC_ERROR_NONE; grpc_pollset_worker *worker = specific_worker; if (worker != NULL) { @@ -1027,7 +1060,7 @@ static void pollset_kick(grpc_pollset *p, for (worker = p->root_worker.next; worker != &p->root_worker; worker = worker->next) { if (gpr_tls_get(&g_current_thread_worker) != (intptr_t)worker) { - pollset_worker_kick(worker); + kick_append_error(&error, pollset_worker_kick(worker)); } } } else { @@ -1037,7 +1070,7 @@ static void pollset_kick(grpc_pollset *p, } else { GPR_TIMER_MARK("kicked_specifically", 0); if (gpr_tls_get(&g_current_thread_worker) != (intptr_t)worker) { - pollset_worker_kick(worker); + kick_append_error(&error, pollset_worker_kick(worker)); } } } else if (gpr_tls_get(&g_current_thread_pollset) != (intptr_t)p) { @@ -1053,7 +1086,7 @@ static void pollset_kick(grpc_pollset *p, if (worker != NULL) { GPR_TIMER_MARK("finally_kick", 0); push_back_worker(p, worker); - pollset_worker_kick(worker); + kick_append_error(&error, pollset_worker_kick(worker)); } else { GPR_TIMER_MARK("kicked_no_pollers", 0); p->kicked_without_pollers = true; @@ -1061,9 +1094,13 @@ static void pollset_kick(grpc_pollset *p, } GPR_TIMER_END("pollset_kick", 0); + GRPC_LOG_IF_ERROR("pollset_kick", GRPC_ERROR_REF(error)); + return error; } -static void kick_poller(void) { grpc_wakeup_fd_wakeup(&grpc_global_wakeup_fd); } +static grpc_error *kick_poller(void) { + return grpc_wakeup_fd_wakeup(&grpc_global_wakeup_fd); +} static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { gpr_mu_init(&pollset->mu); @@ -1139,7 +1176,7 @@ static void finish_shutdown_locked(grpc_exec_ctx *exec_ctx, /* Release the ref and set pollset->polling_island to NULL */ pollset_release_polling_island(pollset, "ps_shutdown"); - grpc_exec_ctx_enqueue(exec_ctx, pollset->shutdown_done, true, NULL); + grpc_exec_ctx_sched(exec_ctx, pollset->shutdown_done, GRPC_ERROR_NONE, NULL); } /* pollset->mu lock must be held by the caller before calling this */ @@ -1181,14 +1218,23 @@ static void pollset_reset(grpc_pollset *pollset) { pollset_release_polling_island(pollset, "ps_reset"); } +static void work_combine_error(grpc_error **composite, grpc_error *error) { + if (error == GRPC_ERROR_NONE) return; + if (*composite == GRPC_ERROR_NONE) { + *composite = GRPC_ERROR_CREATE("pollset_work"); + } + *composite = grpc_error_add_child(*composite, error); +} + #define GRPC_EPOLL_MAX_EVENTS 1000 -static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, - grpc_pollset *pollset, int timeout_ms, - sigset_t *sig_mask) { +static grpc_error *pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset, + int timeout_ms, sigset_t *sig_mask) { struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; int epoll_fd = -1; int ep_rv; polling_island *pi = NULL; + grpc_error *error = GRPC_ERROR_NONE; GPR_TIMER_BEGIN("pollset_work_and_unlock", 0); /* We need to get the epoll_fd to wait on. The epoll_fd is in inside the @@ -1232,6 +1278,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, if (ep_rv < 0) { if (errno != EINTR) { gpr_log(GPR_ERROR, "epoll_pwait() failed: %s", strerror(errno)); + work_combine_error(&error, GRPC_OS_ERROR(errno, "epoll_pwait")); } else { /* We were interrupted. Save an interation by doing a zero timeout epoll_wait to see if there are any other events of interest */ @@ -1247,7 +1294,8 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, for (int i = 0; i < ep_rv; ++i) { void *data_ptr = ep_ev[i].data.ptr; if (data_ptr == &grpc_global_wakeup_fd) { - grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd); + work_combine_error( + &error, grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd)); } else if (data_ptr == &polling_island_wakeup_fd) { /* This means that our polling island is merged with a different island. We do not have to do anything here since the subsequent call @@ -1278,16 +1326,18 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, PI_UNREF(pi, "ps_work"); GPR_TIMER_END("pollset_work_and_unlock", 0); + return error; } /* pollset->mu lock must be held by the caller before calling this. The function pollset_work() may temporarily release the lock (pollset->mu) during the course of its execution but it will always re-acquire the lock and ensure that it is held by the time the function returns */ -static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, - grpc_pollset_worker **worker_hdl, gpr_timespec now, - gpr_timespec deadline) { +static grpc_error *pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, + grpc_pollset_worker **worker_hdl, + gpr_timespec now, gpr_timespec deadline) { GPR_TIMER_BEGIN("pollset_work", 0); + grpc_error *error = GRPC_ERROR_NONE; int timeout_ms = poll_deadline_to_millis_timeout(deadline, now); sigset_t new_mask; @@ -1316,7 +1366,7 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, push_front_worker(pollset, &worker); - pollset_work_and_unlock(exec_ctx, pollset, timeout_ms, &orig_mask); + error = pollset_work_and_unlock(exec_ctx, pollset, timeout_ms, &orig_mask); grpc_exec_ctx_flush(exec_ctx); gpr_mu_lock(&pollset->mu); @@ -1345,6 +1395,8 @@ static void pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, gpr_tls_set(&g_current_thread_pollset, (intptr_t)0); gpr_tls_set(&g_current_thread_worker, (intptr_t)0); GPR_TIMER_END("pollset_work", 0); + GRPC_LOG_IF_ERROR("pollset_work", GRPC_ERROR_REF(error)); + return error; } static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, @@ -1659,8 +1711,16 @@ const grpc_event_engine_vtable *grpc_init_epoll_linux(void) { } fd_global_init(); - pollset_global_init(); - polling_island_global_init(); + + if (!GRPC_LOG_IF_ERROR("pollset_global_init", pollset_global_init())) { + return NULL; + } + + if (!GRPC_LOG_IF_ERROR("polling_island_global_init", + polling_island_global_init())) { + return NULL; + } + return &vtable; } diff --git a/test/core/iomgr/ev_epoll_linux_test.c b/test/core/iomgr/ev_epoll_linux_test.c index 51da15faa7a..034f17fd58b 100644 --- a/test/core/iomgr/ev_epoll_linux_test.c +++ b/test/core/iomgr/ev_epoll_linux_test.c @@ -88,7 +88,8 @@ static void test_pollset_init(test_pollset *pollsets, int num_pollsets) { } } -static void destroy_pollset(grpc_exec_ctx *exec_ctx, void *p, bool success) { +static void destroy_pollset(grpc_exec_ctx *exec_ctx, void *p, + grpc_error *error) { grpc_pollset_destroy(p); } From 0100b2f1c0b08800ba0f7f53fe9cb5fbec7881a7 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Tue, 21 Jun 2016 17:38:13 -0700 Subject: [PATCH 0155/1039] Make fd_shutdown idempotent --- src/core/lib/iomgr/ev_epoll_linux.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index d625b096a10..c077987c01b 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -942,15 +942,19 @@ static grpc_pollset *fd_get_read_notifier_pollset(grpc_exec_ctx *exec_ctx, return notifier; } +/* Might be called multiple times */ static void fd_shutdown(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { gpr_mu_lock(&fd->mu); - GPR_ASSERT(!fd->shutdown); - fd->shutdown = true; - - /* Flush any pending read and write closures. Since fd->shutdown is 'true' at - this point, the closures would be called with 'success = false' */ - set_ready_locked(exec_ctx, fd, &fd->read_closure); - set_ready_locked(exec_ctx, fd, &fd->write_closure); + /* Do the actual shutdown only once */ + if (!fd->shutdown) { + fd->shutdown = true; + + shutdown(fd->fd, SHUT_RDWR); + /* Flush any pending read and write closures. Since fd->shutdown is 'true' + at this point, the closures would be called with 'success = false' */ + set_ready_locked(exec_ctx, fd, &fd->read_closure); + set_ready_locked(exec_ctx, fd, &fd->write_closure); + } gpr_mu_unlock(&fd->mu); } From 24b6eae1fc71a4f5d18eb2e7c1cbca5b4e54a46f Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Tue, 21 Jun 2016 18:01:14 -0700 Subject: [PATCH 0156/1039] Add missing function fd_is_shutdown --- src/core/lib/iomgr/ev_epoll_linux.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index c077987c01b..3a774a8876f 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -942,6 +942,13 @@ static grpc_pollset *fd_get_read_notifier_pollset(grpc_exec_ctx *exec_ctx, return notifier; } +static bool fd_is_shutdown(grpc_fd *fd) { + gpr_mu_lock(&fd->mu); + const bool r = fd->shutdown; + gpr_mu_unlock(&fd->mu); + return r; +} + /* Might be called multiple times */ static void fd_shutdown(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { gpr_mu_lock(&fd->mu); @@ -1659,6 +1666,7 @@ static const grpc_event_engine_vtable vtable = { .fd_wrapped_fd = fd_wrapped_fd, .fd_orphan = fd_orphan, .fd_shutdown = fd_shutdown, + .fd_is_shutdown = fd_is_shutdown, .fd_notify_on_read = fd_notify_on_read, .fd_notify_on_write = fd_notify_on_write, .fd_get_read_notifier_pollset = fd_get_read_notifier_pollset, From bd28936c8648cde481fa20ca64ba00d9281cb119 Mon Sep 17 00:00:00 2001 From: vjpai Date: Tue, 21 Jun 2016 18:03:37 -0700 Subject: [PATCH 0157/1039] Properly handle reviewer comment re concurrent Read --- include/grpc++/impl/codegen/async_stream.h | 7 +++++-- include/grpc++/impl/codegen/sync_stream.h | 5 +++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/include/grpc++/impl/codegen/async_stream.h b/include/grpc++/impl/codegen/async_stream.h index cbc5d944295..e96d224ddbe 100644 --- a/include/grpc++/impl/codegen/async_stream.h +++ b/include/grpc++/impl/codegen/async_stream.h @@ -74,8 +74,11 @@ class AsyncReaderInterface { /// Read a message of type \a R into \a msg. Completion will be notified by \a /// tag on the associated completion queue. - /// This is thread-safe with respect to other streaming APIs except for \a - /// Finish on the same stream. + /// This is thread-safe with respect to \a Write or \a WritesDone methods. It + /// should not be called concurrently with other streaming APIs + /// on the same stream. It is not meaningful to call it concurrently + /// with another \a Read on the same stream since reads on the same stream + /// are delivered in order. /// /// \param[out] msg Where to eventually store the read message. /// \param[in] tag The tag identifying the operation. diff --git a/include/grpc++/impl/codegen/sync_stream.h b/include/grpc++/impl/codegen/sync_stream.h index e53717cabfc..cbfa4106995 100644 --- a/include/grpc++/impl/codegen/sync_stream.h +++ b/include/grpc++/impl/codegen/sync_stream.h @@ -71,8 +71,9 @@ class ReaderInterface { virtual ~ReaderInterface() {} /// Blocking read a message and parse to \a msg. Returns \a true on success. - /// This is thread-safe with respect to other streaming APIs except for \a - /// Finish on the same stream. (\a Finish must be called as described above.) + /// This is thread-safe with respect to \a Write or \WritesDone methods on + /// the same stream. It should not be called concurrently with another \a + /// Read on the same stream as the order of delivery will not be defined. /// /// \param[out] msg The read message. /// From 229533b1e68c4a4b8a67148f7fe25543584131f6 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Tue, 21 Jun 2016 20:42:52 -0700 Subject: [PATCH 0158/1039] Remove pollset->pi_mu since it is redundant. Also do not get polling island lock in the fast-path --- src/core/lib/iomgr/ev_epoll_linux.c | 82 ++++++++++++++++------------- src/core/lib/iomgr/ev_posix.c | 6 +-- 2 files changed, 46 insertions(+), 42 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 3a774a8876f..6464d3ba348 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -207,12 +207,7 @@ struct grpc_pollset { bool finish_shutdown_called; /* Is the 'finish_shutdown_locked()' called ? */ grpc_closure *shutdown_done; /* Called after after shutdown is complete */ - /* The polling island to which this pollset belongs to and the mutex - protecting the field */ - /* TODO: sreek: This lock might actually be adding more overhead to the - critical path (i.e pollset_work() function). Consider removing this lock - and just using the overall pollset lock */ - gpr_mu pi_mu; + /* The polling island to which this pollset belongs to */ struct polling_island *polling_island; }; @@ -488,31 +483,47 @@ static void polling_island_delete(polling_island *pi) { gpr_mu_unlock(&g_pi_freelist_mu); } +/* Attempts to gets the last polling island in the linked list (liked by the + * 'merged_to' field). Since this does not lock the polling island, there are no + * guarantees that the island returned is the last island */ +static polling_island *polling_island_maybe_get_latest(polling_island *pi) { + polling_island *next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); + while (next != NULL) { + pi = next; + next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); + } + + return pi; +} + /* Gets the lock on the *latest* polling island i.e the last polling island in - the linked list (linked by 'merged_to' link). Call gpr_mu_unlock on the + the linked list (linked by the 'merged_to' field). Call gpr_mu_unlock on the returned polling island's mu. Usage: To lock/unlock polling island "pi", do the following: polling_island *pi_latest = polling_island_lock(pi); ... ... critical section .. ... - gpr_mu_unlock(&pi_latest->mu); //NOTE: use pi_latest->mu. NOT pi->mu */ -polling_island *polling_island_lock(polling_island *pi) { + gpr_mu_unlock(&pi_latest->mu); // NOTE: use pi_latest->mu. NOT pi->mu */ +static polling_island *polling_island_lock(polling_island *pi) { polling_island *next = NULL; + while (true) { next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); if (next == NULL) { - /* pi is the last node in the linked list. Get the lock and check again - (under the pi->mu lock) that pi is still the last node (because a merge - may have happend after the (next == NULL) check above and before - getting the pi->mu lock. - If pi is the last node, we are done. If not, unlock and continue - traversing the list */ + /* Looks like 'pi' is the last node in the linked list but unless we check + this by holding the pi->mu lock, we cannot be sure (i.e without the + pi->mu lock, we don't prevent island merges). + To be absolutely sure, check once more by holding the pi->mu lock */ gpr_mu_lock(&pi->mu); next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); if (next == NULL) { + /* pi is infact the last node and we have the pi->mu lock. we're done */ break; } + + /* pi->merged_to is not NULL i.e pi isn't the last node anymore. pi->mu + * isn't the lock we are interested in. Continue traversing the list */ gpr_mu_unlock(&pi->mu); } @@ -526,11 +537,11 @@ polling_island *polling_island_lock(polling_island *pi) { This function is needed because calling the following block of code to obtain locks on polling islands (*p and *q) is prone to deadlocks. { - polling_island_lock(*p); - polling_island_lock(*q); + polling_island_lock(*p, true); + polling_island_lock(*q, true); } - Usage/exmaple: + Usage/example: polling_island *p1; polling_island *p2; .. @@ -551,7 +562,7 @@ polling_island *polling_island_lock(polling_island *pi) { } */ -void polling_island_lock_pair(polling_island **p, polling_island **q) { +static void polling_island_lock_pair(polling_island **p, polling_island **q) { polling_island *pi_1 = *p; polling_island *pi_2 = *q; polling_island *next_1 = NULL; @@ -611,7 +622,8 @@ void polling_island_lock_pair(polling_island **p, polling_island **q) { *q = pi_2; } -polling_island *polling_island_merge(polling_island *p, polling_island *q) { +static polling_island *polling_island_merge(polling_island *p, + polling_island *q) { /* Get locks on both the polling islands */ polling_island_lock_pair(&p, &q); @@ -1124,7 +1136,6 @@ static void pollset_init(grpc_pollset *pollset, gpr_mu **mu) { pollset->finish_shutdown_called = false; pollset->shutdown_done = NULL; - gpr_mu_init(&pollset->pi_mu); pollset->polling_island = NULL; } @@ -1170,12 +1181,10 @@ static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { } static void pollset_release_polling_island(grpc_pollset *ps, char *reason) { - gpr_mu_lock(&ps->pi_mu); if (ps->polling_island != NULL) { PI_UNREF(ps->polling_island, reason); } ps->polling_island = NULL; - gpr_mu_unlock(&ps->pi_mu); } static void finish_shutdown_locked(grpc_exec_ctx *exec_ctx, @@ -1215,7 +1224,6 @@ static void pollset_shutdown(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, * here */ static void pollset_destroy(grpc_pollset *pollset) { GPR_ASSERT(!pollset_has_workers(pollset)); - gpr_mu_destroy(&pollset->pi_mu); gpr_mu_destroy(&pollset->mu); } @@ -1250,22 +1258,25 @@ static grpc_error *pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, /* We need to get the epoll_fd to wait on. The epoll_fd is in inside the latest polling island pointed by pollset->polling_island. - Acquire the following locks: - - pollset->mu (which we already have) - - pollset->pi_mu - - pollset->polling_island lock */ - gpr_mu_lock(&pollset->pi_mu); + + Since epoll_fd is immutable, we can read it without obtaining the polling + island lock. There is however a possibility that the polling island (from + which we got the epoll_fd) got merged with another island while we are + in this function. This is still okay because in such a case, we will wakeup + right-away from epoll_wait() and pick up the latest polling_island the next + this function (i.e pollset_work_and_unlock()) is called. + */ if (pollset->polling_island == NULL) { pollset->polling_island = polling_island_create(NULL); PI_ADD_REF(pollset->polling_island, "ps"); } - pi = polling_island_lock(pollset->polling_island); + pi = polling_island_maybe_get_latest(pollset->polling_island); epoll_fd = pi->epoll_fd; /* Update the pollset->polling_island since the island being pointed by - pollset->polling_island may not be the latest (i.e pi) */ + pollset->polling_island maybe older than the one pointed by pi) */ if (pollset->polling_island != pi) { /* Always do PI_ADD_REF before PI_UNREF because PI_UNREF may cause the polling island to be deleted */ @@ -1278,9 +1289,6 @@ static grpc_error *pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, the epoll_fd won't be closed) while we are are doing an epoll_wait() on the epoll_fd */ PI_ADD_REF(pi, "ps_work"); - - gpr_mu_unlock(&pi->mu); - gpr_mu_unlock(&pollset->pi_mu); gpr_mu_unlock(&pollset->mu); do { @@ -1413,7 +1421,6 @@ static grpc_error *pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { gpr_mu_lock(&pollset->mu); - gpr_mu_lock(&pollset->pi_mu); gpr_mu_lock(&fd->pi_mu); polling_island *pi_new = NULL; @@ -1465,7 +1472,6 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, } gpr_mu_unlock(&fd->pi_mu); - gpr_mu_unlock(&pollset->pi_mu); gpr_mu_unlock(&pollset->mu); } @@ -1627,9 +1633,9 @@ void *grpc_fd_get_polling_island(grpc_fd *fd) { void *grpc_pollset_get_polling_island(grpc_pollset *ps) { polling_island *pi; - gpr_mu_lock(&ps->pi_mu); + gpr_mu_lock(&ps->mu); pi = ps->polling_island; - gpr_mu_unlock(&ps->pi_mu); + gpr_mu_unlock(&ps->mu); return pi; } diff --git a/src/core/lib/iomgr/ev_posix.c b/src/core/lib/iomgr/ev_posix.c index 4cdd13bbdba..a3c1e9db9a0 100644 --- a/src/core/lib/iomgr/ev_posix.c +++ b/src/core/lib/iomgr/ev_posix.c @@ -54,7 +54,7 @@ grpc_poll_function_type grpc_poll_function = poll; static const grpc_event_engine_vtable *g_event_engine; -static const char* g_poll_strategy_name = NULL; +static const char *g_poll_strategy_name = NULL; typedef const grpc_event_engine_vtable *(*event_engine_factory_fn)(void); @@ -111,9 +111,7 @@ static void try_engine(const char *engine) { } /* Call this only after calling grpc_event_engine_init() */ -const char *grpc_get_poll_strategy_name() { - return g_poll_strategy_name; -} +const char *grpc_get_poll_strategy_name() { return g_poll_strategy_name; } void grpc_event_engine_init(void) { char *s = gpr_getenv("GRPC_POLL_STRATEGY"); From caca0a15ded56b3732e0f67fe7d1507d9a7abf07 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Tue, 21 Jun 2016 23:07:49 -0700 Subject: [PATCH 0159/1039] Build objc examples as part of automatic tests --- src/objective-c/tests/build_example_test.sh | 56 +++++++++++++++++++ src/objective-c/tests/build_one_example.sh | 62 +++++++++++++++++++++ tools/run_tests/run_tests.py | 4 +- 3 files changed, 121 insertions(+), 1 deletion(-) create mode 100755 src/objective-c/tests/build_example_test.sh create mode 100755 src/objective-c/tests/build_one_example.sh diff --git a/src/objective-c/tests/build_example_test.sh b/src/objective-c/tests/build_example_test.sh new file mode 100755 index 00000000000..3cc0c552349 --- /dev/null +++ b/src/objective-c/tests/build_example_test.sh @@ -0,0 +1,56 @@ +#!/bin/bash +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# Don't run this script standalone. Instead, run from the repository root: +# ./tools/run_tests/run_tests.py -l objc + +set -eo pipefail + +cd `dirname $0` + +EXAMPLE_PATH=examples/objective-c/helloworld \ +SCHEME=HelloWorld \ +./build_one_example.sh + +EXAMPLE_PATH=examples/objective-c/route_guide \ +SCHEME=RouteGuideClient \ +./build_one_example.sh + +EXAMPLE_PATH=examples/objective-c/auth_sample \ +SCHEME=AuthSample \ +./build_one_example.sh + +EXAMPLE_PATH=src/objective-c/examples/Sample \ +SCHEME=Sample \ +./build_one_example.sh + +EXAMPLE_PATH=src/objective-c/examples/SwiftSample \ +SCHEME=SwiftSample \ +./build_one_example.sh diff --git a/src/objective-c/tests/build_one_example.sh b/src/objective-c/tests/build_one_example.sh new file mode 100755 index 00000000000..131fbd780df --- /dev/null +++ b/src/objective-c/tests/build_one_example.sh @@ -0,0 +1,62 @@ +#!/bin/bash +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# Don't run this script standalone. Instead, run from the repository root: +# ./tools/run_tests/run_tests.py -l objc + +set -e + +# Params: +# EXAMPLE_PATH - directory of the example +# SCHEME - scheme of the example, used by xcodebuild + +# CocoaPods requires the terminal to be using UTF-8 encoding. +export LANG=en_US.UTF-8 + +cd `dirname $0`/../../.. + +cd $EXAMPLE_PATH + +# clean the directory +git checkout . +git clean -df . +rm -rf Pods + +pod install + +set -o pipefail +XCODEBUILD_FILTER='(^===|^\*\*|\bfatal\b|\berror\b|\bwarning\b|\bfail)' +xcodebuild \ + clean build \ + -workspace *.xcworkspace \ + -scheme $SCHEME \ + -destination name="iPhone 6" \ + | egrep "$XCODEBUILD_FILTER" \ + | egrep -v "(GPBDictionary|GPBArray)" - diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index 9ef12c5b07e..0d77f0dbf97 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -425,7 +425,7 @@ class PythonLanguage(object): return [] def build_steps(self): - return [['tools/run_tests/build_python.sh', tox_env] + return [['tools/run_tests/build_python.sh', tox_env] for tox_env in self._tox_envs] def post_tests_steps(self): @@ -602,6 +602,8 @@ class ObjCLanguage(object): def test_specs(self): return [self.config.job_spec(['src/objective-c/tests/run_tests.sh'], None, + environ=_FORCE_ENVIRON_FOR_WRAPPERS), + self.config.job_spec(['src/objective-c/tests/build_example_test.sh'], None, environ=_FORCE_ENVIRON_FOR_WRAPPERS)] def pre_build_steps(self): From f95a4898abd5888c34b31354cdbe9a909958aa08 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Tue, 21 Jun 2016 23:27:46 -0700 Subject: [PATCH 0160/1039] Fixed format issues --- src/objective-c/tests/build_example_test.sh | 2 +- src/objective-c/tests/build_one_example.sh | 2 +- tools/run_tests/run_tests.py | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/objective-c/tests/build_example_test.sh b/src/objective-c/tests/build_example_test.sh index 3cc0c552349..73405695e8e 100755 --- a/src/objective-c/tests/build_example_test.sh +++ b/src/objective-c/tests/build_example_test.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Copyright 2015, Google Inc. +# Copyright 2016, Google Inc. # All rights reserved. # # Redistribution and use in source and binary forms, with or without diff --git a/src/objective-c/tests/build_one_example.sh b/src/objective-c/tests/build_one_example.sh index 131fbd780df..56f686319e1 100755 --- a/src/objective-c/tests/build_one_example.sh +++ b/src/objective-c/tests/build_one_example.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Copyright 2015, Google Inc. +# Copyright 2016, Google Inc. # All rights reserved. # # Redistribution and use in source and binary forms, with or without diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index 0d77f0dbf97..580afc32de4 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -603,8 +603,8 @@ class ObjCLanguage(object): def test_specs(self): return [self.config.job_spec(['src/objective-c/tests/run_tests.sh'], None, environ=_FORCE_ENVIRON_FOR_WRAPPERS), - self.config.job_spec(['src/objective-c/tests/build_example_test.sh'], None, - environ=_FORCE_ENVIRON_FOR_WRAPPERS)] + self.config.job_spec(['src/objective-c/tests/build_example_test.sh'], + None, environ=_FORCE_ENVIRON_FOR_WRAPPERS)] def pre_build_steps(self): return [] From db3ffe13cbb37ea797890c455a4543c9d030adf5 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Wed, 22 Jun 2016 01:12:42 -0700 Subject: [PATCH 0161/1039] Update examples with Cocoapods 1.0.0 --- .../AuthSample.xcodeproj/project.pbxproj | 28 ++++++++--- .../HelloWorld.xcodeproj/project.pbxproj | 24 ++++----- .../project.pbxproj | 18 +++---- .../Sample/Sample.xcodeproj/project.pbxproj | 50 ++++++++++++------- .../SwiftSample.xcodeproj/project.pbxproj | 42 ++++++++-------- 5 files changed, 95 insertions(+), 67 deletions(-) diff --git a/examples/objective-c/auth_sample/AuthSample.xcodeproj/project.pbxproj b/examples/objective-c/auth_sample/AuthSample.xcodeproj/project.pbxproj index 51a39c578c7..ab7419c9bcd 100644 --- a/examples/objective-c/auth_sample/AuthSample.xcodeproj/project.pbxproj +++ b/examples/objective-c/auth_sample/AuthSample.xcodeproj/project.pbxproj @@ -116,11 +116,12 @@ isa = PBXNativeTarget; buildConfigurationList = 63E1E9A21B28CB2100EF0978 /* Build configuration list for PBXNativeTarget "AuthSample" */; buildPhases = ( - DAABBA7B5788A39108D7CA83 /* Check Pods Manifest.lock */, + DAABBA7B5788A39108D7CA83 /* [CP] Check Pods Manifest.lock */, 63E1E9781B28CB2000EF0978 /* Sources */, 63E1E9791B28CB2000EF0978 /* Frameworks */, 63E1E97A1B28CB2000EF0978 /* Resources */, - AEFCCC69DD59CE8F6EB769D7 /* Copy Pods Resources */, + AEFCCC69DD59CE8F6EB769D7 /* [CP] Copy Pods Resources */, + D24F6598302C412D4B863D6F /* [CP] Embed Pods Frameworks */, ); buildRules = ( ); @@ -177,14 +178,14 @@ /* End PBXResourcesBuildPhase section */ /* Begin PBXShellScriptBuildPhase section */ - AEFCCC69DD59CE8F6EB769D7 /* Copy Pods Resources */ = { + AEFCCC69DD59CE8F6EB769D7 /* [CP] Copy Pods Resources */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Copy Pods Resources"; + name = "[CP] Copy Pods Resources"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; @@ -192,14 +193,29 @@ shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-AuthSample/Pods-AuthSample-resources.sh\"\n"; showEnvVarsInLog = 0; }; - DAABBA7B5788A39108D7CA83 /* Check Pods Manifest.lock */ = { + D24F6598302C412D4B863D6F /* [CP] Embed Pods Frameworks */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Check Pods Manifest.lock"; + name = "[CP] Embed Pods Frameworks"; + outputPaths = ( + ); + runOnlyForDeploymentPostprocessing = 0; + shellPath = /bin/sh; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-AuthSample/Pods-AuthSample-frameworks.sh\"\n"; + showEnvVarsInLog = 0; + }; + DAABBA7B5788A39108D7CA83 /* [CP] Check Pods Manifest.lock */ = { + isa = PBXShellScriptBuildPhase; + buildActionMask = 2147483647; + files = ( + ); + inputPaths = ( + ); + name = "[CP] Check Pods Manifest.lock"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; diff --git a/examples/objective-c/helloworld/HelloWorld.xcodeproj/project.pbxproj b/examples/objective-c/helloworld/HelloWorld.xcodeproj/project.pbxproj index 250f009996f..df5c40cda20 100644 --- a/examples/objective-c/helloworld/HelloWorld.xcodeproj/project.pbxproj +++ b/examples/objective-c/helloworld/HelloWorld.xcodeproj/project.pbxproj @@ -7,7 +7,6 @@ objects = { /* Begin PBXBuildFile section */ - 3EF35C14BDC2B65E21837F02 /* libPods.a in Frameworks */ = {isa = PBXBuildFile; fileRef = 43AB08B32839A6700EA00DD4 /* libPods.a */; }; 5E3690661B2A23800040F884 /* main.m in Sources */ = {isa = PBXBuildFile; fileRef = 5E3690651B2A23800040F884 /* main.m */; }; 5E3690691B2A23800040F884 /* AppDelegate.m in Sources */ = {isa = PBXBuildFile; fileRef = 5E3690681B2A23800040F884 /* AppDelegate.m */; }; 5E36906C1B2A23800040F884 /* ViewController.m in Sources */ = {isa = PBXBuildFile; fileRef = 5E36906B1B2A23800040F884 /* ViewController.m */; }; @@ -18,7 +17,6 @@ /* Begin PBXFileReference section */ 0C432EF610DB15C0F47A66BB /* Pods-HelloWorld.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-HelloWorld.release.xcconfig"; path = "Pods/Target Support Files/Pods-HelloWorld/Pods-HelloWorld.release.xcconfig"; sourceTree = ""; }; - 43AB08B32839A6700EA00DD4 /* libPods.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = libPods.a; sourceTree = BUILT_PRODUCTS_DIR; }; 5E3690601B2A23800040F884 /* HelloWorld.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = HelloWorld.app; sourceTree = BUILT_PRODUCTS_DIR; }; 5E3690641B2A23800040F884 /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = ""; }; 5E3690651B2A23800040F884 /* main.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = main.m; sourceTree = ""; }; @@ -37,7 +35,6 @@ buildActionMask = 2147483647; files = ( EF61CF6AE2536A31D47F0E63 /* libPods-HelloWorld.a in Frameworks */, - 3EF35C14BDC2B65E21837F02 /* libPods.a in Frameworks */, ); runOnlyForDeploymentPostprocessing = 0; }; @@ -88,7 +85,6 @@ isa = PBXGroup; children = ( 6B4E1F55F8A2EC95A0E7EE88 /* libPods-HelloWorld.a */, - 43AB08B32839A6700EA00DD4 /* libPods.a */, ); name = Frameworks; sourceTree = ""; @@ -109,12 +105,12 @@ isa = PBXNativeTarget; buildConfigurationList = 5E3690831B2A23810040F884 /* Build configuration list for PBXNativeTarget "HelloWorld" */; buildPhases = ( - ACF9162361FB8F24C70657DE /* Check Pods Manifest.lock */, + ACF9162361FB8F24C70657DE /* [CP] Check Pods Manifest.lock */, 5E36905C1B2A23800040F884 /* Sources */, 5E36905D1B2A23800040F884 /* Frameworks */, 5E36905E1B2A23800040F884 /* Resources */, - 4C7D815378D98AB3BFC1A7D5 /* Copy Pods Resources */, - BB76529986A8BFAF19A385B1 /* Embed Pods Frameworks */, + 4C7D815378D98AB3BFC1A7D5 /* [CP] Copy Pods Resources */, + BB76529986A8BFAF19A385B1 /* [CP] Embed Pods Frameworks */, ); buildRules = ( ); @@ -170,14 +166,14 @@ /* End PBXResourcesBuildPhase section */ /* Begin PBXShellScriptBuildPhase section */ - 4C7D815378D98AB3BFC1A7D5 /* Copy Pods Resources */ = { + 4C7D815378D98AB3BFC1A7D5 /* [CP] Copy Pods Resources */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Copy Pods Resources"; + name = "[CP] Copy Pods Resources"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; @@ -185,14 +181,14 @@ shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-HelloWorld/Pods-HelloWorld-resources.sh\"\n"; showEnvVarsInLog = 0; }; - ACF9162361FB8F24C70657DE /* Check Pods Manifest.lock */ = { + ACF9162361FB8F24C70657DE /* [CP] Check Pods Manifest.lock */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Check Pods Manifest.lock"; + name = "[CP] Check Pods Manifest.lock"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; @@ -200,19 +196,19 @@ shellScript = "diff \"${PODS_ROOT}/../Podfile.lock\" \"${PODS_ROOT}/Manifest.lock\" > /dev/null\nif [[ $? != 0 ]] ; then\n cat << EOM\nerror: The sandbox is not in sync with the Podfile.lock. Run 'pod install' or update your CocoaPods installation.\nEOM\n exit 1\nfi\n"; showEnvVarsInLog = 0; }; - BB76529986A8BFAF19A385B1 /* Embed Pods Frameworks */ = { + BB76529986A8BFAF19A385B1 /* [CP] Embed Pods Frameworks */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Embed Pods Frameworks"; + name = "[CP] Embed Pods Frameworks"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods/Pods-frameworks.sh\"\n"; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-HelloWorld/Pods-HelloWorld-frameworks.sh\"\n"; showEnvVarsInLog = 0; }; /* End PBXShellScriptBuildPhase section */ diff --git a/examples/objective-c/route_guide/RouteGuideClient.xcodeproj/project.pbxproj b/examples/objective-c/route_guide/RouteGuideClient.xcodeproj/project.pbxproj index f99775562c0..0bb84b3b905 100644 --- a/examples/objective-c/route_guide/RouteGuideClient.xcodeproj/project.pbxproj +++ b/examples/objective-c/route_guide/RouteGuideClient.xcodeproj/project.pbxproj @@ -116,12 +116,12 @@ isa = PBXNativeTarget; buildConfigurationList = 632527A31B1D0396003073D9 /* Build configuration list for PBXNativeTarget "RouteGuideClient" */; buildPhases = ( - C6FC30AD2376EC04317237C5 /* Check Pods Manifest.lock */, + C6FC30AD2376EC04317237C5 /* [CP] Check Pods Manifest.lock */, 632527791B1D0395003073D9 /* Sources */, 6325277A1B1D0395003073D9 /* Frameworks */, 6325277B1B1D0395003073D9 /* Resources */, - FFE0BCF30339E7A50A989EAB /* Copy Pods Resources */, - B5388EC5A25E89021740B916 /* Embed Pods Frameworks */, + FFE0BCF30339E7A50A989EAB /* [CP] Copy Pods Resources */, + B5388EC5A25E89021740B916 /* [CP] Embed Pods Frameworks */, ); buildRules = ( ); @@ -178,14 +178,14 @@ /* End PBXResourcesBuildPhase section */ /* Begin PBXShellScriptBuildPhase section */ - B5388EC5A25E89021740B916 /* Embed Pods Frameworks */ = { + B5388EC5A25E89021740B916 /* [CP] Embed Pods Frameworks */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Embed Pods Frameworks"; + name = "[CP] Embed Pods Frameworks"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; @@ -193,14 +193,14 @@ shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-RouteGuideClient/Pods-RouteGuideClient-frameworks.sh\"\n"; showEnvVarsInLog = 0; }; - C6FC30AD2376EC04317237C5 /* Check Pods Manifest.lock */ = { + C6FC30AD2376EC04317237C5 /* [CP] Check Pods Manifest.lock */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Check Pods Manifest.lock"; + name = "[CP] Check Pods Manifest.lock"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; @@ -208,14 +208,14 @@ shellScript = "diff \"${PODS_ROOT}/../Podfile.lock\" \"${PODS_ROOT}/Manifest.lock\" > /dev/null\nif [[ $? != 0 ]] ; then\n cat << EOM\nerror: The sandbox is not in sync with the Podfile.lock. Run 'pod install' or update your CocoaPods installation.\nEOM\n exit 1\nfi\n"; showEnvVarsInLog = 0; }; - FFE0BCF30339E7A50A989EAB /* Copy Pods Resources */ = { + FFE0BCF30339E7A50A989EAB /* [CP] Copy Pods Resources */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Copy Pods Resources"; + name = "[CP] Copy Pods Resources"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; diff --git a/src/objective-c/examples/Sample/Sample.xcodeproj/project.pbxproj b/src/objective-c/examples/Sample/Sample.xcodeproj/project.pbxproj index 611eb6032d5..5c2a6d14f96 100644 --- a/src/objective-c/examples/Sample/Sample.xcodeproj/project.pbxproj +++ b/src/objective-c/examples/Sample/Sample.xcodeproj/project.pbxproj @@ -7,16 +7,16 @@ objects = { /* Begin PBXBuildFile section */ + 426A5020E0E158A101BCA1D9 /* libPods-Sample.a in Frameworks */ = {isa = PBXBuildFile; fileRef = C20055928615A6F8434E26B4 /* libPods-Sample.a */; }; 6369A2701A9322E20015FC5C /* main.m in Sources */ = {isa = PBXBuildFile; fileRef = 6369A26F1A9322E20015FC5C /* main.m */; }; 6369A2731A9322E20015FC5C /* AppDelegate.m in Sources */ = {isa = PBXBuildFile; fileRef = 6369A2721A9322E20015FC5C /* AppDelegate.m */; }; 6369A2761A9322E20015FC5C /* ViewController.m in Sources */ = {isa = PBXBuildFile; fileRef = 6369A2751A9322E20015FC5C /* ViewController.m */; }; 6369A2791A9322E20015FC5C /* Main.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 6369A2771A9322E20015FC5C /* Main.storyboard */; }; 6369A27B1A9322E20015FC5C /* Images.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 6369A27A1A9322E20015FC5C /* Images.xcassets */; }; - FC81FE63CA655031F3524EC0 /* libPods.a in Frameworks */ = {isa = PBXBuildFile; fileRef = 2DC7B7C4C0410F43B9621631 /* libPods.a */; }; /* End PBXBuildFile section */ /* Begin PBXFileReference section */ - 2DC7B7C4C0410F43B9621631 /* libPods.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = libPods.a; sourceTree = BUILT_PRODUCTS_DIR; }; + 5A8C9F4B28733B249DE4AB6D /* Pods-Sample.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-Sample.release.xcconfig"; path = "Pods/Target Support Files/Pods-Sample/Pods-Sample.release.xcconfig"; sourceTree = ""; }; 6369A26A1A9322E20015FC5C /* Sample.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = Sample.app; sourceTree = BUILT_PRODUCTS_DIR; }; 6369A26E1A9322E20015FC5C /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = ""; }; 6369A26F1A9322E20015FC5C /* main.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = main.m; sourceTree = ""; }; @@ -26,8 +26,8 @@ 6369A2751A9322E20015FC5C /* ViewController.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = ViewController.m; sourceTree = ""; }; 6369A2781A9322E20015FC5C /* Base */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; name = Base; path = Base.lproj/Main.storyboard; sourceTree = ""; }; 6369A27A1A9322E20015FC5C /* Images.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Images.xcassets; sourceTree = ""; }; - AC29DD6FCDF962F519FEBB0D /* Pods.debug.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = Pods.debug.xcconfig; path = "Pods/Target Support Files/Pods/Pods.debug.xcconfig"; sourceTree = ""; }; - C68330F8D451CC6ACEABA09F /* Pods.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = Pods.release.xcconfig; path = "Pods/Target Support Files/Pods/Pods.release.xcconfig"; sourceTree = ""; }; + C20055928615A6F8434E26B4 /* libPods-Sample.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = "libPods-Sample.a"; sourceTree = BUILT_PRODUCTS_DIR; }; + E3C01DF315C4E7433BCEC6E6 /* Pods-Sample.debug.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-Sample.debug.xcconfig"; path = "Pods/Target Support Files/Pods-Sample/Pods-Sample.debug.xcconfig"; sourceTree = ""; }; /* End PBXFileReference section */ /* Begin PBXFrameworksBuildPhase section */ @@ -35,7 +35,7 @@ isa = PBXFrameworksBuildPhase; buildActionMask = 2147483647; files = ( - FC81FE63CA655031F3524EC0 /* libPods.a in Frameworks */, + 426A5020E0E158A101BCA1D9 /* libPods-Sample.a in Frameworks */, ); runOnlyForDeploymentPostprocessing = 0; }; @@ -86,8 +86,8 @@ AB3331C9AE6488E61B2B094E /* Pods */ = { isa = PBXGroup; children = ( - AC29DD6FCDF962F519FEBB0D /* Pods.debug.xcconfig */, - C68330F8D451CC6ACEABA09F /* Pods.release.xcconfig */, + E3C01DF315C4E7433BCEC6E6 /* Pods-Sample.debug.xcconfig */, + 5A8C9F4B28733B249DE4AB6D /* Pods-Sample.release.xcconfig */, ); name = Pods; sourceTree = ""; @@ -95,7 +95,7 @@ C4C2C5219053E079C9EFB930 /* Frameworks */ = { isa = PBXGroup; children = ( - 2DC7B7C4C0410F43B9621631 /* libPods.a */, + C20055928615A6F8434E26B4 /* libPods-Sample.a */, ); name = Frameworks; sourceTree = ""; @@ -107,11 +107,12 @@ isa = PBXNativeTarget; buildConfigurationList = 6369A28D1A9322E20015FC5C /* Build configuration list for PBXNativeTarget "Sample" */; buildPhases = ( - 41F7486D8F66994B0BFB84AF /* Check Pods Manifest.lock */, + 41F7486D8F66994B0BFB84AF /* [CP] Check Pods Manifest.lock */, 6369A2661A9322E20015FC5C /* Sources */, 6369A2671A9322E20015FC5C /* Frameworks */, 6369A2681A9322E20015FC5C /* Resources */, - 04554623324BE4A838846086 /* Copy Pods Resources */, + 04554623324BE4A838846086 /* [CP] Copy Pods Resources */, + C7FAD018D05AB5F0B0FE81E2 /* [CP] Embed Pods Frameworks */, ); buildRules = ( ); @@ -167,29 +168,29 @@ /* End PBXResourcesBuildPhase section */ /* Begin PBXShellScriptBuildPhase section */ - 04554623324BE4A838846086 /* Copy Pods Resources */ = { + 04554623324BE4A838846086 /* [CP] Copy Pods Resources */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Copy Pods Resources"; + name = "[CP] Copy Pods Resources"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods/Pods-resources.sh\"\n"; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-Sample/Pods-Sample-resources.sh\"\n"; showEnvVarsInLog = 0; }; - 41F7486D8F66994B0BFB84AF /* Check Pods Manifest.lock */ = { + 41F7486D8F66994B0BFB84AF /* [CP] Check Pods Manifest.lock */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Check Pods Manifest.lock"; + name = "[CP] Check Pods Manifest.lock"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; @@ -197,6 +198,21 @@ shellScript = "diff \"${PODS_ROOT}/../Podfile.lock\" \"${PODS_ROOT}/Manifest.lock\" > /dev/null\nif [[ $? != 0 ]] ; then\n cat << EOM\nerror: The sandbox is not in sync with the Podfile.lock. Run 'pod install' or update your CocoaPods installation.\nEOM\n exit 1\nfi\n"; showEnvVarsInLog = 0; }; + C7FAD018D05AB5F0B0FE81E2 /* [CP] Embed Pods Frameworks */ = { + isa = PBXShellScriptBuildPhase; + buildActionMask = 2147483647; + files = ( + ); + inputPaths = ( + ); + name = "[CP] Embed Pods Frameworks"; + outputPaths = ( + ); + runOnlyForDeploymentPostprocessing = 0; + shellPath = /bin/sh; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-Sample/Pods-Sample-frameworks.sh\"\n"; + showEnvVarsInLog = 0; + }; /* End PBXShellScriptBuildPhase section */ /* Begin PBXSourcesBuildPhase section */ @@ -304,7 +320,7 @@ }; 6369A28E1A9322E20015FC5C /* Debug */ = { isa = XCBuildConfiguration; - baseConfigurationReference = AC29DD6FCDF962F519FEBB0D /* Pods.debug.xcconfig */; + baseConfigurationReference = E3C01DF315C4E7433BCEC6E6 /* Pods-Sample.debug.xcconfig */; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; INFOPLIST_FILE = Sample/Info.plist; @@ -315,7 +331,7 @@ }; 6369A28F1A9322E20015FC5C /* Release */ = { isa = XCBuildConfiguration; - baseConfigurationReference = C68330F8D451CC6ACEABA09F /* Pods.release.xcconfig */; + baseConfigurationReference = 5A8C9F4B28733B249DE4AB6D /* Pods-Sample.release.xcconfig */; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; INFOPLIST_FILE = Sample/Info.plist; diff --git a/src/objective-c/examples/SwiftSample/SwiftSample.xcodeproj/project.pbxproj b/src/objective-c/examples/SwiftSample/SwiftSample.xcodeproj/project.pbxproj index 2f5716082bf..2a1b30f2cf9 100644 --- a/src/objective-c/examples/SwiftSample/SwiftSample.xcodeproj/project.pbxproj +++ b/src/objective-c/examples/SwiftSample/SwiftSample.xcodeproj/project.pbxproj @@ -7,15 +7,14 @@ objects = { /* Begin PBXBuildFile section */ - 253D3A297105CA46DA960A11 /* libPods.a in Frameworks */ = {isa = PBXBuildFile; fileRef = DC58ACA18DCCB1553531B885 /* libPods.a */; }; 633BFFC81B950B210007E424 /* AppDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = 633BFFC71B950B210007E424 /* AppDelegate.swift */; }; 633BFFCA1B950B210007E424 /* ViewController.swift in Sources */ = {isa = PBXBuildFile; fileRef = 633BFFC91B950B210007E424 /* ViewController.swift */; }; 633BFFCD1B950B210007E424 /* Main.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 633BFFCB1B950B210007E424 /* Main.storyboard */; }; 633BFFCF1B950B210007E424 /* Images.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 633BFFCE1B950B210007E424 /* Images.xcassets */; }; + 92EDB1408A1E1E7DDAB25D9C /* libPods-SwiftSample.a in Frameworks */ = {isa = PBXBuildFile; fileRef = 69BB5C6CA3C1F97E007AC527 /* libPods-SwiftSample.a */; }; /* End PBXBuildFile section */ /* Begin PBXFileReference section */ - 12C7B447AA80E624D93B5C54 /* Pods.debug.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = Pods.debug.xcconfig; path = "Pods/Target Support Files/Pods/Pods.debug.xcconfig"; sourceTree = ""; }; 633BFFC21B950B210007E424 /* SwiftSample.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = SwiftSample.app; sourceTree = BUILT_PRODUCTS_DIR; }; 633BFFC61B950B210007E424 /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = ""; }; 633BFFC71B950B210007E424 /* AppDelegate.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AppDelegate.swift; sourceTree = ""; }; @@ -23,8 +22,9 @@ 633BFFCC1B950B210007E424 /* Base */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; name = Base; path = Base.lproj/Main.storyboard; sourceTree = ""; }; 633BFFCE1B950B210007E424 /* Images.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Images.xcassets; sourceTree = ""; }; 6367AD231B951655007FD3A4 /* Bridging-Header.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = "Bridging-Header.h"; sourceTree = ""; }; - C335CBC4C160E0D9EDEE646B /* Pods.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = Pods.release.xcconfig; path = "Pods/Target Support Files/Pods/Pods.release.xcconfig"; sourceTree = ""; }; - DC58ACA18DCCB1553531B885 /* libPods.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = libPods.a; sourceTree = BUILT_PRODUCTS_DIR; }; + 69BB5C6CA3C1F97E007AC527 /* libPods-SwiftSample.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = "libPods-SwiftSample.a"; sourceTree = BUILT_PRODUCTS_DIR; }; + A7E614A494D89D01BB395761 /* Pods-SwiftSample.debug.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-SwiftSample.debug.xcconfig"; path = "Pods/Target Support Files/Pods-SwiftSample/Pods-SwiftSample.debug.xcconfig"; sourceTree = ""; }; + C314E3E246AF23AC29B38FCF /* Pods-SwiftSample.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-SwiftSample.release.xcconfig"; path = "Pods/Target Support Files/Pods-SwiftSample/Pods-SwiftSample.release.xcconfig"; sourceTree = ""; }; /* End PBXFileReference section */ /* Begin PBXFrameworksBuildPhase section */ @@ -32,7 +32,7 @@ isa = PBXFrameworksBuildPhase; buildActionMask = 2147483647; files = ( - 253D3A297105CA46DA960A11 /* libPods.a in Frameworks */, + 92EDB1408A1E1E7DDAB25D9C /* libPods-SwiftSample.a in Frameworks */, ); runOnlyForDeploymentPostprocessing = 0; }; @@ -42,8 +42,8 @@ 31F283C976AE97586C17CCD9 /* Pods */ = { isa = PBXGroup; children = ( - 12C7B447AA80E624D93B5C54 /* Pods.debug.xcconfig */, - C335CBC4C160E0D9EDEE646B /* Pods.release.xcconfig */, + A7E614A494D89D01BB395761 /* Pods-SwiftSample.debug.xcconfig */, + C314E3E246AF23AC29B38FCF /* Pods-SwiftSample.release.xcconfig */, ); name = Pods; sourceTree = ""; @@ -90,7 +90,7 @@ 9D63A7F6423989BA306810CA /* Frameworks */ = { isa = PBXGroup; children = ( - DC58ACA18DCCB1553531B885 /* libPods.a */, + 69BB5C6CA3C1F97E007AC527 /* libPods-SwiftSample.a */, ); name = Frameworks; sourceTree = ""; @@ -102,12 +102,12 @@ isa = PBXNativeTarget; buildConfigurationList = 633BFFE11B950B210007E424 /* Build configuration list for PBXNativeTarget "SwiftSample" */; buildPhases = ( - 6BEEB33CA2705D7D2F2210E6 /* Check Pods Manifest.lock */, + 6BEEB33CA2705D7D2F2210E6 /* [CP] Check Pods Manifest.lock */, 633BFFBE1B950B210007E424 /* Sources */, 633BFFBF1B950B210007E424 /* Frameworks */, 633BFFC01B950B210007E424 /* Resources */, - AC2F6F9AB1C090BB0BEE6E4D /* Copy Pods Resources */, - A1738A987353B0BF2C64F0F7 /* Embed Pods Frameworks */, + AC2F6F9AB1C090BB0BEE6E4D /* [CP] Copy Pods Resources */, + A1738A987353B0BF2C64F0F7 /* [CP] Embed Pods Frameworks */, ); buildRules = ( ); @@ -164,14 +164,14 @@ /* End PBXResourcesBuildPhase section */ /* Begin PBXShellScriptBuildPhase section */ - 6BEEB33CA2705D7D2F2210E6 /* Check Pods Manifest.lock */ = { + 6BEEB33CA2705D7D2F2210E6 /* [CP] Check Pods Manifest.lock */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Check Pods Manifest.lock"; + name = "[CP] Check Pods Manifest.lock"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; @@ -179,34 +179,34 @@ shellScript = "diff \"${PODS_ROOT}/../Podfile.lock\" \"${PODS_ROOT}/Manifest.lock\" > /dev/null\nif [[ $? != 0 ]] ; then\n cat << EOM\nerror: The sandbox is not in sync with the Podfile.lock. Run 'pod install' or update your CocoaPods installation.\nEOM\n exit 1\nfi\n"; showEnvVarsInLog = 0; }; - A1738A987353B0BF2C64F0F7 /* Embed Pods Frameworks */ = { + A1738A987353B0BF2C64F0F7 /* [CP] Embed Pods Frameworks */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Embed Pods Frameworks"; + name = "[CP] Embed Pods Frameworks"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods/Pods-frameworks.sh\"\n"; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-SwiftSample/Pods-SwiftSample-frameworks.sh\"\n"; showEnvVarsInLog = 0; }; - AC2F6F9AB1C090BB0BEE6E4D /* Copy Pods Resources */ = { + AC2F6F9AB1C090BB0BEE6E4D /* [CP] Copy Pods Resources */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( ); inputPaths = ( ); - name = "Copy Pods Resources"; + name = "[CP] Copy Pods Resources"; outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods/Pods-resources.sh\"\n"; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-SwiftSample/Pods-SwiftSample-resources.sh\"\n"; showEnvVarsInLog = 0; }; /* End PBXShellScriptBuildPhase section */ @@ -320,7 +320,7 @@ }; 633BFFE21B950B210007E424 /* Debug */ = { isa = XCBuildConfiguration; - baseConfigurationReference = 12C7B447AA80E624D93B5C54 /* Pods.debug.xcconfig */; + baseConfigurationReference = A7E614A494D89D01BB395761 /* Pods-SwiftSample.debug.xcconfig */; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; INFOPLIST_FILE = Info.plist; @@ -333,7 +333,7 @@ }; 633BFFE31B950B210007E424 /* Release */ = { isa = XCBuildConfiguration; - baseConfigurationReference = C335CBC4C160E0D9EDEE646B /* Pods.release.xcconfig */; + baseConfigurationReference = C314E3E246AF23AC29B38FCF /* Pods-SwiftSample.release.xcconfig */; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; INFOPLIST_FILE = Info.plist; From c0668c837b0ee48f64d0a912e2c6120f2ab719c5 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Wed, 22 Jun 2016 01:30:47 -0700 Subject: [PATCH 0162/1039] Increase timeout_seconds for build_example_test --- tools/run_tests/run_tests.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index 580afc32de4..5c8f6f2a0e7 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -604,7 +604,8 @@ class ObjCLanguage(object): return [self.config.job_spec(['src/objective-c/tests/run_tests.sh'], None, environ=_FORCE_ENVIRON_FOR_WRAPPERS), self.config.job_spec(['src/objective-c/tests/build_example_test.sh'], - None, environ=_FORCE_ENVIRON_FOR_WRAPPERS)] + timeout_seconds=15*60, None, + environ=_FORCE_ENVIRON_FOR_WRAPPERS)] def pre_build_steps(self): return [] From c82324fa499c17c0458abaaea9caac49c1553356 Mon Sep 17 00:00:00 2001 From: Tamas Berghammer Date: Wed, 22 Jun 2016 13:23:27 +0100 Subject: [PATCH 0163/1039] Remove some unneccessary dependency from build.yaml This is the copy of the build.yaml changes from the following CL: https://github.com/grpc/grpc/pull/4124/files#diff-5b123ecec7bf9d216a1323f790a0602a --- Makefile | 21 ++++++------ build.yaml | 10 ++---- tools/run_tests/sources_and_headers.json | 12 ++----- vsprojects/buildtests_c.sln | 8 +---- vsprojects/grpc.sln | 33 ------------------- .../grpc_create_jwt/grpc_create_jwt.vcxproj | 6 ---- ...c_print_google_default_creds_token.vcxproj | 6 ---- .../grpc_verify_jwt/grpc_verify_jwt.vcxproj | 6 ---- .../grpc_fetch_oauth2.vcxproj | 27 +++++++++++++-- .../grpc_fetch_oauth2.vcxproj.filters | 0 10 files changed, 43 insertions(+), 86 deletions(-) rename vsprojects/vcxproj/{ => test}/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj (67%) rename vsprojects/vcxproj/{ => test}/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj.filters (100%) diff --git a/Makefile b/Makefile index ca642429519..3d6566c8110 100644 --- a/Makefile +++ b/Makefile @@ -1279,6 +1279,7 @@ buildtests_c: privatelibs_c \ $(BINDIR)/$(CONFIG)/grpc_channel_stack_test \ $(BINDIR)/$(CONFIG)/grpc_completion_queue_test \ $(BINDIR)/$(CONFIG)/grpc_credentials_test \ + $(BINDIR)/$(CONFIG)/grpc_fetch_oauth2 \ $(BINDIR)/$(CONFIG)/grpc_invalid_channel_args_test \ $(BINDIR)/$(CONFIG)/grpc_json_token_test \ $(BINDIR)/$(CONFIG)/grpc_jwt_verifier_test \ @@ -1837,7 +1838,7 @@ test_python: static_c tools: tools_c tools_cxx -tools_c: privatelibs_c $(BINDIR)/$(CONFIG)/gen_hpack_tables $(BINDIR)/$(CONFIG)/gen_legal_metadata_characters $(BINDIR)/$(CONFIG)/grpc_create_jwt $(BINDIR)/$(CONFIG)/grpc_fetch_oauth2 $(BINDIR)/$(CONFIG)/grpc_print_google_default_creds_token $(BINDIR)/$(CONFIG)/grpc_verify_jwt +tools_c: privatelibs_c $(BINDIR)/$(CONFIG)/gen_hpack_tables $(BINDIR)/$(CONFIG)/gen_legal_metadata_characters $(BINDIR)/$(CONFIG)/grpc_create_jwt $(BINDIR)/$(CONFIG)/grpc_print_google_default_creds_token $(BINDIR)/$(CONFIG)/grpc_verify_jwt tools_cxx: privatelibs_cxx @@ -8171,14 +8172,14 @@ else -$(BINDIR)/$(CONFIG)/grpc_create_jwt: $(GRPC_CREATE_JWT_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a +$(BINDIR)/$(CONFIG)/grpc_create_jwt: $(GRPC_CREATE_JWT_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr.a $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` - $(Q) $(LD) $(LDFLAGS) $(GRPC_CREATE_JWT_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/grpc_create_jwt + $(Q) $(LD) $(LDFLAGS) $(GRPC_CREATE_JWT_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/grpc_create_jwt endif -$(OBJDIR)/$(CONFIG)/test/core/security/create_jwt.o: $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a +$(OBJDIR)/$(CONFIG)/test/core/security/create_jwt.o: $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr.a deps_grpc_create_jwt: $(GRPC_CREATE_JWT_OBJS:.o=.dep) @@ -8363,14 +8364,14 @@ else -$(BINDIR)/$(CONFIG)/grpc_print_google_default_creds_token: $(GRPC_PRINT_GOOGLE_DEFAULT_CREDS_TOKEN_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a +$(BINDIR)/$(CONFIG)/grpc_print_google_default_creds_token: $(GRPC_PRINT_GOOGLE_DEFAULT_CREDS_TOKEN_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr.a $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` - $(Q) $(LD) $(LDFLAGS) $(GRPC_PRINT_GOOGLE_DEFAULT_CREDS_TOKEN_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/grpc_print_google_default_creds_token + $(Q) $(LD) $(LDFLAGS) $(GRPC_PRINT_GOOGLE_DEFAULT_CREDS_TOKEN_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/grpc_print_google_default_creds_token endif -$(OBJDIR)/$(CONFIG)/test/core/security/print_google_default_creds_token.o: $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a +$(OBJDIR)/$(CONFIG)/test/core/security/print_google_default_creds_token.o: $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr.a deps_grpc_print_google_default_creds_token: $(GRPC_PRINT_GOOGLE_DEFAULT_CREDS_TOKEN_OBJS:.o=.dep) @@ -8427,14 +8428,14 @@ else -$(BINDIR)/$(CONFIG)/grpc_verify_jwt: $(GRPC_VERIFY_JWT_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a +$(BINDIR)/$(CONFIG)/grpc_verify_jwt: $(GRPC_VERIFY_JWT_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr.a $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` - $(Q) $(LD) $(LDFLAGS) $(GRPC_VERIFY_JWT_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/grpc_verify_jwt + $(Q) $(LD) $(LDFLAGS) $(GRPC_VERIFY_JWT_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/grpc_verify_jwt endif -$(OBJDIR)/$(CONFIG)/test/core/security/verify_jwt.o: $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a +$(OBJDIR)/$(CONFIG)/test/core/security/verify_jwt.o: $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr.a deps_grpc_verify_jwt: $(GRPC_VERIFY_JWT_OBJS:.o=.dep) diff --git a/build.yaml b/build.yaml index 59f7d19dfcc..2d93e20fa62 100644 --- a/build.yaml +++ b/build.yaml @@ -1706,10 +1706,9 @@ targets: src: - test/core/security/create_jwt.c deps: - - grpc_test_util - grpc - - gpr_test_util - gpr + secure: true - name: grpc_credentials_test build: test language: c @@ -1721,7 +1720,8 @@ targets: - gpr_test_util - gpr - name: grpc_fetch_oauth2 - build: tool + build: test + run: false language: c src: - test/core/security/fetch_oauth2.c @@ -1770,9 +1770,7 @@ targets: src: - test/core/security/print_google_default_creds_token.c deps: - - grpc_test_util - grpc - - gpr_test_util - gpr - name: grpc_security_connector_test build: test @@ -1790,9 +1788,7 @@ targets: src: - test/core/security/verify_jwt.c deps: - - grpc_test_util - grpc - - gpr_test_util - gpr - name: hpack_parser_fuzzer_test build: fuzzer diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 4cfd1c51f8f..c3a219b06e6 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -789,9 +789,7 @@ { "deps": [ "gpr", - "gpr_test_util", - "grpc", - "grpc_test_util" + "grpc" ], "headers": [], "language": "c", @@ -885,9 +883,7 @@ { "deps": [ "gpr", - "gpr_test_util", - "grpc", - "grpc_test_util" + "grpc" ], "headers": [], "language": "c", @@ -917,9 +913,7 @@ { "deps": [ "gpr", - "gpr_test_util", - "grpc", - "grpc_test_util" + "grpc" ], "headers": [], "language": "c", diff --git a/vsprojects/buildtests_c.sln b/vsprojects/buildtests_c.sln index a847add7730..0ead8845ed9 100644 --- a/vsprojects/buildtests_c.sln +++ b/vsprojects/buildtests_c.sln @@ -522,9 +522,7 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_create_jwt", "vcxproj\ lib = "False" EndProjectSection ProjectSection(ProjectDependencies) = postProject - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject @@ -539,7 +537,7 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_credentials_test", "vc {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_fetch_oauth2", "vcxproj\.\grpc_fetch_oauth2\grpc_fetch_oauth2.vcxproj", "{43722E98-54EC-5058-3DAC-327F45964971}" +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_fetch_oauth2", "vcxproj\test\grpc_fetch_oauth2\grpc_fetch_oauth2.vcxproj", "{43722E98-54EC-5058-3DAC-327F45964971}" ProjectSection(myProperties) = preProject lib = "False" EndProjectSection @@ -577,9 +575,7 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_print_google_default_c lib = "False" EndProjectSection ProjectSection(ProjectDependencies) = postProject - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject @@ -628,9 +624,7 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_verify_jwt", "vcxproj\ lib = "False" EndProjectSection ProjectSection(ProjectDependencies) = postProject - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject diff --git a/vsprojects/grpc.sln b/vsprojects/grpc.sln index 771fefc56bd..6105f724c9f 100644 --- a/vsprojects/grpc.sln +++ b/vsprojects/grpc.sln @@ -73,9 +73,7 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_create_jwt", "vcxproj\ lib = "False" EndProjectSection ProjectSection(ProjectDependencies) = postProject - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject @@ -88,25 +86,12 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_dll", "vcxproj\.\grpc_ {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} EndProjectSection EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_fetch_oauth2", "vcxproj\.\grpc_fetch_oauth2\grpc_fetch_oauth2.vcxproj", "{43722E98-54EC-5058-3DAC-327F45964971}" - ProjectSection(myProperties) = preProject - lib = "False" - EndProjectSection - ProjectSection(ProjectDependencies) = postProject - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} - {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} - {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} - EndProjectSection -EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_print_google_default_creds_token", "vcxproj\.\grpc_print_google_default_creds_token\grpc_print_google_default_creds_token.vcxproj", "{C002965C-8457-CCE5-B1BA-E748FF9A11B6}" ProjectSection(myProperties) = preProject lib = "False" EndProjectSection ProjectSection(ProjectDependencies) = postProject - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject @@ -144,9 +129,7 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_verify_jwt", "vcxproj\ lib = "False" EndProjectSection ProjectSection(ProjectDependencies) = postProject - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject @@ -366,22 +349,6 @@ Global {A2F6CBBA-A553-41B3-A7DE-F26DECCC27F0}.Release-DLL|Win32.Build.0 = Release|Win32 {A2F6CBBA-A553-41B3-A7DE-F26DECCC27F0}.Release-DLL|x64.ActiveCfg = Release|x64 {A2F6CBBA-A553-41B3-A7DE-F26DECCC27F0}.Release-DLL|x64.Build.0 = Release|x64 - {43722E98-54EC-5058-3DAC-327F45964971}.Debug|Win32.ActiveCfg = Debug|Win32 - {43722E98-54EC-5058-3DAC-327F45964971}.Debug|x64.ActiveCfg = Debug|x64 - {43722E98-54EC-5058-3DAC-327F45964971}.Release|Win32.ActiveCfg = Release|Win32 - {43722E98-54EC-5058-3DAC-327F45964971}.Release|x64.ActiveCfg = Release|x64 - {43722E98-54EC-5058-3DAC-327F45964971}.Debug|Win32.Build.0 = Debug|Win32 - {43722E98-54EC-5058-3DAC-327F45964971}.Debug|x64.Build.0 = Debug|x64 - {43722E98-54EC-5058-3DAC-327F45964971}.Release|Win32.Build.0 = Release|Win32 - {43722E98-54EC-5058-3DAC-327F45964971}.Release|x64.Build.0 = Release|x64 - {43722E98-54EC-5058-3DAC-327F45964971}.Debug-DLL|Win32.ActiveCfg = Debug|Win32 - {43722E98-54EC-5058-3DAC-327F45964971}.Debug-DLL|Win32.Build.0 = Debug|Win32 - {43722E98-54EC-5058-3DAC-327F45964971}.Debug-DLL|x64.ActiveCfg = Debug|x64 - {43722E98-54EC-5058-3DAC-327F45964971}.Debug-DLL|x64.Build.0 = Debug|x64 - {43722E98-54EC-5058-3DAC-327F45964971}.Release-DLL|Win32.ActiveCfg = Release|Win32 - {43722E98-54EC-5058-3DAC-327F45964971}.Release-DLL|Win32.Build.0 = Release|Win32 - {43722E98-54EC-5058-3DAC-327F45964971}.Release-DLL|x64.ActiveCfg = Release|x64 - {43722E98-54EC-5058-3DAC-327F45964971}.Release-DLL|x64.Build.0 = Release|x64 {C002965C-8457-CCE5-B1BA-E748FF9A11B6}.Debug|Win32.ActiveCfg = Debug|Win32 {C002965C-8457-CCE5-B1BA-E748FF9A11B6}.Debug|x64.ActiveCfg = Debug|x64 {C002965C-8457-CCE5-B1BA-E748FF9A11B6}.Release|Win32.ActiveCfg = Release|Win32 diff --git a/vsprojects/vcxproj/grpc_create_jwt/grpc_create_jwt.vcxproj b/vsprojects/vcxproj/grpc_create_jwt/grpc_create_jwt.vcxproj index ec4ec4a461f..4e8c088e2d7 100644 --- a/vsprojects/vcxproj/grpc_create_jwt/grpc_create_jwt.vcxproj +++ b/vsprojects/vcxproj/grpc_create_jwt/grpc_create_jwt.vcxproj @@ -151,15 +151,9 @@ - - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} - {29D16885-7228-4C31-81ED-5F9187C7F2A9} - - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} - {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} diff --git a/vsprojects/vcxproj/grpc_print_google_default_creds_token/grpc_print_google_default_creds_token.vcxproj b/vsprojects/vcxproj/grpc_print_google_default_creds_token/grpc_print_google_default_creds_token.vcxproj index 87a4a6e9e42..ed0b98c6122 100644 --- a/vsprojects/vcxproj/grpc_print_google_default_creds_token/grpc_print_google_default_creds_token.vcxproj +++ b/vsprojects/vcxproj/grpc_print_google_default_creds_token/grpc_print_google_default_creds_token.vcxproj @@ -151,15 +151,9 @@ - - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} - {29D16885-7228-4C31-81ED-5F9187C7F2A9} - - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} - {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} diff --git a/vsprojects/vcxproj/grpc_verify_jwt/grpc_verify_jwt.vcxproj b/vsprojects/vcxproj/grpc_verify_jwt/grpc_verify_jwt.vcxproj index 27b166582a7..e75143bee6d 100644 --- a/vsprojects/vcxproj/grpc_verify_jwt/grpc_verify_jwt.vcxproj +++ b/vsprojects/vcxproj/grpc_verify_jwt/grpc_verify_jwt.vcxproj @@ -151,15 +151,9 @@ - - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} - {29D16885-7228-4C31-81ED-5F9187C7F2A9} - - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} - {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} diff --git a/vsprojects/vcxproj/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj b/vsprojects/vcxproj/test/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj similarity index 67% rename from vsprojects/vcxproj/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj rename to vsprojects/vcxproj/test/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj index 393f8e59024..52bb2697662 100644 --- a/vsprojects/vcxproj/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj +++ b/vsprojects/vcxproj/test/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj @@ -1,5 +1,6 @@ + Debug @@ -37,12 +38,12 @@ v140 - StaticLibrary + Application true Unicode - StaticLibrary + Application false true Unicode @@ -53,14 +54,24 @@ + + grpc_fetch_oauth2 + static + Debug + static + Debug grpc_fetch_oauth2 + static + Release + static + Release @@ -164,13 +175,25 @@ {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + diff --git a/vsprojects/vcxproj/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj.filters b/vsprojects/vcxproj/test/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj.filters similarity index 100% rename from vsprojects/vcxproj/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj.filters rename to vsprojects/vcxproj/test/grpc_fetch_oauth2/grpc_fetch_oauth2.vcxproj.filters From df6a44cd93ba069b6927006cd761b44921e6da41 Mon Sep 17 00:00:00 2001 From: Tamas Berghammer Date: Wed, 22 Jun 2016 13:38:55 +0100 Subject: [PATCH 0164/1039] Improve the generated cmake build files * Add project name and version number * Fix zlib dependency of grpc * Include more target (all, protoc, tool groups) --- CMakeLists.txt | 108 ++++++++++++++++++++++++++++-- templates/CMakeLists.txt.template | 21 ++++-- 2 files changed, 119 insertions(+), 10 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 5bd48b78ea9..a81517a6a0b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -38,13 +38,20 @@ cmake_minimum_required(VERSION 2.8) -if (NOT BORINGSSL_ROOT_DIR) +set(PACKAGE_NAME "grpc") +set(PACKAGE_VERSION "0.15.0-dev") +set(PACKAGE_STRING "${PACKAGE_NAME} ${PACKAGE_VERSION}") +set(PACKAGE_TARNAME "${PACKAGE_NAME}-${PACKAGE_VERSION}") +set(PACKAGE_BUGREPORT "https://github.com/grpc/grpc/issues/") +project(${PACKAGE_NAME} C CXX) + +if(NOT BORINGSSL_ROOT_DIR) set(BORINGSSL_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/third_party/boringssl) endif() -if (NOT PROTOBUF_ROOT_DIR) +if(NOT PROTOBUF_ROOT_DIR) set(PROTOBUF_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/third_party/protobuf) endif() -if (NOT ZLIB_ROOT_DIR) +if(NOT ZLIB_ROOT_DIR) set(ZLIB_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/third_party/zlib) endif() @@ -52,7 +59,7 @@ add_subdirectory(${BORINGSSL_ROOT_DIR} third_party/boringssl) add_subdirectory(${PROTOBUF_ROOT_DIR}/cmake third_party/protobuf) add_subdirectory(${ZLIB_ROOT_DIR} third_party/zlib) -set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=c11") +set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=c11") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11") @@ -313,6 +320,7 @@ target_include_directories(grpc target_link_libraries(grpc ssl + zlibstatic gpr ) @@ -831,6 +839,98 @@ target_link_libraries(grpc_csharp_ext +add_executable(gen_hpack_tables + tools/codegen/core/gen_hpack_tables.c +) + +target_include_directories(gen_hpack_tables + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(gen_hpack_tables + gpr + grpc +) + + +add_executable(gen_legal_metadata_characters + tools/codegen/core/gen_legal_metadata_characters.c +) + +target_include_directories(gen_legal_metadata_characters + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + + + +add_executable(grpc_create_jwt + test/core/security/create_jwt.c +) + +target_include_directories(grpc_create_jwt + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_create_jwt + ssl + grpc + gpr +) + + +add_executable(grpc_print_google_default_creds_token + test/core/security/print_google_default_creds_token.c +) + +target_include_directories(grpc_print_google_default_creds_token + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_print_google_default_creds_token + grpc + gpr +) + + +add_executable(grpc_verify_jwt + test/core/security/verify_jwt.c +) + +target_include_directories(grpc_verify_jwt + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include + PRIVATE ${BORINGSSL_ROOT_DIR}/include + PRIVATE ${PROTOBUF_ROOT_DIR}/src + PRIVATE ${ZLIB_ROOT_DIR} + PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/third_party/zlib +) + +target_link_libraries(grpc_verify_jwt + grpc + gpr +) + + add_executable(grpc_cpp_plugin src/compiler/cpp_plugin.cc ) diff --git a/templates/CMakeLists.txt.template b/templates/CMakeLists.txt.template index 00a37de3cb9..06085eb722b 100644 --- a/templates/CMakeLists.txt.template +++ b/templates/CMakeLists.txt.template @@ -45,6 +45,8 @@ deps = ["ssl"] if target_dict['name'] in ['grpc++', 'grpc++_unsecure', 'grpc++_codegen_lib']: deps.append("libprotobuf") + elif target_dict['name'] in ['grpc']: + deps.append("zlibstatic") for d in target_dict.get('deps', []): deps.append(d) return deps @@ -52,13 +54,20 @@ cmake_minimum_required(VERSION 2.8) - if (NOT BORINGSSL_ROOT_DIR) + set(PACKAGE_NAME "grpc") + set(PACKAGE_VERSION "${settings.core_version}") + set(PACKAGE_STRING "<%text>${PACKAGE_NAME} ${PACKAGE_VERSION}") + set(PACKAGE_TARNAME "<%text>${PACKAGE_NAME}-${PACKAGE_VERSION}") + set(PACKAGE_BUGREPORT "https://github.com/grpc/grpc/issues/") + project(<%text>${PACKAGE_NAME} C CXX) + + if(NOT BORINGSSL_ROOT_DIR) set(BORINGSSL_ROOT_DIR <%text>${CMAKE_CURRENT_SOURCE_DIR}/third_party/boringssl) endif() - if (NOT PROTOBUF_ROOT_DIR) + if(NOT PROTOBUF_ROOT_DIR) set(PROTOBUF_ROOT_DIR <%text>${CMAKE_CURRENT_SOURCE_DIR}/third_party/protobuf) endif() - if (NOT ZLIB_ROOT_DIR) + if(NOT ZLIB_ROOT_DIR) set(ZLIB_ROOT_DIR <%text>${CMAKE_CURRENT_SOURCE_DIR}/third_party/zlib) endif() @@ -66,17 +75,17 @@ add_subdirectory(<%text>${PROTOBUF_ROOT_DIR}/cmake third_party/protobuf) add_subdirectory(<%text>${ZLIB_ROOT_DIR} third_party/zlib) - set(CMAKE_C_FLAGS "<%text>${CMAKE_C_FLAGS} -std=c11") + set(CMAKE_C_FLAGS "<%text>${CMAKE_C_FLAGS} -std=c11") set(CMAKE_CXX_FLAGS "<%text>${CMAKE_CXX_FLAGS} -std=c++11") % for lib in libs: - % if lib.build in ["all", "protoc"]: + % if lib.build in ["all", "protoc", "tool"]: ${cc_library(lib)} % endif % endfor % for tgt in targets: - % if tgt.build == 'protoc': + % if tgt.build in ["all", "protoc", "tool"]: ${cc_binary(tgt)} % endif % endfor From 7e5d46496b6ad47aa84583260cf7886f7f51da5f Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 22 Jun 2016 08:17:27 -0700 Subject: [PATCH 0165/1039] Better distribution --- test/cpp/qps/driver.cc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/test/cpp/qps/driver.cc b/test/cpp/qps/driver.cc index 147c540840e..2efdc1288ff 100644 --- a/test/cpp/qps/driver.cc +++ b/test/cpp/qps/driver.cc @@ -345,9 +345,7 @@ std::unique_ptr RunScenario( // Reduce channel count so that total channels specified is held regardless // of the number of clients available size_t num_channels = - (i == num_clients - 1) - ? channels_allocated - client_config.client_channels() - : client_config.client_channels() / num_clients; + (client_channels - channels_allocated) / (num_clients - i); channels_allocated += num_channels; per_client_config.set_client_channels(num_channels); From a2dd8385f95259b8a6886902dac87d5df7c7b953 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 22 Jun 2016 10:26:03 -0700 Subject: [PATCH 0166/1039] Use pipe fds instead of event fds for the test --- test/core/iomgr/ev_epoll_linux_test.c | 55 +++++++++++++++++---------- 1 file changed, 34 insertions(+), 21 deletions(-) diff --git a/test/core/iomgr/ev_epoll_linux_test.c b/test/core/iomgr/ev_epoll_linux_test.c index 034f17fd58b..35eb6791302 100644 --- a/test/core/iomgr/ev_epoll_linux_test.c +++ b/test/core/iomgr/ev_epoll_linux_test.c @@ -34,9 +34,8 @@ #include "src/core/lib/iomgr/ev_epoll_linux.h" #include "src/core/lib/iomgr/ev_posix.h" -#include +#include #include -#include #include #include @@ -55,28 +54,29 @@ typedef struct test_fd { grpc_fd *fd; } test_fd; -static void test_fd_init(test_fd *fds, int num_fds) { +/* num_fds should be an even number */ +static void test_fd_init(test_fd *tfds, int *fds, int num_fds) { int i; for (i = 0; i < num_fds; i++) { - fds[i].inner_fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); - fds[i].fd = grpc_fd_create(fds[i].inner_fd, "test_fd"); + tfds[i].inner_fd = fds[i]; + tfds[i].fd = grpc_fd_create(fds[i], "test_fd"); } } -static void test_fd_cleanup(grpc_exec_ctx *exec_ctx, test_fd *fds, +static void test_fd_cleanup(grpc_exec_ctx *exec_ctx, test_fd *tfds, int num_fds) { int release_fd; int i; for (i = 0; i < num_fds; i++) { - grpc_fd_shutdown(exec_ctx, fds[i].fd); + grpc_fd_shutdown(exec_ctx, tfds[i].fd); grpc_exec_ctx_flush(exec_ctx); - grpc_fd_orphan(exec_ctx, fds[i].fd, NULL, &release_fd, "test_fd_cleanup"); + grpc_fd_orphan(exec_ctx, tfds[i].fd, NULL, &release_fd, "test_fd_cleanup"); grpc_exec_ctx_flush(exec_ctx); - GPR_ASSERT(release_fd == fds[i].inner_fd); - close(fds[i].inner_fd); + GPR_ASSERT(release_fd == tfds[i].inner_fd); + close(tfds[i].inner_fd); } } @@ -121,12 +121,25 @@ static void test_pollset_cleanup(grpc_exec_ctx *exec_ctx, * */ static void test_add_fd_to_pollset() { grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; - test_fd fds[NUM_FDS]; + test_fd tfds[NUM_FDS]; + int fds[NUM_FDS]; test_pollset pollsets[NUM_POLLSETS]; void *expected_pi = NULL; int i; + int r; + + /* Create some dummy file descriptors (using pipe fds for this test. Could be + anything). Also NUM_FDS should be even for this test. */ + for (i = 0; i < NUM_FDS; i = i + 2) { + r = pipe(fds + i); + if (r != 0) { + gpr_log(GPR_ERROR, "Error in creating pipe. %d (%s)", errno, + strerror(errno)); + return; + } + } - test_fd_init(fds, NUM_FDS); + test_fd_init(tfds, fds, NUM_FDS); test_pollset_init(pollsets, NUM_POLLSETS); /*Step 1. @@ -156,41 +169,41 @@ static void test_add_fd_to_pollset() { /* == Step 1 == */ for (i = 0; i <= 2; i++) { - grpc_pollset_add_fd(&exec_ctx, pollsets[0].pollset, fds[i].fd); + grpc_pollset_add_fd(&exec_ctx, pollsets[0].pollset, tfds[i].fd); grpc_exec_ctx_flush(&exec_ctx); } for (i = 3; i <= 4; i++) { - grpc_pollset_add_fd(&exec_ctx, pollsets[1].pollset, fds[i].fd); + grpc_pollset_add_fd(&exec_ctx, pollsets[1].pollset, tfds[i].fd); grpc_exec_ctx_flush(&exec_ctx); } for (i = 5; i <= 7; i++) { - grpc_pollset_add_fd(&exec_ctx, pollsets[2].pollset, fds[i].fd); + grpc_pollset_add_fd(&exec_ctx, pollsets[2].pollset, tfds[i].fd); grpc_exec_ctx_flush(&exec_ctx); } /* == Step 2 == */ for (i = 0; i <= 1; i++) { - grpc_pollset_add_fd(&exec_ctx, pollsets[3].pollset, fds[i].fd); + grpc_pollset_add_fd(&exec_ctx, pollsets[3].pollset, tfds[i].fd); grpc_exec_ctx_flush(&exec_ctx); } /* == Step 3 == */ - grpc_pollset_add_fd(&exec_ctx, pollsets[1].pollset, fds[0].fd); + grpc_pollset_add_fd(&exec_ctx, pollsets[1].pollset, tfds[0].fd); grpc_exec_ctx_flush(&exec_ctx); /* == Step 4 == */ - grpc_pollset_add_fd(&exec_ctx, pollsets[2].pollset, fds[3].fd); + grpc_pollset_add_fd(&exec_ctx, pollsets[2].pollset, tfds[3].fd); grpc_exec_ctx_flush(&exec_ctx); /* All polling islands are merged at this point */ /* Compare Fd:0's polling island with that of all other Fds */ - expected_pi = grpc_fd_get_polling_island(fds[0].fd); + expected_pi = grpc_fd_get_polling_island(tfds[0].fd); for (i = 1; i < NUM_FDS; i++) { GPR_ASSERT(grpc_are_polling_islands_equal( - expected_pi, grpc_fd_get_polling_island(fds[i].fd))); + expected_pi, grpc_fd_get_polling_island(tfds[i].fd))); } /* Compare Fd:0's polling island with that of all other pollsets */ @@ -199,7 +212,7 @@ static void test_add_fd_to_pollset() { expected_pi, grpc_pollset_get_polling_island(pollsets[i].pollset))); } - test_fd_cleanup(&exec_ctx, fds, NUM_FDS); + test_fd_cleanup(&exec_ctx, tfds, NUM_FDS); test_pollset_cleanup(&exec_ctx, pollsets, NUM_POLLSETS); grpc_exec_ctx_finish(&exec_ctx); } From 82393efe0c930f7756e9d0e427d2e9ae61e0a88e Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Wed, 22 Jun 2016 10:29:17 -0700 Subject: [PATCH 0167/1039] Fix run_test.py --- tools/run_tests/run_tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index 5c8f6f2a0e7..2a404df0d41 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -604,7 +604,7 @@ class ObjCLanguage(object): return [self.config.job_spec(['src/objective-c/tests/run_tests.sh'], None, environ=_FORCE_ENVIRON_FOR_WRAPPERS), self.config.job_spec(['src/objective-c/tests/build_example_test.sh'], - timeout_seconds=15*60, None, + None, timeout_seconds=15*60, environ=_FORCE_ENVIRON_FOR_WRAPPERS)] def pre_build_steps(self): From d9c6ac09625e548faf286b73f0b0d74fb1b3bb1d Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Wed, 22 Jun 2016 11:11:46 -0700 Subject: [PATCH 0168/1039] Memset ops before being used, destroy metadata_recv after beign used --- test/core/end2end/bad_server_response_test.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/test/core/end2end/bad_server_response_test.c b/test/core/end2end/bad_server_response_test.c index 4a2355b5f6d..ebf2caa0702 100644 --- a/test/core/end2end/bad_server_response_test.c +++ b/test/core/end2end/bad_server_response_test.c @@ -55,13 +55,13 @@ #define HTTP2_RESP(STATUS_CODE) \ "\x00\x00\x00\x04\x00\x00\x00\x00\x00" \ - "\x00\x00" \ - "7\x01\x04\x00\x00\x00\x01" \ + "\x00\x00>\x01\x04\x00\x00\x00\x01" \ "\x10\x0e" \ "content-length\x01" \ "0" \ "\x10\x0c" \ - "content-type\x09text/html" \ + "content-type\x10" \ + "application/grpc" \ "\x10\x07:status\x03" #STATUS_CODE #define UNPARSEABLE_RESP "Bad Request\n" @@ -119,7 +119,7 @@ static void handle_read(grpc_exec_ctx *exec_ctx, void *arg, grpc_error *error) { for (i = 0; i < state.temp_incoming_buffer.count; i++) { char *dump = gpr_dump_slice(state.temp_incoming_buffer.slices[i], GPR_DUMP_HEX | GPR_DUMP_ASCII); - gpr_log(GPR_DEBUG, "%s", dump); + gpr_log(GPR_DEBUG, "Server received: %s", dump); gpr_free(dump); } @@ -175,6 +175,7 @@ static void start_rpc(int target_port, grpc_status_code expected_status, grpc_metadata_array_init(&initial_metadata_recv); grpc_metadata_array_init(&trailing_metadata_recv); + memset(ops, 0, sizeof(ops)); op = ops; op->op = GRPC_OP_SEND_INITIAL_METADATA; op->data.send_initial_metadata.count = 0; @@ -209,6 +210,9 @@ static void start_rpc(int target_port, grpc_status_code expected_status, gpr_log(GPR_DEBUG, "Rpc status: %d, details: %s", status, details); GPR_ASSERT(status == expected_status); GPR_ASSERT(0 == strcmp(details, expected_detail)); + + grpc_metadata_array_destroy(&initial_metadata_recv); + grpc_metadata_array_destroy(&trailing_metadata_recv); gpr_free(details); cq_verifier_destroy(cqv); } From 0d896ef906f6a57f10832764611598d457d0e947 Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Tue, 21 Jun 2016 18:17:27 -0700 Subject: [PATCH 0169/1039] fix reading of compressed byte_buffer in C# --- src/csharp/ext/grpc_csharp_ext.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/csharp/ext/grpc_csharp_ext.c b/src/csharp/ext/grpc_csharp_ext.c index 4782654250c..9b8d050ea51 100644 --- a/src/csharp/ext/grpc_csharp_ext.c +++ b/src/csharp/ext/grpc_csharp_ext.c @@ -249,10 +249,12 @@ grpcsharp_batch_context_recv_initial_metadata( GPR_EXPORT intptr_t GPR_CALLTYPE grpcsharp_batch_context_recv_message_length( const grpcsharp_batch_context *ctx) { + grpc_byte_buffer_reader reader; if (!ctx->recv_message) { return -1; } - return (intptr_t)grpc_byte_buffer_length(ctx->recv_message); + grpc_byte_buffer_reader_init(&reader, ctx->recv_message); + return (intptr_t)grpc_byte_buffer_length(reader.buffer_out); } /* From 4047d5d4b642a3eb33100feda4b82d04179263f6 Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Tue, 21 Jun 2016 20:53:56 -0700 Subject: [PATCH 0170/1039] add test that C# can read compressed messages --- src/csharp/Grpc.Core.Tests/CompressionTest.cs | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/src/csharp/Grpc.Core.Tests/CompressionTest.cs b/src/csharp/Grpc.Core.Tests/CompressionTest.cs index 378c81851c0..f2f2931e480 100644 --- a/src/csharp/Grpc.Core.Tests/CompressionTest.cs +++ b/src/csharp/Grpc.Core.Tests/CompressionTest.cs @@ -34,6 +34,7 @@ using System; using System.Diagnostics; using System.Linq; +using System.Text; using System.Threading; using System.Threading.Tasks; using Grpc.Core; @@ -118,5 +119,30 @@ namespace Grpc.Core.Tests await call.ResponseStream.ToListAsync(); } + + [Test] + public void CanReadCompressedMessages() + { + var compressionMetadata = new Metadata + { + { new Metadata.Entry("grpc-internal-encoding-request", "gzip") } + }; + + helper.UnaryHandler = new UnaryServerMethod(async (req, context) => + { + await context.WriteResponseHeadersAsync(compressionMetadata); + return req; + }); + + var stringBuilder = new StringBuilder(); + for (int i = 0; i < 200000; i++) + { + stringBuilder.Append('a'); + } + var request = stringBuilder.ToString(); + var response = Calls.BlockingUnaryCall(helper.CreateUnaryCall(new CallOptions(compressionMetadata)), request); + + Assert.AreEqual(request, response); + } } } From 67ceba531971a2344a11be21ef5685b66e2609f1 Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Tue, 21 Jun 2016 18:29:09 -0700 Subject: [PATCH 0171/1039] regenerate messages.proto --- .../Grpc.IntegrationTesting/Messages.cs | 414 +++++++++++++----- 1 file changed, 301 insertions(+), 113 deletions(-) diff --git a/src/csharp/Grpc.IntegrationTesting/Messages.cs b/src/csharp/Grpc.IntegrationTesting/Messages.cs index d42501aa5b8..1240db128b0 100644 --- a/src/csharp/Grpc.IntegrationTesting/Messages.cs +++ b/src/csharp/Grpc.IntegrationTesting/Messages.cs @@ -24,46 +24,48 @@ namespace Grpc.Testing { byte[] descriptorData = global::System.Convert.FromBase64String( string.Concat( "CiVzcmMvcHJvdG8vZ3JwYy90ZXN0aW5nL21lc3NhZ2VzLnByb3RvEgxncnBj", - "LnRlc3RpbmciQAoHUGF5bG9hZBInCgR0eXBlGAEgASgOMhkuZ3JwYy50ZXN0", - "aW5nLlBheWxvYWRUeXBlEgwKBGJvZHkYAiABKAwiKwoKRWNob1N0YXR1cxIM", - "CgRjb2RlGAEgASgFEg8KB21lc3NhZ2UYAiABKAkioQIKDVNpbXBsZVJlcXVl", - "c3QSMAoNcmVzcG9uc2VfdHlwZRgBIAEoDjIZLmdycGMudGVzdGluZy5QYXls", - "b2FkVHlwZRIVCg1yZXNwb25zZV9zaXplGAIgASgFEiYKB3BheWxvYWQYAyAB", - "KAsyFS5ncnBjLnRlc3RpbmcuUGF5bG9hZBIVCg1maWxsX3VzZXJuYW1lGAQg", - "ASgIEhgKEGZpbGxfb2F1dGhfc2NvcGUYBSABKAgSOwoUcmVzcG9uc2VfY29t", - "cHJlc3Npb24YBiABKA4yHS5ncnBjLnRlc3RpbmcuQ29tcHJlc3Npb25UeXBl", - "EjEKD3Jlc3BvbnNlX3N0YXR1cxgHIAEoCzIYLmdycGMudGVzdGluZy5FY2hv", - "U3RhdHVzIl8KDlNpbXBsZVJlc3BvbnNlEiYKB3BheWxvYWQYASABKAsyFS5n", - "cnBjLnRlc3RpbmcuUGF5bG9hZBIQCgh1c2VybmFtZRgCIAEoCRITCgtvYXV0", - "aF9zY29wZRgDIAEoCSJDChlTdHJlYW1pbmdJbnB1dENhbGxSZXF1ZXN0EiYK", - "B3BheWxvYWQYASABKAsyFS5ncnBjLnRlc3RpbmcuUGF5bG9hZCI9ChpTdHJl", - "YW1pbmdJbnB1dENhbGxSZXNwb25zZRIfChdhZ2dyZWdhdGVkX3BheWxvYWRf", - "c2l6ZRgBIAEoBSI3ChJSZXNwb25zZVBhcmFtZXRlcnMSDAoEc2l6ZRgBIAEo", - "BRITCgtpbnRlcnZhbF91cxgCIAEoBSKlAgoaU3RyZWFtaW5nT3V0cHV0Q2Fs", - "bFJlcXVlc3QSMAoNcmVzcG9uc2VfdHlwZRgBIAEoDjIZLmdycGMudGVzdGlu", - "Zy5QYXlsb2FkVHlwZRI9ChNyZXNwb25zZV9wYXJhbWV0ZXJzGAIgAygLMiAu", - "Z3JwYy50ZXN0aW5nLlJlc3BvbnNlUGFyYW1ldGVycxImCgdwYXlsb2FkGAMg", - "ASgLMhUuZ3JwYy50ZXN0aW5nLlBheWxvYWQSOwoUcmVzcG9uc2VfY29tcHJl", - "c3Npb24YBiABKA4yHS5ncnBjLnRlc3RpbmcuQ29tcHJlc3Npb25UeXBlEjEK", - "D3Jlc3BvbnNlX3N0YXR1cxgHIAEoCzIYLmdycGMudGVzdGluZy5FY2hvU3Rh", - "dHVzIkUKG1N0cmVhbWluZ091dHB1dENhbGxSZXNwb25zZRImCgdwYXlsb2Fk", - "GAEgASgLMhUuZ3JwYy50ZXN0aW5nLlBheWxvYWQiMwoPUmVjb25uZWN0UGFy", - "YW1zEiAKGG1heF9yZWNvbm5lY3RfYmFja29mZl9tcxgBIAEoBSIzCg1SZWNv", - "bm5lY3RJbmZvEg4KBnBhc3NlZBgBIAEoCBISCgpiYWNrb2ZmX21zGAIgAygF", - "Kj8KC1BheWxvYWRUeXBlEhAKDENPTVBSRVNTQUJMRRAAEhIKDlVOQ09NUFJF", - "U1NBQkxFEAESCgoGUkFORE9NEAIqMgoPQ29tcHJlc3Npb25UeXBlEggKBE5P", - "TkUQABIICgRHWklQEAESCwoHREVGTEFURRACYgZwcm90bzM=")); + "LnRlc3RpbmciGgoJQm9vbFZhbHVlEg0KBXZhbHVlGAEgASgIIkAKB1BheWxv", + "YWQSJwoEdHlwZRgBIAEoDjIZLmdycGMudGVzdGluZy5QYXlsb2FkVHlwZRIM", + "CgRib2R5GAIgASgMIisKCkVjaG9TdGF0dXMSDAoEY29kZRgBIAEoBRIPCgdt", + "ZXNzYWdlGAIgASgJIs4CCg1TaW1wbGVSZXF1ZXN0EjAKDXJlc3BvbnNlX3R5", + "cGUYASABKA4yGS5ncnBjLnRlc3RpbmcuUGF5bG9hZFR5cGUSFQoNcmVzcG9u", + "c2Vfc2l6ZRgCIAEoBRImCgdwYXlsb2FkGAMgASgLMhUuZ3JwYy50ZXN0aW5n", + "LlBheWxvYWQSFQoNZmlsbF91c2VybmFtZRgEIAEoCBIYChBmaWxsX29hdXRo", + "X3Njb3BlGAUgASgIEjQKE3Jlc3BvbnNlX2NvbXByZXNzZWQYBiABKAsyFy5n", + "cnBjLnRlc3RpbmcuQm9vbFZhbHVlEjEKD3Jlc3BvbnNlX3N0YXR1cxgHIAEo", + "CzIYLmdycGMudGVzdGluZy5FY2hvU3RhdHVzEjIKEWV4cGVjdF9jb21wcmVz", + "c2VkGAggASgLMhcuZ3JwYy50ZXN0aW5nLkJvb2xWYWx1ZSJfCg5TaW1wbGVS", + "ZXNwb25zZRImCgdwYXlsb2FkGAEgASgLMhUuZ3JwYy50ZXN0aW5nLlBheWxv", + "YWQSEAoIdXNlcm5hbWUYAiABKAkSEwoLb2F1dGhfc2NvcGUYAyABKAkidwoZ", + "U3RyZWFtaW5nSW5wdXRDYWxsUmVxdWVzdBImCgdwYXlsb2FkGAEgASgLMhUu", + "Z3JwYy50ZXN0aW5nLlBheWxvYWQSMgoRZXhwZWN0X2NvbXByZXNzZWQYAiAB", + "KAsyFy5ncnBjLnRlc3RpbmcuQm9vbFZhbHVlIj0KGlN0cmVhbWluZ0lucHV0", + "Q2FsbFJlc3BvbnNlEh8KF2FnZ3JlZ2F0ZWRfcGF5bG9hZF9zaXplGAEgASgF", + "ImQKElJlc3BvbnNlUGFyYW1ldGVycxIMCgRzaXplGAEgASgFEhMKC2ludGVy", + "dmFsX3VzGAIgASgFEisKCmNvbXByZXNzZWQYAyABKAsyFy5ncnBjLnRlc3Rp", + "bmcuQm9vbFZhbHVlIugBChpTdHJlYW1pbmdPdXRwdXRDYWxsUmVxdWVzdBIw", + "Cg1yZXNwb25zZV90eXBlGAEgASgOMhkuZ3JwYy50ZXN0aW5nLlBheWxvYWRU", + "eXBlEj0KE3Jlc3BvbnNlX3BhcmFtZXRlcnMYAiADKAsyIC5ncnBjLnRlc3Rp", + "bmcuUmVzcG9uc2VQYXJhbWV0ZXJzEiYKB3BheWxvYWQYAyABKAsyFS5ncnBj", + "LnRlc3RpbmcuUGF5bG9hZBIxCg9yZXNwb25zZV9zdGF0dXMYByABKAsyGC5n", + "cnBjLnRlc3RpbmcuRWNob1N0YXR1cyJFChtTdHJlYW1pbmdPdXRwdXRDYWxs", + "UmVzcG9uc2USJgoHcGF5bG9hZBgBIAEoCzIVLmdycGMudGVzdGluZy5QYXls", + "b2FkIjMKD1JlY29ubmVjdFBhcmFtcxIgChhtYXhfcmVjb25uZWN0X2JhY2tv", + "ZmZfbXMYASABKAUiMwoNUmVjb25uZWN0SW5mbxIOCgZwYXNzZWQYASABKAgS", + "EgoKYmFja29mZl9tcxgCIAMoBSofCgtQYXlsb2FkVHlwZRIQCgxDT01QUkVT", + "U0FCTEUQAGIGcHJvdG8z")); descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData, new pbr::FileDescriptor[] { }, - new pbr::GeneratedClrTypeInfo(new[] {typeof(global::Grpc.Testing.PayloadType), typeof(global::Grpc.Testing.CompressionType), }, new pbr::GeneratedClrTypeInfo[] { + new pbr::GeneratedClrTypeInfo(new[] {typeof(global::Grpc.Testing.PayloadType), }, new pbr::GeneratedClrTypeInfo[] { + new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.BoolValue), global::Grpc.Testing.BoolValue.Parser, new[]{ "Value" }, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.Payload), global::Grpc.Testing.Payload.Parser, new[]{ "Type", "Body" }, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.EchoStatus), global::Grpc.Testing.EchoStatus.Parser, new[]{ "Code", "Message" }, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.SimpleRequest), global::Grpc.Testing.SimpleRequest.Parser, new[]{ "ResponseType", "ResponseSize", "Payload", "FillUsername", "FillOauthScope", "ResponseCompression", "ResponseStatus" }, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.SimpleRequest), global::Grpc.Testing.SimpleRequest.Parser, new[]{ "ResponseType", "ResponseSize", "Payload", "FillUsername", "FillOauthScope", "ResponseCompressed", "ResponseStatus", "ExpectCompressed" }, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.SimpleResponse), global::Grpc.Testing.SimpleResponse.Parser, new[]{ "Payload", "Username", "OauthScope" }, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.StreamingInputCallRequest), global::Grpc.Testing.StreamingInputCallRequest.Parser, new[]{ "Payload" }, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.StreamingInputCallRequest), global::Grpc.Testing.StreamingInputCallRequest.Parser, new[]{ "Payload", "ExpectCompressed" }, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.StreamingInputCallResponse), global::Grpc.Testing.StreamingInputCallResponse.Parser, new[]{ "AggregatedPayloadSize" }, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.ResponseParameters), global::Grpc.Testing.ResponseParameters.Parser, new[]{ "Size", "IntervalUs" }, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.StreamingOutputCallRequest), global::Grpc.Testing.StreamingOutputCallRequest.Parser, new[]{ "ResponseType", "ResponseParameters", "Payload", "ResponseCompression", "ResponseStatus" }, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.ResponseParameters), global::Grpc.Testing.ResponseParameters.Parser, new[]{ "Size", "IntervalUs", "Compressed" }, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.StreamingOutputCallRequest), global::Grpc.Testing.StreamingOutputCallRequest.Parser, new[]{ "ResponseType", "ResponseParameters", "Payload", "ResponseStatus" }, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.StreamingOutputCallResponse), global::Grpc.Testing.StreamingOutputCallResponse.Parser, new[]{ "Payload" }, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.ReconnectParams), global::Grpc.Testing.ReconnectParams.Parser, new[]{ "MaxReconnectBackoffMs" }, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::Grpc.Testing.ReconnectInfo), global::Grpc.Testing.ReconnectInfo.Parser, new[]{ "Passed", "BackoffMs" }, null, null, null) @@ -74,6 +76,7 @@ namespace Grpc.Testing { } #region Enums /// + /// DEPRECATED, don't use. To be removed shortly. /// The type of payload that should be returned. /// public enum PayloadType { @@ -81,31 +84,122 @@ namespace Grpc.Testing { /// Compressable text format. /// [pbr::OriginalName("COMPRESSABLE")] Compressable = 0, - /// - /// Uncompressable binary format. - /// - [pbr::OriginalName("UNCOMPRESSABLE")] Uncompressable = 1, - /// - /// Randomly chosen from all other formats defined in this enum. - /// - [pbr::OriginalName("RANDOM")] Random = 2, } + #endregion + + #region Messages /// - /// Compression algorithms + /// TODO(dgq): Go back to using well-known types once + /// https://github.com/grpc/grpc/issues/6980 has been fixed. + /// import "google/protobuf/wrappers.proto"; /// - public enum CompressionType { + [global::System.Diagnostics.DebuggerNonUserCodeAttribute()] + public sealed partial class BoolValue : pb::IMessage { + private static readonly pb::MessageParser _parser = new pb::MessageParser(() => new BoolValue()); + public static pb::MessageParser Parser { get { return _parser; } } + + public static pbr::MessageDescriptor Descriptor { + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[0]; } + } + + pbr::MessageDescriptor pb::IMessage.Descriptor { + get { return Descriptor; } + } + + public BoolValue() { + OnConstruction(); + } + + partial void OnConstruction(); + + public BoolValue(BoolValue other) : this() { + value_ = other.value_; + } + + public BoolValue Clone() { + return new BoolValue(this); + } + + /// Field number for the "value" field. + public const int ValueFieldNumber = 1; + private bool value_; /// - /// No compression + /// The bool value. /// - [pbr::OriginalName("NONE")] None = 0, - [pbr::OriginalName("GZIP")] Gzip = 1, - [pbr::OriginalName("DEFLATE")] Deflate = 2, - } + public bool Value { + get { return value_; } + set { + value_ = value; + } + } - #endregion + public override bool Equals(object other) { + return Equals(other as BoolValue); + } + + public bool Equals(BoolValue other) { + if (ReferenceEquals(other, null)) { + return false; + } + if (ReferenceEquals(other, this)) { + return true; + } + if (Value != other.Value) return false; + return true; + } + + public override int GetHashCode() { + int hash = 1; + if (Value != false) hash ^= Value.GetHashCode(); + return hash; + } + + public override string ToString() { + return pb::JsonFormatter.ToDiagnosticString(this); + } + + public void WriteTo(pb::CodedOutputStream output) { + if (Value != false) { + output.WriteRawTag(8); + output.WriteBool(Value); + } + } + + public int CalculateSize() { + int size = 0; + if (Value != false) { + size += 1 + 1; + } + return size; + } + + public void MergeFrom(BoolValue other) { + if (other == null) { + return; + } + if (other.Value != false) { + Value = other.Value; + } + } + + public void MergeFrom(pb::CodedInputStream input) { + uint tag; + while ((tag = input.ReadTag()) != 0) { + switch(tag) { + default: + input.SkipLastField(); + break; + case 8: { + Value = input.ReadBool(); + break; + } + } + } + } + + } - #region Messages /// /// A block of data, to simply increase gRPC message size. /// @@ -115,7 +209,7 @@ namespace Grpc.Testing { public static pb::MessageParser Parser { get { return _parser; } } public static pbr::MessageDescriptor Descriptor { - get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[0]; } + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[1]; } } pbr::MessageDescriptor pb::IMessage.Descriptor { @@ -141,6 +235,7 @@ namespace Grpc.Testing { public const int TypeFieldNumber = 1; private global::Grpc.Testing.PayloadType type_ = 0; /// + /// DEPRECATED, don't use. To be removed shortly. /// The type of data in body. /// public global::Grpc.Testing.PayloadType Type { @@ -255,7 +350,7 @@ namespace Grpc.Testing { public static pb::MessageParser Parser { get { return _parser; } } public static pbr::MessageDescriptor Descriptor { - get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[1]; } + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[2]; } } pbr::MessageDescriptor pb::IMessage.Descriptor { @@ -388,7 +483,7 @@ namespace Grpc.Testing { public static pb::MessageParser Parser { get { return _parser; } } public static pbr::MessageDescriptor Descriptor { - get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[2]; } + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[3]; } } pbr::MessageDescriptor pb::IMessage.Descriptor { @@ -407,8 +502,9 @@ namespace Grpc.Testing { Payload = other.payload_ != null ? other.Payload.Clone() : null; fillUsername_ = other.fillUsername_; fillOauthScope_ = other.fillOauthScope_; - responseCompression_ = other.responseCompression_; + ResponseCompressed = other.responseCompressed_ != null ? other.ResponseCompressed.Clone() : null; ResponseStatus = other.responseStatus_ != null ? other.ResponseStatus.Clone() : null; + ExpectCompressed = other.expectCompressed_ != null ? other.ExpectCompressed.Clone() : null; } public SimpleRequest Clone() { @@ -419,6 +515,7 @@ namespace Grpc.Testing { public const int ResponseTypeFieldNumber = 1; private global::Grpc.Testing.PayloadType responseType_ = 0; /// + /// DEPRECATED, don't use. To be removed shortly. /// Desired payload type in the response from the server. /// If response_type is RANDOM, server randomly chooses one from other formats. /// @@ -434,7 +531,6 @@ namespace Grpc.Testing { private int responseSize_; /// /// Desired payload size in the response from the server. - /// If response_type is COMPRESSABLE, this denotes the size before compression. /// public int ResponseSize { get { return responseSize_; } @@ -482,16 +578,19 @@ namespace Grpc.Testing { } } - /// Field number for the "response_compression" field. - public const int ResponseCompressionFieldNumber = 6; - private global::Grpc.Testing.CompressionType responseCompression_ = 0; + /// Field number for the "response_compressed" field. + public const int ResponseCompressedFieldNumber = 6; + private global::Grpc.Testing.BoolValue responseCompressed_; /// - /// Compression algorithm to be used by the server for the response (stream) + /// Whether to request the server to compress the response. This field is + /// "nullable" in order to interoperate seamlessly with clients not able to + /// implement the full compression tests by introspecting the call to verify + /// the response's compression status. /// - public global::Grpc.Testing.CompressionType ResponseCompression { - get { return responseCompression_; } + public global::Grpc.Testing.BoolValue ResponseCompressed { + get { return responseCompressed_; } set { - responseCompression_ = value; + responseCompressed_ = value; } } @@ -508,6 +607,19 @@ namespace Grpc.Testing { } } + /// Field number for the "expect_compressed" field. + public const int ExpectCompressedFieldNumber = 8; + private global::Grpc.Testing.BoolValue expectCompressed_; + /// + /// Whether the server should expect this request to be compressed. + /// + public global::Grpc.Testing.BoolValue ExpectCompressed { + get { return expectCompressed_; } + set { + expectCompressed_ = value; + } + } + public override bool Equals(object other) { return Equals(other as SimpleRequest); } @@ -524,8 +636,9 @@ namespace Grpc.Testing { if (!object.Equals(Payload, other.Payload)) return false; if (FillUsername != other.FillUsername) return false; if (FillOauthScope != other.FillOauthScope) return false; - if (ResponseCompression != other.ResponseCompression) return false; + if (!object.Equals(ResponseCompressed, other.ResponseCompressed)) return false; if (!object.Equals(ResponseStatus, other.ResponseStatus)) return false; + if (!object.Equals(ExpectCompressed, other.ExpectCompressed)) return false; return true; } @@ -536,8 +649,9 @@ namespace Grpc.Testing { if (payload_ != null) hash ^= Payload.GetHashCode(); if (FillUsername != false) hash ^= FillUsername.GetHashCode(); if (FillOauthScope != false) hash ^= FillOauthScope.GetHashCode(); - if (ResponseCompression != 0) hash ^= ResponseCompression.GetHashCode(); + if (responseCompressed_ != null) hash ^= ResponseCompressed.GetHashCode(); if (responseStatus_ != null) hash ^= ResponseStatus.GetHashCode(); + if (expectCompressed_ != null) hash ^= ExpectCompressed.GetHashCode(); return hash; } @@ -566,14 +680,18 @@ namespace Grpc.Testing { output.WriteRawTag(40); output.WriteBool(FillOauthScope); } - if (ResponseCompression != 0) { - output.WriteRawTag(48); - output.WriteEnum((int) ResponseCompression); + if (responseCompressed_ != null) { + output.WriteRawTag(50); + output.WriteMessage(ResponseCompressed); } if (responseStatus_ != null) { output.WriteRawTag(58); output.WriteMessage(ResponseStatus); } + if (expectCompressed_ != null) { + output.WriteRawTag(66); + output.WriteMessage(ExpectCompressed); + } } public int CalculateSize() { @@ -593,12 +711,15 @@ namespace Grpc.Testing { if (FillOauthScope != false) { size += 1 + 1; } - if (ResponseCompression != 0) { - size += 1 + pb::CodedOutputStream.ComputeEnumSize((int) ResponseCompression); + if (responseCompressed_ != null) { + size += 1 + pb::CodedOutputStream.ComputeMessageSize(ResponseCompressed); } if (responseStatus_ != null) { size += 1 + pb::CodedOutputStream.ComputeMessageSize(ResponseStatus); } + if (expectCompressed_ != null) { + size += 1 + pb::CodedOutputStream.ComputeMessageSize(ExpectCompressed); + } return size; } @@ -624,8 +745,11 @@ namespace Grpc.Testing { if (other.FillOauthScope != false) { FillOauthScope = other.FillOauthScope; } - if (other.ResponseCompression != 0) { - ResponseCompression = other.ResponseCompression; + if (other.responseCompressed_ != null) { + if (responseCompressed_ == null) { + responseCompressed_ = new global::Grpc.Testing.BoolValue(); + } + ResponseCompressed.MergeFrom(other.ResponseCompressed); } if (other.responseStatus_ != null) { if (responseStatus_ == null) { @@ -633,6 +757,12 @@ namespace Grpc.Testing { } ResponseStatus.MergeFrom(other.ResponseStatus); } + if (other.expectCompressed_ != null) { + if (expectCompressed_ == null) { + expectCompressed_ = new global::Grpc.Testing.BoolValue(); + } + ExpectCompressed.MergeFrom(other.ExpectCompressed); + } } public void MergeFrom(pb::CodedInputStream input) { @@ -665,8 +795,11 @@ namespace Grpc.Testing { FillOauthScope = input.ReadBool(); break; } - case 48: { - responseCompression_ = (global::Grpc.Testing.CompressionType) input.ReadEnum(); + case 50: { + if (responseCompressed_ == null) { + responseCompressed_ = new global::Grpc.Testing.BoolValue(); + } + input.ReadMessage(responseCompressed_); break; } case 58: { @@ -676,6 +809,13 @@ namespace Grpc.Testing { input.ReadMessage(responseStatus_); break; } + case 66: { + if (expectCompressed_ == null) { + expectCompressed_ = new global::Grpc.Testing.BoolValue(); + } + input.ReadMessage(expectCompressed_); + break; + } } } } @@ -691,7 +831,7 @@ namespace Grpc.Testing { public static pb::MessageParser Parser { get { return _parser; } } public static pbr::MessageDescriptor Descriptor { - get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[3]; } + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[4]; } } pbr::MessageDescriptor pb::IMessage.Descriptor { @@ -867,7 +1007,7 @@ namespace Grpc.Testing { public static pb::MessageParser Parser { get { return _parser; } } public static pbr::MessageDescriptor Descriptor { - get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[4]; } + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[5]; } } pbr::MessageDescriptor pb::IMessage.Descriptor { @@ -882,6 +1022,7 @@ namespace Grpc.Testing { public StreamingInputCallRequest(StreamingInputCallRequest other) : this() { Payload = other.payload_ != null ? other.Payload.Clone() : null; + ExpectCompressed = other.expectCompressed_ != null ? other.ExpectCompressed.Clone() : null; } public StreamingInputCallRequest Clone() { @@ -901,6 +1042,22 @@ namespace Grpc.Testing { } } + /// Field number for the "expect_compressed" field. + public const int ExpectCompressedFieldNumber = 2; + private global::Grpc.Testing.BoolValue expectCompressed_; + /// + /// Whether the server should expect this request to be compressed. This field + /// is "nullable" in order to interoperate seamlessly with servers not able to + /// implement the full compression tests by introspecting the call to verify + /// the request's compression status. + /// + public global::Grpc.Testing.BoolValue ExpectCompressed { + get { return expectCompressed_; } + set { + expectCompressed_ = value; + } + } + public override bool Equals(object other) { return Equals(other as StreamingInputCallRequest); } @@ -913,12 +1070,14 @@ namespace Grpc.Testing { return true; } if (!object.Equals(Payload, other.Payload)) return false; + if (!object.Equals(ExpectCompressed, other.ExpectCompressed)) return false; return true; } public override int GetHashCode() { int hash = 1; if (payload_ != null) hash ^= Payload.GetHashCode(); + if (expectCompressed_ != null) hash ^= ExpectCompressed.GetHashCode(); return hash; } @@ -931,6 +1090,10 @@ namespace Grpc.Testing { output.WriteRawTag(10); output.WriteMessage(Payload); } + if (expectCompressed_ != null) { + output.WriteRawTag(18); + output.WriteMessage(ExpectCompressed); + } } public int CalculateSize() { @@ -938,6 +1101,9 @@ namespace Grpc.Testing { if (payload_ != null) { size += 1 + pb::CodedOutputStream.ComputeMessageSize(Payload); } + if (expectCompressed_ != null) { + size += 1 + pb::CodedOutputStream.ComputeMessageSize(ExpectCompressed); + } return size; } @@ -951,6 +1117,12 @@ namespace Grpc.Testing { } Payload.MergeFrom(other.Payload); } + if (other.expectCompressed_ != null) { + if (expectCompressed_ == null) { + expectCompressed_ = new global::Grpc.Testing.BoolValue(); + } + ExpectCompressed.MergeFrom(other.ExpectCompressed); + } } public void MergeFrom(pb::CodedInputStream input) { @@ -967,6 +1139,13 @@ namespace Grpc.Testing { input.ReadMessage(payload_); break; } + case 18: { + if (expectCompressed_ == null) { + expectCompressed_ = new global::Grpc.Testing.BoolValue(); + } + input.ReadMessage(expectCompressed_); + break; + } } } } @@ -982,7 +1161,7 @@ namespace Grpc.Testing { public static pb::MessageParser Parser { get { return _parser; } } public static pbr::MessageDescriptor Descriptor { - get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[5]; } + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[6]; } } pbr::MessageDescriptor pb::IMessage.Descriptor { @@ -1091,7 +1270,7 @@ namespace Grpc.Testing { public static pb::MessageParser Parser { get { return _parser; } } public static pbr::MessageDescriptor Descriptor { - get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[6]; } + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[7]; } } pbr::MessageDescriptor pb::IMessage.Descriptor { @@ -1107,6 +1286,7 @@ namespace Grpc.Testing { public ResponseParameters(ResponseParameters other) : this() { size_ = other.size_; intervalUs_ = other.intervalUs_; + Compressed = other.compressed_ != null ? other.Compressed.Clone() : null; } public ResponseParameters Clone() { @@ -1118,7 +1298,6 @@ namespace Grpc.Testing { private int size_; /// /// Desired payload sizes in responses from the server. - /// If response_type is COMPRESSABLE, this denotes the size before compression. /// public int Size { get { return size_; } @@ -1141,6 +1320,22 @@ namespace Grpc.Testing { } } + /// Field number for the "compressed" field. + public const int CompressedFieldNumber = 3; + private global::Grpc.Testing.BoolValue compressed_; + /// + /// Whether to request the server to compress the response. This field is + /// "nullable" in order to interoperate seamlessly with clients not able to + /// implement the full compression tests by introspecting the call to verify + /// the response's compression status. + /// + public global::Grpc.Testing.BoolValue Compressed { + get { return compressed_; } + set { + compressed_ = value; + } + } + public override bool Equals(object other) { return Equals(other as ResponseParameters); } @@ -1154,6 +1349,7 @@ namespace Grpc.Testing { } if (Size != other.Size) return false; if (IntervalUs != other.IntervalUs) return false; + if (!object.Equals(Compressed, other.Compressed)) return false; return true; } @@ -1161,6 +1357,7 @@ namespace Grpc.Testing { int hash = 1; if (Size != 0) hash ^= Size.GetHashCode(); if (IntervalUs != 0) hash ^= IntervalUs.GetHashCode(); + if (compressed_ != null) hash ^= Compressed.GetHashCode(); return hash; } @@ -1177,6 +1374,10 @@ namespace Grpc.Testing { output.WriteRawTag(16); output.WriteInt32(IntervalUs); } + if (compressed_ != null) { + output.WriteRawTag(26); + output.WriteMessage(Compressed); + } } public int CalculateSize() { @@ -1187,6 +1388,9 @@ namespace Grpc.Testing { if (IntervalUs != 0) { size += 1 + pb::CodedOutputStream.ComputeInt32Size(IntervalUs); } + if (compressed_ != null) { + size += 1 + pb::CodedOutputStream.ComputeMessageSize(Compressed); + } return size; } @@ -1200,6 +1404,12 @@ namespace Grpc.Testing { if (other.IntervalUs != 0) { IntervalUs = other.IntervalUs; } + if (other.compressed_ != null) { + if (compressed_ == null) { + compressed_ = new global::Grpc.Testing.BoolValue(); + } + Compressed.MergeFrom(other.Compressed); + } } public void MergeFrom(pb::CodedInputStream input) { @@ -1217,6 +1427,13 @@ namespace Grpc.Testing { IntervalUs = input.ReadInt32(); break; } + case 26: { + if (compressed_ == null) { + compressed_ = new global::Grpc.Testing.BoolValue(); + } + input.ReadMessage(compressed_); + break; + } } } } @@ -1232,7 +1449,7 @@ namespace Grpc.Testing { public static pb::MessageParser Parser { get { return _parser; } } public static pbr::MessageDescriptor Descriptor { - get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[7]; } + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[8]; } } pbr::MessageDescriptor pb::IMessage.Descriptor { @@ -1249,7 +1466,6 @@ namespace Grpc.Testing { responseType_ = other.responseType_; responseParameters_ = other.responseParameters_.Clone(); Payload = other.payload_ != null ? other.Payload.Clone() : null; - responseCompression_ = other.responseCompression_; ResponseStatus = other.responseStatus_ != null ? other.ResponseStatus.Clone() : null; } @@ -1261,6 +1477,7 @@ namespace Grpc.Testing { public const int ResponseTypeFieldNumber = 1; private global::Grpc.Testing.PayloadType responseType_ = 0; /// + /// DEPRECATED, don't use. To be removed shortly. /// Desired payload type in the response from the server. /// If response_type is RANDOM, the payload from each response in the stream /// might be of different types. This is to simulate a mixed type of payload @@ -1298,19 +1515,6 @@ namespace Grpc.Testing { } } - /// Field number for the "response_compression" field. - public const int ResponseCompressionFieldNumber = 6; - private global::Grpc.Testing.CompressionType responseCompression_ = 0; - /// - /// Compression algorithm to be used by the server for the response (stream) - /// - public global::Grpc.Testing.CompressionType ResponseCompression { - get { return responseCompression_; } - set { - responseCompression_ = value; - } - } - /// Field number for the "response_status" field. public const int ResponseStatusFieldNumber = 7; private global::Grpc.Testing.EchoStatus responseStatus_; @@ -1338,7 +1542,6 @@ namespace Grpc.Testing { if (ResponseType != other.ResponseType) return false; if(!responseParameters_.Equals(other.responseParameters_)) return false; if (!object.Equals(Payload, other.Payload)) return false; - if (ResponseCompression != other.ResponseCompression) return false; if (!object.Equals(ResponseStatus, other.ResponseStatus)) return false; return true; } @@ -1348,7 +1551,6 @@ namespace Grpc.Testing { if (ResponseType != 0) hash ^= ResponseType.GetHashCode(); hash ^= responseParameters_.GetHashCode(); if (payload_ != null) hash ^= Payload.GetHashCode(); - if (ResponseCompression != 0) hash ^= ResponseCompression.GetHashCode(); if (responseStatus_ != null) hash ^= ResponseStatus.GetHashCode(); return hash; } @@ -1367,10 +1569,6 @@ namespace Grpc.Testing { output.WriteRawTag(26); output.WriteMessage(Payload); } - if (ResponseCompression != 0) { - output.WriteRawTag(48); - output.WriteEnum((int) ResponseCompression); - } if (responseStatus_ != null) { output.WriteRawTag(58); output.WriteMessage(ResponseStatus); @@ -1386,9 +1584,6 @@ namespace Grpc.Testing { if (payload_ != null) { size += 1 + pb::CodedOutputStream.ComputeMessageSize(Payload); } - if (ResponseCompression != 0) { - size += 1 + pb::CodedOutputStream.ComputeEnumSize((int) ResponseCompression); - } if (responseStatus_ != null) { size += 1 + pb::CodedOutputStream.ComputeMessageSize(ResponseStatus); } @@ -1409,9 +1604,6 @@ namespace Grpc.Testing { } Payload.MergeFrom(other.Payload); } - if (other.ResponseCompression != 0) { - ResponseCompression = other.ResponseCompression; - } if (other.responseStatus_ != null) { if (responseStatus_ == null) { responseStatus_ = new global::Grpc.Testing.EchoStatus(); @@ -1442,10 +1634,6 @@ namespace Grpc.Testing { input.ReadMessage(payload_); break; } - case 48: { - responseCompression_ = (global::Grpc.Testing.CompressionType) input.ReadEnum(); - break; - } case 58: { if (responseStatus_ == null) { responseStatus_ = new global::Grpc.Testing.EchoStatus(); @@ -1468,7 +1656,7 @@ namespace Grpc.Testing { public static pb::MessageParser Parser { get { return _parser; } } public static pbr::MessageDescriptor Descriptor { - get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[8]; } + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[9]; } } pbr::MessageDescriptor pb::IMessage.Descriptor { @@ -1584,7 +1772,7 @@ namespace Grpc.Testing { public static pb::MessageParser Parser { get { return _parser; } } public static pbr::MessageDescriptor Descriptor { - get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[9]; } + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[10]; } } pbr::MessageDescriptor pb::IMessage.Descriptor { @@ -1692,7 +1880,7 @@ namespace Grpc.Testing { public static pb::MessageParser Parser { get { return _parser; } } public static pbr::MessageDescriptor Descriptor { - get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[10]; } + get { return global::Grpc.Testing.MessagesReflection.Descriptor.MessageTypes[11]; } } pbr::MessageDescriptor pb::IMessage.Descriptor { From f8135f6b05238237ec5ed635ecf1f5a4ba9b3301 Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Tue, 21 Jun 2016 18:41:09 -0700 Subject: [PATCH 0172/1039] remove occurences of compressable payload type --- .../Grpc.IntegrationTesting/InteropClient.cs | 23 ------------------- 1 file changed, 23 deletions(-) diff --git a/src/csharp/Grpc.IntegrationTesting/InteropClient.cs b/src/csharp/Grpc.IntegrationTesting/InteropClient.cs index d273867a6a7..834c276c64d 100644 --- a/src/csharp/Grpc.IntegrationTesting/InteropClient.cs +++ b/src/csharp/Grpc.IntegrationTesting/InteropClient.cs @@ -240,13 +240,11 @@ namespace Grpc.IntegrationTesting Console.WriteLine("running large_unary"); var request = new SimpleRequest { - ResponseType = PayloadType.Compressable, ResponseSize = 314159, Payload = CreateZerosPayload(271828) }; var response = client.UnaryCall(request); - Assert.AreEqual(PayloadType.Compressable, response.Payload.Type); Assert.AreEqual(314159, response.Payload.Body.Length); Console.WriteLine("Passed!"); } @@ -275,17 +273,12 @@ namespace Grpc.IntegrationTesting var request = new StreamingOutputCallRequest { - ResponseType = PayloadType.Compressable, ResponseParameters = { bodySizes.Select((size) => new ResponseParameters { Size = size }) } }; using (var call = client.StreamingOutputCall(request)) { var responseList = await call.ResponseStream.ToListAsync(); - foreach (var res in responseList) - { - Assert.AreEqual(PayloadType.Compressable, res.Payload.Type); - } CollectionAssert.AreEqual(bodySizes, responseList.Select((item) => item.Payload.Body.Length)); } Console.WriteLine("Passed!"); @@ -299,46 +292,38 @@ namespace Grpc.IntegrationTesting { await call.RequestStream.WriteAsync(new StreamingOutputCallRequest { - ResponseType = PayloadType.Compressable, ResponseParameters = { new ResponseParameters { Size = 31415 } }, Payload = CreateZerosPayload(27182) }); Assert.IsTrue(await call.ResponseStream.MoveNext()); - Assert.AreEqual(PayloadType.Compressable, call.ResponseStream.Current.Payload.Type); Assert.AreEqual(31415, call.ResponseStream.Current.Payload.Body.Length); await call.RequestStream.WriteAsync(new StreamingOutputCallRequest { - ResponseType = PayloadType.Compressable, ResponseParameters = { new ResponseParameters { Size = 9 } }, Payload = CreateZerosPayload(8) }); Assert.IsTrue(await call.ResponseStream.MoveNext()); - Assert.AreEqual(PayloadType.Compressable, call.ResponseStream.Current.Payload.Type); Assert.AreEqual(9, call.ResponseStream.Current.Payload.Body.Length); await call.RequestStream.WriteAsync(new StreamingOutputCallRequest { - ResponseType = PayloadType.Compressable, ResponseParameters = { new ResponseParameters { Size = 2653 } }, Payload = CreateZerosPayload(1828) }); Assert.IsTrue(await call.ResponseStream.MoveNext()); - Assert.AreEqual(PayloadType.Compressable, call.ResponseStream.Current.Payload.Type); Assert.AreEqual(2653, call.ResponseStream.Current.Payload.Body.Length); await call.RequestStream.WriteAsync(new StreamingOutputCallRequest { - ResponseType = PayloadType.Compressable, ResponseParameters = { new ResponseParameters { Size = 58979 } }, Payload = CreateZerosPayload(45904) }); Assert.IsTrue(await call.ResponseStream.MoveNext()); - Assert.AreEqual(PayloadType.Compressable, call.ResponseStream.Current.Payload.Type); Assert.AreEqual(58979, call.ResponseStream.Current.Payload.Body.Length); await call.RequestStream.CompleteAsync(); @@ -367,7 +352,6 @@ namespace Grpc.IntegrationTesting var request = new SimpleRequest { - ResponseType = PayloadType.Compressable, ResponseSize = 314159, Payload = CreateZerosPayload(271828), FillUsername = true, @@ -377,7 +361,6 @@ namespace Grpc.IntegrationTesting // not setting credentials here because they were set on channel already var response = client.UnaryCall(request); - Assert.AreEqual(PayloadType.Compressable, response.Payload.Type); Assert.AreEqual(314159, response.Payload.Body.Length); Assert.False(string.IsNullOrEmpty(response.OauthScope)); Assert.True(oauthScope.Contains(response.OauthScope)); @@ -391,7 +374,6 @@ namespace Grpc.IntegrationTesting var request = new SimpleRequest { - ResponseType = PayloadType.Compressable, ResponseSize = 314159, Payload = CreateZerosPayload(271828), FillUsername = true, @@ -400,7 +382,6 @@ namespace Grpc.IntegrationTesting // not setting credentials here because they were set on channel already var response = client.UnaryCall(request); - Assert.AreEqual(PayloadType.Compressable, response.Payload.Type); Assert.AreEqual(314159, response.Payload.Body.Length); Assert.AreEqual(GetEmailFromServiceAccountFile(), response.Username); Console.WriteLine("Passed!"); @@ -480,13 +461,11 @@ namespace Grpc.IntegrationTesting { await call.RequestStream.WriteAsync(new StreamingOutputCallRequest { - ResponseType = PayloadType.Compressable, ResponseParameters = { new ResponseParameters { Size = 31415 } }, Payload = CreateZerosPayload(27182) }); Assert.IsTrue(await call.ResponseStream.MoveNext()); - Assert.AreEqual(PayloadType.Compressable, call.ResponseStream.Current.Payload.Type); Assert.AreEqual(31415, call.ResponseStream.Current.Payload.Body.Length); cts.Cancel(); @@ -546,7 +525,6 @@ namespace Grpc.IntegrationTesting // step 1: test unary call var request = new SimpleRequest { - ResponseType = PayloadType.Compressable, ResponseSize = 314159, Payload = CreateZerosPayload(271828) }; @@ -565,7 +543,6 @@ namespace Grpc.IntegrationTesting // step 2: test full duplex call var request = new StreamingOutputCallRequest { - ResponseType = PayloadType.Compressable, ResponseParameters = { new ResponseParameters { Size = 31415 } }, Payload = CreateZerosPayload(27182) }; From a7daf1edc2b336908918589c4de632197ebb92dc Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Tue, 21 Jun 2016 19:11:49 -0700 Subject: [PATCH 0173/1039] implement C# client compression interop tests --- .../Grpc.IntegrationTesting/InteropClient.cs | 108 ++++++++++++++++++ 1 file changed, 108 insertions(+) diff --git a/src/csharp/Grpc.IntegrationTesting/InteropClient.cs b/src/csharp/Grpc.IntegrationTesting/InteropClient.cs index 834c276c64d..17ef587d168 100644 --- a/src/csharp/Grpc.IntegrationTesting/InteropClient.cs +++ b/src/csharp/Grpc.IntegrationTesting/InteropClient.cs @@ -222,6 +222,12 @@ namespace Grpc.IntegrationTesting case "unimplemented_method": RunUnimplementedMethod(new UnimplementedService.UnimplementedServiceClient(channel)); break; + case "client_compressed_unary": + RunClientCompressedUnary(client); + break; + case "client_compressed_streaming": + await RunClientCompressedStreamingAsync(client); + break; default: throw new ArgumentException("Unknown test case " + options.TestCase); } @@ -615,11 +621,113 @@ namespace Grpc.IntegrationTesting Console.WriteLine("Passed!"); } + public static void RunClientCompressedUnary(TestService.TestServiceClient client) + { + Console.WriteLine("running client_compressed_unary"); + var probeRequest = new SimpleRequest + { + ExpectCompressed = new BoolValue + { + Value = true // lie about compression + }, + ResponseSize = 314159, + Payload = CreateZerosPayload(271828) + }; + var e = Assert.Throws(() => client.UnaryCall(probeRequest, CreateClientCompressionMetadata(false))); + Assert.AreEqual(StatusCode.InvalidArgument, e.Status.StatusCode); + + var compressedRequest = new SimpleRequest + { + ExpectCompressed = new BoolValue + { + Value = true + }, + ResponseSize = 314159, + Payload = CreateZerosPayload(271828) + }; + var response1 = client.UnaryCall(compressedRequest, CreateClientCompressionMetadata(true)); + Assert.AreEqual(314159, response1.Payload.Body.Length); + + var uncompressedRequest = new SimpleRequest + { + ExpectCompressed = new BoolValue + { + Value = false + }, + ResponseSize = 314159, + Payload = CreateZerosPayload(271828) + }; + var response2 = client.UnaryCall(uncompressedRequest, CreateClientCompressionMetadata(false)); + Assert.AreEqual(314159, response2.Payload.Body.Length); + + Console.WriteLine("Passed!"); + } + + public static async Task RunClientCompressedStreamingAsync(TestService.TestServiceClient client) + { + Console.WriteLine("running client_compressed_streaming"); + try + { + var probeCall = client.StreamingInputCall(CreateClientCompressionMetadata(false)); + await probeCall.RequestStream.WriteAsync(new StreamingInputCallRequest + { + ExpectCompressed = new BoolValue + { + Value = true + }, + Payload = CreateZerosPayload(27182) + }); + + // cannot use Assert.ThrowsAsync because it uses Task.Wait and would deadlock. + await probeCall; + Assert.Fail(); + } + catch (RpcException e) + { + Assert.AreEqual(StatusCode.InvalidArgument, e.Status.StatusCode); + } + + var call = client.StreamingInputCall(CreateClientCompressionMetadata(true)); + await call.RequestStream.WriteAsync(new StreamingInputCallRequest + { + ExpectCompressed = new BoolValue + { + Value = true + }, + Payload = CreateZerosPayload(27182) + }); + + call.RequestStream.WriteOptions = new WriteOptions(WriteFlags.NoCompress); + await call.RequestStream.WriteAsync(new StreamingInputCallRequest + { + ExpectCompressed = new BoolValue + { + Value = false + }, + Payload = CreateZerosPayload(45904) + }); + await call.RequestStream.CompleteAsync(); + + var response = await call.ResponseAsync; + Assert.AreEqual(73086, response.AggregatedPayloadSize); + + Console.WriteLine("Passed!"); + } + private static Payload CreateZerosPayload(int size) { return new Payload { Body = ByteString.CopyFrom(new byte[size]) }; } + private static Metadata CreateClientCompressionMetadata(bool compressed) + { + var algorithmName = compressed ? "gzip" : "identity"; + return new Metadata + { + { new Metadata.Entry("grpc-internal-encoding-request", algorithmName) } + }; + } + // extracts the client_email field from service account file used for auth test cases private static string GetEmailFromServiceAccountFile() { From 9fc079fddd2fc1bad958d6f7f7063363437426a5 Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Tue, 21 Jun 2016 19:53:31 -0700 Subject: [PATCH 0174/1039] enable client compression interop tests for C# --- tools/run_tests/run_interop_tests.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/tools/run_tests/run_interop_tests.py b/tools/run_tests/run_interop_tests.py index e3af721ee53..d76dd4b7d26 100755 --- a/tools/run_tests/run_interop_tests.py +++ b/tools/run_tests/run_interop_tests.py @@ -54,10 +54,13 @@ os.chdir(ROOT) _DEFAULT_SERVER_PORT=8080 -_SKIP_COMPRESSION = ['client_compressed_unary', - 'client_compressed_streaming', - 'server_compressed_unary', - 'server_compressed_streaming'] +_SKIP_CLIENT_COMPRESSION = ['client_compressed_unary', + 'client_compressed_streaming'] + +_SKIP_SERVER_COMPRESSION = ['server_compressed_unary', + 'server_compressed_streaming'] + +_SKIP_COMPRESSION = _SKIP_CLIENT_COMPRESSION + _SKIP_SERVER_COMPRESSION _SKIP_ADVANCED = ['custom_metadata', 'status_code_and_message', 'unimplemented_method'] @@ -113,7 +116,7 @@ class CSharpLanguage: return {} def unimplemented_test_cases(self): - return _SKIP_COMPRESSION + return _SKIP_SERVER_COMPRESSION def unimplemented_test_cases_server(self): return _SKIP_COMPRESSION From 606e35a4fb2a6e7dd711e7a2eec6c59bf6f2258b Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Wed, 22 Jun 2016 12:26:36 -0700 Subject: [PATCH 0175/1039] add C# constant for GRPC_COMPRESSION_REQUEST_ALGORITHM_MD_KEY --- src/csharp/Grpc.Core.Tests/CompressionTest.cs | 2 +- src/csharp/Grpc.Core/Metadata.cs | 7 +++++++ src/csharp/Grpc.IntegrationTesting/InteropClient.cs | 2 +- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/src/csharp/Grpc.Core.Tests/CompressionTest.cs b/src/csharp/Grpc.Core.Tests/CompressionTest.cs index f2f2931e480..16be210508a 100644 --- a/src/csharp/Grpc.Core.Tests/CompressionTest.cs +++ b/src/csharp/Grpc.Core.Tests/CompressionTest.cs @@ -125,7 +125,7 @@ namespace Grpc.Core.Tests { var compressionMetadata = new Metadata { - { new Metadata.Entry("grpc-internal-encoding-request", "gzip") } + { new Metadata.Entry(Metadata.CompressionRequestAlgorithmMetadataKey, "gzip") } }; helper.UnaryHandler = new UnaryServerMethod(async (req, context) => diff --git a/src/csharp/Grpc.Core/Metadata.cs b/src/csharp/Grpc.Core/Metadata.cs index f73f720094a..915bec146c9 100644 --- a/src/csharp/Grpc.Core/Metadata.cs +++ b/src/csharp/Grpc.Core/Metadata.cs @@ -63,6 +63,13 @@ namespace Grpc.Core /// public static readonly Metadata Empty = new Metadata().Freeze(); + /// + /// To be used in initial metadata to request specific compression algorithm + /// for given call. Direct selection of compression algorithms is an internal + /// feature and is not part of public API. + /// + internal const string CompressionRequestAlgorithmMetadataKey = "grpc-internal-encoding-request"; + readonly List entries; bool readOnly; diff --git a/src/csharp/Grpc.IntegrationTesting/InteropClient.cs b/src/csharp/Grpc.IntegrationTesting/InteropClient.cs index 17ef587d168..e27fe5b3d80 100644 --- a/src/csharp/Grpc.IntegrationTesting/InteropClient.cs +++ b/src/csharp/Grpc.IntegrationTesting/InteropClient.cs @@ -724,7 +724,7 @@ namespace Grpc.IntegrationTesting var algorithmName = compressed ? "gzip" : "identity"; return new Metadata { - { new Metadata.Entry("grpc-internal-encoding-request", algorithmName) } + { new Metadata.Entry(Metadata.CompressionRequestAlgorithmMetadataKey, algorithmName) } }; } From b16e4518b765f82b9b65222bd56738cda44e7259 Mon Sep 17 00:00:00 2001 From: Nathaniel Manista Date: Tue, 21 Jun 2016 23:24:06 +0000 Subject: [PATCH 0176/1039] Add a test of metadata, status code, and details --- src/python/grpcio/tests/tests.json | 1 + .../tests/unit/_metadata_code_details_test.py | 523 ++++++++++++++++++ 2 files changed, 524 insertions(+) create mode 100644 src/python/grpcio/tests/unit/_metadata_code_details_test.py diff --git a/src/python/grpcio/tests/tests.json b/src/python/grpcio/tests/tests.json index 8e509621a84..e384a2fc135 100644 --- a/src/python/grpcio/tests/tests.json +++ b/src/python/grpcio/tests/tests.json @@ -24,6 +24,7 @@ "_implementations_test.ChannelCredentialsTest", "_insecure_interop_test.InsecureInteropTest", "_logging_pool_test.LoggingPoolTest", + "_metadata_code_details_test.MetadataCodeDetailsTest", "_metadata_test.MetadataTest", "_not_found_test.NotFoundTest", "_python_plugin_test.PythonPluginTest", diff --git a/src/python/grpcio/tests/unit/_metadata_code_details_test.py b/src/python/grpcio/tests/unit/_metadata_code_details_test.py new file mode 100644 index 00000000000..dd74268cbf1 --- /dev/null +++ b/src/python/grpcio/tests/unit/_metadata_code_details_test.py @@ -0,0 +1,523 @@ +# Copyright 2016, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +"""Tests application-provided metadata, status code, and details.""" + +import threading +import unittest + +import grpc +from grpc.framework.foundation import logging_pool + +from tests.unit import test_common +from tests.unit.framework.common import test_constants +from tests.unit.framework.common import test_control + +_SERIALIZED_REQUEST = b'\x46\x47\x48' +_SERIALIZED_RESPONSE = b'\x49\x50\x51' + +_REQUEST_SERIALIZER = lambda unused_request: _SERIALIZED_REQUEST +_REQUEST_DESERIALIZER = lambda unused_serialized_request: object() +_RESPONSE_SERIALIZER = lambda unused_response: _SERIALIZED_RESPONSE +_RESPONSE_DESERIALIZER = lambda unused_serialized_resopnse: object() + +_SERVICE = b'test.TestService' +_UNARY_UNARY = b'UnaryUnary' +_UNARY_STREAM = b'UnaryStream' +_STREAM_UNARY = b'StreamUnary' +_STREAM_STREAM = b'StreamStream' + +_CLIENT_METADATA = ( + (b'client-md-key', b'client-md-key'), + (b'client-md-key-bin', b'\x00\x01') +) + +_SERVER_INITIAL_METADATA = ( + (b'server-initial-md-key', b'server-initial-md-value'), + (b'server-initial-md-key-bin', b'\x00\x02') +) + +_SERVER_TRAILING_METADATA = ( + (b'server-trailing-md-key', b'server-trailing-md-value'), + (b'server-trailing-md-key-bin', b'\x00\x03') +) + +_NON_OK_CODE = grpc.StatusCode.NOT_FOUND +_DETAILS = b'Test details!' + + +class _Servicer(object): + + def __init__(self): + self._lock = threading.Lock() + self._code = None + self._details = None + self._exception = False + self._return_none = False + self._received_client_metadata = None + + def unary_unary(self, request, context): + with self._lock: + self._received_client_metadata = context.invocation_metadata() + context.send_initial_metadata(_SERVER_INITIAL_METADATA) + context.set_trailing_metadata(_SERVER_TRAILING_METADATA) + if self._code is not None: + context.set_code(self._code) + if self._details is not None: + context.set_details(self._details) + if self._exception: + raise test_control.Defect() + else: + return None if self._return_none else object() + + def unary_stream(self, request, context): + with self._lock: + self._received_client_metadata = context.invocation_metadata() + context.send_initial_metadata(_SERVER_INITIAL_METADATA) + context.set_trailing_metadata(_SERVER_TRAILING_METADATA) + if self._code is not None: + context.set_code(self._code) + if self._details is not None: + context.set_details(self._details) + for _ in range(test_constants.STREAM_LENGTH // 2): + yield _SERIALIZED_RESPONSE + if self._exception: + raise test_control.Defect() + + def stream_unary(self, request_iterator, context): + with self._lock: + self._received_client_metadata = context.invocation_metadata() + context.send_initial_metadata(_SERVER_INITIAL_METADATA) + context.set_trailing_metadata(_SERVER_TRAILING_METADATA) + if self._code is not None: + context.set_code(self._code) + if self._details is not None: + context.set_details(self._details) + # TODO(https://github.com/grpc/grpc/issues/6891): just ignore the + # request iterator. + for ignored_request in request_iterator: + pass + if self._exception: + raise test_control.Defect() + else: + return None if self._return_none else _SERIALIZED_RESPONSE + + def stream_stream(self, request_iterator, context): + with self._lock: + self._received_client_metadata = context.invocation_metadata() + context.send_initial_metadata(_SERVER_INITIAL_METADATA) + context.set_trailing_metadata(_SERVER_TRAILING_METADATA) + if self._code is not None: + context.set_code(self._code) + if self._details is not None: + context.set_details(self._details) + # TODO(https://github.com/grpc/grpc/issues/6891): just ignore the + # request iterator. + for ignored_request in request_iterator: + pass + for _ in range(test_constants.STREAM_LENGTH // 3): + yield object() + if self._exception: + raise test_control.Defect() + + def set_code(self, code): + with self._lock: + self._code = code + + def set_details(self, details): + with self._lock: + self._details = details + + def set_exception(self): + with self._lock: + self._exception = True + + def set_return_none(self): + with self._lock: + self._return_none = True + + def received_client_metadata(self): + with self._lock: + return self._received_client_metadata + + +def _generic_handler(servicer): + method_handlers = { + _UNARY_UNARY: grpc.unary_unary_rpc_method_handler( + servicer.unary_unary, request_deserializer=_REQUEST_DESERIALIZER, + response_serializer=_RESPONSE_SERIALIZER), + _UNARY_STREAM: grpc.unary_stream_rpc_method_handler( + servicer.unary_stream), + _STREAM_UNARY: grpc.stream_unary_rpc_method_handler( + servicer.stream_unary), + _STREAM_STREAM: grpc.stream_stream_rpc_method_handler( + servicer.stream_stream, request_deserializer=_REQUEST_DESERIALIZER, + response_serializer=_RESPONSE_SERIALIZER), + } + return grpc.method_handlers_generic_handler(_SERVICE, method_handlers) + + +class MetadataCodeDetailsTest(unittest.TestCase): + + def setUp(self): + self._servicer = _Servicer() + self._server_pool = logging_pool.pool(test_constants.THREAD_CONCURRENCY) + self._server = grpc.server( + (_generic_handler(self._servicer),), self._server_pool) + port = self._server.add_insecure_port('[::]:0') + self._server.start() + + channel = grpc.insecure_channel('localhost:{}'.format(port)) + self._unary_unary = channel.unary_unary( + b'/'.join((b'', _SERVICE, _UNARY_UNARY,)), + request_serializer=_REQUEST_SERIALIZER, + response_deserializer=_RESPONSE_DESERIALIZER,) + self._unary_stream = channel.unary_stream( + b'/'.join((b'', _SERVICE, _UNARY_STREAM,)),) + self._stream_unary = channel.stream_unary( + b'/'.join((b'', _SERVICE, _STREAM_UNARY,)),) + self._stream_stream = channel.stream_stream( + b'/'.join((b'', _SERVICE, _STREAM_STREAM,)), + request_serializer=_REQUEST_SERIALIZER, + response_deserializer=_RESPONSE_DESERIALIZER,) + + + def testSuccessfulUnaryUnary(self): + self._servicer.set_details(_DETAILS) + + unused_response, call = self._unary_unary.with_call( + object(), metadata=_CLIENT_METADATA) + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, call.initial_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, call.trailing_metadata())) + self.assertIs(grpc.StatusCode.OK, call.code()) + self.assertEqual(_DETAILS, call.details()) + + def testSuccessfulUnaryStream(self): + self._servicer.set_details(_DETAILS) + + call = self._unary_stream(_SERIALIZED_REQUEST, metadata=_CLIENT_METADATA) + received_initial_metadata = call.initial_metadata() + for _ in call: + pass + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, received_initial_metadata)) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, call.trailing_metadata())) + self.assertIs(grpc.StatusCode.OK, call.code()) + self.assertEqual(_DETAILS, call.details()) + + def testSuccessfulStreamUnary(self): + self._servicer.set_details(_DETAILS) + + unused_response, call = self._stream_unary.with_call( + iter([_SERIALIZED_REQUEST] * test_constants.STREAM_LENGTH), + metadata=_CLIENT_METADATA) + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, call.initial_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, call.trailing_metadata())) + self.assertIs(grpc.StatusCode.OK, call.code()) + self.assertEqual(_DETAILS, call.details()) + + def testSuccessfulStreamStream(self): + self._servicer.set_details(_DETAILS) + + call = self._stream_stream( + iter([object()] * test_constants.STREAM_LENGTH), + metadata=_CLIENT_METADATA) + received_initial_metadata = call.initial_metadata() + for _ in call: + pass + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, received_initial_metadata)) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, call.trailing_metadata())) + self.assertIs(grpc.StatusCode.OK, call.code()) + self.assertEqual(_DETAILS, call.details()) + + def testCustomCodeUnaryUnary(self): + self._servicer.set_code(_NON_OK_CODE) + self._servicer.set_details(_DETAILS) + + with self.assertRaises(grpc.RpcError) as exception_context: + self._unary_unary.with_call(object(), metadata=_CLIENT_METADATA) + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, + exception_context.exception.initial_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, + exception_context.exception.trailing_metadata())) + self.assertIs(_NON_OK_CODE, exception_context.exception.code()) + self.assertEqual(_DETAILS, exception_context.exception.details()) + + def testCustomCodeUnaryStream(self): + self._servicer.set_code(_NON_OK_CODE) + self._servicer.set_details(_DETAILS) + + call = self._unary_stream(_SERIALIZED_REQUEST, metadata=_CLIENT_METADATA) + received_initial_metadata = call.initial_metadata() + with self.assertRaises(grpc.RpcError): + for _ in call: + pass + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, received_initial_metadata)) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, call.trailing_metadata())) + self.assertIs(_NON_OK_CODE, call.code()) + self.assertEqual(_DETAILS, call.details()) + + def testCustomCodeStreamUnary(self): + self._servicer.set_code(_NON_OK_CODE) + self._servicer.set_details(_DETAILS) + + with self.assertRaises(grpc.RpcError) as exception_context: + self._stream_unary.with_call( + iter([_SERIALIZED_REQUEST] * test_constants.STREAM_LENGTH), + metadata=_CLIENT_METADATA) + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, + exception_context.exception.initial_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, + exception_context.exception.trailing_metadata())) + self.assertIs(_NON_OK_CODE, exception_context.exception.code()) + self.assertEqual(_DETAILS, exception_context.exception.details()) + + def testCustomCodeStreamStream(self): + self._servicer.set_code(_NON_OK_CODE) + self._servicer.set_details(_DETAILS) + + call = self._stream_stream( + iter([object()] * test_constants.STREAM_LENGTH), + metadata=_CLIENT_METADATA) + received_initial_metadata = call.initial_metadata() + with self.assertRaises(grpc.RpcError) as exception_context: + for _ in call: + pass + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, received_initial_metadata)) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, + exception_context.exception.trailing_metadata())) + self.assertIs(_NON_OK_CODE, exception_context.exception.code()) + self.assertEqual(_DETAILS, exception_context.exception.details()) + + def testCustomCodeExceptionUnaryUnary(self): + self._servicer.set_code(_NON_OK_CODE) + self._servicer.set_details(_DETAILS) + self._servicer.set_exception() + + with self.assertRaises(grpc.RpcError) as exception_context: + self._unary_unary.with_call(object(), metadata=_CLIENT_METADATA) + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, + exception_context.exception.initial_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, + exception_context.exception.trailing_metadata())) + self.assertIs(_NON_OK_CODE, exception_context.exception.code()) + self.assertEqual(_DETAILS, exception_context.exception.details()) + + def testCustomCodeExceptionUnaryStream(self): + self._servicer.set_code(_NON_OK_CODE) + self._servicer.set_details(_DETAILS) + self._servicer.set_exception() + + call = self._unary_stream(_SERIALIZED_REQUEST, metadata=_CLIENT_METADATA) + received_initial_metadata = call.initial_metadata() + with self.assertRaises(grpc.RpcError): + for _ in call: + pass + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, received_initial_metadata)) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, call.trailing_metadata())) + self.assertIs(_NON_OK_CODE, call.code()) + self.assertEqual(_DETAILS, call.details()) + + def testCustomCodeExceptionStreamUnary(self): + self._servicer.set_code(_NON_OK_CODE) + self._servicer.set_details(_DETAILS) + self._servicer.set_exception() + + with self.assertRaises(grpc.RpcError) as exception_context: + self._stream_unary.with_call( + iter([_SERIALIZED_REQUEST] * test_constants.STREAM_LENGTH), + metadata=_CLIENT_METADATA) + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, + exception_context.exception.initial_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, + exception_context.exception.trailing_metadata())) + self.assertIs(_NON_OK_CODE, exception_context.exception.code()) + self.assertEqual(_DETAILS, exception_context.exception.details()) + + def testCustomCodeExceptionStreamStream(self): + self._servicer.set_code(_NON_OK_CODE) + self._servicer.set_details(_DETAILS) + self._servicer.set_exception() + + call = self._stream_stream( + iter([object()] * test_constants.STREAM_LENGTH), + metadata=_CLIENT_METADATA) + received_initial_metadata = call.initial_metadata() + with self.assertRaises(grpc.RpcError): + for _ in call: + pass + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, received_initial_metadata)) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, call.trailing_metadata())) + self.assertIs(_NON_OK_CODE, call.code()) + self.assertEqual(_DETAILS, call.details()) + + def testCustomCodeReturnNoneUnaryUnary(self): + self._servicer.set_code(_NON_OK_CODE) + self._servicer.set_details(_DETAILS) + self._servicer.set_return_none() + + with self.assertRaises(grpc.RpcError) as exception_context: + self._unary_unary.with_call(object(), metadata=_CLIENT_METADATA) + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, + exception_context.exception.initial_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, + exception_context.exception.trailing_metadata())) + self.assertIs(_NON_OK_CODE, exception_context.exception.code()) + self.assertEqual(_DETAILS, exception_context.exception.details()) + + def testCustomCodeReturnNoneStreamUnary(self): + self._servicer.set_code(_NON_OK_CODE) + self._servicer.set_details(_DETAILS) + self._servicer.set_return_none() + + with self.assertRaises(grpc.RpcError) as exception_context: + self._stream_unary.with_call( + iter([_SERIALIZED_REQUEST] * test_constants.STREAM_LENGTH), + metadata=_CLIENT_METADATA) + + self.assertTrue( + test_common.metadata_transmitted( + _CLIENT_METADATA, self._servicer.received_client_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_INITIAL_METADATA, + exception_context.exception.initial_metadata())) + self.assertTrue( + test_common.metadata_transmitted( + _SERVER_TRAILING_METADATA, + exception_context.exception.trailing_metadata())) + self.assertIs(_NON_OK_CODE, exception_context.exception.code()) + self.assertEqual(_DETAILS, exception_context.exception.details()) + + +if __name__ == '__main__': + unittest.main(verbosity=2) From edfebc909a7e4a2ada5e07d2012a227fb15d4064 Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Tue, 21 Jun 2016 16:39:21 -0700 Subject: [PATCH 0177/1039] dont generate NewClient method for C# services --- src/compiler/csharp_generator.cc | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/src/compiler/csharp_generator.cc b/src/compiler/csharp_generator.cc index fc8feaf0fcc..8c1c6f2c826 100644 --- a/src/compiler/csharp_generator.cc +++ b/src/compiler/csharp_generator.cc @@ -485,20 +485,6 @@ void GenerateBindServiceMethod(Printer* out, const ServiceDescriptor *service) { out->Print("\n"); } -void GenerateNewStubMethods(Printer* out, const ServiceDescriptor *service) { - out->Print("/// Creates a new client for $servicename$\n", - "servicename", GetServiceClassName(service)); - out->Print("public static $classname$ NewClient(Channel channel)\n", - "classname", GetClientClassName(service)); - out->Print("{\n"); - out->Indent(); - out->Print("return new $classname$(channel);\n", "classname", - GetClientClassName(service)); - out->Outdent(); - out->Print("}\n"); - out->Print("\n"); -} - void GenerateService(Printer* out, const ServiceDescriptor *service, bool generate_client, bool generate_server, bool internal_access) { @@ -524,7 +510,6 @@ void GenerateService(Printer* out, const ServiceDescriptor *service, } if (generate_client) { GenerateClientStub(out, service); - GenerateNewStubMethods(out, service); } if (generate_server) { GenerateBindServiceMethod(out, service); From 614944324246b2b4eef9fc2e989af20375c49cd5 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Wed, 22 Jun 2016 13:34:59 -0700 Subject: [PATCH 0178/1039] add code to unregister endpoints --- src/core/lib/iomgr/network_status_tracker.c | 46 ++++++++++++++++++++- src/core/lib/iomgr/network_status_tracker.h | 1 + src/core/lib/iomgr/tcp_posix.c | 1 + src/core/lib/iomgr/tcp_windows.c | 1 + 4 files changed, 47 insertions(+), 2 deletions(-) diff --git a/src/core/lib/iomgr/network_status_tracker.c b/src/core/lib/iomgr/network_status_tracker.c index 6e432368cf1..f4f8e97a252 100644 --- a/src/core/lib/iomgr/network_status_tracker.c +++ b/src/core/lib/iomgr/network_status_tracker.c @@ -33,6 +33,7 @@ #include "src/core/lib/iomgr/endpoint.h" #include +#include typedef struct endpoint_ll_node { grpc_endpoint *ep; @@ -40,9 +41,13 @@ typedef struct endpoint_ll_node { } endpoint_ll_node; static endpoint_ll_node *head = NULL; +static gpr_mu g_endpoint_mutex; +static bool g_init_done = false; -// TODO(makarandd): Install callback with OS to monitor network status. void grpc_initialize_network_status_monitor() { + g_init_done = true; + gpr_mu_init(&g_endpoint_mutex); + // TODO(makarandd): Install callback with OS to monitor network status. } void grpc_destroy_network_status_monitor() { @@ -51,9 +56,15 @@ void grpc_destroy_network_status_monitor() { gpr_free(curr); curr = next; } + gpr_mu_destroy(&g_endpoint_mutex); } void grpc_network_status_register_endpoint(grpc_endpoint *ep) { + if (!g_init_done) { + grpc_initialize_network_status_monitor(); + } + gpr_mu_lock(&g_endpoint_mutex); + gpr_log(GPR_DEBUG, "Register endpoint %p", ep); if (head == NULL) { head = (endpoint_ll_node *)gpr_malloc(sizeof(endpoint_ll_node)); head->ep = ep; @@ -64,19 +75,50 @@ void grpc_network_status_register_endpoint(grpc_endpoint *ep) { head->ep = ep; head->next = prev_head; } + gpr_mu_unlock(&g_endpoint_mutex); +} + +void grpc_network_status_unregister_endpoint(grpc_endpoint *ep) { + gpr_mu_lock(&g_endpoint_mutex); + GPR_ASSERT(head); + gpr_log(GPR_DEBUG, "Unregister endpoint %p", ep); + bool found = false; + endpoint_ll_node *prev = head; + // if we're unregistering the head, just move head to the next + if (ep == head->ep) { + head = head->next; + gpr_free(prev); + found = true; + } else { + for (endpoint_ll_node *curr = head->next; curr != NULL; curr = curr->next) { + if (ep == curr->ep) { + prev->next = curr->next; + gpr_free(curr); + found = true; + break; + } + prev = curr; + } + } + gpr_mu_unlock(&g_endpoint_mutex); + GPR_ASSERT(found); } // Walk the linked-list from head and execute shutdown. It is possible that // other threads might be in the process of shutdown as well, but that has -// no side effect. +// no side effect since endpoint shutdown is idempotent. void grpc_network_status_shutdown_all_endpoints() { + gpr_mu_lock(&g_endpoint_mutex); if (head == NULL) { + gpr_mu_unlock(&g_endpoint_mutex); return; } grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; for (endpoint_ll_node *curr = head; curr != NULL; curr = curr->next) { + gpr_log(GPR_DEBUG, "Shutting down endpoint %p", curr->ep); curr->ep->vtable->shutdown(&exec_ctx, curr->ep); } + gpr_mu_unlock(&g_endpoint_mutex); grpc_exec_ctx_finish(&exec_ctx); } diff --git a/src/core/lib/iomgr/network_status_tracker.h b/src/core/lib/iomgr/network_status_tracker.h index 4154151927f..74a1aa8135f 100644 --- a/src/core/lib/iomgr/network_status_tracker.h +++ b/src/core/lib/iomgr/network_status_tracker.h @@ -36,5 +36,6 @@ #include "src/core/lib/iomgr/endpoint.h" void grpc_network_status_register_endpoint(grpc_endpoint *ep); +void grpc_network_status_unregister_endpoint(grpc_endpoint *ep); void grpc_network_status_shutdown_all_endpoints(); #endif /* GRPC_CORE_LIB_IOMGR_NETWORK_STATUS_TRACKER_H */ diff --git a/src/core/lib/iomgr/tcp_posix.c b/src/core/lib/iomgr/tcp_posix.c index f7818353b0a..56c1eef024c 100644 --- a/src/core/lib/iomgr/tcp_posix.c +++ b/src/core/lib/iomgr/tcp_posix.c @@ -153,6 +153,7 @@ static void tcp_ref(grpc_tcp *tcp) { gpr_ref(&tcp->refcount); } #endif static void tcp_destroy(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep) { + grpc_network_status_unregister_endpoint(ep); grpc_tcp *tcp = (grpc_tcp *)ep; TCP_UNREF(exec_ctx, tcp, "destroy"); } diff --git a/src/core/lib/iomgr/tcp_windows.c b/src/core/lib/iomgr/tcp_windows.c index 8140d1d8cd2..37ab59021e3 100644 --- a/src/core/lib/iomgr/tcp_windows.c +++ b/src/core/lib/iomgr/tcp_windows.c @@ -379,6 +379,7 @@ static void win_shutdown(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep) { } static void win_destroy(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep) { + grpc_network_status_unregister_endpoint(ep); grpc_tcp *tcp = (grpc_tcp *)ep; TCP_UNREF(tcp, "destroy"); } From f7c7f330a8efa7f264a52973a0c62dc1016f0648 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Wed, 22 Jun 2016 13:37:02 -0700 Subject: [PATCH 0179/1039] Init grpc for each test --- test/core/end2end/bad_server_response_test.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/test/core/end2end/bad_server_response_test.c b/test/core/end2end/bad_server_response_test.c index ebf2caa0702..878dfaed5af 100644 --- a/test/core/end2end/bad_server_response_test.c +++ b/test/core/end2end/bad_server_response_test.c @@ -270,11 +270,12 @@ static void run_test(const char *response_payload, grpc_status_code expected_status, const char *expected_detail) { test_tcp_server test_server; - server_port = grpc_pick_unused_port_or_die(); grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; gpr_event ev; - gpr_event_init(&ev); + grpc_init(); + gpr_event_init(&ev); + server_port = grpc_pick_unused_port_or_die(); test_tcp_server_init(&test_server, on_connect, &test_server); test_tcp_server_start(&test_server, server_port); state.response_payload = response_payload; @@ -291,11 +292,12 @@ static void run_test(const char *response_payload, grpc_exec_ctx_finish(&exec_ctx); cleanup_rpc(); test_tcp_server_destroy(&test_server); + + grpc_shutdown(); } int main(int argc, char **argv) { grpc_test_init(argc, argv); - grpc_init(); /* status defined in hpack static table */ run_test(HTTP2_RESP(204), sizeof(HTTP2_RESP(204)) - 1, GRPC_STATUS_CANCELLED, @@ -334,6 +336,5 @@ int main(int argc, char **argv) { run_test(HTTP1_RESP, sizeof(HTTP1_RESP) - 1, GRPC_STATUS_UNAVAILABLE, HTTP1_DETAIL_MSG); - grpc_shutdown(); return 0; } From 76a0795b73ad2632c435fc338bb49368d1d68d9f Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 22 Jun 2016 15:09:06 -0700 Subject: [PATCH 0180/1039] Fix build errors on some configurations --- src/core/lib/iomgr/ev_epoll_linux.c | 13 +++++++------ test/core/iomgr/ev_epoll_linux_test.c | 10 ++++++++-- 2 files changed, 15 insertions(+), 8 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 6464d3ba348..88cbc586349 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -34,6 +34,7 @@ #include #include +/* This polling engine is only relevant on linux kernels supporting epoll() */ #ifdef GPR_LINUX_EPOLL #include "src/core/lib/iomgr/ev_epoll_linux.h" @@ -322,7 +323,7 @@ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, #ifdef GRPC_TSAN /* See the definition of g_epoll_sync for more context */ - gpr_atm_rel_store(&g_epoll_sync, 0); + gpr_atm_rel_store(&g_epoll_sync, (gpr_atm) 0); #endif /* defined(GRPC_TSAN) */ for (i = 0; i < fd_count; i++) { @@ -442,8 +443,8 @@ static polling_island *polling_island_create(grpc_fd *initial_fd) { pi->fds = NULL; } - gpr_atm_rel_store(&pi->ref_count, 0); - gpr_atm_rel_store(&pi->merged_to, NULL); + gpr_atm_rel_store(&pi->ref_count, (gpr_atm) 0); + gpr_atm_rel_store(&pi->merged_to, (gpr_atm) NULL); pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); @@ -472,7 +473,7 @@ static polling_island *polling_island_create(grpc_fd *initial_fd) { static void polling_island_delete(polling_island *pi) { GPR_ASSERT(pi->fd_cnt == 0); - gpr_atm_rel_store(&pi->merged_to, NULL); + gpr_atm_rel_store(&pi->merged_to, (gpr_atm) NULL); close(pi->epoll_fd); pi->epoll_fd = -1; @@ -648,7 +649,7 @@ static polling_island *polling_island_merge(polling_island *p, polling_island_add_wakeup_fd_locked(p, &polling_island_wakeup_fd); /* Add the 'merged_to' link from p --> q */ - gpr_atm_rel_store(&p->merged_to, q); + gpr_atm_rel_store(&p->merged_to, (gpr_atm) q); PI_ADD_REF(q, "pi_merge"); /* To account for the new incoming ref from p */ gpr_mu_unlock(&p->mu); @@ -810,7 +811,7 @@ static grpc_fd *fd_create(int fd, const char *name) { holding a lock to it anyway. */ gpr_mu_lock(&new_fd->mu); - gpr_atm_rel_store(&new_fd->refst, 1); + gpr_atm_rel_store(&new_fd->refst, (gpr_atm) 1); new_fd->fd = fd; new_fd->shutdown = false; new_fd->orphaned = false; diff --git a/test/core/iomgr/ev_epoll_linux_test.c b/test/core/iomgr/ev_epoll_linux_test.c index 35eb6791302..66a69f52cdb 100644 --- a/test/core/iomgr/ev_epoll_linux_test.c +++ b/test/core/iomgr/ev_epoll_linux_test.c @@ -30,7 +30,10 @@ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * */ +#include +/* This test only relevant on linux systems where epoll() is available */ +#ifdef GPR_LINUX_EPOLL #include "src/core/lib/iomgr/ev_epoll_linux.h" #include "src/core/lib/iomgr/ev_posix.h" @@ -128,8 +131,10 @@ static void test_add_fd_to_pollset() { int i; int r; - /* Create some dummy file descriptors (using pipe fds for this test. Could be - anything). Also NUM_FDS should be even for this test. */ + /* Create some dummy file descriptors. Currently using pipe file descriptors + * for this test but we could use any other type of file descriptors. Also, + * since pipe() used in this test creates two fds in each call, NUM_FDS should + * be an even number */ for (i = 0; i < NUM_FDS; i = i + 2) { r = pipe(fds + i); if (r != 0) { @@ -234,3 +239,4 @@ int main(int argc, char **argv) { grpc_iomgr_shutdown(); return 0; } +#endif /* defined(GPR_LINUX_EPOLL) */ From fcb304f493f72fa5e487b364f78126e01c550288 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 22 Jun 2016 15:25:41 -0700 Subject: [PATCH 0181/1039] Compilation failure --- test/core/iomgr/ev_epoll_linux_test.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/test/core/iomgr/ev_epoll_linux_test.c b/test/core/iomgr/ev_epoll_linux_test.c index 66a69f52cdb..2547dc98718 100644 --- a/test/core/iomgr/ev_epoll_linux_test.c +++ b/test/core/iomgr/ev_epoll_linux_test.c @@ -239,4 +239,6 @@ int main(int argc, char **argv) { grpc_iomgr_shutdown(); return 0; } -#endif /* defined(GPR_LINUX_EPOLL) */ +#else /* defined(GPR_LINUX_EPOLL) */ +int main(int argc, char **argv) { return 0; } +#endif /* !defined(GPR_LINUX_EPOLL) */ From 932d02725200028415f8420bdaf35d9484f419d4 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Wed, 22 Jun 2016 16:17:15 -0700 Subject: [PATCH 0182/1039] clang-format fixes --- src/core/lib/iomgr/network_status_tracker.c | 10 +++++----- test/core/end2end/tests/network_status_change.c | 1 - 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/src/core/lib/iomgr/network_status_tracker.c b/src/core/lib/iomgr/network_status_tracker.c index f4f8e97a252..0e34605c81d 100644 --- a/src/core/lib/iomgr/network_status_tracker.c +++ b/src/core/lib/iomgr/network_status_tracker.c @@ -31,9 +31,9 @@ * */ -#include "src/core/lib/iomgr/endpoint.h" #include #include +#include "src/core/lib/iomgr/endpoint.h" typedef struct endpoint_ll_node { grpc_endpoint *ep; @@ -51,7 +51,7 @@ void grpc_initialize_network_status_monitor() { } void grpc_destroy_network_status_monitor() { - for (endpoint_ll_node *curr = head; curr != NULL; ) { + for (endpoint_ll_node *curr = head; curr != NULL;) { endpoint_ll_node *next = curr->next; gpr_free(curr); curr = next; @@ -86,9 +86,9 @@ void grpc_network_status_unregister_endpoint(grpc_endpoint *ep) { endpoint_ll_node *prev = head; // if we're unregistering the head, just move head to the next if (ep == head->ep) { - head = head->next; - gpr_free(prev); - found = true; + head = head->next; + gpr_free(prev); + found = true; } else { for (endpoint_ll_node *curr = head->next; curr != NULL; curr = curr->next) { if (ep == curr->ep) { diff --git a/test/core/end2end/tests/network_status_change.c b/test/core/end2end/tests/network_status_change.c index bbc85007022..7e7628391de 100644 --- a/test/core/end2end/tests/network_status_change.c +++ b/test/core/end2end/tests/network_status_change.c @@ -217,7 +217,6 @@ static void test_invoke_network_status_change(grpc_end2end_test_config config) { GPR_ASSERT(0 == strcmp(call_details.host, "foo.test.google.fr")); GPR_ASSERT(was_cancelled == 0); - gpr_free(details); grpc_metadata_array_destroy(&initial_metadata_recv); grpc_metadata_array_destroy(&trailing_metadata_recv); From 5856dee7d6728bcdd0aaf8ac98abef4690252d4b Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 22 Jun 2016 16:30:22 -0700 Subject: [PATCH 0183/1039] Fix compile --- test/cpp/qps/driver.cc | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/test/cpp/qps/driver.cc b/test/cpp/qps/driver.cc index 6b31225f20a..b59c1efe127 100644 --- a/test/cpp/qps/driver.cc +++ b/test/cpp/qps/driver.cc @@ -31,6 +31,7 @@ * */ +#include #include #include #include @@ -345,9 +346,11 @@ std::unique_ptr RunScenario( // Reduce channel count so that total channels specified is held regardless // of the number of clients available size_t num_channels = - (client_channels - channels_allocated) / (num_clients - i); + (client_config.client_channels() - channels_allocated) / + (num_clients - i); channels_allocated += num_channels; - gpr_log(GPR_DEBUG, "Client %d gets %d channels", i, num_channels); + gpr_log(GPR_DEBUG, "Client %" PRIdPTR " gets %" PRIdPTR " channels", i, + num_channels); per_client_config.set_client_channels(num_channels); ClientArgs args; From 8891ef50eef2e858b2290de36691c4b535de3acc Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Wed, 22 Jun 2016 16:47:52 -0700 Subject: [PATCH 0184/1039] Initialize port to -1 --- src/core/lib/iomgr/tcp_server_windows.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/core/lib/iomgr/tcp_server_windows.c b/src/core/lib/iomgr/tcp_server_windows.c index 2a51671ec71..86982bc1836 100644 --- a/src/core/lib/iomgr/tcp_server_windows.c +++ b/src/core/lib/iomgr/tcp_server_windows.c @@ -396,7 +396,7 @@ static grpc_error *add_socket_to_server(grpc_tcp_server *s, SOCKET sock, size_t addr_len, unsigned port_index, grpc_tcp_listener **listener) { grpc_tcp_listener *sp = NULL; - int port; + int port = -1; int status; GUID guid = WSAID_ACCEPTEX; DWORD ioctl_num_bytes; From 9c578c7a734937f67b928f4511dd40448c621174 Mon Sep 17 00:00:00 2001 From: Bill Clarke Date: Wed, 22 Jun 2016 16:48:08 -0700 Subject: [PATCH 0185/1039] Const correctness for ClientContext and ServerContext getters --- include/grpc++/impl/codegen/client_context.h | 10 +++++----- include/grpc++/impl/codegen/server_context.h | 7 ++++--- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/include/grpc++/impl/codegen/client_context.h b/include/grpc++/impl/codegen/client_context.h index e23fd4eda34..a132c9a57aa 100644 --- a/include/grpc++/impl/codegen/client_context.h +++ b/include/grpc++/impl/codegen/client_context.h @@ -193,7 +193,7 @@ class ClientContext { /// /// \return A multimap of initial metadata key-value pairs from the server. const std::multimap& - GetServerInitialMetadata() { + GetServerInitialMetadata() const { GPR_CODEGEN_ASSERT(initial_metadata_received_); return recv_initial_metadata_; } @@ -205,7 +205,7 @@ class ClientContext { /// /// \return A multimap of metadata trailing key-value pairs from the server. const std::multimap& - GetServerTrailingMetadata() { + GetServerTrailingMetadata() const { // TODO(yangg) check finished return trailing_metadata_; } @@ -230,13 +230,13 @@ class ClientContext { #ifndef GRPC_CXX0X_NO_CHRONO /// Return the deadline for the client call. - std::chrono::system_clock::time_point deadline() { + std::chrono::system_clock::time_point deadline() const { return Timespec2Timepoint(deadline_); } #endif // !GRPC_CXX0X_NO_CHRONO /// Return a \a gpr_timespec representation of the client call's deadline. - gpr_timespec raw_deadline() { return deadline_; } + gpr_timespec raw_deadline() const { return deadline_; } /// Set the per call authority header (see /// https://tools.ietf.org/html/rfc7540#section-8.1.2.3). @@ -337,7 +337,7 @@ class ClientContext { const InputMessage& request, OutputMessage* result); - grpc_call* call() { return call_; } + grpc_call* call() const { return call_; } void set_call(grpc_call* call, const std::shared_ptr& channel); uint32_t initial_metadata_flags() const { diff --git a/include/grpc++/impl/codegen/server_context.h b/include/grpc++/impl/codegen/server_context.h index a1e1ed176f6..cea13a513f6 100644 --- a/include/grpc++/impl/codegen/server_context.h +++ b/include/grpc++/impl/codegen/server_context.h @@ -94,12 +94,12 @@ class ServerContext { ~ServerContext(); #ifndef GRPC_CXX0X_NO_CHRONO - std::chrono::system_clock::time_point deadline() { + std::chrono::system_clock::time_point deadline() const { return Timespec2Timepoint(deadline_); } #endif // !GRPC_CXX0X_NO_CHRONO - gpr_timespec raw_deadline() { return deadline_; } + gpr_timespec raw_deadline() const { return deadline_; } void AddInitialMetadata(const grpc::string& key, const grpc::string& value); void AddTrailingMetadata(const grpc::string& key, const grpc::string& value); @@ -122,7 +122,8 @@ class ServerContext { // was called. void TryCancel() const; - const std::multimap& client_metadata() { + const std::multimap& client_metadata() + const { return client_metadata_; } From c072c927a885e49cb13afaad11a93379a1255b9f Mon Sep 17 00:00:00 2001 From: Bill Clarke Date: Wed, 22 Jun 2016 16:48:08 -0700 Subject: [PATCH 0186/1039] Const correctness for ClientContext and ServerContext getters --- include/grpc++/impl/codegen/client_context.h | 10 +++++----- include/grpc++/impl/codegen/server_context.h | 7 ++++--- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/include/grpc++/impl/codegen/client_context.h b/include/grpc++/impl/codegen/client_context.h index e23fd4eda34..a132c9a57aa 100644 --- a/include/grpc++/impl/codegen/client_context.h +++ b/include/grpc++/impl/codegen/client_context.h @@ -193,7 +193,7 @@ class ClientContext { /// /// \return A multimap of initial metadata key-value pairs from the server. const std::multimap& - GetServerInitialMetadata() { + GetServerInitialMetadata() const { GPR_CODEGEN_ASSERT(initial_metadata_received_); return recv_initial_metadata_; } @@ -205,7 +205,7 @@ class ClientContext { /// /// \return A multimap of metadata trailing key-value pairs from the server. const std::multimap& - GetServerTrailingMetadata() { + GetServerTrailingMetadata() const { // TODO(yangg) check finished return trailing_metadata_; } @@ -230,13 +230,13 @@ class ClientContext { #ifndef GRPC_CXX0X_NO_CHRONO /// Return the deadline for the client call. - std::chrono::system_clock::time_point deadline() { + std::chrono::system_clock::time_point deadline() const { return Timespec2Timepoint(deadline_); } #endif // !GRPC_CXX0X_NO_CHRONO /// Return a \a gpr_timespec representation of the client call's deadline. - gpr_timespec raw_deadline() { return deadline_; } + gpr_timespec raw_deadline() const { return deadline_; } /// Set the per call authority header (see /// https://tools.ietf.org/html/rfc7540#section-8.1.2.3). @@ -337,7 +337,7 @@ class ClientContext { const InputMessage& request, OutputMessage* result); - grpc_call* call() { return call_; } + grpc_call* call() const { return call_; } void set_call(grpc_call* call, const std::shared_ptr& channel); uint32_t initial_metadata_flags() const { diff --git a/include/grpc++/impl/codegen/server_context.h b/include/grpc++/impl/codegen/server_context.h index a1e1ed176f6..cea13a513f6 100644 --- a/include/grpc++/impl/codegen/server_context.h +++ b/include/grpc++/impl/codegen/server_context.h @@ -94,12 +94,12 @@ class ServerContext { ~ServerContext(); #ifndef GRPC_CXX0X_NO_CHRONO - std::chrono::system_clock::time_point deadline() { + std::chrono::system_clock::time_point deadline() const { return Timespec2Timepoint(deadline_); } #endif // !GRPC_CXX0X_NO_CHRONO - gpr_timespec raw_deadline() { return deadline_; } + gpr_timespec raw_deadline() const { return deadline_; } void AddInitialMetadata(const grpc::string& key, const grpc::string& value); void AddTrailingMetadata(const grpc::string& key, const grpc::string& value); @@ -122,7 +122,8 @@ class ServerContext { // was called. void TryCancel() const; - const std::multimap& client_metadata() { + const std::multimap& client_metadata() + const { return client_metadata_; } From 17233ceaa9a0150eeb0959cd7953ac8be100b413 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Wed, 22 Jun 2016 17:16:15 -0700 Subject: [PATCH 0187/1039] Ignore unused variables in ruby compilation --- Rakefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Rakefile b/Rakefile index f208a24fd33..f44946fe937 100755 --- a/Rakefile +++ b/Rakefile @@ -77,7 +77,7 @@ task 'dlls' do grpc_config = ENV['GRPC_CONFIG'] || 'opt' verbose = ENV['V'] || '0' - env = 'CPPFLAGS="-D_WIN32_WINNT=0x600 -DUNICODE -D_UNICODE" ' + env = 'CPPFLAGS="-D_WIN32_WINNT=0x600 -DUNICODE -D_UNICODE -Wno-unused-variable -Wno-unused-result" ' env += 'LDFLAGS=-static ' env += 'SYSTEM=MINGW32 ' env += 'EMBED_ZLIB=true ' From f41ebc3968ef06439d6d1468256ac5000940ea8f Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Wed, 22 Jun 2016 12:47:14 -0700 Subject: [PATCH 0188/1039] remove occurences of NewClient factory method --- examples/csharp/helloworld/GreeterClient/Program.cs | 2 +- examples/csharp/route_guide/RouteGuideClient/Program.cs | 2 +- src/csharp/Grpc.Examples.Tests/MathClientServerTests.cs | 2 +- src/csharp/Grpc.HealthCheck.Tests/HealthClientServerTest.cs | 2 +- src/csharp/Grpc.IntegrationTesting/ClientRunners.cs | 6 +++--- .../Grpc.IntegrationTesting/GeneratedServiceBaseTest.cs | 2 +- .../Grpc.IntegrationTesting/InteropClientServerTest.cs | 4 ++-- .../Grpc.IntegrationTesting/MetadataCredentialsTest.cs | 4 ++-- src/csharp/Grpc.IntegrationTesting/SslCredentialsTest.cs | 2 +- src/csharp/Grpc.IntegrationTesting/StressTestClient.cs | 2 +- 10 files changed, 14 insertions(+), 14 deletions(-) diff --git a/examples/csharp/helloworld/GreeterClient/Program.cs b/examples/csharp/helloworld/GreeterClient/Program.cs index 4393f2f3c29..444d4735095 100644 --- a/examples/csharp/helloworld/GreeterClient/Program.cs +++ b/examples/csharp/helloworld/GreeterClient/Program.cs @@ -39,7 +39,7 @@ namespace GreeterClient { Channel channel = new Channel("127.0.0.1:50051", ChannelCredentials.Insecure); - var client = Greeter.NewClient(channel); + var client = new Greeter.GreeterClient(channel); String user = "you"; var reply = client.SayHello(new HelloRequest { Name = user }); diff --git a/examples/csharp/route_guide/RouteGuideClient/Program.cs b/examples/csharp/route_guide/RouteGuideClient/Program.cs index ee51fbe8e04..8a173dcc3b3 100644 --- a/examples/csharp/route_guide/RouteGuideClient/Program.cs +++ b/examples/csharp/route_guide/RouteGuideClient/Program.cs @@ -231,7 +231,7 @@ namespace Routeguide static void Main(string[] args) { var channel = new Channel("127.0.0.1:50052", ChannelCredentials.Insecure); - var client = new RouteGuideClient(RouteGuide.NewClient(channel)); + var client = new RouteGuideClient(new RouteGuide.RouteGuideClient(channel)); // Looking for a valid feature client.GetFeature(409146138, -746188906); diff --git a/src/csharp/Grpc.Examples.Tests/MathClientServerTests.cs b/src/csharp/Grpc.Examples.Tests/MathClientServerTests.cs index 324c209ca1d..50dacc2eaaf 100644 --- a/src/csharp/Grpc.Examples.Tests/MathClientServerTests.cs +++ b/src/csharp/Grpc.Examples.Tests/MathClientServerTests.cs @@ -62,7 +62,7 @@ namespace Math.Tests }; server.Start(); channel = new Channel(Host, server.Ports.Single().BoundPort, ChannelCredentials.Insecure); - client = Math.NewClient(channel); + client = new Math.MathClient(channel); } [TestFixtureTearDown] diff --git a/src/csharp/Grpc.HealthCheck.Tests/HealthClientServerTest.cs b/src/csharp/Grpc.HealthCheck.Tests/HealthClientServerTest.cs index 070674bae9d..25a58fb3864 100644 --- a/src/csharp/Grpc.HealthCheck.Tests/HealthClientServerTest.cs +++ b/src/csharp/Grpc.HealthCheck.Tests/HealthClientServerTest.cs @@ -65,7 +65,7 @@ namespace Grpc.HealthCheck.Tests server.Start(); channel = new Channel(Host, server.Ports.Single().BoundPort, ChannelCredentials.Insecure); - client = Grpc.Health.V1.Health.NewClient(channel); + client = new Grpc.Health.V1.Health.HealthClient(channel); } [TestFixtureTearDown] diff --git a/src/csharp/Grpc.IntegrationTesting/ClientRunners.cs b/src/csharp/Grpc.IntegrationTesting/ClientRunners.cs index 39b9ae08e64..b9c0fe6d0d8 100644 --- a/src/csharp/Grpc.IntegrationTesting/ClientRunners.cs +++ b/src/csharp/Grpc.IntegrationTesting/ClientRunners.cs @@ -211,7 +211,7 @@ namespace Grpc.IntegrationTesting bool profilerReset = false; - var client = BenchmarkService.NewClient(channel); + var client = new BenchmarkService.BenchmarkServiceClient(channel); var request = CreateSimpleRequest(); var stopwatch = new Stopwatch(); @@ -237,7 +237,7 @@ namespace Grpc.IntegrationTesting private async Task RunUnaryAsync(Channel channel, IInterarrivalTimer timer) { - var client = BenchmarkService.NewClient(channel); + var client = new BenchmarkService.BenchmarkServiceClient(channel); var request = CreateSimpleRequest(); var stopwatch = new Stopwatch(); @@ -256,7 +256,7 @@ namespace Grpc.IntegrationTesting private async Task RunStreamingPingPongAsync(Channel channel, IInterarrivalTimer timer) { - var client = BenchmarkService.NewClient(channel); + var client = new BenchmarkService.BenchmarkServiceClient(channel); var request = CreateSimpleRequest(); var stopwatch = new Stopwatch(); diff --git a/src/csharp/Grpc.IntegrationTesting/GeneratedServiceBaseTest.cs b/src/csharp/Grpc.IntegrationTesting/GeneratedServiceBaseTest.cs index 001533ce315..4216dc1d6be 100644 --- a/src/csharp/Grpc.IntegrationTesting/GeneratedServiceBaseTest.cs +++ b/src/csharp/Grpc.IntegrationTesting/GeneratedServiceBaseTest.cs @@ -61,7 +61,7 @@ namespace Grpc.IntegrationTesting }; server.Start(); channel = new Channel(Host, server.Ports.Single().BoundPort, ChannelCredentials.Insecure); - client = TestService.NewClient(channel); + client = new TestService.TestServiceClient(channel); } [TearDown] diff --git a/src/csharp/Grpc.IntegrationTesting/InteropClientServerTest.cs b/src/csharp/Grpc.IntegrationTesting/InteropClientServerTest.cs index 4ee1ff5ec8a..f907f630dab 100644 --- a/src/csharp/Grpc.IntegrationTesting/InteropClientServerTest.cs +++ b/src/csharp/Grpc.IntegrationTesting/InteropClientServerTest.cs @@ -69,7 +69,7 @@ namespace Grpc.IntegrationTesting }; int port = server.Ports.Single().BoundPort; channel = new Channel(Host, port, TestCredentials.CreateSslCredentials(), options); - client = TestService.NewClient(channel); + client = new TestService.TestServiceClient(channel); } [TestFixtureTearDown] @@ -148,7 +148,7 @@ namespace Grpc.IntegrationTesting [Test] public void UnimplementedMethod() { - InteropClient.RunUnimplementedMethod(UnimplementedService.NewClient(channel)); + InteropClient.RunUnimplementedMethod(new UnimplementedService.UnimplementedServiceClient(channel)); } } } diff --git a/src/csharp/Grpc.IntegrationTesting/MetadataCredentialsTest.cs b/src/csharp/Grpc.IntegrationTesting/MetadataCredentialsTest.cs index 9fd575f1900..eb3bb8a28bb 100644 --- a/src/csharp/Grpc.IntegrationTesting/MetadataCredentialsTest.cs +++ b/src/csharp/Grpc.IntegrationTesting/MetadataCredentialsTest.cs @@ -88,7 +88,7 @@ namespace Grpc.IntegrationTesting var channelCredentials = ChannelCredentials.Create(TestCredentials.CreateSslCredentials(), CallCredentials.FromInterceptor(asyncAuthInterceptor)); channel = new Channel(Host, server.Ports.Single().BoundPort, channelCredentials, options); - client = TestService.NewClient(channel); + client = new TestService.TestServiceClient(channel); client.UnaryCall(new SimpleRequest { }); } @@ -97,7 +97,7 @@ namespace Grpc.IntegrationTesting public void MetadataCredentials_PerCall() { channel = new Channel(Host, server.Ports.Single().BoundPort, TestCredentials.CreateSslCredentials(), options); - client = TestService.NewClient(channel); + client = new TestService.TestServiceClient(channel); var callCredentials = CallCredentials.FromInterceptor(asyncAuthInterceptor); client.UnaryCall(new SimpleRequest { }, new CallOptions(credentials: callCredentials)); diff --git a/src/csharp/Grpc.IntegrationTesting/SslCredentialsTest.cs b/src/csharp/Grpc.IntegrationTesting/SslCredentialsTest.cs index 3df45b5f708..f85e272711f 100644 --- a/src/csharp/Grpc.IntegrationTesting/SslCredentialsTest.cs +++ b/src/csharp/Grpc.IntegrationTesting/SslCredentialsTest.cs @@ -79,7 +79,7 @@ namespace Grpc.IntegrationTesting }; channel = new Channel(Host, server.Ports.Single().BoundPort, clientCredentials, options); - client = TestService.NewClient(channel); + client = new TestService.TestServiceClient(channel); } [TestFixtureTearDown] diff --git a/src/csharp/Grpc.IntegrationTesting/StressTestClient.cs b/src/csharp/Grpc.IntegrationTesting/StressTestClient.cs index 4d6ca7ece57..74ee040ae49 100644 --- a/src/csharp/Grpc.IntegrationTesting/StressTestClient.cs +++ b/src/csharp/Grpc.IntegrationTesting/StressTestClient.cs @@ -148,7 +148,7 @@ namespace Grpc.IntegrationTesting channels.Add(channel); for (int j = 0; j < options.NumStubsPerChannel; j++) { - var client = TestService.NewClient(channel); + var client = new TestService.TestServiceClient(channel); var task = Task.Factory.StartNew(() => RunBodyAsync(client).GetAwaiter().GetResult(), TaskCreationOptions.LongRunning); tasks.Add(task); From bfcd6b627ece5ddab1aa690e4bf205a6c66d8fe3 Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Wed, 22 Jun 2016 16:52:46 -0700 Subject: [PATCH 0189/1039] generate comments for C# client constructors --- src/compiler/csharp_generator.cc | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/compiler/csharp_generator.cc b/src/compiler/csharp_generator.cc index 8c1c6f2c826..f5a0876cf98 100644 --- a/src/compiler/csharp_generator.cc +++ b/src/compiler/csharp_generator.cc @@ -333,22 +333,28 @@ void GenerateClientStub(Printer* out, const ServiceDescriptor *service) { out->Indent(); // constructors + out->Print("/// Creates a new client for $servicename$\n" + "/// The channel to use to make remote calls.\n", + "servicename", GetServiceClassName(service)); out->Print("public $name$(Channel channel) : base(channel)\n", "name", GetClientClassName(service)); out->Print("{\n"); out->Print("}\n"); + out->Print("/// Creates a new client for $servicename$ that uses a custom CallInvoker.\n" + "/// The callInvoker to use to make remote calls.\n", + "servicename", GetServiceClassName(service)); out->Print("public $name$(CallInvoker callInvoker) : base(callInvoker)\n", "name", GetClientClassName(service)); out->Print("{\n"); out->Print("}\n"); - out->Print("///Protected parameterless constructor to allow creation" + out->Print("/// Protected parameterless constructor to allow creation" " of test doubles.\n"); out->Print("protected $name$() : base()\n", "name", GetClientClassName(service)); out->Print("{\n"); out->Print("}\n"); - out->Print("///Protected constructor to allow creation of configured" - " clients.\n"); + out->Print("/// Protected constructor to allow creation of configured clients.\n" + "/// The client configuration.\n"); out->Print("protected $name$(ClientBaseConfiguration configuration)" " : base(configuration)\n", "name", GetClientClassName(service)); From ed6ea4c691193483223fdb3a7ada5095b8a2494e Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Wed, 22 Jun 2016 12:38:44 -0700 Subject: [PATCH 0190/1039] regenerate protos --- src/csharp/Grpc.Examples/MathGrpc.cs | 15 +++---- src/csharp/Grpc.HealthCheck/HealthGrpc.cs | 15 +++---- .../Grpc.IntegrationTesting/MetricsGrpc.cs | 15 +++---- .../Grpc.IntegrationTesting/ServicesGrpc.cs | 30 ++++++------- .../Grpc.IntegrationTesting/TestGrpc.cs | 45 +++++++++---------- 5 files changed, 56 insertions(+), 64 deletions(-) diff --git a/src/csharp/Grpc.Examples/MathGrpc.cs b/src/csharp/Grpc.Examples/MathGrpc.cs index 4bbefcbe01a..25abc514195 100644 --- a/src/csharp/Grpc.Examples/MathGrpc.cs +++ b/src/csharp/Grpc.Examples/MathGrpc.cs @@ -128,17 +128,22 @@ namespace Math { /// Client for Math public class MathClient : ClientBase { + /// Creates a new client for Math + /// The channel to use to make remote calls. public MathClient(Channel channel) : base(channel) { } + /// Creates a new client for Math that uses a custom CallInvoker. + /// The callInvoker to use to make remote calls. public MathClient(CallInvoker callInvoker) : base(callInvoker) { } - ///Protected parameterless constructor to allow creation of test doubles. + /// Protected parameterless constructor to allow creation of test doubles. protected MathClient() : base() { } - ///Protected constructor to allow creation of configured clients. + /// Protected constructor to allow creation of configured clients. + /// The client configuration. protected MathClient(ClientBaseConfiguration configuration) : base(configuration) { } @@ -235,12 +240,6 @@ namespace Math { } } - /// Creates a new client for Math - public static MathClient NewClient(Channel channel) - { - return new MathClient(channel); - } - /// Creates service definition that can be registered with a server public static ServerServiceDefinition BindService(MathBase serviceImpl) { diff --git a/src/csharp/Grpc.HealthCheck/HealthGrpc.cs b/src/csharp/Grpc.HealthCheck/HealthGrpc.cs index d0ade7d02be..43eea0f22f5 100644 --- a/src/csharp/Grpc.HealthCheck/HealthGrpc.cs +++ b/src/csharp/Grpc.HealthCheck/HealthGrpc.cs @@ -71,17 +71,22 @@ namespace Grpc.Health.V1 { /// Client for Health public class HealthClient : ClientBase { + /// Creates a new client for Health + /// The channel to use to make remote calls. public HealthClient(Channel channel) : base(channel) { } + /// Creates a new client for Health that uses a custom CallInvoker. + /// The callInvoker to use to make remote calls. public HealthClient(CallInvoker callInvoker) : base(callInvoker) { } - ///Protected parameterless constructor to allow creation of test doubles. + /// Protected parameterless constructor to allow creation of test doubles. protected HealthClient() : base() { } - ///Protected constructor to allow creation of configured clients. + /// Protected constructor to allow creation of configured clients. + /// The client configuration. protected HealthClient(ClientBaseConfiguration configuration) : base(configuration) { } @@ -108,12 +113,6 @@ namespace Grpc.Health.V1 { } } - /// Creates a new client for Health - public static HealthClient NewClient(Channel channel) - { - return new HealthClient(channel); - } - /// Creates service definition that can be registered with a server public static ServerServiceDefinition BindService(HealthBase serviceImpl) { diff --git a/src/csharp/Grpc.IntegrationTesting/MetricsGrpc.cs b/src/csharp/Grpc.IntegrationTesting/MetricsGrpc.cs index 22bd27ec0aa..040798e3c21 100644 --- a/src/csharp/Grpc.IntegrationTesting/MetricsGrpc.cs +++ b/src/csharp/Grpc.IntegrationTesting/MetricsGrpc.cs @@ -97,17 +97,22 @@ namespace Grpc.Testing { /// Client for MetricsService public class MetricsServiceClient : ClientBase { + /// Creates a new client for MetricsService + /// The channel to use to make remote calls. public MetricsServiceClient(Channel channel) : base(channel) { } + /// Creates a new client for MetricsService that uses a custom CallInvoker. + /// The callInvoker to use to make remote calls. public MetricsServiceClient(CallInvoker callInvoker) : base(callInvoker) { } - ///Protected parameterless constructor to allow creation of test doubles. + /// Protected parameterless constructor to allow creation of test doubles. protected MetricsServiceClient() : base() { } - ///Protected constructor to allow creation of configured clients. + /// Protected constructor to allow creation of configured clients. + /// The client configuration. protected MetricsServiceClient(ClientBaseConfiguration configuration) : base(configuration) { } @@ -162,12 +167,6 @@ namespace Grpc.Testing { } } - /// Creates a new client for MetricsService - public static MetricsServiceClient NewClient(Channel channel) - { - return new MetricsServiceClient(channel); - } - /// Creates service definition that can be registered with a server public static ServerServiceDefinition BindService(MetricsServiceBase serviceImpl) { diff --git a/src/csharp/Grpc.IntegrationTesting/ServicesGrpc.cs b/src/csharp/Grpc.IntegrationTesting/ServicesGrpc.cs index 9c99296115c..e205dea93ea 100644 --- a/src/csharp/Grpc.IntegrationTesting/ServicesGrpc.cs +++ b/src/csharp/Grpc.IntegrationTesting/ServicesGrpc.cs @@ -93,17 +93,22 @@ namespace Grpc.Testing { /// Client for BenchmarkService public class BenchmarkServiceClient : ClientBase { + /// Creates a new client for BenchmarkService + /// The channel to use to make remote calls. public BenchmarkServiceClient(Channel channel) : base(channel) { } + /// Creates a new client for BenchmarkService that uses a custom CallInvoker. + /// The callInvoker to use to make remote calls. public BenchmarkServiceClient(CallInvoker callInvoker) : base(callInvoker) { } - ///Protected parameterless constructor to allow creation of test doubles. + /// Protected parameterless constructor to allow creation of test doubles. protected BenchmarkServiceClient() : base() { } - ///Protected constructor to allow creation of configured clients. + /// Protected constructor to allow creation of configured clients. + /// The client configuration. protected BenchmarkServiceClient(ClientBaseConfiguration configuration) : base(configuration) { } @@ -162,12 +167,6 @@ namespace Grpc.Testing { } } - /// Creates a new client for BenchmarkService - public static BenchmarkServiceClient NewClient(Channel channel) - { - return new BenchmarkServiceClient(channel); - } - /// Creates service definition that can be registered with a server public static ServerServiceDefinition BindService(BenchmarkServiceBase serviceImpl) { @@ -273,17 +272,22 @@ namespace Grpc.Testing { /// Client for WorkerService public class WorkerServiceClient : ClientBase { + /// Creates a new client for WorkerService + /// The channel to use to make remote calls. public WorkerServiceClient(Channel channel) : base(channel) { } + /// Creates a new client for WorkerService that uses a custom CallInvoker. + /// The callInvoker to use to make remote calls. public WorkerServiceClient(CallInvoker callInvoker) : base(callInvoker) { } - ///Protected parameterless constructor to allow creation of test doubles. + /// Protected parameterless constructor to allow creation of test doubles. protected WorkerServiceClient() : base() { } - ///Protected constructor to allow creation of configured clients. + /// Protected constructor to allow creation of configured clients. + /// The client configuration. protected WorkerServiceClient(ClientBaseConfiguration configuration) : base(configuration) { } @@ -398,12 +402,6 @@ namespace Grpc.Testing { } } - /// Creates a new client for WorkerService - public static WorkerServiceClient NewClient(Channel channel) - { - return new WorkerServiceClient(channel); - } - /// Creates service definition that can be registered with a server public static ServerServiceDefinition BindService(WorkerServiceBase serviceImpl) { diff --git a/src/csharp/Grpc.IntegrationTesting/TestGrpc.cs b/src/csharp/Grpc.IntegrationTesting/TestGrpc.cs index 6c252013f84..3e149da3e01 100644 --- a/src/csharp/Grpc.IntegrationTesting/TestGrpc.cs +++ b/src/csharp/Grpc.IntegrationTesting/TestGrpc.cs @@ -168,17 +168,22 @@ namespace Grpc.Testing { /// Client for TestService public class TestServiceClient : ClientBase { + /// Creates a new client for TestService + /// The channel to use to make remote calls. public TestServiceClient(Channel channel) : base(channel) { } + /// Creates a new client for TestService that uses a custom CallInvoker. + /// The callInvoker to use to make remote calls. public TestServiceClient(CallInvoker callInvoker) : base(callInvoker) { } - ///Protected parameterless constructor to allow creation of test doubles. + /// Protected parameterless constructor to allow creation of test doubles. protected TestServiceClient() : base() { } - ///Protected constructor to allow creation of configured clients. + /// Protected constructor to allow creation of configured clients. + /// The client configuration. protected TestServiceClient(ClientBaseConfiguration configuration) : base(configuration) { } @@ -315,12 +320,6 @@ namespace Grpc.Testing { } } - /// Creates a new client for TestService - public static TestServiceClient NewClient(Channel channel) - { - return new TestServiceClient(channel); - } - /// Creates service definition that can be registered with a server public static ServerServiceDefinition BindService(TestServiceBase serviceImpl) { @@ -373,17 +372,22 @@ namespace Grpc.Testing { /// Client for UnimplementedService public class UnimplementedServiceClient : ClientBase { + /// Creates a new client for UnimplementedService + /// The channel to use to make remote calls. public UnimplementedServiceClient(Channel channel) : base(channel) { } + /// Creates a new client for UnimplementedService that uses a custom CallInvoker. + /// The callInvoker to use to make remote calls. public UnimplementedServiceClient(CallInvoker callInvoker) : base(callInvoker) { } - ///Protected parameterless constructor to allow creation of test doubles. + /// Protected parameterless constructor to allow creation of test doubles. protected UnimplementedServiceClient() : base() { } - ///Protected constructor to allow creation of configured clients. + /// Protected constructor to allow creation of configured clients. + /// The client configuration. protected UnimplementedServiceClient(ClientBaseConfiguration configuration) : base(configuration) { } @@ -422,12 +426,6 @@ namespace Grpc.Testing { } } - /// Creates a new client for UnimplementedService - public static UnimplementedServiceClient NewClient(Channel channel) - { - return new UnimplementedServiceClient(channel); - } - /// Creates service definition that can be registered with a server public static ServerServiceDefinition BindService(UnimplementedServiceBase serviceImpl) { @@ -485,17 +483,22 @@ namespace Grpc.Testing { /// Client for ReconnectService public class ReconnectServiceClient : ClientBase { + /// Creates a new client for ReconnectService + /// The channel to use to make remote calls. public ReconnectServiceClient(Channel channel) : base(channel) { } + /// Creates a new client for ReconnectService that uses a custom CallInvoker. + /// The callInvoker to use to make remote calls. public ReconnectServiceClient(CallInvoker callInvoker) : base(callInvoker) { } - ///Protected parameterless constructor to allow creation of test doubles. + /// Protected parameterless constructor to allow creation of test doubles. protected ReconnectServiceClient() : base() { } - ///Protected constructor to allow creation of configured clients. + /// Protected constructor to allow creation of configured clients. + /// The client configuration. protected ReconnectServiceClient(ClientBaseConfiguration configuration) : base(configuration) { } @@ -538,12 +541,6 @@ namespace Grpc.Testing { } } - /// Creates a new client for ReconnectService - public static ReconnectServiceClient NewClient(Channel channel) - { - return new ReconnectServiceClient(channel); - } - /// Creates service definition that can be registered with a server public static ServerServiceDefinition BindService(ReconnectServiceBase serviceImpl) { From 0224dcc2dcda932a171776de325fa2e66c95478f Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 22 Jun 2016 18:04:00 -0700 Subject: [PATCH 0191/1039] clang format --- src/core/lib/iomgr/ev_epoll_linux.c | 12 ++++++------ src/core/lib/iomgr/ev_posix.h | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 88cbc586349..b1e9ac8a631 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -323,7 +323,7 @@ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, #ifdef GRPC_TSAN /* See the definition of g_epoll_sync for more context */ - gpr_atm_rel_store(&g_epoll_sync, (gpr_atm) 0); + gpr_atm_rel_store(&g_epoll_sync, (gpr_atm)0); #endif /* defined(GRPC_TSAN) */ for (i = 0; i < fd_count; i++) { @@ -443,8 +443,8 @@ static polling_island *polling_island_create(grpc_fd *initial_fd) { pi->fds = NULL; } - gpr_atm_rel_store(&pi->ref_count, (gpr_atm) 0); - gpr_atm_rel_store(&pi->merged_to, (gpr_atm) NULL); + gpr_atm_rel_store(&pi->ref_count, (gpr_atm)0); + gpr_atm_rel_store(&pi->merged_to, (gpr_atm)NULL); pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); @@ -473,7 +473,7 @@ static polling_island *polling_island_create(grpc_fd *initial_fd) { static void polling_island_delete(polling_island *pi) { GPR_ASSERT(pi->fd_cnt == 0); - gpr_atm_rel_store(&pi->merged_to, (gpr_atm) NULL); + gpr_atm_rel_store(&pi->merged_to, (gpr_atm)NULL); close(pi->epoll_fd); pi->epoll_fd = -1; @@ -649,7 +649,7 @@ static polling_island *polling_island_merge(polling_island *p, polling_island_add_wakeup_fd_locked(p, &polling_island_wakeup_fd); /* Add the 'merged_to' link from p --> q */ - gpr_atm_rel_store(&p->merged_to, (gpr_atm) q); + gpr_atm_rel_store(&p->merged_to, (gpr_atm)q); PI_ADD_REF(q, "pi_merge"); /* To account for the new incoming ref from p */ gpr_mu_unlock(&p->mu); @@ -811,7 +811,7 @@ static grpc_fd *fd_create(int fd, const char *name) { holding a lock to it anyway. */ gpr_mu_lock(&new_fd->mu); - gpr_atm_rel_store(&new_fd->refst, (gpr_atm) 1); + gpr_atm_rel_store(&new_fd->refst, (gpr_atm)1); new_fd->fd = fd; new_fd->shutdown = false; new_fd->orphaned = false; diff --git a/src/core/lib/iomgr/ev_posix.h b/src/core/lib/iomgr/ev_posix.h index 32260fe2ee5..579c84ef707 100644 --- a/src/core/lib/iomgr/ev_posix.h +++ b/src/core/lib/iomgr/ev_posix.h @@ -100,7 +100,7 @@ void grpc_event_engine_init(void); void grpc_event_engine_shutdown(void); /* Return the name of the poll strategy */ -const char* grpc_get_poll_strategy_name(); +const char *grpc_get_poll_strategy_name(); /* Create a wrapped file descriptor. Requires fd is a non-blocking file descriptor. From aaba1313469440a887ad70be276d8b790e91146a Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 22 Jun 2016 18:10:37 -0700 Subject: [PATCH 0192/1039] finally got rid of external deps to generate the lb response proto --- test/cpp/grpclb/grpclb_test.cc | 102 +++++++++++++++++---------------- 1 file changed, 52 insertions(+), 50 deletions(-) diff --git a/test/cpp/grpclb/grpclb_test.cc b/test/cpp/grpclb/grpclb_test.cc index 4805bf024d5..8a366a49284 100644 --- a/test/cpp/grpclb/grpclb_test.cc +++ b/test/cpp/grpclb/grpclb_test.cc @@ -31,8 +31,9 @@ * */ -#include -#include +#include +#include +#include extern "C" { #include @@ -55,8 +56,13 @@ extern "C" { #include "test/core/util/test_config.h" } +#include "src/proto/grpc/lb/v1/load_balancer.pb.h" + #define NUM_BACKENDS 4 +namespace grpc { +namespace { + typedef struct client_fixture { grpc_channel *client; char *server_uri; @@ -86,8 +92,9 @@ static gpr_timespec n_seconds_time(int n) { static void *tag(intptr_t t) { return (void *)t; } -static gpr_slice build_response_payload_slice(const char *host, int *ports, - size_t nports) { +static gpr_slice build_response_payload_slice( + const char *host, int *ports, size_t nports, + int64_t expiration_interval_secs, int32_t expiration_interval_nanos) { /* server_list { servers { @@ -97,45 +104,30 @@ static gpr_slice build_response_payload_slice(const char *host, int *ports, } ... } */ - char **hostports_vec = - static_cast(gpr_malloc(sizeof(char *) * nports)); - for (size_t i = 0; i < nports; i++) { - gpr_join_host_port(&hostports_vec[i], "127.0.0.1", ports[i]); - } - char *hostports_str = - gpr_strjoin_sep((const char **)hostports_vec, nports, " ", NULL); - gpr_log(GPR_INFO, "generating response for %s", hostports_str); - - char *output_fname; - FILE *tmpfd = gpr_tmpfile("grpclb_test", &output_fname); - fclose(tmpfd); - char *cmdline; - gpr_asprintf(&cmdline, - "./tools/codegen/core/gen_grpclb_test_response.py --lb_proto " - "src/proto/grpc/lb/v1/load_balancer.proto %s " - "--output %s --quiet", - hostports_str, output_fname); - GPR_ASSERT(system(cmdline) == 0); - FILE *f = fopen(output_fname, "rb"); - fseek(f, 0, SEEK_END); - const size_t fsize = (size_t)ftell(f); - rewind(f); - - char *serialized_response = static_cast(gpr_malloc(fsize)); - GPR_ASSERT(fread(serialized_response, fsize, 1, f) == 1); - fclose(f); - gpr_free(output_fname); - gpr_free(cmdline); + grpc::lb::v1::LoadBalanceResponse response; + auto *serverlist = response.mutable_server_list(); + if (expiration_interval_secs > 0 || expiration_interval_nanos > 0) { + auto *expiration_interval = serverlist->mutable_expiration_interval(); + if (expiration_interval_secs > 0) { + expiration_interval->set_seconds(expiration_interval_secs); + } + if (expiration_interval_nanos > 0) { + expiration_interval->set_nanos(expiration_interval_nanos); + } + } for (size_t i = 0; i < nports; i++) { - gpr_free(hostports_vec[i]); + auto *server = serverlist->add_servers(); + server->set_ip_address(host); + server->set_port(ports[i]); + server->set_load_balance_token("token" + std::to_string(ports[i])); } - gpr_free(hostports_vec); - gpr_free(hostports_str); + + gpr_log(GPR_INFO, "generating response: %s", + response.ShortDebugString().c_str()); const gpr_slice response_slice = - gpr_slice_from_copied_buffer(serialized_response, fsize); - gpr_free(serialized_response); + gpr_slice_from_copied_string(response.SerializeAsString().c_str()); return response_slice; } @@ -211,12 +203,13 @@ static void start_lb_server(server_fixture *sf, int *ports, size_t nports, if (i == 0) { // First half of the ports. response_payload_slice = - build_response_payload_slice("127.0.0.1", ports, nports / 2); + build_response_payload_slice("127.0.0.1", ports, nports / 2, -1, -1); } else { // Second half of the ports. sleep_ms(update_delay_ms); - response_payload_slice = build_response_payload_slice( - "127.0.0.1", ports + (nports / 2), (nports + 1) / 2 /* ceil */); + response_payload_slice = + build_response_payload_slice("127.0.0.1", ports + (nports / 2), + (nports + 1) / 2 /* ceil */, -1, -1); } response_payload = grpc_raw_byte_buffer_create(&response_payload_slice, 1); @@ -606,11 +599,16 @@ static void teardown_test_fixture(test_fixture *tf) { teardown_server(&tf->lb_server); } -// The LB server will send two updates: batch 1 and batch 2. Each batch contains -// two addresses, both of a valid and running backend server. Batch 1 is readily -// available and provided as soon as the client establishes the streaming call. -// Batch 2 is sent after a delay of \a lb_server_update_delay_ms milliseconds. +// The LB server will send two updates: batch 1 and batch 2. Each batch +// contains +// two addresses, both of a valid and running backend server. Batch 1 is +// readily +// available and provided as soon as the client establishes the streaming +// call. +// Batch 2 is sent after a delay of \a lb_server_update_delay_ms +// milliseconds. static test_fixture test_update(int lb_server_update_delay_ms) { + gpr_log(GPR_INFO, "start %s(%d)", __func__, lb_server_update_delay_ms); test_fixture tf; memset(&tf, 0, sizeof(tf)); setup_test_fixture(&tf, lb_server_update_delay_ms); @@ -625,14 +623,18 @@ static test_fixture test_update(int lb_server_update_delay_ms) { &tf.client); // "consumes" 2nd backend server of 2nd serverlist teardown_test_fixture(&tf); + gpr_log(GPR_INFO, "end %s(%d)", __func__, lb_server_update_delay_ms); return tf; } +} // namespace +} // namespace grpc + int main(int argc, char **argv) { grpc_test_init(argc, argv); grpc_init(); - test_fixture tf_result; + grpc::test_fixture tf_result; // Clients take a bit over one second to complete a call (the last part of the // call sleeps for 1 second while verifying the client's completion queue is // empty). Therefore: @@ -641,7 +643,7 @@ int main(int argc, char **argv) { // before the first client request is done, skipping the second server from // batch 1 altogether: the 2nd client request will go to the 1st server of // batch 2 (ie, the third one out of the four total servers). - tf_result = test_update(800); + tf_result = grpc::test_update(800); GPR_ASSERT(tf_result.lb_backends[0].num_calls_serviced == 1); GPR_ASSERT(tf_result.lb_backends[1].num_calls_serviced == 0); GPR_ASSERT(tf_result.lb_backends[2].num_calls_serviced == 2); @@ -650,17 +652,17 @@ int main(int argc, char **argv) { // If the LB server waits 1500ms, the update arrives after having picked the // 2nd server from batch 1 but before the next pick for the first server of // batch 2. All server are used. - tf_result = test_update(1500); + tf_result = grpc::test_update(1500); GPR_ASSERT(tf_result.lb_backends[0].num_calls_serviced == 1); GPR_ASSERT(tf_result.lb_backends[1].num_calls_serviced == 1); GPR_ASSERT(tf_result.lb_backends[2].num_calls_serviced == 1); GPR_ASSERT(tf_result.lb_backends[3].num_calls_serviced == 1); - // If the LB server waits >= 2000ms, the update arrives after the first two + // If the LB server waits > 2000ms, the update arrives after the first two // request are done and the third pick is performed, which returns, in RR // fashion, the 1st server of the 1st update. Therefore, the second server of // batch 1 is hit twice, whereas the first server of batch 2 is never hit. - tf_result = test_update(2000); + tf_result = grpc::test_update(2100); GPR_ASSERT(tf_result.lb_backends[0].num_calls_serviced == 2); GPR_ASSERT(tf_result.lb_backends[1].num_calls_serviced == 1); GPR_ASSERT(tf_result.lb_backends[2].num_calls_serviced == 1); From 86a00a67c40ff1f4d574d08a8b8421af3cf38e7a Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 22 Jun 2016 18:10:55 -0700 Subject: [PATCH 0193/1039] regenerated projects --- Makefile | 3 +++ src/node/tools/package.json | 1 - .../vcxproj/grpc_test_util/grpc_test_util.vcxproj | 6 ++++++ .../grpc_test_util/grpc_test_util.vcxproj.filters | 12 ++++++++++++ 4 files changed, 21 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index c21c24d422d..385330cc74c 100644 --- a/Makefile +++ b/Makefile @@ -1493,6 +1493,7 @@ buildtests_cxx: buildtests_zookeeper privatelibs_cxx \ $(BINDIR)/$(CONFIG)/golden_file_test \ $(BINDIR)/$(CONFIG)/grpc_cli \ $(BINDIR)/$(CONFIG)/grpclb_api_test \ + $(BINDIR)/$(CONFIG)/grpclb_test \ $(BINDIR)/$(CONFIG)/hybrid_end2end_test \ $(BINDIR)/$(CONFIG)/interop_client \ $(BINDIR)/$(CONFIG)/interop_server \ @@ -3101,6 +3102,7 @@ LIBGRPC_TEST_UTIL_SRC = \ src/core/lib/iomgr/endpoint.c \ src/core/lib/iomgr/endpoint_pair_posix.c \ src/core/lib/iomgr/endpoint_pair_windows.c \ + src/core/lib/iomgr/error.c \ src/core/lib/iomgr/ev_poll_and_epoll_posix.c \ src/core/lib/iomgr/ev_poll_posix.c \ src/core/lib/iomgr/ev_posix.c \ @@ -3110,6 +3112,7 @@ LIBGRPC_TEST_UTIL_SRC = \ src/core/lib/iomgr/iomgr.c \ src/core/lib/iomgr/iomgr_posix.c \ src/core/lib/iomgr/iomgr_windows.c \ + src/core/lib/iomgr/load_file.c \ src/core/lib/iomgr/polling_entity.c \ src/core/lib/iomgr/pollset_set_windows.c \ src/core/lib/iomgr/pollset_windows.c \ diff --git a/src/node/tools/package.json b/src/node/tools/package.json index c34b259e2e0..d4849c2e388 100644 --- a/src/node/tools/package.json +++ b/src/node/tools/package.json @@ -34,7 +34,6 @@ "index.js", "bin/protoc.js", "bin/protoc_plugin.js", - "bin/google/protobuf", "LICENSE" ], "main": "index.js" diff --git a/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj b/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj index fd37c566921..17e6876830e 100644 --- a/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj +++ b/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj @@ -206,6 +206,7 @@ + @@ -215,6 +216,7 @@ + @@ -334,6 +336,8 @@ + + @@ -352,6 +356,8 @@ + + diff --git a/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj.filters b/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj.filters index f04c6ea773c..bb30a294e80 100644 --- a/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj.filters +++ b/vsprojects/vcxproj/grpc_test_util/grpc_test_util.vcxproj.filters @@ -103,6 +103,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr @@ -130,6 +133,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr @@ -476,6 +482,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr @@ -503,6 +512,9 @@ src\core\lib\iomgr + + src\core\lib\iomgr + src\core\lib\iomgr From d84cec7380cbc264cb1b620ebd39c3099b6cbe12 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 22 Jun 2016 18:15:01 -0700 Subject: [PATCH 0194/1039] removed unnecessary lb response generator --- .../codegen/core/gen_grpclb_test_response.py | 97 ------------------- 1 file changed, 97 deletions(-) delete mode 100755 tools/codegen/core/gen_grpclb_test_response.py diff --git a/tools/codegen/core/gen_grpclb_test_response.py b/tools/codegen/core/gen_grpclb_test_response.py deleted file mode 100755 index 4535efc738c..00000000000 --- a/tools/codegen/core/gen_grpclb_test_response.py +++ /dev/null @@ -1,97 +0,0 @@ -#!/usr/bin/env python2.7 - -# Copyright 2015-2016, Google Inc. -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are -# met: -# -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above -# copyright notice, this list of conditions and the following disclaimer -# in the documentation and/or other materials provided with the -# distribution. -# * Neither the name of Google Inc. nor the names of its -# contributors may be used to endorse or promote products derived from -# this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -from __future__ import print_function -import argparse -import subprocess -import sys -import os.path -import sys -import tempfile -import importlib - -# Example: tools/codegen/core/gen_grpclb_test_response.py \ -# --lb_proto src/proto/grpc/lb/v1/load_balancer.proto \ -# 127.0.0.1:1234 10.0.0.1:4321 - -# 1) Compile src/proto/grpc/lb/v1/load_balancer.proto to a temp location -parser = argparse.ArgumentParser() -parser.add_argument('--lb_proto', required=True) -parser.add_argument('-e', '--expiration_interval_secs', type=int) -parser.add_argument('-o', '--output') -parser.add_argument('-q', '--quiet', default=False, action='store_true') -parser.add_argument('ipports', nargs='+') -args = parser.parse_args() - -if not os.path.isfile(args.lb_proto): - print("ERROR: file '{}' cannot be accessed (not found, no permissions, etc.)" - .format(args.lb_proto), file=sys.stderr) - sys.exit(1) - -proto_dirname = os.path.dirname(args.lb_proto) -output_dir = tempfile.mkdtemp() - -protoc_cmd = 'protoc -I{} --python_out={} {}'.format( - proto_dirname, output_dir, args.lb_proto) - -with tempfile.TemporaryFile() as stderr_tmpfile: - if subprocess.call(protoc_cmd, stderr=stderr_tmpfile, shell=True) != 0: - stderr_tmpfile.seek(0) - print("ERROR: while running '{}': {}". - format(protoc_cmd, stderr_tmpfile.read())) - sys.exit(2) - -# 2) import the output .py file. -module_name = os.path.splitext(os.path.basename(args.lb_proto))[0] + '_pb2' -sys.path.append(output_dir) -pb_module = importlib.import_module(module_name) - -# 3) Generate! -lb_response = pb_module.LoadBalanceResponse() -if args.expiration_interval_secs: - lb_response.server_list.expiration_interval.seconds = \ - args.expiration_interval_secs - -for ipport in args.ipports: - ip, port = ipport.split(':') - server = lb_response.server_list.servers.add() - server.ip_address = ip - server.port = int(port) - server.load_balance_token = b'token{}'.format(port) - -serialized_bytes = lb_response.SerializeToString() -serialized_hex = ''.join('\\x{:02x}'.format(ord(c)) for c in serialized_bytes) -if args.output: - with open(args.output, 'w') as f: - f.write(serialized_bytes) -if not args.quiet: - print(str(lb_response)) - print(serialized_hex) From f9f856bb890246d39e6f8d25a664269b8d10f8e6 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 22 Jun 2016 18:25:53 -0700 Subject: [PATCH 0195/1039] Added todos for more grpclb tests. --- test/cpp/grpclb/grpclb_test.cc | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/test/cpp/grpclb/grpclb_test.cc b/test/cpp/grpclb/grpclb_test.cc index 8a366a49284..faf8d878848 100644 --- a/test/cpp/grpclb/grpclb_test.cc +++ b/test/cpp/grpclb/grpclb_test.cc @@ -60,6 +60,11 @@ extern "C" { #define NUM_BACKENDS 4 +// TODO(dgq): Other scenarios in need of testing: +// - Send identical serverlist update +// - Test reception of invalid serverlist +// - Test pinging + namespace grpc { namespace { @@ -95,15 +100,14 @@ static void *tag(intptr_t t) { return (void *)t; } static gpr_slice build_response_payload_slice( const char *host, int *ports, size_t nports, int64_t expiration_interval_secs, int32_t expiration_interval_nanos) { - /* - server_list { - servers { - ip_address: "127.0.0.1" - port: ... - load_balance_token: "token..." - } - ... - } */ + // server_list { + // servers { + // ip_address: "127.0.0.1" + // port: ... + // load_balance_token: "token..." + // } + // ... + // } grpc::lb::v1::LoadBalanceResponse response; auto *serverlist = response.mutable_server_list(); @@ -184,7 +188,7 @@ static void start_lb_server(server_fixture *sf, int *ports, size_t nports, GPR_ASSERT(GRPC_CALL_OK == error); gpr_log(GPR_INFO, "LB Server[%s] after tag 201", sf->servers_hostport); - /* receive request for backends */ + // receive request for backends op = ops; op->op = GRPC_OP_RECV_MESSAGE; op->data.recv_message = &request_payload_recv; From cc00745e8c6e160665e753c9fda6bf2650b1823d Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Thu, 23 Jun 2016 00:18:15 -0700 Subject: [PATCH 0196/1039] brought spec up to date --- doc/compression.md | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/doc/compression.md b/doc/compression.md index 15fae4d29bf..9229ef9af10 100644 --- a/doc/compression.md +++ b/doc/compression.md @@ -42,13 +42,13 @@ and RPC settings (for example, if compression would result in small or negative gains). When a message from a client compressed with an unsupported algorithm is -processed by a server, it WILL result in an INVALID\_ARGUMENT error on the +processed by a server, it WILL result in a `UNIMPLEMENTED` error status on the server. The server will then include in its response a `grpc-accept-encoding` -header specifying the algorithms it does accept. If an INTERNAL error is -returned from the server despite having used one of the algorithms from the -`grpc-accept-encoding` header, the cause MUST NOT be related to compression. -Data sent from a server compressed with an algorithm not supported by the client -WILL result in an INTERNAL error on the client side. +header specifying the algorithms it does accept. If an `UNIMPLEMENTED` error +status is returned from the server despite having used one of the algorithms +from the `grpc-accept-encoding` header, the cause MUST NOT be related to +compression. Data sent from a server compressed with an algorithm not supported +by the client WILL result in an `INTERNAL` error status on the client side. Note that a peer MAY choose to not disclose all the encodings it supports. However, if it receives a message compressed in an undisclosed but supported @@ -99,13 +99,20 @@ compressed. channel compression configuration MUST be used. 1. When a compression method (including no compression) is specified for an outgoing message, the message MUST be compressed accordingly. -1. A message compressed in a way not supported by its endpoint MUST fail with -INVALID\_ARGUMENT status, its associated description indicating the unsupported -condition as well as the supported ones. The returned `grpc-accept-encoding` -header MUST NOT contain the compression method (encoding) used. +1. A message compressed by a client in a way not supported by its server MUST +fail with status `UNIMPLEMENTED`, its associated description indicating the +unsupported condition as well as the supported ones. The returned +`grpc-accept-encoding` header MUST NOT contain the compression method +(encoding) used. +1. A message compressed by a server in a way not supported by its client MUST +fail with status `INTERNAL`, its associated description indicating the +unsupported condition as well as the supported ones. The returned +`grpc-accept-encoding` header MUST NOT contain the compression method +(encoding) used. 1. An ill-constructed message with its [Compressed-Flag bit](PROTOCOL-HTTP2.md#compressed-flag) set but lacking a "[grpc-encoding](PROTOCOL-HTTP2.md#message-encoding)" -entry different from _identity_ in its metadata MUST fail with INTERNAL status, -its associated description indicating the invalid Compressed-Flag condition. +entry different from _identity_ in its metadata MUST fail with `INTERNAL` +status, its associated description indicating the invalid Compressed-Flag +condition. From c9b7ad0c0a541ac7559bb7081c260de195fe8315 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Thu, 23 Jun 2016 02:38:20 -0700 Subject: [PATCH 0197/1039] Updated grpcio c extension build for windows to use mingw. The default compiler for Python c extensions is VS2008 or VS2010 depending on the Python version. Since c-core moved onto the c99 standard, these compilers are not compatible. --- tools/run_tests/build_artifact_python.bat | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/tools/run_tests/build_artifact_python.bat b/tools/run_tests/build_artifact_python.bat index fea0275426f..295347e947c 100644 --- a/tools/run_tests/build_artifact_python.bat +++ b/tools/run_tests/build_artifact_python.bat @@ -37,10 +37,11 @@ set NUGET=C:\nuget\nuget.exe mkdir src\python\grpcio\grpc\_cython\_windows +@rem TODO(atash): maybe we could avoid the grpc_c.(32|64).python shim below if +@rem this used the right python build? copy /Y vsprojects\Release\grpc_dll.dll src\python\grpcio\grpc\_cython\_windows\grpc_c.32.python || goto :error copy /Y vsprojects\x64\Release\grpc_dll.dll src\python\grpcio\grpc\_cython\_windows\grpc_c.64.python || goto :error - set PATH=C:\%1;C:\%1\scripts;C:\msys64\mingw%2\bin;%PATH% pip install --upgrade six @@ -50,12 +51,6 @@ pip install -rrequirements.txt set GRPC_PYTHON_USE_CUSTOM_BDIST=0 set GRPC_PYTHON_BUILD_WITH_CYTHON=1 -@rem TODO(atash): maybe we could avoid the grpc_c.(32|64).python shim above if -@rem this used the right python build? -python setup.py bdist_wheel - -@rem Build gRPC Python tools -@rem @rem Because this is windows and *everything seems to hate Windows* we have to @rem set all of these flags ourselves because Python won't help us (see the @rem setup.py of the grpcio_tools project). @@ -70,6 +65,18 @@ set GRPC_PYTHON_CFLAGS=-fno-wrapv -frtti -std=c++11 python -c "from distutils.cygwinccompiler import get_msvcr; print(get_msvcr()[0])" > temp.txt set /p PYTHON_MSVCR= Date: Thu, 23 Jun 2016 08:22:30 -0700 Subject: [PATCH 0198/1039] Re-enable accidentally disabled traces --- src/core/lib/iomgr/tcp_posix.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/core/lib/iomgr/tcp_posix.c b/src/core/lib/iomgr/tcp_posix.c index 017f52c3677..1046b60bcc9 100644 --- a/src/core/lib/iomgr/tcp_posix.c +++ b/src/core/lib/iomgr/tcp_posix.c @@ -160,7 +160,7 @@ static void call_read_cb(grpc_exec_ctx *exec_ctx, grpc_tcp *tcp, grpc_error *error) { grpc_closure *cb = tcp->read_cb; - if (false && grpc_tcp_trace) { + if (grpc_tcp_trace) { size_t i; const char *str = grpc_error_string(error); gpr_log(GPR_DEBUG, "read: error=%s", str); @@ -394,7 +394,7 @@ static void tcp_write(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep, grpc_tcp *tcp = (grpc_tcp *)ep; grpc_error *error = GRPC_ERROR_NONE; - if (false && grpc_tcp_trace) { + if (grpc_tcp_trace) { size_t i; for (i = 0; i < buf->count; i++) { From 0badbe8b119c7375dfb0f8e6f853180009a2ffb7 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 23 Jun 2016 10:15:12 -0700 Subject: [PATCH 0199/1039] Change grpc_channel_filter init_call_elem() method to return grpc_error. --- src/core/ext/census/grpc_filter.c | 14 ++++++++------ src/core/ext/client_config/client_channel.c | 6 ++++-- src/core/ext/client_config/subchannel.c | 7 +++++-- .../load_reporting/load_reporting_filter.c | 6 ++++-- src/core/lib/channel/channel_stack.c | 19 ++++++++++++------- src/core/lib/channel/channel_stack.h | 18 ++++++++++-------- src/core/lib/channel/compress_filter.c | 7 +++++-- src/core/lib/channel/connected_channel.c | 13 +++++++------ src/core/lib/channel/http_client_filter.c | 6 ++++-- src/core/lib/channel/http_server_filter.c | 6 ++++-- .../security/transport/client_auth_filter.c | 6 ++++-- .../security/transport/server_auth_filter.c | 7 +++++-- src/core/lib/surface/call.c | 9 ++++++--- src/core/lib/surface/lame_client.c | 7 +++++-- src/core/lib/surface/server.c | 6 ++++-- test/core/channel/channel_stack_test.c | 12 ++++++++---- test/core/end2end/tests/filter_causes_close.c | 7 +++++-- 17 files changed, 100 insertions(+), 56 deletions(-) diff --git a/src/core/ext/census/grpc_filter.c b/src/core/ext/census/grpc_filter.c index 72e4e5427eb..dd2e2e01244 100644 --- a/src/core/ext/census/grpc_filter.c +++ b/src/core/ext/census/grpc_filter.c @@ -124,13 +124,14 @@ static void server_start_transport_op(grpc_exec_ctx *exec_ctx, grpc_call_next_op(exec_ctx, elem, op); } -static void client_init_call_elem(grpc_exec_ctx *exec_ctx, - grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* client_init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { call_data *d = elem->call_data; GPR_ASSERT(d != NULL); memset(d, 0, sizeof(*d)); d->start_ts = gpr_now(GPR_CLOCK_REALTIME); + return GRPC_ERROR_NONE; } static void client_destroy_call_elem(grpc_exec_ctx *exec_ctx, @@ -142,15 +143,16 @@ static void client_destroy_call_elem(grpc_exec_ctx *exec_ctx, /* TODO(hongyu): record rpc client stats and census_rpc_end_op here */ } -static void server_init_call_elem(grpc_exec_ctx *exec_ctx, - grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* server_init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { call_data *d = elem->call_data; GPR_ASSERT(d != NULL); memset(d, 0, sizeof(*d)); d->start_ts = gpr_now(GPR_CLOCK_REALTIME); /* TODO(hongyu): call census_tracing_start_op here. */ grpc_closure_init(&d->finish_recv, server_on_done_recv, elem); + return GRPC_ERROR_NONE; } static void server_destroy_call_elem(grpc_exec_ctx *exec_ctx, diff --git a/src/core/ext/client_config/client_channel.c b/src/core/ext/client_config/client_channel.c index 1d5a7d52246..b0a09dab932 100644 --- a/src/core/ext/client_config/client_channel.c +++ b/src/core/ext/client_config/client_channel.c @@ -430,10 +430,12 @@ static int cc_pick_subchannel(grpc_exec_ctx *exec_ctx, void *elemp, } /* Constructor for call_data */ -static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { grpc_subchannel_call_holder_init(elem->call_data, cc_pick_subchannel, elem, args->call_stack); + return GRPC_ERROR_NONE; } /* Destructor for call_data */ diff --git a/src/core/ext/client_config/subchannel.c b/src/core/ext/client_config/subchannel.c index 468067ea57c..2e9cf7d7d4e 100644 --- a/src/core/ext/client_config/subchannel.c +++ b/src/core/ext/client_config/subchannel.c @@ -709,8 +709,11 @@ grpc_subchannel_call *grpc_connected_subchannel_create_call( grpc_call_stack *callstk = SUBCHANNEL_CALL_TO_CALL_STACK(call); call->connection = con; GRPC_CONNECTED_SUBCHANNEL_REF(con, "subchannel_call"); - grpc_call_stack_init(exec_ctx, chanstk, 1, subchannel_call_destroy, call, - NULL, NULL, callstk); + grpc_error* error = grpc_call_stack_init(exec_ctx, chanstk, 1, + subchannel_call_destroy, call, + NULL, NULL, callstk); +// FIXME: handle error (probably requires changing this function's API) +GPR_ASSERT(error == GRPC_ERROR_NONE); grpc_call_stack_set_pollset_or_pollset_set(exec_ctx, callstk, pollent); return call; } diff --git a/src/core/ext/load_reporting/load_reporting_filter.c b/src/core/ext/load_reporting/load_reporting_filter.c index f372f88c3a6..cfe752d6e2b 100644 --- a/src/core/ext/load_reporting/load_reporting_filter.c +++ b/src/core/ext/load_reporting/load_reporting_filter.c @@ -56,10 +56,12 @@ static void invoke_lr_fn_locked(grpc_load_reporting_config *lrc, } /* Constructor for call_data */ -static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { call_data *calld = elem->call_data; memset(calld, 0, sizeof(call_data)); + return GRPC_ERROR_NONE; } /* Destructor for call_data */ diff --git a/src/core/lib/channel/channel_stack.c b/src/core/lib/channel/channel_stack.c index bbba85d80ba..0613f94c299 100644 --- a/src/core/lib/channel/channel_stack.c +++ b/src/core/lib/channel/channel_stack.c @@ -157,12 +157,13 @@ void grpc_channel_stack_destroy(grpc_exec_ctx *exec_ctx, } } -void grpc_call_stack_init(grpc_exec_ctx *exec_ctx, - grpc_channel_stack *channel_stack, int initial_refs, - grpc_iomgr_cb_func destroy, void *destroy_arg, - grpc_call_context_element *context, - const void *transport_server_data, - grpc_call_stack *call_stack) { +grpc_error* grpc_call_stack_init(grpc_exec_ctx *exec_ctx, + grpc_channel_stack *channel_stack, + int initial_refs, grpc_iomgr_cb_func destroy, + void *destroy_arg, + grpc_call_context_element *context, + const void *transport_server_data, + grpc_call_stack *call_stack) { grpc_channel_element *channel_elems = CHANNEL_ELEMS_FROM_STACK(channel_stack); grpc_call_element_args args; size_t count = channel_stack->count; @@ -185,10 +186,14 @@ void grpc_call_stack_init(grpc_exec_ctx *exec_ctx, call_elems[i].filter = channel_elems[i].filter; call_elems[i].channel_data = channel_elems[i].channel_data; call_elems[i].call_data = user_data; - call_elems[i].filter->init_call_elem(exec_ctx, &call_elems[i], &args); + grpc_error* error = call_elems[i].filter->init_call_elem( + exec_ctx, &call_elems[i], &args); + if (error != GRPC_ERROR_NONE) + return error; user_data += ROUND_UP_TO_ALIGNMENT_SIZE(call_elems[i].filter->sizeof_call_data); } + return GRPC_ERROR_NONE; } void grpc_call_stack_set_pollset_or_pollset_set(grpc_exec_ctx *exec_ctx, diff --git a/src/core/lib/channel/channel_stack.h b/src/core/lib/channel/channel_stack.h index 41dd4a0d8a1..f56c6540674 100644 --- a/src/core/lib/channel/channel_stack.h +++ b/src/core/lib/channel/channel_stack.h @@ -110,8 +110,9 @@ typedef struct { on a client; if it is non-NULL, then it points to memory owned by the transport and is on the server. Most filters want to ignore this argument. */ - void (*init_call_elem)(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args); + grpc_error* (*init_call_elem)(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args); void (*set_pollset_or_pollset_set)(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_polling_entity *pollent); @@ -209,12 +210,13 @@ void grpc_channel_stack_destroy(grpc_exec_ctx *exec_ctx, /* Initialize a call stack given a channel stack. transport_server_data is expected to be NULL on a client, or an opaque transport owned pointer on the server. */ -void grpc_call_stack_init(grpc_exec_ctx *exec_ctx, - grpc_channel_stack *channel_stack, int initial_refs, - grpc_iomgr_cb_func destroy, void *destroy_arg, - grpc_call_context_element *context, - const void *transport_server_data, - grpc_call_stack *call_stack); +grpc_error* grpc_call_stack_init(grpc_exec_ctx *exec_ctx, + grpc_channel_stack *channel_stack, + int initial_refs, grpc_iomgr_cb_func destroy, + void *destroy_arg, + grpc_call_context_element *context, + const void *transport_server_data, + grpc_call_stack *call_stack); /* Set a pollset or a pollset_set for a call stack: must occur before the first * op is started */ void grpc_call_stack_set_pollset_or_pollset_set(grpc_exec_ctx *exec_ctx, diff --git a/src/core/lib/channel/compress_filter.c b/src/core/lib/channel/compress_filter.c index 32ebe53ee64..bcc8e17a9db 100644 --- a/src/core/lib/channel/compress_filter.c +++ b/src/core/lib/channel/compress_filter.c @@ -256,8 +256,9 @@ static void compress_start_transport_stream_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { /* grab pointers to our data from the call element */ call_data *calld = elem->call_data; @@ -266,6 +267,8 @@ static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, calld->has_compression_algorithm = 0; grpc_closure_init(&calld->got_slice, got_slice, elem); grpc_closure_init(&calld->send_done, send_done, elem); + + return GRPC_ERROR_NONE; } /* Destructor for call_data */ diff --git a/src/core/lib/channel/connected_channel.c b/src/core/lib/channel/connected_channel.c index 0a7d27a1dc5..8dde688df2c 100644 --- a/src/core/lib/channel/connected_channel.c +++ b/src/core/lib/channel/connected_channel.c @@ -81,16 +81,17 @@ static void con_start_transport_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { call_data *calld = elem->call_data; channel_data *chand = elem->channel_data; - int r; - - r = grpc_transport_init_stream( + int r = grpc_transport_init_stream( exec_ctx, chand->transport, TRANSPORT_STREAM_FROM_CALL_DATA(calld), &args->call_stack->refcount, args->server_transport_data); - GPR_ASSERT(r == 0); + return r == 0 + ? GRPC_ERROR_NONE + : GRPC_ERROR_CREATE("transport initialization failed"); } static void set_pollset_or_pollset_set(grpc_exec_ctx *exec_ctx, diff --git a/src/core/lib/channel/http_client_filter.c b/src/core/lib/channel/http_client_filter.c index ab6c6c9ef00..30f4ed1f6fc 100644 --- a/src/core/lib/channel/http_client_filter.c +++ b/src/core/lib/channel/http_client_filter.c @@ -169,11 +169,13 @@ static void hc_start_transport_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { call_data *calld = elem->call_data; calld->on_done_recv = NULL; grpc_closure_init(&calld->hc_on_recv, hc_on_recv, elem); + return GRPC_ERROR_NONE; } /* Destructor for call_data */ diff --git a/src/core/lib/channel/http_server_filter.c b/src/core/lib/channel/http_server_filter.c index d0beebd817f..960d1ce45e5 100644 --- a/src/core/lib/channel/http_server_filter.c +++ b/src/core/lib/channel/http_server_filter.c @@ -224,13 +224,15 @@ static void hs_start_transport_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { /* grab pointers to our data from the call element */ call_data *calld = elem->call_data; /* initialize members */ memset(calld, 0, sizeof(*calld)); grpc_closure_init(&calld->hs_on_recv, hs_on_recv, elem); + return GRPC_ERROR_NONE; } /* Destructor for call_data */ diff --git a/src/core/lib/security/transport/client_auth_filter.c b/src/core/lib/security/transport/client_auth_filter.c index 76be2acd728..6179b9e18cf 100644 --- a/src/core/lib/security/transport/client_auth_filter.c +++ b/src/core/lib/security/transport/client_auth_filter.c @@ -264,10 +264,12 @@ static void auth_start_transport_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { call_data *calld = elem->call_data; memset(calld, 0, sizeof(*calld)); + return GRPC_ERROR_NONE; } static void set_pollset_or_pollset_set(grpc_exec_ctx *exec_ctx, diff --git a/src/core/lib/security/transport/server_auth_filter.c b/src/core/lib/security/transport/server_auth_filter.c index 12e789bde92..379247131b6 100644 --- a/src/core/lib/security/transport/server_auth_filter.c +++ b/src/core/lib/security/transport/server_auth_filter.c @@ -199,8 +199,9 @@ static void auth_start_transport_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { /* grab pointers to our data from the call element */ call_data *calld = elem->call_data; channel_data *chand = elem->channel_data; @@ -222,6 +223,8 @@ static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, args->context[GRPC_CONTEXT_SECURITY].value = server_ctx; args->context[GRPC_CONTEXT_SECURITY].destroy = grpc_server_security_context_destroy; + + return GRPC_ERROR_NONE; } /* Destructor for call_data */ diff --git a/src/core/lib/surface/call.c b/src/core/lib/surface/call.c index 04291b0ee0a..d64ca64a15f 100644 --- a/src/core/lib/surface/call.c +++ b/src/core/lib/surface/call.c @@ -262,9 +262,12 @@ grpc_call *grpc_call_create( call->send_deadline = send_deadline; GRPC_CHANNEL_INTERNAL_REF(channel, "call"); /* initial refcount dropped by grpc_call_destroy */ - grpc_call_stack_init(&exec_ctx, channel_stack, 1, destroy_call, call, - call->context, server_transport_data, - CALL_STACK_FROM_CALL(call)); + grpc_error* error = grpc_call_stack_init(&exec_ctx, channel_stack, 1, + destroy_call, call, call->context, + server_transport_data, + CALL_STACK_FROM_CALL(call)); +// FIXME: handle error (probably requires changing this function's API) +GPR_ASSERT(error == GRPC_ERROR_NONE); if (cq != NULL) { GPR_ASSERT( pollset_set_alternative == NULL && diff --git a/src/core/lib/surface/lame_client.c b/src/core/lib/surface/lame_client.c index 5ea4cba5d1a..7d9168f2e0b 100644 --- a/src/core/lib/surface/lame_client.c +++ b/src/core/lib/surface/lame_client.c @@ -107,8 +107,11 @@ static void lame_start_transport_op(grpc_exec_ctx *exec_ctx, GRPC_ERROR_UNREF(op->disconnect_with_error); } -static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) {} +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { + return GRPC_ERROR_NONE; +} static void destroy_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, const grpc_call_stats *stats, diff --git a/src/core/lib/surface/server.c b/src/core/lib/surface/server.c index def6e5068b0..0a0c224b4d6 100644 --- a/src/core/lib/surface/server.c +++ b/src/core/lib/surface/server.c @@ -848,8 +848,9 @@ static void channel_connectivity_changed(grpc_exec_ctx *exec_ctx, void *cd, } } -static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { call_data *calld = elem->call_data; channel_data *chand = elem->channel_data; memset(calld, 0, sizeof(call_data)); @@ -861,6 +862,7 @@ static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, server_on_recv_initial_metadata, elem); server_ref(chand->server); + return GRPC_ERROR_NONE; } static void destroy_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, diff --git a/test/core/channel/channel_stack_test.c b/test/core/channel/channel_stack_test.c index f9561bed707..9c6a47eb520 100644 --- a/test/core/channel/channel_stack_test.c +++ b/test/core/channel/channel_stack_test.c @@ -53,10 +53,12 @@ static void channel_init_func(grpc_exec_ctx *exec_ctx, *(int *)(elem->channel_data) = 0; } -static void call_init_func(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) { +static grpc_error* call_init_func(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { ++*(int *)(elem->channel_data); *(int *)(elem->call_data) = 0; + return GRPC_ERROR_NONE; } static void channel_destroy_func(grpc_exec_ctx *exec_ctx, @@ -132,8 +134,10 @@ static void test_create_channel_stack(void) { GPR_ASSERT(*channel_data == 0); call_stack = gpr_malloc(channel_stack->call_stack_size); - grpc_call_stack_init(&exec_ctx, channel_stack, 1, free_call, call_stack, NULL, - NULL, call_stack); + grpc_error* error = grpc_call_stack_init(&exec_ctx, channel_stack, 1, + free_call, call_stack, NULL, NULL, + call_stack); + GPR_ASSERT(error == GRPC_ERROR_NONE); GPR_ASSERT(call_stack->count == 1); call_elem = grpc_call_stack_element(call_stack, 0); GPR_ASSERT(call_elem->filter == channel_elem->filter); diff --git a/test/core/end2end/tests/filter_causes_close.c b/test/core/end2end/tests/filter_causes_close.c index 526c05ca3e8..3ff0abed637 100644 --- a/test/core/end2end/tests/filter_causes_close.c +++ b/test/core/end2end/tests/filter_causes_close.c @@ -233,8 +233,11 @@ static void start_transport_stream_op(grpc_exec_ctx *exec_ctx, grpc_call_next_op(exec_ctx, elem, op); } -static void init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, - grpc_call_element_args *args) {} +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { + return GRPC_ERROR_NONE; +} static void destroy_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, const grpc_call_stats *stats, From 75f6828636a432ff32435ac06a6121b8396c67fd Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Thu, 23 Jun 2016 10:16:16 -0700 Subject: [PATCH 0200/1039] Fixed typo --- doc/compression.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/compression.md b/doc/compression.md index 9229ef9af10..de245d90fee 100644 --- a/doc/compression.md +++ b/doc/compression.md @@ -42,7 +42,7 @@ and RPC settings (for example, if compression would result in small or negative gains). When a message from a client compressed with an unsupported algorithm is -processed by a server, it WILL result in a `UNIMPLEMENTED` error status on the +processed by a server, it WILL result in an `UNIMPLEMENTED` error status on the server. The server will then include in its response a `grpc-accept-encoding` header specifying the algorithms it does accept. If an `UNIMPLEMENTED` error status is returned from the server despite having used one of the algorithms From dde4fc66fb55291434a017c667d6a961332801e6 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Thu, 23 Jun 2016 10:35:19 -0700 Subject: [PATCH 0201/1039] changed service to _service, and added a TODO TODO for removing boilerplate code --- .../objective-c/route_guide/ViewControllers.m | 27 ++++++++++--------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/examples/objective-c/route_guide/ViewControllers.m b/examples/objective-c/route_guide/ViewControllers.m index 5fd411e3ba6..a5c03e9cfca 100644 --- a/examples/objective-c/route_guide/ViewControllers.m +++ b/examples/objective-c/route_guide/ViewControllers.m @@ -81,7 +81,7 @@ static NSString * const kHostAddress = @"localhost:50051"; * not to have a feature. */ @interface GetFeatureViewController : UIViewController { - RTGRouteGuide *service; + RTGRouteGuide *_service; } @property (weak, nonatomic) IBOutlet UILabel *outputLabel; @end @@ -90,6 +90,7 @@ static NSString * const kHostAddress = @"localhost:50051"; - (void)execRequest { void (^handler)(RTGFeature *response, NSError *error) = ^(RTGFeature *response, NSError *error) { + // TODO(makdharma): Remove boilerplate by consolidating into one log function. if (response.name.length) { NSString *str =[NSString stringWithFormat:@"%@\nFound feature called %@ at %@.", self.outputLabel.text, response.location, response.name]; self.outputLabel.text = str; @@ -109,8 +110,8 @@ static NSString * const kHostAddress = @"localhost:50051"; point.latitude = 409146138; point.longitude = -746188906; - [service getFeatureWithRequest:point handler:handler]; - [service getFeatureWithRequest:[RTGPoint message] handler:handler]; + [_service getFeatureWithRequest:point handler:handler]; + [_service getFeatureWithRequest:[RTGPoint message] handler:handler]; } - (void)viewDidLoad { @@ -119,7 +120,7 @@ static NSString * const kHostAddress = @"localhost:50051"; // This only needs to be done once per host, before creating service objects for that host. [GRPCCall useInsecureConnectionsForHost:kHostAddress]; - service = [[RTGRouteGuide alloc] initWithHost:kHostAddress]; + _service = [[RTGRouteGuide alloc] initWithHost:kHostAddress]; } - (void)viewDidAppear:(BOOL)animated { @@ -139,7 +140,7 @@ static NSString * const kHostAddress = @"localhost:50051"; * the pre-generated database. Prints each response as it comes in. */ @interface ListFeaturesViewController : UIViewController { - RTGRouteGuide *service; + RTGRouteGuide *_service; } @property (weak, nonatomic) IBOutlet UILabel *outputLabel; @@ -155,7 +156,7 @@ static NSString * const kHostAddress = @"localhost:50051"; rectangle.hi.longitude = -745E6; NSLog(@"Looking for features between %@ and %@", rectangle.lo, rectangle.hi); - [service listFeaturesWithRequest:rectangle + [_service listFeaturesWithRequest:rectangle eventHandler:^(BOOL done, RTGFeature *response, NSError *error) { if (response) { NSString *str =[NSString stringWithFormat:@"%@\nFound feature at %@ called %@.", self.outputLabel.text, response.location, response.name]; @@ -172,7 +173,7 @@ static NSString * const kHostAddress = @"localhost:50051"; - (void)viewDidLoad { [super viewDidLoad]; - service = [[RTGRouteGuide alloc] initWithHost:kHostAddress]; + _service = [[RTGRouteGuide alloc] initWithHost:kHostAddress]; } - (void)viewDidAppear:(BOOL)animated { @@ -193,7 +194,7 @@ static NSString * const kHostAddress = @"localhost:50051"; * server. */ @interface RecordRouteViewController : UIViewController { - RTGRouteGuide *service; + RTGRouteGuide *_service; } @property (weak, nonatomic) IBOutlet UILabel *outputLabel; @@ -217,7 +218,7 @@ static NSString * const kHostAddress = @"localhost:50051"; return location; }]; - [service recordRouteWithRequestsWriter:locations + [_service recordRouteWithRequestsWriter:locations handler:^(RTGRouteSummary *response, NSError *error) { if (response) { NSString *str =[NSString stringWithFormat: @@ -241,7 +242,7 @@ static NSString * const kHostAddress = @"localhost:50051"; - (void)viewDidLoad { [super viewDidLoad]; - service = [[RTGRouteGuide alloc] initWithHost:kHostAddress]; + _service = [[RTGRouteGuide alloc] initWithHost:kHostAddress]; } - (void)viewDidAppear:(BOOL)animated { @@ -261,7 +262,7 @@ static NSString * const kHostAddress = @"localhost:50051"; * the server. */ @interface RouteChatViewController : UIViewController { - RTGRouteGuide *service; + RTGRouteGuide *_service; } @property (weak, nonatomic) IBOutlet UILabel *outputLabel; @@ -279,7 +280,7 @@ static NSString * const kHostAddress = @"localhost:50051"; return note; }]; - [service routeChatWithRequestsWriter:notesWriter + [_service routeChatWithRequestsWriter:notesWriter eventHandler:^(BOOL done, RTGRouteNote *note, NSError *error) { if (note) { NSString *str =[NSString stringWithFormat:@"%@\nGot message %@ at %@", @@ -300,7 +301,7 @@ static NSString * const kHostAddress = @"localhost:50051"; - (void)viewDidLoad { [super viewDidLoad]; - service = [[RTGRouteGuide alloc] initWithHost:kHostAddress]; + _service = [[RTGRouteGuide alloc] initWithHost:kHostAddress]; } - (void)viewDidAppear:(BOOL)animated { From d139da8b9a9c482a83b23c926690f01b49025640 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Thu, 23 Jun 2016 10:37:21 -0700 Subject: [PATCH 0202/1039] Remove unnecessary changes --- tools/run_tests/run_tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index 2a404df0d41..532f7187408 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -425,7 +425,7 @@ class PythonLanguage(object): return [] def build_steps(self): - return [['tools/run_tests/build_python.sh', tox_env] + return [['tools/run_tests/build_python.sh', tox_env] for tox_env in self._tox_envs] def post_tests_steps(self): From 9f97cca37b21fbac12d840fa2337f1fba251e77c Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 23 Jun 2016 10:47:05 -0700 Subject: [PATCH 0203/1039] Fix error handling in grpc_call_create(). --- src/core/lib/surface/call.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/src/core/lib/surface/call.c b/src/core/lib/surface/call.c index d64ca64a15f..2e89393815f 100644 --- a/src/core/lib/surface/call.c +++ b/src/core/lib/surface/call.c @@ -266,8 +266,18 @@ grpc_call *grpc_call_create( destroy_call, call, call->context, server_transport_data, CALL_STACK_FROM_CALL(call)); -// FIXME: handle error (probably requires changing this function's API) -GPR_ASSERT(error == GRPC_ERROR_NONE); + if (error != GRPC_ERROR_NONE) { + intptr_t status; + if (!grpc_error_get_int(error, GRPC_ERROR_INT_GRPC_STATUS, &status)) + status = GRPC_STATUS_UNKNOWN; + const char* error_string = grpc_error_string(error); + received_status* status_struct = &call->status[STATUS_FROM_CORE]; + status_struct->is_set = true; + status_struct->code = status; + status_struct->details = grpc_mdstr_from_string(error_string); + grpc_error_free_string(error_string); + grpc_error_unref(error); + } if (cq != NULL) { GPR_ASSERT( pollset_set_alternative == NULL && From 09e669878f57e4e4a56bb920d4e3e48615fde55d Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 23 Jun 2016 11:14:18 -0700 Subject: [PATCH 0204/1039] Propagate error up through grpc_connected_subchannel_create_call(). --- src/core/ext/client_config/subchannel.c | 23 ++++++----- src/core/ext/client_config/subchannel.h | 4 +- .../client_config/subchannel_call_holder.c | 38 ++++++++++++++----- 3 files changed, 44 insertions(+), 21 deletions(-) diff --git a/src/core/ext/client_config/subchannel.c b/src/core/ext/client_config/subchannel.c index 2e9cf7d7d4e..80cf154f96a 100644 --- a/src/core/ext/client_config/subchannel.c +++ b/src/core/ext/client_config/subchannel.c @@ -700,22 +700,25 @@ grpc_connected_subchannel *grpc_subchannel_get_connected_subchannel( return GET_CONNECTED_SUBCHANNEL(c, acq); } -grpc_subchannel_call *grpc_connected_subchannel_create_call( +grpc_error *grpc_connected_subchannel_create_call( grpc_exec_ctx *exec_ctx, grpc_connected_subchannel *con, - grpc_polling_entity *pollent) { + grpc_polling_entity *pollent, grpc_subchannel_call **call) { grpc_channel_stack *chanstk = CHANNEL_STACK_FROM_CONNECTION(con); - grpc_subchannel_call *call = - gpr_malloc(sizeof(grpc_subchannel_call) + chanstk->call_stack_size); - grpc_call_stack *callstk = SUBCHANNEL_CALL_TO_CALL_STACK(call); - call->connection = con; + *call = gpr_malloc(sizeof(grpc_subchannel_call) + chanstk->call_stack_size); + grpc_call_stack *callstk = SUBCHANNEL_CALL_TO_CALL_STACK(*call); + (*call)->connection = con; GRPC_CONNECTED_SUBCHANNEL_REF(con, "subchannel_call"); grpc_error* error = grpc_call_stack_init(exec_ctx, chanstk, 1, - subchannel_call_destroy, call, + subchannel_call_destroy, *call, NULL, NULL, callstk); -// FIXME: handle error (probably requires changing this function's API) -GPR_ASSERT(error == GRPC_ERROR_NONE); + if (error != GRPC_ERROR_NONE) { + const char* error_string = grpc_error_string(error); + gpr_log(GPR_ERROR, "error: %s", error_string); + grpc_error_free_string(error_string); + return error; + } grpc_call_stack_set_pollset_or_pollset_set(exec_ctx, callstk, pollent); - return call; + return GRPC_ERROR_NONE; } grpc_call_stack *grpc_subchannel_call_get_call_stack( diff --git a/src/core/ext/client_config/subchannel.h b/src/core/ext/client_config/subchannel.h index b6d39f5dc52..21454cb7a4d 100644 --- a/src/core/ext/client_config/subchannel.h +++ b/src/core/ext/client_config/subchannel.h @@ -108,9 +108,9 @@ void grpc_subchannel_call_unref(grpc_exec_ctx *exec_ctx, GRPC_SUBCHANNEL_REF_EXTRA_ARGS); /** construct a subchannel call */ -grpc_subchannel_call *grpc_connected_subchannel_create_call( +grpc_error *grpc_connected_subchannel_create_call( grpc_exec_ctx *exec_ctx, grpc_connected_subchannel *connected_subchannel, - grpc_polling_entity *pollent); + grpc_polling_entity *pollent, grpc_subchannel_call** subchannel_call); /** process a transport level op */ void grpc_connected_subchannel_process_transport_op( diff --git a/src/core/ext/client_config/subchannel_call_holder.c b/src/core/ext/client_config/subchannel_call_holder.c index e31800edd98..25bd9798f0a 100644 --- a/src/core/ext/client_config/subchannel_call_holder.c +++ b/src/core/ext/client_config/subchannel_call_holder.c @@ -84,6 +84,11 @@ void grpc_subchannel_call_holder_destroy(grpc_exec_ctx *exec_ctx, gpr_free(holder->waiting_ops); } +// The logic here is fairly complicated, due to (a) the fact that we +// need to handle the case where we receive the send op before the +// initial metadata op, and (b) the need for efficiency, especially in +// the streaming case. +// TODO(ctiller): Explain this more thoroughly. void grpc_subchannel_call_holder_perform_op(grpc_exec_ctx *exec_ctx, grpc_subchannel_call_holder *holder, grpc_transport_stream_op *op) { @@ -121,7 +126,8 @@ retry: } /* if this is a cancellation, then we can raise our cancelled flag */ if (op->cancel_with_status != GRPC_STATUS_OK) { - if (!gpr_atm_rel_cas(&holder->subchannel_call, 0, 1)) { + if (!gpr_atm_rel_cas(&holder->subchannel_call, 0, + (gpr_atm)(uintptr_t)CANCELLED_CALL)) { goto retry; } else { switch (holder->creation_phase) { @@ -161,10 +167,17 @@ retry: /* if we've got a subchannel, then let's ask it to create a call */ if (holder->creation_phase == GRPC_SUBCHANNEL_CALL_HOLDER_NOT_CREATING && holder->connected_subchannel != NULL) { - gpr_atm_rel_store( - &holder->subchannel_call, - (gpr_atm)(uintptr_t)grpc_connected_subchannel_create_call( - exec_ctx, holder->connected_subchannel, holder->pollent)); + grpc_subchannel_call* subchannel_call = NULL; + grpc_error* error = grpc_connected_subchannel_create_call( + exec_ctx, holder->connected_subchannel, holder->pollent, + &subchannel_call); + if (error != GRPC_ERROR_NONE) { + subchannel_call = CANCELLED_CALL; + fail_locked(exec_ctx, holder, error); + grpc_transport_stream_op_finish_with_failure(exec_ctx, op, error); + } + gpr_atm_rel_store(&holder->subchannel_call, + (gpr_atm)(uintptr_t)subchannel_call); retry_waiting_locked(exec_ctx, holder); goto retry; } @@ -192,10 +205,17 @@ static void subchannel_ready(grpc_exec_ctx *exec_ctx, void *arg, GRPC_ERROR_CREATE_REFERENCING( "Cancelled before creating subchannel", &error, 1)); } else { - gpr_atm_rel_store( - &holder->subchannel_call, - (gpr_atm)(uintptr_t)grpc_connected_subchannel_create_call( - exec_ctx, holder->connected_subchannel, holder->pollent)); + grpc_subchannel_call* subchannel_call = NULL; + grpc_error* new_error = grpc_connected_subchannel_create_call( + exec_ctx, holder->connected_subchannel, holder->pollent, + &subchannel_call); + if (new_error != GRPC_ERROR_NONE) { + grpc_error_add_child(new_error, error); + subchannel_call = CANCELLED_CALL; + fail_locked(exec_ctx, holder, new_error); + } + gpr_atm_rel_store(&holder->subchannel_call, + (gpr_atm)(uintptr_t)subchannel_call); retry_waiting_locked(exec_ctx, holder); } gpr_mu_unlock(&holder->mu); From 820c1f3fda95853e791ffe396b15b32d1bda356b Mon Sep 17 00:00:00 2001 From: Robbie Shade Date: Thu, 23 Jun 2016 14:31:28 -0400 Subject: [PATCH 0205/1039] ASSERT vector size before directly accessing first element --- test/cpp/end2end/end2end_test.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/cpp/end2end/end2end_test.cc b/test/cpp/end2end/end2end_test.cc index 8de9d339f63..ea04ab97db2 100644 --- a/test/cpp/end2end/end2end_test.cc +++ b/test/cpp/end2end/end2end_test.cc @@ -1407,7 +1407,7 @@ TEST_P(SecureEnd2endTest, ClientAuthContext) { std::shared_ptr auth_ctx = context.auth_context(); std::vector tst = auth_ctx->FindPropertyValues("transport_security_type"); - EXPECT_EQ(1u, tst.size()); + ASSERT_EQ(1u, tst.size()); EXPECT_EQ(GetParam().credentials_type, ToString(tst[0])); if (GetParam().credentials_type == kTlsCredentialsType) { EXPECT_EQ("x509_subject_alternative_name", From 83a6a828b047cc24922a6d143e30f4ab6eb63239 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Thu, 23 Jun 2016 11:33:13 -0700 Subject: [PATCH 0206/1039] Fixed error messages for C++ interop client. --- test/cpp/interop/interop_client.cc | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/test/cpp/interop/interop_client.cc b/test/cpp/interop/interop_client.cc index 89f841dbe96..ec0729d9c5d 100644 --- a/test/cpp/interop/interop_client.cc +++ b/test/cpp/interop/interop_client.cc @@ -451,7 +451,7 @@ bool InteropClient::DoResponseStreaming() { // most likely due to connection failure. gpr_log(GPR_ERROR, "DoResponseStreaming(): Read fewer streams (%d) than " - "response_stream_sizes.size() (%" PRIuPTR ")", + "response_stream_sizes.size() (%zu)", i, response_stream_sizes.size()); return TransientFailureOrAbort(); } @@ -576,11 +576,10 @@ bool InteropClient::DoServerCompressedStreaming() { if (k < sizes.size()) { // stream->Read() failed before reading all the expected messages. This // is most likely due to a connection failure. - gpr_log(GPR_ERROR, "%s(): Responses read (k=%" PRIuPTR - ") is " - "less than the expected messages (i.e " - "response_stream_sizes.size() (%" PRIuPTR ")).", - __func__, k, response_stream_sizes.size()); + gpr_log(GPR_ERROR, + "%s(): Responses read (k=%zu) is less than the expected number of " + "messages (%zu).", + __func__, k, sizes.size()); return TransientFailureOrAbort(); } @@ -665,7 +664,7 @@ bool InteropClient::DoHalfDuplex() { // most likely due to a connection failure gpr_log(GPR_ERROR, "DoHalfDuplex(): Responses read (i=%d) are less than the expected " - "number of messages response_stream_sizes.size() (%" PRIuPTR ")", + "number of messages response_stream_sizes.size() (%zu)", i, response_stream_sizes.size()); return TransientFailureOrAbort(); } From 674f55a9ac7536b001abcac73db89e4260e935e2 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Thu, 23 Jun 2016 11:40:53 -0700 Subject: [PATCH 0207/1039] Fixed missing variable propagation --- src/ruby/lib/grpc/generic/active_call.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/ruby/lib/grpc/generic/active_call.rb b/src/ruby/lib/grpc/generic/active_call.rb index 18126c44323..a3ac0d48a3b 100644 --- a/src/ruby/lib/grpc/generic/active_call.rb +++ b/src/ruby/lib/grpc/generic/active_call.rb @@ -412,7 +412,8 @@ module GRPC # # @param gen_each_reply [Proc] generates the BiDi stream replies def run_server_bidi(gen_each_reply) - bd = BidiCall.new(@call, @marshal, @unmarshal) + bd = BidiCall.new(@call, @marshal, @unmarshal, + metadata_received: @metadata_received) bd.run_on_server(gen_each_reply) end From a4b34c290d5f91866a3eff2f9793c00eb12f16f0 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Wed, 22 Jun 2016 15:31:15 -0700 Subject: [PATCH 0208/1039] Changed %lld to use mingw supported PRId64 --- src/core/ext/client_config/channel_connectivity.c | 6 +++--- src/core/lib/profiling/basic_timers.c | 4 ++-- .../lib/security/credentials/jwt/jwt_credentials.c | 4 ++-- src/core/lib/surface/channel.c | 10 +++++----- src/core/lib/surface/completion_queue.c | 8 ++++---- src/core/lib/transport/transport_op_string.c | 4 ++-- 6 files changed, 18 insertions(+), 18 deletions(-) diff --git a/src/core/ext/client_config/channel_connectivity.c b/src/core/ext/client_config/channel_connectivity.c index 20c01a9a7cf..2fd8e0e1232 100644 --- a/src/core/ext/client_config/channel_connectivity.c +++ b/src/core/ext/client_config/channel_connectivity.c @@ -189,10 +189,10 @@ void grpc_channel_watch_connectivity_state( GRPC_API_TRACE( "grpc_channel_watch_connectivity_state(" "channel=%p, last_observed_state=%d, " - "deadline=gpr_timespec { tv_sec: %lld, tv_nsec: %d, clock_type: %d }, " + "deadline=gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " "cq=%p, tag=%p)", - 7, (channel, (int)last_observed_state, (long long)deadline.tv_sec, - (int)deadline.tv_nsec, (int)deadline.clock_type, cq, tag)); + 7, (channel, (int)last_observed_state, deadline.tv_sec, + deadline.tv_nsec, (int)deadline.clock_type, cq, tag)); grpc_cq_begin_op(cq, tag); diff --git a/src/core/lib/profiling/basic_timers.c b/src/core/lib/profiling/basic_timers.c index 50082cd7ee4..ee464f310f4 100644 --- a/src/core/lib/profiling/basic_timers.c +++ b/src/core/lib/profiling/basic_timers.c @@ -141,9 +141,9 @@ static void write_log(gpr_timer_log *log) { entry->tm = gpr_time_0(entry->tm.clock_type); } fprintf(output_file, - "{\"t\": %lld.%09d, \"thd\": \"%d\", \"type\": \"%c\", \"tag\": " + "{\"t\": %"PRId64".%09d, \"thd\": \"%d\", \"type\": \"%c\", \"tag\": " "\"%s\", \"file\": \"%s\", \"line\": %d, \"imp\": %d}\n", - (long long)entry->tm.tv_sec, (int)entry->tm.tv_nsec, entry->thd, + entry->tm.tv_sec, entry->tm.tv_nsec, entry->thd, entry->type, entry->tagstr, entry->file, entry->line, entry->important); } diff --git a/src/core/lib/security/credentials/jwt/jwt_credentials.c b/src/core/lib/security/credentials/jwt/jwt_credentials.c index 973fb75eaab..f4dd9208610 100644 --- a/src/core/lib/security/credentials/jwt/jwt_credentials.c +++ b/src/core/lib/security/credentials/jwt/jwt_credentials.c @@ -149,10 +149,10 @@ grpc_call_credentials *grpc_service_account_jwt_access_credentials_create( "grpc_service_account_jwt_access_credentials_create(" "json_key=%s, " "token_lifetime=" - "gpr_timespec { tv_sec: %lld, tv_nsec: %d, clock_type: %d }, " + "gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " "reserved=%p)", 5, - (json_key, (long long)token_lifetime.tv_sec, (int)token_lifetime.tv_nsec, + (json_key, token_lifetime.tv_sec, token_lifetime.tv_nsec, (int)token_lifetime.clock_type, reserved)); GPR_ASSERT(reserved == NULL); return grpc_service_account_jwt_access_credentials_create_from_auth_json_key( diff --git a/src/core/lib/surface/channel.c b/src/core/lib/surface/channel.c index 8936e7164f4..952a769de1d 100644 --- a/src/core/lib/surface/channel.c +++ b/src/core/lib/surface/channel.c @@ -223,10 +223,10 @@ grpc_call *grpc_channel_create_call(grpc_channel *channel, "grpc_channel_create_call(" "channel=%p, parent_call=%p, propagation_mask=%x, cq=%p, method=%s, " "host=%s, " - "deadline=gpr_timespec { tv_sec: %lld, tv_nsec: %d, clock_type: %d }, " + "deadline=gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " "reserved=%p)", 10, (channel, parent_call, (unsigned)propagation_mask, cq, method, host, - (long long)deadline.tv_sec, (int)deadline.tv_nsec, + deadline.tv_sec, deadline.tv_nsec, (int)deadline.clock_type, reserved)); GPR_ASSERT(!reserved); return grpc_channel_create_call_internal( @@ -282,11 +282,11 @@ grpc_call *grpc_channel_create_registered_call( "grpc_channel_create_registered_call(" "channel=%p, parent_call=%p, propagation_mask=%x, completion_queue=%p, " "registered_call_handle=%p, " - "deadline=gpr_timespec { tv_sec: %lld, tv_nsec: %d, clock_type: %d }, " + "deadline=gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " "reserved=%p)", 9, (channel, parent_call, (unsigned)propagation_mask, completion_queue, - registered_call_handle, (long long)deadline.tv_sec, - (int)deadline.tv_nsec, (int)deadline.clock_type, reserved)); + registered_call_handle, deadline.tv_sec, + deadline.tv_nsec, (int)deadline.clock_type, reserved)); GPR_ASSERT(!reserved); return grpc_channel_create_call_internal( channel, parent_call, propagation_mask, completion_queue, NULL, diff --git a/src/core/lib/surface/completion_queue.c b/src/core/lib/surface/completion_queue.c index 2cc6aa74e02..aa35ae02fe2 100644 --- a/src/core/lib/surface/completion_queue.c +++ b/src/core/lib/surface/completion_queue.c @@ -316,9 +316,9 @@ grpc_event grpc_completion_queue_next(grpc_completion_queue *cc, GRPC_API_TRACE( "grpc_completion_queue_next(" "cc=%p, " - "deadline=gpr_timespec { tv_sec: %lld, tv_nsec: %d, clock_type: %d }, " + "deadline=gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " "reserved=%p)", - 5, (cc, (long long)deadline.tv_sec, (int)deadline.tv_nsec, + 5, (cc, deadline.tv_sec, deadline.tv_nsec, (int)deadline.clock_type, reserved)); GPR_ASSERT(!reserved); @@ -428,9 +428,9 @@ grpc_event grpc_completion_queue_pluck(grpc_completion_queue *cc, void *tag, GRPC_API_TRACE( "grpc_completion_queue_pluck(" "cc=%p, tag=%p, " - "deadline=gpr_timespec { tv_sec: %lld, tv_nsec: %d, clock_type: %d }, " + "deadline=gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " "reserved=%p)", - 6, (cc, tag, (long long)deadline.tv_sec, (int)deadline.tv_nsec, + 6, (cc, tag, deadline.tv_sec, deadline.tv_nsec, (int)deadline.clock_type, reserved)); GPR_ASSERT(!reserved); diff --git a/src/core/lib/transport/transport_op_string.c b/src/core/lib/transport/transport_op_string.c index df04c611270..d8fae7805e4 100644 --- a/src/core/lib/transport/transport_op_string.c +++ b/src/core/lib/transport/transport_op_string.c @@ -63,8 +63,8 @@ static void put_metadata_list(gpr_strvec *b, grpc_metadata_batch md) { } if (gpr_time_cmp(md.deadline, gpr_inf_future(md.deadline.clock_type)) != 0) { char *tmp; - gpr_asprintf(&tmp, " deadline=%lld.%09d", (long long)md.deadline.tv_sec, - (int)md.deadline.tv_nsec); + gpr_asprintf(&tmp, " deadline=%"PRId64".%09d", md.deadline.tv_sec, + md.deadline.tv_nsec); gpr_strvec_add(b, tmp); } } From fe754b4e996415300ff29b6e4c378d3d5704852a Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Thu, 23 Jun 2016 12:32:07 -0700 Subject: [PATCH 0209/1039] Use GRPC_PYTHON_CFLAGS/GRPC_PYTHON_LDFLAGS in setup.py. This is needed for building grpcio with mingw, see https://github.com/grpc/grpc/pull/7012. --- setup.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py index f73501b8b35..7e625b937a7 100644 --- a/setup.py +++ b/setup.py @@ -31,6 +31,7 @@ import os import os.path +import shlex import shutil import sys import sysconfig @@ -99,8 +100,9 @@ if not "win32" in sys.platform: DEFINE_MACROS = (('OPENSSL_NO_ASM', 1), ('_WIN32_WINNT', 0x600), ('GPR_BACKWARDS_COMPATIBILITY_MODE', 1),) -LDFLAGS = () -CFLAGS = () +LDFLAGS = shlex.split(os.environ.get('GRPC_PYTHON_LDFLAGS', '')) +CFLAGS = shlex.split(os.environ.get('GRPC_PYTHON_CFLAGS', '')) + if "linux" in sys.platform: LDFLAGS += ('-Wl,-wrap,memcpy',) if "linux" in sys.platform or "darwin" in sys.platform: From 51d4f0194963e60cca2e28c08a53070f2135b4f0 Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Thu, 23 Jun 2016 13:01:43 -0700 Subject: [PATCH 0210/1039] prevent race between grpcsharp_server_request_call and grpc_completion_queue_shutdown --- .../Grpc.Core/Internal/ServerSafeHandle.cs | 23 +++++++++++++------ 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/src/csharp/Grpc.Core/Internal/ServerSafeHandle.cs b/src/csharp/Grpc.Core/Internal/ServerSafeHandle.cs index 85813027064..014a8db78f2 100644 --- a/src/csharp/Grpc.Core/Internal/ServerSafeHandle.cs +++ b/src/csharp/Grpc.Core/Internal/ServerSafeHandle.cs @@ -54,7 +54,10 @@ namespace Grpc.Core.Internal public void RegisterCompletionQueue(CompletionQueueSafeHandle cq) { - Native.grpcsharp_server_register_completion_queue(this, cq); + using (cq.NewScope()) + { + Native.grpcsharp_server_register_completion_queue(this, cq); + } } public int AddInsecurePort(string addr) @@ -74,16 +77,22 @@ namespace Grpc.Core.Internal public void ShutdownAndNotify(BatchCompletionDelegate callback, CompletionQueueSafeHandle completionQueue) { - var ctx = BatchContextSafeHandle.Create(); - completionQueue.CompletionRegistry.RegisterBatchCompletion(ctx, callback); - Native.grpcsharp_server_shutdown_and_notify_callback(this, completionQueue, ctx); + using (completionQueue.NewScope()) + { + var ctx = BatchContextSafeHandle.Create(); + completionQueue.CompletionRegistry.RegisterBatchCompletion(ctx, callback); + Native.grpcsharp_server_shutdown_and_notify_callback(this, completionQueue, ctx); + } } public void RequestCall(BatchCompletionDelegate callback, CompletionQueueSafeHandle completionQueue) { - var ctx = BatchContextSafeHandle.Create(); - completionQueue.CompletionRegistry.RegisterBatchCompletion(ctx, callback); - Native.grpcsharp_server_request_call(this, completionQueue, ctx).CheckOk(); + using (completionQueue.NewScope()) + { + var ctx = BatchContextSafeHandle.Create(); + completionQueue.CompletionRegistry.RegisterBatchCompletion(ctx, callback); + Native.grpcsharp_server_request_call(this, completionQueue, ctx).CheckOk(); + } } protected override bool ReleaseHandle() From 5d11e43ce34b371fb0ab1fa69e541a4d513025b8 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 23 Jun 2016 13:14:05 -0700 Subject: [PATCH 0211/1039] Added test for filter whose call initialization fails, and fixed bugs uncovered by the test. --- Makefile | 2 + src/core/lib/channel/channel_stack.c | 7 +- src/core/lib/iomgr/error.c | 6 + src/core/lib/iomgr/error.h | 3 + src/core/lib/surface/call.c | 10 +- test/core/end2end/end2end_nosec_tests.c | 8 + test/core/end2end/end2end_tests.c | 8 + test/core/end2end/gen_build_yaml.py | 1 + .../end2end/tests/filter_call_init_fails.c | 263 ++++++++ tools/run_tests/sources_and_headers.json | 2 + tools/run_tests/tests.json | 627 +++++++++++++++++- .../end2end_nosec_tests.vcxproj | 2 + .../end2end_nosec_tests.vcxproj.filters | 3 + .../tests/end2end_tests/end2end_tests.vcxproj | 2 + .../end2end_tests.vcxproj.filters | 3 + 15 files changed, 929 insertions(+), 18 deletions(-) create mode 100644 test/core/end2end/tests/filter_call_init_fails.c diff --git a/Makefile b/Makefile index 9be3e5784cc..8e5d0102c3f 100644 --- a/Makefile +++ b/Makefile @@ -6366,6 +6366,7 @@ LIBEND2END_TESTS_SRC = \ test/core/end2end/tests/default_host.c \ test/core/end2end/tests/disappearing_server.c \ test/core/end2end/tests/empty_batch.c \ + test/core/end2end/tests/filter_call_init_fails.c \ test/core/end2end/tests/filter_causes_close.c \ test/core/end2end/tests/graceful_server_shutdown.c \ test/core/end2end/tests/high_initial_seqno.c \ @@ -6443,6 +6444,7 @@ LIBEND2END_NOSEC_TESTS_SRC = \ test/core/end2end/tests/default_host.c \ test/core/end2end/tests/disappearing_server.c \ test/core/end2end/tests/empty_batch.c \ + test/core/end2end/tests/filter_call_init_fails.c \ test/core/end2end/tests/filter_causes_close.c \ test/core/end2end/tests/graceful_server_shutdown.c \ test/core/end2end/tests/high_initial_seqno.c \ diff --git a/src/core/lib/channel/channel_stack.c b/src/core/lib/channel/channel_stack.c index 0613f94c299..715f6316254 100644 --- a/src/core/lib/channel/channel_stack.c +++ b/src/core/lib/channel/channel_stack.c @@ -179,6 +179,7 @@ grpc_error* grpc_call_stack_init(grpc_exec_ctx *exec_ctx, ROUND_UP_TO_ALIGNMENT_SIZE(count * sizeof(grpc_call_element)); /* init per-filter data */ + grpc_error* first_error = GRPC_ERROR_NONE; for (i = 0; i < count; i++) { args.call_stack = call_stack; args.server_transport_data = transport_server_data; @@ -188,12 +189,12 @@ grpc_error* grpc_call_stack_init(grpc_exec_ctx *exec_ctx, call_elems[i].call_data = user_data; grpc_error* error = call_elems[i].filter->init_call_elem( exec_ctx, &call_elems[i], &args); - if (error != GRPC_ERROR_NONE) - return error; + if (error != GRPC_ERROR_NONE && first_error == GRPC_ERROR_NONE) + first_error = error; user_data += ROUND_UP_TO_ALIGNMENT_SIZE(call_elems[i].filter->sizeof_call_data); } - return GRPC_ERROR_NONE; + return first_error; } void grpc_call_stack_set_pollset_or_pollset_set(grpc_exec_ctx *exec_ctx, diff --git a/src/core/lib/iomgr/error.c b/src/core/lib/iomgr/error.c index 540fb4fa7e4..1b15963e57d 100644 --- a/src/core/lib/iomgr/error.c +++ b/src/core/lib/iomgr/error.c @@ -286,6 +286,12 @@ grpc_error *grpc_error_set_str(grpc_error *src, grpc_error_strs which, return new; } +const char *grpc_error_get_str(grpc_error *error, grpc_error_strs which) { + void *s = NULL; + gpr_avl_maybe_get(error->strs, (void *)(uintptr_t)which, &s); + return s; +} + grpc_error *grpc_error_add_child(grpc_error *src, grpc_error *child) { grpc_error *new = copy_error_and_unref(src); new->errs = gpr_avl_add(new->errs, (void *)(new->next_err++), child); diff --git a/src/core/lib/iomgr/error.h b/src/core/lib/iomgr/error.h index 69cdf3028e4..4bff591fe31 100644 --- a/src/core/lib/iomgr/error.h +++ b/src/core/lib/iomgr/error.h @@ -169,6 +169,9 @@ grpc_error *grpc_error_set_time(grpc_error *src, grpc_error_times which, gpr_timespec value); grpc_error *grpc_error_set_str(grpc_error *src, grpc_error_strs which, const char *value); +/// Returns NULL if the specified string is not set. +/// Caller does NOT own return value. +const char *grpc_error_get_str(grpc_error *error, grpc_error_strs which); /// Add a child error: an error that is believed to have contributed to this /// error occurring. Allows root causing high level errors from lower level /// errors that contributed to them. diff --git a/src/core/lib/surface/call.c b/src/core/lib/surface/call.c index 2e89393815f..d68afff39f7 100644 --- a/src/core/lib/surface/call.c +++ b/src/core/lib/surface/call.c @@ -270,12 +270,10 @@ grpc_call *grpc_call_create( intptr_t status; if (!grpc_error_get_int(error, GRPC_ERROR_INT_GRPC_STATUS, &status)) status = GRPC_STATUS_UNKNOWN; - const char* error_string = grpc_error_string(error); - received_status* status_struct = &call->status[STATUS_FROM_CORE]; - status_struct->is_set = true; - status_struct->code = status; - status_struct->details = grpc_mdstr_from_string(error_string); - grpc_error_free_string(error_string); + const char* error_str = grpc_error_get_str(error, + GRPC_ERROR_STR_DESCRIPTION); + close_with_status(&exec_ctx, call, status, + error_str == NULL ? "unknown error" : error_str); grpc_error_unref(error); } if (cq != NULL) { diff --git a/test/core/end2end/end2end_nosec_tests.c b/test/core/end2end/end2end_nosec_tests.c index 2893bddc43c..03959308987 100644 --- a/test/core/end2end/end2end_nosec_tests.c +++ b/test/core/end2end/end2end_nosec_tests.c @@ -69,6 +69,8 @@ extern void disappearing_server(grpc_end2end_test_config config); extern void disappearing_server_pre_init(void); extern void empty_batch(grpc_end2end_test_config config); extern void empty_batch_pre_init(void); +extern void filter_call_init_fails(grpc_end2end_test_config config); +extern void filter_call_init_fails_pre_init(void); extern void filter_causes_close(grpc_end2end_test_config config); extern void filter_causes_close_pre_init(void); extern void graceful_server_shutdown(grpc_end2end_test_config config); @@ -136,6 +138,7 @@ void grpc_end2end_tests_pre_init(void) { default_host_pre_init(); disappearing_server_pre_init(); empty_batch_pre_init(); + filter_call_init_fails_pre_init(); filter_causes_close_pre_init(); graceful_server_shutdown_pre_init(); high_initial_seqno_pre_init(); @@ -183,6 +186,7 @@ void grpc_end2end_tests(int argc, char **argv, default_host(config); disappearing_server(config); empty_batch(config); + filter_call_init_fails(config); filter_causes_close(config); graceful_server_shutdown(config); high_initial_seqno(config); @@ -264,6 +268,10 @@ void grpc_end2end_tests(int argc, char **argv, empty_batch(config); continue; } + if (0 == strcmp("filter_call_init_fails", argv[i])) { + filter_call_init_fails(config); + continue; + } if (0 == strcmp("filter_causes_close", argv[i])) { filter_causes_close(config); continue; diff --git a/test/core/end2end/end2end_tests.c b/test/core/end2end/end2end_tests.c index 96a38e76dcc..1d6ada72540 100644 --- a/test/core/end2end/end2end_tests.c +++ b/test/core/end2end/end2end_tests.c @@ -71,6 +71,8 @@ extern void disappearing_server(grpc_end2end_test_config config); extern void disappearing_server_pre_init(void); extern void empty_batch(grpc_end2end_test_config config); extern void empty_batch_pre_init(void); +extern void filter_call_init_fails(grpc_end2end_test_config config); +extern void filter_call_init_fails_pre_init(void); extern void filter_causes_close(grpc_end2end_test_config config); extern void filter_causes_close_pre_init(void); extern void graceful_server_shutdown(grpc_end2end_test_config config); @@ -139,6 +141,7 @@ void grpc_end2end_tests_pre_init(void) { default_host_pre_init(); disappearing_server_pre_init(); empty_batch_pre_init(); + filter_call_init_fails_pre_init(); filter_causes_close_pre_init(); graceful_server_shutdown_pre_init(); high_initial_seqno_pre_init(); @@ -187,6 +190,7 @@ void grpc_end2end_tests(int argc, char **argv, default_host(config); disappearing_server(config); empty_batch(config); + filter_call_init_fails(config); filter_causes_close(config); graceful_server_shutdown(config); high_initial_seqno(config); @@ -272,6 +276,10 @@ void grpc_end2end_tests(int argc, char **argv, empty_batch(config); continue; } + if (0 == strcmp("filter_call_init_fails", argv[i])) { + filter_call_init_fails(config); + continue; + } if (0 == strcmp("filter_causes_close", argv[i])) { filter_causes_close(config); continue; diff --git a/test/core/end2end/gen_build_yaml.py b/test/core/end2end/gen_build_yaml.py index 6d3d8f8d3c4..5037559fea0 100755 --- a/test/core/end2end/gen_build_yaml.py +++ b/test/core/end2end/gen_build_yaml.py @@ -102,6 +102,7 @@ END2END_TESTS = { 'disappearing_server': connectivity_test_options, 'empty_batch': default_test_options, 'filter_causes_close': default_test_options, + 'filter_call_init_fails': default_test_options, 'graceful_server_shutdown': default_test_options._replace(cpu_cost=LOWCPU), 'hpack_size': default_test_options._replace(proxyable=False, traceable=False), diff --git a/test/core/end2end/tests/filter_call_init_fails.c b/test/core/end2end/tests/filter_call_init_fails.c new file mode 100644 index 00000000000..d97369fa98b --- /dev/null +++ b/test/core/end2end/tests/filter_call_init_fails.c @@ -0,0 +1,263 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include "test/core/end2end/end2end_tests.h" + +#include +#include +#include + +#include +#include +#include +#include +#include +#include "src/core/lib/channel/channel_stack_builder.h" +#include "src/core/lib/surface/channel_init.h" +#include "test/core/end2end/cq_verifier.h" + +enum { TIMEOUT = 200000 }; + +static bool g_enable_filter = false; + +static void *tag(intptr_t t) { return (void *)t; } + +static grpc_end2end_test_fixture begin_test(grpc_end2end_test_config config, + const char *test_name, + grpc_channel_args *client_args, + grpc_channel_args *server_args) { + grpc_end2end_test_fixture f; + gpr_log(GPR_INFO, "%s/%s", test_name, config.name); + f = config.create_fixture(client_args, server_args); + config.init_server(&f, server_args); + config.init_client(&f, client_args); + return f; +} + +static gpr_timespec n_seconds_time(int n) { + return GRPC_TIMEOUT_SECONDS_TO_DEADLINE(n); +} + +static gpr_timespec five_seconds_time(void) { return n_seconds_time(5); } + +static void drain_cq(grpc_completion_queue *cq) { + grpc_event ev; + do { + ev = grpc_completion_queue_next(cq, five_seconds_time(), NULL); + } while (ev.type != GRPC_QUEUE_SHUTDOWN); +} + +static void shutdown_server(grpc_end2end_test_fixture *f) { + if (!f->server) return; + grpc_server_shutdown_and_notify(f->server, f->cq, tag(1000)); + GPR_ASSERT(grpc_completion_queue_pluck( + f->cq, tag(1000), GRPC_TIMEOUT_SECONDS_TO_DEADLINE(5), NULL) + .type == GRPC_OP_COMPLETE); + grpc_server_destroy(f->server); + f->server = NULL; +} + +static void shutdown_client(grpc_end2end_test_fixture *f) { + if (!f->client) return; + grpc_channel_destroy(f->client); + f->client = NULL; +} + +static void end_test(grpc_end2end_test_fixture *f) { + shutdown_server(f); + shutdown_client(f); + + grpc_completion_queue_shutdown(f->cq); + drain_cq(f->cq); + grpc_completion_queue_destroy(f->cq); +} + +// Simple request via a server filter that always fails to initialize +// the call. +static void test_request(grpc_end2end_test_config config) { + grpc_call *c; + grpc_call *s; + gpr_slice request_payload_slice = gpr_slice_from_copied_string("hello world"); + grpc_byte_buffer *request_payload = + grpc_raw_byte_buffer_create(&request_payload_slice, 1); + gpr_timespec deadline = five_seconds_time(); + grpc_end2end_test_fixture f = + begin_test(config, "filter_call_init_fails", NULL, NULL); + cq_verifier *cqv = cq_verifier_create(f.cq); + grpc_op ops[6]; + grpc_op *op; + grpc_metadata_array initial_metadata_recv; + grpc_metadata_array trailing_metadata_recv; + grpc_metadata_array request_metadata_recv; + grpc_byte_buffer *request_payload_recv = NULL; + grpc_call_details call_details; + grpc_status_code status; + grpc_call_error error; + char *details = NULL; + size_t details_capacity = 0; + + c = grpc_channel_create_call(f.client, NULL, GRPC_PROPAGATE_DEFAULTS, f.cq, + "/foo", "foo.test.google.fr", deadline, NULL); + GPR_ASSERT(c); + + grpc_metadata_array_init(&initial_metadata_recv); + grpc_metadata_array_init(&trailing_metadata_recv); + grpc_metadata_array_init(&request_metadata_recv); + grpc_call_details_init(&call_details); + + memset(ops, 0, sizeof(ops)); + op = ops; + op->op = GRPC_OP_SEND_INITIAL_METADATA; + op->data.send_initial_metadata.count = 0; + op->data.send_initial_metadata.metadata = NULL; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_SEND_MESSAGE; + op->data.send_message = request_payload; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_SEND_CLOSE_FROM_CLIENT; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_INITIAL_METADATA; + op->data.recv_initial_metadata = &initial_metadata_recv; + op->flags = 0; + op->reserved = NULL; + op++; + op->op = GRPC_OP_RECV_STATUS_ON_CLIENT; + op->data.recv_status_on_client.trailing_metadata = &trailing_metadata_recv; + op->data.recv_status_on_client.status = &status; + op->data.recv_status_on_client.status_details = &details; + op->data.recv_status_on_client.status_details_capacity = &details_capacity; + op->flags = 0; + op->reserved = NULL; + op++; + error = grpc_call_start_batch(c, ops, (size_t)(op - ops), tag(1), NULL); + GPR_ASSERT(GRPC_CALL_OK == error); + + error = + grpc_server_request_call(f.server, &s, &call_details, + &request_metadata_recv, f.cq, f.cq, tag(101)); + GPR_ASSERT(GRPC_CALL_OK == error); + + cq_expect_completion(cqv, tag(1), 1); + cq_verify(cqv); + + GPR_ASSERT(status == GRPC_STATUS_PERMISSION_DENIED); + GPR_ASSERT(0 == strcmp(details, "access denied")); + + gpr_free(details); + grpc_metadata_array_destroy(&initial_metadata_recv); + grpc_metadata_array_destroy(&trailing_metadata_recv); + grpc_metadata_array_destroy(&request_metadata_recv); + grpc_call_details_destroy(&call_details); + + grpc_call_destroy(c); + + cq_verifier_destroy(cqv); + + grpc_byte_buffer_destroy(request_payload); + grpc_byte_buffer_destroy(request_payload_recv); + + end_test(&f); + config.tear_down_data(&f); +} + +/******************************************************************************* + * Test filter - always fails to initialize a call + */ + +static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, + grpc_call_element *elem, + grpc_call_element_args *args) { + return grpc_error_set_int(GRPC_ERROR_CREATE("access denied"), + GRPC_ERROR_INT_GRPC_STATUS, + GRPC_STATUS_PERMISSION_DENIED); +} + +static void destroy_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, + const grpc_call_stats *stats, + void *and_free_memory) {} + +static void init_channel_elem(grpc_exec_ctx *exec_ctx, + grpc_channel_element *elem, + grpc_channel_element_args *args) {} + +static void destroy_channel_elem(grpc_exec_ctx *exec_ctx, + grpc_channel_element *elem) {} + +static const grpc_channel_filter test_filter = { + grpc_call_next_op, + grpc_channel_next_op, + 0, + init_call_elem, + grpc_call_stack_ignore_set_pollset_or_pollset_set, + destroy_call_elem, + 0, + init_channel_elem, + destroy_channel_elem, + grpc_call_next_get_peer, + "filter_call_init_fails"}; + +/******************************************************************************* + * Registration + */ + +static bool maybe_add_filter(grpc_channel_stack_builder *builder, void *arg) { + if (g_enable_filter) { + return grpc_channel_stack_builder_prepend_filter(builder, &test_filter, + NULL, NULL); + } else { + return true; + } +} + +static void init_plugin(void) { + grpc_channel_init_register_stage(GRPC_SERVER_CHANNEL, 0, maybe_add_filter, + NULL); +} + +static void destroy_plugin(void) {} + +void filter_call_init_fails(grpc_end2end_test_config config) { + g_enable_filter = true; + test_request(config); + g_enable_filter = false; +} + +void filter_call_init_fails_pre_init(void) { + grpc_register_plugin(init_plugin, destroy_plugin); +} diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 00018834af9..2bea121baad 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -5377,6 +5377,7 @@ "test/core/end2end/tests/default_host.c", "test/core/end2end/tests/disappearing_server.c", "test/core/end2end/tests/empty_batch.c", + "test/core/end2end/tests/filter_call_init_fails.c", "test/core/end2end/tests/filter_causes_close.c", "test/core/end2end/tests/graceful_server_shutdown.c", "test/core/end2end/tests/high_initial_seqno.c", @@ -5436,6 +5437,7 @@ "test/core/end2end/tests/default_host.c", "test/core/end2end/tests/disappearing_server.c", "test/core/end2end/tests/empty_batch.c", + "test/core/end2end/tests/filter_call_init_fails.c", "test/core/end2end/tests/filter_causes_close.c", "test/core/end2end/tests/graceful_server_shutdown.c", "test/core/end2end/tests/high_initial_seqno.c", diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index c62058ede4e..381931db0fa 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -4680,6 +4680,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_census_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -5538,6 +5560,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_compress_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -6382,6 +6426,27 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_fakesec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -7127,6 +7192,26 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_fd_test", + "platforms": [ + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -7895,6 +7980,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -8669,6 +8776,22 @@ "linux" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "linux" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full+pipe_test", + "platforms": [ + "linux" + ] + }, { "args": [ "filter_causes_close" @@ -9377,6 +9500,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full+trace_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -10213,6 +10358,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_loadreporting_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -11057,6 +11224,27 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_oauth2_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -11834,6 +12022,27 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_proxy_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -12506,6 +12715,27 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_sockpair_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -13220,6 +13450,27 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_sockpair+trace_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -13915,7 +14166,7 @@ }, { "args": [ - "filter_causes_close" + "filter_call_init_fails" ], "ci_platforms": [ "windows", @@ -13936,14 +14187,14 @@ }, { "args": [ - "graceful_server_shutdown" + "filter_causes_close" ], "ci_platforms": [ "windows", "linux", "posix" ], - "cpu_cost": 0.1, + "cpu_cost": 1.0, "exclude_configs": [], "flaky": false, "language": "c", @@ -13957,14 +14208,14 @@ }, { "args": [ - "high_initial_seqno" + "graceful_server_shutdown" ], "ci_platforms": [ "windows", "linux", "posix" ], - "cpu_cost": 1.0, + "cpu_cost": 0.1, "exclude_configs": [], "flaky": false, "language": "c", @@ -13978,7 +14229,7 @@ }, { "args": [ - "hpack_size" + "high_initial_seqno" ], "ci_platforms": [ "windows", @@ -13999,7 +14250,7 @@ }, { "args": [ - "idempotent_request" + "hpack_size" ], "ci_platforms": [ "windows", @@ -14020,7 +14271,7 @@ }, { "args": [ - "invoke_large_request" + "idempotent_request" ], "ci_platforms": [ "windows", @@ -14041,7 +14292,28 @@ }, { "args": [ - "large_metadata" + "invoke_large_request" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_sockpair_1byte_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, + { + "args": [ + "large_metadata" ], "ci_platforms": [ "windows", @@ -14704,6 +14976,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_ssl_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -15562,6 +15856,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_ssl_cert_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -16364,6 +16680,27 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_ssl_proxy_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -17065,6 +17402,26 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_uds_test", + "platforms": [ + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -17851,6 +18208,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_census_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -18687,6 +19066,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_compress_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -19437,6 +19838,26 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_fd_nosec_test", + "platforms": [ + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -20183,6 +20604,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -20941,6 +21384,22 @@ "linux" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "linux" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full+pipe_nosec_test", + "platforms": [ + "linux" + ] + }, { "args": [ "filter_causes_close" @@ -21627,6 +22086,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_full+trace_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -22441,6 +22922,28 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_loadreporting_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -23222,6 +23725,27 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_proxy_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -23873,6 +24397,27 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_sockpair_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -24566,6 +25111,27 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_sockpair+trace_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -25258,6 +25824,29 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "windows", + "linux", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [ + "msan" + ], + "flaky": false, + "language": "c", + "name": "h2_sockpair_1byte_nosec_test", + "platforms": [ + "windows", + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" @@ -26027,6 +26616,26 @@ "posix" ] }, + { + "args": [ + "filter_call_init_fails" + ], + "ci_platforms": [ + "linux", + "mac", + "posix" + ], + "cpu_cost": 1.0, + "exclude_configs": [], + "flaky": false, + "language": "c", + "name": "h2_uds_nosec_test", + "platforms": [ + "linux", + "mac", + "posix" + ] + }, { "args": [ "filter_causes_close" diff --git a/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj b/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj index 923c1d1ab41..eb5f285c663 100644 --- a/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj +++ b/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj @@ -179,6 +179,8 @@ + + diff --git a/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj.filters b/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj.filters index 6533eaa057b..9995b92ae56 100644 --- a/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj.filters +++ b/vsprojects/vcxproj/test/end2end/tests/end2end_nosec_tests/end2end_nosec_tests.vcxproj.filters @@ -43,6 +43,9 @@ test\core\end2end\tests + + test\core\end2end\tests + test\core\end2end\tests diff --git a/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj b/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj index 0b859e25ce4..33132e1138f 100644 --- a/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj +++ b/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj @@ -181,6 +181,8 @@ + + diff --git a/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj.filters b/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj.filters index ea1c5e3c237..484fb0c9cc5 100644 --- a/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj.filters +++ b/vsprojects/vcxproj/test/end2end/tests/end2end_tests/end2end_tests.vcxproj.filters @@ -46,6 +46,9 @@ test\core\end2end\tests + + test\core\end2end\tests + test\core\end2end\tests From 76d24420d7a6471dc3b135b62318277991ebdb08 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 23 Jun 2016 13:22:10 -0700 Subject: [PATCH 0212/1039] clang-format --- src/core/ext/census/grpc_filter.c | 4 ++-- src/core/ext/client_config/client_channel.c | 2 +- src/core/ext/client_config/subchannel.c | 8 ++++---- src/core/ext/client_config/subchannel.h | 2 +- src/core/ext/client_config/subchannel_call_holder.c | 8 ++++---- src/core/ext/load_reporting/load_reporting_filter.c | 2 +- src/core/lib/channel/channel_stack.c | 8 ++++---- src/core/lib/channel/channel_stack.h | 4 ++-- src/core/lib/channel/compress_filter.c | 2 +- src/core/lib/channel/connected_channel.c | 7 +++---- src/core/lib/channel/http_client_filter.c | 2 +- src/core/lib/channel/http_server_filter.c | 2 +- .../lib/security/transport/client_auth_filter.c | 2 +- .../lib/security/transport/server_auth_filter.c | 2 +- src/core/lib/surface/call.c | 13 ++++++------- src/core/lib/surface/lame_client.c | 2 +- src/core/lib/surface/server.c | 2 +- test/core/channel/channel_stack_test.c | 8 ++++---- test/core/end2end/tests/filter_call_init_fails.c | 2 +- test/core/end2end/tests/filter_causes_close.c | 2 +- 20 files changed, 41 insertions(+), 43 deletions(-) diff --git a/src/core/ext/census/grpc_filter.c b/src/core/ext/census/grpc_filter.c index dd2e2e01244..c5a6e6d3dc2 100644 --- a/src/core/ext/census/grpc_filter.c +++ b/src/core/ext/census/grpc_filter.c @@ -124,7 +124,7 @@ static void server_start_transport_op(grpc_exec_ctx *exec_ctx, grpc_call_next_op(exec_ctx, elem, op); } -static grpc_error* client_init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *client_init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { call_data *d = elem->call_data; @@ -143,7 +143,7 @@ static void client_destroy_call_elem(grpc_exec_ctx *exec_ctx, /* TODO(hongyu): record rpc client stats and census_rpc_end_op here */ } -static grpc_error* server_init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *server_init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { call_data *d = elem->call_data; diff --git a/src/core/ext/client_config/client_channel.c b/src/core/ext/client_config/client_channel.c index b0a09dab932..84e3fb4da65 100644 --- a/src/core/ext/client_config/client_channel.c +++ b/src/core/ext/client_config/client_channel.c @@ -430,7 +430,7 @@ static int cc_pick_subchannel(grpc_exec_ctx *exec_ctx, void *elemp, } /* Constructor for call_data */ -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { grpc_subchannel_call_holder_init(elem->call_data, cc_pick_subchannel, elem, diff --git a/src/core/ext/client_config/subchannel.c b/src/core/ext/client_config/subchannel.c index 80cf154f96a..8a1ac68c6e3 100644 --- a/src/core/ext/client_config/subchannel.c +++ b/src/core/ext/client_config/subchannel.c @@ -708,11 +708,11 @@ grpc_error *grpc_connected_subchannel_create_call( grpc_call_stack *callstk = SUBCHANNEL_CALL_TO_CALL_STACK(*call); (*call)->connection = con; GRPC_CONNECTED_SUBCHANNEL_REF(con, "subchannel_call"); - grpc_error* error = grpc_call_stack_init(exec_ctx, chanstk, 1, - subchannel_call_destroy, *call, - NULL, NULL, callstk); + grpc_error *error = + grpc_call_stack_init(exec_ctx, chanstk, 1, subchannel_call_destroy, *call, + NULL, NULL, callstk); if (error != GRPC_ERROR_NONE) { - const char* error_string = grpc_error_string(error); + const char *error_string = grpc_error_string(error); gpr_log(GPR_ERROR, "error: %s", error_string); grpc_error_free_string(error_string); return error; diff --git a/src/core/ext/client_config/subchannel.h b/src/core/ext/client_config/subchannel.h index 21454cb7a4d..ae1d96e6400 100644 --- a/src/core/ext/client_config/subchannel.h +++ b/src/core/ext/client_config/subchannel.h @@ -110,7 +110,7 @@ void grpc_subchannel_call_unref(grpc_exec_ctx *exec_ctx, /** construct a subchannel call */ grpc_error *grpc_connected_subchannel_create_call( grpc_exec_ctx *exec_ctx, grpc_connected_subchannel *connected_subchannel, - grpc_polling_entity *pollent, grpc_subchannel_call** subchannel_call); + grpc_polling_entity *pollent, grpc_subchannel_call **subchannel_call); /** process a transport level op */ void grpc_connected_subchannel_process_transport_op( diff --git a/src/core/ext/client_config/subchannel_call_holder.c b/src/core/ext/client_config/subchannel_call_holder.c index 25bd9798f0a..5b542331538 100644 --- a/src/core/ext/client_config/subchannel_call_holder.c +++ b/src/core/ext/client_config/subchannel_call_holder.c @@ -167,8 +167,8 @@ retry: /* if we've got a subchannel, then let's ask it to create a call */ if (holder->creation_phase == GRPC_SUBCHANNEL_CALL_HOLDER_NOT_CREATING && holder->connected_subchannel != NULL) { - grpc_subchannel_call* subchannel_call = NULL; - grpc_error* error = grpc_connected_subchannel_create_call( + grpc_subchannel_call *subchannel_call = NULL; + grpc_error *error = grpc_connected_subchannel_create_call( exec_ctx, holder->connected_subchannel, holder->pollent, &subchannel_call); if (error != GRPC_ERROR_NONE) { @@ -205,8 +205,8 @@ static void subchannel_ready(grpc_exec_ctx *exec_ctx, void *arg, GRPC_ERROR_CREATE_REFERENCING( "Cancelled before creating subchannel", &error, 1)); } else { - grpc_subchannel_call* subchannel_call = NULL; - grpc_error* new_error = grpc_connected_subchannel_create_call( + grpc_subchannel_call *subchannel_call = NULL; + grpc_error *new_error = grpc_connected_subchannel_create_call( exec_ctx, holder->connected_subchannel, holder->pollent, &subchannel_call); if (new_error != GRPC_ERROR_NONE) { diff --git a/src/core/ext/load_reporting/load_reporting_filter.c b/src/core/ext/load_reporting/load_reporting_filter.c index cfe752d6e2b..b584e31c5dd 100644 --- a/src/core/ext/load_reporting/load_reporting_filter.c +++ b/src/core/ext/load_reporting/load_reporting_filter.c @@ -56,7 +56,7 @@ static void invoke_lr_fn_locked(grpc_load_reporting_config *lrc, } /* Constructor for call_data */ -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { call_data *calld = elem->call_data; diff --git a/src/core/lib/channel/channel_stack.c b/src/core/lib/channel/channel_stack.c index 715f6316254..ff824c781f1 100644 --- a/src/core/lib/channel/channel_stack.c +++ b/src/core/lib/channel/channel_stack.c @@ -157,7 +157,7 @@ void grpc_channel_stack_destroy(grpc_exec_ctx *exec_ctx, } } -grpc_error* grpc_call_stack_init(grpc_exec_ctx *exec_ctx, +grpc_error *grpc_call_stack_init(grpc_exec_ctx *exec_ctx, grpc_channel_stack *channel_stack, int initial_refs, grpc_iomgr_cb_func destroy, void *destroy_arg, @@ -179,7 +179,7 @@ grpc_error* grpc_call_stack_init(grpc_exec_ctx *exec_ctx, ROUND_UP_TO_ALIGNMENT_SIZE(count * sizeof(grpc_call_element)); /* init per-filter data */ - grpc_error* first_error = GRPC_ERROR_NONE; + grpc_error *first_error = GRPC_ERROR_NONE; for (i = 0; i < count; i++) { args.call_stack = call_stack; args.server_transport_data = transport_server_data; @@ -187,8 +187,8 @@ grpc_error* grpc_call_stack_init(grpc_exec_ctx *exec_ctx, call_elems[i].filter = channel_elems[i].filter; call_elems[i].channel_data = channel_elems[i].channel_data; call_elems[i].call_data = user_data; - grpc_error* error = call_elems[i].filter->init_call_elem( - exec_ctx, &call_elems[i], &args); + grpc_error *error = + call_elems[i].filter->init_call_elem(exec_ctx, &call_elems[i], &args); if (error != GRPC_ERROR_NONE && first_error == GRPC_ERROR_NONE) first_error = error; user_data += diff --git a/src/core/lib/channel/channel_stack.h b/src/core/lib/channel/channel_stack.h index f56c6540674..ba7baeb23fc 100644 --- a/src/core/lib/channel/channel_stack.h +++ b/src/core/lib/channel/channel_stack.h @@ -110,7 +110,7 @@ typedef struct { on a client; if it is non-NULL, then it points to memory owned by the transport and is on the server. Most filters want to ignore this argument. */ - grpc_error* (*init_call_elem)(grpc_exec_ctx *exec_ctx, + grpc_error *(*init_call_elem)(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args); void (*set_pollset_or_pollset_set)(grpc_exec_ctx *exec_ctx, @@ -210,7 +210,7 @@ void grpc_channel_stack_destroy(grpc_exec_ctx *exec_ctx, /* Initialize a call stack given a channel stack. transport_server_data is expected to be NULL on a client, or an opaque transport owned pointer on the server. */ -grpc_error* grpc_call_stack_init(grpc_exec_ctx *exec_ctx, +grpc_error *grpc_call_stack_init(grpc_exec_ctx *exec_ctx, grpc_channel_stack *channel_stack, int initial_refs, grpc_iomgr_cb_func destroy, void *destroy_arg, diff --git a/src/core/lib/channel/compress_filter.c b/src/core/lib/channel/compress_filter.c index bcc8e17a9db..5be32929dad 100644 --- a/src/core/lib/channel/compress_filter.c +++ b/src/core/lib/channel/compress_filter.c @@ -256,7 +256,7 @@ static void compress_start_transport_stream_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { /* grab pointers to our data from the call element */ diff --git a/src/core/lib/channel/connected_channel.c b/src/core/lib/channel/connected_channel.c index 8dde688df2c..ecafcc99c79 100644 --- a/src/core/lib/channel/connected_channel.c +++ b/src/core/lib/channel/connected_channel.c @@ -81,7 +81,7 @@ static void con_start_transport_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { call_data *calld = elem->call_data; @@ -89,9 +89,8 @@ static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, int r = grpc_transport_init_stream( exec_ctx, chand->transport, TRANSPORT_STREAM_FROM_CALL_DATA(calld), &args->call_stack->refcount, args->server_transport_data); - return r == 0 - ? GRPC_ERROR_NONE - : GRPC_ERROR_CREATE("transport initialization failed"); + return r == 0 ? GRPC_ERROR_NONE + : GRPC_ERROR_CREATE("transport initialization failed"); } static void set_pollset_or_pollset_set(grpc_exec_ctx *exec_ctx, diff --git a/src/core/lib/channel/http_client_filter.c b/src/core/lib/channel/http_client_filter.c index 30f4ed1f6fc..7f8ca1dc9f4 100644 --- a/src/core/lib/channel/http_client_filter.c +++ b/src/core/lib/channel/http_client_filter.c @@ -169,7 +169,7 @@ static void hc_start_transport_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { call_data *calld = elem->call_data; diff --git a/src/core/lib/channel/http_server_filter.c b/src/core/lib/channel/http_server_filter.c index 960d1ce45e5..e1f76510b63 100644 --- a/src/core/lib/channel/http_server_filter.c +++ b/src/core/lib/channel/http_server_filter.c @@ -224,7 +224,7 @@ static void hs_start_transport_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { /* grab pointers to our data from the call element */ diff --git a/src/core/lib/security/transport/client_auth_filter.c b/src/core/lib/security/transport/client_auth_filter.c index 6179b9e18cf..bccb8f755e5 100644 --- a/src/core/lib/security/transport/client_auth_filter.c +++ b/src/core/lib/security/transport/client_auth_filter.c @@ -264,7 +264,7 @@ static void auth_start_transport_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { call_data *calld = elem->call_data; diff --git a/src/core/lib/security/transport/server_auth_filter.c b/src/core/lib/security/transport/server_auth_filter.c index 379247131b6..86d4d616371 100644 --- a/src/core/lib/security/transport/server_auth_filter.c +++ b/src/core/lib/security/transport/server_auth_filter.c @@ -199,7 +199,7 @@ static void auth_start_transport_op(grpc_exec_ctx *exec_ctx, } /* Constructor for call_data */ -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { /* grab pointers to our data from the call element */ diff --git a/src/core/lib/surface/call.c b/src/core/lib/surface/call.c index d68afff39f7..f862e8dee99 100644 --- a/src/core/lib/surface/call.c +++ b/src/core/lib/surface/call.c @@ -262,18 +262,17 @@ grpc_call *grpc_call_create( call->send_deadline = send_deadline; GRPC_CHANNEL_INTERNAL_REF(channel, "call"); /* initial refcount dropped by grpc_call_destroy */ - grpc_error* error = grpc_call_stack_init(&exec_ctx, channel_stack, 1, - destroy_call, call, call->context, - server_transport_data, - CALL_STACK_FROM_CALL(call)); + grpc_error *error = grpc_call_stack_init( + &exec_ctx, channel_stack, 1, destroy_call, call, call->context, + server_transport_data, CALL_STACK_FROM_CALL(call)); if (error != GRPC_ERROR_NONE) { intptr_t status; if (!grpc_error_get_int(error, GRPC_ERROR_INT_GRPC_STATUS, &status)) status = GRPC_STATUS_UNKNOWN; - const char* error_str = grpc_error_get_str(error, - GRPC_ERROR_STR_DESCRIPTION); + const char *error_str = + grpc_error_get_str(error, GRPC_ERROR_STR_DESCRIPTION); close_with_status(&exec_ctx, call, status, - error_str == NULL ? "unknown error" : error_str); + error_str == NULL ? "unknown error" : error_str); grpc_error_unref(error); } if (cq != NULL) { diff --git a/src/core/lib/surface/lame_client.c b/src/core/lib/surface/lame_client.c index 7d9168f2e0b..1cf18ba0605 100644 --- a/src/core/lib/surface/lame_client.c +++ b/src/core/lib/surface/lame_client.c @@ -107,7 +107,7 @@ static void lame_start_transport_op(grpc_exec_ctx *exec_ctx, GRPC_ERROR_UNREF(op->disconnect_with_error); } -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { return GRPC_ERROR_NONE; diff --git a/src/core/lib/surface/server.c b/src/core/lib/surface/server.c index 0a0c224b4d6..165d8aa647a 100644 --- a/src/core/lib/surface/server.c +++ b/src/core/lib/surface/server.c @@ -848,7 +848,7 @@ static void channel_connectivity_changed(grpc_exec_ctx *exec_ctx, void *cd, } } -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { call_data *calld = elem->call_data; diff --git a/test/core/channel/channel_stack_test.c b/test/core/channel/channel_stack_test.c index 9c6a47eb520..d6c8a9a1428 100644 --- a/test/core/channel/channel_stack_test.c +++ b/test/core/channel/channel_stack_test.c @@ -53,7 +53,7 @@ static void channel_init_func(grpc_exec_ctx *exec_ctx, *(int *)(elem->channel_data) = 0; } -static grpc_error* call_init_func(grpc_exec_ctx *exec_ctx, +static grpc_error *call_init_func(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { ++*(int *)(elem->channel_data); @@ -134,9 +134,9 @@ static void test_create_channel_stack(void) { GPR_ASSERT(*channel_data == 0); call_stack = gpr_malloc(channel_stack->call_stack_size); - grpc_error* error = grpc_call_stack_init(&exec_ctx, channel_stack, 1, - free_call, call_stack, NULL, NULL, - call_stack); + grpc_error *error = + grpc_call_stack_init(&exec_ctx, channel_stack, 1, free_call, call_stack, + NULL, NULL, call_stack); GPR_ASSERT(error == GRPC_ERROR_NONE); GPR_ASSERT(call_stack->count == 1); call_elem = grpc_call_stack_element(call_stack, 0); diff --git a/test/core/end2end/tests/filter_call_init_fails.c b/test/core/end2end/tests/filter_call_init_fails.c index d97369fa98b..e8ceb33eb7e 100644 --- a/test/core/end2end/tests/filter_call_init_fails.c +++ b/test/core/end2end/tests/filter_call_init_fails.c @@ -200,7 +200,7 @@ static void test_request(grpc_end2end_test_config config) { * Test filter - always fails to initialize a call */ -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { return grpc_error_set_int(GRPC_ERROR_CREATE("access denied"), diff --git a/test/core/end2end/tests/filter_causes_close.c b/test/core/end2end/tests/filter_causes_close.c index 3ff0abed637..8b0c8dba61d 100644 --- a/test/core/end2end/tests/filter_causes_close.c +++ b/test/core/end2end/tests/filter_causes_close.c @@ -233,7 +233,7 @@ static void start_transport_stream_op(grpc_exec_ctx *exec_ctx, grpc_call_next_op(exec_ctx, elem, op); } -static grpc_error* init_call_elem(grpc_exec_ctx *exec_ctx, +static grpc_error *init_call_elem(grpc_exec_ctx *exec_ctx, grpc_call_element *elem, grpc_call_element_args *args) { return GRPC_ERROR_NONE; From abd285aed813ff55329e76a9dc19eb3a1bc86166 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Thu, 23 Jun 2016 13:30:19 -0700 Subject: [PATCH 0213/1039] Added missing todo and moved _service To Implementation --- .../objective-c/route_guide/ViewControllers.m | 28 ++++++++++--------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/examples/objective-c/route_guide/ViewControllers.m b/examples/objective-c/route_guide/ViewControllers.m index a5c03e9cfca..26ca9d6220b 100644 --- a/examples/objective-c/route_guide/ViewControllers.m +++ b/examples/objective-c/route_guide/ViewControllers.m @@ -80,13 +80,14 @@ static NSString * const kHostAddress = @"localhost:50051"; * Run the getFeature demo. Calls getFeature with a point known to have a feature and a point known * not to have a feature. */ -@interface GetFeatureViewController : UIViewController { - RTGRouteGuide *_service; -} +@interface GetFeatureViewController : UIViewController + @property (weak, nonatomic) IBOutlet UILabel *outputLabel; + @end @implementation GetFeatureViewController +RTGRouteGuide *_service; - (void)execRequest { void (^handler)(RTGFeature *response, NSError *error) = ^(RTGFeature *response, NSError *error) { @@ -139,14 +140,14 @@ static NSString * const kHostAddress = @"localhost:50051"; * Run the listFeatures demo. Calls listFeatures with a rectangle containing all of the features in * the pre-generated database. Prints each response as it comes in. */ -@interface ListFeaturesViewController : UIViewController { - RTGRouteGuide *_service; -} +@interface ListFeaturesViewController : UIViewController + @property (weak, nonatomic) IBOutlet UILabel *outputLabel; @end @implementation ListFeaturesViewController +RTGRouteGuide *_service; - (void)execRequest { RTGRectangle *rectangle = [RTGRectangle message]; @@ -193,14 +194,14 @@ static NSString * const kHostAddress = @"localhost:50051"; * database with a variable delay in between. Prints the statistics when they are sent from the * server. */ -@interface RecordRouteViewController : UIViewController { - RTGRouteGuide *_service; -} +@interface RecordRouteViewController : UIViewController + @property (weak, nonatomic) IBOutlet UILabel *outputLabel; @end @implementation RecordRouteViewController +RTGRouteGuide *_service; - (void)execRequest { NSString *dataBasePath = [NSBundle.mainBundle pathForResource:@"route_guide_db" @@ -233,7 +234,7 @@ static NSString * const kHostAddress = @"localhost:50051"; NSLog(@"It took %i seconds", response.elapsedTime); } else { NSString *str =[NSString stringWithFormat:@"%@\nRPC error: %@", self.outputLabel.text, error]; - self.outputLabel.text = str; + self.outputLabel.text = str; NSLog(@"RPC error: %@", error); } }]; @@ -261,14 +262,14 @@ static NSString * const kHostAddress = @"localhost:50051"; * Run the routeChat demo. Send some chat messages, and print any chat messages that are sent from * the server. */ -@interface RouteChatViewController : UIViewController { - RTGRouteGuide *_service; -} +@interface RouteChatViewController : UIViewController + @property (weak, nonatomic) IBOutlet UILabel *outputLabel; @end @implementation RouteChatViewController +RTGRouteGuide *_service; - (void)execRequest { NSArray *notes = @[[RTGRouteNote noteWithMessage:@"First message" latitude:0 longitude:0], @@ -305,6 +306,7 @@ static NSString * const kHostAddress = @"localhost:50051"; } - (void)viewDidAppear:(BOOL)animated { + // TODO(makarandd): Set these properties through UI builder self.outputLabel.text = @"RPC log:"; self.outputLabel.numberOfLines = 0; self.outputLabel.font = [UIFont fontWithName:@"Helvetica Neue" size:8.0]; From f0f70a8a68c107f9fcff2dd35867fd762c2c13d6 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 23 Jun 2016 13:55:06 -0700 Subject: [PATCH 0214/1039] Make transport-level errors be reflected in status messages on calls Allows us to eliminate logging those errors by default (since they are explicitly passed up to the application). Required plumbing errors through the stack a little more deeply than we had previously. --- .../client_config/subchannel_call_holder.c | 7 +- .../chttp2/transport/chttp2_transport.c | 229 +++++++++++------- .../ext/transport/chttp2/transport/internal.h | 3 - src/core/lib/channel/channel_stack.c | 2 +- src/core/lib/iomgr/error.c | 14 ++ src/core/lib/iomgr/error.h | 16 +- .../security/transport/client_auth_filter.c | 3 +- src/core/lib/surface/call.c | 105 ++++---- src/core/lib/surface/completion_queue.c | 2 +- src/core/lib/transport/transport.c | 65 ++--- src/core/lib/transport/transport.h | 9 +- src/core/lib/transport/transport_op_string.c | 15 +- 12 files changed, 279 insertions(+), 191 deletions(-) diff --git a/src/core/ext/client_config/subchannel_call_holder.c b/src/core/ext/client_config/subchannel_call_holder.c index e31800edd98..b96a0ad0937 100644 --- a/src/core/ext/client_config/subchannel_call_holder.c +++ b/src/core/ext/client_config/subchannel_call_holder.c @@ -120,16 +120,13 @@ retry: return; } /* if this is a cancellation, then we can raise our cancelled flag */ - if (op->cancel_with_status != GRPC_STATUS_OK) { + if (op->cancel_error != GRPC_ERROR_NONE) { if (!gpr_atm_rel_cas(&holder->subchannel_call, 0, 1)) { goto retry; } else { switch (holder->creation_phase) { case GRPC_SUBCHANNEL_CALL_HOLDER_NOT_CREATING: - fail_locked(exec_ctx, holder, - grpc_error_set_int(GRPC_ERROR_CREATE("Cancelled"), - GRPC_ERROR_INT_GRPC_STATUS, - op->cancel_with_status)); + fail_locked(exec_ctx, holder, GRPC_ERROR_REF(op->cancel_error)); break; case GRPC_SUBCHANNEL_CALL_HOLDER_PICKING_SUBCHANNEL: holder->pick_subchannel(exec_ctx, holder->pick_subchannel_arg, NULL, diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index 9c99ff883af..f34ab98b036 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -106,14 +106,12 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, static void cancel_from_api(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global, - grpc_status_code status, - gpr_slice *optional_message); + grpc_error *error); static void close_from_api(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global, - grpc_status_code status, - gpr_slice *optional_message); + grpc_error *error); /** Add endpoint from this transport to pollset */ static void add_to_pollset_locked(grpc_exec_ctx *exec_ctx, @@ -163,8 +161,6 @@ static void destruct_transport(grpc_exec_ctx *exec_ctx, GPR_ASSERT(t->ep == NULL); - gpr_slice_unref(t->optional_drop_message); - gpr_slice_buffer_destroy(&t->global.qbuf); gpr_slice_buffer_destroy(&t->writing.outbuf); @@ -266,7 +262,6 @@ static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, is_client ? GRPC_DTS_FH_0 : GRPC_DTS_CLIENT_PREFIX_0; t->parsing.is_first_frame = true; t->writing.is_client = is_client; - t->optional_drop_message = gpr_empty_slice(); grpc_connectivity_state_init( &t->channel_callback.state_tracker, GRPC_CHANNEL_READY, is_client ? "client_transport" : "server_transport"); @@ -876,7 +871,9 @@ static void maybe_start_some_streams( grpc_chttp2_list_pop_waiting_for_concurrency(transport_global, &stream_global)) { cancel_from_api(exec_ctx, transport_global, stream_global, - GRPC_STATUS_UNAVAILABLE, NULL); + grpc_error_set_int( + GRPC_ERROR_CREATE("Stream IDs exhausted"), + GRPC_ERROR_INT_GRPC_STATUS, GRPC_STATUS_UNAVAILABLE)); } } @@ -958,14 +955,14 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, on_complete->next_data.scratch |= CLOSURE_BARRIER_STATS_BIT; } - if (op->cancel_with_status != GRPC_STATUS_OK) { + if (op->cancel_error != GRPC_ERROR_NONE) { cancel_from_api(exec_ctx, transport_global, stream_global, - op->cancel_with_status, op->optional_close_message); + GRPC_ERROR_REF(op->cancel_error)); } - if (op->close_with_status != GRPC_STATUS_OK) { + if (op->close_error != GRPC_ERROR_NONE) { close_from_api(exec_ctx, transport_global, stream_global, - op->close_with_status, op->optional_close_message); + GRPC_ERROR_REF(op->close_error)); } if (op->send_initial_metadata != NULL) { @@ -979,12 +976,16 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, transport_global->settings[GRPC_PEER_SETTINGS] [GRPC_CHTTP2_SETTINGS_MAX_HEADER_LIST_SIZE]; if (metadata_size > metadata_peer_limit) { - gpr_log(GPR_DEBUG, - "to-be-sent initial metadata size exceeds peer limit " - "(%" PRIuPTR " vs. %" PRIuPTR ")", - metadata_size, metadata_peer_limit); - cancel_from_api(exec_ctx, transport_global, stream_global, - GRPC_STATUS_RESOURCE_EXHAUSTED, NULL); + cancel_from_api( + exec_ctx, transport_global, stream_global, + grpc_error_set_int( + grpc_error_set_int( + grpc_error_set_int( + GRPC_ERROR_CREATE("to-be-sent initial metadata size " + "exceeds peer limit"), + GRPC_ERROR_INT_SIZE, (intptr_t)metadata_size), + GRPC_ERROR_INT_LIMIT, (intptr_t)metadata_peer_limit), + GRPC_ERROR_INT_GRPC_STATUS, GRPC_STATUS_RESOURCE_EXHAUSTED)); } else { if (contains_non_ok_status(transport_global, op->send_initial_metadata)) { stream_global->seen_error = true; @@ -1038,12 +1039,16 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, transport_global->settings[GRPC_PEER_SETTINGS] [GRPC_CHTTP2_SETTINGS_MAX_HEADER_LIST_SIZE]; if (metadata_size > metadata_peer_limit) { - gpr_log(GPR_DEBUG, - "to-be-sent trailing metadata size exceeds peer limit " - "(%" PRIuPTR " vs. %" PRIuPTR ")", - metadata_size, metadata_peer_limit); - cancel_from_api(exec_ctx, transport_global, stream_global, - GRPC_STATUS_RESOURCE_EXHAUSTED, NULL); + cancel_from_api( + exec_ctx, transport_global, stream_global, + grpc_error_set_int( + grpc_error_set_int( + grpc_error_set_int( + GRPC_ERROR_CREATE("to-be-sent trailing metadata size " + "exceeds peer limit"), + GRPC_ERROR_INT_SIZE, (intptr_t)metadata_size), + GRPC_ERROR_INT_LIMIT, (intptr_t)metadata_peer_limit), + GRPC_ERROR_INT_GRPC_STATUS, GRPC_STATUS_RESOURCE_EXHAUSTED)); } else { if (contains_non_ok_status(transport_global, op->send_trailing_metadata)) { @@ -1235,8 +1240,12 @@ static void check_read_ops(grpc_exec_ctx *exec_ctx, incoming_byte_stream_destroy_locked(exec_ctx, NULL, NULL, bs); } if (stream_global->exceeded_metadata_size) { - cancel_from_api(exec_ctx, transport_global, stream_global, - GRPC_STATUS_RESOURCE_EXHAUSTED, NULL); + cancel_from_api( + exec_ctx, transport_global, stream_global, + grpc_error_set_int( + GRPC_ERROR_CREATE( + "received initial metadata size exceeds limit"), + GRPC_ERROR_INT_GRPC_STATUS, GRPC_STATUS_RESOURCE_EXHAUSTED)); } } grpc_chttp2_incoming_metadata_buffer_publish( @@ -1275,8 +1284,12 @@ static void check_read_ops(grpc_exec_ctx *exec_ctx, incoming_byte_stream_destroy_locked(exec_ctx, NULL, NULL, bs); } if (stream_global->exceeded_metadata_size) { - cancel_from_api(exec_ctx, transport_global, stream_global, - GRPC_STATUS_RESOURCE_EXHAUSTED, NULL); + cancel_from_api( + exec_ctx, transport_global, stream_global, + grpc_error_set_int( + GRPC_ERROR_CREATE( + "received trailing metadata size exceeds limit"), + GRPC_ERROR_INT_GRPC_STATUS, GRPC_STATUS_RESOURCE_EXHAUSTED)); } } if (stream_global->all_incoming_byte_streams_finished) { @@ -1340,35 +1353,67 @@ static void remove_stream(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, GRPC_ERROR_UNREF(error); } +static void status_codes_from_error(grpc_error *error, + grpc_chttp2_error_code *http2_error, + grpc_status_code *grpc_status) { + intptr_t ip_http; + intptr_t ip_grpc; + bool have_http = + grpc_error_get_int(error, GRPC_ERROR_INT_HTTP2_ERROR, &ip_http); + bool have_grpc = + grpc_error_get_int(error, GRPC_ERROR_INT_GRPC_STATUS, &ip_grpc); + if (have_http) { + *http2_error = (grpc_chttp2_error_code)ip_http; + } else if (have_grpc) { + *http2_error = + grpc_chttp2_grpc_status_to_http2_error((grpc_status_code)ip_grpc); + } else { + *http2_error = GRPC_CHTTP2_INTERNAL_ERROR; + } + if (have_grpc) { + *grpc_status = (grpc_status_code)ip_grpc; + } else if (have_http) { + *grpc_status = + grpc_chttp2_http2_error_to_grpc_status((grpc_chttp2_error_code)ip_http); + } else { + *grpc_status = GRPC_STATUS_INTERNAL; + } +} + static void cancel_from_api(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global, - grpc_status_code status, - gpr_slice *optional_message) { + grpc_error *due_to_error) { if (!stream_global->read_closed || !stream_global->write_closed) { + grpc_status_code grpc_status; + grpc_chttp2_error_code http_error; + status_codes_from_error(due_to_error, &http_error, &grpc_status); + if (stream_global->id != 0) { gpr_slice_buffer_add( &transport_global->qbuf, - grpc_chttp2_rst_stream_create( - stream_global->id, - (uint32_t)grpc_chttp2_grpc_status_to_http2_error(status), - &stream_global->stats.outgoing)); + grpc_chttp2_rst_stream_create(stream_global->id, (uint32_t)http_error, + &stream_global->stats.outgoing)); } - if (optional_message) { - gpr_slice_ref(*optional_message); + const char *msg = + grpc_error_get_str(due_to_error, GRPC_ERROR_STR_GRPC_MESSAGE); + bool free_msg = false; + if (msg == NULL) { + free_msg = true; + msg = grpc_error_string(due_to_error); } - grpc_chttp2_fake_status(exec_ctx, transport_global, stream_global, status, - optional_message); + gpr_slice msg_slice = gpr_slice_from_copied_string(msg); + grpc_chttp2_fake_status(exec_ctx, transport_global, stream_global, + grpc_status, &msg_slice); + if (free_msg) grpc_error_free_string(msg); } - if (status != GRPC_STATUS_OK && !stream_global->seen_error) { + if (due_to_error != GRPC_ERROR_NONE && !stream_global->seen_error) { stream_global->seen_error = true; grpc_chttp2_list_add_check_read_ops(transport_global, stream_global); } - grpc_chttp2_mark_stream_closed( - exec_ctx, transport_global, stream_global, 1, 1, - grpc_error_set_int(GRPC_ERROR_CREATE("Cancelled"), - GRPC_ERROR_INT_GRPC_STATUS, status)); + grpc_chttp2_mark_stream_closed(exec_ctx, transport_global, stream_global, 1, + 1, due_to_error); } void grpc_chttp2_fake_status(grpc_exec_ctx *exec_ctx, @@ -1469,15 +1514,17 @@ void grpc_chttp2_mark_stream_closed( static void close_from_api(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global, - grpc_status_code status, - gpr_slice *optional_message) { + grpc_error *error) { gpr_slice hdr; gpr_slice status_hdr; gpr_slice message_pfx; uint8_t *p; uint32_t len = 0; + grpc_status_code grpc_status; + grpc_chttp2_error_code http_error; + status_codes_from_error(error, &http_error, &grpc_status); - GPR_ASSERT(status >= 0 && (int)status < 100); + GPR_ASSERT(grpc_status >= 0 && (int)grpc_status < 100); if (stream_global->id != 0 && !transport_global->is_client) { /* Hand roll a header block. @@ -1487,7 +1534,7 @@ static void close_from_api(grpc_exec_ctx *exec_ctx, time we got around to sending this, so instead we ignore HPACK compression and just write the uncompressed bytes onto the wire. */ - status_hdr = gpr_slice_malloc(15 + (status >= 10)); + status_hdr = gpr_slice_malloc(15 + (grpc_status >= 10)); p = GPR_SLICE_START_PTR(status_hdr); *p++ = 0x40; /* literal header */ *p++ = 11; /* len(grpc-status) */ @@ -1502,19 +1549,23 @@ static void close_from_api(grpc_exec_ctx *exec_ctx, *p++ = 't'; *p++ = 'u'; *p++ = 's'; - if (status < 10) { + if (grpc_status < 10) { *p++ = 1; - *p++ = (uint8_t)('0' + status); + *p++ = (uint8_t)('0' + grpc_status); } else { *p++ = 2; - *p++ = (uint8_t)('0' + (status / 10)); - *p++ = (uint8_t)('0' + (status % 10)); + *p++ = (uint8_t)('0' + (grpc_status / 10)); + *p++ = (uint8_t)('0' + (grpc_status % 10)); } GPR_ASSERT(p == GPR_SLICE_END_PTR(status_hdr)); len += (uint32_t)GPR_SLICE_LENGTH(status_hdr); - if (optional_message) { - GPR_ASSERT(GPR_SLICE_LENGTH(*optional_message) < 127); + const char *optional_message = + grpc_error_get_str(error, GRPC_ERROR_STR_GRPC_MESSAGE); + + if (optional_message != NULL) { + size_t msg_len = strlen(optional_message); + GPR_ASSERT(msg_len < 127); message_pfx = gpr_slice_malloc(15); p = GPR_SLICE_START_PTR(message_pfx); *p++ = 0x40; @@ -1531,10 +1582,10 @@ static void close_from_api(grpc_exec_ctx *exec_ctx, *p++ = 'a'; *p++ = 'g'; *p++ = 'e'; - *p++ = (uint8_t)GPR_SLICE_LENGTH(*optional_message); + *p++ = (uint8_t)msg_len; GPR_ASSERT(p == GPR_SLICE_END_PTR(message_pfx)); len += (uint32_t)GPR_SLICE_LENGTH(message_pfx); - len += (uint32_t)GPR_SLICE_LENGTH(*optional_message); + len += (uint32_t)msg_len; } hdr = gpr_slice_malloc(9); @@ -1555,53 +1606,54 @@ static void close_from_api(grpc_exec_ctx *exec_ctx, if (optional_message) { gpr_slice_buffer_add(&transport_global->qbuf, message_pfx); gpr_slice_buffer_add(&transport_global->qbuf, - gpr_slice_ref(*optional_message)); + gpr_slice_from_copied_string(optional_message)); } gpr_slice_buffer_add( &transport_global->qbuf, grpc_chttp2_rst_stream_create(stream_global->id, GRPC_CHTTP2_NO_ERROR, &stream_global->stats.outgoing)); - - if (optional_message) { - gpr_slice_ref(*optional_message); - } } - grpc_chttp2_fake_status(exec_ctx, transport_global, stream_global, status, - optional_message); - grpc_error *err = GRPC_ERROR_CREATE("Stream closed"); - err = grpc_error_set_int(err, GRPC_ERROR_INT_GRPC_STATUS, status); - if (optional_message) { - char *str = - gpr_dump_slice(*optional_message, GPR_DUMP_HEX | GPR_DUMP_ASCII); - err = grpc_error_set_str(err, GRPC_ERROR_STR_GRPC_MESSAGE, str); - gpr_free(str); + const char *msg = grpc_error_get_str(error, GRPC_ERROR_STR_GRPC_MESSAGE); + bool free_msg = false; + if (msg == NULL) { + free_msg = true; + msg = grpc_error_string(error); } + gpr_slice msg_slice = gpr_slice_from_copied_string(msg); + grpc_chttp2_fake_status(exec_ctx, transport_global, stream_global, + grpc_status, &msg_slice); + if (free_msg) grpc_error_free_string(msg); + grpc_chttp2_mark_stream_closed(exec_ctx, transport_global, stream_global, 1, - 1, err); + 1, error); } +typedef struct { + grpc_exec_ctx *exec_ctx; + grpc_error *error; +} cancel_stream_cb_args; + static void cancel_stream_cb(grpc_chttp2_transport_global *transport_global, void *user_data, grpc_chttp2_stream_global *stream_global) { - grpc_chttp2_transport *transport = TRANSPORT_FROM_GLOBAL(transport_global); - cancel_from_api(user_data, transport_global, stream_global, - GRPC_STATUS_UNAVAILABLE, - GPR_SLICE_IS_EMPTY(transport->optional_drop_message) - ? NULL - : &transport->optional_drop_message); + cancel_stream_cb_args *args = user_data; + cancel_from_api(args->exec_ctx, transport_global, stream_global, + GRPC_ERROR_REF(args->error)); } -static void end_all_the_calls(grpc_exec_ctx *exec_ctx, - grpc_chttp2_transport *t) { - grpc_chttp2_for_all_streams(&t->global, exec_ctx, cancel_stream_cb); +static void end_all_the_calls(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, + grpc_error *error) { + cancel_stream_cb_args args = {exec_ctx, error}; + grpc_chttp2_for_all_streams(&t->global, &args, cancel_stream_cb); + GRPC_ERROR_UNREF(error); } static void drop_connection(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, grpc_error *error) { - close_transport_locked(exec_ctx, t, error); - end_all_the_calls(exec_ctx, t); + close_transport_locked(exec_ctx, t, GRPC_ERROR_REF(error)); + end_all_the_calls(exec_ctx, t, error); } /** update window from a settings change */ @@ -1708,15 +1760,7 @@ static void parsing_action(grpc_exec_ctx *exec_ctx, void *arg, t->read_buffer.slices[i]); }; if (i != t->read_buffer.count) { - gpr_slice_unref(t->optional_drop_message); errors[2] = try_http_parsing(exec_ctx, t); - if (errors[2] != GRPC_ERROR_NONE) { - t->optional_drop_message = gpr_slice_from_copied_string( - "Connection dropped: received http1.x response"); - } else { - t->optional_drop_message = gpr_slice_from_copied_string( - "Connection dropped: received unparseable response"); - } } grpc_error *err = errors[0] == GRPC_ERROR_NONE && errors[1] == GRPC_ERROR_NONE && @@ -1784,6 +1828,10 @@ static void post_reading_action_locked(grpc_exec_ctx *exec_ctx, error = GRPC_ERROR_CREATE("Transport closed"); } if (error != GRPC_ERROR_NONE) { + if (!grpc_error_get_int(error, GRPC_ERROR_INT_GRPC_STATUS, NULL)) { + error = grpc_error_set_int(error, GRPC_ERROR_INT_GRPC_STATUS, + GRPC_STATUS_UNAVAILABLE); + } drop_connection(exec_ctx, t, GRPC_ERROR_REF(error)); t->endpoint_reading = 0; if (!t->executor.writing_active && t->ep) { @@ -1798,6 +1846,7 @@ static void post_reading_action_locked(grpc_exec_ctx *exec_ctx, prevent_endpoint_shutdown(t); } gpr_slice_buffer_reset_and_unref(&t->read_buffer); + GRPC_ERROR_UNREF(error); if (keep_reading) { grpc_endpoint_read(exec_ctx, t->ep, &t->read_buffer, &t->reading_action); @@ -1806,8 +1855,6 @@ static void post_reading_action_locked(grpc_exec_ctx *exec_ctx, } else { UNREF_TRANSPORT(exec_ctx, t, "reading_action"); } - - GRPC_LOG_IF_ERROR("close_transport", error); } /******************************************************************************* diff --git a/src/core/ext/transport/chttp2/transport/internal.h b/src/core/ext/transport/chttp2/transport/internal.h index d63170e350b..7e281f1b1a2 100644 --- a/src/core/ext/transport/chttp2/transport/internal.h +++ b/src/core/ext/transport/chttp2/transport/internal.h @@ -384,9 +384,6 @@ struct grpc_chttp2_transport { /** Transport op to be applied post-parsing */ grpc_transport_op *post_parsing_op; - - /** Message explaining the reason of dropping connection */ - gpr_slice optional_drop_message; }; typedef struct { diff --git a/src/core/lib/channel/channel_stack.c b/src/core/lib/channel/channel_stack.c index bbba85d80ba..42075b127bd 100644 --- a/src/core/lib/channel/channel_stack.c +++ b/src/core/lib/channel/channel_stack.c @@ -263,6 +263,6 @@ void grpc_call_element_send_cancel(grpc_exec_ctx *exec_ctx, grpc_call_element *cur_elem) { grpc_transport_stream_op op; memset(&op, 0, sizeof(op)); - op.cancel_with_status = GRPC_STATUS_CANCELLED; + op.cancel_error = GRPC_ERROR_CANCELLED; grpc_call_next_op(exec_ctx, cur_elem, &op); } diff --git a/src/core/lib/iomgr/error.c b/src/core/lib/iomgr/error.c index 540fb4fa7e4..da0d6972360 100644 --- a/src/core/lib/iomgr/error.c +++ b/src/core/lib/iomgr/error.c @@ -37,6 +37,7 @@ #include #include +#include #include #include #include @@ -115,6 +116,8 @@ static const char *error_int_name(grpc_error_ints key) { return "wsa_error"; case GRPC_ERROR_INT_HTTP_STATUS: return "http_status"; + case GRPC_ERROR_INT_LIMIT: + return "limit"; } GPR_UNREACHABLE_CODE(return "unknown"); } @@ -271,6 +274,12 @@ grpc_error *grpc_error_set_int(grpc_error *src, grpc_error_ints which, bool grpc_error_get_int(grpc_error *err, grpc_error_ints which, intptr_t *p) { void *pp; + if (is_special(err)) { + if (err == GRPC_ERROR_CANCELLED && which == GRPC_ERROR_INT_GRPC_STATUS) { + return GRPC_STATUS_CANCELLED; + } + return false; + } if (gpr_avl_maybe_get(err->ints, (void *)(uintptr_t)which, &pp)) { if (p != NULL) *p = (intptr_t)pp; return true; @@ -286,6 +295,11 @@ grpc_error *grpc_error_set_str(grpc_error *src, grpc_error_strs which, return new; } +const char *grpc_error_get_str(grpc_error *err, grpc_error_strs which) { + if (is_special(err)) return NULL; + return gpr_avl_get(err->strs, (void *)(uintptr_t)which); +} + grpc_error *grpc_error_add_child(grpc_error *src, grpc_error *child) { grpc_error *new = copy_error_and_unref(src); new->errs = gpr_avl_add(new->errs, (void *)(new->next_err++), child); diff --git a/src/core/lib/iomgr/error.h b/src/core/lib/iomgr/error.h index 69cdf3028e4..13f898e31ad 100644 --- a/src/core/lib/iomgr/error.h +++ b/src/core/lib/iomgr/error.h @@ -92,6 +92,8 @@ typedef enum { GRPC_ERROR_INT_FD, /// HTTP status (i.e. 404) GRPC_ERROR_INT_HTTP_STATUS, + /// context sensitive limit associated with the error + GRPC_ERROR_INT_LIMIT, } grpc_error_ints; typedef enum { @@ -163,23 +165,25 @@ void grpc_error_unref(grpc_error *err); #endif grpc_error *grpc_error_set_int(grpc_error *src, grpc_error_ints which, - intptr_t value); + intptr_t value) GRPC_MUST_USE_RESULT; bool grpc_error_get_int(grpc_error *error, grpc_error_ints which, intptr_t *p); grpc_error *grpc_error_set_time(grpc_error *src, grpc_error_times which, - gpr_timespec value); + gpr_timespec value) GRPC_MUST_USE_RESULT; grpc_error *grpc_error_set_str(grpc_error *src, grpc_error_strs which, - const char *value); + const char *value) GRPC_MUST_USE_RESULT; +const char *grpc_error_get_str(grpc_error *error, grpc_error_strs which); /// Add a child error: an error that is believed to have contributed to this /// error occurring. Allows root causing high level errors from lower level /// errors that contributed to them. -grpc_error *grpc_error_add_child(grpc_error *src, grpc_error *child); +grpc_error *grpc_error_add_child(grpc_error *src, + grpc_error *child) GRPC_MUST_USE_RESULT; grpc_error *grpc_os_error(const char *file, int line, int err, - const char *call_name); + const char *call_name) GRPC_MUST_USE_RESULT; /// create an error associated with errno!=0 (an 'operating system' error) #define GRPC_OS_ERROR(err, call_name) \ grpc_os_error(__FILE__, __LINE__, err, call_name) grpc_error *grpc_wsa_error(const char *file, int line, int err, - const char *call_name); + const char *call_name) GRPC_MUST_USE_RESULT; /// windows only: create an error associated with WSAGetLastError()!=0 #define GRPC_WSA_ERROR(err, call_name) \ grpc_wsa_error(__FILE__, __LINE__, err, call_name) diff --git a/src/core/lib/security/transport/client_auth_filter.c b/src/core/lib/security/transport/client_auth_filter.c index 76be2acd728..e053afc745c 100644 --- a/src/core/lib/security/transport/client_auth_filter.c +++ b/src/core/lib/security/transport/client_auth_filter.c @@ -220,8 +220,7 @@ static void auth_start_transport_op(grpc_exec_ctx *exec_ctx, grpc_linked_mdelem *l; grpc_client_security_context *sec_ctx = NULL; - if (calld->security_context_set == 0 && - op->cancel_with_status == GRPC_STATUS_OK) { + if (calld->security_context_set == 0 && op->cancel_error == GRPC_ERROR_NONE) { calld->security_context_set = 1; GPR_ASSERT(op->context); if (op->context[GRPC_CONTEXT_SECURITY].value == NULL) { diff --git a/src/core/lib/surface/call.c b/src/core/lib/surface/call.c index 04291b0ee0a..77c17a4975b 100644 --- a/src/core/lib/surface/call.c +++ b/src/core/lib/surface/call.c @@ -402,8 +402,50 @@ static void set_status_code(grpc_call *call, status_source source, call->status[source].is_set = 1; call->status[source].code = (grpc_status_code)status; +} + +static void set_status_details(grpc_call *call, status_source source, + grpc_mdstr *status) { + if (call->status[source].details != NULL) { + GRPC_MDSTR_UNREF(call->status[source].details); + } + call->status[source].details = status; +} + +static void get_final_status(grpc_call *call, + void (*set_value)(grpc_status_code code, + void *user_data), + void *set_value_user_data) { + int i; + for (i = 0; i < STATUS_SOURCE_COUNT; i++) { + if (call->status[i].is_set) { + set_value(call->status[i].code, set_value_user_data); + return; + } + } + if (call->is_client) { + set_value(GRPC_STATUS_UNKNOWN, set_value_user_data); + } else { + set_value(GRPC_STATUS_OK, set_value_user_data); + } +} - /* TODO(ctiller): what to do about the flush that was previously here */ +static void set_status_from_error(grpc_call *call, status_source source, + grpc_error *error) { + intptr_t status; + if (grpc_error_get_int(error, GRPC_ERROR_INT_GRPC_STATUS, &status)) { + set_status_code(call, source, (uint32_t)status); + } else { + set_status_code(call, source, GRPC_STATUS_INTERNAL); + } + const char *msg = grpc_error_get_str(error, GRPC_ERROR_STR_GRPC_MESSAGE); + bool free_msg = false; + if (msg == NULL) { + free_msg = true; + msg = grpc_error_string(error); + } + set_status_details(call, source, grpc_mdstr_from_string(msg)); + if (free_msg) grpc_error_free_string(msg); } static void set_incoming_compression_algorithm( @@ -492,32 +534,6 @@ uint32_t grpc_call_test_only_get_encodings_accepted_by_peer(grpc_call *call) { return encodings_accepted_by_peer; } -static void set_status_details(grpc_call *call, status_source source, - grpc_mdstr *status) { - if (call->status[source].details != NULL) { - GRPC_MDSTR_UNREF(call->status[source].details); - } - call->status[source].details = status; -} - -static void get_final_status(grpc_call *call, - void (*set_value)(grpc_status_code code, - void *user_data), - void *set_value_user_data) { - int i; - for (i = 0; i < STATUS_SOURCE_COUNT; i++) { - if (call->status[i].is_set) { - set_value(call->status[i].code, set_value_user_data); - return; - } - } - if (call->is_client) { - set_value(GRPC_STATUS_UNKNOWN, set_value_user_data); - } else { - set_value(GRPC_STATUS_OK, set_value_user_data); - } -} - static void get_final_details(grpc_call *call, char **out_details, size_t *out_details_capacity) { int i; @@ -741,8 +757,7 @@ grpc_call_error grpc_call_cancel_with_status(grpc_call *c, typedef struct termination_closure { grpc_closure closure; grpc_call *call; - grpc_status_code status; - gpr_slice optional_message; + grpc_error *error; grpc_closure *op_closure; enum { TC_CANCEL, TC_CLOSE } type; } termination_closure; @@ -758,7 +773,7 @@ static void done_termination(grpc_exec_ctx *exec_ctx, void *tcp, GRPC_CALL_INTERNAL_UNREF(exec_ctx, tc->call, "close"); break; } - gpr_slice_unref(tc->optional_message); + GRPC_ERROR_UNREF(tc->error); grpc_exec_ctx_sched(exec_ctx, tc->op_closure, GRPC_ERROR_NONE, NULL); gpr_free(tc); } @@ -767,7 +782,7 @@ static void send_cancel(grpc_exec_ctx *exec_ctx, void *tcp, grpc_error *error) { grpc_transport_stream_op op; termination_closure *tc = tcp; memset(&op, 0, sizeof(op)); - op.cancel_with_status = tc->status; + op.cancel_error = tc->error; /* reuse closure to catch completion */ grpc_closure_init(&tc->closure, done_termination, tc); op.on_complete = &tc->closure; @@ -778,8 +793,7 @@ static void send_close(grpc_exec_ctx *exec_ctx, void *tcp, grpc_error *error) { grpc_transport_stream_op op; termination_closure *tc = tcp; memset(&op, 0, sizeof(op)); - tc->optional_message = gpr_slice_ref(tc->optional_message); - grpc_transport_stream_op_add_close(&op, tc->status, &tc->optional_message); + op.close_error = tc->error; /* reuse closure to catch completion */ grpc_closure_init(&tc->closure, done_termination, tc); tc->op_closure = op.on_complete; @@ -789,14 +803,7 @@ static void send_close(grpc_exec_ctx *exec_ctx, void *tcp, grpc_error *error) { static grpc_call_error terminate_with_status(grpc_exec_ctx *exec_ctx, termination_closure *tc) { - grpc_mdstr *details = NULL; - if (GPR_SLICE_LENGTH(tc->optional_message) > 0) { - tc->optional_message = gpr_slice_ref(tc->optional_message); - details = grpc_mdstr_from_slice(tc->optional_message); - } - - set_status_code(tc->call, STATUS_FROM_API_OVERRIDE, (uint32_t)tc->status); - set_status_details(tc->call, STATUS_FROM_API_OVERRIDE, details); + set_status_from_error(tc->call, STATUS_FROM_API_OVERRIDE, tc->error); if (tc->type == TC_CANCEL) { grpc_closure_init(&tc->closure, send_cancel, tc); @@ -812,13 +819,15 @@ static grpc_call_error terminate_with_status(grpc_exec_ctx *exec_ctx, static grpc_call_error cancel_with_status(grpc_exec_ctx *exec_ctx, grpc_call *c, grpc_status_code status, const char *description) { + GPR_ASSERT(status != GRPC_STATUS_OK); termination_closure *tc = gpr_malloc(sizeof(*tc)); memset(tc, 0, sizeof(termination_closure)); tc->type = TC_CANCEL; tc->call = c; - tc->optional_message = gpr_slice_from_copied_string(description); - GPR_ASSERT(status != GRPC_STATUS_OK); - tc->status = status; + tc->error = grpc_error_set_int( + grpc_error_set_str(GRPC_ERROR_CREATE(description), + GRPC_ERROR_STR_GRPC_MESSAGE, description), + GRPC_ERROR_INT_GRPC_STATUS, status); return terminate_with_status(exec_ctx, tc); } @@ -826,13 +835,15 @@ static grpc_call_error cancel_with_status(grpc_exec_ctx *exec_ctx, grpc_call *c, static grpc_call_error close_with_status(grpc_exec_ctx *exec_ctx, grpc_call *c, grpc_status_code status, const char *description) { + GPR_ASSERT(status != GRPC_STATUS_OK); termination_closure *tc = gpr_malloc(sizeof(*tc)); memset(tc, 0, sizeof(termination_closure)); tc->type = TC_CLOSE; tc->call = c; - tc->optional_message = gpr_slice_from_copied_string(description); - GPR_ASSERT(status != GRPC_STATUS_OK); - tc->status = status; + tc->error = grpc_error_set_int( + grpc_error_set_str(GRPC_ERROR_CREATE(description), + GRPC_ERROR_STR_GRPC_MESSAGE, description), + GRPC_ERROR_INT_GRPC_STATUS, status); return terminate_with_status(exec_ctx, tc); } diff --git a/src/core/lib/surface/completion_queue.c b/src/core/lib/surface/completion_queue.c index 2cc6aa74e02..db8010ef9a1 100644 --- a/src/core/lib/surface/completion_queue.c +++ b/src/core/lib/surface/completion_queue.c @@ -240,7 +240,7 @@ void grpc_cq_end_op(grpc_exec_ctx *exec_ctx, grpc_completion_queue *cc, "grpc_cq_end_op(exec_ctx=%p, cc=%p, tag=%p, error=%s, done=%p, " "done_arg=%p, storage=%p)", 7, (exec_ctx, cc, tag, errmsg, done, done_arg, storage)); - if (grpc_trace_operation_failures) { + if (grpc_trace_operation_failures && error != GRPC_ERROR_NONE) { gpr_log(GPR_ERROR, "Operation failed: tag=%p, error=%s", tag, errmsg); } grpc_error_free_string(errmsg); diff --git a/src/core/lib/transport/transport.c b/src/core/lib/transport/transport.c index 1105494a85c..67920b05279 100644 --- a/src/core/lib/transport/transport.c +++ b/src/core/lib/transport/transport.c @@ -36,6 +36,7 @@ #include #include #include +#include "src/core/lib/support/string.h" #include "src/core/lib/transport/transport_impl.h" #ifdef GRPC_STREAM_REFCOUNT_DEBUG @@ -162,55 +163,63 @@ void grpc_transport_stream_op_finish_with_failure(grpc_exec_ctx *exec_ctx, grpc_exec_ctx_sched(exec_ctx, op->on_complete, error, NULL); } -void grpc_transport_stream_op_add_cancellation(grpc_transport_stream_op *op, - grpc_status_code status) { - GPR_ASSERT(status != GRPC_STATUS_OK); - if (op->cancel_with_status == GRPC_STATUS_OK) { - op->cancel_with_status = status; - } - if (op->close_with_status != GRPC_STATUS_OK) { - op->close_with_status = GRPC_STATUS_OK; - if (op->optional_close_message != NULL) { - gpr_slice_unref(*op->optional_close_message); - op->optional_close_message = NULL; - } - } -} - typedef struct { - gpr_slice message; + grpc_error *error; grpc_closure *then_call; grpc_closure closure; } close_message_data; static void free_message(grpc_exec_ctx *exec_ctx, void *p, grpc_error *error) { close_message_data *cmd = p; - gpr_slice_unref(cmd->message); + GRPC_ERROR_UNREF(cmd->error); if (cmd->then_call != NULL) { cmd->then_call->cb(exec_ctx, cmd->then_call->cb_arg, GRPC_ERROR_REF(error)); } gpr_free(cmd); } +static void add_error(grpc_transport_stream_op *op, grpc_error **which, + grpc_error *error) { + close_message_data *cmd; + cmd = gpr_malloc(sizeof(*cmd)); + cmd->error = error; + cmd->then_call = op->on_complete; + grpc_closure_init(&cmd->closure, free_message, cmd); + op->on_complete = &cmd->closure; + *which = error; +} + +void grpc_transport_stream_op_add_cancellation(grpc_transport_stream_op *op, + grpc_status_code status) { + GPR_ASSERT(status != GRPC_STATUS_OK); + if (op->cancel_error == GRPC_STATUS_OK) { + op->cancel_error = grpc_error_set_int(GRPC_ERROR_CANCELLED, + GRPC_ERROR_INT_GRPC_STATUS, status); + op->close_error = GRPC_ERROR_NONE; + } +} + void grpc_transport_stream_op_add_close(grpc_transport_stream_op *op, grpc_status_code status, gpr_slice *optional_message) { - close_message_data *cmd; GPR_ASSERT(status != GRPC_STATUS_OK); - if (op->cancel_with_status != GRPC_STATUS_OK || - op->close_with_status != GRPC_STATUS_OK) { + if (op->cancel_error != GRPC_ERROR_NONE || + op->close_error != GRPC_ERROR_NONE) { if (optional_message) { gpr_slice_unref(*optional_message); } return; } - if (optional_message) { - cmd = gpr_malloc(sizeof(*cmd)); - cmd->message = *optional_message; - cmd->then_call = op->on_complete; - grpc_closure_init(&cmd->closure, free_message, cmd); - op->on_complete = &cmd->closure; - op->optional_close_message = &cmd->message; + grpc_error *error; + if (optional_message != NULL) { + char *msg = gpr_dump_slice(*optional_message, GPR_DUMP_ASCII); + error = grpc_error_set_str(GRPC_ERROR_CREATE(msg), + GRPC_ERROR_STR_GRPC_MESSAGE, msg); + gpr_free(msg); + gpr_slice_unref(*optional_message); + } else { + error = GRPC_ERROR_CREATE("Call force closed"); } - op->close_with_status = status; + error = grpc_error_set_int(error, GRPC_ERROR_INT_GRPC_STATUS, status); + add_error(op, &op->close_error, error); } diff --git a/src/core/lib/transport/transport.h b/src/core/lib/transport/transport.h index a46ccb643ce..d2f6344ee3c 100644 --- a/src/core/lib/transport/transport.h +++ b/src/core/lib/transport/transport.h @@ -135,13 +135,12 @@ typedef struct grpc_transport_stream_op { /** Collect any stats into provided buffer, zero internal stat counters */ grpc_transport_stream_stats *collect_stats; - /** If != GRPC_STATUS_OK, cancel this stream */ - grpc_status_code cancel_with_status; + /** If != GRPC_ERROR_NONE, cancel this stream */ + grpc_error *cancel_error; - /** If != GRPC_STATUS_OK, send grpc-status, grpc-message, and close this + /** If != GRPC_ERROR, send grpc-status, grpc-message, and close this stream for both reading and writing */ - grpc_status_code close_with_status; - gpr_slice *optional_close_message; + grpc_error *close_error; /* Indexes correspond to grpc_context_index enum values */ grpc_call_context_element *context; diff --git a/src/core/lib/transport/transport_op_string.c b/src/core/lib/transport/transport_op_string.c index df04c611270..a862401df2c 100644 --- a/src/core/lib/transport/transport_op_string.c +++ b/src/core/lib/transport/transport_op_string.c @@ -119,10 +119,21 @@ char *grpc_transport_stream_op_string(grpc_transport_stream_op *op) { gpr_strvec_add(&b, gpr_strdup("RECV_TRAILING_METADATA")); } - if (op->cancel_with_status != GRPC_STATUS_OK) { + if (op->cancel_error != GRPC_STATUS_OK) { if (!first) gpr_strvec_add(&b, gpr_strdup(" ")); first = 0; - gpr_asprintf(&tmp, "CANCEL:%d", op->cancel_with_status); + const char *msg = grpc_error_string(op->cancel_error); + gpr_asprintf(&tmp, "CANCEL:%s", msg); + grpc_error_free_string(msg); + gpr_strvec_add(&b, tmp); + } + + if (op->close_error != GRPC_STATUS_OK) { + if (!first) gpr_strvec_add(&b, gpr_strdup(" ")); + first = 0; + const char *msg = grpc_error_string(op->close_error); + gpr_asprintf(&tmp, "CLOSE:%s", msg); + grpc_error_free_string(msg); gpr_strvec_add(&b, tmp); } From 5a3c6389edeaf19e7e0f0a6c0771c7548f0e0997 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Thu, 23 Jun 2016 14:02:33 -0700 Subject: [PATCH 0215/1039] Added braces around _service --- .../objective-c/route_guide/ViewControllers.m | 20 +++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/examples/objective-c/route_guide/ViewControllers.m b/examples/objective-c/route_guide/ViewControllers.m index 26ca9d6220b..b2f99c437e7 100644 --- a/examples/objective-c/route_guide/ViewControllers.m +++ b/examples/objective-c/route_guide/ViewControllers.m @@ -86,8 +86,9 @@ static NSString * const kHostAddress = @"localhost:50051"; @end -@implementation GetFeatureViewController -RTGRouteGuide *_service; +@implementation GetFeatureViewController { + RTGRouteGuide *_service; +} - (void)execRequest { void (^handler)(RTGFeature *response, NSError *error) = ^(RTGFeature *response, NSError *error) { @@ -146,8 +147,9 @@ RTGRouteGuide *_service; @end -@implementation ListFeaturesViewController -RTGRouteGuide *_service; +@implementation ListFeaturesViewController { + RTGRouteGuide *_service; +} - (void)execRequest { RTGRectangle *rectangle = [RTGRectangle message]; @@ -200,8 +202,9 @@ RTGRouteGuide *_service; @end -@implementation RecordRouteViewController -RTGRouteGuide *_service; +@implementation RecordRouteViewController { + RTGRouteGuide *_service; +} - (void)execRequest { NSString *dataBasePath = [NSBundle.mainBundle pathForResource:@"route_guide_db" @@ -268,8 +271,9 @@ RTGRouteGuide *_service; @end -@implementation RouteChatViewController -RTGRouteGuide *_service; +@implementation RouteChatViewController { + RTGRouteGuide *_service; +} - (void)execRequest { NSArray *notes = @[[RTGRouteNote noteWithMessage:@"First message" latitude:0 longitude:0], From 733e3fc209f78695896b2d721069afad866cd2f7 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 23 Jun 2016 14:07:11 -0700 Subject: [PATCH 0216/1039] Fix comparison --- src/core/lib/transport/transport_op_string.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/core/lib/transport/transport_op_string.c b/src/core/lib/transport/transport_op_string.c index a862401df2c..aede337e7cb 100644 --- a/src/core/lib/transport/transport_op_string.c +++ b/src/core/lib/transport/transport_op_string.c @@ -119,7 +119,7 @@ char *grpc_transport_stream_op_string(grpc_transport_stream_op *op) { gpr_strvec_add(&b, gpr_strdup("RECV_TRAILING_METADATA")); } - if (op->cancel_error != GRPC_STATUS_OK) { + if (op->cancel_error != GRPC_ERROR_NONE) { if (!first) gpr_strvec_add(&b, gpr_strdup(" ")); first = 0; const char *msg = grpc_error_string(op->cancel_error); @@ -128,7 +128,7 @@ char *grpc_transport_stream_op_string(grpc_transport_stream_op *op) { gpr_strvec_add(&b, tmp); } - if (op->close_error != GRPC_STATUS_OK) { + if (op->close_error != GRPC_ERROR_NONE) { if (!first) gpr_strvec_add(&b, gpr_strdup(" ")); first = 0; const char *msg = grpc_error_string(op->close_error); From ea0dc6af37469a45c2750df4bf852d12d75e9240 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Thu, 23 Jun 2016 14:10:38 -0700 Subject: [PATCH 0217/1039] Added a comment --- src/objective-c/GRPCClient/private/GRPCChannel.m | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/objective-c/GRPCClient/private/GRPCChannel.m b/src/objective-c/GRPCClient/private/GRPCChannel.m index 6cd4cb0ca30..7b7b79e1c62 100644 --- a/src/objective-c/GRPCClient/private/GRPCChannel.m +++ b/src/objective-c/GRPCClient/private/GRPCChannel.m @@ -199,7 +199,7 @@ grpc_channel_args * buildChannelArgs(NSDictionary *dictionary) { NULL, GRPC_PROPAGATE_DEFAULTS, queue.unmanagedQueue, path.UTF8String, - NULL, + NULL, // Passing NULL for host gpr_inf_future(GPR_CLOCK_REALTIME), NULL); } From eb429c3067e54b0dcdae21d005a4e98a022caad8 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Thu, 23 Jun 2016 14:18:49 -0700 Subject: [PATCH 0218/1039] Removed gpr_log statement --- src/core/lib/iomgr/network_status_tracker.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/core/lib/iomgr/network_status_tracker.c b/src/core/lib/iomgr/network_status_tracker.c index 0e34605c81d..38a1c9b7d41 100644 --- a/src/core/lib/iomgr/network_status_tracker.c +++ b/src/core/lib/iomgr/network_status_tracker.c @@ -64,7 +64,6 @@ void grpc_network_status_register_endpoint(grpc_endpoint *ep) { grpc_initialize_network_status_monitor(); } gpr_mu_lock(&g_endpoint_mutex); - gpr_log(GPR_DEBUG, "Register endpoint %p", ep); if (head == NULL) { head = (endpoint_ll_node *)gpr_malloc(sizeof(endpoint_ll_node)); head->ep = ep; @@ -81,7 +80,6 @@ void grpc_network_status_register_endpoint(grpc_endpoint *ep) { void grpc_network_status_unregister_endpoint(grpc_endpoint *ep) { gpr_mu_lock(&g_endpoint_mutex); GPR_ASSERT(head); - gpr_log(GPR_DEBUG, "Unregister endpoint %p", ep); bool found = false; endpoint_ll_node *prev = head; // if we're unregistering the head, just move head to the next @@ -116,7 +114,6 @@ void grpc_network_status_shutdown_all_endpoints() { grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; for (endpoint_ll_node *curr = head; curr != NULL; curr = curr->next) { - gpr_log(GPR_DEBUG, "Shutting down endpoint %p", curr->ep); curr->ep->vtable->shutdown(&exec_ctx, curr->ep); } gpr_mu_unlock(&g_endpoint_mutex); From 565c6a0dacf0e1c8434dc88617e8c838108791d4 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Thu, 23 Jun 2016 14:22:12 -0700 Subject: [PATCH 0219/1039] Bump Protobuf to beta-3.1 --- .gitmodules | 2 +- gRPC-ProtoRPC.podspec | 2 +- src/objective-c/examples/RemoteTestClient/RemoteTest.podspec | 2 +- src/objective-c/tests/RemoteTestClient/RemoteTest.podspec | 2 +- third_party/protobuf | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.gitmodules b/.gitmodules index c85a53943a7..1e1876b724e 100644 --- a/.gitmodules +++ b/.gitmodules @@ -4,7 +4,7 @@ [submodule "third_party/protobuf"] path = third_party/protobuf url = https://github.com/google/protobuf.git - branch = v3.0.0-beta-2 + branch = v3.0.0-beta-3.1 [submodule "third_party/gflags"] path = third_party/gflags url = https://github.com/gflags/gflags.git diff --git a/gRPC-ProtoRPC.podspec b/gRPC-ProtoRPC.podspec index 6f66f928e6b..9cc33c7dbd0 100644 --- a/gRPC-ProtoRPC.podspec +++ b/gRPC-ProtoRPC.podspec @@ -61,7 +61,7 @@ Pod::Spec.new do |s| s.dependency 'gRPC', version s.dependency 'gRPC-RxLibrary', version - s.dependency 'Protobuf', '~> 3.0.0-beta-2' + s.dependency 'Protobuf', '~> 3.0.0-beta-3.1' # This is needed by all pods that depend on Protobuf: s.pod_target_xcconfig = { 'GCC_PREPROCESSOR_DEFINITIONS' => '$(inherited) GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1', diff --git a/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec b/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec index 18a9443944a..e3b50ddea50 100644 --- a/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec +++ b/src/objective-c/examples/RemoteTestClient/RemoteTest.podspec @@ -19,7 +19,7 @@ Pod::Spec.new do |s| ms.source_files = '*.pbobjc.{h,m}' ms.header_mappings_dir = '.' ms.requires_arc = false - ms.dependency 'Protobuf', '~> 3.0.0-beta-2' + ms.dependency 'Protobuf', '~> 3.0.0-beta-3.1' # This is needed by all pods that depend on Protobuf: ms.pod_target_xcconfig = { 'GCC_PREPROCESSOR_DEFINITIONS' => '$(inherited) GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1', diff --git a/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec b/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec index 8e72b2c8fe3..887380eb55f 100644 --- a/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec +++ b/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec @@ -22,7 +22,7 @@ Pod::Spec.new do |s| ms.source_files = "*.pbobjc.{h,m}" ms.header_mappings_dir = "." ms.requires_arc = false - ms.dependency "Protobuf", "~> 3.0.0-beta-2" + ms.dependency "Protobuf", "~> 3.0.0-beta-3.1" # This is needed by all pods that depend on Protobuf: ms.pod_target_xcconfig = { 'GCC_PREPROCESSOR_DEFINITIONS' => '$(inherited) GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS=1', diff --git a/third_party/protobuf b/third_party/protobuf index 3470b6895aa..137d6a09bbb 160000 --- a/third_party/protobuf +++ b/third_party/protobuf @@ -1 +1 @@ -Subproject commit 3470b6895aa659b7559ed678e029a5338e535f14 +Subproject commit 137d6a09bbbbfa801d653224703ddc59e3700704 From 67e2525892c57be3b20d517810c25a5e861cb515 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Thu, 23 Jun 2016 14:30:40 -0700 Subject: [PATCH 0220/1039] Fix handling of one error in bidi calls, and one interop server method --- src/ruby/lib/grpc/generic/bidi_call.rb | 8 ++++++++ src/ruby/pb/test/server.rb | 12 +++++++----- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/src/ruby/lib/grpc/generic/bidi_call.rb b/src/ruby/lib/grpc/generic/bidi_call.rb index f4f9d1b3ddc..e2a026cacc0 100644 --- a/src/ruby/lib/grpc/generic/bidi_call.rb +++ b/src/ruby/lib/grpc/generic/bidi_call.rb @@ -173,6 +173,14 @@ module GRPC finished end GRPC.logger.debug('bidi-write-loop: finished') + rescue GRPC::Core::CallError => e + # This is almost definitely caused by a status arriving while still + # writing. Don't re-throw the error + GRPC.logger.warn('bidi-write-loop: ended with error') + GRPC.logger.warn(e) + notify_done + @writes_complete = true + finished rescue StandardError => e GRPC.logger.warn('bidi-write-loop: failed') GRPC.logger.warn(e) diff --git a/src/ruby/pb/test/server.rb b/src/ruby/pb/test/server.rb index 914c7cc79d7..088f281dc47 100755 --- a/src/ruby/pb/test/server.rb +++ b/src/ruby/pb/test/server.rb @@ -188,11 +188,13 @@ class TestTarget < Grpc::Testing::TestService::Service begin GRPC.logger.info('interop-server: started receiving') reqs.each do |req| - resp_size = req.response_parameters[0].size - GRPC.logger.info("read a req, response size is #{resp_size}") - resp = cls.new(payload: Payload.new(type: req.response_type, - body: nulls(resp_size))) - q.push(resp) + req.response_parameters.each do |params| + resp_size = params.size + GRPC.logger.info("read a req, response size is #{resp_size}") + resp = cls.new(payload: Payload.new(type: req.response_type, + body: nulls(resp_size))) + q.push(resp) + end end GRPC.logger.info('interop-server: finished receiving') q.push(self) From d4d4c6f6f62a547da9136f57ddd6406facaf4ce5 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 23 Jun 2016 14:37:30 -0700 Subject: [PATCH 0221/1039] Fix comparison --- src/core/lib/transport/transport.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/core/lib/transport/transport.c b/src/core/lib/transport/transport.c index 67920b05279..79a20e12626 100644 --- a/src/core/lib/transport/transport.c +++ b/src/core/lib/transport/transport.c @@ -192,7 +192,7 @@ static void add_error(grpc_transport_stream_op *op, grpc_error **which, void grpc_transport_stream_op_add_cancellation(grpc_transport_stream_op *op, grpc_status_code status) { GPR_ASSERT(status != GRPC_STATUS_OK); - if (op->cancel_error == GRPC_STATUS_OK) { + if (op->cancel_error == GRPC_ERROR_NONE) { op->cancel_error = grpc_error_set_int(GRPC_ERROR_CANCELLED, GRPC_ERROR_INT_GRPC_STATUS, status); op->close_error = GRPC_ERROR_NONE; From 20d0a167beb287f61a7f33943fddfc34cae75860 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 23 Jun 2016 15:14:03 -0700 Subject: [PATCH 0222/1039] Better error handling and add polling_island_unlock_pair() helper --- src/core/lib/iomgr/ev_epoll_linux.c | 300 +++++++++++++++++----------- 1 file changed, 182 insertions(+), 118 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index b1e9ac8a631..a77044edc50 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -237,6 +237,19 @@ struct grpc_pollset_set { grpc_fd **fds; }; +/******************************************************************************* + * Common helpers + */ + +static void append_error(grpc_error **composite, grpc_error *error, + const char *desc) { + if (error == GRPC_ERROR_NONE) return; + if (*composite == GRPC_ERROR_NONE) { + *composite = GRPC_ERROR_CREATE(desc); + } + *composite = grpc_error_add_child(*composite, error); +} + /******************************************************************************* * Polling island Definitions */ @@ -316,10 +329,13 @@ long pi_unref(polling_island *pi, int ref_cnt) { /* The caller is expected to hold pi->mu lock before calling this function */ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, - size_t fd_count, bool add_fd_refs) { + size_t fd_count, bool add_fd_refs, + grpc_error **error) { int err; size_t i; struct epoll_event ev; + char *err_msg; + const char *err_desc = "polling_island_add_fds"; #ifdef GRPC_TSAN /* See the definition of g_epoll_sync for more context */ @@ -333,10 +349,12 @@ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, if (err < 0) { if (errno != EEXIST) { - /* TODO: sreek - We need a better way to bubble up this error instead of - just logging a message */ - gpr_log(GPR_ERROR, "epoll_ctl add for fd: %d failed with error: %s", - fds[i]->fd, strerror(errno)); + gpr_asprintf( + &err_msg, + "epoll_ctl (epoll_fd: %d) add fd: %d failed with error: %d (%s)", + pi->epoll_fd, fds[i]->fd, errno, strerror(errno)); + append_error(error, GRPC_OS_ERROR(errno, err_msg), err_desc); + gpr_free(err_msg); } continue; @@ -356,37 +374,47 @@ static void polling_island_add_fds_locked(polling_island *pi, grpc_fd **fds, /* The caller is expected to hold pi->mu before calling this */ static void polling_island_add_wakeup_fd_locked(polling_island *pi, - grpc_wakeup_fd *wakeup_fd) { + grpc_wakeup_fd *wakeup_fd, + grpc_error **error) { struct epoll_event ev; int err; + char *err_msg; + const char *err_desc = "polling_island_add_wakeup_fd"; ev.events = (uint32_t)(EPOLLIN | EPOLLET); ev.data.ptr = wakeup_fd; err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_ADD, GRPC_WAKEUP_FD_GET_READ_FD(wakeup_fd), &ev); - if (err < 0) { - gpr_log(GPR_ERROR, - "Failed to add grpc_wake_up_fd (%d) to the epoll set (epoll_fd: %d)" - ". Error: %s", - GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), pi->epoll_fd, - strerror(errno)); + if (err < 0 && errno != EEXIST) { + gpr_asprintf(&err_msg, + "epoll_ctl (epoll_fd: %d) add wakeup fd: %d failed with " + "error: %d (%s)", + pi->epoll_fd, + GRPC_WAKEUP_FD_GET_READ_FD(&grpc_global_wakeup_fd), errno, + strerror(errno)); + append_error(error, GRPC_OS_ERROR(errno, err_msg), err_desc); + gpr_free(err_msg); } } /* The caller is expected to hold pi->mu lock before calling this function */ static void polling_island_remove_all_fds_locked(polling_island *pi, - bool remove_fd_refs) { + bool remove_fd_refs, + grpc_error **error) { int err; size_t i; + char *err_msg; + const char *err_desc = "polling_island_remove_fds"; for (i = 0; i < pi->fd_cnt; i++) { err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_DEL, pi->fds[i]->fd, NULL); if (err < 0 && errno != ENOENT) { - /* TODO: sreek - We need a better way to bubble up this error instead of - * just logging a message */ - gpr_log(GPR_ERROR, - "epoll_ctl deleting fds[%zu]: %d failed with error: %s", i, - pi->fds[i]->fd, strerror(errno)); + gpr_asprintf(&err_msg, + "epoll_ctl (epoll_fd: %d) delete fds[%zu]: %d failed with " + "error: %d (%s)", + pi->epoll_fd, i, pi->fds[i]->fd, errno, strerror(errno)); + append_error(error, GRPC_OS_ERROR(errno, err_msg), err_desc); + gpr_free(err_msg); } if (remove_fd_refs) { @@ -399,17 +427,24 @@ static void polling_island_remove_all_fds_locked(polling_island *pi, /* The caller is expected to hold pi->mu lock before calling this function */ static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd, - bool is_fd_closed) { + bool is_fd_closed, + grpc_error **error) { int err; size_t i; + char *err_msg; + const char *err_desc = "polling_island_remove_fd"; /* If fd is already closed, then it would have been automatically been removed from the epoll set */ if (!is_fd_closed) { err = epoll_ctl(pi->epoll_fd, EPOLL_CTL_DEL, fd->fd, NULL); if (err < 0 && errno != ENOENT) { - gpr_log(GPR_ERROR, "epoll_ctl deleting fd: %d failed with error; %s", - fd->fd, strerror(errno)); + gpr_asprintf( + &err_msg, + "epoll_ctl (epoll_fd: %d) del fd: %d failed with error: %d (%s)", + pi->epoll_fd, fd->fd, errno, strerror(errno)); + append_error(error, GRPC_OS_ERROR(errno, err_msg), err_desc); + gpr_free(err_msg); } } @@ -422,8 +457,12 @@ static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd, } } -static polling_island *polling_island_create(grpc_fd *initial_fd) { +/* Might return NULL in case of an error */ +static polling_island *polling_island_create(grpc_fd *initial_fd, + grpc_error **error) { polling_island *pi = NULL; + char *err_msg; + const char *err_desc = "polling_island_create"; /* Try to get one from the polling island freelist */ gpr_mu_lock(&g_pi_freelist_mu); @@ -449,22 +488,22 @@ static polling_island *polling_island_create(grpc_fd *initial_fd) { pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); if (pi->epoll_fd < 0) { - gpr_log(GPR_ERROR, "epoll_create1() failed with error: %s", - strerror(errno)); - } - GPR_ASSERT(pi->epoll_fd >= 0); - - polling_island_add_wakeup_fd_locked(pi, &grpc_global_wakeup_fd); - - pi->next_free = NULL; + gpr_asprintf(&err_msg, "epoll_create1 failed with error %d (%s)", errno, + strerror(errno)); + append_error(error, GRPC_OS_ERROR(errno, err_msg), err_desc); + gpr_free(err_msg); + } else { + polling_island_add_wakeup_fd_locked(pi, &grpc_global_wakeup_fd, error); + pi->next_free = NULL; - if (initial_fd != NULL) { - /* Lock the polling island here just in case we got this structure from the - freelist and the polling island lock was not released yet (by the code - that adds the polling island to the freelist) */ - gpr_mu_lock(&pi->mu); - polling_island_add_fds_locked(pi, &initial_fd, 1, true); - gpr_mu_unlock(&pi->mu); + if (initial_fd != NULL) { + /* Lock the polling island here just in case we got this structure from + the freelist and the polling island lock was not released yet (by the + code that adds the polling island to the freelist) */ + gpr_mu_lock(&pi->mu); + polling_island_add_fds_locked(pi, &initial_fd, 1, true, error); + gpr_mu_unlock(&pi->mu); + } } return pi; @@ -534,7 +573,9 @@ static polling_island *polling_island_lock(polling_island *pi) { return pi; } -/* Gets the lock on the *latest* polling islands pointed by *p and *q. +/* Gets the lock on the *latest* polling islands in the linked lists pointed by + *p and *q (and also updates *p and *q to point to the latest polling islands) + This function is needed because calling the following block of code to obtain locks on polling islands (*p and *q) is prone to deadlocks. { @@ -550,18 +591,8 @@ static polling_island *polling_island_lock(polling_island *pi) { .. .. Critical section with both p1 and p2 locked .. - // Release locks - // **IMPORTANT**: Make sure you check p1 == p2 AFTER the function - // polling_island_lock_pair() was called and if so, release the lock only - // once. Note: Even if p1 != p2 beforec calling polling_island_lock_pair(), - // they might be after the function returns: - if (p1 == p2) { - gpr_mu_unlock(&p1->mu) - } else { - gpr_mu_unlock(&p1->mu); - gpr_mu_unlock(&p2->mu); - } - + // Release locks: Always call polling_island_unlock_pair() to release locks + polling_island_unlock_pair(p1, p2); */ static void polling_island_lock_pair(polling_island **p, polling_island **q) { polling_island *pi_1 = *p; @@ -623,39 +654,46 @@ static void polling_island_lock_pair(polling_island **p, polling_island **q) { *q = pi_2; } -static polling_island *polling_island_merge(polling_island *p, - polling_island *q) { - /* Get locks on both the polling islands */ - polling_island_lock_pair(&p, &q); - +static void polling_island_unlock_pair(polling_island *p, polling_island *q) { if (p == q) { - /* Nothing needs to be done here */ gpr_mu_unlock(&p->mu); - return p; + } else { + gpr_mu_unlock(&p->mu); + gpr_mu_unlock(&q->mu); } +} - /* Make sure that p points to the polling island with fewer fds than q */ - if (p->fd_cnt > q->fd_cnt) { - GPR_SWAP(polling_island *, p, q); - } +static polling_island *polling_island_merge(polling_island *p, + polling_island *q, + grpc_error **error) { + /* Get locks on both the polling islands */ + polling_island_lock_pair(&p, &q); - /* "Merge" p with q i.e move all the fds from p (The one with fewer fds) to q - Note that the refcounts on the fds being moved will not change here. This - is why the last parameter in the following two functions is 'false') */ - polling_island_add_fds_locked(q, p->fds, p->fd_cnt, false); - polling_island_remove_all_fds_locked(p, false); + if (p != q) { + /* Make sure that p points to the polling island with fewer fds than q */ + if (p->fd_cnt > q->fd_cnt) { + GPR_SWAP(polling_island *, p, q); + } + + /* Merge p with q i.e move all the fds from p (The one with fewer fds) to q + Note that the refcounts on the fds being moved will not change here. + This is why the last param in the following two functions is 'false') */ + polling_island_add_fds_locked(q, p->fds, p->fd_cnt, false, error); + polling_island_remove_all_fds_locked(p, false, error); - /* Wakeup all the pollers (if any) on p so that they can pickup this change */ - polling_island_add_wakeup_fd_locked(p, &polling_island_wakeup_fd); + /* Wakeup all the pollers (if any) on p so that they pickup this change */ + polling_island_add_wakeup_fd_locked(p, &polling_island_wakeup_fd, error); - /* Add the 'merged_to' link from p --> q */ - gpr_atm_rel_store(&p->merged_to, (gpr_atm)q); - PI_ADD_REF(q, "pi_merge"); /* To account for the new incoming ref from p */ + /* Add the 'merged_to' link from p --> q */ + gpr_atm_rel_store(&p->merged_to, (gpr_atm)q); + PI_ADD_REF(q, "pi_merge"); /* To account for the new incoming ref from p */ + } + /* else if p == q, nothing needs to be done */ - gpr_mu_unlock(&p->mu); - gpr_mu_unlock(&q->mu); + polling_island_unlock_pair(p, q); - /* Return the merged polling island */ + /* Return the merged polling island (Note that no merge would have happened + if p == q which is ok) */ return q; } @@ -853,6 +891,8 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure *on_done, int *release_fd, const char *reason) { bool is_fd_closed = false; + grpc_error *error = GRPC_ERROR_NONE; + gpr_mu_lock(&fd->mu); fd->on_done_closure = on_done; @@ -882,7 +922,7 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, gpr_mu_lock(&fd->pi_mu); if (fd->polling_island != NULL) { polling_island *pi_latest = polling_island_lock(fd->polling_island); - polling_island_remove_fd_locked(pi_latest, fd, is_fd_closed); + polling_island_remove_fd_locked(pi_latest, fd, is_fd_closed, &error); gpr_mu_unlock(&pi_latest->mu); PI_UNREF(fd->polling_island, "fd_orphan"); @@ -890,10 +930,11 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, } gpr_mu_unlock(&fd->pi_mu); - grpc_exec_ctx_sched(exec_ctx, fd->on_done_closure, GRPC_ERROR_NONE, NULL); + grpc_exec_ctx_sched(exec_ctx, fd->on_done_closure, error, NULL); gpr_mu_unlock(&fd->mu); UNREF_BY(fd, 2, reason); /* Drop the reference */ + GRPC_LOG_IF_ERROR("fd_orphan", GRPC_ERROR_REF(error)); } static grpc_error *fd_shutdown_error(bool shutdown) { @@ -1062,19 +1103,12 @@ static void push_front_worker(grpc_pollset *p, grpc_pollset_worker *worker) { worker->prev->next = worker->next->prev = worker; } -static void kick_append_error(grpc_error **composite, grpc_error *error) { - if (error == GRPC_ERROR_NONE) return; - if (*composite == GRPC_ERROR_NONE) { - *composite = GRPC_ERROR_CREATE("Kick Failure"); - } - *composite = grpc_error_add_child(*composite, error); -} - /* p->mu must be held before calling this function */ static grpc_error *pollset_kick(grpc_pollset *p, grpc_pollset_worker *specific_worker) { GPR_TIMER_BEGIN("pollset_kick", 0); grpc_error *error = GRPC_ERROR_NONE; + const char *err_desc = "Kick Failure"; grpc_pollset_worker *worker = specific_worker; if (worker != NULL) { @@ -1084,7 +1118,7 @@ static grpc_error *pollset_kick(grpc_pollset *p, for (worker = p->root_worker.next; worker != &p->root_worker; worker = worker->next) { if (gpr_tls_get(&g_current_thread_worker) != (intptr_t)worker) { - kick_append_error(&error, pollset_worker_kick(worker)); + append_error(&error, pollset_worker_kick(worker), err_desc); } } } else { @@ -1094,7 +1128,7 @@ static grpc_error *pollset_kick(grpc_pollset *p, } else { GPR_TIMER_MARK("kicked_specifically", 0); if (gpr_tls_get(&g_current_thread_worker) != (intptr_t)worker) { - kick_append_error(&error, pollset_worker_kick(worker)); + append_error(&error, pollset_worker_kick(worker), err_desc); } } } else if (gpr_tls_get(&g_current_thread_pollset) != (intptr_t)p) { @@ -1110,7 +1144,7 @@ static grpc_error *pollset_kick(grpc_pollset *p, if (worker != NULL) { GPR_TIMER_MARK("finally_kick", 0); push_back_worker(p, worker); - kick_append_error(&error, pollset_worker_kick(worker)); + append_error(&error, pollset_worker_kick(worker), err_desc); } else { GPR_TIMER_MARK("kicked_no_pollers", 0); p->kicked_without_pollers = true; @@ -1238,23 +1272,17 @@ static void pollset_reset(grpc_pollset *pollset) { pollset_release_polling_island(pollset, "ps_reset"); } -static void work_combine_error(grpc_error **composite, grpc_error *error) { - if (error == GRPC_ERROR_NONE) return; - if (*composite == GRPC_ERROR_NONE) { - *composite = GRPC_ERROR_CREATE("pollset_work"); - } - *composite = grpc_error_add_child(*composite, error); -} - #define GRPC_EPOLL_MAX_EVENTS 1000 -static grpc_error *pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, - grpc_pollset *pollset, - int timeout_ms, sigset_t *sig_mask) { +/* Note: sig_mask contains the signal mask to use *during* epoll_wait() */ +static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, + grpc_pollset *pollset, int timeout_ms, + sigset_t *sig_mask, grpc_error **error) { struct epoll_event ep_ev[GRPC_EPOLL_MAX_EVENTS]; int epoll_fd = -1; int ep_rv; polling_island *pi = NULL; - grpc_error *error = GRPC_ERROR_NONE; + char *err_msg; + const char *err_desc = "pollset_work_and_unlock"; GPR_TIMER_BEGIN("pollset_work_and_unlock", 0); /* We need to get the epoll_fd to wait on. The epoll_fd is in inside the @@ -1265,11 +1293,15 @@ static grpc_error *pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, which we got the epoll_fd) got merged with another island while we are in this function. This is still okay because in such a case, we will wakeup right-away from epoll_wait() and pick up the latest polling_island the next - this function (i.e pollset_work_and_unlock()) is called. - */ + this function (i.e pollset_work_and_unlock()) is called */ if (pollset->polling_island == NULL) { - pollset->polling_island = polling_island_create(NULL); + pollset->polling_island = polling_island_create(NULL, error); + if (pollset->polling_island == NULL) { + GPR_TIMER_END("pollset_work_and_unlock", 0); + return; /* Fatal error. We cannot continue */ + } + PI_ADD_REF(pollset->polling_island, "ps"); } @@ -1297,8 +1329,10 @@ static grpc_error *pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, sig_mask); if (ep_rv < 0) { if (errno != EINTR) { - gpr_log(GPR_ERROR, "epoll_pwait() failed: %s", strerror(errno)); - work_combine_error(&error, GRPC_OS_ERROR(errno, "epoll_pwait")); + gpr_asprintf(&err_msg, + "epoll_wait() epoll fd: %d failed with error: %d (%s)", + epoll_fd, errno, strerror(errno)); + append_error(error, GRPC_OS_ERROR(errno, err_msg), err_desc); } else { /* We were interrupted. Save an interation by doing a zero timeout epoll_wait to see if there are any other events of interest */ @@ -1314,8 +1348,9 @@ static grpc_error *pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, for (int i = 0; i < ep_rv; ++i) { void *data_ptr = ep_ev[i].data.ptr; if (data_ptr == &grpc_global_wakeup_fd) { - work_combine_error( - &error, grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd)); + append_error(error, + grpc_wakeup_fd_consume_wakeup(&grpc_global_wakeup_fd), + err_desc); } else if (data_ptr == &polling_island_wakeup_fd) { /* This means that our polling island is merged with a different island. We do not have to do anything here since the subsequent call @@ -1346,7 +1381,6 @@ static grpc_error *pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, PI_UNREF(pi, "ps_work"); GPR_TIMER_END("pollset_work_and_unlock", 0); - return error; } /* pollset->mu lock must be held by the caller before calling this. @@ -1368,6 +1402,7 @@ static grpc_error *pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, worker.pt_id = pthread_self(); *worker_hdl = &worker; + gpr_tls_set(&g_current_thread_pollset, (intptr_t)pollset); gpr_tls_set(&g_current_thread_worker, (intptr_t)&worker); @@ -1379,14 +1414,37 @@ static grpc_error *pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, GPR_TIMER_MARK("pollset_work.kicked_without_pollers", 0); pollset->kicked_without_pollers = 0; } else if (!pollset->shutting_down) { + /* We use the posix-signal with number 'grpc_wakeup_signal' for waking up + (i.e 'kicking') a worker in the pollset. + A 'kick' is a way to inform that worker that there is some pending work + that needs immediate attention (like an event on the completion queue, + or a polling island merge that results in a new epoll-fd to wait on) and + that the worker should not spend time waiting in epoll_pwait(). + + A kick can come at anytime (i.e before/during or after the worker calls + epoll_pwait()) but in all cases we have to make sure that when a worker + gets a kick, it does not spend time in epoll_pwait(). In other words, one + kick should result in skipping/exiting of one epoll_pwait(); + + To accomplish this, we mask 'grpc_wakeup_signal' on this worker at all + times *except* when it is in epoll_pwait(). This way, the worker never + misses acting on a kick */ + sigemptyset(&new_mask); sigaddset(&new_mask, grpc_wakeup_signal); pthread_sigmask(SIG_BLOCK, &new_mask, &orig_mask); sigdelset(&orig_mask, grpc_wakeup_signal); + /* new_mask: The new thread mask which blocks 'grpc_wakeup_signal'. This is + the mask used at all times *except during epoll_wait()*" + orig_mask: The thread mask which allows 'grpc_wakeup_signal' and this is + the mask to use *during epoll_wait()* + + The new_mask is set on the worker before it is added to the pollset (i.e + before it can be kicked) */ - push_front_worker(pollset, &worker); + push_front_worker(pollset, &worker); /* Add worker to pollset */ - error = pollset_work_and_unlock(exec_ctx, pollset, timeout_ms, &orig_mask); + pollset_work_and_unlock(exec_ctx, pollset, timeout_ms, &orig_mask, &error); grpc_exec_ctx_flush(exec_ctx); gpr_mu_lock(&pollset->mu); @@ -1412,15 +1470,20 @@ static grpc_error *pollset_work(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, } *worker_hdl = NULL; + gpr_tls_set(&g_current_thread_pollset, (intptr_t)0); gpr_tls_set(&g_current_thread_worker, (intptr_t)0); + GPR_TIMER_END("pollset_work", 0); + GRPC_LOG_IF_ERROR("pollset_work", GRPC_ERROR_REF(error)); return error; } static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_fd *fd) { + grpc_error *error = GRPC_ERROR_NONE; + gpr_mu_lock(&pollset->mu); gpr_mu_lock(&fd->pi_mu); @@ -1443,19 +1506,23 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, if (fd->polling_island == pollset->polling_island) { pi_new = fd->polling_island; if (pi_new == NULL) { - pi_new = polling_island_create(fd); + pi_new = polling_island_create(fd, &error); } } else if (fd->polling_island == NULL) { pi_new = polling_island_lock(pollset->polling_island); - polling_island_add_fds_locked(pi_new, &fd, 1, true); + polling_island_add_fds_locked(pi_new, &fd, 1, true, &error); gpr_mu_unlock(&pi_new->mu); } else if (pollset->polling_island == NULL) { pi_new = polling_island_lock(fd->polling_island); gpr_mu_unlock(&pi_new->mu); } else { - pi_new = polling_island_merge(fd->polling_island, pollset->polling_island); + pi_new = polling_island_merge(fd->polling_island, pollset->polling_island, + &error); } + /* At this point, pi_new is the polling island that both fd->polling_island + and pollset->polling_island must be pointing to */ + if (fd->polling_island != pi_new) { PI_ADD_REF(pi_new, "fd"); if (fd->polling_island != NULL) { @@ -1645,13 +1712,10 @@ bool grpc_are_polling_islands_equal(void *p, void *q) { polling_island *p1 = p; polling_island *p2 = q; + /* Note: polling_island_lock_pair() may change p1 and p2 to point to the + latest polling islands in their respective linked lists */ polling_island_lock_pair(&p1, &p2); - if (p1 == p2) { - gpr_mu_unlock(&p1->mu); - } else { - gpr_mu_unlock(&p1->mu); - gpr_mu_unlock(&p2->mu); - } + polling_island_unlock_pair(p1, p2); return p1 == p2; } From 74189cd94b49be086e9320bf9536ab4bacfa6d61 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 23 Jun 2016 15:39:06 -0700 Subject: [PATCH 0223/1039] Remove caching of results by run_tests SIGNIFICANTLY increases the performance of actually running tests... --- tools/run_tests/jobset.py | 68 ++++++++---------------------------- tools/run_tests/run_tests.py | 63 ++------------------------------- 2 files changed, 18 insertions(+), 113 deletions(-) diff --git a/tools/run_tests/jobset.py b/tools/run_tests/jobset.py index d3259e724df..40409c43948 100755 --- a/tools/run_tests/jobset.py +++ b/tools/run_tests/jobset.py @@ -29,7 +29,6 @@ """Run a group of subprocesses and then finish.""" -import hashlib import multiprocessing import os import platform @@ -149,7 +148,7 @@ def which(filename): class JobSpec(object): """Specifies what to run for a job.""" - def __init__(self, cmdline, shortname=None, environ=None, hash_targets=None, + def __init__(self, cmdline, shortname=None, environ=None, cwd=None, shell=False, timeout_seconds=5*60, flake_retries=0, timeout_retries=0, kill_handler=None, cpu_cost=1.0, verbose_success=False): @@ -157,19 +156,14 @@ class JobSpec(object): Arguments: cmdline: a list of arguments to pass as the command line environ: a dictionary of environment variables to set in the child process - hash_targets: which files to include in the hash representing the jobs version - (or empty, indicating the job should not be hashed) kill_handler: a handler that will be called whenever job.kill() is invoked cpu_cost: number of cores per second this job needs """ if environ is None: environ = {} - if hash_targets is None: - hash_targets = [] self.cmdline = cmdline self.environ = environ self.shortname = cmdline[0] if shortname is None else shortname - self.hash_targets = hash_targets or [] self.cwd = cwd self.shell = shell self.timeout_seconds = timeout_seconds @@ -180,7 +174,7 @@ class JobSpec(object): self.verbose_success = verbose_success def identity(self): - return '%r %r %r' % (self.cmdline, self.environ, self.hash_targets) + return '%r %r' % (self.cmdline, self.environ) def __hash__(self): return hash(self.identity()) @@ -205,9 +199,8 @@ class JobResult(object): class Job(object): """Manages one job.""" - def __init__(self, spec, bin_hash, newline_on_success, travis, add_env): + def __init__(self, spec, newline_on_success, travis, add_env): self._spec = spec - self._bin_hash = bin_hash self._newline_on_success = newline_on_success self._travis = travis self._add_env = add_env.copy() @@ -249,7 +242,7 @@ class Job(object): self._process = try_start() self._state = _RUNNING - def state(self, update_cache): + def state(self): """Poll current state of the job. Prints messages at completion.""" def stdout(self=self): self._tempfile.seek(0) @@ -293,8 +286,6 @@ class Job(object): stdout() if self._spec.verbose_success else None, do_newline=self._newline_on_success or self._travis) self.result.state = 'PASSED' - if self._bin_hash: - update_cache.finished(self._spec.identity(), self._bin_hash) elif (self._state == _RUNNING and self._spec.timeout_seconds is not None and time.time() - self._start > self._spec.timeout_seconds): @@ -329,7 +320,7 @@ class Jobset(object): """Manages one run of jobs.""" def __init__(self, check_cancelled, maxjobs, newline_on_success, travis, - stop_on_failure, add_env, cache): + stop_on_failure, add_env): self._running = set() self._check_cancelled = check_cancelled self._cancelled = False @@ -338,9 +329,7 @@ class Jobset(object): self._maxjobs = maxjobs self._newline_on_success = newline_on_success self._travis = travis - self._cache = cache self._stop_on_failure = stop_on_failure - self._hashes = {} self._add_env = add_env self.resultset = {} self._remaining = None @@ -367,37 +356,21 @@ class Jobset(object): if current_cpu_cost + spec.cpu_cost <= self._maxjobs: break self.reap() if self.cancelled(): return False - if spec.hash_targets: - if spec.identity() in self._hashes: - bin_hash = self._hashes[spec.identity()] - else: - bin_hash = hashlib.sha1() - for fn in spec.hash_targets: - with open(which(fn)) as f: - bin_hash.update(f.read()) - bin_hash = bin_hash.hexdigest() - self._hashes[spec.identity()] = bin_hash - should_run = self._cache.should_run(spec.identity(), bin_hash) - else: - bin_hash = None - should_run = True - if should_run: - job = Job(spec, - bin_hash, - self._newline_on_success, - self._travis, - self._add_env) - self._running.add(job) - if not self.resultset.has_key(job.GetSpec().shortname): - self.resultset[job.GetSpec().shortname] = [] - return True + job = Job(spec, + self._newline_on_success, + self._travis, + self._add_env) + self._running.add(job) + if not self.resultset.has_key(job.GetSpec().shortname): + self.resultset[job.GetSpec().shortname] = [] + return True def reap(self): """Collect the dead jobs.""" while self._running: dead = set() for job in self._running: - st = job.state(self._cache) + st = job.state() if st == _RUNNING: continue if st == _FAILURE or st == _KILLED: self._failures += 1 @@ -450,15 +423,6 @@ def _never_cancelled(): return False -# cache class that caches nothing -class NoCache(object): - def should_run(self, cmdline, bin_hash): - return True - - def finished(self, cmdline, bin_hash): - pass - - def tag_remaining(xs): staging = [] for x in xs: @@ -477,12 +441,10 @@ def run(cmdlines, travis=False, infinite_runs=False, stop_on_failure=False, - cache=None, add_env={}): js = Jobset(check_cancelled, maxjobs if maxjobs is not None else _DEFAULT_MAX_JOBS, - newline_on_success, travis, stop_on_failure, add_env, - cache if cache is not None else NoCache()) + newline_on_success, travis, stop_on_failure, add_env) for cmdline, remaining in tag_remaining(cmdlines): if not js.start(cmdline): break diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index e4779e3a4e8..c571a81eb28 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -33,7 +33,6 @@ import argparse import ast import glob -import hashlib import itertools import json import multiprocessing @@ -78,24 +77,18 @@ class Config(object): if environ is None: environ = {} self.build_config = config - self.allow_hashing = (config != 'gcov') self.environ = environ self.environ['CONFIG'] = config self.tool_prefix = tool_prefix self.timeout_multiplier = timeout_multiplier - def job_spec(self, cmdline, hash_targets, timeout_seconds=5*60, + def job_spec(self, cmdline, timeout_seconds=5*60, shortname=None, environ={}, cpu_cost=1.0, flaky=False): """Construct a jobset.JobSpec for a test under this config Args: cmdline: a list of strings specifying the command line the test would like to run - hash_targets: either None (don't do caching of test results), or - a list of strings specifying files to include in a - binary hash to check if a test has changed - -- if used, all artifacts needed to run the test must - be listed """ actual_environ = self.environ.copy() for k, v in environ.iteritems(): @@ -105,8 +98,6 @@ class Config(object): environ=actual_environ, cpu_cost=cpu_cost, timeout_seconds=(self.timeout_multiplier * timeout_seconds if timeout_seconds else None), - hash_targets=hash_targets - if self.allow_hashing else None, flake_retries=5 if flaky or args.allow_flakes else 0, timeout_retries=3 if args.allow_flakes else 0) @@ -425,7 +416,7 @@ class PythonLanguage(object): return [] def build_steps(self): - return [['tools/run_tests/build_python.sh', tox_env] + return [['tools/run_tests/build_python.sh', tox_env] for tox_env in self._tox_envs] def post_tests_steps(self): @@ -1058,46 +1049,6 @@ runs_per_test = args.runs_per_test forever = args.forever -class TestCache(object): - """Cache for running tests.""" - - def __init__(self, use_cache_results): - self._last_successful_run = {} - self._use_cache_results = use_cache_results - self._last_save = time.time() - - def should_run(self, cmdline, bin_hash): - if cmdline not in self._last_successful_run: - return True - if self._last_successful_run[cmdline] != bin_hash: - return True - if not self._use_cache_results: - return True - return False - - def finished(self, cmdline, bin_hash): - self._last_successful_run[cmdline] = bin_hash - if time.time() - self._last_save > 1: - self.save() - - def dump(self): - return [{'cmdline': k, 'hash': v} - for k, v in self._last_successful_run.iteritems()] - - def parse(self, exdump): - self._last_successful_run = dict((o['cmdline'], o['hash']) for o in exdump) - - def save(self): - with open('.run_tests_cache', 'w') as f: - f.write(json.dumps(self.dump())) - self._last_save = time.time() - - def maybe_load(self): - if os.path.exists('.run_tests_cache'): - with open('.run_tests_cache') as f: - self.parse(json.loads(f.read())) - - def _start_port_server(port_server_port): # check if a compatible port server is running # if incompatible (version mismatch) ==> start a new one @@ -1217,7 +1168,7 @@ class BuildAndRunError(object): # returns a list of things that failed (or an empty list on success) def _build_and_run( - check_cancelled, newline_on_success, cache, xml_report=None, build_only=False): + check_cancelled, newline_on_success, xml_report=None, build_only=False): """Do one pass of building & running tests.""" # build latest sequentially num_failures, resultset = jobset.run( @@ -1266,7 +1217,6 @@ def _build_and_run( all_runs, check_cancelled, newline_on_success=newline_on_success, travis=args.travis, infinite_runs=infinite_runs, maxjobs=args.jobs, stop_on_failure=args.stop_on_failure, - cache=cache if not xml_report else None, add_env={'GRPC_TEST_PORT_SERVER': 'localhost:%d' % port_server_port}) if resultset: for k, v in sorted(resultset.items()): @@ -1295,14 +1245,9 @@ def _build_and_run( if num_test_failures: out.append(BuildAndRunError.TEST) - if cache: cache.save() - return out -test_cache = TestCache(runs_per_test == 1) -test_cache.maybe_load() - if forever: success = True while True: @@ -1312,7 +1257,6 @@ if forever: previous_success = success errors = _build_and_run(check_cancelled=have_files_changed, newline_on_success=False, - cache=test_cache, build_only=args.build_only) == 0 if not previous_success and not errors: jobset.message('SUCCESS', @@ -1324,7 +1268,6 @@ if forever: else: errors = _build_and_run(check_cancelled=lambda: False, newline_on_success=args.newline_on_success, - cache=test_cache, xml_report=args.xml_report, build_only=args.build_only) if not errors: From 98d31d1a40fe0d320f2488794d97a0bad5a8060d Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 23 Jun 2016 15:43:22 -0700 Subject: [PATCH 0224/1039] Fix special value lookup --- src/core/lib/iomgr/error.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/core/lib/iomgr/error.c b/src/core/lib/iomgr/error.c index da0d6972360..9f5ba76fd6c 100644 --- a/src/core/lib/iomgr/error.c +++ b/src/core/lib/iomgr/error.c @@ -276,7 +276,8 @@ bool grpc_error_get_int(grpc_error *err, grpc_error_ints which, intptr_t *p) { void *pp; if (is_special(err)) { if (err == GRPC_ERROR_CANCELLED && which == GRPC_ERROR_INT_GRPC_STATUS) { - return GRPC_STATUS_CANCELLED; + *p = GRPC_STATUS_CANCELLED; + return true; } return false; } From 6a29545c8c5ef61346af3a9b0bdd2ddb39ba15c8 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 23 Jun 2016 15:53:10 -0700 Subject: [PATCH 0225/1039] Change the type of 'ref_count' in polling_island from gpr_atm to gpr_refcount --- src/core/lib/iomgr/ev_epoll_linux.c | 69 +++++++++++++---------------- 1 file changed, 31 insertions(+), 38 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index a77044edc50..4dca551e1e1 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -121,6 +121,7 @@ struct grpc_fd { }; /* Reference counting for fds */ +// #define GRPC_FD_REF_COUNT_DEBUG #ifdef GRPC_FD_REF_COUNT_DEBUG static void fd_ref(grpc_fd *fd, const char *reason, const char *file, int line); static void fd_unref(grpc_fd *fd, const char *reason, const char *file, @@ -147,13 +148,13 @@ static void fd_global_shutdown(void); // #define GRPC_PI_REF_COUNT_DEBUG #ifdef GRPC_PI_REF_COUNT_DEBUG -#define PI_ADD_REF(p, r) pi_add_ref_dbg((p), 1, (r), __FILE__, __LINE__) -#define PI_UNREF(p, r) pi_unref_dbg((p), 1, (r), __FILE__, __LINE__) +#define PI_ADD_REF(p, r) pi_add_ref_dbg((p), (r), __FILE__, __LINE__) +#define PI_UNREF(p, r) pi_unref_dbg((p), (r), __FILE__, __LINE__) #else /* defined(GRPC_PI_REF_COUNT_DEBUG) */ -#define PI_ADD_REF(p, r) pi_add_ref((p), 1) -#define PI_UNREF(p, r) pi_unref((p), 1) +#define PI_ADD_REF(p, r) pi_add_ref((p)) +#define PI_UNREF(p, r) pi_unref((p)) #endif /* !defined(GPRC_PI_REF_COUNT_DEBUG) */ @@ -164,7 +165,7 @@ typedef struct polling_island { Once the ref count becomes zero, this structure is destroyed which means we should ensure that there is never a scenario where a PI_ADD_REF() is racing with a PI_UNREF() that just made the ref_count zero. */ - gpr_atm ref_count; + gpr_refcount ref_count; /* Pointer to the polling_island this merged into. * merged_to value is only set once in polling_island's lifetime (and that too @@ -281,50 +282,42 @@ gpr_atm g_epoll_sync; #endif /* defined(GRPC_TSAN) */ #ifdef GRPC_PI_REF_COUNT_DEBUG -long pi_add_ref(polling_island *pi, int ref_cnt); -long pi_unref(polling_island *pi, int ref_cnt); +void pi_add_ref(polling_island *pi); +void pi_unref(polling_island *pi); -void pi_add_ref_dbg(polling_island *pi, int ref_cnt, char *reason, char *file, - int line) { - long old_cnt = pi_add_ref(pi, ref_cnt); - gpr_log(GPR_DEBUG, "Add ref pi: %p, old:%ld -> new:%ld (%s) - (%s, %d)", - (void *)pi, old_cnt, (old_cnt + ref_cnt), reason, file, line); +void pi_add_ref_dbg(polling_island *pi, char *reason, char *file, int line) { + long old_cnt = gpr_atm_acq_load(&(pi->ref_count.count)); + pi_add_ref(pi); + gpr_log(GPR_DEBUG, "Add ref pi: %p, old: %ld -> new:%ld (%s) - (%s, %d)", + (void *)pi, old_cnt, old_cnt + 1, reason, file, line); } -void pi_unref_dbg(polling_island *pi, int ref_cnt, char *reason, char *file, - int line) { - long old_cnt = pi_unref(pi, ref_cnt); +void pi_unref_dbg(polling_island *pi, char *reason, char *file, int line) { + long old_cnt = gpr_atm_acq_load(&(pi->ref_count.count)); + pi_unref(pi); gpr_log(GPR_DEBUG, "Unref pi: %p, old:%ld -> new:%ld (%s) - (%s, %d)", - (void *)pi, old_cnt, (old_cnt - ref_cnt), reason, file, line); + (void *)pi, old_cnt, (old_cnt - 1), reason, file, line); } #endif -long pi_add_ref(polling_island *pi, int ref_cnt) { - return gpr_atm_full_fetch_add(&pi->ref_count, ref_cnt); -} - -long pi_unref(polling_island *pi, int ref_cnt) { - long old_cnt = gpr_atm_full_fetch_add(&pi->ref_count, -ref_cnt); +void pi_add_ref(polling_island *pi) { gpr_ref(&pi->ref_count); } - /* If ref count went to zero, delete the polling island. Note that this need - not be done under a lock. Once the ref count goes to zero, we are - guaranteed that no one else holds a reference to the polling island (and - that there is no racing pi_add_ref() call either. +void pi_unref(polling_island *pi) { + /* If ref count went to zero, delete the polling island. + Note that this deletion not be done under a lock. Once the ref count goes + to zero, we are guaranteed that no one else holds a reference to the + polling island (and that there is no racing pi_add_ref() call either). Also, if we are deleting the polling island and the merged_to field is non-empty, we should remove a ref to the merged_to polling island */ - if (old_cnt == ref_cnt) { + if (gpr_unref(&pi->ref_count)) { polling_island *next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); polling_island_delete(pi); if (next != NULL) { PI_UNREF(next, "pi_delete"); /* Recursive call */ } - } else { - GPR_ASSERT(old_cnt > ref_cnt); } - - return old_cnt; } /* The caller is expected to hold pi->mu lock before calling this function */ @@ -482,7 +475,7 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, pi->fds = NULL; } - gpr_atm_rel_store(&pi->ref_count, (gpr_atm)0); + gpr_ref_init(&pi->ref_count, 0); gpr_atm_rel_store(&pi->merged_to, (gpr_atm)NULL); pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); @@ -762,8 +755,8 @@ static gpr_mu fd_freelist_mu; #define UNREF_BY(fd, n, reason) unref_by(fd, n, reason, __FILE__, __LINE__) static void ref_by(grpc_fd *fd, int n, const char *reason, const char *file, int line) { - gpr_log(GPR_DEBUG, "FD %d %p ref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, - gpr_atm_no_barrier_load(&fd->refst), + gpr_log(GPR_DEBUG, "FD %d %p ref %d %ld -> %ld [%s; %s:%d]", fd->fd, + (void *)fd, n, gpr_atm_no_barrier_load(&fd->refst), gpr_atm_no_barrier_load(&fd->refst) + n, reason, file, line); #else #define REF_BY(fd, n, reason) ref_by(fd, n) @@ -777,8 +770,8 @@ static void ref_by(grpc_fd *fd, int n) { static void unref_by(grpc_fd *fd, int n, const char *reason, const char *file, int line) { gpr_atm old; - gpr_log(GPR_DEBUG, "FD %d %p unref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, - gpr_atm_no_barrier_load(&fd->refst), + gpr_log(GPR_DEBUG, "FD %d %p unref %d %ld -> %ld [%s; %s:%d]", fd->fd, + (void *)fd, n, gpr_atm_no_barrier_load(&fd->refst), gpr_atm_no_barrier_load(&fd->refst) - n, reason, file, line); #else static void unref_by(grpc_fd *fd, int n) { @@ -865,10 +858,10 @@ static grpc_fd *fd_create(int fd, const char *name) { char *fd_name; gpr_asprintf(&fd_name, "%s fd=%d", name, fd); grpc_iomgr_register_object(&new_fd->iomgr_object, fd_name); - gpr_free(fd_name); #ifdef GRPC_FD_REF_COUNT_DEBUG - gpr_log(GPR_DEBUG, "FD %d %p create %s", fd, r, fd_name); + gpr_log(GPR_DEBUG, "FD %d %p create %s", fd, (void *)new_fd, fd_name); #endif + gpr_free(fd_name); return new_fd; } From eb5e43762be40f78775dfd3728e105daf1169d90 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 23 Jun 2016 16:16:10 -0700 Subject: [PATCH 0226/1039] Fix ruby tests --- tools/run_tests/run_tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index c571a81eb28..61cef0a9d57 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -451,7 +451,7 @@ class RubyLanguage(object): _check_compiler(self.args.compiler, ['default']) def test_specs(self): - return [self.config.job_spec(['tools/run_tests/run_ruby.sh'], None, + return [self.config.job_spec(['tools/run_tests/run_ruby.sh'], timeout_seconds=10*60, environ=_FORCE_ENVIRON_FOR_WRAPPERS)] From 2e1a1fe56f4276d670362c5aae1b493df0834c2e Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 23 Jun 2016 16:18:31 -0700 Subject: [PATCH 0227/1039] Fixes --- tools/run_tests/dockerjob.py | 4 ++-- tools/run_tests/run_performance_tests.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tools/run_tests/dockerjob.py b/tools/run_tests/dockerjob.py index 326c4faed95..e4ca3b7faaf 100755 --- a/tools/run_tests/dockerjob.py +++ b/tools/run_tests/dockerjob.py @@ -104,7 +104,7 @@ class DockerJob: def __init__(self, spec): self._spec = spec - self._job = jobset.Job(spec, bin_hash=None, newline_on_success=True, travis=True, add_env={}) + self._job = jobset.Job(spec, newline_on_success=True, travis=True, add_env={}) self._container_name = spec.container_name def mapped_port(self, port): @@ -118,4 +118,4 @@ class DockerJob: def is_running(self): """Polls a job and returns True if given job is still running.""" - return self._job.state(jobset.NoCache()) == jobset._RUNNING + return self._job.state() == jobset._RUNNING diff --git a/tools/run_tests/run_performance_tests.py b/tools/run_tests/run_performance_tests.py index f037d0d17d9..14901caf07f 100755 --- a/tools/run_tests/run_performance_tests.py +++ b/tools/run_tests/run_performance_tests.py @@ -61,11 +61,11 @@ class QpsWorkerJob: self._spec = spec self.language = language self.host_and_port = host_and_port - self._job = jobset.Job(spec, bin_hash=None, newline_on_success=True, travis=True, add_env={}) + self._job = jobset.Job(spec, newline_on_success=True, travis=True, add_env={}) def is_running(self): """Polls a job and returns True if given job is still running.""" - return self._job.state(jobset.NoCache()) == jobset._RUNNING + return self._job.state() == jobset._RUNNING def kill(self): return self._job.kill() From d6f8f0b7cd9c1e92219027919eb8a562763e029b Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Thu, 23 Jun 2016 17:33:01 -0700 Subject: [PATCH 0228/1039] Add TODO --- src/objective-c/examples/SwiftSample/ViewController.swift | 1 + 1 file changed, 1 insertion(+) diff --git a/src/objective-c/examples/SwiftSample/ViewController.swift b/src/objective-c/examples/SwiftSample/ViewController.swift index 80d7a47917f..2a95d2de516 100644 --- a/src/objective-c/examples/SwiftSample/ViewController.swift +++ b/src/objective-c/examples/SwiftSample/ViewController.swift @@ -71,6 +71,7 @@ class ViewController: UIViewController { NSLog("2. Response trailers: \(RPC.responseTrailers)") } + // TODO(jcanizales): Revert to using subscript syntax once XCode 8 is released. RPC.requestHeaders.setObject("My value", forKey: "My-Header") RPC.start() From 7f8db257974a81e16d6865e8bf030a0bd69c10d3 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 24 Jun 2016 08:24:15 -0700 Subject: [PATCH 0229/1039] Bool-ify a couple of fields in grpc_transport_op. --- src/core/lib/surface/channel.c | 2 +- src/core/lib/surface/server.c | 12 +++++++----- src/core/lib/transport/transport.h | 4 ++-- 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/src/core/lib/surface/channel.c b/src/core/lib/surface/channel.c index f0b3c2e15d1..905b4a97dd5 100644 --- a/src/core/lib/surface/channel.c +++ b/src/core/lib/surface/channel.c @@ -336,7 +336,7 @@ void grpc_channel_destroy(grpc_channel *channel) { grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; GRPC_API_TRACE("grpc_channel_destroy(channel=%p)", 1, (channel)); memset(&op, 0, sizeof(op)); - op.disconnect = 1; + op.disconnect = true; elem = grpc_channel_stack_element(CHANNEL_STACK_FROM_CHANNEL(channel), 0); elem->filter->start_transport_op(&exec_ctx, elem, &op); diff --git a/src/core/lib/surface/server.c b/src/core/lib/surface/server.c index 88d898bc1c0..6677e65c4c8 100644 --- a/src/core/lib/surface/server.c +++ b/src/core/lib/surface/server.c @@ -270,7 +270,7 @@ static void shutdown_cleanup(grpc_exec_ctx *exec_ctx, void *arg, } static void send_shutdown(grpc_exec_ctx *exec_ctx, grpc_channel *channel, - int send_goaway, int send_disconnect) { + bool send_goaway, bool send_disconnect) { grpc_transport_op op; struct shutdown_cleanup_args *sc; grpc_channel_element *elem; @@ -291,8 +291,8 @@ static void send_shutdown(grpc_exec_ctx *exec_ctx, grpc_channel *channel, static void channel_broadcaster_shutdown(grpc_exec_ctx *exec_ctx, channel_broadcaster *cb, - int send_goaway, - int force_disconnect) { + bool send_goaway, + bool force_disconnect) { size_t i; for (i = 0; i < cb->num_channels; i++) { @@ -1217,7 +1217,8 @@ void grpc_server_shutdown_and_notify(grpc_server *server, l->destroy(&exec_ctx, server, l->arg, &l->destroy_done); } - channel_broadcaster_shutdown(&exec_ctx, &broadcaster, 1, 0); + channel_broadcaster_shutdown(&exec_ctx, &broadcaster, true /* send_goaway */, + false /* force_disconnect */); done: grpc_exec_ctx_finish(&exec_ctx); @@ -1233,7 +1234,8 @@ void grpc_server_cancel_all_calls(grpc_server *server) { channel_broadcaster_init(server, &broadcaster); gpr_mu_unlock(&server->mu_global); - channel_broadcaster_shutdown(&exec_ctx, &broadcaster, 0, 1); + channel_broadcaster_shutdown(&exec_ctx, &broadcaster, false /* send_goaway */, + true /* force_disconnect */); grpc_exec_ctx_finish(&exec_ctx); } diff --git a/src/core/lib/transport/transport.h b/src/core/lib/transport/transport.h index 107bb802f66..b170014dc59 100644 --- a/src/core/lib/transport/transport.h +++ b/src/core/lib/transport/transport.h @@ -159,11 +159,11 @@ typedef struct grpc_transport_op { grpc_closure *on_connectivity_state_change; grpc_connectivity_state *connectivity_state; /** should the transport be disconnected */ - int disconnect; + bool disconnect; /** should we send a goaway? after a goaway is sent, once there are no more active calls on the transport, the transport should disconnect */ - int send_goaway; + bool send_goaway; /** what should the goaway contain? */ grpc_status_code goaway_status; gpr_slice *goaway_message; From 33b767a7701c373767bf29a49c93c9e515eb17f8 Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Fri, 24 Jun 2016 09:13:03 -0700 Subject: [PATCH 0230/1039] fix build node package --- tools/run_tests/build_package_node.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/run_tests/build_package_node.sh b/tools/run_tests/build_package_node.sh index 6f7211b53fe..ff4cfdb8bf9 100755 --- a/tools/run_tests/build_package_node.sh +++ b/tools/run_tests/build_package_node.sh @@ -86,6 +86,7 @@ for arch in {x86,x64}; do cp $input_dir/protoc* bin/ cp $input_dir/grpc_node_plugin* bin/ mkdir -p bin/google/protobuf + mkdir -p bin/google/protobuf/compiler # needed for plugin.proto for proto in "${well_known_protos[@]}"; do cp $base/third_party/protobuf/src/google/protobuf/$proto.proto bin/google/protobuf/$proto.proto done From e07b83ba9fde162fe0e61f97c657492f4d948e95 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Fri, 24 Jun 2016 09:27:00 -0700 Subject: [PATCH 0231/1039] generate_projects.sh --- src/node/tools/package.json | 1 + 1 file changed, 1 insertion(+) diff --git a/src/node/tools/package.json b/src/node/tools/package.json index d4849c2e388..c34b259e2e0 100644 --- a/src/node/tools/package.json +++ b/src/node/tools/package.json @@ -34,6 +34,7 @@ "index.js", "bin/protoc.js", "bin/protoc_plugin.js", + "bin/google/protobuf", "LICENSE" ], "main": "index.js" From 886c512832464a5aa5e61b51f1fcd1da356ca847 Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Fri, 24 Jun 2016 09:48:58 -0700 Subject: [PATCH 0232/1039] fix C# nuget build --- src/csharp/Grpc.Core/Grpc.Core.nuspec | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/src/csharp/Grpc.Core/Grpc.Core.nuspec b/src/csharp/Grpc.Core/Grpc.Core.nuspec index fa2c1fbff22..47593f787b1 100644 --- a/src/csharp/Grpc.Core/Grpc.Core.nuspec +++ b/src/csharp/Grpc.Core/Grpc.Core.nuspec @@ -24,11 +24,12 @@ - - - - - - + + + + + + + From 13e343464bd8abd01cec1df2b43f624906552055 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 24 Jun 2016 10:24:43 -0700 Subject: [PATCH 0233/1039] Another boolification. --- src/core/lib/transport/metadata_batch.c | 3 ++- src/core/lib/transport/metadata_batch.h | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/src/core/lib/transport/metadata_batch.c b/src/core/lib/transport/metadata_batch.c index e4398abeb78..84b5a74d513 100644 --- a/src/core/lib/transport/metadata_batch.c +++ b/src/core/lib/transport/metadata_batch.c @@ -33,6 +33,7 @@ #include "src/core/lib/transport/metadata_batch.h" +#include #include #include @@ -187,7 +188,7 @@ void grpc_metadata_batch_clear(grpc_metadata_batch *batch) { grpc_metadata_batch_filter(batch, no_metadata_for_you, NULL); } -int grpc_metadata_batch_is_empty(grpc_metadata_batch *batch) { +bool grpc_metadata_batch_is_empty(grpc_metadata_batch *batch) { return batch->list.head == NULL && gpr_time_cmp(gpr_inf_future(batch->deadline.clock_type), batch->deadline) == 0; diff --git a/src/core/lib/transport/metadata_batch.h b/src/core/lib/transport/metadata_batch.h index 7af823f7ca4..dad0cc55100 100644 --- a/src/core/lib/transport/metadata_batch.h +++ b/src/core/lib/transport/metadata_batch.h @@ -34,6 +34,8 @@ #ifndef GRPC_CORE_LIB_TRANSPORT_METADATA_BATCH_H #define GRPC_CORE_LIB_TRANSPORT_METADATA_BATCH_H +#include + #include #include #include @@ -64,7 +66,7 @@ typedef struct grpc_metadata_batch { void grpc_metadata_batch_init(grpc_metadata_batch *batch); void grpc_metadata_batch_destroy(grpc_metadata_batch *batch); void grpc_metadata_batch_clear(grpc_metadata_batch *batch); -int grpc_metadata_batch_is_empty(grpc_metadata_batch *batch); +bool grpc_metadata_batch_is_empty(grpc_metadata_batch *batch); /* Returns the transport size of the batch. */ size_t grpc_metadata_batch_size(grpc_metadata_batch *batch); From d29a3bf004e23eb481b33b9a69fd10ae9221fc98 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Fri, 24 Jun 2016 10:26:14 -0700 Subject: [PATCH 0234/1039] Update master branch to 0.16.0-dev --- Makefile | 2 +- build.yaml | 2 +- package.json | 2 +- package.xml | 4 ++-- src/core/lib/surface/version.c | 2 +- src/csharp/Grpc.Auth/project.json | 4 ++-- src/csharp/Grpc.Core/VersionInfo.cs | 4 ++-- src/csharp/Grpc.Core/project.json | 2 +- src/csharp/Grpc.HealthCheck/project.json | 4 ++-- src/csharp/build_packages.bat | 2 +- src/node/tools/package.json | 2 +- src/python/grpcio/grpc_version.py | 2 +- src/ruby/lib/grpc/version.rb | 2 +- src/ruby/tools/version.rb | 2 +- tools/distrib/python/grpcio_tools/grpc_version.py | 2 +- tools/doxygen/Doxyfile.c++ | 2 +- tools/doxygen/Doxyfile.c++.internal | 2 +- tools/doxygen/Doxyfile.core | 2 +- tools/doxygen/Doxyfile.core.internal | 2 +- 19 files changed, 23 insertions(+), 23 deletions(-) diff --git a/Makefile b/Makefile index 9be3e5784cc..2f49e7712fd 100644 --- a/Makefile +++ b/Makefile @@ -414,7 +414,7 @@ E = @echo Q = @ endif -VERSION = 0.15.0-dev +VERSION = 0.16.0-dev CPPFLAGS_NO_ARCH += $(addprefix -I, $(INCLUDES)) $(addprefix -D, $(DEFINES)) CPPFLAGS += $(CPPFLAGS_NO_ARCH) $(ARCH_FLAGS) diff --git a/build.yaml b/build.yaml index 1f06e20cf9b..7d570e4dab9 100644 --- a/build.yaml +++ b/build.yaml @@ -7,7 +7,7 @@ settings: '#3': Use "-preN" suffixes to identify pre-release versions '#4': Per-language overrides are possible with (eg) ruby_version tag here '#5': See the expand_version.py for all the quirks here - version: 0.15.0-dev + version: 0.16.0-dev filegroups: - name: census public_headers: diff --git a/package.json b/package.json index 5bdaa761e28..68a31d794c5 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "grpc", - "version": "0.15.0-dev", + "version": "0.16.0-dev", "author": "Google Inc.", "description": "gRPC Library for Node", "homepage": "http://www.grpc.io/", diff --git a/package.xml b/package.xml index 67e9bb2c282..c1ecae8b806 100644 --- a/package.xml +++ b/package.xml @@ -13,8 +13,8 @@ 2016-05-19 - 0.15.0 - 0.15.0 + 0.16.0 + 0.16.0 beta diff --git a/src/core/lib/surface/version.c b/src/core/lib/surface/version.c index aca76d2bb79..53f3c438541 100644 --- a/src/core/lib/surface/version.c +++ b/src/core/lib/surface/version.c @@ -36,4 +36,4 @@ #include -const char *grpc_version_string(void) { return "0.15.0-dev"; } +const char *grpc_version_string(void) { return "0.16.0-dev"; } diff --git a/src/csharp/Grpc.Auth/project.json b/src/csharp/Grpc.Auth/project.json index 1677565824b..4c5c960204e 100644 --- a/src/csharp/Grpc.Auth/project.json +++ b/src/csharp/Grpc.Auth/project.json @@ -1,5 +1,5 @@ { - "version": "0.15.0-dev", + "version": "0.16.0-dev", "title": "gRPC C# Auth", "authors": [ "Google Inc." ], "copyright": "Copyright 2015, Google Inc.", @@ -13,7 +13,7 @@ "tags": [ "gRPC RPC Protocol HTTP/2 Auth OAuth2" ], }, "dependencies": { - "Grpc.Core": "0.15.0-dev", + "Grpc.Core": "0.16.0-dev", "Google.Apis.Auth": "1.11.1" }, "frameworks": { diff --git a/src/csharp/Grpc.Core/VersionInfo.cs b/src/csharp/Grpc.Core/VersionInfo.cs index e1609341d9a..cb20967680e 100644 --- a/src/csharp/Grpc.Core/VersionInfo.cs +++ b/src/csharp/Grpc.Core/VersionInfo.cs @@ -48,11 +48,11 @@ namespace Grpc.Core /// /// Current AssemblyFileVersion of gRPC C# assemblies /// - public const string CurrentAssemblyFileVersion = "0.15.0.0"; + public const string CurrentAssemblyFileVersion = "0.16.0.0"; /// /// Current version of gRPC C# /// - public const string CurrentVersion = "0.15.0-dev"; + public const string CurrentVersion = "0.16.0-dev"; } } diff --git a/src/csharp/Grpc.Core/project.json b/src/csharp/Grpc.Core/project.json index 7253107e04a..4729a9346cb 100644 --- a/src/csharp/Grpc.Core/project.json +++ b/src/csharp/Grpc.Core/project.json @@ -1,5 +1,5 @@ { - "version": "0.15.0-dev", + "version": "0.16.0-dev", "title": "gRPC C# Core", "authors": [ "Google Inc." ], "copyright": "Copyright 2015, Google Inc.", diff --git a/src/csharp/Grpc.HealthCheck/project.json b/src/csharp/Grpc.HealthCheck/project.json index eb57608957a..c4895c2ad3c 100644 --- a/src/csharp/Grpc.HealthCheck/project.json +++ b/src/csharp/Grpc.HealthCheck/project.json @@ -1,5 +1,5 @@ { - "version": "0.15.0-dev", + "version": "0.16.0-dev", "title": "gRPC C# Healthchecking", "authors": [ "Google Inc." ], "copyright": "Copyright 2015, Google Inc.", @@ -13,7 +13,7 @@ "tags": [ "gRPC health check" ] }, "dependencies": { - "Grpc.Core": "0.15.0-dev", + "Grpc.Core": "0.16.0-dev", "Google.Protobuf": "3.0.0-beta3" }, "frameworks": { diff --git a/src/csharp/build_packages.bat b/src/csharp/build_packages.bat index 63f8c30bc7e..272b30f385d 100644 --- a/src/csharp/build_packages.bat +++ b/src/csharp/build_packages.bat @@ -30,7 +30,7 @@ @rem Builds gRPC NuGet packages @rem Current package versions -set VERSION=0.15.0-dev +set VERSION=0.16.0-dev set PROTOBUF_VERSION=3.0.0-beta3 @rem Packages that depend on prerelease packages (like Google.Protobuf) need to have prerelease suffix as well. diff --git a/src/node/tools/package.json b/src/node/tools/package.json index c34b259e2e0..7c256d7ba0d 100644 --- a/src/node/tools/package.json +++ b/src/node/tools/package.json @@ -1,6 +1,6 @@ { "name": "grpc-tools", - "version": "0.15.0-dev", + "version": "0.16.0-dev", "author": "Google Inc.", "description": "Tools for developing with gRPC on Node.js", "homepage": "http://www.grpc.io/", diff --git a/src/python/grpcio/grpc_version.py b/src/python/grpcio/grpc_version.py index 0c13104d9da..0f4db9d972e 100644 --- a/src/python/grpcio/grpc_version.py +++ b/src/python/grpcio/grpc_version.py @@ -29,4 +29,4 @@ # AUTO-GENERATED FROM `$REPO_ROOT/templates/src/python/grpcio/grpc_version.py.template`!!! -VERSION='0.15.0.dev0' +VERSION='0.16.0.dev0' diff --git a/src/ruby/lib/grpc/version.rb b/src/ruby/lib/grpc/version.rb index 01c8c5ac8f1..5e6aaef2eb5 100644 --- a/src/ruby/lib/grpc/version.rb +++ b/src/ruby/lib/grpc/version.rb @@ -29,5 +29,5 @@ # GRPC contains the General RPC module. module GRPC - VERSION = '0.15.0.dev' + VERSION = '0.16.0.dev' end diff --git a/src/ruby/tools/version.rb b/src/ruby/tools/version.rb index dca7fd7e72c..68c1bf369d2 100644 --- a/src/ruby/tools/version.rb +++ b/src/ruby/tools/version.rb @@ -29,6 +29,6 @@ module GRPC module Tools - VERSION = '0.15.0.dev' + VERSION = '0.16.0.dev' end end diff --git a/tools/distrib/python/grpcio_tools/grpc_version.py b/tools/distrib/python/grpcio_tools/grpc_version.py index 1267d0e45dc..4b1e7fcd584 100644 --- a/tools/distrib/python/grpcio_tools/grpc_version.py +++ b/tools/distrib/python/grpcio_tools/grpc_version.py @@ -29,4 +29,4 @@ # AUTO-GENERATED FROM `$REPO_ROOT/templates/tools/distrib/python/grpcio_tools/grpc_version.py.template`!!! -VERSION='0.15.0.dev0' +VERSION='0.16.0.dev0' diff --git a/tools/doxygen/Doxyfile.c++ b/tools/doxygen/Doxyfile.c++ index 7f9d2df6f6c..de7acd7777e 100644 --- a/tools/doxygen/Doxyfile.c++ +++ b/tools/doxygen/Doxyfile.c++ @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC C++" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0-dev +PROJECT_NUMBER = 0.16.0-dev # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a diff --git a/tools/doxygen/Doxyfile.c++.internal b/tools/doxygen/Doxyfile.c++.internal index dcf1a4c8c40..76bb3b6c59e 100644 --- a/tools/doxygen/Doxyfile.c++.internal +++ b/tools/doxygen/Doxyfile.c++.internal @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC C++" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0-dev +PROJECT_NUMBER = 0.16.0-dev # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a diff --git a/tools/doxygen/Doxyfile.core b/tools/doxygen/Doxyfile.core index 72102b2fc50..53ae4e4cf4b 100644 --- a/tools/doxygen/Doxyfile.core +++ b/tools/doxygen/Doxyfile.core @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC Core" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0-dev +PROJECT_NUMBER = 0.16.0-dev # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index e1555930e91..1c75c941c27 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC Core" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0-dev +PROJECT_NUMBER = 0.16.0-dev # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a From f4c1bff0d8caedf716b460c4ee6114ac126e8872 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Sun, 12 Jun 2016 15:23:36 -0700 Subject: [PATCH 0235/1039] Moved grpc_shutdown to end of Py_Finalize() We currently rely on the __del__ method of a module scope object to call grpc_shutdown(). __del__ methods are not guaranteed to be called, and furthermore there are no guarantees about ordering, leading to shutdown race conditions. This moves grpc_shutdown to Py_Finalize(), which gets called after the Python context is completely cleaned up. --- .../grpcio/grpc/_cython/_cygrpc/grpc.pxi | 1 + src/python/grpcio/grpc/_cython/cygrpc.pyx | 32 +++++++------------ src/python/grpcio/grpc/_cython/loader.c | 7 ++++ src/python/grpcio/grpc/_cython/loader.h | 5 +++ 4 files changed, 24 insertions(+), 21 deletions(-) diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi index 05b8886df73..35e394d02f8 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi @@ -37,6 +37,7 @@ cdef extern from "grpc/_cython/loader.h": ctypedef long int64_t int pygrpc_load_core(char*) + int pygrpc_initialize_core() void *gpr_malloc(size_t size) nogil void gpr_free(void *ptr) nogil diff --git a/src/python/grpcio/grpc/_cython/cygrpc.pyx b/src/python/grpcio/grpc/_cython/cygrpc.pyx index cf146f5a048..c92a8d19a77 100644 --- a/src/python/grpcio/grpc/_cython/cygrpc.pyx +++ b/src/python/grpcio/grpc/_cython/cygrpc.pyx @@ -45,30 +45,20 @@ include "grpc/_cython/_cygrpc/security.pyx.pxi" include "grpc/_cython/_cygrpc/server.pyx.pxi" # -# Global state +# initialize gRPC # -cdef class _ModuleState: - cdef bint is_loaded +def _initialize(): + if 'win32' in sys.platform: + filename = pkg_resources.resource_filename( + 'grpc._cython', '_windows/grpc_c.64.python') + if not pygrpc_load_core(filename): + raise ImportError('failed to load core gRPC library') + if not pygrpc_initialize_core(): + raise ImportError('failed to initialize core gRPC library') - def __cinit__(self): - if 'win32' in sys.platform: - filename = pkg_resources.resource_filename( - 'grpc._cython', '_windows/grpc_c.64.python') - if not pygrpc_load_core(filename): - raise ImportError('failed to load core gRPC library') - with nogil: - grpc_init() - self.is_loaded = True - with nogil: - grpc_set_ssl_roots_override_callback( + grpc_set_ssl_roots_override_callback( ssl_roots_override_callback) - def __dealloc__(self): - if self.is_loaded: - with nogil: - grpc_shutdown() - -_module_state = _ModuleState() - +_initialize() diff --git a/src/python/grpcio/grpc/_cython/loader.c b/src/python/grpcio/grpc/_cython/loader.c index b909ad594ed..86b70dbb02d 100644 --- a/src/python/grpcio/grpc/_cython/loader.c +++ b/src/python/grpcio/grpc/_cython/loader.c @@ -31,6 +31,7 @@ * */ +#include #include "loader.h" #ifdef __cplusplus @@ -62,6 +63,12 @@ int pygrpc_load_core(char *path) { return 1; } #endif /* !GPR_WINDOWS */ +// Cython doesn't have Py_AtExit bindings, so we call the C_API directly +int pygrpc_initialize_core(void) { + grpc_init(); + return Py_AtExit(grpc_shutdown) < 0 ? 0 : 1; +} + #ifdef __cplusplus } #endif /* __cpluslus */ diff --git a/src/python/grpcio/grpc/_cython/loader.h b/src/python/grpcio/grpc/_cython/loader.h index 3b8796d39f7..eb4b1a1b018 100644 --- a/src/python/grpcio/grpc/_cython/loader.h +++ b/src/python/grpcio/grpc/_cython/loader.h @@ -46,6 +46,11 @@ extern "C" { /* Attempts to load the core if necessary, and return non-zero upon succes. */ int pygrpc_load_core(char *path); +/* Initializes grpc and registers grpc_shutdown() to be called right before + * interpreter exit. Returns non-zero upon success. + */ +int pygrpc_initialize_core(void); + #ifdef __cplusplus } #endif /* __cpluslus */ From 2dfcf14705f24bdf6d95d1b16265beb52fe33b02 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Fri, 24 Jun 2016 11:06:28 -0700 Subject: [PATCH 0236/1039] Fix Node Windows build error --- src/node/ext/node_grpc.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/node/ext/node_grpc.cc b/src/node/ext/node_grpc.cc index ce988f9dfa9..745b5023d59 100644 --- a/src/node/ext/node_grpc.cc +++ b/src/node/ext/node_grpc.cc @@ -265,8 +265,8 @@ void InitLogConstants(Local exports) { Nan::Set(log_verbosity, Nan::New("DEBUG").ToLocalChecked(), DEBUG); Local INFO(Nan::New(GPR_LOG_SEVERITY_INFO)); Nan::Set(log_verbosity, Nan::New("INFO").ToLocalChecked(), INFO); - Local ERROR(Nan::New(GPR_LOG_SEVERITY_ERROR)); - Nan::Set(log_verbosity, Nan::New("ERROR").ToLocalChecked(), ERROR); + Local LOG_ERROR(Nan::New(GPR_LOG_SEVERITY_ERROR)); + Nan::Set(log_verbosity, Nan::New("ERROR").ToLocalChecked(), LOG_ERROR); } NAN_METHOD(MetadataKeyIsLegal) { From f519df8147e80b6ef21a576e8e08742e4f957e87 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Fri, 24 Jun 2016 11:12:03 -0700 Subject: [PATCH 0237/1039] Update version to 0.15.0 --- Makefile | 2 +- build.yaml | 2 +- package.json | 2 +- src/core/lib/surface/version.c | 2 +- src/csharp/Grpc.Auth/project.json | 4 ++-- src/csharp/Grpc.Core/VersionInfo.cs | 2 +- src/csharp/Grpc.Core/project.json | 2 +- src/csharp/Grpc.HealthCheck/project.json | 4 ++-- src/csharp/build_packages.bat | 2 +- src/node/tools/package.json | 2 +- src/python/grpcio/grpc_version.py | 2 +- src/ruby/lib/grpc/version.rb | 2 +- src/ruby/tools/version.rb | 2 +- tools/distrib/python/grpcio_tools/grpc_version.py | 2 +- tools/doxygen/Doxyfile.c++ | 2 +- tools/doxygen/Doxyfile.c++.internal | 2 +- tools/doxygen/Doxyfile.core | 2 +- tools/doxygen/Doxyfile.core.internal | 2 +- 18 files changed, 20 insertions(+), 20 deletions(-) diff --git a/Makefile b/Makefile index 9be3e5784cc..d369653be3c 100644 --- a/Makefile +++ b/Makefile @@ -414,7 +414,7 @@ E = @echo Q = @ endif -VERSION = 0.15.0-dev +VERSION = 0.15.0 CPPFLAGS_NO_ARCH += $(addprefix -I, $(INCLUDES)) $(addprefix -D, $(DEFINES)) CPPFLAGS += $(CPPFLAGS_NO_ARCH) $(ARCH_FLAGS) diff --git a/build.yaml b/build.yaml index 1f06e20cf9b..ffc681d9028 100644 --- a/build.yaml +++ b/build.yaml @@ -7,7 +7,7 @@ settings: '#3': Use "-preN" suffixes to identify pre-release versions '#4': Per-language overrides are possible with (eg) ruby_version tag here '#5': See the expand_version.py for all the quirks here - version: 0.15.0-dev + version: 0.15.0 filegroups: - name: census public_headers: diff --git a/package.json b/package.json index 5bdaa761e28..1b2920c6bc4 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "grpc", - "version": "0.15.0-dev", + "version": "0.15.0", "author": "Google Inc.", "description": "gRPC Library for Node", "homepage": "http://www.grpc.io/", diff --git a/src/core/lib/surface/version.c b/src/core/lib/surface/version.c index aca76d2bb79..e4a3358c351 100644 --- a/src/core/lib/surface/version.c +++ b/src/core/lib/surface/version.c @@ -36,4 +36,4 @@ #include -const char *grpc_version_string(void) { return "0.15.0-dev"; } +const char *grpc_version_string(void) { return "0.15.0"; } diff --git a/src/csharp/Grpc.Auth/project.json b/src/csharp/Grpc.Auth/project.json index 1677565824b..ae83c3be2b8 100644 --- a/src/csharp/Grpc.Auth/project.json +++ b/src/csharp/Grpc.Auth/project.json @@ -1,5 +1,5 @@ { - "version": "0.15.0-dev", + "version": "0.15.0", "title": "gRPC C# Auth", "authors": [ "Google Inc." ], "copyright": "Copyright 2015, Google Inc.", @@ -13,7 +13,7 @@ "tags": [ "gRPC RPC Protocol HTTP/2 Auth OAuth2" ], }, "dependencies": { - "Grpc.Core": "0.15.0-dev", + "Grpc.Core": "0.15.0", "Google.Apis.Auth": "1.11.1" }, "frameworks": { diff --git a/src/csharp/Grpc.Core/VersionInfo.cs b/src/csharp/Grpc.Core/VersionInfo.cs index e1609341d9a..d89a2b5c6ee 100644 --- a/src/csharp/Grpc.Core/VersionInfo.cs +++ b/src/csharp/Grpc.Core/VersionInfo.cs @@ -53,6 +53,6 @@ namespace Grpc.Core /// /// Current version of gRPC C# /// - public const string CurrentVersion = "0.15.0-dev"; + public const string CurrentVersion = "0.15.0"; } } diff --git a/src/csharp/Grpc.Core/project.json b/src/csharp/Grpc.Core/project.json index 7253107e04a..c7f2bf4fb12 100644 --- a/src/csharp/Grpc.Core/project.json +++ b/src/csharp/Grpc.Core/project.json @@ -1,5 +1,5 @@ { - "version": "0.15.0-dev", + "version": "0.15.0", "title": "gRPC C# Core", "authors": [ "Google Inc." ], "copyright": "Copyright 2015, Google Inc.", diff --git a/src/csharp/Grpc.HealthCheck/project.json b/src/csharp/Grpc.HealthCheck/project.json index eb57608957a..98ea21a436a 100644 --- a/src/csharp/Grpc.HealthCheck/project.json +++ b/src/csharp/Grpc.HealthCheck/project.json @@ -1,5 +1,5 @@ { - "version": "0.15.0-dev", + "version": "0.15.0", "title": "gRPC C# Healthchecking", "authors": [ "Google Inc." ], "copyright": "Copyright 2015, Google Inc.", @@ -13,7 +13,7 @@ "tags": [ "gRPC health check" ] }, "dependencies": { - "Grpc.Core": "0.15.0-dev", + "Grpc.Core": "0.15.0", "Google.Protobuf": "3.0.0-beta3" }, "frameworks": { diff --git a/src/csharp/build_packages.bat b/src/csharp/build_packages.bat index 63f8c30bc7e..e387efcc2da 100644 --- a/src/csharp/build_packages.bat +++ b/src/csharp/build_packages.bat @@ -30,7 +30,7 @@ @rem Builds gRPC NuGet packages @rem Current package versions -set VERSION=0.15.0-dev +set VERSION=0.15.0 set PROTOBUF_VERSION=3.0.0-beta3 @rem Packages that depend on prerelease packages (like Google.Protobuf) need to have prerelease suffix as well. diff --git a/src/node/tools/package.json b/src/node/tools/package.json index c34b259e2e0..b2cadd3f47a 100644 --- a/src/node/tools/package.json +++ b/src/node/tools/package.json @@ -1,6 +1,6 @@ { "name": "grpc-tools", - "version": "0.15.0-dev", + "version": "0.15.0", "author": "Google Inc.", "description": "Tools for developing with gRPC on Node.js", "homepage": "http://www.grpc.io/", diff --git a/src/python/grpcio/grpc_version.py b/src/python/grpcio/grpc_version.py index 0c13104d9da..c6c07afb443 100644 --- a/src/python/grpcio/grpc_version.py +++ b/src/python/grpcio/grpc_version.py @@ -29,4 +29,4 @@ # AUTO-GENERATED FROM `$REPO_ROOT/templates/src/python/grpcio/grpc_version.py.template`!!! -VERSION='0.15.0.dev0' +VERSION='0.15.0' diff --git a/src/ruby/lib/grpc/version.rb b/src/ruby/lib/grpc/version.rb index 01c8c5ac8f1..7f512e47aab 100644 --- a/src/ruby/lib/grpc/version.rb +++ b/src/ruby/lib/grpc/version.rb @@ -29,5 +29,5 @@ # GRPC contains the General RPC module. module GRPC - VERSION = '0.15.0.dev' + VERSION = '0.15.0' end diff --git a/src/ruby/tools/version.rb b/src/ruby/tools/version.rb index dca7fd7e72c..6a7a1d5bd33 100644 --- a/src/ruby/tools/version.rb +++ b/src/ruby/tools/version.rb @@ -29,6 +29,6 @@ module GRPC module Tools - VERSION = '0.15.0.dev' + VERSION = '0.15.0' end end diff --git a/tools/distrib/python/grpcio_tools/grpc_version.py b/tools/distrib/python/grpcio_tools/grpc_version.py index 1267d0e45dc..9a33c6e5d14 100644 --- a/tools/distrib/python/grpcio_tools/grpc_version.py +++ b/tools/distrib/python/grpcio_tools/grpc_version.py @@ -29,4 +29,4 @@ # AUTO-GENERATED FROM `$REPO_ROOT/templates/tools/distrib/python/grpcio_tools/grpc_version.py.template`!!! -VERSION='0.15.0.dev0' +VERSION='0.15.0' diff --git a/tools/doxygen/Doxyfile.c++ b/tools/doxygen/Doxyfile.c++ index 7f9d2df6f6c..066d29ac001 100644 --- a/tools/doxygen/Doxyfile.c++ +++ b/tools/doxygen/Doxyfile.c++ @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC C++" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0-dev +PROJECT_NUMBER = 0.15.0 # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a diff --git a/tools/doxygen/Doxyfile.c++.internal b/tools/doxygen/Doxyfile.c++.internal index dcf1a4c8c40..6a0e8b2129c 100644 --- a/tools/doxygen/Doxyfile.c++.internal +++ b/tools/doxygen/Doxyfile.c++.internal @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC C++" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0-dev +PROJECT_NUMBER = 0.15.0 # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a diff --git a/tools/doxygen/Doxyfile.core b/tools/doxygen/Doxyfile.core index 72102b2fc50..fa9fd5a3123 100644 --- a/tools/doxygen/Doxyfile.core +++ b/tools/doxygen/Doxyfile.core @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC Core" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0-dev +PROJECT_NUMBER = 0.15.0 # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index e1555930e91..e4c9f991d31 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC Core" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0-dev +PROJECT_NUMBER = 0.15.0 # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a From 5e824fa42e691d8e04c6218eb75b26ef80e86918 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Fri, 24 Jun 2016 11:32:51 -0700 Subject: [PATCH 0238/1039] Restore fix undid by https://github.com/grpc/grpc/pull/5893 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit That PR wasn’t tested nor reviewed adecuately. --- src/objective-c/GRPCClient/GRPCCall.m | 1 + src/objective-c/GRPCClient/private/GRPCWrappedCall.m | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/objective-c/GRPCClient/GRPCCall.m b/src/objective-c/GRPCClient/GRPCCall.m index 0eb10656ddc..c34df99d9f4 100644 --- a/src/objective-c/GRPCClient/GRPCCall.m +++ b/src/objective-c/GRPCClient/GRPCCall.m @@ -378,6 +378,7 @@ NSString * const kGRPCTrailersKey = @"io.grpc.TrailersKey"; [strongSelf finishWithError:[NSError errorWithDomain:kGRPCErrorDomain code:GRPCErrorCodeUnavailable userInfo:@{NSLocalizedDescriptionKey: @"Connectivity lost."}]]; + [[GRPCHost hostWithAddress:strongSelf->_host] discardChannel:channel]; } }]; } diff --git a/src/objective-c/GRPCClient/private/GRPCWrappedCall.m b/src/objective-c/GRPCClient/private/GRPCWrappedCall.m index 16e5bff7ff7..160344a7a69 100644 --- a/src/objective-c/GRPCClient/private/GRPCWrappedCall.m +++ b/src/objective-c/GRPCClient/private/GRPCWrappedCall.m @@ -250,7 +250,7 @@ // Each completion queue consumes one thread. There's a trade to be made between creating and // consuming too many threads and having contention of multiple calls in a single completion - // queue. Currently we favor latency and use one per call. + // queue. Currently we use a singleton queue. _queue = [GRPCCompletionQueue completionQueue]; _call = [[GRPCHost hostWithAddress:host] unmanagedCallWithPath:path completionQueue:_queue]; From 06b4d6c353455a2af9fee3bdc57307cbb426d2cf Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Fri, 24 Jun 2016 11:35:32 -0700 Subject: [PATCH 0239/1039] =?UTF-8?q?Don=E2=80=99t=20checkout=20specific?= =?UTF-8?q?=20proto=20versions=20in=20Travis=20anymore?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Now that the submodule points to 3.0.0-beta-3.1, which is what we need. --- .travis.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/.travis.yml b/.travis.yml index 16c6390a54a..75b14406306 100644 --- a/.travis.yml +++ b/.travis.yml @@ -11,9 +11,6 @@ before_install: - gem install cocoapods -v '1.0.0' - pod --version - brew install gflags - - pushd third_party/protobuf - - git checkout v3.0.0-beta-3 - - popd install: - make grpc_objective_c_plugin - pushd src/objective-c/tests From a324c4fea63840eb06dadb4f0d920226b034404e Mon Sep 17 00:00:00 2001 From: yang-g Date: Fri, 24 Jun 2016 11:57:07 -0700 Subject: [PATCH 0240/1039] Add API to get c slice from c++ Slice. --- include/grpc++/support/slice.h | 3 +++ test/cpp/util/slice_test.cc | 10 ++++++++++ 2 files changed, 13 insertions(+) diff --git a/include/grpc++/support/slice.h b/include/grpc++/support/slice.h index cec9062d4fd..5874b4f5ae3 100644 --- a/include/grpc++/support/slice.h +++ b/include/grpc++/support/slice.h @@ -77,6 +77,9 @@ class Slice GRPC_FINAL { /// Raw pointer to the end (one byte \em past the last element) of the slice. const uint8_t* end() const { return GPR_SLICE_END_PTR(slice_); } + /// Raw C slice. Caller needs to call gpr_slice_unref when done. + gpr_slice c_slice() const { return gpr_slice_ref(slice_); } + private: friend class ByteBuffer; diff --git a/test/cpp/util/slice_test.cc b/test/cpp/util/slice_test.cc index de7ff031ab8..45799ae157c 100644 --- a/test/cpp/util/slice_test.cc +++ b/test/cpp/util/slice_test.cc @@ -68,6 +68,16 @@ TEST_F(SliceTest, Empty) { CheckSlice(empty_slice, ""); } +TEST_F(SliceTest, Cslice) { + gpr_slice s = gpr_slice_from_copied_string(kContent); + Slice spp(s, Slice::STEAL_REF); + CheckSlice(spp, kContent); + gpr_slice c_slice = spp.c_slice(); + EXPECT_EQ(GPR_SLICE_START_PTR(s), GPR_SLICE_START_PTR(c_slice)); + EXPECT_EQ(GPR_SLICE_END_PTR(s), GPR_SLICE_END_PTR(c_slice)); + gpr_slice_unref(c_slice); +} + } // namespace } // namespace grpc From be1b9a718a09722e95ab04b08ccaf353dc3a51e9 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 24 Jun 2016 13:22:11 -0700 Subject: [PATCH 0241/1039] Fixes --- src/core/lib/iomgr/error.c | 5 ++++- src/core/lib/iomgr/tcp_posix.c | 4 ++-- src/core/lib/surface/call.c | 5 +++-- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/src/core/lib/iomgr/error.c b/src/core/lib/iomgr/error.c index 9f5ba76fd6c..aa94bb7fdf7 100644 --- a/src/core/lib/iomgr/error.c +++ b/src/core/lib/iomgr/error.c @@ -248,7 +248,10 @@ static grpc_error *copy_error_and_unref(grpc_error *in) { if (is_special(in)) { if (in == GRPC_ERROR_NONE) return GRPC_ERROR_CREATE("no error"); if (in == GRPC_ERROR_OOM) return GRPC_ERROR_CREATE("oom"); - if (in == GRPC_ERROR_CANCELLED) return GRPC_ERROR_CREATE("cancelled"); + if (in == GRPC_ERROR_CANCELLED) + return grpc_error_set_int(GRPC_ERROR_CREATE("cancelled"), + GRPC_ERROR_INT_GRPC_STATUS, + GRPC_STATUS_CANCELLED); return GRPC_ERROR_CREATE("unknown"); } grpc_error *out = gpr_malloc(sizeof(*out)); diff --git a/src/core/lib/iomgr/tcp_posix.c b/src/core/lib/iomgr/tcp_posix.c index 017f52c3677..1046b60bcc9 100644 --- a/src/core/lib/iomgr/tcp_posix.c +++ b/src/core/lib/iomgr/tcp_posix.c @@ -160,7 +160,7 @@ static void call_read_cb(grpc_exec_ctx *exec_ctx, grpc_tcp *tcp, grpc_error *error) { grpc_closure *cb = tcp->read_cb; - if (false && grpc_tcp_trace) { + if (grpc_tcp_trace) { size_t i; const char *str = grpc_error_string(error); gpr_log(GPR_DEBUG, "read: error=%s", str); @@ -394,7 +394,7 @@ static void tcp_write(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep, grpc_tcp *tcp = (grpc_tcp *)ep; grpc_error *error = GRPC_ERROR_NONE; - if (false && grpc_tcp_trace) { + if (grpc_tcp_trace) { size_t i; for (i = 0; i < buf->count; i++) { diff --git a/src/core/lib/surface/call.c b/src/core/lib/surface/call.c index 77c17a4975b..708ea3502af 100644 --- a/src/core/lib/surface/call.c +++ b/src/core/lib/surface/call.c @@ -407,9 +407,10 @@ static void set_status_code(grpc_call *call, status_source source, static void set_status_details(grpc_call *call, status_source source, grpc_mdstr *status) { if (call->status[source].details != NULL) { - GRPC_MDSTR_UNREF(call->status[source].details); + GRPC_MDSTR_UNREF(status); + } else { + call->status[source].details = status; } - call->status[source].details = status; } static void get_final_status(grpc_call *call, From bd96e8a6a38c0afe01c8bfcb0849d7f10ff634e8 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Fri, 24 Jun 2016 11:53:54 -0700 Subject: [PATCH 0242/1039] Fix clang formatting --- .../ext/client_config/channel_connectivity.c | 7 ++++--- src/core/lib/profiling/basic_timers.c | 8 ++++---- .../security/credentials/jwt/jwt_credentials.c | 8 ++++---- src/core/lib/surface/channel.c | 16 +++++++++------- src/core/lib/surface/completion_queue.c | 14 ++++++++------ src/core/lib/transport/transport_op_string.c | 2 +- 6 files changed, 30 insertions(+), 25 deletions(-) diff --git a/src/core/ext/client_config/channel_connectivity.c b/src/core/ext/client_config/channel_connectivity.c index 2fd8e0e1232..c1220e3a8c3 100644 --- a/src/core/ext/client_config/channel_connectivity.c +++ b/src/core/ext/client_config/channel_connectivity.c @@ -189,10 +189,11 @@ void grpc_channel_watch_connectivity_state( GRPC_API_TRACE( "grpc_channel_watch_connectivity_state(" "channel=%p, last_observed_state=%d, " - "deadline=gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " + "deadline=gpr_timespec { tv_sec: %" PRId64 + ", tv_nsec: %d, clock_type: %d }, " "cq=%p, tag=%p)", - 7, (channel, (int)last_observed_state, deadline.tv_sec, - deadline.tv_nsec, (int)deadline.clock_type, cq, tag)); + 7, (channel, (int)last_observed_state, deadline.tv_sec, deadline.tv_nsec, + (int)deadline.clock_type, cq, tag)); grpc_cq_begin_op(cq, tag); diff --git a/src/core/lib/profiling/basic_timers.c b/src/core/lib/profiling/basic_timers.c index ee464f310f4..51813d04617 100644 --- a/src/core/lib/profiling/basic_timers.c +++ b/src/core/lib/profiling/basic_timers.c @@ -141,11 +141,11 @@ static void write_log(gpr_timer_log *log) { entry->tm = gpr_time_0(entry->tm.clock_type); } fprintf(output_file, - "{\"t\": %"PRId64".%09d, \"thd\": \"%d\", \"type\": \"%c\", \"tag\": " + "{\"t\": %" PRId64 + ".%09d, \"thd\": \"%d\", \"type\": \"%c\", \"tag\": " "\"%s\", \"file\": \"%s\", \"line\": %d, \"imp\": %d}\n", - entry->tm.tv_sec, entry->tm.tv_nsec, entry->thd, - entry->type, entry->tagstr, entry->file, entry->line, - entry->important); + entry->tm.tv_sec, entry->tm.tv_nsec, entry->thd, entry->type, + entry->tagstr, entry->file, entry->line, entry->important); } } diff --git a/src/core/lib/security/credentials/jwt/jwt_credentials.c b/src/core/lib/security/credentials/jwt/jwt_credentials.c index f4dd9208610..1ae9f701fbd 100644 --- a/src/core/lib/security/credentials/jwt/jwt_credentials.c +++ b/src/core/lib/security/credentials/jwt/jwt_credentials.c @@ -149,11 +149,11 @@ grpc_call_credentials *grpc_service_account_jwt_access_credentials_create( "grpc_service_account_jwt_access_credentials_create(" "json_key=%s, " "token_lifetime=" - "gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " + "gpr_timespec { tv_sec: %" PRId64 + ", tv_nsec: %d, clock_type: %d }, " "reserved=%p)", - 5, - (json_key, token_lifetime.tv_sec, token_lifetime.tv_nsec, - (int)token_lifetime.clock_type, reserved)); + 5, (json_key, token_lifetime.tv_sec, token_lifetime.tv_nsec, + (int)token_lifetime.clock_type, reserved)); GPR_ASSERT(reserved == NULL); return grpc_service_account_jwt_access_credentials_create_from_auth_json_key( grpc_auth_json_key_create_from_string(json_key), token_lifetime); diff --git a/src/core/lib/surface/channel.c b/src/core/lib/surface/channel.c index 952a769de1d..2cf6d8890a3 100644 --- a/src/core/lib/surface/channel.c +++ b/src/core/lib/surface/channel.c @@ -223,11 +223,12 @@ grpc_call *grpc_channel_create_call(grpc_channel *channel, "grpc_channel_create_call(" "channel=%p, parent_call=%p, propagation_mask=%x, cq=%p, method=%s, " "host=%s, " - "deadline=gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " + "deadline=gpr_timespec { tv_sec: %" PRId64 + ", tv_nsec: %d, clock_type: %d }, " "reserved=%p)", - 10, (channel, parent_call, (unsigned)propagation_mask, cq, method, host, - deadline.tv_sec, deadline.tv_nsec, - (int)deadline.clock_type, reserved)); + 10, + (channel, parent_call, (unsigned)propagation_mask, cq, method, host, + deadline.tv_sec, deadline.tv_nsec, (int)deadline.clock_type, reserved)); GPR_ASSERT(!reserved); return grpc_channel_create_call_internal( channel, parent_call, propagation_mask, cq, NULL, @@ -282,11 +283,12 @@ grpc_call *grpc_channel_create_registered_call( "grpc_channel_create_registered_call(" "channel=%p, parent_call=%p, propagation_mask=%x, completion_queue=%p, " "registered_call_handle=%p, " - "deadline=gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " + "deadline=gpr_timespec { tv_sec: %" PRId64 + ", tv_nsec: %d, clock_type: %d }, " "reserved=%p)", 9, (channel, parent_call, (unsigned)propagation_mask, completion_queue, - registered_call_handle, deadline.tv_sec, - deadline.tv_nsec, (int)deadline.clock_type, reserved)); + registered_call_handle, deadline.tv_sec, deadline.tv_nsec, + (int)deadline.clock_type, reserved)); GPR_ASSERT(!reserved); return grpc_channel_create_call_internal( channel, parent_call, propagation_mask, completion_queue, NULL, diff --git a/src/core/lib/surface/completion_queue.c b/src/core/lib/surface/completion_queue.c index aa35ae02fe2..b5f2f65e5c0 100644 --- a/src/core/lib/surface/completion_queue.c +++ b/src/core/lib/surface/completion_queue.c @@ -316,10 +316,11 @@ grpc_event grpc_completion_queue_next(grpc_completion_queue *cc, GRPC_API_TRACE( "grpc_completion_queue_next(" "cc=%p, " - "deadline=gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " + "deadline=gpr_timespec { tv_sec: %" PRId64 + ", tv_nsec: %d, clock_type: %d }, " "reserved=%p)", - 5, (cc, deadline.tv_sec, deadline.tv_nsec, - (int)deadline.clock_type, reserved)); + 5, (cc, deadline.tv_sec, deadline.tv_nsec, (int)deadline.clock_type, + reserved)); GPR_ASSERT(!reserved); deadline = gpr_convert_clock_type(deadline, GPR_CLOCK_MONOTONIC); @@ -428,10 +429,11 @@ grpc_event grpc_completion_queue_pluck(grpc_completion_queue *cc, void *tag, GRPC_API_TRACE( "grpc_completion_queue_pluck(" "cc=%p, tag=%p, " - "deadline=gpr_timespec { tv_sec: %"PRId64", tv_nsec: %d, clock_type: %d }, " + "deadline=gpr_timespec { tv_sec: %" PRId64 + ", tv_nsec: %d, clock_type: %d }, " "reserved=%p)", - 6, (cc, tag, deadline.tv_sec, deadline.tv_nsec, - (int)deadline.clock_type, reserved)); + 6, (cc, tag, deadline.tv_sec, deadline.tv_nsec, (int)deadline.clock_type, + reserved)); GPR_ASSERT(!reserved); deadline = gpr_convert_clock_type(deadline, GPR_CLOCK_MONOTONIC); diff --git a/src/core/lib/transport/transport_op_string.c b/src/core/lib/transport/transport_op_string.c index d8fae7805e4..aeaba5339fd 100644 --- a/src/core/lib/transport/transport_op_string.c +++ b/src/core/lib/transport/transport_op_string.c @@ -63,7 +63,7 @@ static void put_metadata_list(gpr_strvec *b, grpc_metadata_batch md) { } if (gpr_time_cmp(md.deadline, gpr_inf_future(md.deadline.clock_type)) != 0) { char *tmp; - gpr_asprintf(&tmp, " deadline=%"PRId64".%09d", md.deadline.tv_sec, + gpr_asprintf(&tmp, " deadline=%" PRId64 ".%09d", md.deadline.tv_sec, md.deadline.tv_nsec); gpr_strvec_add(b, tmp); } From ddf02c1c3ff6efb15a9c47b5da7bd76b3654ad94 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 24 Jun 2016 14:04:01 -0700 Subject: [PATCH 0243/1039] Fix bad indentation --- tools/run_tests/jobset.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/run_tests/jobset.py b/tools/run_tests/jobset.py index 40409c43948..4fe77487f96 100755 --- a/tools/run_tests/jobset.py +++ b/tools/run_tests/jobset.py @@ -363,7 +363,7 @@ class Jobset(object): self._running.add(job) if not self.resultset.has_key(job.GetSpec().shortname): self.resultset[job.GetSpec().shortname] = [] - return True + return True def reap(self): """Collect the dead jobs.""" From d3b0dda9b8c5ac7dda843e46b240e99a3973d3b6 Mon Sep 17 00:00:00 2001 From: Jan Tattermusch Date: Fri, 24 Jun 2016 15:13:08 -0700 Subject: [PATCH 0244/1039] update grpc nugets to 0.15 --- examples/csharp/helloworld/.nuget/packages.config | 2 +- examples/csharp/helloworld/Greeter/Greeter.csproj | 10 +++++----- examples/csharp/helloworld/Greeter/packages.config | 6 +++--- .../helloworld/GreeterClient/GreeterClient.csproj | 10 +++++----- .../csharp/helloworld/GreeterClient/packages.config | 6 +++--- .../helloworld/GreeterServer/GreeterServer.csproj | 10 +++++----- .../csharp/helloworld/GreeterServer/packages.config | 6 +++--- examples/csharp/helloworld/generate_protos.bat | 2 +- examples/csharp/route_guide/.nuget/packages.config | 2 +- .../csharp/route_guide/RouteGuide/RouteGuide.csproj | 10 +++++----- examples/csharp/route_guide/RouteGuide/packages.config | 6 +++--- .../RouteGuideClient/RouteGuideClient.csproj | 10 +++++----- .../route_guide/RouteGuideClient/packages.config | 6 +++--- .../RouteGuideServer/RouteGuideServer.csproj | 10 +++++----- .../route_guide/RouteGuideServer/packages.config | 6 +++--- examples/csharp/route_guide/generate_protos.bat | 2 +- 16 files changed, 52 insertions(+), 52 deletions(-) diff --git a/examples/csharp/helloworld/.nuget/packages.config b/examples/csharp/helloworld/.nuget/packages.config index bfd6c6723d9..aa060800c19 100644 --- a/examples/csharp/helloworld/.nuget/packages.config +++ b/examples/csharp/helloworld/.nuget/packages.config @@ -1,4 +1,4 @@  - + \ No newline at end of file diff --git a/examples/csharp/helloworld/Greeter/Greeter.csproj b/examples/csharp/helloworld/Greeter/Greeter.csproj index 0270cc25f7a..20b85db8b60 100644 --- a/examples/csharp/helloworld/Greeter/Greeter.csproj +++ b/examples/csharp/helloworld/Greeter/Greeter.csproj @@ -10,7 +10,7 @@ Greeter Greeter v4.5 - 745ac60f + 2669b4f2 true @@ -33,11 +33,11 @@ False - ..\packages\Google.Protobuf.3.0.0-beta2\lib\portable-net45+netcore45+wpa81+wp8\Google.Protobuf.dll + ..\packages\Google.Protobuf.3.0.0-beta3\lib\portable-net45+netcore45+wpa81+wp8\Google.Protobuf.dll False - ..\packages\Grpc.Core.0.14.0\lib\net45\Grpc.Core.dll + ..\packages\Grpc.Core.0.15.0\lib\net45\Grpc.Core.dll @@ -61,11 +61,11 @@ - + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - + \ No newline at end of file diff --git a/examples/csharp/helloworld/Greeter/packages.config b/examples/csharp/helloworld/Greeter/packages.config index 617fe6da7be..ff9d6bbf73f 100644 --- a/examples/csharp/helloworld/Greeter/packages.config +++ b/examples/csharp/helloworld/Greeter/packages.config @@ -1,7 +1,7 @@  - - - + + + \ No newline at end of file diff --git a/examples/csharp/helloworld/GreeterClient/GreeterClient.csproj b/examples/csharp/helloworld/GreeterClient/GreeterClient.csproj index 877c450a50e..2b38ce290e9 100644 --- a/examples/csharp/helloworld/GreeterClient/GreeterClient.csproj +++ b/examples/csharp/helloworld/GreeterClient/GreeterClient.csproj @@ -10,7 +10,7 @@ GreeterClient GreeterClient v4.5 - 63b59176 + 5e942a7d true @@ -33,11 +33,11 @@ False - ..\packages\Google.Protobuf.3.0.0-beta2\lib\portable-net45+netcore45+wpa81+wp8\Google.Protobuf.dll + ..\packages\Google.Protobuf.3.0.0-beta3\lib\portable-net45+netcore45+wpa81+wp8\Google.Protobuf.dll False - ..\packages\Grpc.Core.0.14.0\lib\net45\Grpc.Core.dll + ..\packages\Grpc.Core.0.15.0\lib\net45\Grpc.Core.dll @@ -59,11 +59,11 @@ - + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - + \ No newline at end of file diff --git a/examples/csharp/helloworld/GreeterClient/packages.config b/examples/csharp/helloworld/GreeterClient/packages.config index 617fe6da7be..ff9d6bbf73f 100644 --- a/examples/csharp/helloworld/GreeterClient/packages.config +++ b/examples/csharp/helloworld/GreeterClient/packages.config @@ -1,7 +1,7 @@  - - - + + + \ No newline at end of file diff --git a/examples/csharp/helloworld/GreeterServer/GreeterServer.csproj b/examples/csharp/helloworld/GreeterServer/GreeterServer.csproj index 4d792dcf32e..43c633678b1 100644 --- a/examples/csharp/helloworld/GreeterServer/GreeterServer.csproj +++ b/examples/csharp/helloworld/GreeterServer/GreeterServer.csproj @@ -10,7 +10,7 @@ GreeterServer GreeterServer v4.5 - 25ac2e80 + 9c7b2963 true @@ -33,11 +33,11 @@ False - ..\packages\Google.Protobuf.3.0.0-beta2\lib\portable-net45+netcore45+wpa81+wp8\Google.Protobuf.dll + ..\packages\Google.Protobuf.3.0.0-beta3\lib\portable-net45+netcore45+wpa81+wp8\Google.Protobuf.dll False - ..\packages\Grpc.Core.0.14.0\lib\net45\Grpc.Core.dll + ..\packages\Grpc.Core.0.15.0\lib\net45\Grpc.Core.dll @@ -59,11 +59,11 @@ - + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - + \ No newline at end of file diff --git a/examples/csharp/helloworld/GreeterServer/packages.config b/examples/csharp/helloworld/GreeterServer/packages.config index 617fe6da7be..ff9d6bbf73f 100644 --- a/examples/csharp/helloworld/GreeterServer/packages.config +++ b/examples/csharp/helloworld/GreeterServer/packages.config @@ -1,7 +1,7 @@  - - - + + + \ No newline at end of file diff --git a/examples/csharp/helloworld/generate_protos.bat b/examples/csharp/helloworld/generate_protos.bat index b9f68d6745a..a952bb46cdc 100644 --- a/examples/csharp/helloworld/generate_protos.bat +++ b/examples/csharp/helloworld/generate_protos.bat @@ -34,7 +34,7 @@ setlocal @rem enter this directory cd /d %~dp0 -set TOOLS_PATH=packages\Grpc.Tools.0.14.0\tools\windows_x86 +set TOOLS_PATH=packages\Grpc.Tools.0.15.0\tools\windows_x86 %TOOLS_PATH%\protoc.exe -I../../protos --csharp_out Greeter ../../protos/helloworld.proto --grpc_out Greeter --plugin=protoc-gen-grpc=%TOOLS_PATH%\grpc_csharp_plugin.exe diff --git a/examples/csharp/route_guide/.nuget/packages.config b/examples/csharp/route_guide/.nuget/packages.config index bfd6c6723d9..aa060800c19 100644 --- a/examples/csharp/route_guide/.nuget/packages.config +++ b/examples/csharp/route_guide/.nuget/packages.config @@ -1,4 +1,4 @@  - + \ No newline at end of file diff --git a/examples/csharp/route_guide/RouteGuide/RouteGuide.csproj b/examples/csharp/route_guide/RouteGuide/RouteGuide.csproj index 4f7222ebba3..601d16ba24d 100644 --- a/examples/csharp/route_guide/RouteGuide/RouteGuide.csproj +++ b/examples/csharp/route_guide/RouteGuide/RouteGuide.csproj @@ -11,7 +11,7 @@ RouteGuide v4.5 512 - 0a9fcb7a + de2137f9 true @@ -33,11 +33,11 @@ False - ..\packages\Google.Protobuf.3.0.0-beta2\lib\portable-net45+netcore45+wpa81+wp8\Google.Protobuf.dll + ..\packages\Google.Protobuf.3.0.0-beta3\lib\portable-net45+netcore45+wpa81+wp8\Google.Protobuf.dll False - ..\packages\Grpc.Core.0.14.0\lib\net45\Grpc.Core.dll + ..\packages\Grpc.Core.0.15.0\lib\net45\Grpc.Core.dll False @@ -74,12 +74,12 @@ - + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - + + + + + + + + + diff --git a/src/objective-c/tests/Connectivity/ConnectivityTestingApp.xcodeproj/project.pbxproj b/src/objective-c/tests/Connectivity/ConnectivityTestingApp.xcodeproj/project.pbxproj new file mode 100644 index 00000000000..1b1aca74d3b --- /dev/null +++ b/src/objective-c/tests/Connectivity/ConnectivityTestingApp.xcodeproj/project.pbxproj @@ -0,0 +1,274 @@ +// !$*UTF8*$! +{ + archiveVersion = 1; + classes = { + }; + objectVersion = 46; + objects = { + +/* Begin PBXBuildFile section */ + 63BFB9CC1D2478DD00E17927 /* main.m in Sources */ = {isa = PBXBuildFile; fileRef = 63BFB9CB1D2478DD00E17927 /* main.m */; }; + 63BFB9D21D2478DD00E17927 /* ViewController.m in Sources */ = {isa = PBXBuildFile; fileRef = 63BFB9D11D2478DD00E17927 /* ViewController.m */; }; + 63BFB9D51D2478DD00E17927 /* Main.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 63BFB9D31D2478DD00E17927 /* Main.storyboard */; }; +/* End PBXBuildFile section */ + +/* Begin PBXFileReference section */ + 63BFB9C71D2478DD00E17927 /* ConnectivityTestingApp.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = ConnectivityTestingApp.app; sourceTree = BUILT_PRODUCTS_DIR; }; + 63BFB9CB1D2478DD00E17927 /* main.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = main.m; sourceTree = SOURCE_ROOT; }; + 63BFB9D11D2478DD00E17927 /* ViewController.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = ViewController.m; sourceTree = SOURCE_ROOT; }; + 63BFB9D41D2478DD00E17927 /* Base */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; name = Base; path = Base.lproj/Main.storyboard; sourceTree = ""; }; + 63BFB9DB1D2478DD00E17927 /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = SOURCE_ROOT; }; +/* End PBXFileReference section */ + +/* Begin PBXFrameworksBuildPhase section */ + 63BFB9C41D2478DD00E17927 /* Frameworks */ = { + isa = PBXFrameworksBuildPhase; + buildActionMask = 2147483647; + files = ( + ); + runOnlyForDeploymentPostprocessing = 0; + }; +/* End PBXFrameworksBuildPhase section */ + +/* Begin PBXGroup section */ + 63BFB9BE1D2478DD00E17927 = { + isa = PBXGroup; + children = ( + 63BFB9D11D2478DD00E17927 /* ViewController.m */, + 63BFB9D31D2478DD00E17927 /* Main.storyboard */, + 63BFB9DB1D2478DD00E17927 /* Info.plist */, + 63BFB9CB1D2478DD00E17927 /* main.m */, + 63BFB9C81D2478DD00E17927 /* Products */, + ); + sourceTree = ""; + }; + 63BFB9C81D2478DD00E17927 /* Products */ = { + isa = PBXGroup; + children = ( + 63BFB9C71D2478DD00E17927 /* ConnectivityTestingApp.app */, + ); + name = Products; + sourceTree = ""; + }; +/* End PBXGroup section */ + +/* Begin PBXNativeTarget section */ + 63BFB9C61D2478DD00E17927 /* ConnectivityTestingApp */ = { + isa = PBXNativeTarget; + buildConfigurationList = 63BFB9DE1D2478DD00E17927 /* Build configuration list for PBXNativeTarget "ConnectivityTestingApp" */; + buildPhases = ( + 63BFB9C31D2478DD00E17927 /* Sources */, + 63BFB9C41D2478DD00E17927 /* Frameworks */, + 63BFB9C51D2478DD00E17927 /* Resources */, + ); + buildRules = ( + ); + dependencies = ( + ); + name = ConnectivityTestingApp; + productName = ConnectivityTestingApp; + productReference = 63BFB9C71D2478DD00E17927 /* ConnectivityTestingApp.app */; + productType = "com.apple.product-type.application"; + }; +/* End PBXNativeTarget section */ + +/* Begin PBXProject section */ + 63BFB9BF1D2478DD00E17927 /* Project object */ = { + isa = PBXProject; + attributes = { + LastUpgradeCheck = 0800; + ORGANIZATIONNAME = gRPC; + TargetAttributes = { + 63BFB9C61D2478DD00E17927 = { + CreatedOnToolsVersion = 8.0; + DevelopmentTeam = EQHXZ8M8AV; + DevelopmentTeamName = "Google, Inc."; + ProvisioningStyle = Automatic; + }; + }; + }; + buildConfigurationList = 63BFB9C21D2478DD00E17927 /* Build configuration list for PBXProject "ConnectivityTestingApp" */; + compatibilityVersion = "Xcode 3.2"; + developmentRegion = English; + hasScannedForEncodings = 0; + knownRegions = ( + en, + Base, + ); + mainGroup = 63BFB9BE1D2478DD00E17927; + productRefGroup = 63BFB9C81D2478DD00E17927 /* Products */; + projectDirPath = ""; + projectRoot = ""; + targets = ( + 63BFB9C61D2478DD00E17927 /* ConnectivityTestingApp */, + ); + }; +/* End PBXProject section */ + +/* Begin PBXResourcesBuildPhase section */ + 63BFB9C51D2478DD00E17927 /* Resources */ = { + isa = PBXResourcesBuildPhase; + buildActionMask = 2147483647; + files = ( + 63BFB9D51D2478DD00E17927 /* Main.storyboard in Resources */, + ); + runOnlyForDeploymentPostprocessing = 0; + }; +/* End PBXResourcesBuildPhase section */ + +/* Begin PBXSourcesBuildPhase section */ + 63BFB9C31D2478DD00E17927 /* Sources */ = { + isa = PBXSourcesBuildPhase; + buildActionMask = 2147483647; + files = ( + 63BFB9D21D2478DD00E17927 /* ViewController.m in Sources */, + 63BFB9CC1D2478DD00E17927 /* main.m in Sources */, + ); + runOnlyForDeploymentPostprocessing = 0; + }; +/* End PBXSourcesBuildPhase section */ + +/* Begin PBXVariantGroup section */ + 63BFB9D31D2478DD00E17927 /* Main.storyboard */ = { + isa = PBXVariantGroup; + children = ( + 63BFB9D41D2478DD00E17927 /* Base */, + ); + name = Main.storyboard; + path = .; + sourceTree = SOURCE_ROOT; + }; +/* End PBXVariantGroup section */ + +/* Begin XCBuildConfiguration section */ + 63BFB9DC1D2478DD00E17927 /* Debug */ = { + isa = XCBuildConfiguration; + buildSettings = { + ALWAYS_SEARCH_USER_PATHS = NO; + CLANG_ANALYZER_NONNULL = YES; + CLANG_CXX_LANGUAGE_STANDARD = "gnu++0x"; + CLANG_CXX_LIBRARY = "libc++"; + CLANG_ENABLE_MODULES = YES; + CLANG_ENABLE_OBJC_ARC = YES; + CLANG_WARN_BOOL_CONVERSION = YES; + CLANG_WARN_CONSTANT_CONVERSION = YES; + CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; + CLANG_WARN_DOCUMENTATION_COMMENTS = YES; + CLANG_WARN_EMPTY_BODY = YES; + CLANG_WARN_ENUM_CONVERSION = YES; + CLANG_WARN_INT_CONVERSION = YES; + CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; + CLANG_WARN_UNREACHABLE_CODE = YES; + CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; + "CODE_SIGN_IDENTITY[sdk=iphoneos*]" = "iPhone Developer"; + COPY_PHASE_STRIP = NO; + DEBUG_INFORMATION_FORMAT = dwarf; + ENABLE_STRICT_OBJC_MSGSEND = YES; + ENABLE_TESTABILITY = YES; + GCC_C_LANGUAGE_STANDARD = gnu99; + GCC_DYNAMIC_NO_PIC = NO; + GCC_NO_COMMON_BLOCKS = YES; + GCC_OPTIMIZATION_LEVEL = 0; + GCC_PREPROCESSOR_DEFINITIONS = ( + "DEBUG=1", + "$(inherited)", + ); + GCC_WARN_64_TO_32_BIT_CONVERSION = YES; + GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; + GCC_WARN_UNDECLARED_SELECTOR = YES; + GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; + GCC_WARN_UNUSED_FUNCTION = YES; + GCC_WARN_UNUSED_VARIABLE = YES; + IPHONEOS_DEPLOYMENT_TARGET = 10.0; + MTL_ENABLE_DEBUG_INFO = YES; + ONLY_ACTIVE_ARCH = YES; + SDKROOT = iphoneos; + }; + name = Debug; + }; + 63BFB9DD1D2478DD00E17927 /* Release */ = { + isa = XCBuildConfiguration; + buildSettings = { + ALWAYS_SEARCH_USER_PATHS = NO; + CLANG_ANALYZER_NONNULL = YES; + CLANG_CXX_LANGUAGE_STANDARD = "gnu++0x"; + CLANG_CXX_LIBRARY = "libc++"; + CLANG_ENABLE_MODULES = YES; + CLANG_ENABLE_OBJC_ARC = YES; + CLANG_WARN_BOOL_CONVERSION = YES; + CLANG_WARN_CONSTANT_CONVERSION = YES; + CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; + CLANG_WARN_DOCUMENTATION_COMMENTS = YES; + CLANG_WARN_EMPTY_BODY = YES; + CLANG_WARN_ENUM_CONVERSION = YES; + CLANG_WARN_INT_CONVERSION = YES; + CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; + CLANG_WARN_UNREACHABLE_CODE = YES; + CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; + "CODE_SIGN_IDENTITY[sdk=iphoneos*]" = "iPhone Developer"; + COPY_PHASE_STRIP = NO; + DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym"; + ENABLE_NS_ASSERTIONS = NO; + ENABLE_STRICT_OBJC_MSGSEND = YES; + GCC_C_LANGUAGE_STANDARD = gnu99; + GCC_NO_COMMON_BLOCKS = YES; + GCC_WARN_64_TO_32_BIT_CONVERSION = YES; + GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; + GCC_WARN_UNDECLARED_SELECTOR = YES; + GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; + GCC_WARN_UNUSED_FUNCTION = YES; + GCC_WARN_UNUSED_VARIABLE = YES; + IPHONEOS_DEPLOYMENT_TARGET = 10.0; + MTL_ENABLE_DEBUG_INFO = NO; + SDKROOT = iphoneos; + VALIDATE_PRODUCT = YES; + }; + name = Release; + }; + 63BFB9DF1D2478DD00E17927 /* Debug */ = { + isa = XCBuildConfiguration; + buildSettings = { + ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; + INFOPLIST_FILE = Info.plist; + LD_RUNPATH_SEARCH_PATHS = "$(inherited) @executable_path/Frameworks"; + PRODUCT_BUNDLE_IDENTIFIER = io.grpc.ConnectivityTestingApp; + PRODUCT_NAME = "$(TARGET_NAME)"; + }; + name = Debug; + }; + 63BFB9E01D2478DD00E17927 /* Release */ = { + isa = XCBuildConfiguration; + buildSettings = { + ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; + INFOPLIST_FILE = Info.plist; + LD_RUNPATH_SEARCH_PATHS = "$(inherited) @executable_path/Frameworks"; + PRODUCT_BUNDLE_IDENTIFIER = io.grpc.ConnectivityTestingApp; + PRODUCT_NAME = "$(TARGET_NAME)"; + }; + name = Release; + }; +/* End XCBuildConfiguration section */ + +/* Begin XCConfigurationList section */ + 63BFB9C21D2478DD00E17927 /* Build configuration list for PBXProject "ConnectivityTestingApp" */ = { + isa = XCConfigurationList; + buildConfigurations = ( + 63BFB9DC1D2478DD00E17927 /* Debug */, + 63BFB9DD1D2478DD00E17927 /* Release */, + ); + defaultConfigurationIsVisible = 0; + defaultConfigurationName = Release; + }; + 63BFB9DE1D2478DD00E17927 /* Build configuration list for PBXNativeTarget "ConnectivityTestingApp" */ = { + isa = XCConfigurationList; + buildConfigurations = ( + 63BFB9DF1D2478DD00E17927 /* Debug */, + 63BFB9E01D2478DD00E17927 /* Release */, + ); + defaultConfigurationIsVisible = 0; + defaultConfigurationName = Release; + }; +/* End XCConfigurationList section */ + }; + rootObject = 63BFB9BF1D2478DD00E17927 /* Project object */; +} diff --git a/src/objective-c/tests/Connectivity/ConnectivityTestingApp.xcodeproj/project.xcworkspace/contents.xcworkspacedata b/src/objective-c/tests/Connectivity/ConnectivityTestingApp.xcodeproj/project.xcworkspace/contents.xcworkspacedata new file mode 100644 index 00000000000..b541b4b44d5 --- /dev/null +++ b/src/objective-c/tests/Connectivity/ConnectivityTestingApp.xcodeproj/project.xcworkspace/contents.xcworkspacedata @@ -0,0 +1,7 @@ + + + + + diff --git a/src/objective-c/tests/Connectivity/Info.plist b/src/objective-c/tests/Connectivity/Info.plist new file mode 100644 index 00000000000..8a9fb887013 --- /dev/null +++ b/src/objective-c/tests/Connectivity/Info.plist @@ -0,0 +1,40 @@ + + + + + CFBundleDevelopmentRegion + en + CFBundleExecutable + $(EXECUTABLE_NAME) + CFBundleIdentifier + $(PRODUCT_BUNDLE_IDENTIFIER) + CFBundleInfoDictionaryVersion + 6.0 + CFBundleName + $(PRODUCT_NAME) + CFBundlePackageType + APPL + CFBundleShortVersionString + 1.0 + CFBundleSignature + ???? + CFBundleVersion + 1 + LSRequiresIPhoneOS + + UILaunchStoryboardName + Main + UIMainStoryboardFile + Main + UIRequiredDeviceCapabilities + + armv7 + + UISupportedInterfaceOrientations + + UIInterfaceOrientationPortrait + UIInterfaceOrientationLandscapeLeft + UIInterfaceOrientationLandscapeRight + + + diff --git a/src/objective-c/tests/Connectivity/ViewController.m b/src/objective-c/tests/Connectivity/ViewController.m new file mode 100644 index 00000000000..cb0321c0c10 --- /dev/null +++ b/src/objective-c/tests/Connectivity/ViewController.m @@ -0,0 +1,43 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#import + +@interface ViewController : UIViewController +@end + +@implementation ViewController +- (void)viewDidLoad { + [super viewDidLoad]; +} +@end diff --git a/src/objective-c/tests/Connectivity/main.m b/src/objective-c/tests/Connectivity/main.m new file mode 100644 index 00000000000..5e09196d549 --- /dev/null +++ b/src/objective-c/tests/Connectivity/main.m @@ -0,0 +1,46 @@ +/* + * + * Copyright 2015, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#import + +@interface AppDelegate : UIResponder +@property (strong, nonatomic) UIWindow *window; +@end +@implementation AppDelegate +@end + +int main(int argc, char * argv[]) { + @autoreleasepool { + return UIApplicationMain(argc, argv, nil, NSStringFromClass(AppDelegate.class)); + } +} From 3325c0b2a248c28666d9a0a7c9a27e1dd33099e9 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Wed, 29 Jun 2016 17:02:16 -0700 Subject: [PATCH 0337/1039] Add Podfile and pod install --- .../project.pbxproj | 79 +++++++++++++++++++ src/objective-c/tests/Connectivity/Podfile | 10 +++ 2 files changed, 89 insertions(+) create mode 100644 src/objective-c/tests/Connectivity/Podfile diff --git a/src/objective-c/tests/Connectivity/ConnectivityTestingApp.xcodeproj/project.pbxproj b/src/objective-c/tests/Connectivity/ConnectivityTestingApp.xcodeproj/project.pbxproj index 1b1aca74d3b..2a9466c03f1 100644 --- a/src/objective-c/tests/Connectivity/ConnectivityTestingApp.xcodeproj/project.pbxproj +++ b/src/objective-c/tests/Connectivity/ConnectivityTestingApp.xcodeproj/project.pbxproj @@ -7,6 +7,7 @@ objects = { /* Begin PBXBuildFile section */ + 500A4E0AC9D489EB214D1ED4 /* libPods-ConnectivityTestingApp.a in Frameworks */ = {isa = PBXBuildFile; fileRef = C2AF815D8242A2172891621D /* libPods-ConnectivityTestingApp.a */; }; 63BFB9CC1D2478DD00E17927 /* main.m in Sources */ = {isa = PBXBuildFile; fileRef = 63BFB9CB1D2478DD00E17927 /* main.m */; }; 63BFB9D21D2478DD00E17927 /* ViewController.m in Sources */ = {isa = PBXBuildFile; fileRef = 63BFB9D11D2478DD00E17927 /* ViewController.m */; }; 63BFB9D51D2478DD00E17927 /* Main.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 63BFB9D31D2478DD00E17927 /* Main.storyboard */; }; @@ -18,6 +19,9 @@ 63BFB9D11D2478DD00E17927 /* ViewController.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = ViewController.m; sourceTree = SOURCE_ROOT; }; 63BFB9D41D2478DD00E17927 /* Base */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; name = Base; path = Base.lproj/Main.storyboard; sourceTree = ""; }; 63BFB9DB1D2478DD00E17927 /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = SOURCE_ROOT; }; + BA96CBC1612BD2F70E66246C /* Pods-ConnectivityTestingApp.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-ConnectivityTestingApp.release.xcconfig"; path = "Pods/Target Support Files/Pods-ConnectivityTestingApp/Pods-ConnectivityTestingApp.release.xcconfig"; sourceTree = ""; }; + C2AF815D8242A2172891621D /* libPods-ConnectivityTestingApp.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = "libPods-ConnectivityTestingApp.a"; sourceTree = BUILT_PRODUCTS_DIR; }; + FC9BD3AE427396EDB4CD13E3 /* Pods-ConnectivityTestingApp.debug.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-ConnectivityTestingApp.debug.xcconfig"; path = "Pods/Target Support Files/Pods-ConnectivityTestingApp/Pods-ConnectivityTestingApp.debug.xcconfig"; sourceTree = ""; }; /* End PBXFileReference section */ /* Begin PBXFrameworksBuildPhase section */ @@ -25,12 +29,30 @@ isa = PBXFrameworksBuildPhase; buildActionMask = 2147483647; files = ( + 500A4E0AC9D489EB214D1ED4 /* libPods-ConnectivityTestingApp.a in Frameworks */, ); runOnlyForDeploymentPostprocessing = 0; }; /* End PBXFrameworksBuildPhase section */ /* Begin PBXGroup section */ + 16E6C67F2E48B42376DFFD2A /* Pods */ = { + isa = PBXGroup; + children = ( + FC9BD3AE427396EDB4CD13E3 /* Pods-ConnectivityTestingApp.debug.xcconfig */, + BA96CBC1612BD2F70E66246C /* Pods-ConnectivityTestingApp.release.xcconfig */, + ); + name = Pods; + sourceTree = ""; + }; + 48F8EC18C66D3416A41F76F5 /* Frameworks */ = { + isa = PBXGroup; + children = ( + C2AF815D8242A2172891621D /* libPods-ConnectivityTestingApp.a */, + ); + name = Frameworks; + sourceTree = ""; + }; 63BFB9BE1D2478DD00E17927 = { isa = PBXGroup; children = ( @@ -39,6 +61,8 @@ 63BFB9DB1D2478DD00E17927 /* Info.plist */, 63BFB9CB1D2478DD00E17927 /* main.m */, 63BFB9C81D2478DD00E17927 /* Products */, + 16E6C67F2E48B42376DFFD2A /* Pods */, + 48F8EC18C66D3416A41F76F5 /* Frameworks */, ); sourceTree = ""; }; @@ -57,9 +81,12 @@ isa = PBXNativeTarget; buildConfigurationList = 63BFB9DE1D2478DD00E17927 /* Build configuration list for PBXNativeTarget "ConnectivityTestingApp" */; buildPhases = ( + 4DCA2703A0AA5DC1BD2751B8 /* [CP] Check Pods Manifest.lock */, 63BFB9C31D2478DD00E17927 /* Sources */, 63BFB9C41D2478DD00E17927 /* Frameworks */, 63BFB9C51D2478DD00E17927 /* Resources */, + 8593A2388A8F7BF5A7E98D26 /* [CP] Embed Pods Frameworks */, + 5347BF6C41E7888C1C05CD88 /* [CP] Copy Pods Resources */, ); buildRules = ( ); @@ -116,6 +143,54 @@ }; /* End PBXResourcesBuildPhase section */ +/* Begin PBXShellScriptBuildPhase section */ + 4DCA2703A0AA5DC1BD2751B8 /* [CP] Check Pods Manifest.lock */ = { + isa = PBXShellScriptBuildPhase; + buildActionMask = 2147483647; + files = ( + ); + inputPaths = ( + ); + name = "[CP] Check Pods Manifest.lock"; + outputPaths = ( + ); + runOnlyForDeploymentPostprocessing = 0; + shellPath = /bin/sh; + shellScript = "diff \"${PODS_ROOT}/../Podfile.lock\" \"${PODS_ROOT}/Manifest.lock\" > /dev/null\nif [[ $? != 0 ]] ; then\n cat << EOM\nerror: The sandbox is not in sync with the Podfile.lock. Run 'pod install' or update your CocoaPods installation.\nEOM\n exit 1\nfi\n"; + showEnvVarsInLog = 0; + }; + 5347BF6C41E7888C1C05CD88 /* [CP] Copy Pods Resources */ = { + isa = PBXShellScriptBuildPhase; + buildActionMask = 2147483647; + files = ( + ); + inputPaths = ( + ); + name = "[CP] Copy Pods Resources"; + outputPaths = ( + ); + runOnlyForDeploymentPostprocessing = 0; + shellPath = /bin/sh; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-ConnectivityTestingApp/Pods-ConnectivityTestingApp-resources.sh\"\n"; + showEnvVarsInLog = 0; + }; + 8593A2388A8F7BF5A7E98D26 /* [CP] Embed Pods Frameworks */ = { + isa = PBXShellScriptBuildPhase; + buildActionMask = 2147483647; + files = ( + ); + inputPaths = ( + ); + name = "[CP] Embed Pods Frameworks"; + outputPaths = ( + ); + runOnlyForDeploymentPostprocessing = 0; + shellPath = /bin/sh; + shellScript = "\"${SRCROOT}/Pods/Target Support Files/Pods-ConnectivityTestingApp/Pods-ConnectivityTestingApp-frameworks.sh\"\n"; + showEnvVarsInLog = 0; + }; +/* End PBXShellScriptBuildPhase section */ + /* Begin PBXSourcesBuildPhase section */ 63BFB9C31D2478DD00E17927 /* Sources */ = { isa = PBXSourcesBuildPhase; @@ -227,9 +302,11 @@ }; 63BFB9DF1D2478DD00E17927 /* Debug */ = { isa = XCBuildConfiguration; + baseConfigurationReference = FC9BD3AE427396EDB4CD13E3 /* Pods-ConnectivityTestingApp.debug.xcconfig */; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; INFOPLIST_FILE = Info.plist; + IPHONEOS_DEPLOYMENT_TARGET = 9.3; LD_RUNPATH_SEARCH_PATHS = "$(inherited) @executable_path/Frameworks"; PRODUCT_BUNDLE_IDENTIFIER = io.grpc.ConnectivityTestingApp; PRODUCT_NAME = "$(TARGET_NAME)"; @@ -238,9 +315,11 @@ }; 63BFB9E01D2478DD00E17927 /* Release */ = { isa = XCBuildConfiguration; + baseConfigurationReference = BA96CBC1612BD2F70E66246C /* Pods-ConnectivityTestingApp.release.xcconfig */; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; INFOPLIST_FILE = Info.plist; + IPHONEOS_DEPLOYMENT_TARGET = 9.3; LD_RUNPATH_SEARCH_PATHS = "$(inherited) @executable_path/Frameworks"; PRODUCT_BUNDLE_IDENTIFIER = io.grpc.ConnectivityTestingApp; PRODUCT_NAME = "$(TARGET_NAME)"; diff --git a/src/objective-c/tests/Connectivity/Podfile b/src/objective-c/tests/Connectivity/Podfile new file mode 100644 index 00000000000..f9224d9e4e9 --- /dev/null +++ b/src/objective-c/tests/Connectivity/Podfile @@ -0,0 +1,10 @@ +install! 'cocoapods', :deterministic_uuids => false + +# Location of gRPC's repo root relative to this file. +GRPC_LOCAL_SRC = '../../../..' + +target 'ConnectivityTestingApp' do + pod 'gRPC', :path => GRPC_LOCAL_SRC + pod 'Protobuf', :path => "#{GRPC_LOCAL_SRC}/third_party/protobuf" + pod 'BoringSSL', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c" +end From 10cc76cbe7f4a717e5da800837ee9d69144684f1 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Wed, 29 Jun 2016 17:02:45 -0700 Subject: [PATCH 0338/1039] Make RPCs on a loop --- .../tests/Connectivity/ViewController.m | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/src/objective-c/tests/Connectivity/ViewController.m b/src/objective-c/tests/Connectivity/ViewController.m index cb0321c0c10..2b199c9617f 100644 --- a/src/objective-c/tests/Connectivity/ViewController.m +++ b/src/objective-c/tests/Connectivity/ViewController.m @@ -33,11 +33,50 @@ #import +#import +#import +#import +#import + @interface ViewController : UIViewController @end @implementation ViewController - (void)viewDidLoad { [super viewDidLoad]; + + NSString *host = @"grpc-test.sandbox.googleapis.com"; + + GRPCProtoMethod *method = [[GRPCProtoMethod alloc] initWithPackage:@"grpc.testing" + service:@"TestService" + method:@"StreamingOutputCall"]; + + __block void (^startCall)() = ^{ + GRXWriter *loggingRequestWriter = [[GRXWriter writerWithValue:[NSData data]] map:^id(id value) { + NSLog(@"Sending request."); + return value; + }]; + + GRPCCall *call = [[GRPCCall alloc] initWithHost:host + path:method.HTTPPath + requestsWriter:loggingRequestWriter]; + + [call startWithWriteable:[GRXWriteable writeableWithEventHandler:^(BOOL done, id value, + NSError *error) { + if (!done) { + return; + } + if (error) { + NSLog(@"Finished with error %@", error); + } else { + NSLog(@"Finished successfully."); + } + + dispatch_time_t oneSecond = dispatch_time(DISPATCH_TIME_NOW, (int64_t)(1 * NSEC_PER_SEC)); + dispatch_after(oneSecond, dispatch_get_main_queue(), startCall); + }]]; + }; + + startCall(); } @end From 28820575ff722d3332cebcae11d2b77d5be67316 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Wed, 29 Jun 2016 17:09:11 -0700 Subject: [PATCH 0339/1039] Add README.md explaining use of the app --- src/objective-c/tests/Connectivity/README.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 src/objective-c/tests/Connectivity/README.md diff --git a/src/objective-c/tests/Connectivity/README.md b/src/objective-c/tests/Connectivity/README.md new file mode 100644 index 00000000000..851cb9d1dab --- /dev/null +++ b/src/objective-c/tests/Connectivity/README.md @@ -0,0 +1,16 @@ +This app can be used to manually test gRPC under changing network conditions. + +It makes RPCs in a loop, logging when the request is sent and the response is received. + +To test on the simulator, run `pod install`, open the workspace created by Cocoapods, and run the app. +Once running, disable WiFi (or ethernet) _in your computer_, then enable it again after a while. Don't +bother with the simulator's WiFi or cell settings, as they have no effect: Simulator apps are just Mac +apps running within the simulator UI. + +The expected result is to never see a "hanged" RPC: success or failure should happen almost immediately +after sending the request. Symptom of a hanged RPC is a log like the following being the last in your +console: + +``` +2016-06-29 16:51:29.443 ConnectivityTestingApp[73129:3567949] Sending request. +``` From 42812723893b08c466b830b0515c20bd5edb21dc Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 29 Jun 2016 17:39:07 -0700 Subject: [PATCH 0340/1039] Updated git clone URL --- INSTALL.md | 2 +- examples/cpp/README.md | 2 +- examples/cpp/cpptutorial.md | 2 +- examples/cpp/helloworld/README.md | 2 +- examples/node/README.md | 2 +- examples/objective-c/helloworld/README.md | 2 +- examples/php/README.md | 2 +- src/cpp/README.md | 2 +- src/php/README.md | 4 ++-- src/python/grpcio/README.rst | 2 +- tools/distrib/python/grpcio_tools/README.rst | 2 +- 11 files changed, 12 insertions(+), 12 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 66e6c33a775..6718aad120d 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -49,7 +49,7 @@ For developers who are interested to contribute, here is how to compile the gRPC C Core library. ```sh - $ git clone https://github.com/grpc/grpc.git + $ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc $ cd grpc $ git submodule update --init $ make diff --git a/examples/cpp/README.md b/examples/cpp/README.md index d93cbacf7b2..3fa7ad4c784 100644 --- a/examples/cpp/README.md +++ b/examples/cpp/README.md @@ -14,7 +14,7 @@ following command: ```sh -$ git clone https://github.com/grpc/grpc.git +$ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc ``` Change your current directory to examples/cpp/helloworld diff --git a/examples/cpp/cpptutorial.md b/examples/cpp/cpptutorial.md index ef9ca99c0fe..80fef07192e 100644 --- a/examples/cpp/cpptutorial.md +++ b/examples/cpp/cpptutorial.md @@ -20,7 +20,7 @@ With gRPC we can define our service once in a .proto file and implement clients The example code for our tutorial is in [examples/cpp/route_guide](route_guide). To download the example, clone this repository by running the following command: ```shell -$ git clone https://github.com/grpc/grpc.git +$ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc ``` Then change your current directory to `examples/cpp/route_guide`: diff --git a/examples/cpp/helloworld/README.md b/examples/cpp/helloworld/README.md index 04283eabc35..db953f53628 100644 --- a/examples/cpp/helloworld/README.md +++ b/examples/cpp/helloworld/README.md @@ -12,7 +12,7 @@ following command: ```sh -$ git clone https://github.com/grpc/grpc.git +$ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc ``` Change your current directory to examples/cpp/helloworld diff --git a/examples/node/README.md b/examples/node/README.md index 59fb4a17f59..b92ca383c47 100644 --- a/examples/node/README.md +++ b/examples/node/README.md @@ -12,7 +12,7 @@ INSTALL ```sh $ # Get the gRPC repository $ export REPO_ROOT=grpc # REPO root can be any directory of your choice - $ git clone https://github.com/grpc/grpc.git $REPO_ROOT + $ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc $REPO_ROOT $ cd $REPO_ROOT $ cd examples/node diff --git a/examples/objective-c/helloworld/README.md b/examples/objective-c/helloworld/README.md index 81c5aaa7bcb..fdff938a978 100644 --- a/examples/objective-c/helloworld/README.md +++ b/examples/objective-c/helloworld/README.md @@ -16,7 +16,7 @@ this repository to your local machine by running the following commands: ```sh -$ git clone https://github.com/grpc/grpc.git +$ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc $ cd grpc $ git submodule update --init ``` diff --git a/examples/php/README.md b/examples/php/README.md index ea9ccb67908..e56b0178730 100644 --- a/examples/php/README.md +++ b/examples/php/README.md @@ -17,7 +17,7 @@ INSTALL - Clone this repository ```sh - $ git clone https://github.com/grpc/grpc.git + $ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc ``` - Install composer diff --git a/src/cpp/README.md b/src/cpp/README.md index f2935e52d9e..8c0f85e5ff7 100644 --- a/src/cpp/README.md +++ b/src/cpp/README.md @@ -51,7 +51,7 @@ below. #Build from Source ```sh - $ git clone https://github.com/grpc/grpc.git + $ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc $ cd grpc $ git submodule update --init $ make diff --git a/src/php/README.md b/src/php/README.md index cf8f2c11b0f..6cc1ba4d462 100644 --- a/src/php/README.md +++ b/src/php/README.md @@ -58,7 +58,7 @@ To run tests with generated stub code from `.proto` files, you will also need th Clone this repository ```sh -$ git clone https://github.com/grpc/grpc.git +$ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc ``` Build and install the gRPC C core library @@ -101,7 +101,7 @@ extension=grpc.so You will need the source code to run tests ```sh -$ git clone https://github.com/grpc/grpc.git +$ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc $ cd grpc $ git pull --recurse-submodules && git submodule update --init --recursive ``` diff --git a/src/python/grpcio/README.rst b/src/python/grpcio/README.rst index afc4fe6a370..3fc318539ea 100644 --- a/src/python/grpcio/README.rst +++ b/src/python/grpcio/README.rst @@ -46,7 +46,7 @@ package named :code:`python-dev`). :: $ export REPO_ROOT=grpc # REPO_ROOT can be any directory of your choice - $ git clone https://github.com/grpc/grpc.git $REPO_ROOT + $ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc $REPO_ROOT $ cd $REPO_ROOT $ git submodule update --init diff --git a/tools/distrib/python/grpcio_tools/README.rst b/tools/distrib/python/grpcio_tools/README.rst index e9f137493ba..8946a1d5b31 100644 --- a/tools/distrib/python/grpcio_tools/README.rst +++ b/tools/distrib/python/grpcio_tools/README.rst @@ -53,7 +53,7 @@ GCC-like stuff, but you may end up having a bad time. :: $ export REPO_ROOT=grpc # REPO_ROOT can be any directory of your choice - $ git clone https://github.com/grpc/grpc.git $REPO_ROOT + $ git clone -b $(curl -L http://grpc.io/release) https://github.com/grpc/grpc $REPO_ROOT $ cd $REPO_ROOT $ git submodule update --init From 6a654dd400cc204a1ceb059785821c8ea15acde9 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Mon, 27 Jun 2016 14:12:08 -0700 Subject: [PATCH 0341/1039] Changed default string type to be str This impacts the following APIs: Metadata: Key is always a str, Value is bytes for binary metadata, str otherwise Call Details: str type gRPC method: str type hostname/target: str type --- src/python/grpcio/grpc/_channel.py | 46 +++++---- src/python/grpcio/grpc/_common.py | 43 ++++++--- .../grpc/_cython/_cygrpc/channel.pyx.pxi | 5 +- .../grpc/_cython/_cygrpc/credentials.pyx.pxi | 8 +- .../grpc/_cython/_cygrpc/records.pyx.pxi | 36 +++---- .../grpc/_cython/_cygrpc/server.pyx.pxi | 2 +- src/python/grpcio/grpc/_plugin_wrapping.py | 14 +-- src/python/grpcio/grpc/_server.py | 42 ++++---- .../grpcio/grpc/beta/_server_adaptations.py | 3 +- .../grpcio/tests/unit/_compression_test.py | 4 +- .../unit/_cython/_cancel_many_calls_test.py | 4 +- .../tests/unit/_cython/_channel_test.py | 2 +- .../_read_some_but_not_all_responses_test.py | 4 +- .../grpcio/tests/unit/_cython/cygrpc_test.py | 40 ++++---- .../grpcio/tests/unit/_empty_message_test.py | 8 +- .../tests/unit/_metadata_code_details_test.py | 32 +++---- .../grpcio/tests/unit/_metadata_test.py | 24 ++--- src/python/grpcio/tests/unit/_rpc_test.py | 96 +++++++++---------- .../grpcio/tests/unit/beta/_utilities_test.py | 21 +--- .../grpcio/tests/unit/beta/test_utilities.py | 2 +- src/python/grpcio/tests/unit/test_common.py | 18 ++-- 21 files changed, 225 insertions(+), 229 deletions(-) diff --git a/src/python/grpcio/grpc/_channel.py b/src/python/grpcio/grpc/_channel.py index cf6175d0314..a89b5013030 100644 --- a/src/python/grpcio/grpc/_channel.py +++ b/src/python/grpcio/grpc/_channel.py @@ -364,13 +364,13 @@ class _Rendezvous(grpc.RpcError, grpc.Future, grpc.Call): with self._state.condition: while self._state.initial_metadata is None: self._state.condition.wait() - return self._state.initial_metadata + return _common.application_metadata(self._state.initial_metadata) def trailing_metadata(self): with self._state.condition: while self._state.trailing_metadata is None: self._state.condition.wait() - return self._state.trailing_metadata + return _common.application_metadata(self._state.trailing_metadata) def code(self): with self._state.condition: @@ -382,7 +382,7 @@ class _Rendezvous(grpc.RpcError, grpc.Future, grpc.Call): with self._state.condition: while self._state.details is None: self._state.condition.wait() - return self._state.details + return _common.decode(self._state.details) def _repr(self): with self._state.condition: @@ -390,7 +390,7 @@ class _Rendezvous(grpc.RpcError, grpc.Future, grpc.Call): return '<_Rendezvous object of in-flight RPC>' else: return '<_Rendezvous of RPC that terminated with ({}, {})>'.format( - self._state.code, self._state.details) + self._state.code, _common.decode(self._state.details)) def __repr__(self): return self._repr() @@ -451,7 +451,7 @@ class _UnaryUnaryMultiCallable(grpc.UnaryUnaryMultiCallable): state = _RPCState(_UNARY_UNARY_INITIAL_DUE, None, None, None, None) operations = ( cygrpc.operation_send_initial_metadata( - _common.metadata(metadata), _EMPTY_FLAGS), + _common.cygrpc_metadata(metadata), _EMPTY_FLAGS), cygrpc.operation_send_message(serialized_request, _EMPTY_FLAGS), cygrpc.operation_send_close_from_client(_EMPTY_FLAGS), cygrpc.operation_receive_initial_metadata(_EMPTY_FLAGS), @@ -529,7 +529,7 @@ class _UnaryStreamMultiCallable(grpc.UnaryStreamMultiCallable): event_handler) operations = ( cygrpc.operation_send_initial_metadata( - _common.metadata(metadata), _EMPTY_FLAGS), + _common.cygrpc_metadata(metadata), _EMPTY_FLAGS), cygrpc.operation_send_message(serialized_request, _EMPTY_FLAGS), cygrpc.operation_send_close_from_client(_EMPTY_FLAGS), cygrpc.operation_receive_status_on_client(_EMPTY_FLAGS), @@ -564,7 +564,7 @@ class _StreamUnaryMultiCallable(grpc.StreamUnaryMultiCallable): None) operations = ( cygrpc.operation_send_initial_metadata( - _common.metadata(metadata), _EMPTY_FLAGS), + _common.cygrpc_metadata(metadata), _EMPTY_FLAGS), cygrpc.operation_receive_message(_EMPTY_FLAGS), cygrpc.operation_receive_status_on_client(_EMPTY_FLAGS), ) @@ -608,7 +608,7 @@ class _StreamUnaryMultiCallable(grpc.StreamUnaryMultiCallable): event_handler) operations = ( cygrpc.operation_send_initial_metadata( - _common.metadata(metadata), _EMPTY_FLAGS), + _common.cygrpc_metadata(metadata), _EMPTY_FLAGS), cygrpc.operation_receive_message(_EMPTY_FLAGS), cygrpc.operation_receive_status_on_client(_EMPTY_FLAGS), ) @@ -645,7 +645,7 @@ class _StreamStreamMultiCallable(grpc.StreamStreamMultiCallable): event_handler) operations = ( cygrpc.operation_send_initial_metadata( - _common.metadata(metadata), _EMPTY_FLAGS), + _common.cygrpc_metadata(metadata), _EMPTY_FLAGS), cygrpc.operation_receive_status_on_client(_EMPTY_FLAGS), ) call.start_batch(cygrpc.Operations(operations), event_handler) @@ -846,8 +846,13 @@ def _options(options): else: pairs = list(options) + [ (cygrpc.ChannelArgKey.primary_user_agent_string, _USER_AGENT)] - return cygrpc.ChannelArgs( - cygrpc.ChannelArg(arg_name, arg_value) for arg_name, arg_value in pairs) + encoded_pairs = [ + (_common.encode(arg_name), arg_value) if isinstance(arg_value, int) + else (_common.encode(arg_name), _common.encode(arg_value)) + for arg_name, arg_value in pairs] + return cygrpc.ChannelArgs([ + cygrpc.ChannelArg(arg_name, arg_value) + for arg_name, arg_value in encoded_pairs]) class Channel(grpc.Channel): @@ -860,7 +865,8 @@ class Channel(grpc.Channel): options: Configuration options for the channel. credentials: A cygrpc.ChannelCredentials or None. """ - self._channel = cygrpc.Channel(target, _options(options), credentials) + self._channel = cygrpc.Channel( + _common.encode(target), _options(options), credentials) self._call_state = _ChannelCallState(self._channel) self._connectivity_state = _ChannelConnectivityState(self._channel) @@ -873,26 +879,26 @@ class Channel(grpc.Channel): def unary_unary( self, method, request_serializer=None, response_deserializer=None): return _UnaryUnaryMultiCallable( - self._channel, _create_channel_managed_call(self._call_state), method, - request_serializer, response_deserializer) + self._channel, _create_channel_managed_call(self._call_state), + _common.encode(method), request_serializer, response_deserializer) def unary_stream( self, method, request_serializer=None, response_deserializer=None): return _UnaryStreamMultiCallable( - self._channel, _create_channel_managed_call(self._call_state), method, - request_serializer, response_deserializer) + self._channel, _create_channel_managed_call(self._call_state), + _common.encode(method), request_serializer, response_deserializer) def stream_unary( self, method, request_serializer=None, response_deserializer=None): return _StreamUnaryMultiCallable( - self._channel, _create_channel_managed_call(self._call_state), method, - request_serializer, response_deserializer) + self._channel, _create_channel_managed_call(self._call_state), + _common.encode(method), request_serializer, response_deserializer) def stream_stream( self, method, request_serializer=None, response_deserializer=None): return _StreamStreamMultiCallable( - self._channel, _create_channel_managed_call(self._call_state), method, - request_serializer, response_deserializer) + self._channel, _create_channel_managed_call(self._call_state), + _common.encode(method), request_serializer, response_deserializer) def __del__(self): _moot(self._connectivity_state) diff --git a/src/python/grpcio/grpc/_common.py b/src/python/grpcio/grpc/_common.py index f351bea9e39..4d7d5214195 100644 --- a/src/python/grpcio/grpc/_common.py +++ b/src/python/grpcio/grpc/_common.py @@ -76,9 +76,37 @@ STATUS_CODE_TO_CYGRPC_STATUS_CODE = { } -def metadata(application_metadata): +def encode(s): + if isinstance(s, bytes): + return s + else: + return s.encode('ascii') + + +def decode(b): + if isinstance(b, str): + return b + else: + try: + return b.decode('utf8') + except UnicodeDecodeError: + logging.exception('Invalid encoding on {}'.format(b)) + return b.decode('latin1') + + +def cygrpc_metadata(application_metadata): return _EMPTY_METADATA if application_metadata is None else cygrpc.Metadata( - cygrpc.Metadatum(key, value) for key, value in application_metadata) + cygrpc.Metadatum(encode(key), encode(value)) + for key, value in application_metadata) + + +def application_metadata(cygrpc_metadata): + if cygrpc_metadata is None: + return () + else: + return tuple( + (decode(key), value if key[-4:] == b'-bin' else decode(value)) + for key, value in cygrpc_metadata) def _transform(message, transformer, exception_message): @@ -101,17 +129,8 @@ def deserialize(serialized_message, deserializer): 'Exception deserializing message!') -def _encode(s): - if isinstance(s, bytes): - return s - else: - return s.encode('ascii') - - def fully_qualified_method(group, method): - group = _encode(group) - method = _encode(method) - return b'/' + group + b'/' + method + return '/{}/{}'.format(group, method) class CleanupThread(threading.Thread): diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi index 866cff0d019..14066965104 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi @@ -32,9 +32,8 @@ cimport cpython cdef class Channel: - def __cinit__(self, target, ChannelArgs arguments=None, + def __cinit__(self, bytes target, ChannelArgs arguments=None, ChannelCredentials channel_credentials=None): - target = str_to_bytes(target) cdef grpc_channel_args *c_arguments = NULL cdef char *c_target = NULL self.c_channel = NULL @@ -57,8 +56,6 @@ cdef class Channel: def create_call(self, Call parent, int flags, CompletionQueue queue not None, method, host, Timespec deadline not None): - method = str_to_bytes(method) - host = str_to_bytes(host) if queue.is_shutting_down: raise ValueError("queue must not be shutting down or shutdown") cdef char *method_c_string = method diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi index 470382d6091..b24e69243e9 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi @@ -82,7 +82,7 @@ cdef class ServerCredentials: cdef class CredentialsMetadataPlugin: - def __cinit__(self, object plugin_callback, name): + def __cinit__(self, object plugin_callback, bytes name): """ Args: plugin_callback (callable): Callback accepting a service URL (str/bytes) @@ -91,9 +91,8 @@ cdef class CredentialsMetadataPlugin: when called should be non-blocking and eventually call the callback object with the appropriate status code/details and metadata (if successful). - name (str): Plugin name. + name (bytes): Plugin name. """ - name = str_to_bytes(name) if not callable(plugin_callback): raise ValueError('expected callable plugin_callback') self.plugin_callback = plugin_callback @@ -130,8 +129,7 @@ cdef void plugin_get_metadata( grpc_credentials_plugin_metadata_cb cb, void *user_data) with gil: def python_callback( Metadata metadata, grpc_status_code status, - error_details): - error_details = str_to_bytes(error_details) + bytes error_details): cb(user_data, metadata.c_metadata_array.metadata, metadata.c_metadata_array.count, status, error_details) cdef CredentialsMetadataPlugin self = state diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi index 0055d0d3a28..8e651e880f3 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi @@ -231,17 +231,10 @@ cdef class Event: cdef class ByteBuffer: - def __cinit__(self, data): + def __cinit__(self, bytes data): if data is None: self.c_byte_buffer = NULL return - if isinstance(data, ByteBuffer): - data = (data).bytes() - if data is None: - self.c_byte_buffer = NULL - return - else: - data = str_to_bytes(data) cdef char *c_data = data cdef gpr_slice data_slice @@ -296,26 +289,28 @@ cdef class ByteBuffer: cdef class SslPemKeyCertPair: - def __cinit__(self, private_key, certificate_chain): - self.private_key = str_to_bytes(private_key) - self.certificate_chain = str_to_bytes(certificate_chain) + def __cinit__(self, bytes private_key, bytes certificate_chain): + self.private_key = private_key + self.certificate_chain = certificate_chain self.c_pair.private_key = self.private_key self.c_pair.certificate_chain = self.certificate_chain cdef class ChannelArg: - def __cinit__(self, key, value): - self.key = str_to_bytes(key) + def __cinit__(self, bytes key, value): + self.key = key self.c_arg.key = self.key if isinstance(value, int): - self.value = int(value) + self.value = value self.c_arg.type = GRPC_ARG_INTEGER self.c_arg.value.integer = self.value - else: - self.value = str_to_bytes(value) + elif isinstance(value, bytes): + self.value = value self.c_arg.type = GRPC_ARG_STRING self.c_arg.value.string = self.value + else: + raise TypeError('Expected int or bytes, got {}'.format(type(value))) cdef class ChannelArgs: @@ -347,9 +342,9 @@ cdef class ChannelArgs: cdef class Metadatum: - def __cinit__(self, key, value): - self._key = str_to_bytes(key) - self._value = str_to_bytes(value) + def __cinit__(self, bytes key, bytes value): + self._key = key + self._value = value self.c_metadata.key = self._key self.c_metadata.value = self._value self.c_metadata.value_length = len(self._value) @@ -563,8 +558,7 @@ def operation_send_close_from_client(int flags): return op def operation_send_status_from_server( - Metadata metadata, grpc_status_code code, details, int flags): - details = str_to_bytes(details) + Metadata metadata, grpc_status_code code, bytes details, int flags): cdef Operation op = Operation() op.c_op.type = GRPC_OP_SEND_STATUS_FROM_SERVER op.c_op.flags = flags diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/server.pyx.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/server.pyx.pxi index 42afeb84987..3e03b6efe10 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/server.pyx.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/server.pyx.pxi @@ -101,7 +101,7 @@ cdef class Server: # Ensure the core has gotten a chance to do the start-up work self.backup_shutdown_queue.poll(Timespec(None)) - def add_http2_port(self, address, + def add_http2_port(self, bytes address, ServerCredentials server_credentials=None): address = str_to_bytes(address) self.references.append(address) diff --git a/src/python/grpcio/grpc/_plugin_wrapping.py b/src/python/grpcio/grpc/_plugin_wrapping.py index 4e9cfe710cd..7cb5218c22b 100644 --- a/src/python/grpcio/grpc/_plugin_wrapping.py +++ b/src/python/grpcio/grpc/_plugin_wrapping.py @@ -31,6 +31,7 @@ import collections import threading import grpc +from grpc import _common from grpc._cython import cygrpc @@ -62,17 +63,16 @@ class _WrappedCygrpcCallback(object): # TODO(atash) translate different Exception superclasses into different # status codes. self.cygrpc_callback( - cygrpc.Metadata([]), cygrpc.StatusCode.internal, error.message) + _common.EMPTY_METADATA, cygrpc.StatusCode.internal, + _common.encode(str(error))) def _invoke_success(self, metadata): try: - cygrpc_metadata = cygrpc.Metadata( - cygrpc.Metadatum(key, value) - for key, value in metadata) + cygrpc_metadata = _common.cygrpc_metadata(metadata) except Exception as error: self._invoke_failure(error) return - self.cygrpc_callback(cygrpc_metadata, cygrpc.StatusCode.ok, '') + self.cygrpc_callback(cygrpc_metadata, cygrpc.StatusCode.ok, b'') def __call__(self, metadata, error): with self.is_called_lock: @@ -101,7 +101,7 @@ class _WrappedPlugin(object): def __call__(self, context, cygrpc_callback): wrapped_cygrpc_callback = _WrappedCygrpcCallback(cygrpc_callback) wrapped_context = AuthMetadataContext( - context.service_url, context.method_name) + _common.decode(context.service_url), _common.decode(context.method_name)) try: self.plugin( wrapped_context, AuthMetadataPluginCallback(wrapped_cygrpc_callback)) @@ -120,4 +120,4 @@ def call_credentials_metadata_plugin(plugin, name): plugin's invocation must be non-blocking. """ return cygrpc.call_credentials_metadata_plugin( - cygrpc.CredentialsMetadataPlugin(_WrappedPlugin(plugin), name)) + cygrpc.CredentialsMetadataPlugin(_WrappedPlugin(plugin), _common.encode(name))) diff --git a/src/python/grpcio/grpc/_server.py b/src/python/grpcio/grpc/_server.py index bf20f15f729..f4c114056fe 100644 --- a/src/python/grpcio/grpc/_server.py +++ b/src/python/grpcio/grpc/_server.py @@ -87,7 +87,7 @@ def _abortion_code(state, code): def _details(state): - return '' if state.details is None else state.details + return b'' if state.details is None else state.details class _HandlerCallDetails( @@ -146,14 +146,14 @@ def _abort(state, call, code, details): cygrpc.operation_send_initial_metadata( _EMPTY_METADATA, _EMPTY_FLAGS), cygrpc.operation_send_status_from_server( - _common.metadata(state.trailing_metadata), effective_code, + _common.cygrpc_metadata(state.trailing_metadata), effective_code, effective_details, _EMPTY_FLAGS), ) token = _SEND_INITIAL_METADATA_AND_SEND_STATUS_FROM_SERVER_TOKEN else: operations = ( cygrpc.operation_send_status_from_server( - _common.metadata(state.trailing_metadata), effective_code, + _common.cygrpc_metadata(state.trailing_metadata), effective_code, effective_details, _EMPTY_FLAGS), ) token = _SEND_STATUS_FROM_SERVER_TOKEN @@ -191,7 +191,7 @@ def _receive_message(state, call, request_deserializer): if request is None: _abort( state, call, cygrpc.StatusCode.internal, - 'Exception deserializing request!') + b'Exception deserializing request!') else: state.request = request state.condition.notify_all() @@ -244,10 +244,10 @@ class _Context(grpc.ServicerContext): self._state.disable_next_compression = True def invocation_metadata(self): - return self._rpc_event.request_metadata + return _common.application_metadata(self._rpc_event.request_metadata) def peer(self): - return self._rpc_event.operation_call.peer() + return _common.decode(self._rpc_event.operation_call.peer()) def send_initial_metadata(self, initial_metadata): with self._state.condition: @@ -256,7 +256,7 @@ class _Context(grpc.ServicerContext): else: if self._state.initial_metadata_allowed: operation = cygrpc.operation_send_initial_metadata( - _common.metadata(initial_metadata), _EMPTY_FLAGS) + _common.cygrpc_metadata(initial_metadata), _EMPTY_FLAGS) self._rpc_event.operation_call.start_batch( cygrpc.Operations((operation,)), _send_initial_metadata(self._state)) @@ -267,7 +267,8 @@ class _Context(grpc.ServicerContext): def set_trailing_metadata(self, trailing_metadata): with self._state.condition: - self._state.trailing_metadata = trailing_metadata + self._state.trailing_metadata = _common.cygrpc_metadata( + trailing_metadata) def set_code(self, code): with self._state.condition: @@ -275,7 +276,7 @@ class _Context(grpc.ServicerContext): def set_details(self, details): with self._state.condition: - self._state.details = details + self._state.details = _common.encode(details) class _RequestIterator(object): @@ -346,7 +347,7 @@ def _unary_request(rpc_event, state, request_deserializer): rpc_event.request_call_details.method) _abort( state, rpc_event.operation_call, - cygrpc.StatusCode.unimplemented, details) + cygrpc.StatusCode.unimplemented, _common.encode(details)) return None elif state.client is _CANCELLED: return None @@ -366,8 +367,8 @@ def _call_behavior(rpc_event, state, behavior, argument, request_deserializer): if e not in state.rpc_errors: details = 'Exception calling application: {}'.format(e) logging.exception(details) - _abort( - state, rpc_event.operation_call, cygrpc.StatusCode.unknown, details) + _abort(state, rpc_event.operation_call, + cygrpc.StatusCode.unknown, _common.encode(details)) return None, False @@ -381,8 +382,8 @@ def _take_response_from_response_iterator(rpc_event, state, response_iterator): if e not in state.rpc_errors: details = 'Exception iterating responses: {}'.format(e) logging.exception(details) - _abort( - state, rpc_event.operation_call, cygrpc.StatusCode.unknown, details) + _abort(state, rpc_event.operation_call, + cygrpc.StatusCode.unknown, _common.encode(details)) return None, False @@ -392,7 +393,7 @@ def _serialize_response(rpc_event, state, response, response_serializer): with state.condition: _abort( state, rpc_event.operation_call, cygrpc.StatusCode.internal, - 'Failed to serialize response!') + b'Failed to serialize response!') return None else: return serialized_response @@ -428,7 +429,7 @@ def _send_response(rpc_event, state, serialized_response): def _status(rpc_event, state, serialized_response): with state.condition: if state.client is not _CANCELLED: - trailing_metadata = _common.metadata(state.trailing_metadata) + trailing_metadata = _common.cygrpc_metadata(state.trailing_metadata) code = _completion_code(state) details = _details(state) operations = [ @@ -532,7 +533,8 @@ def _find_method_handler(rpc_event, generic_handlers): for generic_handler in generic_handlers: method_handler = generic_handler.service( _HandlerCallDetails( - rpc_event.request_call_details.method, rpc_event.request_metadata)) + _common.decode(rpc_event.request_call_details.method), + rpc_event.request_metadata)) if method_handler is not None: return method_handler else: @@ -545,7 +547,7 @@ def _handle_unrecognized_method(rpc_event): cygrpc.operation_receive_close_on_server(_EMPTY_FLAGS), cygrpc.operation_send_status_from_server( _EMPTY_METADATA, cygrpc.StatusCode.unimplemented, - 'Method not found!', _EMPTY_FLAGS), + b'Method not found!', _EMPTY_FLAGS), ) rpc_state = _RPCState() rpc_event.operation_call.start_batch( @@ -740,10 +742,10 @@ class Server(grpc.Server): _add_generic_handlers(self._state, generic_rpc_handlers) def add_insecure_port(self, address): - return _add_insecure_port(self._state, address) + return _add_insecure_port(self._state, _common.encode(address)) def add_secure_port(self, address, server_credentials): - return _add_secure_port(self._state, address, server_credentials) + return _add_secure_port(self._state, _common.encode(address), server_credentials) def start(self): _start(self._state) diff --git a/src/python/grpcio/grpc/beta/_server_adaptations.py b/src/python/grpcio/grpc/beta/_server_adaptations.py index c695434dacf..1e1f80156ac 100644 --- a/src/python/grpcio/grpc/beta/_server_adaptations.py +++ b/src/python/grpcio/grpc/beta/_server_adaptations.py @@ -79,7 +79,8 @@ class _FaceServicerContext(face.ServicerContext): return _ServerProtocolContext(self._servicer_context) def invocation_metadata(self): - return self._servicer_context.invocation_metadata() + return _common.cygrpc_metadata( + self._servicer_context.invocation_metadata()) def initial_metadata(self, initial_metadata): self._servicer_context.send_initial_metadata(initial_metadata) diff --git a/src/python/grpcio/tests/unit/_compression_test.py b/src/python/grpcio/tests/unit/_compression_test.py index 6eddb6f83f3..9e8b8578c1e 100644 --- a/src/python/grpcio/tests/unit/_compression_test.py +++ b/src/python/grpcio/tests/unit/_compression_test.py @@ -37,8 +37,8 @@ from grpc.framework.foundation import logging_pool from tests.unit import test_common from tests.unit.framework.common import test_constants -_UNARY_UNARY = b'/test/UnaryUnary' -_STREAM_STREAM = b'/test/StreamStream' +_UNARY_UNARY = '/test/UnaryUnary' +_STREAM_STREAM = '/test/StreamStream' def handle_unary(request, servicer_context): diff --git a/src/python/grpcio/tests/unit/_cython/_cancel_many_calls_test.py b/src/python/grpcio/tests/unit/_cython/_cancel_many_calls_test.py index c1de7790145..cac0c8b3b93 100644 --- a/src/python/grpcio/tests/unit/_cython/_cancel_many_calls_test.py +++ b/src/python/grpcio/tests/unit/_cython/_cancel_many_calls_test.py @@ -159,9 +159,9 @@ class CancelManyCallsTest(unittest.TestCase): server_completion_queue = cygrpc.CompletionQueue() server = cygrpc.Server() server.register_completion_queue(server_completion_queue) - port = server.add_http2_port('[::]:0') + port = server.add_http2_port(b'[::]:0') server.start() - channel = cygrpc.Channel('localhost:{}'.format(port)) + channel = cygrpc.Channel('localhost:{}'.format(port).encode()) state = _State() diff --git a/src/python/grpcio/tests/unit/_cython/_channel_test.py b/src/python/grpcio/tests/unit/_cython/_channel_test.py index 3dc7a246aee..f9c8a3ac62b 100644 --- a/src/python/grpcio/tests/unit/_cython/_channel_test.py +++ b/src/python/grpcio/tests/unit/_cython/_channel_test.py @@ -37,7 +37,7 @@ from tests.unit.framework.common import test_constants def _channel_and_completion_queue(): - channel = cygrpc.Channel('localhost:54321', cygrpc.ChannelArgs(())) + channel = cygrpc.Channel(b'localhost:54321', cygrpc.ChannelArgs(())) completion_queue = cygrpc.CompletionQueue() return channel, completion_queue diff --git a/src/python/grpcio/tests/unit/_cython/_read_some_but_not_all_responses_test.py b/src/python/grpcio/tests/unit/_cython/_read_some_but_not_all_responses_test.py index 6ae7a90fbed..27fcee0d6f1 100644 --- a/src/python/grpcio/tests/unit/_cython/_read_some_but_not_all_responses_test.py +++ b/src/python/grpcio/tests/unit/_cython/_read_some_but_not_all_responses_test.py @@ -126,9 +126,9 @@ class ReadSomeButNotAllResponsesTest(unittest.TestCase): server_completion_queue = cygrpc.CompletionQueue() server = cygrpc.Server() server.register_completion_queue(server_completion_queue) - port = server.add_http2_port('[::]:0') + port = server.add_http2_port(b'[::]:0') server.start() - channel = cygrpc.Channel('localhost:{}'.format(port)) + channel = cygrpc.Channel('localhost:{}'.format(port).encode()) server_shutdown_tag = 'server_shutdown_tag' server_driver = _ServerDriver(server_completion_queue, server_shutdown_tag) diff --git a/src/python/grpcio/tests/unit/_cython/cygrpc_test.py b/src/python/grpcio/tests/unit/_cython/cygrpc_test.py index a006a20ce3f..b740695e35b 100644 --- a/src/python/grpcio/tests/unit/_cython/cygrpc_test.py +++ b/src/python/grpcio/tests/unit/_cython/cygrpc_test.py @@ -46,38 +46,38 @@ def _metadata_plugin_callback(context, callback): callback(cygrpc.Metadata( [cygrpc.Metadatum(_CALL_CREDENTIALS_METADATA_KEY, _CALL_CREDENTIALS_METADATA_VALUE)]), - cygrpc.StatusCode.ok, '') + cygrpc.StatusCode.ok, b'') class TypeSmokeTest(unittest.TestCase): def testStringsInUtilitiesUpDown(self): self.assertEqual(0, cygrpc.StatusCode.ok) - metadatum = cygrpc.Metadatum('a', 'b') - self.assertEqual('a'.encode(), metadatum.key) - self.assertEqual('b'.encode(), metadatum.value) + metadatum = cygrpc.Metadatum(b'a', b'b') + self.assertEqual(b'a', metadatum.key) + self.assertEqual(b'b', metadatum.value) metadata = cygrpc.Metadata([metadatum]) self.assertEqual(1, len(metadata)) self.assertEqual(metadatum.key, metadata[0].key) def testMetadataIteration(self): metadata = cygrpc.Metadata([ - cygrpc.Metadatum('a', 'b'), cygrpc.Metadatum('c', 'd')]) + cygrpc.Metadatum(b'a', b'b'), cygrpc.Metadatum(b'c', b'd')]) iterator = iter(metadata) metadatum = next(iterator) self.assertIsInstance(metadatum, cygrpc.Metadatum) - self.assertEqual(metadatum.key, 'a'.encode()) - self.assertEqual(metadatum.value, 'b'.encode()) + self.assertEqual(metadatum.key, b'a') + self.assertEqual(metadatum.value, b'b') metadatum = next(iterator) self.assertIsInstance(metadatum, cygrpc.Metadatum) - self.assertEqual(metadatum.key, 'c'.encode()) - self.assertEqual(metadatum.value, 'd'.encode()) + self.assertEqual(metadatum.key, b'c') + self.assertEqual(metadatum.value, b'd') with self.assertRaises(StopIteration): next(iterator) def testOperationsIteration(self): operations = cygrpc.Operations([ - cygrpc.operation_send_message('asdf', _EMPTY_FLAGS)]) + cygrpc.operation_send_message(b'asdf', _EMPTY_FLAGS)]) iterator = iter(operations) operation = next(iterator) self.assertIsInstance(operation, cygrpc.Operation) @@ -87,7 +87,7 @@ class TypeSmokeTest(unittest.TestCase): next(iterator) def testOperationFlags(self): - operation = cygrpc.operation_send_message('asdf', + operation = cygrpc.operation_send_message(b'asdf', cygrpc.WriteFlag.no_compress) self.assertEqual(cygrpc.WriteFlag.no_compress, operation.flags) @@ -105,16 +105,16 @@ class TypeSmokeTest(unittest.TestCase): del server def testChannelUpDown(self): - channel = cygrpc.Channel('[::]:0', cygrpc.ChannelArgs([])) + channel = cygrpc.Channel(b'[::]:0', cygrpc.ChannelArgs([])) del channel def testCredentialsMetadataPluginUpDown(self): plugin = cygrpc.CredentialsMetadataPlugin( - lambda ignored_a, ignored_b: None, '') + lambda ignored_a, ignored_b: None, b'') del plugin def testCallCredentialsFromPluginUpDown(self): - plugin = cygrpc.CredentialsMetadataPlugin(_metadata_plugin_callback, '') + plugin = cygrpc.CredentialsMetadataPlugin(_metadata_plugin_callback, b'') call_credentials = cygrpc.call_credentials_metadata_plugin(plugin) del plugin del call_credentials @@ -123,7 +123,7 @@ class TypeSmokeTest(unittest.TestCase): server = cygrpc.Server() completion_queue = cygrpc.CompletionQueue() server.register_completion_queue(completion_queue) - port = server.add_http2_port('[::]:0') + port = server.add_http2_port(b'[::]:0') self.assertIsInstance(port, int) server.start() del server @@ -131,7 +131,7 @@ class TypeSmokeTest(unittest.TestCase): def testServerStartShutdown(self): completion_queue = cygrpc.CompletionQueue() server = cygrpc.Server() - server.add_http2_port('[::]:0') + server.add_http2_port(b'[::]:0') server.register_completion_queue(completion_queue) server.start() shutdown_tag = object() @@ -150,9 +150,9 @@ class ServerClientMixin(object): self.server = cygrpc.Server() self.server.register_completion_queue(self.server_completion_queue) if server_credentials: - self.port = self.server.add_http2_port('[::]:0', server_credentials) + self.port = self.server.add_http2_port(b'[::]:0', server_credentials) else: - self.port = self.server.add_http2_port('[::]:0') + self.port = self.server.add_http2_port(b'[::]:0') self.server.start() self.client_completion_queue = cygrpc.CompletionQueue() if client_credentials: @@ -160,10 +160,10 @@ class ServerClientMixin(object): cygrpc.ChannelArg(cygrpc.ChannelArgKey.ssl_target_name_override, host_override)]) self.client_channel = cygrpc.Channel( - 'localhost:{}'.format(self.port), client_channel_arguments, + 'localhost:{}'.format(self.port).encode(), client_channel_arguments, client_credentials) else: - self.client_channel = cygrpc.Channel('localhost:{}'.format(self.port)) + self.client_channel = cygrpc.Channel('localhost:{}'.format(self.port).encode()) if host_override: self.host_argument = None # default host self.expected_host = host_override diff --git a/src/python/grpcio/tests/unit/_empty_message_test.py b/src/python/grpcio/tests/unit/_empty_message_test.py index f324f6216b7..8c7d697728b 100644 --- a/src/python/grpcio/tests/unit/_empty_message_test.py +++ b/src/python/grpcio/tests/unit/_empty_message_test.py @@ -37,10 +37,10 @@ from tests.unit.framework.common import test_constants _REQUEST = b'' _RESPONSE = b'' -_UNARY_UNARY = b'/test/UnaryUnary' -_UNARY_STREAM = b'/test/UnaryStream' -_STREAM_UNARY = b'/test/StreamUnary' -_STREAM_STREAM = b'/test/StreamStream' +_UNARY_UNARY = '/test/UnaryUnary' +_UNARY_STREAM = '/test/UnaryStream' +_STREAM_UNARY = '/test/StreamUnary' +_STREAM_STREAM = '/test/StreamStream' def handle_unary_unary(request, servicer_context): diff --git a/src/python/grpcio/tests/unit/_metadata_code_details_test.py b/src/python/grpcio/tests/unit/_metadata_code_details_test.py index dd74268cbf1..0fd02d2a227 100644 --- a/src/python/grpcio/tests/unit/_metadata_code_details_test.py +++ b/src/python/grpcio/tests/unit/_metadata_code_details_test.py @@ -47,29 +47,29 @@ _REQUEST_DESERIALIZER = lambda unused_serialized_request: object() _RESPONSE_SERIALIZER = lambda unused_response: _SERIALIZED_RESPONSE _RESPONSE_DESERIALIZER = lambda unused_serialized_resopnse: object() -_SERVICE = b'test.TestService' -_UNARY_UNARY = b'UnaryUnary' -_UNARY_STREAM = b'UnaryStream' -_STREAM_UNARY = b'StreamUnary' -_STREAM_STREAM = b'StreamStream' +_SERVICE = 'test.TestService' +_UNARY_UNARY = 'UnaryUnary' +_UNARY_STREAM = 'UnaryStream' +_STREAM_UNARY = 'StreamUnary' +_STREAM_STREAM = 'StreamStream' _CLIENT_METADATA = ( - (b'client-md-key', b'client-md-key'), - (b'client-md-key-bin', b'\x00\x01') + ('client-md-key', 'client-md-key'), + ('client-md-key-bin', b'\x00\x01') ) _SERVER_INITIAL_METADATA = ( - (b'server-initial-md-key', b'server-initial-md-value'), - (b'server-initial-md-key-bin', b'\x00\x02') + ('server-initial-md-key', 'server-initial-md-value'), + ('server-initial-md-key-bin', b'\x00\x02') ) _SERVER_TRAILING_METADATA = ( - (b'server-trailing-md-key', b'server-trailing-md-value'), - (b'server-trailing-md-key-bin', b'\x00\x03') + ('server-trailing-md-key', 'server-trailing-md-value'), + ('server-trailing-md-key-bin', b'\x00\x03') ) _NON_OK_CODE = grpc.StatusCode.NOT_FOUND -_DETAILS = b'Test details!' +_DETAILS = 'Test details!' class _Servicer(object): @@ -195,15 +195,15 @@ class MetadataCodeDetailsTest(unittest.TestCase): channel = grpc.insecure_channel('localhost:{}'.format(port)) self._unary_unary = channel.unary_unary( - b'/'.join((b'', _SERVICE, _UNARY_UNARY,)), + '/'.join(('', _SERVICE, _UNARY_UNARY,)), request_serializer=_REQUEST_SERIALIZER, response_deserializer=_RESPONSE_DESERIALIZER,) self._unary_stream = channel.unary_stream( - b'/'.join((b'', _SERVICE, _UNARY_STREAM,)),) + '/'.join(('', _SERVICE, _UNARY_STREAM,)),) self._stream_unary = channel.stream_unary( - b'/'.join((b'', _SERVICE, _STREAM_UNARY,)),) + '/'.join(('', _SERVICE, _STREAM_UNARY,)),) self._stream_stream = channel.stream_stream( - b'/'.join((b'', _SERVICE, _STREAM_STREAM,)), + '/'.join(('', _SERVICE, _STREAM_STREAM,)), request_serializer=_REQUEST_SERIALIZER, response_deserializer=_RESPONSE_DESERIALIZER,) diff --git a/src/python/grpcio/tests/unit/_metadata_test.py b/src/python/grpcio/tests/unit/_metadata_test.py index 2cb13f236bd..c637a28039d 100644 --- a/src/python/grpcio/tests/unit/_metadata_test.py +++ b/src/python/grpcio/tests/unit/_metadata_test.py @@ -44,33 +44,33 @@ _CHANNEL_ARGS = (('grpc.primary_user_agent', 'primary-agent'), _REQUEST = b'\x00\x00\x00' _RESPONSE = b'\x00\x00\x00' -_UNARY_UNARY = b'/test/UnaryUnary' -_UNARY_STREAM = b'/test/UnaryStream' -_STREAM_UNARY = b'/test/StreamUnary' -_STREAM_STREAM = b'/test/StreamStream' +_UNARY_UNARY = '/test/UnaryUnary' +_UNARY_STREAM = '/test/UnaryStream' +_STREAM_UNARY = '/test/StreamUnary' +_STREAM_STREAM = '/test/StreamStream' _USER_AGENT = 'Python-gRPC-{}'.format(_grpcio_metadata.__version__) _CLIENT_METADATA = ( - (b'client-md-key', b'client-md-key'), - (b'client-md-key-bin', b'\x00\x01') + ('client-md-key', 'client-md-key'), + ('client-md-key-bin', b'\x00\x01') ) _SERVER_INITIAL_METADATA = ( - (b'server-initial-md-key', b'server-initial-md-value'), - (b'server-initial-md-key-bin', b'\x00\x02') + ('server-initial-md-key', 'server-initial-md-value'), + ('server-initial-md-key-bin', b'\x00\x02') ) _SERVER_TRAILING_METADATA = ( - (b'server-trailing-md-key', b'server-trailing-md-value'), - (b'server-trailing-md-key-bin', b'\x00\x03') + ('server-trailing-md-key', 'server-trailing-md-value'), + ('server-trailing-md-key-bin', b'\x00\x03') ) def user_agent(metadata): for key, val in metadata: - if key == b'user-agent': - return val.decode('ascii') + if key == 'user-agent': + return val raise KeyError('No user agent!') diff --git a/src/python/grpcio/tests/unit/_rpc_test.py b/src/python/grpcio/tests/unit/_rpc_test.py index 9814504edff..c70d65a6dfb 100644 --- a/src/python/grpcio/tests/unit/_rpc_test.py +++ b/src/python/grpcio/tests/unit/_rpc_test.py @@ -45,10 +45,10 @@ _DESERIALIZE_REQUEST = lambda bytestring: bytestring[len(bytestring) // 2:] _SERIALIZE_RESPONSE = lambda bytestring: bytestring * 3 _DESERIALIZE_RESPONSE = lambda bytestring: bytestring[:len(bytestring) // 3] -_UNARY_UNARY = b'/test/UnaryUnary' -_UNARY_STREAM = b'/test/UnaryStream' -_STREAM_UNARY = b'/test/StreamUnary' -_STREAM_STREAM = b'/test/StreamStream' +_UNARY_UNARY = '/test/UnaryUnary' +_UNARY_STREAM = '/test/UnaryStream' +_STREAM_UNARY = '/test/StreamUnary' +_STREAM_STREAM = '/test/StreamStream' class _Callback(object): @@ -79,7 +79,7 @@ class _Handler(object): def handle_unary_unary(self, request, servicer_context): self._control.control() if servicer_context is not None: - servicer_context.set_trailing_metadata(((b'testkey', b'testvalue',),)) + servicer_context.set_trailing_metadata((('testkey', 'testvalue',),)) return request def handle_unary_stream(self, request, servicer_context): @@ -88,7 +88,7 @@ class _Handler(object): yield request self._control.control() if servicer_context is not None: - servicer_context.set_trailing_metadata(((b'testkey', b'testvalue',),)) + servicer_context.set_trailing_metadata((('testkey', 'testvalue',),)) def handle_stream_unary(self, request_iterator, servicer_context): if servicer_context is not None: @@ -100,13 +100,13 @@ class _Handler(object): response_elements.append(request) self._control.control() if servicer_context is not None: - servicer_context.set_trailing_metadata(((b'testkey', b'testvalue',),)) + servicer_context.set_trailing_metadata((('testkey', 'testvalue',),)) return b''.join(response_elements) def handle_stream_stream(self, request_iterator, servicer_context): self._control.control() if servicer_context is not None: - servicer_context.set_trailing_metadata(((b'testkey', b'testvalue',),)) + servicer_context.set_trailing_metadata((('testkey', 'testvalue',),)) for request in request_iterator: self._control.control() yield request @@ -185,7 +185,7 @@ class RPCTest(unittest.TestCase): self._server_pool = logging_pool.pool(test_constants.THREAD_CONCURRENCY) self._server = grpc.server((), self._server_pool) - port = self._server.add_insecure_port(b'[::]:0') + port = self._server.add_insecure_port('[::]:0') self._server.add_generic_rpc_handlers((_GenericHandler(self._handler),)) self._server.start() @@ -195,7 +195,7 @@ class RPCTest(unittest.TestCase): request = b'abc' with self.assertRaises(grpc.RpcError) as exception_context: - self._channel.unary_unary(b'NoSuchMethod')(request) + self._channel.unary_unary('NoSuchMethod')(request) self.assertEqual( grpc.StatusCode.UNIMPLEMENTED, exception_context.exception.code()) @@ -207,7 +207,7 @@ class RPCTest(unittest.TestCase): multi_callable = _unary_unary_multi_callable(self._channel) response = multi_callable( request, metadata=( - (b'test', b'SuccessfulUnaryRequestBlockingUnaryResponse'),)) + ('test', 'SuccessfulUnaryRequestBlockingUnaryResponse'),)) self.assertEqual(expected_response, response) @@ -218,7 +218,7 @@ class RPCTest(unittest.TestCase): multi_callable = _unary_unary_multi_callable(self._channel) response, call = multi_callable.with_call( request, metadata=( - (b'test', b'SuccessfulUnaryRequestBlockingUnaryResponseWithCall'),)) + ('test', 'SuccessfulUnaryRequestBlockingUnaryResponseWithCall'),)) self.assertEqual(expected_response, response) self.assertIs(grpc.StatusCode.OK, call.code()) @@ -230,7 +230,7 @@ class RPCTest(unittest.TestCase): multi_callable = _unary_unary_multi_callable(self._channel) response_future = multi_callable.future( request, metadata=( - (b'test', b'SuccessfulUnaryRequestFutureUnaryResponse'),)) + ('test', 'SuccessfulUnaryRequestFutureUnaryResponse'),)) response = response_future.result() self.assertEqual(expected_response, response) @@ -242,7 +242,7 @@ class RPCTest(unittest.TestCase): multi_callable = _unary_stream_multi_callable(self._channel) response_iterator = multi_callable( request, - metadata=((b'test', b'SuccessfulUnaryRequestStreamResponse'),)) + metadata=(('test', 'SuccessfulUnaryRequestStreamResponse'),)) responses = tuple(response_iterator) self.assertSequenceEqual(expected_responses, responses) @@ -255,7 +255,7 @@ class RPCTest(unittest.TestCase): multi_callable = _stream_unary_multi_callable(self._channel) response = multi_callable( request_iterator, - metadata=((b'test', b'SuccessfulStreamRequestBlockingUnaryResponse'),)) + metadata=(('test', 'SuccessfulStreamRequestBlockingUnaryResponse'),)) self.assertEqual(expected_response, response) @@ -268,7 +268,7 @@ class RPCTest(unittest.TestCase): response, call = multi_callable.with_call( request_iterator, metadata=( - (b'test', b'SuccessfulStreamRequestBlockingUnaryResponseWithCall'), + ('test', 'SuccessfulStreamRequestBlockingUnaryResponseWithCall'), )) self.assertEqual(expected_response, response) @@ -283,7 +283,7 @@ class RPCTest(unittest.TestCase): response_future = multi_callable.future( request_iterator, metadata=( - (b'test', b'SuccessfulStreamRequestFutureUnaryResponse'),)) + ('test', 'SuccessfulStreamRequestFutureUnaryResponse'),)) response = response_future.result() self.assertEqual(expected_response, response) @@ -297,7 +297,7 @@ class RPCTest(unittest.TestCase): multi_callable = _stream_stream_multi_callable(self._channel) response_iterator = multi_callable( request_iterator, - metadata=((b'test', b'SuccessfulStreamRequestStreamResponse'),)) + metadata=(('test', 'SuccessfulStreamRequestStreamResponse'),)) responses = tuple(response_iterator) self.assertSequenceEqual(expected_responses, responses) @@ -312,9 +312,9 @@ class RPCTest(unittest.TestCase): multi_callable = _unary_unary_multi_callable(self._channel) first_response = multi_callable( - first_request, metadata=((b'test', b'SequentialInvocations'),)) + first_request, metadata=(('test', 'SequentialInvocations'),)) second_response = multi_callable( - second_request, metadata=((b'test', b'SequentialInvocations'),)) + second_request, metadata=(('test', 'SequentialInvocations'),)) self.assertEqual(expected_first_response, first_response) self.assertEqual(expected_second_response, second_response) @@ -331,7 +331,7 @@ class RPCTest(unittest.TestCase): request_iterator = iter(requests) response_future = pool.submit( multi_callable, request_iterator, - metadata=((b'test', b'ConcurrentBlockingInvocations'),)) + metadata=(('test', 'ConcurrentBlockingInvocations'),)) response_futures[index] = response_future responses = tuple( response_future.result() for response_future in response_futures) @@ -350,7 +350,7 @@ class RPCTest(unittest.TestCase): request_iterator = iter(requests) response_future = multi_callable.future( request_iterator, - metadata=((b'test', b'ConcurrentFutureInvocations'),)) + metadata=(('test', 'ConcurrentFutureInvocations'),)) response_futures[index] = response_future responses = tuple( response_future.result() for response_future in response_futures) @@ -380,8 +380,8 @@ class RPCTest(unittest.TestCase): inner_response_future = multi_callable.future( request, metadata=( - (b'test', - b'WaitingForSomeButNotAllConcurrentFutureInvocations'),)) + ('test', + 'WaitingForSomeButNotAllConcurrentFutureInvocations'),)) outer_response_future = pool.submit(wrap_future(inner_response_future)) response_futures[index] = outer_response_future @@ -400,7 +400,7 @@ class RPCTest(unittest.TestCase): response_iterator = multi_callable( request, metadata=( - (b'test', b'ConsumingOneStreamResponseUnaryRequest'),)) + ('test', 'ConsumingOneStreamResponseUnaryRequest'),)) next(response_iterator) def testConsumingSomeButNotAllStreamResponsesUnaryRequest(self): @@ -410,7 +410,7 @@ class RPCTest(unittest.TestCase): response_iterator = multi_callable( request, metadata=( - (b'test', b'ConsumingSomeButNotAllStreamResponsesUnaryRequest'),)) + ('test', 'ConsumingSomeButNotAllStreamResponsesUnaryRequest'),)) for _ in range(test_constants.STREAM_LENGTH // 2): next(response_iterator) @@ -422,7 +422,7 @@ class RPCTest(unittest.TestCase): response_iterator = multi_callable( request_iterator, metadata=( - (b'test', b'ConsumingSomeButNotAllStreamResponsesStreamRequest'),)) + ('test', 'ConsumingSomeButNotAllStreamResponsesStreamRequest'),)) for _ in range(test_constants.STREAM_LENGTH // 2): next(response_iterator) @@ -434,7 +434,7 @@ class RPCTest(unittest.TestCase): response_iterator = multi_callable( request_iterator, metadata=( - (b'test', b'ConsumingTooManyStreamResponsesStreamRequest'),)) + ('test', 'ConsumingTooManyStreamResponsesStreamRequest'),)) for _ in range(test_constants.STREAM_LENGTH): next(response_iterator) for _ in range(test_constants.STREAM_LENGTH): @@ -453,7 +453,7 @@ class RPCTest(unittest.TestCase): with self._control.pause(): response_future = multi_callable.future( request, - metadata=((b'test', b'CancelledUnaryRequestUnaryResponse'),)) + metadata=(('test', 'CancelledUnaryRequestUnaryResponse'),)) response_future.cancel() self.assertTrue(response_future.cancelled()) @@ -468,7 +468,7 @@ class RPCTest(unittest.TestCase): with self._control.pause(): response_iterator = multi_callable( request, - metadata=((b'test', b'CancelledUnaryRequestStreamResponse'),)) + metadata=(('test', 'CancelledUnaryRequestStreamResponse'),)) self._control.block_until_paused() response_iterator.cancel() @@ -488,7 +488,7 @@ class RPCTest(unittest.TestCase): with self._control.pause(): response_future = multi_callable.future( request_iterator, - metadata=((b'test', b'CancelledStreamRequestUnaryResponse'),)) + metadata=(('test', 'CancelledStreamRequestUnaryResponse'),)) self._control.block_until_paused() response_future.cancel() @@ -508,7 +508,7 @@ class RPCTest(unittest.TestCase): with self._control.pause(): response_iterator = multi_callable( request_iterator, - metadata=((b'test', b'CancelledStreamRequestStreamResponse'),)) + metadata=(('test', 'CancelledStreamRequestStreamResponse'),)) response_iterator.cancel() with self.assertRaises(grpc.RpcError): @@ -526,7 +526,7 @@ class RPCTest(unittest.TestCase): with self.assertRaises(grpc.RpcError) as exception_context: multi_callable.with_call( request, timeout=test_constants.SHORT_TIMEOUT, - metadata=((b'test', b'ExpiredUnaryRequestBlockingUnaryResponse'),)) + metadata=(('test', 'ExpiredUnaryRequestBlockingUnaryResponse'),)) self.assertIsNotNone(exception_context.exception.initial_metadata()) self.assertIs( @@ -542,7 +542,7 @@ class RPCTest(unittest.TestCase): with self._control.pause(): response_future = multi_callable.future( request, timeout=test_constants.SHORT_TIMEOUT, - metadata=((b'test', b'ExpiredUnaryRequestFutureUnaryResponse'),)) + metadata=(('test', 'ExpiredUnaryRequestFutureUnaryResponse'),)) response_future.add_done_callback(callback) value_passed_to_callback = callback.value() @@ -567,7 +567,7 @@ class RPCTest(unittest.TestCase): with self.assertRaises(grpc.RpcError) as exception_context: response_iterator = multi_callable( request, timeout=test_constants.SHORT_TIMEOUT, - metadata=((b'test', b'ExpiredUnaryRequestStreamResponse'),)) + metadata=(('test', 'ExpiredUnaryRequestStreamResponse'),)) next(response_iterator) self.assertIs( @@ -583,7 +583,7 @@ class RPCTest(unittest.TestCase): with self.assertRaises(grpc.RpcError) as exception_context: multi_callable( request_iterator, timeout=test_constants.SHORT_TIMEOUT, - metadata=((b'test', b'ExpiredStreamRequestBlockingUnaryResponse'),)) + metadata=(('test', 'ExpiredStreamRequestBlockingUnaryResponse'),)) self.assertIsNotNone(exception_context.exception.initial_metadata()) self.assertIs( @@ -600,7 +600,7 @@ class RPCTest(unittest.TestCase): with self._control.pause(): response_future = multi_callable.future( request_iterator, timeout=test_constants.SHORT_TIMEOUT, - metadata=((b'test', b'ExpiredStreamRequestFutureUnaryResponse'),)) + metadata=(('test', 'ExpiredStreamRequestFutureUnaryResponse'),)) response_future.add_done_callback(callback) value_passed_to_callback = callback.value() @@ -625,7 +625,7 @@ class RPCTest(unittest.TestCase): with self.assertRaises(grpc.RpcError) as exception_context: response_iterator = multi_callable( request_iterator, timeout=test_constants.SHORT_TIMEOUT, - metadata=((b'test', b'ExpiredStreamRequestStreamResponse'),)) + metadata=(('test', 'ExpiredStreamRequestStreamResponse'),)) next(response_iterator) self.assertIs( @@ -640,7 +640,7 @@ class RPCTest(unittest.TestCase): with self.assertRaises(grpc.RpcError) as exception_context: multi_callable.with_call( request, - metadata=((b'test', b'FailedUnaryRequestBlockingUnaryResponse'),)) + metadata=(('test', 'FailedUnaryRequestBlockingUnaryResponse'),)) self.assertIs(grpc.StatusCode.UNKNOWN, exception_context.exception.code()) @@ -652,7 +652,7 @@ class RPCTest(unittest.TestCase): with self._control.fail(): response_future = multi_callable.future( request, - metadata=((b'test', b'FailedUnaryRequestFutureUnaryResponse'),)) + metadata=(('test', 'FailedUnaryRequestFutureUnaryResponse'),)) response_future.add_done_callback(callback) value_passed_to_callback = callback.value() @@ -672,7 +672,7 @@ class RPCTest(unittest.TestCase): with self._control.fail(): response_iterator = multi_callable( request, - metadata=((b'test', b'FailedUnaryRequestStreamResponse'),)) + metadata=(('test', 'FailedUnaryRequestStreamResponse'),)) next(response_iterator) self.assertIs(grpc.StatusCode.UNKNOWN, exception_context.exception.code()) @@ -686,7 +686,7 @@ class RPCTest(unittest.TestCase): with self.assertRaises(grpc.RpcError) as exception_context: multi_callable( request_iterator, - metadata=((b'test', b'FailedStreamRequestBlockingUnaryResponse'),)) + metadata=(('test', 'FailedStreamRequestBlockingUnaryResponse'),)) self.assertIs(grpc.StatusCode.UNKNOWN, exception_context.exception.code()) @@ -699,7 +699,7 @@ class RPCTest(unittest.TestCase): with self._control.fail(): response_future = multi_callable.future( request_iterator, - metadata=((b'test', b'FailedStreamRequestFutureUnaryResponse'),)) + metadata=(('test', 'FailedStreamRequestFutureUnaryResponse'),)) response_future.add_done_callback(callback) value_passed_to_callback = callback.value() @@ -720,7 +720,7 @@ class RPCTest(unittest.TestCase): with self.assertRaises(grpc.RpcError) as exception_context: response_iterator = multi_callable( request_iterator, - metadata=((b'test', b'FailedStreamRequestStreamResponse'),)) + metadata=(('test', 'FailedStreamRequestStreamResponse'),)) tuple(response_iterator) self.assertIs(grpc.StatusCode.UNKNOWN, exception_context.exception.code()) @@ -732,7 +732,7 @@ class RPCTest(unittest.TestCase): multi_callable = _unary_unary_multi_callable(self._channel) multi_callable.future( request, - metadata=((b'test', b'IgnoredUnaryRequestFutureUnaryResponse'),)) + metadata=(('test', 'IgnoredUnaryRequestFutureUnaryResponse'),)) def testIgnoredUnaryRequestStreamResponse(self): request = b'\x37\x17' @@ -740,7 +740,7 @@ class RPCTest(unittest.TestCase): multi_callable = _unary_stream_multi_callable(self._channel) multi_callable( request, - metadata=((b'test', b'IgnoredUnaryRequestStreamResponse'),)) + metadata=(('test', 'IgnoredUnaryRequestStreamResponse'),)) def testIgnoredStreamRequestFutureUnaryResponse(self): requests = tuple(b'\x07\x18' for _ in range(test_constants.STREAM_LENGTH)) @@ -749,7 +749,7 @@ class RPCTest(unittest.TestCase): multi_callable = _stream_unary_multi_callable(self._channel) multi_callable.future( request_iterator, - metadata=((b'test', b'IgnoredStreamRequestFutureUnaryResponse'),)) + metadata=(('test', 'IgnoredStreamRequestFutureUnaryResponse'),)) def testIgnoredStreamRequestStreamResponse(self): requests = tuple(b'\x67\x88' for _ in range(test_constants.STREAM_LENGTH)) @@ -758,7 +758,7 @@ class RPCTest(unittest.TestCase): multi_callable = _stream_stream_multi_callable(self._channel) multi_callable( request_iterator, - metadata=((b'test', b'IgnoredStreamRequestStreamResponse'),)) + metadata=(('test', 'IgnoredStreamRequestStreamResponse'),)) if __name__ == '__main__': diff --git a/src/python/grpcio/tests/unit/beta/_utilities_test.py b/src/python/grpcio/tests/unit/beta/_utilities_test.py index 08ce98e7516..90fe10c77ce 100644 --- a/src/python/grpcio/tests/unit/beta/_utilities_test.py +++ b/src/python/grpcio/tests/unit/beta/_utilities_test.py @@ -33,21 +33,12 @@ import threading import time import unittest -from grpc._adapter import _low -from grpc._adapter import _types from grpc.beta import implementations from grpc.beta import utilities from grpc.framework.foundation import future from tests.unit.framework.common import test_constants -def _drive_completion_queue(completion_queue): - while True: - event = completion_queue.next(time.time() + 24 * 60 * 60) - if event.type == _types.EventType.QUEUE_SHUTDOWN: - break - - class _Callback(object): def __init__(self): @@ -87,13 +78,9 @@ class ChannelConnectivityTest(unittest.TestCase): self.assertFalse(ready_future.running()) def test_immediately_connectable_channel_connectivity(self): - server_completion_queue = _low.CompletionQueue() - server = _low.Server(server_completion_queue, []) - port = server.add_http2_port('[::]:0') + server = implementations.server({}) + port = server.add_insecure_port('[::]:0') server.start() - server_completion_queue_thread = threading.Thread( - target=_drive_completion_queue, args=(server_completion_queue,)) - server_completion_queue_thread.start() channel = implementations.insecure_channel('localhost', port) callback = _Callback() @@ -114,9 +101,7 @@ class ChannelConnectivityTest(unittest.TestCase): self.assertFalse(ready_future.running()) finally: ready_future.cancel() - server.shutdown() - server_completion_queue.shutdown() - server_completion_queue_thread.join() + server.stop(0) if __name__ == '__main__': diff --git a/src/python/grpcio/tests/unit/beta/test_utilities.py b/src/python/grpcio/tests/unit/beta/test_utilities.py index 8ccad04e056..692da9c97d7 100644 --- a/src/python/grpcio/tests/unit/beta/test_utilities.py +++ b/src/python/grpcio/tests/unit/beta/test_utilities.py @@ -51,5 +51,5 @@ def not_really_secure_channel( target = '%s:%d' % (host, port) channel = grpc.secure_channel( target, channel_credentials, - ((b'grpc.ssl_target_name_override', server_host_override,),)) + (('grpc.ssl_target_name_override', server_host_override,),)) return implementations.Channel(channel) diff --git a/src/python/grpcio/tests/unit/test_common.py b/src/python/grpcio/tests/unit/test_common.py index b779f65e7e9..c8886bf4ca5 100644 --- a/src/python/grpcio/tests/unit/test_common.py +++ b/src/python/grpcio/tests/unit/test_common.py @@ -33,10 +33,10 @@ import collections import six -INVOCATION_INITIAL_METADATA = ((b'0', b'abc'), (b'1', b'def'), (b'2', b'ghi'),) -SERVICE_INITIAL_METADATA = ((b'3', b'jkl'), (b'4', b'mno'), (b'5', b'pqr'),) -SERVICE_TERMINAL_METADATA = ((b'6', b'stu'), (b'7', b'vwx'), (b'8', b'yza'),) -DETAILS = b'test details' +INVOCATION_INITIAL_METADATA = (('0', 'abc'), ('1', 'def'), ('2', 'ghi'),) +SERVICE_INITIAL_METADATA = (('3', 'jkl'), ('4', 'mno'), ('5', 'pqr'),) +SERVICE_TERMINAL_METADATA = (('6', 'stu'), ('7', 'vwx'), ('8', 'yza'),) +DETAILS = 'test details' def metadata_transmitted(original_metadata, transmitted_metadata): @@ -59,16 +59,10 @@ def metadata_transmitted(original_metadata, transmitted_metadata): original_metadata after having been transmitted via gRPC. """ original = collections.defaultdict(list) - for key_value_pair in original_metadata: - key, value = tuple(key_value_pair) - if not isinstance(key, bytes): - key = key.encode() - if not isinstance(value, bytes): - value = value.encode() + for key, value in original_metadata: original[key].append(value) transmitted = collections.defaultdict(list) - for key_value_pair in transmitted_metadata: - key, value = tuple(key_value_pair) + for key, value in transmitted_metadata: transmitted[key].append(value) for key, values in six.iteritems(original): From a45e24eb3e3a8f437303ad982abdd0c0a7327905 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 29 Jun 2016 17:55:01 -0700 Subject: [PATCH 0342/1039] Removed invalid sentence --- doc/interop-test-descriptions.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/doc/interop-test-descriptions.md b/doc/interop-test-descriptions.md index a4f9abecfae..b23ceac8e1b 100644 --- a/doc/interop-test-descriptions.md +++ b/doc/interop-test-descriptions.md @@ -30,8 +30,7 @@ Clients should accept these arguments: [ca.pem](https://github.com/grpc/grpc/blob/master/src/core/lib/tsi/test_creds/ca.pem) as the CA root * --default_service_account=ACCOUNT_EMAIL - * Email of the GCE default service account. Only applicable - for compute_engine_creds test. + * Email of the GCE default service account. * --oauth_scope=SCOPE * OAuth scope. For example, "https://www.googleapis.com/auth/xapi.zoo" * --service_account_key_file=PATH From 0287be8e609c0dd3974025509a590720ab947c6b Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Wed, 29 Jun 2016 17:55:14 -0700 Subject: [PATCH 0343/1039] Use protoc and objc plugins built by run_test --- src/objective-c/tests/build_example_test.sh | 43 ++++++++++++++------- 1 file changed, 28 insertions(+), 15 deletions(-) diff --git a/src/objective-c/tests/build_example_test.sh b/src/objective-c/tests/build_example_test.sh index 73405695e8e..93af0a1cfc2 100755 --- a/src/objective-c/tests/build_example_test.sh +++ b/src/objective-c/tests/build_example_test.sh @@ -35,22 +35,35 @@ set -eo pipefail cd `dirname $0` -EXAMPLE_PATH=examples/objective-c/helloworld \ -SCHEME=HelloWorld \ -./build_one_example.sh +BINDIR=`pwd`/../../../bins/$CONFIG +TMP_PATH=$PATH +hash protoc 2>/dev/null || TMP_PATH=$BINDIR/protobuf:$TMP_PATH +PATH=$TMP_PATH hash protoc-gen-objcgrpc 2>/dev/null || { + ln -sf $BINDIR/grpc_objective_c_plugin $BINDIR/protoc-gen-objcgrpc + TMP_PATH=$BINDIR:$TMP_PATH +} -EXAMPLE_PATH=examples/objective-c/route_guide \ -SCHEME=RouteGuideClient \ -./build_one_example.sh +SCHEME=HelloWorld \ + EXAMPLE_PATH=examples/objective-c/helloworld \ + PATH=$TMP_PATH \ + ./build_one_example.sh -EXAMPLE_PATH=examples/objective-c/auth_sample \ -SCHEME=AuthSample \ -./build_one_example.sh +SCHEME=RouteGuideClient \ + EXAMPLE_PATH=examples/objective-c/route_guide \ + PATH=$TMP_PATH \ + ./build_one_example.sh -EXAMPLE_PATH=src/objective-c/examples/Sample \ -SCHEME=Sample \ -./build_one_example.sh +SCHEME=AuthSample \ + EXAMPLE_PATH=examples/objective-c/auth_sample \ + PATH=$TMP_PATH \ + ./build_one_example.sh -EXAMPLE_PATH=src/objective-c/examples/SwiftSample \ -SCHEME=SwiftSample \ -./build_one_example.sh +SCHEME=Sample \ + EXAMPLE_PATH=src/objective-c/examples/Sample \ + PATH=$TMP_PATH \ + ./build_one_example.sh + +SCHEME=SwiftSample \ + EXAMPLE_PATH=src/objective-c/examples/SwiftSample \ + PATH=$TMP_PATH \ + ./build_one_example.sh From 1e1a816c3f72b83e5ba304f32db57557a61e0f2f Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 29 Jun 2016 20:12:40 -0700 Subject: [PATCH 0344/1039] fixed size_t format string --- test/core/network_benchmarks/low_level_ping_pong.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/core/network_benchmarks/low_level_ping_pong.c b/test/core/network_benchmarks/low_level_ping_pong.c index 1b40895a719..9038d076755 100644 --- a/test/core/network_benchmarks/low_level_ping_pong.c +++ b/test/core/network_benchmarks/low_level_ping_pong.c @@ -583,7 +583,7 @@ static int run_benchmark(char *socket_type, thread_args *client_args, return rv; } - gpr_log(GPR_INFO, "Starting test %s %s %d", client_args->strategy_name, + gpr_log(GPR_INFO, "Starting test %s %s %zu", client_args->strategy_name, socket_type, client_args->msg_size); gpr_thd_new(&tid, server_thread_wrap, server_args, NULL); From 314d3350d928179cfd810b72e95c87ee86cebb5c Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Thu, 30 Jun 2016 09:51:02 -0700 Subject: [PATCH 0345/1039] removed unused version of status_code_enum.h --- .../impl/codegen/impl/status_code_enum.h | 152 ------------------ 1 file changed, 152 deletions(-) delete mode 100644 include/grpc++/impl/codegen/impl/status_code_enum.h diff --git a/include/grpc++/impl/codegen/impl/status_code_enum.h b/include/grpc++/impl/codegen/impl/status_code_enum.h deleted file mode 100644 index f8caec0c119..00000000000 --- a/include/grpc++/impl/codegen/impl/status_code_enum.h +++ /dev/null @@ -1,152 +0,0 @@ -/* - * - * Copyright 2016, Google Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following disclaimer - * in the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Google Inc. nor the names of its - * contributors may be used to endorse or promote products derived from - * this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - */ - -#ifndef GRPCXX_IMPL_CODEGEN_IMPL_STATUS_CODE_ENUM_H -#define GRPCXX_IMPL_CODEGEN_IMPL_STATUS_CODE_ENUM_H - -namespace grpc { - -enum StatusCode { - /// Not an error; returned on success. - OK = 0, - - /// The operation was cancelled (typically by the caller). - CANCELLED = 1, - - /// Unknown error. An example of where this error may be returned is if a - /// Status value received from another address space belongs to an error-space - /// that is not known in this address space. Also errors raised by APIs that - /// do not return enough error information may be converted to this error. - UNKNOWN = 2, - - /// Client specified an invalid argument. Note that this differs from - /// FAILED_PRECONDITION. INVALID_ARGUMENT indicates arguments that are - /// problematic regardless of the state of the system (e.g., a malformed file - /// name). - INVALID_ARGUMENT = 3, - - /// Deadline expired before operation could complete. For operations that - /// change the state of the system, this error may be returned even if the - /// operation has completed successfully. For example, a successful response - /// from a server could have been delayed long enough for the deadline to - /// expire. - DEADLINE_EXCEEDED = 4, - - /// Some requested entity (e.g., file or directory) was not found. - NOT_FOUND = 5, - - /// Some entity that we attempted to create (e.g., file or directory) already - /// exists. - ALREADY_EXISTS = 6, - - /// The caller does not have permission to execute the specified operation. - /// PERMISSION_DENIED must not be used for rejections caused by exhausting - /// some resource (use RESOURCE_EXHAUSTED instead for those errors). - /// PERMISSION_DENIED must not be used if the caller can not be identified - /// (use UNAUTHENTICATED instead for those errors). - PERMISSION_DENIED = 7, - - /// The request does not have valid authentication credentials for the - /// operation. - UNAUTHENTICATED = 16, - - /// Some resource has been exhausted, perhaps a per-user quota, or perhaps the - /// entire file system is out of space. - RESOURCE_EXHAUSTED = 8, - - /// Operation was rejected because the system is not in a state required for - /// the operation's execution. For example, directory to be deleted may be - /// non-empty, an rmdir operation is applied to a non-directory, etc. - /// - /// A litmus test that may help a service implementor in deciding - /// between FAILED_PRECONDITION, ABORTED, and UNAVAILABLE: - /// (a) Use UNAVAILABLE if the client can retry just the failing call. - /// (b) Use ABORTED if the client should retry at a higher-level - /// (e.g., restarting a read-modify-write sequence). - /// (c) Use FAILED_PRECONDITION if the client should not retry until - /// the system state has been explicitly fixed. E.g., if an "rmdir" - /// fails because the directory is non-empty, FAILED_PRECONDITION - /// should be returned since the client should not retry unless - /// they have first fixed up the directory by deleting files from it. - /// (d) Use FAILED_PRECONDITION if the client performs conditional - /// REST Get/Update/Delete on a resource and the resource on the - /// server does not match the condition. E.g., conflicting - /// read-modify-write on the same resource. - FAILED_PRECONDITION = 9, - - /// The operation was aborted, typically due to a concurrency issue like - /// sequencer check failures, transaction aborts, etc. - /// - /// See litmus test above for deciding between FAILED_PRECONDITION, ABORTED, - /// and UNAVAILABLE. - ABORTED = 10, - - /// Operation was attempted past the valid range. E.g., seeking or reading - /// past end of file. - /// - /// Unlike INVALID_ARGUMENT, this error indicates a problem that may be fixed - /// if the system state changes. For example, a 32-bit file system will - /// generate INVALID_ARGUMENT if asked to read at an offset that is not in the - /// range [0,2^32-1], but it will generate OUT_OF_RANGE if asked to read from - /// an offset past the current file size. - /// - /// There is a fair bit of overlap between FAILED_PRECONDITION and - /// OUT_OF_RANGE. We recommend using OUT_OF_RANGE (the more specific error) - /// when it applies so that callers who are iterating through a space can - /// easily look for an OUT_OF_RANGE error to detect when they are done. - OUT_OF_RANGE = 11, - - /// Operation is not implemented or not supported/enabled in this service. - UNIMPLEMENTED = 12, - - /// Internal errors. Means some invariants expected by underlying System has - /// been broken. If you see one of these errors, Something is very broken. - INTERNAL = 13, - - /// The service is currently unavailable. This is a most likely a transient - /// condition and may be corrected by retrying with a backoff. - /// - /// See litmus test above for deciding between FAILED_PRECONDITION, ABORTED, - /// and UNAVAILABLE. - UNAVAILABLE = 14, - - /// Unrecoverable data loss or corruption. - DATA_LOSS = 15, - - /// Force users to include a default branch: - DO_NOT_USE = -1 -}; - -} // namespace grpc - -#endif // GRPCXX_IMPL_CODEGEN_IMPL_STATUS_CODE_ENUM_H From 1ff168ac810011a43e446bbbc8ee908ee0098269 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Thu, 30 Jun 2016 10:25:04 -0700 Subject: [PATCH 0346/1039] Initialize variables --- src/core/lib/surface/call.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/core/lib/surface/call.c b/src/core/lib/surface/call.c index 708ea3502af..e5668be47fe 100644 --- a/src/core/lib/surface/call.c +++ b/src/core/lib/surface/call.c @@ -1206,7 +1206,7 @@ static void validate_filtered_metadata(grpc_exec_ctx *exec_ctx, } else if (grpc_compression_options_is_algorithm_enabled( &compression_options, algo) == 0) { /* check if algorithm is supported by current channel config */ - char *algo_name; + char *algo_name = NULL; grpc_compression_algorithm_name(algo, &algo_name); gpr_asprintf(&error_msg, "Compression algorithm '%s' is disabled.", algo_name); @@ -1225,7 +1225,7 @@ static void validate_filtered_metadata(grpc_exec_ctx *exec_ctx, call->incoming_compression_algorithm)) { extern int grpc_compression_trace; if (grpc_compression_trace) { - char *algo_name; + char *algo_name = NULL; grpc_compression_algorithm_name(call->incoming_compression_algorithm, &algo_name); gpr_log(GPR_ERROR, @@ -1426,7 +1426,7 @@ static grpc_call_error call_start_batch(grpc_exec_ctx *exec_ctx, const grpc_compression_algorithm calgo = compression_algorithm_for_level_locked( call, effective_compression_level); - char *calgo_name; + char *calgo_name = NULL; grpc_compression_algorithm_name(calgo, &calgo_name); // the following will be picked up by the compress filter and used as // the call's compression algorithm. From f329395514182ef61f01552daabba471ae28085e Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Thu, 30 Jun 2016 10:44:30 -0700 Subject: [PATCH 0347/1039] Add comments in build_example_test.sh --- src/objective-c/tests/build_example_test.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/objective-c/tests/build_example_test.sh b/src/objective-c/tests/build_example_test.sh index 93af0a1cfc2..5c3766b4c0b 100755 --- a/src/objective-c/tests/build_example_test.sh +++ b/src/objective-c/tests/build_example_test.sh @@ -37,7 +37,12 @@ cd `dirname $0` BINDIR=`pwd`/../../../bins/$CONFIG TMP_PATH=$PATH + +# If `protoc` is not found, add bins/$CONFIG/protobuf/protoc to the search path hash protoc 2>/dev/null || TMP_PATH=$BINDIR/protobuf:$TMP_PATH + +# If `protoc-gen-objcgrpc` is not found, make a symlink from +# bins/$CONGIF/grpc_objective_c_plugin and add it to the search path PATH=$TMP_PATH hash protoc-gen-objcgrpc 2>/dev/null || { ln -sf $BINDIR/grpc_objective_c_plugin $BINDIR/protoc-gen-objcgrpc TMP_PATH=$BINDIR:$TMP_PATH From 97d1cd87670f6eb082d2243bc7cfd4a21d33a78f Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 30 Jun 2016 11:31:23 -0700 Subject: [PATCH 0348/1039] Fix memory leak, fix error.c refcount reporting --- src/core/lib/iomgr/error.c | 8 ++++---- src/core/lib/transport/transport.c | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/core/lib/iomgr/error.c b/src/core/lib/iomgr/error.c index e20a0169c50..149c55663c0 100644 --- a/src/core/lib/iomgr/error.c +++ b/src/core/lib/iomgr/error.c @@ -174,8 +174,8 @@ static bool is_special(grpc_error *err) { grpc_error *grpc_error_ref(grpc_error *err, const char *file, int line, const char *func) { if (is_special(err)) return err; - gpr_log(GPR_DEBUG, "%p: %d -> %d [%s:%d %s]", err, err->refs.count, - err->refs.count + 1, file, line, func); + gpr_log(GPR_DEBUG, "%p: %" PRIdPTR " -> %" PRIdPTR " [%s:%d %s]", err, + err->refs.count, err->refs.count + 1, file, line, func); gpr_ref(&err->refs); return err; } @@ -200,8 +200,8 @@ static void error_destroy(grpc_error *err) { void grpc_error_unref(grpc_error *err, const char *file, int line, const char *func) { if (is_special(err)) return; - gpr_log(GPR_DEBUG, "%p: %d -> %d [%s:%d %s]", err, err->refs.count, - err->refs.count - 1, file, line, func); + gpr_log(GPR_DEBUG, "%p: %" PRIdPTR " -> %" PRIdPTR " [%s:%d %s]", err, + err->refs.count, err->refs.count - 1, file, line, func); if (gpr_unref(&err->refs)) { error_destroy(err); } diff --git a/src/core/lib/transport/transport.c b/src/core/lib/transport/transport.c index 79a20e12626..f8ddf5774a1 100644 --- a/src/core/lib/transport/transport.c +++ b/src/core/lib/transport/transport.c @@ -173,7 +173,7 @@ static void free_message(grpc_exec_ctx *exec_ctx, void *p, grpc_error *error) { close_message_data *cmd = p; GRPC_ERROR_UNREF(cmd->error); if (cmd->then_call != NULL) { - cmd->then_call->cb(exec_ctx, cmd->then_call->cb_arg, GRPC_ERROR_REF(error)); + cmd->then_call->cb(exec_ctx, cmd->then_call->cb_arg, error); } gpr_free(cmd); } From 2cb69ad2039345e71b7b2d9eb2530bc806568df0 Mon Sep 17 00:00:00 2001 From: Stanley Cheung Date: Thu, 30 Jun 2016 11:47:14 -0700 Subject: [PATCH 0349/1039] php: update package.xml --- package.xml | 26 ++++++++++++++++++++++---- templates/package.xml.template | 26 ++++++++++++++++++++++---- 2 files changed, 44 insertions(+), 8 deletions(-) diff --git a/package.xml b/package.xml index ebe4af90e43..49a6d900f12 100644 --- a/package.xml +++ b/package.xml @@ -10,7 +10,7 @@ grpc-packages@google.com yes - 2016-05-19 + 2016-06-30 0.16.0 @@ -22,7 +22,7 @@ BSD -- TBD +- Fix shutdown hang problem #4017 @@ -1034,6 +1034,7 @@ Update to wrap gRPC C Core version 0.10.0 BSD - Simplify gRPC PHP installation #4517 +- Wrap gRPC core library version 0.13 @@ -1063,13 +1064,14 @@ Update to wrap gRPC C Core version 0.10.0 2016-04-19 BSD +- wrap grpc C core version 0.14.0 - destroy grpc_byte_buffer after startBatch #6096 - 0.14.2 - 0.14.2 + 0.15.0 + 0.15.0 beta @@ -1079,6 +1081,22 @@ Update to wrap gRPC C Core version 0.10.0 BSD - Updated functions with TSRM macros for ZTS support #6607 +- Load default roots.pem via grpc_set_ssl_roots_override_callback #6848 + + + + + 0.15.1 + 0.15.1 + + + beta + beta + + 2016-06-30 + BSD + +- Fix shutdown hang problem #4017 diff --git a/templates/package.xml.template b/templates/package.xml.template index a620a2d6a50..d8155cdd823 100644 --- a/templates/package.xml.template +++ b/templates/package.xml.template @@ -12,7 +12,7 @@ grpc-packages@google.com yes - 2016-05-19 + 2016-06-30 ${settings.php_version.php()} @@ -24,7 +24,7 @@ BSD - - TBD + - Fix shutdown hang problem #4017 @@ -153,6 +153,7 @@ BSD - Simplify gRPC PHP installation #4517 + - Wrap gRPC core library version 0.13 @@ -182,13 +183,14 @@ 2016-04-19 BSD + - wrap grpc C core version 0.14.0 - destroy grpc_byte_buffer after startBatch #6096 - 0.14.2 - 0.14.2 + 0.15.0 + 0.15.0 beta @@ -198,6 +200,22 @@ BSD - Updated functions with TSRM macros for ZTS support #6607 + - Load default roots.pem via grpc_set_ssl_roots_override_callback #6848 + + + + + 0.15.1 + 0.15.1 + + + beta + beta + + 2016-06-30 + BSD + + - Fix shutdown hang problem #4017 From 6f7f00b5736196d067ac91cab87fa1d043264e68 Mon Sep 17 00:00:00 2001 From: Stanley Cheung Date: Thu, 30 Jun 2016 11:49:30 -0700 Subject: [PATCH 0350/1039] php: update package.xml --- package.xml | 26 ++++++++++++++++++++++---- templates/package.xml.template | 26 ++++++++++++++++++++++---- 2 files changed, 44 insertions(+), 8 deletions(-) diff --git a/package.xml b/package.xml index 67e9bb2c282..1304eb06fa3 100644 --- a/package.xml +++ b/package.xml @@ -10,7 +10,7 @@ grpc-packages@google.com yes - 2016-05-19 + 2016-06-30 0.15.0 @@ -22,7 +22,7 @@ BSD -- TBD +- Fix shutdown hang problem #4017 @@ -1031,6 +1031,7 @@ Update to wrap gRPC C Core version 0.10.0 BSD - Simplify gRPC PHP installation #4517 +- Wrap gRPC core library version 0.13 @@ -1060,13 +1061,14 @@ Update to wrap gRPC C Core version 0.10.0 2016-04-19 BSD +- wrap grpc C core version 0.14.0 - destroy grpc_byte_buffer after startBatch #6096 - 0.14.2 - 0.14.2 + 0.15.0 + 0.15.0 beta @@ -1076,6 +1078,22 @@ Update to wrap gRPC C Core version 0.10.0 BSD - Updated functions with TSRM macros for ZTS support #6607 +- Load default roots.pem via grpc_set_ssl_roots_override_callback #6848 + + + + + 0.15.1 + 0.15.1 + + + beta + beta + + 2016-06-30 + BSD + +- Fix shutdown hang problem #4017 diff --git a/templates/package.xml.template b/templates/package.xml.template index a620a2d6a50..d8155cdd823 100644 --- a/templates/package.xml.template +++ b/templates/package.xml.template @@ -12,7 +12,7 @@ grpc-packages@google.com yes - 2016-05-19 + 2016-06-30 ${settings.php_version.php()} @@ -24,7 +24,7 @@ BSD - - TBD + - Fix shutdown hang problem #4017 @@ -153,6 +153,7 @@ BSD - Simplify gRPC PHP installation #4517 + - Wrap gRPC core library version 0.13 @@ -182,13 +183,14 @@ 2016-04-19 BSD + - wrap grpc C core version 0.14.0 - destroy grpc_byte_buffer after startBatch #6096 - 0.14.2 - 0.14.2 + 0.15.0 + 0.15.0 beta @@ -198,6 +200,22 @@ BSD - Updated functions with TSRM macros for ZTS support #6607 + - Load default roots.pem via grpc_set_ssl_roots_override_callback #6848 + + + + + 0.15.1 + 0.15.1 + + + beta + beta + + 2016-06-30 + BSD + + - Fix shutdown hang problem #4017 From 5c88bc660cb1d5b3569ed85ec9eed30d10257470 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Thu, 30 Jun 2016 11:54:50 -0700 Subject: [PATCH 0351/1039] Updated example codegen to use grpcio-tools --- examples/python/helloworld/run_codegen.sh | 2 +- examples/python/route_guide/run_codegen.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/python/helloworld/run_codegen.sh b/examples/python/helloworld/run_codegen.sh index 42b58e50217..34224e5c418 100755 --- a/examples/python/helloworld/run_codegen.sh +++ b/examples/python/helloworld/run_codegen.sh @@ -29,4 +29,4 @@ # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # Runs the protoc with gRPC plugin to generate protocol messages and gRPC stubs. -protoc -I ../../protos --python_out=. --grpc_out=. --plugin=protoc-gen-grpc=`which grpc_python_plugin` ../../protos/helloworld.proto +python -m grpc.tools.protoc -I../../protos --python_out=. --grpc_python_out=. ../../protos/helloworld.proto diff --git a/examples/python/route_guide/run_codegen.sh b/examples/python/route_guide/run_codegen.sh index d9d56c2d7ab..a377a1ab409 100755 --- a/examples/python/route_guide/run_codegen.sh +++ b/examples/python/route_guide/run_codegen.sh @@ -29,4 +29,4 @@ # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # Runs the protoc with gRPC plugin to generate protocol messages and gRPC stubs. -protoc -I ../../protos --python_out=. --grpc_out=. --plugin=protoc-gen-grpc=`which grpc_python_plugin` ../../protos/route_guide.proto +python -m grpc.tools.protoc -I../../protos --python_out=. --grpc_python_out=. ../../protos/route_guide.proto From 9acc40f9b6753a0d43434b30393c4996c8002b28 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Thu, 30 Jun 2016 13:54:09 -0700 Subject: [PATCH 0352/1039] Simplified Ruby completion queue destruction code --- src/ruby/ext/grpc/rb_channel.c | 1 + src/ruby/ext/grpc/rb_completion_queue.c | 57 +---------------------- src/ruby/ext/grpc/rb_server.c | 1 + src/ruby/ext/grpc/rb_server_credentials.c | 1 + 4 files changed, 5 insertions(+), 55 deletions(-) diff --git a/src/ruby/ext/grpc/rb_channel.c b/src/ruby/ext/grpc/rb_channel.c index 346b0d12381..18a15d01252 100644 --- a/src/ruby/ext/grpc/rb_channel.c +++ b/src/ruby/ext/grpc/rb_channel.c @@ -39,6 +39,7 @@ #include #include #include +#include #include "rb_grpc.h" #include "rb_call.h" #include "rb_channel_args.h" diff --git a/src/ruby/ext/grpc/rb_completion_queue.c b/src/ruby/ext/grpc/rb_completion_queue.c index 1a69a175c4c..87ff9083ff8 100644 --- a/src/ruby/ext/grpc/rb_completion_queue.c +++ b/src/ruby/ext/grpc/rb_completion_queue.c @@ -40,6 +40,7 @@ #include #include +#include #include "rb_grpc.h" /* Used to allow grpc_completion_queue_next call to release the GIL */ @@ -51,23 +52,6 @@ typedef struct next_call_stack { volatile int interrupted; } next_call_stack; -/* Calls grpc_completion_queue_next without holding the ruby GIL */ -static void *grpc_rb_completion_queue_next_no_gil(void *param) { - next_call_stack *const next_call = (next_call_stack*)param; - gpr_timespec increment = gpr_time_from_millis(20, GPR_TIMESPAN); - gpr_timespec deadline; - do { - deadline = gpr_time_add(gpr_now(GPR_CLOCK_REALTIME), increment); - next_call->event = grpc_completion_queue_next(next_call->cq, - deadline, NULL); - if (next_call->event.type != GRPC_QUEUE_TIMEOUT || - gpr_time_cmp(deadline, next_call->timeout) > 0) { - break; - } - } while (!next_call->interrupted); - return NULL; -} - /* Calls grpc_completion_queue_pluck without holding the ruby GIL */ static void *grpc_rb_completion_queue_pluck_no_gil(void *param) { next_call_stack *const next_call = (next_call_stack*)param; @@ -86,46 +70,9 @@ static void *grpc_rb_completion_queue_pluck_no_gil(void *param) { return NULL; } -/* Shuts down and drains the completion queue if necessary. - * - * This is done when the ruby completion queue object is about to be GCed. - */ -static void grpc_rb_completion_queue_shutdown_drain(grpc_completion_queue *cq) { - next_call_stack next_call; - grpc_completion_type type; - int drained = 0; - MEMZERO(&next_call, next_call_stack, 1); - - grpc_completion_queue_shutdown(cq); - next_call.cq = cq; - next_call.event.type = GRPC_QUEUE_TIMEOUT; - /* TODO: the timeout should be a module level constant that defaults - * to gpr_inf_future(GPR_CLOCK_REALTIME). - * - * - at the moment this does not work, it stalls. Using a small timeout like - * this one works, and leads to fast test run times; a longer timeout was - * causing unnecessary delays in the test runs. - * - * - investigate further, this is probably another example of C-level cleanup - * not working consistently in all cases. - */ - next_call.timeout = gpr_time_add(gpr_now(GPR_CLOCK_REALTIME), - gpr_time_from_micros(5e3, GPR_TIMESPAN)); - do { - rb_thread_call_without_gvl(grpc_rb_completion_queue_next_no_gil, - (void *)&next_call, NULL, NULL); - type = next_call.event.type; - if (type == GRPC_QUEUE_TIMEOUT) break; - if (type != GRPC_QUEUE_SHUTDOWN) { - ++drained; - rb_warning("completion queue shutdown: %d undrained events", drained); - } - } while (type != GRPC_QUEUE_SHUTDOWN); -} - /* Helper function to free a completion queue. */ void grpc_rb_completion_queue_destroy(grpc_completion_queue *cq) { - grpc_rb_completion_queue_shutdown_drain(cq); + grpc_completion_queue_shutdown(cq); grpc_completion_queue_destroy(cq); } diff --git a/src/ruby/ext/grpc/rb_server.c b/src/ruby/ext/grpc/rb_server.c index a7d53350920..bf26841fd22 100644 --- a/src/ruby/ext/grpc/rb_server.c +++ b/src/ruby/ext/grpc/rb_server.c @@ -38,6 +38,7 @@ #include #include +#include #include "rb_call.h" #include "rb_channel_args.h" #include "rb_completion_queue.h" diff --git a/src/ruby/ext/grpc/rb_server_credentials.c b/src/ruby/ext/grpc/rb_server_credentials.c index cb7545b3645..a44ce715ae8 100644 --- a/src/ruby/ext/grpc/rb_server_credentials.c +++ b/src/ruby/ext/grpc/rb_server_credentials.c @@ -38,6 +38,7 @@ #include #include +#include #include "rb_grpc.h" From 5756c6862bc4e65102ff5409d46e85fbdb09ccea Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Thu, 30 Jun 2016 13:58:18 -0700 Subject: [PATCH 0353/1039] Added a comment about Ruby queue destruction --- src/ruby/ext/grpc/rb_completion_queue.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/ruby/ext/grpc/rb_completion_queue.c b/src/ruby/ext/grpc/rb_completion_queue.c index 87ff9083ff8..fd75d2f691f 100644 --- a/src/ruby/ext/grpc/rb_completion_queue.c +++ b/src/ruby/ext/grpc/rb_completion_queue.c @@ -72,6 +72,10 @@ static void *grpc_rb_completion_queue_pluck_no_gil(void *param) { /* Helper function to free a completion queue. */ void grpc_rb_completion_queue_destroy(grpc_completion_queue *cq) { + /* Every function that adds an event to a queue also synchronously plucks + that event from the queue, and holds a reference to the Ruby object that + holds the queue, so we only get to this point if all of those functions + have completed, and the queue is empty */ grpc_completion_queue_shutdown(cq); grpc_completion_queue_destroy(cq); } From 70bd4839bc93feab530494d7a75c821b540be906 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 30 Jun 2016 14:20:46 -0700 Subject: [PATCH 0354/1039] Allow returning a workqueue somehow associated with an endpoint --- src/core/lib/iomgr/endpoint.c | 4 ++++ src/core/lib/iomgr/endpoint.h | 4 ++++ src/core/lib/iomgr/ev_epoll_linux.c | 3 +++ src/core/lib/iomgr/ev_poll_and_epoll_posix.c | 3 +++ src/core/lib/iomgr/ev_poll_posix.c | 3 +++ src/core/lib/iomgr/ev_posix.c | 4 ++++ src/core/lib/iomgr/ev_posix.h | 4 ++++ src/core/lib/iomgr/tcp_posix.c | 16 +++++++++++++--- src/core/lib/iomgr/workqueue.h | 4 ++-- .../lib/security/transport/secure_endpoint.c | 18 +++++++++++++----- test/core/internal_api_canaries/iomgr.c | 13 ++++++++----- test/core/util/mock_endpoint.c | 12 ++++++++++-- test/core/util/passthru_endpoint.c | 12 ++++++++++-- 13 files changed, 81 insertions(+), 19 deletions(-) diff --git a/src/core/lib/iomgr/endpoint.c b/src/core/lib/iomgr/endpoint.c index 1ab3733d381..f901fcf9622 100644 --- a/src/core/lib/iomgr/endpoint.c +++ b/src/core/lib/iomgr/endpoint.c @@ -65,3 +65,7 @@ void grpc_endpoint_destroy(grpc_exec_ctx* exec_ctx, grpc_endpoint* ep) { char* grpc_endpoint_get_peer(grpc_endpoint* ep) { return ep->vtable->get_peer(ep); } + +grpc_workqueue* grpc_endpoint_get_workqueue(grpc_endpoint* ep) { + return ep->vtable->get_workqueue(ep); +} diff --git a/src/core/lib/iomgr/endpoint.h b/src/core/lib/iomgr/endpoint.h index f9808bbda18..894efc0b238 100644 --- a/src/core/lib/iomgr/endpoint.h +++ b/src/core/lib/iomgr/endpoint.h @@ -51,6 +51,7 @@ struct grpc_endpoint_vtable { gpr_slice_buffer *slices, grpc_closure *cb); void (*write)(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep, gpr_slice_buffer *slices, grpc_closure *cb); + grpc_workqueue *(*get_workqueue)(grpc_endpoint *ep); void (*add_to_pollset)(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep, grpc_pollset *pollset); void (*add_to_pollset_set)(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep, @@ -69,6 +70,9 @@ void grpc_endpoint_read(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep, char *grpc_endpoint_get_peer(grpc_endpoint *ep); +/* Retrieve a reference to the workqueue associated with this endpoint */ +grpc_workqueue *grpc_endpoint_get_workqueue(grpc_endpoint *ep); + /* Write slices out to the socket. If the connection is ready for more data after the end of the call, it diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index cf0fe736a0b..116de76e6a0 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -1037,6 +1037,8 @@ static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, gpr_mu_unlock(&fd->mu); } +static grpc_workqueue *fd_get_workqueue(grpc_fd *fd) { return NULL; } + /******************************************************************************* * Pollset Definitions */ @@ -1794,6 +1796,7 @@ static const grpc_event_engine_vtable vtable = { .fd_notify_on_read = fd_notify_on_read, .fd_notify_on_write = fd_notify_on_write, .fd_get_read_notifier_pollset = fd_get_read_notifier_pollset, + .fd_get_workqueue = fd_get_workqueue, .pollset_init = pollset_init, .pollset_shutdown = pollset_shutdown, diff --git a/src/core/lib/iomgr/ev_poll_and_epoll_posix.c b/src/core/lib/iomgr/ev_poll_and_epoll_posix.c index 9e306af5fac..c2107e5e393 100644 --- a/src/core/lib/iomgr/ev_poll_and_epoll_posix.c +++ b/src/core/lib/iomgr/ev_poll_and_epoll_posix.c @@ -725,6 +725,8 @@ static void fd_end_poll(grpc_exec_ctx *exec_ctx, grpc_fd_watcher *watcher, GRPC_FD_UNREF(fd, "poll"); } +static grpc_workqueue *fd_get_workqueue(grpc_fd *fd) { return NULL; } + /******************************************************************************* * pollset_posix.c */ @@ -2006,6 +2008,7 @@ static const grpc_event_engine_vtable vtable = { .fd_notify_on_read = fd_notify_on_read, .fd_notify_on_write = fd_notify_on_write, .fd_get_read_notifier_pollset = fd_get_read_notifier_pollset, + .fd_get_workqueue = fd_get_workqueue, .pollset_init = pollset_init, .pollset_shutdown = pollset_shutdown, diff --git a/src/core/lib/iomgr/ev_poll_posix.c b/src/core/lib/iomgr/ev_poll_posix.c index 45c0a5e9546..4b593f4b2c2 100644 --- a/src/core/lib/iomgr/ev_poll_posix.c +++ b/src/core/lib/iomgr/ev_poll_posix.c @@ -617,6 +617,8 @@ static void fd_end_poll(grpc_exec_ctx *exec_ctx, grpc_fd_watcher *watcher, GRPC_FD_UNREF(fd, "poll"); } +static grpc_workqueue *fd_get_workqueue(grpc_fd *fd) { return NULL; } + /******************************************************************************* * pollset_posix.c */ @@ -1234,6 +1236,7 @@ static const grpc_event_engine_vtable vtable = { .fd_notify_on_read = fd_notify_on_read, .fd_notify_on_write = fd_notify_on_write, .fd_get_read_notifier_pollset = fd_get_read_notifier_pollset, + .fd_get_workqueue = fd_get_workqueue, .pollset_init = pollset_init, .pollset_shutdown = pollset_shutdown, diff --git a/src/core/lib/iomgr/ev_posix.c b/src/core/lib/iomgr/ev_posix.c index a3c1e9db9a0..65366726859 100644 --- a/src/core/lib/iomgr/ev_posix.c +++ b/src/core/lib/iomgr/ev_posix.c @@ -148,6 +148,10 @@ grpc_fd *grpc_fd_create(int fd, const char *name) { return g_event_engine->fd_create(fd, name); } +grpc_workqueue *grpc_fd_get_workqueue(grpc_fd *fd) { + return g_event_engine->fd_get_workqueue(fd); +} + int grpc_fd_wrapped_fd(grpc_fd *fd) { return g_event_engine->fd_wrapped_fd(fd); } diff --git a/src/core/lib/iomgr/ev_posix.h b/src/core/lib/iomgr/ev_posix.h index 579c84ef707..c2aa1756ea2 100644 --- a/src/core/lib/iomgr/ev_posix.h +++ b/src/core/lib/iomgr/ev_posix.h @@ -56,6 +56,7 @@ typedef struct grpc_event_engine_vtable { void (*fd_notify_on_write)(grpc_exec_ctx *exec_ctx, grpc_fd *fd, grpc_closure *closure); bool (*fd_is_shutdown)(grpc_fd *fd); + grpc_workqueue *(*fd_get_workqueue)(grpc_fd *fd); grpc_pollset *(*fd_get_read_notifier_pollset)(grpc_exec_ctx *exec_ctx, grpc_fd *fd); @@ -107,6 +108,9 @@ const char *grpc_get_poll_strategy_name(); This takes ownership of closing fd. */ grpc_fd *grpc_fd_create(int fd, const char *name); +/* Get a workqueue that's associated with this fd */ +grpc_workqueue *grpc_fd_get_workqueue(grpc_fd *fd); + /* Return the wrapped fd, or -1 if it has been released or closed. */ int grpc_fd_wrapped_fd(grpc_fd *fd); diff --git a/src/core/lib/iomgr/tcp_posix.c b/src/core/lib/iomgr/tcp_posix.c index 2ab45e33ce3..b5bac152fb6 100644 --- a/src/core/lib/iomgr/tcp_posix.c +++ b/src/core/lib/iomgr/tcp_posix.c @@ -450,9 +450,19 @@ static char *tcp_get_peer(grpc_endpoint *ep) { return gpr_strdup(tcp->peer_string); } -static const grpc_endpoint_vtable vtable = { - tcp_read, tcp_write, tcp_add_to_pollset, tcp_add_to_pollset_set, - tcp_shutdown, tcp_destroy, tcp_get_peer}; +static grpc_workqueue *tcp_get_workqueue(grpc_endpoint *ep) { + grpc_tcp *tcp = (grpc_tcp *)ep; + return grpc_fd_get_workqueue(tcp->em_fd); +} + +static const grpc_endpoint_vtable vtable = {tcp_read, + tcp_write, + tcp_get_workqueue, + tcp_add_to_pollset, + tcp_add_to_pollset_set, + tcp_shutdown, + tcp_destroy, + tcp_get_peer}; grpc_endpoint *grpc_tcp_create(grpc_fd *em_fd, size_t slice_size, const char *peer_string) { diff --git a/src/core/lib/iomgr/workqueue.h b/src/core/lib/iomgr/workqueue.h index 5cc40eea505..498b7a300a5 100644 --- a/src/core/lib/iomgr/workqueue.h +++ b/src/core/lib/iomgr/workqueue.h @@ -58,7 +58,7 @@ void grpc_workqueue_flush(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue); #define GRPC_WORKQUEUE_REFCOUNT_DEBUG #ifdef GRPC_WORKQUEUE_REFCOUNT_DEBUG #define GRPC_WORKQUEUE_REF(p, r) \ - grpc_workqueue_ref((p), __FILE__, __LINE__, (r)) + (grpc_workqueue_ref((p), __FILE__, __LINE__, (r)), (p)) #define GRPC_WORKQUEUE_UNREF(cl, p, r) \ grpc_workqueue_unref((cl), (p), __FILE__, __LINE__, (r)) void grpc_workqueue_ref(grpc_workqueue *workqueue, const char *file, int line, @@ -66,7 +66,7 @@ void grpc_workqueue_ref(grpc_workqueue *workqueue, const char *file, int line, void grpc_workqueue_unref(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, const char *file, int line, const char *reason); #else -#define GRPC_WORKQUEUE_REF(p, r) grpc_workqueue_ref((p)) +#define GRPC_WORKQUEUE_REF(p, r) (grpc_workqueue_ref((p)), (p)) #define GRPC_WORKQUEUE_UNREF(cl, p, r) grpc_workqueue_unref((cl), (p)) void grpc_workqueue_ref(grpc_workqueue *workqueue); void grpc_workqueue_unref(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue); diff --git a/src/core/lib/security/transport/secure_endpoint.c b/src/core/lib/security/transport/secure_endpoint.c index 7650d68e892..bc50f9d1b00 100644 --- a/src/core/lib/security/transport/secure_endpoint.c +++ b/src/core/lib/security/transport/secure_endpoint.c @@ -360,11 +360,19 @@ static char *endpoint_get_peer(grpc_endpoint *secure_ep) { return grpc_endpoint_get_peer(ep->wrapped_ep); } -static const grpc_endpoint_vtable vtable = { - endpoint_read, endpoint_write, - endpoint_add_to_pollset, endpoint_add_to_pollset_set, - endpoint_shutdown, endpoint_destroy, - endpoint_get_peer}; +static grpc_workqueue *endpoint_get_workqueue(grpc_endpoint *secure_ep) { + secure_endpoint *ep = (secure_endpoint *)secure_ep; + return grpc_endpoint_get_workqueue(ep->wrapped_ep); +} + +static const grpc_endpoint_vtable vtable = {endpoint_read, + endpoint_write, + endpoint_get_workqueue, + endpoint_add_to_pollset, + endpoint_add_to_pollset_set, + endpoint_shutdown, + endpoint_destroy, + endpoint_get_peer}; grpc_endpoint *grpc_secure_endpoint_create( struct tsi_frame_protector *protector, grpc_endpoint *transport, diff --git a/test/core/internal_api_canaries/iomgr.c b/test/core/internal_api_canaries/iomgr.c index 5e86c423095..27d630623ed 100644 --- a/test/core/internal_api_canaries/iomgr.c +++ b/test/core/internal_api_canaries/iomgr.c @@ -77,11 +77,14 @@ static void test_code(void) { /* endpoint.h */ grpc_endpoint endpoint; - grpc_endpoint_vtable vtable = { - grpc_endpoint_read, grpc_endpoint_write, - grpc_endpoint_add_to_pollset, grpc_endpoint_add_to_pollset_set, - grpc_endpoint_shutdown, grpc_endpoint_destroy, - grpc_endpoint_get_peer}; + grpc_endpoint_vtable vtable = {grpc_endpoint_read, + grpc_endpoint_write, + grpc_endpoint_get_workqueue, + grpc_endpoint_add_to_pollset, + grpc_endpoint_add_to_pollset_set, + grpc_endpoint_shutdown, + grpc_endpoint_destroy, + grpc_endpoint_get_peer}; endpoint.vtable = &vtable; grpc_endpoint_read(&exec_ctx, &endpoint, NULL, NULL); diff --git a/test/core/util/mock_endpoint.c b/test/core/util/mock_endpoint.c index ed9545e9df2..13e0e918fbe 100644 --- a/test/core/util/mock_endpoint.c +++ b/test/core/util/mock_endpoint.c @@ -95,9 +95,17 @@ static char *me_get_peer(grpc_endpoint *ep) { return gpr_strdup("fake:mock_endpoint"); } +static grpc_workqueue *me_get_workqueue(grpc_endpoint *ep) { return NULL; } + static const grpc_endpoint_vtable vtable = { - me_read, me_write, me_add_to_pollset, me_add_to_pollset_set, - me_shutdown, me_destroy, me_get_peer, + me_read, + me_write, + me_get_workqueue, + me_add_to_pollset, + me_add_to_pollset_set, + me_shutdown, + me_destroy, + me_get_peer, }; grpc_endpoint *grpc_mock_endpoint_create(void (*on_write)(gpr_slice slice)) { diff --git a/test/core/util/passthru_endpoint.c b/test/core/util/passthru_endpoint.c index a39f3dd66e2..7ed9e97bd6a 100644 --- a/test/core/util/passthru_endpoint.c +++ b/test/core/util/passthru_endpoint.c @@ -140,9 +140,17 @@ static char *me_get_peer(grpc_endpoint *ep) { return gpr_strdup("fake:mock_endpoint"); } +static grpc_workqueue *me_get_workqueue(grpc_endpoint *ep) { return NULL; } + static const grpc_endpoint_vtable vtable = { - me_read, me_write, me_add_to_pollset, me_add_to_pollset_set, - me_shutdown, me_destroy, me_get_peer, + me_read, + me_write, + me_get_workqueue, + me_add_to_pollset, + me_add_to_pollset_set, + me_shutdown, + me_destroy, + me_get_peer, }; static void half_init(half *m, passthru_endpoint *parent) { From efe7a6e3380c9ddaa7d70b516fda281a441a7e7f Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 30 Jun 2016 14:34:04 -0700 Subject: [PATCH 0355/1039] Use endpoint workqueue via execution context --- .../transport/chttp2/transport/chttp2_transport.c | 12 ++---------- src/core/ext/transport/chttp2/transport/internal.h | 1 - 2 files changed, 2 insertions(+), 11 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index 197cee7e0ce..92722b50f4f 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -204,8 +204,6 @@ static void destruct_transport(grpc_exec_ctx *exec_ctx, gpr_free(ping); } - GRPC_WORKQUEUE_UNREF(exec_ctx, t->executor.workqueue, "transport"); - gpr_free(t->peer_string); gpr_free(t); } @@ -257,9 +255,6 @@ static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, /* ref is dropped at transport close() */ gpr_ref_init(&t->shutdown_ep_refs, 1); gpr_mu_init(&t->executor.mu); - GPR_ASSERT(GRPC_LOG_IF_ERROR( - "workqueue_create", - grpc_workqueue_create(exec_ctx, &t->executor.workqueue))); t->peer_string = grpc_endpoint_get_peer(ep); t->endpoint_reading = 1; t->global.next_stream_id = is_client ? 1 : 2; @@ -715,8 +710,8 @@ static void finish_global_actions(grpc_exec_ctx *exec_ctx, set_write_state(t, GRPC_CHTTP2_WRITE_SCHEDULED, "unlocking"); REF_TRANSPORT(t, "initiate_writing"); gpr_mu_unlock(&t->executor.mu); - grpc_workqueue_enqueue(exec_ctx, t->executor.workqueue, - &t->initiate_writing, GRPC_ERROR_NONE); + grpc_exec_ctx_sched(exec_ctx, &t->initiate_writing, GRPC_ERROR_NONE, + grpc_endpoint_get_workqueue(t->ep)); break; case GRPC_CHTTP2_WRITE_REQUESTED_NO_POLLER: start_writing(exec_ctx, t); @@ -2082,7 +2077,6 @@ static void add_to_pollset_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_stream *s_unused, void *pollset) { if (t->ep) { grpc_endpoint_add_to_pollset(exec_ctx, t->ep, pollset); - grpc_workqueue_add_to_pollset(exec_ctx, t->executor.workqueue, pollset); } } @@ -2092,8 +2086,6 @@ static void add_to_pollset_set_locked(grpc_exec_ctx *exec_ctx, void *pollset_set) { if (t->ep) { grpc_endpoint_add_to_pollset_set(exec_ctx, t->ep, pollset_set); - grpc_workqueue_add_to_pollset_set(exec_ctx, t->executor.workqueue, - pollset_set); } } diff --git a/src/core/ext/transport/chttp2/transport/internal.h b/src/core/ext/transport/chttp2/transport/internal.h index 2ef1c9df7a6..4e0c31c1110 100644 --- a/src/core/ext/transport/chttp2/transport/internal.h +++ b/src/core/ext/transport/chttp2/transport/internal.h @@ -332,7 +332,6 @@ struct grpc_chttp2_transport { struct { gpr_mu mu; - grpc_workqueue *workqueue; /** is a thread currently in the global lock */ bool global_active; From 5e4fb0eedeb0a92f00ec10942ab65408550138bf Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 30 Jun 2016 14:53:51 -0700 Subject: [PATCH 0356/1039] Windows implementation of get_workqueue --- src/core/lib/iomgr/tcp_windows.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/src/core/lib/iomgr/tcp_windows.c b/src/core/lib/iomgr/tcp_windows.c index 37ab59021e3..35054c42b55 100644 --- a/src/core/lib/iomgr/tcp_windows.c +++ b/src/core/lib/iomgr/tcp_windows.c @@ -389,9 +389,16 @@ static char *win_get_peer(grpc_endpoint *ep) { return gpr_strdup(tcp->peer_string); } -static grpc_endpoint_vtable vtable = { - win_read, win_write, win_add_to_pollset, win_add_to_pollset_set, - win_shutdown, win_destroy, win_get_peer}; +static grpc_workqueue *win_get_workqueue(grpc_endpoint *ep) { return NULL; } + +static grpc_endpoint_vtable vtable = {win_read, + win_write, + win_get_workqueue, + win_add_to_pollset, + win_add_to_pollset_set, + win_shutdown, + win_destroy, + win_get_peer}; grpc_endpoint *grpc_tcp_create(grpc_winsocket *socket, char *peer_string) { grpc_tcp *tcp = (grpc_tcp *)gpr_malloc(sizeof(grpc_tcp)); From 7be0595600013003fd856c057a039655e7da7fc6 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 30 Jun 2016 15:04:32 -0700 Subject: [PATCH 0357/1039] Implement minimal exec_ctx offloading --- src/core/lib/iomgr/exec_ctx.c | 9 ++++++--- src/core/lib/iomgr/exec_ctx.h | 6 +++++- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/src/core/lib/iomgr/exec_ctx.c b/src/core/lib/iomgr/exec_ctx.c index c44aafcddf0..059133f8796 100644 --- a/src/core/lib/iomgr/exec_ctx.c +++ b/src/core/lib/iomgr/exec_ctx.c @@ -85,14 +85,17 @@ void grpc_exec_ctx_finish(grpc_exec_ctx *exec_ctx) { void grpc_exec_ctx_sched(grpc_exec_ctx *exec_ctx, grpc_closure *closure, grpc_error *error, grpc_workqueue *offload_target_or_null) { - GPR_ASSERT(offload_target_or_null == NULL); - grpc_closure_list_append(&exec_ctx->closure_list, closure, error); + if (offload_target_or_null == NULL) { + grpc_closure_list_append(&exec_ctx->closure_list, closure, error); + } else { + grpc_workqueue_enqueue(exec_ctx, offload_target_or_null, closure, error); + GRPC_WORKQUEUE_UNREF(exec_ctx, offload_target_or_null, "exec_ctx_sched"); + } } void grpc_exec_ctx_enqueue_list(grpc_exec_ctx *exec_ctx, grpc_closure_list *list, grpc_workqueue *offload_target_or_null) { - GPR_ASSERT(offload_target_or_null == NULL); grpc_closure_list_move(list, &exec_ctx->closure_list); } diff --git a/src/core/lib/iomgr/exec_ctx.h b/src/core/lib/iomgr/exec_ctx.h index 38f27d9b136..917f332f03d 100644 --- a/src/core/lib/iomgr/exec_ctx.h +++ b/src/core/lib/iomgr/exec_ctx.h @@ -93,7 +93,11 @@ bool grpc_exec_ctx_flush(grpc_exec_ctx *exec_ctx); /** Finish any pending work for a grpc_exec_ctx. Must be called before * the instance is destroyed, or work may be lost. */ void grpc_exec_ctx_finish(grpc_exec_ctx *exec_ctx); -/** Add a closure to be executed at the next flush/finish point */ +/** Add a closure to be executed in the future. + If \a offload_target_or_null is NULL, the closure will be executed at the + next exec_ctx.{finish,flush} point. + If \a offload_target_or_null is non-NULL, the closure will be scheduled + against the workqueue, and a reference to the workqueue will be consumed. */ void grpc_exec_ctx_sched(grpc_exec_ctx *exec_ctx, grpc_closure *closure, grpc_error *error, grpc_workqueue *offload_target_or_null); From 099eb6ee4bd9c2a9484d5a794ccf6e97d353ad2e Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 30 Jun 2016 15:10:12 -0700 Subject: [PATCH 0358/1039] Get it compiling --- src/core/lib/iomgr/exec_ctx.c | 1 + 1 file changed, 1 insertion(+) diff --git a/src/core/lib/iomgr/exec_ctx.c b/src/core/lib/iomgr/exec_ctx.c index 059133f8796..ac7785ec135 100644 --- a/src/core/lib/iomgr/exec_ctx.c +++ b/src/core/lib/iomgr/exec_ctx.c @@ -37,6 +37,7 @@ #include #include +#include "src/core/lib/iomgr/workqueue.h" #include "src/core/lib/profiling/timers.h" bool grpc_exec_ctx_ready_to_finish(grpc_exec_ctx *exec_ctx) { From b39307d2bcf859b9e7dfa485fae263f6a73e35b6 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 30 Jun 2016 15:39:13 -0700 Subject: [PATCH 0359/1039] Add workqueues to polling_islands Also: - remove pooling of polling_islands (it's unnecessary) - add some 'statics' to function calls --- src/core/lib/iomgr/ev_epoll_linux.c | 162 +++++++++++++--------------- 1 file changed, 75 insertions(+), 87 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 116de76e6a0..ac6a2157256 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -57,6 +57,7 @@ #include "src/core/lib/iomgr/ev_posix.h" #include "src/core/lib/iomgr/iomgr_internal.h" #include "src/core/lib/iomgr/wakeup_fd_posix.h" +#include "src/core/lib/iomgr/workqueue.h" #include "src/core/lib/profiling/timers.h" #include "src/core/lib/support/block_annotate.h" @@ -156,12 +157,13 @@ static void fd_global_shutdown(void); #ifdef GRPC_PI_REF_COUNT_DEBUG #define PI_ADD_REF(p, r) pi_add_ref_dbg((p), (r), __FILE__, __LINE__) -#define PI_UNREF(p, r) pi_unref_dbg((p), (r), __FILE__, __LINE__) +#define PI_UNREF(exec_ctx, p, r) \ + pi_unref_dbg((exec_ctx), (p), (r), __FILE__, __LINE__) #else /* defined(GRPC_PI_REF_COUNT_DEBUG) */ #define PI_ADD_REF(p, r) pi_add_ref((p)) -#define PI_UNREF(p, r) pi_unref((p)) +#define PI_UNREF(exec_ctx, p, r) pi_unref((exec_ctx), (p)) #endif /* !defined(GPRC_PI_REF_COUNT_DEBUG) */ @@ -184,6 +186,9 @@ typedef struct polling_island { * (except mu and ref_count) are invalid and must be ignored. */ gpr_atm merged_to; + /* The workqueue associated with this polling island */ + grpc_workqueue *workqueue; + /* The fd of the underlying epoll set */ int epoll_fd; @@ -191,11 +196,6 @@ typedef struct polling_island { size_t fd_cnt; size_t fd_capacity; grpc_fd **fds; - - /* Polling islands that are no longer needed are kept in a freelist so that - they can be reused. This field points to the next polling island in the - free list */ - struct polling_island *next_free; } polling_island; /******************************************************************************* @@ -275,11 +275,8 @@ static void append_error(grpc_error **composite, grpc_error *error, threads that woke up MUST NOT call grpc_wakeup_fd_consume_wakeup() */ static grpc_wakeup_fd polling_island_wakeup_fd; -/* Polling island freelist */ -static gpr_mu g_pi_freelist_mu; -static polling_island *g_pi_freelist = NULL; - -static void polling_island_delete(); /* Forward declaration */ +/* Forward declaration */ +static void polling_island_delete(grpc_exec_ctx *exec_ctx); #ifdef GRPC_TSAN /* Currently TSAN may incorrectly flag data races between epoll_ctl and @@ -293,27 +290,29 @@ gpr_atm g_epoll_sync; #endif /* defined(GRPC_TSAN) */ #ifdef GRPC_PI_REF_COUNT_DEBUG -void pi_add_ref(polling_island *pi); -void pi_unref(polling_island *pi); +static void pi_add_ref(polling_island *pi); +static void pi_unref(grpc_exec_ctx *exec_ctx, polling_island *pi); -void pi_add_ref_dbg(polling_island *pi, char *reason, char *file, int line) { +static void pi_add_ref_dbg(polling_island *pi, char *reason, char *file, + int line) { long old_cnt = gpr_atm_acq_load(&(pi->ref_count.count)); pi_add_ref(pi); gpr_log(GPR_DEBUG, "Add ref pi: %p, old: %ld -> new:%ld (%s) - (%s, %d)", (void *)pi, old_cnt, old_cnt + 1, reason, file, line); } -void pi_unref_dbg(polling_island *pi, char *reason, char *file, int line) { +static void pi_unref_dbg(grpc_exec_ctx *exec_ctx, polling_island *pi, + char *reason, char *file, int line) { long old_cnt = gpr_atm_acq_load(&(pi->ref_count.count)); - pi_unref(pi); + pi_unref(exec_ctx, pi); gpr_log(GPR_DEBUG, "Unref pi: %p, old:%ld -> new:%ld (%s) - (%s, %d)", (void *)pi, old_cnt, (old_cnt - 1), reason, file, line); } #endif -void pi_add_ref(polling_island *pi) { gpr_ref(&pi->ref_count); } +static void pi_add_ref(polling_island *pi) { gpr_ref(&pi->ref_count); } -void pi_unref(polling_island *pi) { +static void pi_unref(grpc_exec_ctx *exec_ctx, polling_island *pi) { /* If ref count went to zero, delete the polling island. Note that this deletion not be done under a lock. Once the ref count goes to zero, we are guaranteed that no one else holds a reference to the @@ -324,7 +323,7 @@ void pi_unref(polling_island *pi) { */ if (gpr_unref(&pi->ref_count)) { polling_island *next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); - polling_island_delete(pi); + polling_island_delete(exec_ctx, pi); if (next != NULL) { PI_UNREF(next, "pi_delete"); /* Recursive call */ } @@ -462,29 +461,22 @@ static void polling_island_remove_fd_locked(polling_island *pi, grpc_fd *fd, } /* Might return NULL in case of an error */ -static polling_island *polling_island_create(grpc_fd *initial_fd, +static polling_island *polling_island_create(grpc_exec_ctx *exec_ctx, + grpc_fd *initial_fd, grpc_error **error) { polling_island *pi = NULL; char *err_msg; const char *err_desc = "polling_island_create"; - /* Try to get one from the polling island freelist */ - gpr_mu_lock(&g_pi_freelist_mu); - if (g_pi_freelist != NULL) { - pi = g_pi_freelist; - g_pi_freelist = g_pi_freelist->next_free; - pi->next_free = NULL; - } - gpr_mu_unlock(&g_pi_freelist_mu); - - /* Create new polling island if we could not get one from the free list */ - if (pi == NULL) { - pi = gpr_malloc(sizeof(*pi)); - gpr_mu_init(&pi->mu); - pi->fd_cnt = 0; - pi->fd_capacity = 0; - pi->fds = NULL; - } + *error = GRPC_ERROR_NONE; + + pi = gpr_malloc(sizeof(*pi)); + gpr_mu_init(&pi->mu); + pi->fd_cnt = 0; + pi->fd_capacity = 0; + pi->fds = NULL; + pi->epoll_fd = -1; + pi->workqueue = NULL; gpr_ref_init(&pi->ref_count, 0); gpr_atm_rel_store(&pi->merged_to, (gpr_atm)NULL); @@ -492,39 +484,49 @@ static polling_island *polling_island_create(grpc_fd *initial_fd, pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); if (pi->epoll_fd < 0) { - gpr_asprintf(&err_msg, "epoll_create1 failed with error %d (%s)", errno, - strerror(errno)); - append_error(error, GRPC_OS_ERROR(errno, err_msg), err_desc); - gpr_free(err_msg); - } else { - polling_island_add_wakeup_fd_locked(pi, &grpc_global_wakeup_fd, error); - pi->next_free = NULL; + append_error(error, GRPC_OS_ERROR(errno, "epoll_create1"), err_desc); + goto done; + } - if (initial_fd != NULL) { - /* Lock the polling island here just in case we got this structure from - the freelist and the polling island lock was not released yet (by the - code that adds the polling island to the freelist) */ - gpr_mu_lock(&pi->mu); - polling_island_add_fds_locked(pi, &initial_fd, 1, true, error); - gpr_mu_unlock(&pi->mu); - } + polling_island_add_wakeup_fd_locked(pi, &grpc_global_wakeup_fd, error); + + if (initial_fd != NULL) { + /* Lock the polling island here just in case we got this structure from + the freelist and the polling island lock was not released yet (by the + code that adds the polling island to the freelist) */ + gpr_mu_lock(&pi->mu); + polling_island_add_fds_locked(pi, &initial_fd, 1, true, error); + gpr_mu_unlock(&pi->mu); } + append_error(error, grpc_workqueue_create(exec_ctx, &pi->workqueue), + err_desc); + +done: + if (*error != GRPC_ERROR_NONE) { + if (pi->epoll_fd < 0) { + close(pi->epoll_fd); + } + if (pi->workqueue != NULL) { + GRPC_WORKQUEUE_UNREF(exec_ctx, pi->workqueue, "polling_island"); + } + gpr_mu_destroy(&pi->mu); + gpr_free(pi); + pi = NULL; + } return pi; } -static void polling_island_delete(polling_island *pi) { +static void polling_island_delete(grpc_exec_ctx *exec_ctx, polling_island *pi) { GPR_ASSERT(pi->fd_cnt == 0); gpr_atm_rel_store(&pi->merged_to, (gpr_atm)NULL); close(pi->epoll_fd); - pi->epoll_fd = -1; - - gpr_mu_lock(&g_pi_freelist_mu); - pi->next_free = g_pi_freelist; - g_pi_freelist = pi; - gpr_mu_unlock(&g_pi_freelist_mu); + GRPC_WORKQUEUE_UNREF(exec_ctx, pi->workqueue, "polling_island"); + gpr_mu_destroy(&pi->mu); + gpr_free(pi->fds); + gpr_free(pi); } /* Attempts to gets the last polling island in the linked list (liked by the @@ -704,9 +706,6 @@ static polling_island *polling_island_merge(polling_island *p, static grpc_error *polling_island_global_init() { grpc_error *error = GRPC_ERROR_NONE; - gpr_mu_init(&g_pi_freelist_mu); - g_pi_freelist = NULL; - error = grpc_wakeup_fd_init(&polling_island_wakeup_fd); if (error == GRPC_ERROR_NONE) { error = grpc_wakeup_fd_wakeup(&polling_island_wakeup_fd); @@ -716,18 +715,6 @@ static grpc_error *polling_island_global_init() { } static void polling_island_global_shutdown() { - polling_island *next; - gpr_mu_lock(&g_pi_freelist_mu); - gpr_mu_unlock(&g_pi_freelist_mu); - while (g_pi_freelist != NULL) { - next = g_pi_freelist->next_free; - gpr_mu_destroy(&g_pi_freelist->mu); - gpr_free(g_pi_freelist->fds); - gpr_free(g_pi_freelist); - g_pi_freelist = next; - } - gpr_mu_destroy(&g_pi_freelist_mu); - grpc_wakeup_fd_destroy(&polling_island_wakeup_fd); } @@ -929,7 +916,7 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, polling_island_remove_fd_locked(pi_latest, fd, is_fd_closed, &error); gpr_mu_unlock(&pi_latest->mu); - PI_UNREF(fd->polling_island, "fd_orphan"); + PI_UNREF(exec_ctx, fd->polling_island, "fd_orphan"); fd->polling_island = NULL; } gpr_mu_unlock(&fd->pi_mu); @@ -1229,9 +1216,10 @@ static void fd_become_writable(grpc_exec_ctx *exec_ctx, grpc_fd *fd) { gpr_mu_unlock(&fd->mu); } -static void pollset_release_polling_island(grpc_pollset *ps, char *reason) { +static void pollset_release_polling_island(grpc_exec_ctx *exec_ctx, + grpc_pollset *ps, char *reason) { if (ps->polling_island != NULL) { - PI_UNREF(ps->polling_island, reason); + PI_UNREF(exec_ctx, ps->polling_island, reason); } ps->polling_island = NULL; } @@ -1244,7 +1232,7 @@ static void finish_shutdown_locked(grpc_exec_ctx *exec_ctx, pollset->finish_shutdown_called = true; /* Release the ref and set pollset->polling_island to NULL */ - pollset_release_polling_island(pollset, "ps_shutdown"); + pollset_release_polling_island(exec_ctx, pollset, "ps_shutdown"); grpc_exec_ctx_sched(exec_ctx, pollset->shutdown_done, GRPC_ERROR_NONE, NULL); } @@ -1276,14 +1264,14 @@ static void pollset_destroy(grpc_pollset *pollset) { gpr_mu_destroy(&pollset->mu); } -static void pollset_reset(grpc_pollset *pollset) { +static void pollset_reset(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset) { GPR_ASSERT(pollset->shutting_down); GPR_ASSERT(!pollset_has_workers(pollset)); pollset->shutting_down = false; pollset->finish_shutdown_called = false; pollset->kicked_without_pollers = false; pollset->shutdown_done = NULL; - pollset_release_polling_island(pollset, "ps_reset"); + GPR_ASSERT(pollset->polling_island == NULL); } #define GRPC_EPOLL_MAX_EVENTS 1000 @@ -1311,7 +1299,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, this function (i.e pollset_work_and_unlock()) is called */ if (pollset->polling_island == NULL) { - pollset->polling_island = polling_island_create(NULL, error); + pollset->polling_island = polling_island_create(exec_ctx, NULL, error); if (pollset->polling_island == NULL) { GPR_TIMER_END("pollset_work_and_unlock", 0); return; /* Fatal error. We cannot continue */ @@ -1331,7 +1319,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, /* Always do PI_ADD_REF before PI_UNREF because PI_UNREF may cause the polling island to be deleted */ PI_ADD_REF(pi, "ps"); - PI_UNREF(pollset->polling_island, "ps"); + PI_UNREF(exec_ctx, pollset->polling_island, "ps"); pollset->polling_island = pi; } @@ -1402,7 +1390,7 @@ static void pollset_work_and_unlock(grpc_exec_ctx *exec_ctx, that we got before releasing the polling island lock). This is because pollset->polling_island pointer might get udpated in other parts of the code when there is an island merge while we are doing epoll_wait() above */ - PI_UNREF(pi, "ps_work"); + PI_UNREF(exec_ctx, pi, "ps_work"); GPR_TIMER_END("pollset_work_and_unlock", 0); } @@ -1540,7 +1528,7 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, if (fd->polling_island == pollset->polling_island) { pi_new = fd->polling_island; if (pi_new == NULL) { - pi_new = polling_island_create(fd, &error); + pi_new = polling_island_create(exec_ctx, fd, &error); GRPC_POLLING_TRACE( "pollset_add_fd: Created new polling island. pi_new: %p (fd: %d, " @@ -1581,7 +1569,7 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, if (fd->polling_island != pi_new) { PI_ADD_REF(pi_new, "fd"); if (fd->polling_island != NULL) { - PI_UNREF(fd->polling_island, "fd"); + PI_UNREF(exec_ctx, fd->polling_island, "fd"); } fd->polling_island = pi_new; } @@ -1589,7 +1577,7 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, if (pollset->polling_island != pi_new) { PI_ADD_REF(pi_new, "ps"); if (pollset->polling_island != NULL) { - PI_UNREF(pollset->polling_island, "ps"); + PI_UNREF(exec_ctx, pollset->polling_island, "ps"); } pollset->polling_island = pi_new; } From d6ba6192b048fd3f8be1e6d13dd9d95489ce2065 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 30 Jun 2016 15:42:41 -0700 Subject: [PATCH 0360/1039] Add accessor for fd related workqueue --- src/core/lib/iomgr/ev_epoll_linux.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index ac6a2157256..6dfd363b0b5 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -1024,7 +1024,16 @@ static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, gpr_mu_unlock(&fd->mu); } -static grpc_workqueue *fd_get_workqueue(grpc_fd *fd) { return NULL; } +static grpc_workqueue *fd_get_workqueue(grpc_fd *fd) { + gpr_mu_lock(&fd->pi_mu); + grpc_workqueue *workqueue = NULL; + if (fd->polling_island != NULL) { + workqueue = + GRPC_WORKQUEUE_REF(fd->polling_island->workqueue, "get_workqueue"); + } + gpr_mu_unlock(&fd->pi_mu); + return workqueue; +} /******************************************************************************* * Pollset Definitions From 8a15dd2fae54ebb809402c24ebaa86cb8da36597 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Wed, 29 Jun 2016 11:41:24 -0700 Subject: [PATCH 0361/1039] Use open-source defaults to propagate Python plugin configuration --- src/compiler/python_generator.cc | 3 +++ src/compiler/python_generator.h | 1 + src/compiler/python_plugin.cc | 2 -- tools/distrib/python/grpcio_tools/grpc/tools/main.cc | 1 - 4 files changed, 4 insertions(+), 3 deletions(-) diff --git a/src/compiler/python_generator.cc b/src/compiler/python_generator.cc index 15cda474ccd..c5f1ed80611 100644 --- a/src/compiler/python_generator.cc +++ b/src/compiler/python_generator.cc @@ -66,6 +66,9 @@ using std::vector; namespace grpc_python_generator { +GeneratorConfiguration::GeneratorConfiguration() + : grpc_package_root("grpc"), beta_package_root("grpc.beta") {} + PythonGrpcGenerator::PythonGrpcGenerator(const GeneratorConfiguration& config) : config_(config) {} diff --git a/src/compiler/python_generator.h b/src/compiler/python_generator.h index fc51b48daec..7ed99eff0bf 100644 --- a/src/compiler/python_generator.h +++ b/src/compiler/python_generator.h @@ -43,6 +43,7 @@ namespace grpc_python_generator { // Data pertaining to configuration of the generator with respect to anything // that may be used internally at Google. struct GeneratorConfiguration { + GeneratorConfiguration(); grpc::string grpc_package_root; grpc::string beta_package_root; }; diff --git a/src/compiler/python_plugin.cc b/src/compiler/python_plugin.cc index fc76ee5ec6d..3457aa631a3 100644 --- a/src/compiler/python_plugin.cc +++ b/src/compiler/python_plugin.cc @@ -38,8 +38,6 @@ int main(int argc, char* argv[]) { grpc_python_generator::GeneratorConfiguration config; - config.grpc_package_root = "grpc"; - config.beta_package_root = "grpc.beta"; grpc_python_generator::PythonGrpcGenerator generator(config); return grpc::protobuf::compiler::PluginMain(argc, argv, &generator); } diff --git a/tools/distrib/python/grpcio_tools/grpc/tools/main.cc b/tools/distrib/python/grpcio_tools/grpc/tools/main.cc index 81675b4e6fc..83918395135 100644 --- a/tools/distrib/python/grpcio_tools/grpc/tools/main.cc +++ b/tools/distrib/python/grpcio_tools/grpc/tools/main.cc @@ -45,7 +45,6 @@ int protoc_main(int argc, char* argv[]) { // gRPC Python grpc_python_generator::GeneratorConfiguration grpc_py_config; - grpc_py_config.beta_package_root = "grpc.beta"; grpc_python_generator::PythonGrpcGenerator grpc_py_generator(grpc_py_config); cli.RegisterGenerator("--grpc_python_out", &grpc_py_generator, "Generate Python source file."); From 6721d4f0f18c71f5e4ab1cef904944185ed86b89 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Thu, 30 Jun 2016 17:17:23 -0700 Subject: [PATCH 0362/1039] Return success status of grpc_byte_buffer_reader --- include/grpc++/impl/codegen/core_codegen.h | 4 ++-- .../impl/codegen/core_codegen_interface.h | 5 +++-- include/grpc++/impl/codegen/proto_utils.h | 17 +++++++++++++-- include/grpc++/support/byte_buffer.h | 2 +- include/grpc/impl/codegen/byte_buffer.h | 7 ++++--- src/core/lib/surface/byte_buffer_reader.c | 8 ++++--- src/cpp/common/core_codegen.cc | 6 +++--- src/cpp/util/byte_buffer.cc | 10 ++++++--- src/csharp/ext/grpc_csharp_ext.c | 2 ++ src/node/ext/byte_buffer.cc | 5 ++++- .../GRPCClient/private/NSData+GRPC.m | 6 +++++- src/php/ext/grpc/byte_buffer.c | 5 ++--- .../grpc/_cython/_cygrpc/records.pyx.pxi | 1 + src/ruby/ext/grpc/rb_byte_buffer.c | 5 ++++- test/core/end2end/cq_verifier.c | 3 ++- test/core/surface/byte_buffer_reader_test.c | 21 ++++++++++++------- 16 files changed, 74 insertions(+), 33 deletions(-) diff --git a/include/grpc++/impl/codegen/core_codegen.h b/include/grpc++/impl/codegen/core_codegen.h index b0c4c57e66a..9699abfb438 100644 --- a/include/grpc++/impl/codegen/core_codegen.h +++ b/include/grpc++/impl/codegen/core_codegen.h @@ -54,8 +54,8 @@ class CoreCodegen : public CoreCodegenInterface { void grpc_byte_buffer_destroy(grpc_byte_buffer* bb) GRPC_OVERRIDE; - void grpc_byte_buffer_reader_init(grpc_byte_buffer_reader* reader, - grpc_byte_buffer* buffer) GRPC_OVERRIDE; + int grpc_byte_buffer_reader_init(grpc_byte_buffer_reader* reader, + grpc_byte_buffer* buffer) GRPC_OVERRIDE; void grpc_byte_buffer_reader_destroy(grpc_byte_buffer_reader* reader) GRPC_OVERRIDE; int grpc_byte_buffer_reader_next(grpc_byte_buffer_reader* reader, diff --git a/include/grpc++/impl/codegen/core_codegen_interface.h b/include/grpc++/impl/codegen/core_codegen_interface.h index 64d882ed5db..f9a8f9b980b 100644 --- a/include/grpc++/impl/codegen/core_codegen_interface.h +++ b/include/grpc++/impl/codegen/core_codegen_interface.h @@ -65,8 +65,9 @@ class CoreCodegenInterface { virtual void grpc_byte_buffer_destroy(grpc_byte_buffer* bb) = 0; - virtual void grpc_byte_buffer_reader_init(grpc_byte_buffer_reader* reader, - grpc_byte_buffer* buffer) = 0; + virtual int grpc_byte_buffer_reader_init(grpc_byte_buffer_reader* reader, + grpc_byte_buffer* buffer) + GRPC_MUST_USE_RESULT = 0; virtual void grpc_byte_buffer_reader_destroy( grpc_byte_buffer_reader* reader) = 0; virtual int grpc_byte_buffer_reader_next(grpc_byte_buffer_reader* reader, diff --git a/include/grpc++/impl/codegen/proto_utils.h b/include/grpc++/impl/codegen/proto_utils.h index 3bad468a74e..d4599c5ffff 100644 --- a/include/grpc++/impl/codegen/proto_utils.h +++ b/include/grpc++/impl/codegen/proto_utils.h @@ -111,14 +111,21 @@ class GrpcBufferReader GRPC_FINAL : public ::grpc::protobuf::io::ZeroCopyInputStream { public: explicit GrpcBufferReader(grpc_byte_buffer* buffer) - : byte_count_(0), backup_count_(0) { - g_core_codegen_interface->grpc_byte_buffer_reader_init(&reader_, buffer); + : byte_count_(0), backup_count_(0), status_() { + if (!g_core_codegen_interface->grpc_byte_buffer_reader_init(&reader_, + buffer)) { + status_ = Status(StatusCode::INTERNAL, + "Couldn't initialize byte buffer reader"); + } } ~GrpcBufferReader() GRPC_OVERRIDE { g_core_codegen_interface->grpc_byte_buffer_reader_destroy(&reader_); } bool Next(const void** data, int* size) GRPC_OVERRIDE { + if (!status_.ok()) { + return false; + } if (backup_count_ > 0) { *data = GPR_SLICE_START_PTR(slice_) + GPR_SLICE_LENGTH(slice_) - backup_count_; @@ -139,6 +146,8 @@ class GrpcBufferReader GRPC_FINAL return true; } + Status status() const { return status_; } + void BackUp(int count) GRPC_OVERRIDE { backup_count_ = count; } bool Skip(int count) GRPC_OVERRIDE { @@ -165,6 +174,7 @@ class GrpcBufferReader GRPC_FINAL int64_t backup_count_; grpc_byte_buffer_reader reader_; gpr_slice slice_; + Status status_; }; } // namespace internal @@ -202,6 +212,9 @@ class SerializationTraitsok(); { internal::GrpcBufferReader reader(buffer); + if (!reader.status().ok()) { + return reader.status(); + } ::grpc::protobuf::io::CodedInputStream decoder(&reader); if (max_message_size > 0) { decoder.SetTotalBytesLimit(max_message_size, max_message_size); diff --git a/include/grpc++/support/byte_buffer.h b/include/grpc++/support/byte_buffer.h index f6eb09638f5..20bd4071091 100644 --- a/include/grpc++/support/byte_buffer.h +++ b/include/grpc++/support/byte_buffer.h @@ -64,7 +64,7 @@ class ByteBuffer GRPC_FINAL { ByteBuffer& operator=(const ByteBuffer&); /// Dump (read) the buffer contents into \a slices. - void Dump(std::vector* slices) const; + Status Dump(std::vector* slices) const; /// Remove all data. void Clear(); diff --git a/include/grpc/impl/codegen/byte_buffer.h b/include/grpc/impl/codegen/byte_buffer.h index 3ae8ac50ba1..4cec21d1312 100644 --- a/include/grpc/impl/codegen/byte_buffer.h +++ b/include/grpc/impl/codegen/byte_buffer.h @@ -93,9 +93,10 @@ GRPCAPI void grpc_byte_buffer_destroy(grpc_byte_buffer *byte_buffer); struct grpc_byte_buffer_reader; typedef struct grpc_byte_buffer_reader grpc_byte_buffer_reader; -/** Initialize \a reader to read over \a buffer */ -GRPCAPI void grpc_byte_buffer_reader_init(grpc_byte_buffer_reader *reader, - grpc_byte_buffer *buffer); +/** Initialize \a reader to read over \a buffer. + * Returns 1 upon success, 0 otherwise. */ +GRPCAPI int grpc_byte_buffer_reader_init(grpc_byte_buffer_reader *reader, + grpc_byte_buffer *buffer); /** Cleanup and destroy \a reader */ GRPCAPI void grpc_byte_buffer_reader_destroy(grpc_byte_buffer_reader *reader); diff --git a/src/core/lib/surface/byte_buffer_reader.c b/src/core/lib/surface/byte_buffer_reader.c index c97079f6385..3a85f1f1f93 100644 --- a/src/core/lib/surface/byte_buffer_reader.c +++ b/src/core/lib/surface/byte_buffer_reader.c @@ -54,8 +54,8 @@ static int is_compressed(grpc_byte_buffer *buffer) { return 1 /* GPR_TRUE */; } -void grpc_byte_buffer_reader_init(grpc_byte_buffer_reader *reader, - grpc_byte_buffer *buffer) { +int grpc_byte_buffer_reader_init(grpc_byte_buffer_reader *reader, + grpc_byte_buffer *buffer) { gpr_slice_buffer decompressed_slices_buffer; reader->buffer_in = buffer; switch (reader->buffer_in->type) { @@ -69,7 +69,8 @@ void grpc_byte_buffer_reader_init(grpc_byte_buffer_reader *reader, "Unexpected error decompressing data for algorithm with enum " "value '%d'. Reading data as if it were uncompressed.", reader->buffer_in->data.raw.compression); - reader->buffer_out = reader->buffer_in; + return 0; + memset(reader, 0, sizeof(*reader)); } else { /* all fine */ reader->buffer_out = grpc_raw_byte_buffer_create(decompressed_slices_buffer.slices, @@ -82,6 +83,7 @@ void grpc_byte_buffer_reader_init(grpc_byte_buffer_reader *reader, reader->current.index = 0; break; } + return 1; } void grpc_byte_buffer_reader_destroy(grpc_byte_buffer_reader *reader) { diff --git a/src/cpp/common/core_codegen.cc b/src/cpp/common/core_codegen.cc index cc35aa69bab..3d6780bcb8c 100644 --- a/src/cpp/common/core_codegen.cc +++ b/src/cpp/common/core_codegen.cc @@ -74,9 +74,9 @@ void CoreCodegen::grpc_byte_buffer_destroy(grpc_byte_buffer* bb) { ::grpc_byte_buffer_destroy(bb); } -void CoreCodegen::grpc_byte_buffer_reader_init(grpc_byte_buffer_reader* reader, - grpc_byte_buffer* buffer) { - ::grpc_byte_buffer_reader_init(reader, buffer); +int CoreCodegen::grpc_byte_buffer_reader_init(grpc_byte_buffer_reader* reader, + grpc_byte_buffer* buffer) { + return ::grpc_byte_buffer_reader_init(reader, buffer); } void CoreCodegen::grpc_byte_buffer_reader_destroy( diff --git a/src/cpp/util/byte_buffer.cc b/src/cpp/util/byte_buffer.cc index c0a14de418a..c2cd20ee07f 100644 --- a/src/cpp/util/byte_buffer.cc +++ b/src/cpp/util/byte_buffer.cc @@ -58,18 +58,22 @@ void ByteBuffer::Clear() { } } -void ByteBuffer::Dump(std::vector* slices) const { +Status ByteBuffer::Dump(std::vector* slices) const { slices->clear(); if (!buffer_) { - return; + return Status(StatusCode::FAILED_PRECONDITION, "Buffer not initialized"); } grpc_byte_buffer_reader reader; - grpc_byte_buffer_reader_init(&reader, buffer_); + if (!grpc_byte_buffer_reader_init(&reader, buffer_)) { + return Status(StatusCode::INTERNAL, + "Couldn't initialize byte buffer reader"); + } gpr_slice s; while (grpc_byte_buffer_reader_next(&reader, &s)) { slices->push_back(Slice(s, Slice::STEAL_REF)); } grpc_byte_buffer_reader_destroy(&reader); + return Status::OK; } size_t ByteBuffer::Length() const { diff --git a/src/csharp/ext/grpc_csharp_ext.c b/src/csharp/ext/grpc_csharp_ext.c index 9b8d050ea51..4009748157b 100644 --- a/src/csharp/ext/grpc_csharp_ext.c +++ b/src/csharp/ext/grpc_csharp_ext.c @@ -253,6 +253,7 @@ GPR_EXPORT intptr_t GPR_CALLTYPE grpcsharp_batch_context_recv_message_length( if (!ctx->recv_message) { return -1; } + /* TODO(dgq): check the return value of grpc_byte_buffer_reader_init. */ grpc_byte_buffer_reader_init(&reader, ctx->recv_message); return (intptr_t)grpc_byte_buffer_length(reader.buffer_out); } @@ -267,6 +268,7 @@ GPR_EXPORT void GPR_CALLTYPE grpcsharp_batch_context_recv_message_to_buffer( gpr_slice slice; size_t offset = 0; + /* TODO(dgq): check the return value of grpc_byte_buffer_reader_init. */ grpc_byte_buffer_reader_init(&reader, ctx->recv_message); while (grpc_byte_buffer_reader_next(&reader, &slice)) { diff --git a/src/node/ext/byte_buffer.cc b/src/node/ext/byte_buffer.cc index 3479a677029..a3f678f32c6 100644 --- a/src/node/ext/byte_buffer.cc +++ b/src/node/ext/byte_buffer.cc @@ -73,7 +73,10 @@ Local ByteBufferToBuffer(grpc_byte_buffer *buffer) { return scope.Escape(Nan::Null()); } grpc_byte_buffer_reader reader; - grpc_byte_buffer_reader_init(&reader, buffer); + if (!grpc_byte_buffer_reader_init(&reader, buffer)) { + Nan::ThrowError("Error initializing byte buffer reader."); + return scope.Escape(Nan::Undefined()); + } gpr_slice slice = grpc_byte_buffer_reader_readall(&reader); size_t length = GPR_SLICE_LENGTH(slice); char *result = new char[length]; diff --git a/src/objective-c/GRPCClient/private/NSData+GRPC.m b/src/objective-c/GRPCClient/private/NSData+GRPC.m index 1238374af3d..ee72e9068ee 100644 --- a/src/objective-c/GRPCClient/private/NSData+GRPC.m +++ b/src/objective-c/GRPCClient/private/NSData+GRPC.m @@ -42,7 +42,11 @@ static void MallocAndCopyByteBufferToCharArray(grpc_byte_buffer *buffer, size_t *length, char **array) { grpc_byte_buffer_reader reader; - grpc_byte_buffer_reader_init(&reader, buffer); + if (!grpc_byte_buffer_reader_init(&reader, buffer)) { + *array = NULL; + *lenght = 0; + return; + } // The slice contains uncompressed data even if compressed data was received // because the reader takes care of automatically decompressing it gpr_slice slice = grpc_byte_buffer_reader_readall(&reader); diff --git a/src/php/ext/grpc/byte_buffer.c b/src/php/ext/grpc/byte_buffer.c index 7a726de5db4..9e20a4ef9b2 100644 --- a/src/php/ext/grpc/byte_buffer.c +++ b/src/php/ext/grpc/byte_buffer.c @@ -58,7 +58,8 @@ grpc_byte_buffer *string_to_byte_buffer(char *string, size_t length) { void byte_buffer_to_string(grpc_byte_buffer *buffer, char **out_string, size_t *out_length) { - if (buffer == NULL) { + grpc_byte_buffer_reader reader; + if (buffer == NULL || !grpc_byte_buffer_reader_init(&reader, buffer)) { *out_string = NULL; *out_length = 0; return; @@ -66,8 +67,6 @@ void byte_buffer_to_string(grpc_byte_buffer *buffer, char **out_string, size_t length = grpc_byte_buffer_length(buffer); char *string = ecalloc(length + 1, sizeof(char)); size_t offset = 0; - grpc_byte_buffer_reader reader; - grpc_byte_buffer_reader_init(&reader, buffer); gpr_slice next; while (grpc_byte_buffer_reader_next(&reader, &next) != 0) { memcpy(string + offset, GPR_SLICE_START_PTR(next), GPR_SLICE_LENGTH(next)); diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi index 8e651e880f3..337cf7fa3bb 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi @@ -254,6 +254,7 @@ cdef class ByteBuffer: cdef void *data_slice_pointer if self.c_byte_buffer != NULL: with nogil: + # TODO(dgq): check the return value of grpc_byte_buffer_reader_init. grpc_byte_buffer_reader_init(&reader, self.c_byte_buffer) result = bytearray() with nogil: diff --git a/src/ruby/ext/grpc/rb_byte_buffer.c b/src/ruby/ext/grpc/rb_byte_buffer.c index 1172691116c..61b7c30315d 100644 --- a/src/ruby/ext/grpc/rb_byte_buffer.c +++ b/src/ruby/ext/grpc/rb_byte_buffer.c @@ -56,7 +56,10 @@ VALUE grpc_rb_byte_buffer_to_s(grpc_byte_buffer *buffer) { return Qnil; } rb_string = rb_str_buf_new(grpc_byte_buffer_length(buffer)); - grpc_byte_buffer_reader_init(&reader, buffer); + if (!grpc_byte_buffer_reader_init(&reader, buffer)) { + rb_raise(rb_eRuntimeError, "Error initializing byte buffer reader."); + return Qnil; + } while (grpc_byte_buffer_reader_next(&reader, &next) != 0) { rb_str_cat(rb_string, (const char *) GPR_SLICE_START_PTR(next), GPR_SLICE_LENGTH(next)); diff --git a/test/core/end2end/cq_verifier.c b/test/core/end2end/cq_verifier.c index 8e9fa70b0ec..890309c44a8 100644 --- a/test/core/end2end/cq_verifier.c +++ b/test/core/end2end/cq_verifier.c @@ -149,7 +149,8 @@ int byte_buffer_eq_string(grpc_byte_buffer *bb, const char *str) { grpc_byte_buffer *rbb; int res; - grpc_byte_buffer_reader_init(&reader, bb); + GPR_ASSERT(grpc_byte_buffer_reader_init(&reader, bb) && + "Couldn't init byte buffer reader"); rbb = grpc_raw_byte_buffer_from_reader(&reader); res = byte_buffer_eq_slice(rbb, gpr_slice_from_copied_string(str)); grpc_byte_buffer_reader_destroy(&reader); diff --git a/test/core/surface/byte_buffer_reader_test.c b/test/core/surface/byte_buffer_reader_test.c index 9c6734e1799..1c779a8a847 100644 --- a/test/core/surface/byte_buffer_reader_test.c +++ b/test/core/surface/byte_buffer_reader_test.c @@ -59,7 +59,8 @@ static void test_read_one_slice(void) { slice = gpr_slice_from_copied_string("test"); buffer = grpc_raw_byte_buffer_create(&slice, 1); gpr_slice_unref(slice); - grpc_byte_buffer_reader_init(&reader, buffer); + GPR_ASSERT(grpc_byte_buffer_reader_init(&reader, buffer) && + "Couldn't init byte buffer reader"); first_code = grpc_byte_buffer_reader_next(&reader, &first_slice); GPR_ASSERT(first_code != 0); GPR_ASSERT(memcmp(GPR_SLICE_START_PTR(first_slice), "test", 4) == 0); @@ -81,7 +82,8 @@ static void test_read_one_slice_malloc(void) { memcpy(GPR_SLICE_START_PTR(slice), "test", 4); buffer = grpc_raw_byte_buffer_create(&slice, 1); gpr_slice_unref(slice); - grpc_byte_buffer_reader_init(&reader, buffer); + GPR_ASSERT(grpc_byte_buffer_reader_init(&reader, buffer) && + "Couldn't init byte buffer reader"); first_code = grpc_byte_buffer_reader_next(&reader, &first_slice); GPR_ASSERT(first_code != 0); GPR_ASSERT(memcmp(GPR_SLICE_START_PTR(first_slice), "test", 4) == 0); @@ -102,7 +104,8 @@ static void test_read_none_compressed_slice(void) { slice = gpr_slice_from_copied_string("test"); buffer = grpc_raw_byte_buffer_create(&slice, 1); gpr_slice_unref(slice); - grpc_byte_buffer_reader_init(&reader, buffer); + GPR_ASSERT(grpc_byte_buffer_reader_init(&reader, buffer) && + "Couldn't init byte buffer reader"); first_code = grpc_byte_buffer_reader_next(&reader, &first_slice); GPR_ASSERT(first_code != 0); GPR_ASSERT(memcmp(GPR_SLICE_START_PTR(first_slice), "test", 4) == 0); @@ -132,7 +135,8 @@ static void read_compressed_slice(grpc_compression_algorithm algorithm, buffer = grpc_raw_compressed_byte_buffer_create(sliceb_out.slices, sliceb_out.count, algorithm); - grpc_byte_buffer_reader_init(&reader, buffer); + GPR_ASSERT(grpc_byte_buffer_reader_init(&reader, buffer) && + "Couldn't init byte buffer reader"); while (grpc_byte_buffer_reader_next(&reader, &read_slice)) { GPR_ASSERT(memcmp(GPR_SLICE_START_PTR(read_slice), @@ -170,7 +174,8 @@ static void test_byte_buffer_from_reader(void) { memcpy(GPR_SLICE_START_PTR(slice), "test", 4); buffer = grpc_raw_byte_buffer_create(&slice, 1); gpr_slice_unref(slice); - grpc_byte_buffer_reader_init(&reader, buffer); + GPR_ASSERT(grpc_byte_buffer_reader_init(&reader, buffer) && + "Couldn't init byte buffer reader"); buffer_from_reader = grpc_raw_byte_buffer_from_reader(&reader); GPR_ASSERT(buffer->type == buffer_from_reader->type); @@ -206,7 +211,8 @@ static void test_readall(void) { gpr_slice_unref(slices[0]); gpr_slice_unref(slices[1]); - grpc_byte_buffer_reader_init(&reader, buffer); + GPR_ASSERT(grpc_byte_buffer_reader_init(&reader, buffer) && + "Couldn't init byte buffer reader"); slice_out = grpc_byte_buffer_reader_readall(&reader); GPR_ASSERT(GPR_SLICE_LENGTH(slice_out) == 512 + 1024); @@ -241,7 +247,8 @@ static void test_byte_buffer_copy(void) { gpr_slice_unref(slices[1]); copied_buffer = grpc_byte_buffer_copy(buffer); - grpc_byte_buffer_reader_init(&reader, copied_buffer); + GPR_ASSERT(grpc_byte_buffer_reader_init(&reader, buffer) && + "Couldn't init byte buffer reader"); slice_out = grpc_byte_buffer_reader_readall(&reader); GPR_ASSERT(GPR_SLICE_LENGTH(slice_out) == 512 + 1024); From 9ac997acf2f67eb76ce43359d7a7881f320f4c9c Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Thu, 30 Jun 2016 17:59:01 -0700 Subject: [PATCH 0363/1039] Added TODO for php. --- src/objective-c/GRPCClient/private/NSData+GRPC.m | 2 +- src/php/ext/grpc/byte_buffer.c | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/objective-c/GRPCClient/private/NSData+GRPC.m b/src/objective-c/GRPCClient/private/NSData+GRPC.m index ee72e9068ee..99c882ee922 100644 --- a/src/objective-c/GRPCClient/private/NSData+GRPC.m +++ b/src/objective-c/GRPCClient/private/NSData+GRPC.m @@ -44,7 +44,7 @@ static void MallocAndCopyByteBufferToCharArray(grpc_byte_buffer *buffer, grpc_byte_buffer_reader reader; if (!grpc_byte_buffer_reader_init(&reader, buffer)) { *array = NULL; - *lenght = 0; + *length = 0; return; } // The slice contains uncompressed data even if compressed data was received diff --git a/src/php/ext/grpc/byte_buffer.c b/src/php/ext/grpc/byte_buffer.c index 9e20a4ef9b2..2bf9e003050 100644 --- a/src/php/ext/grpc/byte_buffer.c +++ b/src/php/ext/grpc/byte_buffer.c @@ -60,6 +60,7 @@ void byte_buffer_to_string(grpc_byte_buffer *buffer, char **out_string, size_t *out_length) { grpc_byte_buffer_reader reader; if (buffer == NULL || !grpc_byte_buffer_reader_init(&reader, buffer)) { + /* TODO(dgq): distinguish between the error cases. */ *out_string = NULL; *out_length = 0; return; From 00d3f2789b3610d2dedb4b098b6e229ac2055009 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Thu, 30 Jun 2016 18:14:40 -0700 Subject: [PATCH 0364/1039] updated copyright --- include/grpc/impl/codegen/byte_buffer.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/grpc/impl/codegen/byte_buffer.h b/include/grpc/impl/codegen/byte_buffer.h index 4cec21d1312..fe1e2159793 100644 --- a/include/grpc/impl/codegen/byte_buffer.h +++ b/include/grpc/impl/codegen/byte_buffer.h @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without From 9a2320e40e105ec2ffc1d9b892b12b5afb8f9e17 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Thu, 30 Jun 2016 18:15:12 -0700 Subject: [PATCH 0365/1039] regenerated projects --- src/python/grpcio/grpc/_cython/imports.generated.h | 2 +- src/ruby/ext/grpc/rb_grpc_imports.generated.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/python/grpcio/grpc/_cython/imports.generated.h b/src/python/grpcio/grpc/_cython/imports.generated.h index b3e341fe25e..f87c4da7878 100644 --- a/src/python/grpcio/grpc/_cython/imports.generated.h +++ b/src/python/grpcio/grpc/_cython/imports.generated.h @@ -470,7 +470,7 @@ extern grpc_byte_buffer_length_type grpc_byte_buffer_length_import; typedef void(*grpc_byte_buffer_destroy_type)(grpc_byte_buffer *byte_buffer); extern grpc_byte_buffer_destroy_type grpc_byte_buffer_destroy_import; #define grpc_byte_buffer_destroy grpc_byte_buffer_destroy_import -typedef void(*grpc_byte_buffer_reader_init_type)(grpc_byte_buffer_reader *reader, grpc_byte_buffer *buffer); +typedef int(*grpc_byte_buffer_reader_init_type)(grpc_byte_buffer_reader *reader, grpc_byte_buffer *buffer); extern grpc_byte_buffer_reader_init_type grpc_byte_buffer_reader_init_import; #define grpc_byte_buffer_reader_init grpc_byte_buffer_reader_init_import typedef void(*grpc_byte_buffer_reader_destroy_type)(grpc_byte_buffer_reader *reader); diff --git a/src/ruby/ext/grpc/rb_grpc_imports.generated.h b/src/ruby/ext/grpc/rb_grpc_imports.generated.h index 13f961495cf..6f0974e31b4 100644 --- a/src/ruby/ext/grpc/rb_grpc_imports.generated.h +++ b/src/ruby/ext/grpc/rb_grpc_imports.generated.h @@ -470,7 +470,7 @@ extern grpc_byte_buffer_length_type grpc_byte_buffer_length_import; typedef void(*grpc_byte_buffer_destroy_type)(grpc_byte_buffer *byte_buffer); extern grpc_byte_buffer_destroy_type grpc_byte_buffer_destroy_import; #define grpc_byte_buffer_destroy grpc_byte_buffer_destroy_import -typedef void(*grpc_byte_buffer_reader_init_type)(grpc_byte_buffer_reader *reader, grpc_byte_buffer *buffer); +typedef int(*grpc_byte_buffer_reader_init_type)(grpc_byte_buffer_reader *reader, grpc_byte_buffer *buffer); extern grpc_byte_buffer_reader_init_type grpc_byte_buffer_reader_init_import; #define grpc_byte_buffer_reader_init grpc_byte_buffer_reader_init_import typedef void(*grpc_byte_buffer_reader_destroy_type)(grpc_byte_buffer_reader *reader); From 89d8f1697c2bf4a6691203d4a5f6fd495241696e Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Thu, 30 Jun 2016 18:17:27 -0700 Subject: [PATCH 0366/1039] Added comment for obj-c --- src/objective-c/GRPCClient/private/NSData+GRPC.m | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/objective-c/GRPCClient/private/NSData+GRPC.m b/src/objective-c/GRPCClient/private/NSData+GRPC.m index 99c882ee922..98337799e91 100644 --- a/src/objective-c/GRPCClient/private/NSData+GRPC.m +++ b/src/objective-c/GRPCClient/private/NSData+GRPC.m @@ -43,6 +43,10 @@ static void MallocAndCopyByteBufferToCharArray(grpc_byte_buffer *buffer, size_t *length, char **array) { grpc_byte_buffer_reader reader; if (!grpc_byte_buffer_reader_init(&reader, buffer)) { + // grpc_byte_buffer_reader_init can fail if the data sent by the server + // could not be decompressed for any reason. This is an issue with the data + // coming from the server and thus we want the RPC to fail with error code + // INTERNAL. *array = NULL; *length = 0; return; From 3bdaa5e72684a271445f945f3d6aa6ea33681dd1 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Thu, 30 Jun 2016 19:15:21 -0700 Subject: [PATCH 0367/1039] Regenerate list of gRPC-Core files --- gRPC-Core.podspec | 177 ++++++++++++++++++++++++++++++---------------- 1 file changed, 117 insertions(+), 60 deletions(-) diff --git a/gRPC-Core.podspec b/gRPC-Core.podspec index 7e9fe0a9afd..e10e05387b1 100644 --- a/gRPC-Core.podspec +++ b/gRPC-Core.podspec @@ -118,14 +118,14 @@ Pod::Spec.new do |s| 'include/grpc/support/atm.h', 'include/grpc/support/atm_gcc_atomic.h', 'include/grpc/support/atm_gcc_sync.h', - 'include/grpc/support/atm_win32.h', + 'include/grpc/support/atm_windows.h', 'include/grpc/support/avl.h', 'include/grpc/support/cmdline.h', 'include/grpc/support/cpu.h', 'include/grpc/support/histogram.h', 'include/grpc/support/host_port.h', 'include/grpc/support/log.h', - 'include/grpc/support/log_win32.h', + 'include/grpc/support/log_windows.h', 'include/grpc/support/port_platform.h', 'include/grpc/support/slice.h', 'include/grpc/support/slice_buffer.h', @@ -134,7 +134,7 @@ Pod::Spec.new do |s| 'include/grpc/support/sync.h', 'include/grpc/support/sync_generic.h', 'include/grpc/support/sync_posix.h', - 'include/grpc/support/sync_win32.h', + 'include/grpc/support/sync_windows.h', 'include/grpc/support/thd.h', 'include/grpc/support/time.h', 'include/grpc/support/tls.h', @@ -146,7 +146,7 @@ Pod::Spec.new do |s| 'include/grpc/impl/codegen/atm.h', 'include/grpc/impl/codegen/atm_gcc_atomic.h', 'include/grpc/impl/codegen/atm_gcc_sync.h', - 'include/grpc/impl/codegen/atm_win32.h', + 'include/grpc/impl/codegen/atm_windows.h', 'include/grpc/impl/codegen/log.h', 'include/grpc/impl/codegen/port_platform.h', 'include/grpc/impl/codegen/slice.h', @@ -154,12 +154,13 @@ Pod::Spec.new do |s| 'include/grpc/impl/codegen/sync.h', 'include/grpc/impl/codegen/sync_generic.h', 'include/grpc/impl/codegen/sync_posix.h', - 'include/grpc/impl/codegen/sync_win32.h', + 'include/grpc/impl/codegen/sync_windows.h', 'include/grpc/impl/codegen/time.h', 'include/grpc/byte_buffer.h', 'include/grpc/byte_buffer_reader.h', 'include/grpc/compression.h', 'include/grpc/grpc.h', + 'include/grpc/grpc_posix.h', 'include/grpc/status.h', 'include/grpc/impl/codegen/byte_buffer.h', 'include/grpc/impl/codegen/byte_buffer_reader.h', @@ -172,7 +173,7 @@ Pod::Spec.new do |s| 'include/grpc/impl/codegen/atm.h', 'include/grpc/impl/codegen/atm_gcc_atomic.h', 'include/grpc/impl/codegen/atm_gcc_sync.h', - 'include/grpc/impl/codegen/atm_win32.h', + 'include/grpc/impl/codegen/atm_windows.h', 'include/grpc/impl/codegen/log.h', 'include/grpc/impl/codegen/port_platform.h', 'include/grpc/impl/codegen/slice.h', @@ -180,7 +181,7 @@ Pod::Spec.new do |s| 'include/grpc/impl/codegen/sync.h', 'include/grpc/impl/codegen/sync_generic.h', 'include/grpc/impl/codegen/sync_posix.h', - 'include/grpc/impl/codegen/sync_win32.h', + 'include/grpc/impl/codegen/sync_windows.h', 'include/grpc/impl/codegen/time.h', 'include/grpc/grpc_security.h', 'include/grpc/grpc_security_constants.h', @@ -197,11 +198,10 @@ Pod::Spec.new do |s| 'src/core/lib/support/backoff.h', 'src/core/lib/support/block_annotate.h', 'src/core/lib/support/env.h', - 'src/core/lib/support/load_file.h', 'src/core/lib/support/murmur_hash.h', 'src/core/lib/support/stack_lockfree.h', 'src/core/lib/support/string.h', - 'src/core/lib/support/string_win32.h', + 'src/core/lib/support/string_windows.h', 'src/core/lib/support/thd_internal.h', 'src/core/lib/support/time_precise.h', 'src/core/lib/support/tmpfile.h', @@ -217,39 +217,38 @@ Pod::Spec.new do |s| 'src/core/lib/support/cpu_windows.c', 'src/core/lib/support/env_linux.c', 'src/core/lib/support/env_posix.c', - 'src/core/lib/support/env_win32.c', + 'src/core/lib/support/env_windows.c', 'src/core/lib/support/histogram.c', 'src/core/lib/support/host_port.c', - 'src/core/lib/support/load_file.c', 'src/core/lib/support/log.c', 'src/core/lib/support/log_android.c', 'src/core/lib/support/log_linux.c', 'src/core/lib/support/log_posix.c', - 'src/core/lib/support/log_win32.c', + 'src/core/lib/support/log_windows.c', 'src/core/lib/support/murmur_hash.c', 'src/core/lib/support/slice.c', 'src/core/lib/support/slice_buffer.c', 'src/core/lib/support/stack_lockfree.c', 'src/core/lib/support/string.c', 'src/core/lib/support/string_posix.c', - 'src/core/lib/support/string_util_win32.c', - 'src/core/lib/support/string_win32.c', + 'src/core/lib/support/string_util_windows.c', + 'src/core/lib/support/string_windows.c', 'src/core/lib/support/subprocess_posix.c', 'src/core/lib/support/subprocess_windows.c', 'src/core/lib/support/sync.c', 'src/core/lib/support/sync_posix.c', - 'src/core/lib/support/sync_win32.c', + 'src/core/lib/support/sync_windows.c', 'src/core/lib/support/thd.c', 'src/core/lib/support/thd_posix.c', - 'src/core/lib/support/thd_win32.c', + 'src/core/lib/support/thd_windows.c', 'src/core/lib/support/time.c', 'src/core/lib/support/time_posix.c', 'src/core/lib/support/time_precise.c', - 'src/core/lib/support/time_win32.c', + 'src/core/lib/support/time_windows.c', 'src/core/lib/support/tls_pthread.c', 'src/core/lib/support/tmpfile_msys.c', 'src/core/lib/support/tmpfile_posix.c', - 'src/core/lib/support/tmpfile_win32.c', + 'src/core/lib/support/tmpfile_windows.c', 'src/core/lib/support/wrap_memcpy.c', 'src/core/lib/channel/channel_args.h', 'src/core/lib/channel/channel_stack.h', @@ -268,7 +267,10 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/closure.h', 'src/core/lib/iomgr/endpoint.h', 'src/core/lib/iomgr/endpoint_pair.h', + 'src/core/lib/iomgr/error.h', + 'src/core/lib/iomgr/ev_epoll_linux.h', 'src/core/lib/iomgr/ev_poll_and_epoll_posix.h', + 'src/core/lib/iomgr/ev_poll_posix.h', 'src/core/lib/iomgr/ev_posix.h', 'src/core/lib/iomgr/exec_ctx.h', 'src/core/lib/iomgr/executor.h', @@ -276,6 +278,9 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/iomgr.h', 'src/core/lib/iomgr/iomgr_internal.h', 'src/core/lib/iomgr/iomgr_posix.h', + 'src/core/lib/iomgr/load_file.h', + 'src/core/lib/iomgr/network_status_tracker.h', + 'src/core/lib/iomgr/polling_entity.h', 'src/core/lib/iomgr/pollset.h', 'src/core/lib/iomgr/pollset_set.h', 'src/core/lib/iomgr/pollset_set_windows.h', @@ -284,7 +289,7 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/sockaddr.h', 'src/core/lib/iomgr/sockaddr_posix.h', 'src/core/lib/iomgr/sockaddr_utils.h', - 'src/core/lib/iomgr/sockaddr_win32.h', + 'src/core/lib/iomgr/sockaddr_windows.h', 'src/core/lib/iomgr/socket_utils_posix.h', 'src/core/lib/iomgr/socket_windows.h', 'src/core/lib/iomgr/tcp_client.h', @@ -316,7 +321,6 @@ Pod::Spec.new do |s| 'src/core/lib/surface/init.h', 'src/core/lib/surface/lame_client.h', 'src/core/lib/surface/server.h', - 'src/core/lib/surface/surface_trace.h', 'src/core/lib/transport/byte_stream.h', 'src/core/lib/transport/connectivity_state.h', 'src/core/lib/transport/metadata.h', @@ -324,6 +328,7 @@ Pod::Spec.new do |s| 'src/core/lib/transport/static_metadata.h', 'src/core/lib/transport/transport.h', 'src/core/lib/transport/transport_impl.h', + 'src/core/ext/transport/chttp2/transport/bin_decoder.h', 'src/core/ext/transport/chttp2/transport/bin_encoder.h', 'src/core/ext/transport/chttp2/transport/chttp2_transport.h', 'src/core/ext/transport/chttp2/transport/frame.h', @@ -345,15 +350,25 @@ Pod::Spec.new do |s| 'src/core/ext/transport/chttp2/transport/timeout_encoding.h', 'src/core/ext/transport/chttp2/transport/varint.h', 'src/core/ext/transport/chttp2/alpn/alpn.h', - 'src/core/lib/security/auth_filters.h', - 'src/core/lib/security/b64.h', - 'src/core/lib/security/credentials.h', - 'src/core/lib/security/handshake.h', - 'src/core/lib/security/json_token.h', - 'src/core/lib/security/jwt_verifier.h', - 'src/core/lib/security/secure_endpoint.h', - 'src/core/lib/security/security_connector.h', - 'src/core/lib/security/security_context.h', + 'src/core/lib/security/context/security_context.h', + 'src/core/lib/security/credentials/composite/composite_credentials.h', + 'src/core/lib/security/credentials/credentials.h', + 'src/core/lib/security/credentials/fake/fake_credentials.h', + 'src/core/lib/security/credentials/google_default/google_default_credentials.h', + 'src/core/lib/security/credentials/iam/iam_credentials.h', + 'src/core/lib/security/credentials/jwt/json_token.h', + 'src/core/lib/security/credentials/jwt/jwt_credentials.h', + 'src/core/lib/security/credentials/jwt/jwt_verifier.h', + 'src/core/lib/security/credentials/oauth2/oauth2_credentials.h', + 'src/core/lib/security/credentials/plugin/plugin_credentials.h', + 'src/core/lib/security/credentials/ssl/ssl_credentials.h', + 'src/core/lib/security/transport/auth_filters.h', + 'src/core/lib/security/transport/handshake.h', + 'src/core/lib/security/transport/secure_endpoint.h', + 'src/core/lib/security/transport/security_connector.h', + 'src/core/lib/security/transport/tsi_error.h', + 'src/core/lib/security/util/b64.h', + 'src/core/lib/security/util/json_util.h', 'src/core/lib/tsi/fake_transport_security.h', 'src/core/lib/tsi/ssl_transport_security.h', 'src/core/lib/tsi/ssl_types.h', @@ -376,14 +391,17 @@ Pod::Spec.new do |s| 'src/core/ext/client_config/subchannel_index.h', 'src/core/ext/client_config/uri_parser.h', 'src/core/ext/lb_policy/grpclb/load_balancer_api.h', - 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v0/load_balancer.pb.h', + 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h', 'third_party/nanopb/pb.h', 'third_party/nanopb/pb_common.h', 'third_party/nanopb/pb_decode.h', 'third_party/nanopb/pb_encode.h', + 'src/core/ext/load_reporting/load_reporting.h', + 'src/core/ext/load_reporting/load_reporting_filter.h', 'src/core/ext/census/aggregation.h', 'src/core/ext/census/census_interface.h', 'src/core/ext/census/census_rpc_stats.h', + 'src/core/ext/census/gen/census.pb.h', 'src/core/ext/census/grpc_filter.h', 'src/core/ext/census/mlog.h', 'src/core/ext/census/rpc_metric_id.h', @@ -395,7 +413,7 @@ Pod::Spec.new do |s| 'src/core/lib/channel/connected_channel.c', 'src/core/lib/channel/http_client_filter.c', 'src/core/lib/channel/http_server_filter.c', - 'src/core/lib/compression/compression_algorithm.c', + 'src/core/lib/compression/compression.c', 'src/core/lib/compression/message_compress.c', 'src/core/lib/debug/trace.c', 'src/core/lib/http/format_request.c', @@ -405,7 +423,10 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/endpoint.c', 'src/core/lib/iomgr/endpoint_pair_posix.c', 'src/core/lib/iomgr/endpoint_pair_windows.c', + 'src/core/lib/iomgr/error.c', + 'src/core/lib/iomgr/ev_epoll_linux.c', 'src/core/lib/iomgr/ev_poll_and_epoll_posix.c', + 'src/core/lib/iomgr/ev_poll_posix.c', 'src/core/lib/iomgr/ev_posix.c', 'src/core/lib/iomgr/exec_ctx.c', 'src/core/lib/iomgr/executor.c', @@ -413,6 +434,9 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/iomgr.c', 'src/core/lib/iomgr/iomgr_posix.c', 'src/core/lib/iomgr/iomgr_windows.c', + 'src/core/lib/iomgr/load_file.c', + 'src/core/lib/iomgr/network_status_tracker.c', + 'src/core/lib/iomgr/polling_entity.c', 'src/core/lib/iomgr/pollset_set_windows.c', 'src/core/lib/iomgr/pollset_windows.c', 'src/core/lib/iomgr/resolve_address_posix.c', @@ -470,6 +494,7 @@ Pod::Spec.new do |s| 'src/core/lib/transport/transport.c', 'src/core/lib/transport/transport_op_string.c', 'src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c', + 'src/core/ext/transport/chttp2/transport/bin_decoder.c', 'src/core/ext/transport/chttp2/transport/bin_encoder.c', 'src/core/ext/transport/chttp2/transport/chttp2_plugin.c', 'src/core/ext/transport/chttp2/transport/chttp2_transport.c', @@ -493,20 +518,29 @@ Pod::Spec.new do |s| 'src/core/ext/transport/chttp2/transport/writing.c', 'src/core/ext/transport/chttp2/alpn/alpn.c', 'src/core/lib/http/httpcli_security_connector.c', - 'src/core/lib/security/b64.c', - 'src/core/lib/security/client_auth_filter.c', - 'src/core/lib/security/credentials.c', - 'src/core/lib/security/credentials_metadata.c', - 'src/core/lib/security/credentials_posix.c', - 'src/core/lib/security/credentials_win32.c', - 'src/core/lib/security/google_default_credentials.c', - 'src/core/lib/security/handshake.c', - 'src/core/lib/security/json_token.c', - 'src/core/lib/security/jwt_verifier.c', - 'src/core/lib/security/secure_endpoint.c', - 'src/core/lib/security/security_connector.c', - 'src/core/lib/security/security_context.c', - 'src/core/lib/security/server_auth_filter.c', + 'src/core/lib/security/context/security_context.c', + 'src/core/lib/security/credentials/composite/composite_credentials.c', + 'src/core/lib/security/credentials/credentials.c', + 'src/core/lib/security/credentials/credentials_metadata.c', + 'src/core/lib/security/credentials/fake/fake_credentials.c', + 'src/core/lib/security/credentials/google_default/credentials_posix.c', + 'src/core/lib/security/credentials/google_default/credentials_windows.c', + 'src/core/lib/security/credentials/google_default/google_default_credentials.c', + 'src/core/lib/security/credentials/iam/iam_credentials.c', + 'src/core/lib/security/credentials/jwt/json_token.c', + 'src/core/lib/security/credentials/jwt/jwt_credentials.c', + 'src/core/lib/security/credentials/jwt/jwt_verifier.c', + 'src/core/lib/security/credentials/oauth2/oauth2_credentials.c', + 'src/core/lib/security/credentials/plugin/plugin_credentials.c', + 'src/core/lib/security/credentials/ssl/ssl_credentials.c', + 'src/core/lib/security/transport/client_auth_filter.c', + 'src/core/lib/security/transport/handshake.c', + 'src/core/lib/security/transport/secure_endpoint.c', + 'src/core/lib/security/transport/security_connector.c', + 'src/core/lib/security/transport/server_auth_filter.c', + 'src/core/lib/security/transport/tsi_error.c', + 'src/core/lib/security/util/b64.c', + 'src/core/lib/security/util/json_util.c', 'src/core/lib/surface/init_secure.c', 'src/core/lib/tsi/fake_transport_security.c', 'src/core/lib/tsi/ssl_transport_security.c', @@ -532,9 +566,11 @@ Pod::Spec.new do |s| 'src/core/ext/client_config/subchannel_index.c', 'src/core/ext/client_config/uri_parser.c', 'src/core/ext/transport/chttp2/server/insecure/server_chttp2.c', + 'src/core/ext/transport/chttp2/server/insecure/server_chttp2_posix.c', 'src/core/ext/transport/chttp2/client/insecure/channel_create.c', + 'src/core/ext/transport/chttp2/client/insecure/channel_create_posix.c', 'src/core/ext/lb_policy/grpclb/load_balancer_api.c', - 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v0/load_balancer.pb.c', + 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.c', 'third_party/nanopb/pb_common.c', 'third_party/nanopb/pb_decode.c', 'third_party/nanopb/pb_encode.c', @@ -542,7 +578,10 @@ Pod::Spec.new do |s| 'src/core/ext/lb_policy/round_robin/round_robin.c', 'src/core/ext/resolver/dns/native/dns_resolver.c', 'src/core/ext/resolver/sockaddr/sockaddr_resolver.c', + 'src/core/ext/load_reporting/load_reporting.c', + 'src/core/ext/load_reporting/load_reporting_filter.c', 'src/core/ext/census/context.c', + 'src/core/ext/census/gen/census.pb.c', 'src/core/ext/census/grpc_context.c', 'src/core/ext/census/grpc_filter.c', 'src/core/ext/census/grpc_plugin.c', @@ -557,11 +596,10 @@ Pod::Spec.new do |s| 'src/core/lib/support/backoff.h', 'src/core/lib/support/block_annotate.h', 'src/core/lib/support/env.h', - 'src/core/lib/support/load_file.h', 'src/core/lib/support/murmur_hash.h', 'src/core/lib/support/stack_lockfree.h', 'src/core/lib/support/string.h', - 'src/core/lib/support/string_win32.h', + 'src/core/lib/support/string_windows.h', 'src/core/lib/support/thd_internal.h', 'src/core/lib/support/time_precise.h', 'src/core/lib/support/tmpfile.h', @@ -582,7 +620,10 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/closure.h', 'src/core/lib/iomgr/endpoint.h', 'src/core/lib/iomgr/endpoint_pair.h', + 'src/core/lib/iomgr/error.h', + 'src/core/lib/iomgr/ev_epoll_linux.h', 'src/core/lib/iomgr/ev_poll_and_epoll_posix.h', + 'src/core/lib/iomgr/ev_poll_posix.h', 'src/core/lib/iomgr/ev_posix.h', 'src/core/lib/iomgr/exec_ctx.h', 'src/core/lib/iomgr/executor.h', @@ -590,6 +631,9 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/iomgr.h', 'src/core/lib/iomgr/iomgr_internal.h', 'src/core/lib/iomgr/iomgr_posix.h', + 'src/core/lib/iomgr/load_file.h', + 'src/core/lib/iomgr/network_status_tracker.h', + 'src/core/lib/iomgr/polling_entity.h', 'src/core/lib/iomgr/pollset.h', 'src/core/lib/iomgr/pollset_set.h', 'src/core/lib/iomgr/pollset_set_windows.h', @@ -598,7 +642,7 @@ Pod::Spec.new do |s| 'src/core/lib/iomgr/sockaddr.h', 'src/core/lib/iomgr/sockaddr_posix.h', 'src/core/lib/iomgr/sockaddr_utils.h', - 'src/core/lib/iomgr/sockaddr_win32.h', + 'src/core/lib/iomgr/sockaddr_windows.h', 'src/core/lib/iomgr/socket_utils_posix.h', 'src/core/lib/iomgr/socket_windows.h', 'src/core/lib/iomgr/tcp_client.h', @@ -630,7 +674,6 @@ Pod::Spec.new do |s| 'src/core/lib/surface/init.h', 'src/core/lib/surface/lame_client.h', 'src/core/lib/surface/server.h', - 'src/core/lib/surface/surface_trace.h', 'src/core/lib/transport/byte_stream.h', 'src/core/lib/transport/connectivity_state.h', 'src/core/lib/transport/metadata.h', @@ -638,6 +681,7 @@ Pod::Spec.new do |s| 'src/core/lib/transport/static_metadata.h', 'src/core/lib/transport/transport.h', 'src/core/lib/transport/transport_impl.h', + 'src/core/ext/transport/chttp2/transport/bin_decoder.h', 'src/core/ext/transport/chttp2/transport/bin_encoder.h', 'src/core/ext/transport/chttp2/transport/chttp2_transport.h', 'src/core/ext/transport/chttp2/transport/frame.h', @@ -659,15 +703,25 @@ Pod::Spec.new do |s| 'src/core/ext/transport/chttp2/transport/timeout_encoding.h', 'src/core/ext/transport/chttp2/transport/varint.h', 'src/core/ext/transport/chttp2/alpn/alpn.h', - 'src/core/lib/security/auth_filters.h', - 'src/core/lib/security/b64.h', - 'src/core/lib/security/credentials.h', - 'src/core/lib/security/handshake.h', - 'src/core/lib/security/json_token.h', - 'src/core/lib/security/jwt_verifier.h', - 'src/core/lib/security/secure_endpoint.h', - 'src/core/lib/security/security_connector.h', - 'src/core/lib/security/security_context.h', + 'src/core/lib/security/context/security_context.h', + 'src/core/lib/security/credentials/composite/composite_credentials.h', + 'src/core/lib/security/credentials/credentials.h', + 'src/core/lib/security/credentials/fake/fake_credentials.h', + 'src/core/lib/security/credentials/google_default/google_default_credentials.h', + 'src/core/lib/security/credentials/iam/iam_credentials.h', + 'src/core/lib/security/credentials/jwt/json_token.h', + 'src/core/lib/security/credentials/jwt/jwt_credentials.h', + 'src/core/lib/security/credentials/jwt/jwt_verifier.h', + 'src/core/lib/security/credentials/oauth2/oauth2_credentials.h', + 'src/core/lib/security/credentials/plugin/plugin_credentials.h', + 'src/core/lib/security/credentials/ssl/ssl_credentials.h', + 'src/core/lib/security/transport/auth_filters.h', + 'src/core/lib/security/transport/handshake.h', + 'src/core/lib/security/transport/secure_endpoint.h', + 'src/core/lib/security/transport/security_connector.h', + 'src/core/lib/security/transport/tsi_error.h', + 'src/core/lib/security/util/b64.h', + 'src/core/lib/security/util/json_util.h', 'src/core/lib/tsi/fake_transport_security.h', 'src/core/lib/tsi/ssl_transport_security.h', 'src/core/lib/tsi/ssl_types.h', @@ -690,14 +744,17 @@ Pod::Spec.new do |s| 'src/core/ext/client_config/subchannel_index.h', 'src/core/ext/client_config/uri_parser.h', 'src/core/ext/lb_policy/grpclb/load_balancer_api.h', - 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v0/load_balancer.pb.h', + 'src/core/ext/lb_policy/grpclb/proto/grpc/lb/v1/load_balancer.pb.h', 'third_party/nanopb/pb.h', 'third_party/nanopb/pb_common.h', 'third_party/nanopb/pb_decode.h', 'third_party/nanopb/pb_encode.h', + 'src/core/ext/load_reporting/load_reporting.h', + 'src/core/ext/load_reporting/load_reporting_filter.h', 'src/core/ext/census/aggregation.h', 'src/core/ext/census/census_interface.h', 'src/core/ext/census/census_rpc_stats.h', + 'src/core/ext/census/gen/census.pb.h', 'src/core/ext/census/grpc_filter.h', 'src/core/ext/census/mlog.h', 'src/core/ext/census/rpc_metric_id.h' From fa51de5d3dcd030de2d462130aa74ad83a01e16a Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Thu, 30 Jun 2016 23:50:48 -0700 Subject: [PATCH 0368/1039] Change port_server.py to use port 32766 32767 is used by filenet-powsrm --- tools/run_tests/port_server.py | 4 ++-- tools/run_tests/run_tests.py | 18 +++++++++++++++++- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/tools/run_tests/port_server.py b/tools/run_tests/port_server.py index 14e82b601ea..e2be26d182c 100755 --- a/tools/run_tests/port_server.py +++ b/tools/run_tests/port_server.py @@ -42,7 +42,7 @@ import time # increment this number whenever making a change to ensure that # the changes are picked up by running CI servers # note that all changes must be backwards compatible -_MY_VERSION = 7 +_MY_VERSION = 8 if len(sys.argv) == 2 and sys.argv[1] == 'dump_version': @@ -70,7 +70,7 @@ in_use = {} def refill_pool(max_timeout, req): """Scan for ports not marked for being in use""" - for i in range(1025, 32767): + for i in range(1025, 32766): if len(pool) > 100: break if i in in_use: age = time.time() - in_use[i] diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index c1254275e6c..413d4145601 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -1050,7 +1050,23 @@ runs_per_test = args.runs_per_test forever = args.forever +def _shut_down_legacy_server(legacy_server_port): + try: + version = int(urllib2.urlopen( + 'http://localhost:%d/version_number' % legacy_server_port, + timeout=10).read()) + except: + pass + else: + urllib2.urlopen( + 'http://localhost:%d/quitquitquit' % legacy_server_port).read() + + def _start_port_server(port_server_port): + # Temporary patch to switch the port_server port + # see https://github.com/grpc/grpc/issues/7145 + _shut_down_legacy_server(32767) + # check if a compatible port server is running # if incompatible (version mismatch) ==> start a new one # if not running ==> start a new one @@ -1186,7 +1202,7 @@ def _build_and_run( # start antagonists antagonists = [subprocess.Popen(['tools/run_tests/antagonist.py']) for _ in range(0, args.antagonists)] - port_server_port = 32767 + port_server_port = 32766 _start_port_server(port_server_port) resultset = None num_test_failures = 0 From ce14c30460394336c42b8304ad2fd3e0d775d61f Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 1 Jul 2016 09:06:38 -0700 Subject: [PATCH 0369/1039] Fix race in network status monitor --- src/core/lib/iomgr/network_status_tracker.c | 27 +++++++++------------ 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/src/core/lib/iomgr/network_status_tracker.c b/src/core/lib/iomgr/network_status_tracker.c index 38a1c9b7d41..d3f0ea53f87 100644 --- a/src/core/lib/iomgr/network_status_tracker.c +++ b/src/core/lib/iomgr/network_status_tracker.c @@ -42,27 +42,24 @@ typedef struct endpoint_ll_node { static endpoint_ll_node *head = NULL; static gpr_mu g_endpoint_mutex; -static bool g_init_done = false; +static gpr_once g_once_init = GPR_ONCE_INIT; -void grpc_initialize_network_status_monitor() { - g_init_done = true; - gpr_mu_init(&g_endpoint_mutex); - // TODO(makarandd): Install callback with OS to monitor network status. -} - -void grpc_destroy_network_status_monitor() { - for (endpoint_ll_node *curr = head; curr != NULL;) { - endpoint_ll_node *next = curr->next; - gpr_free(curr); - curr = next; +static void destroy_network_status_monitor() { + if (head != NULL) { + gpr_log(GPR_ERROR, + "Memory leaked as all network endpoints were not shut down"); } gpr_mu_destroy(&g_endpoint_mutex); } +static void initialize_network_status_monitor() { + gpr_mu_init(&g_endpoint_mutex); + atexit(destroy_network_status_monitor); + // TODO(makarandd): Install callback with OS to monitor network status. +} + void grpc_network_status_register_endpoint(grpc_endpoint *ep) { - if (!g_init_done) { - grpc_initialize_network_status_monitor(); - } + gpr_once_init(&g_once_init, initialize_network_status_monitor); gpr_mu_lock(&g_endpoint_mutex); if (head == NULL) { head = (endpoint_ll_node *)gpr_malloc(sizeof(endpoint_ll_node)); From 356758a7eec71b3abf4336bd0665476e7756fd2f Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Fri, 1 Jul 2016 09:56:50 -0700 Subject: [PATCH 0370/1039] Split Node health check code into a separate package and make it use static codegen --- package.json | 1 - src/node/health_check/health.js | 18 +++--- src/node/test/health_test.js | 60 +++++++++++-------- templates/package.json.template | 1 - .../node/health_check/package.json.template | 31 ++++++++++ 5 files changed, 76 insertions(+), 35 deletions(-) create mode 100644 templates/src/node/health_check/package.json.template diff --git a/package.json b/package.json index 68a31d794c5..1fec9cb40f8 100644 --- a/package.json +++ b/package.json @@ -59,7 +59,6 @@ "files": [ "LICENSE", "src/node/README.md", - "src/node/health_check", "src/proto", "etc", "src/node/index.js", diff --git a/src/node/health_check/health.js b/src/node/health_check/health.js index 52366830881..64ba9fb9601 100644 --- a/src/node/health_check/health.js +++ b/src/node/health_check/health.js @@ -33,14 +33,12 @@ 'use strict'; -var grpc = require('../'); +var grpc = require('grpc'); var _ = require('lodash'); -var health_proto = grpc.load(__dirname + - '/../../proto/grpc/health/v1/health.proto'); - -var HealthClient = health_proto.grpc.health.v1.Health; +var health_messages = require('./v1/health_pb'); +var health_service = require('./v1/health_grpc_pb'); function HealthImplementation(statusMap) { this.statusMap = _.clone(statusMap); @@ -51,17 +49,19 @@ HealthImplementation.prototype.setStatus = function(service, status) { }; HealthImplementation.prototype.check = function(call, callback){ - var service = call.request.service; + var service = call.request.getService(); var status = _.get(this.statusMap, service, null); if (status === null) { callback({code:grpc.status.NOT_FOUND}); } else { - callback(null, {status: status}); + var response = new health_messages.HealthCheckResponse(); + response.setStatus(status); + callback(null, response); } }; module.exports = { - Client: HealthClient, - service: HealthClient.service, + Client: health_service.HealthClient, + service: health_service.HealthService, Implementation: HealthImplementation }; diff --git a/src/node/test/health_test.js b/src/node/test/health_test.js index c93b528d42f..efbca46c2dc 100644 --- a/src/node/test/health_test.js +++ b/src/node/test/health_test.js @@ -35,15 +35,19 @@ var assert = require('assert'); -var health = require('../health_check/health.js'); +var health = require('../health_check/health'); + +var health_messages = require('../health_check/v1/health_pb'); + +var ServingStatus = health_messages.HealthCheckResponse.ServingStatus; var grpc = require('../'); describe('Health Checking', function() { var statusMap = { - '': 'SERVING', - 'grpc.test.TestServiceNotServing': 'NOT_SERVING', - 'grpc.test.TestServiceServing': 'SERVING' + '': ServingStatus.SERVING, + 'grpc.test.TestServiceNotServing': ServingStatus.NOT_SERVING, + 'grpc.test.TestServiceServing': ServingStatus.SERVING }; var healthServer; var healthImpl; @@ -51,7 +55,7 @@ describe('Health Checking', function() { before(function() { healthServer = new grpc.Server(); healthImpl = new health.Implementation(statusMap); - healthServer.addProtoService(health.service, healthImpl); + healthServer.addService(health.service, healthImpl); var port_num = healthServer.bind('0.0.0.0:0', grpc.ServerCredentials.createInsecure()); healthServer.start(); @@ -62,43 +66,51 @@ describe('Health Checking', function() { healthServer.forceShutdown(); }); it('should say an enabled service is SERVING', function(done) { - healthClient.check({service: ''}, function(err, response) { + var request = new health_messages.HealthCheckRequest(); + request.setService(''); + healthClient.check(request, function(err, response) { assert.ifError(err); - assert.strictEqual(response.status, 'SERVING'); + assert.strictEqual(response.getStatus(), ServingStatus.SERVING); done(); }); }); it('should say that a disabled service is NOT_SERVING', function(done) { - healthClient.check({service: 'grpc.test.TestServiceNotServing'}, - function(err, response) { - assert.ifError(err); - assert.strictEqual(response.status, 'NOT_SERVING'); - done(); - }); + var request = new health_messages.HealthCheckRequest(); + request.setService('grpc.test.TestServiceNotServing'); + healthClient.check(request, function(err, response) { + assert.ifError(err); + assert.strictEqual(response.getStatus(), ServingStatus.NOT_SERVING); + done(); + }); }); it('should say that an enabled service is SERVING', function(done) { - healthClient.check({service: 'grpc.test.TestServiceServing'}, - function(err, response) { - assert.ifError(err); - assert.strictEqual(response.status, 'SERVING'); - done(); - }); + var request = new health_messages.HealthCheckRequest(); + request.setService('grpc.test.TestServiceServing'); + healthClient.check(request, function(err, response) { + assert.ifError(err); + assert.strictEqual(response.getStatus(), ServingStatus.SERVING); + done(); + }); }); it('should get NOT_FOUND if the service is not registered', function(done) { - healthClient.check({service: 'not_registered'}, function(err, response) { + var request = new health_messages.HealthCheckRequest(); + request.setService('not_registered'); + healthClient.check(request, function(err, response) { assert(err); assert.strictEqual(err.code, grpc.status.NOT_FOUND); done(); }); }); it('should get a different response if the status changes', function(done) { - healthClient.check({service: 'transient'}, function(err, response) { + var request = new health_messages.HealthCheckRequest(); + request.setService('transient'); + healthClient.check(request, function(err, response) { assert(err); assert.strictEqual(err.code, grpc.status.NOT_FOUND); - healthImpl.setStatus('transient', 'SERVING'); - healthClient.check({service: 'transient'}, function(err, response) { + healthImpl.setStatus('transient', ServingStatus.SERVING); + healthClient.check(request, function(err, response) { assert.ifError(err); - assert.strictEqual(response.status, 'SERVING'); + assert.strictEqual(response.getStatus(), ServingStatus.SERVING); done(); }); }); diff --git a/templates/package.json.template b/templates/package.json.template index 9d19ca06293..f68f64d0475 100644 --- a/templates/package.json.template +++ b/templates/package.json.template @@ -61,7 +61,6 @@ "files": [ "LICENSE", "src/node/README.md", - "src/node/health_check", "src/proto", "etc", "src/node/index.js", diff --git a/templates/src/node/health_check/package.json.template b/templates/src/node/health_check/package.json.template new file mode 100644 index 00000000000..1248ced1e16 --- /dev/null +++ b/templates/src/node/health_check/package.json.template @@ -0,0 +1,31 @@ +%YAML 1.2 +--- | + { + "name": "grpc-health-check", + "version": "${settings.node_version}", + "author": "Google Inc.", + "description": "Health check service for use with gRPC", + "repository": { + "type": "git", + "url": "https://github.com/grpc/grpc.git" + }, + "bugs": "https://github.com/grpc/grpc/issues", + "contributors": [ + { + "name": "Michael Lumish", + "email": "mlumish@google.com" + } + ], + "dependencies": { + "grpc": "^0.15.0", + "lodash": "^3.9.3", + "google-protobuf": "^3.0.0-alpha.5" + }, + "files": { + "LICENSE", + "health.js", + "v1" + }, + "main": "src/node/index.js", + "license": "BSD-3-Clause" + } From c9579be13f56f889caaf8e910a374a9ba428da93 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Fri, 1 Jul 2016 10:02:32 -0700 Subject: [PATCH 0371/1039] Add Node health check package.json --- src/node/health_check/package.json | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 src/node/health_check/package.json diff --git a/src/node/health_check/package.json b/src/node/health_check/package.json new file mode 100644 index 00000000000..ad65b319177 --- /dev/null +++ b/src/node/health_check/package.json @@ -0,0 +1,29 @@ +{ + "name": "grpc-health-check", + "version": "0.16.0-dev", + "author": "Google Inc.", + "description": "Health check service for use with gRPC", + "repository": { + "type": "git", + "url": "https://github.com/grpc/grpc.git" + }, + "bugs": "https://github.com/grpc/grpc/issues", + "contributors": [ + { + "name": "Michael Lumish", + "email": "mlumish@google.com" + } + ], + "dependencies": { + "grpc": "^0.15.0", + "lodash": "^3.9.3", + "google-protobuf": "^3.0.0-alpha.5" + }, + "files": { + "LICENSE", + "health.js", + "v1" + }, + "main": "src/node/index.js", + "license": "BSD-3-Clause" +} From bb160258407289e9ab0dbb60c7d369c801bb3311 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Fri, 1 Jul 2016 10:09:16 -0700 Subject: [PATCH 0372/1039] Merge fixup: version, deployment target & spec name --- gRPC.podspec | 2 +- src/objective-c/CronetFramework.podspec | 2 +- src/objective-c/tests/Podfile | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/gRPC.podspec b/gRPC.podspec index f5744cdac55..e5556cc5448 100644 --- a/gRPC.podspec +++ b/gRPC.podspec @@ -36,7 +36,7 @@ Pod::Spec.new do |s| s.name = 'gRPC' - version = '0.12.0' + version = '0.14.0' s.version = version s.summary = 'gRPC client library for iOS/OSX' s.homepage = 'http://www.grpc.io' diff --git a/src/objective-c/CronetFramework.podspec b/src/objective-c/CronetFramework.podspec index 20af7647f70..3ebcacf0554 100644 --- a/src/objective-c/CronetFramework.podspec +++ b/src/objective-c/CronetFramework.podspec @@ -36,7 +36,7 @@ Pod::Spec.new do |s| s.license = { :type => 'BSD' } s.vendored_framework = "Cronet.framework" s.author = "The Chromium Authors" - s.ios.deployment_target = "8.0" + s.ios.deployment_target = "7.1" s.source = { :http => 'https://storage.googleapis.com/grpc-precompiled-binaries/cronet/Cronet.framework.zip' } s.preserve_paths = "Cronet.framework" s.public_header_files = "Cronet.framework/Headers/**/*{.h}" diff --git a/src/objective-c/tests/Podfile b/src/objective-c/tests/Podfile index 64b0009eb15..30a34260d40 100644 --- a/src/objective-c/tests/Podfile +++ b/src/objective-c/tests/Podfile @@ -63,7 +63,7 @@ post_install do |installer| target.build_configurations.each do |config| config.build_settings['GCC_TREAT_WARNINGS_AS_ERRORS'] = 'YES' end - if target.name == 'gRPC' + if target.name == 'gRPC-Core' target.build_configurations.each do |config| # TODO(zyc) Remove this setting after the issue is resolved # GPR_UNREACHABLE_CODE causes "Control may reach end of non-void From d5fee35f932c5aa3234d0aafaaf1e7ac7af82f3e Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Wed, 29 Jun 2016 11:14:53 -0700 Subject: [PATCH 0373/1039] Build Python3 grpcio-tools on OS X --- tools/distrib/python/grpcio_tools/setup.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/tools/distrib/python/grpcio_tools/setup.py b/tools/distrib/python/grpcio_tools/setup.py index fbe69f43d56..f9cb0534d54 100644 --- a/tools/distrib/python/grpcio_tools/setup.py +++ b/tools/distrib/python/grpcio_tools/setup.py @@ -31,9 +31,11 @@ from distutils import extension import errno import os import os.path +import pkg_resources import shlex import shutil import sys +import sysconfig import setuptools from setuptools.command import build_ext @@ -43,6 +45,8 @@ from setuptools.command import build_ext os.chdir(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, os.path.abspath('.')) +PY3 = sys.version_info.major == 3 + # There are some situations (like on Windows) where CC, CFLAGS, and LDFLAGS are # entirely ignored/dropped/forgotten by distutils and its Cygwin/MinGW support. # We use these environment variables to thus get around that without locking @@ -59,6 +63,15 @@ GRPC_PYTHON_PROTO_RESOURCES_NAME = '_proto' import protoc_lib_deps import grpc_version +# By default, Python3 distutils enforces compatibility of +# c plugins (.so files) with the OSX version Python3 was built with. +# For Python3.4, this is OSX 10.6, but we need Thread Local Support (__thread) +if 'darwin' in sys.platform and PY3: + mac_target = sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET') + if mac_target and (pkg_resources.parse_version(mac_target) < + pkg_resources.parse_version('10.9.0')): + os.environ['MACOSX_DEPLOYMENT_TARGET'] = '10.9' + def package_data(): tools_path = GRPC_PYTHON_TOOLS_PACKAGE.replace('.', os.path.sep) proto_resources_path = os.path.join(tools_path, From ebf81e7a0540a01079ac03661426cd3771664de7 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Wed, 1 Jun 2016 15:57:20 -0700 Subject: [PATCH 0374/1039] Add programmatic access to protoc in grpcio-tools --- ...otoc_compiler.pyx => _protoc_compiler.pyx} | 0 .../python/grpcio_tools/grpc/tools/command.py | 70 +++++++++++++++++++ .../python/grpcio_tools/grpc/tools/protoc.py | 13 +++- tools/distrib/python/grpcio_tools/setup.py | 4 +- 4 files changed, 82 insertions(+), 5 deletions(-) rename tools/distrib/python/grpcio_tools/grpc/tools/{protoc_compiler.pyx => _protoc_compiler.pyx} (100%) create mode 100644 tools/distrib/python/grpcio_tools/grpc/tools/command.py diff --git a/tools/distrib/python/grpcio_tools/grpc/tools/protoc_compiler.pyx b/tools/distrib/python/grpcio_tools/grpc/tools/_protoc_compiler.pyx similarity index 100% rename from tools/distrib/python/grpcio_tools/grpc/tools/protoc_compiler.pyx rename to tools/distrib/python/grpcio_tools/grpc/tools/_protoc_compiler.pyx diff --git a/tools/distrib/python/grpcio_tools/grpc/tools/command.py b/tools/distrib/python/grpcio_tools/grpc/tools/command.py new file mode 100644 index 00000000000..ccf38b7d569 --- /dev/null +++ b/tools/distrib/python/grpcio_tools/grpc/tools/command.py @@ -0,0 +1,70 @@ +# Copyright 2016, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +import os +import sys + +import setuptools + +from grpc.tools import protoc + + +class BuildProtoModules(setuptools.Command): + """Command to generate project *_pb2.py modules from proto files.""" + + description = 'build grpc protobuf modules' + user_options = [] + + def initialize_options(self): + pass + + def finalize_options(self): + pass + + def run(self): + # due to limitations of the proto generator, we require that only *one* + # directory is provided as an 'include' directory. We assume it's the '' key + # to `self.distribution.package_dir` (and get a key error if it's not + # there). + proto_files = [] + inclusion_root = os.path.abspath(self.distribution.package_dir['']) + for root, _, files in os.walk(inclusion_root): + for filename in files: + if filename.endswith('.proto'): + proto_files.append(os.path.abspath(os.path.join(root, filename))) + + for proto_file in proto_files: + command = [ + 'grpc.tools.protoc', + '--proto_path={}'.format(inclusion_root), + '--python_out={}'.format(inclusion_root), + '--grpc_python_out={}'.format(inclusion_root), + ] + [proto_file] + if protoc.main(command) != 0: + sys.stderr.write('warning: {} failed'.format(command)) diff --git a/tools/distrib/python/grpcio_tools/grpc/tools/protoc.py b/tools/distrib/python/grpcio_tools/grpc/tools/protoc.py index 1c69e78920c..931fda27d26 100644 --- a/tools/distrib/python/grpcio_tools/grpc/tools/protoc.py +++ b/tools/distrib/python/grpcio_tools/grpc/tools/protoc.py @@ -32,10 +32,17 @@ import pkg_resources import sys -from grpc.tools import protoc_compiler +from grpc.tools import _protoc_compiler +def main(command_arguments): + """Run the protocol buffer compiler with the given command-line arguments. + + Args: + command_arguments: a list of strings representing command line arguments to + `protoc`. + """ + return _protoc_compiler.run_main(command_arguments) if __name__ == '__main__': proto_include = pkg_resources.resource_filename('grpc.tools', '_proto') - protoc_compiler.run_main( - sys.argv + ['-I{}'.format(proto_include)]) + sys.exit(main(sys.argv + ['-I{}'.format(proto_include)])) diff --git a/tools/distrib/python/grpcio_tools/setup.py b/tools/distrib/python/grpcio_tools/setup.py index f9cb0534d54..afb6063906e 100644 --- a/tools/distrib/python/grpcio_tools/setup.py +++ b/tools/distrib/python/grpcio_tools/setup.py @@ -99,8 +99,8 @@ def protoc_ext_module(): os.path.join(protoc_lib_deps.CC_INCLUDE, cc_file) for cc_file in protoc_lib_deps.CC_FILES] plugin_ext = extension.Extension( - name='grpc.tools.protoc_compiler', - sources=['grpc/tools/protoc_compiler.pyx'] + plugin_sources, + name='grpc.tools._protoc_compiler', + sources=['grpc/tools/_protoc_compiler.pyx'] + plugin_sources, include_dirs=[ '.', 'grpc_root', From b926ef2fb7e10df1f45166296e9e3a8a44f7060f Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Fri, 3 Jun 2016 12:26:48 -0700 Subject: [PATCH 0375/1039] Ignore cython debug information --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index ca61bda124c..26d51236705 100644 --- a/.gitignore +++ b/.gitignore @@ -5,6 +5,7 @@ libs objs # Python items +cython_debug/ python_build/ .coverage* .eggs From 1ff429da2a94bc79300ebce3f8aae7efb10e9a75 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Thu, 2 Jun 2016 16:39:20 -0700 Subject: [PATCH 0376/1039] Organize Python tests to use grpcio-tools directly Moves all tests into a separate package. This does not change existing supported means of running tests (e.g. through run_tests.py). --- .gitignore | 2 + setup.py | 44 +--- src/python/grpcio/commands.py | 146 ----------- .../grpcio_health_checking/health_commands.py | 47 +--- src/python/grpcio_health_checking/setup.py | 34 ++- src/python/grpcio_tests/.gitignore | 4 + src/python/grpcio_tests/commands.py | 217 ++++++++++++++++ src/python/grpcio_tests/grpc_version.py | 32 +++ src/python/grpcio_tests/setup.py | 124 ++++++++++ .../tests/__init__.py | 0 .../{grpcio => grpcio_tests}/tests/_loader.py | 0 .../{grpcio => grpcio_tests}/tests/_result.py | 0 .../{grpcio => grpcio_tests}/tests/_runner.py | 0 .../tests/health_check/__init__.py | 0 .../health_check/_health_servicer_test.py | 0 .../tests/interop/__init__.py | 0 .../tests/interop/_insecure_interop_test.py | 0 .../tests/interop/_interop_test_case.py | 0 .../tests/interop/_secure_interop_test.py | 0 .../tests/interop/client.py | 0 .../tests/interop/credentials/README | 0 .../tests/interop/credentials/ca.pem | 0 .../tests/interop/credentials/server1.key | 0 .../tests/interop/credentials/server1.pem | 0 .../tests/interop/methods.py | 0 .../tests/interop/resources.py | 0 .../tests/interop/server.py | 0 .../tests/protoc_plugin/__init__.py | 0 .../protoc_plugin/_python_plugin_test.py | 232 ++++++----------- .../protoc_plugin/beta_python_plugin_test.py | 234 ++++++------------ .../tests/protoc_plugin/protos}/__init__.py | 0 .../protoc_plugin/protos/payload}/__init__.py | 0 .../protos/payload/test_payload.proto | 0 .../protos/requests}/__init__.py | 0 .../protos/requests/r}/__init__.py | 0 .../protos/requests/r/test_requests.proto | 2 +- .../protos/responses}/__init__.py | 0 .../protos/responses/test_responses.proto | 2 +- .../protoc_plugin/protos/service}/__init__.py | 0 .../protos/service/test_service.proto | 4 +- .../tests/qps/__init__.py | 0 .../tests/qps/benchmark_client.py | 0 .../tests/qps/benchmark_server.py | 0 .../tests/qps/client_runner.py | 0 .../tests/qps/histogram.py | 0 .../tests/qps/qps_worker.py | 0 .../tests/qps/worker_server.py | 0 .../tests/stress/__init__.py | 0 .../tests/stress/client.py | 0 .../tests/stress/metrics_server.py | 0 .../tests/stress/test_runner.py | 0 .../{grpcio => grpcio_tests}/tests/tests.json | 0 .../tests/unit}/__init__.py | 0 .../tests/unit/_adapter/.gitignore | 0 .../tests/unit/_adapter}/__init__.py | 0 .../tests/unit/_adapter/_proto_scenarios.py | 0 .../tests/unit/_api_test.py | 0 .../tests/unit/_auth_test.py | 0 .../tests/unit/_channel_connectivity_test.py | 0 .../tests/unit/_channel_ready_future_test.py | 0 .../tests/unit/_compression_test.py | 0 .../tests/unit/_cython/.gitignore | 0 .../tests/unit/_cython/__init__.py | 0 .../unit/_cython/_cancel_many_calls_test.py | 0 .../tests/unit/_cython/_channel_test.py | 0 .../_read_some_but_not_all_responses_test.py | 0 .../tests/unit/_cython/cygrpc_test.py | 0 .../tests/unit/_cython/test_utilities.py | 0 .../tests/unit/_empty_message_test.py | 0 .../tests/unit/_exit_scenarios.py | 0 .../tests/unit/_exit_test.py | 0 .../tests/unit/_from_grpc_import_star.py | 0 .../tests/unit/_junkdrawer}/__init__.py | 0 .../tests/unit/_junkdrawer/math_pb2.py | 0 .../tests/unit/_junkdrawer/stock_pb2.py | 0 .../tests/unit/_links}/__init__.py | 0 .../tests/unit/_links/_proto_scenarios.py | 0 .../tests/unit/_metadata_code_details_test.py | 0 .../tests/unit/_metadata_test.py | 0 .../tests/unit/_rpc_test.py | 0 .../tests/unit/_sanity/__init__.py | 0 .../tests/unit/_sanity/_sanity_test.py | 9 +- .../tests/unit/_thread_cleanup_test.py | 0 .../tests/unit/beta}/__init__.py | 0 .../tests/unit/beta/_beta_features_test.py | 0 .../unit/beta/_connectivity_channel_test.py | 0 .../tests/unit/beta/_face_interface_test.py | 0 .../tests/unit/beta/_implementations_test.py | 0 .../tests/unit/beta/_not_found_test.py | 0 .../tests/unit/beta/_utilities_test.py | 0 .../tests/unit/beta/test_utilities.py | 0 .../tests/unit/credentials/README | 0 .../tests/unit/credentials/ca.pem | 0 .../tests/unit/credentials/server1.key | 0 .../tests/unit/credentials/server1.pem | 0 .../tests/unit/framework}/__init__.py | 0 .../tests/unit/framework/common}/__init__.py | 0 .../unit/framework/common/test_constants.py | 0 .../unit/framework/common/test_control.py | 0 .../unit/framework/common/test_coverage.py | 0 .../tests/unit/framework/core/__init__.py | 30 +++ .../unit/framework/foundation/__init__.py | 30 +++ .../foundation/_logging_pool_test.py | 0 .../framework/foundation/stream_testing.py | 0 .../unit/framework/interfaces/__init__.py | 30 +++ .../framework/interfaces/base/__init__.py | 30 +++ .../framework/interfaces/base/_control.py | 0 .../framework/interfaces/base/_sequence.py | 0 .../unit/framework/interfaces/base/_state.py | 0 .../framework/interfaces/base/test_cases.py | 0 .../interfaces/base/test_interfaces.py | 0 .../interfaces/face/_3069_test_constant.py | 0 .../framework/interfaces/face/__init__.py | 30 +++ .../_blocking_invocation_inline_service.py | 0 .../unit/framework/interfaces/face/_digest.py | 0 ...e_invocation_asynchronous_event_service.py | 0 .../framework/interfaces/face/_invocation.py | 0 .../framework/interfaces/face/_receiver.py | 0 .../framework/interfaces/face/_service.py | 0 .../interfaces/face/_stock_service.py | 0 .../framework/interfaces/face/test_cases.py | 0 .../interfaces/face/test_interfaces.py | 0 .../framework/interfaces/links/__init__.py | 30 +++ .../framework/interfaces/links/test_cases.py | 0 .../interfaces/links/test_utilities.py | 0 .../tests/unit/resources.py | 0 .../tests/unit/test_common.py | 0 .../grpcio_tests/grpc_version.py.template | 34 +++ tools/distrib/python/grpcio_tools/.gitignore | 1 + tools/run_tests/build_python.sh | 37 +-- .../performance/run_worker_python.sh | 2 +- tools/run_tests/run_python.sh | 2 +- tools/run_tests/run_tests.py | 5 +- tox.ini | 4 +- 134 files changed, 787 insertions(+), 611 deletions(-) create mode 100644 src/python/grpcio_tests/.gitignore create mode 100644 src/python/grpcio_tests/commands.py create mode 100644 src/python/grpcio_tests/grpc_version.py create mode 100644 src/python/grpcio_tests/setup.py rename src/python/{grpcio => grpcio_tests}/tests/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/_loader.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/_result.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/_runner.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/health_check/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/health_check/_health_servicer_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/_insecure_interop_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/_interop_test_case.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/_secure_interop_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/client.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/credentials/README (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/credentials/ca.pem (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/credentials/server1.key (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/credentials/server1.pem (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/methods.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/resources.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/interop/server.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/protoc_plugin/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/protoc_plugin/_python_plugin_test.py (67%) rename src/python/{grpcio => grpcio_tests}/tests/protoc_plugin/beta_python_plugin_test.py (64%) rename src/python/{grpcio/tests/unit => grpcio_tests/tests/protoc_plugin/protos}/__init__.py (100%) rename src/python/{grpcio/tests/unit/_adapter => grpcio_tests/tests/protoc_plugin/protos/payload}/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/protoc_plugin/protos/payload/test_payload.proto (100%) rename src/python/{grpcio/tests/unit/_junkdrawer => grpcio_tests/tests/protoc_plugin/protos/requests}/__init__.py (100%) rename src/python/{grpcio/tests/unit/_links => grpcio_tests/tests/protoc_plugin/protos/requests/r}/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/protoc_plugin/protos/requests/r/test_requests.proto (97%) rename src/python/{grpcio/tests/unit/beta => grpcio_tests/tests/protoc_plugin/protos/responses}/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/protoc_plugin/protos/responses/test_responses.proto (96%) rename src/python/{grpcio/tests/unit/framework => grpcio_tests/tests/protoc_plugin/protos/service}/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/protoc_plugin/protos/service/test_service.proto (95%) rename src/python/{grpcio => grpcio_tests}/tests/qps/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/qps/benchmark_client.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/qps/benchmark_server.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/qps/client_runner.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/qps/histogram.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/qps/qps_worker.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/qps/worker_server.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/stress/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/stress/client.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/stress/metrics_server.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/stress/test_runner.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/tests.json (100%) rename src/python/{grpcio/tests/unit/framework/common => grpcio_tests/tests/unit}/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_adapter/.gitignore (100%) rename src/python/{grpcio/tests/unit/framework/core => grpcio_tests/tests/unit/_adapter}/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_adapter/_proto_scenarios.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_api_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_auth_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_channel_connectivity_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_channel_ready_future_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_compression_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_cython/.gitignore (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_cython/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_cython/_cancel_many_calls_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_cython/_channel_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_cython/_read_some_but_not_all_responses_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_cython/cygrpc_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_cython/test_utilities.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_empty_message_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_exit_scenarios.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_exit_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_from_grpc_import_star.py (100%) rename src/python/{grpcio/tests/unit/framework/foundation => grpcio_tests/tests/unit/_junkdrawer}/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_junkdrawer/math_pb2.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_junkdrawer/stock_pb2.py (100%) rename src/python/{grpcio/tests/unit/framework/interfaces => grpcio_tests/tests/unit/_links}/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_links/_proto_scenarios.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_metadata_code_details_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_metadata_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_rpc_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_sanity/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_sanity/_sanity_test.py (90%) rename src/python/{grpcio => grpcio_tests}/tests/unit/_thread_cleanup_test.py (100%) rename src/python/{grpcio/tests/unit/framework/interfaces/base => grpcio_tests/tests/unit/beta}/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/beta/_beta_features_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/beta/_connectivity_channel_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/beta/_face_interface_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/beta/_implementations_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/beta/_not_found_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/beta/_utilities_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/beta/test_utilities.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/credentials/README (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/credentials/ca.pem (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/credentials/server1.key (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/credentials/server1.pem (100%) rename src/python/{grpcio/tests/unit/framework/interfaces/face => grpcio_tests/tests/unit/framework}/__init__.py (100%) rename src/python/{grpcio/tests/unit/framework/interfaces/links => grpcio_tests/tests/unit/framework/common}/__init__.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/common/test_constants.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/common/test_control.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/common/test_coverage.py (100%) create mode 100644 src/python/grpcio_tests/tests/unit/framework/core/__init__.py create mode 100644 src/python/grpcio_tests/tests/unit/framework/foundation/__init__.py rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/foundation/_logging_pool_test.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/foundation/stream_testing.py (100%) create mode 100644 src/python/grpcio_tests/tests/unit/framework/interfaces/__init__.py create mode 100644 src/python/grpcio_tests/tests/unit/framework/interfaces/base/__init__.py rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/base/_control.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/base/_sequence.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/base/_state.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/base/test_cases.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/base/test_interfaces.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/face/_3069_test_constant.py (100%) create mode 100644 src/python/grpcio_tests/tests/unit/framework/interfaces/face/__init__.py rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/face/_blocking_invocation_inline_service.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/face/_digest.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/face/_invocation.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/face/_receiver.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/face/_service.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/face/_stock_service.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/face/test_cases.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/face/test_interfaces.py (100%) create mode 100644 src/python/grpcio_tests/tests/unit/framework/interfaces/links/__init__.py rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/links/test_cases.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/framework/interfaces/links/test_utilities.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/resources.py (100%) rename src/python/{grpcio => grpcio_tests}/tests/unit/test_common.py (100%) create mode 100644 templates/src/python/grpcio_tests/grpc_version.py.template diff --git a/.gitignore b/.gitignore index 26d51236705..d985a797949 100644 --- a/.gitignore +++ b/.gitignore @@ -13,6 +13,8 @@ python_build/ htmlcov/ dist/ *.egg +py27/ +py34/ # Node installation output ^node_modules diff --git a/setup.py b/setup.py index c1341187cad..d9c46ba77a1 100644 --- a/setup.py +++ b/setup.py @@ -72,10 +72,6 @@ BUILD_WITH_CYTHON = os.environ.get('GRPC_PYTHON_BUILD_WITH_CYTHON', False) ENABLE_CYTHON_TRACING = os.environ.get( 'GRPC_PYTHON_ENABLE_CYTHON_TRACING', False) -# Environment variable to determine whether or not to include the test files in -# the installation. -INSTALL_TESTS = os.environ.get('GRPC_PYTHON_INSTALL_TESTS', False) - CYTHON_EXTENSION_PACKAGE_NAMES = () CYTHON_EXTENSION_MODULE_NAMES = ('grpc._cython.cygrpc',) @@ -183,13 +179,10 @@ SETUP_REQUIRES = INSTALL_REQUIRES + ( COMMAND_CLASS = { 'doc': commands.SphinxDocumentation, - 'build_proto_modules': commands.BuildProtoModules, 'build_project_metadata': commands.BuildProjectMetadata, 'build_py': commands.BuildPy, 'build_ext': commands.BuildExt, 'gather': commands.Gather, - 'run_interop': commands.RunInterop, - 'test_lite': commands.TestLite } # Ensure that package data is copied over before any commands have been run: @@ -200,32 +193,6 @@ except OSError: pass shutil.copyfile('etc/roots.pem', os.path.join(credentials_dir, 'roots.pem')) -TEST_PACKAGE_DATA = { - 'tests.interop': [ - 'credentials/ca.pem', - 'credentials/server1.key', - 'credentials/server1.pem', - ], - 'tests.protoc_plugin': [ - 'protoc_plugin_test.proto', - ], - 'tests.unit': [ - 'credentials/ca.pem', - 'credentials/server1.key', - 'credentials/server1.pem', - ], -} - -TESTS_REQUIRE = ( - 'oauth2client>=2.1.0', - 'protobuf>=3.0.0a3', - 'coverage>=4.0', -) + INSTALL_REQUIRES - -TEST_SUITE = 'tests' -TEST_LOADER = 'tests:Loader' -TEST_RUNNER = 'tests:Runner' - PACKAGE_DATA = { # Binaries that may or may not be present in the final installation, but are # mentioned here for completeness. @@ -235,12 +202,7 @@ PACKAGE_DATA = { '_windows/grpc_c.64.python', ], } -if INSTALL_TESTS: - PACKAGE_DATA = dict(PACKAGE_DATA, **TEST_PACKAGE_DATA) - PACKAGES = setuptools.find_packages(PYTHON_STEM) -else: - PACKAGES = setuptools.find_packages( - PYTHON_STEM, exclude=['tests', 'tests.*']) +PACKAGES = setuptools.find_packages(PYTHON_STEM) setuptools.setup( name='grpcio', @@ -253,8 +215,4 @@ setuptools.setup( install_requires=INSTALL_REQUIRES, setup_requires=SETUP_REQUIRES, cmdclass=COMMAND_CLASS, - tests_require=TESTS_REQUIRE, - test_suite=TEST_SUITE, - test_loader=TEST_LOADER, - test_runner=TEST_RUNNER, ) diff --git a/src/python/grpcio/commands.py b/src/python/grpcio/commands.py index f498ed41901..3f91954d5f0 100644 --- a/src/python/grpcio/commands.py +++ b/src/python/grpcio/commands.py @@ -134,75 +134,6 @@ class SphinxDocumentation(setuptools.Command): sphinx.main(['', os.path.join('doc', 'src'), os.path.join('doc', 'build')]) -class BuildProtoModules(setuptools.Command): - """Command to generate project *_pb2.py modules from proto files.""" - - description = 'build protobuf modules' - user_options = [ - ('include=', None, 'path patterns to include in protobuf generation'), - ('exclude=', None, 'path patterns to exclude from protobuf generation') - ] - - def initialize_options(self): - self.exclude = None - self.include = r'.*\.proto$' - self.protoc_command = None - self.grpc_python_plugin_command = None - - def finalize_options(self): - self.protoc_command = distutils.spawn.find_executable('protoc') - self.grpc_python_plugin_command = distutils.spawn.find_executable( - 'grpc_python_plugin') - - def run(self): - if not self.protoc_command: - raise CommandError('could not find protoc') - if not self.grpc_python_plugin_command: - raise CommandError('could not find grpc_python_plugin ' - '(protoc plugin for GRPC Python)') - - if not os.path.exists(PROTO_GEN_STEM): - os.makedirs(PROTO_GEN_STEM) - - include_regex = re.compile(self.include) - exclude_regex = re.compile(self.exclude) if self.exclude else None - paths = [] - for walk_root, directories, filenames in os.walk(PROTO_STEM): - for filename in filenames: - path = os.path.join(walk_root, filename) - if include_regex.match(path) and not ( - exclude_regex and exclude_regex.match(path)): - paths.append(path) - - # TODO(kpayson): It would be nice to do this in a batch command, - # but we currently have name conflicts in src/proto - for path in paths: - command = [ - self.protoc_command, - '--plugin=protoc-gen-python-grpc={}'.format( - self.grpc_python_plugin_command), - '-I {}'.format(GRPC_STEM), - '-I .', - '-I {}/third_party/protobuf/src'.format(GRPC_STEM), - '--python_out={}'.format(PROTO_GEN_STEM), - '--python-grpc_out={}'.format(PROTO_GEN_STEM), - ] + [path] - try: - subprocess.check_output(' '.join(command), cwd=PYTHON_STEM, shell=True, - stderr=subprocess.STDOUT) - except subprocess.CalledProcessError as e: - sys.stderr.write( - 'warning: Command:\n{}\nMessage:\n{}\nOutput:\n{}'.format( - command, str(e), e.output)) - - # Generated proto directories dont include __init__.py, but - # these are needed for python package resolution - for walk_root, _, _ in os.walk(PROTO_GEN_STEM): - if walk_root != PROTO_GEN_STEM: - path = os.path.join(walk_root, '__init__.py') - open(path, 'a').close() - - class BuildProjectMetadata(setuptools.Command): """Command to generate project metadata in a module.""" @@ -225,10 +156,6 @@ class BuildPy(build_py.build_py): """Custom project build command.""" def run(self): - try: - self.run_command('build_proto_modules') - except CommandError as error: - sys.stderr.write('warning: %s\n' % error.message) self.run_command('build_project_metadata') build_py.build_py.run(self) @@ -281,76 +208,3 @@ class Gather(setuptools.Command): self.distribution.fetch_build_eggs(self.distribution.install_requires) if self.test and self.distribution.tests_require: self.distribution.fetch_build_eggs(self.distribution.tests_require) - - -class TestLite(setuptools.Command): - """Command to run tests without fetching or building anything.""" - - description = 'run tests without fetching or building anything.' - user_options = [] - - def initialize_options(self): - pass - - def finalize_options(self): - # distutils requires this override. - pass - - def run(self): - self._add_eggs_to_path() - - import tests - loader = tests.Loader() - loader.loadTestsFromNames(['tests']) - runner = tests.Runner() - result = runner.run(loader.suite) - if not result.wasSuccessful(): - sys.exit('Test failure') - - def _add_eggs_to_path(self): - """Fetch install and test requirements""" - self.distribution.fetch_build_eggs(self.distribution.install_requires) - self.distribution.fetch_build_eggs(self.distribution.tests_require) - - -class RunInterop(test.test): - - description = 'run interop test client/server' - user_options = [ - ('args=', 'a', 'pass-thru arguments for the client/server'), - ('client', 'c', 'flag indicating to run the client'), - ('server', 's', 'flag indicating to run the server') - ] - - def initialize_options(self): - self.args = '' - self.client = False - self.server = False - - def finalize_options(self): - if self.client and self.server: - raise DistutilsOptionError('you may only specify one of client or server') - - def run(self): - if self.distribution.install_requires: - self.distribution.fetch_build_eggs(self.distribution.install_requires) - if self.distribution.tests_require: - self.distribution.fetch_build_eggs(self.distribution.tests_require) - if self.client: - self.run_client() - elif self.server: - self.run_server() - - def run_server(self): - # We import here to ensure that our setuptools parent has had a chance to - # edit the Python system path. - from tests.interop import server - sys.argv[1:] = self.args.split() - server.serve() - - def run_client(self): - # We import here to ensure that our setuptools parent has had a chance to - # edit the Python system path. - from tests.interop import client - sys.argv[1:] = self.args.split() - client.test_interoperability() diff --git a/src/python/grpcio_health_checking/health_commands.py b/src/python/grpcio_health_checking/health_commands.py index 631066f3310..a7a59f6974b 100644 --- a/src/python/grpcio_health_checking/health_commands.py +++ b/src/python/grpcio_health_checking/health_commands.py @@ -39,51 +39,17 @@ import sys import setuptools from setuptools.command import build_py -from setuptools.command import sdist ROOT_DIR = os.path.abspath(os.path.dirname(os.path.abspath(__file__))) HEALTH_PROTO = os.path.join(ROOT_DIR, '../../proto/grpc/health/v1/health.proto') -class BuildProtoModules(setuptools.Command): - """Command to generate project *_pb2.py modules from proto files.""" +class CopyProtoModules(setuptools.Command): + """Command to copy proto modules from grpc/src/proto.""" description = '' user_options = [] - def initialize_options(self): - pass - - def finalize_options(self): - self.protoc_command = distutils.spawn.find_executable('protoc') - self.grpc_python_plugin_command = distutils.spawn.find_executable( - 'grpc_python_plugin') - - def run(self): - paths = [] - root_directory = os.getcwd() - for walk_root, directories, filenames in os.walk(root_directory): - for filename in filenames: - if filename.endswith('.proto'): - paths.append(os.path.join(walk_root, filename)) - command = [ - self.protoc_command, - '--plugin=protoc-gen-python-grpc={}'.format( - self.grpc_python_plugin_command), - '-I {}'.format(root_directory), - '--python_out={}'.format(root_directory), - '--python-grpc_out={}'.format(root_directory), - ] + paths - try: - subprocess.check_output(' '.join(command), cwd=root_directory, shell=True, - stderr=subprocess.STDOUT) - except subprocess.CalledProcessError as e: - raise Exception('{}\nOutput:\n{}'.format(e.message, e.output)) - - -class CopyProtoModules(setuptools.Command): - """Command to copy proto modules from grpc/src/proto.""" - def initialize_options(self): pass @@ -101,14 +67,5 @@ class BuildPy(build_py.build_py): """Custom project build command.""" def run(self): - self.run_command('copy_proto_modules') self.run_command('build_proto_modules') build_py.build_py.run(self) - - -class SDist(sdist.sdist): - """Custom project build command.""" - - def run(self): - self.run_command('copy_proto_modules') - sdist.sdist.run(self) diff --git a/src/python/grpcio_health_checking/setup.py b/src/python/grpcio_health_checking/setup.py index d68a7ced8ed..70b4575bf5d 100644 --- a/src/python/grpcio_health_checking/setup.py +++ b/src/python/grpcio_health_checking/setup.py @@ -36,36 +36,44 @@ import sys from distutils import core as _core import setuptools +import grpc.tools.command + # Ensure we're in the proper directory whether or not we're being used by pip. os.chdir(os.path.dirname(os.path.abspath(__file__))) # Break import-style to ensure we can actually find our commands module. import health_commands -_PACKAGES = ( +PACKAGES = ( setuptools.find_packages('.') ) -_PACKAGE_DIRECTORIES = { +PACKAGE_DIRECTORIES = { '': '.', } -_INSTALL_REQUIRES = ( +SETUP_REQUIRES = ( + 'grpcio-tools>=0.14.0', +) + +INSTALL_REQUIRES = ( 'grpcio>=0.13.1', ) -_COMMAND_CLASS = { - 'copy_proto_modules': health_commands.CopyProtoModules, - 'build_proto_modules': health_commands.BuildProtoModules, +COMMAND_CLASS = { + # Run preprocess from the repository *before* doing any packaging! + 'preprocess': health_commands.CopyProtoModules, + + 'build_proto_modules': grpc.tools.command.BuildProtoModules, 'build_py': health_commands.BuildPy, - 'sdist': health_commands.SDist, } setuptools.setup( - name='grpcio_health_checking', - version='0.14.0b0', - packages=list(_PACKAGES), - package_dir=_PACKAGE_DIRECTORIES, - install_requires=_INSTALL_REQUIRES, - cmdclass=_COMMAND_CLASS + name='grpcio-health-checking', + version='0.14.0', + packages=list(PACKAGES), + package_dir=PACKAGE_DIRECTORIES, + install_requires=INSTALL_REQUIRES, + setup_requires=SETUP_REQUIRES, + cmdclass=COMMAND_CLASS ) diff --git a/src/python/grpcio_tests/.gitignore b/src/python/grpcio_tests/.gitignore new file mode 100644 index 00000000000..fc620135dc7 --- /dev/null +++ b/src/python/grpcio_tests/.gitignore @@ -0,0 +1,4 @@ +proto/ +src/ +*_pb2.py +*.egg-info/ diff --git a/src/python/grpcio_tests/commands.py b/src/python/grpcio_tests/commands.py new file mode 100644 index 00000000000..171829b62fc --- /dev/null +++ b/src/python/grpcio_tests/commands.py @@ -0,0 +1,217 @@ +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +"""Provides distutils command classes for the gRPC Python setup process.""" + +import distutils +import glob +import os +import os.path +import platform +import re +import shutil +import subprocess +import sys +import traceback + +import setuptools +from setuptools.command import build_ext +from setuptools.command import build_py +from setuptools.command import easy_install +from setuptools.command import install +from setuptools.command import test + +PYTHON_STEM = os.path.dirname(os.path.abspath(__file__)) +GRPC_STEM = os.path.abspath(PYTHON_STEM + '../../../../') +GRPC_PROTO_STEM = os.path.join(GRPC_STEM, 'src', 'proto') +PROTO_STEM = os.path.join(PYTHON_STEM, 'src', 'proto') +PYTHON_PROTO_TOP_LEVEL = os.path.join(PYTHON_STEM, 'src') + + +class CommandError(object): + pass + + +class GatherProto(setuptools.Command): + + description = 'gather proto dependencies' + user_options = [] + + def initialize_options(self): + pass + + def finalize_options(self): + pass + + def run(self): + # TODO(atash) ensure that we're running from the repository directory when + # this command is used + try: + shutil.rmtree(PROTO_STEM) + except Exception as error: + # We don't care if this command fails + pass + shutil.copytree(GRPC_PROTO_STEM, PROTO_STEM) + for root, _, _ in os.walk(PYTHON_PROTO_TOP_LEVEL): + path = os.path.join(root, '__init__.py') + open(path, 'a').close() + + +class BuildProtoModules(setuptools.Command): + """Command to generate project *_pb2.py modules from proto files.""" + + description = 'build protobuf modules' + user_options = [ + ('include=', None, 'path patterns to include in protobuf generation'), + ('exclude=', None, 'path patterns to exclude from protobuf generation') + ] + + def initialize_options(self): + self.exclude = None + self.include = r'.*\.proto$' + + def finalize_options(self): + pass + + def run(self): + import grpc.tools.protoc as protoc + + include_regex = re.compile(self.include) + exclude_regex = re.compile(self.exclude) if self.exclude else None + paths = [] + for walk_root, directories, filenames in os.walk(PROTO_STEM): + for filename in filenames: + path = os.path.join(walk_root, filename) + if include_regex.match(path) and not ( + exclude_regex and exclude_regex.match(path)): + paths.append(path) + + # TODO(kpayson): It would be nice to do this in a batch command, + # but we currently have name conflicts in src/proto + for path in paths: + command = [ + 'grpc.tools.protoc', + '-I {}'.format(PROTO_STEM), + '--python_out={}'.format(PROTO_STEM), + '--grpc_python_out={}'.format(PROTO_STEM), + ] + [path] + if protoc.main(command) != 0: + sys.stderr.write( + 'warning: Command:\n{}\nFailed'.format( + command)) + + # Generated proto directories dont include __init__.py, but + # these are needed for python package resolution + for walk_root, _, _ in os.walk(PROTO_STEM): + path = os.path.join(walk_root, '__init__.py') + open(path, 'a').close() + + +class BuildPy(build_py.build_py): + """Custom project build command.""" + + def run(self): + try: + self.run_command('build_proto_modules') + except CommandError as error: + sys.stderr.write('warning: %s\n' % error.message) + build_py.build_py.run(self) + + +class TestLite(setuptools.Command): + """Command to run tests without fetching or building anything.""" + + description = 'run tests without fetching or building anything.' + user_options = [] + + def initialize_options(self): + pass + + def finalize_options(self): + # distutils requires this override. + pass + + def run(self): + self._add_eggs_to_path() + + import tests + loader = tests.Loader() + loader.loadTestsFromNames(['tests']) + runner = tests.Runner() + result = runner.run(loader.suite) + if not result.wasSuccessful(): + sys.exit('Test failure') + + def _add_eggs_to_path(self): + """Fetch install and test requirements""" + self.distribution.fetch_build_eggs(self.distribution.install_requires) + self.distribution.fetch_build_eggs(self.distribution.tests_require) + + +class RunInterop(test.test): + + description = 'run interop test client/server' + user_options = [ + ('args=', 'a', 'pass-thru arguments for the client/server'), + ('client', 'c', 'flag indicating to run the client'), + ('server', 's', 'flag indicating to run the server') + ] + + def initialize_options(self): + self.args = '' + self.client = False + self.server = False + + def finalize_options(self): + if self.client and self.server: + raise DistutilsOptionError('you may only specify one of client or server') + + def run(self): + if self.distribution.install_requires: + self.distribution.fetch_build_eggs(self.distribution.install_requires) + if self.distribution.tests_require: + self.distribution.fetch_build_eggs(self.distribution.tests_require) + if self.client: + self.run_client() + elif self.server: + self.run_server() + + def run_server(self): + # We import here to ensure that our setuptools parent has had a chance to + # edit the Python system path. + from tests.interop import server + sys.argv[1:] = self.args.split() + server.serve() + + def run_client(self): + # We import here to ensure that our setuptools parent has had a chance to + # edit the Python system path. + from tests.interop import client + sys.argv[1:] = self.args.split() + client.test_interoperability() diff --git a/src/python/grpcio_tests/grpc_version.py b/src/python/grpcio_tests/grpc_version.py new file mode 100644 index 00000000000..478b5d62d6a --- /dev/null +++ b/src/python/grpcio_tests/grpc_version.py @@ -0,0 +1,32 @@ +# Copyright 2016, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# AUTO-GENERATED FROM `$REPO_ROOT/templates/src/python/grpcio_tests/grpc_version.py.template`!!! + +VERSION='0.15.0.dev0' diff --git a/src/python/grpcio_tests/setup.py b/src/python/grpcio_tests/setup.py new file mode 100644 index 00000000000..7eef420bdb0 --- /dev/null +++ b/src/python/grpcio_tests/setup.py @@ -0,0 +1,124 @@ +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +"""A setup module for the gRPC Python package.""" + +import os +import os.path +import shutil +import sys + +from distutils import core as _core +from distutils import extension as _extension +import setuptools +from setuptools.command import egg_info + +import grpc.tools.command + +PY3 = sys.version_info.major == 3 + +# Ensure we're in the proper directory whether or not we're being used by pip. +os.chdir(os.path.dirname(os.path.abspath(__file__))) + +# Break import-style to ensure we can actually find our in-repo dependencies. +import commands +import grpc_version + +LICENSE = '3-clause BSD' + +PACKAGE_DIRECTORIES = { + '': '.', +} + +INSTALL_REQUIRES = ( + 'coverage>=4.0', + 'enum34>=1.0.4', + 'futures>=2.2.0', + 'grpcio>=0.14.0', + 'grpcio-health-checking>=0.14.0', + 'oauth2client>=1.4.7', + 'protobuf>=3.0.0a3', + 'six>=1.10', +) + +SETUP_REQUIRES = ( + 'grpcio-tools>=0.14.0', +) + +COMMAND_CLASS = { + # Run `preprocess` *before* doing any packaging! + 'preprocess': commands.GatherProto, + + 'build_proto_modules': grpc.tools.command.BuildProtoModules, + 'build_py': commands.BuildPy, + 'run_interop': commands.RunInterop, + 'test_lite': commands.TestLite +} + +PACKAGE_DATA = { + 'tests.interop': [ + 'credentials/ca.pem', + 'credentials/server1.key', + 'credentials/server1.pem', + ], + 'tests.protoc_plugin': [ + 'protoc_plugin_test.proto', + ], + 'tests.unit': [ + 'credentials/ca.pem', + 'credentials/server1.key', + 'credentials/server1.pem', + ], + 'tests': [ + 'tests.json' + ], +} + +TEST_SUITE = 'tests' +TEST_LOADER = 'tests:Loader' +TEST_RUNNER = 'tests:Runner' +TESTS_REQUIRE = INSTALL_REQUIRES + +PACKAGES = setuptools.find_packages('.') + +setuptools.setup( + name='grpcio-tests', + version=grpc_version.VERSION, + license=LICENSE, + packages=list(PACKAGES), + package_dir=PACKAGE_DIRECTORIES, + package_data=PACKAGE_DATA, + install_requires=INSTALL_REQUIRES, + setup_requires=SETUP_REQUIRES, + cmdclass=COMMAND_CLASS, + tests_require=TESTS_REQUIRE, + test_suite=TEST_SUITE, + test_loader=TEST_LOADER, + test_runner=TEST_RUNNER, +) diff --git a/src/python/grpcio/tests/__init__.py b/src/python/grpcio_tests/tests/__init__.py similarity index 100% rename from src/python/grpcio/tests/__init__.py rename to src/python/grpcio_tests/tests/__init__.py diff --git a/src/python/grpcio/tests/_loader.py b/src/python/grpcio_tests/tests/_loader.py similarity index 100% rename from src/python/grpcio/tests/_loader.py rename to src/python/grpcio_tests/tests/_loader.py diff --git a/src/python/grpcio/tests/_result.py b/src/python/grpcio_tests/tests/_result.py similarity index 100% rename from src/python/grpcio/tests/_result.py rename to src/python/grpcio_tests/tests/_result.py diff --git a/src/python/grpcio/tests/_runner.py b/src/python/grpcio_tests/tests/_runner.py similarity index 100% rename from src/python/grpcio/tests/_runner.py rename to src/python/grpcio_tests/tests/_runner.py diff --git a/src/python/grpcio/tests/health_check/__init__.py b/src/python/grpcio_tests/tests/health_check/__init__.py similarity index 100% rename from src/python/grpcio/tests/health_check/__init__.py rename to src/python/grpcio_tests/tests/health_check/__init__.py diff --git a/src/python/grpcio/tests/health_check/_health_servicer_test.py b/src/python/grpcio_tests/tests/health_check/_health_servicer_test.py similarity index 100% rename from src/python/grpcio/tests/health_check/_health_servicer_test.py rename to src/python/grpcio_tests/tests/health_check/_health_servicer_test.py diff --git a/src/python/grpcio/tests/interop/__init__.py b/src/python/grpcio_tests/tests/interop/__init__.py similarity index 100% rename from src/python/grpcio/tests/interop/__init__.py rename to src/python/grpcio_tests/tests/interop/__init__.py diff --git a/src/python/grpcio/tests/interop/_insecure_interop_test.py b/src/python/grpcio_tests/tests/interop/_insecure_interop_test.py similarity index 100% rename from src/python/grpcio/tests/interop/_insecure_interop_test.py rename to src/python/grpcio_tests/tests/interop/_insecure_interop_test.py diff --git a/src/python/grpcio/tests/interop/_interop_test_case.py b/src/python/grpcio_tests/tests/interop/_interop_test_case.py similarity index 100% rename from src/python/grpcio/tests/interop/_interop_test_case.py rename to src/python/grpcio_tests/tests/interop/_interop_test_case.py diff --git a/src/python/grpcio/tests/interop/_secure_interop_test.py b/src/python/grpcio_tests/tests/interop/_secure_interop_test.py similarity index 100% rename from src/python/grpcio/tests/interop/_secure_interop_test.py rename to src/python/grpcio_tests/tests/interop/_secure_interop_test.py diff --git a/src/python/grpcio/tests/interop/client.py b/src/python/grpcio_tests/tests/interop/client.py similarity index 100% rename from src/python/grpcio/tests/interop/client.py rename to src/python/grpcio_tests/tests/interop/client.py diff --git a/src/python/grpcio/tests/interop/credentials/README b/src/python/grpcio_tests/tests/interop/credentials/README similarity index 100% rename from src/python/grpcio/tests/interop/credentials/README rename to src/python/grpcio_tests/tests/interop/credentials/README diff --git a/src/python/grpcio/tests/interop/credentials/ca.pem b/src/python/grpcio_tests/tests/interop/credentials/ca.pem similarity index 100% rename from src/python/grpcio/tests/interop/credentials/ca.pem rename to src/python/grpcio_tests/tests/interop/credentials/ca.pem diff --git a/src/python/grpcio/tests/interop/credentials/server1.key b/src/python/grpcio_tests/tests/interop/credentials/server1.key similarity index 100% rename from src/python/grpcio/tests/interop/credentials/server1.key rename to src/python/grpcio_tests/tests/interop/credentials/server1.key diff --git a/src/python/grpcio/tests/interop/credentials/server1.pem b/src/python/grpcio_tests/tests/interop/credentials/server1.pem similarity index 100% rename from src/python/grpcio/tests/interop/credentials/server1.pem rename to src/python/grpcio_tests/tests/interop/credentials/server1.pem diff --git a/src/python/grpcio/tests/interop/methods.py b/src/python/grpcio_tests/tests/interop/methods.py similarity index 100% rename from src/python/grpcio/tests/interop/methods.py rename to src/python/grpcio_tests/tests/interop/methods.py diff --git a/src/python/grpcio/tests/interop/resources.py b/src/python/grpcio_tests/tests/interop/resources.py similarity index 100% rename from src/python/grpcio/tests/interop/resources.py rename to src/python/grpcio_tests/tests/interop/resources.py diff --git a/src/python/grpcio/tests/interop/server.py b/src/python/grpcio_tests/tests/interop/server.py similarity index 100% rename from src/python/grpcio/tests/interop/server.py rename to src/python/grpcio_tests/tests/interop/server.py diff --git a/src/python/grpcio/tests/protoc_plugin/__init__.py b/src/python/grpcio_tests/tests/protoc_plugin/__init__.py similarity index 100% rename from src/python/grpcio/tests/protoc_plugin/__init__.py rename to src/python/grpcio_tests/tests/protoc_plugin/__init__.py diff --git a/src/python/grpcio/tests/protoc_plugin/_python_plugin_test.py b/src/python/grpcio_tests/tests/protoc_plugin/_python_plugin_test.py similarity index 67% rename from src/python/grpcio/tests/protoc_plugin/_python_plugin_test.py rename to src/python/grpcio_tests/tests/protoc_plugin/_python_plugin_test.py index 1c9cbb0d0c3..bf09380c85b 100644 --- a/src/python/grpcio/tests/protoc_plugin/_python_plugin_test.py +++ b/src/python/grpcio_tests/tests/protoc_plugin/_python_plugin_test.py @@ -45,6 +45,11 @@ from six import moves import grpc from tests.unit.framework.common import test_constants +import tests.protoc_plugin.protos.payload.test_payload_pb2 as payload_pb2 +import tests.protoc_plugin.protos.requests.r.test_requests_pb2 as request_pb2 +import tests.protoc_plugin.protos.responses.test_responses_pb2 as response_pb2 +import tests.protoc_plugin.protos.service.test_service_pb2 as service_pb2 + # Identifiers of entities we expect to find in the generated module. STUB_IDENTIFIER = 'TestServiceStub' SERVICER_IDENTIFIER = 'TestServiceServicer' @@ -53,12 +58,10 @@ ADD_SERVICER_TO_SERVER_IDENTIFIER = 'add_TestServiceServicer_to_server' class _ServicerMethods(object): - def __init__(self, response_pb2, payload_pb2): + def __init__(self): self._condition = threading.Condition() self._paused = False self._fail = False - self._response_pb2 = response_pb2 - self._payload_pb2 = payload_pb2 @contextlib.contextmanager def pause(self): # pylint: disable=invalid-name @@ -85,22 +88,22 @@ class _ServicerMethods(object): self._condition.wait() def UnaryCall(self, request, unused_rpc_context): - response = self._response_pb2.SimpleResponse() - response.payload.payload_type = self._payload_pb2.COMPRESSABLE + response = response_pb2.SimpleResponse() + response.payload.payload_type = payload_pb2.COMPRESSABLE response.payload.payload_compressable = 'a' * request.response_size self._control() return response def StreamingOutputCall(self, request, unused_rpc_context): for parameter in request.response_parameters: - response = self._response_pb2.StreamingOutputCallResponse() - response.payload.payload_type = self._payload_pb2.COMPRESSABLE + response = response_pb2.StreamingOutputCallResponse() + response.payload.payload_type = payload_pb2.COMPRESSABLE response.payload.payload_compressable = 'a' * parameter.size self._control() yield response def StreamingInputCall(self, request_iter, unused_rpc_context): - response = self._response_pb2.StreamingInputCallResponse() + response = response_pb2.StreamingInputCallResponse() aggregated_payload_size = 0 for request in request_iter: aggregated_payload_size += len(request.payload.payload_compressable) @@ -111,8 +114,8 @@ class _ServicerMethods(object): def FullDuplexCall(self, request_iter, unused_rpc_context): for request in request_iter: for parameter in request.response_parameters: - response = self._response_pb2.StreamingOutputCallResponse() - response.payload.payload_type = self._payload_pb2.COMPRESSABLE + response = response_pb2.StreamingOutputCallResponse() + response.payload.payload_type = payload_pb2.COMPRESSABLE response.payload.payload_compressable = 'a' * parameter.size self._control() yield response @@ -121,8 +124,8 @@ class _ServicerMethods(object): responses = [] for request in request_iter: for parameter in request.response_parameters: - response = self._response_pb2.StreamingOutputCallResponse() - response.payload.payload_type = self._payload_pb2.COMPRESSABLE + response = response_pb2.StreamingOutputCallResponse() + response.payload.payload_type = payload_pb2.COMPRESSABLE response.payload.payload_compressable = 'a' * parameter.size self._control() responses.append(response) @@ -142,18 +145,13 @@ class _Service( """ -def _CreateService(service_pb2, response_pb2, payload_pb2): +def _CreateService(): """Provides a servicer backend and a stub. - Args: - service_pb2: The service_pb2 module generated by this test. - response_pb2: The response_pb2 module generated by this test. - payload_pb2: The payload_pb2 module generated by this test. - Returns: A _Service with which to test RPCs. """ - servicer_methods = _ServicerMethods(response_pb2, payload_pb2) + servicer_methods = _ServicerMethods() class Servicer(getattr(service_pb2, SERVICER_IDENTIFIER)): @@ -182,12 +180,9 @@ def _CreateService(service_pb2, response_pb2, payload_pb2): return _Service(servicer_methods, server, stub) -def _CreateIncompleteService(service_pb2): +def _CreateIncompleteService(): """Provides a servicer backend that fails to implement methods and its stub. - Args: - service_pb2: The service_pb2 module generated by this test. - Returns: A _Service with which to test RPCs. The returned _Service's servicer_methods implements none of the methods required of it. @@ -206,7 +201,7 @@ def _CreateIncompleteService(service_pb2): return _Service(None, server, stub) -def _streaming_input_request_iterator(request_pb2, payload_pb2): +def _streaming_input_request_iterator(): for _ in range(3): request = request_pb2.StreamingInputCallRequest() request.payload.payload_type = payload_pb2.COMPRESSABLE @@ -214,7 +209,7 @@ def _streaming_input_request_iterator(request_pb2, payload_pb2): yield request -def _streaming_output_request(request_pb2): +def _streaming_output_request(): request = request_pb2.StreamingOutputCallRequest() sizes = [1, 2, 3] request.response_parameters.add(size=sizes[0], interval_us=0) @@ -223,7 +218,7 @@ def _streaming_output_request(request_pb2): return request -def _full_duplex_request_iterator(request_pb2): +def _full_duplex_request_iterator(): request = request_pb2.StreamingOutputCallRequest() request.response_parameters.add(size=1, interval_us=0) yield request @@ -241,102 +236,40 @@ class PythonPluginTest(unittest.TestCase): methods and does not exist for response-streaming methods. """ - def setUp(self): - # Assume that the appropriate protoc and grpc_python_plugins are on the - # path. - protoc_command = 'protoc' - protoc_plugin_filename = distutils.spawn.find_executable( - 'grpc_python_plugin') - if not os.path.isfile(protoc_command): - # Assume that if we haven't built protoc that it's on the system. - protoc_command = 'protoc' - - # Ensure that the output directory exists. - self.outdir = tempfile.mkdtemp() - - # Find all proto files - paths = [] - root_dir = os.path.dirname(os.path.realpath(__file__)) - proto_dir = os.path.join(root_dir, 'protos') - for walk_root, _, filenames in os.walk(proto_dir): - for filename in filenames: - if filename.endswith('.proto'): - path = os.path.join(walk_root, filename) - paths.append(path) - - # Invoke protoc with the plugin. - cmd = [ - protoc_command, - '--plugin=protoc-gen-python-grpc=%s' % protoc_plugin_filename, - '-I %s' % root_dir, - '--python_out=%s' % self.outdir, - '--python-grpc_out=%s' % self.outdir - ] + paths - subprocess.check_call(' '.join(cmd), shell=True, env=os.environ, - cwd=os.path.dirname(os.path.realpath(__file__))) - - # Generated proto directories dont include __init__.py, but - # these are needed for python package resolution - for walk_root, _, _ in os.walk(os.path.join(self.outdir, 'protos')): - path = os.path.join(walk_root, '__init__.py') - open(path, 'a').close() - - sys.path.insert(0, self.outdir) - - import protos.payload.test_payload_pb2 as payload_pb2 - import protos.requests.r.test_requests_pb2 as request_pb2 - import protos.responses.test_responses_pb2 as response_pb2 - import protos.service.test_service_pb2 as service_pb2 - self._payload_pb2 = payload_pb2 - self._request_pb2 = request_pb2 - self._response_pb2 = response_pb2 - self._service_pb2 = service_pb2 - - def tearDown(self): - try: - shutil.rmtree(self.outdir) - except OSError as exc: - if exc.errno != errno.ENOENT: - raise - sys.path.remove(self.outdir) - def testImportAttributes(self): # check that we can access the generated module and its members. self.assertIsNotNone( - getattr(self._service_pb2, STUB_IDENTIFIER, None)) + getattr(service_pb2, STUB_IDENTIFIER, None)) self.assertIsNotNone( - getattr(self._service_pb2, SERVICER_IDENTIFIER, None)) + getattr(service_pb2, SERVICER_IDENTIFIER, None)) self.assertIsNotNone( - getattr(self._service_pb2, ADD_SERVICER_TO_SERVER_IDENTIFIER, None)) + getattr(service_pb2, ADD_SERVICER_TO_SERVER_IDENTIFIER, None)) def testUpDown(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) + service = _CreateService() self.assertIsNotNone(service.servicer_methods) self.assertIsNotNone(service.server) self.assertIsNotNone(service.stub) def testIncompleteServicer(self): - service = _CreateIncompleteService(self._service_pb2) - request = self._request_pb2.SimpleRequest(response_size=13) + service = _CreateIncompleteService() + request = request_pb2.SimpleRequest(response_size=13) with self.assertRaises(grpc.RpcError) as exception_context: service.stub.UnaryCall(request) self.assertIs( exception_context.exception.code(), grpc.StatusCode.UNIMPLEMENTED) def testUnaryCall(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) - request = self._request_pb2.SimpleRequest(response_size=13) + service = _CreateService() + request = request_pb2.SimpleRequest(response_size=13) response = service.stub.UnaryCall(request) expected_response = service.servicer_methods.UnaryCall( request, 'not a real context!') self.assertEqual(expected_response, response) def testUnaryCallFuture(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) - request = self._request_pb2.SimpleRequest(response_size=13) + service = _CreateService() + request = request_pb2.SimpleRequest(response_size=13) # Check that the call does not block waiting for the server to respond. with service.servicer_methods.pause(): response_future = service.stub.UnaryCall.future(request) @@ -346,9 +279,8 @@ class PythonPluginTest(unittest.TestCase): self.assertEqual(expected_response, response) def testUnaryCallFutureExpired(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) - request = self._request_pb2.SimpleRequest(response_size=13) + service = _CreateService() + request = request_pb2.SimpleRequest(response_size=13) with service.servicer_methods.pause(): response_future = service.stub.UnaryCall.future( request, timeout=test_constants.SHORT_TIMEOUT) @@ -359,9 +291,8 @@ class PythonPluginTest(unittest.TestCase): self.assertIs(response_future.code(), grpc.StatusCode.DEADLINE_EXCEEDED) def testUnaryCallFutureCancelled(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) - request = self._request_pb2.SimpleRequest(response_size=13) + service = _CreateService() + request = request_pb2.SimpleRequest(response_size=13) with service.servicer_methods.pause(): response_future = service.stub.UnaryCall.future(request) response_future.cancel() @@ -369,18 +300,16 @@ class PythonPluginTest(unittest.TestCase): self.assertIs(response_future.code(), grpc.StatusCode.CANCELLED) def testUnaryCallFutureFailed(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) - request = self._request_pb2.SimpleRequest(response_size=13) + service = _CreateService() + request = request_pb2.SimpleRequest(response_size=13) with service.servicer_methods.fail(): response_future = service.stub.UnaryCall.future(request) self.assertIsNotNone(response_future.exception()) self.assertIs(response_future.code(), grpc.StatusCode.UNKNOWN) def testStreamingOutputCall(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) - request = _streaming_output_request(self._request_pb2) + service = _CreateService() + request = _streaming_output_request() responses = service.stub.StreamingOutputCall(request) expected_responses = service.servicer_methods.StreamingOutputCall( request, 'not a real RpcContext!') @@ -389,9 +318,8 @@ class PythonPluginTest(unittest.TestCase): self.assertEqual(expected_response, response) def testStreamingOutputCallExpired(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) - request = _streaming_output_request(self._request_pb2) + service = _CreateService() + request = _streaming_output_request() with service.servicer_methods.pause(): responses = service.stub.StreamingOutputCall( request, timeout=test_constants.SHORT_TIMEOUT) @@ -401,9 +329,8 @@ class PythonPluginTest(unittest.TestCase): exception_context.exception.code(), grpc.StatusCode.DEADLINE_EXCEEDED) def testStreamingOutputCallCancelled(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) - request = _streaming_output_request(self._request_pb2) + service = _CreateService() + request = _streaming_output_request() responses = service.stub.StreamingOutputCall(request) next(responses) responses.cancel() @@ -412,9 +339,8 @@ class PythonPluginTest(unittest.TestCase): self.assertIs(responses.code(), grpc.StatusCode.CANCELLED) def testStreamingOutputCallFailed(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) - request = _streaming_output_request(self._request_pb2) + service = _CreateService() + request = _streaming_output_request() with service.servicer_methods.fail(): responses = service.stub.StreamingOutputCall(request) self.assertIsNotNone(responses) @@ -423,36 +349,30 @@ class PythonPluginTest(unittest.TestCase): self.assertIs(exception_context.exception.code(), grpc.StatusCode.UNKNOWN) def testStreamingInputCall(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) + service = _CreateService() response = service.stub.StreamingInputCall( - _streaming_input_request_iterator( - self._request_pb2, self._payload_pb2)) + _streaming_input_request_iterator()) expected_response = service.servicer_methods.StreamingInputCall( - _streaming_input_request_iterator(self._request_pb2, self._payload_pb2), + _streaming_input_request_iterator(), 'not a real RpcContext!') self.assertEqual(expected_response, response) def testStreamingInputCallFuture(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) + service = _CreateService() with service.servicer_methods.pause(): response_future = service.stub.StreamingInputCall.future( - _streaming_input_request_iterator( - self._request_pb2, self._payload_pb2)) + _streaming_input_request_iterator()) response = response_future.result() expected_response = service.servicer_methods.StreamingInputCall( - _streaming_input_request_iterator(self._request_pb2, self._payload_pb2), + _streaming_input_request_iterator(), 'not a real RpcContext!') self.assertEqual(expected_response, response) def testStreamingInputCallFutureExpired(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) + service = _CreateService() with service.servicer_methods.pause(): response_future = service.stub.StreamingInputCall.future( - _streaming_input_request_iterator( - self._request_pb2, self._payload_pb2), + _streaming_input_request_iterator(), timeout=test_constants.SHORT_TIMEOUT) with self.assertRaises(grpc.RpcError) as exception_context: response_future.result() @@ -463,43 +383,37 @@ class PythonPluginTest(unittest.TestCase): exception_context.exception.code(), grpc.StatusCode.DEADLINE_EXCEEDED) def testStreamingInputCallFutureCancelled(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) + service = _CreateService() with service.servicer_methods.pause(): response_future = service.stub.StreamingInputCall.future( - _streaming_input_request_iterator( - self._request_pb2, self._payload_pb2)) + _streaming_input_request_iterator()) response_future.cancel() self.assertTrue(response_future.cancelled()) with self.assertRaises(grpc.FutureCancelledError): response_future.result() def testStreamingInputCallFutureFailed(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) + service = _CreateService() with service.servicer_methods.fail(): response_future = service.stub.StreamingInputCall.future( - _streaming_input_request_iterator( - self._request_pb2, self._payload_pb2)) + _streaming_input_request_iterator()) self.assertIsNotNone(response_future.exception()) self.assertIs(response_future.code(), grpc.StatusCode.UNKNOWN) def testFullDuplexCall(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) + service = _CreateService() responses = service.stub.FullDuplexCall( - _full_duplex_request_iterator(self._request_pb2)) + _full_duplex_request_iterator()) expected_responses = service.servicer_methods.FullDuplexCall( - _full_duplex_request_iterator(self._request_pb2), + _full_duplex_request_iterator(), 'not a real RpcContext!') for expected_response, response in moves.zip_longest( expected_responses, responses): self.assertEqual(expected_response, response) def testFullDuplexCallExpired(self): - request_iterator = _full_duplex_request_iterator(self._request_pb2) - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) + request_iterator = _full_duplex_request_iterator() + service = _CreateService() with service.servicer_methods.pause(): responses = service.stub.FullDuplexCall( request_iterator, timeout=test_constants.SHORT_TIMEOUT) @@ -509,9 +423,8 @@ class PythonPluginTest(unittest.TestCase): exception_context.exception.code(), grpc.StatusCode.DEADLINE_EXCEEDED) def testFullDuplexCallCancelled(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) - request_iterator = _full_duplex_request_iterator(self._request_pb2) + service = _CreateService() + request_iterator = _full_duplex_request_iterator() responses = service.stub.FullDuplexCall(request_iterator) next(responses) responses.cancel() @@ -521,9 +434,8 @@ class PythonPluginTest(unittest.TestCase): exception_context.exception.code(), grpc.StatusCode.CANCELLED) def testFullDuplexCallFailed(self): - request_iterator = _full_duplex_request_iterator(self._request_pb2) - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) + request_iterator = _full_duplex_request_iterator() + service = _CreateService() with service.servicer_methods.fail(): responses = service.stub.FullDuplexCall(request_iterator) with self.assertRaises(grpc.RpcError) as exception_context: @@ -531,13 +443,12 @@ class PythonPluginTest(unittest.TestCase): self.assertIs(exception_context.exception.code(), grpc.StatusCode.UNKNOWN) def testHalfDuplexCall(self): - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) + service = _CreateService() def half_duplex_request_iterator(): - request = self._request_pb2.StreamingOutputCallRequest() + request = request_pb2.StreamingOutputCallRequest() request.response_parameters.add(size=1, interval_us=0) yield request - request = self._request_pb2.StreamingOutputCallRequest() + request = request_pb2.StreamingOutputCallRequest() request.response_parameters.add(size=2, interval_us=0) request.response_parameters.add(size=3, interval_us=0) yield request @@ -561,14 +472,13 @@ class PythonPluginTest(unittest.TestCase): wait_cell[0] = False condition.notify_all() def half_duplex_request_iterator(): - request = self._request_pb2.StreamingOutputCallRequest() + request = request_pb2.StreamingOutputCallRequest() request.response_parameters.add(size=1, interval_us=0) yield request with condition: while wait_cell[0]: condition.wait() - service = _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2) + service = _CreateService() with wait(): responses = service.stub.HalfDuplexCall( half_duplex_request_iterator(), timeout=test_constants.SHORT_TIMEOUT) diff --git a/src/python/grpcio/tests/protoc_plugin/beta_python_plugin_test.py b/src/python/grpcio_tests/tests/protoc_plugin/beta_python_plugin_test.py similarity index 64% rename from src/python/grpcio/tests/protoc_plugin/beta_python_plugin_test.py rename to src/python/grpcio_tests/tests/protoc_plugin/beta_python_plugin_test.py index 7466f880597..1eba9c93543 100644 --- a/src/python/grpcio/tests/protoc_plugin/beta_python_plugin_test.py +++ b/src/python/grpcio_tests/tests/protoc_plugin/beta_python_plugin_test.py @@ -50,6 +50,11 @@ from grpc.framework.foundation import future from grpc.framework.interfaces.face import face from tests.unit.framework.common import test_constants +import tests.protoc_plugin.protos.payload.test_payload_pb2 as payload_pb2 +import tests.protoc_plugin.protos.requests.r.test_requests_pb2 as request_pb2 +import tests.protoc_plugin.protos.responses.test_responses_pb2 as response_pb2 +import tests.protoc_plugin.protos.service.test_service_pb2 as service_pb2 + # Identifiers of entities we expect to find in the generated module. SERVICER_IDENTIFIER = 'BetaTestServiceServicer' STUB_IDENTIFIER = 'BetaTestServiceStub' @@ -59,12 +64,10 @@ STUB_FACTORY_IDENTIFIER = 'beta_create_TestService_stub' class _ServicerMethods(object): - def __init__(self, response_pb2, payload_pb2): + def __init__(self): self._condition = threading.Condition() self._paused = False self._fail = False - self._response_pb2 = response_pb2 - self._payload_pb2 = payload_pb2 @contextlib.contextmanager def pause(self): # pylint: disable=invalid-name @@ -91,22 +94,22 @@ class _ServicerMethods(object): self._condition.wait() def UnaryCall(self, request, unused_rpc_context): - response = self._response_pb2.SimpleResponse() - response.payload.payload_type = self._payload_pb2.COMPRESSABLE + response = response_pb2.SimpleResponse() + response.payload.payload_type = payload_pb2.COMPRESSABLE response.payload.payload_compressable = 'a' * request.response_size self._control() return response def StreamingOutputCall(self, request, unused_rpc_context): for parameter in request.response_parameters: - response = self._response_pb2.StreamingOutputCallResponse() - response.payload.payload_type = self._payload_pb2.COMPRESSABLE + response = response_pb2.StreamingOutputCallResponse() + response.payload.payload_type = payload_pb2.COMPRESSABLE response.payload.payload_compressable = 'a' * parameter.size self._control() yield response def StreamingInputCall(self, request_iter, unused_rpc_context): - response = self._response_pb2.StreamingInputCallResponse() + response = response_pb2.StreamingInputCallResponse() aggregated_payload_size = 0 for request in request_iter: aggregated_payload_size += len(request.payload.payload_compressable) @@ -117,8 +120,8 @@ class _ServicerMethods(object): def FullDuplexCall(self, request_iter, unused_rpc_context): for request in request_iter: for parameter in request.response_parameters: - response = self._response_pb2.StreamingOutputCallResponse() - response.payload.payload_type = self._payload_pb2.COMPRESSABLE + response = response_pb2.StreamingOutputCallResponse() + response.payload.payload_type = payload_pb2.COMPRESSABLE response.payload.payload_compressable = 'a' * parameter.size self._control() yield response @@ -127,8 +130,8 @@ class _ServicerMethods(object): responses = [] for request in request_iter: for parameter in request.response_parameters: - response = self._response_pb2.StreamingOutputCallResponse() - response.payload.payload_type = self._payload_pb2.COMPRESSABLE + response = response_pb2.StreamingOutputCallResponse() + response.payload.payload_type = payload_pb2.COMPRESSABLE response.payload.payload_compressable = 'a' * parameter.size self._control() responses.append(response) @@ -137,23 +140,18 @@ class _ServicerMethods(object): @contextlib.contextmanager -def _CreateService(service_pb2, response_pb2, payload_pb2): +def _CreateService(): """Provides a servicer backend and a stub. The servicer is just the implementation of the actual servicer passed to the face player of the python RPC implementation; the two are detached. - Args: - service_pb2: The service_pb2 module generated by this test. - response_pb2: The response_pb2 module generated by this test - payload_pb2: The payload_pb2 module generated by this test - Yields: A (servicer_methods, stub) pair where servicer_methods is the back-end of the service bound to the stub and and stub is the stub on which to invoke RPCs. """ - servicer_methods = _ServicerMethods(response_pb2, payload_pb2) + servicer_methods = _ServicerMethods() class Servicer(getattr(service_pb2, SERVICER_IDENTIFIER)): @@ -183,7 +181,7 @@ def _CreateService(service_pb2, response_pb2, payload_pb2): @contextlib.contextmanager -def _CreateIncompleteService(service_pb2): +def _CreateIncompleteService(): """Provides a servicer backend that fails to implement methods and its stub. The servicer is just the implementation of the actual servicer passed to the @@ -209,7 +207,7 @@ def _CreateIncompleteService(service_pb2): server.stop(0) -def _streaming_input_request_iterator(request_pb2, payload_pb2): +def _streaming_input_request_iterator(): for _ in range(3): request = request_pb2.StreamingInputCallRequest() request.payload.payload_type = payload_pb2.COMPRESSABLE @@ -217,7 +215,7 @@ def _streaming_input_request_iterator(request_pb2, payload_pb2): yield request -def _streaming_output_request(request_pb2): +def _streaming_output_request(): request = request_pb2.StreamingOutputCallRequest() sizes = [1, 2, 3] request.response_parameters.add(size=sizes[0], interval_us=0) @@ -226,7 +224,7 @@ def _streaming_output_request(request_pb2): return request -def _full_duplex_request_iterator(request_pb2): +def _full_duplex_request_iterator(): request = request_pb2.StreamingOutputCallRequest() request.response_parameters.add(size=1, interval_us=0) yield request @@ -244,101 +242,39 @@ class PythonPluginTest(unittest.TestCase): methods and does not exist for response-streaming methods. """ - def setUp(self): - # Assume that the appropriate protoc and grpc_python_plugins are on the - # path. - protoc_command = 'protoc' - protoc_plugin_filename = distutils.spawn.find_executable( - 'grpc_python_plugin') - if not os.path.isfile(protoc_command): - # Assume that if we haven't built protoc that it's on the system. - protoc_command = 'protoc' - - # Ensure that the output directory exists. - self.outdir = tempfile.mkdtemp() - - # Find all proto files - paths = [] - root_dir = os.path.dirname(os.path.realpath(__file__)) - proto_dir = os.path.join(root_dir, 'protos') - for walk_root, _, filenames in os.walk(proto_dir): - for filename in filenames: - if filename.endswith('.proto'): - path = os.path.join(walk_root, filename) - paths.append(path) - - # Invoke protoc with the plugin. - cmd = [ - protoc_command, - '--plugin=protoc-gen-python-grpc=%s' % protoc_plugin_filename, - '-I %s' % root_dir, - '--python_out=%s' % self.outdir, - '--python-grpc_out=%s' % self.outdir - ] + paths - subprocess.check_call(' '.join(cmd), shell=True, env=os.environ, - cwd=os.path.dirname(os.path.realpath(__file__))) - - # Generated proto directories dont include __init__.py, but - # these are needed for python package resolution - for walk_root, _, _ in os.walk(os.path.join(self.outdir, 'protos')): - path = os.path.join(walk_root, '__init__.py') - open(path, 'a').close() - - sys.path.insert(0, self.outdir) - - import protos.payload.test_payload_pb2 as payload_pb2 # pylint: disable=g-import-not-at-top - import protos.requests.r.test_requests_pb2 as request_pb2 # pylint: disable=g-import-not-at-top - import protos.responses.test_responses_pb2 as response_pb2 # pylint: disable=g-import-not-at-top - import protos.service.test_service_pb2 as service_pb2 # pylint: disable=g-import-not-at-top - self._payload_pb2 = payload_pb2 - self._request_pb2 = request_pb2 - self._response_pb2 = response_pb2 - self._service_pb2 = service_pb2 - - def tearDown(self): - try: - shutil.rmtree(self.outdir) - except OSError as exc: - if exc.errno != errno.ENOENT: - raise - sys.path.remove(self.outdir) - def testImportAttributes(self): # check that we can access the generated module and its members. self.assertIsNotNone( - getattr(self._service_pb2, SERVICER_IDENTIFIER, None)) + getattr(service_pb2, SERVICER_IDENTIFIER, None)) self.assertIsNotNone( - getattr(self._service_pb2, STUB_IDENTIFIER, None)) + getattr(service_pb2, STUB_IDENTIFIER, None)) self.assertIsNotNone( - getattr(self._service_pb2, SERVER_FACTORY_IDENTIFIER, None)) + getattr(service_pb2, SERVER_FACTORY_IDENTIFIER, None)) self.assertIsNotNone( - getattr(self._service_pb2, STUB_FACTORY_IDENTIFIER, None)) + getattr(service_pb2, STUB_FACTORY_IDENTIFIER, None)) def testUpDown(self): - with _CreateService( - self._service_pb2, self._response_pb2, self._payload_pb2): - self._request_pb2.SimpleRequest(response_size=13) + with _CreateService(): + request_pb2.SimpleRequest(response_size=13) def testIncompleteServicer(self): - with _CreateIncompleteService(self._service_pb2) as (_, stub): - request = self._request_pb2.SimpleRequest(response_size=13) + with _CreateIncompleteService() as (_, stub): + request = request_pb2.SimpleRequest(response_size=13) try: stub.UnaryCall(request, test_constants.LONG_TIMEOUT) except face.AbortionError as error: self.assertEqual(interfaces.StatusCode.UNIMPLEMENTED, error.code) def testUnaryCall(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): - request = self._request_pb2.SimpleRequest(response_size=13) + with _CreateService() as (methods, stub): + request = request_pb2.SimpleRequest(response_size=13) response = stub.UnaryCall(request, test_constants.LONG_TIMEOUT) expected_response = methods.UnaryCall(request, 'not a real context!') self.assertEqual(expected_response, response) def testUnaryCallFuture(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): - request = self._request_pb2.SimpleRequest(response_size=13) + with _CreateService() as (methods, stub): + request = request_pb2.SimpleRequest(response_size=13) # Check that the call does not block waiting for the server to respond. with methods.pause(): response_future = stub.UnaryCall.future( @@ -348,9 +284,8 @@ class PythonPluginTest(unittest.TestCase): self.assertEqual(expected_response, response) def testUnaryCallFutureExpired(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): - request = self._request_pb2.SimpleRequest(response_size=13) + with _CreateService() as (methods, stub): + request = request_pb2.SimpleRequest(response_size=13) with methods.pause(): response_future = stub.UnaryCall.future( request, test_constants.SHORT_TIMEOUT) @@ -358,27 +293,24 @@ class PythonPluginTest(unittest.TestCase): response_future.result() def testUnaryCallFutureCancelled(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): - request = self._request_pb2.SimpleRequest(response_size=13) + with _CreateService() as (methods, stub): + request = request_pb2.SimpleRequest(response_size=13) with methods.pause(): response_future = stub.UnaryCall.future(request, 1) response_future.cancel() self.assertTrue(response_future.cancelled()) def testUnaryCallFutureFailed(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): - request = self._request_pb2.SimpleRequest(response_size=13) + with _CreateService() as (methods, stub): + request = request_pb2.SimpleRequest(response_size=13) with methods.fail(): response_future = stub.UnaryCall.future( request, test_constants.LONG_TIMEOUT) self.assertIsNotNone(response_future.exception()) def testStreamingOutputCall(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): - request = _streaming_output_request(self._request_pb2) + with _CreateService() as (methods, stub): + request = _streaming_output_request() responses = stub.StreamingOutputCall( request, test_constants.LONG_TIMEOUT) expected_responses = methods.StreamingOutputCall( @@ -388,9 +320,8 @@ class PythonPluginTest(unittest.TestCase): self.assertEqual(expected_response, response) def testStreamingOutputCallExpired(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): - request = _streaming_output_request(self._request_pb2) + with _CreateService() as (methods, stub): + request = _streaming_output_request() with methods.pause(): responses = stub.StreamingOutputCall( request, test_constants.SHORT_TIMEOUT) @@ -398,9 +329,8 @@ class PythonPluginTest(unittest.TestCase): list(responses) def testStreamingOutputCallCancelled(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): - request = _streaming_output_request(self._request_pb2) + with _CreateService() as (methods, stub): + request = _streaming_output_request() responses = stub.StreamingOutputCall( request, test_constants.LONG_TIMEOUT) next(responses) @@ -409,9 +339,8 @@ class PythonPluginTest(unittest.TestCase): next(responses) def testStreamingOutputCallFailed(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): - request = _streaming_output_request(self._request_pb2) + with _CreateService() as (methods, stub): + request = _streaming_output_request() with methods.fail(): responses = stub.StreamingOutputCall(request, 1) self.assertIsNotNone(responses) @@ -419,38 +348,32 @@ class PythonPluginTest(unittest.TestCase): next(responses) def testStreamingInputCall(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): + with _CreateService() as (methods, stub): response = stub.StreamingInputCall( - _streaming_input_request_iterator( - self._request_pb2, self._payload_pb2), + _streaming_input_request_iterator(), test_constants.LONG_TIMEOUT) expected_response = methods.StreamingInputCall( - _streaming_input_request_iterator(self._request_pb2, self._payload_pb2), + _streaming_input_request_iterator(), 'not a real RpcContext!') self.assertEqual(expected_response, response) def testStreamingInputCallFuture(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): + with _CreateService() as (methods, stub): with methods.pause(): response_future = stub.StreamingInputCall.future( - _streaming_input_request_iterator( - self._request_pb2, self._payload_pb2), + _streaming_input_request_iterator(), test_constants.LONG_TIMEOUT) response = response_future.result() expected_response = methods.StreamingInputCall( - _streaming_input_request_iterator(self._request_pb2, self._payload_pb2), + _streaming_input_request_iterator(), 'not a real RpcContext!') self.assertEqual(expected_response, response) def testStreamingInputCallFutureExpired(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): + with _CreateService() as (methods, stub): with methods.pause(): response_future = stub.StreamingInputCall.future( - _streaming_input_request_iterator( - self._request_pb2, self._payload_pb2), + _streaming_input_request_iterator(), test_constants.SHORT_TIMEOUT) with self.assertRaises(face.ExpirationError): response_future.result() @@ -458,12 +381,10 @@ class PythonPluginTest(unittest.TestCase): response_future.exception(), face.ExpirationError) def testStreamingInputCallFutureCancelled(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): + with _CreateService() as (methods, stub): with methods.pause(): response_future = stub.StreamingInputCall.future( - _streaming_input_request_iterator( - self._request_pb2, self._payload_pb2), + _streaming_input_request_iterator(), test_constants.LONG_TIMEOUT) response_future.cancel() self.assertTrue(response_future.cancelled()) @@ -471,32 +392,28 @@ class PythonPluginTest(unittest.TestCase): response_future.result() def testStreamingInputCallFutureFailed(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): + with _CreateService() as (methods, stub): with methods.fail(): response_future = stub.StreamingInputCall.future( - _streaming_input_request_iterator( - self._request_pb2, self._payload_pb2), + _streaming_input_request_iterator(), test_constants.LONG_TIMEOUT) self.assertIsNotNone(response_future.exception()) def testFullDuplexCall(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): + with _CreateService() as (methods, stub): responses = stub.FullDuplexCall( - _full_duplex_request_iterator(self._request_pb2), + _full_duplex_request_iterator(), test_constants.LONG_TIMEOUT) expected_responses = methods.FullDuplexCall( - _full_duplex_request_iterator(self._request_pb2), + _full_duplex_request_iterator(), 'not a real RpcContext!') for expected_response, response in moves.zip_longest( expected_responses, responses): self.assertEqual(expected_response, response) def testFullDuplexCallExpired(self): - request_iterator = _full_duplex_request_iterator(self._request_pb2) - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): + request_iterator = _full_duplex_request_iterator() + with _CreateService() as (methods, stub): with methods.pause(): responses = stub.FullDuplexCall( request_iterator, test_constants.SHORT_TIMEOUT) @@ -504,9 +421,8 @@ class PythonPluginTest(unittest.TestCase): list(responses) def testFullDuplexCallCancelled(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): - request_iterator = _full_duplex_request_iterator(self._request_pb2) + with _CreateService() as (methods, stub): + request_iterator = _full_duplex_request_iterator() responses = stub.FullDuplexCall( request_iterator, test_constants.LONG_TIMEOUT) next(responses) @@ -515,9 +431,8 @@ class PythonPluginTest(unittest.TestCase): next(responses) def testFullDuplexCallFailed(self): - request_iterator = _full_duplex_request_iterator(self._request_pb2) - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): + request_iterator = _full_duplex_request_iterator() + with _CreateService() as (methods, stub): with methods.fail(): responses = stub.FullDuplexCall( request_iterator, test_constants.LONG_TIMEOUT) @@ -526,13 +441,12 @@ class PythonPluginTest(unittest.TestCase): next(responses) def testHalfDuplexCall(self): - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): + with _CreateService() as (methods, stub): def half_duplex_request_iterator(): - request = self._request_pb2.StreamingOutputCallRequest() + request = request_pb2.StreamingOutputCallRequest() request.response_parameters.add(size=1, interval_us=0) yield request - request = self._request_pb2.StreamingOutputCallRequest() + request = request_pb2.StreamingOutputCallRequest() request.response_parameters.add(size=2, interval_us=0) request.response_parameters.add(size=3, interval_us=0) yield request @@ -557,14 +471,13 @@ class PythonPluginTest(unittest.TestCase): wait_cell[0] = False condition.notify_all() def half_duplex_request_iterator(): - request = self._request_pb2.StreamingOutputCallRequest() + request = request_pb2.StreamingOutputCallRequest() request.response_parameters.add(size=1, interval_us=0) yield request with condition: while wait_cell[0]: condition.wait() - with _CreateService(self._service_pb2, self._response_pb2, - self._payload_pb2) as (methods, stub): + with _CreateService() as (methods, stub): with wait(): responses = stub.HalfDuplexCall( half_duplex_request_iterator(), test_constants.SHORT_TIMEOUT) @@ -574,5 +487,4 @@ class PythonPluginTest(unittest.TestCase): if __name__ == '__main__': - #os.chdir(os.path.dirname(sys.argv[0])) unittest.main(verbosity=2) diff --git a/src/python/grpcio/tests/unit/__init__.py b/src/python/grpcio_tests/tests/protoc_plugin/protos/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/__init__.py rename to src/python/grpcio_tests/tests/protoc_plugin/protos/__init__.py diff --git a/src/python/grpcio/tests/unit/_adapter/__init__.py b/src/python/grpcio_tests/tests/protoc_plugin/protos/payload/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/_adapter/__init__.py rename to src/python/grpcio_tests/tests/protoc_plugin/protos/payload/__init__.py diff --git a/src/python/grpcio/tests/protoc_plugin/protos/payload/test_payload.proto b/src/python/grpcio_tests/tests/protoc_plugin/protos/payload/test_payload.proto similarity index 100% rename from src/python/grpcio/tests/protoc_plugin/protos/payload/test_payload.proto rename to src/python/grpcio_tests/tests/protoc_plugin/protos/payload/test_payload.proto diff --git a/src/python/grpcio/tests/unit/_junkdrawer/__init__.py b/src/python/grpcio_tests/tests/protoc_plugin/protos/requests/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/_junkdrawer/__init__.py rename to src/python/grpcio_tests/tests/protoc_plugin/protos/requests/__init__.py diff --git a/src/python/grpcio/tests/unit/_links/__init__.py b/src/python/grpcio_tests/tests/protoc_plugin/protos/requests/r/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/_links/__init__.py rename to src/python/grpcio_tests/tests/protoc_plugin/protos/requests/r/__init__.py diff --git a/src/python/grpcio/tests/protoc_plugin/protos/requests/r/test_requests.proto b/src/python/grpcio_tests/tests/protoc_plugin/protos/requests/r/test_requests.proto similarity index 97% rename from src/python/grpcio/tests/protoc_plugin/protos/requests/r/test_requests.proto rename to src/python/grpcio_tests/tests/protoc_plugin/protos/requests/r/test_requests.proto index 54105df6a56..365ae738e14 100644 --- a/src/python/grpcio/tests/protoc_plugin/protos/requests/r/test_requests.proto +++ b/src/python/grpcio_tests/tests/protoc_plugin/protos/requests/r/test_requests.proto @@ -29,7 +29,7 @@ syntax = "proto3"; -import "protos/payload/test_payload.proto"; +import "tests/protoc_plugin/protos/payload/test_payload.proto"; package grpc_protoc_plugin; diff --git a/src/python/grpcio/tests/unit/beta/__init__.py b/src/python/grpcio_tests/tests/protoc_plugin/protos/responses/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/beta/__init__.py rename to src/python/grpcio_tests/tests/protoc_plugin/protos/responses/__init__.py diff --git a/src/python/grpcio/tests/protoc_plugin/protos/responses/test_responses.proto b/src/python/grpcio_tests/tests/protoc_plugin/protos/responses/test_responses.proto similarity index 96% rename from src/python/grpcio/tests/protoc_plugin/protos/responses/test_responses.proto rename to src/python/grpcio_tests/tests/protoc_plugin/protos/responses/test_responses.proto index 734fbda86e9..1d54d58db1e 100644 --- a/src/python/grpcio/tests/protoc_plugin/protos/responses/test_responses.proto +++ b/src/python/grpcio_tests/tests/protoc_plugin/protos/responses/test_responses.proto @@ -29,7 +29,7 @@ syntax = "proto3"; -import "protos/payload/test_payload.proto"; +import "tests/protoc_plugin/protos/payload/test_payload.proto"; package grpc_protoc_plugin; diff --git a/src/python/grpcio/tests/unit/framework/__init__.py b/src/python/grpcio_tests/tests/protoc_plugin/protos/service/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/__init__.py rename to src/python/grpcio_tests/tests/protoc_plugin/protos/service/__init__.py diff --git a/src/python/grpcio/tests/protoc_plugin/protos/service/test_service.proto b/src/python/grpcio_tests/tests/protoc_plugin/protos/service/test_service.proto similarity index 95% rename from src/python/grpcio/tests/protoc_plugin/protos/service/test_service.proto rename to src/python/grpcio_tests/tests/protoc_plugin/protos/service/test_service.proto index fe715ee7f92..003dbbb9eb4 100644 --- a/src/python/grpcio/tests/protoc_plugin/protos/service/test_service.proto +++ b/src/python/grpcio_tests/tests/protoc_plugin/protos/service/test_service.proto @@ -29,8 +29,8 @@ syntax = "proto3"; -import "protos/requests/r/test_requests.proto"; -import "protos/responses/test_responses.proto"; +import "tests/protoc_plugin/protos/requests/r/test_requests.proto"; +import "tests/protoc_plugin/protos/responses/test_responses.proto"; package grpc_protoc_plugin; diff --git a/src/python/grpcio/tests/qps/__init__.py b/src/python/grpcio_tests/tests/qps/__init__.py similarity index 100% rename from src/python/grpcio/tests/qps/__init__.py rename to src/python/grpcio_tests/tests/qps/__init__.py diff --git a/src/python/grpcio/tests/qps/benchmark_client.py b/src/python/grpcio_tests/tests/qps/benchmark_client.py similarity index 100% rename from src/python/grpcio/tests/qps/benchmark_client.py rename to src/python/grpcio_tests/tests/qps/benchmark_client.py diff --git a/src/python/grpcio/tests/qps/benchmark_server.py b/src/python/grpcio_tests/tests/qps/benchmark_server.py similarity index 100% rename from src/python/grpcio/tests/qps/benchmark_server.py rename to src/python/grpcio_tests/tests/qps/benchmark_server.py diff --git a/src/python/grpcio/tests/qps/client_runner.py b/src/python/grpcio_tests/tests/qps/client_runner.py similarity index 100% rename from src/python/grpcio/tests/qps/client_runner.py rename to src/python/grpcio_tests/tests/qps/client_runner.py diff --git a/src/python/grpcio/tests/qps/histogram.py b/src/python/grpcio_tests/tests/qps/histogram.py similarity index 100% rename from src/python/grpcio/tests/qps/histogram.py rename to src/python/grpcio_tests/tests/qps/histogram.py diff --git a/src/python/grpcio/tests/qps/qps_worker.py b/src/python/grpcio_tests/tests/qps/qps_worker.py similarity index 100% rename from src/python/grpcio/tests/qps/qps_worker.py rename to src/python/grpcio_tests/tests/qps/qps_worker.py diff --git a/src/python/grpcio/tests/qps/worker_server.py b/src/python/grpcio_tests/tests/qps/worker_server.py similarity index 100% rename from src/python/grpcio/tests/qps/worker_server.py rename to src/python/grpcio_tests/tests/qps/worker_server.py diff --git a/src/python/grpcio/tests/stress/__init__.py b/src/python/grpcio_tests/tests/stress/__init__.py similarity index 100% rename from src/python/grpcio/tests/stress/__init__.py rename to src/python/grpcio_tests/tests/stress/__init__.py diff --git a/src/python/grpcio/tests/stress/client.py b/src/python/grpcio_tests/tests/stress/client.py similarity index 100% rename from src/python/grpcio/tests/stress/client.py rename to src/python/grpcio_tests/tests/stress/client.py diff --git a/src/python/grpcio/tests/stress/metrics_server.py b/src/python/grpcio_tests/tests/stress/metrics_server.py similarity index 100% rename from src/python/grpcio/tests/stress/metrics_server.py rename to src/python/grpcio_tests/tests/stress/metrics_server.py diff --git a/src/python/grpcio/tests/stress/test_runner.py b/src/python/grpcio_tests/tests/stress/test_runner.py similarity index 100% rename from src/python/grpcio/tests/stress/test_runner.py rename to src/python/grpcio_tests/tests/stress/test_runner.py diff --git a/src/python/grpcio/tests/tests.json b/src/python/grpcio_tests/tests/tests.json similarity index 100% rename from src/python/grpcio/tests/tests.json rename to src/python/grpcio_tests/tests/tests.json diff --git a/src/python/grpcio/tests/unit/framework/common/__init__.py b/src/python/grpcio_tests/tests/unit/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/common/__init__.py rename to src/python/grpcio_tests/tests/unit/__init__.py diff --git a/src/python/grpcio/tests/unit/_adapter/.gitignore b/src/python/grpcio_tests/tests/unit/_adapter/.gitignore similarity index 100% rename from src/python/grpcio/tests/unit/_adapter/.gitignore rename to src/python/grpcio_tests/tests/unit/_adapter/.gitignore diff --git a/src/python/grpcio/tests/unit/framework/core/__init__.py b/src/python/grpcio_tests/tests/unit/_adapter/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/core/__init__.py rename to src/python/grpcio_tests/tests/unit/_adapter/__init__.py diff --git a/src/python/grpcio/tests/unit/_adapter/_proto_scenarios.py b/src/python/grpcio_tests/tests/unit/_adapter/_proto_scenarios.py similarity index 100% rename from src/python/grpcio/tests/unit/_adapter/_proto_scenarios.py rename to src/python/grpcio_tests/tests/unit/_adapter/_proto_scenarios.py diff --git a/src/python/grpcio/tests/unit/_api_test.py b/src/python/grpcio_tests/tests/unit/_api_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_api_test.py rename to src/python/grpcio_tests/tests/unit/_api_test.py diff --git a/src/python/grpcio/tests/unit/_auth_test.py b/src/python/grpcio_tests/tests/unit/_auth_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_auth_test.py rename to src/python/grpcio_tests/tests/unit/_auth_test.py diff --git a/src/python/grpcio/tests/unit/_channel_connectivity_test.py b/src/python/grpcio_tests/tests/unit/_channel_connectivity_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_channel_connectivity_test.py rename to src/python/grpcio_tests/tests/unit/_channel_connectivity_test.py diff --git a/src/python/grpcio/tests/unit/_channel_ready_future_test.py b/src/python/grpcio_tests/tests/unit/_channel_ready_future_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_channel_ready_future_test.py rename to src/python/grpcio_tests/tests/unit/_channel_ready_future_test.py diff --git a/src/python/grpcio/tests/unit/_compression_test.py b/src/python/grpcio_tests/tests/unit/_compression_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_compression_test.py rename to src/python/grpcio_tests/tests/unit/_compression_test.py diff --git a/src/python/grpcio/tests/unit/_cython/.gitignore b/src/python/grpcio_tests/tests/unit/_cython/.gitignore similarity index 100% rename from src/python/grpcio/tests/unit/_cython/.gitignore rename to src/python/grpcio_tests/tests/unit/_cython/.gitignore diff --git a/src/python/grpcio/tests/unit/_cython/__init__.py b/src/python/grpcio_tests/tests/unit/_cython/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/_cython/__init__.py rename to src/python/grpcio_tests/tests/unit/_cython/__init__.py diff --git a/src/python/grpcio/tests/unit/_cython/_cancel_many_calls_test.py b/src/python/grpcio_tests/tests/unit/_cython/_cancel_many_calls_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_cython/_cancel_many_calls_test.py rename to src/python/grpcio_tests/tests/unit/_cython/_cancel_many_calls_test.py diff --git a/src/python/grpcio/tests/unit/_cython/_channel_test.py b/src/python/grpcio_tests/tests/unit/_cython/_channel_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_cython/_channel_test.py rename to src/python/grpcio_tests/tests/unit/_cython/_channel_test.py diff --git a/src/python/grpcio/tests/unit/_cython/_read_some_but_not_all_responses_test.py b/src/python/grpcio_tests/tests/unit/_cython/_read_some_but_not_all_responses_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_cython/_read_some_but_not_all_responses_test.py rename to src/python/grpcio_tests/tests/unit/_cython/_read_some_but_not_all_responses_test.py diff --git a/src/python/grpcio/tests/unit/_cython/cygrpc_test.py b/src/python/grpcio_tests/tests/unit/_cython/cygrpc_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_cython/cygrpc_test.py rename to src/python/grpcio_tests/tests/unit/_cython/cygrpc_test.py diff --git a/src/python/grpcio/tests/unit/_cython/test_utilities.py b/src/python/grpcio_tests/tests/unit/_cython/test_utilities.py similarity index 100% rename from src/python/grpcio/tests/unit/_cython/test_utilities.py rename to src/python/grpcio_tests/tests/unit/_cython/test_utilities.py diff --git a/src/python/grpcio/tests/unit/_empty_message_test.py b/src/python/grpcio_tests/tests/unit/_empty_message_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_empty_message_test.py rename to src/python/grpcio_tests/tests/unit/_empty_message_test.py diff --git a/src/python/grpcio/tests/unit/_exit_scenarios.py b/src/python/grpcio_tests/tests/unit/_exit_scenarios.py similarity index 100% rename from src/python/grpcio/tests/unit/_exit_scenarios.py rename to src/python/grpcio_tests/tests/unit/_exit_scenarios.py diff --git a/src/python/grpcio/tests/unit/_exit_test.py b/src/python/grpcio_tests/tests/unit/_exit_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_exit_test.py rename to src/python/grpcio_tests/tests/unit/_exit_test.py diff --git a/src/python/grpcio/tests/unit/_from_grpc_import_star.py b/src/python/grpcio_tests/tests/unit/_from_grpc_import_star.py similarity index 100% rename from src/python/grpcio/tests/unit/_from_grpc_import_star.py rename to src/python/grpcio_tests/tests/unit/_from_grpc_import_star.py diff --git a/src/python/grpcio/tests/unit/framework/foundation/__init__.py b/src/python/grpcio_tests/tests/unit/_junkdrawer/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/foundation/__init__.py rename to src/python/grpcio_tests/tests/unit/_junkdrawer/__init__.py diff --git a/src/python/grpcio/tests/unit/_junkdrawer/math_pb2.py b/src/python/grpcio_tests/tests/unit/_junkdrawer/math_pb2.py similarity index 100% rename from src/python/grpcio/tests/unit/_junkdrawer/math_pb2.py rename to src/python/grpcio_tests/tests/unit/_junkdrawer/math_pb2.py diff --git a/src/python/grpcio/tests/unit/_junkdrawer/stock_pb2.py b/src/python/grpcio_tests/tests/unit/_junkdrawer/stock_pb2.py similarity index 100% rename from src/python/grpcio/tests/unit/_junkdrawer/stock_pb2.py rename to src/python/grpcio_tests/tests/unit/_junkdrawer/stock_pb2.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/__init__.py b/src/python/grpcio_tests/tests/unit/_links/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/__init__.py rename to src/python/grpcio_tests/tests/unit/_links/__init__.py diff --git a/src/python/grpcio/tests/unit/_links/_proto_scenarios.py b/src/python/grpcio_tests/tests/unit/_links/_proto_scenarios.py similarity index 100% rename from src/python/grpcio/tests/unit/_links/_proto_scenarios.py rename to src/python/grpcio_tests/tests/unit/_links/_proto_scenarios.py diff --git a/src/python/grpcio/tests/unit/_metadata_code_details_test.py b/src/python/grpcio_tests/tests/unit/_metadata_code_details_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_metadata_code_details_test.py rename to src/python/grpcio_tests/tests/unit/_metadata_code_details_test.py diff --git a/src/python/grpcio/tests/unit/_metadata_test.py b/src/python/grpcio_tests/tests/unit/_metadata_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_metadata_test.py rename to src/python/grpcio_tests/tests/unit/_metadata_test.py diff --git a/src/python/grpcio/tests/unit/_rpc_test.py b/src/python/grpcio_tests/tests/unit/_rpc_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_rpc_test.py rename to src/python/grpcio_tests/tests/unit/_rpc_test.py diff --git a/src/python/grpcio/tests/unit/_sanity/__init__.py b/src/python/grpcio_tests/tests/unit/_sanity/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/_sanity/__init__.py rename to src/python/grpcio_tests/tests/unit/_sanity/__init__.py diff --git a/src/python/grpcio/tests/unit/_sanity/_sanity_test.py b/src/python/grpcio_tests/tests/unit/_sanity/_sanity_test.py similarity index 90% rename from src/python/grpcio/tests/unit/_sanity/_sanity_test.py rename to src/python/grpcio_tests/tests/unit/_sanity/_sanity_test.py index 0a5a715c0e1..e9fdf217aef 100644 --- a/src/python/grpcio/tests/unit/_sanity/_sanity_test.py +++ b/src/python/grpcio_tests/tests/unit/_sanity/_sanity_test.py @@ -30,6 +30,9 @@ import json import unittest +import pkg_resources +import six + import tests @@ -44,8 +47,10 @@ class Sanity(unittest.TestCase): for test_case_class in tests._loader.iterate_suite_cases(loader.suite)] test_suite_names = sorted(set(test_suite_names)) - with open('src/python/grpcio/tests/tests.json') as tests_json_file: - tests_json = json.load(tests_json_file) + tests_json_string = pkg_resources.resource_string('tests', 'tests.json') + if six.PY3: + tests_json_string = tests_json_string.decode() + tests_json = json.loads(tests_json_string) self.assertListEqual(test_suite_names, tests_json) diff --git a/src/python/grpcio/tests/unit/_thread_cleanup_test.py b/src/python/grpcio_tests/tests/unit/_thread_cleanup_test.py similarity index 100% rename from src/python/grpcio/tests/unit/_thread_cleanup_test.py rename to src/python/grpcio_tests/tests/unit/_thread_cleanup_test.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/base/__init__.py b/src/python/grpcio_tests/tests/unit/beta/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/base/__init__.py rename to src/python/grpcio_tests/tests/unit/beta/__init__.py diff --git a/src/python/grpcio/tests/unit/beta/_beta_features_test.py b/src/python/grpcio_tests/tests/unit/beta/_beta_features_test.py similarity index 100% rename from src/python/grpcio/tests/unit/beta/_beta_features_test.py rename to src/python/grpcio_tests/tests/unit/beta/_beta_features_test.py diff --git a/src/python/grpcio/tests/unit/beta/_connectivity_channel_test.py b/src/python/grpcio_tests/tests/unit/beta/_connectivity_channel_test.py similarity index 100% rename from src/python/grpcio/tests/unit/beta/_connectivity_channel_test.py rename to src/python/grpcio_tests/tests/unit/beta/_connectivity_channel_test.py diff --git a/src/python/grpcio/tests/unit/beta/_face_interface_test.py b/src/python/grpcio_tests/tests/unit/beta/_face_interface_test.py similarity index 100% rename from src/python/grpcio/tests/unit/beta/_face_interface_test.py rename to src/python/grpcio_tests/tests/unit/beta/_face_interface_test.py diff --git a/src/python/grpcio/tests/unit/beta/_implementations_test.py b/src/python/grpcio_tests/tests/unit/beta/_implementations_test.py similarity index 100% rename from src/python/grpcio/tests/unit/beta/_implementations_test.py rename to src/python/grpcio_tests/tests/unit/beta/_implementations_test.py diff --git a/src/python/grpcio/tests/unit/beta/_not_found_test.py b/src/python/grpcio_tests/tests/unit/beta/_not_found_test.py similarity index 100% rename from src/python/grpcio/tests/unit/beta/_not_found_test.py rename to src/python/grpcio_tests/tests/unit/beta/_not_found_test.py diff --git a/src/python/grpcio/tests/unit/beta/_utilities_test.py b/src/python/grpcio_tests/tests/unit/beta/_utilities_test.py similarity index 100% rename from src/python/grpcio/tests/unit/beta/_utilities_test.py rename to src/python/grpcio_tests/tests/unit/beta/_utilities_test.py diff --git a/src/python/grpcio/tests/unit/beta/test_utilities.py b/src/python/grpcio_tests/tests/unit/beta/test_utilities.py similarity index 100% rename from src/python/grpcio/tests/unit/beta/test_utilities.py rename to src/python/grpcio_tests/tests/unit/beta/test_utilities.py diff --git a/src/python/grpcio/tests/unit/credentials/README b/src/python/grpcio_tests/tests/unit/credentials/README similarity index 100% rename from src/python/grpcio/tests/unit/credentials/README rename to src/python/grpcio_tests/tests/unit/credentials/README diff --git a/src/python/grpcio/tests/unit/credentials/ca.pem b/src/python/grpcio_tests/tests/unit/credentials/ca.pem similarity index 100% rename from src/python/grpcio/tests/unit/credentials/ca.pem rename to src/python/grpcio_tests/tests/unit/credentials/ca.pem diff --git a/src/python/grpcio/tests/unit/credentials/server1.key b/src/python/grpcio_tests/tests/unit/credentials/server1.key similarity index 100% rename from src/python/grpcio/tests/unit/credentials/server1.key rename to src/python/grpcio_tests/tests/unit/credentials/server1.key diff --git a/src/python/grpcio/tests/unit/credentials/server1.pem b/src/python/grpcio_tests/tests/unit/credentials/server1.pem similarity index 100% rename from src/python/grpcio/tests/unit/credentials/server1.pem rename to src/python/grpcio_tests/tests/unit/credentials/server1.pem diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/__init__.py b/src/python/grpcio_tests/tests/unit/framework/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/face/__init__.py rename to src/python/grpcio_tests/tests/unit/framework/__init__.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/links/__init__.py b/src/python/grpcio_tests/tests/unit/framework/common/__init__.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/links/__init__.py rename to src/python/grpcio_tests/tests/unit/framework/common/__init__.py diff --git a/src/python/grpcio/tests/unit/framework/common/test_constants.py b/src/python/grpcio_tests/tests/unit/framework/common/test_constants.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/common/test_constants.py rename to src/python/grpcio_tests/tests/unit/framework/common/test_constants.py diff --git a/src/python/grpcio/tests/unit/framework/common/test_control.py b/src/python/grpcio_tests/tests/unit/framework/common/test_control.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/common/test_control.py rename to src/python/grpcio_tests/tests/unit/framework/common/test_control.py diff --git a/src/python/grpcio/tests/unit/framework/common/test_coverage.py b/src/python/grpcio_tests/tests/unit/framework/common/test_coverage.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/common/test_coverage.py rename to src/python/grpcio_tests/tests/unit/framework/common/test_coverage.py diff --git a/src/python/grpcio_tests/tests/unit/framework/core/__init__.py b/src/python/grpcio_tests/tests/unit/framework/core/__init__.py new file mode 100644 index 00000000000..70865191060 --- /dev/null +++ b/src/python/grpcio_tests/tests/unit/framework/core/__init__.py @@ -0,0 +1,30 @@ +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + diff --git a/src/python/grpcio_tests/tests/unit/framework/foundation/__init__.py b/src/python/grpcio_tests/tests/unit/framework/foundation/__init__.py new file mode 100644 index 00000000000..70865191060 --- /dev/null +++ b/src/python/grpcio_tests/tests/unit/framework/foundation/__init__.py @@ -0,0 +1,30 @@ +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + diff --git a/src/python/grpcio/tests/unit/framework/foundation/_logging_pool_test.py b/src/python/grpcio_tests/tests/unit/framework/foundation/_logging_pool_test.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/foundation/_logging_pool_test.py rename to src/python/grpcio_tests/tests/unit/framework/foundation/_logging_pool_test.py diff --git a/src/python/grpcio/tests/unit/framework/foundation/stream_testing.py b/src/python/grpcio_tests/tests/unit/framework/foundation/stream_testing.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/foundation/stream_testing.py rename to src/python/grpcio_tests/tests/unit/framework/foundation/stream_testing.py diff --git a/src/python/grpcio_tests/tests/unit/framework/interfaces/__init__.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/__init__.py new file mode 100644 index 00000000000..70865191060 --- /dev/null +++ b/src/python/grpcio_tests/tests/unit/framework/interfaces/__init__.py @@ -0,0 +1,30 @@ +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + diff --git a/src/python/grpcio_tests/tests/unit/framework/interfaces/base/__init__.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/base/__init__.py new file mode 100644 index 00000000000..70865191060 --- /dev/null +++ b/src/python/grpcio_tests/tests/unit/framework/interfaces/base/__init__.py @@ -0,0 +1,30 @@ +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + diff --git a/src/python/grpcio/tests/unit/framework/interfaces/base/_control.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/base/_control.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/base/_control.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/base/_control.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/base/_sequence.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/base/_sequence.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/base/_sequence.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/base/_sequence.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/base/_state.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/base/_state.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/base/_state.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/base/_state.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/base/test_cases.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/base/test_cases.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/base/test_cases.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/base/test_cases.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/base/test_interfaces.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/base/test_interfaces.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/base/test_interfaces.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/base/test_interfaces.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/_3069_test_constant.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_3069_test_constant.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/face/_3069_test_constant.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/face/_3069_test_constant.py diff --git a/src/python/grpcio_tests/tests/unit/framework/interfaces/face/__init__.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/__init__.py new file mode 100644 index 00000000000..70865191060 --- /dev/null +++ b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/__init__.py @@ -0,0 +1,30 @@ +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/_blocking_invocation_inline_service.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_blocking_invocation_inline_service.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/face/_blocking_invocation_inline_service.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/face/_blocking_invocation_inline_service.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/_digest.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_digest.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/face/_digest.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/face/_digest.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/_invocation.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_invocation.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/face/_invocation.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/face/_invocation.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/_receiver.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_receiver.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/face/_receiver.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/face/_receiver.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/_service.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_service.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/face/_service.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/face/_service.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/_stock_service.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_stock_service.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/face/_stock_service.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/face/_stock_service.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/test_cases.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/test_cases.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/face/test_cases.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/face/test_cases.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/test_interfaces.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/test_interfaces.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/face/test_interfaces.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/face/test_interfaces.py diff --git a/src/python/grpcio_tests/tests/unit/framework/interfaces/links/__init__.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/links/__init__.py new file mode 100644 index 00000000000..70865191060 --- /dev/null +++ b/src/python/grpcio_tests/tests/unit/framework/interfaces/links/__init__.py @@ -0,0 +1,30 @@ +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + diff --git a/src/python/grpcio/tests/unit/framework/interfaces/links/test_cases.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/links/test_cases.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/links/test_cases.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/links/test_cases.py diff --git a/src/python/grpcio/tests/unit/framework/interfaces/links/test_utilities.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/links/test_utilities.py similarity index 100% rename from src/python/grpcio/tests/unit/framework/interfaces/links/test_utilities.py rename to src/python/grpcio_tests/tests/unit/framework/interfaces/links/test_utilities.py diff --git a/src/python/grpcio/tests/unit/resources.py b/src/python/grpcio_tests/tests/unit/resources.py similarity index 100% rename from src/python/grpcio/tests/unit/resources.py rename to src/python/grpcio_tests/tests/unit/resources.py diff --git a/src/python/grpcio/tests/unit/test_common.py b/src/python/grpcio_tests/tests/unit/test_common.py similarity index 100% rename from src/python/grpcio/tests/unit/test_common.py rename to src/python/grpcio_tests/tests/unit/test_common.py diff --git a/templates/src/python/grpcio_tests/grpc_version.py.template b/templates/src/python/grpcio_tests/grpc_version.py.template new file mode 100644 index 00000000000..1f1bf2956dc --- /dev/null +++ b/templates/src/python/grpcio_tests/grpc_version.py.template @@ -0,0 +1,34 @@ +%YAML 1.2 +--- | + # Copyright 2016, Google Inc. + # All rights reserved. + # + # Redistribution and use in source and binary forms, with or without + # modification, are permitted provided that the following conditions are + # met: + # + # * Redistributions of source code must retain the above copyright + # notice, this list of conditions and the following disclaimer. + # * Redistributions in binary form must reproduce the above + # copyright notice, this list of conditions and the following disclaimer + # in the documentation and/or other materials provided with the + # distribution. + # * Neither the name of Google Inc. nor the names of its + # contributors may be used to endorse or promote products derived from + # this software without specific prior written permission. + # + # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + # AUTO-GENERATED FROM `$REPO_ROOT/templates/src/python/grpcio_tests/grpc_version.py.template`!!! + + VERSION='${settings.python_version.pep440()}' diff --git a/tools/distrib/python/grpcio_tools/.gitignore b/tools/distrib/python/grpcio_tools/.gitignore index 979704d970b..9f3a7360ee6 100644 --- a/tools/distrib/python/grpcio_tools/.gitignore +++ b/tools/distrib/python/grpcio_tools/.gitignore @@ -5,3 +5,4 @@ grpc_root/ *.c *.cpp *.egg-info +*.so diff --git a/tools/run_tests/build_python.sh b/tools/run_tests/build_python.sh index b1c90df824c..3b1c7d2d40e 100755 --- a/tools/run_tests/build_python.sh +++ b/tools/run_tests/build_python.sh @@ -37,13 +37,8 @@ TOX_PYTHON_ENV="$1" PY_VERSION="${TOX_PYTHON_ENV: -2}" ROOT=`pwd` -export LD_LIBRARY_PATH=$ROOT/libs/$CONFIG -export DYLD_LIBRARY_PATH=$ROOT/libs/$CONFIG -export PATH=$ROOT/bins/$CONFIG:$ROOT/bins/$CONFIG/protobuf:$PATH -export CFLAGS="-I$ROOT/include -std=gnu99" -export LDFLAGS="-L$ROOT/libs/$CONFIG" +export CFLAGS="-I$ROOT/include -std=gnu99 -fno-wrapv" export GRPC_PYTHON_BUILD_WITH_CYTHON=1 -export GRPC_PYTHON_USE_PRECOMPILED_BINARIES=0 if [ "$CONFIG" = "gcov" ] then @@ -52,25 +47,11 @@ fi tox -e ${TOX_PYTHON_ENV} --notest -# We force the .so naming convention in PEP 3149 for side by side installation support -# Note this is the default in Python3, but explicitly disabled for Darwin, so we only -# use this hack for our testing environment. -if [ "$PY_VERSION" -gt "27" ] -then - mv $ROOT/src/python/grpcio/grpc/_cython/cygrpc.so $ROOT/src/python/grpcio/grpc/_cython/cygrpc.so.backup || true -fi - -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/setup.py build -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/setup.py build_py -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/setup.py build_ext --inplace -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/setup.py gather --test - -if [ "$PY_VERSION" -gt "27" ] -then - mv $ROOT/src/python/grpcio/grpc/_cython/cygrpc.so $ROOT/src/python/grpcio/grpc/_cython/cygrpc.cpython-${PY_VERSION}m.so || true - mv $ROOT/src/python/grpcio/grpc/_cython/cygrpc.so.backup $ROOT/src/python/grpcio/grpc/_cython/cygrpc.so || true -fi - -# Build the health checker -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/src/python/grpcio_health_checking/setup.py build -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/src/python/grpcio_health_checking/setup.py build_py +$ROOT/.tox/${TOX_PYTHON_ENV}/bin/pip install cython +$ROOT/.tox/${TOX_PYTHON_ENV}/bin/pip install $ROOT +$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/tools/distrib/python/make_grpcio_tools.py +$ROOT/.tox/${TOX_PYTHON_ENV}/bin/pip install $ROOT/tools/distrib/python/grpcio_tools +$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/src/python/grpcio_health_checking/setup.py preprocess +$ROOT/.tox/${TOX_PYTHON_ENV}/bin/pip install $ROOT/src/python/grpcio_health_checking +$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/src/python/grpcio_tests/setup.py preprocess +$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/src/python/grpcio_tests/setup.py build_proto_modules diff --git a/tools/run_tests/performance/run_worker_python.sh b/tools/run_tests/performance/run_worker_python.sh index 0da8deda58e..2bca5e16952 100755 --- a/tools/run_tests/performance/run_worker_python.sh +++ b/tools/run_tests/performance/run_worker_python.sh @@ -32,4 +32,4 @@ set -ex cd $(dirname $0)/../../.. -PYTHONPATH=src/python/grpcio:src/python/gens .tox/py27/bin/python src/python/grpcio/tests/qps/qps_worker.py $@ +PYTHONPATH=src/python/grpcio_tests:src/python/grpcio:src/python/gens .tox/py27/bin/python src/python/grpcio_tests/tests/qps/qps_worker.py $@ diff --git a/tools/run_tests/run_python.sh b/tools/run_tests/run_python.sh index 8059059d41b..b05efd1c76a 100755 --- a/tools/run_tests/run_python.sh +++ b/tools/run_tests/run_python.sh @@ -49,7 +49,7 @@ then export GRPC_PYTHON_ENABLE_CYTHON_TRACING=1 tox -e ${TOX_PYTHON_ENV} else - $ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/setup.py test_lite + $ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/src/python/grpcio_tests/setup.py test_lite fi mkdir -p $ROOT/reports diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index c1254275e6c..f32a621ee46 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -381,12 +381,9 @@ class PythonLanguage(object): def test_specs(self): # load list of known test suites - with open('src/python/grpcio/tests/tests.json') as tests_json_file: + with open('src/python/grpcio_tests/tests/tests.json') as tests_json_file: tests_json = json.load(tests_json_file) environment = dict(_FORCE_ENVIRON_FOR_WRAPPERS) - environment['PYTHONPATH'] = '{}:{}'.format( - os.path.abspath('src/python/gens'), - os.path.abspath('src/python/grpcio_health_checking')) if self.config.build_config != 'gcov': return [self.config.job_spec( ['tools/run_tests/run_python.sh', tox_env], diff --git a/tox.ini b/tox.ini index 66b74a32db0..8f580047e18 100644 --- a/tox.ini +++ b/tox.ini @@ -19,8 +19,8 @@ passenv = * [testenv:interop_client] commands = - {envpython} setup.py run_interop --client --args='{posargs}' + {envpython} src/python/grpcio_tests/setup.py run_interop --client --args='{posargs}' [testenv:interop_server] commands = - {envpython} setup.py run_interop --server --args='{posargs}' + {envpython} src/python/grpcio_tests/setup.py run_interop --server --args='{posargs}' From 3b5b20682bf64e2f7275adec6983d6e390e7caf2 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Thu, 2 Jun 2016 20:27:20 -0700 Subject: [PATCH 0377/1039] Make running individual Python tests less painful Before this change, running Python tests individually required building a tox environment via the run_tests script and then specifying long environment variables to filter out just the test we wanted to run (and then we wouldn't be able to get the output on interrupt, nor would we have an easy way of determining the PID of the process for debugger attachment). Now invoking the build_python.sh script creates a workable python virtual environment that includes all necessary libraries and tests (s.t. running a single test is now possible by just knowing the module name). This does not change existing supported means of running tests (e.g. through run_tests.py). An additional way of running individual tests has been introduced. Following invocation of `./tools/run_tests/build_python.sh` (or run_tests.py), one may invoke ./$VENV/bin/python -m $TEST_MODULE_NAME and acquire a single running process that *is* the test process (rather than a parent of the process). $VENV is the virtual environment name specified to `build_python.sh` (defaults to `py27`) and $TEST_MODULE_NAME is what it says on the tin. --- PYTHON-MANIFEST.in | 1 - tools/run_tests/build_python.sh | 71 +++++++++++++++---- .../performance/run_worker_python.sh | 2 +- tools/run_tests/run_interop_tests.py | 14 ++-- tools/run_tests/run_python.sh | 17 +---- tools/run_tests/run_tests.py | 60 +++++++++++----- 6 files changed, 114 insertions(+), 51 deletions(-) diff --git a/PYTHON-MANIFEST.in b/PYTHON-MANIFEST.in index 3ebba6ec3fc..635e77b875c 100644 --- a/PYTHON-MANIFEST.in +++ b/PYTHON-MANIFEST.in @@ -1,6 +1,5 @@ recursive-include src/python/grpcio/grpc *.c *.h *.py *.pyx *.pxd *.pxi *.python *.pem recursive-exclude src/python/grpcio/grpc/_cython *.so *.pyd -graft src/python/grpcio/tests graft src/python/grpcio/grpcio.egg-info graft src/core graft src/boringssl diff --git a/tools/run_tests/build_python.sh b/tools/run_tests/build_python.sh index 3b1c7d2d40e..9d2813fa192 100755 --- a/tools/run_tests/build_python.sh +++ b/tools/run_tests/build_python.sh @@ -33,25 +33,70 @@ set -ex # change to grpc repo root cd $(dirname $0)/../.. -TOX_PYTHON_ENV="$1" -PY_VERSION="${TOX_PYTHON_ENV: -2}" +PYTHON=${1:-python2.7} +VENV=${2:-py27} +VENV_RELATIVE_PYTHON=${3:-bin/python} +TOOLCHAIN=${4:-unix} ROOT=`pwd` export CFLAGS="-I$ROOT/include -std=gnu99 -fno-wrapv" export GRPC_PYTHON_BUILD_WITH_CYTHON=1 -if [ "$CONFIG" = "gcov" ] -then +# If ccache is available, use it... unless we're on Mac, then all hell breaks +# loose because Python does hacky things to support other hacky things done to +# hacky things on Mac OS X +PLATFORM=`uname -s` +if [ "${PLATFORM/Darwin}" = "$PLATFORM" ]; then + # We're not on Darwin (Mac OS X) + if [ -x "$(command -v ccache)" ]; then + if [ -x "$(command -v gcc)" ]; then + export CC='ccache gcc' + elif [ -x "$(command -v clang)" ]; then + export CC='ccache clang' + fi + fi +fi + +# Find `realpath` +if [ -x "$(command -v realpath)" ]; then + export REALPATH=realpath +elif [ -x "$(command -v grealpath)" ]; then + export REALPATH=grealpath +else + echo 'Couldn'"'"'t find `realpath` or `grealpath`' + exit 1 +fi + +if [ "$CONFIG" = "gcov" ]; then export GRPC_PYTHON_ENABLE_CYTHON_TRACING=1 fi -tox -e ${TOX_PYTHON_ENV} --notest +($PYTHON -m virtualenv $VENV || true) +VENV_PYTHON=`$REALPATH -s "$VENV/$VENV_RELATIVE_PYTHON"` + +# pip-installs the directory specified. Used because on MSYS the vanilla Windows +# Python gets confused when parsing paths. +pip_install_dir() { + PWD=`pwd` + cd $1 + ($VENV_PYTHON setup.py build_ext -c $TOOLCHAIN || true) + # install the dependencies + $VENV_PYTHON -m pip install --upgrade . + # ensure that we've reinstalled the test packages + $VENV_PYTHON -m pip install --upgrade --force-reinstall --no-deps . + cd $PWD +} -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/pip install cython -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/pip install $ROOT -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/tools/distrib/python/make_grpcio_tools.py -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/pip install $ROOT/tools/distrib/python/grpcio_tools -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/src/python/grpcio_health_checking/setup.py preprocess -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/pip install $ROOT/src/python/grpcio_health_checking -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/src/python/grpcio_tests/setup.py preprocess -$ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/src/python/grpcio_tests/setup.py build_proto_modules +$VENV_PYTHON -m pip install --upgrade pip setuptools +$VENV_PYTHON -m pip install cython +pip_install_dir $ROOT +$VENV_PYTHON $ROOT/tools/distrib/python/make_grpcio_tools.py +pip_install_dir $ROOT/tools/distrib/python/grpcio_tools +# TODO(atash) figure out namespace packages and grpcio-tools and auditwheel +# etc... +pip_install_dir $ROOT +$VENV_PYTHON $ROOT/src/python/grpcio_health_checking/setup.py preprocess +pip_install_dir $ROOT/src/python/grpcio_health_checking +$VENV_PYTHON $ROOT/src/python/grpcio_tests/setup.py preprocess +$VENV_PYTHON $ROOT/src/python/grpcio_tests/setup.py build_proto_modules +pip_install_dir $ROOT/src/python/grpcio_tests diff --git a/tools/run_tests/performance/run_worker_python.sh b/tools/run_tests/performance/run_worker_python.sh index 2bca5e16952..3b8ba6f4e4a 100755 --- a/tools/run_tests/performance/run_worker_python.sh +++ b/tools/run_tests/performance/run_worker_python.sh @@ -32,4 +32,4 @@ set -ex cd $(dirname $0)/../../.. -PYTHONPATH=src/python/grpcio_tests:src/python/grpcio:src/python/gens .tox/py27/bin/python src/python/grpcio_tests/tests/qps/qps_worker.py $@ +PYTHONPATH=src/python/grpcio_tests:src/python/grpcio:src/python/gens py27/bin/python src/python/grpcio_tests/tests/qps/qps_worker.py $@ diff --git a/tools/run_tests/run_interop_tests.py b/tools/run_tests/run_interop_tests.py index adf9adaf136..13a4a49325a 100755 --- a/tools/run_tests/run_interop_tests.py +++ b/tools/run_tests/run_interop_tests.py @@ -304,8 +304,11 @@ class PythonLanguage: def client_cmd(self, args): return [ - 'tox -einterop_client --', - ' '.join(args) + 'py27/bin/python', + 'src/python/grpcio_tests/setup.py', + 'run_interop', + '--client', + '--args="{}"'.format(' '.join(args)) ] def cloud_to_prod_env(self): @@ -313,8 +316,11 @@ class PythonLanguage: def server_cmd(self, args): return [ - 'tox -einterop_server --', - ' '.join(args) + ' --use_tls=true' + 'py27/bin/python', + 'src/python/grpcio_tests/setup.py', + 'run_interop', + '--server', + '--args="{}"'.format(' '.join(args) + ' --use_tls=true') ] def global_env(self): diff --git a/tools/run_tests/run_python.sh b/tools/run_tests/run_python.sh index b05efd1c76a..17e0186f2a8 100755 --- a/tools/run_tests/run_python.sh +++ b/tools/run_tests/run_python.sh @@ -33,24 +33,11 @@ set -ex # change to grpc repo root cd $(dirname $0)/../.. -TOX_PYTHON_ENV="$1" +PYTHON=`realpath -s "${1:-py27/bin/python}"` ROOT=`pwd` -export LD_LIBRARY_PATH=$ROOT/libs/$CONFIG -export DYLD_LIBRARY_PATH=$ROOT/libs/$CONFIG -export PATH=$ROOT/bins/$CONFIG:$ROOT/bins/$CONFIG/protobuf:$PATH -export CFLAGS="-I$ROOT/include -std=c89" -export LDFLAGS="-L$ROOT/libs/$CONFIG" -export GRPC_PYTHON_BUILD_WITH_CYTHON=1 -export GRPC_PYTHON_USE_PRECOMPILED_BINARIES=0 -if [ "$CONFIG" = "gcov" ] -then - export GRPC_PYTHON_ENABLE_CYTHON_TRACING=1 - tox -e ${TOX_PYTHON_ENV} -else - $ROOT/.tox/${TOX_PYTHON_ENV}/bin/python $ROOT/src/python/grpcio_tests/setup.py test_lite -fi +$PYTHON $ROOT/src/python/grpcio_tests/setup.py test_lite mkdir -p $ROOT/reports rm -rf $ROOT/reports/python-coverage diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index f32a621ee46..fbc5729c758 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -32,11 +32,13 @@ import argparse import ast +import collections import glob import itertools import json import multiprocessing import os +import os.path import platform import random import re @@ -372,12 +374,20 @@ class PhpLanguage(object): return 'php' +class PythonConfig(collections.namedtuple('PythonConfig', [ + 'python', 'venv', 'venv_relative_python', 'toolchain',])): + + @property + def venv_python(self): + return os.path.abspath('{}/{}'.format(self.venv, self.venv_relative_python)) + + class PythonLanguage(object): def configure(self, config, args): self.config = config self.args = args - self._tox_envs = self._get_tox_envs(self.args.compiler) + self.pythons = self._get_pythons(self.args.compiler) def test_specs(self): # load list of known test suites @@ -386,33 +396,42 @@ class PythonLanguage(object): environment = dict(_FORCE_ENVIRON_FOR_WRAPPERS) if self.config.build_config != 'gcov': return [self.config.job_spec( - ['tools/run_tests/run_python.sh', tox_env], + ['tools/run_tests/run_python.sh', config.venv_python], + None, environ=dict(environment.items() + [('GRPC_PYTHON_TESTRUNNER_FILTER', suite_name)]), - shortname='%s.test.%s' % (tox_env, suite_name), + shortname='%s.test.%s' % (config.venv, suite_name), timeout_seconds=5*60) for suite_name in tests_json - for tox_env in self._tox_envs] + for config in self.pythons] else: - return [self.config.job_spec(['tools/run_tests/run_python.sh', tox_env], - environ=environment, - shortname='%s.test.coverage' % tox_env, - timeout_seconds=15*60) - for tox_env in self._tox_envs] + return [self.config.job_spec( + ['tools/run_tests/run_python.sh', config.venv_python], + None, + environ=environment, + shortname='%s.test.coverage' % config.venv, + timeout_seconds=15*60) + for config in self.pythons] def pre_build_steps(self): return [] def make_targets(self): - return ['static_c', 'grpc_python_plugin', 'shared_c'] + return [] def make_options(self): return [] def build_steps(self): - return [['tools/run_tests/build_python.sh', tox_env] - for tox_env in self._tox_envs] + return [ + [ + 'tools/run_tests/build_python.sh', + config.python, config.venv, + config.venv_relative_python, config.toolchain + ] + for config in self.pythons + ] def post_tests_steps(self): return [] @@ -423,14 +442,21 @@ class PythonLanguage(object): def dockerfile_dir(self): return 'tools/dockerfile/test/python_jessie_%s' % _docker_arch_suffix(self.args.arch) - def _get_tox_envs(self, compiler): - """Returns name of tox environment based on selected compiler.""" + def _get_pythons(self, compiler): + if os.name == 'nt': + venv_relative_python = 'Scripts/python.exe' + toolchain = 'mingw32' + else: + venv_relative_python = 'bin/python' + toolchain = 'unix' + python27_config = PythonConfig('python2.7', 'py27', venv_relative_python, toolchain) + python34_config = PythonConfig('python3.4', 'py34', venv_relative_python, toolchain) if compiler == 'default': - return ('py27', 'py34') + return (python27_config, python34_config,) elif compiler == 'python2.7': - return ('py27',) + return (python27_config,) elif compiler == 'python3.4': - return ('py34',) + return (python34_config,) else: raise Exception('Compiler %s not supported.' % compiler) From ac586ba21e385319a17b8150b1cd681b96807898 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Sat, 4 Jun 2016 03:24:46 -0700 Subject: [PATCH 0378/1039] Remove tox --- .gitignore | 1 - MANIFEST.md | 1 - src/python/grpcio/.gitignore | 1 - .../tools/dockerfile/python_deps.include | 2 +- .../grpc_artifact_linux_x64/Dockerfile | 2 +- .../grpc_artifact_linux_x86/Dockerfile | 2 +- .../grpc_interop_python/Dockerfile | 2 +- .../grpc_interop_stress_python/Dockerfile | 2 +- .../test/multilang_jessie_x64/Dockerfile | 2 +- .../test/python_jessie_x64/Dockerfile | 2 +- tools/gce/linux_performance_worker_init.sh | 2 -- tox.ini | 26 ------------------- 12 files changed, 7 insertions(+), 38 deletions(-) delete mode 100644 tox.ini diff --git a/.gitignore b/.gitignore index d985a797949..f37989176ea 100644 --- a/.gitignore +++ b/.gitignore @@ -9,7 +9,6 @@ cython_debug/ python_build/ .coverage* .eggs -.tox htmlcov/ dist/ *.egg diff --git a/MANIFEST.md b/MANIFEST.md index 77e014002d9..a0e79e85327 100644 --- a/MANIFEST.md +++ b/MANIFEST.md @@ -19,7 +19,6 @@ * [requirements.txt](requirements.txt) * [setup.cfg](setup.cfg) * [setup.py](setup.py) -* [tox.ini](tox.ini) * [PYTHON-MANIFEST.in](PYTHON-MANIFEST.in) ## Ruby diff --git a/src/python/grpcio/.gitignore b/src/python/grpcio/.gitignore index 6e5e65096c8..7cd8fab2735 100644 --- a/src/python/grpcio/.gitignore +++ b/src/python/grpcio/.gitignore @@ -9,7 +9,6 @@ dist/ .coverage .coverage.* .cache/ -.tox/ nosetests.xml doc/ _grpcio_metadata.py diff --git a/templates/tools/dockerfile/python_deps.include b/templates/tools/dockerfile/python_deps.include index a559f96394c..31623640482 100644 --- a/templates/tools/dockerfile/python_deps.include +++ b/templates/tools/dockerfile/python_deps.include @@ -11,4 +11,4 @@ RUN apt-get update && apt-get install -y ${'\\'} # Install Python packages from PyPI RUN pip install pip --upgrade RUN pip install virtualenv -RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 tox +RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 diff --git a/tools/dockerfile/grpc_artifact_linux_x64/Dockerfile b/tools/dockerfile/grpc_artifact_linux_x64/Dockerfile index 4ae4ebdb06d..669e3557b89 100644 --- a/tools/dockerfile/grpc_artifact_linux_x64/Dockerfile +++ b/tools/dockerfile/grpc_artifact_linux_x64/Dockerfile @@ -76,7 +76,7 @@ RUN apt-get update && apt-get install -y \ RUN pip install pip --upgrade RUN pip install virtualenv -RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 tox +RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 ################## diff --git a/tools/dockerfile/grpc_artifact_linux_x86/Dockerfile b/tools/dockerfile/grpc_artifact_linux_x86/Dockerfile index 9c2fd52eee1..860b8f4fb94 100644 --- a/tools/dockerfile/grpc_artifact_linux_x86/Dockerfile +++ b/tools/dockerfile/grpc_artifact_linux_x86/Dockerfile @@ -76,7 +76,7 @@ RUN apt-get update && apt-get install -y \ RUN pip install pip --upgrade RUN pip install virtualenv -RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 tox +RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 ################## diff --git a/tools/dockerfile/interoptest/grpc_interop_python/Dockerfile b/tools/dockerfile/interoptest/grpc_interop_python/Dockerfile index 071fb2c93b1..6ff9aeca95c 100644 --- a/tools/dockerfile/interoptest/grpc_interop_python/Dockerfile +++ b/tools/dockerfile/interoptest/grpc_interop_python/Dockerfile @@ -76,7 +76,7 @@ RUN apt-get update && apt-get install -y \ # Install Python packages from PyPI RUN pip install pip --upgrade RUN pip install virtualenv -RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 tox +RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 # Prepare ccache RUN ln -s /usr/bin/ccache /usr/local/bin/gcc diff --git a/tools/dockerfile/stress_test/grpc_interop_stress_python/Dockerfile b/tools/dockerfile/stress_test/grpc_interop_stress_python/Dockerfile index 606b7654576..ee6249d381e 100644 --- a/tools/dockerfile/stress_test/grpc_interop_stress_python/Dockerfile +++ b/tools/dockerfile/stress_test/grpc_interop_stress_python/Dockerfile @@ -93,7 +93,7 @@ RUN apt-get update && apt-get install -y \ # Install Python packages from PyPI RUN pip install pip --upgrade RUN pip install virtualenv -RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 tox +RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 RUN pip install coverage diff --git a/tools/dockerfile/test/multilang_jessie_x64/Dockerfile b/tools/dockerfile/test/multilang_jessie_x64/Dockerfile index 5c3f77405ea..66d25a02304 100644 --- a/tools/dockerfile/test/multilang_jessie_x64/Dockerfile +++ b/tools/dockerfile/test/multilang_jessie_x64/Dockerfile @@ -137,7 +137,7 @@ RUN apt-get update && apt-get install -y \ # Install Python packages from PyPI RUN pip install pip --upgrade RUN pip install virtualenv -RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 tox +RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 # Prepare ccache RUN ln -s /usr/bin/ccache /usr/local/bin/gcc diff --git a/tools/dockerfile/test/python_jessie_x64/Dockerfile b/tools/dockerfile/test/python_jessie_x64/Dockerfile index 071fb2c93b1..6ff9aeca95c 100644 --- a/tools/dockerfile/test/python_jessie_x64/Dockerfile +++ b/tools/dockerfile/test/python_jessie_x64/Dockerfile @@ -76,7 +76,7 @@ RUN apt-get update && apt-get install -y \ # Install Python packages from PyPI RUN pip install pip --upgrade RUN pip install virtualenv -RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 tox +RUN pip install futures==2.2.0 enum34==1.0.4 protobuf==3.0.0a2 # Prepare ccache RUN ln -s /usr/bin/ccache /usr/local/bin/gcc diff --git a/tools/gce/linux_performance_worker_init.sh b/tools/gce/linux_performance_worker_init.sh index 9b8d1d1eb74..23a8f082b26 100755 --- a/tools/gce/linux_performance_worker_init.sh +++ b/tools/gce/linux_performance_worker_init.sh @@ -90,13 +90,11 @@ sudo apt-get install -y libgflags-dev libgtest-dev libc++-dev clang # Python dependencies sudo pip install tabulate sudo pip install google-api-python-client -sudo pip install tox curl -O https://bootstrap.pypa.io/get-pip.py sudo pypy get-pip.py sudo pypy -m pip install tabulate sudo pip install google-api-python-client -sudo pip install tox # Node dependencies (nvm has to be installed under user jenkins) touch .profile diff --git a/tox.ini b/tox.ini deleted file mode 100644 index 8f580047e18..00000000000 --- a/tox.ini +++ /dev/null @@ -1,26 +0,0 @@ -# GRPC Python tox (test environment) settings -[tox] -skipsdist = true -envlist = py27,py34 - -[testenv] -setenv = - PYGRPC_ROOT = {toxinidir}/src/python/grpcio/ -commands = - {envpython} setup.py build_py - {envpython} setup.py test - {envbindir}/coverage combine -# TODO(atash): we currently ignore cygrpc.pyx due to an insufficiency in Cython's coverage plug-in. Discussion is ongoing. - {envbindir}/coverage html --include='{env:PYGRPC_ROOT}/grpc/*' --omit='{env:PYGRPC_ROOT}/grpc/framework/alpha/*','{env:PYGRPC_ROOT}/grpc/early_adopter/*','{env:PYGRPC_ROOT}/grpc/framework/base/*','{env:PYGRPC_ROOT}/grpc/framework/face/*','{env:PYGRPC_ROOT}/grpc/_adapter/fore.py','{env:PYGRPC_ROOT}/grpc/_adapter/rear.py','{env:PYGRPC_ROOT}/grpc/_cython/cygrpc.pyx' - {envbindir}/coverage report --include='{env:PYGRPC_ROOT}/grpc/*' --omit='{env:PYGRPC_ROOT}/grpc/framework/alpha/*','{env:PYGRPC_ROOT}/grpc/early_adopter/*','{env:PYGRPC_ROOT}/grpc/framework/base/*','{env:PYGRPC_ROOT}/grpc/framework/face/*','{env:PYGRPC_ROOT}/grpc/_adapter/fore.py','{env:PYGRPC_ROOT}/grpc/_adapter/rear.py','{env:PYGRPC_ROOT}/grpc/_cython/cygrpc.pyx' -deps = - -rrequirements.txt -passenv = * - -[testenv:interop_client] -commands = - {envpython} src/python/grpcio_tests/setup.py run_interop --client --args='{posargs}' - -[testenv:interop_server] -commands = - {envpython} src/python/grpcio_tests/setup.py run_interop --server --args='{posargs}' From 6db60b9041ce8d7bf33254d2daa87ccafd493498 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Mon, 13 Jun 2016 12:16:33 -0700 Subject: [PATCH 0379/1039] Sanitize grpcio-tools command arguments --- tools/distrib/python/grpcio_tools/grpc/tools/protoc.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/distrib/python/grpcio_tools/grpc/tools/protoc.py b/tools/distrib/python/grpcio_tools/grpc/tools/protoc.py index 931fda27d26..e1256a7dd9b 100644 --- a/tools/distrib/python/grpcio_tools/grpc/tools/protoc.py +++ b/tools/distrib/python/grpcio_tools/grpc/tools/protoc.py @@ -41,6 +41,7 @@ def main(command_arguments): command_arguments: a list of strings representing command line arguments to `protoc`. """ + command_arguments = [argument.encode() for argument in command_arguments] return _protoc_compiler.run_main(command_arguments) if __name__ == '__main__': From 0bd13ed8d0ee0f5600b0b8d05907e86d24a7ab1b Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Mon, 13 Jun 2016 19:22:43 -0700 Subject: [PATCH 0380/1039] Fall back to default python for test virtualenvs --- tools/run_tests/build_python.sh | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/tools/run_tests/build_python.sh b/tools/run_tests/build_python.sh index 9d2813fa192..1ad928f2da8 100755 --- a/tools/run_tests/build_python.sh +++ b/tools/run_tests/build_python.sh @@ -33,6 +33,7 @@ set -ex # change to grpc repo root cd $(dirname $0)/../.. +# Arguments PYTHON=${1:-python2.7} VENV=${2:-py27} VENV_RELATIVE_PYTHON=${3:-bin/python} @@ -42,6 +43,10 @@ ROOT=`pwd` export CFLAGS="-I$ROOT/include -std=gnu99 -fno-wrapv" export GRPC_PYTHON_BUILD_WITH_CYTHON=1 +# Default python on the host to fall back to when instantiating e.g. the +# virtualenv. +HOST_PYTHON=${HOST_PYTHON:-python} + # If ccache is available, use it... unless we're on Mac, then all hell breaks # loose because Python does hacky things to support other hacky things done to # hacky things on Mac OS X @@ -71,7 +76,14 @@ if [ "$CONFIG" = "gcov" ]; then export GRPC_PYTHON_ENABLE_CYTHON_TRACING=1 fi -($PYTHON -m virtualenv $VENV || true) +# Instnatiate the virtualenv, preferring to do so from the relevant python +# version. Even if these commands fail (e.g. on Windows due to name conflicts) +# it's possible that the virtualenv is still usable and we trust the tester to +# be able to 'figure it out' instead of us e.g. doing potentially expensive and +# unnecessary error recovery by `rm -rf`ing the virtualenv. +($PYTHON -m virtualenv $VENV || + $HOST_PYTHON -m virtualenv -p $PYTHON $VENV || + true) VENV_PYTHON=`$REALPATH -s "$VENV/$VENV_RELATIVE_PYTHON"` # pip-installs the directory specified. Used because on MSYS the vanilla Windows From 1c062bdd8cedccbfdd3474668e4c6f43af098e23 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Mon, 13 Jun 2016 18:41:36 -0700 Subject: [PATCH 0381/1039] Remove gcov special-casing for Python tests We'll need to fix coverage testing in the future anyway (see #6894). --- tools/run_tests/build_python.sh | 4 ---- tools/run_tests/run_tests.py | 27 +++++++++------------------ 2 files changed, 9 insertions(+), 22 deletions(-) diff --git a/tools/run_tests/build_python.sh b/tools/run_tests/build_python.sh index 1ad928f2da8..687b04e954c 100755 --- a/tools/run_tests/build_python.sh +++ b/tools/run_tests/build_python.sh @@ -72,10 +72,6 @@ else exit 1 fi -if [ "$CONFIG" = "gcov" ]; then - export GRPC_PYTHON_ENABLE_CYTHON_TRACING=1 -fi - # Instnatiate the virtualenv, preferring to do so from the relevant python # version. Even if these commands fail (e.g. on Windows due to name conflicts) # it's possible that the virtualenv is still usable and we trust the tester to diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index fbc5729c758..e9c663695b3 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -394,24 +394,15 @@ class PythonLanguage(object): with open('src/python/grpcio_tests/tests/tests.json') as tests_json_file: tests_json = json.load(tests_json_file) environment = dict(_FORCE_ENVIRON_FOR_WRAPPERS) - if self.config.build_config != 'gcov': - return [self.config.job_spec( - ['tools/run_tests/run_python.sh', config.venv_python], - None, - environ=dict(environment.items() + - [('GRPC_PYTHON_TESTRUNNER_FILTER', suite_name)]), - shortname='%s.test.%s' % (config.venv, suite_name), - timeout_seconds=5*60) - for suite_name in tests_json - for config in self.pythons] - else: - return [self.config.job_spec( - ['tools/run_tests/run_python.sh', config.venv_python], - None, - environ=environment, - shortname='%s.test.coverage' % config.venv, - timeout_seconds=15*60) - for config in self.pythons] + return [self.config.job_spec( + ['tools/run_tests/run_python.sh', config.venv_python], + None, + environ=dict(environment.items() + + [('GRPC_PYTHON_TESTRUNNER_FILTER', suite_name)]), + shortname='%s.test.%s' % (config.venv, suite_name), + timeout_seconds=5*60) + for suite_name in tests_json + for config in self.pythons] def pre_build_steps(self): From e6a23e255bb01eb0006a627b261da67983a0c021 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Tue, 28 Jun 2016 13:58:42 -0700 Subject: [PATCH 0382/1039] Fix job_spec invocation for Python run_tests --- tools/run_tests/run_tests.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index e9c663695b3..8913e9ded30 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -396,15 +396,13 @@ class PythonLanguage(object): environment = dict(_FORCE_ENVIRON_FOR_WRAPPERS) return [self.config.job_spec( ['tools/run_tests/run_python.sh', config.venv_python], - None, + timeout_seconds=5*60, environ=dict(environment.items() + [('GRPC_PYTHON_TESTRUNNER_FILTER', suite_name)]), - shortname='%s.test.%s' % (config.venv, suite_name), - timeout_seconds=5*60) + shortname='%s.test.%s' % (config.venv, suite_name),) for suite_name in tests_json for config in self.pythons] - def pre_build_steps(self): return [] From 4763678016a253b5ed2e33579e52b124f1d8fa21 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Thu, 30 Jun 2016 12:55:01 +0000 Subject: [PATCH 0383/1039] Regenerate project files --- src/python/grpcio_tests/grpc_version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/python/grpcio_tests/grpc_version.py b/src/python/grpcio_tests/grpc_version.py index 478b5d62d6a..7aa600728a8 100644 --- a/src/python/grpcio_tests/grpc_version.py +++ b/src/python/grpcio_tests/grpc_version.py @@ -29,4 +29,4 @@ # AUTO-GENERATED FROM `$REPO_ROOT/templates/src/python/grpcio_tests/grpc_version.py.template`!!! -VERSION='0.15.0.dev0' +VERSION='0.16.0.dev0' From c09854b8feb97c2ac741be4e094897e6f74694a6 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Fri, 1 Jul 2016 10:24:17 -0700 Subject: [PATCH 0384/1039] Fix example Podfiles, and document better --- examples/objective-c/auth_sample/Podfile | 18 +++++++++++++++--- examples/objective-c/helloworld/Podfile | 18 +++++++++++++++--- examples/objective-c/route_guide/Podfile | 19 ++++++++++++++++--- src/objective-c/examples/Sample/Podfile | 7 +++++-- src/objective-c/examples/SwiftSample/Podfile | 7 +++++-- 5 files changed, 56 insertions(+), 13 deletions(-) diff --git a/examples/objective-c/auth_sample/Podfile b/examples/objective-c/auth_sample/Podfile index 7affe08743c..5020df9d752 100644 --- a/examples/objective-c/auth_sample/Podfile +++ b/examples/objective-c/auth_sample/Podfile @@ -1,9 +1,10 @@ source 'https://github.com/CocoaPods/Specs.git' platform :ios, '8.0' -pod 'Protobuf', :path => "../../../third_party/protobuf" -pod 'BoringSSL', :podspec => "../../../src/objective-c" -pod 'gRPC', :path => "../../.." +install! 'cocoapods', :deterministic_uuids => false + +# Location of gRPC's repo root relative to this file. +GRPC_LOCAL_SRC = '../../..' target 'AuthSample' do # Depend on the generated AuthTestService library. @@ -11,4 +12,15 @@ target 'AuthSample' do # Depend on Google's OAuth2 library pod 'Google/SignIn' + + # Use the local versions of Protobuf, BoringSSL, and gRPC. You don't need any of the following + # lines in your application. + pod 'Protobuf', :path => "#{GRPC_LOCAL_SRC}/third_party/protobuf" + + pod 'BoringSSL', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c" + + pod 'gRPC', :path => GRPC_LOCAL_SRC + pod 'gRPC-Core', :path => GRPC_LOCAL_SRC + pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC + pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC end diff --git a/examples/objective-c/helloworld/Podfile b/examples/objective-c/helloworld/Podfile index eebf05470d1..a9f309bd15c 100644 --- a/examples/objective-c/helloworld/Podfile +++ b/examples/objective-c/helloworld/Podfile @@ -1,11 +1,23 @@ source 'https://github.com/CocoaPods/Specs.git' platform :ios, '8.0' -pod 'Protobuf', :path => "../../../third_party/protobuf" -pod 'BoringSSL', :podspec => "../../../src/objective-c" -pod 'gRPC', :path => "../../.." +install! 'cocoapods', :deterministic_uuids => false + +# Location of gRPC's repo root relative to this file. +GRPC_LOCAL_SRC = '../../..' target 'HelloWorld' do # Depend on the generated HelloWorld library. pod 'HelloWorld', :path => '.' + + # Use the local versions of Protobuf, BoringSSL, and gRPC. You don't need any of the following + # lines in your application. + pod 'Protobuf', :path => "#{GRPC_LOCAL_SRC}/third_party/protobuf" + + pod 'BoringSSL', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c" + + pod 'gRPC', :path => GRPC_LOCAL_SRC + pod 'gRPC-Core', :path => GRPC_LOCAL_SRC + pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC + pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC end diff --git a/examples/objective-c/route_guide/Podfile b/examples/objective-c/route_guide/Podfile index b9f2fefd6d2..645309052ed 100644 --- a/examples/objective-c/route_guide/Podfile +++ b/examples/objective-c/route_guide/Podfile @@ -1,10 +1,23 @@ source 'https://github.com/CocoaPods/Specs.git' platform :ios, '8.0' +install! 'cocoapods', :deterministic_uuids => false + +# Location of gRPC's repo root relative to this file. +GRPC_LOCAL_SRC = '../../..' + target 'RouteGuideClient' do - pod 'Protobuf', :path => "../../../third_party/protobuf" - pod 'BoringSSL', :podspec => "../../../src/objective-c" - pod 'gRPC', :path => "../../.." # Depend on the generated RouteGuide library. pod 'RouteGuide', :path => '.' + + # Use the local versions of Protobuf, BoringSSL, and gRPC. You don't need any of the following + # lines in your application. + pod 'Protobuf', :path => "#{GRPC_LOCAL_SRC}/third_party/protobuf" + + pod 'BoringSSL', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c" + + pod 'gRPC', :path => GRPC_LOCAL_SRC + pod 'gRPC-Core', :path => GRPC_LOCAL_SRC + pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC + pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC end diff --git a/src/objective-c/examples/Sample/Podfile b/src/objective-c/examples/Sample/Podfile index 77e37e98afd..acb5f7374a3 100644 --- a/src/objective-c/examples/Sample/Podfile +++ b/src/objective-c/examples/Sample/Podfile @@ -7,6 +7,11 @@ install! 'cocoapods', :deterministic_uuids => false GRPC_LOCAL_SRC = '../../../..' target 'Sample' do + # Depend on the generated RemoteTestClient library + pod 'RemoteTest', :path => "../RemoteTestClient" + + # Use the local versions of Protobuf, BoringSSL, and gRPC. You don't need any of the following + # lines in your application. pod 'Protobuf', :path => "#{GRPC_LOCAL_SRC}/third_party/protobuf" pod 'BoringSSL', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c" @@ -15,6 +20,4 @@ target 'Sample' do pod 'gRPC-Core', :path => GRPC_LOCAL_SRC pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC - - pod 'RemoteTest', :path => "../RemoteTestClient" end diff --git a/src/objective-c/examples/SwiftSample/Podfile b/src/objective-c/examples/SwiftSample/Podfile index 2e789c3ef1c..624e5bb9e0f 100644 --- a/src/objective-c/examples/SwiftSample/Podfile +++ b/src/objective-c/examples/SwiftSample/Podfile @@ -7,6 +7,11 @@ install! 'cocoapods', :deterministic_uuids => false GRPC_LOCAL_SRC = '../../../..' target 'SwiftSample' do + # Depend on the generated RemoteTestClient library + pod 'RemoteTest', :path => "../RemoteTestClient" + + # Use the local versions of Protobuf, BoringSSL, and gRPC. You don't need any of the following + # lines in your application. pod 'Protobuf', :path => "#{GRPC_LOCAL_SRC}/third_party/protobuf" pod 'BoringSSL', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c" @@ -15,6 +20,4 @@ target 'SwiftSample' do pod 'gRPC-Core', :path => GRPC_LOCAL_SRC pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC - - pod 'RemoteTest', :path => "../RemoteTestClient" end From b76471d53b93f9f472c0e50b7058e8b2e0738d69 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 1 Jul 2016 10:24:56 -0700 Subject: [PATCH 0385/1039] Fix TSAN failure in tcp_server (shown via qps_openloop_test) --- src/core/lib/iomgr/tcp_server_posix.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/core/lib/iomgr/tcp_server_posix.c b/src/core/lib/iomgr/tcp_server_posix.c index a1a463550ae..d3803c3bd0e 100644 --- a/src/core/lib/iomgr/tcp_server_posix.c +++ b/src/core/lib/iomgr/tcp_server_posix.c @@ -134,7 +134,7 @@ struct grpc_tcp_server { size_t pollset_count; /* next pollset to assign a channel to */ - size_t next_pollset_to_assign; + gpr_atm next_pollset_to_assign; }; static gpr_once check_init = GPR_ONCE_INIT; @@ -181,7 +181,7 @@ grpc_error *grpc_tcp_server_create(grpc_closure *shutdown_complete, s->head = NULL; s->tail = NULL; s->nports = 0; - s->next_pollset_to_assign = 0; + gpr_atm_no_barrier_store(&s->next_pollset_to_assign, 0); *server = s; return GRPC_ERROR_NONE; } @@ -369,7 +369,8 @@ static void on_read(grpc_exec_ctx *exec_ctx, void *arg, grpc_error *err) { } read_notifier_pollset = - sp->server->pollsets[(sp->server->next_pollset_to_assign++) % + sp->server->pollsets[(size_t)gpr_atm_no_barrier_fetch_add( + &sp->server->next_pollset_to_assign, 1) % sp->server->pollset_count]; /* loop until accept4 returns EAGAIN, and then re-arm notification */ From b19ca30b43401cf5b550ba91498e6daced1dde6b Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 1 Jul 2016 11:26:07 -0700 Subject: [PATCH 0386/1039] Enable workqueue_test on Windows --- build.yaml | 4 - tools/run_tests/tests.json | 6 +- vsprojects/buildtests_c.sln | 27 +++ .../workqueue_test/workqueue_test.vcxproj | 199 ++++++++++++++++++ .../workqueue_test.vcxproj.filters | 21 ++ 5 files changed, 251 insertions(+), 6 deletions(-) create mode 100644 vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj create mode 100644 vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj.filters diff --git a/build.yaml b/build.yaml index 2446b9feb67..681ab54d5b1 100644 --- a/build.yaml +++ b/build.yaml @@ -2433,10 +2433,6 @@ targets: - grpc - gpr_test_util - gpr - platforms: - - mac - - linux - - posix - name: alarm_cpp_test gtest: true build: test diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index dedd55774b6..291ae415a49 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -1919,7 +1919,8 @@ "ci_platforms": [ "linux", "mac", - "posix" + "posix", + "windows" ], "cpu_cost": 1.0, "exclude_configs": [], @@ -1930,7 +1931,8 @@ "platforms": [ "linux", "mac", - "posix" + "posix", + "windows" ] }, { diff --git a/vsprojects/buildtests_c.sln b/vsprojects/buildtests_c.sln index 10be520b5f1..fcf9b1fc22a 100644 --- a/vsprojects/buildtests_c.sln +++ b/vsprojects/buildtests_c.sln @@ -1421,6 +1421,17 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "window_overflow_bad_client_ {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "workqueue_test", "vcxproj\test\workqueue_test\workqueue_test.vcxproj", "{1B6E9651-4D88-2ACB-BEE0-26599919361D}" + ProjectSection(myProperties) = preProject + lib = "False" + EndProjectSection + ProjectSection(ProjectDependencies) = postProject + {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} + {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} + {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + EndProjectSection +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Win32 = Debug|Win32 @@ -3545,6 +3556,22 @@ Global {658D7F7F-9628-6545-743C-D949301DC5DC}.Release-DLL|Win32.Build.0 = Release|Win32 {658D7F7F-9628-6545-743C-D949301DC5DC}.Release-DLL|x64.ActiveCfg = Release|x64 {658D7F7F-9628-6545-743C-D949301DC5DC}.Release-DLL|x64.Build.0 = Release|x64 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug|Win32.ActiveCfg = Debug|Win32 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug|x64.ActiveCfg = Debug|x64 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release|Win32.ActiveCfg = Release|Win32 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release|x64.ActiveCfg = Release|x64 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug|Win32.Build.0 = Debug|Win32 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug|x64.Build.0 = Debug|x64 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release|Win32.Build.0 = Release|Win32 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release|x64.Build.0 = Release|x64 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug-DLL|Win32.ActiveCfg = Debug|Win32 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug-DLL|Win32.Build.0 = Debug|Win32 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug-DLL|x64.ActiveCfg = Debug|x64 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug-DLL|x64.Build.0 = Debug|x64 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release-DLL|Win32.ActiveCfg = Release|Win32 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release-DLL|Win32.Build.0 = Release|Win32 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release-DLL|x64.ActiveCfg = Release|x64 + {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release-DLL|x64.Build.0 = Release|x64 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE diff --git a/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj b/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj new file mode 100644 index 00000000000..35665fa1833 --- /dev/null +++ b/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj @@ -0,0 +1,199 @@ + + + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {1B6E9651-4D88-2ACB-BEE0-26599919361D} + true + $(SolutionDir)IntDir\$(MSBuildProjectName)\ + + + + v100 + + + v110 + + + v120 + + + v140 + + + Application + true + Unicode + + + Application + false + true + Unicode + + + + + + + + + + + + + + workqueue_test + static + Debug + static + Debug + + + workqueue_test + static + Release + static + Release + + + + NotUsing + Level3 + Disabled + WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) + true + MultiThreadedDebug + true + None + false + + + Console + true + false + + + + + + NotUsing + Level3 + Disabled + WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) + true + MultiThreadedDebug + true + None + false + + + Console + true + false + + + + + + NotUsing + Level3 + MaxSpeed + WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) + true + true + true + MultiThreaded + true + None + false + + + Console + true + false + true + true + + + + + + NotUsing + Level3 + MaxSpeed + WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) + true + true + true + MultiThreaded + true + None + false + + + Console + true + false + true + true + + + + + + + + + + {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} + + + {29D16885-7228-4C31-81ED-5F9187C7F2A9} + + + {EAB0A629-17A9-44DB-B5FF-E91A721FE037} + + + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + diff --git a/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj.filters b/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj.filters new file mode 100644 index 00000000000..0c5979e66ae --- /dev/null +++ b/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj.filters @@ -0,0 +1,21 @@ + + + + + test\core\iomgr + + + + + + {3c17ce2b-b57b-9278-8494-48ab4df88ec8} + + + {7035b4d7-88e6-ce95-0011-0ea3cc64eddd} + + + {f0fb09d4-0bdc-a53c-1b5f-d71acbf72a5d} + + + + From 0910e4c0f6e99e621c9e11ac900d65ff7b2eeebb Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Fri, 1 Jul 2016 11:25:01 -0700 Subject: [PATCH 0387/1039] pr comments --- src/core/lib/surface/byte_buffer_reader.c | 2 +- src/csharp/ext/grpc_csharp_ext.c | 4 ++-- src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi | 3 ++- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/src/core/lib/surface/byte_buffer_reader.c b/src/core/lib/surface/byte_buffer_reader.c index 3a85f1f1f93..fb38bf51027 100644 --- a/src/core/lib/surface/byte_buffer_reader.c +++ b/src/core/lib/surface/byte_buffer_reader.c @@ -69,8 +69,8 @@ int grpc_byte_buffer_reader_init(grpc_byte_buffer_reader *reader, "Unexpected error decompressing data for algorithm with enum " "value '%d'. Reading data as if it were uncompressed.", reader->buffer_in->data.raw.compression); - return 0; memset(reader, 0, sizeof(*reader)); + return 0; } else { /* all fine */ reader->buffer_out = grpc_raw_byte_buffer_create(decompressed_slices_buffer.slices, diff --git a/src/csharp/ext/grpc_csharp_ext.c b/src/csharp/ext/grpc_csharp_ext.c index 4009748157b..c670ea65c79 100644 --- a/src/csharp/ext/grpc_csharp_ext.c +++ b/src/csharp/ext/grpc_csharp_ext.c @@ -253,7 +253,7 @@ GPR_EXPORT intptr_t GPR_CALLTYPE grpcsharp_batch_context_recv_message_length( if (!ctx->recv_message) { return -1; } - /* TODO(dgq): check the return value of grpc_byte_buffer_reader_init. */ + /* TODO(issue:#7206): check return value of grpc_byte_buffer_reader_init. */ grpc_byte_buffer_reader_init(&reader, ctx->recv_message); return (intptr_t)grpc_byte_buffer_length(reader.buffer_out); } @@ -268,7 +268,7 @@ GPR_EXPORT void GPR_CALLTYPE grpcsharp_batch_context_recv_message_to_buffer( gpr_slice slice; size_t offset = 0; - /* TODO(dgq): check the return value of grpc_byte_buffer_reader_init. */ + /* TODO(issue:#7206): check return value of grpc_byte_buffer_reader_init. */ grpc_byte_buffer_reader_init(&reader, ctx->recv_message); while (grpc_byte_buffer_reader_next(&reader, &slice)) { diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi index 337cf7fa3bb..7510cc98958 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi @@ -254,7 +254,8 @@ cdef class ByteBuffer: cdef void *data_slice_pointer if self.c_byte_buffer != NULL: with nogil: - # TODO(dgq): check the return value of grpc_byte_buffer_reader_init. + # TODO(issue:#7205): check the return value of + # grpc_byte_buffer_reader_init. grpc_byte_buffer_reader_init(&reader, self.c_byte_buffer) result = bytearray() with nogil: From 6ec3b532977550d6343920104eefe71440b94790 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Fri, 1 Jul 2016 11:34:32 -0700 Subject: [PATCH 0388/1039] Fix runtest.py breakage --- tools/run_tests/run_tests.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index c1254275e6c..b2cc213f808 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -621,10 +621,13 @@ class ObjCLanguage(object): _check_compiler(self.args.compiler, ['default']) def test_specs(self): - return [self.config.job_spec(['src/objective-c/tests/run_tests.sh'], None, - environ=_FORCE_ENVIRON_FOR_WRAPPERS), + return [self.config.job_spec(['src/objective-c/tests/run_tests.sh'], + timeout_seconds=None, + shortname='objc-tests', + environ=_FORCE_ENVIRON_FOR_WRAPPERS), self.config.job_spec(['src/objective-c/tests/build_example_test.sh'], - None, timeout_seconds=15*60, + timeout_seconds=15*60, + shortname='objc-examples-build', environ=_FORCE_ENVIRON_FOR_WRAPPERS)] def pre_build_steps(self): From bf6fd294d26e8a504d45f1f96ac1eb909bc367f4 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 1 Jul 2016 11:35:06 -0700 Subject: [PATCH 0389/1039] Add a minimal (but correct) implementation of workqueue to Windows --- src/core/lib/iomgr/workqueue_windows.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/src/core/lib/iomgr/workqueue_windows.c b/src/core/lib/iomgr/workqueue_windows.c index 275f040b1cc..6dde80b0b2e 100644 --- a/src/core/lib/iomgr/workqueue_windows.c +++ b/src/core/lib/iomgr/workqueue_windows.c @@ -37,4 +37,28 @@ #include "src/core/lib/iomgr/workqueue.h" +grpc_error *grpc_workqueue_create(grpc_exec_ctx *exec_ctx, grpc_workqueue **workqueue) { + return GRPC_ERROR_NONE; +} + +void grpc_workqueue_flush(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue) {} + +#ifdef GRPC_WORKQUEUE_REFCOUNT_DEBUG +void grpc_workqueue_ref(grpc_workqueue *workqueue, const char *file, int line, + const char *reason) { +} +void grpc_workqueue_unref(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, + const char *file, int line, const char *reason) { +} +#else +void grpc_workqueue_ref(grpc_workqueue *workqueue) {} +void grpc_workqueue_unref(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue) {} +#endif + +void grpc_workqueue_add_to_pollset(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, grpc_pollset *pollset) {} + +void grpc_workqueue_enqueue(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, grpc_closure *closure, grpc_error *error) { + grpc_exec_ctx_sched(exec_ctx, closure, error, NULL); +} + #endif /* GPR_WINDOWS */ From d7addc50279dc7f679231f0392d375983415f332 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 1 Jul 2016 11:36:04 -0700 Subject: [PATCH 0390/1039] clang-format --- src/core/lib/iomgr/workqueue_windows.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/src/core/lib/iomgr/workqueue_windows.c b/src/core/lib/iomgr/workqueue_windows.c index 6dde80b0b2e..789e5687c6c 100644 --- a/src/core/lib/iomgr/workqueue_windows.c +++ b/src/core/lib/iomgr/workqueue_windows.c @@ -37,7 +37,8 @@ #include "src/core/lib/iomgr/workqueue.h" -grpc_error *grpc_workqueue_create(grpc_exec_ctx *exec_ctx, grpc_workqueue **workqueue) { +grpc_error *grpc_workqueue_create(grpc_exec_ctx *exec_ctx, + grpc_workqueue **workqueue) { return GRPC_ERROR_NONE; } @@ -45,19 +46,20 @@ void grpc_workqueue_flush(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue) {} #ifdef GRPC_WORKQUEUE_REFCOUNT_DEBUG void grpc_workqueue_ref(grpc_workqueue *workqueue, const char *file, int line, - const char *reason) { -} + const char *reason) {} void grpc_workqueue_unref(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, - const char *file, int line, const char *reason) { -} + const char *file, int line, const char *reason) {} #else void grpc_workqueue_ref(grpc_workqueue *workqueue) {} void grpc_workqueue_unref(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue) {} #endif -void grpc_workqueue_add_to_pollset(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, grpc_pollset *pollset) {} +void grpc_workqueue_add_to_pollset(grpc_exec_ctx *exec_ctx, + grpc_workqueue *workqueue, + grpc_pollset *pollset) {} -void grpc_workqueue_enqueue(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, grpc_closure *closure, grpc_error *error) { +void grpc_workqueue_enqueue(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, + grpc_closure *closure, grpc_error *error) { grpc_exec_ctx_sched(exec_ctx, closure, error, NULL); } From be4e6803444f12bb75e21f37f04004280f74b764 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 1 Jul 2016 11:38:07 -0700 Subject: [PATCH 0391/1039] Add comment, make sure object value is set --- src/core/lib/iomgr/workqueue_windows.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/core/lib/iomgr/workqueue_windows.c b/src/core/lib/iomgr/workqueue_windows.c index 789e5687c6c..52a3e47f376 100644 --- a/src/core/lib/iomgr/workqueue_windows.c +++ b/src/core/lib/iomgr/workqueue_windows.c @@ -37,8 +37,14 @@ #include "src/core/lib/iomgr/workqueue.h" +// Minimal implementation of grpc_workqueue for Windows +// Works by directly enqueuing workqueue items onto the current execution +// context, which is at least correct, if not performant or in the spirit of +// workqueues. + grpc_error *grpc_workqueue_create(grpc_exec_ctx *exec_ctx, grpc_workqueue **workqueue) { + *workqueue = (grpc_workqueue *)1; return GRPC_ERROR_NONE; } From 0f213c399c5e4025146cbd6fb2a63925f37e8b82 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Fri, 1 Jul 2016 11:54:35 -0700 Subject: [PATCH 0392/1039] Fix examples to work with local gRPC-Core --- examples/objective-c/auth_sample/Podfile | 20 ++++++++++++++++++++ examples/objective-c/helloworld/Podfile | 20 ++++++++++++++++++++ examples/objective-c/route_guide/Podfile | 20 ++++++++++++++++++++ src/objective-c/examples/Sample/Podfile | 20 ++++++++++++++++++++ src/objective-c/examples/SwiftSample/Podfile | 20 ++++++++++++++++++++ 5 files changed, 100 insertions(+) diff --git a/examples/objective-c/auth_sample/Podfile b/examples/objective-c/auth_sample/Podfile index 5020df9d752..32157a9dcee 100644 --- a/examples/objective-c/auth_sample/Podfile +++ b/examples/objective-c/auth_sample/Podfile @@ -24,3 +24,23 @@ target 'AuthSample' do pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC end + +# This pre_install hook is only needed to use the local version of gRPC-Core. You don't need it in +# your application. +pre_install do |installer| + # This is the gRPC-Core podspec object, as initialized by its podspec file. + grpc_core_spec = installer.pod_targets.find{|t| t.name == 'gRPC-Core'}.root_spec + + # Copied from gRPC-Core.podspec, except for the adjusted src_root: + src_root = "$(PODS_ROOT)/../#{GRPC_LOCAL_SRC}" + grpc_core_spec.pod_target_xcconfig = { + 'GRPC_SRC_ROOT' => src_root, + 'HEADER_SEARCH_PATHS' => '"$(inherited)" "$(GRPC_SRC_ROOT)/include"', + 'USER_HEADER_SEARCH_PATHS' => '"$(GRPC_SRC_ROOT)"', + # If we don't set these two settings, `include/grpc/support/time.h` and + # `src/core/lib/support/string.h` shadow the system `` and ``, breaking the + # build. + 'USE_HEADERMAP' => 'NO', + 'ALWAYS_SEARCH_USER_PATHS' => 'NO', + } +end diff --git a/examples/objective-c/helloworld/Podfile b/examples/objective-c/helloworld/Podfile index a9f309bd15c..e1bb4ddfd5e 100644 --- a/examples/objective-c/helloworld/Podfile +++ b/examples/objective-c/helloworld/Podfile @@ -21,3 +21,23 @@ target 'HelloWorld' do pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC end + +# This pre_install hook is only needed to use the local version of gRPC-Core. You don't need it in +# your application. +pre_install do |installer| + # This is the gRPC-Core podspec object, as initialized by its podspec file. + grpc_core_spec = installer.pod_targets.find{|t| t.name == 'gRPC-Core'}.root_spec + + # Copied from gRPC-Core.podspec, except for the adjusted src_root: + src_root = "$(PODS_ROOT)/../#{GRPC_LOCAL_SRC}" + grpc_core_spec.pod_target_xcconfig = { + 'GRPC_SRC_ROOT' => src_root, + 'HEADER_SEARCH_PATHS' => '"$(inherited)" "$(GRPC_SRC_ROOT)/include"', + 'USER_HEADER_SEARCH_PATHS' => '"$(GRPC_SRC_ROOT)"', + # If we don't set these two settings, `include/grpc/support/time.h` and + # `src/core/lib/support/string.h` shadow the system `` and ``, breaking the + # build. + 'USE_HEADERMAP' => 'NO', + 'ALWAYS_SEARCH_USER_PATHS' => 'NO', + } +end diff --git a/examples/objective-c/route_guide/Podfile b/examples/objective-c/route_guide/Podfile index 645309052ed..943f5464d89 100644 --- a/examples/objective-c/route_guide/Podfile +++ b/examples/objective-c/route_guide/Podfile @@ -21,3 +21,23 @@ target 'RouteGuideClient' do pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC end + +# This pre_install hook is only needed to use the local version of gRPC-Core. You don't need it in +# your application. +pre_install do |installer| + # This is the gRPC-Core podspec object, as initialized by its podspec file. + grpc_core_spec = installer.pod_targets.find{|t| t.name == 'gRPC-Core'}.root_spec + + # Copied from gRPC-Core.podspec, except for the adjusted src_root: + src_root = "$(PODS_ROOT)/../#{GRPC_LOCAL_SRC}" + grpc_core_spec.pod_target_xcconfig = { + 'GRPC_SRC_ROOT' => src_root, + 'HEADER_SEARCH_PATHS' => '"$(inherited)" "$(GRPC_SRC_ROOT)/include"', + 'USER_HEADER_SEARCH_PATHS' => '"$(GRPC_SRC_ROOT)"', + # If we don't set these two settings, `include/grpc/support/time.h` and + # `src/core/lib/support/string.h` shadow the system `` and ``, breaking the + # build. + 'USE_HEADERMAP' => 'NO', + 'ALWAYS_SEARCH_USER_PATHS' => 'NO', + } +end diff --git a/src/objective-c/examples/Sample/Podfile b/src/objective-c/examples/Sample/Podfile index acb5f7374a3..80ab2c320d4 100644 --- a/src/objective-c/examples/Sample/Podfile +++ b/src/objective-c/examples/Sample/Podfile @@ -21,3 +21,23 @@ target 'Sample' do pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC end + +# This pre_install hook is only needed to use the local version of gRPC-Core. You don't need it in +# your application. +pre_install do |installer| + # This is the gRPC-Core podspec object, as initialized by its podspec file. + grpc_core_spec = installer.pod_targets.find{|t| t.name == 'gRPC-Core'}.root_spec + + # Copied from gRPC-Core.podspec, except for the adjusted src_root: + src_root = "$(PODS_ROOT)/../#{GRPC_LOCAL_SRC}" + grpc_core_spec.pod_target_xcconfig = { + 'GRPC_SRC_ROOT' => src_root, + 'HEADER_SEARCH_PATHS' => '"$(inherited)" "$(GRPC_SRC_ROOT)/include"', + 'USER_HEADER_SEARCH_PATHS' => '"$(GRPC_SRC_ROOT)"', + # If we don't set these two settings, `include/grpc/support/time.h` and + # `src/core/lib/support/string.h` shadow the system `` and ``, breaking the + # build. + 'USE_HEADERMAP' => 'NO', + 'ALWAYS_SEARCH_USER_PATHS' => 'NO', + } +end diff --git a/src/objective-c/examples/SwiftSample/Podfile b/src/objective-c/examples/SwiftSample/Podfile index 624e5bb9e0f..b675fd29ef5 100644 --- a/src/objective-c/examples/SwiftSample/Podfile +++ b/src/objective-c/examples/SwiftSample/Podfile @@ -21,3 +21,23 @@ target 'SwiftSample' do pod 'gRPC-RxLibrary', :path => GRPC_LOCAL_SRC pod 'gRPC-ProtoRPC', :path => GRPC_LOCAL_SRC end + +# This pre_install hook is only needed to use the local version of gRPC-Core. You don't need it in +# your application. +pre_install do |installer| + # This is the gRPC-Core podspec object, as initialized by its podspec file. + grpc_core_spec = installer.pod_targets.find{|t| t.name == 'gRPC-Core'}.root_spec + + # Copied from gRPC-Core.podspec, except for the adjusted src_root: + src_root = "$(PODS_ROOT)/../#{GRPC_LOCAL_SRC}" + grpc_core_spec.pod_target_xcconfig = { + 'GRPC_SRC_ROOT' => src_root, + 'HEADER_SEARCH_PATHS' => '"$(inherited)" "$(GRPC_SRC_ROOT)/include"', + 'USER_HEADER_SEARCH_PATHS' => '"$(GRPC_SRC_ROOT)"', + # If we don't set these two settings, `include/grpc/support/time.h` and + # `src/core/lib/support/string.h` shadow the system `` and ``, breaking the + # build. + 'USE_HEADERMAP' => 'NO', + 'ALWAYS_SEARCH_USER_PATHS' => 'NO', + } +end From eda85c6765bf9b201fe5fa3185b704490f14026e Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 1 Jul 2016 12:45:19 -0700 Subject: [PATCH 0393/1039] Expunge all references to zookeeper --- Makefile | 96 +------- doc/naming.md | 22 -- include/grpc/grpc_zookeeper.h | 59 ----- src/core/ext/resolver/zookeeper/README.md | 1 - templates/Makefile.template | 199 +--------------- .../run_tests_addons_nocache.include | 4 - test/build/zookeeper.c | 43 ---- test/cpp/end2end/shutdown_test.cc | 1 - test/cpp/end2end/zookeeper_test.cc | 219 ------------------ .../grpc_check_generated_pb_files/Dockerfile | 5 - .../grpc_interop_csharp/Dockerfile | 4 - .../interoptest/grpc_interop_cxx/Dockerfile | 4 - .../interoptest/grpc_interop_node/Dockerfile | 4 - .../interoptest/grpc_interop_php/Dockerfile | 4 - .../grpc_interop_python/Dockerfile | 4 - .../interoptest/grpc_interop_ruby/Dockerfile | 4 - .../grpc_interop_stress_node/Dockerfile | 4 - .../grpc_interop_stress_php/Dockerfile | 4 - .../test/csharp_coreclr_x64/Dockerfile | 4 - .../test/csharp_jessie_x64/Dockerfile | 4 - .../dockerfile/test/cxx_jessie_x64/Dockerfile | 4 - .../dockerfile/test/cxx_jessie_x86/Dockerfile | 4 - .../test/cxx_ubuntu1404_x64/Dockerfile | 4 - .../test/cxx_ubuntu1604_x64/Dockerfile | 4 - .../dockerfile/test/cxx_wheezy_x64/Dockerfile | 4 - tools/dockerfile/test/fuzzer/Dockerfile | 4 - .../test/multilang_jessie_x64/Dockerfile | 4 - .../test/node_jessie_x64/Dockerfile | 4 - .../dockerfile/test/php_jessie_x64/Dockerfile | 4 - .../test/python_jessie_x64/Dockerfile | 4 - .../test/ruby_jessie_x64/Dockerfile | 4 - 31 files changed, 20 insertions(+), 713 deletions(-) delete mode 100644 include/grpc/grpc_zookeeper.h delete mode 100644 src/core/ext/resolver/zookeeper/README.md delete mode 100644 test/build/zookeeper.c delete mode 100644 test/cpp/end2end/zookeeper_test.cc diff --git a/Makefile b/Makefile index 8fd86e78ed0..f1d908dce73 100644 --- a/Makefile +++ b/Makefile @@ -492,7 +492,6 @@ PROTOC_CHECK_CMD = which protoc > /dev/null PROTOC_CHECK_VERSION_CMD = protoc --version | grep -q libprotoc.3 DTRACE_CHECK_CMD = which dtrace > /dev/null SYSTEMTAP_HEADERS_CHECK_CMD = $(CC) $(CPPFLAGS) $(CFLAGS) -o $(TMPOUT) test/build/systemtap.c $(LDFLAGS) -ZOOKEEPER_CHECK_CMD = $(CC) $(CPPFLAGS) $(CFLAGS) -o $(TMPOUT) test/build/zookeeper.c $(LDFLAGS) -lzookeeper_mt ifndef REQUIRE_CUSTOM_LIBRARIES_$(CONFIG) HAS_SYSTEM_PERFTOOLS ?= $(shell $(PERFTOOLS_CHECK_CMD) 2> /dev/null && echo true || echo false) @@ -560,8 +559,6 @@ ifeq ($(HAS_SYSTEMTAP),true) CACHE_MK += HAS_SYSTEMTAP = true, endif -HAS_ZOOKEEPER = $(shell $(ZOOKEEPER_CHECK_CMD) 2> /dev/null && echo true || echo false) - # Note that for testing purposes, one can do: # make HAS_EMBEDDED_OPENSSL_ALPN=false # to emulate the fact we do not have OpenSSL in the third_party folder. @@ -705,14 +702,6 @@ PC_LIBS_PRIVATE = $(PC_LIBS_GRPC) PC_LIB = -lgrpc GRPC_UNSECURE_PC_FILE := $(PC_TEMPLATE) -# grpc_zookeeper .pc file -PC_NAME = gRPC zookeeper -PC_DESCRIPTION = gRPC's zookeeper plugin -PC_CFLAGS = -PC_REQUIRES_PRIVATE = -PC_LIBS_PRIVATE = -lzookeeper_mt -GRPC_ZOOKEEPER_PC_FILE := $(PC_TEMPLATE) - PROTOBUF_PKG_CONFIG = false PC_REQUIRES_GRPCXX = @@ -1151,7 +1140,6 @@ run_dep_checks: $(PERFTOOLS_CHECK_CMD) || true $(PROTOBUF_CHECK_CMD) || true $(PROTOC_CHECK_VERSION_CMD) || true - $(ZOOKEEPER_CHECK_CMD) || true third_party/protobuf/configure: $(E) "[AUTOGEN] Preparing protobuf" @@ -1170,29 +1158,16 @@ $(LIBDIR)/$(CONFIG)/protobuf/libprotobuf.a: third_party/protobuf/configure static: static_c static_cxx -static_c: pc_c pc_c_unsecure cache.mk pc_c_zookeeper $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgrpc_cronet.a $(LIBDIR)/$(CONFIG)/libgrpc_unsecure.a static_zookeeper_libs - +static_c: pc_c pc_c_unsecure cache.mk $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgrpc_cronet.a $(LIBDIR)/$(CONFIG)/libgrpc_unsecure.a static_cxx: pc_cxx pc_cxx_unsecure cache.mk $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc++_reflection.a $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure.a shared: shared_c shared_cxx -shared_c: pc_c pc_c_unsecure cache.mk pc_c_zookeeper $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)gpr$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc_cronet$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc_unsecure$(SHARED_VERSION).$(SHARED_EXT) shared_zookeeper_libs - +shared_c: pc_c pc_c_unsecure cache.mk $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)gpr$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc_cronet$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc_unsecure$(SHARED_VERSION).$(SHARED_EXT) shared_cxx: pc_cxx pc_cxx_unsecure cache.mk $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc++_reflection$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) shared_csharp: shared_c $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc_csharp_ext$(SHARED_VERSION).$(SHARED_EXT) -ifeq ($(HAS_ZOOKEEPER),true) -static_zookeeper_libs: -shared_zookeeper_libs: -else - -static_zookeeper_libs: - -shared_zookeeper_libs: - -endif - grpc_csharp_ext: shared_csharp plugins: $(PROTOC_PLUGINS) @@ -1204,12 +1179,6 @@ pc_c: $(LIBDIR)/$(CONFIG)/pkgconfig/grpc.pc pc_c_unsecure: $(LIBDIR)/$(CONFIG)/pkgconfig/grpc_unsecure.pc -ifeq ($(HAS_ZOOKEEPER),true) -pc_c_zookeeper: $(LIBDIR)/$(CONFIG)/pkgconfig/grpc_zookeeper.pc -else -pc_c_zookeeper: -endif - pc_cxx: $(LIBDIR)/$(CONFIG)/pkgconfig/grpc++.pc pc_cxx_unsecure: $(LIBDIR)/$(CONFIG)/pkgconfig/grpc++_unsecure.pc @@ -1221,14 +1190,7 @@ privatelibs_cxx: $(LIBDIR)/$(CONFIG)/libgrpc++_test_config.a $(LIBDIR)/$(CONFIG endif -ifeq ($(HAS_ZOOKEEPER),true) -privatelibs_zookeeper: -else -privatelibs_zookeeper: -endif - - -buildtests: buildtests_c buildtests_cxx buildtests_zookeeper +buildtests: buildtests_c buildtests_cxx buildtests_c: privatelibs_c \ $(BINDIR)/$(CONFIG)/alarm_test \ @@ -1388,7 +1350,7 @@ buildtests_c: privatelibs_c \ ifeq ($(EMBED_OPENSSL),true) -buildtests_cxx: buildtests_zookeeper privatelibs_cxx \ +buildtests_cxx: privatelibs_cxx \ $(BINDIR)/$(CONFIG)/alarm_cpp_test \ $(BINDIR)/$(CONFIG)/async_end2end_test \ $(BINDIR)/$(CONFIG)/auth_property_iterator_test \ @@ -1472,7 +1434,7 @@ buildtests_cxx: buildtests_zookeeper privatelibs_cxx \ $(BINDIR)/$(CONFIG)/boringssl_ssl_test \ else -buildtests_cxx: buildtests_zookeeper privatelibs_cxx \ +buildtests_cxx: privatelibs_cxx \ $(BINDIR)/$(CONFIG)/alarm_cpp_test \ $(BINDIR)/$(CONFIG)/async_end2end_test \ $(BINDIR)/$(CONFIG)/auth_property_iterator_test \ @@ -1520,17 +1482,9 @@ buildtests_cxx: buildtests_zookeeper privatelibs_cxx \ endif -ifeq ($(HAS_ZOOKEEPER),true) -buildtests_zookeeper: privatelibs_zookeeper \ - -else -buildtests_zookeeper: -endif - - -test: test_c test_cxx test_zookeeper +test: test_c test_cxx -flaky_test: flaky_test_c flaky_test_cxx flaky_test_zookeeper +flaky_test: flaky_test_c flaky_test_cxx test_c: buildtests_c $(E) "[RUN] Testing alarm_test" @@ -1752,7 +1706,7 @@ flaky_test_c: buildtests_c $(Q) $(BINDIR)/$(CONFIG)/mlog_test || ( echo test mlog_test failed ; exit 1 ) -test_cxx: test_zookeeper buildtests_cxx +test_cxx: buildtests_cxx $(E) "[RUN] Testing alarm_cpp_test" $(Q) $(BINDIR)/$(CONFIG)/alarm_cpp_test || ( echo test alarm_cpp_test failed ; exit 1 ) $(E) "[RUN] Testing async_end2end_test" @@ -1818,18 +1772,6 @@ test_cxx: test_zookeeper buildtests_cxx flaky_test_cxx: buildtests_cxx -ifeq ($(HAS_ZOOKEEPER),true) -test_zookeeper: buildtests_zookeeper - - -flaky_test_zookeeper: buildtests_zookeeper - -else -test_zookeeper: -flaky_test_zookeeper: -endif - - test_python: static_c $(E) "[RUN] Testing python code" $(Q) tools/run_tests/run_tests.py -lpython -c$(CONFIG) @@ -1867,8 +1809,6 @@ ifeq ($(CONFIG),opt) $(Q) $(STRIP) $(LIBDIR)/$(CONFIG)/libgrpc_cronet.a $(E) "[STRIP] Stripping libgrpc_unsecure.a" $(Q) $(STRIP) $(LIBDIR)/$(CONFIG)/libgrpc_unsecure.a -ifeq ($(HAS_ZOOKEEPER),true) -endif endif strip-static_cxx: static_cxx @@ -1891,8 +1831,6 @@ ifeq ($(CONFIG),opt) $(Q) $(STRIP) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc_cronet$(SHARED_VERSION).$(SHARED_EXT) $(E) "[STRIP] Stripping $(SHARED_PREFIX)grpc_unsecure$(SHARED_VERSION).$(SHARED_EXT)" $(Q) $(STRIP) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)grpc_unsecure$(SHARED_VERSION).$(SHARED_EXT) -ifeq ($(HAS_ZOOKEEPER),true) -endif endif strip-shared_cxx: shared_cxx @@ -1925,11 +1863,6 @@ $(LIBDIR)/$(CONFIG)/pkgconfig/grpc_unsecure.pc: $(Q) mkdir -p $(@D) $(Q) echo "$(GRPC_UNSECURE_PC_FILE)" | tr , '\n' >$@ -$(LIBDIR)/$(CONFIG)/pkgconfig/grpc_zookeeper.pc: - $(E) "[MAKE] Generating $@" - $(Q) mkdir -p $(@D) - $(Q) echo "$(GRPC_ZOOKEEPER_PC_FILE)" | tr , '\n' >$@ - $(LIBDIR)/$(CONFIG)/pkgconfig/grpc++.pc: $(E) "[MAKE] Generating $@" $(Q) mkdir -p $(@D) @@ -2205,8 +2138,6 @@ install-static_c: static_c strip-static_c install-pkg-config_c $(E) "[INSTALL] Installing libgrpc_unsecure.a" $(Q) $(INSTALL) -d $(prefix)/lib $(Q) $(INSTALL) $(LIBDIR)/$(CONFIG)/libgrpc_unsecure.a $(prefix)/lib/libgrpc_unsecure.a -ifeq ($(HAS_ZOOKEEPER),true) -endif install-static_cxx: static_cxx strip-static_cxx install-pkg-config_cxx $(E) "[INSTALL] Installing libgrpc++.a" @@ -2258,8 +2189,6 @@ else ifneq ($(SYSTEM),Darwin) $(Q) ln -sf $(SHARED_PREFIX)grpc_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(prefix)/lib/libgrpc_unsecure.so.0 $(Q) ln -sf $(SHARED_PREFIX)grpc_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(prefix)/lib/libgrpc_unsecure.so endif -ifeq ($(HAS_ZOOKEEPER),true) -endif ifneq ($(SYSTEM),MINGW32) ifneq ($(SYSTEM),Darwin) $(Q) ldconfig || true @@ -2295,8 +2224,6 @@ else ifneq ($(SYSTEM),Darwin) $(Q) ln -sf $(SHARED_PREFIX)grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(prefix)/lib/libgrpc++_unsecure.so.0 $(Q) ln -sf $(SHARED_PREFIX)grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(prefix)/lib/libgrpc++_unsecure.so endif -ifeq ($(HAS_ZOOKEEPER),true) -endif ifneq ($(SYSTEM),MINGW32) ifneq ($(SYSTEM),Darwin) $(Q) ldconfig || true @@ -2314,8 +2241,6 @@ else ifneq ($(SYSTEM),Darwin) $(Q) ln -sf $(SHARED_PREFIX)grpc_csharp_ext$(SHARED_VERSION).$(SHARED_EXT) $(prefix)/lib/libgrpc_csharp_ext.so.0 $(Q) ln -sf $(SHARED_PREFIX)grpc_csharp_ext$(SHARED_VERSION).$(SHARED_EXT) $(prefix)/lib/libgrpc_csharp_ext.so endif -ifeq ($(HAS_ZOOKEEPER),true) -endif ifneq ($(SYSTEM),MINGW32) ifneq ($(SYSTEM),Darwin) $(Q) ldconfig || true @@ -2342,14 +2267,11 @@ else $(Q) $(INSTALL) $(BINDIR)/$(CONFIG)/grpc_ruby_plugin $(prefix)/bin/grpc_ruby_plugin endif -install-pkg-config_c: pc_c pc_c_unsecure pc_c_zookeeper +install-pkg-config_c: pc_c pc_c_unsecure $(E) "[INSTALL] Installing C pkg-config files" $(Q) $(INSTALL) -d $(prefix)/lib/pkgconfig $(Q) $(INSTALL) $(LIBDIR)/$(CONFIG)/pkgconfig/grpc.pc $(prefix)/lib/pkgconfig/grpc.pc $(Q) $(INSTALL) $(LIBDIR)/$(CONFIG)/pkgconfig/grpc_unsecure.pc $(prefix)/lib/pkgconfig/grpc_unsecure.pc -ifeq ($(HAS_ZOOKEEPER),true) - $(Q) $(INSTALL) $(LIBDIR)/$(CONFIG)/pkgconfig/grpc_zookeeper.pc $(prefix)/lib/pkgconfig/grpc_zookeeper.pc -endif install-pkg-config_cxx: pc_cxx pc_cxx_unsecure $(E) "[INSTALL] Installing C++ pkg-config files" diff --git a/doc/naming.md b/doc/naming.md index 5ad7e6622ed..d0c892e8d92 100644 --- a/doc/naming.md +++ b/doc/naming.md @@ -16,8 +16,6 @@ Here, scheme indicates the name-system to be used. Example schemes to be support * `dns` -* `zookeeper` - * `etcd` Authority indicates some scheme-specific bootstrap information, e.g., for DNS, the authority may include the IP[:port] of the DNS server to use. Often, a DNS name may used as the authority, since the ability to resolve DNS names is already built into all gRPC client libraries. @@ -30,23 +28,3 @@ The gRPC client library will switch on the scheme to pick the right resolver plu Resolvers should be able to contact the authority and get a resolution that they return back to the gRPC client library. The returned contents include a list of IP:port, an optional config and optional auth config data to be used for channel authentication. The plugin API allows the resolvers to continuously watch an endpoint_name and return updated resolutions as needed. -## Zookeeper - -Apache [ZooKeeper](https://zookeeper.apache.org/) is a popular solution for building name-systems. Curator is a service discovery system built on to of ZooKeeper. We propose to organize names hierarchically as `/path/service/instance` similar to Apache Curator. - -A fully-qualified ZooKeeper name used to construct a gRPC channel will look as follows: - -``` -zookeeper://host:port/path/service/instance -``` -Here `zookeeper` is the scheme identifying the name-system. `host:port` identifies an authoritative name-server for this scheme (i.e., a Zookeeper server). The host can be an IP address or a DNS name. -Finally `/path/service/instance` is the Zookeeper name to be resolved. - -## Service Registration - - -Service providers can register their services in Zookeeper by using a Zookeeper client. - -Each service is a zookeeper node, and each instance is a child node of the corresponding service. For example, a MySQL service may have multiple instances, `/mysql/1`, `/mysql/2`, `/mysql/3`. The name of the service or instance, as well as an optional path is specified by the service provider. - -The data in service nodes is empty. Each instance node stores its address in the format of `host:port`, where host can be either hostname or IP address. diff --git a/include/grpc/grpc_zookeeper.h b/include/grpc/grpc_zookeeper.h deleted file mode 100644 index 2b195c18bff..00000000000 --- a/include/grpc/grpc_zookeeper.h +++ /dev/null @@ -1,59 +0,0 @@ -/* - * - * Copyright 2015, Google Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following disclaimer - * in the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Google Inc. nor the names of its - * contributors may be used to endorse or promote products derived from - * this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - */ - -/** Support zookeeper as alternative name system in addition to DNS - * Zookeeper name in gRPC is represented as a URI: - * zookeeper://host:port/path/service/instance - * - * Where zookeeper is the name system scheme - * host:port is the address of a zookeeper server - * /path/service/instance is the zookeeper name to be resolved - * - * Refer doc/naming.md for more details - */ - -#ifndef GRPC_GRPC_ZOOKEEPER_H -#define GRPC_GRPC_ZOOKEEPER_H - -#ifdef __cplusplus -extern "C" { -#endif - -/** Register zookeeper name resolver in grpc */ -void grpc_zookeeper_register(); - -#ifdef __cplusplus -} -#endif - -#endif /* GRPC_GRPC_ZOOKEEPER_H */ diff --git a/src/core/ext/resolver/zookeeper/README.md b/src/core/ext/resolver/zookeeper/README.md deleted file mode 100644 index ce6f39683bb..00000000000 --- a/src/core/ext/resolver/zookeeper/README.md +++ /dev/null @@ -1 +0,0 @@ -Zookeeper based name resolver: WIP diff --git a/templates/Makefile.template b/templates/Makefile.template index 0e3b9926b7d..0cbd8bfdd55 100644 --- a/templates/Makefile.template +++ b/templates/Makefile.template @@ -380,7 +380,6 @@ PROTOC_CHECK_VERSION_CMD = protoc --version | grep -q libprotoc.3 DTRACE_CHECK_CMD = which dtrace > /dev/null SYSTEMTAP_HEADERS_CHECK_CMD = $(CC) $(CPPFLAGS) $(CFLAGS) -o $(TMPOUT) test/build/systemtap.c $(LDFLAGS) - ZOOKEEPER_CHECK_CMD = $(CC) $(CPPFLAGS) $(CFLAGS) -o $(TMPOUT) test/build/zookeeper.c $(LDFLAGS) -lzookeeper_mt ifndef REQUIRE_CUSTOM_LIBRARIES_$(CONFIG) HAS_SYSTEM_PERFTOOLS ?= $(shell $(PERFTOOLS_CHECK_CMD) 2> /dev/null && echo true || echo false) @@ -448,8 +447,6 @@ CACHE_MK += HAS_SYSTEMTAP = true, endif - HAS_ZOOKEEPER = $(shell $(ZOOKEEPER_CHECK_CMD) 2> /dev/null && echo true || echo false) - # Note that for testing purposes, one can do: # make HAS_EMBEDDED_OPENSSL_ALPN=false # to emulate the fact we do not have OpenSSL in the third_party folder. @@ -593,14 +590,6 @@ PC_LIB = -lgrpc GRPC_UNSECURE_PC_FILE := $(PC_TEMPLATE) - # grpc_zookeeper .pc file - PC_NAME = gRPC zookeeper - PC_DESCRIPTION = gRPC's zookeeper plugin - PC_CFLAGS = - PC_REQUIRES_PRIVATE = - PC_LIBS_PRIVATE = -lzookeeper_mt - GRPC_ZOOKEEPER_PC_FILE := $(PC_TEMPLATE) - PROTOBUF_PKG_CONFIG = false PC_REQUIRES_GRPCXX = @@ -796,7 +785,6 @@ $(PERFTOOLS_CHECK_CMD) || true $(PROTOBUF_CHECK_CMD) || true $(PROTOC_CHECK_VERSION_CMD) || true - $(ZOOKEEPER_CHECK_CMD) || true third_party/protobuf/configure: $(E) "[AUTOGEN] Preparing protobuf" @@ -815,7 +803,7 @@ static: static_c static_cxx - static_c: pc_c pc_c_unsecure cache.mk pc_c_zookeeper\ + static_c: pc_c pc_c_unsecure cache.mk \ % for lib in libs: % if 'Makefile' in lib.get('build_system', ['Makefile']): % if lib.build == 'all' and lib.language == 'c' and not lib.get('external_deps', None): @@ -823,7 +811,6 @@ % endif % endif % endfor - static_zookeeper_libs static_cxx: pc_cxx pc_cxx_unsecure cache.mk \ @@ -838,7 +825,7 @@ shared: shared_c shared_cxx - shared_c: pc_c pc_c_unsecure cache.mk pc_c_zookeeper\ + shared_c: pc_c pc_c_unsecure cache.mk\ % for lib in libs: % if 'Makefile' in lib.get('build_system', ['Makefile']): % if lib.build == 'all' and lib.language == 'c' and not lib.get('external_deps', None): @@ -846,7 +833,6 @@ % endif % endif % endfor - shared_zookeeper_libs shared_cxx: pc_cxx pc_cxx_unsecure cache.mk\ % for lib in libs: @@ -867,33 +853,6 @@ % endif % endfor - ifeq ($(HAS_ZOOKEEPER),true) - static_zookeeper_libs:\ - % for lib in libs: - % if 'Makefile' in lib.get('build_system', ['Makefile']): - % if lib.build == 'all' and lib.language == 'c' and 'zookeeper' in lib.get('external_deps', []): - $(LIBDIR)/$(CONFIG)/lib${lib.name}.a\ - % endif - % endif - % endfor - - shared_zookeeper_libs:\ - % for lib in libs: - % if 'Makefile' in lib.get('build_system', ['Makefile']): - % if lib.build == 'all' and lib.language == 'c' and 'zookeeper' in lib.get('external_deps', []): - $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)${lib.name}$(SHARED_VERSION).$(SHARED_EXT)\ - % endif - % endif - % endfor - - else - - static_zookeeper_libs: - - shared_zookeeper_libs: - - endif - grpc_csharp_ext: shared_csharp plugins: $(PROTOC_PLUGINS) @@ -913,12 +872,6 @@ pc_c_unsecure: $(LIBDIR)/$(CONFIG)/pkgconfig/grpc_unsecure.pc - ifeq ($(HAS_ZOOKEEPER),true) - pc_c_zookeeper: $(LIBDIR)/$(CONFIG)/pkgconfig/grpc_zookeeper.pc - else - pc_c_zookeeper: - endif - pc_cxx: $(LIBDIR)/$(CONFIG)/pkgconfig/grpc++.pc pc_cxx_unsecure: $(LIBDIR)/$(CONFIG)/pkgconfig/grpc++_unsecure.pc @@ -932,7 +885,7 @@ % endif % endif % endfor - + else privatelibs_cxx: \ % for lib in libs: @@ -942,26 +895,11 @@ % endif % endif % endfor - - endif - - - ifeq ($(HAS_ZOOKEEPER),true) - privatelibs_zookeeper: \ - % for lib in libs: - % if 'Makefile' in lib.get('build_system', ['Makefile']): - % if lib.build == 'private' and lib.language == 'c++' and zookeeper in lib.get('external_deps', []): - $(LIBDIR)/$(CONFIG)/lib${lib.name}.a\ - % endif - % endif - % endfor - else - privatelibs_zookeeper: endif - buildtests: buildtests_c buildtests_cxx buildtests_zookeeper + buildtests: buildtests_c buildtests_cxx buildtests_c: privatelibs_c <%text>\ % for tgt in targets: @@ -972,40 +910,27 @@ ifeq ($(EMBED_OPENSSL),true) - buildtests_cxx: buildtests_zookeeper privatelibs_cxx <%text>\ + buildtests_cxx: privatelibs_cxx <%text>\ % for tgt in targets: % if tgt.build == 'test' and tgt.language == 'c++' and not tgt.get('external_deps', None): $(BINDIR)/$(CONFIG)/${tgt.name} <%text>\ % endif % endfor - + else - buildtests_cxx: buildtests_zookeeper privatelibs_cxx <%text>\ + buildtests_cxx: privatelibs_cxx <%text>\ % for tgt in targets: % if tgt.build == 'test' and tgt.language == 'c++' and not tgt.get('external_deps', None) and not tgt.boringssl: $(BINDIR)/$(CONFIG)/${tgt.name} <%text>\ % endif % endfor - - endif - - - ifeq ($(HAS_ZOOKEEPER),true) - buildtests_zookeeper: privatelibs_zookeeper <%text>\ - % for tgt in targets: - % if tgt.build == 'test' and tgt.language == 'c++' and 'zookeeper' in tgt.get('external_deps', []): - $(BINDIR)/$(CONFIG)/${tgt.name} <%text>\ - % endif - % endfor - else - buildtests_zookeeper: endif - test: test_c test_cxx test_zookeeper + test: test_c test_cxx - flaky_test: flaky_test_c flaky_test_cxx flaky_test_zookeeper + flaky_test: flaky_test_c flaky_test_cxx test_c: buildtests_c % for tgt in targets: @@ -1025,7 +950,7 @@ % endfor - test_cxx: test_zookeeper buildtests_cxx + test_cxx: buildtests_cxx % for tgt in targets: % if tgt.build == 'test' and tgt.get('run', True) and tgt.language == 'c++' and not tgt.get('flaky', False) and not tgt.get('external_deps', None): $(E) "[RUN] Testing ${tgt.name}" @@ -1043,30 +968,6 @@ % endfor - ifeq ($(HAS_ZOOKEEPER),true) - test_zookeeper: buildtests_zookeeper - % for tgt in targets: - % if tgt.build == 'test' and tgt.get('run', True) and tgt.language == 'c++' and not tgt.get('flaky', False) and 'zookeeper' in tgt.get('external_deps', []): - $(E) "[RUN] Testing ${tgt.name}" - $(Q) $(BINDIR)/$(CONFIG)/${tgt.name} || ( echo test ${tgt.name} failed ; exit 1 ) - % endif - % endfor - - - flaky_test_zookeeper: buildtests_zookeeper - % for tgt in targets: - % if tgt.build == 'test' and tgt.get('run', True) and tgt.language == 'c++' and tgt.get('flaky', False) and 'zookeeper' in tgt.get('external_deps', []): - $(E) "[RUN] Testing ${tgt.name}" - $(Q) $(BINDIR)/$(CONFIG)/${tgt.name} || ( echo test ${tgt.name} failed ; exit 1 ) - % endif - % endfor - - else - test_zookeeper: - flaky_test_zookeeper: - endif - - test_python: static_c $(E) "[RUN] Testing python code" $(Q) tools/run_tests/run_tests.py -lpython -c$(CONFIG) @@ -1126,20 +1027,6 @@ % endif % endif % endfor - ifeq ($(HAS_ZOOKEEPER),true) - % for lib in libs: - % if 'Makefile' in lib.get('build_system', ['Makefile']): - % if lib.language == "c": - % if lib.build == "all": - % if 'zookeeper' in lib.get('external_deps', []): - $(E) "[STRIP] Stripping lib${lib.name}.a" - $(Q) $(STRIP) $(LIBDIR)/$(CONFIG)/lib${lib.name}.a - % endif - % endif - % endif - % endif - % endfor - endif endif strip-static_cxx: static_cxx @@ -1170,20 +1057,6 @@ % endif % endif % endfor - ifeq ($(HAS_ZOOKEEPER),true) - % for lib in libs: - % if 'Makefile' in lib.get('build_system', ['Makefile']): - % if lib.language == "c": - % if lib.build == "all": - % if 'zookeeper' in lib.get('external_deps', []): - $(E) "[STRIP] Stripping $(SHARED_PREFIX)${lib.name}$(SHARED_VERSION).$(SHARED_EXT)" - $(Q) $(STRIP) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)${lib.name}$(SHARED_VERSION).$(SHARED_EXT) - % endif - % endif - % endif - % endif - % endfor - endif endif strip-shared_cxx: shared_cxx @@ -1228,11 +1101,6 @@ $(Q) mkdir -p $(@D) $(Q) echo "$(GRPC_UNSECURE_PC_FILE)" | tr , '\n' >$@ - $(LIBDIR)/$(CONFIG)/pkgconfig/grpc_zookeeper.pc: - $(E) "[MAKE] Generating $@" - $(Q) mkdir -p $(@D) - $(Q) echo "$(GRPC_ZOOKEEPER_PC_FILE)" | tr , '\n' >$@ - $(LIBDIR)/$(CONFIG)/pkgconfig/grpc++.pc: $(E) "[MAKE] Generating $@" $(Q) mkdir -p $(@D) @@ -1331,21 +1199,6 @@ % endif % endif % endfor - ifeq ($(HAS_ZOOKEEPER),true) - % for lib in libs: - % if 'Makefile' in lib.get('build_system', ['Makefile']): - % if lib.language == "c": - % if lib.build == "all": - % if 'zookeeper' in lib.get('external_deps', []): - $(E) "[INSTALL] Installing lib${lib.name}.a" - $(Q) $(INSTALL) -d $(prefix)/lib - $(Q) $(INSTALL) $(LIBDIR)/$(CONFIG)/lib${lib.name}.a $(prefix)/lib/lib${lib.name}.a - % endif - % endif - % endif - % endif - % endfor - endif install-static_cxx: static_cxx strip-static_cxx install-pkg-config_cxx % for lib in libs: @@ -1380,27 +1233,6 @@ % endif % endif % endfor - ifeq ($(HAS_ZOOKEEPER),true) - % for lib in libs: - % if 'Makefile' in lib.get('build_system', ['Makefile']): - % if lib.language == lang_filter: - % if lib.build == "all": - % if 'zookeeper' in lib.get('external_deps', []): - $(E) "[INSTALL] Installing $(SHARED_PREFIX)${lib.name}$(SHARED_VERSION).$(SHARED_EXT)" - $(Q) $(INSTALL) -d $(prefix)/lib - $(Q) $(INSTALL) $(LIBDIR)/$(CONFIG)/$(SHARED_PREFIX)${lib.name}$(SHARED_VERSION).$(SHARED_EXT) $(prefix)/lib/$(SHARED_PREFIX)${lib.name}$(SHARED_VERSION).$(SHARED_EXT) - ifeq ($(SYSTEM),MINGW32) - $(Q) $(INSTALL) $(LIBDIR)/$(CONFIG)/lib${lib.name}-imp.a $(prefix)/lib/lib${lib.name}-imp.a - else ifneq ($(SYSTEM),Darwin) - $(Q) ln -sf $(SHARED_PREFIX)${lib.name}$(SHARED_VERSION).$(SHARED_EXT) $(prefix)/lib/lib${lib.name}.so.${settings.core_version.major} - $(Q) ln -sf $(SHARED_PREFIX)${lib.name}$(SHARED_VERSION).$(SHARED_EXT) $(prefix)/lib/lib${lib.name}.so - endif - % endif - % endif - % endif - % endif - % endfor - endif ifneq ($(SYSTEM),MINGW32) ifneq ($(SYSTEM),Darwin) $(Q) ldconfig || true @@ -1430,14 +1262,11 @@ % endfor endif - install-pkg-config_c: pc_c pc_c_unsecure pc_c_zookeeper + install-pkg-config_c: pc_c pc_c_unsecure $(E) "[INSTALL] Installing C pkg-config files" $(Q) $(INSTALL) -d $(prefix)/lib/pkgconfig $(Q) $(INSTALL) $(LIBDIR)/$(CONFIG)/pkgconfig/grpc.pc $(prefix)/lib/pkgconfig/grpc.pc $(Q) $(INSTALL) $(LIBDIR)/$(CONFIG)/pkgconfig/grpc_unsecure.pc $(prefix)/lib/pkgconfig/grpc_unsecure.pc - ifeq ($(HAS_ZOOKEEPER),true) - $(Q) $(INSTALL) $(LIBDIR)/$(CONFIG)/pkgconfig/grpc_zookeeper.pc $(prefix)/lib/pkgconfig/grpc_zookeeper.pc - endif install-pkg-config_cxx: pc_cxx pc_cxx_unsecure $(E) "[INSTALL] Installing C++ pkg-config files" @@ -1645,9 +1474,6 @@ for src in lib.src: sources_that_don_t_need_openssl.add(src) - if 'zookeeper' in lib.get('external_deps', []): - libs = libs + ' -lzookeeper_mt' - if lib.get('secure', 'check') == True or lib.get('secure', 'check') == 'check': lib_deps = lib_deps + ' $(OPENSSL_DEP)' mingw_lib_deps = mingw_lib_deps + ' $(OPENSSL_DEP)' @@ -1802,9 +1628,6 @@ % for dep in tgt.deps: $(LIBDIR)/$(CONFIG)/lib${dep}.a\ % endfor - % if 'zookeeper' in tgt.get('external_deps', []): - -lzookeeper_mt\ - % endif % if tgt.language == "c++": % if tgt.build == 'protoc': $(HOST_LDLIBSXX) $(HOST_LDLIBS_PROTOC)\ diff --git a/templates/tools/dockerfile/run_tests_addons_nocache.include b/templates/tools/dockerfile/run_tests_addons_nocache.include index 242a1acfb3a..74b01e386c3 100644 --- a/templates/tools/dockerfile/run_tests_addons_nocache.include +++ b/templates/tools/dockerfile/run_tests_addons_nocache.include @@ -1,6 +1,2 @@ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/test/build/zookeeper.c b/test/build/zookeeper.c deleted file mode 100644 index 7cd3d0da9e9..00000000000 --- a/test/build/zookeeper.c +++ /dev/null @@ -1,43 +0,0 @@ -/* - * - * Copyright 2015, Google Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following disclaimer - * in the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Google Inc. nor the names of its - * contributors may be used to endorse or promote products derived from - * this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - */ - -/* This is just a compilation test, to see if we have Zookeeper C client - library installed. */ - -#include -#include - -int main() { - zookeeper_init(NULL, NULL, 0, 0, 0, 0); - return 0; -} diff --git a/test/cpp/end2end/shutdown_test.cc b/test/cpp/end2end/shutdown_test.cc index aa8d42141d4..3f98de6db7b 100644 --- a/test/cpp/end2end/shutdown_test.cc +++ b/test/cpp/end2end/shutdown_test.cc @@ -123,7 +123,6 @@ class ShutdownTest : public ::testing::Test { TestServiceImpl service_; }; -// Tests zookeeper state change between two RPCs // TODO(ctiller): leaked objects in this test TEST_F(ShutdownTest, ShutdownTest) { ResetStub(); diff --git a/test/cpp/end2end/zookeeper_test.cc b/test/cpp/end2end/zookeeper_test.cc deleted file mode 100644 index fdc500e5352..00000000000 --- a/test/cpp/end2end/zookeeper_test.cc +++ /dev/null @@ -1,219 +0,0 @@ -/* - * - * Copyright 2015, Google Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following disclaimer - * in the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Google Inc. nor the names of its - * contributors may be used to endorse or promote products derived from - * this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "src/core/lib/support/env.h" -#include "src/proto/grpc/testing/echo.grpc.pb.h" -#include "test/core/util/port.h" -#include "test/core/util/test_config.h" - -using grpc::testing::EchoRequest; -using grpc::testing::EchoResponse; - -namespace grpc { -namespace testing { - -class ZookeeperTestServiceImpl - : public ::grpc::testing::EchoTestService::Service { - public: - Status Echo(ServerContext* context, const EchoRequest* request, - EchoResponse* response) GRPC_OVERRIDE { - response->set_message(request->message()); - return Status::OK; - } -}; - -class ZookeeperTest : public ::testing::Test { - protected: - ZookeeperTest() {} - - void SetUp() GRPC_OVERRIDE { - SetUpZookeeper(); - - // Sets up two servers - int port1 = grpc_pick_unused_port_or_die(); - server1_ = SetUpServer(port1); - - int port2 = grpc_pick_unused_port_or_die(); - server2_ = SetUpServer(port2); - - // Registers service /test in zookeeper - RegisterService("/test", "test"); - - // Registers service instance /test/1 in zookeeper - string value = - "{\"host\":\"localhost\",\"port\":\"" + to_string(port1) + "\"}"; - RegisterService("/test/1", value); - - // Registers service instance /test/2 in zookeeper - value = "{\"host\":\"localhost\",\"port\":\"" + to_string(port2) + "\"}"; - RegisterService("/test/2", value); - } - - // Requires zookeeper server running - void SetUpZookeeper() { - // Finds zookeeper server address in environment - // Default is localhost:2181 - zookeeper_address_ = "localhost:2181"; - char* addr = gpr_getenv("GRPC_ZOOKEEPER_SERVER_TEST"); - if (addr != NULL) { - string addr_str(addr); - zookeeper_address_ = addr_str; - gpr_free(addr); - } - gpr_log(GPR_DEBUG, "%s", zookeeper_address_.c_str()); - - // Connects to zookeeper server - zoo_set_debug_level(ZOO_LOG_LEVEL_WARN); - zookeeper_handle_ = - zookeeper_init(zookeeper_address_.c_str(), NULL, 15000, 0, 0, 0); - GPR_ASSERT(zookeeper_handle_ != NULL); - - // Registers zookeeper name resolver in grpc - grpc_zookeeper_register(); - } - - std::unique_ptr SetUpServer(const int port) { - string server_address = "localhost:" + to_string(port); - - ServerBuilder builder; - builder.AddListeningPort(server_address, InsecureServerCredentials()); - builder.RegisterService(&service_); - std::unique_ptr server = builder.BuildAndStart(); - return server; - } - - void RegisterService(const string& name, const string& value) { - char* path = (char*)gpr_malloc(name.size()); - - int status = zoo_exists(zookeeper_handle_, name.c_str(), 0, NULL); - if (status == ZNONODE) { - status = - zoo_create(zookeeper_handle_, name.c_str(), value.c_str(), - value.size(), &ZOO_OPEN_ACL_UNSAFE, 0, path, name.size()); - } else { - status = zoo_set(zookeeper_handle_, name.c_str(), value.c_str(), - value.size(), -1); - } - gpr_free(path); - GPR_ASSERT(status == 0); - } - - void DeleteService(const string& name) { - int status = zoo_delete(zookeeper_handle_, name.c_str(), -1); - GPR_ASSERT(status == 0); - } - - void ChangeZookeeperState() { - server1_->Shutdown(); - DeleteService("/test/1"); - } - - void TearDown() GRPC_OVERRIDE { - server1_->Shutdown(); - server2_->Shutdown(); - zookeeper_close(zookeeper_handle_); - } - - void ResetStub() { - string target = "zookeeper://" + zookeeper_address_ + "/test"; - channel_ = CreateChannel(target, InsecureChannelCredentials()); - stub_ = grpc::testing::EchoTestService::NewStub(channel_); - } - - string to_string(const int number) { - std::stringstream strs; - strs << number; - return strs.str(); - } - - std::shared_ptr channel_; - std::unique_ptr stub_; - std::unique_ptr server1_; - std::unique_ptr server2_; - ZookeeperTestServiceImpl service_; - zhandle_t* zookeeper_handle_; - string zookeeper_address_; -}; - -// Tests zookeeper state change between two RPCs -// TODO(ctiller): leaked objects in this test -TEST_F(ZookeeperTest, ZookeeperStateChangeTwoRpc) { - ResetStub(); - - // First RPC - EchoRequest request1; - EchoResponse response1; - ClientContext context1; - context1.set_authority("test"); - request1.set_message("Hello"); - Status s1 = stub_->Echo(&context1, request1, &response1); - EXPECT_EQ(response1.message(), request1.message()); - EXPECT_TRUE(s1.ok()); - - // Zookeeper state changes - gpr_log(GPR_DEBUG, "Zookeeper state change"); - ChangeZookeeperState(); - // Waits for re-resolving addresses - // TODO(ctiller): RPC will probably fail if not waiting - sleep(1); - - // Second RPC - EchoRequest request2; - EchoResponse response2; - ClientContext context2; - context2.set_authority("test"); - request2.set_message("World"); - Status s2 = stub_->Echo(&context2, request2, &response2); - EXPECT_EQ(response2.message(), request2.message()); - EXPECT_TRUE(s2.ok()); -} - -} // namespace testing -} // namespace grpc - -int main(int argc, char** argv) { - grpc_test_init(argc, argv); - ::testing::InitGoogleTest(&argc, argv); - return RUN_ALL_TESTS(); -} diff --git a/tools/dockerfile/grpc_check_generated_pb_files/Dockerfile b/tools/dockerfile/grpc_check_generated_pb_files/Dockerfile index 7658991462d..d19bc67120f 100644 --- a/tools/dockerfile/grpc_check_generated_pb_files/Dockerfile +++ b/tools/dockerfile/grpc_check_generated_pb_files/Dockerfile @@ -67,11 +67,6 @@ RUN apt-get update && apt-get install -y time && apt-get clean # C++ dependencies RUN apt-get update && apt-get -y install libgflags-dev libgtest-dev libc++-dev clang && apt-get clean -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev - RUN mkdir /var/local/jenkins # Define the default command. diff --git a/tools/dockerfile/interoptest/grpc_interop_csharp/Dockerfile b/tools/dockerfile/interoptest/grpc_interop_csharp/Dockerfile index baab2f56380..150dde4f212 100644 --- a/tools/dockerfile/interoptest/grpc_interop_csharp/Dockerfile +++ b/tools/dockerfile/interoptest/grpc_interop_csharp/Dockerfile @@ -88,10 +88,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/interoptest/grpc_interop_cxx/Dockerfile b/tools/dockerfile/interoptest/grpc_interop_cxx/Dockerfile index 2bbccca9e51..bbd903e2696 100644 --- a/tools/dockerfile/interoptest/grpc_interop_cxx/Dockerfile +++ b/tools/dockerfile/interoptest/grpc_interop_cxx/Dockerfile @@ -75,10 +75,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/interoptest/grpc_interop_node/Dockerfile b/tools/dockerfile/interoptest/grpc_interop_node/Dockerfile index 2a8d35a5dc1..be07094cd27 100644 --- a/tools/dockerfile/interoptest/grpc_interop_node/Dockerfile +++ b/tools/dockerfile/interoptest/grpc_interop_node/Dockerfile @@ -82,10 +82,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/interoptest/grpc_interop_php/Dockerfile b/tools/dockerfile/interoptest/grpc_interop_php/Dockerfile index e27a6a23013..af83ee61646 100644 --- a/tools/dockerfile/interoptest/grpc_interop_php/Dockerfile +++ b/tools/dockerfile/interoptest/grpc_interop_php/Dockerfile @@ -100,10 +100,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/interoptest/grpc_interop_python/Dockerfile b/tools/dockerfile/interoptest/grpc_interop_python/Dockerfile index 071fb2c93b1..db01e301dec 100644 --- a/tools/dockerfile/interoptest/grpc_interop_python/Dockerfile +++ b/tools/dockerfile/interoptest/grpc_interop_python/Dockerfile @@ -86,10 +86,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/interoptest/grpc_interop_ruby/Dockerfile b/tools/dockerfile/interoptest/grpc_interop_ruby/Dockerfile index df8eef54385..88b513032ac 100644 --- a/tools/dockerfile/interoptest/grpc_interop_ruby/Dockerfile +++ b/tools/dockerfile/interoptest/grpc_interop_ruby/Dockerfile @@ -86,10 +86,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/stress_test/grpc_interop_stress_node/Dockerfile b/tools/dockerfile/stress_test/grpc_interop_stress_node/Dockerfile index 4fd7cc29a3c..0738e95e9bc 100644 --- a/tools/dockerfile/stress_test/grpc_interop_stress_node/Dockerfile +++ b/tools/dockerfile/stress_test/grpc_interop_stress_node/Dockerfile @@ -87,10 +87,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/stress_test/grpc_interop_stress_php/Dockerfile b/tools/dockerfile/stress_test/grpc_interop_stress_php/Dockerfile index c29aaf7c3f3..3092bd955e2 100644 --- a/tools/dockerfile/stress_test/grpc_interop_stress_php/Dockerfile +++ b/tools/dockerfile/stress_test/grpc_interop_stress_php/Dockerfile @@ -105,10 +105,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/csharp_coreclr_x64/Dockerfile b/tools/dockerfile/test/csharp_coreclr_x64/Dockerfile index fd7215716d3..98515aa5d73 100644 --- a/tools/dockerfile/test/csharp_coreclr_x64/Dockerfile +++ b/tools/dockerfile/test/csharp_coreclr_x64/Dockerfile @@ -103,10 +103,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/csharp_jessie_x64/Dockerfile b/tools/dockerfile/test/csharp_jessie_x64/Dockerfile index baab2f56380..150dde4f212 100644 --- a/tools/dockerfile/test/csharp_jessie_x64/Dockerfile +++ b/tools/dockerfile/test/csharp_jessie_x64/Dockerfile @@ -88,10 +88,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/cxx_jessie_x64/Dockerfile b/tools/dockerfile/test/cxx_jessie_x64/Dockerfile index 64921589299..a8aa74dd0e0 100644 --- a/tools/dockerfile/test/cxx_jessie_x64/Dockerfile +++ b/tools/dockerfile/test/cxx_jessie_x64/Dockerfile @@ -108,10 +108,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/cxx_jessie_x86/Dockerfile b/tools/dockerfile/test/cxx_jessie_x86/Dockerfile index 92c9c4ce86e..abd3e42f266 100644 --- a/tools/dockerfile/test/cxx_jessie_x86/Dockerfile +++ b/tools/dockerfile/test/cxx_jessie_x86/Dockerfile @@ -75,10 +75,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/cxx_ubuntu1404_x64/Dockerfile b/tools/dockerfile/test/cxx_ubuntu1404_x64/Dockerfile index 5982c9783e0..5ef25e80b47 100644 --- a/tools/dockerfile/test/cxx_ubuntu1404_x64/Dockerfile +++ b/tools/dockerfile/test/cxx_ubuntu1404_x64/Dockerfile @@ -67,10 +67,6 @@ RUN apt-get update && apt-get install -y time && apt-get clean # C++ dependencies RUN apt-get update && apt-get -y install libgflags-dev libgtest-dev libc++-dev clang && apt-get clean -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/cxx_ubuntu1604_x64/Dockerfile b/tools/dockerfile/test/cxx_ubuntu1604_x64/Dockerfile index d356433163e..c65fc619778 100644 --- a/tools/dockerfile/test/cxx_ubuntu1604_x64/Dockerfile +++ b/tools/dockerfile/test/cxx_ubuntu1604_x64/Dockerfile @@ -75,10 +75,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/cxx_wheezy_x64/Dockerfile b/tools/dockerfile/test/cxx_wheezy_x64/Dockerfile index dd9a79b1ed5..9d5dd52c18f 100644 --- a/tools/dockerfile/test/cxx_wheezy_x64/Dockerfile +++ b/tools/dockerfile/test/cxx_wheezy_x64/Dockerfile @@ -86,10 +86,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/fuzzer/Dockerfile b/tools/dockerfile/test/fuzzer/Dockerfile index 6ba31114ab6..3ac134ad7da 100644 --- a/tools/dockerfile/test/fuzzer/Dockerfile +++ b/tools/dockerfile/test/fuzzer/Dockerfile @@ -108,10 +108,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/multilang_jessie_x64/Dockerfile b/tools/dockerfile/test/multilang_jessie_x64/Dockerfile index 5c3f77405ea..5e5969cda2f 100644 --- a/tools/dockerfile/test/multilang_jessie_x64/Dockerfile +++ b/tools/dockerfile/test/multilang_jessie_x64/Dockerfile @@ -147,10 +147,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/node_jessie_x64/Dockerfile b/tools/dockerfile/test/node_jessie_x64/Dockerfile index 2a8d35a5dc1..be07094cd27 100644 --- a/tools/dockerfile/test/node_jessie_x64/Dockerfile +++ b/tools/dockerfile/test/node_jessie_x64/Dockerfile @@ -82,10 +82,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/php_jessie_x64/Dockerfile b/tools/dockerfile/test/php_jessie_x64/Dockerfile index d8d27846c16..e477295722a 100644 --- a/tools/dockerfile/test/php_jessie_x64/Dockerfile +++ b/tools/dockerfile/test/php_jessie_x64/Dockerfile @@ -85,10 +85,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/python_jessie_x64/Dockerfile b/tools/dockerfile/test/python_jessie_x64/Dockerfile index 071fb2c93b1..db01e301dec 100644 --- a/tools/dockerfile/test/python_jessie_x64/Dockerfile +++ b/tools/dockerfile/test/python_jessie_x64/Dockerfile @@ -86,10 +86,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins diff --git a/tools/dockerfile/test/ruby_jessie_x64/Dockerfile b/tools/dockerfile/test/ruby_jessie_x64/Dockerfile index df8eef54385..88b513032ac 100644 --- a/tools/dockerfile/test/ruby_jessie_x64/Dockerfile +++ b/tools/dockerfile/test/ruby_jessie_x64/Dockerfile @@ -86,10 +86,6 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/c++ RUN ln -s /usr/bin/ccache /usr/local/bin/clang RUN ln -s /usr/bin/ccache /usr/local/bin/clang++ -#====================== -# Zookeeper dependencies -# TODO(jtattermusch): is zookeeper still needed? -RUN apt-get install -y libzookeeper-mt-dev RUN mkdir /var/local/jenkins From 98716b3bb71c9005189918f928ff7bf2f4f90e84 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Fri, 1 Jul 2016 13:10:32 -0700 Subject: [PATCH 0394/1039] Change port_server.py to use port 32766 --- tools/run_tests/port_server.py | 4 ++-- tools/run_tests/run_tests.py | 18 +++++++++++++++++- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/tools/run_tests/port_server.py b/tools/run_tests/port_server.py index 14e82b601ea..e2be26d182c 100755 --- a/tools/run_tests/port_server.py +++ b/tools/run_tests/port_server.py @@ -42,7 +42,7 @@ import time # increment this number whenever making a change to ensure that # the changes are picked up by running CI servers # note that all changes must be backwards compatible -_MY_VERSION = 7 +_MY_VERSION = 8 if len(sys.argv) == 2 and sys.argv[1] == 'dump_version': @@ -70,7 +70,7 @@ in_use = {} def refill_pool(max_timeout, req): """Scan for ports not marked for being in use""" - for i in range(1025, 32767): + for i in range(1025, 32766): if len(pool) > 100: break if i in in_use: age = time.time() - in_use[i] diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index 4fc1be199cf..82c835fe5da 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -1031,7 +1031,23 @@ class TestCache(object): self.parse(json.loads(f.read())) +def _shut_down_legacy_server(legacy_server_port): + try: + version = int(urllib2.urlopen( + 'http://localhost:%d/version_number' % legacy_server_port, + timeout=10).read()) + except: + pass + else: + urllib2.urlopen( + 'http://localhost:%d/quitquitquit' % legacy_server_port).read() + + def _start_port_server(port_server_port): + # Temporary patch to switch the port_server port + # see https://github.com/grpc/grpc/issues/7145 + _shut_down_legacy_server(32767) + # check if a compatible port server is running # if incompatible (version mismatch) ==> start a new one # if not running ==> start a new one @@ -1167,7 +1183,7 @@ def _build_and_run( # start antagonists antagonists = [subprocess.Popen(['tools/run_tests/antagonist.py']) for _ in range(0, args.antagonists)] - port_server_port = 32767 + port_server_port = 32766 _start_port_server(port_server_port) resultset = None num_test_failures = 0 From d9dfc673fc5af5b7904833704a3b9f5b44038419 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Thu, 30 Jun 2016 23:50:48 -0700 Subject: [PATCH 0395/1039] Change port_server.py to use port 32766 32767 is used by filenet-powsrm --- tools/run_tests/port_server.py | 4 ++-- tools/run_tests/run_tests.py | 18 +++++++++++++++++- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/tools/run_tests/port_server.py b/tools/run_tests/port_server.py index 14e82b601ea..e2be26d182c 100755 --- a/tools/run_tests/port_server.py +++ b/tools/run_tests/port_server.py @@ -42,7 +42,7 @@ import time # increment this number whenever making a change to ensure that # the changes are picked up by running CI servers # note that all changes must be backwards compatible -_MY_VERSION = 7 +_MY_VERSION = 8 if len(sys.argv) == 2 and sys.argv[1] == 'dump_version': @@ -70,7 +70,7 @@ in_use = {} def refill_pool(max_timeout, req): """Scan for ports not marked for being in use""" - for i in range(1025, 32767): + for i in range(1025, 32766): if len(pool) > 100: break if i in in_use: age = time.time() - in_use[i] diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index e4779e3a4e8..a1aa8d7c097 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -1098,7 +1098,23 @@ class TestCache(object): self.parse(json.loads(f.read())) +def _shut_down_legacy_server(legacy_server_port): + try: + version = int(urllib2.urlopen( + 'http://localhost:%d/version_number' % legacy_server_port, + timeout=10).read()) + except: + pass + else: + urllib2.urlopen( + 'http://localhost:%d/quitquitquit' % legacy_server_port).read() + + def _start_port_server(port_server_port): + # Temporary patch to switch the port_server port + # see https://github.com/grpc/grpc/issues/7145 + _shut_down_legacy_server(32767) + # check if a compatible port server is running # if incompatible (version mismatch) ==> start a new one # if not running ==> start a new one @@ -1234,7 +1250,7 @@ def _build_and_run( # start antagonists antagonists = [subprocess.Popen(['tools/run_tests/antagonist.py']) for _ in range(0, args.antagonists)] - port_server_port = 32767 + port_server_port = 32766 _start_port_server(port_server_port) resultset = None num_test_failures = 0 From 88eb42b2cfa61a8c6214d0a855dfe4da36be8959 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Fri, 1 Jul 2016 13:17:04 -0700 Subject: [PATCH 0396/1039] Add grpc module pointer file for Node health check tests --- src/node/health_check/node_modules/grpc.js | 37 ++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 src/node/health_check/node_modules/grpc.js diff --git a/src/node/health_check/node_modules/grpc.js b/src/node/health_check/node_modules/grpc.js new file mode 100644 index 00000000000..42161198ccd --- /dev/null +++ b/src/node/health_check/node_modules/grpc.js @@ -0,0 +1,37 @@ +/* + * + * Copyright 2016, Google Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google Inc. nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +/* This exists solely to allow the generated code to import the grpc module + * without using a relative path */ + +module.exports = require('../..'); From 2b49ea9d54bb3605f6b39672f68a1fbf6cf95df2 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 1 Jul 2016 13:21:27 -0700 Subject: [PATCH 0397/1039] Fix compile erors --- src/core/lib/iomgr/ev_epoll_linux.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 6dfd363b0b5..af23e35ca3d 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -276,7 +276,7 @@ static void append_error(grpc_error **composite, grpc_error *error, static grpc_wakeup_fd polling_island_wakeup_fd; /* Forward declaration */ -static void polling_island_delete(grpc_exec_ctx *exec_ctx); +static void polling_island_delete(grpc_exec_ctx *exec_ctx, polling_island *pi); #ifdef GRPC_TSAN /* Currently TSAN may incorrectly flag data races between epoll_ctl and @@ -325,7 +325,7 @@ static void pi_unref(grpc_exec_ctx *exec_ctx, polling_island *pi) { polling_island *next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); polling_island_delete(exec_ctx, pi); if (next != NULL) { - PI_UNREF(next, "pi_delete"); /* Recursive call */ + PI_UNREF(exec_ctx, next, "pi_delete"); /* Recursive call */ } } } @@ -465,7 +465,6 @@ static polling_island *polling_island_create(grpc_exec_ctx *exec_ctx, grpc_fd *initial_fd, grpc_error **error) { polling_island *pi = NULL; - char *err_msg; const char *err_desc = "polling_island_create"; *error = GRPC_ERROR_NONE; @@ -1273,7 +1272,7 @@ static void pollset_destroy(grpc_pollset *pollset) { gpr_mu_destroy(&pollset->mu); } -static void pollset_reset(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset) { +static void pollset_reset(grpc_pollset *pollset) { GPR_ASSERT(pollset->shutting_down); GPR_ASSERT(!pollset_has_workers(pollset)); pollset->shutting_down = false; From e38b917f9ee0e9f801384f6c89fe822f72eefa24 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Fri, 1 Jul 2016 13:48:06 -0700 Subject: [PATCH 0398/1039] Check Python ByteBuffer reader init status --- src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi | 4 ++-- src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi | 6 +++++- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi index f3b3d612736..7714504d1bb 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi @@ -132,8 +132,8 @@ cdef extern from "grpc/_cython/loader.h": size_t grpc_byte_buffer_length(grpc_byte_buffer *bb) nogil void grpc_byte_buffer_destroy(grpc_byte_buffer *byte_buffer) nogil - void grpc_byte_buffer_reader_init(grpc_byte_buffer_reader *reader, - grpc_byte_buffer *buffer) nogil + int grpc_byte_buffer_reader_init(grpc_byte_buffer_reader *reader, + grpc_byte_buffer *buffer) nogil int grpc_byte_buffer_reader_next(grpc_byte_buffer_reader *reader, gpr_slice *slice) nogil void grpc_byte_buffer_reader_destroy(grpc_byte_buffer_reader *reader) nogil diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi index 8e651e880f3..b39b2f08de7 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/records.pyx.pxi @@ -252,9 +252,13 @@ cdef class ByteBuffer: cdef gpr_slice data_slice cdef size_t data_slice_length cdef void *data_slice_pointer + cdef bint reader_status if self.c_byte_buffer != NULL: with nogil: - grpc_byte_buffer_reader_init(&reader, self.c_byte_buffer) + reader_status = grpc_byte_buffer_reader_init( + &reader, self.c_byte_buffer) + if not reader_status: + return None result = bytearray() with nogil: while grpc_byte_buffer_reader_next(&reader, &data_slice): From e4123f308342cf6985f04c41e375d8cb995a9121 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Fri, 1 Jul 2016 14:54:52 -0700 Subject: [PATCH 0399/1039] Updated release version to 0.15.1 --- Makefile | 2 +- build.yaml | 2 +- package.json | 2 +- package.xml | 4 ++-- src/core/lib/surface/version.c | 2 +- src/csharp/Grpc.Auth/project.json | 4 ++-- src/csharp/Grpc.Core/VersionInfo.cs | 4 ++-- src/csharp/Grpc.Core/project.json | 2 +- src/csharp/Grpc.HealthCheck/project.json | 4 ++-- src/csharp/build_packages.bat | 2 +- src/node/tools/package.json | 2 +- src/python/grpcio/grpc_version.py | 2 +- src/ruby/lib/grpc/version.rb | 2 +- src/ruby/tools/version.rb | 2 +- tools/distrib/python/grpcio_tools/grpc_version.py | 2 +- tools/distrib/python/grpcio_tools/setup.py | 2 +- tools/doxygen/Doxyfile.c++ | 2 +- tools/doxygen/Doxyfile.c++.internal | 2 +- tools/doxygen/Doxyfile.core | 2 +- tools/doxygen/Doxyfile.core.internal | 2 +- 20 files changed, 24 insertions(+), 24 deletions(-) diff --git a/Makefile b/Makefile index d369653be3c..dabc02bd2e0 100644 --- a/Makefile +++ b/Makefile @@ -414,7 +414,7 @@ E = @echo Q = @ endif -VERSION = 0.15.0 +VERSION = 0.15.1 CPPFLAGS_NO_ARCH += $(addprefix -I, $(INCLUDES)) $(addprefix -D, $(DEFINES)) CPPFLAGS += $(CPPFLAGS_NO_ARCH) $(ARCH_FLAGS) diff --git a/build.yaml b/build.yaml index ffc681d9028..d550c0107fa 100644 --- a/build.yaml +++ b/build.yaml @@ -7,7 +7,7 @@ settings: '#3': Use "-preN" suffixes to identify pre-release versions '#4': Per-language overrides are possible with (eg) ruby_version tag here '#5': See the expand_version.py for all the quirks here - version: 0.15.0 + version: 0.15.1 filegroups: - name: census public_headers: diff --git a/package.json b/package.json index 1b2920c6bc4..f0e1f033f23 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "grpc", - "version": "0.15.0", + "version": "0.15.1", "author": "Google Inc.", "description": "gRPC Library for Node", "homepage": "http://www.grpc.io/", diff --git a/package.xml b/package.xml index 1304eb06fa3..c88314480a4 100644 --- a/package.xml +++ b/package.xml @@ -13,8 +13,8 @@ 2016-06-30 - 0.15.0 - 0.15.0 + 0.15.1 + 0.15.1 beta diff --git a/src/core/lib/surface/version.c b/src/core/lib/surface/version.c index e4a3358c351..8ad93e8ac34 100644 --- a/src/core/lib/surface/version.c +++ b/src/core/lib/surface/version.c @@ -36,4 +36,4 @@ #include -const char *grpc_version_string(void) { return "0.15.0"; } +const char *grpc_version_string(void) { return "0.15.1"; } diff --git a/src/csharp/Grpc.Auth/project.json b/src/csharp/Grpc.Auth/project.json index ae83c3be2b8..7865203ab8a 100644 --- a/src/csharp/Grpc.Auth/project.json +++ b/src/csharp/Grpc.Auth/project.json @@ -1,5 +1,5 @@ { - "version": "0.15.0", + "version": "0.15.1", "title": "gRPC C# Auth", "authors": [ "Google Inc." ], "copyright": "Copyright 2015, Google Inc.", @@ -13,7 +13,7 @@ "tags": [ "gRPC RPC Protocol HTTP/2 Auth OAuth2" ], }, "dependencies": { - "Grpc.Core": "0.15.0", + "Grpc.Core": "0.15.1", "Google.Apis.Auth": "1.11.1" }, "frameworks": { diff --git a/src/csharp/Grpc.Core/VersionInfo.cs b/src/csharp/Grpc.Core/VersionInfo.cs index d89a2b5c6ee..d928f0dc3b7 100644 --- a/src/csharp/Grpc.Core/VersionInfo.cs +++ b/src/csharp/Grpc.Core/VersionInfo.cs @@ -48,11 +48,11 @@ namespace Grpc.Core /// /// Current AssemblyFileVersion of gRPC C# assemblies /// - public const string CurrentAssemblyFileVersion = "0.15.0.0"; + public const string CurrentAssemblyFileVersion = "0.15.1.0"; /// /// Current version of gRPC C# /// - public const string CurrentVersion = "0.15.0"; + public const string CurrentVersion = "0.15.1"; } } diff --git a/src/csharp/Grpc.Core/project.json b/src/csharp/Grpc.Core/project.json index c7f2bf4fb12..030b8680f65 100644 --- a/src/csharp/Grpc.Core/project.json +++ b/src/csharp/Grpc.Core/project.json @@ -1,5 +1,5 @@ { - "version": "0.15.0", + "version": "0.15.1", "title": "gRPC C# Core", "authors": [ "Google Inc." ], "copyright": "Copyright 2015, Google Inc.", diff --git a/src/csharp/Grpc.HealthCheck/project.json b/src/csharp/Grpc.HealthCheck/project.json index 98ea21a436a..853b7131f27 100644 --- a/src/csharp/Grpc.HealthCheck/project.json +++ b/src/csharp/Grpc.HealthCheck/project.json @@ -1,5 +1,5 @@ { - "version": "0.15.0", + "version": "0.15.1", "title": "gRPC C# Healthchecking", "authors": [ "Google Inc." ], "copyright": "Copyright 2015, Google Inc.", @@ -13,7 +13,7 @@ "tags": [ "gRPC health check" ] }, "dependencies": { - "Grpc.Core": "0.15.0", + "Grpc.Core": "0.15.1", "Google.Protobuf": "3.0.0-beta3" }, "frameworks": { diff --git a/src/csharp/build_packages.bat b/src/csharp/build_packages.bat index e387efcc2da..d7631ed4396 100644 --- a/src/csharp/build_packages.bat +++ b/src/csharp/build_packages.bat @@ -30,7 +30,7 @@ @rem Builds gRPC NuGet packages @rem Current package versions -set VERSION=0.15.0 +set VERSION=0.15.1 set PROTOBUF_VERSION=3.0.0-beta3 @rem Packages that depend on prerelease packages (like Google.Protobuf) need to have prerelease suffix as well. diff --git a/src/node/tools/package.json b/src/node/tools/package.json index b2cadd3f47a..ad273d4d5a5 100644 --- a/src/node/tools/package.json +++ b/src/node/tools/package.json @@ -1,6 +1,6 @@ { "name": "grpc-tools", - "version": "0.15.0", + "version": "0.15.1", "author": "Google Inc.", "description": "Tools for developing with gRPC on Node.js", "homepage": "http://www.grpc.io/", diff --git a/src/python/grpcio/grpc_version.py b/src/python/grpcio/grpc_version.py index c6c07afb443..80cc397737c 100644 --- a/src/python/grpcio/grpc_version.py +++ b/src/python/grpcio/grpc_version.py @@ -29,4 +29,4 @@ # AUTO-GENERATED FROM `$REPO_ROOT/templates/src/python/grpcio/grpc_version.py.template`!!! -VERSION='0.15.0' +VERSION='0.15.1' diff --git a/src/ruby/lib/grpc/version.rb b/src/ruby/lib/grpc/version.rb index 7f512e47aab..5c840da5491 100644 --- a/src/ruby/lib/grpc/version.rb +++ b/src/ruby/lib/grpc/version.rb @@ -29,5 +29,5 @@ # GRPC contains the General RPC module. module GRPC - VERSION = '0.15.0' + VERSION = '0.15.1' end diff --git a/src/ruby/tools/version.rb b/src/ruby/tools/version.rb index 6a7a1d5bd33..7b286d1d703 100644 --- a/src/ruby/tools/version.rb +++ b/src/ruby/tools/version.rb @@ -29,6 +29,6 @@ module GRPC module Tools - VERSION = '0.15.0' + VERSION = '0.15.1' end end diff --git a/tools/distrib/python/grpcio_tools/grpc_version.py b/tools/distrib/python/grpcio_tools/grpc_version.py index 9a33c6e5d14..31309923a91 100644 --- a/tools/distrib/python/grpcio_tools/grpc_version.py +++ b/tools/distrib/python/grpcio_tools/grpc_version.py @@ -29,4 +29,4 @@ # AUTO-GENERATED FROM `$REPO_ROOT/templates/tools/distrib/python/grpcio_tools/grpc_version.py.template`!!! -VERSION='0.15.0' +VERSION='0.15.1' diff --git a/tools/distrib/python/grpcio_tools/setup.py b/tools/distrib/python/grpcio_tools/setup.py index fbe69f43d56..a21e090ae34 100644 --- a/tools/distrib/python/grpcio_tools/setup.py +++ b/tools/distrib/python/grpcio_tools/setup.py @@ -116,7 +116,7 @@ setuptools.setup( namespace_packages=['grpc'], install_requires=[ 'protobuf>=3.0.0a3', - 'grpcio>=0.14.0', + 'grpcio>=0.15.0', ], package_data=package_data(), ) diff --git a/tools/doxygen/Doxyfile.c++ b/tools/doxygen/Doxyfile.c++ index 066d29ac001..09bc73f59ef 100644 --- a/tools/doxygen/Doxyfile.c++ +++ b/tools/doxygen/Doxyfile.c++ @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC C++" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0 +PROJECT_NUMBER = 0.15.1 # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a diff --git a/tools/doxygen/Doxyfile.c++.internal b/tools/doxygen/Doxyfile.c++.internal index 6a0e8b2129c..4114d2c755d 100644 --- a/tools/doxygen/Doxyfile.c++.internal +++ b/tools/doxygen/Doxyfile.c++.internal @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC C++" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0 +PROJECT_NUMBER = 0.15.1 # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a diff --git a/tools/doxygen/Doxyfile.core b/tools/doxygen/Doxyfile.core index fa9fd5a3123..cd0dd76b4e2 100644 --- a/tools/doxygen/Doxyfile.core +++ b/tools/doxygen/Doxyfile.core @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC Core" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0 +PROJECT_NUMBER = 0.15.1 # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index e4c9f991d31..93cea985d28 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -40,7 +40,7 @@ PROJECT_NAME = "GRPC Core" # could be handy for archiving the generated documentation or if some version # control system is used. -PROJECT_NUMBER = 0.15.0 +PROJECT_NUMBER = 0.15.1 # Using the PROJECT_BRIEF tag one can provide an optional one line description # for a project that appears at the top of each page and should give viewer a From f975f74c016d5850d10644c4865e76e05e4b7815 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 1 Jul 2016 14:56:27 -0700 Subject: [PATCH 0400/1039] Refactor workqueues a little to make them great again --- Makefile | 36 ---- build.yaml | 10 - src/core/lib/iomgr/ev_epoll_linux.c | 21 +- src/core/lib/iomgr/tcp_posix.c | 2 +- src/core/lib/iomgr/workqueue.h | 9 - src/core/lib/iomgr/workqueue_posix.c | 6 - src/core/lib/iomgr/workqueue_posix.h | 4 + src/core/lib/iomgr/workqueue_windows.c | 10 - test/core/iomgr/workqueue_test.c | 150 ------------- tools/run_tests/sources_and_headers.json | 16 -- tools/run_tests/tests.json | 21 -- vsprojects/buildtests_c.sln | 27 --- .../workqueue_test/workqueue_test.vcxproj | 199 ------------------ .../workqueue_test.vcxproj.filters | 21 -- 14 files changed, 17 insertions(+), 515 deletions(-) delete mode 100644 test/core/iomgr/workqueue_test.c delete mode 100644 vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj delete mode 100644 vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj.filters diff --git a/Makefile b/Makefile index 8fd86e78ed0..d7d7b8f3d90 100644 --- a/Makefile +++ b/Makefile @@ -1001,7 +1001,6 @@ transport_security_test: $(BINDIR)/$(CONFIG)/transport_security_test udp_server_test: $(BINDIR)/$(CONFIG)/udp_server_test uri_fuzzer_test: $(BINDIR)/$(CONFIG)/uri_fuzzer_test uri_parser_test: $(BINDIR)/$(CONFIG)/uri_parser_test -workqueue_test: $(BINDIR)/$(CONFIG)/workqueue_test alarm_cpp_test: $(BINDIR)/$(CONFIG)/alarm_cpp_test async_end2end_test: $(BINDIR)/$(CONFIG)/async_end2end_test auth_property_iterator_test: $(BINDIR)/$(CONFIG)/auth_property_iterator_test @@ -1330,7 +1329,6 @@ buildtests_c: privatelibs_c \ $(BINDIR)/$(CONFIG)/transport_security_test \ $(BINDIR)/$(CONFIG)/udp_server_test \ $(BINDIR)/$(CONFIG)/uri_parser_test \ - $(BINDIR)/$(CONFIG)/workqueue_test \ $(BINDIR)/$(CONFIG)/public_headers_must_be_c89 \ $(BINDIR)/$(CONFIG)/badreq_bad_client_test \ $(BINDIR)/$(CONFIG)/connection_prefix_bad_client_test \ @@ -1715,8 +1713,6 @@ test_c: buildtests_c $(Q) $(BINDIR)/$(CONFIG)/udp_server_test || ( echo test udp_server_test failed ; exit 1 ) $(E) "[RUN] Testing uri_parser_test" $(Q) $(BINDIR)/$(CONFIG)/uri_parser_test || ( echo test uri_parser_test failed ; exit 1 ) - $(E) "[RUN] Testing workqueue_test" - $(Q) $(BINDIR)/$(CONFIG)/workqueue_test || ( echo test workqueue_test failed ; exit 1 ) $(E) "[RUN] Testing public_headers_must_be_c89" $(Q) $(BINDIR)/$(CONFIG)/public_headers_must_be_c89 || ( echo test public_headers_must_be_c89 failed ; exit 1 ) $(E) "[RUN] Testing badreq_bad_client_test" @@ -10216,38 +10212,6 @@ endif endif -WORKQUEUE_TEST_SRC = \ - test/core/iomgr/workqueue_test.c \ - -WORKQUEUE_TEST_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(WORKQUEUE_TEST_SRC)))) -ifeq ($(NO_SECURE),true) - -# You can't build secure targets if you don't have OpenSSL. - -$(BINDIR)/$(CONFIG)/workqueue_test: openssl_dep_error - -else - - - -$(BINDIR)/$(CONFIG)/workqueue_test: $(WORKQUEUE_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a - $(E) "[LD] Linking $@" - $(Q) mkdir -p `dirname $@` - $(Q) $(LD) $(LDFLAGS) $(WORKQUEUE_TEST_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LDLIBS) $(LDLIBS_SECURE) -o $(BINDIR)/$(CONFIG)/workqueue_test - -endif - -$(OBJDIR)/$(CONFIG)/test/core/iomgr/workqueue_test.o: $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a - -deps_workqueue_test: $(WORKQUEUE_TEST_OBJS:.o=.dep) - -ifneq ($(NO_SECURE),true) -ifneq ($(NO_DEPS),true) --include $(WORKQUEUE_TEST_OBJS:.o=.dep) -endif -endif - - ALARM_CPP_TEST_SRC = \ test/cpp/common/alarm_cpp_test.cc \ diff --git a/build.yaml b/build.yaml index 681ab54d5b1..44716927ae4 100644 --- a/build.yaml +++ b/build.yaml @@ -2423,16 +2423,6 @@ targets: - grpc - gpr_test_util - gpr -- name: workqueue_test - build: test - language: c - src: - - test/core/iomgr/workqueue_test.c - deps: - - grpc_test_util - - grpc - - gpr_test_util - - gpr - name: alarm_cpp_test gtest: true build: test diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index af23e35ca3d..e5aab293bd3 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -253,13 +253,14 @@ struct grpc_pollset_set { * Common helpers */ -static void append_error(grpc_error **composite, grpc_error *error, +static bool append_error(grpc_error **composite, grpc_error *error, const char *desc) { - if (error == GRPC_ERROR_NONE) return; + if (error == GRPC_ERROR_NONE) return true; if (*composite == GRPC_ERROR_NONE) { *composite = GRPC_ERROR_CREATE(desc); } *composite = grpc_error_add_child(*composite, error); + return false; } /******************************************************************************* @@ -490,16 +491,18 @@ static polling_island *polling_island_create(grpc_exec_ctx *exec_ctx, polling_island_add_wakeup_fd_locked(pi, &grpc_global_wakeup_fd, error); if (initial_fd != NULL) { - /* Lock the polling island here just in case we got this structure from - the freelist and the polling island lock was not released yet (by the - code that adds the polling island to the freelist) */ - gpr_mu_lock(&pi->mu); polling_island_add_fds_locked(pi, &initial_fd, 1, true, error); - gpr_mu_unlock(&pi->mu); } - append_error(error, grpc_workqueue_create(exec_ctx, &pi->workqueue), - err_desc); + if (append_error(error, grpc_workqueue_create(exec_ctx, &pi->workqueue), + err_desc) && + *error == GRPC_ERROR_NONE) { + polling_island_add_fds_locked(pi, &pi->workqueue->wakeup_read_fd, 1, true, + error); + GPR_ASSERT(pi->workqueue->wakeup_read_fd->polling_island == NULL); + pi->workqueue->wakeup_read_fd->polling_island = pi; + PI_ADD_REF(pi, 1); + } done: if (*error != GRPC_ERROR_NONE) { diff --git a/src/core/lib/iomgr/tcp_posix.c b/src/core/lib/iomgr/tcp_posix.c index b5bac152fb6..ec21e039448 100644 --- a/src/core/lib/iomgr/tcp_posix.c +++ b/src/core/lib/iomgr/tcp_posix.c @@ -284,7 +284,7 @@ static void tcp_read(grpc_exec_ctx *exec_ctx, grpc_endpoint *ep, } /* returns true if done, false if pending; if returning true, *error is set */ -#define MAX_WRITE_IOVEC 16 +#define MAX_WRITE_IOVEC 1024 static bool tcp_flush(grpc_tcp *tcp, grpc_error **error) { struct msghdr msg; struct iovec iov[MAX_WRITE_IOVEC]; diff --git a/src/core/lib/iomgr/workqueue.h b/src/core/lib/iomgr/workqueue.h index 498b7a300a5..064a551ed0a 100644 --- a/src/core/lib/iomgr/workqueue.h +++ b/src/core/lib/iomgr/workqueue.h @@ -49,10 +49,6 @@ /* grpc_workqueue is forward declared in exec_ctx.h */ -/** Create a work queue */ -grpc_error *grpc_workqueue_create(grpc_exec_ctx *exec_ctx, - grpc_workqueue **workqueue); - void grpc_workqueue_flush(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue); #define GRPC_WORKQUEUE_REFCOUNT_DEBUG @@ -72,11 +68,6 @@ void grpc_workqueue_ref(grpc_workqueue *workqueue); void grpc_workqueue_unref(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue); #endif -/** Bind this workqueue to a pollset */ -void grpc_workqueue_add_to_pollset(grpc_exec_ctx *exec_ctx, - grpc_workqueue *workqueue, - grpc_pollset *pollset); - /** Add a work item to a workqueue */ void grpc_workqueue_enqueue(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, grpc_closure *closure, grpc_error *error); diff --git a/src/core/lib/iomgr/workqueue_posix.c b/src/core/lib/iomgr/workqueue_posix.c index 45e0f6063b4..fef1709954f 100644 --- a/src/core/lib/iomgr/workqueue_posix.c +++ b/src/core/lib/iomgr/workqueue_posix.c @@ -100,12 +100,6 @@ void grpc_workqueue_unref(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue) { } } -void grpc_workqueue_add_to_pollset(grpc_exec_ctx *exec_ctx, - grpc_workqueue *workqueue, - grpc_pollset *pollset) { - grpc_pollset_add_fd(exec_ctx, pollset, workqueue->wakeup_read_fd); -} - void grpc_workqueue_flush(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue) { gpr_mu_lock(&workqueue->mu); grpc_exec_ctx_enqueue_list(exec_ctx, &workqueue->closure_list, NULL); diff --git a/src/core/lib/iomgr/workqueue_posix.h b/src/core/lib/iomgr/workqueue_posix.h index dcb47e7b59d..2e8aca18164 100644 --- a/src/core/lib/iomgr/workqueue_posix.h +++ b/src/core/lib/iomgr/workqueue_posix.h @@ -50,4 +50,8 @@ struct grpc_workqueue { grpc_closure read_closure; }; +/** Create a work queue */ +grpc_error *grpc_workqueue_create(grpc_exec_ctx *exec_ctx, + grpc_workqueue **workqueue); + #endif /* GRPC_CORE_LIB_IOMGR_WORKQUEUE_POSIX_H */ diff --git a/src/core/lib/iomgr/workqueue_windows.c b/src/core/lib/iomgr/workqueue_windows.c index 52a3e47f376..23e2dea1859 100644 --- a/src/core/lib/iomgr/workqueue_windows.c +++ b/src/core/lib/iomgr/workqueue_windows.c @@ -42,12 +42,6 @@ // context, which is at least correct, if not performant or in the spirit of // workqueues. -grpc_error *grpc_workqueue_create(grpc_exec_ctx *exec_ctx, - grpc_workqueue **workqueue) { - *workqueue = (grpc_workqueue *)1; - return GRPC_ERROR_NONE; -} - void grpc_workqueue_flush(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue) {} #ifdef GRPC_WORKQUEUE_REFCOUNT_DEBUG @@ -60,10 +54,6 @@ void grpc_workqueue_ref(grpc_workqueue *workqueue) {} void grpc_workqueue_unref(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue) {} #endif -void grpc_workqueue_add_to_pollset(grpc_exec_ctx *exec_ctx, - grpc_workqueue *workqueue, - grpc_pollset *pollset) {} - void grpc_workqueue_enqueue(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue, grpc_closure *closure, grpc_error *error) { grpc_exec_ctx_sched(exec_ctx, closure, error, NULL); diff --git a/test/core/iomgr/workqueue_test.c b/test/core/iomgr/workqueue_test.c deleted file mode 100644 index 76ecfae74b8..00000000000 --- a/test/core/iomgr/workqueue_test.c +++ /dev/null @@ -1,150 +0,0 @@ -/* - * - * Copyright 2015, Google Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following disclaimer - * in the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Google Inc. nor the names of its - * contributors may be used to endorse or promote products derived from - * this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - */ - -#include "src/core/lib/iomgr/workqueue.h" - -#include -#include -#include - -#include "test/core/util/test_config.h" - -static gpr_mu *g_mu; -static grpc_pollset *g_pollset; - -static void must_succeed(grpc_exec_ctx *exec_ctx, void *p, grpc_error *error) { - GPR_ASSERT(error == GRPC_ERROR_NONE); - gpr_mu_lock(g_mu); - *(int *)p = 1; - GPR_ASSERT( - GRPC_LOG_IF_ERROR("pollset_kick", grpc_pollset_kick(g_pollset, NULL))); - gpr_mu_unlock(g_mu); -} - -static void test_ref_unref(void) { - grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; - grpc_workqueue *wq; - GPR_ASSERT(GRPC_LOG_IF_ERROR("grpc_workqueue_create", - grpc_workqueue_create(&exec_ctx, &wq))); - GRPC_WORKQUEUE_REF(wq, "test"); - GRPC_WORKQUEUE_UNREF(&exec_ctx, wq, "test"); - GRPC_WORKQUEUE_UNREF(&exec_ctx, wq, "destroy"); - grpc_exec_ctx_finish(&exec_ctx); -} - -static void test_add_closure(void) { - grpc_closure c; - int done = 0; - grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; - grpc_workqueue *wq; - GPR_ASSERT(GRPC_LOG_IF_ERROR("grpc_workqueue_create", - grpc_workqueue_create(&exec_ctx, &wq))); - gpr_timespec deadline = GRPC_TIMEOUT_SECONDS_TO_DEADLINE(5); - grpc_pollset_worker *worker = NULL; - grpc_closure_init(&c, must_succeed, &done); - - grpc_workqueue_enqueue(&exec_ctx, wq, &c, GRPC_ERROR_NONE); - grpc_workqueue_add_to_pollset(&exec_ctx, wq, g_pollset); - - gpr_mu_lock(g_mu); - GPR_ASSERT(!done); - while (!done) { - GPR_ASSERT(GRPC_LOG_IF_ERROR( - "pollset_work", - grpc_pollset_work(&exec_ctx, g_pollset, &worker, - gpr_now(deadline.clock_type), deadline))); - } - gpr_mu_unlock(g_mu); - grpc_exec_ctx_finish(&exec_ctx); - GPR_ASSERT(done); - - GRPC_WORKQUEUE_UNREF(&exec_ctx, wq, "destroy"); - grpc_exec_ctx_finish(&exec_ctx); -} - -static void test_flush(void) { - grpc_closure c; - int done = 0; - grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; - grpc_workqueue *wq; - GPR_ASSERT(GRPC_LOG_IF_ERROR("grpc_workqueue_create", - grpc_workqueue_create(&exec_ctx, &wq))); - gpr_timespec deadline = GRPC_TIMEOUT_SECONDS_TO_DEADLINE(5); - grpc_pollset_worker *worker = NULL; - grpc_closure_init(&c, must_succeed, &done); - - grpc_exec_ctx_sched(&exec_ctx, &c, GRPC_ERROR_NONE, NULL); - grpc_workqueue_flush(&exec_ctx, wq); - grpc_workqueue_add_to_pollset(&exec_ctx, wq, g_pollset); - - gpr_mu_lock(g_mu); - GPR_ASSERT(!done); - while (!done) { - GPR_ASSERT(GRPC_LOG_IF_ERROR( - "pollset_work", - grpc_pollset_work(&exec_ctx, g_pollset, &worker, - gpr_now(deadline.clock_type), deadline))); - } - gpr_mu_unlock(g_mu); - grpc_exec_ctx_finish(&exec_ctx); - GPR_ASSERT(done); - - GRPC_WORKQUEUE_UNREF(&exec_ctx, wq, "destroy"); - grpc_exec_ctx_finish(&exec_ctx); -} - -static void destroy_pollset(grpc_exec_ctx *exec_ctx, void *p, - grpc_error *error) { - grpc_pollset_destroy(p); -} - -int main(int argc, char **argv) { - grpc_closure destroyed; - grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT; - grpc_test_init(argc, argv); - grpc_init(); - g_pollset = gpr_malloc(grpc_pollset_size()); - grpc_pollset_init(g_pollset, &g_mu); - - test_ref_unref(); - test_add_closure(); - test_flush(); - - grpc_closure_init(&destroyed, destroy_pollset, g_pollset); - grpc_pollset_shutdown(&exec_ctx, g_pollset, &destroyed); - grpc_exec_ctx_finish(&exec_ctx); - grpc_shutdown(); - - gpr_free(g_pollset); - return 0; -} diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 65de89c0ab0..0b3314a4419 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -1807,22 +1807,6 @@ "third_party": false, "type": "target" }, - { - "deps": [ - "gpr", - "gpr_test_util", - "grpc", - "grpc_test_util" - ], - "headers": [], - "language": "c", - "name": "workqueue_test", - "src": [ - "test/core/iomgr/workqueue_test.c" - ], - "third_party": false, - "type": "target" - }, { "deps": [ "gpr", diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index 291ae415a49..8dee7c8f0f3 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -1914,27 +1914,6 @@ "windows" ] }, - { - "args": [], - "ci_platforms": [ - "linux", - "mac", - "posix", - "windows" - ], - "cpu_cost": 1.0, - "exclude_configs": [], - "flaky": false, - "gtest": false, - "language": "c", - "name": "workqueue_test", - "platforms": [ - "linux", - "mac", - "posix", - "windows" - ] - }, { "args": [], "ci_platforms": [ diff --git a/vsprojects/buildtests_c.sln b/vsprojects/buildtests_c.sln index fcf9b1fc22a..10be520b5f1 100644 --- a/vsprojects/buildtests_c.sln +++ b/vsprojects/buildtests_c.sln @@ -1421,17 +1421,6 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "window_overflow_bad_client_ {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "workqueue_test", "vcxproj\test\workqueue_test\workqueue_test.vcxproj", "{1B6E9651-4D88-2ACB-BEE0-26599919361D}" - ProjectSection(myProperties) = preProject - lib = "False" - EndProjectSection - ProjectSection(ProjectDependencies) = postProject - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} = {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} - {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} = {EAB0A629-17A9-44DB-B5FF-E91A721FE037} - {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} - EndProjectSection -EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Win32 = Debug|Win32 @@ -3556,22 +3545,6 @@ Global {658D7F7F-9628-6545-743C-D949301DC5DC}.Release-DLL|Win32.Build.0 = Release|Win32 {658D7F7F-9628-6545-743C-D949301DC5DC}.Release-DLL|x64.ActiveCfg = Release|x64 {658D7F7F-9628-6545-743C-D949301DC5DC}.Release-DLL|x64.Build.0 = Release|x64 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug|Win32.ActiveCfg = Debug|Win32 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug|x64.ActiveCfg = Debug|x64 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release|Win32.ActiveCfg = Release|Win32 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release|x64.ActiveCfg = Release|x64 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug|Win32.Build.0 = Debug|Win32 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug|x64.Build.0 = Debug|x64 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release|Win32.Build.0 = Release|Win32 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release|x64.Build.0 = Release|x64 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug-DLL|Win32.ActiveCfg = Debug|Win32 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug-DLL|Win32.Build.0 = Debug|Win32 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug-DLL|x64.ActiveCfg = Debug|x64 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Debug-DLL|x64.Build.0 = Debug|x64 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release-DLL|Win32.ActiveCfg = Release|Win32 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release-DLL|Win32.Build.0 = Release|Win32 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release-DLL|x64.ActiveCfg = Release|x64 - {1B6E9651-4D88-2ACB-BEE0-26599919361D}.Release-DLL|x64.Build.0 = Release|x64 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE diff --git a/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj b/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj deleted file mode 100644 index 35665fa1833..00000000000 --- a/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj +++ /dev/null @@ -1,199 +0,0 @@ - - - - - - Debug - Win32 - - - Debug - x64 - - - Release - Win32 - - - Release - x64 - - - - {1B6E9651-4D88-2ACB-BEE0-26599919361D} - true - $(SolutionDir)IntDir\$(MSBuildProjectName)\ - - - - v100 - - - v110 - - - v120 - - - v140 - - - Application - true - Unicode - - - Application - false - true - Unicode - - - - - - - - - - - - - - workqueue_test - static - Debug - static - Debug - - - workqueue_test - static - Release - static - Release - - - - NotUsing - Level3 - Disabled - WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) - true - MultiThreadedDebug - true - None - false - - - Console - true - false - - - - - - NotUsing - Level3 - Disabled - WIN32;_DEBUG;_LIB;%(PreprocessorDefinitions) - true - MultiThreadedDebug - true - None - false - - - Console - true - false - - - - - - NotUsing - Level3 - MaxSpeed - WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) - true - true - true - MultiThreaded - true - None - false - - - Console - true - false - true - true - - - - - - NotUsing - Level3 - MaxSpeed - WIN32;NDEBUG;_LIB;%(PreprocessorDefinitions) - true - true - true - MultiThreaded - true - None - false - - - Console - true - false - true - true - - - - - - - - - - {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} - - - {29D16885-7228-4C31-81ED-5F9187C7F2A9} - - - {EAB0A629-17A9-44DB-B5FF-E91A721FE037} - - - {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - diff --git a/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj.filters b/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj.filters deleted file mode 100644 index 0c5979e66ae..00000000000 --- a/vsprojects/vcxproj/test/workqueue_test/workqueue_test.vcxproj.filters +++ /dev/null @@ -1,21 +0,0 @@ - - - - - test\core\iomgr - - - - - - {3c17ce2b-b57b-9278-8494-48ab4df88ec8} - - - {7035b4d7-88e6-ce95-0011-0ea3cc64eddd} - - - {f0fb09d4-0bdc-a53c-1b5f-d71acbf72a5d} - - - - From c4326ef25a5280b4254ab1c97c9f28d68f6ce852 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Fri, 1 Jul 2016 15:20:51 -0700 Subject: [PATCH 0401/1039] Make Node code generator work properly with nested types --- src/compiler/node_generator.cc | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/src/compiler/node_generator.cc b/src/compiler/node_generator.cc index 1fe090d17af..2ac5e772acb 100644 --- a/src/compiler/node_generator.cc +++ b/src/compiler/node_generator.cc @@ -111,8 +111,8 @@ map GetAllMessages(const FileDescriptor *file) const MethodDescriptor* method = service->method(method_num); const Descriptor* input_type = method->input_type(); const Descriptor* output_type = method->output_type(); - message_types[input_type->name()] = input_type; - message_types[output_type->name()] = output_type; + message_types[input_type->full_name()] = input_type; + message_types[output_type->full_name()] = output_type; } } return message_types; @@ -124,7 +124,7 @@ grpc::string MessageIdentifierName(const grpc::string& name) { grpc::string NodeObjectPath(const Descriptor *descriptor) { grpc::string module_alias = ModuleAlias(descriptor->file()->name()); - grpc::string name = descriptor->name(); + grpc::string name = descriptor->full_name(); grpc_generator::StripPrefix(&name, descriptor->file()->package() + "."); return module_alias + "." + name; } @@ -132,8 +132,9 @@ grpc::string NodeObjectPath(const Descriptor *descriptor) { // Prints out the message serializer and deserializer functions void PrintMessageTransformer(const Descriptor *descriptor, Printer *out) { map template_vars; - template_vars["identifier_name"] = MessageIdentifierName(descriptor->name()); - template_vars["name"] = descriptor->name(); + grpc::string full_name = descriptor->full_name(); + template_vars["identifier_name"] = MessageIdentifierName(full_name); + template_vars["name"] = full_name; template_vars["node_name"] = NodeObjectPath(descriptor); // Print the serializer out->Print(template_vars, "function serialize_$identifier_name$(arg) {\n"); @@ -166,9 +167,9 @@ void PrintMethod(const MethodDescriptor *method, Printer *out) { vars["service_name"] = method->service()->full_name(); vars["name"] = method->name(); vars["input_type"] = NodeObjectPath(input_type); - vars["input_type_id"] = MessageIdentifierName(input_type->name()); + vars["input_type_id"] = MessageIdentifierName(input_type->full_name()); vars["output_type"] = NodeObjectPath(output_type); - vars["output_type_id"] = MessageIdentifierName(output_type->name()); + vars["output_type_id"] = MessageIdentifierName(output_type->full_name()); vars["client_stream"] = method->client_streaming() ? "true" : "false"; vars["server_stream"] = method->server_streaming() ? "true" : "false"; out->Print("{\n"); From b360c8acc7b53aabb2af7f76ddf7310f3d025acc Mon Sep 17 00:00:00 2001 From: "Nicolas \"Pixel\" Noble" Date: Sat, 2 Jul 2016 00:54:28 +0200 Subject: [PATCH 0402/1039] Regenerating project files, and adding experimental disclaimer. --- CMakeLists.txt | 13 +++++++++++-- templates/CMakeLists.txt.template | 5 ++++- third_party/protobuf | 2 +- 3 files changed, 16 insertions(+), 4 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index a81517a6a0b..9caf03191fa 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -4,7 +4,10 @@ # Please look at the templates directory instead. # This file can be regenerated from the template by running # tools/buildgen/generate_projects.sh - +# +# Additionally, this is currently very experimental, and unsupported. +# Further work will happen on that file. +# # Copyright 2015, Google Inc. # All rights reserved. # @@ -39,7 +42,7 @@ cmake_minimum_required(VERSION 2.8) set(PACKAGE_NAME "grpc") -set(PACKAGE_VERSION "0.15.0-dev") +set(PACKAGE_VERSION "0.16.0-dev") set(PACKAGE_STRING "${PACKAGE_NAME} ${PACKAGE_VERSION}") set(PACKAGE_TARNAME "${PACKAGE_NAME}-${PACKAGE_VERSION}") set(PACKAGE_BUGREPORT "https://github.com/grpc/grpc/issues/") @@ -142,6 +145,7 @@ add_library(grpc src/core/lib/iomgr/endpoint_pair_posix.c src/core/lib/iomgr/endpoint_pair_windows.c src/core/lib/iomgr/error.c + src/core/lib/iomgr/ev_epoll_linux.c src/core/lib/iomgr/ev_poll_and_epoll_posix.c src/core/lib/iomgr/ev_poll_posix.c src/core/lib/iomgr/ev_posix.c @@ -152,6 +156,7 @@ add_library(grpc src/core/lib/iomgr/iomgr_posix.c src/core/lib/iomgr/iomgr_windows.c src/core/lib/iomgr/load_file.c + src/core/lib/iomgr/network_status_tracker.c src/core/lib/iomgr/polling_entity.c src/core/lib/iomgr/pollset_set_windows.c src/core/lib/iomgr/pollset_windows.c @@ -345,6 +350,7 @@ add_library(grpc_cronet src/core/lib/iomgr/endpoint_pair_posix.c src/core/lib/iomgr/endpoint_pair_windows.c src/core/lib/iomgr/error.c + src/core/lib/iomgr/ev_epoll_linux.c src/core/lib/iomgr/ev_poll_and_epoll_posix.c src/core/lib/iomgr/ev_poll_posix.c src/core/lib/iomgr/ev_posix.c @@ -355,6 +361,7 @@ add_library(grpc_cronet src/core/lib/iomgr/iomgr_posix.c src/core/lib/iomgr/iomgr_windows.c src/core/lib/iomgr/load_file.c + src/core/lib/iomgr/network_status_tracker.c src/core/lib/iomgr/polling_entity.c src/core/lib/iomgr/pollset_set_windows.c src/core/lib/iomgr/pollset_windows.c @@ -525,6 +532,7 @@ add_library(grpc_unsecure src/core/lib/iomgr/endpoint_pair_posix.c src/core/lib/iomgr/endpoint_pair_windows.c src/core/lib/iomgr/error.c + src/core/lib/iomgr/ev_epoll_linux.c src/core/lib/iomgr/ev_poll_and_epoll_posix.c src/core/lib/iomgr/ev_poll_posix.c src/core/lib/iomgr/ev_posix.c @@ -535,6 +543,7 @@ add_library(grpc_unsecure src/core/lib/iomgr/iomgr_posix.c src/core/lib/iomgr/iomgr_windows.c src/core/lib/iomgr/load_file.c + src/core/lib/iomgr/network_status_tracker.c src/core/lib/iomgr/polling_entity.c src/core/lib/iomgr/pollset_set_windows.c src/core/lib/iomgr/pollset_windows.c diff --git a/templates/CMakeLists.txt.template b/templates/CMakeLists.txt.template index 06085eb722b..76299cb21bf 100644 --- a/templates/CMakeLists.txt.template +++ b/templates/CMakeLists.txt.template @@ -6,7 +6,10 @@ # Please look at the templates directory instead. # This file can be regenerated from the template by running # tools/buildgen/generate_projects.sh - + # + # Additionally, this is currently very experimental, and unsupported. + # Further work will happen on that file. + # # Copyright 2015, Google Inc. # All rights reserved. # diff --git a/third_party/protobuf b/third_party/protobuf index d4d13a4349e..3470b6895aa 160000 --- a/third_party/protobuf +++ b/third_party/protobuf @@ -1 +1 @@ -Subproject commit d4d13a4349e4e59d67f311185ddcc1890d956d7a +Subproject commit 3470b6895aa659b7559ed678e029a5338e535f14 From f29e364168240bc472e22a6b55d2f27d501fceea Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Fri, 1 Jul 2016 15:56:57 -0700 Subject: [PATCH 0403/1039] Bump protobuf version to beta-3.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It’s just the merge of beta-3.1 and beta-3.2, both of which we need. --- .gitmodules | 2 +- examples/cpp/helloworld/Makefile | 2 +- examples/cpp/route_guide/Makefile | 2 +- third_party/protobuf | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.gitmodules b/.gitmodules index 1e1876b724e..ce647f3c455 100644 --- a/.gitmodules +++ b/.gitmodules @@ -4,7 +4,7 @@ [submodule "third_party/protobuf"] path = third_party/protobuf url = https://github.com/google/protobuf.git - branch = v3.0.0-beta-3.1 + branch = 3.0.0-beta-3 [submodule "third_party/gflags"] path = third_party/gflags url = https://github.com/gflags/gflags.git diff --git a/examples/cpp/helloworld/Makefile b/examples/cpp/helloworld/Makefile index 780e5e427a7..b45b3c7ee0c 100644 --- a/examples/cpp/helloworld/Makefile +++ b/examples/cpp/helloworld/Makefile @@ -97,7 +97,7 @@ ifneq ($(HAS_VALID_PROTOC),true) @echo "Please install Google protocol buffers 3.0.0 and its compiler." @echo "You can find it here:" @echo - @echo " https://github.com/google/protobuf/releases/tag/v3.0.0-beta-2" + @echo " https://github.com/google/protobuf/releases/tag/v3.0.0-beta-3.3" @echo @echo "Here is what I get when trying to evaluate your version of protoc:" @echo diff --git a/examples/cpp/route_guide/Makefile b/examples/cpp/route_guide/Makefile index 11f2a00cc89..50ecf041f56 100644 --- a/examples/cpp/route_guide/Makefile +++ b/examples/cpp/route_guide/Makefile @@ -86,7 +86,7 @@ ifneq ($(HAS_VALID_PROTOC),true) @echo "Please install Google protocol buffers 3.0.0 and its compiler." @echo "You can find it here:" @echo - @echo " https://github.com/google/protobuf/releases/tag/v3.0.0-beta-2" + @echo " https://github.com/google/protobuf/releases/tag/v3.0.0-beta-3.3" @echo @echo "Here is what I get when trying to evaluate your version of protoc:" @echo diff --git a/third_party/protobuf b/third_party/protobuf index d4d13a4349e..bdeb215cab2 160000 --- a/third_party/protobuf +++ b/third_party/protobuf @@ -1 +1 @@ -Subproject commit d4d13a4349e4e59d67f311185ddcc1890d956d7a +Subproject commit bdeb215cab2985195325fcd5e70c3fa751f46e0f From 3e53e1a5be4ee1d0433a64b73553496fa3f93827 Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Wed, 29 Jun 2016 12:57:44 -0700 Subject: [PATCH 0404/1039] ProtoCompiler podspec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Still needed: - Make it build protoc if it’s not in the root (i.e. if we're developing locally) - Put the gRPC plugin in the GitHub release page, and add a podspec for it that does the same --- src/objective-c/ProtoCompiler.podspec | 56 +++++++++++++++++++ src/objective-c/tests/Podfile | 1 + .../tests/RemoteTestClient/RemoteTest.podspec | 13 +++-- 3 files changed, 66 insertions(+), 4 deletions(-) create mode 100644 src/objective-c/ProtoCompiler.podspec diff --git a/src/objective-c/ProtoCompiler.podspec b/src/objective-c/ProtoCompiler.podspec new file mode 100644 index 00000000000..2f13853b0b5 --- /dev/null +++ b/src/objective-c/ProtoCompiler.podspec @@ -0,0 +1,56 @@ +# BoringSSL CocoaPods podspec + +# Copyright 2015, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Pod::Spec.new do |s| + s.name = 'ProtoCompiler' + v = '3.0.0-beta-3.1' + s.version = v + s.summary = 'The Protobuf Compiler (protoc) generates Objective-C files from .proto files' + s.description = <<-DESC + This podspec only downloads protoc so that local pods generating protos can execute it as part + of their prepare_command. + The generated code will have a dependency on the Protobuf Objective-C runtime of the same + version. The runtime can be obtained as the "Protobuf" pod. + DESC + s.homepage = 'https://github.com/google/protobuf' + s.license = 'New BSD' + # "The name and email addresses of the library maintainers, not the Podspec maintainer." + s.authors = { 'The Protocol Buffers contributors' => 'protobuf@googlegroups.com' } + + s.source = { + :http => "https://github.com/google/protobuf/releases/download/v#{v}/protoc-#{v}-osx-fat.zip", + # TODO(jcanizales): Add sha1 or sha256 + # :sha1 => '??', + } + + s.preserve_paths = 'protoc', + 'google/**/*.proto' # Well-known protobuf types +end diff --git a/src/objective-c/tests/Podfile b/src/objective-c/tests/Podfile index 30a34260d40..168114da37a 100644 --- a/src/objective-c/tests/Podfile +++ b/src/objective-c/tests/Podfile @@ -17,6 +17,7 @@ GRPC_LOCAL_SRC = '../../..' ).each do |target_name| target target_name do pod 'Protobuf', :path => "#{GRPC_LOCAL_SRC}/third_party/protobuf", :inhibit_warnings => true + pod 'ProtoCompiler', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c" pod 'BoringSSL', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c", :inhibit_warnings => true pod 'CronetFramework', :podspec => "#{GRPC_LOCAL_SRC}/src/objective-c" pod 'gRPC', :path => GRPC_LOCAL_SRC diff --git a/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec b/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec index 887380eb55f..e705310370d 100644 --- a/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec +++ b/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec @@ -11,13 +11,18 @@ Pod::Spec.new do |s| s.osx.deployment_target = '10.9' # Run protoc with the Objective-C and gRPC plugins to generate protocol messages and gRPC clients. + repo_root = '../../../..' + pods_root = "#{repo_root}/src/objective-c/tests/Pods" + bin_dir = "#{repo_root}/bins/$CONFIG" + + protoc = "#{pods_root}/ProtoCompiler/protoc" # "#{bin_dir}/protobuf/protoc" + plugin = "#{bin_dir}/grpc_objective_c_plugin" s.prepare_command = <<-CMD - BINDIR=../../../../bins/$CONFIG - PROTOC=$BINDIR/protobuf/protoc - PLUGIN=$BINDIR/grpc_objective_c_plugin - $PROTOC --plugin=protoc-gen-grpc=$PLUGIN --objc_out=. --grpc_out=. *.proto + #{protoc} --plugin=protoc-gen-grpc=#{plugin} --objc_out=. --grpc_out=. *.proto CMD + s.dependency "ProtoCompiler", "~> 3.0.0-beta-3.1" + s.subspec "Messages" do |ms| ms.source_files = "*.pbobjc.{h,m}" ms.header_mappings_dir = "." From cb58105fd616b5e64659368a37624ef5d6530633 Mon Sep 17 00:00:00 2001 From: Nathaniel Manista Date: Sat, 2 Jul 2016 08:18:04 +0000 Subject: [PATCH 0405/1039] Fix _Rendezvous.exception for successful calls --- .../grpcio/grpc/beta/_client_adaptations.py | 5 +++- ...e_invocation_asynchronous_event_service.py | 25 +++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/src/python/grpcio/grpc/beta/_client_adaptations.py b/src/python/grpcio/grpc/beta/_client_adaptations.py index 56456cc117f..73415e0be7f 100644 --- a/src/python/grpcio/grpc/beta/_client_adaptations.py +++ b/src/python/grpcio/grpc/beta/_client_adaptations.py @@ -117,7 +117,10 @@ class _Rendezvous(future.Future, face.Call): def exception(self, timeout=None): try: rpc_error_call = self._future.exception(timeout=timeout) - return _abortion_error(rpc_error_call) + if rpc_error_call is None: + return None + else: + return _abortion_error(rpc_error_call) except grpc.FutureTimeoutError: raise future.TimeoutError() except grpc.FutureCancelledError: diff --git a/src/python/grpcio/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py b/src/python/grpcio/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py index 791620307b5..d32208f9eb0 100644 --- a/src/python/grpcio/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py +++ b/src/python/grpcio/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py @@ -41,6 +41,7 @@ from concurrent import futures import six # test_interfaces is referenced from specification in this module. +from grpc.framework.foundation import future from grpc.framework.foundation import logging_pool from grpc.framework.interfaces.face import face from tests.unit.framework.common import test_constants @@ -159,6 +160,8 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. test_messages.verify(request, response, self) self.assertIs(callback.future(), response_future) + self.assertIsNone(response_future.exception()) + self.assertIsNone(response_future.traceback()) def testSuccessfulUnaryRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -191,6 +194,8 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. test_messages.verify(requests, response, self) self.assertIs(future_passed_to_callback, response_future) + self.assertIsNone(response_future.exception()) + self.assertIsNone(response_future.traceback()) def testSuccessfulStreamRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -301,6 +306,12 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. self.assertIs(callback.future(), response_future) self.assertFalse(cancel_method_return_value) self.assertTrue(response_future.cancelled()) + with self.assertRaises(future.CancelledError): + response_future.result() + with self.assertRaises(future.CancelledError): + response_future.exception() + with self.assertRaises(future.CancelledError): + response_future.traceback() def testCancelledUnaryRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -332,6 +343,12 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. self.assertIs(callback.future(), response_future) self.assertFalse(cancel_method_return_value) self.assertTrue(response_future.cancelled()) + with self.assertRaises(future.CancelledError): + response_future.result() + with self.assertRaises(future.CancelledError): + response_future.exception() + with self.assertRaises(future.CancelledError): + response_future.traceback() def testCancelledStreamRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -363,6 +380,9 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. response_future.exception(), face.ExpirationError) with self.assertRaises(face.ExpirationError): response_future.result() + self.assertIsInstance( + response_future.exception(), face.AbortionError) + self.assertIsNotNone(response_future.traceback()) def testExpiredUnaryRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -392,6 +412,9 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. response_future.exception(), face.ExpirationError) with self.assertRaises(face.ExpirationError): response_future.result() + self.assertIsInstance( + response_future.exception(), face.AbortionError) + self.assertIsNotNone(response_future.traceback()) def testExpiredStreamRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -426,6 +449,7 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. response_future.exception(), face.ExpirationError) with self.assertRaises(face.ExpirationError): response_future.result() + self.assertIsNotNone(response_future.traceback()) def testFailedUnaryRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -463,6 +487,7 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. response_future.exception(), face.ExpirationError) with self.assertRaises(face.ExpirationError): response_future.result() + self.assertIsNotNone(response_future.traceback()) def testFailedStreamRequestStreamResponse(self): for (group, method), test_messages_sequence in ( From ecbe2d52853ee70716479f0eaaef8564667255ec Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Sat, 2 Jul 2016 09:50:17 -0700 Subject: [PATCH 0406/1039] Added test for C --- src/core/lib/surface/byte_buffer_reader.c | 2 +- test/core/surface/byte_buffer_reader_test.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/src/core/lib/surface/byte_buffer_reader.c b/src/core/lib/surface/byte_buffer_reader.c index fb38bf51027..310bacb2c94 100644 --- a/src/core/lib/surface/byte_buffer_reader.c +++ b/src/core/lib/surface/byte_buffer_reader.c @@ -67,7 +67,7 @@ int grpc_byte_buffer_reader_init(grpc_byte_buffer_reader *reader, &decompressed_slices_buffer) == 0) { gpr_log(GPR_ERROR, "Unexpected error decompressing data for algorithm with enum " - "value '%d'. Reading data as if it were uncompressed.", + "value '%d'.", reader->buffer_in->data.raw.compression); memset(reader, 0, sizeof(*reader)); return 0; diff --git a/test/core/surface/byte_buffer_reader_test.c b/test/core/surface/byte_buffer_reader_test.c index 1c779a8a847..4231d899473 100644 --- a/test/core/surface/byte_buffer_reader_test.c +++ b/test/core/surface/byte_buffer_reader_test.c @@ -115,6 +115,20 @@ static void test_read_none_compressed_slice(void) { grpc_byte_buffer_destroy(buffer); } +static void test_read_corrupted_slice(void) { + gpr_slice slice; + grpc_byte_buffer *buffer; + grpc_byte_buffer_reader reader; + + LOG_TEST(__func__); + slice = gpr_slice_from_copied_string("test"); + buffer = grpc_raw_byte_buffer_create(&slice, 1); + buffer->data.raw.compression = GRPC_COMPRESS_GZIP; /* lies! */ + gpr_slice_unref(slice); + GPR_ASSERT(!grpc_byte_buffer_reader_init(&reader, buffer)); + grpc_byte_buffer_destroy(buffer); +} + static void read_compressed_slice(grpc_compression_algorithm algorithm, size_t input_size) { gpr_slice input_slice; @@ -267,6 +281,7 @@ int main(int argc, char **argv) { test_read_none_compressed_slice(); test_read_gzip_compressed_slice(); test_read_deflate_compressed_slice(); + test_read_corrupted_slice(); test_byte_buffer_from_reader(); test_byte_buffer_copy(); test_readall(); From a72712e245ca39d35d67532b0d66cd2dc1959e4d Mon Sep 17 00:00:00 2001 From: Tamas Berghammer Date: Wed, 6 Jul 2016 10:30:25 +0100 Subject: [PATCH 0407/1039] Remove gmock protobuf dependency from cmake build A full build of protobuf depends on gmock even though it is not part of a standrad checkout. This CL explicitly disable the build of the protobuf tests to get rid of this dependency. If somebody want to build the protobuf tests then they have to download gmock to the protobuf directory and specify -Dprotobuf_BUILD_TESTS=ON to the cmake command line. Fixes https://github.com/grpc/grpc/issues/7233 --- CMakeLists.txt | 7 +++++++ templates/CMakeLists.txt.template | 7 +++++++ 2 files changed, 14 insertions(+) diff --git a/CMakeLists.txt b/CMakeLists.txt index 9caf03191fa..dc002a4e342 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -58,6 +58,13 @@ if(NOT ZLIB_ROOT_DIR) set(ZLIB_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/third_party/zlib) endif() +# Building the protobuf tests require gmock what is not part of a standard protobuf checkout. +# Disable them unless they are explicitly requested from the cmake command line (when we assume +# gmock is downloaded to the right location inside protobuf). +if(NOT protobuf_BUILD_TESTS) + set(protobuf_BUILD_TESTS OFF CACHE BOOL "Build protobuf tests") +endif() + add_subdirectory(${BORINGSSL_ROOT_DIR} third_party/boringssl) add_subdirectory(${PROTOBUF_ROOT_DIR}/cmake third_party/protobuf) add_subdirectory(${ZLIB_ROOT_DIR} third_party/zlib) diff --git a/templates/CMakeLists.txt.template b/templates/CMakeLists.txt.template index 76299cb21bf..52e8b866be1 100644 --- a/templates/CMakeLists.txt.template +++ b/templates/CMakeLists.txt.template @@ -74,6 +74,13 @@ set(ZLIB_ROOT_DIR <%text>${CMAKE_CURRENT_SOURCE_DIR}/third_party/zlib) endif() + # Building the protobuf tests require gmock what is not part of a standard protobuf checkout. + # Disable them unless they are explicitly requested from the cmake command line (when we assume + # gmock is downloaded to the right location inside protobuf). + if(NOT protobuf_BUILD_TESTS) + set(protobuf_BUILD_TESTS OFF CACHE BOOL "Build protobuf tests") + endif() + add_subdirectory(<%text>${BORINGSSL_ROOT_DIR} third_party/boringssl) add_subdirectory(<%text>${PROTOBUF_ROOT_DIR}/cmake third_party/protobuf) add_subdirectory(<%text>${ZLIB_ROOT_DIR} third_party/zlib) From 611bfc21c45c89a7fce23f0f26296404590a0b94 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Wed, 6 Jul 2016 09:25:47 -0700 Subject: [PATCH 0408/1039] Add Node health check generated code and license files --- src/node/health_check/LICENSE | 28 ++ src/node/health_check/v1/health_grpc_pb.js | 74 +++++ src/node/health_check/v1/health_pb.js | 342 +++++++++++++++++++++ 3 files changed, 444 insertions(+) create mode 100644 src/node/health_check/LICENSE create mode 100644 src/node/health_check/v1/health_grpc_pb.js create mode 100644 src/node/health_check/v1/health_pb.js diff --git a/src/node/health_check/LICENSE b/src/node/health_check/LICENSE new file mode 100644 index 00000000000..0209b570e10 --- /dev/null +++ b/src/node/health_check/LICENSE @@ -0,0 +1,28 @@ +Copyright 2015, Google Inc. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/src/node/health_check/v1/health_grpc_pb.js b/src/node/health_check/v1/health_grpc_pb.js new file mode 100644 index 00000000000..89bc304e566 --- /dev/null +++ b/src/node/health_check/v1/health_grpc_pb.js @@ -0,0 +1,74 @@ +// GENERATED CODE -- DO NOT EDIT! + +// Original file comments: +// Copyright 2015, Google Inc. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above +// copyright notice, this list of conditions and the following disclaimer +// in the documentation and/or other materials provided with the +// distribution. +// * Neither the name of Google Inc. nor the names of its +// contributors may be used to endorse or promote products derived from +// this software without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +// +'use strict'; +var grpc = require('grpc'); +var v1_health_pb = require('../v1/health_pb.js'); + +function serialize_HealthCheckRequest(arg) { + if (!(arg instanceof v1_health_pb.HealthCheckRequest)) { + throw new Error('Expected argument of type HealthCheckRequest'); + } + return new Buffer(arg.serializeBinary()); +} + +function deserialize_HealthCheckRequest(buffer_arg) { + return v1_health_pb.HealthCheckRequest.deserializeBinary(new Uint8Array(buffer_arg)); +} + +function serialize_HealthCheckResponse(arg) { + if (!(arg instanceof v1_health_pb.HealthCheckResponse)) { + throw new Error('Expected argument of type HealthCheckResponse'); + } + return new Buffer(arg.serializeBinary()); +} + +function deserialize_HealthCheckResponse(buffer_arg) { + return v1_health_pb.HealthCheckResponse.deserializeBinary(new Uint8Array(buffer_arg)); +} + + +var HealthService = exports.HealthService = { + check: { + path: '/grpc.health.v1.Health/Check', + requestStream: false, + responseStream: false, + requestType: v1_health_pb.HealthCheckRequest, + responseType: v1_health_pb.HealthCheckResponse, + requestSerialize: serialize_HealthCheckRequest, + requestDeserialize: deserialize_HealthCheckRequest, + responseSerialize: serialize_HealthCheckResponse, + responseDeserialize: deserialize_HealthCheckResponse, + }, +}; + +exports.HealthClient = grpc.makeGenericClientConstructor(HealthService); diff --git a/src/node/health_check/v1/health_pb.js b/src/node/health_check/v1/health_pb.js new file mode 100644 index 00000000000..b36d47cdbb9 --- /dev/null +++ b/src/node/health_check/v1/health_pb.js @@ -0,0 +1,342 @@ +/** + * @fileoverview + * @enhanceable + * @public + */ +// GENERATED CODE -- DO NOT EDIT! + +var jspb = require('google-protobuf'); +var goog = jspb; +var global = Function('return this')(); + +goog.exportSymbol('proto.grpc.health.v1.HealthCheckRequest', null, global); +goog.exportSymbol('proto.grpc.health.v1.HealthCheckResponse', null, global); +goog.exportSymbol('proto.grpc.health.v1.HealthCheckResponse.ServingStatus', null, global); + +/** + * Generated by JsPbCodeGenerator. + * @param {Array=} opt_data Optional initial data array, typically from a + * server response, or constructed directly in Javascript. The array is used + * in place and becomes part of the constructed object. It is not cloned. + * If no data is provided, the constructed object will be empty, but still + * valid. + * @extends {jspb.Message} + * @constructor + */ +proto.grpc.health.v1.HealthCheckRequest = function(opt_data) { + jspb.Message.initialize(this, opt_data, 0, -1, null, null); +}; +goog.inherits(proto.grpc.health.v1.HealthCheckRequest, jspb.Message); +if (goog.DEBUG && !COMPILED) { + proto.grpc.health.v1.HealthCheckRequest.displayName = 'proto.grpc.health.v1.HealthCheckRequest'; +} + + +if (jspb.Message.GENERATE_TO_OBJECT) { +/** + * Creates an object representation of this proto suitable for use in Soy templates. + * Field names that are reserved in JavaScript and will be renamed to pb_name. + * To access a reserved field use, foo.pb_, eg, foo.pb_default. + * For the list of reserved names please see: + * com.google.apps.jspb.JsClassTemplate.JS_RESERVED_WORDS. + * @param {boolean=} opt_includeInstance Whether to include the JSPB instance + * for transitional soy proto support: http://goto/soy-param-migration + * @return {!Object} + */ +proto.grpc.health.v1.HealthCheckRequest.prototype.toObject = function(opt_includeInstance) { + return proto.grpc.health.v1.HealthCheckRequest.toObject(opt_includeInstance, this); +}; + + +/** + * Static version of the {@see toObject} method. + * @param {boolean|undefined} includeInstance Whether to include the JSPB + * instance for transitional soy proto support: + * http://goto/soy-param-migration + * @param {!proto.grpc.health.v1.HealthCheckRequest} msg The msg instance to transform. + * @return {!Object} + */ +proto.grpc.health.v1.HealthCheckRequest.toObject = function(includeInstance, msg) { + var f, obj = { + service: msg.getService() + }; + + if (includeInstance) { + obj.$jspbMessageInstance = msg; + } + return obj; +}; +} + + +/** + * Deserializes binary data (in protobuf wire format). + * @param {jspb.ByteSource} bytes The bytes to deserialize. + * @return {!proto.grpc.health.v1.HealthCheckRequest} + */ +proto.grpc.health.v1.HealthCheckRequest.deserializeBinary = function(bytes) { + var reader = new jspb.BinaryReader(bytes); + var msg = new proto.grpc.health.v1.HealthCheckRequest; + return proto.grpc.health.v1.HealthCheckRequest.deserializeBinaryFromReader(msg, reader); +}; + + +/** + * Deserializes binary data (in protobuf wire format) from the + * given reader into the given message object. + * @param {!proto.grpc.health.v1.HealthCheckRequest} msg The message object to deserialize into. + * @param {!jspb.BinaryReader} reader The BinaryReader to use. + * @return {!proto.grpc.health.v1.HealthCheckRequest} + */ +proto.grpc.health.v1.HealthCheckRequest.deserializeBinaryFromReader = function(msg, reader) { + while (reader.nextField()) { + if (reader.isEndGroup()) { + break; + } + var field = reader.getFieldNumber(); + switch (field) { + case 1: + var value = /** @type {string} */ (reader.readString()); + msg.setService(value); + break; + default: + reader.skipField(); + break; + } + } + return msg; +}; + + +/** + * Class method variant: serializes the given message to binary data + * (in protobuf wire format), writing to the given BinaryWriter. + * @param {!proto.grpc.health.v1.HealthCheckRequest} message + * @param {!jspb.BinaryWriter} writer + */ +proto.grpc.health.v1.HealthCheckRequest.serializeBinaryToWriter = function(message, writer) { + message.serializeBinaryToWriter(writer); +}; + + +/** + * Serializes the message to binary data (in protobuf wire format). + * @return {!Uint8Array} + */ +proto.grpc.health.v1.HealthCheckRequest.prototype.serializeBinary = function() { + var writer = new jspb.BinaryWriter(); + this.serializeBinaryToWriter(writer); + return writer.getResultBuffer(); +}; + + +/** + * Serializes the message to binary data (in protobuf wire format), + * writing to the given BinaryWriter. + * @param {!jspb.BinaryWriter} writer + */ +proto.grpc.health.v1.HealthCheckRequest.prototype.serializeBinaryToWriter = function (writer) { + var f = undefined; + f = this.getService(); + if (f.length > 0) { + writer.writeString( + 1, + f + ); + } +}; + + +/** + * Creates a deep clone of this proto. No data is shared with the original. + * @return {!proto.grpc.health.v1.HealthCheckRequest} The clone. + */ +proto.grpc.health.v1.HealthCheckRequest.prototype.cloneMessage = function() { + return /** @type {!proto.grpc.health.v1.HealthCheckRequest} */ (jspb.Message.cloneMessage(this)); +}; + + +/** + * optional string service = 1; + * @return {string} + */ +proto.grpc.health.v1.HealthCheckRequest.prototype.getService = function() { + return /** @type {string} */ (jspb.Message.getFieldProto3(this, 1, "")); +}; + + +/** @param {string} value */ +proto.grpc.health.v1.HealthCheckRequest.prototype.setService = function(value) { + jspb.Message.setField(this, 1, value); +}; + + + +/** + * Generated by JsPbCodeGenerator. + * @param {Array=} opt_data Optional initial data array, typically from a + * server response, or constructed directly in Javascript. The array is used + * in place and becomes part of the constructed object. It is not cloned. + * If no data is provided, the constructed object will be empty, but still + * valid. + * @extends {jspb.Message} + * @constructor + */ +proto.grpc.health.v1.HealthCheckResponse = function(opt_data) { + jspb.Message.initialize(this, opt_data, 0, -1, null, null); +}; +goog.inherits(proto.grpc.health.v1.HealthCheckResponse, jspb.Message); +if (goog.DEBUG && !COMPILED) { + proto.grpc.health.v1.HealthCheckResponse.displayName = 'proto.grpc.health.v1.HealthCheckResponse'; +} + + +if (jspb.Message.GENERATE_TO_OBJECT) { +/** + * Creates an object representation of this proto suitable for use in Soy templates. + * Field names that are reserved in JavaScript and will be renamed to pb_name. + * To access a reserved field use, foo.pb_, eg, foo.pb_default. + * For the list of reserved names please see: + * com.google.apps.jspb.JsClassTemplate.JS_RESERVED_WORDS. + * @param {boolean=} opt_includeInstance Whether to include the JSPB instance + * for transitional soy proto support: http://goto/soy-param-migration + * @return {!Object} + */ +proto.grpc.health.v1.HealthCheckResponse.prototype.toObject = function(opt_includeInstance) { + return proto.grpc.health.v1.HealthCheckResponse.toObject(opt_includeInstance, this); +}; + + +/** + * Static version of the {@see toObject} method. + * @param {boolean|undefined} includeInstance Whether to include the JSPB + * instance for transitional soy proto support: + * http://goto/soy-param-migration + * @param {!proto.grpc.health.v1.HealthCheckResponse} msg The msg instance to transform. + * @return {!Object} + */ +proto.grpc.health.v1.HealthCheckResponse.toObject = function(includeInstance, msg) { + var f, obj = { + status: msg.getStatus() + }; + + if (includeInstance) { + obj.$jspbMessageInstance = msg; + } + return obj; +}; +} + + +/** + * Deserializes binary data (in protobuf wire format). + * @param {jspb.ByteSource} bytes The bytes to deserialize. + * @return {!proto.grpc.health.v1.HealthCheckResponse} + */ +proto.grpc.health.v1.HealthCheckResponse.deserializeBinary = function(bytes) { + var reader = new jspb.BinaryReader(bytes); + var msg = new proto.grpc.health.v1.HealthCheckResponse; + return proto.grpc.health.v1.HealthCheckResponse.deserializeBinaryFromReader(msg, reader); +}; + + +/** + * Deserializes binary data (in protobuf wire format) from the + * given reader into the given message object. + * @param {!proto.grpc.health.v1.HealthCheckResponse} msg The message object to deserialize into. + * @param {!jspb.BinaryReader} reader The BinaryReader to use. + * @return {!proto.grpc.health.v1.HealthCheckResponse} + */ +proto.grpc.health.v1.HealthCheckResponse.deserializeBinaryFromReader = function(msg, reader) { + while (reader.nextField()) { + if (reader.isEndGroup()) { + break; + } + var field = reader.getFieldNumber(); + switch (field) { + case 1: + var value = /** @type {!proto.grpc.health.v1.HealthCheckResponse.ServingStatus} */ (reader.readEnum()); + msg.setStatus(value); + break; + default: + reader.skipField(); + break; + } + } + return msg; +}; + + +/** + * Class method variant: serializes the given message to binary data + * (in protobuf wire format), writing to the given BinaryWriter. + * @param {!proto.grpc.health.v1.HealthCheckResponse} message + * @param {!jspb.BinaryWriter} writer + */ +proto.grpc.health.v1.HealthCheckResponse.serializeBinaryToWriter = function(message, writer) { + message.serializeBinaryToWriter(writer); +}; + + +/** + * Serializes the message to binary data (in protobuf wire format). + * @return {!Uint8Array} + */ +proto.grpc.health.v1.HealthCheckResponse.prototype.serializeBinary = function() { + var writer = new jspb.BinaryWriter(); + this.serializeBinaryToWriter(writer); + return writer.getResultBuffer(); +}; + + +/** + * Serializes the message to binary data (in protobuf wire format), + * writing to the given BinaryWriter. + * @param {!jspb.BinaryWriter} writer + */ +proto.grpc.health.v1.HealthCheckResponse.prototype.serializeBinaryToWriter = function (writer) { + var f = undefined; + f = this.getStatus(); + if (f !== 0.0) { + writer.writeEnum( + 1, + f + ); + } +}; + + +/** + * Creates a deep clone of this proto. No data is shared with the original. + * @return {!proto.grpc.health.v1.HealthCheckResponse} The clone. + */ +proto.grpc.health.v1.HealthCheckResponse.prototype.cloneMessage = function() { + return /** @type {!proto.grpc.health.v1.HealthCheckResponse} */ (jspb.Message.cloneMessage(this)); +}; + + +/** + * optional ServingStatus status = 1; + * @return {!proto.grpc.health.v1.HealthCheckResponse.ServingStatus} + */ +proto.grpc.health.v1.HealthCheckResponse.prototype.getStatus = function() { + return /** @type {!proto.grpc.health.v1.HealthCheckResponse.ServingStatus} */ (jspb.Message.getFieldProto3(this, 1, 0)); +}; + + +/** @param {!proto.grpc.health.v1.HealthCheckResponse.ServingStatus} value */ +proto.grpc.health.v1.HealthCheckResponse.prototype.setStatus = function(value) { + jspb.Message.setField(this, 1, value); +}; + + +/** + * @enum {number} + */ +proto.grpc.health.v1.HealthCheckResponse.ServingStatus = { + UNKNOWN: 0, + SERVING: 1, + NOT_SERVING: 2 +}; + +goog.object.extend(exports, proto.grpc.health.v1); From 1500761b075abb5971a8662c2ccb6bff828a6a06 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 6 Jul 2016 09:36:16 -0700 Subject: [PATCH 0409/1039] Prevent polling island + workqueue reference loop --- src/core/lib/iomgr/ev_epoll_linux.c | 48 ++++++++++++++++++++--------- 1 file changed, 33 insertions(+), 15 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index e5aab293bd3..634946a781a 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -153,7 +153,7 @@ static void fd_global_shutdown(void); * Polling island Declarations */ -// #define GRPC_PI_REF_COUNT_DEBUG +//#define GRPC_PI_REF_COUNT_DEBUG #ifdef GRPC_PI_REF_COUNT_DEBUG #define PI_ADD_REF(p, r) pi_add_ref_dbg((p), (r), __FILE__, __LINE__) @@ -174,7 +174,7 @@ typedef struct polling_island { Once the ref count becomes zero, this structure is destroyed which means we should ensure that there is never a scenario where a PI_ADD_REF() is racing with a PI_UNREF() that just made the ref_count zero. */ - gpr_refcount ref_count; + gpr_atm ref_count; /* Pointer to the polling_island this merged into. * merged_to value is only set once in polling_island's lifetime (and that too @@ -296,7 +296,7 @@ static void pi_unref(grpc_exec_ctx *exec_ctx, polling_island *pi); static void pi_add_ref_dbg(polling_island *pi, char *reason, char *file, int line) { - long old_cnt = gpr_atm_acq_load(&(pi->ref_count.count)); + long old_cnt = gpr_atm_acq_load(&pi->ref_count); pi_add_ref(pi); gpr_log(GPR_DEBUG, "Add ref pi: %p, old: %ld -> new:%ld (%s) - (%s, %d)", (void *)pi, old_cnt, old_cnt + 1, reason, file, line); @@ -304,17 +304,22 @@ static void pi_add_ref_dbg(polling_island *pi, char *reason, char *file, static void pi_unref_dbg(grpc_exec_ctx *exec_ctx, polling_island *pi, char *reason, char *file, int line) { - long old_cnt = gpr_atm_acq_load(&(pi->ref_count.count)); + long old_cnt = gpr_atm_acq_load(&pi->ref_count); pi_unref(exec_ctx, pi); gpr_log(GPR_DEBUG, "Unref pi: %p, old:%ld -> new:%ld (%s) - (%s, %d)", (void *)pi, old_cnt, (old_cnt - 1), reason, file, line); } #endif -static void pi_add_ref(polling_island *pi) { gpr_ref(&pi->ref_count); } +static void pi_add_ref(polling_island *pi) { + gpr_atm_no_barrier_fetch_add(&pi->ref_count, 1); +} static void pi_unref(grpc_exec_ctx *exec_ctx, polling_island *pi) { - /* If ref count went to zero, delete the polling island. + /* If ref count went to one, we're back to just the workqueue owning a ref. + Unref the workqueue to break the loop. + + If ref count went to zero, delete the polling island. Note that this deletion not be done under a lock. Once the ref count goes to zero, we are guaranteed that no one else holds a reference to the polling island (and that there is no racing pi_add_ref() call either). @@ -322,12 +327,20 @@ static void pi_unref(grpc_exec_ctx *exec_ctx, polling_island *pi) { Also, if we are deleting the polling island and the merged_to field is non-empty, we should remove a ref to the merged_to polling island */ - if (gpr_unref(&pi->ref_count)) { - polling_island *next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); - polling_island_delete(exec_ctx, pi); - if (next != NULL) { - PI_UNREF(exec_ctx, next, "pi_delete"); /* Recursive call */ + switch (gpr_atm_full_fetch_add(&pi->ref_count, -1)) { + case 2: /* last external ref: the only one now owned is by the workqueue */ + GRPC_WORKQUEUE_UNREF(exec_ctx, pi->workqueue, "polling_island"); + break; + case 1: { + polling_island *next = (polling_island *)gpr_atm_acq_load(&pi->merged_to); + polling_island_delete(exec_ctx, pi); + if (next != NULL) { + PI_UNREF(exec_ctx, next, "pi_delete"); /* Recursive call */ + } + break; } + case 0: + GPR_UNREACHABLE_CODE(return ); } } @@ -478,7 +491,7 @@ static polling_island *polling_island_create(grpc_exec_ctx *exec_ctx, pi->epoll_fd = -1; pi->workqueue = NULL; - gpr_ref_init(&pi->ref_count, 0); + gpr_atm_rel_store(&pi->ref_count, 0); gpr_atm_rel_store(&pi->merged_to, (gpr_atm)NULL); pi->epoll_fd = epoll_create1(EPOLL_CLOEXEC); @@ -501,7 +514,7 @@ static polling_island *polling_island_create(grpc_exec_ctx *exec_ctx, error); GPR_ASSERT(pi->workqueue->wakeup_read_fd->polling_island == NULL); pi->workqueue->wakeup_read_fd->polling_island = pi; - PI_ADD_REF(pi, 1); + PI_ADD_REF(pi, "fd"); } done: @@ -525,7 +538,6 @@ static void polling_island_delete(grpc_exec_ctx *exec_ctx, polling_island *pi) { gpr_atm_rel_store(&pi->merged_to, (gpr_atm)NULL); close(pi->epoll_fd); - GRPC_WORKQUEUE_UNREF(exec_ctx, pi->workqueue, "polling_island"); gpr_mu_destroy(&pi->mu); gpr_free(pi->fds); gpr_free(pi); @@ -885,6 +897,7 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, const char *reason) { bool is_fd_closed = false; grpc_error *error = GRPC_ERROR_NONE; + polling_island *unref_pi = NULL; gpr_mu_lock(&fd->mu); fd->on_done_closure = on_done; @@ -918,7 +931,7 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, polling_island_remove_fd_locked(pi_latest, fd, is_fd_closed, &error); gpr_mu_unlock(&pi_latest->mu); - PI_UNREF(exec_ctx, fd->polling_island, "fd_orphan"); + unref_pi = fd->polling_island; fd->polling_island = NULL; } gpr_mu_unlock(&fd->pi_mu); @@ -927,6 +940,9 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, gpr_mu_unlock(&fd->mu); UNREF_BY(fd, 2, reason); /* Drop the reference */ + if (unref_pi != NULL) { + PI_UNREF(exec_ctx, unref_pi, "fd_orphan"); + } GRPC_LOG_IF_ERROR("fd_orphan", GRPC_ERROR_REF(error)); } @@ -1595,6 +1611,8 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, gpr_mu_unlock(&fd->pi_mu); gpr_mu_unlock(&pollset->mu); + + GRPC_LOG_IF_ERROR("pollset_add_fd", error); } /******************************************************************************* From f0ae3e39f08313cb7105374d8da97b210edc1ef1 Mon Sep 17 00:00:00 2001 From: Julien Boeuf Date: Wed, 6 Jul 2016 11:23:51 -0700 Subject: [PATCH 0410/1039] Fixing #7182: Change the return status codes according to the documentation. --- src/core/lib/security/transport/client_auth_filter.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/core/lib/security/transport/client_auth_filter.c b/src/core/lib/security/transport/client_auth_filter.c index 14ccf72dc95..ed7929aa27e 100644 --- a/src/core/lib/security/transport/client_auth_filter.c +++ b/src/core/lib/security/transport/client_auth_filter.c @@ -176,7 +176,7 @@ static void send_security_metadata(grpc_exec_ctx *exec_ctx, calld->creds = grpc_composite_call_credentials_create(channel_call_creds, ctx->creds, NULL); if (calld->creds == NULL) { - bubble_up_error(exec_ctx, elem, GRPC_STATUS_INTERNAL, + bubble_up_error(exec_ctx, elem, GRPC_STATUS_UNAUTHENTICATED, "Incompatible credentials set on channel and call."); return; } @@ -205,7 +205,7 @@ static void on_host_checked(grpc_exec_ctx *exec_ctx, void *user_data, char *error_msg; gpr_asprintf(&error_msg, "Invalid host %s set in :authority metadata.", grpc_mdstr_as_c_string(calld->host)); - bubble_up_error(exec_ctx, elem, GRPC_STATUS_INTERNAL, error_msg); + bubble_up_error(exec_ctx, elem, GRPC_STATUS_UNAUTHENTICATED, error_msg); gpr_free(error_msg); } } From f83f8ca443778dc00b47b6364053970b30df83f9 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 6 Jul 2016 11:34:08 -0700 Subject: [PATCH 0411/1039] Remove pi_mu --- src/core/lib/iomgr/ev_epoll_linux.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 634946a781a..0c42d6e99a9 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -114,9 +114,7 @@ struct grpc_fd { grpc_closure *read_closure; grpc_closure *write_closure; - /* The polling island to which this fd belongs to and the mutex protecting the - the field */ - gpr_mu pi_mu; + /* The polling island to which this fd belongs to (protected by mu) */ struct polling_island *polling_island; struct grpc_fd *freelist_next; @@ -846,7 +844,6 @@ static grpc_fd *fd_create(int fd, const char *name) { if (new_fd == NULL) { new_fd = gpr_malloc(sizeof(grpc_fd)); gpr_mu_init(&new_fd->mu); - gpr_mu_init(&new_fd->pi_mu); } /* Note: It is not really needed to get the new_fd->mu lock here. If this is a @@ -925,7 +922,6 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, - Unlock the latest polling island - Set fd->polling_island to NULL (but remove the ref on the polling island before doing this.) */ - gpr_mu_lock(&fd->pi_mu); if (fd->polling_island != NULL) { polling_island *pi_latest = polling_island_lock(fd->polling_island); polling_island_remove_fd_locked(pi_latest, fd, is_fd_closed, &error); @@ -934,7 +930,6 @@ static void fd_orphan(grpc_exec_ctx *exec_ctx, grpc_fd *fd, unref_pi = fd->polling_island; fd->polling_island = NULL; } - gpr_mu_unlock(&fd->pi_mu); grpc_exec_ctx_sched(exec_ctx, fd->on_done_closure, error, NULL); @@ -1043,13 +1038,13 @@ static void fd_notify_on_write(grpc_exec_ctx *exec_ctx, grpc_fd *fd, } static grpc_workqueue *fd_get_workqueue(grpc_fd *fd) { - gpr_mu_lock(&fd->pi_mu); + gpr_mu_lock(&fd->mu); grpc_workqueue *workqueue = NULL; if (fd->polling_island != NULL) { workqueue = GRPC_WORKQUEUE_REF(fd->polling_island->workqueue, "get_workqueue"); } - gpr_mu_unlock(&fd->pi_mu); + gpr_mu_unlock(&fd->mu); return workqueue; } @@ -1534,7 +1529,7 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, grpc_error *error = GRPC_ERROR_NONE; gpr_mu_lock(&pollset->mu); - gpr_mu_lock(&fd->pi_mu); + gpr_mu_lock(&fd->mu); polling_island *pi_new = NULL; @@ -1609,7 +1604,7 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, pollset->polling_island = pi_new; } - gpr_mu_unlock(&fd->pi_mu); + gpr_mu_unlock(&fd->mu); gpr_mu_unlock(&pollset->mu); GRPC_LOG_IF_ERROR("pollset_add_fd", error); @@ -1763,9 +1758,9 @@ static void pollset_set_del_pollset_set(grpc_exec_ctx *exec_ctx, void *grpc_fd_get_polling_island(grpc_fd *fd) { polling_island *pi; - gpr_mu_lock(&fd->pi_mu); + gpr_mu_lock(&fd->mu); pi = fd->polling_island; - gpr_mu_unlock(&fd->pi_mu); + gpr_mu_unlock(&fd->mu); return pi; } From 7212c233e84f199742bb98176eef9f8d15ccaabb Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 6 Jul 2016 13:11:09 -0700 Subject: [PATCH 0412/1039] Fix mutex loop --- src/core/lib/iomgr/ev_epoll_linux.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 0c42d6e99a9..30f03c17bc3 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -1533,6 +1533,7 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, polling_island *pi_new = NULL; +retry: /* 1) If fd->polling_island and pollset->polling_island are both non-NULL and * equal, do nothing. * 2) If fd->polling_island and pollset->polling_island are both NULL, create @@ -1550,7 +1551,12 @@ static void pollset_add_fd(grpc_exec_ctx *exec_ctx, grpc_pollset *pollset, if (fd->polling_island == pollset->polling_island) { pi_new = fd->polling_island; if (pi_new == NULL) { + gpr_mu_unlock(&fd->mu); pi_new = polling_island_create(exec_ctx, fd, &error); + gpr_mu_lock(&fd->mu); + if (fd->polling_island != NULL) { + goto retry; + } GRPC_POLLING_TRACE( "pollset_add_fd: Created new polling island. pi_new: %p (fd: %d, " From 27da642ab990f3a69a11255395d80e71ffcbdd21 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 6 Jul 2016 13:14:46 -0700 Subject: [PATCH 0413/1039] Better implementation of backup loop --- src/core/lib/iomgr/ev_epoll_linux.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 30f03c17bc3..0fb4399fa7a 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -1555,13 +1555,19 @@ retry: pi_new = polling_island_create(exec_ctx, fd, &error); gpr_mu_lock(&fd->mu); if (fd->polling_island != NULL) { + GRPC_POLLING_TRACE( + "pollset_add_fd: Raced creating new polling island. pi_new: %p " + "(fd: %d, pollset: %p)", + (void *)pi_new, fd->fd, (void *)pollset); + PI_ADD_REF(pi_new, "dance_of_destruction"); + PI_UNREF(exec_ctx, pi_new, "dance_of_destruction"); goto retry; + } else { + GRPC_POLLING_TRACE( + "pollset_add_fd: Created new polling island. pi_new: %p (fd: %d, " + "pollset: %p)", + (void *)pi_new, fd->fd, (void *)pollset); } - - GRPC_POLLING_TRACE( - "pollset_add_fd: Created new polling island. pi_new: %p (fd: %d, " - "pollset: %p)", - (void *)pi_new, fd->fd, (void *)pollset); } } else if (fd->polling_island == NULL) { pi_new = polling_island_lock(pollset->polling_island); From 19a51a914da88924fa38ce0b6c2beca687d4e3c6 Mon Sep 17 00:00:00 2001 From: yang-g Date: Wed, 6 Jul 2016 13:19:00 -0700 Subject: [PATCH 0414/1039] Remove unused file --- src/core/lib/http/parser.c.orig | 357 -------------------------------- 1 file changed, 357 deletions(-) delete mode 100644 src/core/lib/http/parser.c.orig diff --git a/src/core/lib/http/parser.c.orig b/src/core/lib/http/parser.c.orig deleted file mode 100644 index 74d90fd8bfb..00000000000 --- a/src/core/lib/http/parser.c.orig +++ /dev/null @@ -1,357 +0,0 @@ -/* - * - * Copyright 2015, Google Inc. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are - * met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following disclaimer - * in the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Google Inc. nor the names of its - * contributors may be used to endorse or promote products derived from - * this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - */ - -#include "src/core/lib/http/parser.h" - -#include - -#include -#include -#include - -int grpc_http1_trace = 0; - -static char *buf2str(void *buffer, size_t length) { - char *out = gpr_malloc(length + 1); - memcpy(out, buffer, length); - out[length] = 0; - return out; -} - -static grpc_error *handle_response_line(grpc_http_parser *parser) { - uint8_t *beg = parser->cur_line; - uint8_t *cur = beg; - uint8_t *end = beg + parser->cur_line_length; - - if (cur == end || *cur++ != 'H') return GRPC_ERROR_CREATE("Expected 'H'"); - if (cur == end || *cur++ != 'T') return GRPC_ERROR_CREATE("Expected 'T'"); - if (cur == end || *cur++ != 'T') return GRPC_ERROR_CREATE("Expected 'T'"); - if (cur == end || *cur++ != 'P') return GRPC_ERROR_CREATE("Expected 'P'"); - if (cur == end || *cur++ != '/') return GRPC_ERROR_CREATE("Expected '/'"); - if (cur == end || *cur++ != '1') return GRPC_ERROR_CREATE("Expected '1'"); - if (cur == end || *cur++ != '.') return GRPC_ERROR_CREATE("Expected '.'"); - if (cur == end || *cur < '0' || *cur++ > '1') { - return GRPC_ERROR_CREATE("Expected HTTP/1.0 or HTTP/1.1"); - } - if (cur == end || *cur++ != ' ') return GRPC_ERROR_CREATE("Expected ' '"); - if (cur == end || *cur < '1' || *cur++ > '9') - return GRPC_ERROR_CREATE("Expected status code"); - if (cur == end || *cur < '0' || *cur++ > '9') - return GRPC_ERROR_CREATE("Expected status code"); - if (cur == end || *cur < '0' || *cur++ > '9') - return GRPC_ERROR_CREATE("Expected status code"); - parser->http.response->status = - (cur[-3] - '0') * 100 + (cur[-2] - '0') * 10 + (cur[-1] - '0'); - if (cur == end || *cur++ != ' ') return GRPC_ERROR_CREATE("Expected ' '"); - - /* we don't really care about the status code message */ - - return GRPC_ERROR_NONE; -} - -static grpc_error *handle_request_line(grpc_http_parser *parser) { - uint8_t *beg = parser->cur_line; - uint8_t *cur = beg; - uint8_t *end = beg + parser->cur_line_length; - uint8_t vers_major = 0; - uint8_t vers_minor = 0; - - while (cur != end && *cur++ != ' ') - ; - if (cur == end) return GRPC_ERROR_CREATE("No method on HTTP request line"); - parser->http.request->method = buf2str(beg, (size_t)(cur - beg - 1)); - - beg = cur; - while (cur != end && *cur++ != ' ') - ; - if (cur == end) return GRPC_ERROR_CREATE("No path on HTTP request line"); - parser->http.request->path = buf2str(beg, (size_t)(cur - beg - 1)); - - if (cur == end || *cur++ != 'H') return GRPC_ERROR_CREATE("Expected 'H'"); - if (cur == end || *cur++ != 'T') return GRPC_ERROR_CREATE("Expected 'T'"); - if (cur == end || *cur++ != 'T') return GRPC_ERROR_CREATE("Expected 'T'"); - if (cur == end || *cur++ != 'P') return GRPC_ERROR_CREATE("Expected 'P'"); - if (cur == end || *cur++ != '/') return GRPC_ERROR_CREATE("Expected '/'"); - vers_major = (uint8_t)(*cur++ - '1' + 1); - ++cur; - if (cur == end) - return GRPC_ERROR_CREATE("End of line in HTTP version string"); - vers_minor = (uint8_t)(*cur++ - '1' + 1); - - if (vers_major == 1) { - if (vers_minor == 0) { - parser->http.request->version = GRPC_HTTP_HTTP10; - } else if (vers_minor == 1) { - parser->http.request->version = GRPC_HTTP_HTTP11; - } else { - return GRPC_ERROR_CREATE( - "Expected one of HTTP/1.0, HTTP/1.1, or HTTP/2.0"); - } - } else if (vers_major == 2) { - if (vers_minor == 0) { - parser->http.request->version = GRPC_HTTP_HTTP20; - } else { - return GRPC_ERROR_CREATE( - "Expected one of HTTP/1.0, HTTP/1.1, or HTTP/2.0"); - } - } else { - return GRPC_ERROR_CREATE("Expected one of HTTP/1.0, HTTP/1.1, or HTTP/2.0"); - } - - return GRPC_ERROR_NONE; -} - -static grpc_error *handle_first_line(grpc_http_parser *parser) { - switch (parser->type) { - case GRPC_HTTP_REQUEST: - return handle_request_line(parser); - case GRPC_HTTP_RESPONSE: - return handle_response_line(parser); - } - GPR_UNREACHABLE_CODE(return GRPC_ERROR_CREATE("Should never reach here")); -} - -static grpc_error *add_header(grpc_http_parser *parser) { - uint8_t *beg = parser->cur_line; - uint8_t *cur = beg; - uint8_t *end = beg + parser->cur_line_length; - size_t *hdr_count = NULL; - grpc_http_header **hdrs = NULL; - grpc_http_header hdr = {NULL, NULL}; - grpc_error *error = GRPC_ERROR_NONE; - - GPR_ASSERT(cur != end); - - if (*cur == ' ' || *cur == '\t') { - error = GRPC_ERROR_CREATE("Continued header lines not supported yet"); - goto done; - } - - while (cur != end && *cur != ':') { - cur++; - } - if (cur == end) { -<<<<<<< HEAD - error = GRPC_ERROR_CREATE("Didn't find ':' in header string"); - goto done; -======= - if (grpc_http1_trace) { - gpr_log(GPR_ERROR, "Didn't find ':' in header string"); - } - goto error; ->>>>>>> a709afe241d8b264a1c326315f757b4a8d330207 - } - GPR_ASSERT(cur >= beg); - hdr.key = buf2str(beg, (size_t)(cur - beg)); - cur++; /* skip : */ - - while (cur != end && (*cur == ' ' || *cur == '\t')) { - cur++; - } - GPR_ASSERT((size_t)(end - cur) >= parser->cur_line_end_length); - hdr.value = buf2str(cur, (size_t)(end - cur) - parser->cur_line_end_length); - - switch (parser->type) { - case GRPC_HTTP_RESPONSE: - hdr_count = &parser->http.response->hdr_count; - hdrs = &parser->http.response->hdrs; - break; - case GRPC_HTTP_REQUEST: - hdr_count = &parser->http.request->hdr_count; - hdrs = &parser->http.request->hdrs; - break; - } - - if (*hdr_count == parser->hdr_capacity) { - parser->hdr_capacity = - GPR_MAX(parser->hdr_capacity + 1, parser->hdr_capacity * 3 / 2); - *hdrs = gpr_realloc(*hdrs, parser->hdr_capacity * sizeof(**hdrs)); - } - (*hdrs)[(*hdr_count)++] = hdr; - -done: - if (error != GRPC_ERROR_NONE) { - gpr_free(hdr.key); - gpr_free(hdr.value); - } - return error; -} - -static grpc_error *finish_line(grpc_http_parser *parser) { - grpc_error *err; - switch (parser->state) { - case GRPC_HTTP_FIRST_LINE: - err = handle_first_line(parser); - if (err != GRPC_ERROR_NONE) return err; - parser->state = GRPC_HTTP_HEADERS; - break; - case GRPC_HTTP_HEADERS: - if (parser->cur_line_length == parser->cur_line_end_length) { - parser->state = GRPC_HTTP_BODY; - break; - } - err = add_header(parser); - if (err != GRPC_ERROR_NONE) { - return err; - } - break; - case GRPC_HTTP_BODY: - GPR_UNREACHABLE_CODE(return GRPC_ERROR_CREATE("Should never reach here")); - } - - parser->cur_line_length = 0; - return GRPC_ERROR_NONE; -} - -static grpc_error *addbyte_body(grpc_http_parser *parser, uint8_t byte) { - size_t *body_length = NULL; - char **body = NULL; - - if (parser->type == GRPC_HTTP_RESPONSE) { - body_length = &parser->http.response->body_length; - body = &parser->http.response->body; - } else if (parser->type == GRPC_HTTP_REQUEST) { - body_length = &parser->http.request->body_length; - body = &parser->http.request->body; - } else { - GPR_UNREACHABLE_CODE(return GRPC_ERROR_CREATE("Should never reach here")); - } - - if (*body_length == parser->body_capacity) { - parser->body_capacity = GPR_MAX(8, parser->body_capacity * 3 / 2); - *body = gpr_realloc((void *)*body, parser->body_capacity); - } - (*body)[*body_length] = (char)byte; - (*body_length)++; - - return GRPC_ERROR_NONE; -} - -static bool check_line(grpc_http_parser *parser) { - if (parser->cur_line_length >= 2 && - parser->cur_line[parser->cur_line_length - 2] == '\r' && - parser->cur_line[parser->cur_line_length - 1] == '\n') { - return true; - } - - // HTTP request with \n\r line termiantors. - else if (parser->cur_line_length >= 2 && - parser->cur_line[parser->cur_line_length - 2] == '\n' && - parser->cur_line[parser->cur_line_length - 1] == '\r') { - return true; - } - - // HTTP request with only \n line terminators. - else if (parser->cur_line_length >= 1 && - parser->cur_line[parser->cur_line_length - 1] == '\n') { - parser->cur_line_end_length = 1; - return true; - } - - return false; -} - -static grpc_error *addbyte(grpc_http_parser *parser, uint8_t byte) { - switch (parser->state) { - case GRPC_HTTP_FIRST_LINE: - case GRPC_HTTP_HEADERS: - if (parser->cur_line_length >= GRPC_HTTP_PARSER_MAX_HEADER_LENGTH) { - if (grpc_http1_trace) - gpr_log(GPR_ERROR, "HTTP client max line length (%d) exceeded", - GRPC_HTTP_PARSER_MAX_HEADER_LENGTH); - return 0; - } - parser->cur_line[parser->cur_line_length] = byte; - parser->cur_line_length++; - if (check_line(parser)) { - return finish_line(parser); - } else { - return GRPC_ERROR_NONE; - } - GPR_UNREACHABLE_CODE(return 0); - case GRPC_HTTP_BODY: - return addbyte_body(parser, byte); - } - GPR_UNREACHABLE_CODE(return 0); -} - -void grpc_http_parser_init(grpc_http_parser *parser, grpc_http_type type, - void *request_or_response) { - memset(parser, 0, sizeof(*parser)); - parser->state = GRPC_HTTP_FIRST_LINE; - parser->type = type; - parser->http.request_or_response = request_or_response; - parser->cur_line_end_length = 2; -} - -void grpc_http_parser_destroy(grpc_http_parser *parser) {} - -void grpc_http_request_destroy(grpc_http_request *request) { - size_t i; - gpr_free(request->body); - for (i = 0; i < request->hdr_count; i++) { - gpr_free(request->hdrs[i].key); - gpr_free(request->hdrs[i].value); - } - gpr_free(request->hdrs); - gpr_free(request->method); - gpr_free(request->path); -} - -void grpc_http_response_destroy(grpc_http_response *response) { - size_t i; - gpr_free(response->body); - for (i = 0; i < response->hdr_count; i++) { - gpr_free(response->hdrs[i].key); - gpr_free(response->hdrs[i].value); - } - gpr_free(response->hdrs); -} - -grpc_error *grpc_http_parser_parse(grpc_http_parser *parser, gpr_slice slice) { - size_t i; - - for (i = 0; i < GPR_SLICE_LENGTH(slice); i++) { - grpc_error *err = addbyte(parser, GPR_SLICE_START_PTR(slice)[i]); - if (err != GRPC_ERROR_NONE) return err; - } - - return GRPC_ERROR_NONE; -} - -grpc_error *grpc_http_parser_eof(grpc_http_parser *parser) { - if (parser->state != GRPC_HTTP_BODY) { - return GRPC_ERROR_CREATE("Did not finish headers"); - } - return GRPC_ERROR_NONE; -} From d552dbdf4d47c48aa52b25f963e1ea3c7d87351d Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 6 Jul 2016 13:41:32 -0700 Subject: [PATCH 0415/1039] Cleanup test --- test/core/end2end/tests/network_status_change.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/test/core/end2end/tests/network_status_change.c b/test/core/end2end/tests/network_status_change.c index 10207844ab0..39ddc137543 100644 --- a/test/core/end2end/tests/network_status_change.c +++ b/test/core/end2end/tests/network_status_change.c @@ -186,9 +186,10 @@ static void test_invoke_network_status_change(grpc_end2end_test_config config) { GPR_ASSERT(GRPC_CALL_OK == error); cq_expect_completion(cqv, tag(102), 1); + cq_verify(cqv); + // Simulate the network loss event grpc_network_status_shutdown_all_endpoints(); - cq_verify(cqv); op = ops; op->op = GRPC_OP_RECV_CLOSE_ON_SERVER; @@ -205,7 +206,7 @@ static void test_invoke_network_status_change(grpc_end2end_test_config config) { op++; error = grpc_call_start_batch(s, ops, (size_t)(op - ops), tag(103), NULL); GPR_ASSERT(GRPC_CALL_OK == error); - void shutdown_all_endpoints(); + cq_expect_completion(cqv, tag(103), 1); cq_expect_completion(cqv, tag(1), 1); cq_verify(cqv); From 8db469baf6edcee935c58f42613609f7265020a4 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 6 Jul 2016 13:42:13 -0700 Subject: [PATCH 0416/1039] Remove the sea of green to allow focusing on failures --- tools/run_tests/run_tests.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index c1254275e6c..f61821557c4 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -1228,8 +1228,6 @@ def _build_and_run( jobset.message( 'FLAKE', '%s [%d/%d runs flaked]' % (k, num_failures, num_runs), do_newline=True) - else: - jobset.message('PASSED', k, do_newline=True) finally: for antagonist in antagonists: antagonist.kill() From b0f15e8af3622ebffa8414771fa6934568155179 Mon Sep 17 00:00:00 2001 From: vjpai Date: Wed, 6 Jul 2016 13:57:01 -0700 Subject: [PATCH 0417/1039] Reduce assertions, use status codes, increase verbosity on errors --- test/cpp/qps/client_async.cc | 1 - test/cpp/qps/client_sync.cc | 12 ++-- test/cpp/qps/driver.cc | 131 +++++++++++++++++++++++++---------- test/cpp/qps/qps_worker.cc | 35 +++++----- 4 files changed, 122 insertions(+), 57 deletions(-) diff --git a/test/cpp/qps/client_async.cc b/test/cpp/qps/client_async.cc index 1507d1e3d66..2f987fc80d4 100644 --- a/test/cpp/qps/client_async.cc +++ b/test/cpp/qps/client_async.cc @@ -31,7 +31,6 @@ * */ -#include #include #include #include diff --git a/test/cpp/qps/client_sync.cc b/test/cpp/qps/client_sync.cc index c88e95b80e5..686c8d750c4 100644 --- a/test/cpp/qps/client_sync.cc +++ b/test/cpp/qps/client_sync.cc @@ -31,7 +31,6 @@ * */ -#include #include #include #include @@ -128,11 +127,16 @@ class SynchronousStreamingClient GRPC_FINAL : public SynchronousClient { } ~SynchronousStreamingClient() { EndThreads(); - for (auto stream = &stream_[0]; stream != &stream_[num_threads_]; - stream++) { + for (size_t i = 0; i < num_threads_; i++) { + auto stream = &stream_[i]; if (*stream) { (*stream)->WritesDone(); - EXPECT_TRUE((*stream)->Finish().ok()); + Status s = (*stream)->Finish(); + EXPECT_TRUE(s.ok()); + if (!s.ok()) { + gpr_log(GPR_ERROR, "Stream %zu received an error %s", i, + s.error_message().c_str()); + } } } delete[] stream_; diff --git a/test/cpp/qps/driver.cc b/test/cpp/qps/driver.cc index 08bf0458832..ba38de76f31 100644 --- a/test/cpp/qps/driver.cc +++ b/test/cpp/qps/driver.cc @@ -87,7 +87,7 @@ static std::unordered_map> get_hosts_and_cores( CoreRequest dummy; CoreResponse cores; grpc::Status s = stub->CoreCount(&ctx, dummy, &cores); - assert(s.ok()); + GPR_ASSERT(s.ok()); std::deque dq; for (int i = 0; i < cores.cores(); i++) { dq.push_back(i); @@ -289,9 +289,13 @@ std::unique_ptr RunScenario( *args.mutable_setup() = server_config; servers[i].stream = servers[i].stub->RunServer(runsc::AllocContext(&contexts)); - GPR_ASSERT(servers[i].stream->Write(args)); + if (!servers[i].stream->Write(args)) { + gpr_log(GPR_ERROR, "Could not write args to server %zu", i); + } ServerStatus init_status; - GPR_ASSERT(servers[i].stream->Read(&init_status)); + if (!servers[i].stream->Read(&init_status)) { + gpr_log(GPR_ERROR, "Server %zu did not yield initial status", i); + } gpr_join_host_port(&cli_target, host, init_status.port()); client_config.add_server_targets(cli_target); gpr_free(host); @@ -344,10 +348,14 @@ std::unique_ptr RunScenario( ClientArgs args; *args.mutable_setup() = per_client_config; clients[i].stream = - clients[i].stub->RunClient(runsc::AllocContext(&contexts)); - GPR_ASSERT(clients[i].stream->Write(args)); + clients[i].stub->RunClient(runsc::AllocContext(&contexts)); + if (!clients[i].stream->Write(args)) { + gpr_log(GPR_ERROR, "Could not write args to client %zu", i); + } ClientStatus init_status; - GPR_ASSERT(clients[i].stream->Read(&init_status)); + if (!clients[i].stream->Read(&init_status)) { + gpr_log(GPR_ERROR, "Client %zu did not yield initial status", i); + } } // Let everything warmup @@ -362,19 +370,31 @@ std::unique_ptr RunScenario( server_mark.mutable_mark()->set_reset(true); ClientArgs client_mark; client_mark.mutable_mark()->set_reset(true); - for (auto server = &servers[0]; server != &servers[num_servers]; server++) { - GPR_ASSERT(server->stream->Write(server_mark)); + for (size_t i = 0; i < num_servers; i++) { + auto server = &servers[i]; + if (!server->stream->Write(server_mark)) { + gpr_log(GPR_ERROR, "Couldn't write mark to server %zu", i); + } } - for (auto client = &clients[0]; client != &clients[num_clients]; client++) { - GPR_ASSERT(client->stream->Write(client_mark)); + for (size_t i = 0; i < num_clients; i++) { + auto client = &clients[i]; + if (!client->stream->Write(client_mark)) { + gpr_log(GPR_ERROR, "Couldn't write mark to client %zu", i); + } } ServerStatus server_status; ClientStatus client_status; - for (auto server = &servers[0]; server != &servers[num_servers]; server++) { - GPR_ASSERT(server->stream->Read(&server_status)); + for (size_t i = 0; i < num_servers; i++) { + auto server = &servers[i]; + if (!server->stream->Read(&server_status)) { + gpr_log(GPR_ERROR, "Couldn't get status from server %zu", i); + } } - for (auto client = &clients[0]; client != &clients[num_clients]; client++) { - GPR_ASSERT(client->stream->Read(&client_status)); + for (size_t i = 0; i < num_clients; i++) { + auto client = &clients[i]; + if (!client->stream->Read(&client_status)) { + gpr_log(GPR_ERROR, "Couldn't get status from client %zu", i); + } } // Wait some time @@ -390,37 +410,71 @@ std::unique_ptr RunScenario( Histogram merged_latencies; gpr_log(GPR_INFO, "Finishing clients"); - for (auto client = &clients[0]; client != &clients[num_clients]; client++) { - GPR_ASSERT(client->stream->Write(client_mark)); - GPR_ASSERT(client->stream->WritesDone()); + for (size_t i = 0; i < num_clients; i++) { + auto client = &clients[i]; + if (!client->stream->Write(client_mark)) { + gpr_log(GPR_ERROR, "Couldn't write mark to client %zu", i); + } + if (!client->stream->WritesDone()) { + gpr_log(GPR_ERROR, "Failed WritesDone for client %zu", i); + } } - for (auto client = &clients[0]; client != &clients[num_clients]; client++) { - GPR_ASSERT(client->stream->Read(&client_status)); - const auto& stats = client_status.stats(); - merged_latencies.MergeProto(stats.latencies()); - result->add_client_stats()->CopyFrom(stats); - GPR_ASSERT(!client->stream->Read(&client_status)); + for (size_t i = 0; i < num_clients; i++) { + auto client = &clients[i]; + // Read the client final status + if (client->stream->Read(&client_status)) { + gpr_log(GPR_INFO, "Received final status from client %zu", i); + const auto& stats = client_status.stats(); + merged_latencies.MergeProto(stats.latencies()); + result->add_client_stats()->CopyFrom(stats); + // That final status should be the last message on the client stream + GPR_ASSERT(!client->stream->Read(&client_status)); + } else { + gpr_log(GPR_ERROR, "Couldn't get final status from client %zu", i); + } } - for (auto client = &clients[0]; client != &clients[num_clients]; client++) { - GPR_ASSERT(client->stream->Finish().ok()); + for (size_t i = 0; i < num_clients; i++) { + auto client = &clients[i]; + Status s = client->stream->Finish(); + if (!s.ok()) { + gpr_log(GPR_ERROR, "Client %zu had an error %s", i, + s.error_message().c_str()); + } } delete[] clients; merged_latencies.FillProto(result->mutable_latencies()); gpr_log(GPR_INFO, "Finishing servers"); - for (auto server = &servers[0]; server != &servers[num_servers]; server++) { - GPR_ASSERT(server->stream->Write(server_mark)); - GPR_ASSERT(server->stream->WritesDone()); + for (size_t i = 0; i < num_servers; i++) { + auto server = &servers[i]; + if (!server->stream->Write(server_mark)) { + gpr_log(GPR_ERROR, "Couldn't write mark to server %zu", i); + } + if (!server->stream->WritesDone()) { + gpr_log(GPR_ERROR, "Failed WritesDone for server %zu", i); + } } - for (auto server = &servers[0]; server != &servers[num_servers]; server++) { - GPR_ASSERT(server->stream->Read(&server_status)); - result->add_server_stats()->CopyFrom(server_status.stats()); - result->add_server_cores(server_status.cores()); - GPR_ASSERT(!server->stream->Read(&server_status)); + for (size_t i = 0; i < num_servers; i++) { + auto server = &servers[i]; + // Read the server final status + if (server->stream->Read(&server_status)) { + gpr_log(GPR_INFO, "Received final status from server %zu", i); + result->add_server_stats()->CopyFrom(server_status.stats()); + result->add_server_cores(server_status.cores()); + // That final status should be the last message on the server stream + GPR_ASSERT(!server->stream->Read(&server_status)); + } else { + gpr_log(GPR_ERROR, "Couldn't get final status from server %zu", i); + } } - for (auto server = &servers[0]; server != &servers[num_servers]; server++) { - GPR_ASSERT(server->stream->Finish().ok()); + for (size_t i = 0; i < num_servers; i++) { + auto server = &servers[i]; + Status s = server->stream->Finish(); + if (!s.ok()) { + gpr_log(GPR_ERROR, "Server %zu had an error %s", i, + s.error_message().c_str()); + } } delete[] servers; @@ -438,7 +492,12 @@ void RunQuit() { Void dummy; grpc::ClientContext ctx; ctx.set_fail_fast(false); - GPR_ASSERT(stub->QuitWorker(&ctx, dummy, &dummy).ok()); + Status s = stub->QuitWorker(&ctx, dummy, &dummy); + if (!s.ok()) { + gpr_log(GPR_ERROR, "Worker %zu could not be properly quit because %s", + i, s.error_message().c_str()); + GPR_ASSERT(false); + } } } diff --git a/test/cpp/qps/qps_worker.cc b/test/cpp/qps/qps_worker.cc index f514e23e854..8456fde0ed3 100644 --- a/test/cpp/qps/qps_worker.cc +++ b/test/cpp/qps/qps_worker.cc @@ -33,7 +33,6 @@ #include "test/cpp/qps/qps_worker.h" -#include #include #include #include @@ -124,7 +123,7 @@ class WorkerServiceImpl GRPC_FINAL : public WorkerService::Service { GRPC_OVERRIDE { InstanceGuard g(this); if (!g.Acquired()) { - return Status(StatusCode::RESOURCE_EXHAUSTED, ""); + return Status(StatusCode::RESOURCE_EXHAUSTED, "Client worker busy"); } ScopedProfile profile("qps_client.prof", false); @@ -137,7 +136,7 @@ class WorkerServiceImpl GRPC_FINAL : public WorkerService::Service { GRPC_OVERRIDE { InstanceGuard g(this); if (!g.Acquired()) { - return Status(StatusCode::RESOURCE_EXHAUSTED, ""); + return Status(StatusCode::RESOURCE_EXHAUSTED, "Server worker busy"); } ScopedProfile profile("qps_server.prof", false); @@ -154,7 +153,7 @@ class WorkerServiceImpl GRPC_FINAL : public WorkerService::Service { Status QuitWorker(ServerContext* ctx, const Void*, Void*) GRPC_OVERRIDE { InstanceGuard g(this); if (!g.Acquired()) { - return Status(StatusCode::RESOURCE_EXHAUSTED, ""); + return Status(StatusCode::RESOURCE_EXHAUSTED, "Quitting worker busy"); } worker_->MarkDone(); @@ -197,30 +196,32 @@ class WorkerServiceImpl GRPC_FINAL : public WorkerService::Service { ServerReaderWriter* stream) { ClientArgs args; if (!stream->Read(&args)) { - return Status(StatusCode::INVALID_ARGUMENT, ""); + return Status(StatusCode::INVALID_ARGUMENT, "Couldn't read args"); } if (!args.has_setup()) { - return Status(StatusCode::INVALID_ARGUMENT, ""); + return Status(StatusCode::INVALID_ARGUMENT, "Invalid setup arg"); } gpr_log(GPR_INFO, "RunClientBody: about to create client"); auto client = CreateClient(args.setup()); if (!client) { - return Status(StatusCode::INVALID_ARGUMENT, ""); + return Status(StatusCode::INVALID_ARGUMENT, "Couldn't create client"); } gpr_log(GPR_INFO, "RunClientBody: client created"); ClientStatus status; if (!stream->Write(status)) { - return Status(StatusCode::UNKNOWN, ""); + return Status(StatusCode::UNKNOWN, "Client couldn't report init status"); } gpr_log(GPR_INFO, "RunClientBody: creation status reported"); while (stream->Read(&args)) { gpr_log(GPR_INFO, "RunClientBody: Message read"); if (!args.has_mark()) { gpr_log(GPR_INFO, "RunClientBody: Message is not a mark!"); - return Status(StatusCode::INVALID_ARGUMENT, ""); + return Status(StatusCode::INVALID_ARGUMENT, "Invalid mark"); } *status.mutable_stats() = client->Mark(args.mark().reset()); - stream->Write(status); + if (!stream->Write(status)) { + return Status(StatusCode::UNKNOWN, "Client couldn't respond to mark"); + } gpr_log(GPR_INFO, "RunClientBody: Mark response given"); } @@ -232,10 +233,10 @@ class WorkerServiceImpl GRPC_FINAL : public WorkerService::Service { ServerReaderWriter* stream) { ServerArgs args; if (!stream->Read(&args)) { - return Status(StatusCode::INVALID_ARGUMENT, ""); + return Status(StatusCode::INVALID_ARGUMENT, "Couldn't read server args"); } if (!args.has_setup()) { - return Status(StatusCode::INVALID_ARGUMENT, ""); + return Status(StatusCode::INVALID_ARGUMENT, "Bad server creation args"); } if (server_port_ != 0) { args.mutable_setup()->set_port(server_port_); @@ -243,24 +244,26 @@ class WorkerServiceImpl GRPC_FINAL : public WorkerService::Service { gpr_log(GPR_INFO, "RunServerBody: about to create server"); auto server = CreateServer(args.setup()); if (!server) { - return Status(StatusCode::INVALID_ARGUMENT, ""); + return Status(StatusCode::INVALID_ARGUMENT, "Couldn't create server"); } gpr_log(GPR_INFO, "RunServerBody: server created"); ServerStatus status; status.set_port(server->port()); status.set_cores(server->cores()); if (!stream->Write(status)) { - return Status(StatusCode::UNKNOWN, ""); + return Status(StatusCode::UNKNOWN, "Server couldn't report init status"); } gpr_log(GPR_INFO, "RunServerBody: creation status reported"); while (stream->Read(&args)) { gpr_log(GPR_INFO, "RunServerBody: Message read"); if (!args.has_mark()) { gpr_log(GPR_INFO, "RunServerBody: Message not a mark!"); - return Status(StatusCode::INVALID_ARGUMENT, ""); + return Status(StatusCode::INVALID_ARGUMENT, "Invalid mark"); } *status.mutable_stats() = server->Mark(args.mark().reset()); - stream->Write(status); + if (!stream->Write(status)) { + return Status(StatusCode::UNKNOWN, "Server couldn't respond to mark"); + } gpr_log(GPR_INFO, "RunServerBody: Mark response given"); } From 571f3e55b4a304cb50b211dd3492ce1664d5bf0c Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 6 Jul 2016 14:00:24 -0700 Subject: [PATCH 0418/1039] Prevent a use-after-free --- src/core/ext/transport/chttp2/transport/chttp2_transport.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index 92722b50f4f..b58d93787d7 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -1178,6 +1178,7 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, true); } } else { + stream_global->send_initial_metadata = NULL; grpc_chttp2_complete_closure_step( exec_ctx, transport_global, stream_global, &stream_global->send_initial_metadata_finished, @@ -1233,6 +1234,7 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_list_add_check_read_ops(transport_global, stream_global); } if (stream_global->write_closed) { + stream_global->send_trailing_metadata = NULL; grpc_chttp2_complete_closure_step( exec_ctx, transport_global, stream_global, &stream_global->send_trailing_metadata_finished, From fed3e3b0705c2af5ba66f2c81366e05f07a9a937 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 6 Jul 2016 14:02:34 -0700 Subject: [PATCH 0419/1039] Added section about variable initialization --- doc/c-style-guide.md | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/doc/c-style-guide.md b/doc/c-style-guide.md index d6f9bbd7d4a..517f60a2197 100644 --- a/doc/c-style-guide.md +++ b/doc/c-style-guide.md @@ -9,16 +9,17 @@ Here we document style rules for C usage in the gRPC Core library. General ------- -- Layout rules are defined by clang-format, and all code should be passed through - clang-format. A (docker-based) script to do so is included in - [tools/distrib/clang\_format\_code.sh] (../tools/distrib/clang_format_code.sh). +- Layout rules are defined by clang-format, and all code should be passed + through clang-format. A (docker-based) script to do so is included in + [tools/distrib/clang\_format\_code.sh](../tools/distrib/clang_format_code.sh). Header Files ------------ -- Public header files (those in the include/grpc tree) should compile as pedantic C89 -- Public header files should be includable from C++ programs. That is, they should - include the following: +- Public header files (those in the include/grpc tree) should compile as + pedantic C89 +- Public header files should be includable from C++ programs. That is, they + should include the following: ```c #ifdef __cplusplus extern "C" { @@ -34,24 +35,34 @@ Header Files - All header files should have a #define guard to prevent multiple inclusion. To guarantee uniqueness they should be based on the file's path. - For public headers: include/grpc/grpc.h --> GRPC_GRPC_H + For public headers: `include/grpc/grpc.h` → `GRPC_GRPC_H` + + For private headers: + `src/core/channel/channel_stack.h` → + `GRPC_INTERNAL_CORE_CHANNEL_CHANNEL_STACK_H` + +Variable Initialization +----------------------- + +When declaring a (non-static) pointer variable, always initialize it to `NULL`. +Even in the case of static pointer variables, it's recommended to explicitly +initialize them to `NULL`. - For private headers: - src/core/channel/channel_stack.h --> GRPC_INTERNAL_CORE_CHANNEL_CHANNEL_STACK_H C99 Features ------------ - Variable sized arrays are not allowed - Do not use the 'inline' keyword -- Flexible array members are allowed (https://en.wikipedia.org/wiki/Flexible_array_member) +- Flexible array members are allowed + (https://en.wikipedia.org/wiki/Flexible_array_member) Comments -------- Within public header files, only `/* */` comments are allowed. -Within implementation files and private headers, either single line `//` +Within implementation files and private headers, either single line `//` or multi line `/* */` comments are allowed. Only one comment style per file is allowed however (i.e. if single line comments are used anywhere within a file, ALL comments within that file must be single line comments). From fc99ff6644e51297c4d20b686a62783161c8500b Mon Sep 17 00:00:00 2001 From: Muxi Yan Date: Wed, 6 Jul 2016 15:08:30 -0700 Subject: [PATCH 0420/1039] Fix error code when client run out of memory in Objective C code base Update Status Codes documentation --- doc/statuscodes.md | 1 + src/objective-c/GRPCClient/GRPCCall.m | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/statuscodes.md b/doc/statuscodes.md index c918f9ed9ab..1cd72df30ad 100644 --- a/doc/statuscodes.md +++ b/doc/statuscodes.md @@ -18,6 +18,7 @@ Only a subset of the pre-defined status codes are generated by the gRPC librarie | Could not decompress, but compression algorithm supported (Server -> Client) | INTERNAL | Client | | Compression mechanism used by client not supported at server | UNIMPLEMENTED | Server | | Server temporarily out of resources (e.g., Flow-control resource limits reached) | RESOURCE_EXHAUSTED | Server| +| Client does not have enough memory to hold the server response | RESOURCE_EXHAUSTED | Client | | Flow-control protocol violation | INTERNAL | Both | | Error parsing returned status | UNKNOWN | Client | | Incorrect Auth metadata ( Credentials failed to get metadata, Incompatible credentials set on channel and call, Invalid host set in `:authority` metadata, etc.) | UNAUTHENTICATED | Both | diff --git a/src/objective-c/GRPCClient/GRPCCall.m b/src/objective-c/GRPCClient/GRPCCall.m index e9678f38a9c..71121094afd 100644 --- a/src/objective-c/GRPCClient/GRPCCall.m +++ b/src/objective-c/GRPCClient/GRPCCall.m @@ -213,8 +213,8 @@ NSString * const kGRPCTrailersKey = @"io.grpc.TrailersKey"; // (because it's just a client problem). Use another domain and an // appropriately-documented code. [weakSelf finishWithError:[NSError errorWithDomain:kGRPCErrorDomain - code:GRPCErrorCodeInternal - userInfo:nil]]; + code:GRPCErrorCodeResourceExhausted + userInfo:@{NSLocalizedDescriptionKey: @"Client out of memory."}]]; [weakSelf cancelCall]; return; } From 7ac58464dfa07730c48a63d42290513316496e2b Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 6 Jul 2016 15:42:03 -0700 Subject: [PATCH 0421/1039] removed use of __func__ in test --- test/core/surface/byte_buffer_reader_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/core/surface/byte_buffer_reader_test.c b/test/core/surface/byte_buffer_reader_test.c index 4231d899473..1ab1a06211e 100644 --- a/test/core/surface/byte_buffer_reader_test.c +++ b/test/core/surface/byte_buffer_reader_test.c @@ -120,7 +120,7 @@ static void test_read_corrupted_slice(void) { grpc_byte_buffer *buffer; grpc_byte_buffer_reader reader; - LOG_TEST(__func__); + LOG_TEST("test_read_corrupted_slice"); slice = gpr_slice_from_copied_string("test"); buffer = grpc_raw_byte_buffer_create(&slice, 1); buffer->data.raw.compression = GRPC_COMPRESS_GZIP; /* lies! */ From 9ef0cd81f74eff88191cff541add28765b59ff46 Mon Sep 17 00:00:00 2001 From: yang-g Date: Wed, 6 Jul 2016 16:17:37 -0700 Subject: [PATCH 0422/1039] Initialize variable. --- test/core/surface/server_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/core/surface/server_test.c b/test/core/surface/server_test.c index 6dd8a435aa0..3fd1c2c2663 100644 --- a/test/core/surface/server_test.c +++ b/test/core/surface/server_test.c @@ -139,7 +139,7 @@ void test_bind_server_to_addr(const char *host, bool secure) { } static int external_dns_works(const char *host) { - grpc_resolved_addresses *res; + grpc_resolved_addresses *res = NULL; grpc_error *error = grpc_blocking_resolve_address(host, "80", &res); GRPC_ERROR_UNREF(error); if (res != NULL) { From f710ba0095147646503ae9ae3802375602412175 Mon Sep 17 00:00:00 2001 From: yang-g Date: Wed, 6 Jul 2016 16:17:37 -0700 Subject: [PATCH 0423/1039] Initialize variable. --- test/core/surface/server_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/core/surface/server_test.c b/test/core/surface/server_test.c index 6dd8a435aa0..3fd1c2c2663 100644 --- a/test/core/surface/server_test.c +++ b/test/core/surface/server_test.c @@ -139,7 +139,7 @@ void test_bind_server_to_addr(const char *host, bool secure) { } static int external_dns_works(const char *host) { - grpc_resolved_addresses *res; + grpc_resolved_addresses *res = NULL; grpc_error *error = grpc_blocking_resolve_address(host, "80", &res); GRPC_ERROR_UNREF(error); if (res != NULL) { From b15af069da766abd3764ee06dc9bb7227ab415e6 Mon Sep 17 00:00:00 2001 From: "Nicolas \"Pixel\" Noble" Date: Thu, 7 Jul 2016 01:13:53 +0200 Subject: [PATCH 0424/1039] Pinning to protobuf submodule 3.0.0-beta-3.3. --- third_party/protobuf | 2 +- tools/run_tests/sanity/check_submodules.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/third_party/protobuf b/third_party/protobuf index 3470b6895aa..bdeb215cab2 160000 --- a/third_party/protobuf +++ b/third_party/protobuf @@ -1 +1 @@ -Subproject commit 3470b6895aa659b7559ed678e029a5338e535f14 +Subproject commit bdeb215cab2985195325fcd5e70c3fa751f46e0f diff --git a/tools/run_tests/sanity/check_submodules.sh b/tools/run_tests/sanity/check_submodules.sh index f2d7a1429ea..b602d695649 100755 --- a/tools/run_tests/sanity/check_submodules.sh +++ b/tools/run_tests/sanity/check_submodules.sh @@ -45,7 +45,7 @@ cat << EOF | awk '{ print $1 }' | sort > $want_submodules 05b155ff59114735ec8cd089f669c4c3d8f59029 third_party/gflags (v2.1.0-45-g05b155f) c99458533a9b4c743ed51537e25989ea55944908 third_party/googletest (release-1.7.0) f8ac463766281625ad710900479130c7fcb4d63b third_party/nanopb (nanopb-0.3.4-29-gf8ac463) - d4d13a4349e4e59d67f311185ddcc1890d956d7a third_party/protobuf (v3.0.0-beta-3.2) + bdeb215cab2985195325fcd5e70c3fa751f46e0f third_party/protobuf (v3.0.0-beta-3.3) 50893291621658f355bc5b4d450a8d06a563053d third_party/zlib (v1.2.8) EOF From 4ebfe9056063dc27c8e2cfced886b9d0b05bd8e2 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 6 Jul 2016 16:24:45 -0700 Subject: [PATCH 0425/1039] Properly report large metadata errors --- .../chttp2/transport/chttp2_transport.c | 41 ++++++++++++++++--- .../ext/transport/chttp2/transport/internal.h | 6 ++- src/core/lib/iomgr/workqueue.h | 2 +- 3 files changed, 41 insertions(+), 8 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index b58d93787d7..8b7f91d3af8 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -603,7 +603,8 @@ static void destroy_stream_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_incoming_metadata_buffer_destroy( &s->global.received_trailing_metadata); gpr_slice_buffer_destroy(&s->writing.flow_controlled_buffer); - GRPC_ERROR_UNREF(s->global.removal_error); + GRPC_ERROR_UNREF(s->global.read_closed_error); + GRPC_ERROR_UNREF(s->global.write_closed_error); UNREF_TRANSPORT(exec_ctx, t, "stream"); @@ -1178,7 +1179,7 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, true); } } else { - stream_global->send_initial_metadata = NULL; + stream_global->send_trailing_metadata = NULL; grpc_chttp2_complete_closure_step( exec_ctx, transport_global, stream_global, &stream_global->send_initial_metadata_finished, @@ -1637,10 +1638,38 @@ void grpc_chttp2_fake_status(grpc_exec_ctx *exec_ctx, } } +static void add_error(grpc_error *error, grpc_error **refs, size_t *nrefs) { + if (error == GRPC_ERROR_NONE) return; + for (size_t i = 0; i < *nrefs; i++) { + if (error == refs[i]) { + return; + } + } + refs[*nrefs] = error; + ++*nrefs; +} + +static grpc_error *removal_error(grpc_error *extra_error, + grpc_chttp2_stream_global *stream_global) { + grpc_error *refs[3]; + size_t nrefs = 0; + add_error(stream_global->read_closed_error, refs, &nrefs); + add_error(stream_global->write_closed_error, refs, &nrefs); + add_error(extra_error, refs, &nrefs); + grpc_error *error = GRPC_ERROR_NONE; + if (nrefs > 0) { + error = GRPC_ERROR_CREATE_REFERENCING("Failed due to stream removal", refs, + nrefs); + } + GRPC_ERROR_UNREF(extra_error); + return error; +} + static void fail_pending_writes(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global, grpc_error *error) { + error = removal_error(error, stream_global); grpc_chttp2_complete_closure_step( exec_ctx, transport_global, stream_global, &stream_global->send_initial_metadata_finished, GRPC_ERROR_REF(error)); @@ -1663,12 +1692,14 @@ void grpc_chttp2_mark_stream_closed( } grpc_chttp2_list_add_check_read_ops(transport_global, stream_global); if (close_reads && !stream_global->read_closed) { + stream_global->read_closed_error = GRPC_ERROR_REF(error); stream_global->read_closed = true; stream_global->published_initial_metadata = true; stream_global->published_trailing_metadata = true; decrement_active_streams_locked(exec_ctx, transport_global, stream_global); } if (close_writes && !stream_global->write_closed) { + stream_global->write_closed_error = GRPC_ERROR_REF(error); stream_global->write_closed = true; if (TRANSPORT_FROM_GLOBAL(transport_global)->executor.write_state != GRPC_CHTTP2_WRITING_INACTIVE) { @@ -1681,7 +1712,6 @@ void grpc_chttp2_mark_stream_closed( } } if (stream_global->read_closed && stream_global->write_closed) { - stream_global->removal_error = GRPC_ERROR_REF(error); if (stream_global->id != 0 && TRANSPORT_FROM_GLOBAL(transport_global)->executor.parsing_active) { grpc_chttp2_list_add_closed_waiting_for_parsing(transport_global, @@ -1689,7 +1719,8 @@ void grpc_chttp2_mark_stream_closed( } else { if (stream_global->id != 0) { remove_stream(exec_ctx, TRANSPORT_FROM_GLOBAL(transport_global), - stream_global->id, GRPC_ERROR_REF(error)); + stream_global->id, + removal_error(GRPC_ERROR_REF(error), stream_global)); } GRPC_CHTTP2_STREAM_UNREF(exec_ctx, stream_global, "chttp2"); } @@ -2008,7 +2039,7 @@ static void post_parse_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, GPR_ASSERT(stream_global->write_closed); GPR_ASSERT(stream_global->read_closed); remove_stream(exec_ctx, t, stream_global->id, - GRPC_ERROR_REF(stream_global->removal_error)); + removal_error(GRPC_ERROR_NONE, stream_global)); GRPC_CHTTP2_STREAM_UNREF(exec_ctx, stream_global, "chttp2"); } diff --git a/src/core/ext/transport/chttp2/transport/internal.h b/src/core/ext/transport/chttp2/transport/internal.h index 4e0c31c1110..776017bee21 100644 --- a/src/core/ext/transport/chttp2/transport/internal.h +++ b/src/core/ext/transport/chttp2/transport/internal.h @@ -454,8 +454,10 @@ typedef struct { bool seen_error; bool exceeded_metadata_size; - /** the error that resulted in this stream being removed */ - grpc_error *removal_error; + /** the error that resulted in this stream being read-closed */ + grpc_error *read_closed_error; + /** the error that resulted in this stream being write-closed */ + grpc_error *write_closed_error; bool published_initial_metadata; bool published_trailing_metadata; diff --git a/src/core/lib/iomgr/workqueue.h b/src/core/lib/iomgr/workqueue.h index b4bd5742ddd..9f7219ebf16 100644 --- a/src/core/lib/iomgr/workqueue.h +++ b/src/core/lib/iomgr/workqueue.h @@ -52,7 +52,7 @@ void grpc_workqueue_flush(grpc_exec_ctx *exec_ctx, grpc_workqueue *workqueue); -/*#define GRPC_WORKQUEUE_REFCOUNT_DEBUG*/ +//#define GRPC_WORKQUEUE_REFCOUNT_DEBUG #ifdef GRPC_WORKQUEUE_REFCOUNT_DEBUG #define GRPC_WORKQUEUE_REF(p, r) \ (grpc_workqueue_ref((p), __FILE__, __LINE__, (r)), (p)) From 5c77320ce0108296223dbf087676196c205a5c18 Mon Sep 17 00:00:00 2001 From: David Garcia Quintas Date: Wed, 6 Jul 2016 16:36:45 -0700 Subject: [PATCH 0426/1039] Added missing dependencies to grpc++_base --- BUILD | 455 +++++++++++ CMakeLists.txt | 233 ++++++ Makefile | 286 ++++++- build.yaml | 2 + tools/doxygen/Doxyfile.c++ | 10 +- tools/doxygen/Doxyfile.c++.internal | 227 ++++++ tools/run_tests/sources_and_headers.json | 6 +- vsprojects/grpc.sln | 1 + vsprojects/vcxproj/grpc++/grpc++.vcxproj | 346 ++++++++ .../vcxproj/grpc++/grpc++.vcxproj.filters | 765 ++++++++++++++++++ .../grpc++_unsecure/grpc++_unsecure.vcxproj | 343 ++++++++ .../grpc++_unsecure.vcxproj.filters | 765 ++++++++++++++++++ 12 files changed, 3404 insertions(+), 35 deletions(-) diff --git a/BUILD b/BUILD index 66e582e6520..8c17065927c 100644 --- a/BUILD +++ b/BUILD @@ -1235,6 +1235,109 @@ cc_library( "src/cpp/client/create_channel_internal.h", "src/cpp/server/dynamic_thread_pool.h", "src/cpp/server/thread_pool_interface.h", + "src/core/lib/channel/channel_args.h", + "src/core/lib/channel/channel_stack.h", + "src/core/lib/channel/channel_stack_builder.h", + "src/core/lib/channel/compress_filter.h", + "src/core/lib/channel/connected_channel.h", + "src/core/lib/channel/context.h", + "src/core/lib/channel/http_client_filter.h", + "src/core/lib/channel/http_server_filter.h", + "src/core/lib/compression/algorithm_metadata.h", + "src/core/lib/compression/message_compress.h", + "src/core/lib/debug/trace.h", + "src/core/lib/http/format_request.h", + "src/core/lib/http/httpcli.h", + "src/core/lib/http/parser.h", + "src/core/lib/iomgr/closure.h", + "src/core/lib/iomgr/endpoint.h", + "src/core/lib/iomgr/endpoint_pair.h", + "src/core/lib/iomgr/error.h", + "src/core/lib/iomgr/ev_epoll_linux.h", + "src/core/lib/iomgr/ev_poll_and_epoll_posix.h", + "src/core/lib/iomgr/ev_poll_posix.h", + "src/core/lib/iomgr/ev_posix.h", + "src/core/lib/iomgr/exec_ctx.h", + "src/core/lib/iomgr/executor.h", + "src/core/lib/iomgr/iocp_windows.h", + "src/core/lib/iomgr/iomgr.h", + "src/core/lib/iomgr/iomgr_internal.h", + "src/core/lib/iomgr/iomgr_posix.h", + "src/core/lib/iomgr/load_file.h", + "src/core/lib/iomgr/network_status_tracker.h", + "src/core/lib/iomgr/polling_entity.h", + "src/core/lib/iomgr/pollset.h", + "src/core/lib/iomgr/pollset_set.h", + "src/core/lib/iomgr/pollset_set_windows.h", + "src/core/lib/iomgr/pollset_windows.h", + "src/core/lib/iomgr/resolve_address.h", + "src/core/lib/iomgr/sockaddr.h", + "src/core/lib/iomgr/sockaddr_posix.h", + "src/core/lib/iomgr/sockaddr_utils.h", + "src/core/lib/iomgr/sockaddr_windows.h", + "src/core/lib/iomgr/socket_utils_posix.h", + "src/core/lib/iomgr/socket_windows.h", + "src/core/lib/iomgr/tcp_client.h", + "src/core/lib/iomgr/tcp_posix.h", + "src/core/lib/iomgr/tcp_server.h", + "src/core/lib/iomgr/tcp_windows.h", + "src/core/lib/iomgr/time_averaged_stats.h", + "src/core/lib/iomgr/timer.h", + "src/core/lib/iomgr/timer_heap.h", + "src/core/lib/iomgr/udp_server.h", + "src/core/lib/iomgr/unix_sockets_posix.h", + "src/core/lib/iomgr/wakeup_fd_pipe.h", + "src/core/lib/iomgr/wakeup_fd_posix.h", + "src/core/lib/iomgr/workqueue.h", + "src/core/lib/iomgr/workqueue_posix.h", + "src/core/lib/iomgr/workqueue_windows.h", + "src/core/lib/json/json.h", + "src/core/lib/json/json_common.h", + "src/core/lib/json/json_reader.h", + "src/core/lib/json/json_writer.h", + "src/core/lib/surface/api_trace.h", + "src/core/lib/surface/call.h", + "src/core/lib/surface/call_test_only.h", + "src/core/lib/surface/channel.h", + "src/core/lib/surface/channel_init.h", + "src/core/lib/surface/channel_stack_type.h", + "src/core/lib/surface/completion_queue.h", + "src/core/lib/surface/event_string.h", + "src/core/lib/surface/init.h", + "src/core/lib/surface/lame_client.h", + "src/core/lib/surface/server.h", + "src/core/lib/transport/byte_stream.h", + "src/core/lib/transport/connectivity_state.h", + "src/core/lib/transport/metadata.h", + "src/core/lib/transport/metadata_batch.h", + "src/core/lib/transport/static_metadata.h", + "src/core/lib/transport/transport.h", + "src/core/lib/transport/transport_impl.h", + "src/core/lib/security/context/security_context.h", + "src/core/lib/security/credentials/composite/composite_credentials.h", + "src/core/lib/security/credentials/credentials.h", + "src/core/lib/security/credentials/fake/fake_credentials.h", + "src/core/lib/security/credentials/google_default/google_default_credentials.h", + "src/core/lib/security/credentials/iam/iam_credentials.h", + "src/core/lib/security/credentials/jwt/json_token.h", + "src/core/lib/security/credentials/jwt/jwt_credentials.h", + "src/core/lib/security/credentials/jwt/jwt_verifier.h", + "src/core/lib/security/credentials/oauth2/oauth2_credentials.h", + "src/core/lib/security/credentials/plugin/plugin_credentials.h", + "src/core/lib/security/credentials/ssl/ssl_credentials.h", + "src/core/lib/security/transport/auth_filters.h", + "src/core/lib/security/transport/handshake.h", + "src/core/lib/security/transport/secure_endpoint.h", + "src/core/lib/security/transport/security_connector.h", + "src/core/lib/security/transport/tsi_error.h", + "src/core/lib/security/util/b64.h", + "src/core/lib/security/util/json_util.h", + "src/core/ext/transport/chttp2/alpn/alpn.h", + "src/core/lib/tsi/fake_transport_security.h", + "src/core/lib/tsi/ssl_transport_security.h", + "src/core/lib/tsi/ssl_types.h", + "src/core/lib/tsi/transport_security.h", + "src/core/lib/tsi/transport_security_interface.h", "src/cpp/client/secure_credentials.cc", "src/cpp/common/auth_property_iterator.cc", "src/cpp/common/secure_auth_context.cc", @@ -1267,6 +1370,122 @@ cc_library( "src/cpp/util/status.cc", "src/cpp/util/string_ref.cc", "src/cpp/util/time.cc", + "src/core/lib/channel/channel_args.c", + "src/core/lib/channel/channel_stack.c", + "src/core/lib/channel/channel_stack_builder.c", + "src/core/lib/channel/compress_filter.c", + "src/core/lib/channel/connected_channel.c", + "src/core/lib/channel/http_client_filter.c", + "src/core/lib/channel/http_server_filter.c", + "src/core/lib/compression/compression.c", + "src/core/lib/compression/message_compress.c", + "src/core/lib/debug/trace.c", + "src/core/lib/http/format_request.c", + "src/core/lib/http/httpcli.c", + "src/core/lib/http/parser.c", + "src/core/lib/iomgr/closure.c", + "src/core/lib/iomgr/endpoint.c", + "src/core/lib/iomgr/endpoint_pair_posix.c", + "src/core/lib/iomgr/endpoint_pair_windows.c", + "src/core/lib/iomgr/error.c", + "src/core/lib/iomgr/ev_epoll_linux.c", + "src/core/lib/iomgr/ev_poll_and_epoll_posix.c", + "src/core/lib/iomgr/ev_poll_posix.c", + "src/core/lib/iomgr/ev_posix.c", + "src/core/lib/iomgr/exec_ctx.c", + "src/core/lib/iomgr/executor.c", + "src/core/lib/iomgr/iocp_windows.c", + "src/core/lib/iomgr/iomgr.c", + "src/core/lib/iomgr/iomgr_posix.c", + "src/core/lib/iomgr/iomgr_windows.c", + "src/core/lib/iomgr/load_file.c", + "src/core/lib/iomgr/network_status_tracker.c", + "src/core/lib/iomgr/polling_entity.c", + "src/core/lib/iomgr/pollset_set_windows.c", + "src/core/lib/iomgr/pollset_windows.c", + "src/core/lib/iomgr/resolve_address_posix.c", + "src/core/lib/iomgr/resolve_address_windows.c", + "src/core/lib/iomgr/sockaddr_utils.c", + "src/core/lib/iomgr/socket_utils_common_posix.c", + "src/core/lib/iomgr/socket_utils_linux.c", + "src/core/lib/iomgr/socket_utils_posix.c", + "src/core/lib/iomgr/socket_windows.c", + "src/core/lib/iomgr/tcp_client_posix.c", + "src/core/lib/iomgr/tcp_client_windows.c", + "src/core/lib/iomgr/tcp_posix.c", + "src/core/lib/iomgr/tcp_server_posix.c", + "src/core/lib/iomgr/tcp_server_windows.c", + "src/core/lib/iomgr/tcp_windows.c", + "src/core/lib/iomgr/time_averaged_stats.c", + "src/core/lib/iomgr/timer.c", + "src/core/lib/iomgr/timer_heap.c", + "src/core/lib/iomgr/udp_server.c", + "src/core/lib/iomgr/unix_sockets_posix.c", + "src/core/lib/iomgr/unix_sockets_posix_noop.c", + "src/core/lib/iomgr/wakeup_fd_eventfd.c", + "src/core/lib/iomgr/wakeup_fd_nospecial.c", + "src/core/lib/iomgr/wakeup_fd_pipe.c", + "src/core/lib/iomgr/wakeup_fd_posix.c", + "src/core/lib/iomgr/workqueue_posix.c", + "src/core/lib/iomgr/workqueue_windows.c", + "src/core/lib/json/json.c", + "src/core/lib/json/json_reader.c", + "src/core/lib/json/json_string.c", + "src/core/lib/json/json_writer.c", + "src/core/lib/surface/alarm.c", + "src/core/lib/surface/api_trace.c", + "src/core/lib/surface/byte_buffer.c", + "src/core/lib/surface/byte_buffer_reader.c", + "src/core/lib/surface/call.c", + "src/core/lib/surface/call_details.c", + "src/core/lib/surface/call_log_batch.c", + "src/core/lib/surface/channel.c", + "src/core/lib/surface/channel_init.c", + "src/core/lib/surface/channel_ping.c", + "src/core/lib/surface/channel_stack_type.c", + "src/core/lib/surface/completion_queue.c", + "src/core/lib/surface/event_string.c", + "src/core/lib/surface/lame_client.c", + "src/core/lib/surface/metadata_array.c", + "src/core/lib/surface/server.c", + "src/core/lib/surface/validate_metadata.c", + "src/core/lib/surface/version.c", + "src/core/lib/transport/byte_stream.c", + "src/core/lib/transport/connectivity_state.c", + "src/core/lib/transport/metadata.c", + "src/core/lib/transport/metadata_batch.c", + "src/core/lib/transport/static_metadata.c", + "src/core/lib/transport/transport.c", + "src/core/lib/transport/transport_op_string.c", + "src/core/lib/http/httpcli_security_connector.c", + "src/core/lib/security/context/security_context.c", + "src/core/lib/security/credentials/composite/composite_credentials.c", + "src/core/lib/security/credentials/credentials.c", + "src/core/lib/security/credentials/credentials_metadata.c", + "src/core/lib/security/credentials/fake/fake_credentials.c", + "src/core/lib/security/credentials/google_default/credentials_posix.c", + "src/core/lib/security/credentials/google_default/credentials_windows.c", + "src/core/lib/security/credentials/google_default/google_default_credentials.c", + "src/core/lib/security/credentials/iam/iam_credentials.c", + "src/core/lib/security/credentials/jwt/json_token.c", + "src/core/lib/security/credentials/jwt/jwt_credentials.c", + "src/core/lib/security/credentials/jwt/jwt_verifier.c", + "src/core/lib/security/credentials/oauth2/oauth2_credentials.c", + "src/core/lib/security/credentials/plugin/plugin_credentials.c", + "src/core/lib/security/credentials/ssl/ssl_credentials.c", + "src/core/lib/security/transport/client_auth_filter.c", + "src/core/lib/security/transport/handshake.c", + "src/core/lib/security/transport/secure_endpoint.c", + "src/core/lib/security/transport/security_connector.c", + "src/core/lib/security/transport/server_auth_filter.c", + "src/core/lib/security/transport/tsi_error.c", + "src/core/lib/security/util/b64.c", + "src/core/lib/security/util/json_util.c", + "src/core/lib/surface/init_secure.c", + "src/core/ext/transport/chttp2/alpn/alpn.c", + "src/core/lib/tsi/fake_transport_security.c", + "src/core/lib/tsi/ssl_transport_security.c", + "src/core/lib/tsi/transport_security.c", "src/cpp/codegen/codegen_init.cc", ], hdrs = [ @@ -1368,6 +1587,14 @@ cc_library( "include/grpc/impl/codegen/sync_posix.h", "include/grpc/impl/codegen/sync_windows.h", "include/grpc/impl/codegen/time.h", + "include/grpc/byte_buffer.h", + "include/grpc/byte_buffer_reader.h", + "include/grpc/compression.h", + "include/grpc/grpc.h", + "include/grpc/grpc_posix.h", + "include/grpc/status.h", + "include/grpc/grpc_security.h", + "include/grpc/grpc_security_constants.h", ], includes = [ "include", @@ -1377,6 +1604,7 @@ cc_library( "//external:libssl", "//external:protobuf_clib", ":grpc", + ":gpr", ], ) @@ -1466,6 +1694,109 @@ cc_library( "src/cpp/client/create_channel_internal.h", "src/cpp/server/dynamic_thread_pool.h", "src/cpp/server/thread_pool_interface.h", + "src/core/lib/channel/channel_args.h", + "src/core/lib/channel/channel_stack.h", + "src/core/lib/channel/channel_stack_builder.h", + "src/core/lib/channel/compress_filter.h", + "src/core/lib/channel/connected_channel.h", + "src/core/lib/channel/context.h", + "src/core/lib/channel/http_client_filter.h", + "src/core/lib/channel/http_server_filter.h", + "src/core/lib/compression/algorithm_metadata.h", + "src/core/lib/compression/message_compress.h", + "src/core/lib/debug/trace.h", + "src/core/lib/http/format_request.h", + "src/core/lib/http/httpcli.h", + "src/core/lib/http/parser.h", + "src/core/lib/iomgr/closure.h", + "src/core/lib/iomgr/endpoint.h", + "src/core/lib/iomgr/endpoint_pair.h", + "src/core/lib/iomgr/error.h", + "src/core/lib/iomgr/ev_epoll_linux.h", + "src/core/lib/iomgr/ev_poll_and_epoll_posix.h", + "src/core/lib/iomgr/ev_poll_posix.h", + "src/core/lib/iomgr/ev_posix.h", + "src/core/lib/iomgr/exec_ctx.h", + "src/core/lib/iomgr/executor.h", + "src/core/lib/iomgr/iocp_windows.h", + "src/core/lib/iomgr/iomgr.h", + "src/core/lib/iomgr/iomgr_internal.h", + "src/core/lib/iomgr/iomgr_posix.h", + "src/core/lib/iomgr/load_file.h", + "src/core/lib/iomgr/network_status_tracker.h", + "src/core/lib/iomgr/polling_entity.h", + "src/core/lib/iomgr/pollset.h", + "src/core/lib/iomgr/pollset_set.h", + "src/core/lib/iomgr/pollset_set_windows.h", + "src/core/lib/iomgr/pollset_windows.h", + "src/core/lib/iomgr/resolve_address.h", + "src/core/lib/iomgr/sockaddr.h", + "src/core/lib/iomgr/sockaddr_posix.h", + "src/core/lib/iomgr/sockaddr_utils.h", + "src/core/lib/iomgr/sockaddr_windows.h", + "src/core/lib/iomgr/socket_utils_posix.h", + "src/core/lib/iomgr/socket_windows.h", + "src/core/lib/iomgr/tcp_client.h", + "src/core/lib/iomgr/tcp_posix.h", + "src/core/lib/iomgr/tcp_server.h", + "src/core/lib/iomgr/tcp_windows.h", + "src/core/lib/iomgr/time_averaged_stats.h", + "src/core/lib/iomgr/timer.h", + "src/core/lib/iomgr/timer_heap.h", + "src/core/lib/iomgr/udp_server.h", + "src/core/lib/iomgr/unix_sockets_posix.h", + "src/core/lib/iomgr/wakeup_fd_pipe.h", + "src/core/lib/iomgr/wakeup_fd_posix.h", + "src/core/lib/iomgr/workqueue.h", + "src/core/lib/iomgr/workqueue_posix.h", + "src/core/lib/iomgr/workqueue_windows.h", + "src/core/lib/json/json.h", + "src/core/lib/json/json_common.h", + "src/core/lib/json/json_reader.h", + "src/core/lib/json/json_writer.h", + "src/core/lib/surface/api_trace.h", + "src/core/lib/surface/call.h", + "src/core/lib/surface/call_test_only.h", + "src/core/lib/surface/channel.h", + "src/core/lib/surface/channel_init.h", + "src/core/lib/surface/channel_stack_type.h", + "src/core/lib/surface/completion_queue.h", + "src/core/lib/surface/event_string.h", + "src/core/lib/surface/init.h", + "src/core/lib/surface/lame_client.h", + "src/core/lib/surface/server.h", + "src/core/lib/transport/byte_stream.h", + "src/core/lib/transport/connectivity_state.h", + "src/core/lib/transport/metadata.h", + "src/core/lib/transport/metadata_batch.h", + "src/core/lib/transport/static_metadata.h", + "src/core/lib/transport/transport.h", + "src/core/lib/transport/transport_impl.h", + "src/core/lib/security/context/security_context.h", + "src/core/lib/security/credentials/composite/composite_credentials.h", + "src/core/lib/security/credentials/credentials.h", + "src/core/lib/security/credentials/fake/fake_credentials.h", + "src/core/lib/security/credentials/google_default/google_default_credentials.h", + "src/core/lib/security/credentials/iam/iam_credentials.h", + "src/core/lib/security/credentials/jwt/json_token.h", + "src/core/lib/security/credentials/jwt/jwt_credentials.h", + "src/core/lib/security/credentials/jwt/jwt_verifier.h", + "src/core/lib/security/credentials/oauth2/oauth2_credentials.h", + "src/core/lib/security/credentials/plugin/plugin_credentials.h", + "src/core/lib/security/credentials/ssl/ssl_credentials.h", + "src/core/lib/security/transport/auth_filters.h", + "src/core/lib/security/transport/handshake.h", + "src/core/lib/security/transport/secure_endpoint.h", + "src/core/lib/security/transport/security_connector.h", + "src/core/lib/security/transport/tsi_error.h", + "src/core/lib/security/util/b64.h", + "src/core/lib/security/util/json_util.h", + "src/core/ext/transport/chttp2/alpn/alpn.h", + "src/core/lib/tsi/fake_transport_security.h", + "src/core/lib/tsi/ssl_transport_security.h", + "src/core/lib/tsi/ssl_types.h", + "src/core/lib/tsi/transport_security.h", + "src/core/lib/tsi/transport_security_interface.h", "src/cpp/common/insecure_create_auth_context.cc", "src/cpp/client/channel.cc", "src/cpp/client/client_context.cc", @@ -1493,6 +1824,122 @@ cc_library( "src/cpp/util/status.cc", "src/cpp/util/string_ref.cc", "src/cpp/util/time.cc", + "src/core/lib/channel/channel_args.c", + "src/core/lib/channel/channel_stack.c", + "src/core/lib/channel/channel_stack_builder.c", + "src/core/lib/channel/compress_filter.c", + "src/core/lib/channel/connected_channel.c", + "src/core/lib/channel/http_client_filter.c", + "src/core/lib/channel/http_server_filter.c", + "src/core/lib/compression/compression.c", + "src/core/lib/compression/message_compress.c", + "src/core/lib/debug/trace.c", + "src/core/lib/http/format_request.c", + "src/core/lib/http/httpcli.c", + "src/core/lib/http/parser.c", + "src/core/lib/iomgr/closure.c", + "src/core/lib/iomgr/endpoint.c", + "src/core/lib/iomgr/endpoint_pair_posix.c", + "src/core/lib/iomgr/endpoint_pair_windows.c", + "src/core/lib/iomgr/error.c", + "src/core/lib/iomgr/ev_epoll_linux.c", + "src/core/lib/iomgr/ev_poll_and_epoll_posix.c", + "src/core/lib/iomgr/ev_poll_posix.c", + "src/core/lib/iomgr/ev_posix.c", + "src/core/lib/iomgr/exec_ctx.c", + "src/core/lib/iomgr/executor.c", + "src/core/lib/iomgr/iocp_windows.c", + "src/core/lib/iomgr/iomgr.c", + "src/core/lib/iomgr/iomgr_posix.c", + "src/core/lib/iomgr/iomgr_windows.c", + "src/core/lib/iomgr/load_file.c", + "src/core/lib/iomgr/network_status_tracker.c", + "src/core/lib/iomgr/polling_entity.c", + "src/core/lib/iomgr/pollset_set_windows.c", + "src/core/lib/iomgr/pollset_windows.c", + "src/core/lib/iomgr/resolve_address_posix.c", + "src/core/lib/iomgr/resolve_address_windows.c", + "src/core/lib/iomgr/sockaddr_utils.c", + "src/core/lib/iomgr/socket_utils_common_posix.c", + "src/core/lib/iomgr/socket_utils_linux.c", + "src/core/lib/iomgr/socket_utils_posix.c", + "src/core/lib/iomgr/socket_windows.c", + "src/core/lib/iomgr/tcp_client_posix.c", + "src/core/lib/iomgr/tcp_client_windows.c", + "src/core/lib/iomgr/tcp_posix.c", + "src/core/lib/iomgr/tcp_server_posix.c", + "src/core/lib/iomgr/tcp_server_windows.c", + "src/core/lib/iomgr/tcp_windows.c", + "src/core/lib/iomgr/time_averaged_stats.c", + "src/core/lib/iomgr/timer.c", + "src/core/lib/iomgr/timer_heap.c", + "src/core/lib/iomgr/udp_server.c", + "src/core/lib/iomgr/unix_sockets_posix.c", + "src/core/lib/iomgr/unix_sockets_posix_noop.c", + "src/core/lib/iomgr/wakeup_fd_eventfd.c", + "src/core/lib/iomgr/wakeup_fd_nospecial.c", + "src/core/lib/iomgr/wakeup_fd_pipe.c", + "src/core/lib/iomgr/wakeup_fd_posix.c", + "src/core/lib/iomgr/workqueue_posix.c", + "src/core/lib/iomgr/workqueue_windows.c", + "src/core/lib/json/json.c", + "src/core/lib/json/json_reader.c", + "src/core/lib/json/json_string.c", + "src/core/lib/json/json_writer.c", + "src/core/lib/surface/alarm.c", + "src/core/lib/surface/api_trace.c", + "src/core/lib/surface/byte_buffer.c", + "src/core/lib/surface/byte_buffer_reader.c", + "src/core/lib/surface/call.c", + "src/core/lib/surface/call_details.c", + "src/core/lib/surface/call_log_batch.c", + "src/core/lib/surface/channel.c", + "src/core/lib/surface/channel_init.c", + "src/core/lib/surface/channel_ping.c", + "src/core/lib/surface/channel_stack_type.c", + "src/core/lib/surface/completion_queue.c", + "src/core/lib/surface/event_string.c", + "src/core/lib/surface/lame_client.c", + "src/core/lib/surface/metadata_array.c", + "src/core/lib/surface/server.c", + "src/core/lib/surface/validate_metadata.c", + "src/core/lib/surface/version.c", + "src/core/lib/transport/byte_stream.c", + "src/core/lib/transport/connectivity_state.c", + "src/core/lib/transport/metadata.c", + "src/core/lib/transport/metadata_batch.c", + "src/core/lib/transport/static_metadata.c", + "src/core/lib/transport/transport.c", + "src/core/lib/transport/transport_op_string.c", + "src/core/lib/http/httpcli_security_connector.c", + "src/core/lib/security/context/security_context.c", + "src/core/lib/security/credentials/composite/composite_credentials.c", + "src/core/lib/security/credentials/credentials.c", + "src/core/lib/security/credentials/credentials_metadata.c", + "src/core/lib/security/credentials/fake/fake_credentials.c", + "src/core/lib/security/credentials/google_default/credentials_posix.c", + "src/core/lib/security/credentials/google_default/credentials_windows.c", + "src/core/lib/security/credentials/google_default/google_default_credentials.c", + "src/core/lib/security/credentials/iam/iam_credentials.c", + "src/core/lib/security/credentials/jwt/json_token.c", + "src/core/lib/security/credentials/jwt/jwt_credentials.c", + "src/core/lib/security/credentials/jwt/jwt_verifier.c", + "src/core/lib/security/credentials/oauth2/oauth2_credentials.c", + "src/core/lib/security/credentials/plugin/plugin_credentials.c", + "src/core/lib/security/credentials/ssl/ssl_credentials.c", + "src/core/lib/security/transport/client_auth_filter.c", + "src/core/lib/security/transport/handshake.c", + "src/core/lib/security/transport/secure_endpoint.c", + "src/core/lib/security/transport/security_connector.c", + "src/core/lib/security/transport/server_auth_filter.c", + "src/core/lib/security/transport/tsi_error.c", + "src/core/lib/security/util/b64.c", + "src/core/lib/security/util/json_util.c", + "src/core/lib/surface/init_secure.c", + "src/core/ext/transport/chttp2/alpn/alpn.c", + "src/core/lib/tsi/fake_transport_security.c", + "src/core/lib/tsi/ssl_transport_security.c", + "src/core/lib/tsi/transport_security.c", "src/cpp/codegen/codegen_init.cc", ], hdrs = [ @@ -1594,6 +2041,14 @@ cc_library( "include/grpc/impl/codegen/sync_posix.h", "include/grpc/impl/codegen/sync_windows.h", "include/grpc/impl/codegen/time.h", + "include/grpc/byte_buffer.h", + "include/grpc/byte_buffer_reader.h", + "include/grpc/compression.h", + "include/grpc/grpc.h", + "include/grpc/grpc_posix.h", + "include/grpc/status.h", + "include/grpc/grpc_security.h", + "include/grpc/grpc_security_constants.h", ], includes = [ "include", diff --git a/CMakeLists.txt b/CMakeLists.txt index 9caf03191fa..3dcd1eb23d1 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -718,6 +718,122 @@ add_library(grpc++ src/cpp/util/status.cc src/cpp/util/string_ref.cc src/cpp/util/time.cc + src/core/lib/channel/channel_args.c + src/core/lib/channel/channel_stack.c + src/core/lib/channel/channel_stack_builder.c + src/core/lib/channel/compress_filter.c + src/core/lib/channel/connected_channel.c + src/core/lib/channel/http_client_filter.c + src/core/lib/channel/http_server_filter.c + src/core/lib/compression/compression.c + src/core/lib/compression/message_compress.c + src/core/lib/debug/trace.c + src/core/lib/http/format_request.c + src/core/lib/http/httpcli.c + src/core/lib/http/parser.c + src/core/lib/iomgr/closure.c + src/core/lib/iomgr/endpoint.c + src/core/lib/iomgr/endpoint_pair_posix.c + src/core/lib/iomgr/endpoint_pair_windows.c + src/core/lib/iomgr/error.c + src/core/lib/iomgr/ev_epoll_linux.c + src/core/lib/iomgr/ev_poll_and_epoll_posix.c + src/core/lib/iomgr/ev_poll_posix.c + src/core/lib/iomgr/ev_posix.c + src/core/lib/iomgr/exec_ctx.c + src/core/lib/iomgr/executor.c + src/core/lib/iomgr/iocp_windows.c + src/core/lib/iomgr/iomgr.c + src/core/lib/iomgr/iomgr_posix.c + src/core/lib/iomgr/iomgr_windows.c + src/core/lib/iomgr/load_file.c + src/core/lib/iomgr/network_status_tracker.c + src/core/lib/iomgr/polling_entity.c + src/core/lib/iomgr/pollset_set_windows.c + src/core/lib/iomgr/pollset_windows.c + src/core/lib/iomgr/resolve_address_posix.c + src/core/lib/iomgr/resolve_address_windows.c + src/core/lib/iomgr/sockaddr_utils.c + src/core/lib/iomgr/socket_utils_common_posix.c + src/core/lib/iomgr/socket_utils_linux.c + src/core/lib/iomgr/socket_utils_posix.c + src/core/lib/iomgr/socket_windows.c + src/core/lib/iomgr/tcp_client_posix.c + src/core/lib/iomgr/tcp_client_windows.c + src/core/lib/iomgr/tcp_posix.c + src/core/lib/iomgr/tcp_server_posix.c + src/core/lib/iomgr/tcp_server_windows.c + src/core/lib/iomgr/tcp_windows.c + src/core/lib/iomgr/time_averaged_stats.c + src/core/lib/iomgr/timer.c + src/core/lib/iomgr/timer_heap.c + src/core/lib/iomgr/udp_server.c + src/core/lib/iomgr/unix_sockets_posix.c + src/core/lib/iomgr/unix_sockets_posix_noop.c + src/core/lib/iomgr/wakeup_fd_eventfd.c + src/core/lib/iomgr/wakeup_fd_nospecial.c + src/core/lib/iomgr/wakeup_fd_pipe.c + src/core/lib/iomgr/wakeup_fd_posix.c + src/core/lib/iomgr/workqueue_posix.c + src/core/lib/iomgr/workqueue_windows.c + src/core/lib/json/json.c + src/core/lib/json/json_reader.c + src/core/lib/json/json_string.c + src/core/lib/json/json_writer.c + src/core/lib/surface/alarm.c + src/core/lib/surface/api_trace.c + src/core/lib/surface/byte_buffer.c + src/core/lib/surface/byte_buffer_reader.c + src/core/lib/surface/call.c + src/core/lib/surface/call_details.c + src/core/lib/surface/call_log_batch.c + src/core/lib/surface/channel.c + src/core/lib/surface/channel_init.c + src/core/lib/surface/channel_ping.c + src/core/lib/surface/channel_stack_type.c + src/core/lib/surface/completion_queue.c + src/core/lib/surface/event_string.c + src/core/lib/surface/lame_client.c + src/core/lib/surface/metadata_array.c + src/core/lib/surface/server.c + src/core/lib/surface/validate_metadata.c + src/core/lib/surface/version.c + src/core/lib/transport/byte_stream.c + src/core/lib/transport/connectivity_state.c + src/core/lib/transport/metadata.c + src/core/lib/transport/metadata_batch.c + src/core/lib/transport/static_metadata.c + src/core/lib/transport/transport.c + src/core/lib/transport/transport_op_string.c + src/core/lib/http/httpcli_security_connector.c + src/core/lib/security/context/security_context.c + src/core/lib/security/credentials/composite/composite_credentials.c + src/core/lib/security/credentials/credentials.c + src/core/lib/security/credentials/credentials_metadata.c + src/core/lib/security/credentials/fake/fake_credentials.c + src/core/lib/security/credentials/google_default/credentials_posix.c + src/core/lib/security/credentials/google_default/credentials_windows.c + src/core/lib/security/credentials/google_default/google_default_credentials.c + src/core/lib/security/credentials/iam/iam_credentials.c + src/core/lib/security/credentials/jwt/json_token.c + src/core/lib/security/credentials/jwt/jwt_credentials.c + src/core/lib/security/credentials/jwt/jwt_verifier.c + src/core/lib/security/credentials/oauth2/oauth2_credentials.c + src/core/lib/security/credentials/plugin/plugin_credentials.c + src/core/lib/security/credentials/ssl/ssl_credentials.c + src/core/lib/security/transport/client_auth_filter.c + src/core/lib/security/transport/handshake.c + src/core/lib/security/transport/secure_endpoint.c + src/core/lib/security/transport/security_connector.c + src/core/lib/security/transport/server_auth_filter.c + src/core/lib/security/transport/tsi_error.c + src/core/lib/security/util/b64.c + src/core/lib/security/util/json_util.c + src/core/lib/surface/init_secure.c + src/core/ext/transport/chttp2/alpn/alpn.c + src/core/lib/tsi/fake_transport_security.c + src/core/lib/tsi/ssl_transport_security.c + src/core/lib/tsi/transport_security.c src/cpp/codegen/codegen_init.cc ) @@ -734,6 +850,7 @@ target_link_libraries(grpc++ ssl libprotobuf grpc + gpr ) @@ -786,6 +903,122 @@ add_library(grpc++_unsecure src/cpp/util/status.cc src/cpp/util/string_ref.cc src/cpp/util/time.cc + src/core/lib/channel/channel_args.c + src/core/lib/channel/channel_stack.c + src/core/lib/channel/channel_stack_builder.c + src/core/lib/channel/compress_filter.c + src/core/lib/channel/connected_channel.c + src/core/lib/channel/http_client_filter.c + src/core/lib/channel/http_server_filter.c + src/core/lib/compression/compression.c + src/core/lib/compression/message_compress.c + src/core/lib/debug/trace.c + src/core/lib/http/format_request.c + src/core/lib/http/httpcli.c + src/core/lib/http/parser.c + src/core/lib/iomgr/closure.c + src/core/lib/iomgr/endpoint.c + src/core/lib/iomgr/endpoint_pair_posix.c + src/core/lib/iomgr/endpoint_pair_windows.c + src/core/lib/iomgr/error.c + src/core/lib/iomgr/ev_epoll_linux.c + src/core/lib/iomgr/ev_poll_and_epoll_posix.c + src/core/lib/iomgr/ev_poll_posix.c + src/core/lib/iomgr/ev_posix.c + src/core/lib/iomgr/exec_ctx.c + src/core/lib/iomgr/executor.c + src/core/lib/iomgr/iocp_windows.c + src/core/lib/iomgr/iomgr.c + src/core/lib/iomgr/iomgr_posix.c + src/core/lib/iomgr/iomgr_windows.c + src/core/lib/iomgr/load_file.c + src/core/lib/iomgr/network_status_tracker.c + src/core/lib/iomgr/polling_entity.c + src/core/lib/iomgr/pollset_set_windows.c + src/core/lib/iomgr/pollset_windows.c + src/core/lib/iomgr/resolve_address_posix.c + src/core/lib/iomgr/resolve_address_windows.c + src/core/lib/iomgr/sockaddr_utils.c + src/core/lib/iomgr/socket_utils_common_posix.c + src/core/lib/iomgr/socket_utils_linux.c + src/core/lib/iomgr/socket_utils_posix.c + src/core/lib/iomgr/socket_windows.c + src/core/lib/iomgr/tcp_client_posix.c + src/core/lib/iomgr/tcp_client_windows.c + src/core/lib/iomgr/tcp_posix.c + src/core/lib/iomgr/tcp_server_posix.c + src/core/lib/iomgr/tcp_server_windows.c + src/core/lib/iomgr/tcp_windows.c + src/core/lib/iomgr/time_averaged_stats.c + src/core/lib/iomgr/timer.c + src/core/lib/iomgr/timer_heap.c + src/core/lib/iomgr/udp_server.c + src/core/lib/iomgr/unix_sockets_posix.c + src/core/lib/iomgr/unix_sockets_posix_noop.c + src/core/lib/iomgr/wakeup_fd_eventfd.c + src/core/lib/iomgr/wakeup_fd_nospecial.c + src/core/lib/iomgr/wakeup_fd_pipe.c + src/core/lib/iomgr/wakeup_fd_posix.c + src/core/lib/iomgr/workqueue_posix.c + src/core/lib/iomgr/workqueue_windows.c + src/core/lib/json/json.c + src/core/lib/json/json_reader.c + src/core/lib/json/json_string.c + src/core/lib/json/json_writer.c + src/core/lib/surface/alarm.c + src/core/lib/surface/api_trace.c + src/core/lib/surface/byte_buffer.c + src/core/lib/surface/byte_buffer_reader.c + src/core/lib/surface/call.c + src/core/lib/surface/call_details.c + src/core/lib/surface/call_log_batch.c + src/core/lib/surface/channel.c + src/core/lib/surface/channel_init.c + src/core/lib/surface/channel_ping.c + src/core/lib/surface/channel_stack_type.c + src/core/lib/surface/completion_queue.c + src/core/lib/surface/event_string.c + src/core/lib/surface/lame_client.c + src/core/lib/surface/metadata_array.c + src/core/lib/surface/server.c + src/core/lib/surface/validate_metadata.c + src/core/lib/surface/version.c + src/core/lib/transport/byte_stream.c + src/core/lib/transport/connectivity_state.c + src/core/lib/transport/metadata.c + src/core/lib/transport/metadata_batch.c + src/core/lib/transport/static_metadata.c + src/core/lib/transport/transport.c + src/core/lib/transport/transport_op_string.c + src/core/lib/http/httpcli_security_connector.c + src/core/lib/security/context/security_context.c + src/core/lib/security/credentials/composite/composite_credentials.c + src/core/lib/security/credentials/credentials.c + src/core/lib/security/credentials/credentials_metadata.c + src/core/lib/security/credentials/fake/fake_credentials.c + src/core/lib/security/credentials/google_default/credentials_posix.c + src/core/lib/security/credentials/google_default/credentials_windows.c + src/core/lib/security/credentials/google_default/google_default_credentials.c + src/core/lib/security/credentials/iam/iam_credentials.c + src/core/lib/security/credentials/jwt/json_token.c + src/core/lib/security/credentials/jwt/jwt_credentials.c + src/core/lib/security/credentials/jwt/jwt_verifier.c + src/core/lib/security/credentials/oauth2/oauth2_credentials.c + src/core/lib/security/credentials/plugin/plugin_credentials.c + src/core/lib/security/credentials/ssl/ssl_credentials.c + src/core/lib/security/transport/client_auth_filter.c + src/core/lib/security/transport/handshake.c + src/core/lib/security/transport/secure_endpoint.c + src/core/lib/security/transport/security_connector.c + src/core/lib/security/transport/server_auth_filter.c + src/core/lib/security/transport/tsi_error.c + src/core/lib/security/util/b64.c + src/core/lib/security/util/json_util.c + src/core/lib/surface/init_secure.c + src/core/ext/transport/chttp2/alpn/alpn.c + src/core/lib/tsi/fake_transport_security.c + src/core/lib/tsi/ssl_transport_security.c + src/core/lib/tsi/transport_security.c src/cpp/codegen/codegen_init.cc ) diff --git a/Makefile b/Makefile index 5871c7b39fc..51f5c5e44c3 100644 --- a/Makefile +++ b/Makefile @@ -3421,6 +3421,122 @@ LIBGRPC++_SRC = \ src/cpp/util/status.cc \ src/cpp/util/string_ref.cc \ src/cpp/util/time.cc \ + src/core/lib/channel/channel_args.c \ + src/core/lib/channel/channel_stack.c \ + src/core/lib/channel/channel_stack_builder.c \ + src/core/lib/channel/compress_filter.c \ + src/core/lib/channel/connected_channel.c \ + src/core/lib/channel/http_client_filter.c \ + src/core/lib/channel/http_server_filter.c \ + src/core/lib/compression/compression.c \ + src/core/lib/compression/message_compress.c \ + src/core/lib/debug/trace.c \ + src/core/lib/http/format_request.c \ + src/core/lib/http/httpcli.c \ + src/core/lib/http/parser.c \ + src/core/lib/iomgr/closure.c \ + src/core/lib/iomgr/endpoint.c \ + src/core/lib/iomgr/endpoint_pair_posix.c \ + src/core/lib/iomgr/endpoint_pair_windows.c \ + src/core/lib/iomgr/error.c \ + src/core/lib/iomgr/ev_epoll_linux.c \ + src/core/lib/iomgr/ev_poll_and_epoll_posix.c \ + src/core/lib/iomgr/ev_poll_posix.c \ + src/core/lib/iomgr/ev_posix.c \ + src/core/lib/iomgr/exec_ctx.c \ + src/core/lib/iomgr/executor.c \ + src/core/lib/iomgr/iocp_windows.c \ + src/core/lib/iomgr/iomgr.c \ + src/core/lib/iomgr/iomgr_posix.c \ + src/core/lib/iomgr/iomgr_windows.c \ + src/core/lib/iomgr/load_file.c \ + src/core/lib/iomgr/network_status_tracker.c \ + src/core/lib/iomgr/polling_entity.c \ + src/core/lib/iomgr/pollset_set_windows.c \ + src/core/lib/iomgr/pollset_windows.c \ + src/core/lib/iomgr/resolve_address_posix.c \ + src/core/lib/iomgr/resolve_address_windows.c \ + src/core/lib/iomgr/sockaddr_utils.c \ + src/core/lib/iomgr/socket_utils_common_posix.c \ + src/core/lib/iomgr/socket_utils_linux.c \ + src/core/lib/iomgr/socket_utils_posix.c \ + src/core/lib/iomgr/socket_windows.c \ + src/core/lib/iomgr/tcp_client_posix.c \ + src/core/lib/iomgr/tcp_client_windows.c \ + src/core/lib/iomgr/tcp_posix.c \ + src/core/lib/iomgr/tcp_server_posix.c \ + src/core/lib/iomgr/tcp_server_windows.c \ + src/core/lib/iomgr/tcp_windows.c \ + src/core/lib/iomgr/time_averaged_stats.c \ + src/core/lib/iomgr/timer.c \ + src/core/lib/iomgr/timer_heap.c \ + src/core/lib/iomgr/udp_server.c \ + src/core/lib/iomgr/unix_sockets_posix.c \ + src/core/lib/iomgr/unix_sockets_posix_noop.c \ + src/core/lib/iomgr/wakeup_fd_eventfd.c \ + src/core/lib/iomgr/wakeup_fd_nospecial.c \ + src/core/lib/iomgr/wakeup_fd_pipe.c \ + src/core/lib/iomgr/wakeup_fd_posix.c \ + src/core/lib/iomgr/workqueue_posix.c \ + src/core/lib/iomgr/workqueue_windows.c \ + src/core/lib/json/json.c \ + src/core/lib/json/json_reader.c \ + src/core/lib/json/json_string.c \ + src/core/lib/json/json_writer.c \ + src/core/lib/surface/alarm.c \ + src/core/lib/surface/api_trace.c \ + src/core/lib/surface/byte_buffer.c \ + src/core/lib/surface/byte_buffer_reader.c \ + src/core/lib/surface/call.c \ + src/core/lib/surface/call_details.c \ + src/core/lib/surface/call_log_batch.c \ + src/core/lib/surface/channel.c \ + src/core/lib/surface/channel_init.c \ + src/core/lib/surface/channel_ping.c \ + src/core/lib/surface/channel_stack_type.c \ + src/core/lib/surface/completion_queue.c \ + src/core/lib/surface/event_string.c \ + src/core/lib/surface/lame_client.c \ + src/core/lib/surface/metadata_array.c \ + src/core/lib/surface/server.c \ + src/core/lib/surface/validate_metadata.c \ + src/core/lib/surface/version.c \ + src/core/lib/transport/byte_stream.c \ + src/core/lib/transport/connectivity_state.c \ + src/core/lib/transport/metadata.c \ + src/core/lib/transport/metadata_batch.c \ + src/core/lib/transport/static_metadata.c \ + src/core/lib/transport/transport.c \ + src/core/lib/transport/transport_op_string.c \ + src/core/lib/http/httpcli_security_connector.c \ + src/core/lib/security/context/security_context.c \ + src/core/lib/security/credentials/composite/composite_credentials.c \ + src/core/lib/security/credentials/credentials.c \ + src/core/lib/security/credentials/credentials_metadata.c \ + src/core/lib/security/credentials/fake/fake_credentials.c \ + src/core/lib/security/credentials/google_default/credentials_posix.c \ + src/core/lib/security/credentials/google_default/credentials_windows.c \ + src/core/lib/security/credentials/google_default/google_default_credentials.c \ + src/core/lib/security/credentials/iam/iam_credentials.c \ + src/core/lib/security/credentials/jwt/json_token.c \ + src/core/lib/security/credentials/jwt/jwt_credentials.c \ + src/core/lib/security/credentials/jwt/jwt_verifier.c \ + src/core/lib/security/credentials/oauth2/oauth2_credentials.c \ + src/core/lib/security/credentials/plugin/plugin_credentials.c \ + src/core/lib/security/credentials/ssl/ssl_credentials.c \ + src/core/lib/security/transport/client_auth_filter.c \ + src/core/lib/security/transport/handshake.c \ + src/core/lib/security/transport/secure_endpoint.c \ + src/core/lib/security/transport/security_connector.c \ + src/core/lib/security/transport/server_auth_filter.c \ + src/core/lib/security/transport/tsi_error.c \ + src/core/lib/security/util/b64.c \ + src/core/lib/security/util/json_util.c \ + src/core/lib/surface/init_secure.c \ + src/core/ext/transport/chttp2/alpn/alpn.c \ + src/core/lib/tsi/fake_transport_security.c \ + src/core/lib/tsi/ssl_transport_security.c \ + src/core/lib/tsi/transport_security.c \ src/cpp/codegen/codegen_init.cc \ PUBLIC_HEADERS_CXX += \ @@ -3522,6 +3638,14 @@ PUBLIC_HEADERS_CXX += \ include/grpc/impl/codegen/sync_posix.h \ include/grpc/impl/codegen/sync_windows.h \ include/grpc/impl/codegen/time.h \ + include/grpc/byte_buffer.h \ + include/grpc/byte_buffer_reader.h \ + include/grpc/compression.h \ + include/grpc/grpc.h \ + include/grpc/grpc_posix.h \ + include/grpc/status.h \ + include/grpc/grpc_security.h \ + include/grpc/grpc_security_constants.h \ LIBGRPC++_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(LIBGRPC++_SRC)))) @@ -3558,18 +3682,18 @@ endif ifeq ($(SYSTEM),MINGW32) -$(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/grpc.$(SHARED_EXT) $(OPENSSL_DEP) +$(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/grpc.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/gpr.$(SHARED_EXT) $(OPENSSL_DEP) $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared grpc++.def -Wl,--output-def=$(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).def -Wl,--out-implib=$(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION)-dll.a -o $(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc-imp + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared grpc++.def -Wl,--output-def=$(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).def -Wl,--out-implib=$(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION)-dll.a -o $(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc-imp -lgpr-imp else -$(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/libgrpc.$(SHARED_EXT) $(OPENSSL_DEP) +$(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/libgrpc.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgpr.$(SHARED_EXT) $(OPENSSL_DEP) $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` ifeq ($(SYSTEM),Darwin) - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -install_name $(SHARED_PREFIX)grpc++$(SHARED_VERSION).$(SHARED_EXT) -dynamiclib -o $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -install_name $(SHARED_PREFIX)grpc++$(SHARED_VERSION).$(SHARED_EXT) -dynamiclib -o $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc -lgpr else - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared -Wl,-soname,libgrpc++.so.0 -o $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared -Wl,-soname,libgrpc++.so.0 -o $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc -lgpr $(Q) ln -sf $(SHARED_PREFIX)grpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).so.0 $(Q) ln -sf $(SHARED_PREFIX)grpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).so endif @@ -3908,6 +4032,122 @@ LIBGRPC++_UNSECURE_SRC = \ src/cpp/util/status.cc \ src/cpp/util/string_ref.cc \ src/cpp/util/time.cc \ + src/core/lib/channel/channel_args.c \ + src/core/lib/channel/channel_stack.c \ + src/core/lib/channel/channel_stack_builder.c \ + src/core/lib/channel/compress_filter.c \ + src/core/lib/channel/connected_channel.c \ + src/core/lib/channel/http_client_filter.c \ + src/core/lib/channel/http_server_filter.c \ + src/core/lib/compression/compression.c \ + src/core/lib/compression/message_compress.c \ + src/core/lib/debug/trace.c \ + src/core/lib/http/format_request.c \ + src/core/lib/http/httpcli.c \ + src/core/lib/http/parser.c \ + src/core/lib/iomgr/closure.c \ + src/core/lib/iomgr/endpoint.c \ + src/core/lib/iomgr/endpoint_pair_posix.c \ + src/core/lib/iomgr/endpoint_pair_windows.c \ + src/core/lib/iomgr/error.c \ + src/core/lib/iomgr/ev_epoll_linux.c \ + src/core/lib/iomgr/ev_poll_and_epoll_posix.c \ + src/core/lib/iomgr/ev_poll_posix.c \ + src/core/lib/iomgr/ev_posix.c \ + src/core/lib/iomgr/exec_ctx.c \ + src/core/lib/iomgr/executor.c \ + src/core/lib/iomgr/iocp_windows.c \ + src/core/lib/iomgr/iomgr.c \ + src/core/lib/iomgr/iomgr_posix.c \ + src/core/lib/iomgr/iomgr_windows.c \ + src/core/lib/iomgr/load_file.c \ + src/core/lib/iomgr/network_status_tracker.c \ + src/core/lib/iomgr/polling_entity.c \ + src/core/lib/iomgr/pollset_set_windows.c \ + src/core/lib/iomgr/pollset_windows.c \ + src/core/lib/iomgr/resolve_address_posix.c \ + src/core/lib/iomgr/resolve_address_windows.c \ + src/core/lib/iomgr/sockaddr_utils.c \ + src/core/lib/iomgr/socket_utils_common_posix.c \ + src/core/lib/iomgr/socket_utils_linux.c \ + src/core/lib/iomgr/socket_utils_posix.c \ + src/core/lib/iomgr/socket_windows.c \ + src/core/lib/iomgr/tcp_client_posix.c \ + src/core/lib/iomgr/tcp_client_windows.c \ + src/core/lib/iomgr/tcp_posix.c \ + src/core/lib/iomgr/tcp_server_posix.c \ + src/core/lib/iomgr/tcp_server_windows.c \ + src/core/lib/iomgr/tcp_windows.c \ + src/core/lib/iomgr/time_averaged_stats.c \ + src/core/lib/iomgr/timer.c \ + src/core/lib/iomgr/timer_heap.c \ + src/core/lib/iomgr/udp_server.c \ + src/core/lib/iomgr/unix_sockets_posix.c \ + src/core/lib/iomgr/unix_sockets_posix_noop.c \ + src/core/lib/iomgr/wakeup_fd_eventfd.c \ + src/core/lib/iomgr/wakeup_fd_nospecial.c \ + src/core/lib/iomgr/wakeup_fd_pipe.c \ + src/core/lib/iomgr/wakeup_fd_posix.c \ + src/core/lib/iomgr/workqueue_posix.c \ + src/core/lib/iomgr/workqueue_windows.c \ + src/core/lib/json/json.c \ + src/core/lib/json/json_reader.c \ + src/core/lib/json/json_string.c \ + src/core/lib/json/json_writer.c \ + src/core/lib/surface/alarm.c \ + src/core/lib/surface/api_trace.c \ + src/core/lib/surface/byte_buffer.c \ + src/core/lib/surface/byte_buffer_reader.c \ + src/core/lib/surface/call.c \ + src/core/lib/surface/call_details.c \ + src/core/lib/surface/call_log_batch.c \ + src/core/lib/surface/channel.c \ + src/core/lib/surface/channel_init.c \ + src/core/lib/surface/channel_ping.c \ + src/core/lib/surface/channel_stack_type.c \ + src/core/lib/surface/completion_queue.c \ + src/core/lib/surface/event_string.c \ + src/core/lib/surface/lame_client.c \ + src/core/lib/surface/metadata_array.c \ + src/core/lib/surface/server.c \ + src/core/lib/surface/validate_metadata.c \ + src/core/lib/surface/version.c \ + src/core/lib/transport/byte_stream.c \ + src/core/lib/transport/connectivity_state.c \ + src/core/lib/transport/metadata.c \ + src/core/lib/transport/metadata_batch.c \ + src/core/lib/transport/static_metadata.c \ + src/core/lib/transport/transport.c \ + src/core/lib/transport/transport_op_string.c \ + src/core/lib/http/httpcli_security_connector.c \ + src/core/lib/security/context/security_context.c \ + src/core/lib/security/credentials/composite/composite_credentials.c \ + src/core/lib/security/credentials/credentials.c \ + src/core/lib/security/credentials/credentials_metadata.c \ + src/core/lib/security/credentials/fake/fake_credentials.c \ + src/core/lib/security/credentials/google_default/credentials_posix.c \ + src/core/lib/security/credentials/google_default/credentials_windows.c \ + src/core/lib/security/credentials/google_default/google_default_credentials.c \ + src/core/lib/security/credentials/iam/iam_credentials.c \ + src/core/lib/security/credentials/jwt/json_token.c \ + src/core/lib/security/credentials/jwt/jwt_credentials.c \ + src/core/lib/security/credentials/jwt/jwt_verifier.c \ + src/core/lib/security/credentials/oauth2/oauth2_credentials.c \ + src/core/lib/security/credentials/plugin/plugin_credentials.c \ + src/core/lib/security/credentials/ssl/ssl_credentials.c \ + src/core/lib/security/transport/client_auth_filter.c \ + src/core/lib/security/transport/handshake.c \ + src/core/lib/security/transport/secure_endpoint.c \ + src/core/lib/security/transport/security_connector.c \ + src/core/lib/security/transport/server_auth_filter.c \ + src/core/lib/security/transport/tsi_error.c \ + src/core/lib/security/util/b64.c \ + src/core/lib/security/util/json_util.c \ + src/core/lib/surface/init_secure.c \ + src/core/ext/transport/chttp2/alpn/alpn.c \ + src/core/lib/tsi/fake_transport_security.c \ + src/core/lib/tsi/ssl_transport_security.c \ + src/core/lib/tsi/transport_security.c \ src/cpp/codegen/codegen_init.cc \ PUBLIC_HEADERS_CXX += \ @@ -4009,6 +4249,14 @@ PUBLIC_HEADERS_CXX += \ include/grpc/impl/codegen/sync_posix.h \ include/grpc/impl/codegen/sync_windows.h \ include/grpc/impl/codegen/time.h \ + include/grpc/byte_buffer.h \ + include/grpc/byte_buffer_reader.h \ + include/grpc/compression.h \ + include/grpc/grpc.h \ + include/grpc/grpc_posix.h \ + include/grpc/status.h \ + include/grpc/grpc_security.h \ + include/grpc/grpc_security_constants.h \ LIBGRPC++_UNSECURE_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(LIBGRPC++_UNSECURE_SRC)))) @@ -14851,34 +15099,6 @@ src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c: $(OPENSSL_DE src/core/ext/transport/cronet/client/secure/cronet_channel_create.c: $(OPENSSL_DEP) src/core/ext/transport/cronet/transport/cronet_api_dummy.c: $(OPENSSL_DEP) src/core/ext/transport/cronet/transport/cronet_transport.c: $(OPENSSL_DEP) -src/core/lib/http/httpcli_security_connector.c: $(OPENSSL_DEP) -src/core/lib/security/context/security_context.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/composite/composite_credentials.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/credentials.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/credentials_metadata.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/fake/fake_credentials.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/google_default/credentials_posix.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/google_default/credentials_windows.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/google_default/google_default_credentials.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/iam/iam_credentials.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/jwt/json_token.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/jwt/jwt_credentials.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/jwt/jwt_verifier.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/oauth2/oauth2_credentials.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/plugin/plugin_credentials.c: $(OPENSSL_DEP) -src/core/lib/security/credentials/ssl/ssl_credentials.c: $(OPENSSL_DEP) -src/core/lib/security/transport/client_auth_filter.c: $(OPENSSL_DEP) -src/core/lib/security/transport/handshake.c: $(OPENSSL_DEP) -src/core/lib/security/transport/secure_endpoint.c: $(OPENSSL_DEP) -src/core/lib/security/transport/security_connector.c: $(OPENSSL_DEP) -src/core/lib/security/transport/server_auth_filter.c: $(OPENSSL_DEP) -src/core/lib/security/transport/tsi_error.c: $(OPENSSL_DEP) -src/core/lib/security/util/b64.c: $(OPENSSL_DEP) -src/core/lib/security/util/json_util.c: $(OPENSSL_DEP) -src/core/lib/surface/init_secure.c: $(OPENSSL_DEP) -src/core/lib/tsi/fake_transport_security.c: $(OPENSSL_DEP) -src/core/lib/tsi/ssl_transport_security.c: $(OPENSSL_DEP) -src/core/lib/tsi/transport_security.c: $(OPENSSL_DEP) src/core/plugin_registry/grpc_cronet_plugin_registry.c: $(OPENSSL_DEP) src/core/plugin_registry/grpc_plugin_registry.c: $(OPENSSL_DEP) src/cpp/client/secure_credentials.cc: $(OPENSSL_DEP) diff --git a/build.yaml b/build.yaml index 1c485fd5c9e..dc42a61300e 100644 --- a/build.yaml +++ b/build.yaml @@ -714,6 +714,8 @@ filegroups: - src/cpp/util/time.cc uses: - grpc++_codegen_base + - grpc_base + - grpc_secure - name: grpc++_codegen_base language: c++ public_headers: diff --git a/tools/doxygen/Doxyfile.c++ b/tools/doxygen/Doxyfile.c++ index de7acd7777e..db6b36f8c7b 100644 --- a/tools/doxygen/Doxyfile.c++ +++ b/tools/doxygen/Doxyfile.c++ @@ -857,7 +857,15 @@ include/grpc/impl/codegen/sync.h \ include/grpc/impl/codegen/sync_generic.h \ include/grpc/impl/codegen/sync_posix.h \ include/grpc/impl/codegen/sync_windows.h \ -include/grpc/impl/codegen/time.h +include/grpc/impl/codegen/time.h \ +include/grpc/byte_buffer.h \ +include/grpc/byte_buffer_reader.h \ +include/grpc/compression.h \ +include/grpc/grpc.h \ +include/grpc/grpc_posix.h \ +include/grpc/status.h \ +include/grpc/grpc_security.h \ +include/grpc/grpc_security_constants.h # This tag can be used to specify the character encoding of the source files # that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses diff --git a/tools/doxygen/Doxyfile.c++.internal b/tools/doxygen/Doxyfile.c++.internal index 76bb3b6c59e..660e501d713 100644 --- a/tools/doxygen/Doxyfile.c++.internal +++ b/tools/doxygen/Doxyfile.c++.internal @@ -858,6 +858,14 @@ include/grpc/impl/codegen/sync_generic.h \ include/grpc/impl/codegen/sync_posix.h \ include/grpc/impl/codegen/sync_windows.h \ include/grpc/impl/codegen/time.h \ +include/grpc/byte_buffer.h \ +include/grpc/byte_buffer_reader.h \ +include/grpc/compression.h \ +include/grpc/grpc.h \ +include/grpc/grpc_posix.h \ +include/grpc/status.h \ +include/grpc/grpc_security.h \ +include/grpc/grpc_security_constants.h \ include/grpc++/impl/codegen/core_codegen.h \ src/cpp/client/secure_credentials.h \ src/cpp/common/secure_auth_context.h \ @@ -865,6 +873,109 @@ src/cpp/server/secure_server_credentials.h \ src/cpp/client/create_channel_internal.h \ src/cpp/server/dynamic_thread_pool.h \ src/cpp/server/thread_pool_interface.h \ +src/core/lib/channel/channel_args.h \ +src/core/lib/channel/channel_stack.h \ +src/core/lib/channel/channel_stack_builder.h \ +src/core/lib/channel/compress_filter.h \ +src/core/lib/channel/connected_channel.h \ +src/core/lib/channel/context.h \ +src/core/lib/channel/http_client_filter.h \ +src/core/lib/channel/http_server_filter.h \ +src/core/lib/compression/algorithm_metadata.h \ +src/core/lib/compression/message_compress.h \ +src/core/lib/debug/trace.h \ +src/core/lib/http/format_request.h \ +src/core/lib/http/httpcli.h \ +src/core/lib/http/parser.h \ +src/core/lib/iomgr/closure.h \ +src/core/lib/iomgr/endpoint.h \ +src/core/lib/iomgr/endpoint_pair.h \ +src/core/lib/iomgr/error.h \ +src/core/lib/iomgr/ev_epoll_linux.h \ +src/core/lib/iomgr/ev_poll_and_epoll_posix.h \ +src/core/lib/iomgr/ev_poll_posix.h \ +src/core/lib/iomgr/ev_posix.h \ +src/core/lib/iomgr/exec_ctx.h \ +src/core/lib/iomgr/executor.h \ +src/core/lib/iomgr/iocp_windows.h \ +src/core/lib/iomgr/iomgr.h \ +src/core/lib/iomgr/iomgr_internal.h \ +src/core/lib/iomgr/iomgr_posix.h \ +src/core/lib/iomgr/load_file.h \ +src/core/lib/iomgr/network_status_tracker.h \ +src/core/lib/iomgr/polling_entity.h \ +src/core/lib/iomgr/pollset.h \ +src/core/lib/iomgr/pollset_set.h \ +src/core/lib/iomgr/pollset_set_windows.h \ +src/core/lib/iomgr/pollset_windows.h \ +src/core/lib/iomgr/resolve_address.h \ +src/core/lib/iomgr/sockaddr.h \ +src/core/lib/iomgr/sockaddr_posix.h \ +src/core/lib/iomgr/sockaddr_utils.h \ +src/core/lib/iomgr/sockaddr_windows.h \ +src/core/lib/iomgr/socket_utils_posix.h \ +src/core/lib/iomgr/socket_windows.h \ +src/core/lib/iomgr/tcp_client.h \ +src/core/lib/iomgr/tcp_posix.h \ +src/core/lib/iomgr/tcp_server.h \ +src/core/lib/iomgr/tcp_windows.h \ +src/core/lib/iomgr/time_averaged_stats.h \ +src/core/lib/iomgr/timer.h \ +src/core/lib/iomgr/timer_heap.h \ +src/core/lib/iomgr/udp_server.h \ +src/core/lib/iomgr/unix_sockets_posix.h \ +src/core/lib/iomgr/wakeup_fd_pipe.h \ +src/core/lib/iomgr/wakeup_fd_posix.h \ +src/core/lib/iomgr/workqueue.h \ +src/core/lib/iomgr/workqueue_posix.h \ +src/core/lib/iomgr/workqueue_windows.h \ +src/core/lib/json/json.h \ +src/core/lib/json/json_common.h \ +src/core/lib/json/json_reader.h \ +src/core/lib/json/json_writer.h \ +src/core/lib/surface/api_trace.h \ +src/core/lib/surface/call.h \ +src/core/lib/surface/call_test_only.h \ +src/core/lib/surface/channel.h \ +src/core/lib/surface/channel_init.h \ +src/core/lib/surface/channel_stack_type.h \ +src/core/lib/surface/completion_queue.h \ +src/core/lib/surface/event_string.h \ +src/core/lib/surface/init.h \ +src/core/lib/surface/lame_client.h \ +src/core/lib/surface/server.h \ +src/core/lib/transport/byte_stream.h \ +src/core/lib/transport/connectivity_state.h \ +src/core/lib/transport/metadata.h \ +src/core/lib/transport/metadata_batch.h \ +src/core/lib/transport/static_metadata.h \ +src/core/lib/transport/transport.h \ +src/core/lib/transport/transport_impl.h \ +src/core/lib/security/context/security_context.h \ +src/core/lib/security/credentials/composite/composite_credentials.h \ +src/core/lib/security/credentials/credentials.h \ +src/core/lib/security/credentials/fake/fake_credentials.h \ +src/core/lib/security/credentials/google_default/google_default_credentials.h \ +src/core/lib/security/credentials/iam/iam_credentials.h \ +src/core/lib/security/credentials/jwt/json_token.h \ +src/core/lib/security/credentials/jwt/jwt_credentials.h \ +src/core/lib/security/credentials/jwt/jwt_verifier.h \ +src/core/lib/security/credentials/oauth2/oauth2_credentials.h \ +src/core/lib/security/credentials/plugin/plugin_credentials.h \ +src/core/lib/security/credentials/ssl/ssl_credentials.h \ +src/core/lib/security/transport/auth_filters.h \ +src/core/lib/security/transport/handshake.h \ +src/core/lib/security/transport/secure_endpoint.h \ +src/core/lib/security/transport/security_connector.h \ +src/core/lib/security/transport/tsi_error.h \ +src/core/lib/security/util/b64.h \ +src/core/lib/security/util/json_util.h \ +src/core/ext/transport/chttp2/alpn/alpn.h \ +src/core/lib/tsi/fake_transport_security.h \ +src/core/lib/tsi/ssl_transport_security.h \ +src/core/lib/tsi/ssl_types.h \ +src/core/lib/tsi/transport_security.h \ +src/core/lib/tsi/transport_security_interface.h \ src/cpp/client/secure_credentials.cc \ src/cpp/common/auth_property_iterator.cc \ src/cpp/common/secure_auth_context.cc \ @@ -897,6 +1008,122 @@ src/cpp/util/slice.cc \ src/cpp/util/status.cc \ src/cpp/util/string_ref.cc \ src/cpp/util/time.cc \ +src/core/lib/channel/channel_args.c \ +src/core/lib/channel/channel_stack.c \ +src/core/lib/channel/channel_stack_builder.c \ +src/core/lib/channel/compress_filter.c \ +src/core/lib/channel/connected_channel.c \ +src/core/lib/channel/http_client_filter.c \ +src/core/lib/channel/http_server_filter.c \ +src/core/lib/compression/compression.c \ +src/core/lib/compression/message_compress.c \ +src/core/lib/debug/trace.c \ +src/core/lib/http/format_request.c \ +src/core/lib/http/httpcli.c \ +src/core/lib/http/parser.c \ +src/core/lib/iomgr/closure.c \ +src/core/lib/iomgr/endpoint.c \ +src/core/lib/iomgr/endpoint_pair_posix.c \ +src/core/lib/iomgr/endpoint_pair_windows.c \ +src/core/lib/iomgr/error.c \ +src/core/lib/iomgr/ev_epoll_linux.c \ +src/core/lib/iomgr/ev_poll_and_epoll_posix.c \ +src/core/lib/iomgr/ev_poll_posix.c \ +src/core/lib/iomgr/ev_posix.c \ +src/core/lib/iomgr/exec_ctx.c \ +src/core/lib/iomgr/executor.c \ +src/core/lib/iomgr/iocp_windows.c \ +src/core/lib/iomgr/iomgr.c \ +src/core/lib/iomgr/iomgr_posix.c \ +src/core/lib/iomgr/iomgr_windows.c \ +src/core/lib/iomgr/load_file.c \ +src/core/lib/iomgr/network_status_tracker.c \ +src/core/lib/iomgr/polling_entity.c \ +src/core/lib/iomgr/pollset_set_windows.c \ +src/core/lib/iomgr/pollset_windows.c \ +src/core/lib/iomgr/resolve_address_posix.c \ +src/core/lib/iomgr/resolve_address_windows.c \ +src/core/lib/iomgr/sockaddr_utils.c \ +src/core/lib/iomgr/socket_utils_common_posix.c \ +src/core/lib/iomgr/socket_utils_linux.c \ +src/core/lib/iomgr/socket_utils_posix.c \ +src/core/lib/iomgr/socket_windows.c \ +src/core/lib/iomgr/tcp_client_posix.c \ +src/core/lib/iomgr/tcp_client_windows.c \ +src/core/lib/iomgr/tcp_posix.c \ +src/core/lib/iomgr/tcp_server_posix.c \ +src/core/lib/iomgr/tcp_server_windows.c \ +src/core/lib/iomgr/tcp_windows.c \ +src/core/lib/iomgr/time_averaged_stats.c \ +src/core/lib/iomgr/timer.c \ +src/core/lib/iomgr/timer_heap.c \ +src/core/lib/iomgr/udp_server.c \ +src/core/lib/iomgr/unix_sockets_posix.c \ +src/core/lib/iomgr/unix_sockets_posix_noop.c \ +src/core/lib/iomgr/wakeup_fd_eventfd.c \ +src/core/lib/iomgr/wakeup_fd_nospecial.c \ +src/core/lib/iomgr/wakeup_fd_pipe.c \ +src/core/lib/iomgr/wakeup_fd_posix.c \ +src/core/lib/iomgr/workqueue_posix.c \ +src/core/lib/iomgr/workqueue_windows.c \ +src/core/lib/json/json.c \ +src/core/lib/json/json_reader.c \ +src/core/lib/json/json_string.c \ +src/core/lib/json/json_writer.c \ +src/core/lib/surface/alarm.c \ +src/core/lib/surface/api_trace.c \ +src/core/lib/surface/byte_buffer.c \ +src/core/lib/surface/byte_buffer_reader.c \ +src/core/lib/surface/call.c \ +src/core/lib/surface/call_details.c \ +src/core/lib/surface/call_log_batch.c \ +src/core/lib/surface/channel.c \ +src/core/lib/surface/channel_init.c \ +src/core/lib/surface/channel_ping.c \ +src/core/lib/surface/channel_stack_type.c \ +src/core/lib/surface/completion_queue.c \ +src/core/lib/surface/event_string.c \ +src/core/lib/surface/lame_client.c \ +src/core/lib/surface/metadata_array.c \ +src/core/lib/surface/server.c \ +src/core/lib/surface/validate_metadata.c \ +src/core/lib/surface/version.c \ +src/core/lib/transport/byte_stream.c \ +src/core/lib/transport/connectivity_state.c \ +src/core/lib/transport/metadata.c \ +src/core/lib/transport/metadata_batch.c \ +src/core/lib/transport/static_metadata.c \ +src/core/lib/transport/transport.c \ +src/core/lib/transport/transport_op_string.c \ +src/core/lib/http/httpcli_security_connector.c \ +src/core/lib/security/context/security_context.c \ +src/core/lib/security/credentials/composite/composite_credentials.c \ +src/core/lib/security/credentials/credentials.c \ +src/core/lib/security/credentials/credentials_metadata.c \ +src/core/lib/security/credentials/fake/fake_credentials.c \ +src/core/lib/security/credentials/google_default/credentials_posix.c \ +src/core/lib/security/credentials/google_default/credentials_windows.c \ +src/core/lib/security/credentials/google_default/google_default_credentials.c \ +src/core/lib/security/credentials/iam/iam_credentials.c \ +src/core/lib/security/credentials/jwt/json_token.c \ +src/core/lib/security/credentials/jwt/jwt_credentials.c \ +src/core/lib/security/credentials/jwt/jwt_verifier.c \ +src/core/lib/security/credentials/oauth2/oauth2_credentials.c \ +src/core/lib/security/credentials/plugin/plugin_credentials.c \ +src/core/lib/security/credentials/ssl/ssl_credentials.c \ +src/core/lib/security/transport/client_auth_filter.c \ +src/core/lib/security/transport/handshake.c \ +src/core/lib/security/transport/secure_endpoint.c \ +src/core/lib/security/transport/security_connector.c \ +src/core/lib/security/transport/server_auth_filter.c \ +src/core/lib/security/transport/tsi_error.c \ +src/core/lib/security/util/b64.c \ +src/core/lib/security/util/json_util.c \ +src/core/lib/surface/init_secure.c \ +src/core/ext/transport/chttp2/alpn/alpn.c \ +src/core/lib/tsi/fake_transport_security.c \ +src/core/lib/tsi/ssl_transport_security.c \ +src/core/lib/tsi/transport_security.c \ src/cpp/codegen/codegen_init.cc # This tag can be used to specify the character encoding of the source files diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 6d7cfdaf233..3ebb445b8a8 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -4345,6 +4345,7 @@ }, { "deps": [ + "gpr", "grpc", "grpc++_base", "grpc++_codegen_base", @@ -6514,7 +6515,10 @@ }, { "deps": [ - "grpc++_codegen_base" + "gpr", + "grpc++_codegen_base", + "grpc_base", + "grpc_secure" ], "headers": [ "include/grpc++/alarm.h", diff --git a/vsprojects/grpc.sln b/vsprojects/grpc.sln index 6105f724c9f..8fccc646e5c 100644 --- a/vsprojects/grpc.sln +++ b/vsprojects/grpc.sln @@ -49,6 +49,7 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc++", "vcxproj\.\grpc++\ EndProjectSection ProjectSection(ProjectDependencies) = postProject {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc++_reflection", "vcxproj\.\grpc++_reflection\grpc++_reflection.vcxproj", "{5F575402-3F89-5D1A-6910-9DB8BF5D2BAB}" diff --git a/vsprojects/vcxproj/grpc++/grpc++.vcxproj b/vsprojects/vcxproj/grpc++/grpc++.vcxproj index cb9e41ea22f..a2711ca7a46 100644 --- a/vsprojects/vcxproj/grpc++/grpc++.vcxproj +++ b/vsprojects/vcxproj/grpc++/grpc++.vcxproj @@ -356,6 +356,14 @@ + + + + + + + + @@ -365,6 +373,109 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -431,6 +542,238 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -438,6 +781,9 @@ {29D16885-7228-4C31-81ED-5F9187C7F2A9} + + {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} + diff --git a/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters b/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters index a9051182b3c..f478ac9839c 100644 --- a/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters +++ b/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters @@ -97,6 +97,354 @@ src\cpp\util + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\compression + + + src\core\lib\compression + + + src\core\lib\debug + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\http + + + src\core\lib\security\context + + + src\core\lib\security\credentials\composite + + + src\core\lib\security\credentials + + + src\core\lib\security\credentials + + + src\core\lib\security\credentials\fake + + + src\core\lib\security\credentials\google_default + + + src\core\lib\security\credentials\google_default + + + src\core\lib\security\credentials\google_default + + + src\core\lib\security\credentials\iam + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\oauth2 + + + src\core\lib\security\credentials\plugin + + + src\core\lib\security\credentials\ssl + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\util + + + src\core\lib\security\util + + + src\core\lib\surface + + + src\core\ext\transport\chttp2\alpn + + + src\core\lib\tsi + + + src\core\lib\tsi + + + src\core\lib\tsi + src\cpp\codegen @@ -396,6 +744,30 @@ include\grpc\impl\codegen + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + @@ -419,6 +791,315 @@ src\cpp\server + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\compression + + + src\core\lib\compression + + + src\core\lib\debug + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\security\context + + + src\core\lib\security\credentials\composite + + + src\core\lib\security\credentials + + + src\core\lib\security\credentials\fake + + + src\core\lib\security\credentials\google_default + + + src\core\lib\security\credentials\iam + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\oauth2 + + + src\core\lib\security\credentials\plugin + + + src\core\lib\security\credentials\ssl + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\util + + + src\core\lib\security\util + + + src\core\ext\transport\chttp2\alpn + + + src\core\lib\tsi + + + src\core\lib\tsi + + + src\core\lib\tsi + + + src\core\lib\tsi + + + src\core\lib\tsi + @@ -458,6 +1139,90 @@ {328ff211-2886-406e-56f9-18ba1686f363} + + {d02f1155-7e7e-3736-3c69-dc9146dc523d} + + + {96d09c4a-59f9-3486-6c2f-cbf695b285d8} + + + {202b1172-189f-afc4-f16c-4ca12677b480} + + + {9de393b8-4b6e-6c34-122a-940419ca9989} + + + {efb6b3e6-8c7b-c2a0-12c6-486c68cdb8ec} + + + {80567a8f-622f-a3ce-c12d-aebb63984b07} + + + {e769265c-8abd-cd64-2cc2-a52da484fe7b} + + + {701b2d46-11c6-3640-b189-45287f00bee3} + + + {ada68fd5-8e51-98cb-71a7-baf7989d8ffa} + + + {e770844e-61d4-555e-59be-81288e21a35f} + + + {04dfa1c8-7ffe-4f06-4a7c-37441dc75764} + + + {a5d5bddf-6f19-b655-a03a-f30ff5c253a5} + + + {dbd8cbb6-6308-d6fe-7a36-06cc7045c037} + + + {ecd2c264-808d-0041-2f69-a5200543de91} + + + {0015e481-7e80-8936-a25c-c3fa260cc095} + + + {fad200df-a5e2-1648-7442-cea0f07edd4d} + + + {397464b3-9bbd-15a5-041b-c7deef1662ec} + + + {567691b4-6a06-cc5a-c6ad-e8c080b89ecf} + + + {d5930113-d396-7a70-d273-d07a1feae0ff} + + + {0f6afb67-4b51-6344-9de7-2b1a18a19e7d} + + + {99faa051-ca9f-cb4f-36d5-95f042fb22bc} + + + {b7a9e7e5-2445-6b0f-4677-5095ca10e760} + + + {436bc65a-0c1b-d85a-2c91-6474588c5cb6} + + + {e6a9bf58-3b0f-0b3d-3a35-3ded80d27695} + + + {b4a1cab8-5c2c-909a-8097-7a5c8f0aa9f7} + + + {fb2276d7-5a11-f1d9-82c3-e7c7f1155523} + + + {4bd7971a-68f7-0d5a-f502-6dea3099caaa} + + + {aa0153b8-c9b6-ae1d-ebdd-89754d8579f1} + {2420a905-e4f1-a5aa-a364-6a112878a39e} diff --git a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj index f37fefc65a4..84e709611df 100644 --- a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj @@ -356,11 +356,122 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -417,6 +528,238 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters index ba99bc53c8c..1e54e1595d4 100644 --- a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters +++ b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters @@ -82,6 +82,354 @@ src\cpp\util + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\compression + + + src\core\lib\compression + + + src\core\lib\debug + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\http + + + src\core\lib\security\context + + + src\core\lib\security\credentials\composite + + + src\core\lib\security\credentials + + + src\core\lib\security\credentials + + + src\core\lib\security\credentials\fake + + + src\core\lib\security\credentials\google_default + + + src\core\lib\security\credentials\google_default + + + src\core\lib\security\credentials\google_default + + + src\core\lib\security\credentials\iam + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\oauth2 + + + src\core\lib\security\credentials\plugin + + + src\core\lib\security\credentials\ssl + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\util + + + src\core\lib\security\util + + + src\core\lib\surface + + + src\core\ext\transport\chttp2\alpn + + + src\core\lib\tsi + + + src\core\lib\tsi + + + src\core\lib\tsi + src\cpp\codegen @@ -381,6 +729,30 @@ include\grpc\impl\codegen + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + + + include\grpc + @@ -392,6 +764,315 @@ src\cpp\server + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\channel + + + src\core\lib\compression + + + src\core\lib\compression + + + src\core\lib\debug + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\http + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\iomgr + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\json + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\surface + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\transport + + + src\core\lib\security\context + + + src\core\lib\security\credentials\composite + + + src\core\lib\security\credentials + + + src\core\lib\security\credentials\fake + + + src\core\lib\security\credentials\google_default + + + src\core\lib\security\credentials\iam + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\jwt + + + src\core\lib\security\credentials\oauth2 + + + src\core\lib\security\credentials\plugin + + + src\core\lib\security\credentials\ssl + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\transport + + + src\core\lib\security\util + + + src\core\lib\security\util + + + src\core\ext\transport\chttp2\alpn + + + src\core\lib\tsi + + + src\core\lib\tsi + + + src\core\lib\tsi + + + src\core\lib\tsi + + + src\core\lib\tsi + @@ -431,6 +1112,90 @@ {cce6a85d-1111-3834-6825-31e170d93cff} + + {595f2ea0-aafb-87e5-c938-db3ff0b0c69a} + + + {52eca76b-9502-3d96-9064-6415226a860f} + + + {8e70201f-3b54-d3cb-8b30-ebe0d96a9b2a} + + + {d505ab7b-5e44-f307-5361-500128965cdc} + + + {d54bab94-cab9-803d-2737-5120774f1893} + + + {cf8fd5d8-ff54-331d-2d20-36d6cae0e14b} + + + {7e0225af-000b-4873-1c16-caffffbfd084} + + + {0bbdbf56-83ad-bb4b-c4e2-a6d38c342179} + + + {3875f7d7-ff11-c91d-0f98-810260cb554b} + + + {4bd405b9-af65-f0a6-d67a-433f75900668} + + + {f4b146e4-8fba-83a6-1cc1-1262ebb785e8} + + + {b83c8e70-e491-f6f9-a08c-85f632bb61d2} + + + {7e21ce26-45e2-6baf-037d-8ab4374077a9} + + + {613e655a-e5c0-9f0c-2bb4-62310a7329c0} + + + {30bddf3f-0eda-9f2f-8171-d86b1e4896fc} + + + {b34f8fa3-0fb9-4916-be6d-2a14a0794882} + + + {7e11872b-bfbb-7d23-4783-e56909c520e8} + + + {212855e8-b7bc-d5bb-0734-dd28996f28de} + + + {6d3828d0-5e5f-15c2-7d46-5d4039a88aad} + + + {b31e7015-364c-5701-31d0-644b1a8ae8c9} + + + {43e3cb91-4101-1fee-6833-20f77ab7f4e5} + + + {727c0b51-4544-957f-45f2-00bf42ff7db9} + + + {606a441b-0d57-85d8-8079-1e6e502d18f1} + + + {5b0b16ae-a8ad-81c3-afe4-8ac0b9e15311} + + + {56333427-0f81-b88b-bf49-a1b2f462023d} + + + {1d59dcef-3358-d0ab-fa42-64da74065785} + + + {ba865739-5dd9-6731-6772-48c25d45134f} + + + {dd4e4960-5bc8-395b-09c4-f2cbd6f6432b} + {1e5fd68c-bd87-e803-42b0-75a7fa19b91d} From d716e24c5c97b2afb51ee4066a149c7316154584 Mon Sep 17 00:00:00 2001 From: "Nicolas \"Pixel\" Noble" Date: Thu, 7 Jul 2016 01:40:33 +0200 Subject: [PATCH 0427/1039] Stop using image aliases, as this is getting deprecated. --- tools/gce/create_linux_worker.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/gce/create_linux_worker.sh b/tools/gce/create_linux_worker.sh index c41e4d299bf..8a2df40859b 100755 --- a/tools/gce/create_linux_worker.sh +++ b/tools/gce/create_linux_worker.sh @@ -43,7 +43,8 @@ gcloud compute instances create $INSTANCE_NAME \ --project="$CLOUD_PROJECT" \ --zone "$ZONE" \ --machine-type n1-standard-8 \ - --image ubuntu-15-10 \ + --image-family=ubuntu-1510 \ + --image-project=ubuntu-os-cloud \ --boot-disk-size 1000 echo 'Created GCE instance, waiting 60 seconds for it to come online.' From c7940ba9f01d0f2873c45019a325b6d84b700a9a Mon Sep 17 00:00:00 2001 From: yang-g Date: Wed, 6 Jul 2016 16:48:19 -0700 Subject: [PATCH 0428/1039] init another one --- test/core/end2end/dualstack_socket_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/core/end2end/dualstack_socket_test.c b/test/core/end2end/dualstack_socket_test.c index 65a8deb663d..348b9ed5f02 100644 --- a/test/core/end2end/dualstack_socket_test.c +++ b/test/core/end2end/dualstack_socket_test.c @@ -273,7 +273,7 @@ void test_connect(const char *server_host, const char *client_host, int port, } int external_dns_works(const char *host) { - grpc_resolved_addresses *res; + grpc_resolved_addresses *res = NULL; grpc_error *error = grpc_blocking_resolve_address(host, "80", &res); GRPC_ERROR_UNREF(error); if (res != NULL) { From 9241c6947fa924b6d961d2e90e1a4f7293ec727e Mon Sep 17 00:00:00 2001 From: yang-g Date: Wed, 6 Jul 2016 17:01:10 -0700 Subject: [PATCH 0429/1039] Use test roots.pem in test --- test/core/surface/sequential_connectivity_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/core/surface/sequential_connectivity_test.c b/test/core/surface/sequential_connectivity_test.c index 2fba3927ba7..fe87f119f2f 100644 --- a/test/core/surface/sequential_connectivity_test.c +++ b/test/core/surface/sequential_connectivity_test.c @@ -154,7 +154,7 @@ static void secure_test_add_port(grpc_server *server, const char *addr) { static grpc_channel *secure_test_create_channel(const char *addr) { grpc_channel_credentials *ssl_creds = - grpc_ssl_credentials_create(NULL, NULL, NULL); + grpc_ssl_credentials_create(test_root_cert, NULL, NULL); grpc_arg ssl_name_override = {GRPC_ARG_STRING, GRPC_SSL_TARGET_NAME_OVERRIDE_ARG, {"foo.test.google.fr"}}; From 42ac6dbe2052913727b29f92d5415c4a5c4b845f Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Wed, 6 Jul 2016 17:13:56 -0700 Subject: [PATCH 0430/1039] Handle orphaned fds --- src/core/lib/iomgr/ev_epoll_linux.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 0fb4399fa7a..0e6cba7e4fe 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -1548,6 +1548,14 @@ retry: * polling_island fields in both fd and pollset to point to the merged * polling island. */ + + if (fd->orphaned) { + gpr_mu_unlock(&fd->mu); + gpr_mu_unlock(&pollset->mu); + /* early out */ + return; + } + if (fd->polling_island == pollset->polling_island) { pi_new = fd->polling_island; if (pi_new == NULL) { From 45c0f2b3051bf1642337e109df57e8031cb654c8 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Wed, 6 Jul 2016 19:26:09 -0700 Subject: [PATCH 0431/1039] Migrated python performance tests to use GA API --- .../tests/qps/benchmark_client.py | 37 +++++++++---------- .../tests/qps/benchmark_server.py | 4 +- .../grpcio_tests/tests/qps/qps_worker.py | 7 +++- .../grpcio_tests/tests/qps/worker_server.py | 36 +++++++++++------- .../grpcio_tests/tests/unit/test_common.py | 22 +++++++++++ .../performance/run_worker_python.sh | 2 +- .../run_tests/performance/scenario_config.py | 25 ++++++------- 7 files changed, 83 insertions(+), 50 deletions(-) diff --git a/src/python/grpcio_tests/tests/qps/benchmark_client.py b/src/python/grpcio_tests/tests/qps/benchmark_client.py index 080281415d4..83b46c914e7 100644 --- a/src/python/grpcio_tests/tests/qps/benchmark_client.py +++ b/src/python/grpcio_tests/tests/qps/benchmark_client.py @@ -37,16 +37,23 @@ from concurrent import futures from six.moves import queue import grpc -from grpc.beta import implementations -from grpc.framework.interfaces.face import face from src.proto.grpc.testing import messages_pb2 from src.proto.grpc.testing import services_pb2 from tests.unit import resources -from tests.unit.beta import test_utilities +from tests.unit import test_common _TIMEOUT = 60 * 60 * 24 +class GenericStub(object): + + def __init__(self, channel): + self.UnaryCall = channel.unary_unary( + '/grpc.testing.BenchmarkService/UnaryCall') + self.StreamingCall = channel.stream_stream( + '/grpc.testing.BenchmarkService/StreamingCall') + + class BenchmarkClient: """Benchmark client interface that exposes a non-blocking send_request().""" @@ -54,15 +61,12 @@ class BenchmarkClient: def __init__(self, server, config, hist): # Create the stub - host, port = server.split(':') - port = int(port) if config.HasField('security_params'): - creds = implementations.ssl_channel_credentials( - resources.test_root_certificates()) - channel = test_utilities.not_really_secure_channel( - host, port, creds, config.security_params.server_host_override) + creds = grpc.ssl_channel_credentials(resources.test_root_certificates()) + channel = test_common.test_secure_channel( + server, creds, config.security_params.server_host_override) else: - channel = implementations.insecure_channel(host, port) + channel = grpc.insecure_channel(server) connected_event = threading.Event() def wait_for_ready(connectivity): @@ -73,7 +77,7 @@ class BenchmarkClient: if config.payload_config.WhichOneof('payload') == 'simple_params': self._generic = False - self._stub = services_pb2.beta_create_BenchmarkService_stub(channel) + self._stub = services_pb2.BenchmarkServiceStub(channel) payload = messages_pb2.Payload( body='\0' * config.payload_config.simple_params.req_size) self._request = messages_pb2.SimpleRequest( @@ -81,7 +85,7 @@ class BenchmarkClient: response_size=config.payload_config.simple_params.resp_size) else: self._generic = True - self._stub = implementations.generic_stub(channel) + self._stub = GenericStub(channel) self._request = '\0' * config.payload_config.bytebuf_params.req_size self._hist = hist @@ -166,13 +170,8 @@ class _SyncStream(object): def start(self): self._is_streaming = True - if self._generic: - stream_callable = self._stub.stream_stream( - 'grpc.testing.BenchmarkService', 'StreamingCall') - else: - stream_callable = self._stub.StreamingCall - - response_stream = stream_callable(self._request_generator(), _TIMEOUT) + response_stream = self._stub.StreamingCall( + self._request_generator(), _TIMEOUT) for _ in response_stream: self._handle_response( self, time.time() - self._send_time_queue.get_nowait()) diff --git a/src/python/grpcio_tests/tests/qps/benchmark_server.py b/src/python/grpcio_tests/tests/qps/benchmark_server.py index 8cbf480d58b..2b76b810cdd 100644 --- a/src/python/grpcio_tests/tests/qps/benchmark_server.py +++ b/src/python/grpcio_tests/tests/qps/benchmark_server.py @@ -31,7 +31,7 @@ from src.proto.grpc.testing import messages_pb2 from src.proto.grpc.testing import services_pb2 -class BenchmarkServer(services_pb2.BetaBenchmarkServiceServicer): +class BenchmarkServer(services_pb2.BenchmarkServiceServicer): """Synchronous Server implementation for the Benchmark service.""" def UnaryCall(self, request, context): @@ -44,7 +44,7 @@ class BenchmarkServer(services_pb2.BetaBenchmarkServiceServicer): yield messages_pb2.SimpleResponse(payload=payload) -class GenericBenchmarkServer(services_pb2.BetaBenchmarkServiceServicer): +class GenericBenchmarkServer(services_pb2.BenchmarkServiceServicer): """Generic Server implementation for the Benchmark service.""" def __init__(self, resp_size): diff --git a/src/python/grpcio_tests/tests/qps/qps_worker.py b/src/python/grpcio_tests/tests/qps/qps_worker.py index 16926379a5b..3abf0d08dd9 100644 --- a/src/python/grpcio_tests/tests/qps/qps_worker.py +++ b/src/python/grpcio_tests/tests/qps/qps_worker.py @@ -32,18 +32,21 @@ import argparse import time +from concurrent import futures +import grpc from src.proto.grpc.testing import services_pb2 from tests.qps import worker_server def run_worker_server(port): + server = grpc.server((), futures.ThreadPoolExecutor(max_workers=5)) servicer = worker_server.WorkerServer() - server = services_pb2.beta_create_WorkerService_server(servicer) + services_pb2.add_WorkerServiceServicer_to_server(servicer, server) server.add_insecure_port('[::]:{}'.format(port)) server.start() servicer.wait_for_quit() - server.stop(2) + server.stop(0) if __name__ == '__main__': diff --git a/src/python/grpcio_tests/tests/qps/worker_server.py b/src/python/grpcio_tests/tests/qps/worker_server.py index d41f8377c2a..932a1ffe2b4 100644 --- a/src/python/grpcio_tests/tests/qps/worker_server.py +++ b/src/python/grpcio_tests/tests/qps/worker_server.py @@ -32,8 +32,8 @@ import random import threading import time -from grpc.beta import implementations -from grpc.framework.interfaces.face import utilities +from concurrent import futures +import grpc from src.proto.grpc.testing import control_pb2 from src.proto.grpc.testing import services_pb2 from src.proto.grpc.testing import stats_pb2 @@ -45,7 +45,7 @@ from tests.qps import histogram from tests.unit import resources -class WorkerServer(services_pb2.BetaWorkerServiceServicer): +class WorkerServer(services_pb2.WorkerServiceServicer): """Python Worker Server implementation.""" def __init__(self): @@ -65,7 +65,7 @@ class WorkerServer(services_pb2.BetaWorkerServiceServicer): if request.mark.reset: start_time = end_time yield status - server.stop(0) + server.stop(None) def _get_server_status(self, start_time, end_time, port, cores): end_time = time.time() @@ -76,25 +76,35 @@ class WorkerServer(services_pb2.BetaWorkerServiceServicer): return control_pb2.ServerStatus(stats=stats, port=port, cores=cores) def _create_server(self, config): - if config.server_type == control_pb2.SYNC_SERVER: + if config.async_server_threads == 0: + # This is the default concurrent.futures thread pool size, but + # None doesn't seem to work + server_threads = multiprocessing.cpu_count() * 5 + else: + server_threads = config.async_server_threads + server = grpc.server((), futures.ThreadPoolExecutor( + max_workers=server_threads)) + if config.server_type == control_pb2.ASYNC_SERVER: servicer = benchmark_server.BenchmarkServer() - server = services_pb2.beta_create_BenchmarkService_server(servicer) + services_pb2.add_BenchmarkServiceServicer_to_server(servicer, server) elif config.server_type == control_pb2.ASYNC_GENERIC_SERVER: resp_size = config.payload_config.bytebuf_params.resp_size servicer = benchmark_server.GenericBenchmarkServer(resp_size) method_implementations = { - ('grpc.testing.BenchmarkService', 'StreamingCall'): - utilities.stream_stream_inline(servicer.StreamingCall), - ('grpc.testing.BenchmarkService', 'UnaryCall'): - utilities.unary_unary_inline(servicer.UnaryCall), + 'StreamingCall': + grpc.stream_stream_rpc_method_handler(servicer.StreamingCall), + 'UnaryCall': + grpc.unary_unary_rpc_method_handler(servicer.UnaryCall), } - server = implementations.server(method_implementations) + handler = grpc.method_handlers_generic_handler( + 'grpc.testing.BenchmarkService', method_implementations) + server.add_generic_rpc_handlers((handler,)) else: raise Exception('Unsupported server type {}'.format(config.server_type)) if config.HasField('security_params'): # Use SSL - server_creds = implementations.ssl_server_credentials([( - resources.private_key(), resources.certificate_chain())]) + server_creds = grpc.ssl_server_credentials( + ((resources.private_key(), resources.certificate_chain()),)) port = server.add_secure_port('[::]:{}'.format(config.port), server_creds) else: port = server.add_insecure_port('[::]:{}'.format(config.port)) diff --git a/src/python/grpcio_tests/tests/unit/test_common.py b/src/python/grpcio_tests/tests/unit/test_common.py index c8886bf4ca5..cd71bd80d75 100644 --- a/src/python/grpcio_tests/tests/unit/test_common.py +++ b/src/python/grpcio_tests/tests/unit/test_common.py @@ -31,6 +31,7 @@ import collections +import grpc import six INVOCATION_INITIAL_METADATA = (('0', 'abc'), ('1', 'def'), ('2', 'ghi'),) @@ -78,3 +79,24 @@ def metadata_transmitted(original_metadata, transmitted_metadata): return False else: return True + + +def test_secure_channel( + target, channel_credentials, server_host_override): + """Creates an insecure Channel to a remote host. + + Args: + host: The name of the remote host to which to connect. + port: The port of the remote host to which to connect. + channel_credentials: The implementations.ChannelCredentials with which to + connect. + server_host_override: The target name used for SSL host name checking. + + Returns: + An implementations.Channel to the remote host through which RPCs may be + conducted. + """ + channel = grpc.secure_channel( + target, channel_credentials, + (('grpc.ssl_target_name_override', server_host_override,),)) + return channel diff --git a/tools/run_tests/performance/run_worker_python.sh b/tools/run_tests/performance/run_worker_python.sh index 3b8ba6f4e4a..06cf172d6fe 100755 --- a/tools/run_tests/performance/run_worker_python.sh +++ b/tools/run_tests/performance/run_worker_python.sh @@ -32,4 +32,4 @@ set -ex cd $(dirname $0)/../../.. -PYTHONPATH=src/python/grpcio_tests:src/python/grpcio:src/python/gens py27/bin/python src/python/grpcio_tests/tests/qps/qps_worker.py $@ +PYTHONPATH=src/python/grpcio_tests:src/python/gens py27/bin/python src/python/grpcio_tests/tests/qps/qps_worker.py $@ diff --git a/tools/run_tests/performance/scenario_config.py b/tools/run_tests/performance/scenario_config.py index 2d5130e1e86..4dfd01fc663 100644 --- a/tools/run_tests/performance/scenario_config.py +++ b/tools/run_tests/performance/scenario_config.py @@ -387,45 +387,44 @@ class PythonLanguage: return 500 def scenarios(self): - # TODO(issue #6522): Empty streaming requests does not work for python - #yield _ping_pong_scenario( - # 'python_generic_async_streaming_ping_pong', rpc_type='STREAMING', - # client_type='ASYNC_CLIENT', server_type='ASYNC_GENERIC_SERVER', - # use_generic_payload=True, - # categories=[SMOKETEST]) + yield _ping_pong_scenario( + 'python_generic_sync_streaming_ping_pong', rpc_type='STREAMING', + client_type='SYNC_CLIENT', server_type='ASYNC_GENERIC_SERVER', + use_generic_payload=True, + categories=[SMOKETEST]) yield _ping_pong_scenario( 'python_protobuf_sync_streaming_ping_pong', rpc_type='STREAMING', - client_type='SYNC_CLIENT', server_type='SYNC_SERVER') + client_type='SYNC_CLIENT', server_type='ASYNC_SERVER') yield _ping_pong_scenario( 'python_protobuf_async_unary_ping_pong', rpc_type='UNARY', - client_type='ASYNC_CLIENT', server_type='SYNC_SERVER') + client_type='ASYNC_CLIENT', server_type='ASYNC_SERVER') yield _ping_pong_scenario( 'python_protobuf_sync_unary_ping_pong', rpc_type='UNARY', - client_type='SYNC_CLIENT', server_type='SYNC_SERVER', + client_type='SYNC_CLIENT', server_type='ASYNC_SERVER', categories=[SMOKETEST]) yield _ping_pong_scenario( 'python_protobuf_sync_unary_qps_unconstrained', rpc_type='UNARY', - client_type='SYNC_CLIENT', server_type='SYNC_SERVER', + client_type='SYNC_CLIENT', server_type='ASYNC_SERVER', unconstrained_client='sync') yield _ping_pong_scenario( 'python_protobuf_sync_streaming_qps_unconstrained', rpc_type='STREAMING', - client_type='SYNC_CLIENT', server_type='SYNC_SERVER', + client_type='SYNC_CLIENT', server_type='ASYNC_SERVER', unconstrained_client='sync') yield _ping_pong_scenario( 'python_to_cpp_protobuf_sync_unary_ping_pong', rpc_type='UNARY', - client_type='SYNC_CLIENT', server_type='SYNC_SERVER', + client_type='SYNC_CLIENT', server_type='ASYNC_SERVER', server_language='c++', server_core_limit=1, async_server_threads=1, categories=[SMOKETEST]) yield _ping_pong_scenario( 'python_to_cpp_protobuf_sync_streaming_ping_pong', rpc_type='STREAMING', - client_type='SYNC_CLIENT', server_type='SYNC_SERVER', + client_type='SYNC_CLIENT', server_type='ASYNC_SERVER', server_language='c++', server_core_limit=1, async_server_threads=1) def __str__(self): From ccb184068d21758f905ecc3bfd42a2f9626dd5ee Mon Sep 17 00:00:00 2001 From: Jorge Canizales Date: Thu, 7 Jul 2016 01:47:43 -0700 Subject: [PATCH 0432/1039] Add arm_arch.h back to fix compilation for devices --- src/objective-c/BoringSSL.podspec | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/objective-c/BoringSSL.podspec b/src/objective-c/BoringSSL.podspec index 26a0451f7d3..42b4434d0d3 100644 --- a/src/objective-c/BoringSSL.podspec +++ b/src/objective-c/BoringSSL.podspec @@ -109,8 +109,6 @@ Pod::Spec.new do |s| s.subspec 'Interface' do |ss| ss.header_mappings_dir = 'include/openssl' ss.source_files = 'include/openssl/*.h' - # Doesn't compile correctly; but doesn't seem to be needed: - ss.exclude_files = 'include/openssl/arm_arch.h' end s.subspec 'Implementation' do |ss| ss.header_mappings_dir = '.' @@ -147,6 +145,11 @@ Pod::Spec.new do |s| #include "ssl.h" #include "crypto.h" #include "aes.h" + /* The following macros are defined by base.h. The latter is the first file included by the + other headers. */ + #if defined(OPENSSL_ARM) || defined(OPENSSL_AARCH64) + # include "arm_arch.h" + #endif #include "asn1.h" #include "asn1_mac.h" #include "asn1t.h" From c0e73da8c22550070f328a6fd168abced87342d7 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 7 Jul 2016 09:43:24 -0700 Subject: [PATCH 0433/1039] Fix flow control issue, make debugging in the future easier --- .../chttp2/transport/chttp2_transport.c | 47 ++++++++++--------- .../ext/transport/chttp2/transport/internal.h | 7 ++- .../ext/transport/chttp2/transport/parsing.c | 6 +-- .../transport/chttp2/transport/stream_lists.c | 12 ++--- .../ext/transport/chttp2/transport/writing.c | 6 +-- 5 files changed, 36 insertions(+), 42 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index 8b7f91d3af8..aeedf417b4a 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -295,7 +295,7 @@ static void init_transport(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, gpr_slice_buffer_add( &t->global.qbuf, gpr_slice_from_copied_string(GRPC_CHTTP2_CLIENT_CONNECT_STRING)); - grpc_chttp2_initiate_write(exec_ctx, &t->global, false); + grpc_chttp2_initiate_write(exec_ctx, &t->global, false, "initial_write"); } /* 8 is a random stab in the dark as to a good initial size: it's small enough that it shouldn't waste memory for infrequently used connections, yet @@ -799,14 +799,14 @@ void grpc_chttp2_run_with_global_lock(grpc_exec_ctx *exec_ctx, void grpc_chttp2_initiate_write(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, - bool covered_by_poller) { + bool covered_by_poller, const char *reason) { grpc_chttp2_transport *t = TRANSPORT_FROM_GLOBAL(transport_global); switch (t->executor.write_state) { case GRPC_CHTTP2_WRITING_INACTIVE: set_write_state(t, covered_by_poller ? GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER : GRPC_CHTTP2_WRITE_REQUESTED_NO_POLLER, - "initiate_write"); + reason); break; case GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER: /* nothing to do: write already requested */ @@ -815,7 +815,7 @@ void grpc_chttp2_initiate_write(grpc_exec_ctx *exec_ctx, if (covered_by_poller) { /* upgrade to note poller is available to cover the write */ set_write_state(t, GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER, - "initiate_write"); + reason); } break; case GRPC_CHTTP2_WRITE_SCHEDULED: @@ -825,7 +825,7 @@ void grpc_chttp2_initiate_write(grpc_exec_ctx *exec_ctx, set_write_state(t, covered_by_poller ? GRPC_CHTTP2_WRITING_STALE_WITH_POLLER : GRPC_CHTTP2_WRITING_STALE_NO_POLLER, - "initiate_write"); + reason); break; case GRPC_CHTTP2_WRITING_STALE_WITH_POLLER: /* nothing to do: write already requested */ @@ -834,7 +834,7 @@ void grpc_chttp2_initiate_write(grpc_exec_ctx *exec_ctx, if (covered_by_poller) { /* upgrade to note poller is available to cover the write */ set_write_state(t, GRPC_CHTTP2_WRITING_STALE_WITH_POLLER, - "initiate_write"); + reason); } break; } @@ -881,11 +881,11 @@ static void initiate_writing(grpc_exec_ctx *exec_ctx, void *arg, void grpc_chttp2_become_writable(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global, - bool covered_by_poller) { + bool covered_by_poller, const char *reason) { if (!TRANSPORT_FROM_GLOBAL(transport_global)->closed && grpc_chttp2_list_add_writable_stream(transport_global, stream_global)) { GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing"); - grpc_chttp2_initiate_write(exec_ctx, transport_global, covered_by_poller); + grpc_chttp2_initiate_write(exec_ctx, transport_global, covered_by_poller, reason); } } @@ -901,7 +901,7 @@ static void push_setting(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, if (use_value != t->global.settings[GRPC_LOCAL_SETTINGS][id]) { t->global.settings[GRPC_LOCAL_SETTINGS][id] = use_value; t->global.dirtied_local_settings = 1; - grpc_chttp2_initiate_write(exec_ctx, &t->global, false); + grpc_chttp2_initiate_write(exec_ctx, &t->global, false, "push_setting"); } } @@ -1040,7 +1040,7 @@ static void maybe_start_some_streams( stream_global->in_stream_map = true; transport_global->concurrent_stream_count++; grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, - true); + true, "new_stream"); } /* cancel out streams that will never be started */ while (transport_global->next_stream_id >= MAX_CLIENT_STREAM_ID && @@ -1176,7 +1176,7 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, } else { GPR_ASSERT(stream_global->id != 0); grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, - true); + true, "op.send_initial_metadata"); } } else { stream_global->send_trailing_metadata = NULL; @@ -1202,7 +1202,7 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, stream_global->send_message = op->send_message; if (stream_global->id != 0) { grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, - true); + true, "op.send_message"); } } } @@ -1247,7 +1247,7 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, /* TODO(ctiller): check if there's flow control for any outstanding bytes before going writable */ grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, - true); + true, "op.send_trailing_metadata"); } } } @@ -1313,7 +1313,7 @@ static void send_ping_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, p->id[7] = (uint8_t)(t->global.ping_counter & 0xff); p->on_recv = on_recv; gpr_slice_buffer_add(&t->global.qbuf, grpc_chttp2_ping_create(0, p->id)); - grpc_chttp2_initiate_write(exec_ctx, &t->global, true); + grpc_chttp2_initiate_write(exec_ctx, &t->global, true, "send_ping"); } static void ack_ping_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, @@ -1373,7 +1373,7 @@ static void perform_transport_op_locked(grpc_exec_ctx *exec_ctx, close_transport = grpc_chttp2_has_streams(t) ? GRPC_ERROR_NONE : GRPC_ERROR_CREATE("GOAWAY sent"); - grpc_chttp2_initiate_write(exec_ctx, &t->global, false); + grpc_chttp2_initiate_write(exec_ctx, &t->global, false, "goaway_sent"); } if (op->set_accept_stream) { @@ -1578,7 +1578,7 @@ static void cancel_from_api(grpc_exec_ctx *exec_ctx, &transport_global->qbuf, grpc_chttp2_rst_stream_create(stream_global->id, (uint32_t)http_error, &stream_global->stats.outgoing)); - grpc_chttp2_initiate_write(exec_ctx, transport_global, false); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false, "rst_stream"); } const char *msg = @@ -1844,7 +1844,7 @@ static void close_from_api(grpc_exec_ctx *exec_ctx, grpc_chttp2_mark_stream_closed(exec_ctx, transport_global, stream_global, 1, 1, error); - grpc_chttp2_initiate_write(exec_ctx, transport_global, false); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false, "close_from_api"); } typedef struct { @@ -1896,7 +1896,7 @@ static void update_global_window(void *args, uint32_t id, void *stream) { if (was_zero && !is_zero) { grpc_chttp2_become_writable(a->exec_ctx, transport_global, stream_global, - true); + true, "update_global_window"); } } @@ -2007,7 +2007,7 @@ static void post_parse_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, /* copy parsing qbuf to global qbuf */ if (t->parsing.qbuf.count > 0) { gpr_slice_buffer_move_into(&t->parsing.qbuf, &t->global.qbuf); - grpc_chttp2_initiate_write(exec_ctx, transport_global, false); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false, "parsing_qbuf"); } /* merge stream lists */ grpc_chttp2_stream_map_move_into(&t->new_stream_map, &t->parsing_stream_map); @@ -2176,6 +2176,7 @@ static void incoming_byte_stream_update_flow_control( if (stream_global->max_recv_bytes < max_recv_bytes) { uint32_t add_max_recv_bytes = max_recv_bytes - stream_global->max_recv_bytes; + gpr_log(GPR_DEBUG, "add_max_recv_bytes:%d", add_max_recv_bytes); GRPC_CHTTP2_FLOW_CREDIT_STREAM("op", transport_global, stream_global, max_recv_bytes, add_max_recv_bytes); GRPC_CHTTP2_FLOW_CREDIT_STREAM("op", transport_global, stream_global, @@ -2187,7 +2188,7 @@ static void incoming_byte_stream_update_flow_control( grpc_chttp2_list_add_unannounced_incoming_window_available(transport_global, stream_global); grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, - false); + false, "read_incoming_stream"); } } @@ -2395,7 +2396,7 @@ static char *format_flowctl_context_var(const char *context, const char *var, if (context == NULL) { *scope = NULL; gpr_asprintf(&buf, "%s(%" PRId64 ")", var, val); - result = gpr_leftpad(buf, ' ', 40); + result = gpr_leftpad(buf, ' ', 60); gpr_free(buf); return result; } @@ -2408,7 +2409,7 @@ static char *format_flowctl_context_var(const char *context, const char *var, gpr_free(tmp); } gpr_asprintf(&buf, "%s.%s(%" PRId64 ")", underscore_pos + 1, var, val); - result = gpr_leftpad(buf, ' ', 40); + result = gpr_leftpad(buf, ' ', 60); gpr_free(buf); return result; } @@ -2441,7 +2442,7 @@ void grpc_chttp2_flowctl_trace(const char *file, int line, const char *phase, tmp_phase = gpr_leftpad(phase, ' ', 8); tmp_scope1 = gpr_leftpad(scope1, ' ', 11); - gpr_asprintf(&prefix, "FLOW %s: %s %s ", phase, clisvr, scope1); + gpr_asprintf(&prefix, "FLOW %s: %s %s ", tmp_phase, clisvr, scope1); gpr_free(tmp_phase); gpr_free(tmp_scope1); diff --git a/src/core/ext/transport/chttp2/transport/internal.h b/src/core/ext/transport/chttp2/transport/internal.h index 776017bee21..2a12afad6ce 100644 --- a/src/core/ext/transport/chttp2/transport/internal.h +++ b/src/core/ext/transport/chttp2/transport/internal.h @@ -542,7 +542,7 @@ struct grpc_chttp2_stream { should be performed. */ void grpc_chttp2_initiate_write(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, - bool covered_by_poller); + bool covered_by_poller, const char *reason); /** Someone is unlocking the transport mutex: check to see if writes are required, and schedule them if so */ @@ -631,8 +631,7 @@ void grpc_chttp2_list_add_writing_stalled_by_transport( grpc_chttp2_transport_writing *transport_writing, grpc_chttp2_stream_writing *stream_writing); void grpc_chttp2_list_flush_writing_stalled_by_transport( - grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_writing *transport_writing, - bool is_window_available); + grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_writing *transport_writing); void grpc_chttp2_list_add_stalled_by_transport( grpc_chttp2_transport_writing *transport_writing, @@ -845,6 +844,6 @@ void grpc_chttp2_ack_ping(grpc_exec_ctx *exec_ctx, void grpc_chttp2_become_writable(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global, - bool covered_by_poller); + bool covered_by_poller, const char *reason); #endif /* GRPC_CORE_EXT_TRANSPORT_CHTTP2_TRANSPORT_INTERNAL_H */ diff --git a/src/core/ext/transport/chttp2/transport/parsing.c b/src/core/ext/transport/chttp2/transport/parsing.c index b42d98b3b06..efc27775f00 100644 --- a/src/core/ext/transport/chttp2/transport/parsing.c +++ b/src/core/ext/transport/chttp2/transport/parsing.c @@ -157,7 +157,7 @@ void grpc_chttp2_publish_reads( while (grpc_chttp2_list_pop_stalled_by_transport(transport_global, &stream_global)) { grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, - false); + false, "transport.read_flow_control"); } } @@ -169,7 +169,7 @@ void grpc_chttp2_publish_reads( announce_incoming_window, announce_bytes); GRPC_CHTTP2_FLOW_CREDIT_TRANSPORT("parsed", transport_parsing, incoming_window, announce_bytes); - grpc_chttp2_initiate_write(exec_ctx, transport_global, false); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false, "global incoming window"); } /* for each stream that saw an update, fixup global state */ @@ -193,7 +193,7 @@ void grpc_chttp2_publish_reads( is_zero = stream_global->outgoing_window <= 0; if (was_zero && !is_zero) { grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, - false); + false, "stream.read_flow_control"); } stream_global->max_recv_bytes -= (uint32_t)GPR_MIN( diff --git a/src/core/ext/transport/chttp2/transport/stream_lists.c b/src/core/ext/transport/chttp2/transport/stream_lists.c index f227a0af982..aaa4768c7b0 100644 --- a/src/core/ext/transport/chttp2/transport/stream_lists.c +++ b/src/core/ext/transport/chttp2/transport/stream_lists.c @@ -337,19 +337,13 @@ void grpc_chttp2_list_add_writing_stalled_by_transport( } void grpc_chttp2_list_flush_writing_stalled_by_transport( - grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_writing *transport_writing, - bool is_window_available) { + grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_writing *transport_writing) { grpc_chttp2_stream *stream; grpc_chttp2_transport *transport = TRANSPORT_FROM_WRITING(transport_writing); while (stream_list_pop(transport, &stream, GRPC_CHTTP2_LIST_WRITING_STALLED_BY_TRANSPORT)) { - if (is_window_available) { - grpc_chttp2_become_writable(exec_ctx, &transport->global, &stream->global, - true); - } else { - grpc_chttp2_list_add_stalled_by_transport(transport_writing, - &stream->writing); - } + grpc_chttp2_list_add_stalled_by_transport(transport_writing, + &stream->writing); GRPC_CHTTP2_STREAM_UNREF(exec_ctx, &stream->global, "chttp2_writing_stalled"); } diff --git a/src/core/ext/transport/chttp2/transport/writing.c b/src/core/ext/transport/chttp2/transport/writing.c index b19f5f068df..cbc57d10ad3 100644 --- a/src/core/ext/transport/chttp2/transport/writing.c +++ b/src/core/ext/transport/chttp2/transport/writing.c @@ -75,9 +75,6 @@ int grpc_chttp2_unlocking_check_writes( GRPC_CHTTP2_FLOW_MOVE_TRANSPORT("write", transport_writing, outgoing_window, transport_global, outgoing_window); - bool is_window_available = transport_writing->outgoing_window > 0; - grpc_chttp2_list_flush_writing_stalled_by_transport( - exec_ctx, transport_writing, is_window_available); /* for each grpc_chttp2_stream that's become writable, frame it's data (according to available window sizes) and add to the output buffer */ @@ -331,6 +328,9 @@ void grpc_chttp2_cleanup_writing( grpc_chttp2_stream_writing *stream_writing; grpc_chttp2_stream_global *stream_global; + grpc_chttp2_list_flush_writing_stalled_by_transport(exec_ctx, + transport_writing); + while (grpc_chttp2_list_pop_written_stream( transport_global, transport_writing, &stream_global, &stream_writing)) { if (stream_writing->sent_initial_metadata) { From e940d30f4cd99616329164cfe58ee80d4dcb824b Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 7 Jul 2016 09:45:49 -0700 Subject: [PATCH 0434/1039] Remove spam --- .../chttp2/transport/chttp2_transport.c | 23 ++++++++++--------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index aeedf417b4a..41506094de1 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -814,8 +814,7 @@ void grpc_chttp2_initiate_write(grpc_exec_ctx *exec_ctx, case GRPC_CHTTP2_WRITE_REQUESTED_NO_POLLER: if (covered_by_poller) { /* upgrade to note poller is available to cover the write */ - set_write_state(t, GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER, - reason); + set_write_state(t, GRPC_CHTTP2_WRITE_REQUESTED_WITH_POLLER, reason); } break; case GRPC_CHTTP2_WRITE_SCHEDULED: @@ -833,8 +832,7 @@ void grpc_chttp2_initiate_write(grpc_exec_ctx *exec_ctx, case GRPC_CHTTP2_WRITING_STALE_NO_POLLER: if (covered_by_poller) { /* upgrade to note poller is available to cover the write */ - set_write_state(t, GRPC_CHTTP2_WRITING_STALE_WITH_POLLER, - reason); + set_write_state(t, GRPC_CHTTP2_WRITING_STALE_WITH_POLLER, reason); } break; } @@ -885,7 +883,8 @@ void grpc_chttp2_become_writable(grpc_exec_ctx *exec_ctx, if (!TRANSPORT_FROM_GLOBAL(transport_global)->closed && grpc_chttp2_list_add_writable_stream(transport_global, stream_global)) { GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing"); - grpc_chttp2_initiate_write(exec_ctx, transport_global, covered_by_poller, reason); + grpc_chttp2_initiate_write(exec_ctx, transport_global, covered_by_poller, + reason); } } @@ -1039,8 +1038,8 @@ static void maybe_start_some_streams( stream_global->id, STREAM_FROM_GLOBAL(stream_global)); stream_global->in_stream_map = true; transport_global->concurrent_stream_count++; - grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, - true, "new_stream"); + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, true, + "new_stream"); } /* cancel out streams that will never be started */ while (transport_global->next_stream_id >= MAX_CLIENT_STREAM_ID && @@ -1578,7 +1577,8 @@ static void cancel_from_api(grpc_exec_ctx *exec_ctx, &transport_global->qbuf, grpc_chttp2_rst_stream_create(stream_global->id, (uint32_t)http_error, &stream_global->stats.outgoing)); - grpc_chttp2_initiate_write(exec_ctx, transport_global, false, "rst_stream"); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false, + "rst_stream"); } const char *msg = @@ -1844,7 +1844,8 @@ static void close_from_api(grpc_exec_ctx *exec_ctx, grpc_chttp2_mark_stream_closed(exec_ctx, transport_global, stream_global, 1, 1, error); - grpc_chttp2_initiate_write(exec_ctx, transport_global, false, "close_from_api"); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false, + "close_from_api"); } typedef struct { @@ -2007,7 +2008,8 @@ static void post_parse_locked(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, /* copy parsing qbuf to global qbuf */ if (t->parsing.qbuf.count > 0) { gpr_slice_buffer_move_into(&t->parsing.qbuf, &t->global.qbuf); - grpc_chttp2_initiate_write(exec_ctx, transport_global, false, "parsing_qbuf"); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false, + "parsing_qbuf"); } /* merge stream lists */ grpc_chttp2_stream_map_move_into(&t->new_stream_map, &t->parsing_stream_map); @@ -2176,7 +2178,6 @@ static void incoming_byte_stream_update_flow_control( if (stream_global->max_recv_bytes < max_recv_bytes) { uint32_t add_max_recv_bytes = max_recv_bytes - stream_global->max_recv_bytes; - gpr_log(GPR_DEBUG, "add_max_recv_bytes:%d", add_max_recv_bytes); GRPC_CHTTP2_FLOW_CREDIT_STREAM("op", transport_global, stream_global, max_recv_bytes, add_max_recv_bytes); GRPC_CHTTP2_FLOW_CREDIT_STREAM("op", transport_global, stream_global, From ff6cd70a1b4f8e711d45ed1c127192f92a1bc805 Mon Sep 17 00:00:00 2001 From: Muxi Yan Date: Thu, 7 Jul 2016 09:57:54 -0700 Subject: [PATCH 0435/1039] Update error message when client does not enough memory to hold server response --- src/objective-c/GRPCClient/GRPCCall.m | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/objective-c/GRPCClient/GRPCCall.m b/src/objective-c/GRPCClient/GRPCCall.m index 71121094afd..a5c6751c89c 100644 --- a/src/objective-c/GRPCClient/GRPCCall.m +++ b/src/objective-c/GRPCClient/GRPCCall.m @@ -214,7 +214,7 @@ NSString * const kGRPCTrailersKey = @"io.grpc.TrailersKey"; // appropriately-documented code. [weakSelf finishWithError:[NSError errorWithDomain:kGRPCErrorDomain code:GRPCErrorCodeResourceExhausted - userInfo:@{NSLocalizedDescriptionKey: @"Client out of memory."}]]; + userInfo:@{NSLocalizedDescriptionKey: @"Client does not have enough memory to hold the server response."}]]; [weakSelf cancelCall]; return; } From 6c8619bbe7a0eb8ca65782886e8253ebbec87b54 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 7 Jul 2016 10:41:48 -0700 Subject: [PATCH 0436/1039] Better fix for flow control bug --- src/core/ext/transport/chttp2/transport/internal.h | 2 +- src/core/ext/transport/chttp2/transport/parsing.c | 14 ++++++-------- .../ext/transport/chttp2/transport/stream_lists.c | 9 ++++++++- src/core/ext/transport/chttp2/transport/writing.c | 14 ++++++++++++-- test/cpp/end2end/end2end_test.cc | 3 +++ 5 files changed, 30 insertions(+), 12 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/internal.h b/src/core/ext/transport/chttp2/transport/internal.h index 2a12afad6ce..6b47d702aea 100644 --- a/src/core/ext/transport/chttp2/transport/internal.h +++ b/src/core/ext/transport/chttp2/transport/internal.h @@ -630,7 +630,7 @@ int grpc_chttp2_list_pop_check_read_ops( void grpc_chttp2_list_add_writing_stalled_by_transport( grpc_chttp2_transport_writing *transport_writing, grpc_chttp2_stream_writing *stream_writing); -void grpc_chttp2_list_flush_writing_stalled_by_transport( +bool grpc_chttp2_list_flush_writing_stalled_by_transport( grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_writing *transport_writing); void grpc_chttp2_list_add_stalled_by_transport( diff --git a/src/core/ext/transport/chttp2/transport/parsing.c b/src/core/ext/transport/chttp2/transport/parsing.c index efc27775f00..a8ce1db8477 100644 --- a/src/core/ext/transport/chttp2/transport/parsing.c +++ b/src/core/ext/transport/chttp2/transport/parsing.c @@ -87,8 +87,8 @@ void grpc_chttp2_prepare_to_read( transport_global->settings[GRPC_SENT_SETTINGS], sizeof(transport_parsing->last_sent_settings)); transport_parsing->max_frame_size = - transport_global->settings[GRPC_ACKED_SETTINGS] - [GRPC_CHTTP2_SETTINGS_MAX_FRAME_SIZE]; + transport_global + ->settings[GRPC_ACKED_SETTINGS][GRPC_CHTTP2_SETTINGS_MAX_FRAME_SIZE]; /* update the parsing view of incoming window */ while (grpc_chttp2_list_pop_unannounced_incoming_window_available( @@ -154,11 +154,8 @@ void grpc_chttp2_publish_reads( transport_parsing, outgoing_window); is_zero = transport_global->outgoing_window <= 0; if (was_zero && !is_zero) { - while (grpc_chttp2_list_pop_stalled_by_transport(transport_global, - &stream_global)) { - grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, - false, "transport.read_flow_control"); - } + grpc_chttp2_initiate_write(exec_ctx, transport_global, false, + "new_global_flow_control"); } if (transport_parsing->incoming_window < @@ -169,7 +166,8 @@ void grpc_chttp2_publish_reads( announce_incoming_window, announce_bytes); GRPC_CHTTP2_FLOW_CREDIT_TRANSPORT("parsed", transport_parsing, incoming_window, announce_bytes); - grpc_chttp2_initiate_write(exec_ctx, transport_global, false, "global incoming window"); + grpc_chttp2_initiate_write(exec_ctx, transport_global, false, + "global incoming window"); } /* for each stream that saw an update, fixup global state */ diff --git a/src/core/ext/transport/chttp2/transport/stream_lists.c b/src/core/ext/transport/chttp2/transport/stream_lists.c index aaa4768c7b0..2eb5f5f632e 100644 --- a/src/core/ext/transport/chttp2/transport/stream_lists.c +++ b/src/core/ext/transport/chttp2/transport/stream_lists.c @@ -329,6 +329,7 @@ void grpc_chttp2_list_add_writing_stalled_by_transport( grpc_chttp2_transport_writing *transport_writing, grpc_chttp2_stream_writing *stream_writing) { grpc_chttp2_stream *stream = STREAM_FROM_WRITING(stream_writing); + gpr_log(GPR_DEBUG, "writing stalled %d", stream->global.id); if (!stream->included[GRPC_CHTTP2_LIST_WRITING_STALLED_BY_TRANSPORT]) { GRPC_CHTTP2_STREAM_REF(&stream->global, "chttp2_writing_stalled"); } @@ -336,22 +337,28 @@ void grpc_chttp2_list_add_writing_stalled_by_transport( GRPC_CHTTP2_LIST_WRITING_STALLED_BY_TRANSPORT); } -void grpc_chttp2_list_flush_writing_stalled_by_transport( +bool grpc_chttp2_list_flush_writing_stalled_by_transport( grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_writing *transport_writing) { grpc_chttp2_stream *stream; + bool out = false; grpc_chttp2_transport *transport = TRANSPORT_FROM_WRITING(transport_writing); while (stream_list_pop(transport, &stream, GRPC_CHTTP2_LIST_WRITING_STALLED_BY_TRANSPORT)) { + gpr_log(GPR_DEBUG, "move %d from writing stalled to just stalled", + stream->global.id); grpc_chttp2_list_add_stalled_by_transport(transport_writing, &stream->writing); GRPC_CHTTP2_STREAM_UNREF(exec_ctx, &stream->global, "chttp2_writing_stalled"); + out = true; } + return out; } void grpc_chttp2_list_add_stalled_by_transport( grpc_chttp2_transport_writing *transport_writing, grpc_chttp2_stream_writing *stream_writing) { + gpr_log(GPR_DEBUG, "stalled %d", stream_writing->id); stream_list_add(TRANSPORT_FROM_WRITING(transport_writing), STREAM_FROM_WRITING(stream_writing), GRPC_CHTTP2_LIST_STALLED_BY_TRANSPORT); diff --git a/src/core/ext/transport/chttp2/transport/writing.c b/src/core/ext/transport/chttp2/transport/writing.c index cbc57d10ad3..e0d87725e9d 100644 --- a/src/core/ext/transport/chttp2/transport/writing.c +++ b/src/core/ext/transport/chttp2/transport/writing.c @@ -75,6 +75,13 @@ int grpc_chttp2_unlocking_check_writes( GRPC_CHTTP2_FLOW_MOVE_TRANSPORT("write", transport_writing, outgoing_window, transport_global, outgoing_window); + if (transport_writing->outgoing_window > 0) { + while (grpc_chttp2_list_pop_stalled_by_transport(transport_global, + &stream_global)) { + grpc_chttp2_become_writable(exec_ctx, transport_global, stream_global, + false, "transport.read_flow_control"); + } + } /* for each grpc_chttp2_stream that's become writable, frame it's data (according to available window sizes) and add to the output buffer */ @@ -328,8 +335,11 @@ void grpc_chttp2_cleanup_writing( grpc_chttp2_stream_writing *stream_writing; grpc_chttp2_stream_global *stream_global; - grpc_chttp2_list_flush_writing_stalled_by_transport(exec_ctx, - transport_writing); + if (grpc_chttp2_list_flush_writing_stalled_by_transport(exec_ctx, + transport_writing)) { + grpc_chttp2_initiate_write(exec_ctx, transport_global, false, + "resume_stalled_stream"); + } while (grpc_chttp2_list_pop_written_stream( transport_global, transport_writing, &stream_global, &stream_writing)) { diff --git a/test/cpp/end2end/end2end_test.cc b/test/cpp/end2end/end2end_test.cc index 354a59cedd5..0f87ae3e440 100644 --- a/test/cpp/end2end/end2end_test.cc +++ b/test/cpp/end2end/end2end_test.cc @@ -1166,6 +1166,9 @@ TEST_P(ProxyEnd2endTest, HugeResponse) { request.mutable_param()->set_response_message_length(kResponseSize); ClientContext context; + std::chrono::system_clock::time_point deadline = + std::chrono::system_clock::now() + std::chrono::seconds(20); + context.set_deadline(deadline); Status s = stub_->Echo(&context, request, &response); EXPECT_EQ(kResponseSize, response.message().size()); EXPECT_TRUE(s.ok()); From 8e8027bad6c4b4720a27e29178a1431fc069f86a Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 7 Jul 2016 10:42:09 -0700 Subject: [PATCH 0437/1039] clang-format --- src/core/ext/transport/chttp2/transport/parsing.c | 4 ++-- src/core/lib/iomgr/ev_epoll_linux.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/parsing.c b/src/core/ext/transport/chttp2/transport/parsing.c index a8ce1db8477..fbb44ec54ab 100644 --- a/src/core/ext/transport/chttp2/transport/parsing.c +++ b/src/core/ext/transport/chttp2/transport/parsing.c @@ -87,8 +87,8 @@ void grpc_chttp2_prepare_to_read( transport_global->settings[GRPC_SENT_SETTINGS], sizeof(transport_parsing->last_sent_settings)); transport_parsing->max_frame_size = - transport_global - ->settings[GRPC_ACKED_SETTINGS][GRPC_CHTTP2_SETTINGS_MAX_FRAME_SIZE]; + transport_global->settings[GRPC_ACKED_SETTINGS] + [GRPC_CHTTP2_SETTINGS_MAX_FRAME_SIZE]; /* update the parsing view of incoming window */ while (grpc_chttp2_list_pop_unannounced_incoming_window_available( diff --git a/src/core/lib/iomgr/ev_epoll_linux.c b/src/core/lib/iomgr/ev_epoll_linux.c index 0e6cba7e4fe..4282d01a2bf 100644 --- a/src/core/lib/iomgr/ev_epoll_linux.c +++ b/src/core/lib/iomgr/ev_epoll_linux.c @@ -1548,7 +1548,7 @@ retry: * polling_island fields in both fd and pollset to point to the merged * polling island. */ - + if (fd->orphaned) { gpr_mu_unlock(&fd->mu); gpr_mu_unlock(&pollset->mu); From cbff48224916f1a38c3ecc33ddb500a78a3797aa Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 7 Jul 2016 11:00:01 -0700 Subject: [PATCH 0438/1039] Reduce default max message size --- src/core/lib/surface/channel.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/core/lib/surface/channel.c b/src/core/lib/surface/channel.c index 2cf6d8890a3..6d2b1c49352 100644 --- a/src/core/lib/surface/channel.c +++ b/src/core/lib/surface/channel.c @@ -81,7 +81,7 @@ struct grpc_channel { CHANNEL_FROM_CHANNEL_STACK(grpc_channel_stack_from_top_element(top_elem)) /* the protobuf library will (by default) start warning at 100megs */ -#define DEFAULT_MAX_MESSAGE_LENGTH (100 * 1024 * 1024) +#define DEFAULT_MAX_MESSAGE_LENGTH (4 * 1024 * 1024) static void destroy_channel(grpc_exec_ctx *exec_ctx, void *arg, grpc_error *error); From ceb1a7d79f04bcee8c153b43fbe8bef408160537 Mon Sep 17 00:00:00 2001 From: Vijay Pai Date: Thu, 7 Jul 2016 11:06:04 -0700 Subject: [PATCH 0439/1039] Add more information so that we can have a meaningful exit code --- src/proto/grpc/testing/control.proto | 3 +++ test/cpp/qps/driver.cc | 8 ++++++-- test/cpp/qps/driver.h | 2 +- test/cpp/qps/qps_json_driver.cc | 18 +++++++++++++----- 4 files changed, 23 insertions(+), 8 deletions(-) diff --git a/src/proto/grpc/testing/control.proto b/src/proto/grpc/testing/control.proto index 20496a8116b..ece69108158 100644 --- a/src/proto/grpc/testing/control.proto +++ b/src/proto/grpc/testing/control.proto @@ -229,4 +229,7 @@ message ScenarioResult { repeated int32 server_cores = 5; // An after-the-fact computed summary ScenarioResultSummary summary = 6; + // Information on success or failure of each worker + repeated bool client_success = 7; + repeated bool server_success = 8; } diff --git a/test/cpp/qps/driver.cc b/test/cpp/qps/driver.cc index ba38de76f31..7f12ee9c0e4 100644 --- a/test/cpp/qps/driver.cc +++ b/test/cpp/qps/driver.cc @@ -436,6 +436,7 @@ std::unique_ptr RunScenario( for (size_t i = 0; i < num_clients; i++) { auto client = &clients[i]; Status s = client->stream->Finish(); + result->add_client_success(s.ok()); if (!s.ok()) { gpr_log(GPR_ERROR, "Client %zu had an error %s", i, s.error_message().c_str()); @@ -471,6 +472,7 @@ std::unique_ptr RunScenario( for (size_t i = 0; i < num_servers; i++) { auto server = &servers[i]; Status s = server->stream->Finish(); + result->add_server_success(s.ok()); if (!s.ok()) { gpr_log(GPR_ERROR, "Server %zu had an error %s", i, s.error_message().c_str()); @@ -483,8 +485,9 @@ std::unique_ptr RunScenario( return result; } -void RunQuit() { +bool RunQuit() { // Get client, server lists + bool result = true; auto workers = get_workers("QPS_WORKERS"); for (size_t i = 0; i < workers.size(); i++) { auto stub = WorkerService::NewStub( @@ -496,9 +499,10 @@ void RunQuit() { if (!s.ok()) { gpr_log(GPR_ERROR, "Worker %zu could not be properly quit because %s", i, s.error_message().c_str()); - GPR_ASSERT(false); + result = false; } } + return result; } } // namespace testing diff --git a/test/cpp/qps/driver.h b/test/cpp/qps/driver.h index 3a5cf138f11..93f4370cafa 100644 --- a/test/cpp/qps/driver.h +++ b/test/cpp/qps/driver.h @@ -47,7 +47,7 @@ std::unique_ptr RunScenario( const grpc::testing::ServerConfig& server_config, size_t num_servers, int warmup_seconds, int benchmark_seconds, int spawn_local_worker_count); -void RunQuit(); +bool RunQuit(); } // namespace testing } // namespace grpc diff --git a/test/cpp/qps/qps_json_driver.cc b/test/cpp/qps/qps_json_driver.cc index f5d739f893a..1524ebbc389 100644 --- a/test/cpp/qps/qps_json_driver.cc +++ b/test/cpp/qps/qps_json_driver.cc @@ -53,7 +53,7 @@ DEFINE_bool(quit, false, "Quit the workers"); namespace grpc { namespace testing { -static void QpsDriver() { +static bool QpsDriver() { grpc::string json; bool scfile = (FLAGS_scenarios_file != ""); @@ -81,13 +81,13 @@ static void QpsDriver() { } else if (scjson) { json = FLAGS_scenarios_json.c_str(); } else if (FLAGS_quit) { - RunQuit(); - return; + return RunQuit(); } // Parse into an array of scenarios Scenarios scenarios; ParseJson(json.c_str(), "grpc.testing.Scenarios", &scenarios); + bool success = true; // Make sure that there is at least some valid scenario here GPR_ASSERT(scenarios.scenarios_size() > 0); @@ -109,7 +109,15 @@ static void QpsDriver() { GetReporter()->ReportQPSPerCore(*result); GetReporter()->ReportLatency(*result); GetReporter()->ReportTimes(*result); + + for (int i = 0; success && i < result->client_success_size(); i++) { + success = result->client_success(i); + } + for (int i = 0; success && i < result->server_success_size(); i++) { + success = result->server_success(i); + } } + return success; } } // namespace testing @@ -118,7 +126,7 @@ static void QpsDriver() { int main(int argc, char **argv) { grpc::testing::InitBenchmark(&argc, &argv, true); - grpc::testing::QpsDriver(); + bool ok = grpc::testing::QpsDriver(); - return 0; + return ok ? 0 : 1; } From 256cc7aa034f038f8e82f3e278ca61b64252693b Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Thu, 7 Jul 2016 11:09:49 -0700 Subject: [PATCH 0440/1039] Support server reflection in CLI --- Makefile | 8 ++-- build.yaml | 4 ++ test/cpp/util/grpc_cli.cc | 44 ++++++++++--------- test/cpp/util/proto_file_parser.cc | 42 ++++++++++++++++-- test/cpp/util/proto_file_parser.h | 11 +++++ .../proto_reflection_descriptor_database.cc | 12 ++++- tools/run_tests/sources_and_headers.json | 8 +++- .../grpc_cli_libs/grpc_cli_libs.vcxproj | 3 ++ .../grpc_cli_libs.vcxproj.filters | 6 +++ .../vcxproj/test/grpc_cli/grpc_cli.vcxproj | 3 ++ 10 files changed, 111 insertions(+), 30 deletions(-) diff --git a/Makefile b/Makefile index 70119963711..6ba584cbdc5 100644 --- a/Makefile +++ b/Makefile @@ -4140,6 +4140,7 @@ endif LIBGRPC_CLI_LIBS_SRC = \ test/cpp/util/cli_call.cc \ test/cpp/util/proto_file_parser.cc \ + test/cpp/util/proto_reflection_descriptor_database.cc \ PUBLIC_HEADERS_CXX += \ @@ -11079,16 +11080,16 @@ $(BINDIR)/$(CONFIG)/grpc_cli: protobuf_dep_error else -$(BINDIR)/$(CONFIG)/grpc_cli: $(PROTOBUF_DEP) $(GRPC_CLI_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_cli_libs.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_config.a +$(BINDIR)/$(CONFIG)/grpc_cli: $(PROTOBUF_DEP) $(GRPC_CLI_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_cli_libs.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc++_reflection.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_config.a $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` - $(Q) $(LDXX) $(LDFLAGS) $(GRPC_CLI_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_cli_libs.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_config.a $(LDLIBSXX) $(LDLIBS_PROTOBUF) $(LDLIBS) $(LDLIBS_SECURE) $(GTEST_LIB) -o $(BINDIR)/$(CONFIG)/grpc_cli + $(Q) $(LDXX) $(LDFLAGS) $(GRPC_CLI_OBJS) $(LIBDIR)/$(CONFIG)/libgrpc_cli_libs.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc++_reflection.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_config.a $(LDLIBSXX) $(LDLIBS_PROTOBUF) $(LDLIBS) $(LDLIBS_SECURE) $(GTEST_LIB) -o $(BINDIR)/$(CONFIG)/grpc_cli endif endif -$(OBJDIR)/$(CONFIG)/test/cpp/util/grpc_cli.o: $(LIBDIR)/$(CONFIG)/libgrpc_cli_libs.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_config.a +$(OBJDIR)/$(CONFIG)/test/cpp/util/grpc_cli.o: $(LIBDIR)/$(CONFIG)/libgrpc_cli_libs.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc_test_util.a $(LIBDIR)/$(CONFIG)/libgrpc++_reflection.a $(LIBDIR)/$(CONFIG)/libgrpc++.a $(LIBDIR)/$(CONFIG)/libgrpc.a $(LIBDIR)/$(CONFIG)/libgpr_test_util.a $(LIBDIR)/$(CONFIG)/libgpr.a $(LIBDIR)/$(CONFIG)/libgrpc++_test_config.a deps_grpc_cli: $(GRPC_CLI_OBJS:.o=.dep) @@ -15002,6 +15003,7 @@ test/cpp/util/byte_buffer_proto_helper.cc: $(OPENSSL_DEP) test/cpp/util/cli_call.cc: $(OPENSSL_DEP) test/cpp/util/create_test_channel.cc: $(OPENSSL_DEP) test/cpp/util/proto_file_parser.cc: $(OPENSSL_DEP) +test/cpp/util/proto_reflection_descriptor_database.cc: $(OPENSSL_DEP) test/cpp/util/string_ref_helper.cc: $(OPENSSL_DEP) test/cpp/util/subprocess.cc: $(OPENSSL_DEP) test/cpp/util/test_config.cc: $(OPENSSL_DEP) diff --git a/build.yaml b/build.yaml index 1c485fd5c9e..8fabaad7747 100644 --- a/build.yaml +++ b/build.yaml @@ -1028,10 +1028,13 @@ libs: headers: - test/cpp/util/cli_call.h - test/cpp/util/proto_file_parser.h + - test/cpp/util/proto_reflection_descriptor_database.h src: - test/cpp/util/cli_call.cc - test/cpp/util/proto_file_parser.cc + - test/cpp/util/proto_reflection_descriptor_database.cc deps: + - grpc++_reflection - grpc++ - grpc_plugin_support - name: grpc_plugin_support @@ -2669,6 +2672,7 @@ targets: - grpc_cli_libs - grpc++_test_util - grpc_test_util + - grpc++_reflection - grpc++ - grpc - gpr_test_util diff --git a/test/cpp/util/grpc_cli.cc b/test/cpp/util/grpc_cli.cc index c52e48bae65..71470e5214f 100644 --- a/test/cpp/util/grpc_cli.cc +++ b/test/cpp/util/grpc_cli.cc @@ -86,6 +86,7 @@ DEFINE_string(output_binary_file, "", DEFINE_string(metadata, "", "Metadata to send to server, in the form of key1:val1:key2:val2"); DEFINE_string(proto_path, ".", "Path to look for the proto file."); +DEFINE_string(proto_file, "", "Name of the proto file."); void ParseMetadataFlag( std::multimap* client_metadata) { @@ -129,31 +130,47 @@ void PrintMetadata(const T& m, const grpc::string& message) { int main(int argc, char** argv) { grpc::testing::InitTest(&argc, &argv, true); - if (argc < 4 || argc == 5 || grpc::string(argv[1]) != "call") { + if (argc < 4 || grpc::string(argv[1]) != "call") { std::cout << "Usage: grpc_cli call server_host:port method_name " << "[proto file] [text format request] []" << std::endl; + return 1; } - grpc::string file_name; grpc::string request_text; grpc::string server_address(argv[2]); grpc::string method_name(argv[3]); std::unique_ptr parser; grpc::string serialized_request_proto; - if (argc == 6) { - file_name = argv[4]; + if (argc == 5) { // TODO(yangg) read from stdin as well? - request_text = argv[5]; + request_text = argv[4]; } + std::shared_ptr creds; + if (!FLAGS_enable_ssl) { + creds = grpc::InsecureChannelCredentials(); + } else { + if (FLAGS_use_auth) { + creds = grpc::GoogleDefaultCredentials(); + } else { + creds = grpc::SslCredentials(grpc::SslCredentialsOptions()); + } + } + std::shared_ptr channel = + grpc::CreateChannel(server_address, creds); + if (request_text.empty() && FLAGS_input_binary_file.empty()) { std::cout << "Missing input. Use text format input or " << "--input_binary_file for serialized request" << std::endl; return 1; } else if (!request_text.empty()) { - parser.reset(new grpc::testing::ProtoFileParser(FLAGS_proto_path, file_name, - method_name)); + if (!FLAGS_proto_file.empty()) { + parser.reset(new grpc::testing::ProtoFileParser( + FLAGS_proto_path, FLAGS_proto_file, method_name)); + } else { + parser.reset(new grpc::testing::ProtoFileParser(channel, method_name)); + } method_name = parser->GetFullMethodName(); if (parser->HasError()) { return 1; @@ -175,19 +192,6 @@ int main(int argc, char** argv) { } std::cout << "connecting to " << server_address << std::endl; - std::shared_ptr creds; - if (!FLAGS_enable_ssl) { - creds = grpc::InsecureChannelCredentials(); - } else { - if (FLAGS_use_auth) { - creds = grpc::GoogleDefaultCredentials(); - } else { - creds = grpc::SslCredentials(grpc::SslCredentialsOptions()); - } - } - std::shared_ptr channel = - grpc::CreateChannel(server_address, creds); - grpc::string serialized_response_proto; std::multimap client_metadata; std::multimap server_initial_metadata, diff --git a/test/cpp/util/proto_file_parser.cc b/test/cpp/util/proto_file_parser.cc index 25aec329eb3..2ac19858bf7 100644 --- a/test/cpp/util/proto_file_parser.cc +++ b/test/cpp/util/proto_file_parser.cc @@ -95,9 +95,45 @@ ProtoFileParser::ProtoFileParser(const grpc::string& proto_path, dynamic_factory_.reset( new google::protobuf::DynamicMessageFactory(importer_->pool())); + std::vector service_desc_list; + for (int i = 0; i < file_desc->service_count(); i++) { + service_desc_list.push_back(file_desc->service(i)); + } + InitProtoFileParser(method, service_desc_list); +} + +ProtoFileParser::ProtoFileParser(std::shared_ptr channel, + const grpc::string& method) + : has_error_(false), + desc_db_(new grpc::ProtoReflectionDescriptorDatabase(channel)), + desc_pool_(new google::protobuf::DescriptorPool(desc_db_.get())) { + std::vector service_list; + if (!desc_db_->GetServices(&service_list)) { + LogError("Failed to get services"); + } + if (has_error_) { + return; + } + dynamic_factory_.reset( + new google::protobuf::DynamicMessageFactory(desc_pool_.get())); + + std::vector service_desc_list; + for (auto it = service_list.begin(); it != service_list.end(); it++) { + service_desc_list.push_back(desc_pool_->FindServiceByName(*it)); + } + InitProtoFileParser(method, service_desc_list); +} + +ProtoFileParser::~ProtoFileParser() {} + +void ProtoFileParser::InitProtoFileParser( + const grpc::string& method, + const std::vector + service_desc_list) { const google::protobuf::MethodDescriptor* method_descriptor = nullptr; - for (int i = 0; !method_descriptor && i < file_desc->service_count(); i++) { - const auto* service_desc = file_desc->service(i); + for (auto it = service_desc_list.begin(); it != service_desc_list.end(); + it++) { + const auto* service_desc = *it; for (int j = 0; j < service_desc->method_count(); j++) { const auto* method_desc = service_desc->method(j); if (MethodNameMatch(method_desc->full_name(), method)) { @@ -130,8 +166,6 @@ ProtoFileParser::ProtoFileParser(const grpc::string& proto_path, dynamic_factory_->GetPrototype(method_descriptor->output_type())->New()); } -ProtoFileParser::~ProtoFileParser() {} - grpc::string ProtoFileParser::GetSerializedProto( const grpc::string& text_format_proto, bool is_request) { grpc::string serialized; diff --git a/test/cpp/util/proto_file_parser.h b/test/cpp/util/proto_file_parser.h index 46cdd665038..b442d77db98 100644 --- a/test/cpp/util/proto_file_parser.h +++ b/test/cpp/util/proto_file_parser.h @@ -38,8 +38,10 @@ #include #include +#include #include "src/compiler/config.h" +#include "test/cpp/util/proto_reflection_descriptor_database.h" namespace grpc { namespace testing { @@ -53,6 +55,9 @@ class ProtoFileParser { // even just Method. It will log an error if there is ambiguity. ProtoFileParser(const grpc::string& proto_path, const grpc::string& file_name, const grpc::string& method); + + ProtoFileParser(std::shared_ptr channel, + const grpc::string& method); ~ProtoFileParser(); grpc::string GetFullMethodName() const { return full_method_name_; } @@ -68,12 +73,18 @@ class ProtoFileParser { void LogError(const grpc::string& error_msg); private: + void InitProtoFileParser( + const grpc::string& method, + const std::vector services); + bool has_error_; grpc::string request_text_; grpc::string full_method_name_; google::protobuf::compiler::DiskSourceTree source_tree_; std::unique_ptr error_printer_; std::unique_ptr importer_; + std::unique_ptr desc_db_; + std::unique_ptr desc_pool_; std::unique_ptr dynamic_factory_; std::unique_ptr request_prototype_; std::unique_ptr response_prototype_; diff --git a/test/cpp/util/proto_reflection_descriptor_database.cc b/test/cpp/util/proto_reflection_descriptor_database.cc index 25b720aee0a..48998551a59 100644 --- a/test/cpp/util/proto_reflection_descriptor_database.cc +++ b/test/cpp/util/proto_reflection_descriptor_database.cc @@ -53,7 +53,17 @@ ProtoReflectionDescriptorDatabase::ProtoReflectionDescriptorDatabase( std::shared_ptr channel) : stub_(ServerReflection::NewStub(channel)) {} -ProtoReflectionDescriptorDatabase::~ProtoReflectionDescriptorDatabase() {} +ProtoReflectionDescriptorDatabase::~ProtoReflectionDescriptorDatabase() { + if (!stream_) { + GetStream()->WritesDone(); + Status status = stream_->Finish(); + if (!status.ok()) { + gpr_log(GPR_ERROR, + "ServerReflectionInfo rpc failed. Error code: %d, details: %s", + (int)status.error_code(), status.error_message().c_str()); + } + } +} bool ProtoReflectionDescriptorDatabase::FindFileByName( const string& filename, google::protobuf::FileDescriptorProto* output) { diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 6d7cfdaf233..86383fa0ccc 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -2149,6 +2149,7 @@ "gpr_test_util", "grpc", "grpc++", + "grpc++_reflection", "grpc++_test_config", "grpc++_test_util", "grpc_cli_libs", @@ -4479,7 +4480,8 @@ ], "headers": [ "test/cpp/util/cli_call.h", - "test/cpp/util/proto_file_parser.h" + "test/cpp/util/proto_file_parser.h", + "test/cpp/util/proto_reflection_descriptor_database.h" ], "language": "c++", "name": "grpc_cli_libs", @@ -4487,7 +4489,9 @@ "test/cpp/util/cli_call.cc", "test/cpp/util/cli_call.h", "test/cpp/util/proto_file_parser.cc", - "test/cpp/util/proto_file_parser.h" + "test/cpp/util/proto_file_parser.h", + "test/cpp/util/proto_reflection_descriptor_database.cc", + "test/cpp/util/proto_reflection_descriptor_database.h" ], "third_party": false, "type": "lib" diff --git a/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj b/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj index 39cb1e0cb58..03c82f686cc 100644 --- a/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj +++ b/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj @@ -149,12 +149,15 @@ + + + diff --git a/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj.filters b/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj.filters index 55ef18bf306..4add8ed5e13 100644 --- a/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj.filters +++ b/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj.filters @@ -7,6 +7,9 @@ test\cpp\util + + test\cpp\util + @@ -15,6 +18,9 @@ test\cpp\util + + test\cpp\util + diff --git a/vsprojects/vcxproj/test/grpc_cli/grpc_cli.vcxproj b/vsprojects/vcxproj/test/grpc_cli/grpc_cli.vcxproj index cd844d15794..9c8cdc54c25 100644 --- a/vsprojects/vcxproj/test/grpc_cli/grpc_cli.vcxproj +++ b/vsprojects/vcxproj/test/grpc_cli/grpc_cli.vcxproj @@ -173,6 +173,9 @@ {17BCAFC0-5FDC-4C94-AEB9-95F3E220614B} + + {5F575402-3F89-5D1A-6910-9DB8BF5D2BAB} + {C187A093-A0FE-489D-A40A-6E33DE0F9FEB} From 590092ba28d96ce5107b4e588654af77e414d0fd Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 7 Jul 2016 11:30:10 -0700 Subject: [PATCH 0441/1039] Fix windows compilation --- src/core/lib/iomgr/network_status_tracker.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/core/lib/iomgr/network_status_tracker.c b/src/core/lib/iomgr/network_status_tracker.c index d3f0ea53f87..ccbe136db92 100644 --- a/src/core/lib/iomgr/network_status_tracker.c +++ b/src/core/lib/iomgr/network_status_tracker.c @@ -44,7 +44,7 @@ static endpoint_ll_node *head = NULL; static gpr_mu g_endpoint_mutex; static gpr_once g_once_init = GPR_ONCE_INIT; -static void destroy_network_status_monitor() { +static void destroy_network_status_monitor(void) { if (head != NULL) { gpr_log(GPR_ERROR, "Memory leaked as all network endpoints were not shut down"); @@ -52,7 +52,7 @@ static void destroy_network_status_monitor() { gpr_mu_destroy(&g_endpoint_mutex); } -static void initialize_network_status_monitor() { +static void initialize_network_status_monitor(void) { gpr_mu_init(&g_endpoint_mutex); atexit(destroy_network_status_monitor); // TODO(makarandd): Install callback with OS to monitor network status. From c24e0ee4f00a348a2f9948c820cf08be813c5f8a Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Thu, 7 Jul 2016 11:32:25 -0700 Subject: [PATCH 0442/1039] Update docs --- test/cpp/util/grpc_cli.cc | 21 ++++++++++++--------- test/cpp/util/proto_file_parser.cc | 7 +++++-- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/test/cpp/util/grpc_cli.cc b/test/cpp/util/grpc_cli.cc index 71470e5214f..1a9662694ac 100644 --- a/test/cpp/util/grpc_cli.cc +++ b/test/cpp/util/grpc_cli.cc @@ -34,18 +34,21 @@ /* A command line tool to talk to a grpc server. Example of talking to grpc interop server: - grpc_cli call localhost:50051 UnaryCall src/proto/grpc/testing/test.proto \ - "response_size:10" --enable_ssl=false + grpc_cli call localhost:50051 UnaryCall "response_size:10" \ + --proto_file=src/proto/grpc/testing/test.prot --enable_ssl=false Options: - 1. --proto_path, if your proto file is not under current working directory, + 1. --proto_file, use this flag to provide a proto file if the server does + does not have the reflection service. + 2. --proto_path, if your proto file is not under current working directory, use this flag to provide a search root. It should work similar to the - counterpart in protoc. - 2. --metadata specifies metadata to be sent to the server, such as: + counterpart in protoc. This option is valid only when proto_file is + provided. + 3. --metadata specifies metadata to be sent to the server, such as: --metadata="MyHeaderKey1:Value1:MyHeaderKey2:Value2" - 3. --enable_ssl, whether to use tls. - 4. --use_auth, if set to true, attach a GoogleDefaultCredentials to the call - 3. --input_binary_file, a file containing the serialized request. The file + 4. --enable_ssl, whether to use tls. + 5. --use_auth, if set to true, attach a GoogleDefaultCredentials to the call + 6. --input_binary_file, a file containing the serialized request. The file can be generated by calling something like: protoc --proto_path=src/proto/grpc/testing/ \ --encode=grpc.testing.SimpleRequest \ @@ -53,7 +56,7 @@ < input.txt > input.bin If this is used and no proto file is provided in the argument list, the method string has to be exact in the form of /package.service/method. - 4. --output_binary_file, a file to write binary format response into, it can + 7. --output_binary_file, a file to write binary format response into, it can be later decoded using protoc: protoc --proto_path=src/proto/grpc/testing/ \ --decode=grpc.testing.SimpleResponse \ diff --git a/test/cpp/util/proto_file_parser.cc b/test/cpp/util/proto_file_parser.cc index 2ac19858bf7..b1bf0471e15 100644 --- a/test/cpp/util/proto_file_parser.cc +++ b/test/cpp/util/proto_file_parser.cc @@ -109,7 +109,10 @@ ProtoFileParser::ProtoFileParser(std::shared_ptr channel, desc_pool_(new google::protobuf::DescriptorPool(desc_db_.get())) { std::vector service_list; if (!desc_db_->GetServices(&service_list)) { - LogError("Failed to get services"); + LogError( + "Failed to get services from the server, " + "it may not have the reflection service.\n" + "Please try to use the --proto_file option to provide a proto file."); } if (has_error_) { return; @@ -177,7 +180,7 @@ grpc::string ProtoFileParser::GetSerializedProto( LogError("Failed to parse text format to proto."); return ""; } - ok = request_prototype_->SerializeToString(&serialized); + ok = msg->SerializeToString(&serialized); if (!ok) { LogError("Failed to serialize proto."); return ""; From a17c8d993d3bc4867fcc9d6fcf42e6bfff9c9cd8 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Thu, 7 Jul 2016 11:40:05 -0700 Subject: [PATCH 0443/1039] Fix typos --- test/cpp/util/grpc_cli.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/cpp/util/grpc_cli.cc b/test/cpp/util/grpc_cli.cc index 1a9662694ac..7c8e2cdc639 100644 --- a/test/cpp/util/grpc_cli.cc +++ b/test/cpp/util/grpc_cli.cc @@ -35,7 +35,7 @@ A command line tool to talk to a grpc server. Example of talking to grpc interop server: grpc_cli call localhost:50051 UnaryCall "response_size:10" \ - --proto_file=src/proto/grpc/testing/test.prot --enable_ssl=false + --proto_file=src/proto/grpc/testing/test.proto --enable_ssl=false Options: 1. --proto_file, use this flag to provide a proto file if the server does From fbf03c17355e65255862e91a938f4f5cfbcee51a Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Thu, 7 Jul 2016 13:10:09 -0700 Subject: [PATCH 0444/1039] Fix high seqno tests, up runtimes for benchmark tests to flush more bugs --- src/core/lib/transport/connectivity_state.c | 3 ++ test/core/end2end/tests/high_initial_seqno.c | 6 ++++ test/cpp/qps/gen_build_yaml.py | 2 +- tools/run_tests/tests.json | 32 ++++++++++---------- 4 files changed, 26 insertions(+), 17 deletions(-) diff --git a/src/core/lib/transport/connectivity_state.c b/src/core/lib/transport/connectivity_state.c index 054f112127b..68d05e3a858 100644 --- a/src/core/lib/transport/connectivity_state.c +++ b/src/core/lib/transport/connectivity_state.c @@ -179,6 +179,9 @@ void grpc_connectivity_state_set(grpc_exec_ctx *exec_ctx, while ((w = tracker->watchers) != NULL) { *w->current = tracker->current_state; tracker->watchers = w->next; + if (grpc_connectivity_state_trace) { + gpr_log(GPR_DEBUG, "NOTIFY: %p", w->notify); + } grpc_exec_ctx_sched(exec_ctx, w->notify, GRPC_ERROR_REF(tracker->current_error), NULL); gpr_free(w); diff --git a/test/core/end2end/tests/high_initial_seqno.c b/test/core/end2end/tests/high_initial_seqno.c index 50e3c9cb898..db45f5eb5ad 100644 --- a/test/core/end2end/tests/high_initial_seqno.c +++ b/test/core/end2end/tests/high_initial_seqno.c @@ -203,6 +203,12 @@ static void simple_request_body(grpc_end2end_test_fixture f) { grpc_call_destroy(c); grpc_call_destroy(s); + /* TODO(ctiller): this rate limits the test, and it should be removed when + retry has been implemented; until then cross-thread chatter + may result in some requests needing to be cancelled due to + seqno exhaustion. */ + cq_verify_empty(cqv); + cq_verifier_destroy(cqv); } diff --git a/test/cpp/qps/gen_build_yaml.py b/test/cpp/qps/gen_build_yaml.py index 34b81514411..255968256e3 100755 --- a/test/cpp/qps/gen_build_yaml.py +++ b/test/cpp/qps/gen_build_yaml.py @@ -46,7 +46,7 @@ import performance.scenario_config as scenario_config def _scenario_json_string(scenario_json): # tweak parameters to get fast test times scenario_json['warmup_seconds'] = 1 - scenario_json['benchmark_seconds'] = 1 + scenario_json['benchmark_seconds'] = 10 return json.dumps(scenario_config.remove_nonproto_fields(scenario_json)) def threads_of_type(scenario_json, path): diff --git a/tools/run_tests/tests.json b/tools/run_tests/tests.json index 9c38b7e47af..71641cebc95 100644 --- a/tools/run_tests/tests.json +++ b/tools/run_tests/tests.json @@ -27135,7 +27135,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_generic_async_streaming_ping_pong_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" + "'{\"name\": \"cpp_generic_async_streaming_ping_pong_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" ], "boringssl": true, "ci_platforms": [ @@ -27161,7 +27161,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_protobuf_async_streaming_ping_pong_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" + "'{\"name\": \"cpp_protobuf_async_streaming_ping_pong_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" ], "boringssl": true, "ci_platforms": [ @@ -27187,7 +27187,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_protobuf_async_unary_ping_pong_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" + "'{\"name\": \"cpp_protobuf_async_unary_ping_pong_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" ], "boringssl": true, "ci_platforms": [ @@ -27213,7 +27213,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_protobuf_sync_unary_ping_pong_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"server_type\": \"SYNC_SERVER\"}, \"client_config\": {\"client_type\": \"SYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" + "'{\"name\": \"cpp_protobuf_sync_unary_ping_pong_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"server_type\": \"SYNC_SERVER\"}, \"client_config\": {\"client_type\": \"SYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" ], "boringssl": true, "ci_platforms": [ @@ -27239,7 +27239,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_protobuf_async_unary_qps_unconstrained_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" + "'{\"name\": \"cpp_protobuf_async_unary_qps_unconstrained_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" ], "boringssl": true, "ci_platforms": [ @@ -27265,7 +27265,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_protobuf_async_streaming_qps_unconstrained_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" + "'{\"name\": \"cpp_protobuf_async_streaming_qps_unconstrained_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" ], "boringssl": true, "ci_platforms": [ @@ -27291,7 +27291,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_generic_async_streaming_qps_unconstrained_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" + "'{\"name\": \"cpp_generic_async_streaming_qps_unconstrained_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" ], "boringssl": true, "ci_platforms": [ @@ -27317,7 +27317,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_generic_async_streaming_qps_one_server_core_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" + "'{\"name\": \"cpp_generic_async_streaming_qps_one_server_core_secure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": {\"use_test_ca\": true, \"server_host_override\": \"foo.test.google.fr\"}, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" ], "boringssl": true, "ci_platforms": [ @@ -27343,7 +27343,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_generic_async_streaming_ping_pong_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" + "'{\"name\": \"cpp_generic_async_streaming_ping_pong_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" ], "boringssl": true, "ci_platforms": [ @@ -27369,7 +27369,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_protobuf_async_streaming_ping_pong_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": null, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" + "'{\"name\": \"cpp_protobuf_async_streaming_ping_pong_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": null, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" ], "boringssl": true, "ci_platforms": [ @@ -27395,7 +27395,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_protobuf_async_unary_ping_pong_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": null, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" + "'{\"name\": \"cpp_protobuf_async_unary_ping_pong_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": null, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" ], "boringssl": true, "ci_platforms": [ @@ -27421,7 +27421,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_protobuf_sync_unary_ping_pong_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": null, \"server_type\": \"SYNC_SERVER\"}, \"client_config\": {\"client_type\": \"SYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" + "'{\"name\": \"cpp_protobuf_sync_unary_ping_pong_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": null, \"server_type\": \"SYNC_SERVER\"}, \"client_config\": {\"client_type\": \"SYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 1, \"async_client_threads\": 1, \"outstanding_rpcs_per_channel\": 1, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 1}'" ], "boringssl": true, "ci_platforms": [ @@ -27447,7 +27447,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_protobuf_async_unary_qps_unconstrained_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": null, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" + "'{\"name\": \"cpp_protobuf_async_unary_qps_unconstrained_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": null, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"UNARY\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" ], "boringssl": true, "ci_platforms": [ @@ -27473,7 +27473,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_protobuf_async_streaming_qps_unconstrained_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": null, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" + "'{\"name\": \"cpp_protobuf_async_streaming_qps_unconstrained_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": null, \"server_type\": \"ASYNC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"simple_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" ], "boringssl": true, "ci_platforms": [ @@ -27499,7 +27499,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_generic_async_streaming_qps_unconstrained_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" + "'{\"name\": \"cpp_generic_async_streaming_qps_unconstrained_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 0, \"core_limit\": 0, \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" ], "boringssl": true, "ci_platforms": [ @@ -27525,7 +27525,7 @@ { "args": [ "--scenario_json", - "'{\"name\": \"cpp_generic_async_streaming_qps_one_server_core_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 1, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" + "'{\"name\": \"cpp_generic_async_streaming_qps_one_server_core_insecure\", \"warmup_seconds\": 1, \"benchmark_seconds\": 10, \"num_servers\": 1, \"server_config\": {\"async_server_threads\": 1, \"core_limit\": 1, \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"server_type\": \"ASYNC_GENERIC_SERVER\"}, \"client_config\": {\"client_type\": \"ASYNC_CLIENT\", \"security_params\": null, \"payload_config\": {\"bytebuf_params\": {\"resp_size\": 0, \"req_size\": 0}}, \"client_channels\": 64, \"async_client_threads\": 0, \"outstanding_rpcs_per_channel\": 100, \"rpc_type\": \"STREAMING\", \"load_params\": {\"closed_loop\": {}}, \"histogram_params\": {\"max_possible\": 60000000000.0, \"resolution\": 0.01}}, \"num_clients\": 0}'" ], "boringssl": true, "ci_platforms": [ From 39070fe3a536819708aaa4422ea912b303cbe30f Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Thu, 7 Jul 2016 13:57:12 -0700 Subject: [PATCH 0445/1039] Rerun generate_projects.sh --- tools/run_tests/sources_and_headers.json | 1 + vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj | 3 +++ 2 files changed, 4 insertions(+) diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 86383fa0ccc..141c8982667 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -4476,6 +4476,7 @@ { "deps": [ "grpc++", + "grpc++_reflection", "grpc_plugin_support" ], "headers": [ diff --git a/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj b/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj index 03c82f686cc..d25c692e3e8 100644 --- a/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj +++ b/vsprojects/vcxproj/grpc_cli_libs/grpc_cli_libs.vcxproj @@ -160,6 +160,9 @@ + + {5F575402-3F89-5D1A-6910-9DB8BF5D2BAB} + {C187A093-A0FE-489D-A40A-6E33DE0F9FEB} From a6b2a5a090ff057f77976b22d854ff375883c1b9 Mon Sep 17 00:00:00 2001 From: Nathaniel Manista Date: Thu, 7 Jul 2016 21:06:17 +0000 Subject: [PATCH 0446/1039] Make handlers optional at server construction --- src/python/grpcio/grpc/__init__.py | 16 +++++++--------- src/python/grpcio/grpc/_server.py | 2 +- .../grpcio/grpc/beta/_server_adaptations.py | 3 ++- .../tests/protoc_plugin/_python_plugin_test.py | 4 ++-- .../tests/unit/_channel_connectivity_test.py | 4 ++-- .../tests/unit/_channel_ready_future_test.py | 2 +- .../grpcio_tests/tests/unit/_compression_test.py | 3 ++- .../tests/unit/_empty_message_test.py | 3 ++- .../tests/unit/_metadata_code_details_test.py | 2 +- .../grpcio_tests/tests/unit/_metadata_test.py | 4 ++-- src/python/grpcio_tests/tests/unit/_rpc_test.py | 2 +- 11 files changed, 23 insertions(+), 22 deletions(-) diff --git a/src/python/grpcio/grpc/__init__.py b/src/python/grpcio/grpc/__init__.py index b3eeaad1f73..f7d51d2df7e 100644 --- a/src/python/grpcio/grpc/__init__.py +++ b/src/python/grpcio/grpc/__init__.py @@ -1207,25 +1207,23 @@ def secure_channel(target, credentials, options=None): return _channel.Channel(target, options, credentials._credentials) -def server(generic_rpc_handlers, thread_pool, options=None): +def server(thread_pool, handlers=None): """Creates a Server with which RPCs can be serviced. - The GenericRpcHandlers passed to this function needn't be the only - GenericRpcHandlers that will be used to serve RPCs; others may be added later - by calling add_generic_rpc_handlers any time before the returned server is - started. - Args: - generic_rpc_handlers: Some number of GenericRpcHandlers that will be used - to service RPCs after the returned Server is started. thread_pool: A futures.ThreadPoolExecutor to be used by the returned Server to service RPCs. + handlers: An optional sequence of GenericRpcHandlers to be used to service + RPCs after the returned Server is started. These handlers need not be the + only handlers the returned Server will use to service RPCs; other + handlers may later be added to the returned Server by calling its + add_generic_rpc_handlers method any time before it is started. Returns: A Server with which RPCs can be serviced. """ from grpc import _server - return _server.Server(generic_rpc_handlers, thread_pool) + return _server.Server(thread_pool, () if handlers is None else handlers) ################################### __all__ ################################# diff --git a/src/python/grpcio/grpc/_server.py b/src/python/grpcio/grpc/_server.py index f4c114056fe..f8a9d5afd30 100644 --- a/src/python/grpcio/grpc/_server.py +++ b/src/python/grpcio/grpc/_server.py @@ -731,7 +731,7 @@ def _start(state): class Server(grpc.Server): - def __init__(self, generic_handlers, thread_pool): + def __init__(self, thread_pool, generic_handlers): completion_queue = cygrpc.CompletionQueue() server = cygrpc.Server() server.register_completion_queue(completion_queue) diff --git a/src/python/grpcio/grpc/beta/_server_adaptations.py b/src/python/grpcio/grpc/beta/_server_adaptations.py index 1e1f80156ac..cca4a1797a1 100644 --- a/src/python/grpcio/grpc/beta/_server_adaptations.py +++ b/src/python/grpcio/grpc/beta/_server_adaptations.py @@ -371,4 +371,5 @@ def server( _DEFAULT_POOL_SIZE if thread_pool_size is None else thread_pool_size) else: effective_thread_pool = thread_pool - return _Server(grpc.server((generic_rpc_handler,), effective_thread_pool)) + return _Server( + grpc.server(effective_thread_pool, handlers=(generic_rpc_handler,))) diff --git a/src/python/grpcio_tests/tests/protoc_plugin/_python_plugin_test.py b/src/python/grpcio_tests/tests/protoc_plugin/_python_plugin_test.py index bf09380c85b..7ca2bcff383 100644 --- a/src/python/grpcio_tests/tests/protoc_plugin/_python_plugin_test.py +++ b/src/python/grpcio_tests/tests/protoc_plugin/_python_plugin_test.py @@ -171,7 +171,7 @@ def _CreateService(): return servicer_methods.HalfDuplexCall(request_iter, context) server = grpc.server( - (), futures.ThreadPoolExecutor(max_workers=test_constants.POOL_SIZE)) + futures.ThreadPoolExecutor(max_workers=test_constants.POOL_SIZE)) getattr(service_pb2, ADD_SERVICER_TO_SERVER_IDENTIFIER)(Servicer(), server) port = server.add_insecure_port('[::]:0') server.start() @@ -192,7 +192,7 @@ def _CreateIncompleteService(): pass server = grpc.server( - (), futures.ThreadPoolExecutor(max_workers=test_constants.POOL_SIZE)) + futures.ThreadPoolExecutor(max_workers=test_constants.POOL_SIZE)) getattr(service_pb2, ADD_SERVICER_TO_SERVER_IDENTIFIER)(Servicer(), server) port = server.add_insecure_port('[::]:0') server.start() diff --git a/src/python/grpcio_tests/tests/unit/_channel_connectivity_test.py b/src/python/grpcio_tests/tests/unit/_channel_connectivity_test.py index ae8de523ecf..3c00f686cec 100644 --- a/src/python/grpcio_tests/tests/unit/_channel_connectivity_test.py +++ b/src/python/grpcio_tests/tests/unit/_channel_connectivity_test.py @@ -104,7 +104,7 @@ class ChannelConnectivityTest(unittest.TestCase): grpc.ChannelConnectivity.READY, fifth_connectivities) def test_immediately_connectable_channel_connectivity(self): - server = _server.Server((), futures.ThreadPoolExecutor(max_workers=0)) + server = _server.Server(futures.ThreadPoolExecutor(max_workers=0), ()) port = server.add_insecure_port('[::]:0') server.start() first_callback = _Callback() @@ -143,7 +143,7 @@ class ChannelConnectivityTest(unittest.TestCase): grpc.ChannelConnectivity.SHUTDOWN, fourth_connectivities) def test_reachable_then_unreachable_channel_connectivity(self): - server = _server.Server((), futures.ThreadPoolExecutor(max_workers=0)) + server = _server.Server(futures.ThreadPoolExecutor(max_workers=0), ()) port = server.add_insecure_port('[::]:0') server.start() callback = _Callback() diff --git a/src/python/grpcio_tests/tests/unit/_channel_ready_future_test.py b/src/python/grpcio_tests/tests/unit/_channel_ready_future_test.py index b84bc0197a9..e8982ed2ded 100644 --- a/src/python/grpcio_tests/tests/unit/_channel_ready_future_test.py +++ b/src/python/grpcio_tests/tests/unit/_channel_ready_future_test.py @@ -78,7 +78,7 @@ class ChannelReadyFutureTest(unittest.TestCase): self.assertFalse(ready_future.running()) def test_immediately_connectable_channel_connectivity(self): - server = _server.Server((), futures.ThreadPoolExecutor(max_workers=0)) + server = _server.Server(futures.ThreadPoolExecutor(max_workers=0), ()) port = server.add_insecure_port('[::]:0') server.start() channel = grpc.insecure_channel('localhost:{}'.format(port)) diff --git a/src/python/grpcio_tests/tests/unit/_compression_test.py b/src/python/grpcio_tests/tests/unit/_compression_test.py index 9e8b8578c1e..83b91094666 100644 --- a/src/python/grpcio_tests/tests/unit/_compression_test.py +++ b/src/python/grpcio_tests/tests/unit/_compression_test.py @@ -88,7 +88,8 @@ class CompressionTest(unittest.TestCase): def setUp(self): self._server_pool = logging_pool.pool(test_constants.THREAD_CONCURRENCY) - self._server = grpc.server((_GenericHandler(),), self._server_pool) + self._server = grpc.server( + self._server_pool, handlers=(_GenericHandler(),)) self._port = self._server.add_insecure_port('[::]:0') self._server.start() diff --git a/src/python/grpcio_tests/tests/unit/_empty_message_test.py b/src/python/grpcio_tests/tests/unit/_empty_message_test.py index 8c7d697728b..131f6e94525 100644 --- a/src/python/grpcio_tests/tests/unit/_empty_message_test.py +++ b/src/python/grpcio_tests/tests/unit/_empty_message_test.py @@ -103,7 +103,8 @@ class EmptyMessageTest(unittest.TestCase): def setUp(self): self._server_pool = logging_pool.pool(test_constants.THREAD_CONCURRENCY) - self._server = grpc.server((_GenericHandler(),), self._server_pool) + self._server = grpc.server( + self._server_pool, handlers=(_GenericHandler(),)) port = self._server.add_insecure_port('[::]:0') self._server.start() self._channel = grpc.insecure_channel('localhost:%d' % port) diff --git a/src/python/grpcio_tests/tests/unit/_metadata_code_details_test.py b/src/python/grpcio_tests/tests/unit/_metadata_code_details_test.py index 0fd02d2a227..fb3e5477815 100644 --- a/src/python/grpcio_tests/tests/unit/_metadata_code_details_test.py +++ b/src/python/grpcio_tests/tests/unit/_metadata_code_details_test.py @@ -189,7 +189,7 @@ class MetadataCodeDetailsTest(unittest.TestCase): self._servicer = _Servicer() self._server_pool = logging_pool.pool(test_constants.THREAD_CONCURRENCY) self._server = grpc.server( - (_generic_handler(self._servicer),), self._server_pool) + self._server_pool, handlers=(_generic_handler(self._servicer),)) port = self._server.add_insecure_port('[::]:0') self._server.start() diff --git a/src/python/grpcio_tests/tests/unit/_metadata_test.py b/src/python/grpcio_tests/tests/unit/_metadata_test.py index c637a28039d..da734769299 100644 --- a/src/python/grpcio_tests/tests/unit/_metadata_test.py +++ b/src/python/grpcio_tests/tests/unit/_metadata_test.py @@ -161,8 +161,8 @@ class MetadataTest(unittest.TestCase): def setUp(self): self._server_pool = logging_pool.pool(test_constants.THREAD_CONCURRENCY) - self._server = grpc.server((_GenericHandler(weakref.proxy(self)),), - self._server_pool) + self._server = grpc.server( + self._server_pool, handlers=(_GenericHandler(weakref.proxy(self)),)) port = self._server.add_insecure_port('[::]:0') self._server.start() self._channel = grpc.insecure_channel('localhost:%d' % port, diff --git a/src/python/grpcio_tests/tests/unit/_rpc_test.py b/src/python/grpcio_tests/tests/unit/_rpc_test.py index c70d65a6dfb..59bf240d286 100644 --- a/src/python/grpcio_tests/tests/unit/_rpc_test.py +++ b/src/python/grpcio_tests/tests/unit/_rpc_test.py @@ -184,7 +184,7 @@ class RPCTest(unittest.TestCase): self._handler = _Handler(self._control) self._server_pool = logging_pool.pool(test_constants.THREAD_CONCURRENCY) - self._server = grpc.server((), self._server_pool) + self._server = grpc.server(self._server_pool) port = self._server.add_insecure_port('[::]:0') self._server.add_generic_rpc_handlers((_GenericHandler(self._handler),)) self._server.start() From 22869a00fd19459f936ca31ea31fc3c8d16abfd7 Mon Sep 17 00:00:00 2001 From: Nathaniel Manista Date: Thu, 7 Jul 2016 21:10:56 +0000 Subject: [PATCH 0447/1039] Composition of arbitrarily many CallCredentials --- src/python/grpcio/grpc/__init__.py | 34 +++++---- .../grpcio/grpc/_credential_composition.py | 48 +++++++++++++ src/python/grpcio_tests/tests/tests.json | 1 + .../tests/unit/_credentials_test.py | 72 +++++++++++++++++++ 4 files changed, 140 insertions(+), 15 deletions(-) create mode 100644 src/python/grpcio/grpc/_credential_composition.py create mode 100644 src/python/grpcio_tests/tests/unit/_credentials_test.py diff --git a/src/python/grpcio/grpc/__init__.py b/src/python/grpcio/grpc/__init__.py index b3eeaad1f73..0efd233d79a 100644 --- a/src/python/grpcio/grpc/__init__.py +++ b/src/python/grpcio/grpc/__init__.py @@ -1091,37 +1091,41 @@ def access_token_call_credentials(access_token): _auth.AccessTokenCallCredentials(access_token)) -def composite_call_credentials(call_credentials, additional_call_credentials): - """Compose two CallCredentials to make a new one. +def composite_call_credentials(*call_credentials): + """Compose multiple CallCredentials to make a new CallCredentials. Args: - call_credentials: A CallCredentials object. - additional_call_credentials: Another CallCredentials object to compose on - top of call_credentials. + *call_credentials: At least two CallCredentials objects. Returns: - A new CallCredentials composed of the two given CallCredentials. + A CallCredentials object composed of the given CallCredentials objects. """ + from grpc import _credential_composition + cygrpc_call_credentials = tuple( + single_call_credentials._credentials + for single_call_credentials in call_credentials) return CallCredentials( - _cygrpc.call_credentials_composite( - call_credentials._credentials, - additional_call_credentials._credentials)) + _credential_composition.call(cygrpc_call_credentials)) -def composite_channel_credentials(channel_credentials, call_credentials): - """Compose a ChannelCredentials and a CallCredentials. +def composite_channel_credentials(channel_credentials, *call_credentials): + """Compose a ChannelCredentials and one or more CallCredentials objects. Args: channel_credentials: A ChannelCredentials. - call_credentials: A CallCredentials. + *call_credentials: One or more CallCredentials objects. Returns: A ChannelCredentials composed of the given ChannelCredentials and - CallCredentials. + CallCredentials objects. """ + from grpc import _credential_composition + cygrpc_call_credentials = tuple( + single_call_credentials._credentials + for single_call_credentials in call_credentials) return ChannelCredentials( - _cygrpc.channel_credentials_composite( - channel_credentials._credentials, call_credentials._credentials)) + _credential_composition.channel( + channel_credentials._credentials, cygrpc_call_credentials)) def ssl_server_credentials( diff --git a/src/python/grpcio/grpc/_credential_composition.py b/src/python/grpcio/grpc/_credential_composition.py new file mode 100644 index 00000000000..9cb5508e27c --- /dev/null +++ b/src/python/grpcio/grpc/_credential_composition.py @@ -0,0 +1,48 @@ +# Copyright 2016, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from grpc._cython import cygrpc + + +def _call(call_credentialses): + call_credentials_iterator = iter(call_credentialses) + composition = next(call_credentials_iterator) + for additional_call_credentials in call_credentials_iterator: + composition = cygrpc.call_credentials_composite( + composition, additional_call_credentials) + return composition + + +def call(call_credentialses): + return _call(call_credentialses) + + +def channel(channel_credentials, call_credentialses): + return cygrpc.channel_credentials_composite( + channel_credentials, _call(call_credentialses)) diff --git a/src/python/grpcio_tests/tests/tests.json b/src/python/grpcio_tests/tests/tests.json index 45eb75b242c..dcaef0db1fa 100644 --- a/src/python/grpcio_tests/tests/tests.json +++ b/src/python/grpcio_tests/tests/tests.json @@ -12,6 +12,7 @@ "_channel_test.ChannelTest", "_compression_test.CompressionTest", "_connectivity_channel_test.ConnectivityStatesTest", + "_credentials_test.CredentialsTest", "_empty_message_test.EmptyMessageTest", "_exit_test.ExitTest", "_face_interface_test.DynamicInvokerBlockingInvocationInlineServiceTest", diff --git a/src/python/grpcio_tests/tests/unit/_credentials_test.py b/src/python/grpcio_tests/tests/unit/_credentials_test.py new file mode 100644 index 00000000000..87af85a0b9b --- /dev/null +++ b/src/python/grpcio_tests/tests/unit/_credentials_test.py @@ -0,0 +1,72 @@ +# Copyright 2016, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +"""Tests of credentials.""" + +import unittest + +import grpc + + +class CredentialsTest(unittest.TestCase): + + def test_call_credentials_composition(self): + first = grpc.access_token_call_credentials('abc') + second = grpc.access_token_call_credentials('def') + third = grpc.access_token_call_credentials('ghi') + + first_and_second = grpc.composite_call_credentials(first, second) + first_second_and_third = grpc.composite_call_credentials( + first, second, third) + + self.assertIsInstance(first_and_second, grpc.CallCredentials) + self.assertIsInstance(first_second_and_third, grpc.CallCredentials) + + def test_channel_credentials_composition(self): + first_call_credentials = grpc.access_token_call_credentials('abc') + second_call_credentials = grpc.access_token_call_credentials('def') + third_call_credentials = grpc.access_token_call_credentials('ghi') + channel_credentials = grpc.ssl_channel_credentials() + + channel_and_first = grpc.composite_channel_credentials( + channel_credentials, first_call_credentials) + channel_first_and_second = grpc.composite_channel_credentials( + channel_credentials, first_call_credentials, second_call_credentials) + channel_first_second_and_third = grpc.composite_channel_credentials( + channel_credentials, first_call_credentials, second_call_credentials, + third_call_credentials) + + self.assertIsInstance(channel_and_first, grpc.ChannelCredentials) + self.assertIsInstance(channel_first_and_second, grpc.ChannelCredentials) + self.assertIsInstance( + channel_first_second_and_third, grpc.ChannelCredentials) + + +if __name__ == '__main__': + unittest.main(verbosity=2) From 1c58bd221da3f67cc612149b99a7efe49bb884cb Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Thu, 7 Jul 2016 14:17:32 -0700 Subject: [PATCH 0448/1039] Enable server reflection in c++ examples --- examples/cpp/helloworld/Makefile | 4 +++- examples/cpp/route_guide/Makefile | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/examples/cpp/helloworld/Makefile b/examples/cpp/helloworld/Makefile index 780e5e427a7..a632f1b1262 100644 --- a/examples/cpp/helloworld/Makefile +++ b/examples/cpp/helloworld/Makefile @@ -32,7 +32,9 @@ CXX = g++ CPPFLAGS += -I/usr/local/include -pthread CXXFLAGS += -std=c++11 -LDFLAGS += -L/usr/local/lib `pkg-config --libs grpc++ grpc` -lprotobuf -lpthread -ldl +LDFLAGS += -L/usr/local/lib `pkg-config --libs grpc++ grpc` \ + -Wl,--no-as-needed -lgrpc++_reflection -Wl,--as-needed \ + -lprotobuf -lpthread -ldl PROTOC = protoc GRPC_CPP_PLUGIN = grpc_cpp_plugin GRPC_CPP_PLUGIN_PATH ?= `which $(GRPC_CPP_PLUGIN)` diff --git a/examples/cpp/route_guide/Makefile b/examples/cpp/route_guide/Makefile index 11f2a00cc89..237152ecd32 100644 --- a/examples/cpp/route_guide/Makefile +++ b/examples/cpp/route_guide/Makefile @@ -32,7 +32,9 @@ CXX = g++ CPPFLAGS += -I/usr/local/include -pthread CXXFLAGS += -std=c++11 -LDFLAGS += -L/usr/local/lib `pkg-config --libs grpc++` -lprotobuf -lpthread -ldl +LDFLAGS += -L/usr/local/lib `pkg-config --libs grpc++` \ + -Wl,--no-as-needed -lgrpc++_reflection -Wl,--as-needed \ + -lprotobuf -lpthread -ldl PROTOC = protoc GRPC_CPP_PLUGIN = grpc_cpp_plugin GRPC_CPP_PLUGIN_PATH ?= `which $(GRPC_CPP_PLUGIN)` From b156671023197b2aa015325365a0e6723fdf8b44 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Thu, 7 Jul 2016 15:16:03 -0700 Subject: [PATCH 0449/1039] Change cancelled to cancelled? --- src/ruby/lib/grpc/generic/active_call.rb | 8 ++++---- src/ruby/pb/test/client.rb | 4 ++-- src/ruby/spec/generic/active_call_spec.rb | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/src/ruby/lib/grpc/generic/active_call.rb b/src/ruby/lib/grpc/generic/active_call.rb index a3ac0d48a3b..6e411a2b941 100644 --- a/src/ruby/lib/grpc/generic/active_call.rb +++ b/src/ruby/lib/grpc/generic/active_call.rb @@ -120,7 +120,7 @@ module GRPC end # cancelled indicates if the call was cancelled - def cancelled + def cancelled? !@call.status.nil? && @call.status.code == Core::StatusCodes::CANCELLED end @@ -455,17 +455,17 @@ module GRPC # SingleReqView limits access to an ActiveCall's methods for use in server # handlers that receive just one request. - SingleReqView = view_class(:cancelled, :deadline, :metadata, + SingleReqView = view_class(:cancelled?, :deadline, :metadata, :output_metadata, :peer, :peer_cert) # MultiReqView limits access to an ActiveCall's methods for use in # server client_streamer handlers. - MultiReqView = view_class(:cancelled, :deadline, :each_queued_msg, + MultiReqView = view_class(:cancelled?, :deadline, :each_queued_msg, :each_remote_read, :metadata, :output_metadata) # Operation limits access to an ActiveCall's methods for use as # a Operation on the client. - Operation = view_class(:cancel, :cancelled, :deadline, :execute, + Operation = view_class(:cancel, :cancelled?, :deadline, :execute, :metadata, :status, :start_call, :wait, :write_flag, :write_flag=) end diff --git a/src/ruby/pb/test/client.rb b/src/ruby/pb/test/client.rb index 146623e0abf..066a7bb90f1 100755 --- a/src/ruby/pb/test/client.rb +++ b/src/ruby/pb/test/client.rb @@ -369,7 +369,7 @@ class NamedTests op.execute fail 'Should have raised GRPC:Cancelled' rescue GRPC::Cancelled - assert("#{__callee__}: call operation should be CANCELLED") { op.cancelled } + assert("#{__callee__}: call operation should be CANCELLED") { op.cancelled? } end def cancel_after_first_response @@ -380,7 +380,7 @@ class NamedTests op.execute.each { |r| ppp.queue.push(r) } fail 'Should have raised GRPC:Cancelled' rescue GRPC::Cancelled - assert("#{__callee__}: call operation should be CANCELLED") { op.cancelled } + assert("#{__callee__}: call operation should be CANCELLED") { op.cancelled? } op.wait end diff --git a/src/ruby/spec/generic/active_call_spec.rb b/src/ruby/spec/generic/active_call_spec.rb index b4e6f9ee028..018580e0dfa 100644 --- a/src/ruby/spec/generic/active_call_spec.rb +++ b/src/ruby/spec/generic/active_call_spec.rb @@ -61,7 +61,7 @@ describe GRPC::ActiveCall do describe '#multi_req_view' do it 'exposes a fixed subset of the ActiveCall methods' do - want = %w(cancelled, deadline, each_remote_read, metadata, shutdown) + want = %w(cancelled?, deadline, each_remote_read, metadata, shutdown) v = @client_call.multi_req_view want.each do |w| expect(v.methods.include?(w)) @@ -71,7 +71,7 @@ describe GRPC::ActiveCall do describe '#single_req_view' do it 'exposes a fixed subset of the ActiveCall methods' do - want = %w(cancelled, deadline, metadata, shutdown) + want = %w(cancelled?, deadline, metadata, shutdown) v = @client_call.single_req_view want.each do |w| expect(v.methods.include?(w)) From 4dd9ca9a6edb96ac5c880fc98656109c78e8a615 Mon Sep 17 00:00:00 2001 From: Nathaniel Manista Date: Thu, 7 Jul 2016 22:20:33 +0000 Subject: [PATCH 0450/1039] Fix _Rendezvous.exception for successful calls --- .../grpcio/grpc/beta/_client_adaptations.py | 5 +++- ...e_invocation_asynchronous_event_service.py | 25 +++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/src/python/grpcio/grpc/beta/_client_adaptations.py b/src/python/grpcio/grpc/beta/_client_adaptations.py index 56456cc117f..73415e0be7f 100644 --- a/src/python/grpcio/grpc/beta/_client_adaptations.py +++ b/src/python/grpcio/grpc/beta/_client_adaptations.py @@ -117,7 +117,10 @@ class _Rendezvous(future.Future, face.Call): def exception(self, timeout=None): try: rpc_error_call = self._future.exception(timeout=timeout) - return _abortion_error(rpc_error_call) + if rpc_error_call is None: + return None + else: + return _abortion_error(rpc_error_call) except grpc.FutureTimeoutError: raise future.TimeoutError() except grpc.FutureCancelledError: diff --git a/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py index 791620307b5..d32208f9eb0 100644 --- a/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py +++ b/src/python/grpcio_tests/tests/unit/framework/interfaces/face/_future_invocation_asynchronous_event_service.py @@ -41,6 +41,7 @@ from concurrent import futures import six # test_interfaces is referenced from specification in this module. +from grpc.framework.foundation import future from grpc.framework.foundation import logging_pool from grpc.framework.interfaces.face import face from tests.unit.framework.common import test_constants @@ -159,6 +160,8 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. test_messages.verify(request, response, self) self.assertIs(callback.future(), response_future) + self.assertIsNone(response_future.exception()) + self.assertIsNone(response_future.traceback()) def testSuccessfulUnaryRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -191,6 +194,8 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. test_messages.verify(requests, response, self) self.assertIs(future_passed_to_callback, response_future) + self.assertIsNone(response_future.exception()) + self.assertIsNone(response_future.traceback()) def testSuccessfulStreamRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -301,6 +306,12 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. self.assertIs(callback.future(), response_future) self.assertFalse(cancel_method_return_value) self.assertTrue(response_future.cancelled()) + with self.assertRaises(future.CancelledError): + response_future.result() + with self.assertRaises(future.CancelledError): + response_future.exception() + with self.assertRaises(future.CancelledError): + response_future.traceback() def testCancelledUnaryRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -332,6 +343,12 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. self.assertIs(callback.future(), response_future) self.assertFalse(cancel_method_return_value) self.assertTrue(response_future.cancelled()) + with self.assertRaises(future.CancelledError): + response_future.result() + with self.assertRaises(future.CancelledError): + response_future.exception() + with self.assertRaises(future.CancelledError): + response_future.traceback() def testCancelledStreamRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -363,6 +380,9 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. response_future.exception(), face.ExpirationError) with self.assertRaises(face.ExpirationError): response_future.result() + self.assertIsInstance( + response_future.exception(), face.AbortionError) + self.assertIsNotNone(response_future.traceback()) def testExpiredUnaryRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -392,6 +412,9 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. response_future.exception(), face.ExpirationError) with self.assertRaises(face.ExpirationError): response_future.result() + self.assertIsInstance( + response_future.exception(), face.AbortionError) + self.assertIsNotNone(response_future.traceback()) def testExpiredStreamRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -426,6 +449,7 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. response_future.exception(), face.ExpirationError) with self.assertRaises(face.ExpirationError): response_future.result() + self.assertIsNotNone(response_future.traceback()) def testFailedUnaryRequestStreamResponse(self): for (group, method), test_messages_sequence in ( @@ -463,6 +487,7 @@ class TestCase(six.with_metaclass(abc.ABCMeta, test_coverage.Coverage, unittest. response_future.exception(), face.ExpirationError) with self.assertRaises(face.ExpirationError): response_future.result() + self.assertIsNotNone(response_future.traceback()) def testFailedStreamRequestStreamResponse(self): for (group, method), test_messages_sequence in ( From e69f088cd973e61da445eddae61c67de6edbca17 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Thu, 7 Jul 2016 15:52:27 -0700 Subject: [PATCH 0451/1039] Split incoming initial and trailing metadata in Ruby calls --- src/ruby/ext/grpc/rb_call.c | 33 ++++++++++++++++++++++++ src/ruby/lib/grpc/generic/active_call.rb | 22 ++++++++-------- src/ruby/spec/generic/rpc_server_spec.rb | 4 +-- 3 files changed, 46 insertions(+), 13 deletions(-) diff --git a/src/ruby/ext/grpc/rb_call.c b/src/ruby/ext/grpc/rb_call.c index f62397e79f5..21261244436 100644 --- a/src/ruby/ext/grpc/rb_call.c +++ b/src/ruby/ext/grpc/rb_call.c @@ -71,6 +71,10 @@ static ID id_credentials; * received by the call and subsequently saved on it. */ static ID id_metadata; +/* id_trailing_metadata is the name of the attribute used to access the trailing + * metadata hash received by the call and subsequently saved on it. */ +static ID id_trailing_metadata; + /* id_status is name of the attribute used to access the status object * received by the call and subsequently saved on it. */ static ID id_status; @@ -296,6 +300,30 @@ static VALUE grpc_rb_call_set_metadata(VALUE self, VALUE metadata) { return rb_ivar_set(self, id_metadata, metadata); } +/* + call-seq: + trailing_metadata = call.trailing_metadata + + Gets the trailing metadata object saved on the call */ +static VALUE grpc_rb_call_get_trailing_metadata(VALUE self) { + return rb_ivar_get(self, id_trailing_metadata); +} + +/* + call-seq: + call.trailing_metadata = trailing_metadata + + Saves the trailing metadata hash on the call. */ +static VALUE grpc_rb_call_set_trailing_metadata(VALUE self, VALUE metadata) { + if (!NIL_P(metadata) && TYPE(metadata) != T_HASH) { + rb_raise(rb_eTypeError, "bad metadata: got:<%s> want: ", + rb_obj_classname(metadata)); + return Qnil; + } + + return rb_ivar_set(self, id_trailing_metadata, metadata); +} + /* call-seq: write_flag = call.write_flag @@ -908,6 +936,10 @@ void Init_grpc_call() { rb_define_method(grpc_rb_cCall, "status=", grpc_rb_call_set_status, 1); rb_define_method(grpc_rb_cCall, "metadata", grpc_rb_call_get_metadata, 0); rb_define_method(grpc_rb_cCall, "metadata=", grpc_rb_call_set_metadata, 1); + rb_define_method(grpc_rb_cCall, "trailing_metadata", + grpc_rb_call_get_trailing_metadata, 0); + rb_define_method(grpc_rb_cCall, "trailing_metadata=", + grpc_rb_call_set_trailing_metadata, 1); rb_define_method(grpc_rb_cCall, "write_flag", grpc_rb_call_get_write_flag, 0); rb_define_method(grpc_rb_cCall, "write_flag=", grpc_rb_call_set_write_flag, 1); @@ -916,6 +948,7 @@ void Init_grpc_call() { /* Ids used to support call attributes */ id_metadata = rb_intern("metadata"); + id_trailing_metadata = rb_intern("trailing_metadata"); id_status = rb_intern("status"); id_write_flag = rb_intern("write_flag"); diff --git a/src/ruby/lib/grpc/generic/active_call.rb b/src/ruby/lib/grpc/generic/active_call.rb index a3ac0d48a3b..5c142fb1619 100644 --- a/src/ruby/lib/grpc/generic/active_call.rb +++ b/src/ruby/lib/grpc/generic/active_call.rb @@ -43,8 +43,7 @@ class Struct GRPC.logger.debug("Failing with status #{status}") # raise BadStatus, propagating the metadata if present. md = status.metadata - with_sym_keys = Hash[md.each_pair.collect { |x, y| [x.to_sym, y] }] - fail GRPC::BadStatus.new(status.code, status.details, with_sym_keys) + fail GRPC::BadStatus.new(status.code, status.details, md) end status end @@ -61,7 +60,7 @@ module GRPC extend Forwardable attr_reader(:deadline) def_delegators :@call, :cancel, :metadata, :write_flag, :write_flag=, - :peer, :peer_cert + :peer, :peer_cert, :trailing_metadata # client_invoke begins a client invocation. # @@ -158,6 +157,9 @@ module GRPC ops[RECV_STATUS_ON_CLIENT] = nil if assert_finished batch_result = @call.run_batch(ops) return unless assert_finished + unless batch_result.status.nil? + @call.trailing_metadata = batch_result.status.metadata + end @call.status = batch_result.status op_is_done batch_result.check_status @@ -169,11 +171,7 @@ module GRPC def finished batch_result = @call.run_batch(RECV_STATUS_ON_CLIENT => nil) unless batch_result.status.nil? - if @call.metadata.nil? - @call.metadata = batch_result.status.metadata - else - @call.metadata.merge!(batch_result.status.metadata) - end + @call.trailing_metadata = batch_result.status.metadata end @call.status = batch_result.status op_is_done @@ -456,17 +454,19 @@ module GRPC # SingleReqView limits access to an ActiveCall's methods for use in server # handlers that receive just one request. SingleReqView = view_class(:cancelled, :deadline, :metadata, - :output_metadata, :peer, :peer_cert) + :output_metadata, :peer, :peer_cert, + :trailing_metadata) # MultiReqView limits access to an ActiveCall's methods for use in # server client_streamer handlers. MultiReqView = view_class(:cancelled, :deadline, :each_queued_msg, - :each_remote_read, :metadata, :output_metadata) + :each_remote_read, :metadata, :output_metadata, + :trailing_metadata) # Operation limits access to an ActiveCall's methods for use as # a Operation on the client. Operation = view_class(:cancel, :cancelled, :deadline, :execute, :metadata, :status, :start_call, :wait, :write_flag, - :write_flag=) + :write_flag=, :trailing_metadata) end end diff --git a/src/ruby/spec/generic/rpc_server_spec.rb b/src/ruby/spec/generic/rpc_server_spec.rb index 901c84fc783..31157cf161e 100644 --- a/src/ruby/spec/generic/rpc_server_spec.rb +++ b/src/ruby/spec/generic/rpc_server_spec.rb @@ -95,7 +95,7 @@ class FailingService def initialize(_default_var = 'ignored') @details = 'app error' @code = 101 - @md = { failed_method: 'an_rpc' } + @md = { 'failed_method' => 'an_rpc' } end def an_rpc(_req, _call) @@ -515,7 +515,7 @@ describe GRPC::RpcServer do op = stub.an_rpc(req, return_op: true, metadata: { k1: 'v1', k2: 'v2' }) expect(op.metadata).to be nil expect(op.execute).to be_a(EchoMsg) - expect(op.metadata).to eq(wanted_trailers) + expect(op.trailing_metadata).to eq(wanted_trailers) @srv.stop t.join end From c68640f05cea2cf79bf2703d83da6b76fa6dc5e6 Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Thu, 7 Jul 2016 14:13:35 -0700 Subject: [PATCH 0452/1039] Read from stdin Read from stdin if the request text and binary file are not provided --- test/cpp/util/grpc_cli.cc | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/test/cpp/util/grpc_cli.cc b/test/cpp/util/grpc_cli.cc index 7c8e2cdc639..fdb1a7c2a00 100644 --- a/test/cpp/util/grpc_cli.cc +++ b/test/cpp/util/grpc_cli.cc @@ -64,6 +64,7 @@ < output.bin > output.txt */ +#include #include #include #include @@ -146,7 +147,6 @@ int main(int argc, char** argv) { grpc::string serialized_request_proto; if (argc == 5) { - // TODO(yangg) read from stdin as well? request_text = argv[4]; } @@ -164,10 +164,15 @@ int main(int argc, char** argv) { grpc::CreateChannel(server_address, creds); if (request_text.empty() && FLAGS_input_binary_file.empty()) { - std::cout << "Missing input. Use text format input or " - << "--input_binary_file for serialized request" << std::endl; - return 1; - } else if (!request_text.empty()) { + if (isatty(STDIN_FILENO)) { + std::cout << "reading request message from stdin..." << std::endl; + } + std::stringstream input_stream; + input_stream << std::cin.rdbuf(); + request_text = input_stream.str(); + } + + if (!request_text.empty()) { if (!FLAGS_proto_file.empty()) { parser.reset(new grpc::testing::ProtoFileParser( FLAGS_proto_path, FLAGS_proto_file, method_name)); @@ -178,6 +183,12 @@ int main(int argc, char** argv) { if (parser->HasError()) { return 1; } + + if (!FLAGS_input_binary_file.empty()) { + std::cout + << "warning: request given in argv, ignoring --input_binary_file" + << std::endl; + } } if (parser) { @@ -226,7 +237,7 @@ int main(int argc, char** argv) { } } else { std::cout << "Rpc failed with status code " << s.error_code() - << " error message " << s.error_message() << std::endl; + << ", error message: " << s.error_message() << std::endl; } return 0; From 8b249088b0116d57a489c9882514f646bee0e5ee Mon Sep 17 00:00:00 2001 From: Stanley Cheung Date: Wed, 29 Jun 2016 15:13:06 -0700 Subject: [PATCH 0453/1039] php: add warning about shutdown function --- src/core/lib/iomgr/ev_poll_posix.c | 4 ++-- src/core/lib/iomgr/tcp_posix.c | 4 ++-- src/php/ext/grpc/php_grpc.c | 2 ++ 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/src/core/lib/iomgr/ev_poll_posix.c b/src/core/lib/iomgr/ev_poll_posix.c index 45c0a5e9546..3b57a0bdfb3 100644 --- a/src/core/lib/iomgr/ev_poll_posix.c +++ b/src/core/lib/iomgr/ev_poll_posix.c @@ -254,7 +254,7 @@ struct grpc_pollset_set { #define UNREF_BY(fd, n, reason) unref_by(fd, n, reason, __FILE__, __LINE__) static void ref_by(grpc_fd *fd, int n, const char *reason, const char *file, int line) { - gpr_log(GPR_DEBUG, "FD %d %p ref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, + gpr_log(GPR_DEBUG, "FD %d %p ref %d %ld -> %ld [%s; %s:%d]", fd->fd, fd, n, gpr_atm_no_barrier_load(&fd->refst), gpr_atm_no_barrier_load(&fd->refst) + n, reason, file, line); #else @@ -269,7 +269,7 @@ static void ref_by(grpc_fd *fd, int n) { static void unref_by(grpc_fd *fd, int n, const char *reason, const char *file, int line) { gpr_atm old; - gpr_log(GPR_DEBUG, "FD %d %p unref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, + gpr_log(GPR_DEBUG, "FD %d %p unref %d %ld -> %ld [%s; %s:%d]", fd->fd, fd, n, gpr_atm_no_barrier_load(&fd->refst), gpr_atm_no_barrier_load(&fd->refst) - n, reason, file, line); #else diff --git a/src/core/lib/iomgr/tcp_posix.c b/src/core/lib/iomgr/tcp_posix.c index 2ab45e33ce3..27fd21ba091 100644 --- a/src/core/lib/iomgr/tcp_posix.c +++ b/src/core/lib/iomgr/tcp_posix.c @@ -127,7 +127,7 @@ static void tcp_free(grpc_exec_ctx *exec_ctx, grpc_tcp *tcp) { #define TCP_REF(tcp, reason) tcp_ref((tcp), (reason), __FILE__, __LINE__) static void tcp_unref(grpc_exec_ctx *exec_ctx, grpc_tcp *tcp, const char *reason, const char *file, int line) { - gpr_log(file, line, GPR_LOG_SEVERITY_DEBUG, "TCP unref %p : %s %d -> %d", tcp, + gpr_log(file, line, GPR_LOG_SEVERITY_DEBUG, "TCP unref %p : %s %ld -> %ld", tcp, reason, tcp->refcount.count, tcp->refcount.count - 1); if (gpr_unref(&tcp->refcount)) { tcp_free(exec_ctx, tcp); @@ -136,7 +136,7 @@ static void tcp_unref(grpc_exec_ctx *exec_ctx, grpc_tcp *tcp, static void tcp_ref(grpc_tcp *tcp, const char *reason, const char *file, int line) { - gpr_log(file, line, GPR_LOG_SEVERITY_DEBUG, "TCP ref %p : %s %d -> %d", tcp, + gpr_log(file, line, GPR_LOG_SEVERITY_DEBUG, "TCP ref %p : %s %ld -> %ld", tcp, reason, tcp->refcount.count, tcp->refcount.count + 1); gpr_ref(&tcp->refcount); } diff --git a/src/php/ext/grpc/php_grpc.c b/src/php/ext/grpc/php_grpc.c index f4cb5b28cc8..449ba3cd47b 100644 --- a/src/php/ext/grpc/php_grpc.c +++ b/src/php/ext/grpc/php_grpc.c @@ -248,6 +248,8 @@ PHP_MSHUTDOWN_FUNCTION(grpc) { /* uncomment this line if you have INI entries UNREGISTER_INI_ENTRIES(); */ + // WARNING: This function IS being called by PHP when the extension + // is unloaded but the logs were somehow suppressed. grpc_shutdown_timeval(TSRMLS_C); grpc_php_shutdown_completion_queue(TSRMLS_C); grpc_shutdown(); From 692a38f7fa412f7fc90c23d778f13d4995abf9fa Mon Sep 17 00:00:00 2001 From: Stanley Cheung Date: Thu, 30 Jun 2016 15:13:29 -0700 Subject: [PATCH 0454/1039] php: update example composer.json --- examples/php/composer.json | 2 +- src/php/composer.json | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/examples/php/composer.json b/examples/php/composer.json index c837bf7ac0f..950e11367d0 100644 --- a/examples/php/composer.json +++ b/examples/php/composer.json @@ -9,6 +9,6 @@ } ], "require": { - "grpc/grpc": "dev-release-0_13" + "grpc/grpc": "v0.15.0" } } diff --git a/src/php/composer.json b/src/php/composer.json index 01674a25db8..2ad73223c65 100644 --- a/src/php/composer.json +++ b/src/php/composer.json @@ -2,7 +2,6 @@ "name": "grpc/grpc", "type": "library", "description": "gRPC library for PHP", - "version": "0.14.0", "keywords": ["rpc"], "homepage": "http://grpc.io", "license": "BSD-3-Clause", From 69f9e448f8bcb52e27fe944f5ac335ef1ef5d93d Mon Sep 17 00:00:00 2001 From: Stanley Cheung Date: Fri, 1 Jul 2016 08:13:16 -0700 Subject: [PATCH 0455/1039] revert debug log change --- src/core/lib/iomgr/ev_poll_posix.c | 4 ++-- src/core/lib/iomgr/tcp_posix.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/core/lib/iomgr/ev_poll_posix.c b/src/core/lib/iomgr/ev_poll_posix.c index 3b57a0bdfb3..45c0a5e9546 100644 --- a/src/core/lib/iomgr/ev_poll_posix.c +++ b/src/core/lib/iomgr/ev_poll_posix.c @@ -254,7 +254,7 @@ struct grpc_pollset_set { #define UNREF_BY(fd, n, reason) unref_by(fd, n, reason, __FILE__, __LINE__) static void ref_by(grpc_fd *fd, int n, const char *reason, const char *file, int line) { - gpr_log(GPR_DEBUG, "FD %d %p ref %d %ld -> %ld [%s; %s:%d]", fd->fd, fd, n, + gpr_log(GPR_DEBUG, "FD %d %p ref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, gpr_atm_no_barrier_load(&fd->refst), gpr_atm_no_barrier_load(&fd->refst) + n, reason, file, line); #else @@ -269,7 +269,7 @@ static void ref_by(grpc_fd *fd, int n) { static void unref_by(grpc_fd *fd, int n, const char *reason, const char *file, int line) { gpr_atm old; - gpr_log(GPR_DEBUG, "FD %d %p unref %d %ld -> %ld [%s; %s:%d]", fd->fd, fd, n, + gpr_log(GPR_DEBUG, "FD %d %p unref %d %d -> %d [%s; %s:%d]", fd->fd, fd, n, gpr_atm_no_barrier_load(&fd->refst), gpr_atm_no_barrier_load(&fd->refst) - n, reason, file, line); #else diff --git a/src/core/lib/iomgr/tcp_posix.c b/src/core/lib/iomgr/tcp_posix.c index 27fd21ba091..2ab45e33ce3 100644 --- a/src/core/lib/iomgr/tcp_posix.c +++ b/src/core/lib/iomgr/tcp_posix.c @@ -127,7 +127,7 @@ static void tcp_free(grpc_exec_ctx *exec_ctx, grpc_tcp *tcp) { #define TCP_REF(tcp, reason) tcp_ref((tcp), (reason), __FILE__, __LINE__) static void tcp_unref(grpc_exec_ctx *exec_ctx, grpc_tcp *tcp, const char *reason, const char *file, int line) { - gpr_log(file, line, GPR_LOG_SEVERITY_DEBUG, "TCP unref %p : %s %ld -> %ld", tcp, + gpr_log(file, line, GPR_LOG_SEVERITY_DEBUG, "TCP unref %p : %s %d -> %d", tcp, reason, tcp->refcount.count, tcp->refcount.count - 1); if (gpr_unref(&tcp->refcount)) { tcp_free(exec_ctx, tcp); @@ -136,7 +136,7 @@ static void tcp_unref(grpc_exec_ctx *exec_ctx, grpc_tcp *tcp, static void tcp_ref(grpc_tcp *tcp, const char *reason, const char *file, int line) { - gpr_log(file, line, GPR_LOG_SEVERITY_DEBUG, "TCP ref %p : %s %ld -> %ld", tcp, + gpr_log(file, line, GPR_LOG_SEVERITY_DEBUG, "TCP ref %p : %s %d -> %d", tcp, reason, tcp->refcount.count, tcp->refcount.count + 1); gpr_ref(&tcp->refcount); } From ea1b16f82f348cc58cf5f932d22d3eb6777b3bd1 Mon Sep 17 00:00:00 2001 From: Ken Payson Date: Thu, 7 Jul 2016 16:44:33 -0700 Subject: [PATCH 0456/1039] Removed cython client-side call tracking This ensures sync calls get cancelled after a keyboard interrupt, as well as all calls getting destroyed before grpc_shutdown() --- src/python/grpcio/grpc/_channel.py | 27 ++++++++++--------- .../grpcio/grpc/_cython/_cygrpc/call.pyx.pxi | 15 +++++++++-- .../grpc/_cython/_cygrpc/records.pxd.pxi | 6 ++--- src/python/grpcio/grpc/_server.py | 16 +++++------ .../unit/_cython/_cancel_many_calls_test.py | 8 +++--- .../_read_some_but_not_all_responses_test.py | 22 +++++++-------- .../tests/unit/_cython/cygrpc_test.py | 9 ++++--- 7 files changed, 58 insertions(+), 45 deletions(-) diff --git a/src/python/grpcio/grpc/_channel.py b/src/python/grpcio/grpc/_channel.py index a89b5013030..29dbc3a668b 100644 --- a/src/python/grpcio/grpc/_channel.py +++ b/src/python/grpcio/grpc/_channel.py @@ -195,7 +195,8 @@ def _consume_request_iterator( cygrpc.operation_send_message( serialized_request, _EMPTY_FLAGS), ) - call.start_batch(cygrpc.Operations(operations), event_handler) + call.start_client_batch(cygrpc.Operations(operations), + event_handler) state.due.add(cygrpc.OperationType.send_message) while True: state.condition.wait() @@ -211,7 +212,7 @@ def _consume_request_iterator( operations = ( cygrpc.operation_send_close_from_client(_EMPTY_FLAGS), ) - call.start_batch(cygrpc.Operations(operations), event_handler) + call.start_client_batch(cygrpc.Operations(operations), event_handler) state.due.add(cygrpc.OperationType.send_close_from_client) def stop_consumption_thread(timeout): @@ -312,7 +313,7 @@ class _Rendezvous(grpc.RpcError, grpc.Future, grpc.Call): if self._state.code is None: event_handler = _event_handler( self._state, self._call, self._response_deserializer) - self._call.start_batch( + self._call.start_client_batch( cygrpc.Operations( (cygrpc.operation_receive_message(_EMPTY_FLAGS),)), event_handler) @@ -471,7 +472,7 @@ class _UnaryUnaryMultiCallable(grpc.UnaryUnaryMultiCallable): None, 0, completion_queue, self._method, None, deadline_timespec) if credentials is not None: call.set_credentials(credentials._credentials) - call.start_batch(cygrpc.Operations(operations), None) + call.start_client_batch(cygrpc.Operations(operations), None) _handle_event(completion_queue.poll(), state, self._response_deserializer) return state, deadline @@ -495,7 +496,7 @@ class _UnaryUnaryMultiCallable(grpc.UnaryUnaryMultiCallable): call.set_credentials(credentials._credentials) event_handler = _event_handler(state, call, self._response_deserializer) with state.condition: - call.start_batch(cygrpc.Operations(operations), event_handler) + call.start_client_batch(cygrpc.Operations(operations), event_handler) return _Rendezvous(state, call, self._response_deserializer, deadline) @@ -523,7 +524,7 @@ class _UnaryStreamMultiCallable(grpc.UnaryStreamMultiCallable): call.set_credentials(credentials._credentials) event_handler = _event_handler(state, call, self._response_deserializer) with state.condition: - call.start_batch( + call.start_client_batch( cygrpc.Operations( (cygrpc.operation_receive_initial_metadata(_EMPTY_FLAGS),)), event_handler) @@ -534,7 +535,7 @@ class _UnaryStreamMultiCallable(grpc.UnaryStreamMultiCallable): cygrpc.operation_send_close_from_client(_EMPTY_FLAGS), cygrpc.operation_receive_status_on_client(_EMPTY_FLAGS), ) - call.start_batch(cygrpc.Operations(operations), event_handler) + call.start_client_batch(cygrpc.Operations(operations), event_handler) return _Rendezvous(state, call, self._response_deserializer, deadline) @@ -558,7 +559,7 @@ class _StreamUnaryMultiCallable(grpc.StreamUnaryMultiCallable): if credentials is not None: call.set_credentials(credentials._credentials) with state.condition: - call.start_batch( + call.start_client_batch( cygrpc.Operations( (cygrpc.operation_receive_initial_metadata(_EMPTY_FLAGS),)), None) @@ -568,7 +569,7 @@ class _StreamUnaryMultiCallable(grpc.StreamUnaryMultiCallable): cygrpc.operation_receive_message(_EMPTY_FLAGS), cygrpc.operation_receive_status_on_client(_EMPTY_FLAGS), ) - call.start_batch(cygrpc.Operations(operations), None) + call.start_client_batch(cygrpc.Operations(operations), None) _consume_request_iterator( request_iterator, state, call, self._request_serializer) while True: @@ -602,7 +603,7 @@ class _StreamUnaryMultiCallable(grpc.StreamUnaryMultiCallable): call.set_credentials(credentials._credentials) event_handler = _event_handler(state, call, self._response_deserializer) with state.condition: - call.start_batch( + call.start_client_batch( cygrpc.Operations( (cygrpc.operation_receive_initial_metadata(_EMPTY_FLAGS),)), event_handler) @@ -612,7 +613,7 @@ class _StreamUnaryMultiCallable(grpc.StreamUnaryMultiCallable): cygrpc.operation_receive_message(_EMPTY_FLAGS), cygrpc.operation_receive_status_on_client(_EMPTY_FLAGS), ) - call.start_batch(cygrpc.Operations(operations), event_handler) + call.start_client_batch(cygrpc.Operations(operations), event_handler) _consume_request_iterator( request_iterator, state, call, self._request_serializer) return _Rendezvous(state, call, self._response_deserializer, deadline) @@ -639,7 +640,7 @@ class _StreamStreamMultiCallable(grpc.StreamStreamMultiCallable): call.set_credentials(credentials._credentials) event_handler = _event_handler(state, call, self._response_deserializer) with state.condition: - call.start_batch( + call.start_client_batch( cygrpc.Operations( (cygrpc.operation_receive_initial_metadata(_EMPTY_FLAGS),)), event_handler) @@ -648,7 +649,7 @@ class _StreamStreamMultiCallable(grpc.StreamStreamMultiCallable): _common.cygrpc_metadata(metadata), _EMPTY_FLAGS), cygrpc.operation_receive_status_on_client(_EMPTY_FLAGS), ) - call.start_batch(cygrpc.Operations(operations), event_handler) + call.start_client_batch(cygrpc.Operations(operations), event_handler) _consume_request_iterator( request_iterator, state, call, self._request_serializer) return _Rendezvous(state, call, self._response_deserializer, deadline) diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/call.pyx.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/call.pyx.pxi index a09bbc75fe6..6570dcdb852 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/call.pyx.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/call.pyx.pxi @@ -37,13 +37,16 @@ cdef class Call: self.c_call = NULL self.references = [] - def start_batch(self, operations, tag): + def _start_batch(self, operations, tag, retain_self): if not self.is_valid: raise ValueError("invalid call object cannot be used from Python") cdef grpc_call_error result cdef Operations cy_operations = Operations(operations) cdef OperationTag operation_tag = OperationTag(tag) - operation_tag.operation_call = self + if retain_self: + operation_tag.operation_call = self + else: + operation_tag.operation_call = None operation_tag.batch_operations = cy_operations cpython.Py_INCREF(operation_tag) with nogil: @@ -52,6 +55,14 @@ cdef class Call: operation_tag, NULL) return result + def start_client_batch(self, operations, tag): + # We don't reference this call in the operations tag because + # it should be cancelled when it goes out of scope + return self._start_batch(operations, tag, False) + + def start_server_batch(self, operations, tag): + return self._start_batch(operations, tag, True) + def cancel( self, grpc_status_code error_code=GRPC_STATUS__DO_NOT_USE, details=None): diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/records.pxd.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/records.pxd.pxi index 0474697af82..96c5b02bc2e 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/records.pxd.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/records.pxd.pxi @@ -58,14 +58,14 @@ cdef class Event: cdef readonly bint success cdef readonly object tag - # For operations with calls - cdef readonly Call operation_call - # For Server.request_call cdef readonly bint is_new_request cdef readonly CallDetails request_call_details cdef readonly Metadata request_metadata + # For server calls + cdef readonly Call operation_call + # For Call.start_batch cdef readonly Operations batch_operations diff --git a/src/python/grpcio/grpc/_server.py b/src/python/grpcio/grpc/_server.py index f4c114056fe..5d805b53e40 100644 --- a/src/python/grpcio/grpc/_server.py +++ b/src/python/grpcio/grpc/_server.py @@ -157,7 +157,7 @@ def _abort(state, call, code, details): effective_details, _EMPTY_FLAGS), ) token = _SEND_STATUS_FROM_SERVER_TOKEN - call.start_batch( + call.start_server_batch( cygrpc.Operations(operations), _send_status_from_server(state, token)) state.statused = True @@ -257,7 +257,7 @@ class _Context(grpc.ServicerContext): if self._state.initial_metadata_allowed: operation = cygrpc.operation_send_initial_metadata( _common.cygrpc_metadata(initial_metadata), _EMPTY_FLAGS) - self._rpc_event.operation_call.start_batch( + self._rpc_event.operation_call.start_server_batch( cygrpc.Operations((operation,)), _send_initial_metadata(self._state)) self._state.initial_metadata_allowed = False @@ -292,7 +292,7 @@ class _RequestIterator(object): elif self._state.client is _CLOSED or self._state.statused: raise StopIteration() else: - self._call.start_batch( + self._call.start_server_batch( cygrpc.Operations((cygrpc.operation_receive_message(_EMPTY_FLAGS),)), _receive_message(self._state, self._call, self._request_deserializer)) self._state.due.add(_RECEIVE_MESSAGE_TOKEN) @@ -333,7 +333,7 @@ def _unary_request(rpc_event, state, request_deserializer): if state.client is _CANCELLED or state.statused: return None else: - start_batch_result = rpc_event.operation_call.start_batch( + start_server_batch_result = rpc_event.operation_call.start_server_batch( cygrpc.Operations( (cygrpc.operation_receive_message(_EMPTY_FLAGS),)), _receive_message( @@ -417,7 +417,7 @@ def _send_response(rpc_event, state, serialized_response): cygrpc.operation_send_message(serialized_response, _EMPTY_FLAGS), ) token = _SEND_MESSAGE_TOKEN - rpc_event.operation_call.start_batch( + rpc_event.operation_call.start_server_batch( cygrpc.Operations(operations), _send_message(state, token)) state.due.add(token) while True: @@ -443,7 +443,7 @@ def _status(rpc_event, state, serialized_response): if serialized_response is not None: operations.append(cygrpc.operation_send_message( serialized_response, _EMPTY_FLAGS)) - rpc_event.operation_call.start_batch( + rpc_event.operation_call.start_server_batch( cygrpc.Operations(operations), _send_status_from_server(state, _SEND_STATUS_FROM_SERVER_TOKEN)) state.statused = True @@ -550,7 +550,7 @@ def _handle_unrecognized_method(rpc_event): b'Method not found!', _EMPTY_FLAGS), ) rpc_state = _RPCState() - rpc_event.operation_call.start_batch( + rpc_event.operation_call.start_server_batch( operations, lambda ignored_event: (rpc_state, (),)) return rpc_state @@ -558,7 +558,7 @@ def _handle_unrecognized_method(rpc_event): def _handle_with_method_handler(rpc_event, method_handler, thread_pool): state = _RPCState() with state.condition: - rpc_event.operation_call.start_batch( + rpc_event.operation_call.start_server_batch( cygrpc.Operations( (cygrpc.operation_receive_close_on_server(_EMPTY_FLAGS),)), _receive_close_on_server(state)) diff --git a/src/python/grpcio_tests/tests/unit/_cython/_cancel_many_calls_test.py b/src/python/grpcio_tests/tests/unit/_cython/_cancel_many_calls_test.py index cac0c8b3b93..cf212c56539 100644 --- a/src/python/grpcio_tests/tests/unit/_cython/_cancel_many_calls_test.py +++ b/src/python/grpcio_tests/tests/unit/_cython/_cancel_many_calls_test.py @@ -81,11 +81,11 @@ class _Handler(object): self._state.condition.wait() with self._lock: - self._call.start_batch( + self._call.start_server_batch( cygrpc.Operations( (cygrpc.operation_receive_close_on_server(_EMPTY_FLAGS),)), _RECEIVE_CLOSE_ON_SERVER_TAG) - self._call.start_batch( + self._call.start_server_batch( cygrpc.Operations((cygrpc.operation_receive_message(_EMPTY_FLAGS),)), _RECEIVE_MESSAGE_TAG) first_event = self._completion_queue.poll() @@ -101,7 +101,7 @@ class _Handler(object): _EMPTY_METADATA, cygrpc.StatusCode.ok, b'test details!', _EMPTY_FLAGS), ) - self._call.start_batch( + self._call.start_server_batch( cygrpc.Operations(operations), _SERVER_COMPLETE_CALL_TAG) self._completion_queue.poll() self._completion_queue.poll() @@ -193,7 +193,7 @@ class CancelManyCallsTest(unittest.TestCase): cygrpc.operation_receive_status_on_client(_EMPTY_FLAGS), ) tag = 'client_complete_call_{0:04d}_tag'.format(index) - client_call.start_batch(cygrpc.Operations(operations), tag) + client_call.start_client_batch(cygrpc.Operations(operations), tag) client_due.add(tag) client_calls.append(client_call) diff --git a/src/python/grpcio_tests/tests/unit/_cython/_read_some_but_not_all_responses_test.py b/src/python/grpcio_tests/tests/unit/_cython/_read_some_but_not_all_responses_test.py index 27fcee0d6f1..152d8edde3b 100644 --- a/src/python/grpcio_tests/tests/unit/_cython/_read_some_but_not_all_responses_test.py +++ b/src/python/grpcio_tests/tests/unit/_cython/_read_some_but_not_all_responses_test.py @@ -168,12 +168,12 @@ class ReadSomeButNotAllResponsesTest(unittest.TestCase): client_complete_rpc_tag = 'client_complete_rpc_tag' with client_condition: client_receive_initial_metadata_start_batch_result = ( - client_call.start_batch(cygrpc.Operations([ + client_call.start_client_batch(cygrpc.Operations([ cygrpc.operation_receive_initial_metadata(_EMPTY_FLAGS), ]), client_receive_initial_metadata_tag)) client_due.add(client_receive_initial_metadata_tag) client_complete_rpc_start_batch_result = ( - client_call.start_batch(cygrpc.Operations([ + client_call.start_client_batch(cygrpc.Operations([ cygrpc.operation_send_initial_metadata( _EMPTY_METADATA, _EMPTY_FLAGS), cygrpc.operation_send_close_from_client(_EMPTY_FLAGS), @@ -185,30 +185,30 @@ class ReadSomeButNotAllResponsesTest(unittest.TestCase): with server_call_condition: server_send_initial_metadata_start_batch_result = ( - server_rpc_event.operation_call.start_batch(cygrpc.Operations([ + server_rpc_event.operation_call.start_server_batch([ cygrpc.operation_send_initial_metadata( _EMPTY_METADATA, _EMPTY_FLAGS), - ]), server_send_initial_metadata_tag)) + ], server_send_initial_metadata_tag)) server_send_first_message_start_batch_result = ( - server_rpc_event.operation_call.start_batch(cygrpc.Operations([ + server_rpc_event.operation_call.start_server_batch([ cygrpc.operation_send_message(b'\x07', _EMPTY_FLAGS), - ]), server_send_first_message_tag)) + ], server_send_first_message_tag)) server_send_initial_metadata_event = server_call_driver.event_with_tag( server_send_initial_metadata_tag) server_send_first_message_event = server_call_driver.event_with_tag( server_send_first_message_tag) with server_call_condition: server_send_second_message_start_batch_result = ( - server_rpc_event.operation_call.start_batch(cygrpc.Operations([ + server_rpc_event.operation_call.start_server_batch([ cygrpc.operation_send_message(b'\x07', _EMPTY_FLAGS), - ]), server_send_second_message_tag)) + ], server_send_second_message_tag)) server_complete_rpc_start_batch_result = ( - server_rpc_event.operation_call.start_batch(cygrpc.Operations([ + server_rpc_event.operation_call.start_server_batch([ cygrpc.operation_receive_close_on_server(_EMPTY_FLAGS), cygrpc.operation_send_status_from_server( cygrpc.Metadata(()), cygrpc.StatusCode.ok, b'test details', _EMPTY_FLAGS), - ]), server_complete_rpc_tag)) + ], server_complete_rpc_tag)) server_send_second_message_event = server_call_driver.event_with_tag( server_send_second_message_tag) server_complete_rpc_event = server_call_driver.event_with_tag( @@ -218,7 +218,7 @@ class ReadSomeButNotAllResponsesTest(unittest.TestCase): with client_condition: client_receive_first_message_tag = 'client_receive_first_message_tag' client_receive_first_message_start_batch_result = ( - client_call.start_batch(cygrpc.Operations([ + client_call.start_client_batch(cygrpc.Operations([ cygrpc.operation_receive_message(_EMPTY_FLAGS), ]), client_receive_first_message_tag)) client_due.add(client_receive_first_message_tag) diff --git a/src/python/grpcio_tests/tests/unit/_cython/cygrpc_test.py b/src/python/grpcio_tests/tests/unit/_cython/cygrpc_test.py index b740695e35b..9d1dbc189b8 100644 --- a/src/python/grpcio_tests/tests/unit/_cython/cygrpc_test.py +++ b/src/python/grpcio_tests/tests/unit/_cython/cygrpc_test.py @@ -186,7 +186,8 @@ class ServerClientMixin(object): def performer(): tag = object() try: - call_result = call.start_batch(cygrpc.Operations(operations), tag) + call_result = call.start_client_batch( + cygrpc.Operations(operations), tag) self.assertEqual(cygrpc.CallError.ok, call_result) event = queue.poll(deadline) self.assertEqual(cygrpc.CompletionType.operation_complete, event.type) @@ -231,7 +232,7 @@ class ServerClientMixin(object): cygrpc.Metadatum(CLIENT_METADATA_ASCII_KEY, CLIENT_METADATA_ASCII_VALUE), cygrpc.Metadatum(CLIENT_METADATA_BIN_KEY, CLIENT_METADATA_BIN_VALUE)]) - client_start_batch_result = client_call.start_batch(cygrpc.Operations([ + client_start_batch_result = client_call.start_client_batch([ cygrpc.operation_send_initial_metadata(client_initial_metadata, _EMPTY_FLAGS), cygrpc.operation_send_message(REQUEST, _EMPTY_FLAGS), @@ -239,7 +240,7 @@ class ServerClientMixin(object): cygrpc.operation_receive_initial_metadata(_EMPTY_FLAGS), cygrpc.operation_receive_message(_EMPTY_FLAGS), cygrpc.operation_receive_status_on_client(_EMPTY_FLAGS) - ]), client_call_tag) + ], client_call_tag) self.assertEqual(cygrpc.CallError.ok, client_start_batch_result) client_event_future = test_utilities.CompletionQueuePollFuture( self.client_completion_queue, cygrpc_deadline) @@ -268,7 +269,7 @@ class ServerClientMixin(object): server_trailing_metadata = cygrpc.Metadata([ cygrpc.Metadatum(SERVER_TRAILING_METADATA_KEY, SERVER_TRAILING_METADATA_VALUE)]) - server_start_batch_result = server_call.start_batch([ + server_start_batch_result = server_call.start_server_batch([ cygrpc.operation_send_initial_metadata(server_initial_metadata, _EMPTY_FLAGS), cygrpc.operation_receive_message(_EMPTY_FLAGS), From ae466c8a8d5586b08ef5078febc115775220b8b9 Mon Sep 17 00:00:00 2001 From: murgatroid99 Date: Fri, 8 Jul 2016 09:28:25 -0700 Subject: [PATCH 0457/1039] Revert changes to SingleReqView and MultiReqView --- src/ruby/lib/grpc/generic/active_call.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/ruby/lib/grpc/generic/active_call.rb b/src/ruby/lib/grpc/generic/active_call.rb index 5c142fb1619..47f24989365 100644 --- a/src/ruby/lib/grpc/generic/active_call.rb +++ b/src/ruby/lib/grpc/generic/active_call.rb @@ -454,14 +454,12 @@ module GRPC # SingleReqView limits access to an ActiveCall's methods for use in server # handlers that receive just one request. SingleReqView = view_class(:cancelled, :deadline, :metadata, - :output_metadata, :peer, :peer_cert, - :trailing_metadata) + :output_metadata, :peer, :peer_cert) # MultiReqView limits access to an ActiveCall's methods for use in # server client_streamer handlers. MultiReqView = view_class(:cancelled, :deadline, :each_queued_msg, - :each_remote_read, :metadata, :output_metadata, - :trailing_metadata) + :each_remote_read, :metadata, :output_metadata) # Operation limits access to an ActiveCall's methods for use as # a Operation on the client. From f373f2cf8b9d6f8975e1dd976cddb1e3618e8ff9 Mon Sep 17 00:00:00 2001 From: Vijay Pai Date: Fri, 8 Jul 2016 09:40:55 -0700 Subject: [PATCH 0458/1039] Stop holding histogram for a long time --- test/cpp/qps/client.h | 35 +++++++++++++++++++---------------- test/cpp/qps/client_async.cc | 19 +++++++++---------- test/cpp/qps/client_sync.cc | 10 ++++------ 3 files changed, 32 insertions(+), 32 deletions(-) diff --git a/test/cpp/qps/client.h b/test/cpp/qps/client.h index 047bd164082..38478be5d9f 100644 --- a/test/cpp/qps/client.h +++ b/test/cpp/qps/client.h @@ -112,6 +112,17 @@ class ClientRequestCreator { } }; +class HistogramEntry GRPC_FINAL { + public: + HistogramEntry(): used_(false) {} + bool used() const {return used_;} + double value() const {return value_;} + void set_value(double v) {used_ = true; value_ = v;} + private: + bool used_; + double value_; +}; + class Client { public: Client() : timer_(new UsageTimer), interarrival_timer_() {} @@ -162,7 +173,7 @@ class Client { void EndThreads() { threads_.clear(); } - virtual bool ThreadFunc(Histogram* histogram, size_t thread_idx) = 0; + virtual bool ThreadFunc(HistogramEntry* histogram, size_t thread_idx) = 0; void SetupLoadTest(const ClientConfig& config, size_t num_threads) { // Set up the load distribution based on the number of threads @@ -215,7 +226,6 @@ class Client { public: Thread(Client* client, size_t idx) : done_(false), - new_stats_(nullptr), client_(client), idx_(idx), impl_(&Thread::ThreadFunc, this) {} @@ -230,14 +240,10 @@ class Client { void BeginSwap(Histogram* n) { std::lock_guard g(mu_); - new_stats_ = n; + n->Swap(&histogram_); } void EndSwap() { - std::unique_lock g(mu_); - while (new_stats_ != nullptr) { - cv_.wait(g); - }; } void MergeStatsInto(Histogram* hist) { @@ -252,9 +258,13 @@ class Client { void ThreadFunc() { for (;;) { // run the loop body - const bool thread_still_ok = client_->ThreadFunc(&histogram_, idx_); - // lock, see if we're done + HistogramEntry entry; + const bool thread_still_ok = client_->ThreadFunc(&entry, idx_); + // lock, update histogram if needed and see if we're done std::lock_guard g(mu_); + if (entry.used()) { + histogram_.Add(entry.value()); + } if (!thread_still_ok) { gpr_log(GPR_ERROR, "Finishing client thread due to RPC error"); done_ = true; @@ -262,17 +272,10 @@ class Client { if (done_) { return; } - // check if we're resetting stats, swap out the histogram if so - if (new_stats_) { - new_stats_->Swap(&histogram_); - new_stats_ = nullptr; - cv_.notify_one(); - } } } std::mutex mu_; - std::condition_variable cv_; bool done_; Histogram* new_stats_; Histogram histogram_; diff --git a/test/cpp/qps/client_async.cc b/test/cpp/qps/client_async.cc index 1507d1e3d66..c2b69337a3e 100644 --- a/test/cpp/qps/client_async.cc +++ b/test/cpp/qps/client_async.cc @@ -48,7 +48,6 @@ #include #include #include -#include #include #include "src/proto/grpc/testing/services.grpc.pb.h" @@ -64,7 +63,7 @@ class ClientRpcContext { ClientRpcContext() {} virtual ~ClientRpcContext() {} // next state, return false if done. Collect stats when appropriate - virtual bool RunNextState(bool, Histogram* hist) = 0; + virtual bool RunNextState(bool, HistogramEntry* entry) = 0; virtual ClientRpcContext* StartNewClone() = 0; static void* tag(ClientRpcContext* c) { return reinterpret_cast(c); } static ClientRpcContext* detag(void* t) { @@ -104,7 +103,7 @@ class ClientRpcContextUnaryImpl : public ClientRpcContext { alarm_.reset(new Alarm(cq_, next_issue_(), ClientRpcContext::tag(this))); } } - bool RunNextState(bool ok, Histogram* hist) GRPC_OVERRIDE { + bool RunNextState(bool ok, HistogramEntry* entry) GRPC_OVERRIDE { switch (next_state_) { case State::READY: start_ = UsageTimer::Now(); @@ -114,7 +113,7 @@ class ClientRpcContextUnaryImpl : public ClientRpcContext { next_state_ = State::RESP_DONE; return true; case State::RESP_DONE: - hist->Add((UsageTimer::Now() - start_) * 1e9); + entry->set_value((UsageTimer::Now() - start_) * 1e9); callback_(status_, &response_); next_state_ = State::INVALID; return false; @@ -201,7 +200,7 @@ class AsyncClient : public ClientImpl { } } - bool ThreadFunc(Histogram* histogram, + bool ThreadFunc(HistogramEntry* entry, size_t thread_idx) GRPC_OVERRIDE GRPC_FINAL { void* got_tag; bool ok; @@ -209,7 +208,7 @@ class AsyncClient : public ClientImpl { if (cli_cqs_[thread_idx]->Next(&got_tag, &ok)) { // Got a regular event, so process it ClientRpcContext* ctx = ClientRpcContext::detag(got_tag); - if (!ctx->RunNextState(ok, histogram)) { + if (!ctx->RunNextState(ok, entry)) { // The RPC and callback are done, so clone the ctx // and kickstart the new one auto clone = ctx->StartNewClone(); @@ -298,7 +297,7 @@ class ClientRpcContextStreamingImpl : public ClientRpcContext { stream_ = start_req_(stub_, &context_, cq, ClientRpcContext::tag(this)); next_state_ = State::STREAM_IDLE; } - bool RunNextState(bool ok, Histogram* hist) GRPC_OVERRIDE { + bool RunNextState(bool ok, HistogramEntry* entry) GRPC_OVERRIDE { while (true) { switch (next_state_) { case State::STREAM_IDLE: @@ -330,7 +329,7 @@ class ClientRpcContextStreamingImpl : public ClientRpcContext { return true; break; case State::READ_DONE: - hist->Add((UsageTimer::Now() - start_) * 1e9); + entry->set_value((UsageTimer::Now() - start_) * 1e9); callback_(status_, &response_); next_state_ = State::STREAM_IDLE; break; // loop around @@ -430,7 +429,7 @@ class ClientRpcContextGenericStreamingImpl : public ClientRpcContext { ClientRpcContext::tag(this)); next_state_ = State::STREAM_IDLE; } - bool RunNextState(bool ok, Histogram* hist) GRPC_OVERRIDE { + bool RunNextState(bool ok, HistogramEntry* entry) GRPC_OVERRIDE { while (true) { switch (next_state_) { case State::STREAM_IDLE: @@ -462,7 +461,7 @@ class ClientRpcContextGenericStreamingImpl : public ClientRpcContext { return true; break; case State::READ_DONE: - hist->Add((UsageTimer::Now() - start_) * 1e9); + entry->set_value((UsageTimer::Now() - start_) * 1e9); callback_(status_, &response_); next_state_ = State::STREAM_IDLE; break; // loop around diff --git a/test/cpp/qps/client_sync.cc b/test/cpp/qps/client_sync.cc index c88e95b80e5..f328f492e3d 100644 --- a/test/cpp/qps/client_sync.cc +++ b/test/cpp/qps/client_sync.cc @@ -46,7 +46,6 @@ #include #include #include -#include #include #include #include @@ -55,7 +54,6 @@ #include "src/core/lib/profiling/timers.h" #include "src/proto/grpc/testing/services.grpc.pb.h" #include "test/cpp/qps/client.h" -#include "test/cpp/qps/histogram.h" #include "test/cpp/qps/interarrival.h" #include "test/cpp/qps/usage_timer.h" @@ -100,7 +98,7 @@ class SynchronousUnaryClient GRPC_FINAL : public SynchronousClient { } ~SynchronousUnaryClient() { EndThreads(); } - bool ThreadFunc(Histogram* histogram, size_t thread_idx) GRPC_OVERRIDE { + bool ThreadFunc(HistogramEntry* entry, size_t thread_idx) GRPC_OVERRIDE { WaitToIssue(thread_idx); auto* stub = channels_[thread_idx % channels_.size()].get_stub(); double start = UsageTimer::Now(); @@ -108,7 +106,7 @@ class SynchronousUnaryClient GRPC_FINAL : public SynchronousClient { grpc::ClientContext context; grpc::Status s = stub->UnaryCall(&context, request_, &responses_[thread_idx]); - histogram->Add((UsageTimer::Now() - start) * 1e9); + entry->set_value((UsageTimer::Now() - start) * 1e9); return s.ok(); } }; @@ -139,13 +137,13 @@ class SynchronousStreamingClient GRPC_FINAL : public SynchronousClient { delete[] context_; } - bool ThreadFunc(Histogram* histogram, size_t thread_idx) GRPC_OVERRIDE { + bool ThreadFunc(HistogramEntry* entry, size_t thread_idx) GRPC_OVERRIDE { WaitToIssue(thread_idx); GPR_TIMER_SCOPE("SynchronousStreamingClient::ThreadFunc", 0); double start = UsageTimer::Now(); if (stream_[thread_idx]->Write(request_) && stream_[thread_idx]->Read(&responses_[thread_idx])) { - histogram->Add((UsageTimer::Now() - start) * 1e9); + entry->set_value((UsageTimer::Now() - start) * 1e9); return true; } return false; From f782465fba11864293d858ba91d5e715fc481d7d Mon Sep 17 00:00:00 2001 From: Vijay Pai Date: Fri, 8 Jul 2016 10:33:10 -0700 Subject: [PATCH 0459/1039] Fix some shutdown errors related to CQ/join ordering --- test/cpp/qps/client_async.cc | 43 +++++++++++++++++++++++++----------- test/cpp/qps/qps_worker.cc | 2 ++ 2 files changed, 32 insertions(+), 13 deletions(-) diff --git a/test/cpp/qps/client_async.cc b/test/cpp/qps/client_async.cc index 963a1e1cd05..057e5a0d6be 100644 --- a/test/cpp/qps/client_async.cc +++ b/test/cpp/qps/client_async.cc @@ -189,14 +189,7 @@ class AsyncClient : public ClientImpl { } } virtual ~AsyncClient() { - for (auto cq = cli_cqs_.begin(); cq != cli_cqs_.end(); cq++) { - (*cq)->Shutdown(); - void* got_tag; - bool ok; - while ((*cq)->Next(&got_tag, &ok)) { - delete ClientRpcContext::detag(got_tag); - } - } + FinalShutdownCQs(); } bool ThreadFunc(HistogramEntry* entry, @@ -216,14 +209,29 @@ class AsyncClient : public ClientImpl { delete ctx; } return true; - } else { // queue is shutting down - return false; + } else { // queue is shutting down, so we must be done + return true; } } protected: const int num_async_threads_; + void ShutdownCQs() { + for (auto cq = cli_cqs_.begin(); cq != cli_cqs_.end(); cq++) { + (*cq)->Shutdown(); + } + } + void FinalShutdownCQs() { + for (auto cq = cli_cqs_.begin(); cq != cli_cqs_.end(); cq++) { + void* got_tag; + bool ok; + while ((*cq)->Next(&got_tag, &ok)) { + delete ClientRpcContext::detag(got_tag); + } + } + } + private: int NumThreads(const ClientConfig& config) { int num_threads = config.async_client_threads(); @@ -251,7 +259,10 @@ class AsyncUnaryClient GRPC_FINAL config, SetupCtx, BenchmarkStubCreator) { StartThreads(num_async_threads_); } - ~AsyncUnaryClient() GRPC_OVERRIDE { EndThreads(); } + ~AsyncUnaryClient() GRPC_OVERRIDE { + ShutdownCQs(); + EndThreads(); + } private: static void CheckDone(grpc::Status s, SimpleResponse* response) {} @@ -380,7 +391,10 @@ class AsyncStreamingClient GRPC_FINAL StartThreads(num_async_threads_); } - ~AsyncStreamingClient() GRPC_OVERRIDE { EndThreads(); } + ~AsyncStreamingClient() GRPC_OVERRIDE { + ShutdownCQs(); + EndThreads(); + } private: static void CheckDone(grpc::Status s, SimpleResponse* response) {} @@ -516,7 +530,10 @@ class GenericAsyncStreamingClient GRPC_FINAL StartThreads(num_async_threads_); } - ~GenericAsyncStreamingClient() GRPC_OVERRIDE { EndThreads(); } + ~GenericAsyncStreamingClient() GRPC_OVERRIDE { + ShutdownCQs(); + EndThreads(); + } private: static void CheckDone(grpc::Status s, ByteBuffer* response) {} diff --git a/test/cpp/qps/qps_worker.cc b/test/cpp/qps/qps_worker.cc index 8456fde0ed3..49ef52895c2 100644 --- a/test/cpp/qps/qps_worker.cc +++ b/test/cpp/qps/qps_worker.cc @@ -128,6 +128,7 @@ class WorkerServiceImpl GRPC_FINAL : public WorkerService::Service { ScopedProfile profile("qps_client.prof", false); Status ret = RunClientBody(ctx, stream); + gpr_log(GPR_INFO, "RunClient: Returning"); return ret; } @@ -141,6 +142,7 @@ class WorkerServiceImpl GRPC_FINAL : public WorkerService::Service { ScopedProfile profile("qps_server.prof", false); Status ret = RunServerBody(ctx, stream); + gpr_log(GPR_INFO, "RunServer: Returning"); return ret; } From bcd1015320200ecf3222f885023e2c50003f39c4 Mon Sep 17 00:00:00 2001 From: Stanley Cheung Date: Fri, 8 Jul 2016 10:40:33 -0700 Subject: [PATCH 0460/1039] php: update composer.json require-dev --- composer.json | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/composer.json b/composer.json index 05ac0037146..6e7f24b451e 100644 --- a/composer.json +++ b/composer.json @@ -13,8 +13,10 @@ ], "require": { "php": ">=5.5.0", - "datto/protobuf-php": "dev-master", - "google/auth": "v0.7" + "datto/protobuf-php": "dev-master" + }, + "require-dev": { + "google/auth": "v0.9" }, "autoload": { "psr-4": { From 2b220f8be7c80d4d51f9506f359608fce3b78ccb Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Fri, 8 Jul 2016 10:47:57 -0700 Subject: [PATCH 0461/1039] fixes #7219 Modified objc plugin to add two types of import statements for well known protos. Deleted unnecessary empty.proto as we'll be using wellknown empty.proto Modified test to use the well known empty.proto. --- src/compiler/objective_c_plugin.cc | 20 ++++++++- src/objective-c/tests/InteropTests.m | 12 ++--- .../tests/RemoteTestClient/RemoteTest.podspec | 4 +- .../tests/RemoteTestClient/empty.proto | 44 ------------------- .../tests/RemoteTestClient/test.proto | 4 +- 5 files changed, 29 insertions(+), 55 deletions(-) delete mode 100644 src/objective-c/tests/RemoteTestClient/empty.proto diff --git a/src/compiler/objective_c_plugin.cc b/src/compiler/objective_c_plugin.cc index 3ccfd5b037c..43e4b3647d0 100644 --- a/src/compiler/objective_c_plugin.cc +++ b/src/compiler/objective_c_plugin.cc @@ -39,6 +39,8 @@ #include "src/compiler/objective_c_generator.h" #include "src/compiler/objective_c_generator_helpers.h" +#include + class ObjectiveCGrpcGenerator : public grpc::protobuf::compiler::CodeGenerator { public: ObjectiveCGrpcGenerator() {} @@ -72,7 +74,21 @@ class ObjectiveCGrpcGenerator : public grpc::protobuf::compiler::CodeGenerator { for (int i = 0; i < file->dependency_count(); i++) { ::grpc::string header = grpc_objective_c_generator::MessageHeaderName( file->dependency(i)); - proto_imports += ::grpc::string("#import \"") + header + "\"\n"; + const grpc::protobuf::FileDescriptor *dependency = file->dependency(i); + if (::google::protobuf::compiler::objectivec::IsProtobufLibraryBundledProtoFile(dependency)) { + ::grpc::string base_name = header; + grpc_generator::StripPrefix(&base_name, "google/protobuf/"); + proto_imports += + ::grpc::string("#if GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS\n") + + ::grpc::string(" #import <") + + ::google::protobuf::compiler::objectivec::ProtobufLibraryFrameworkName + + ("/") + base_name + ">\n" + + ::grpc::string("#else\n") + + ::grpc::string(" #import \"") + header + "\"\n" + + ::grpc::string("#endif\n"); + } else { + proto_imports += ::grpc::string("#import \"") + header + "\"\n"; + } } ::grpc::string declarations; @@ -85,7 +101,7 @@ class ObjectiveCGrpcGenerator : public grpc::protobuf::compiler::CodeGenerator { static const ::grpc::string kNonNullEnd = "\nNS_ASSUME_NONNULL_END\n"; Write(context, file_name + ".pbrpc.h", - imports + '\n' + proto_imports + '\n' + kNonNullBegin + + imports + '\n' + proto_imports + '\n' + kNonNullBegin + declarations + kNonNullEnd); } diff --git a/src/objective-c/tests/InteropTests.m b/src/objective-c/tests/InteropTests.m index a503f020591..a2f63ac40ca 100644 --- a/src/objective-c/tests/InteropTests.m +++ b/src/objective-c/tests/InteropTests.m @@ -110,12 +110,12 @@ static cronet_engine *cronetEngine = NULL; XCTAssertNotNil(self.class.host); __weak XCTestExpectation *expectation = [self expectationWithDescription:@"EmptyUnary"]; - RMTEmpty *request = [RMTEmpty message]; + GPBEmpty *request = [GPBEmpty message]; - [_service emptyCallWithRequest:request handler:^(RMTEmpty *response, NSError *error) { + [_service emptyCallWithRequest:request handler:^(GPBEmpty *response, NSError *error) { XCTAssertNil(error, @"Finished with unexpected error: %@", error); - id expectedResponse = [RMTEmpty message]; + id expectedResponse = [GPBEmpty message]; XCTAssertEqualObjects(response, expectedResponse); [expectation fulfill]; @@ -343,9 +343,9 @@ static cronet_engine *cronetEngine = NULL; __weak XCTestExpectation *expectation = [self expectationWithDescription:@"RPC after closing connection"]; - RMTEmpty *request = [RMTEmpty message]; + GPBEmpty *request = [GPBEmpty message]; - [_service emptyCallWithRequest:request handler:^(RMTEmpty *response, NSError *error) { + [_service emptyCallWithRequest:request handler:^(GPBEmpty *response, NSError *error) { XCTAssertNil(error, @"First RPC finished with unexpected error: %@", error); #pragma clang diagnostic push @@ -353,7 +353,7 @@ static cronet_engine *cronetEngine = NULL; [GRPCCall closeOpenConnections]; #pragma clang diagnostic pop - [_service emptyCallWithRequest:request handler:^(RMTEmpty *response, NSError *error) { + [_service emptyCallWithRequest:request handler:^(GPBEmpty *response, NSError *error) { XCTAssertNil(error, @"Second RPC finished with unexpected error: %@", error); [expectation fulfill]; }]; diff --git a/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec b/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec index 887380eb55f..25c9c7f8418 100644 --- a/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec +++ b/src/objective-c/tests/RemoteTestClient/RemoteTest.podspec @@ -15,7 +15,9 @@ Pod::Spec.new do |s| BINDIR=../../../../bins/$CONFIG PROTOC=$BINDIR/protobuf/protoc PLUGIN=$BINDIR/grpc_objective_c_plugin - $PROTOC --plugin=protoc-gen-grpc=$PLUGIN --objc_out=. --grpc_out=. *.proto + # we use this path to locate well-known proto files + PROTO_SRC=../../../../third_party/protobuf/src + $PROTOC --plugin=protoc-gen-grpc=$PLUGIN --objc_out=. --grpc_out=. *.proto -I $PROTO_SRC -I . CMD s.subspec "Messages" do |ms| diff --git a/src/objective-c/tests/RemoteTestClient/empty.proto b/src/objective-c/tests/RemoteTestClient/empty.proto deleted file mode 100644 index a678048289e..00000000000 --- a/src/objective-c/tests/RemoteTestClient/empty.proto +++ /dev/null @@ -1,44 +0,0 @@ -// Copyright 2015, Google Inc. -// All rights reserved. -// -// Redistribution and use in source and binary forms, with or without -// modification, are permitted provided that the following conditions are -// met: -// -// * Redistributions of source code must retain the above copyright -// notice, this list of conditions and the following disclaimer. -// * Redistributions in binary form must reproduce the above -// copyright notice, this list of conditions and the following disclaimer -// in the documentation and/or other materials provided with the -// distribution. -// * Neither the name of Google Inc. nor the names of its -// contributors may be used to endorse or promote products derived from -// this software without specific prior written permission. -// -// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -syntax = "proto3"; - -package grpc.testing; - -option objc_class_prefix = "RMT"; - -// An empty message that you can re-use to avoid defining duplicated empty -// messages in your project. A typical example is to use it as argument or the -// return value of a service API. For instance: -// -// service Foo { -// rpc Bar (grpc.testing.Empty) returns (grpc.testing.Empty) { }; -// }; -// -message Empty {} diff --git a/src/objective-c/tests/RemoteTestClient/test.proto b/src/objective-c/tests/RemoteTestClient/test.proto index 514c3b80955..5c359c5c129 100644 --- a/src/objective-c/tests/RemoteTestClient/test.proto +++ b/src/objective-c/tests/RemoteTestClient/test.proto @@ -31,7 +31,7 @@ // of unary/streaming requests/responses. syntax = "proto3"; -import "empty.proto"; +import "google/protobuf/empty.proto"; import "messages.proto"; package grpc.testing; @@ -42,7 +42,7 @@ option objc_class_prefix = "RMT"; // performance with various types of payload. service TestService { // One empty request followed by one empty response. - rpc EmptyCall(grpc.testing.Empty) returns (grpc.testing.Empty); + rpc EmptyCall(google.protobuf.Empty) returns (google.protobuf.Empty); // One request followed by one response. rpc UnaryCall(SimpleRequest) returns (SimpleResponse); From 771dc7546a34fe2d93159c443cbb58244b0b2274 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Mon, 6 Jun 2016 19:25:36 -0700 Subject: [PATCH 0462/1039] Remove misleading diagnostics message --- src/python/grpcio/support.py | 1 - 1 file changed, 1 deletion(-) diff --git a/src/python/grpcio/support.py b/src/python/grpcio/support.py index 33244eb388e..7730374df08 100644 --- a/src/python/grpcio/support.py +++ b/src/python/grpcio/support.py @@ -50,7 +50,6 @@ Could not find . This could mean the following: (check your environment variables or try re-installing?) * You're on Windows and your Python installation was somehow corrupted (check your environment variables or try re-installing?) - * Note: Windows users should look into installing `vcpython27`. """ C_CHECKS = { From af26ce6f4383a6c0e70d4c58f276ec4f2a722dfa Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Sun, 5 Jun 2016 16:47:37 -0700 Subject: [PATCH 0463/1039] Remove unnecessary fcntl module import --- src/python/grpcio_tests/tests/_runner.py | 1 - 1 file changed, 1 deletion(-) diff --git a/src/python/grpcio_tests/tests/_runner.py b/src/python/grpcio_tests/tests/_runner.py index f0718573e2c..81c6100feda 100644 --- a/src/python/grpcio_tests/tests/_runner.py +++ b/src/python/grpcio_tests/tests/_runner.py @@ -30,7 +30,6 @@ from __future__ import absolute_import import collections -import fcntl import multiprocessing import os import select From 46cc9eebb17b2711515007d7af9108cd0d78788a Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Fri, 8 Jul 2016 11:42:07 -0700 Subject: [PATCH 0464/1039] addressed feedback using 'using'. Removed unnecessary grpc::string. fixed indentation. --- src/compiler/objective_c_plugin.cc | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/src/compiler/objective_c_plugin.cc b/src/compiler/objective_c_plugin.cc index 43e4b3647d0..3878a332515 100644 --- a/src/compiler/objective_c_plugin.cc +++ b/src/compiler/objective_c_plugin.cc @@ -41,6 +41,9 @@ #include +using ::google::protobuf::compiler::objectivec::ProtobufLibraryFrameworkName; +using ::google::protobuf::compiler::objectivec::IsProtobufLibraryBundledProtoFile; + class ObjectiveCGrpcGenerator : public grpc::protobuf::compiler::CodeGenerator { public: ObjectiveCGrpcGenerator() {} @@ -75,17 +78,16 @@ class ObjectiveCGrpcGenerator : public grpc::protobuf::compiler::CodeGenerator { ::grpc::string header = grpc_objective_c_generator::MessageHeaderName( file->dependency(i)); const grpc::protobuf::FileDescriptor *dependency = file->dependency(i); - if (::google::protobuf::compiler::objectivec::IsProtobufLibraryBundledProtoFile(dependency)) { + if (IsProtobufLibraryBundledProtoFile(dependency)) { ::grpc::string base_name = header; grpc_generator::StripPrefix(&base_name, "google/protobuf/"); proto_imports += - ::grpc::string("#if GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS\n") + - ::grpc::string(" #import <") + - ::google::protobuf::compiler::objectivec::ProtobufLibraryFrameworkName + - ("/") + base_name + ">\n" + - ::grpc::string("#else\n") + - ::grpc::string(" #import \"") + header + "\"\n" + - ::grpc::string("#endif\n"); + "#if GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS\n" + " #import <" + ::grpc::string(ProtobufLibraryFrameworkName) + + "/" + base_name + ">\n" + "#else\n" + " #import \"" + header + "\"\n" + "#endif\n"; } else { proto_imports += ::grpc::string("#import \"") + header + "\"\n"; } From f17f0f6b071fe42f0d57e6b2f08d797bcbcad88a Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Fri, 8 Jul 2016 12:07:49 -0700 Subject: [PATCH 0465/1039] Fallback to generating files if not generated Even if GRPC_PYTHON_BUILD_WITH_CYTHON is not specified, if the files are not present then we will fall back to generating with Cython. This relegates GRPC_PYTHON_BUILD_WITH_CYTHON to providing a regeneration option rather than being a necessary build environment variable. --- setup.py | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/setup.py b/setup.py index d9c46ba77a1..2eeb37db830 100644 --- a/setup.py +++ b/setup.py @@ -64,7 +64,9 @@ LICENSE = '3-clause BSD' # Environment variable to determine whether or not the Cython extension should # *use* Cython or use the generated C files. Note that this requires the C files -# to have been generated by building first *with* Cython support. +# to have been generated by building first *with* Cython support. Even if this +# is set to false, if the script detects that the generated `.c` file isn't +# present, then it will still attempt to use Cython. BUILD_WITH_CYTHON = os.environ.get('GRPC_PYTHON_BUILD_WITH_CYTHON', False) # Environment variable to determine whether or not to enable coverage analysis @@ -129,10 +131,20 @@ def cython_extensions(module_names, extra_sources, include_dirs, if ENABLE_CYTHON_TRACING: define_macros = define_macros + [('CYTHON_TRACE_NOGIL', 1)] cython_compiler_directives['linetrace'] = True - file_extension = 'pyx' if build_with_cython else 'c' - module_files = [os.path.join(PYTHON_STEM, - name.replace('.', '/') + '.' + file_extension) - for name in module_names] + pyx_module_files = [os.path.join(PYTHON_STEM, + name.replace('.', '/') + '.pyx') + for name in module_names] + c_module_files = [os.path.join(PYTHON_STEM, + name.replace('.', '/') + '.c') + for name in module_names] + if not build_with_cython: + for module_file in c_module_files: + if not os.path.isfile(module_file): + sys.stderr.write('Cython-generated files are missing; ' + 'forcing Cython build...\n') + build_with_cython = True + break + module_files = pyx_module_files if build_with_cython else c_module_files extensions = [ _extension.Extension( name=module_name, From 6c54078d2ee377a1c12ded8e031353ee5125ac2f Mon Sep 17 00:00:00 2001 From: Dan Born Date: Tue, 28 Jun 2016 16:34:41 -0700 Subject: [PATCH 0466/1039] Set siblings for server clones properly. --- src/core/lib/iomgr/tcp_server_posix.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/core/lib/iomgr/tcp_server_posix.c b/src/core/lib/iomgr/tcp_server_posix.c index a1a463550ae..5d2ebe2e7cf 100644 --- a/src/core/lib/iomgr/tcp_server_posix.c +++ b/src/core/lib/iomgr/tcp_server_posix.c @@ -512,8 +512,9 @@ static grpc_error *clone_port(grpc_tcp_listener *listener, unsigned count) { sp->port = port; sp->port_index = listener->port_index; sp->fd_index = listener->fd_index + count - i; + listener->sibling = sp; sp->is_sibling = 1; - sp->sibling = listener->is_sibling ? listener->sibling : listener; + sp->sibling = listener->sibling; GPR_ASSERT(sp->emfd); while (listener->server->tail->next != NULL) { listener->server->tail = listener->server->tail->next; From 586e3835fe01257299a8573df3e7540e5778bc45 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Fri, 3 Jun 2016 19:29:12 -0700 Subject: [PATCH 0467/1039] Make Python build standalone on Windows --- PYTHON-MANIFEST.in | 1 + setup.py | 29 +- src/python/grpcio/build.py | 117 +++ .../grpcio/grpc/_cython/imports.generated.c | 557 +----------- .../grpcio/grpc/_cython/imports.generated.h | 852 +----------------- src/python/grpcio/grpc/_cython/loader.c | 29 +- src/python/grpcio/grpc/_cython/loader.h | 2 + .../grpc/_cython/imports.generated.c.template | 22 +- .../grpc/_cython/imports.generated.h.template | 29 +- tools/distrib/python/grpcio_tools/setup.py | 15 +- tools/run_tests/build_python.sh | 15 +- 11 files changed, 172 insertions(+), 1496 deletions(-) create mode 100644 src/python/grpcio/build.py diff --git a/PYTHON-MANIFEST.in b/PYTHON-MANIFEST.in index 635e77b875c..175a47f1576 100644 --- a/PYTHON-MANIFEST.in +++ b/PYTHON-MANIFEST.in @@ -7,6 +7,7 @@ graft include/grpc graft third_party/boringssl graft third_party/nanopb graft third_party/zlib +include src/python/grpcio/build.py include src/python/grpcio/commands.py include src/python/grpcio/grpc_version.py include src/python/grpcio/grpc_core_dependencies.py diff --git a/setup.py b/setup.py index d9c46ba77a1..c6d3da72994 100644 --- a/setup.py +++ b/setup.py @@ -31,6 +31,7 @@ import os import os.path +import platform import shlex import shutil import sys @@ -56,10 +57,15 @@ os.chdir(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, os.path.abspath(PYTHON_STEM)) # Break import-style to ensure we can actually find our in-repo dependencies. +import build import commands import grpc_core_dependencies import grpc_version +# TODO(atash) make this conditional on being on a mingw32 build +build.monkeypatch_unix_compiler() + + LICENSE = '3-clause BSD' # Environment variable to determine whether or not the Cython extension should @@ -72,6 +78,14 @@ BUILD_WITH_CYTHON = os.environ.get('GRPC_PYTHON_BUILD_WITH_CYTHON', False) ENABLE_CYTHON_TRACING = os.environ.get( 'GRPC_PYTHON_ENABLE_CYTHON_TRACING', False) +# There are some situations (like on Windows) where CC, CFLAGS, and LDFLAGS are +# entirely ignored/dropped/forgotten by distutils and its Cygwin/MinGW support. +# We use these environment variables to thus get around that without locking +# ourselves in w.r.t. the multitude of operating systems this ought to build on. +# By default we assume a GCC-like compiler. +EXTRA_COMPILE_ARGS = shlex.split(os.environ.get('GRPC_PYTHON_CFLAGS', '')) +EXTRA_LINK_ARGS = shlex.split(os.environ.get('GRPC_PYTHON_LDFLAGS', '')) + CYTHON_EXTENSION_PACKAGE_NAMES = () CYTHON_EXTENSION_MODULE_NAMES = ('grpc._cython.cygrpc',) @@ -81,9 +95,7 @@ CYTHON_HELPER_C_FILES = ( os.path.join(PYTHON_STEM, 'grpc/_cython/imports.generated.c'), ) -CORE_C_FILES = () -if not "win32" in sys.platform: - CORE_C_FILES += tuple(grpc_core_dependencies.CORE_SOURCE_FILES) +CORE_C_FILES = tuple(grpc_core_dependencies.CORE_SOURCE_FILES) EXTENSION_INCLUDE_DIRECTORIES = ( (PYTHON_STEM,) + CORE_INCLUDE + BORINGSSL_INCLUDE + ZLIB_INCLUDE) @@ -93,12 +105,17 @@ if "linux" in sys.platform: EXTENSION_LIBRARIES += ('rt',) if not "win32" in sys.platform: EXTENSION_LIBRARIES += ('m',) +if "win32" in sys.platform: + EXTENSION_LIBRARIES += ('ws2_32',) DEFINE_MACROS = (('OPENSSL_NO_ASM', 1), ('_WIN32_WINNT', 0x600), ('GPR_BACKWARDS_COMPATIBILITY_MODE', 1),) +if "win32" in sys.platform: + DEFINE_MACROS += (('OPENSSL_WINDOWS', 1), ('WIN32_LEAN_AND_MEAN', 1),) + if '64bit' in platform.architecture()[0]: + DEFINE_MACROS += (('MS_WIN64', 1),) -LDFLAGS = shlex.split(os.environ.get('GRPC_PYTHON_LDFLAGS', '')) -CFLAGS = shlex.split(os.environ.get('GRPC_PYTHON_CFLAGS', '')) - +LDFLAGS = tuple(EXTRA_LINK_ARGS) +CFLAGS = tuple(EXTRA_COMPILE_ARGS) if "linux" in sys.platform: LDFLAGS += ('-Wl,-wrap,memcpy',) if "linux" in sys.platform or "darwin" in sys.platform: diff --git a/src/python/grpcio/build.py b/src/python/grpcio/build.py new file mode 100644 index 00000000000..df5b54cf69c --- /dev/null +++ b/src/python/grpcio/build.py @@ -0,0 +1,117 @@ +# Copyright 2016, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +"""Covers inadequacies in distutils.""" + +from distutils import ccompiler +from distutils import errors +from distutils import unixccompiler +import os +import os.path +import shutil +import sys +import tempfile + + +def _unix_piecemeal_link( + self, target_desc, objects, output_filename, output_dir=None, + libraries=None, library_dirs=None, runtime_library_dirs=None, + export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, + build_temp=None, target_lang=None): + """`link` externalized method taken almost verbatim from UnixCCompiler. + + Modifies the link command for unix-like compilers by using a command file so + that long command line argument strings don't break the command shell's + ARG_MAX character limit. + """ + objects, output_dir = self._fix_object_args(objects, output_dir) + libraries, library_dirs, runtime_library_dirs = self._fix_lib_args( + libraries, library_dirs, runtime_library_dirs) + # filter out standard library paths, which are not explicitely needed + # for linking + library_dirs = [dir for dir in library_dirs + if not dir in ('/lib', '/lib64', '/usr/lib', '/usr/lib64')] + runtime_library_dirs = [dir for dir in runtime_library_dirs + if not dir in ('/lib', '/lib64', '/usr/lib', '/usr/lib64')] + lib_opts = ccompiler.gen_lib_options(self, library_dirs, runtime_library_dirs, + libraries) + if not isinstance(output_dir, basestring) and output_dir is not None: + raise TypeError, "'output_dir' must be a string or None" + if output_dir is not None: + output_filename = os.path.join(output_dir, output_filename) + + if self._need_link(objects, output_filename): + ld_args = (objects + self.objects + + lib_opts + ['-o', output_filename]) + if debug: + ld_args[:0] = ['-g'] + if extra_preargs: + ld_args[:0] = extra_preargs + if extra_postargs: + ld_args.extend(extra_postargs) + self.mkpath(os.path.dirname(output_filename)) + try: + if target_desc == ccompiler.CCompiler.EXECUTABLE: + linker = self.linker_exe[:] + else: + linker = self.linker_so[:] + if target_lang == "c++" and self.compiler_cxx: + # skip over environment variable settings if /usr/bin/env + # is used to set up the linker's environment. + # This is needed on OSX. Note: this assumes that the + # normal and C++ compiler have the same environment + # settings. + i = 0 + if os.path.basename(linker[0]) == "env": + i = 1 + while '=' in linker[i]: + i = i + 1 + + linker[i] = self.compiler_cxx[i] + + if sys.platform == 'darwin': + import _osx_support + linker = _osx_support.compiler_fixup(linker, ld_args) + + temporary_directory = tempfile.mkdtemp() + command_filename = os.path.abspath( + os.path.join(temporary_directory, 'command')) + with open(command_filename, 'w') as command_file: + escaped_ld_args = [arg.replace('\\', '\\\\') for arg in ld_args] + command_file.write(' '.join(escaped_ld_args)) + self.spawn(linker + ['@{}'.format(command_filename)]) + except errors.DistutilsExecError, msg: + raise ccompiler.LinkError, msg + else: + log.debug("skipping %s (up-to-date)", output_filename) + +def monkeypatch_unix_compiler(): + """Monkeypatching is dumb, but it's either that or we become maintainers of + something much, much bigger.""" + unixccompiler.UnixCCompiler.link = _unix_piecemeal_link diff --git a/src/python/grpcio/grpc/_cython/imports.generated.c b/src/python/grpcio/grpc/_cython/imports.generated.c index d78ec2f66ed..c0080b5a47a 100644 --- a/src/python/grpcio/grpc/_cython/imports.generated.c +++ b/src/python/grpcio/grpc/_cython/imports.generated.c @@ -31,562 +31,7 @@ * */ +/* TODO(atash) remove cruft */ #include #include "imports.generated.h" - -#ifdef GPR_WINDOWS - -census_initialize_type census_initialize_import; -census_shutdown_type census_shutdown_import; -census_supported_type census_supported_import; -census_enabled_type census_enabled_import; -census_context_create_type census_context_create_import; -census_context_destroy_type census_context_destroy_import; -census_context_get_status_type census_context_get_status_import; -census_context_initialize_iterator_type census_context_initialize_iterator_import; -census_context_next_tag_type census_context_next_tag_import; -census_context_get_tag_type census_context_get_tag_import; -census_context_encode_type census_context_encode_import; -census_context_decode_type census_context_decode_import; -census_trace_mask_type census_trace_mask_import; -census_set_trace_mask_type census_set_trace_mask_import; -census_start_rpc_op_timestamp_type census_start_rpc_op_timestamp_import; -census_start_client_rpc_op_type census_start_client_rpc_op_import; -census_set_rpc_client_peer_type census_set_rpc_client_peer_import; -census_start_server_rpc_op_type census_start_server_rpc_op_import; -census_start_op_type census_start_op_import; -census_end_op_type census_end_op_import; -census_trace_print_type census_trace_print_import; -census_trace_scan_start_type census_trace_scan_start_import; -census_get_trace_record_type census_get_trace_record_import; -census_trace_scan_end_type census_trace_scan_end_import; -census_record_values_type census_record_values_import; -census_view_create_type census_view_create_import; -census_view_delete_type census_view_delete_import; -census_view_metric_type census_view_metric_import; -census_view_naggregations_type census_view_naggregations_import; -census_view_tags_type census_view_tags_import; -census_view_aggregrations_type census_view_aggregrations_import; -census_view_get_data_type census_view_get_data_import; -census_view_reset_type census_view_reset_import; -grpc_compression_algorithm_parse_type grpc_compression_algorithm_parse_import; -grpc_compression_algorithm_name_type grpc_compression_algorithm_name_import; -grpc_compression_algorithm_for_level_type grpc_compression_algorithm_for_level_import; -grpc_compression_options_init_type grpc_compression_options_init_import; -grpc_compression_options_enable_algorithm_type grpc_compression_options_enable_algorithm_import; -grpc_compression_options_disable_algorithm_type grpc_compression_options_disable_algorithm_import; -grpc_compression_options_is_algorithm_enabled_type grpc_compression_options_is_algorithm_enabled_import; -grpc_metadata_array_init_type grpc_metadata_array_init_import; -grpc_metadata_array_destroy_type grpc_metadata_array_destroy_import; -grpc_call_details_init_type grpc_call_details_init_import; -grpc_call_details_destroy_type grpc_call_details_destroy_import; -grpc_register_plugin_type grpc_register_plugin_import; -grpc_init_type grpc_init_import; -grpc_shutdown_type grpc_shutdown_import; -grpc_version_string_type grpc_version_string_import; -grpc_completion_queue_create_type grpc_completion_queue_create_import; -grpc_completion_queue_next_type grpc_completion_queue_next_import; -grpc_completion_queue_pluck_type grpc_completion_queue_pluck_import; -grpc_completion_queue_shutdown_type grpc_completion_queue_shutdown_import; -grpc_completion_queue_destroy_type grpc_completion_queue_destroy_import; -grpc_alarm_create_type grpc_alarm_create_import; -grpc_alarm_cancel_type grpc_alarm_cancel_import; -grpc_alarm_destroy_type grpc_alarm_destroy_import; -grpc_channel_check_connectivity_state_type grpc_channel_check_connectivity_state_import; -grpc_channel_watch_connectivity_state_type grpc_channel_watch_connectivity_state_import; -grpc_channel_create_call_type grpc_channel_create_call_import; -grpc_channel_ping_type grpc_channel_ping_import; -grpc_channel_register_call_type grpc_channel_register_call_import; -grpc_channel_create_registered_call_type grpc_channel_create_registered_call_import; -grpc_call_start_batch_type grpc_call_start_batch_import; -grpc_call_get_peer_type grpc_call_get_peer_import; -grpc_census_call_set_context_type grpc_census_call_set_context_import; -grpc_census_call_get_context_type grpc_census_call_get_context_import; -grpc_channel_get_target_type grpc_channel_get_target_import; -grpc_insecure_channel_create_type grpc_insecure_channel_create_import; -grpc_lame_client_channel_create_type grpc_lame_client_channel_create_import; -grpc_channel_destroy_type grpc_channel_destroy_import; -grpc_call_cancel_type grpc_call_cancel_import; -grpc_call_cancel_with_status_type grpc_call_cancel_with_status_import; -grpc_call_destroy_type grpc_call_destroy_import; -grpc_server_request_call_type grpc_server_request_call_import; -grpc_server_register_method_type grpc_server_register_method_import; -grpc_server_request_registered_call_type grpc_server_request_registered_call_import; -grpc_server_create_type grpc_server_create_import; -grpc_server_register_completion_queue_type grpc_server_register_completion_queue_import; -grpc_server_register_non_listening_completion_queue_type grpc_server_register_non_listening_completion_queue_import; -grpc_server_add_insecure_http2_port_type grpc_server_add_insecure_http2_port_import; -grpc_server_start_type grpc_server_start_import; -grpc_server_shutdown_and_notify_type grpc_server_shutdown_and_notify_import; -grpc_server_cancel_all_calls_type grpc_server_cancel_all_calls_import; -grpc_server_destroy_type grpc_server_destroy_import; -grpc_tracer_set_enabled_type grpc_tracer_set_enabled_import; -grpc_header_key_is_legal_type grpc_header_key_is_legal_import; -grpc_header_nonbin_value_is_legal_type grpc_header_nonbin_value_is_legal_import; -grpc_is_binary_header_type grpc_is_binary_header_import; -grpc_call_error_to_string_type grpc_call_error_to_string_import; -grpc_insecure_channel_create_from_fd_type grpc_insecure_channel_create_from_fd_import; -grpc_server_add_insecure_channel_from_fd_type grpc_server_add_insecure_channel_from_fd_import; -grpc_use_signal_type grpc_use_signal_import; -grpc_auth_property_iterator_next_type grpc_auth_property_iterator_next_import; -grpc_auth_context_property_iterator_type grpc_auth_context_property_iterator_import; -grpc_auth_context_peer_identity_type grpc_auth_context_peer_identity_import; -grpc_auth_context_find_properties_by_name_type grpc_auth_context_find_properties_by_name_import; -grpc_auth_context_peer_identity_property_name_type grpc_auth_context_peer_identity_property_name_import; -grpc_auth_context_peer_is_authenticated_type grpc_auth_context_peer_is_authenticated_import; -grpc_call_auth_context_type grpc_call_auth_context_import; -grpc_auth_context_release_type grpc_auth_context_release_import; -grpc_auth_context_add_property_type grpc_auth_context_add_property_import; -grpc_auth_context_add_cstring_property_type grpc_auth_context_add_cstring_property_import; -grpc_auth_context_set_peer_identity_property_name_type grpc_auth_context_set_peer_identity_property_name_import; -grpc_channel_credentials_release_type grpc_channel_credentials_release_import; -grpc_google_default_credentials_create_type grpc_google_default_credentials_create_import; -grpc_set_ssl_roots_override_callback_type grpc_set_ssl_roots_override_callback_import; -grpc_ssl_credentials_create_type grpc_ssl_credentials_create_import; -grpc_call_credentials_release_type grpc_call_credentials_release_import; -grpc_composite_channel_credentials_create_type grpc_composite_channel_credentials_create_import; -grpc_composite_call_credentials_create_type grpc_composite_call_credentials_create_import; -grpc_google_compute_engine_credentials_create_type grpc_google_compute_engine_credentials_create_import; -grpc_max_auth_token_lifetime_type grpc_max_auth_token_lifetime_import; -grpc_service_account_jwt_access_credentials_create_type grpc_service_account_jwt_access_credentials_create_import; -grpc_google_refresh_token_credentials_create_type grpc_google_refresh_token_credentials_create_import; -grpc_access_token_credentials_create_type grpc_access_token_credentials_create_import; -grpc_google_iam_credentials_create_type grpc_google_iam_credentials_create_import; -grpc_metadata_credentials_create_from_plugin_type grpc_metadata_credentials_create_from_plugin_import; -grpc_secure_channel_create_type grpc_secure_channel_create_import; -grpc_server_credentials_release_type grpc_server_credentials_release_import; -grpc_ssl_server_credentials_create_type grpc_ssl_server_credentials_create_import; -grpc_ssl_server_credentials_create_ex_type grpc_ssl_server_credentials_create_ex_import; -grpc_server_add_secure_http2_port_type grpc_server_add_secure_http2_port_import; -grpc_call_set_credentials_type grpc_call_set_credentials_import; -grpc_server_credentials_set_auth_metadata_processor_type grpc_server_credentials_set_auth_metadata_processor_import; -gpr_malloc_type gpr_malloc_import; -gpr_free_type gpr_free_import; -gpr_realloc_type gpr_realloc_import; -gpr_malloc_aligned_type gpr_malloc_aligned_import; -gpr_free_aligned_type gpr_free_aligned_import; -gpr_set_allocation_functions_type gpr_set_allocation_functions_import; -gpr_get_allocation_functions_type gpr_get_allocation_functions_import; -grpc_raw_byte_buffer_create_type grpc_raw_byte_buffer_create_import; -grpc_raw_compressed_byte_buffer_create_type grpc_raw_compressed_byte_buffer_create_import; -grpc_byte_buffer_copy_type grpc_byte_buffer_copy_import; -grpc_byte_buffer_length_type grpc_byte_buffer_length_import; -grpc_byte_buffer_destroy_type grpc_byte_buffer_destroy_import; -grpc_byte_buffer_reader_init_type grpc_byte_buffer_reader_init_import; -grpc_byte_buffer_reader_destroy_type grpc_byte_buffer_reader_destroy_import; -grpc_byte_buffer_reader_next_type grpc_byte_buffer_reader_next_import; -grpc_byte_buffer_reader_readall_type grpc_byte_buffer_reader_readall_import; -grpc_raw_byte_buffer_from_reader_type grpc_raw_byte_buffer_from_reader_import; -gpr_log_type gpr_log_import; -gpr_log_message_type gpr_log_message_import; -gpr_set_log_verbosity_type gpr_set_log_verbosity_import; -gpr_log_verbosity_init_type gpr_log_verbosity_init_import; -gpr_set_log_function_type gpr_set_log_function_import; -gpr_slice_ref_type gpr_slice_ref_import; -gpr_slice_unref_type gpr_slice_unref_import; -gpr_slice_new_type gpr_slice_new_import; -gpr_slice_new_with_len_type gpr_slice_new_with_len_import; -gpr_slice_malloc_type gpr_slice_malloc_import; -gpr_slice_from_copied_string_type gpr_slice_from_copied_string_import; -gpr_slice_from_copied_buffer_type gpr_slice_from_copied_buffer_import; -gpr_slice_from_static_string_type gpr_slice_from_static_string_import; -gpr_slice_sub_type gpr_slice_sub_import; -gpr_slice_sub_no_ref_type gpr_slice_sub_no_ref_import; -gpr_slice_split_tail_type gpr_slice_split_tail_import; -gpr_slice_split_head_type gpr_slice_split_head_import; -gpr_empty_slice_type gpr_empty_slice_import; -gpr_slice_cmp_type gpr_slice_cmp_import; -gpr_slice_str_cmp_type gpr_slice_str_cmp_import; -gpr_slice_buffer_init_type gpr_slice_buffer_init_import; -gpr_slice_buffer_destroy_type gpr_slice_buffer_destroy_import; -gpr_slice_buffer_add_type gpr_slice_buffer_add_import; -gpr_slice_buffer_add_indexed_type gpr_slice_buffer_add_indexed_import; -gpr_slice_buffer_addn_type gpr_slice_buffer_addn_import; -gpr_slice_buffer_tiny_add_type gpr_slice_buffer_tiny_add_import; -gpr_slice_buffer_pop_type gpr_slice_buffer_pop_import; -gpr_slice_buffer_reset_and_unref_type gpr_slice_buffer_reset_and_unref_import; -gpr_slice_buffer_swap_type gpr_slice_buffer_swap_import; -gpr_slice_buffer_move_into_type gpr_slice_buffer_move_into_import; -gpr_slice_buffer_trim_end_type gpr_slice_buffer_trim_end_import; -gpr_slice_buffer_move_first_type gpr_slice_buffer_move_first_import; -gpr_slice_buffer_take_first_type gpr_slice_buffer_take_first_import; -gpr_mu_init_type gpr_mu_init_import; -gpr_mu_destroy_type gpr_mu_destroy_import; -gpr_mu_lock_type gpr_mu_lock_import; -gpr_mu_unlock_type gpr_mu_unlock_import; -gpr_mu_trylock_type gpr_mu_trylock_import; -gpr_cv_init_type gpr_cv_init_import; -gpr_cv_destroy_type gpr_cv_destroy_import; -gpr_cv_wait_type gpr_cv_wait_import; -gpr_cv_signal_type gpr_cv_signal_import; -gpr_cv_broadcast_type gpr_cv_broadcast_import; -gpr_once_init_type gpr_once_init_import; -gpr_event_init_type gpr_event_init_import; -gpr_event_set_type gpr_event_set_import; -gpr_event_get_type gpr_event_get_import; -gpr_event_wait_type gpr_event_wait_import; -gpr_ref_init_type gpr_ref_init_import; -gpr_ref_type gpr_ref_import; -gpr_ref_non_zero_type gpr_ref_non_zero_import; -gpr_refn_type gpr_refn_import; -gpr_unref_type gpr_unref_import; -gpr_stats_init_type gpr_stats_init_import; -gpr_stats_inc_type gpr_stats_inc_import; -gpr_stats_read_type gpr_stats_read_import; -gpr_time_0_type gpr_time_0_import; -gpr_inf_future_type gpr_inf_future_import; -gpr_inf_past_type gpr_inf_past_import; -gpr_time_init_type gpr_time_init_import; -gpr_now_type gpr_now_import; -gpr_convert_clock_type_type gpr_convert_clock_type_import; -gpr_time_cmp_type gpr_time_cmp_import; -gpr_time_max_type gpr_time_max_import; -gpr_time_min_type gpr_time_min_import; -gpr_time_add_type gpr_time_add_import; -gpr_time_sub_type gpr_time_sub_import; -gpr_time_from_micros_type gpr_time_from_micros_import; -gpr_time_from_nanos_type gpr_time_from_nanos_import; -gpr_time_from_millis_type gpr_time_from_millis_import; -gpr_time_from_seconds_type gpr_time_from_seconds_import; -gpr_time_from_minutes_type gpr_time_from_minutes_import; -gpr_time_from_hours_type gpr_time_from_hours_import; -gpr_time_to_millis_type gpr_time_to_millis_import; -gpr_time_similar_type gpr_time_similar_import; -gpr_sleep_until_type gpr_sleep_until_import; -gpr_timespec_to_micros_type gpr_timespec_to_micros_import; -gpr_avl_create_type gpr_avl_create_import; -gpr_avl_ref_type gpr_avl_ref_import; -gpr_avl_unref_type gpr_avl_unref_import; -gpr_avl_add_type gpr_avl_add_import; -gpr_avl_remove_type gpr_avl_remove_import; -gpr_avl_get_type gpr_avl_get_import; -gpr_avl_maybe_get_type gpr_avl_maybe_get_import; -gpr_avl_is_empty_type gpr_avl_is_empty_import; -gpr_cmdline_create_type gpr_cmdline_create_import; -gpr_cmdline_add_int_type gpr_cmdline_add_int_import; -gpr_cmdline_add_flag_type gpr_cmdline_add_flag_import; -gpr_cmdline_add_string_type gpr_cmdline_add_string_import; -gpr_cmdline_on_extra_arg_type gpr_cmdline_on_extra_arg_import; -gpr_cmdline_set_survive_failure_type gpr_cmdline_set_survive_failure_import; -gpr_cmdline_parse_type gpr_cmdline_parse_import; -gpr_cmdline_destroy_type gpr_cmdline_destroy_import; -gpr_cmdline_usage_string_type gpr_cmdline_usage_string_import; -gpr_cpu_num_cores_type gpr_cpu_num_cores_import; -gpr_cpu_current_cpu_type gpr_cpu_current_cpu_import; -gpr_histogram_create_type gpr_histogram_create_import; -gpr_histogram_destroy_type gpr_histogram_destroy_import; -gpr_histogram_add_type gpr_histogram_add_import; -gpr_histogram_merge_type gpr_histogram_merge_import; -gpr_histogram_percentile_type gpr_histogram_percentile_import; -gpr_histogram_mean_type gpr_histogram_mean_import; -gpr_histogram_stddev_type gpr_histogram_stddev_import; -gpr_histogram_variance_type gpr_histogram_variance_import; -gpr_histogram_maximum_type gpr_histogram_maximum_import; -gpr_histogram_minimum_type gpr_histogram_minimum_import; -gpr_histogram_count_type gpr_histogram_count_import; -gpr_histogram_sum_type gpr_histogram_sum_import; -gpr_histogram_sum_of_squares_type gpr_histogram_sum_of_squares_import; -gpr_histogram_get_contents_type gpr_histogram_get_contents_import; -gpr_histogram_merge_contents_type gpr_histogram_merge_contents_import; -gpr_join_host_port_type gpr_join_host_port_import; -gpr_split_host_port_type gpr_split_host_port_import; -gpr_format_message_type gpr_format_message_import; -gpr_strdup_type gpr_strdup_import; -gpr_asprintf_type gpr_asprintf_import; -gpr_subprocess_binary_extension_type gpr_subprocess_binary_extension_import; -gpr_subprocess_create_type gpr_subprocess_create_import; -gpr_subprocess_destroy_type gpr_subprocess_destroy_import; -gpr_subprocess_join_type gpr_subprocess_join_import; -gpr_subprocess_interrupt_type gpr_subprocess_interrupt_import; -gpr_thd_new_type gpr_thd_new_import; -gpr_thd_options_default_type gpr_thd_options_default_import; -gpr_thd_options_set_detached_type gpr_thd_options_set_detached_import; -gpr_thd_options_set_joinable_type gpr_thd_options_set_joinable_import; -gpr_thd_options_is_detached_type gpr_thd_options_is_detached_import; -gpr_thd_options_is_joinable_type gpr_thd_options_is_joinable_import; -gpr_thd_currentid_type gpr_thd_currentid_import; -gpr_thd_join_type gpr_thd_join_import; - -#ifdef __cplusplus -extern "C" { -#endif /* __cpluslus */ - -void pygrpc_load_imports(HMODULE library) { - census_initialize_import = (census_initialize_type) GetProcAddress(library, "census_initialize"); - census_shutdown_import = (census_shutdown_type) GetProcAddress(library, "census_shutdown"); - census_supported_import = (census_supported_type) GetProcAddress(library, "census_supported"); - census_enabled_import = (census_enabled_type) GetProcAddress(library, "census_enabled"); - census_context_create_import = (census_context_create_type) GetProcAddress(library, "census_context_create"); - census_context_destroy_import = (census_context_destroy_type) GetProcAddress(library, "census_context_destroy"); - census_context_get_status_import = (census_context_get_status_type) GetProcAddress(library, "census_context_get_status"); - census_context_initialize_iterator_import = (census_context_initialize_iterator_type) GetProcAddress(library, "census_context_initialize_iterator"); - census_context_next_tag_import = (census_context_next_tag_type) GetProcAddress(library, "census_context_next_tag"); - census_context_get_tag_import = (census_context_get_tag_type) GetProcAddress(library, "census_context_get_tag"); - census_context_encode_import = (census_context_encode_type) GetProcAddress(library, "census_context_encode"); - census_context_decode_import = (census_context_decode_type) GetProcAddress(library, "census_context_decode"); - census_trace_mask_import = (census_trace_mask_type) GetProcAddress(library, "census_trace_mask"); - census_set_trace_mask_import = (census_set_trace_mask_type) GetProcAddress(library, "census_set_trace_mask"); - census_start_rpc_op_timestamp_import = (census_start_rpc_op_timestamp_type) GetProcAddress(library, "census_start_rpc_op_timestamp"); - census_start_client_rpc_op_import = (census_start_client_rpc_op_type) GetProcAddress(library, "census_start_client_rpc_op"); - census_set_rpc_client_peer_import = (census_set_rpc_client_peer_type) GetProcAddress(library, "census_set_rpc_client_peer"); - census_start_server_rpc_op_import = (census_start_server_rpc_op_type) GetProcAddress(library, "census_start_server_rpc_op"); - census_start_op_import = (census_start_op_type) GetProcAddress(library, "census_start_op"); - census_end_op_import = (census_end_op_type) GetProcAddress(library, "census_end_op"); - census_trace_print_import = (census_trace_print_type) GetProcAddress(library, "census_trace_print"); - census_trace_scan_start_import = (census_trace_scan_start_type) GetProcAddress(library, "census_trace_scan_start"); - census_get_trace_record_import = (census_get_trace_record_type) GetProcAddress(library, "census_get_trace_record"); - census_trace_scan_end_import = (census_trace_scan_end_type) GetProcAddress(library, "census_trace_scan_end"); - census_record_values_import = (census_record_values_type) GetProcAddress(library, "census_record_values"); - census_view_create_import = (census_view_create_type) GetProcAddress(library, "census_view_create"); - census_view_delete_import = (census_view_delete_type) GetProcAddress(library, "census_view_delete"); - census_view_metric_import = (census_view_metric_type) GetProcAddress(library, "census_view_metric"); - census_view_naggregations_import = (census_view_naggregations_type) GetProcAddress(library, "census_view_naggregations"); - census_view_tags_import = (census_view_tags_type) GetProcAddress(library, "census_view_tags"); - census_view_aggregrations_import = (census_view_aggregrations_type) GetProcAddress(library, "census_view_aggregrations"); - census_view_get_data_import = (census_view_get_data_type) GetProcAddress(library, "census_view_get_data"); - census_view_reset_import = (census_view_reset_type) GetProcAddress(library, "census_view_reset"); - grpc_compression_algorithm_parse_import = (grpc_compression_algorithm_parse_type) GetProcAddress(library, "grpc_compression_algorithm_parse"); - grpc_compression_algorithm_name_import = (grpc_compression_algorithm_name_type) GetProcAddress(library, "grpc_compression_algorithm_name"); - grpc_compression_algorithm_for_level_import = (grpc_compression_algorithm_for_level_type) GetProcAddress(library, "grpc_compression_algorithm_for_level"); - grpc_compression_options_init_import = (grpc_compression_options_init_type) GetProcAddress(library, "grpc_compression_options_init"); - grpc_compression_options_enable_algorithm_import = (grpc_compression_options_enable_algorithm_type) GetProcAddress(library, "grpc_compression_options_enable_algorithm"); - grpc_compression_options_disable_algorithm_import = (grpc_compression_options_disable_algorithm_type) GetProcAddress(library, "grpc_compression_options_disable_algorithm"); - grpc_compression_options_is_algorithm_enabled_import = (grpc_compression_options_is_algorithm_enabled_type) GetProcAddress(library, "grpc_compression_options_is_algorithm_enabled"); - grpc_metadata_array_init_import = (grpc_metadata_array_init_type) GetProcAddress(library, "grpc_metadata_array_init"); - grpc_metadata_array_destroy_import = (grpc_metadata_array_destroy_type) GetProcAddress(library, "grpc_metadata_array_destroy"); - grpc_call_details_init_import = (grpc_call_details_init_type) GetProcAddress(library, "grpc_call_details_init"); - grpc_call_details_destroy_import = (grpc_call_details_destroy_type) GetProcAddress(library, "grpc_call_details_destroy"); - grpc_register_plugin_import = (grpc_register_plugin_type) GetProcAddress(library, "grpc_register_plugin"); - grpc_init_import = (grpc_init_type) GetProcAddress(library, "grpc_init"); - grpc_shutdown_import = (grpc_shutdown_type) GetProcAddress(library, "grpc_shutdown"); - grpc_version_string_import = (grpc_version_string_type) GetProcAddress(library, "grpc_version_string"); - grpc_completion_queue_create_import = (grpc_completion_queue_create_type) GetProcAddress(library, "grpc_completion_queue_create"); - grpc_completion_queue_next_import = (grpc_completion_queue_next_type) GetProcAddress(library, "grpc_completion_queue_next"); - grpc_completion_queue_pluck_import = (grpc_completion_queue_pluck_type) GetProcAddress(library, "grpc_completion_queue_pluck"); - grpc_completion_queue_shutdown_import = (grpc_completion_queue_shutdown_type) GetProcAddress(library, "grpc_completion_queue_shutdown"); - grpc_completion_queue_destroy_import = (grpc_completion_queue_destroy_type) GetProcAddress(library, "grpc_completion_queue_destroy"); - grpc_alarm_create_import = (grpc_alarm_create_type) GetProcAddress(library, "grpc_alarm_create"); - grpc_alarm_cancel_import = (grpc_alarm_cancel_type) GetProcAddress(library, "grpc_alarm_cancel"); - grpc_alarm_destroy_import = (grpc_alarm_destroy_type) GetProcAddress(library, "grpc_alarm_destroy"); - grpc_channel_check_connectivity_state_import = (grpc_channel_check_connectivity_state_type) GetProcAddress(library, "grpc_channel_check_connectivity_state"); - grpc_channel_watch_connectivity_state_import = (grpc_channel_watch_connectivity_state_type) GetProcAddress(library, "grpc_channel_watch_connectivity_state"); - grpc_channel_create_call_import = (grpc_channel_create_call_type) GetProcAddress(library, "grpc_channel_create_call"); - grpc_channel_ping_import = (grpc_channel_ping_type) GetProcAddress(library, "grpc_channel_ping"); - grpc_channel_register_call_import = (grpc_channel_register_call_type) GetProcAddress(library, "grpc_channel_register_call"); - grpc_channel_create_registered_call_import = (grpc_channel_create_registered_call_type) GetProcAddress(library, "grpc_channel_create_registered_call"); - grpc_call_start_batch_import = (grpc_call_start_batch_type) GetProcAddress(library, "grpc_call_start_batch"); - grpc_call_get_peer_import = (grpc_call_get_peer_type) GetProcAddress(library, "grpc_call_get_peer"); - grpc_census_call_set_context_import = (grpc_census_call_set_context_type) GetProcAddress(library, "grpc_census_call_set_context"); - grpc_census_call_get_context_import = (grpc_census_call_get_context_type) GetProcAddress(library, "grpc_census_call_get_context"); - grpc_channel_get_target_import = (grpc_channel_get_target_type) GetProcAddress(library, "grpc_channel_get_target"); - grpc_insecure_channel_create_import = (grpc_insecure_channel_create_type) GetProcAddress(library, "grpc_insecure_channel_create"); - grpc_lame_client_channel_create_import = (grpc_lame_client_channel_create_type) GetProcAddress(library, "grpc_lame_client_channel_create"); - grpc_channel_destroy_import = (grpc_channel_destroy_type) GetProcAddress(library, "grpc_channel_destroy"); - grpc_call_cancel_import = (grpc_call_cancel_type) GetProcAddress(library, "grpc_call_cancel"); - grpc_call_cancel_with_status_import = (grpc_call_cancel_with_status_type) GetProcAddress(library, "grpc_call_cancel_with_status"); - grpc_call_destroy_import = (grpc_call_destroy_type) GetProcAddress(library, "grpc_call_destroy"); - grpc_server_request_call_import = (grpc_server_request_call_type) GetProcAddress(library, "grpc_server_request_call"); - grpc_server_register_method_import = (grpc_server_register_method_type) GetProcAddress(library, "grpc_server_register_method"); - grpc_server_request_registered_call_import = (grpc_server_request_registered_call_type) GetProcAddress(library, "grpc_server_request_registered_call"); - grpc_server_create_import = (grpc_server_create_type) GetProcAddress(library, "grpc_server_create"); - grpc_server_register_completion_queue_import = (grpc_server_register_completion_queue_type) GetProcAddress(library, "grpc_server_register_completion_queue"); - grpc_server_register_non_listening_completion_queue_import = (grpc_server_register_non_listening_completion_queue_type) GetProcAddress(library, "grpc_server_register_non_listening_completion_queue"); - grpc_server_add_insecure_http2_port_import = (grpc_server_add_insecure_http2_port_type) GetProcAddress(library, "grpc_server_add_insecure_http2_port"); - grpc_server_start_import = (grpc_server_start_type) GetProcAddress(library, "grpc_server_start"); - grpc_server_shutdown_and_notify_import = (grpc_server_shutdown_and_notify_type) GetProcAddress(library, "grpc_server_shutdown_and_notify"); - grpc_server_cancel_all_calls_import = (grpc_server_cancel_all_calls_type) GetProcAddress(library, "grpc_server_cancel_all_calls"); - grpc_server_destroy_import = (grpc_server_destroy_type) GetProcAddress(library, "grpc_server_destroy"); - grpc_tracer_set_enabled_import = (grpc_tracer_set_enabled_type) GetProcAddress(library, "grpc_tracer_set_enabled"); - grpc_header_key_is_legal_import = (grpc_header_key_is_legal_type) GetProcAddress(library, "grpc_header_key_is_legal"); - grpc_header_nonbin_value_is_legal_import = (grpc_header_nonbin_value_is_legal_type) GetProcAddress(library, "grpc_header_nonbin_value_is_legal"); - grpc_is_binary_header_import = (grpc_is_binary_header_type) GetProcAddress(library, "grpc_is_binary_header"); - grpc_call_error_to_string_import = (grpc_call_error_to_string_type) GetProcAddress(library, "grpc_call_error_to_string"); - grpc_insecure_channel_create_from_fd_import = (grpc_insecure_channel_create_from_fd_type) GetProcAddress(library, "grpc_insecure_channel_create_from_fd"); - grpc_server_add_insecure_channel_from_fd_import = (grpc_server_add_insecure_channel_from_fd_type) GetProcAddress(library, "grpc_server_add_insecure_channel_from_fd"); - grpc_use_signal_import = (grpc_use_signal_type) GetProcAddress(library, "grpc_use_signal"); - grpc_auth_property_iterator_next_import = (grpc_auth_property_iterator_next_type) GetProcAddress(library, "grpc_auth_property_iterator_next"); - grpc_auth_context_property_iterator_import = (grpc_auth_context_property_iterator_type) GetProcAddress(library, "grpc_auth_context_property_iterator"); - grpc_auth_context_peer_identity_import = (grpc_auth_context_peer_identity_type) GetProcAddress(library, "grpc_auth_context_peer_identity"); - grpc_auth_context_find_properties_by_name_import = (grpc_auth_context_find_properties_by_name_type) GetProcAddress(library, "grpc_auth_context_find_properties_by_name"); - grpc_auth_context_peer_identity_property_name_import = (grpc_auth_context_peer_identity_property_name_type) GetProcAddress(library, "grpc_auth_context_peer_identity_property_name"); - grpc_auth_context_peer_is_authenticated_import = (grpc_auth_context_peer_is_authenticated_type) GetProcAddress(library, "grpc_auth_context_peer_is_authenticated"); - grpc_call_auth_context_import = (grpc_call_auth_context_type) GetProcAddress(library, "grpc_call_auth_context"); - grpc_auth_context_release_import = (grpc_auth_context_release_type) GetProcAddress(library, "grpc_auth_context_release"); - grpc_auth_context_add_property_import = (grpc_auth_context_add_property_type) GetProcAddress(library, "grpc_auth_context_add_property"); - grpc_auth_context_add_cstring_property_import = (grpc_auth_context_add_cstring_property_type) GetProcAddress(library, "grpc_auth_context_add_cstring_property"); - grpc_auth_context_set_peer_identity_property_name_import = (grpc_auth_context_set_peer_identity_property_name_type) GetProcAddress(library, "grpc_auth_context_set_peer_identity_property_name"); - grpc_channel_credentials_release_import = (grpc_channel_credentials_release_type) GetProcAddress(library, "grpc_channel_credentials_release"); - grpc_google_default_credentials_create_import = (grpc_google_default_credentials_create_type) GetProcAddress(library, "grpc_google_default_credentials_create"); - grpc_set_ssl_roots_override_callback_import = (grpc_set_ssl_roots_override_callback_type) GetProcAddress(library, "grpc_set_ssl_roots_override_callback"); - grpc_ssl_credentials_create_import = (grpc_ssl_credentials_create_type) GetProcAddress(library, "grpc_ssl_credentials_create"); - grpc_call_credentials_release_import = (grpc_call_credentials_release_type) GetProcAddress(library, "grpc_call_credentials_release"); - grpc_composite_channel_credentials_create_import = (grpc_composite_channel_credentials_create_type) GetProcAddress(library, "grpc_composite_channel_credentials_create"); - grpc_composite_call_credentials_create_import = (grpc_composite_call_credentials_create_type) GetProcAddress(library, "grpc_composite_call_credentials_create"); - grpc_google_compute_engine_credentials_create_import = (grpc_google_compute_engine_credentials_create_type) GetProcAddress(library, "grpc_google_compute_engine_credentials_create"); - grpc_max_auth_token_lifetime_import = (grpc_max_auth_token_lifetime_type) GetProcAddress(library, "grpc_max_auth_token_lifetime"); - grpc_service_account_jwt_access_credentials_create_import = (grpc_service_account_jwt_access_credentials_create_type) GetProcAddress(library, "grpc_service_account_jwt_access_credentials_create"); - grpc_google_refresh_token_credentials_create_import = (grpc_google_refresh_token_credentials_create_type) GetProcAddress(library, "grpc_google_refresh_token_credentials_create"); - grpc_access_token_credentials_create_import = (grpc_access_token_credentials_create_type) GetProcAddress(library, "grpc_access_token_credentials_create"); - grpc_google_iam_credentials_create_import = (grpc_google_iam_credentials_create_type) GetProcAddress(library, "grpc_google_iam_credentials_create"); - grpc_metadata_credentials_create_from_plugin_import = (grpc_metadata_credentials_create_from_plugin_type) GetProcAddress(library, "grpc_metadata_credentials_create_from_plugin"); - grpc_secure_channel_create_import = (grpc_secure_channel_create_type) GetProcAddress(library, "grpc_secure_channel_create"); - grpc_server_credentials_release_import = (grpc_server_credentials_release_type) GetProcAddress(library, "grpc_server_credentials_release"); - grpc_ssl_server_credentials_create_import = (grpc_ssl_server_credentials_create_type) GetProcAddress(library, "grpc_ssl_server_credentials_create"); - grpc_ssl_server_credentials_create_ex_import = (grpc_ssl_server_credentials_create_ex_type) GetProcAddress(library, "grpc_ssl_server_credentials_create_ex"); - grpc_server_add_secure_http2_port_import = (grpc_server_add_secure_http2_port_type) GetProcAddress(library, "grpc_server_add_secure_http2_port"); - grpc_call_set_credentials_import = (grpc_call_set_credentials_type) GetProcAddress(library, "grpc_call_set_credentials"); - grpc_server_credentials_set_auth_metadata_processor_import = (grpc_server_credentials_set_auth_metadata_processor_type) GetProcAddress(library, "grpc_server_credentials_set_auth_metadata_processor"); - gpr_malloc_import = (gpr_malloc_type) GetProcAddress(library, "gpr_malloc"); - gpr_free_import = (gpr_free_type) GetProcAddress(library, "gpr_free"); - gpr_realloc_import = (gpr_realloc_type) GetProcAddress(library, "gpr_realloc"); - gpr_malloc_aligned_import = (gpr_malloc_aligned_type) GetProcAddress(library, "gpr_malloc_aligned"); - gpr_free_aligned_import = (gpr_free_aligned_type) GetProcAddress(library, "gpr_free_aligned"); - gpr_set_allocation_functions_import = (gpr_set_allocation_functions_type) GetProcAddress(library, "gpr_set_allocation_functions"); - gpr_get_allocation_functions_import = (gpr_get_allocation_functions_type) GetProcAddress(library, "gpr_get_allocation_functions"); - grpc_raw_byte_buffer_create_import = (grpc_raw_byte_buffer_create_type) GetProcAddress(library, "grpc_raw_byte_buffer_create"); - grpc_raw_compressed_byte_buffer_create_import = (grpc_raw_compressed_byte_buffer_create_type) GetProcAddress(library, "grpc_raw_compressed_byte_buffer_create"); - grpc_byte_buffer_copy_import = (grpc_byte_buffer_copy_type) GetProcAddress(library, "grpc_byte_buffer_copy"); - grpc_byte_buffer_length_import = (grpc_byte_buffer_length_type) GetProcAddress(library, "grpc_byte_buffer_length"); - grpc_byte_buffer_destroy_import = (grpc_byte_buffer_destroy_type) GetProcAddress(library, "grpc_byte_buffer_destroy"); - grpc_byte_buffer_reader_init_import = (grpc_byte_buffer_reader_init_type) GetProcAddress(library, "grpc_byte_buffer_reader_init"); - grpc_byte_buffer_reader_destroy_import = (grpc_byte_buffer_reader_destroy_type) GetProcAddress(library, "grpc_byte_buffer_reader_destroy"); - grpc_byte_buffer_reader_next_import = (grpc_byte_buffer_reader_next_type) GetProcAddress(library, "grpc_byte_buffer_reader_next"); - grpc_byte_buffer_reader_readall_import = (grpc_byte_buffer_reader_readall_type) GetProcAddress(library, "grpc_byte_buffer_reader_readall"); - grpc_raw_byte_buffer_from_reader_import = (grpc_raw_byte_buffer_from_reader_type) GetProcAddress(library, "grpc_raw_byte_buffer_from_reader"); - gpr_log_import = (gpr_log_type) GetProcAddress(library, "gpr_log"); - gpr_log_message_import = (gpr_log_message_type) GetProcAddress(library, "gpr_log_message"); - gpr_set_log_verbosity_import = (gpr_set_log_verbosity_type) GetProcAddress(library, "gpr_set_log_verbosity"); - gpr_log_verbosity_init_import = (gpr_log_verbosity_init_type) GetProcAddress(library, "gpr_log_verbosity_init"); - gpr_set_log_function_import = (gpr_set_log_function_type) GetProcAddress(library, "gpr_set_log_function"); - gpr_slice_ref_import = (gpr_slice_ref_type) GetProcAddress(library, "gpr_slice_ref"); - gpr_slice_unref_import = (gpr_slice_unref_type) GetProcAddress(library, "gpr_slice_unref"); - gpr_slice_new_import = (gpr_slice_new_type) GetProcAddress(library, "gpr_slice_new"); - gpr_slice_new_with_len_import = (gpr_slice_new_with_len_type) GetProcAddress(library, "gpr_slice_new_with_len"); - gpr_slice_malloc_import = (gpr_slice_malloc_type) GetProcAddress(library, "gpr_slice_malloc"); - gpr_slice_from_copied_string_import = (gpr_slice_from_copied_string_type) GetProcAddress(library, "gpr_slice_from_copied_string"); - gpr_slice_from_copied_buffer_import = (gpr_slice_from_copied_buffer_type) GetProcAddress(library, "gpr_slice_from_copied_buffer"); - gpr_slice_from_static_string_import = (gpr_slice_from_static_string_type) GetProcAddress(library, "gpr_slice_from_static_string"); - gpr_slice_sub_import = (gpr_slice_sub_type) GetProcAddress(library, "gpr_slice_sub"); - gpr_slice_sub_no_ref_import = (gpr_slice_sub_no_ref_type) GetProcAddress(library, "gpr_slice_sub_no_ref"); - gpr_slice_split_tail_import = (gpr_slice_split_tail_type) GetProcAddress(library, "gpr_slice_split_tail"); - gpr_slice_split_head_import = (gpr_slice_split_head_type) GetProcAddress(library, "gpr_slice_split_head"); - gpr_empty_slice_import = (gpr_empty_slice_type) GetProcAddress(library, "gpr_empty_slice"); - gpr_slice_cmp_import = (gpr_slice_cmp_type) GetProcAddress(library, "gpr_slice_cmp"); - gpr_slice_str_cmp_import = (gpr_slice_str_cmp_type) GetProcAddress(library, "gpr_slice_str_cmp"); - gpr_slice_buffer_init_import = (gpr_slice_buffer_init_type) GetProcAddress(library, "gpr_slice_buffer_init"); - gpr_slice_buffer_destroy_import = (gpr_slice_buffer_destroy_type) GetProcAddress(library, "gpr_slice_buffer_destroy"); - gpr_slice_buffer_add_import = (gpr_slice_buffer_add_type) GetProcAddress(library, "gpr_slice_buffer_add"); - gpr_slice_buffer_add_indexed_import = (gpr_slice_buffer_add_indexed_type) GetProcAddress(library, "gpr_slice_buffer_add_indexed"); - gpr_slice_buffer_addn_import = (gpr_slice_buffer_addn_type) GetProcAddress(library, "gpr_slice_buffer_addn"); - gpr_slice_buffer_tiny_add_import = (gpr_slice_buffer_tiny_add_type) GetProcAddress(library, "gpr_slice_buffer_tiny_add"); - gpr_slice_buffer_pop_import = (gpr_slice_buffer_pop_type) GetProcAddress(library, "gpr_slice_buffer_pop"); - gpr_slice_buffer_reset_and_unref_import = (gpr_slice_buffer_reset_and_unref_type) GetProcAddress(library, "gpr_slice_buffer_reset_and_unref"); - gpr_slice_buffer_swap_import = (gpr_slice_buffer_swap_type) GetProcAddress(library, "gpr_slice_buffer_swap"); - gpr_slice_buffer_move_into_import = (gpr_slice_buffer_move_into_type) GetProcAddress(library, "gpr_slice_buffer_move_into"); - gpr_slice_buffer_trim_end_import = (gpr_slice_buffer_trim_end_type) GetProcAddress(library, "gpr_slice_buffer_trim_end"); - gpr_slice_buffer_move_first_import = (gpr_slice_buffer_move_first_type) GetProcAddress(library, "gpr_slice_buffer_move_first"); - gpr_slice_buffer_take_first_import = (gpr_slice_buffer_take_first_type) GetProcAddress(library, "gpr_slice_buffer_take_first"); - gpr_mu_init_import = (gpr_mu_init_type) GetProcAddress(library, "gpr_mu_init"); - gpr_mu_destroy_import = (gpr_mu_destroy_type) GetProcAddress(library, "gpr_mu_destroy"); - gpr_mu_lock_import = (gpr_mu_lock_type) GetProcAddress(library, "gpr_mu_lock"); - gpr_mu_unlock_import = (gpr_mu_unlock_type) GetProcAddress(library, "gpr_mu_unlock"); - gpr_mu_trylock_import = (gpr_mu_trylock_type) GetProcAddress(library, "gpr_mu_trylock"); - gpr_cv_init_import = (gpr_cv_init_type) GetProcAddress(library, "gpr_cv_init"); - gpr_cv_destroy_import = (gpr_cv_destroy_type) GetProcAddress(library, "gpr_cv_destroy"); - gpr_cv_wait_import = (gpr_cv_wait_type) GetProcAddress(library, "gpr_cv_wait"); - gpr_cv_signal_import = (gpr_cv_signal_type) GetProcAddress(library, "gpr_cv_signal"); - gpr_cv_broadcast_import = (gpr_cv_broadcast_type) GetProcAddress(library, "gpr_cv_broadcast"); - gpr_once_init_import = (gpr_once_init_type) GetProcAddress(library, "gpr_once_init"); - gpr_event_init_import = (gpr_event_init_type) GetProcAddress(library, "gpr_event_init"); - gpr_event_set_import = (gpr_event_set_type) GetProcAddress(library, "gpr_event_set"); - gpr_event_get_import = (gpr_event_get_type) GetProcAddress(library, "gpr_event_get"); - gpr_event_wait_import = (gpr_event_wait_type) GetProcAddress(library, "gpr_event_wait"); - gpr_ref_init_import = (gpr_ref_init_type) GetProcAddress(library, "gpr_ref_init"); - gpr_ref_import = (gpr_ref_type) GetProcAddress(library, "gpr_ref"); - gpr_ref_non_zero_import = (gpr_ref_non_zero_type) GetProcAddress(library, "gpr_ref_non_zero"); - gpr_refn_import = (gpr_refn_type) GetProcAddress(library, "gpr_refn"); - gpr_unref_import = (gpr_unref_type) GetProcAddress(library, "gpr_unref"); - gpr_stats_init_import = (gpr_stats_init_type) GetProcAddress(library, "gpr_stats_init"); - gpr_stats_inc_import = (gpr_stats_inc_type) GetProcAddress(library, "gpr_stats_inc"); - gpr_stats_read_import = (gpr_stats_read_type) GetProcAddress(library, "gpr_stats_read"); - gpr_time_0_import = (gpr_time_0_type) GetProcAddress(library, "gpr_time_0"); - gpr_inf_future_import = (gpr_inf_future_type) GetProcAddress(library, "gpr_inf_future"); - gpr_inf_past_import = (gpr_inf_past_type) GetProcAddress(library, "gpr_inf_past"); - gpr_time_init_import = (gpr_time_init_type) GetProcAddress(library, "gpr_time_init"); - gpr_now_import = (gpr_now_type) GetProcAddress(library, "gpr_now"); - gpr_convert_clock_type_import = (gpr_convert_clock_type_type) GetProcAddress(library, "gpr_convert_clock_type"); - gpr_time_cmp_import = (gpr_time_cmp_type) GetProcAddress(library, "gpr_time_cmp"); - gpr_time_max_import = (gpr_time_max_type) GetProcAddress(library, "gpr_time_max"); - gpr_time_min_import = (gpr_time_min_type) GetProcAddress(library, "gpr_time_min"); - gpr_time_add_import = (gpr_time_add_type) GetProcAddress(library, "gpr_time_add"); - gpr_time_sub_import = (gpr_time_sub_type) GetProcAddress(library, "gpr_time_sub"); - gpr_time_from_micros_import = (gpr_time_from_micros_type) GetProcAddress(library, "gpr_time_from_micros"); - gpr_time_from_nanos_import = (gpr_time_from_nanos_type) GetProcAddress(library, "gpr_time_from_nanos"); - gpr_time_from_millis_import = (gpr_time_from_millis_type) GetProcAddress(library, "gpr_time_from_millis"); - gpr_time_from_seconds_import = (gpr_time_from_seconds_type) GetProcAddress(library, "gpr_time_from_seconds"); - gpr_time_from_minutes_import = (gpr_time_from_minutes_type) GetProcAddress(library, "gpr_time_from_minutes"); - gpr_time_from_hours_import = (gpr_time_from_hours_type) GetProcAddress(library, "gpr_time_from_hours"); - gpr_time_to_millis_import = (gpr_time_to_millis_type) GetProcAddress(library, "gpr_time_to_millis"); - gpr_time_similar_import = (gpr_time_similar_type) GetProcAddress(library, "gpr_time_similar"); - gpr_sleep_until_import = (gpr_sleep_until_type) GetProcAddress(library, "gpr_sleep_until"); - gpr_timespec_to_micros_import = (gpr_timespec_to_micros_type) GetProcAddress(library, "gpr_timespec_to_micros"); - gpr_avl_create_import = (gpr_avl_create_type) GetProcAddress(library, "gpr_avl_create"); - gpr_avl_ref_import = (gpr_avl_ref_type) GetProcAddress(library, "gpr_avl_ref"); - gpr_avl_unref_import = (gpr_avl_unref_type) GetProcAddress(library, "gpr_avl_unref"); - gpr_avl_add_import = (gpr_avl_add_type) GetProcAddress(library, "gpr_avl_add"); - gpr_avl_remove_import = (gpr_avl_remove_type) GetProcAddress(library, "gpr_avl_remove"); - gpr_avl_get_import = (gpr_avl_get_type) GetProcAddress(library, "gpr_avl_get"); - gpr_avl_maybe_get_import = (gpr_avl_maybe_get_type) GetProcAddress(library, "gpr_avl_maybe_get"); - gpr_avl_is_empty_import = (gpr_avl_is_empty_type) GetProcAddress(library, "gpr_avl_is_empty"); - gpr_cmdline_create_import = (gpr_cmdline_create_type) GetProcAddress(library, "gpr_cmdline_create"); - gpr_cmdline_add_int_import = (gpr_cmdline_add_int_type) GetProcAddress(library, "gpr_cmdline_add_int"); - gpr_cmdline_add_flag_import = (gpr_cmdline_add_flag_type) GetProcAddress(library, "gpr_cmdline_add_flag"); - gpr_cmdline_add_string_import = (gpr_cmdline_add_string_type) GetProcAddress(library, "gpr_cmdline_add_string"); - gpr_cmdline_on_extra_arg_import = (gpr_cmdline_on_extra_arg_type) GetProcAddress(library, "gpr_cmdline_on_extra_arg"); - gpr_cmdline_set_survive_failure_import = (gpr_cmdline_set_survive_failure_type) GetProcAddress(library, "gpr_cmdline_set_survive_failure"); - gpr_cmdline_parse_import = (gpr_cmdline_parse_type) GetProcAddress(library, "gpr_cmdline_parse"); - gpr_cmdline_destroy_import = (gpr_cmdline_destroy_type) GetProcAddress(library, "gpr_cmdline_destroy"); - gpr_cmdline_usage_string_import = (gpr_cmdline_usage_string_type) GetProcAddress(library, "gpr_cmdline_usage_string"); - gpr_cpu_num_cores_import = (gpr_cpu_num_cores_type) GetProcAddress(library, "gpr_cpu_num_cores"); - gpr_cpu_current_cpu_import = (gpr_cpu_current_cpu_type) GetProcAddress(library, "gpr_cpu_current_cpu"); - gpr_histogram_create_import = (gpr_histogram_create_type) GetProcAddress(library, "gpr_histogram_create"); - gpr_histogram_destroy_import = (gpr_histogram_destroy_type) GetProcAddress(library, "gpr_histogram_destroy"); - gpr_histogram_add_import = (gpr_histogram_add_type) GetProcAddress(library, "gpr_histogram_add"); - gpr_histogram_merge_import = (gpr_histogram_merge_type) GetProcAddress(library, "gpr_histogram_merge"); - gpr_histogram_percentile_import = (gpr_histogram_percentile_type) GetProcAddress(library, "gpr_histogram_percentile"); - gpr_histogram_mean_import = (gpr_histogram_mean_type) GetProcAddress(library, "gpr_histogram_mean"); - gpr_histogram_stddev_import = (gpr_histogram_stddev_type) GetProcAddress(library, "gpr_histogram_stddev"); - gpr_histogram_variance_import = (gpr_histogram_variance_type) GetProcAddress(library, "gpr_histogram_variance"); - gpr_histogram_maximum_import = (gpr_histogram_maximum_type) GetProcAddress(library, "gpr_histogram_maximum"); - gpr_histogram_minimum_import = (gpr_histogram_minimum_type) GetProcAddress(library, "gpr_histogram_minimum"); - gpr_histogram_count_import = (gpr_histogram_count_type) GetProcAddress(library, "gpr_histogram_count"); - gpr_histogram_sum_import = (gpr_histogram_sum_type) GetProcAddress(library, "gpr_histogram_sum"); - gpr_histogram_sum_of_squares_import = (gpr_histogram_sum_of_squares_type) GetProcAddress(library, "gpr_histogram_sum_of_squares"); - gpr_histogram_get_contents_import = (gpr_histogram_get_contents_type) GetProcAddress(library, "gpr_histogram_get_contents"); - gpr_histogram_merge_contents_import = (gpr_histogram_merge_contents_type) GetProcAddress(library, "gpr_histogram_merge_contents"); - gpr_join_host_port_import = (gpr_join_host_port_type) GetProcAddress(library, "gpr_join_host_port"); - gpr_split_host_port_import = (gpr_split_host_port_type) GetProcAddress(library, "gpr_split_host_port"); - gpr_format_message_import = (gpr_format_message_type) GetProcAddress(library, "gpr_format_message"); - gpr_strdup_import = (gpr_strdup_type) GetProcAddress(library, "gpr_strdup"); - gpr_asprintf_import = (gpr_asprintf_type) GetProcAddress(library, "gpr_asprintf"); - gpr_subprocess_binary_extension_import = (gpr_subprocess_binary_extension_type) GetProcAddress(library, "gpr_subprocess_binary_extension"); - gpr_subprocess_create_import = (gpr_subprocess_create_type) GetProcAddress(library, "gpr_subprocess_create"); - gpr_subprocess_destroy_import = (gpr_subprocess_destroy_type) GetProcAddress(library, "gpr_subprocess_destroy"); - gpr_subprocess_join_import = (gpr_subprocess_join_type) GetProcAddress(library, "gpr_subprocess_join"); - gpr_subprocess_interrupt_import = (gpr_subprocess_interrupt_type) GetProcAddress(library, "gpr_subprocess_interrupt"); - gpr_thd_new_import = (gpr_thd_new_type) GetProcAddress(library, "gpr_thd_new"); - gpr_thd_options_default_import = (gpr_thd_options_default_type) GetProcAddress(library, "gpr_thd_options_default"); - gpr_thd_options_set_detached_import = (gpr_thd_options_set_detached_type) GetProcAddress(library, "gpr_thd_options_set_detached"); - gpr_thd_options_set_joinable_import = (gpr_thd_options_set_joinable_type) GetProcAddress(library, "gpr_thd_options_set_joinable"); - gpr_thd_options_is_detached_import = (gpr_thd_options_is_detached_type) GetProcAddress(library, "gpr_thd_options_is_detached"); - gpr_thd_options_is_joinable_import = (gpr_thd_options_is_joinable_type) GetProcAddress(library, "gpr_thd_options_is_joinable"); - gpr_thd_currentid_import = (gpr_thd_currentid_type) GetProcAddress(library, "gpr_thd_currentid"); - gpr_thd_join_import = (gpr_thd_join_type) GetProcAddress(library, "gpr_thd_join"); -} - -#ifdef __cplusplus -} -#endif /* __cpluslus */ - -#endif /* !GPR_WINDOWS */ diff --git a/src/python/grpcio/grpc/_cython/imports.generated.h b/src/python/grpcio/grpc/_cython/imports.generated.h index f87c4da7878..8e5c9a8ce2b 100644 --- a/src/python/grpcio/grpc/_cython/imports.generated.h +++ b/src/python/grpcio/grpc/_cython/imports.generated.h @@ -31,860 +31,12 @@ * */ +/* TODO(atash) remove cruft */ #ifndef PYGRPC_CYTHON_WINDOWS_IMPORTS_H_ #define PYGRPC_CYTHON_WINDOWS_IMPORTS_H_ #include -#ifdef GPR_WINDOWS - -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -typedef int(*census_initialize_type)(int features); -extern census_initialize_type census_initialize_import; -#define census_initialize census_initialize_import -typedef void(*census_shutdown_type)(void); -extern census_shutdown_type census_shutdown_import; -#define census_shutdown census_shutdown_import -typedef int(*census_supported_type)(void); -extern census_supported_type census_supported_import; -#define census_supported census_supported_import -typedef int(*census_enabled_type)(void); -extern census_enabled_type census_enabled_import; -#define census_enabled census_enabled_import -typedef census_context *(*census_context_create_type)(const census_context *base, const census_tag *tags, int ntags, census_context_status const **status); -extern census_context_create_type census_context_create_import; -#define census_context_create census_context_create_import -typedef void(*census_context_destroy_type)(census_context *context); -extern census_context_destroy_type census_context_destroy_import; -#define census_context_destroy census_context_destroy_import -typedef const census_context_status *(*census_context_get_status_type)(const census_context *context); -extern census_context_get_status_type census_context_get_status_import; -#define census_context_get_status census_context_get_status_import -typedef void(*census_context_initialize_iterator_type)(const census_context *context, census_context_iterator *iterator); -extern census_context_initialize_iterator_type census_context_initialize_iterator_import; -#define census_context_initialize_iterator census_context_initialize_iterator_import -typedef int(*census_context_next_tag_type)(census_context_iterator *iterator, census_tag *tag); -extern census_context_next_tag_type census_context_next_tag_import; -#define census_context_next_tag census_context_next_tag_import -typedef int(*census_context_get_tag_type)(const census_context *context, const char *key, census_tag *tag); -extern census_context_get_tag_type census_context_get_tag_import; -#define census_context_get_tag census_context_get_tag_import -typedef size_t(*census_context_encode_type)(const census_context *context, char *buffer, size_t buf_size); -extern census_context_encode_type census_context_encode_import; -#define census_context_encode census_context_encode_import -typedef census_context *(*census_context_decode_type)(const char *buffer, size_t size); -extern census_context_decode_type census_context_decode_import; -#define census_context_decode census_context_decode_import -typedef int(*census_trace_mask_type)(const census_context *context); -extern census_trace_mask_type census_trace_mask_import; -#define census_trace_mask census_trace_mask_import -typedef void(*census_set_trace_mask_type)(int trace_mask); -extern census_set_trace_mask_type census_set_trace_mask_import; -#define census_set_trace_mask census_set_trace_mask_import -typedef census_timestamp(*census_start_rpc_op_timestamp_type)(void); -extern census_start_rpc_op_timestamp_type census_start_rpc_op_timestamp_import; -#define census_start_rpc_op_timestamp census_start_rpc_op_timestamp_import -typedef census_context *(*census_start_client_rpc_op_type)(const census_context *context, int64_t rpc_name_id, const census_rpc_name_info *rpc_name_info, const char *peer, int trace_mask, const census_timestamp *start_time); -extern census_start_client_rpc_op_type census_start_client_rpc_op_import; -#define census_start_client_rpc_op census_start_client_rpc_op_import -typedef void(*census_set_rpc_client_peer_type)(census_context *context, const char *peer); -extern census_set_rpc_client_peer_type census_set_rpc_client_peer_import; -#define census_set_rpc_client_peer census_set_rpc_client_peer_import -typedef census_context *(*census_start_server_rpc_op_type)(const char *buffer, int64_t rpc_name_id, const census_rpc_name_info *rpc_name_info, const char *peer, int trace_mask, census_timestamp *start_time); -extern census_start_server_rpc_op_type census_start_server_rpc_op_import; -#define census_start_server_rpc_op census_start_server_rpc_op_import -typedef census_context *(*census_start_op_type)(census_context *context, const char *family, const char *name, int trace_mask); -extern census_start_op_type census_start_op_import; -#define census_start_op census_start_op_import -typedef void(*census_end_op_type)(census_context *context, int status); -extern census_end_op_type census_end_op_import; -#define census_end_op census_end_op_import -typedef void(*census_trace_print_type)(census_context *context, uint32_t type, const char *buffer, size_t n); -extern census_trace_print_type census_trace_print_import; -#define census_trace_print census_trace_print_import -typedef int(*census_trace_scan_start_type)(int consume); -extern census_trace_scan_start_type census_trace_scan_start_import; -#define census_trace_scan_start census_trace_scan_start_import -typedef int(*census_get_trace_record_type)(census_trace_record *trace_record); -extern census_get_trace_record_type census_get_trace_record_import; -#define census_get_trace_record census_get_trace_record_import -typedef void(*census_trace_scan_end_type)(); -extern census_trace_scan_end_type census_trace_scan_end_import; -#define census_trace_scan_end census_trace_scan_end_import -typedef void(*census_record_values_type)(census_context *context, census_value *values, size_t nvalues); -extern census_record_values_type census_record_values_import; -#define census_record_values census_record_values_import -typedef census_view *(*census_view_create_type)(uint32_t metric_id, const census_context *tags, const census_aggregation *aggregations, size_t naggregations); -extern census_view_create_type census_view_create_import; -#define census_view_create census_view_create_import -typedef void(*census_view_delete_type)(census_view *view); -extern census_view_delete_type census_view_delete_import; -#define census_view_delete census_view_delete_import -typedef size_t(*census_view_metric_type)(const census_view *view); -extern census_view_metric_type census_view_metric_import; -#define census_view_metric census_view_metric_import -typedef size_t(*census_view_naggregations_type)(const census_view *view); -extern census_view_naggregations_type census_view_naggregations_import; -#define census_view_naggregations census_view_naggregations_import -typedef const census_context *(*census_view_tags_type)(const census_view *view); -extern census_view_tags_type census_view_tags_import; -#define census_view_tags census_view_tags_import -typedef const census_aggregation *(*census_view_aggregrations_type)(const census_view *view); -extern census_view_aggregrations_type census_view_aggregrations_import; -#define census_view_aggregrations census_view_aggregrations_import -typedef const census_view_data *(*census_view_get_data_type)(const census_view *view); -extern census_view_get_data_type census_view_get_data_import; -#define census_view_get_data census_view_get_data_import -typedef void(*census_view_reset_type)(census_view *view); -extern census_view_reset_type census_view_reset_import; -#define census_view_reset census_view_reset_import -typedef int(*grpc_compression_algorithm_parse_type)(const char *name, size_t name_length, grpc_compression_algorithm *algorithm); -extern grpc_compression_algorithm_parse_type grpc_compression_algorithm_parse_import; -#define grpc_compression_algorithm_parse grpc_compression_algorithm_parse_import -typedef int(*grpc_compression_algorithm_name_type)(grpc_compression_algorithm algorithm, char **name); -extern grpc_compression_algorithm_name_type grpc_compression_algorithm_name_import; -#define grpc_compression_algorithm_name grpc_compression_algorithm_name_import -typedef grpc_compression_algorithm(*grpc_compression_algorithm_for_level_type)(grpc_compression_level level, uint32_t accepted_encodings); -extern grpc_compression_algorithm_for_level_type grpc_compression_algorithm_for_level_import; -#define grpc_compression_algorithm_for_level grpc_compression_algorithm_for_level_import -typedef void(*grpc_compression_options_init_type)(grpc_compression_options *opts); -extern grpc_compression_options_init_type grpc_compression_options_init_import; -#define grpc_compression_options_init grpc_compression_options_init_import -typedef void(*grpc_compression_options_enable_algorithm_type)(grpc_compression_options *opts, grpc_compression_algorithm algorithm); -extern grpc_compression_options_enable_algorithm_type grpc_compression_options_enable_algorithm_import; -#define grpc_compression_options_enable_algorithm grpc_compression_options_enable_algorithm_import -typedef void(*grpc_compression_options_disable_algorithm_type)(grpc_compression_options *opts, grpc_compression_algorithm algorithm); -extern grpc_compression_options_disable_algorithm_type grpc_compression_options_disable_algorithm_import; -#define grpc_compression_options_disable_algorithm grpc_compression_options_disable_algorithm_import -typedef int(*grpc_compression_options_is_algorithm_enabled_type)(const grpc_compression_options *opts, grpc_compression_algorithm algorithm); -extern grpc_compression_options_is_algorithm_enabled_type grpc_compression_options_is_algorithm_enabled_import; -#define grpc_compression_options_is_algorithm_enabled grpc_compression_options_is_algorithm_enabled_import -typedef void(*grpc_metadata_array_init_type)(grpc_metadata_array *array); -extern grpc_metadata_array_init_type grpc_metadata_array_init_import; -#define grpc_metadata_array_init grpc_metadata_array_init_import -typedef void(*grpc_metadata_array_destroy_type)(grpc_metadata_array *array); -extern grpc_metadata_array_destroy_type grpc_metadata_array_destroy_import; -#define grpc_metadata_array_destroy grpc_metadata_array_destroy_import -typedef void(*grpc_call_details_init_type)(grpc_call_details *details); -extern grpc_call_details_init_type grpc_call_details_init_import; -#define grpc_call_details_init grpc_call_details_init_import -typedef void(*grpc_call_details_destroy_type)(grpc_call_details *details); -extern grpc_call_details_destroy_type grpc_call_details_destroy_import; -#define grpc_call_details_destroy grpc_call_details_destroy_import -typedef void(*grpc_register_plugin_type)(void (*init)(void), void (*destroy)(void)); -extern grpc_register_plugin_type grpc_register_plugin_import; -#define grpc_register_plugin grpc_register_plugin_import -typedef void(*grpc_init_type)(void); -extern grpc_init_type grpc_init_import; -#define grpc_init grpc_init_import -typedef void(*grpc_shutdown_type)(void); -extern grpc_shutdown_type grpc_shutdown_import; -#define grpc_shutdown grpc_shutdown_import -typedef const char *(*grpc_version_string_type)(void); -extern grpc_version_string_type grpc_version_string_import; -#define grpc_version_string grpc_version_string_import -typedef grpc_completion_queue *(*grpc_completion_queue_create_type)(void *reserved); -extern grpc_completion_queue_create_type grpc_completion_queue_create_import; -#define grpc_completion_queue_create grpc_completion_queue_create_import -typedef grpc_event(*grpc_completion_queue_next_type)(grpc_completion_queue *cq, gpr_timespec deadline, void *reserved); -extern grpc_completion_queue_next_type grpc_completion_queue_next_import; -#define grpc_completion_queue_next grpc_completion_queue_next_import -typedef grpc_event(*grpc_completion_queue_pluck_type)(grpc_completion_queue *cq, void *tag, gpr_timespec deadline, void *reserved); -extern grpc_completion_queue_pluck_type grpc_completion_queue_pluck_import; -#define grpc_completion_queue_pluck grpc_completion_queue_pluck_import -typedef void(*grpc_completion_queue_shutdown_type)(grpc_completion_queue *cq); -extern grpc_completion_queue_shutdown_type grpc_completion_queue_shutdown_import; -#define grpc_completion_queue_shutdown grpc_completion_queue_shutdown_import -typedef void(*grpc_completion_queue_destroy_type)(grpc_completion_queue *cq); -extern grpc_completion_queue_destroy_type grpc_completion_queue_destroy_import; -#define grpc_completion_queue_destroy grpc_completion_queue_destroy_import -typedef grpc_alarm *(*grpc_alarm_create_type)(grpc_completion_queue *cq, gpr_timespec deadline, void *tag); -extern grpc_alarm_create_type grpc_alarm_create_import; -#define grpc_alarm_create grpc_alarm_create_import -typedef void(*grpc_alarm_cancel_type)(grpc_alarm *alarm); -extern grpc_alarm_cancel_type grpc_alarm_cancel_import; -#define grpc_alarm_cancel grpc_alarm_cancel_import -typedef void(*grpc_alarm_destroy_type)(grpc_alarm *alarm); -extern grpc_alarm_destroy_type grpc_alarm_destroy_import; -#define grpc_alarm_destroy grpc_alarm_destroy_import -typedef grpc_connectivity_state(*grpc_channel_check_connectivity_state_type)(grpc_channel *channel, int try_to_connect); -extern grpc_channel_check_connectivity_state_type grpc_channel_check_connectivity_state_import; -#define grpc_channel_check_connectivity_state grpc_channel_check_connectivity_state_import -typedef void(*grpc_channel_watch_connectivity_state_type)(grpc_channel *channel, grpc_connectivity_state last_observed_state, gpr_timespec deadline, grpc_completion_queue *cq, void *tag); -extern grpc_channel_watch_connectivity_state_type grpc_channel_watch_connectivity_state_import; -#define grpc_channel_watch_connectivity_state grpc_channel_watch_connectivity_state_import -typedef grpc_call *(*grpc_channel_create_call_type)(grpc_channel *channel, grpc_call *parent_call, uint32_t propagation_mask, grpc_completion_queue *completion_queue, const char *method, const char *host, gpr_timespec deadline, void *reserved); -extern grpc_channel_create_call_type grpc_channel_create_call_import; -#define grpc_channel_create_call grpc_channel_create_call_import -typedef void(*grpc_channel_ping_type)(grpc_channel *channel, grpc_completion_queue *cq, void *tag, void *reserved); -extern grpc_channel_ping_type grpc_channel_ping_import; -#define grpc_channel_ping grpc_channel_ping_import -typedef void *(*grpc_channel_register_call_type)(grpc_channel *channel, const char *method, const char *host, void *reserved); -extern grpc_channel_register_call_type grpc_channel_register_call_import; -#define grpc_channel_register_call grpc_channel_register_call_import -typedef grpc_call *(*grpc_channel_create_registered_call_type)(grpc_channel *channel, grpc_call *parent_call, uint32_t propagation_mask, grpc_completion_queue *completion_queue, void *registered_call_handle, gpr_timespec deadline, void *reserved); -extern grpc_channel_create_registered_call_type grpc_channel_create_registered_call_import; -#define grpc_channel_create_registered_call grpc_channel_create_registered_call_import -typedef grpc_call_error(*grpc_call_start_batch_type)(grpc_call *call, const grpc_op *ops, size_t nops, void *tag, void *reserved); -extern grpc_call_start_batch_type grpc_call_start_batch_import; -#define grpc_call_start_batch grpc_call_start_batch_import -typedef char *(*grpc_call_get_peer_type)(grpc_call *call); -extern grpc_call_get_peer_type grpc_call_get_peer_import; -#define grpc_call_get_peer grpc_call_get_peer_import -typedef void(*grpc_census_call_set_context_type)(grpc_call *call, struct census_context *context); -extern grpc_census_call_set_context_type grpc_census_call_set_context_import; -#define grpc_census_call_set_context grpc_census_call_set_context_import -typedef struct census_context *(*grpc_census_call_get_context_type)(grpc_call *call); -extern grpc_census_call_get_context_type grpc_census_call_get_context_import; -#define grpc_census_call_get_context grpc_census_call_get_context_import -typedef char *(*grpc_channel_get_target_type)(grpc_channel *channel); -extern grpc_channel_get_target_type grpc_channel_get_target_import; -#define grpc_channel_get_target grpc_channel_get_target_import -typedef grpc_channel *(*grpc_insecure_channel_create_type)(const char *target, const grpc_channel_args *args, void *reserved); -extern grpc_insecure_channel_create_type grpc_insecure_channel_create_import; -#define grpc_insecure_channel_create grpc_insecure_channel_create_import -typedef grpc_channel *(*grpc_lame_client_channel_create_type)(const char *target, grpc_status_code error_code, const char *error_message); -extern grpc_lame_client_channel_create_type grpc_lame_client_channel_create_import; -#define grpc_lame_client_channel_create grpc_lame_client_channel_create_import -typedef void(*grpc_channel_destroy_type)(grpc_channel *channel); -extern grpc_channel_destroy_type grpc_channel_destroy_import; -#define grpc_channel_destroy grpc_channel_destroy_import -typedef grpc_call_error(*grpc_call_cancel_type)(grpc_call *call, void *reserved); -extern grpc_call_cancel_type grpc_call_cancel_import; -#define grpc_call_cancel grpc_call_cancel_import -typedef grpc_call_error(*grpc_call_cancel_with_status_type)(grpc_call *call, grpc_status_code status, const char *description, void *reserved); -extern grpc_call_cancel_with_status_type grpc_call_cancel_with_status_import; -#define grpc_call_cancel_with_status grpc_call_cancel_with_status_import -typedef void(*grpc_call_destroy_type)(grpc_call *call); -extern grpc_call_destroy_type grpc_call_destroy_import; -#define grpc_call_destroy grpc_call_destroy_import -typedef grpc_call_error(*grpc_server_request_call_type)(grpc_server *server, grpc_call **call, grpc_call_details *details, grpc_metadata_array *request_metadata, grpc_completion_queue *cq_bound_to_call, grpc_completion_queue *cq_for_notification, void *tag_new); -extern grpc_server_request_call_type grpc_server_request_call_import; -#define grpc_server_request_call grpc_server_request_call_import -typedef void *(*grpc_server_register_method_type)(grpc_server *server, const char *method, const char *host, grpc_server_register_method_payload_handling payload_handling, uint32_t flags); -extern grpc_server_register_method_type grpc_server_register_method_import; -#define grpc_server_register_method grpc_server_register_method_import -typedef grpc_call_error(*grpc_server_request_registered_call_type)(grpc_server *server, void *registered_method, grpc_call **call, gpr_timespec *deadline, grpc_metadata_array *request_metadata, grpc_byte_buffer **optional_payload, grpc_completion_queue *cq_bound_to_call, grpc_completion_queue *cq_for_notification, void *tag_new); -extern grpc_server_request_registered_call_type grpc_server_request_registered_call_import; -#define grpc_server_request_registered_call grpc_server_request_registered_call_import -typedef grpc_server *(*grpc_server_create_type)(const grpc_channel_args *args, void *reserved); -extern grpc_server_create_type grpc_server_create_import; -#define grpc_server_create grpc_server_create_import -typedef void(*grpc_server_register_completion_queue_type)(grpc_server *server, grpc_completion_queue *cq, void *reserved); -extern grpc_server_register_completion_queue_type grpc_server_register_completion_queue_import; -#define grpc_server_register_completion_queue grpc_server_register_completion_queue_import -typedef void(*grpc_server_register_non_listening_completion_queue_type)(grpc_server *server, grpc_completion_queue *q, void *reserved); -extern grpc_server_register_non_listening_completion_queue_type grpc_server_register_non_listening_completion_queue_import; -#define grpc_server_register_non_listening_completion_queue grpc_server_register_non_listening_completion_queue_import -typedef int(*grpc_server_add_insecure_http2_port_type)(grpc_server *server, const char *addr); -extern grpc_server_add_insecure_http2_port_type grpc_server_add_insecure_http2_port_import; -#define grpc_server_add_insecure_http2_port grpc_server_add_insecure_http2_port_import -typedef void(*grpc_server_start_type)(grpc_server *server); -extern grpc_server_start_type grpc_server_start_import; -#define grpc_server_start grpc_server_start_import -typedef void(*grpc_server_shutdown_and_notify_type)(grpc_server *server, grpc_completion_queue *cq, void *tag); -extern grpc_server_shutdown_and_notify_type grpc_server_shutdown_and_notify_import; -#define grpc_server_shutdown_and_notify grpc_server_shutdown_and_notify_import -typedef void(*grpc_server_cancel_all_calls_type)(grpc_server *server); -extern grpc_server_cancel_all_calls_type grpc_server_cancel_all_calls_import; -#define grpc_server_cancel_all_calls grpc_server_cancel_all_calls_import -typedef void(*grpc_server_destroy_type)(grpc_server *server); -extern grpc_server_destroy_type grpc_server_destroy_import; -#define grpc_server_destroy grpc_server_destroy_import -typedef int(*grpc_tracer_set_enabled_type)(const char *name, int enabled); -extern grpc_tracer_set_enabled_type grpc_tracer_set_enabled_import; -#define grpc_tracer_set_enabled grpc_tracer_set_enabled_import -typedef int(*grpc_header_key_is_legal_type)(const char *key, size_t length); -extern grpc_header_key_is_legal_type grpc_header_key_is_legal_import; -#define grpc_header_key_is_legal grpc_header_key_is_legal_import -typedef int(*grpc_header_nonbin_value_is_legal_type)(const char *value, size_t length); -extern grpc_header_nonbin_value_is_legal_type grpc_header_nonbin_value_is_legal_import; -#define grpc_header_nonbin_value_is_legal grpc_header_nonbin_value_is_legal_import -typedef int(*grpc_is_binary_header_type)(const char *key, size_t length); -extern grpc_is_binary_header_type grpc_is_binary_header_import; -#define grpc_is_binary_header grpc_is_binary_header_import -typedef const char *(*grpc_call_error_to_string_type)(grpc_call_error error); -extern grpc_call_error_to_string_type grpc_call_error_to_string_import; -#define grpc_call_error_to_string grpc_call_error_to_string_import -typedef grpc_channel *(*grpc_insecure_channel_create_from_fd_type)(const char *target, int fd, const grpc_channel_args *args); -extern grpc_insecure_channel_create_from_fd_type grpc_insecure_channel_create_from_fd_import; -#define grpc_insecure_channel_create_from_fd grpc_insecure_channel_create_from_fd_import -typedef void(*grpc_server_add_insecure_channel_from_fd_type)(grpc_server *server, grpc_completion_queue *cq, int fd); -extern grpc_server_add_insecure_channel_from_fd_type grpc_server_add_insecure_channel_from_fd_import; -#define grpc_server_add_insecure_channel_from_fd grpc_server_add_insecure_channel_from_fd_import -typedef void(*grpc_use_signal_type)(int signum); -extern grpc_use_signal_type grpc_use_signal_import; -#define grpc_use_signal grpc_use_signal_import -typedef const grpc_auth_property *(*grpc_auth_property_iterator_next_type)(grpc_auth_property_iterator *it); -extern grpc_auth_property_iterator_next_type grpc_auth_property_iterator_next_import; -#define grpc_auth_property_iterator_next grpc_auth_property_iterator_next_import -typedef grpc_auth_property_iterator(*grpc_auth_context_property_iterator_type)(const grpc_auth_context *ctx); -extern grpc_auth_context_property_iterator_type grpc_auth_context_property_iterator_import; -#define grpc_auth_context_property_iterator grpc_auth_context_property_iterator_import -typedef grpc_auth_property_iterator(*grpc_auth_context_peer_identity_type)(const grpc_auth_context *ctx); -extern grpc_auth_context_peer_identity_type grpc_auth_context_peer_identity_import; -#define grpc_auth_context_peer_identity grpc_auth_context_peer_identity_import -typedef grpc_auth_property_iterator(*grpc_auth_context_find_properties_by_name_type)(const grpc_auth_context *ctx, const char *name); -extern grpc_auth_context_find_properties_by_name_type grpc_auth_context_find_properties_by_name_import; -#define grpc_auth_context_find_properties_by_name grpc_auth_context_find_properties_by_name_import -typedef const char *(*grpc_auth_context_peer_identity_property_name_type)(const grpc_auth_context *ctx); -extern grpc_auth_context_peer_identity_property_name_type grpc_auth_context_peer_identity_property_name_import; -#define grpc_auth_context_peer_identity_property_name grpc_auth_context_peer_identity_property_name_import -typedef int(*grpc_auth_context_peer_is_authenticated_type)(const grpc_auth_context *ctx); -extern grpc_auth_context_peer_is_authenticated_type grpc_auth_context_peer_is_authenticated_import; -#define grpc_auth_context_peer_is_authenticated grpc_auth_context_peer_is_authenticated_import -typedef grpc_auth_context *(*grpc_call_auth_context_type)(grpc_call *call); -extern grpc_call_auth_context_type grpc_call_auth_context_import; -#define grpc_call_auth_context grpc_call_auth_context_import -typedef void(*grpc_auth_context_release_type)(grpc_auth_context *context); -extern grpc_auth_context_release_type grpc_auth_context_release_import; -#define grpc_auth_context_release grpc_auth_context_release_import -typedef void(*grpc_auth_context_add_property_type)(grpc_auth_context *ctx, const char *name, const char *value, size_t value_length); -extern grpc_auth_context_add_property_type grpc_auth_context_add_property_import; -#define grpc_auth_context_add_property grpc_auth_context_add_property_import -typedef void(*grpc_auth_context_add_cstring_property_type)(grpc_auth_context *ctx, const char *name, const char *value); -extern grpc_auth_context_add_cstring_property_type grpc_auth_context_add_cstring_property_import; -#define grpc_auth_context_add_cstring_property grpc_auth_context_add_cstring_property_import -typedef int(*grpc_auth_context_set_peer_identity_property_name_type)(grpc_auth_context *ctx, const char *name); -extern grpc_auth_context_set_peer_identity_property_name_type grpc_auth_context_set_peer_identity_property_name_import; -#define grpc_auth_context_set_peer_identity_property_name grpc_auth_context_set_peer_identity_property_name_import -typedef void(*grpc_channel_credentials_release_type)(grpc_channel_credentials *creds); -extern grpc_channel_credentials_release_type grpc_channel_credentials_release_import; -#define grpc_channel_credentials_release grpc_channel_credentials_release_import -typedef grpc_channel_credentials *(*grpc_google_default_credentials_create_type)(void); -extern grpc_google_default_credentials_create_type grpc_google_default_credentials_create_import; -#define grpc_google_default_credentials_create grpc_google_default_credentials_create_import -typedef void(*grpc_set_ssl_roots_override_callback_type)(grpc_ssl_roots_override_callback cb); -extern grpc_set_ssl_roots_override_callback_type grpc_set_ssl_roots_override_callback_import; -#define grpc_set_ssl_roots_override_callback grpc_set_ssl_roots_override_callback_import -typedef grpc_channel_credentials *(*grpc_ssl_credentials_create_type)(const char *pem_root_certs, grpc_ssl_pem_key_cert_pair *pem_key_cert_pair, void *reserved); -extern grpc_ssl_credentials_create_type grpc_ssl_credentials_create_import; -#define grpc_ssl_credentials_create grpc_ssl_credentials_create_import -typedef void(*grpc_call_credentials_release_type)(grpc_call_credentials *creds); -extern grpc_call_credentials_release_type grpc_call_credentials_release_import; -#define grpc_call_credentials_release grpc_call_credentials_release_import -typedef grpc_channel_credentials *(*grpc_composite_channel_credentials_create_type)(grpc_channel_credentials *channel_creds, grpc_call_credentials *call_creds, void *reserved); -extern grpc_composite_channel_credentials_create_type grpc_composite_channel_credentials_create_import; -#define grpc_composite_channel_credentials_create grpc_composite_channel_credentials_create_import -typedef grpc_call_credentials *(*grpc_composite_call_credentials_create_type)(grpc_call_credentials *creds1, grpc_call_credentials *creds2, void *reserved); -extern grpc_composite_call_credentials_create_type grpc_composite_call_credentials_create_import; -#define grpc_composite_call_credentials_create grpc_composite_call_credentials_create_import -typedef grpc_call_credentials *(*grpc_google_compute_engine_credentials_create_type)(void *reserved); -extern grpc_google_compute_engine_credentials_create_type grpc_google_compute_engine_credentials_create_import; -#define grpc_google_compute_engine_credentials_create grpc_google_compute_engine_credentials_create_import -typedef gpr_timespec(*grpc_max_auth_token_lifetime_type)(); -extern grpc_max_auth_token_lifetime_type grpc_max_auth_token_lifetime_import; -#define grpc_max_auth_token_lifetime grpc_max_auth_token_lifetime_import -typedef grpc_call_credentials *(*grpc_service_account_jwt_access_credentials_create_type)(const char *json_key, gpr_timespec token_lifetime, void *reserved); -extern grpc_service_account_jwt_access_credentials_create_type grpc_service_account_jwt_access_credentials_create_import; -#define grpc_service_account_jwt_access_credentials_create grpc_service_account_jwt_access_credentials_create_import -typedef grpc_call_credentials *(*grpc_google_refresh_token_credentials_create_type)(const char *json_refresh_token, void *reserved); -extern grpc_google_refresh_token_credentials_create_type grpc_google_refresh_token_credentials_create_import; -#define grpc_google_refresh_token_credentials_create grpc_google_refresh_token_credentials_create_import -typedef grpc_call_credentials *(*grpc_access_token_credentials_create_type)(const char *access_token, void *reserved); -extern grpc_access_token_credentials_create_type grpc_access_token_credentials_create_import; -#define grpc_access_token_credentials_create grpc_access_token_credentials_create_import -typedef grpc_call_credentials *(*grpc_google_iam_credentials_create_type)(const char *authorization_token, const char *authority_selector, void *reserved); -extern grpc_google_iam_credentials_create_type grpc_google_iam_credentials_create_import; -#define grpc_google_iam_credentials_create grpc_google_iam_credentials_create_import -typedef grpc_call_credentials *(*grpc_metadata_credentials_create_from_plugin_type)(grpc_metadata_credentials_plugin plugin, void *reserved); -extern grpc_metadata_credentials_create_from_plugin_type grpc_metadata_credentials_create_from_plugin_import; -#define grpc_metadata_credentials_create_from_plugin grpc_metadata_credentials_create_from_plugin_import -typedef grpc_channel *(*grpc_secure_channel_create_type)(grpc_channel_credentials *creds, const char *target, const grpc_channel_args *args, void *reserved); -extern grpc_secure_channel_create_type grpc_secure_channel_create_import; -#define grpc_secure_channel_create grpc_secure_channel_create_import -typedef void(*grpc_server_credentials_release_type)(grpc_server_credentials *creds); -extern grpc_server_credentials_release_type grpc_server_credentials_release_import; -#define grpc_server_credentials_release grpc_server_credentials_release_import -typedef grpc_server_credentials *(*grpc_ssl_server_credentials_create_type)(const char *pem_root_certs, grpc_ssl_pem_key_cert_pair *pem_key_cert_pairs, size_t num_key_cert_pairs, int force_client_auth, void *reserved); -extern grpc_ssl_server_credentials_create_type grpc_ssl_server_credentials_create_import; -#define grpc_ssl_server_credentials_create grpc_ssl_server_credentials_create_import -typedef grpc_server_credentials *(*grpc_ssl_server_credentials_create_ex_type)(const char *pem_root_certs, grpc_ssl_pem_key_cert_pair *pem_key_cert_pairs, size_t num_key_cert_pairs, grpc_ssl_client_certificate_request_type client_certificate_request, void *reserved); -extern grpc_ssl_server_credentials_create_ex_type grpc_ssl_server_credentials_create_ex_import; -#define grpc_ssl_server_credentials_create_ex grpc_ssl_server_credentials_create_ex_import -typedef int(*grpc_server_add_secure_http2_port_type)(grpc_server *server, const char *addr, grpc_server_credentials *creds); -extern grpc_server_add_secure_http2_port_type grpc_server_add_secure_http2_port_import; -#define grpc_server_add_secure_http2_port grpc_server_add_secure_http2_port_import -typedef grpc_call_error(*grpc_call_set_credentials_type)(grpc_call *call, grpc_call_credentials *creds); -extern grpc_call_set_credentials_type grpc_call_set_credentials_import; -#define grpc_call_set_credentials grpc_call_set_credentials_import -typedef void(*grpc_server_credentials_set_auth_metadata_processor_type)(grpc_server_credentials *creds, grpc_auth_metadata_processor processor); -extern grpc_server_credentials_set_auth_metadata_processor_type grpc_server_credentials_set_auth_metadata_processor_import; -#define grpc_server_credentials_set_auth_metadata_processor grpc_server_credentials_set_auth_metadata_processor_import -typedef void *(*gpr_malloc_type)(size_t size); -extern gpr_malloc_type gpr_malloc_import; -#define gpr_malloc gpr_malloc_import -typedef void(*gpr_free_type)(void *ptr); -extern gpr_free_type gpr_free_import; -#define gpr_free gpr_free_import -typedef void *(*gpr_realloc_type)(void *p, size_t size); -extern gpr_realloc_type gpr_realloc_import; -#define gpr_realloc gpr_realloc_import -typedef void *(*gpr_malloc_aligned_type)(size_t size, size_t alignment_log); -extern gpr_malloc_aligned_type gpr_malloc_aligned_import; -#define gpr_malloc_aligned gpr_malloc_aligned_import -typedef void(*gpr_free_aligned_type)(void *ptr); -extern gpr_free_aligned_type gpr_free_aligned_import; -#define gpr_free_aligned gpr_free_aligned_import -typedef void(*gpr_set_allocation_functions_type)(gpr_allocation_functions functions); -extern gpr_set_allocation_functions_type gpr_set_allocation_functions_import; -#define gpr_set_allocation_functions gpr_set_allocation_functions_import -typedef gpr_allocation_functions(*gpr_get_allocation_functions_type)(); -extern gpr_get_allocation_functions_type gpr_get_allocation_functions_import; -#define gpr_get_allocation_functions gpr_get_allocation_functions_import -typedef grpc_byte_buffer *(*grpc_raw_byte_buffer_create_type)(gpr_slice *slices, size_t nslices); -extern grpc_raw_byte_buffer_create_type grpc_raw_byte_buffer_create_import; -#define grpc_raw_byte_buffer_create grpc_raw_byte_buffer_create_import -typedef grpc_byte_buffer *(*grpc_raw_compressed_byte_buffer_create_type)(gpr_slice *slices, size_t nslices, grpc_compression_algorithm compression); -extern grpc_raw_compressed_byte_buffer_create_type grpc_raw_compressed_byte_buffer_create_import; -#define grpc_raw_compressed_byte_buffer_create grpc_raw_compressed_byte_buffer_create_import -typedef grpc_byte_buffer *(*grpc_byte_buffer_copy_type)(grpc_byte_buffer *bb); -extern grpc_byte_buffer_copy_type grpc_byte_buffer_copy_import; -#define grpc_byte_buffer_copy grpc_byte_buffer_copy_import -typedef size_t(*grpc_byte_buffer_length_type)(grpc_byte_buffer *bb); -extern grpc_byte_buffer_length_type grpc_byte_buffer_length_import; -#define grpc_byte_buffer_length grpc_byte_buffer_length_import -typedef void(*grpc_byte_buffer_destroy_type)(grpc_byte_buffer *byte_buffer); -extern grpc_byte_buffer_destroy_type grpc_byte_buffer_destroy_import; -#define grpc_byte_buffer_destroy grpc_byte_buffer_destroy_import -typedef int(*grpc_byte_buffer_reader_init_type)(grpc_byte_buffer_reader *reader, grpc_byte_buffer *buffer); -extern grpc_byte_buffer_reader_init_type grpc_byte_buffer_reader_init_import; -#define grpc_byte_buffer_reader_init grpc_byte_buffer_reader_init_import -typedef void(*grpc_byte_buffer_reader_destroy_type)(grpc_byte_buffer_reader *reader); -extern grpc_byte_buffer_reader_destroy_type grpc_byte_buffer_reader_destroy_import; -#define grpc_byte_buffer_reader_destroy grpc_byte_buffer_reader_destroy_import -typedef int(*grpc_byte_buffer_reader_next_type)(grpc_byte_buffer_reader *reader, gpr_slice *slice); -extern grpc_byte_buffer_reader_next_type grpc_byte_buffer_reader_next_import; -#define grpc_byte_buffer_reader_next grpc_byte_buffer_reader_next_import -typedef gpr_slice(*grpc_byte_buffer_reader_readall_type)(grpc_byte_buffer_reader *reader); -extern grpc_byte_buffer_reader_readall_type grpc_byte_buffer_reader_readall_import; -#define grpc_byte_buffer_reader_readall grpc_byte_buffer_reader_readall_import -typedef grpc_byte_buffer *(*grpc_raw_byte_buffer_from_reader_type)(grpc_byte_buffer_reader *reader); -extern grpc_raw_byte_buffer_from_reader_type grpc_raw_byte_buffer_from_reader_import; -#define grpc_raw_byte_buffer_from_reader grpc_raw_byte_buffer_from_reader_import -typedef void(*gpr_log_type)(const char *file, int line, gpr_log_severity severity, const char *format, ...) GPRC_PRINT_FORMAT_CHECK(4, 5); -extern gpr_log_type gpr_log_import; -#define gpr_log gpr_log_import -typedef void(*gpr_log_message_type)(const char *file, int line, gpr_log_severity severity, const char *message); -extern gpr_log_message_type gpr_log_message_import; -#define gpr_log_message gpr_log_message_import -typedef void(*gpr_set_log_verbosity_type)(gpr_log_severity min_severity_to_print); -extern gpr_set_log_verbosity_type gpr_set_log_verbosity_import; -#define gpr_set_log_verbosity gpr_set_log_verbosity_import -typedef void(*gpr_log_verbosity_init_type)(); -extern gpr_log_verbosity_init_type gpr_log_verbosity_init_import; -#define gpr_log_verbosity_init gpr_log_verbosity_init_import -typedef void(*gpr_set_log_function_type)(gpr_log_func func); -extern gpr_set_log_function_type gpr_set_log_function_import; -#define gpr_set_log_function gpr_set_log_function_import -typedef gpr_slice(*gpr_slice_ref_type)(gpr_slice s); -extern gpr_slice_ref_type gpr_slice_ref_import; -#define gpr_slice_ref gpr_slice_ref_import -typedef void(*gpr_slice_unref_type)(gpr_slice s); -extern gpr_slice_unref_type gpr_slice_unref_import; -#define gpr_slice_unref gpr_slice_unref_import -typedef gpr_slice(*gpr_slice_new_type)(void *p, size_t len, void (*destroy)(void *)); -extern gpr_slice_new_type gpr_slice_new_import; -#define gpr_slice_new gpr_slice_new_import -typedef gpr_slice(*gpr_slice_new_with_len_type)(void *p, size_t len, void (*destroy)(void *, size_t)); -extern gpr_slice_new_with_len_type gpr_slice_new_with_len_import; -#define gpr_slice_new_with_len gpr_slice_new_with_len_import -typedef gpr_slice(*gpr_slice_malloc_type)(size_t length); -extern gpr_slice_malloc_type gpr_slice_malloc_import; -#define gpr_slice_malloc gpr_slice_malloc_import -typedef gpr_slice(*gpr_slice_from_copied_string_type)(const char *source); -extern gpr_slice_from_copied_string_type gpr_slice_from_copied_string_import; -#define gpr_slice_from_copied_string gpr_slice_from_copied_string_import -typedef gpr_slice(*gpr_slice_from_copied_buffer_type)(const char *source, size_t len); -extern gpr_slice_from_copied_buffer_type gpr_slice_from_copied_buffer_import; -#define gpr_slice_from_copied_buffer gpr_slice_from_copied_buffer_import -typedef gpr_slice(*gpr_slice_from_static_string_type)(const char *source); -extern gpr_slice_from_static_string_type gpr_slice_from_static_string_import; -#define gpr_slice_from_static_string gpr_slice_from_static_string_import -typedef gpr_slice(*gpr_slice_sub_type)(gpr_slice s, size_t begin, size_t end); -extern gpr_slice_sub_type gpr_slice_sub_import; -#define gpr_slice_sub gpr_slice_sub_import -typedef gpr_slice(*gpr_slice_sub_no_ref_type)(gpr_slice s, size_t begin, size_t end); -extern gpr_slice_sub_no_ref_type gpr_slice_sub_no_ref_import; -#define gpr_slice_sub_no_ref gpr_slice_sub_no_ref_import -typedef gpr_slice(*gpr_slice_split_tail_type)(gpr_slice *s, size_t split); -extern gpr_slice_split_tail_type gpr_slice_split_tail_import; -#define gpr_slice_split_tail gpr_slice_split_tail_import -typedef gpr_slice(*gpr_slice_split_head_type)(gpr_slice *s, size_t split); -extern gpr_slice_split_head_type gpr_slice_split_head_import; -#define gpr_slice_split_head gpr_slice_split_head_import -typedef gpr_slice(*gpr_empty_slice_type)(void); -extern gpr_empty_slice_type gpr_empty_slice_import; -#define gpr_empty_slice gpr_empty_slice_import -typedef int(*gpr_slice_cmp_type)(gpr_slice a, gpr_slice b); -extern gpr_slice_cmp_type gpr_slice_cmp_import; -#define gpr_slice_cmp gpr_slice_cmp_import -typedef int(*gpr_slice_str_cmp_type)(gpr_slice a, const char *b); -extern gpr_slice_str_cmp_type gpr_slice_str_cmp_import; -#define gpr_slice_str_cmp gpr_slice_str_cmp_import -typedef void(*gpr_slice_buffer_init_type)(gpr_slice_buffer *sb); -extern gpr_slice_buffer_init_type gpr_slice_buffer_init_import; -#define gpr_slice_buffer_init gpr_slice_buffer_init_import -typedef void(*gpr_slice_buffer_destroy_type)(gpr_slice_buffer *sb); -extern gpr_slice_buffer_destroy_type gpr_slice_buffer_destroy_import; -#define gpr_slice_buffer_destroy gpr_slice_buffer_destroy_import -typedef void(*gpr_slice_buffer_add_type)(gpr_slice_buffer *sb, gpr_slice slice); -extern gpr_slice_buffer_add_type gpr_slice_buffer_add_import; -#define gpr_slice_buffer_add gpr_slice_buffer_add_import -typedef size_t(*gpr_slice_buffer_add_indexed_type)(gpr_slice_buffer *sb, gpr_slice slice); -extern gpr_slice_buffer_add_indexed_type gpr_slice_buffer_add_indexed_import; -#define gpr_slice_buffer_add_indexed gpr_slice_buffer_add_indexed_import -typedef void(*gpr_slice_buffer_addn_type)(gpr_slice_buffer *sb, gpr_slice *slices, size_t n); -extern gpr_slice_buffer_addn_type gpr_slice_buffer_addn_import; -#define gpr_slice_buffer_addn gpr_slice_buffer_addn_import -typedef uint8_t *(*gpr_slice_buffer_tiny_add_type)(gpr_slice_buffer *sb, size_t len); -extern gpr_slice_buffer_tiny_add_type gpr_slice_buffer_tiny_add_import; -#define gpr_slice_buffer_tiny_add gpr_slice_buffer_tiny_add_import -typedef void(*gpr_slice_buffer_pop_type)(gpr_slice_buffer *sb); -extern gpr_slice_buffer_pop_type gpr_slice_buffer_pop_import; -#define gpr_slice_buffer_pop gpr_slice_buffer_pop_import -typedef void(*gpr_slice_buffer_reset_and_unref_type)(gpr_slice_buffer *sb); -extern gpr_slice_buffer_reset_and_unref_type gpr_slice_buffer_reset_and_unref_import; -#define gpr_slice_buffer_reset_and_unref gpr_slice_buffer_reset_and_unref_import -typedef void(*gpr_slice_buffer_swap_type)(gpr_slice_buffer *a, gpr_slice_buffer *b); -extern gpr_slice_buffer_swap_type gpr_slice_buffer_swap_import; -#define gpr_slice_buffer_swap gpr_slice_buffer_swap_import -typedef void(*gpr_slice_buffer_move_into_type)(gpr_slice_buffer *src, gpr_slice_buffer *dst); -extern gpr_slice_buffer_move_into_type gpr_slice_buffer_move_into_import; -#define gpr_slice_buffer_move_into gpr_slice_buffer_move_into_import -typedef void(*gpr_slice_buffer_trim_end_type)(gpr_slice_buffer *src, size_t n, gpr_slice_buffer *garbage); -extern gpr_slice_buffer_trim_end_type gpr_slice_buffer_trim_end_import; -#define gpr_slice_buffer_trim_end gpr_slice_buffer_trim_end_import -typedef void(*gpr_slice_buffer_move_first_type)(gpr_slice_buffer *src, size_t n, gpr_slice_buffer *dst); -extern gpr_slice_buffer_move_first_type gpr_slice_buffer_move_first_import; -#define gpr_slice_buffer_move_first gpr_slice_buffer_move_first_import -typedef gpr_slice(*gpr_slice_buffer_take_first_type)(gpr_slice_buffer *src); -extern gpr_slice_buffer_take_first_type gpr_slice_buffer_take_first_import; -#define gpr_slice_buffer_take_first gpr_slice_buffer_take_first_import -typedef void(*gpr_mu_init_type)(gpr_mu *mu); -extern gpr_mu_init_type gpr_mu_init_import; -#define gpr_mu_init gpr_mu_init_import -typedef void(*gpr_mu_destroy_type)(gpr_mu *mu); -extern gpr_mu_destroy_type gpr_mu_destroy_import; -#define gpr_mu_destroy gpr_mu_destroy_import -typedef void(*gpr_mu_lock_type)(gpr_mu *mu); -extern gpr_mu_lock_type gpr_mu_lock_import; -#define gpr_mu_lock gpr_mu_lock_import -typedef void(*gpr_mu_unlock_type)(gpr_mu *mu); -extern gpr_mu_unlock_type gpr_mu_unlock_import; -#define gpr_mu_unlock gpr_mu_unlock_import -typedef int(*gpr_mu_trylock_type)(gpr_mu *mu); -extern gpr_mu_trylock_type gpr_mu_trylock_import; -#define gpr_mu_trylock gpr_mu_trylock_import -typedef void(*gpr_cv_init_type)(gpr_cv *cv); -extern gpr_cv_init_type gpr_cv_init_import; -#define gpr_cv_init gpr_cv_init_import -typedef void(*gpr_cv_destroy_type)(gpr_cv *cv); -extern gpr_cv_destroy_type gpr_cv_destroy_import; -#define gpr_cv_destroy gpr_cv_destroy_import -typedef int(*gpr_cv_wait_type)(gpr_cv *cv, gpr_mu *mu, gpr_timespec abs_deadline); -extern gpr_cv_wait_type gpr_cv_wait_import; -#define gpr_cv_wait gpr_cv_wait_import -typedef void(*gpr_cv_signal_type)(gpr_cv *cv); -extern gpr_cv_signal_type gpr_cv_signal_import; -#define gpr_cv_signal gpr_cv_signal_import -typedef void(*gpr_cv_broadcast_type)(gpr_cv *cv); -extern gpr_cv_broadcast_type gpr_cv_broadcast_import; -#define gpr_cv_broadcast gpr_cv_broadcast_import -typedef void(*gpr_once_init_type)(gpr_once *once, void (*init_routine)(void)); -extern gpr_once_init_type gpr_once_init_import; -#define gpr_once_init gpr_once_init_import -typedef void(*gpr_event_init_type)(gpr_event *ev); -extern gpr_event_init_type gpr_event_init_import; -#define gpr_event_init gpr_event_init_import -typedef void(*gpr_event_set_type)(gpr_event *ev, void *value); -extern gpr_event_set_type gpr_event_set_import; -#define gpr_event_set gpr_event_set_import -typedef void *(*gpr_event_get_type)(gpr_event *ev); -extern gpr_event_get_type gpr_event_get_import; -#define gpr_event_get gpr_event_get_import -typedef void *(*gpr_event_wait_type)(gpr_event *ev, gpr_timespec abs_deadline); -extern gpr_event_wait_type gpr_event_wait_import; -#define gpr_event_wait gpr_event_wait_import -typedef void(*gpr_ref_init_type)(gpr_refcount *r, int n); -extern gpr_ref_init_type gpr_ref_init_import; -#define gpr_ref_init gpr_ref_init_import -typedef void(*gpr_ref_type)(gpr_refcount *r); -extern gpr_ref_type gpr_ref_import; -#define gpr_ref gpr_ref_import -typedef void(*gpr_ref_non_zero_type)(gpr_refcount *r); -extern gpr_ref_non_zero_type gpr_ref_non_zero_import; -#define gpr_ref_non_zero gpr_ref_non_zero_import -typedef void(*gpr_refn_type)(gpr_refcount *r, int n); -extern gpr_refn_type gpr_refn_import; -#define gpr_refn gpr_refn_import -typedef int(*gpr_unref_type)(gpr_refcount *r); -extern gpr_unref_type gpr_unref_import; -#define gpr_unref gpr_unref_import -typedef void(*gpr_stats_init_type)(gpr_stats_counter *c, intptr_t n); -extern gpr_stats_init_type gpr_stats_init_import; -#define gpr_stats_init gpr_stats_init_import -typedef void(*gpr_stats_inc_type)(gpr_stats_counter *c, intptr_t inc); -extern gpr_stats_inc_type gpr_stats_inc_import; -#define gpr_stats_inc gpr_stats_inc_import -typedef intptr_t(*gpr_stats_read_type)(const gpr_stats_counter *c); -extern gpr_stats_read_type gpr_stats_read_import; -#define gpr_stats_read gpr_stats_read_import -typedef gpr_timespec(*gpr_time_0_type)(gpr_clock_type type); -extern gpr_time_0_type gpr_time_0_import; -#define gpr_time_0 gpr_time_0_import -typedef gpr_timespec(*gpr_inf_future_type)(gpr_clock_type type); -extern gpr_inf_future_type gpr_inf_future_import; -#define gpr_inf_future gpr_inf_future_import -typedef gpr_timespec(*gpr_inf_past_type)(gpr_clock_type type); -extern gpr_inf_past_type gpr_inf_past_import; -#define gpr_inf_past gpr_inf_past_import -typedef void(*gpr_time_init_type)(void); -extern gpr_time_init_type gpr_time_init_import; -#define gpr_time_init gpr_time_init_import -typedef gpr_timespec(*gpr_now_type)(gpr_clock_type clock); -extern gpr_now_type gpr_now_import; -#define gpr_now gpr_now_import -typedef gpr_timespec(*gpr_convert_clock_type_type)(gpr_timespec t, gpr_clock_type target_clock); -extern gpr_convert_clock_type_type gpr_convert_clock_type_import; -#define gpr_convert_clock_type gpr_convert_clock_type_import -typedef int(*gpr_time_cmp_type)(gpr_timespec a, gpr_timespec b); -extern gpr_time_cmp_type gpr_time_cmp_import; -#define gpr_time_cmp gpr_time_cmp_import -typedef gpr_timespec(*gpr_time_max_type)(gpr_timespec a, gpr_timespec b); -extern gpr_time_max_type gpr_time_max_import; -#define gpr_time_max gpr_time_max_import -typedef gpr_timespec(*gpr_time_min_type)(gpr_timespec a, gpr_timespec b); -extern gpr_time_min_type gpr_time_min_import; -#define gpr_time_min gpr_time_min_import -typedef gpr_timespec(*gpr_time_add_type)(gpr_timespec a, gpr_timespec b); -extern gpr_time_add_type gpr_time_add_import; -#define gpr_time_add gpr_time_add_import -typedef gpr_timespec(*gpr_time_sub_type)(gpr_timespec a, gpr_timespec b); -extern gpr_time_sub_type gpr_time_sub_import; -#define gpr_time_sub gpr_time_sub_import -typedef gpr_timespec(*gpr_time_from_micros_type)(int64_t x, gpr_clock_type clock_type); -extern gpr_time_from_micros_type gpr_time_from_micros_import; -#define gpr_time_from_micros gpr_time_from_micros_import -typedef gpr_timespec(*gpr_time_from_nanos_type)(int64_t x, gpr_clock_type clock_type); -extern gpr_time_from_nanos_type gpr_time_from_nanos_import; -#define gpr_time_from_nanos gpr_time_from_nanos_import -typedef gpr_timespec(*gpr_time_from_millis_type)(int64_t x, gpr_clock_type clock_type); -extern gpr_time_from_millis_type gpr_time_from_millis_import; -#define gpr_time_from_millis gpr_time_from_millis_import -typedef gpr_timespec(*gpr_time_from_seconds_type)(int64_t x, gpr_clock_type clock_type); -extern gpr_time_from_seconds_type gpr_time_from_seconds_import; -#define gpr_time_from_seconds gpr_time_from_seconds_import -typedef gpr_timespec(*gpr_time_from_minutes_type)(int64_t x, gpr_clock_type clock_type); -extern gpr_time_from_minutes_type gpr_time_from_minutes_import; -#define gpr_time_from_minutes gpr_time_from_minutes_import -typedef gpr_timespec(*gpr_time_from_hours_type)(int64_t x, gpr_clock_type clock_type); -extern gpr_time_from_hours_type gpr_time_from_hours_import; -#define gpr_time_from_hours gpr_time_from_hours_import -typedef int32_t(*gpr_time_to_millis_type)(gpr_timespec timespec); -extern gpr_time_to_millis_type gpr_time_to_millis_import; -#define gpr_time_to_millis gpr_time_to_millis_import -typedef int(*gpr_time_similar_type)(gpr_timespec a, gpr_timespec b, gpr_timespec threshold); -extern gpr_time_similar_type gpr_time_similar_import; -#define gpr_time_similar gpr_time_similar_import -typedef void(*gpr_sleep_until_type)(gpr_timespec until); -extern gpr_sleep_until_type gpr_sleep_until_import; -#define gpr_sleep_until gpr_sleep_until_import -typedef double(*gpr_timespec_to_micros_type)(gpr_timespec t); -extern gpr_timespec_to_micros_type gpr_timespec_to_micros_import; -#define gpr_timespec_to_micros gpr_timespec_to_micros_import -typedef gpr_avl(*gpr_avl_create_type)(const gpr_avl_vtable *vtable); -extern gpr_avl_create_type gpr_avl_create_import; -#define gpr_avl_create gpr_avl_create_import -typedef gpr_avl(*gpr_avl_ref_type)(gpr_avl avl); -extern gpr_avl_ref_type gpr_avl_ref_import; -#define gpr_avl_ref gpr_avl_ref_import -typedef void(*gpr_avl_unref_type)(gpr_avl avl); -extern gpr_avl_unref_type gpr_avl_unref_import; -#define gpr_avl_unref gpr_avl_unref_import -typedef gpr_avl(*gpr_avl_add_type)(gpr_avl avl, void *key, void *value); -extern gpr_avl_add_type gpr_avl_add_import; -#define gpr_avl_add gpr_avl_add_import -typedef gpr_avl(*gpr_avl_remove_type)(gpr_avl avl, void *key); -extern gpr_avl_remove_type gpr_avl_remove_import; -#define gpr_avl_remove gpr_avl_remove_import -typedef void *(*gpr_avl_get_type)(gpr_avl avl, void *key); -extern gpr_avl_get_type gpr_avl_get_import; -#define gpr_avl_get gpr_avl_get_import -typedef int(*gpr_avl_maybe_get_type)(gpr_avl avl, void *key, void **value); -extern gpr_avl_maybe_get_type gpr_avl_maybe_get_import; -#define gpr_avl_maybe_get gpr_avl_maybe_get_import -typedef int(*gpr_avl_is_empty_type)(gpr_avl avl); -extern gpr_avl_is_empty_type gpr_avl_is_empty_import; -#define gpr_avl_is_empty gpr_avl_is_empty_import -typedef gpr_cmdline *(*gpr_cmdline_create_type)(const char *description); -extern gpr_cmdline_create_type gpr_cmdline_create_import; -#define gpr_cmdline_create gpr_cmdline_create_import -typedef void(*gpr_cmdline_add_int_type)(gpr_cmdline *cl, const char *name, const char *help, int *value); -extern gpr_cmdline_add_int_type gpr_cmdline_add_int_import; -#define gpr_cmdline_add_int gpr_cmdline_add_int_import -typedef void(*gpr_cmdline_add_flag_type)(gpr_cmdline *cl, const char *name, const char *help, int *value); -extern gpr_cmdline_add_flag_type gpr_cmdline_add_flag_import; -#define gpr_cmdline_add_flag gpr_cmdline_add_flag_import -typedef void(*gpr_cmdline_add_string_type)(gpr_cmdline *cl, const char *name, const char *help, char **value); -extern gpr_cmdline_add_string_type gpr_cmdline_add_string_import; -#define gpr_cmdline_add_string gpr_cmdline_add_string_import -typedef void(*gpr_cmdline_on_extra_arg_type)(gpr_cmdline *cl, const char *name, const char *help, void (*on_extra_arg)(void *user_data, const char *arg), void *user_data); -extern gpr_cmdline_on_extra_arg_type gpr_cmdline_on_extra_arg_import; -#define gpr_cmdline_on_extra_arg gpr_cmdline_on_extra_arg_import -typedef void(*gpr_cmdline_set_survive_failure_type)(gpr_cmdline *cl); -extern gpr_cmdline_set_survive_failure_type gpr_cmdline_set_survive_failure_import; -#define gpr_cmdline_set_survive_failure gpr_cmdline_set_survive_failure_import -typedef int(*gpr_cmdline_parse_type)(gpr_cmdline *cl, int argc, char **argv); -extern gpr_cmdline_parse_type gpr_cmdline_parse_import; -#define gpr_cmdline_parse gpr_cmdline_parse_import -typedef void(*gpr_cmdline_destroy_type)(gpr_cmdline *cl); -extern gpr_cmdline_destroy_type gpr_cmdline_destroy_import; -#define gpr_cmdline_destroy gpr_cmdline_destroy_import -typedef char *(*gpr_cmdline_usage_string_type)(gpr_cmdline *cl, const char *argv0); -extern gpr_cmdline_usage_string_type gpr_cmdline_usage_string_import; -#define gpr_cmdline_usage_string gpr_cmdline_usage_string_import -typedef unsigned(*gpr_cpu_num_cores_type)(void); -extern gpr_cpu_num_cores_type gpr_cpu_num_cores_import; -#define gpr_cpu_num_cores gpr_cpu_num_cores_import -typedef unsigned(*gpr_cpu_current_cpu_type)(void); -extern gpr_cpu_current_cpu_type gpr_cpu_current_cpu_import; -#define gpr_cpu_current_cpu gpr_cpu_current_cpu_import -typedef gpr_histogram *(*gpr_histogram_create_type)(double resolution, double max_bucket_start); -extern gpr_histogram_create_type gpr_histogram_create_import; -#define gpr_histogram_create gpr_histogram_create_import -typedef void(*gpr_histogram_destroy_type)(gpr_histogram *h); -extern gpr_histogram_destroy_type gpr_histogram_destroy_import; -#define gpr_histogram_destroy gpr_histogram_destroy_import -typedef void(*gpr_histogram_add_type)(gpr_histogram *h, double x); -extern gpr_histogram_add_type gpr_histogram_add_import; -#define gpr_histogram_add gpr_histogram_add_import -typedef int(*gpr_histogram_merge_type)(gpr_histogram *dst, const gpr_histogram *src); -extern gpr_histogram_merge_type gpr_histogram_merge_import; -#define gpr_histogram_merge gpr_histogram_merge_import -typedef double(*gpr_histogram_percentile_type)(gpr_histogram *histogram, double percentile); -extern gpr_histogram_percentile_type gpr_histogram_percentile_import; -#define gpr_histogram_percentile gpr_histogram_percentile_import -typedef double(*gpr_histogram_mean_type)(gpr_histogram *histogram); -extern gpr_histogram_mean_type gpr_histogram_mean_import; -#define gpr_histogram_mean gpr_histogram_mean_import -typedef double(*gpr_histogram_stddev_type)(gpr_histogram *histogram); -extern gpr_histogram_stddev_type gpr_histogram_stddev_import; -#define gpr_histogram_stddev gpr_histogram_stddev_import -typedef double(*gpr_histogram_variance_type)(gpr_histogram *histogram); -extern gpr_histogram_variance_type gpr_histogram_variance_import; -#define gpr_histogram_variance gpr_histogram_variance_import -typedef double(*gpr_histogram_maximum_type)(gpr_histogram *histogram); -extern gpr_histogram_maximum_type gpr_histogram_maximum_import; -#define gpr_histogram_maximum gpr_histogram_maximum_import -typedef double(*gpr_histogram_minimum_type)(gpr_histogram *histogram); -extern gpr_histogram_minimum_type gpr_histogram_minimum_import; -#define gpr_histogram_minimum gpr_histogram_minimum_import -typedef double(*gpr_histogram_count_type)(gpr_histogram *histogram); -extern gpr_histogram_count_type gpr_histogram_count_import; -#define gpr_histogram_count gpr_histogram_count_import -typedef double(*gpr_histogram_sum_type)(gpr_histogram *histogram); -extern gpr_histogram_sum_type gpr_histogram_sum_import; -#define gpr_histogram_sum gpr_histogram_sum_import -typedef double(*gpr_histogram_sum_of_squares_type)(gpr_histogram *histogram); -extern gpr_histogram_sum_of_squares_type gpr_histogram_sum_of_squares_import; -#define gpr_histogram_sum_of_squares gpr_histogram_sum_of_squares_import -typedef const uint32_t *(*gpr_histogram_get_contents_type)(gpr_histogram *histogram, size_t *count); -extern gpr_histogram_get_contents_type gpr_histogram_get_contents_import; -#define gpr_histogram_get_contents gpr_histogram_get_contents_import -typedef void(*gpr_histogram_merge_contents_type)(gpr_histogram *histogram, const uint32_t *data, size_t data_count, double min_seen, double max_seen, double sum, double sum_of_squares, double count); -extern gpr_histogram_merge_contents_type gpr_histogram_merge_contents_import; -#define gpr_histogram_merge_contents gpr_histogram_merge_contents_import -typedef int(*gpr_join_host_port_type)(char **out, const char *host, int port); -extern gpr_join_host_port_type gpr_join_host_port_import; -#define gpr_join_host_port gpr_join_host_port_import -typedef int(*gpr_split_host_port_type)(const char *name, char **host, char **port); -extern gpr_split_host_port_type gpr_split_host_port_import; -#define gpr_split_host_port gpr_split_host_port_import -typedef char *(*gpr_format_message_type)(int messageid); -extern gpr_format_message_type gpr_format_message_import; -#define gpr_format_message gpr_format_message_import -typedef char *(*gpr_strdup_type)(const char *src); -extern gpr_strdup_type gpr_strdup_import; -#define gpr_strdup gpr_strdup_import -typedef int(*gpr_asprintf_type)(char **strp, const char *format, ...) GPRC_PRINT_FORMAT_CHECK(2, 3); -extern gpr_asprintf_type gpr_asprintf_import; -#define gpr_asprintf gpr_asprintf_import -typedef const char *(*gpr_subprocess_binary_extension_type)(); -extern gpr_subprocess_binary_extension_type gpr_subprocess_binary_extension_import; -#define gpr_subprocess_binary_extension gpr_subprocess_binary_extension_import -typedef gpr_subprocess *(*gpr_subprocess_create_type)(int argc, const char **argv); -extern gpr_subprocess_create_type gpr_subprocess_create_import; -#define gpr_subprocess_create gpr_subprocess_create_import -typedef void(*gpr_subprocess_destroy_type)(gpr_subprocess *p); -extern gpr_subprocess_destroy_type gpr_subprocess_destroy_import; -#define gpr_subprocess_destroy gpr_subprocess_destroy_import -typedef int(*gpr_subprocess_join_type)(gpr_subprocess *p); -extern gpr_subprocess_join_type gpr_subprocess_join_import; -#define gpr_subprocess_join gpr_subprocess_join_import -typedef void(*gpr_subprocess_interrupt_type)(gpr_subprocess *p); -extern gpr_subprocess_interrupt_type gpr_subprocess_interrupt_import; -#define gpr_subprocess_interrupt gpr_subprocess_interrupt_import -typedef int(*gpr_thd_new_type)(gpr_thd_id *t, void (*thd_body)(void *arg), void *arg, const gpr_thd_options *options); -extern gpr_thd_new_type gpr_thd_new_import; -#define gpr_thd_new gpr_thd_new_import -typedef gpr_thd_options(*gpr_thd_options_default_type)(void); -extern gpr_thd_options_default_type gpr_thd_options_default_import; -#define gpr_thd_options_default gpr_thd_options_default_import -typedef void(*gpr_thd_options_set_detached_type)(gpr_thd_options *options); -extern gpr_thd_options_set_detached_type gpr_thd_options_set_detached_import; -#define gpr_thd_options_set_detached gpr_thd_options_set_detached_import -typedef void(*gpr_thd_options_set_joinable_type)(gpr_thd_options *options); -extern gpr_thd_options_set_joinable_type gpr_thd_options_set_joinable_import; -#define gpr_thd_options_set_joinable gpr_thd_options_set_joinable_import -typedef int(*gpr_thd_options_is_detached_type)(const gpr_thd_options *options); -extern gpr_thd_options_is_detached_type gpr_thd_options_is_detached_import; -#define gpr_thd_options_is_detached gpr_thd_options_is_detached_import -typedef int(*gpr_thd_options_is_joinable_type)(const gpr_thd_options *options); -extern gpr_thd_options_is_joinable_type gpr_thd_options_is_joinable_import; -#define gpr_thd_options_is_joinable gpr_thd_options_is_joinable_import -typedef gpr_thd_id(*gpr_thd_currentid_type)(void); -extern gpr_thd_currentid_type gpr_thd_currentid_import; -#define gpr_thd_currentid gpr_thd_currentid_import -typedef void(*gpr_thd_join_type)(gpr_thd_id t); -extern gpr_thd_join_type gpr_thd_join_import; -#define gpr_thd_join gpr_thd_join_import - -#ifdef __cplusplus -extern "C" { -#endif /* __cpluslus */ - -void pygrpc_load_imports(HMODULE library); - -#ifdef __cplusplus -} -#endif /* __cpluslus */ - -#else /* !GPR_WINDOWS */ - #include #include #include @@ -895,6 +47,4 @@ void pygrpc_load_imports(HMODULE library); #include #include -#endif /* !GPR_WINDOWS */ - #endif diff --git a/src/python/grpcio/grpc/_cython/loader.c b/src/python/grpcio/grpc/_cython/loader.c index 86b70dbb02d..750a9f6fa38 100644 --- a/src/python/grpcio/grpc/_cython/loader.c +++ b/src/python/grpcio/grpc/_cython/loader.c @@ -38,37 +38,10 @@ extern "C" { #endif /* __cpluslus */ -#if GPR_WINDOWS - -int pygrpc_load_core(char *path) { - HMODULE grpc_c; -#ifdef GPR_ARCH_32 - /* Close your eyes for a moment, it'll all be over soon. */ - char *six = strrchr(path, '6'); - *six++ = '3'; - *six = '2'; -#endif - grpc_c = LoadLibraryA(path); - if (grpc_c) { - pygrpc_load_imports(grpc_c); - return 1; - } - - return 0; -} - -#else +/* TODO(atash) remove cruft */ int pygrpc_load_core(char *path) { return 1; } -#endif /* !GPR_WINDOWS */ - -// Cython doesn't have Py_AtExit bindings, so we call the C_API directly -int pygrpc_initialize_core(void) { - grpc_init(); - return Py_AtExit(grpc_shutdown) < 0 ? 0 : 1; -} - #ifdef __cplusplus } #endif /* __cpluslus */ diff --git a/src/python/grpcio/grpc/_cython/loader.h b/src/python/grpcio/grpc/_cython/loader.h index eb4b1a1b018..62fd2252047 100644 --- a/src/python/grpcio/grpc/_cython/loader.h +++ b/src/python/grpcio/grpc/_cython/loader.h @@ -39,6 +39,8 @@ /* Additional inclusions not covered by "imports.generated.h" */ #include +/* TODO(atash) remove cruft */ + #ifdef __cplusplus extern "C" { #endif /* __cpluslus */ diff --git a/templates/src/python/grpcio/grpc/_cython/imports.generated.c.template b/templates/src/python/grpcio/grpc/_cython/imports.generated.c.template index 84fa5e62bfe..d83bccad1db 100644 --- a/templates/src/python/grpcio/grpc/_cython/imports.generated.c.template +++ b/templates/src/python/grpcio/grpc/_cython/imports.generated.c.template @@ -33,29 +33,9 @@ * */ + /* TODO(atash) remove cruft */ #include #include "imports.generated.h" - #ifdef GPR_WINDOWS - - %for api in c_apis: - ${api.name}_type ${api.name}_import; - %endfor - - #ifdef __cplusplus - extern "C" { - #endif /* __cpluslus */ - - void pygrpc_load_imports(HMODULE library) { - %for api in c_apis: - ${api.name}_import = (${api.name}_type) GetProcAddress(library, "${api.name}"); - %endfor - } - - #ifdef __cplusplus - } - #endif /* __cpluslus */ - - #endif /* !GPR_WINDOWS */ diff --git a/templates/src/python/grpcio/grpc/_cython/imports.generated.h.template b/templates/src/python/grpcio/grpc/_cython/imports.generated.h.template index d0f60dc0a50..b85bc3dbd8b 100644 --- a/templates/src/python/grpcio/grpc/_cython/imports.generated.h.template +++ b/templates/src/python/grpcio/grpc/_cython/imports.generated.h.template @@ -33,37 +33,12 @@ * */ + /* TODO(atash) remove cruft */ #ifndef PYGRPC_CYTHON_WINDOWS_IMPORTS_H_ #define PYGRPC_CYTHON_WINDOWS_IMPORTS_H_ #include - #ifdef GPR_WINDOWS - - #include - - %for header in sorted(set(api.header for api in c_apis)): - #include <${'/'.join(header.split('/')[1:])}> - %endfor - - %for api in c_apis: - typedef ${api.return_type}(*${api.name}_type)(${api.arguments}); - extern ${api.name}_type ${api.name}_import; - #define ${api.name} ${api.name}_import - %endfor - - #ifdef __cplusplus - extern "C" { - #endif /* __cpluslus */ - - void pygrpc_load_imports(HMODULE library); - - #ifdef __cplusplus - } - #endif /* __cpluslus */ - - #else /* !GPR_WINDOWS */ - #include #include #include @@ -74,6 +49,4 @@ #include #include - #endif /* !GPR_WINDOWS */ - #endif diff --git a/tools/distrib/python/grpcio_tools/setup.py b/tools/distrib/python/grpcio_tools/setup.py index afb6063906e..e025158a82b 100644 --- a/tools/distrib/python/grpcio_tools/setup.py +++ b/tools/distrib/python/grpcio_tools/setup.py @@ -32,6 +32,7 @@ import errno import os import os.path import pkg_resources +import platform import shlex import shutil import sys @@ -45,6 +46,9 @@ from setuptools.command import build_ext os.chdir(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, os.path.abspath('.')) +import protoc_lib_deps +import grpc_version + PY3 = sys.version_info.major == 3 # There are some situations (like on Windows) where CC, CFLAGS, and LDFLAGS are @@ -60,8 +64,9 @@ EXTRA_LINK_ARGS = shlex.split(os.environ.get('GRPC_PYTHON_LDFLAGS', GRPC_PYTHON_TOOLS_PACKAGE = 'grpc.tools' GRPC_PYTHON_PROTO_RESOURCES_NAME = '_proto' -import protoc_lib_deps -import grpc_version +DEFINE_MACROS = (('HAVE_PTHREAD', 1),) +if "win32" in sys.platform and '64bit' in platform.architecture()[0]: + DEFINE_MACROS += (('MS_WIN64', 1),) # By default, Python3 distutils enforces compatibility of # c plugins (.so files) with the OSX version Python3 was built with. @@ -108,9 +113,9 @@ def protoc_ext_module(): protoc_lib_deps.CC_INCLUDE, ], language='c++', - define_macros=[('HAVE_PTHREAD', 1)], - extra_compile_args=EXTRA_COMPILE_ARGS, - extra_link_args=EXTRA_LINK_ARGS, + define_macros=list(DEFINE_MACROS), + extra_compile_args=list(EXTRA_COMPILE_ARGS), + extra_link_args=list(EXTRA_LINK_ARGS), ) return plugin_ext diff --git a/tools/run_tests/build_python.sh b/tools/run_tests/build_python.sh index 687b04e954c..63b15dd09e4 100755 --- a/tools/run_tests/build_python.sh +++ b/tools/run_tests/build_python.sh @@ -40,7 +40,7 @@ VENV_RELATIVE_PYTHON=${3:-bin/python} TOOLCHAIN=${4:-unix} ROOT=`pwd` -export CFLAGS="-I$ROOT/include -std=gnu99 -fno-wrapv" +export CFLAGS="-I$ROOT/include -std=gnu99 -fno-wrapv $CFLAGS" export GRPC_PYTHON_BUILD_WITH_CYTHON=1 # Default python on the host to fall back to when instantiating e.g. the @@ -61,6 +61,19 @@ if [ "${PLATFORM/Darwin}" = "$PLATFORM" ]; then fi fi fi +# TODO(atash) consider conceptualizing MinGW as a first-class platform and move +# these flags into our `setup.py`s +if [ "${PLATFORM/MINGW}" != "$PLATFORM" ]; then + # We're on MinGW, and our CFLAGS and LDFLAGS will be eaten by the void. Use + # our work-around environment variables instead. + PYTHON_MSVCR=`$PYTHON -c "from distutils.cygwinccompiler import get_msvcr; print(get_msvcr()[0])"` + export GRPC_PYTHON_LDFLAGS="-static-libgcc -static-libstdc++ -mcrtdll=$PYTHON_MSVCR -static -lpthread" + # See https://sourceforge.net/p/mingw-w64/bugs/363/ + export GRPC_PYTHON_CFLAGS="-D_ftime=_ftime64 -D_timeb=__timeb64" + # TODO(atash) set these flags for only grpcio-tools (they don't do any harm to + # grpcio, but they result in noisy warnings). + export GRPC_PYTHON_CFLAGS="-frtti -std=c++11 $GRPC_PYTHON_CFLAGS" +fi # Find `realpath` if [ -x "$(command -v realpath)" ]; then From 06c857cb86586755ab5f01f1a8fecc614f23dffa Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Mon, 13 Jun 2016 20:53:02 -0700 Subject: [PATCH 0468/1039] Patch monkeypatch link function to work in Python3 The modified link command was originally taken from a Python 2.x distutils. --- setup.py | 4 ++-- .../grpcio/{build.py => _unixccompiler_patch.py} | 12 ++++++++---- 2 files changed, 10 insertions(+), 6 deletions(-) rename src/python/grpcio/{build.py => _unixccompiler_patch.py} (91%) diff --git a/setup.py b/setup.py index c6d3da72994..700515b894d 100644 --- a/setup.py +++ b/setup.py @@ -57,13 +57,13 @@ os.chdir(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, os.path.abspath(PYTHON_STEM)) # Break import-style to ensure we can actually find our in-repo dependencies. -import build +import _unixccompiler_patch import commands import grpc_core_dependencies import grpc_version # TODO(atash) make this conditional on being on a mingw32 build -build.monkeypatch_unix_compiler() +_unixccompiler_patch.monkeypatch_unix_compiler() LICENSE = '3-clause BSD' diff --git a/src/python/grpcio/build.py b/src/python/grpcio/_unixccompiler_patch.py similarity index 91% rename from src/python/grpcio/build.py rename to src/python/grpcio/_unixccompiler_patch.py index df5b54cf69c..9a697989b30 100644 --- a/src/python/grpcio/build.py +++ b/src/python/grpcio/_unixccompiler_patch.py @@ -61,8 +61,9 @@ def _unix_piecemeal_link( if not dir in ('/lib', '/lib64', '/usr/lib', '/usr/lib64')] lib_opts = ccompiler.gen_lib_options(self, library_dirs, runtime_library_dirs, libraries) - if not isinstance(output_dir, basestring) and output_dir is not None: - raise TypeError, "'output_dir' must be a string or None" + if (not (isinstance(output_dir, str) or isinstance(output_dir, bytes)) + and output_dir is not None): + raise TypeError("'output_dir' must be a string or None") if output_dir is not None: output_filename = os.path.join(output_dir, output_filename) @@ -106,11 +107,14 @@ def _unix_piecemeal_link( escaped_ld_args = [arg.replace('\\', '\\\\') for arg in ld_args] command_file.write(' '.join(escaped_ld_args)) self.spawn(linker + ['@{}'.format(command_filename)]) - except errors.DistutilsExecError, msg: - raise ccompiler.LinkError, msg + except errors.DistutilsExecError: + raise ccompiler.LinkError else: log.debug("skipping %s (up-to-date)", output_filename) +# TODO(atash) try replacing this monkeypatch of the compiler harness' link +# operation with a monkeypatch of the distutils `spawn` that applies +# command-argument-file hacks where it can. Might be cleaner. def monkeypatch_unix_compiler(): """Monkeypatching is dumb, but it's either that or we become maintainers of something much, much bigger.""" From 639bb3996fa5b4f9d1785376c19ab69f747e0da8 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Sun, 5 Jun 2016 17:04:44 -0700 Subject: [PATCH 0469/1039] Build Python distributions standalone for Windows --- src/python/grpcio/grpc/_cython/cygrpc.pyx | 7 ---- src/python/grpcio/grpc/_cython/loader.c | 8 +++- tools/run_tests/build_artifact_python.bat | 48 ++++++++--------------- 3 files changed, 23 insertions(+), 40 deletions(-) diff --git a/src/python/grpcio/grpc/_cython/cygrpc.pyx b/src/python/grpcio/grpc/_cython/cygrpc.pyx index 7a8d0dd8a1d..e055d321bc7 100644 --- a/src/python/grpcio/grpc/_cython/cygrpc.pyx +++ b/src/python/grpcio/grpc/_cython/cygrpc.pyx @@ -50,13 +50,6 @@ include "grpc/_cython/_cygrpc/server.pyx.pxi" def _initialize(): - if 'win32' in sys.platform: - filename = pkg_resources.resource_filename( - 'grpc._cython', '_windows/grpc_c.64.python') - if not isinstance(filename, bytes): - filename = filename.encode() - if not pygrpc_load_core(filename): - raise ImportError('failed to load core gRPC library') if not pygrpc_initialize_core(): raise ImportError('failed to initialize core gRPC library') diff --git a/src/python/grpcio/grpc/_cython/loader.c b/src/python/grpcio/grpc/_cython/loader.c index 750a9f6fa38..34bd8975495 100644 --- a/src/python/grpcio/grpc/_cython/loader.c +++ b/src/python/grpcio/grpc/_cython/loader.c @@ -38,10 +38,14 @@ extern "C" { #endif /* __cpluslus */ -/* TODO(atash) remove cruft */ - int pygrpc_load_core(char *path) { return 1; } +// Cython doesn't have Py_AtExit bindings, so we call the C_API directly +int pygrpc_initialize_core(void) { + grpc_init(); + return Py_AtExit(grpc_shutdown) < 0 ? 0 : 1; +} + #ifdef __cplusplus } #endif /* __cpluslus */ diff --git a/tools/run_tests/build_artifact_python.bat b/tools/run_tests/build_artifact_python.bat index 295347e947c..7c8c2aa12d7 100644 --- a/tools/run_tests/build_artifact_python.bat +++ b/tools/run_tests/build_artifact_python.bat @@ -28,33 +28,24 @@ @rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -set NUGET=C:\nuget\nuget.exe -%NUGET% restore vsprojects\grpc.sln || goto :error - - -@call vsprojects\build_vs2013.bat vsprojects\grpc.sln /t:grpc_dll /p:Configuration=Release /p:PlatformToolset=v120 /p:Platform=Win32 || goto :error -@call vsprojects\build_vs2013.bat vsprojects\grpc.sln /t:grpc_dll /p:Configuration=Release /p:PlatformToolset=v120 /p:Platform=x64 || goto :error - -mkdir src\python\grpcio\grpc\_cython\_windows - -@rem TODO(atash): maybe we could avoid the grpc_c.(32|64).python shim below if -@rem this used the right python build? -copy /Y vsprojects\Release\grpc_dll.dll src\python\grpcio\grpc\_cython\_windows\grpc_c.32.python || goto :error -copy /Y vsprojects\x64\Release\grpc_dll.dll src\python\grpcio\grpc\_cython\_windows\grpc_c.64.python || goto :error - set PATH=C:\%1;C:\%1\scripts;C:\msys64\mingw%2\bin;%PATH% pip install --upgrade six pip install --upgrade setuptools pip install -rrequirements.txt -set GRPC_PYTHON_USE_CUSTOM_BDIST=0 -set GRPC_PYTHON_BUILD_WITH_CYTHON=1 - @rem Because this is windows and *everything seems to hate Windows* we have to @rem set all of these flags ourselves because Python won't help us (see the @rem setup.py of the grpcio_tools project). set GRPC_PYTHON_CFLAGS=-fno-wrapv -frtti -std=c++11 + +@rem See https://sourceforge.net/p/mingw-w64/bugs/363/ +if %2 == 32 ( + set GRPC_PYTHON_CFLAGS=%GRPC_PYTHON_CFLAGS% -D_ftime=_ftime32 -D_timeb=__timeb32 -D_ftime_s=_ftime32_s +) else ( + set GRPC_PYTHON_CFLAGS=%GRPC_PYTHON_CFLAGS% -D_ftime=_ftime64 -D_timeb=__timeb64 +) + @rem Further confusing things, MSYS2's mingw64 tries to dynamically link @rem libgcc, libstdc++, and winpthreads. We have to override this or our @rem extensions end up linking to MSYS2 DLLs, which the normal Python on @@ -66,23 +57,18 @@ python -c "from distutils.cygwinccompiler import get_msvcr; print(get_msvcr()[0] set /p PYTHON_MSVCR= Date: Tue, 28 Jun 2016 09:09:31 -0700 Subject: [PATCH 0470/1039] Make Python tests run on Windows --- src/python/grpcio_tests/tests/_runner.py | 19 +++--- tools/run_tests/build_python_msys2.sh | 36 ++++++++++++ tools/run_tests/run_tests.py | 74 ++++++++++++++---------- 3 files changed, 93 insertions(+), 36 deletions(-) create mode 100644 tools/run_tests/build_python_msys2.sh diff --git a/src/python/grpcio_tests/tests/_runner.py b/src/python/grpcio_tests/tests/_runner.py index 81c6100feda..926dcbe23a1 100644 --- a/src/python/grpcio_tests/tests/_runner.py +++ b/src/python/grpcio_tests/tests/_runner.py @@ -177,15 +177,20 @@ class Runner(object): stderr_pipe.write_bypass( '\ninterrupted stderr:\n{}\n'.format(stderr_pipe.output().decode())) os._exit(1) - signal.signal(signal.SIGINT, sigint_handler) - signal.signal(signal.SIGSEGV, fault_handler) - signal.signal(signal.SIGBUS, fault_handler) - signal.signal(signal.SIGABRT, fault_handler) - signal.signal(signal.SIGFPE, fault_handler) - signal.signal(signal.SIGILL, fault_handler) + def try_set_handler(name, handler): + try: + signal.signal(getattr(signal, name), handler) + except AttributeError: + pass + try_set_handler('SIGINT', sigint_handler) + try_set_handler('SIGSEGV', fault_handler) + try_set_handler('SIGBUS', fault_handler) + try_set_handler('SIGABRT', fault_handler) + try_set_handler('SIGFPE', fault_handler) + try_set_handler('SIGILL', fault_handler) # Sometimes output will lag after a test has successfully finished; we # ignore such writes to our pipes. - signal.signal(signal.SIGPIPE, signal.SIG_IGN) + try_set_handler('SIGPIPE', signal.SIG_IGN) # Run the tests result.startTestRun() diff --git a/tools/run_tests/build_python_msys2.sh b/tools/run_tests/build_python_msys2.sh new file mode 100644 index 00000000000..6e9d3690180 --- /dev/null +++ b/tools/run_tests/build_python_msys2.sh @@ -0,0 +1,36 @@ +#!/bin/bash +# Copyright 2016, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +set -ex + +BUILD_PYTHON=`realpath "$(dirname $0)/build_python.sh"` +export MSYSTEM=$1 +shift 1 +bash --login $BUILD_PYTHON "$@" diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index cbe17ee2ad7..3bc83c24799 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -375,19 +375,15 @@ class PhpLanguage(object): class PythonConfig(collections.namedtuple('PythonConfig', [ - 'python', 'venv', 'venv_relative_python', 'toolchain',])): - - @property - def venv_python(self): - return os.path.abspath('{}/{}'.format(self.venv, self.venv_relative_python)) - + 'name', 'build', 'run'])): + """Tuple of commands (named s.t. 'what it says on the tin' applies)""" class PythonLanguage(object): def configure(self, config, args): self.config = config self.args = args - self.pythons = self._get_pythons(self.args.compiler) + self.pythons = self._get_pythons(self.args) def test_specs(self): # load list of known test suites @@ -395,11 +391,11 @@ class PythonLanguage(object): tests_json = json.load(tests_json_file) environment = dict(_FORCE_ENVIRON_FOR_WRAPPERS) return [self.config.job_spec( - ['tools/run_tests/run_python.sh', config.venv_python], + config.run, timeout_seconds=5*60, environ=dict(environment.items() + [('GRPC_PYTHON_TESTRUNNER_FILTER', suite_name)]), - shortname='%s.test.%s' % (config.venv, suite_name),) + shortname='%s.test.%s' % (config.name, suite_name),) for suite_name in tests_json for config in self.pythons] @@ -413,14 +409,7 @@ class PythonLanguage(object): return [] def build_steps(self): - return [ - [ - 'tools/run_tests/build_python.sh', - config.python, config.venv, - config.venv_relative_python, config.toolchain - ] - for config in self.pythons - ] + return [config.build for config in self.pythons] def post_tests_steps(self): return [] @@ -431,23 +420,50 @@ class PythonLanguage(object): def dockerfile_dir(self): return 'tools/dockerfile/test/python_jessie_%s' % _docker_arch_suffix(self.args.arch) - def _get_pythons(self, compiler): + def _get_pythons(self, args): + if args.arch == 'x86': + bits = '32' + else: + bits = '64' if os.name == 'nt': - venv_relative_python = 'Scripts/python.exe' - toolchain = 'mingw32' + shell = ['bash'] + builder = [os.path.abspath('tools/run_tests/build_python_msys2.sh')] + builder_prefix_arguments = ['MINGW{}'.format(bits)] + venv_relative_python = ['Scripts/python.exe'] + toolchain = ['mingw32'] + python_pattern_function = lambda major, minor, bits: ( + '/c/Python{major}{minor}/python.exe'.format(major=major, minor=minor, bits=bits) + if bits == '64' else + '/c/Python{major}{minor}_{bits}bits/python.exe'.format( + major=major, minor=minor, bits=bits)) else: - venv_relative_python = 'bin/python' - toolchain = 'unix' - python27_config = PythonConfig('python2.7', 'py27', venv_relative_python, toolchain) - python34_config = PythonConfig('python3.4', 'py34', venv_relative_python, toolchain) - if compiler == 'default': - return (python27_config, python34_config,) - elif compiler == 'python2.7': + shell = [] + builder = [os.path.abspath('tools/run_tests/build_python.sh')] + builder_prefix_arguments = [] + venv_relative_python = ['bin/python'] + toolchain = ['unix'] + # Bit-ness is handled by the test machine's environment + python_pattern_function = lambda major, minor, bits: 'python{major}.{minor}'.format(major=major, minor=minor) + runner = [os.path.abspath('tools/run_tests/run_python.sh')] + python_config_generator = lambda name, major, minor, bits: PythonConfig( + name, + shell + builder + builder_prefix_arguments + + [python_pattern_function(major=major, minor=minor, bits=bits)] + + [name] + venv_relative_python + toolchain, + shell + runner + [os.path.join(name, venv_relative_python[0])]) + python27_config = python_config_generator(name='py27', major='2', minor='7', bits=bits) + python34_config = python_config_generator(name='py34', major='3', minor='4', bits=bits) + if args.compiler == 'default': + if os.name == 'nt': + return (python27_config,) + else: + return (python27_config, python34_config,) + elif args.compiler == 'python2.7': return (python27_config,) - elif compiler == 'python3.4': + elif args.compiler == 'python3.4': return (python34_config,) else: - raise Exception('Compiler %s not supported.' % compiler) + raise Exception('Compiler %s not supported.' % args.compiler) def __str__(self): return 'python' From 768b1db4df9726bf241690b70cf06e01e7fb126e Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Mon, 6 Jun 2016 16:45:19 -0700 Subject: [PATCH 0471/1039] Sanitize environment variables in run_tests `jobset` --- tools/run_tests/jobset.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tools/run_tests/jobset.py b/tools/run_tests/jobset.py index 4fe77487f96..3999537c40e 100755 --- a/tools/run_tests/jobset.py +++ b/tools/run_tests/jobset.py @@ -47,6 +47,12 @@ measure_cpu_costs = False _DEFAULT_MAX_JOBS = 16 * multiprocessing.cpu_count() _MAX_RESULT_SIZE = 8192 +def sanitized_environment(env): + sanitized = {} + for key, value in env.items(): + sanitized[str(key).encode()] = str(value).encode() + return sanitized + def platform_string(): if platform.system() == 'Windows': return 'windows' @@ -219,6 +225,7 @@ class Job(object): env = dict(os.environ) env.update(self._spec.environ) env.update(self._add_env) + env = sanitized_environment(env) self._start = time.time() cmdline = self._spec.cmdline if measure_cpu_costs: From fbf15e436ff316c8ec77de7bd37e0e66d53599e9 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Tue, 7 Jun 2016 19:13:11 -0700 Subject: [PATCH 0472/1039] Make build_python.sh script smarter Now reasonable defaults are auto-detected by platform (and by specific Python implementation). --- tools/run_tests/build_python.sh | 100 +++++++++++++++++++++++++------- 1 file changed, 80 insertions(+), 20 deletions(-) diff --git a/tools/run_tests/build_python.sh b/tools/run_tests/build_python.sh index 63b15dd09e4..a3fa8200d5d 100755 --- a/tools/run_tests/build_python.sh +++ b/tools/run_tests/build_python.sh @@ -33,11 +33,80 @@ set -ex # change to grpc repo root cd $(dirname $0)/../.. -# Arguments +########################## +# Portability operations # +########################## + +PLATFORM=`uname -s` + +function is_mingw() { + if [ "${PLATFORM/MINGW}" != "$PLATFORM" ]; then + echo true + else + exit 1 + fi +} + +function is_darwin() { + if [ "${PLATFORM/Darwin}" != "$PLATFORM" ]; then + echo true + else + exit 1 + fi +} + +function is_linux() { + if [ "${PLATFORM/Linux}" != "$PLATFORM" ]; then + echo true + else + exit 1 + fi +} + +# Associated virtual environment name for the given python command. +function venv() { + $1 -c "import sys; print('py{}{}'.format(*sys.version_info[:2]))" +} + +# Path to python executable within a virtual environment depending on the +# system. +function venv_relative_python() { + if [ $(is_mingw) ]; then + echo 'Scripts/python.exe' + else + echo 'bin/python' + fi +} + +# Distutils toolchain to use depending on the system. +function toolchain() { + if [ $(is_mingw) ]; then + echo 'mingw32' + else + echo 'unix' + fi +} + +# Command to invoke the linux command `realpath` or equivalent. +function script_realpath() { + # Find `realpath` + if [ -x "$(command -v realpath)" ]; then + realpath "$@" + elif [ -x "$(command -v grealpath)" ]; then + grealpath "$@" + else + exit 1 + fi +} + +#################### +# Script Arguments # +#################### + PYTHON=${1:-python2.7} -VENV=${2:-py27} -VENV_RELATIVE_PYTHON=${3:-bin/python} -TOOLCHAIN=${4:-unix} +VENV=${2:-$(venv $PYTHON)} +VENV_RELATIVE_PYTHON=${3:-$(venv_relative_python)} +TOOLCHAIN=${4:-$(toolchain)} ROOT=`pwd` export CFLAGS="-I$ROOT/include -std=gnu99 -fno-wrapv $CFLAGS" @@ -47,11 +116,8 @@ export GRPC_PYTHON_BUILD_WITH_CYTHON=1 # virtualenv. HOST_PYTHON=${HOST_PYTHON:-python} -# If ccache is available, use it... unless we're on Mac, then all hell breaks -# loose because Python does hacky things to support other hacky things done to -# hacky things on Mac OS X -PLATFORM=`uname -s` -if [ "${PLATFORM/Darwin}" = "$PLATFORM" ]; then +# If ccache is available on Linux, use it. +if [ $(is_linux) ]; then # We're not on Darwin (Mac OS X) if [ -x "$(command -v ccache)" ]; then if [ -x "$(command -v gcc)" ]; then @@ -63,7 +129,7 @@ if [ "${PLATFORM/Darwin}" = "$PLATFORM" ]; then fi # TODO(atash) consider conceptualizing MinGW as a first-class platform and move # these flags into our `setup.py`s -if [ "${PLATFORM/MINGW}" != "$PLATFORM" ]; then +if [ $(is_mingw) ]; then # We're on MinGW, and our CFLAGS and LDFLAGS will be eaten by the void. Use # our work-around environment variables instead. PYTHON_MSVCR=`$PYTHON -c "from distutils.cygwinccompiler import get_msvcr; print(get_msvcr()[0])"` @@ -75,15 +141,9 @@ if [ "${PLATFORM/MINGW}" != "$PLATFORM" ]; then export GRPC_PYTHON_CFLAGS="-frtti -std=c++11 $GRPC_PYTHON_CFLAGS" fi -# Find `realpath` -if [ -x "$(command -v realpath)" ]; then - export REALPATH=realpath -elif [ -x "$(command -v grealpath)" ]; then - export REALPATH=grealpath -else - echo 'Couldn'"'"'t find `realpath` or `grealpath`' - exit 1 -fi +############################ +# Perform build operations # +############################ # Instnatiate the virtualenv, preferring to do so from the relevant python # version. Even if these commands fail (e.g. on Windows due to name conflicts) @@ -93,7 +153,7 @@ fi ($PYTHON -m virtualenv $VENV || $HOST_PYTHON -m virtualenv -p $PYTHON $VENV || true) -VENV_PYTHON=`$REALPATH -s "$VENV/$VENV_RELATIVE_PYTHON"` +VENV_PYTHON=`script_realpath -s "$VENV/$VENV_RELATIVE_PYTHON"` # pip-installs the directory specified. Used because on MSYS the vanilla Windows # Python gets confused when parsing paths. From 3a9e6d9770c3c8fb86135a6afea4ef3514996a10 Mon Sep 17 00:00:00 2001 From: Masood Malekghassemi Date: Fri, 1 Jul 2016 06:52:13 -0700 Subject: [PATCH 0473/1039] Fix interop tests on Windows --- src/python/grpcio_tests/tests/interop/_insecure_interop_test.py | 2 +- src/python/grpcio_tests/tests/interop/_secure_interop_test.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/python/grpcio_tests/tests/interop/_insecure_interop_test.py b/src/python/grpcio_tests/tests/interop/_insecure_interop_test.py index 91519b6fba2..c753d6faf05 100644 --- a/src/python/grpcio_tests/tests/interop/_insecure_interop_test.py +++ b/src/python/grpcio_tests/tests/interop/_insecure_interop_test.py @@ -48,7 +48,7 @@ class InsecureInteropTest( port = self.server.add_insecure_port('[::]:0') self.server.start() self.stub = test_pb2.beta_create_TestService_stub( - implementations.insecure_channel('[::]', port)) + implementations.insecure_channel('localhost', port)) def tearDown(self): self.server.stop(0) diff --git a/src/python/grpcio_tests/tests/interop/_secure_interop_test.py b/src/python/grpcio_tests/tests/interop/_secure_interop_test.py index c61547b9778..cb09f54a347 100644 --- a/src/python/grpcio_tests/tests/interop/_secure_interop_test.py +++ b/src/python/grpcio_tests/tests/interop/_secure_interop_test.py @@ -55,7 +55,7 @@ class SecureInteropTest( self.server.start() self.stub = test_pb2.beta_create_TestService_stub( test_utilities.not_really_secure_channel( - '[::]', port, implementations.ssl_channel_credentials( + 'localhost', port, implementations.ssl_channel_credentials( resources.test_root_certificates()), _SERVER_HOST_OVERRIDE)) From 86202529a81ce45a1da160ef1dcc58540785d4b2 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Fri, 8 Jul 2016 12:41:12 -0700 Subject: [PATCH 0474/1039] fixed minor indent and resubmit --- src/compiler/objective_c_plugin.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/compiler/objective_c_plugin.cc b/src/compiler/objective_c_plugin.cc index 3878a332515..5026088db1e 100644 --- a/src/compiler/objective_c_plugin.cc +++ b/src/compiler/objective_c_plugin.cc @@ -88,9 +88,9 @@ class ObjectiveCGrpcGenerator : public grpc::protobuf::compiler::CodeGenerator { "#else\n" " #import \"" + header + "\"\n" "#endif\n"; - } else { + } else { proto_imports += ::grpc::string("#import \"") + header + "\"\n"; - } + } } ::grpc::string declarations; From c22e31fb053d7229aed69e43972cf0c94817a841 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 8 Jul 2016 13:22:18 -0700 Subject: [PATCH 0475/1039] Make it more likely to correctly report deadline exceeded --- .../chttp2/transport/chttp2_transport.c | 18 +++-- .../ext/transport/chttp2/transport/internal.h | 2 + .../ext/transport/chttp2/transport/parsing.c | 11 +-- .../chttp2/transport/status_conversion.c | 10 ++- .../chttp2/transport/status_conversion.h | 2 +- .../transport/chttp2/status_conversion_test.c | 68 ++++++++++++++----- 6 files changed, 81 insertions(+), 30 deletions(-) diff --git a/src/core/ext/transport/chttp2/transport/chttp2_transport.c b/src/core/ext/transport/chttp2/transport/chttp2_transport.c index 38e782b9b4f..5aae753c07d 100644 --- a/src/core/ext/transport/chttp2/transport/chttp2_transport.c +++ b/src/core/ext/transport/chttp2/transport/chttp2_transport.c @@ -513,6 +513,7 @@ static int init_stream(grpc_exec_ctx *exec_ctx, grpc_transport *gt, &s->global.received_trailing_metadata); grpc_chttp2_data_parser_init(&s->parsing.data_parser); gpr_slice_buffer_init(&s->writing.flow_controlled_buffer); + s->global.deadline = gpr_inf_future(GPR_CLOCK_MONOTONIC); REF_TRANSPORT(t, "stream"); @@ -988,6 +989,11 @@ static void perform_stream_op_locked(grpc_exec_ctx *exec_ctx, const size_t metadata_peer_limit = transport_global->settings[GRPC_PEER_SETTINGS] [GRPC_CHTTP2_SETTINGS_MAX_HEADER_LIST_SIZE]; + if (transport_global->is_client) { + stream_global->deadline = + gpr_time_min(stream_global->deadline, + stream_global->send_initial_metadata->deadline); + } if (metadata_size > metadata_peer_limit) { cancel_from_api( exec_ctx, transport_global, stream_global, @@ -1366,7 +1372,7 @@ static void remove_stream(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, GRPC_ERROR_UNREF(error); } -static void status_codes_from_error(grpc_error *error, +static void status_codes_from_error(grpc_error *error, gpr_timespec deadline, grpc_chttp2_error_code *http2_error, grpc_status_code *grpc_status) { intptr_t ip_http; @@ -1386,8 +1392,8 @@ static void status_codes_from_error(grpc_error *error, if (have_grpc) { *grpc_status = (grpc_status_code)ip_grpc; } else if (have_http) { - *grpc_status = - grpc_chttp2_http2_error_to_grpc_status((grpc_chttp2_error_code)ip_http); + *grpc_status = grpc_chttp2_http2_error_to_grpc_status( + (grpc_chttp2_error_code)ip_http, deadline); } else { *grpc_status = GRPC_STATUS_INTERNAL; } @@ -1400,7 +1406,8 @@ static void cancel_from_api(grpc_exec_ctx *exec_ctx, if (!stream_global->read_closed || !stream_global->write_closed) { grpc_status_code grpc_status; grpc_chttp2_error_code http_error; - status_codes_from_error(due_to_error, &http_error, &grpc_status); + status_codes_from_error(due_to_error, stream_global->deadline, &http_error, + &grpc_status); if (stream_global->id != 0) { gpr_slice_buffer_add( @@ -1536,7 +1543,8 @@ static void close_from_api(grpc_exec_ctx *exec_ctx, uint32_t len = 0; grpc_status_code grpc_status; grpc_chttp2_error_code http_error; - status_codes_from_error(error, &http_error, &grpc_status); + status_codes_from_error(error, stream_global->deadline, &http_error, + &grpc_status); GPR_ASSERT(grpc_status >= 0 && (int)grpc_status < 100); diff --git a/src/core/ext/transport/chttp2/transport/internal.h b/src/core/ext/transport/chttp2/transport/internal.h index b5180c6fc86..8d79e93ceb3 100644 --- a/src/core/ext/transport/chttp2/transport/internal.h +++ b/src/core/ext/transport/chttp2/transport/internal.h @@ -447,6 +447,8 @@ typedef struct { grpc_chttp2_incoming_metadata_buffer received_trailing_metadata; grpc_chttp2_incoming_frame_queue incoming_frames; + + gpr_timespec deadline; } grpc_chttp2_stream_global; typedef struct { diff --git a/src/core/ext/transport/chttp2/transport/parsing.c b/src/core/ext/transport/chttp2/transport/parsing.c index 991d7729af4..c5240ce38a7 100644 --- a/src/core/ext/transport/chttp2/transport/parsing.c +++ b/src/core/ext/transport/chttp2/transport/parsing.c @@ -87,8 +87,8 @@ void grpc_chttp2_prepare_to_read( transport_global->settings[GRPC_SENT_SETTINGS], sizeof(transport_parsing->last_sent_settings)); transport_parsing->max_frame_size = - transport_global->settings[GRPC_ACKED_SETTINGS] - [GRPC_CHTTP2_SETTINGS_MAX_FRAME_SIZE]; + transport_global + ->settings[GRPC_ACKED_SETTINGS][GRPC_CHTTP2_SETTINGS_MAX_FRAME_SIZE]; /* update the parsing view of incoming window */ while (grpc_chttp2_list_pop_unannounced_incoming_window_available( @@ -236,9 +236,10 @@ void grpc_chttp2_publish_reads( GRPC_ERROR_INT_HTTP2_ERROR, &reason); if (has_reason && reason != GRPC_CHTTP2_NO_ERROR) { grpc_status_code status_code = - has_reason ? grpc_chttp2_http2_error_to_grpc_status( - (grpc_chttp2_error_code)reason) - : GRPC_STATUS_INTERNAL; + has_reason + ? grpc_chttp2_http2_error_to_grpc_status( + (grpc_chttp2_error_code)reason, stream_global->deadline) + : GRPC_STATUS_INTERNAL; const char *status_details = grpc_error_string(stream_parsing->forced_close_error); gpr_slice slice_details = gpr_slice_from_copied_string(status_details); diff --git a/src/core/ext/transport/chttp2/transport/status_conversion.c b/src/core/ext/transport/chttp2/transport/status_conversion.c index c42fb9b3a1e..5dce2f2d0cc 100644 --- a/src/core/ext/transport/chttp2/transport/status_conversion.c +++ b/src/core/ext/transport/chttp2/transport/status_conversion.c @@ -39,6 +39,8 @@ int grpc_chttp2_grpc_status_to_http2_error(grpc_status_code status) { return GRPC_CHTTP2_NO_ERROR; case GRPC_STATUS_CANCELLED: return GRPC_CHTTP2_CANCEL; + case GRPC_STATUS_DEADLINE_EXCEEDED: + return GRPC_CHTTP2_CANCEL; case GRPC_STATUS_RESOURCE_EXHAUSTED: return GRPC_CHTTP2_ENHANCE_YOUR_CALM; case GRPC_STATUS_PERMISSION_DENIED: @@ -51,13 +53,17 @@ int grpc_chttp2_grpc_status_to_http2_error(grpc_status_code status) { } grpc_status_code grpc_chttp2_http2_error_to_grpc_status( - grpc_chttp2_error_code error) { + grpc_chttp2_error_code error, gpr_timespec deadline) { switch (error) { case GRPC_CHTTP2_NO_ERROR: /* should never be received */ return GRPC_STATUS_INTERNAL; case GRPC_CHTTP2_CANCEL: - return GRPC_STATUS_CANCELLED; + /* http2 cancel translates to STATUS_CANCELLED iff deadline hasn't been + * exceeded */ + return gpr_time_cmp(gpr_now(deadline.clock_type), deadline) >= 0 + ? GRPC_STATUS_DEADLINE_EXCEEDED + : GRPC_STATUS_CANCELLED; case GRPC_CHTTP2_ENHANCE_YOUR_CALM: return GRPC_STATUS_RESOURCE_EXHAUSTED; case GRPC_CHTTP2_INADEQUATE_SECURITY: diff --git a/src/core/ext/transport/chttp2/transport/status_conversion.h b/src/core/ext/transport/chttp2/transport/status_conversion.h index e7285e6fd51..953bc9f1e1f 100644 --- a/src/core/ext/transport/chttp2/transport/status_conversion.h +++ b/src/core/ext/transport/chttp2/transport/status_conversion.h @@ -41,7 +41,7 @@ grpc_chttp2_error_code grpc_chttp2_grpc_status_to_http2_error( grpc_status_code status); grpc_status_code grpc_chttp2_http2_error_to_grpc_status( - grpc_chttp2_error_code error); + grpc_chttp2_error_code error, gpr_timespec deadline); /* Conversion of HTTP status codes (:status) to grpc status codes */ grpc_status_code grpc_chttp2_http2_status_to_grpc_status(int status); diff --git a/test/core/transport/chttp2/status_conversion_test.c b/test/core/transport/chttp2/status_conversion_test.c index e6fc7857281..f5a5cd1395b 100644 --- a/test/core/transport/chttp2/status_conversion_test.c +++ b/test/core/transport/chttp2/status_conversion_test.c @@ -37,8 +37,8 @@ #define GRPC_STATUS_TO_HTTP2_ERROR(a, b) \ GPR_ASSERT(grpc_chttp2_grpc_status_to_http2_error(a) == (b)) -#define HTTP2_ERROR_TO_GRPC_STATUS(a, b) \ - GPR_ASSERT(grpc_chttp2_http2_error_to_grpc_status(a) == (b)) +#define HTTP2_ERROR_TO_GRPC_STATUS(a, deadline, b) \ + GPR_ASSERT(grpc_chttp2_http2_error_to_grpc_status(a, deadline) == (b)) #define GRPC_STATUS_TO_HTTP2_STATUS(a, b) \ GPR_ASSERT(grpc_chttp2_grpc_status_to_http2_status(a) == (b)) #define HTTP2_STATUS_TO_GRPC_STATUS(a, b) \ @@ -54,8 +54,7 @@ int main(int argc, char **argv) { GRPC_STATUS_TO_HTTP2_ERROR(GRPC_STATUS_UNKNOWN, GRPC_CHTTP2_INTERNAL_ERROR); GRPC_STATUS_TO_HTTP2_ERROR(GRPC_STATUS_INVALID_ARGUMENT, GRPC_CHTTP2_INTERNAL_ERROR); - GRPC_STATUS_TO_HTTP2_ERROR(GRPC_STATUS_DEADLINE_EXCEEDED, - GRPC_CHTTP2_INTERNAL_ERROR); + GRPC_STATUS_TO_HTTP2_ERROR(GRPC_STATUS_DEADLINE_EXCEEDED, GRPC_CHTTP2_CANCEL); GRPC_STATUS_TO_HTTP2_ERROR(GRPC_STATUS_NOT_FOUND, GRPC_CHTTP2_INTERNAL_ERROR); GRPC_STATUS_TO_HTTP2_ERROR(GRPC_STATUS_ALREADY_EXISTS, GRPC_CHTTP2_INTERNAL_ERROR); @@ -95,25 +94,60 @@ int main(int argc, char **argv) { GRPC_STATUS_TO_HTTP2_STATUS(GRPC_STATUS_UNAVAILABLE, 200); GRPC_STATUS_TO_HTTP2_STATUS(GRPC_STATUS_DATA_LOSS, 200); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_NO_ERROR, GRPC_STATUS_INTERNAL); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_PROTOCOL_ERROR, GRPC_STATUS_INTERNAL); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_INTERNAL_ERROR, GRPC_STATUS_INTERNAL); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_FLOW_CONTROL_ERROR, + const gpr_timespec before_deadline = gpr_inf_future(GPR_CLOCK_MONOTONIC); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_NO_ERROR, before_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_PROTOCOL_ERROR, before_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_INTERNAL_ERROR, before_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_FLOW_CONTROL_ERROR, before_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_SETTINGS_TIMEOUT, before_deadline, GRPC_STATUS_INTERNAL); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_SETTINGS_TIMEOUT, + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_STREAM_CLOSED, before_deadline, GRPC_STATUS_INTERNAL); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_STREAM_CLOSED, GRPC_STATUS_INTERNAL); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_FRAME_SIZE_ERROR, + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_FRAME_SIZE_ERROR, before_deadline, GRPC_STATUS_INTERNAL); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_REFUSED_STREAM, + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_REFUSED_STREAM, before_deadline, GRPC_STATUS_UNAVAILABLE); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_CANCEL, GRPC_STATUS_CANCELLED); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_COMPRESSION_ERROR, + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_CANCEL, before_deadline, + GRPC_STATUS_CANCELLED); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_COMPRESSION_ERROR, before_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_CONNECT_ERROR, before_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_ENHANCE_YOUR_CALM, before_deadline, + GRPC_STATUS_RESOURCE_EXHAUSTED); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_INADEQUATE_SECURITY, before_deadline, + GRPC_STATUS_PERMISSION_DENIED); + + const gpr_timespec after_deadline = gpr_inf_past(GPR_CLOCK_MONOTONIC); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_NO_ERROR, after_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_PROTOCOL_ERROR, after_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_INTERNAL_ERROR, after_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_FLOW_CONTROL_ERROR, after_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_SETTINGS_TIMEOUT, after_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_STREAM_CLOSED, after_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_FRAME_SIZE_ERROR, after_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_REFUSED_STREAM, after_deadline, + GRPC_STATUS_UNAVAILABLE); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_CANCEL, after_deadline, + GRPC_STATUS_DEADLINE_EXCEEDED); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_COMPRESSION_ERROR, after_deadline, + GRPC_STATUS_INTERNAL); + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_CONNECT_ERROR, after_deadline, GRPC_STATUS_INTERNAL); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_CONNECT_ERROR, GRPC_STATUS_INTERNAL); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_ENHANCE_YOUR_CALM, + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_ENHANCE_YOUR_CALM, after_deadline, GRPC_STATUS_RESOURCE_EXHAUSTED); - HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_INADEQUATE_SECURITY, + HTTP2_ERROR_TO_GRPC_STATUS(GRPC_CHTTP2_INADEQUATE_SECURITY, after_deadline, GRPC_STATUS_PERMISSION_DENIED); HTTP2_STATUS_TO_GRPC_STATUS(200, GRPC_STATUS_OK); From ad1f31ff8198b1956284f55f14c6229a12d4d513 Mon Sep 17 00:00:00 2001 From: Dan Born Date: Fri, 8 Jul 2016 13:24:49 -0700 Subject: [PATCH 0476/1039] Code comments for siblings. --- src/core/lib/iomgr/tcp_server_posix.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/src/core/lib/iomgr/tcp_server_posix.c b/src/core/lib/iomgr/tcp_server_posix.c index 5d2ebe2e7cf..684bb73e21e 100644 --- a/src/core/lib/iomgr/tcp_server_posix.c +++ b/src/core/lib/iomgr/tcp_server_posix.c @@ -90,10 +90,12 @@ struct grpc_tcp_listener { grpc_closure read_closure; grpc_closure destroyed_closure; struct grpc_tcp_listener *next; - /* When we add a listener, more than one can be created, mainly because of - IPv6. A sibling will still be in the normal list, but will be flagged - as such. Any action, such as ref or unref, will affect all of the - siblings in the list. */ + /* sibling is a linked list of all listeners for a given port. add_port and + clone_port place all new listeners in the same sibling list. A member of + the 'sibling' list is also a member of the 'next' list. The head of each + sibling list has is_sibling==0, and subsequent members of sibling lists + have is_sibling==1. is_sibling allows separate sibling lists to be + identified while iterating through 'next'. */ struct grpc_tcp_listener *sibling; int is_sibling; }; @@ -479,6 +481,9 @@ static grpc_error *add_socket_to_server(grpc_tcp_server *s, int fd, return err; } +/* Insert count new listeners after listener. Every new listener will have the + same listen address as listener (SO_REUSEPORT must be enabled). Every new + listener is a sibling of listener. */ static grpc_error *clone_port(grpc_tcp_listener *listener, unsigned count) { grpc_tcp_listener *sp = NULL; char *addr_str; @@ -504,6 +509,11 @@ static grpc_error *clone_port(grpc_tcp_listener *listener, unsigned count) { sp = gpr_malloc(sizeof(grpc_tcp_listener)); sp->next = listener->next; listener->next = sp; + /* sp (the new listener) is a sibling of 'listener' (the original + listener). */ + sp->is_sibling = 1; + sp->sibling = listener->sibling; + listener->sibling = sp; sp->server = listener->server; sp->fd = fd; sp->emfd = grpc_fd_create(fd, name); @@ -512,9 +522,6 @@ static grpc_error *clone_port(grpc_tcp_listener *listener, unsigned count) { sp->port = port; sp->port_index = listener->port_index; sp->fd_index = listener->fd_index + count - i; - listener->sibling = sp; - sp->is_sibling = 1; - sp->sibling = listener->sibling; GPR_ASSERT(sp->emfd); while (listener->server->tail->next != NULL) { listener->server->tail = listener->server->tail->next; From f0ec5b673cc525502bec1c945d514b7f021d799c Mon Sep 17 00:00:00 2001 From: Yuchen Zeng Date: Fri, 8 Jul 2016 13:40:40 -0700 Subject: [PATCH 0477/1039] Fix mac build --- BUILD | 456 +---------- CMakeLists.txt | 234 +----- Makefile | 296 +------ build.yaml | 4 +- tools/doxygen/Doxyfile.c++ | 10 +- tools/doxygen/Doxyfile.c++.internal | 227 ------ tools/run_tests/sources_and_headers.json | 8 +- vsprojects/grpc.sln | 2 +- vsprojects/vcxproj/grpc++/grpc++.vcxproj | 346 -------- .../vcxproj/grpc++/grpc++.vcxproj.filters | 765 ------------------ .../grpc++_unsecure/grpc++_unsecure.vcxproj | 346 +------- .../grpc++_unsecure.vcxproj.filters | 765 ------------------ 12 files changed, 50 insertions(+), 3409 deletions(-) diff --git a/BUILD b/BUILD index 8c17065927c..33323be229d 100644 --- a/BUILD +++ b/BUILD @@ -1235,109 +1235,6 @@ cc_library( "src/cpp/client/create_channel_internal.h", "src/cpp/server/dynamic_thread_pool.h", "src/cpp/server/thread_pool_interface.h", - "src/core/lib/channel/channel_args.h", - "src/core/lib/channel/channel_stack.h", - "src/core/lib/channel/channel_stack_builder.h", - "src/core/lib/channel/compress_filter.h", - "src/core/lib/channel/connected_channel.h", - "src/core/lib/channel/context.h", - "src/core/lib/channel/http_client_filter.h", - "src/core/lib/channel/http_server_filter.h", - "src/core/lib/compression/algorithm_metadata.h", - "src/core/lib/compression/message_compress.h", - "src/core/lib/debug/trace.h", - "src/core/lib/http/format_request.h", - "src/core/lib/http/httpcli.h", - "src/core/lib/http/parser.h", - "src/core/lib/iomgr/closure.h", - "src/core/lib/iomgr/endpoint.h", - "src/core/lib/iomgr/endpoint_pair.h", - "src/core/lib/iomgr/error.h", - "src/core/lib/iomgr/ev_epoll_linux.h", - "src/core/lib/iomgr/ev_poll_and_epoll_posix.h", - "src/core/lib/iomgr/ev_poll_posix.h", - "src/core/lib/iomgr/ev_posix.h", - "src/core/lib/iomgr/exec_ctx.h", - "src/core/lib/iomgr/executor.h", - "src/core/lib/iomgr/iocp_windows.h", - "src/core/lib/iomgr/iomgr.h", - "src/core/lib/iomgr/iomgr_internal.h", - "src/core/lib/iomgr/iomgr_posix.h", - "src/core/lib/iomgr/load_file.h", - "src/core/lib/iomgr/network_status_tracker.h", - "src/core/lib/iomgr/polling_entity.h", - "src/core/lib/iomgr/pollset.h", - "src/core/lib/iomgr/pollset_set.h", - "src/core/lib/iomgr/pollset_set_windows.h", - "src/core/lib/iomgr/pollset_windows.h", - "src/core/lib/iomgr/resolve_address.h", - "src/core/lib/iomgr/sockaddr.h", - "src/core/lib/iomgr/sockaddr_posix.h", - "src/core/lib/iomgr/sockaddr_utils.h", - "src/core/lib/iomgr/sockaddr_windows.h", - "src/core/lib/iomgr/socket_utils_posix.h", - "src/core/lib/iomgr/socket_windows.h", - "src/core/lib/iomgr/tcp_client.h", - "src/core/lib/iomgr/tcp_posix.h", - "src/core/lib/iomgr/tcp_server.h", - "src/core/lib/iomgr/tcp_windows.h", - "src/core/lib/iomgr/time_averaged_stats.h", - "src/core/lib/iomgr/timer.h", - "src/core/lib/iomgr/timer_heap.h", - "src/core/lib/iomgr/udp_server.h", - "src/core/lib/iomgr/unix_sockets_posix.h", - "src/core/lib/iomgr/wakeup_fd_pipe.h", - "src/core/lib/iomgr/wakeup_fd_posix.h", - "src/core/lib/iomgr/workqueue.h", - "src/core/lib/iomgr/workqueue_posix.h", - "src/core/lib/iomgr/workqueue_windows.h", - "src/core/lib/json/json.h", - "src/core/lib/json/json_common.h", - "src/core/lib/json/json_reader.h", - "src/core/lib/json/json_writer.h", - "src/core/lib/surface/api_trace.h", - "src/core/lib/surface/call.h", - "src/core/lib/surface/call_test_only.h", - "src/core/lib/surface/channel.h", - "src/core/lib/surface/channel_init.h", - "src/core/lib/surface/channel_stack_type.h", - "src/core/lib/surface/completion_queue.h", - "src/core/lib/surface/event_string.h", - "src/core/lib/surface/init.h", - "src/core/lib/surface/lame_client.h", - "src/core/lib/surface/server.h", - "src/core/lib/transport/byte_stream.h", - "src/core/lib/transport/connectivity_state.h", - "src/core/lib/transport/metadata.h", - "src/core/lib/transport/metadata_batch.h", - "src/core/lib/transport/static_metadata.h", - "src/core/lib/transport/transport.h", - "src/core/lib/transport/transport_impl.h", - "src/core/lib/security/context/security_context.h", - "src/core/lib/security/credentials/composite/composite_credentials.h", - "src/core/lib/security/credentials/credentials.h", - "src/core/lib/security/credentials/fake/fake_credentials.h", - "src/core/lib/security/credentials/google_default/google_default_credentials.h", - "src/core/lib/security/credentials/iam/iam_credentials.h", - "src/core/lib/security/credentials/jwt/json_token.h", - "src/core/lib/security/credentials/jwt/jwt_credentials.h", - "src/core/lib/security/credentials/jwt/jwt_verifier.h", - "src/core/lib/security/credentials/oauth2/oauth2_credentials.h", - "src/core/lib/security/credentials/plugin/plugin_credentials.h", - "src/core/lib/security/credentials/ssl/ssl_credentials.h", - "src/core/lib/security/transport/auth_filters.h", - "src/core/lib/security/transport/handshake.h", - "src/core/lib/security/transport/secure_endpoint.h", - "src/core/lib/security/transport/security_connector.h", - "src/core/lib/security/transport/tsi_error.h", - "src/core/lib/security/util/b64.h", - "src/core/lib/security/util/json_util.h", - "src/core/ext/transport/chttp2/alpn/alpn.h", - "src/core/lib/tsi/fake_transport_security.h", - "src/core/lib/tsi/ssl_transport_security.h", - "src/core/lib/tsi/ssl_types.h", - "src/core/lib/tsi/transport_security.h", - "src/core/lib/tsi/transport_security_interface.h", "src/cpp/client/secure_credentials.cc", "src/cpp/common/auth_property_iterator.cc", "src/cpp/common/secure_auth_context.cc", @@ -1370,122 +1267,6 @@ cc_library( "src/cpp/util/status.cc", "src/cpp/util/string_ref.cc", "src/cpp/util/time.cc", - "src/core/lib/channel/channel_args.c", - "src/core/lib/channel/channel_stack.c", - "src/core/lib/channel/channel_stack_builder.c", - "src/core/lib/channel/compress_filter.c", - "src/core/lib/channel/connected_channel.c", - "src/core/lib/channel/http_client_filter.c", - "src/core/lib/channel/http_server_filter.c", - "src/core/lib/compression/compression.c", - "src/core/lib/compression/message_compress.c", - "src/core/lib/debug/trace.c", - "src/core/lib/http/format_request.c", - "src/core/lib/http/httpcli.c", - "src/core/lib/http/parser.c", - "src/core/lib/iomgr/closure.c", - "src/core/lib/iomgr/endpoint.c", - "src/core/lib/iomgr/endpoint_pair_posix.c", - "src/core/lib/iomgr/endpoint_pair_windows.c", - "src/core/lib/iomgr/error.c", - "src/core/lib/iomgr/ev_epoll_linux.c", - "src/core/lib/iomgr/ev_poll_and_epoll_posix.c", - "src/core/lib/iomgr/ev_poll_posix.c", - "src/core/lib/iomgr/ev_posix.c", - "src/core/lib/iomgr/exec_ctx.c", - "src/core/lib/iomgr/executor.c", - "src/core/lib/iomgr/iocp_windows.c", - "src/core/lib/iomgr/iomgr.c", - "src/core/lib/iomgr/iomgr_posix.c", - "src/core/lib/iomgr/iomgr_windows.c", - "src/core/lib/iomgr/load_file.c", - "src/core/lib/iomgr/network_status_tracker.c", - "src/core/lib/iomgr/polling_entity.c", - "src/core/lib/iomgr/pollset_set_windows.c", - "src/core/lib/iomgr/pollset_windows.c", - "src/core/lib/iomgr/resolve_address_posix.c", - "src/core/lib/iomgr/resolve_address_windows.c", - "src/core/lib/iomgr/sockaddr_utils.c", - "src/core/lib/iomgr/socket_utils_common_posix.c", - "src/core/lib/iomgr/socket_utils_linux.c", - "src/core/lib/iomgr/socket_utils_posix.c", - "src/core/lib/iomgr/socket_windows.c", - "src/core/lib/iomgr/tcp_client_posix.c", - "src/core/lib/iomgr/tcp_client_windows.c", - "src/core/lib/iomgr/tcp_posix.c", - "src/core/lib/iomgr/tcp_server_posix.c", - "src/core/lib/iomgr/tcp_server_windows.c", - "src/core/lib/iomgr/tcp_windows.c", - "src/core/lib/iomgr/time_averaged_stats.c", - "src/core/lib/iomgr/timer.c", - "src/core/lib/iomgr/timer_heap.c", - "src/core/lib/iomgr/udp_server.c", - "src/core/lib/iomgr/unix_sockets_posix.c", - "src/core/lib/iomgr/unix_sockets_posix_noop.c", - "src/core/lib/iomgr/wakeup_fd_eventfd.c", - "src/core/lib/iomgr/wakeup_fd_nospecial.c", - "src/core/lib/iomgr/wakeup_fd_pipe.c", - "src/core/lib/iomgr/wakeup_fd_posix.c", - "src/core/lib/iomgr/workqueue_posix.c", - "src/core/lib/iomgr/workqueue_windows.c", - "src/core/lib/json/json.c", - "src/core/lib/json/json_reader.c", - "src/core/lib/json/json_string.c", - "src/core/lib/json/json_writer.c", - "src/core/lib/surface/alarm.c", - "src/core/lib/surface/api_trace.c", - "src/core/lib/surface/byte_buffer.c", - "src/core/lib/surface/byte_buffer_reader.c", - "src/core/lib/surface/call.c", - "src/core/lib/surface/call_details.c", - "src/core/lib/surface/call_log_batch.c", - "src/core/lib/surface/channel.c", - "src/core/lib/surface/channel_init.c", - "src/core/lib/surface/channel_ping.c", - "src/core/lib/surface/channel_stack_type.c", - "src/core/lib/surface/completion_queue.c", - "src/core/lib/surface/event_string.c", - "src/core/lib/surface/lame_client.c", - "src/core/lib/surface/metadata_array.c", - "src/core/lib/surface/server.c", - "src/core/lib/surface/validate_metadata.c", - "src/core/lib/surface/version.c", - "src/core/lib/transport/byte_stream.c", - "src/core/lib/transport/connectivity_state.c", - "src/core/lib/transport/metadata.c", - "src/core/lib/transport/metadata_batch.c", - "src/core/lib/transport/static_metadata.c", - "src/core/lib/transport/transport.c", - "src/core/lib/transport/transport_op_string.c", - "src/core/lib/http/httpcli_security_connector.c", - "src/core/lib/security/context/security_context.c", - "src/core/lib/security/credentials/composite/composite_credentials.c", - "src/core/lib/security/credentials/credentials.c", - "src/core/lib/security/credentials/credentials_metadata.c", - "src/core/lib/security/credentials/fake/fake_credentials.c", - "src/core/lib/security/credentials/google_default/credentials_posix.c", - "src/core/lib/security/credentials/google_default/credentials_windows.c", - "src/core/lib/security/credentials/google_default/google_default_credentials.c", - "src/core/lib/security/credentials/iam/iam_credentials.c", - "src/core/lib/security/credentials/jwt/json_token.c", - "src/core/lib/security/credentials/jwt/jwt_credentials.c", - "src/core/lib/security/credentials/jwt/jwt_verifier.c", - "src/core/lib/security/credentials/oauth2/oauth2_credentials.c", - "src/core/lib/security/credentials/plugin/plugin_credentials.c", - "src/core/lib/security/credentials/ssl/ssl_credentials.c", - "src/core/lib/security/transport/client_auth_filter.c", - "src/core/lib/security/transport/handshake.c", - "src/core/lib/security/transport/secure_endpoint.c", - "src/core/lib/security/transport/security_connector.c", - "src/core/lib/security/transport/server_auth_filter.c", - "src/core/lib/security/transport/tsi_error.c", - "src/core/lib/security/util/b64.c", - "src/core/lib/security/util/json_util.c", - "src/core/lib/surface/init_secure.c", - "src/core/ext/transport/chttp2/alpn/alpn.c", - "src/core/lib/tsi/fake_transport_security.c", - "src/core/lib/tsi/ssl_transport_security.c", - "src/core/lib/tsi/transport_security.c", "src/cpp/codegen/codegen_init.cc", ], hdrs = [ @@ -1587,14 +1368,6 @@ cc_library( "include/grpc/impl/codegen/sync_posix.h", "include/grpc/impl/codegen/sync_windows.h", "include/grpc/impl/codegen/time.h", - "include/grpc/byte_buffer.h", - "include/grpc/byte_buffer_reader.h", - "include/grpc/compression.h", - "include/grpc/grpc.h", - "include/grpc/grpc_posix.h", - "include/grpc/status.h", - "include/grpc/grpc_security.h", - "include/grpc/grpc_security_constants.h", ], includes = [ "include", @@ -1604,7 +1377,6 @@ cc_library( "//external:libssl", "//external:protobuf_clib", ":grpc", - ":gpr", ], ) @@ -1694,109 +1466,6 @@ cc_library( "src/cpp/client/create_channel_internal.h", "src/cpp/server/dynamic_thread_pool.h", "src/cpp/server/thread_pool_interface.h", - "src/core/lib/channel/channel_args.h", - "src/core/lib/channel/channel_stack.h", - "src/core/lib/channel/channel_stack_builder.h", - "src/core/lib/channel/compress_filter.h", - "src/core/lib/channel/connected_channel.h", - "src/core/lib/channel/context.h", - "src/core/lib/channel/http_client_filter.h", - "src/core/lib/channel/http_server_filter.h", - "src/core/lib/compression/algorithm_metadata.h", - "src/core/lib/compression/message_compress.h", - "src/core/lib/debug/trace.h", - "src/core/lib/http/format_request.h", - "src/core/lib/http/httpcli.h", - "src/core/lib/http/parser.h", - "src/core/lib/iomgr/closure.h", - "src/core/lib/iomgr/endpoint.h", - "src/core/lib/iomgr/endpoint_pair.h", - "src/core/lib/iomgr/error.h", - "src/core/lib/iomgr/ev_epoll_linux.h", - "src/core/lib/iomgr/ev_poll_and_epoll_posix.h", - "src/core/lib/iomgr/ev_poll_posix.h", - "src/core/lib/iomgr/ev_posix.h", - "src/core/lib/iomgr/exec_ctx.h", - "src/core/lib/iomgr/executor.h", - "src/core/lib/iomgr/iocp_windows.h", - "src/core/lib/iomgr/iomgr.h", - "src/core/lib/iomgr/iomgr_internal.h", - "src/core/lib/iomgr/iomgr_posix.h", - "src/core/lib/iomgr/load_file.h", - "src/core/lib/iomgr/network_status_tracker.h", - "src/core/lib/iomgr/polling_entity.h", - "src/core/lib/iomgr/pollset.h", - "src/core/lib/iomgr/pollset_set.h", - "src/core/lib/iomgr/pollset_set_windows.h", - "src/core/lib/iomgr/pollset_windows.h", - "src/core/lib/iomgr/resolve_address.h", - "src/core/lib/iomgr/sockaddr.h", - "src/core/lib/iomgr/sockaddr_posix.h", - "src/core/lib/iomgr/sockaddr_utils.h", - "src/core/lib/iomgr/sockaddr_windows.h", - "src/core/lib/iomgr/socket_utils_posix.h", - "src/core/lib/iomgr/socket_windows.h", - "src/core/lib/iomgr/tcp_client.h", - "src/core/lib/iomgr/tcp_posix.h", - "src/core/lib/iomgr/tcp_server.h", - "src/core/lib/iomgr/tcp_windows.h", - "src/core/lib/iomgr/time_averaged_stats.h", - "src/core/lib/iomgr/timer.h", - "src/core/lib/iomgr/timer_heap.h", - "src/core/lib/iomgr/udp_server.h", - "src/core/lib/iomgr/unix_sockets_posix.h", - "src/core/lib/iomgr/wakeup_fd_pipe.h", - "src/core/lib/iomgr/wakeup_fd_posix.h", - "src/core/lib/iomgr/workqueue.h", - "src/core/lib/iomgr/workqueue_posix.h", - "src/core/lib/iomgr/workqueue_windows.h", - "src/core/lib/json/json.h", - "src/core/lib/json/json_common.h", - "src/core/lib/json/json_reader.h", - "src/core/lib/json/json_writer.h", - "src/core/lib/surface/api_trace.h", - "src/core/lib/surface/call.h", - "src/core/lib/surface/call_test_only.h", - "src/core/lib/surface/channel.h", - "src/core/lib/surface/channel_init.h", - "src/core/lib/surface/channel_stack_type.h", - "src/core/lib/surface/completion_queue.h", - "src/core/lib/surface/event_string.h", - "src/core/lib/surface/init.h", - "src/core/lib/surface/lame_client.h", - "src/core/lib/surface/server.h", - "src/core/lib/transport/byte_stream.h", - "src/core/lib/transport/connectivity_state.h", - "src/core/lib/transport/metadata.h", - "src/core/lib/transport/metadata_batch.h", - "src/core/lib/transport/static_metadata.h", - "src/core/lib/transport/transport.h", - "src/core/lib/transport/transport_impl.h", - "src/core/lib/security/context/security_context.h", - "src/core/lib/security/credentials/composite/composite_credentials.h", - "src/core/lib/security/credentials/credentials.h", - "src/core/lib/security/credentials/fake/fake_credentials.h", - "src/core/lib/security/credentials/google_default/google_default_credentials.h", - "src/core/lib/security/credentials/iam/iam_credentials.h", - "src/core/lib/security/credentials/jwt/json_token.h", - "src/core/lib/security/credentials/jwt/jwt_credentials.h", - "src/core/lib/security/credentials/jwt/jwt_verifier.h", - "src/core/lib/security/credentials/oauth2/oauth2_credentials.h", - "src/core/lib/security/credentials/plugin/plugin_credentials.h", - "src/core/lib/security/credentials/ssl/ssl_credentials.h", - "src/core/lib/security/transport/auth_filters.h", - "src/core/lib/security/transport/handshake.h", - "src/core/lib/security/transport/secure_endpoint.h", - "src/core/lib/security/transport/security_connector.h", - "src/core/lib/security/transport/tsi_error.h", - "src/core/lib/security/util/b64.h", - "src/core/lib/security/util/json_util.h", - "src/core/ext/transport/chttp2/alpn/alpn.h", - "src/core/lib/tsi/fake_transport_security.h", - "src/core/lib/tsi/ssl_transport_security.h", - "src/core/lib/tsi/ssl_types.h", - "src/core/lib/tsi/transport_security.h", - "src/core/lib/tsi/transport_security_interface.h", "src/cpp/common/insecure_create_auth_context.cc", "src/cpp/client/channel.cc", "src/cpp/client/client_context.cc", @@ -1824,122 +1493,6 @@ cc_library( "src/cpp/util/status.cc", "src/cpp/util/string_ref.cc", "src/cpp/util/time.cc", - "src/core/lib/channel/channel_args.c", - "src/core/lib/channel/channel_stack.c", - "src/core/lib/channel/channel_stack_builder.c", - "src/core/lib/channel/compress_filter.c", - "src/core/lib/channel/connected_channel.c", - "src/core/lib/channel/http_client_filter.c", - "src/core/lib/channel/http_server_filter.c", - "src/core/lib/compression/compression.c", - "src/core/lib/compression/message_compress.c", - "src/core/lib/debug/trace.c", - "src/core/lib/http/format_request.c", - "src/core/lib/http/httpcli.c", - "src/core/lib/http/parser.c", - "src/core/lib/iomgr/closure.c", - "src/core/lib/iomgr/endpoint.c", - "src/core/lib/iomgr/endpoint_pair_posix.c", - "src/core/lib/iomgr/endpoint_pair_windows.c", - "src/core/lib/iomgr/error.c", - "src/core/lib/iomgr/ev_epoll_linux.c", - "src/core/lib/iomgr/ev_poll_and_epoll_posix.c", - "src/core/lib/iomgr/ev_poll_posix.c", - "src/core/lib/iomgr/ev_posix.c", - "src/core/lib/iomgr/exec_ctx.c", - "src/core/lib/iomgr/executor.c", - "src/core/lib/iomgr/iocp_windows.c", - "src/core/lib/iomgr/iomgr.c", - "src/core/lib/iomgr/iomgr_posix.c", - "src/core/lib/iomgr/iomgr_windows.c", - "src/core/lib/iomgr/load_file.c", - "src/core/lib/iomgr/network_status_tracker.c", - "src/core/lib/iomgr/polling_entity.c", - "src/core/lib/iomgr/pollset_set_windows.c", - "src/core/lib/iomgr/pollset_windows.c", - "src/core/lib/iomgr/resolve_address_posix.c", - "src/core/lib/iomgr/resolve_address_windows.c", - "src/core/lib/iomgr/sockaddr_utils.c", - "src/core/lib/iomgr/socket_utils_common_posix.c", - "src/core/lib/iomgr/socket_utils_linux.c", - "src/core/lib/iomgr/socket_utils_posix.c", - "src/core/lib/iomgr/socket_windows.c", - "src/core/lib/iomgr/tcp_client_posix.c", - "src/core/lib/iomgr/tcp_client_windows.c", - "src/core/lib/iomgr/tcp_posix.c", - "src/core/lib/iomgr/tcp_server_posix.c", - "src/core/lib/iomgr/tcp_server_windows.c", - "src/core/lib/iomgr/tcp_windows.c", - "src/core/lib/iomgr/time_averaged_stats.c", - "src/core/lib/iomgr/timer.c", - "src/core/lib/iomgr/timer_heap.c", - "src/core/lib/iomgr/udp_server.c", - "src/core/lib/iomgr/unix_sockets_posix.c", - "src/core/lib/iomgr/unix_sockets_posix_noop.c", - "src/core/lib/iomgr/wakeup_fd_eventfd.c", - "src/core/lib/iomgr/wakeup_fd_nospecial.c", - "src/core/lib/iomgr/wakeup_fd_pipe.c", - "src/core/lib/iomgr/wakeup_fd_posix.c", - "src/core/lib/iomgr/workqueue_posix.c", - "src/core/lib/iomgr/workqueue_windows.c", - "src/core/lib/json/json.c", - "src/core/lib/json/json_reader.c", - "src/core/lib/json/json_string.c", - "src/core/lib/json/json_writer.c", - "src/core/lib/surface/alarm.c", - "src/core/lib/surface/api_trace.c", - "src/core/lib/surface/byte_buffer.c", - "src/core/lib/surface/byte_buffer_reader.c", - "src/core/lib/surface/call.c", - "src/core/lib/surface/call_details.c", - "src/core/lib/surface/call_log_batch.c", - "src/core/lib/surface/channel.c", - "src/core/lib/surface/channel_init.c", - "src/core/lib/surface/channel_ping.c", - "src/core/lib/surface/channel_stack_type.c", - "src/core/lib/surface/completion_queue.c", - "src/core/lib/surface/event_string.c", - "src/core/lib/surface/lame_client.c", - "src/core/lib/surface/metadata_array.c", - "src/core/lib/surface/server.c", - "src/core/lib/surface/validate_metadata.c", - "src/core/lib/surface/version.c", - "src/core/lib/transport/byte_stream.c", - "src/core/lib/transport/connectivity_state.c", - "src/core/lib/transport/metadata.c", - "src/core/lib/transport/metadata_batch.c", - "src/core/lib/transport/static_metadata.c", - "src/core/lib/transport/transport.c", - "src/core/lib/transport/transport_op_string.c", - "src/core/lib/http/httpcli_security_connector.c", - "src/core/lib/security/context/security_context.c", - "src/core/lib/security/credentials/composite/composite_credentials.c", - "src/core/lib/security/credentials/credentials.c", - "src/core/lib/security/credentials/credentials_metadata.c", - "src/core/lib/security/credentials/fake/fake_credentials.c", - "src/core/lib/security/credentials/google_default/credentials_posix.c", - "src/core/lib/security/credentials/google_default/credentials_windows.c", - "src/core/lib/security/credentials/google_default/google_default_credentials.c", - "src/core/lib/security/credentials/iam/iam_credentials.c", - "src/core/lib/security/credentials/jwt/json_token.c", - "src/core/lib/security/credentials/jwt/jwt_credentials.c", - "src/core/lib/security/credentials/jwt/jwt_verifier.c", - "src/core/lib/security/credentials/oauth2/oauth2_credentials.c", - "src/core/lib/security/credentials/plugin/plugin_credentials.c", - "src/core/lib/security/credentials/ssl/ssl_credentials.c", - "src/core/lib/security/transport/client_auth_filter.c", - "src/core/lib/security/transport/handshake.c", - "src/core/lib/security/transport/secure_endpoint.c", - "src/core/lib/security/transport/security_connector.c", - "src/core/lib/security/transport/server_auth_filter.c", - "src/core/lib/security/transport/tsi_error.c", - "src/core/lib/security/util/b64.c", - "src/core/lib/security/util/json_util.c", - "src/core/lib/surface/init_secure.c", - "src/core/ext/transport/chttp2/alpn/alpn.c", - "src/core/lib/tsi/fake_transport_security.c", - "src/core/lib/tsi/ssl_transport_security.c", - "src/core/lib/tsi/transport_security.c", "src/cpp/codegen/codegen_init.cc", ], hdrs = [ @@ -2041,14 +1594,6 @@ cc_library( "include/grpc/impl/codegen/sync_posix.h", "include/grpc/impl/codegen/sync_windows.h", "include/grpc/impl/codegen/time.h", - "include/grpc/byte_buffer.h", - "include/grpc/byte_buffer_reader.h", - "include/grpc/compression.h", - "include/grpc/grpc.h", - "include/grpc/grpc_posix.h", - "include/grpc/status.h", - "include/grpc/grpc_security.h", - "include/grpc/grpc_security_constants.h", ], includes = [ "include", @@ -2058,6 +1603,7 @@ cc_library( "//external:protobuf_clib", ":gpr", ":grpc_unsecure", + ":grpc", ], ) diff --git a/CMakeLists.txt b/CMakeLists.txt index 3dcd1eb23d1..9910bd330cc 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -718,122 +718,6 @@ add_library(grpc++ src/cpp/util/status.cc src/cpp/util/string_ref.cc src/cpp/util/time.cc - src/core/lib/channel/channel_args.c - src/core/lib/channel/channel_stack.c - src/core/lib/channel/channel_stack_builder.c - src/core/lib/channel/compress_filter.c - src/core/lib/channel/connected_channel.c - src/core/lib/channel/http_client_filter.c - src/core/lib/channel/http_server_filter.c - src/core/lib/compression/compression.c - src/core/lib/compression/message_compress.c - src/core/lib/debug/trace.c - src/core/lib/http/format_request.c - src/core/lib/http/httpcli.c - src/core/lib/http/parser.c - src/core/lib/iomgr/closure.c - src/core/lib/iomgr/endpoint.c - src/core/lib/iomgr/endpoint_pair_posix.c - src/core/lib/iomgr/endpoint_pair_windows.c - src/core/lib/iomgr/error.c - src/core/lib/iomgr/ev_epoll_linux.c - src/core/lib/iomgr/ev_poll_and_epoll_posix.c - src/core/lib/iomgr/ev_poll_posix.c - src/core/lib/iomgr/ev_posix.c - src/core/lib/iomgr/exec_ctx.c - src/core/lib/iomgr/executor.c - src/core/lib/iomgr/iocp_windows.c - src/core/lib/iomgr/iomgr.c - src/core/lib/iomgr/iomgr_posix.c - src/core/lib/iomgr/iomgr_windows.c - src/core/lib/iomgr/load_file.c - src/core/lib/iomgr/network_status_tracker.c - src/core/lib/iomgr/polling_entity.c - src/core/lib/iomgr/pollset_set_windows.c - src/core/lib/iomgr/pollset_windows.c - src/core/lib/iomgr/resolve_address_posix.c - src/core/lib/iomgr/resolve_address_windows.c - src/core/lib/iomgr/sockaddr_utils.c - src/core/lib/iomgr/socket_utils_common_posix.c - src/core/lib/iomgr/socket_utils_linux.c - src/core/lib/iomgr/socket_utils_posix.c - src/core/lib/iomgr/socket_windows.c - src/core/lib/iomgr/tcp_client_posix.c - src/core/lib/iomgr/tcp_client_windows.c - src/core/lib/iomgr/tcp_posix.c - src/core/lib/iomgr/tcp_server_posix.c - src/core/lib/iomgr/tcp_server_windows.c - src/core/lib/iomgr/tcp_windows.c - src/core/lib/iomgr/time_averaged_stats.c - src/core/lib/iomgr/timer.c - src/core/lib/iomgr/timer_heap.c - src/core/lib/iomgr/udp_server.c - src/core/lib/iomgr/unix_sockets_posix.c - src/core/lib/iomgr/unix_sockets_posix_noop.c - src/core/lib/iomgr/wakeup_fd_eventfd.c - src/core/lib/iomgr/wakeup_fd_nospecial.c - src/core/lib/iomgr/wakeup_fd_pipe.c - src/core/lib/iomgr/wakeup_fd_posix.c - src/core/lib/iomgr/workqueue_posix.c - src/core/lib/iomgr/workqueue_windows.c - src/core/lib/json/json.c - src/core/lib/json/json_reader.c - src/core/lib/json/json_string.c - src/core/lib/json/json_writer.c - src/core/lib/surface/alarm.c - src/core/lib/surface/api_trace.c - src/core/lib/surface/byte_buffer.c - src/core/lib/surface/byte_buffer_reader.c - src/core/lib/surface/call.c - src/core/lib/surface/call_details.c - src/core/lib/surface/call_log_batch.c - src/core/lib/surface/channel.c - src/core/lib/surface/channel_init.c - src/core/lib/surface/channel_ping.c - src/core/lib/surface/channel_stack_type.c - src/core/lib/surface/completion_queue.c - src/core/lib/surface/event_string.c - src/core/lib/surface/lame_client.c - src/core/lib/surface/metadata_array.c - src/core/lib/surface/server.c - src/core/lib/surface/validate_metadata.c - src/core/lib/surface/version.c - src/core/lib/transport/byte_stream.c - src/core/lib/transport/connectivity_state.c - src/core/lib/transport/metadata.c - src/core/lib/transport/metadata_batch.c - src/core/lib/transport/static_metadata.c - src/core/lib/transport/transport.c - src/core/lib/transport/transport_op_string.c - src/core/lib/http/httpcli_security_connector.c - src/core/lib/security/context/security_context.c - src/core/lib/security/credentials/composite/composite_credentials.c - src/core/lib/security/credentials/credentials.c - src/core/lib/security/credentials/credentials_metadata.c - src/core/lib/security/credentials/fake/fake_credentials.c - src/core/lib/security/credentials/google_default/credentials_posix.c - src/core/lib/security/credentials/google_default/credentials_windows.c - src/core/lib/security/credentials/google_default/google_default_credentials.c - src/core/lib/security/credentials/iam/iam_credentials.c - src/core/lib/security/credentials/jwt/json_token.c - src/core/lib/security/credentials/jwt/jwt_credentials.c - src/core/lib/security/credentials/jwt/jwt_verifier.c - src/core/lib/security/credentials/oauth2/oauth2_credentials.c - src/core/lib/security/credentials/plugin/plugin_credentials.c - src/core/lib/security/credentials/ssl/ssl_credentials.c - src/core/lib/security/transport/client_auth_filter.c - src/core/lib/security/transport/handshake.c - src/core/lib/security/transport/secure_endpoint.c - src/core/lib/security/transport/security_connector.c - src/core/lib/security/transport/server_auth_filter.c - src/core/lib/security/transport/tsi_error.c - src/core/lib/security/util/b64.c - src/core/lib/security/util/json_util.c - src/core/lib/surface/init_secure.c - src/core/ext/transport/chttp2/alpn/alpn.c - src/core/lib/tsi/fake_transport_security.c - src/core/lib/tsi/ssl_transport_security.c - src/core/lib/tsi/transport_security.c src/cpp/codegen/codegen_init.cc ) @@ -850,7 +734,6 @@ target_link_libraries(grpc++ ssl libprotobuf grpc - gpr ) @@ -903,122 +786,6 @@ add_library(grpc++_unsecure src/cpp/util/status.cc src/cpp/util/string_ref.cc src/cpp/util/time.cc - src/core/lib/channel/channel_args.c - src/core/lib/channel/channel_stack.c - src/core/lib/channel/channel_stack_builder.c - src/core/lib/channel/compress_filter.c - src/core/lib/channel/connected_channel.c - src/core/lib/channel/http_client_filter.c - src/core/lib/channel/http_server_filter.c - src/core/lib/compression/compression.c - src/core/lib/compression/message_compress.c - src/core/lib/debug/trace.c - src/core/lib/http/format_request.c - src/core/lib/http/httpcli.c - src/core/lib/http/parser.c - src/core/lib/iomgr/closure.c - src/core/lib/iomgr/endpoint.c - src/core/lib/iomgr/endpoint_pair_posix.c - src/core/lib/iomgr/endpoint_pair_windows.c - src/core/lib/iomgr/error.c - src/core/lib/iomgr/ev_epoll_linux.c - src/core/lib/iomgr/ev_poll_and_epoll_posix.c - src/core/lib/iomgr/ev_poll_posix.c - src/core/lib/iomgr/ev_posix.c - src/core/lib/iomgr/exec_ctx.c - src/core/lib/iomgr/executor.c - src/core/lib/iomgr/iocp_windows.c - src/core/lib/iomgr/iomgr.c - src/core/lib/iomgr/iomgr_posix.c - src/core/lib/iomgr/iomgr_windows.c - src/core/lib/iomgr/load_file.c - src/core/lib/iomgr/network_status_tracker.c - src/core/lib/iomgr/polling_entity.c - src/core/lib/iomgr/pollset_set_windows.c - src/core/lib/iomgr/pollset_windows.c - src/core/lib/iomgr/resolve_address_posix.c - src/core/lib/iomgr/resolve_address_windows.c - src/core/lib/iomgr/sockaddr_utils.c - src/core/lib/iomgr/socket_utils_common_posix.c - src/core/lib/iomgr/socket_utils_linux.c - src/core/lib/iomgr/socket_utils_posix.c - src/core/lib/iomgr/socket_windows.c - src/core/lib/iomgr/tcp_client_posix.c - src/core/lib/iomgr/tcp_client_windows.c - src/core/lib/iomgr/tcp_posix.c - src/core/lib/iomgr/tcp_server_posix.c - src/core/lib/iomgr/tcp_server_windows.c - src/core/lib/iomgr/tcp_windows.c - src/core/lib/iomgr/time_averaged_stats.c - src/core/lib/iomgr/timer.c - src/core/lib/iomgr/timer_heap.c - src/core/lib/iomgr/udp_server.c - src/core/lib/iomgr/unix_sockets_posix.c - src/core/lib/iomgr/unix_sockets_posix_noop.c - src/core/lib/iomgr/wakeup_fd_eventfd.c - src/core/lib/iomgr/wakeup_fd_nospecial.c - src/core/lib/iomgr/wakeup_fd_pipe.c - src/core/lib/iomgr/wakeup_fd_posix.c - src/core/lib/iomgr/workqueue_posix.c - src/core/lib/iomgr/workqueue_windows.c - src/core/lib/json/json.c - src/core/lib/json/json_reader.c - src/core/lib/json/json_string.c - src/core/lib/json/json_writer.c - src/core/lib/surface/alarm.c - src/core/lib/surface/api_trace.c - src/core/lib/surface/byte_buffer.c - src/core/lib/surface/byte_buffer_reader.c - src/core/lib/surface/call.c - src/core/lib/surface/call_details.c - src/core/lib/surface/call_log_batch.c - src/core/lib/surface/channel.c - src/core/lib/surface/channel_init.c - src/core/lib/surface/channel_ping.c - src/core/lib/surface/channel_stack_type.c - src/core/lib/surface/completion_queue.c - src/core/lib/surface/event_string.c - src/core/lib/surface/lame_client.c - src/core/lib/surface/metadata_array.c - src/core/lib/surface/server.c - src/core/lib/surface/validate_metadata.c - src/core/lib/surface/version.c - src/core/lib/transport/byte_stream.c - src/core/lib/transport/connectivity_state.c - src/core/lib/transport/metadata.c - src/core/lib/transport/metadata_batch.c - src/core/lib/transport/static_metadata.c - src/core/lib/transport/transport.c - src/core/lib/transport/transport_op_string.c - src/core/lib/http/httpcli_security_connector.c - src/core/lib/security/context/security_context.c - src/core/lib/security/credentials/composite/composite_credentials.c - src/core/lib/security/credentials/credentials.c - src/core/lib/security/credentials/credentials_metadata.c - src/core/lib/security/credentials/fake/fake_credentials.c - src/core/lib/security/credentials/google_default/credentials_posix.c - src/core/lib/security/credentials/google_default/credentials_windows.c - src/core/lib/security/credentials/google_default/google_default_credentials.c - src/core/lib/security/credentials/iam/iam_credentials.c - src/core/lib/security/credentials/jwt/json_token.c - src/core/lib/security/credentials/jwt/jwt_credentials.c - src/core/lib/security/credentials/jwt/jwt_verifier.c - src/core/lib/security/credentials/oauth2/oauth2_credentials.c - src/core/lib/security/credentials/plugin/plugin_credentials.c - src/core/lib/security/credentials/ssl/ssl_credentials.c - src/core/lib/security/transport/client_auth_filter.c - src/core/lib/security/transport/handshake.c - src/core/lib/security/transport/secure_endpoint.c - src/core/lib/security/transport/security_connector.c - src/core/lib/security/transport/server_auth_filter.c - src/core/lib/security/transport/tsi_error.c - src/core/lib/security/util/b64.c - src/core/lib/security/util/json_util.c - src/core/lib/surface/init_secure.c - src/core/ext/transport/chttp2/alpn/alpn.c - src/core/lib/tsi/fake_transport_security.c - src/core/lib/tsi/ssl_transport_security.c - src/core/lib/tsi/transport_security.c src/cpp/codegen/codegen_init.cc ) @@ -1035,6 +802,7 @@ target_link_libraries(grpc++_unsecure libprotobuf gpr grpc_unsecure + grpc ) diff --git a/Makefile b/Makefile index 51f5c5e44c3..db479c95849 100644 --- a/Makefile +++ b/Makefile @@ -3421,122 +3421,6 @@ LIBGRPC++_SRC = \ src/cpp/util/status.cc \ src/cpp/util/string_ref.cc \ src/cpp/util/time.cc \ - src/core/lib/channel/channel_args.c \ - src/core/lib/channel/channel_stack.c \ - src/core/lib/channel/channel_stack_builder.c \ - src/core/lib/channel/compress_filter.c \ - src/core/lib/channel/connected_channel.c \ - src/core/lib/channel/http_client_filter.c \ - src/core/lib/channel/http_server_filter.c \ - src/core/lib/compression/compression.c \ - src/core/lib/compression/message_compress.c \ - src/core/lib/debug/trace.c \ - src/core/lib/http/format_request.c \ - src/core/lib/http/httpcli.c \ - src/core/lib/http/parser.c \ - src/core/lib/iomgr/closure.c \ - src/core/lib/iomgr/endpoint.c \ - src/core/lib/iomgr/endpoint_pair_posix.c \ - src/core/lib/iomgr/endpoint_pair_windows.c \ - src/core/lib/iomgr/error.c \ - src/core/lib/iomgr/ev_epoll_linux.c \ - src/core/lib/iomgr/ev_poll_and_epoll_posix.c \ - src/core/lib/iomgr/ev_poll_posix.c \ - src/core/lib/iomgr/ev_posix.c \ - src/core/lib/iomgr/exec_ctx.c \ - src/core/lib/iomgr/executor.c \ - src/core/lib/iomgr/iocp_windows.c \ - src/core/lib/iomgr/iomgr.c \ - src/core/lib/iomgr/iomgr_posix.c \ - src/core/lib/iomgr/iomgr_windows.c \ - src/core/lib/iomgr/load_file.c \ - src/core/lib/iomgr/network_status_tracker.c \ - src/core/lib/iomgr/polling_entity.c \ - src/core/lib/iomgr/pollset_set_windows.c \ - src/core/lib/iomgr/pollset_windows.c \ - src/core/lib/iomgr/resolve_address_posix.c \ - src/core/lib/iomgr/resolve_address_windows.c \ - src/core/lib/iomgr/sockaddr_utils.c \ - src/core/lib/iomgr/socket_utils_common_posix.c \ - src/core/lib/iomgr/socket_utils_linux.c \ - src/core/lib/iomgr/socket_utils_posix.c \ - src/core/lib/iomgr/socket_windows.c \ - src/core/lib/iomgr/tcp_client_posix.c \ - src/core/lib/iomgr/tcp_client_windows.c \ - src/core/lib/iomgr/tcp_posix.c \ - src/core/lib/iomgr/tcp_server_posix.c \ - src/core/lib/iomgr/tcp_server_windows.c \ - src/core/lib/iomgr/tcp_windows.c \ - src/core/lib/iomgr/time_averaged_stats.c \ - src/core/lib/iomgr/timer.c \ - src/core/lib/iomgr/timer_heap.c \ - src/core/lib/iomgr/udp_server.c \ - src/core/lib/iomgr/unix_sockets_posix.c \ - src/core/lib/iomgr/unix_sockets_posix_noop.c \ - src/core/lib/iomgr/wakeup_fd_eventfd.c \ - src/core/lib/iomgr/wakeup_fd_nospecial.c \ - src/core/lib/iomgr/wakeup_fd_pipe.c \ - src/core/lib/iomgr/wakeup_fd_posix.c \ - src/core/lib/iomgr/workqueue_posix.c \ - src/core/lib/iomgr/workqueue_windows.c \ - src/core/lib/json/json.c \ - src/core/lib/json/json_reader.c \ - src/core/lib/json/json_string.c \ - src/core/lib/json/json_writer.c \ - src/core/lib/surface/alarm.c \ - src/core/lib/surface/api_trace.c \ - src/core/lib/surface/byte_buffer.c \ - src/core/lib/surface/byte_buffer_reader.c \ - src/core/lib/surface/call.c \ - src/core/lib/surface/call_details.c \ - src/core/lib/surface/call_log_batch.c \ - src/core/lib/surface/channel.c \ - src/core/lib/surface/channel_init.c \ - src/core/lib/surface/channel_ping.c \ - src/core/lib/surface/channel_stack_type.c \ - src/core/lib/surface/completion_queue.c \ - src/core/lib/surface/event_string.c \ - src/core/lib/surface/lame_client.c \ - src/core/lib/surface/metadata_array.c \ - src/core/lib/surface/server.c \ - src/core/lib/surface/validate_metadata.c \ - src/core/lib/surface/version.c \ - src/core/lib/transport/byte_stream.c \ - src/core/lib/transport/connectivity_state.c \ - src/core/lib/transport/metadata.c \ - src/core/lib/transport/metadata_batch.c \ - src/core/lib/transport/static_metadata.c \ - src/core/lib/transport/transport.c \ - src/core/lib/transport/transport_op_string.c \ - src/core/lib/http/httpcli_security_connector.c \ - src/core/lib/security/context/security_context.c \ - src/core/lib/security/credentials/composite/composite_credentials.c \ - src/core/lib/security/credentials/credentials.c \ - src/core/lib/security/credentials/credentials_metadata.c \ - src/core/lib/security/credentials/fake/fake_credentials.c \ - src/core/lib/security/credentials/google_default/credentials_posix.c \ - src/core/lib/security/credentials/google_default/credentials_windows.c \ - src/core/lib/security/credentials/google_default/google_default_credentials.c \ - src/core/lib/security/credentials/iam/iam_credentials.c \ - src/core/lib/security/credentials/jwt/json_token.c \ - src/core/lib/security/credentials/jwt/jwt_credentials.c \ - src/core/lib/security/credentials/jwt/jwt_verifier.c \ - src/core/lib/security/credentials/oauth2/oauth2_credentials.c \ - src/core/lib/security/credentials/plugin/plugin_credentials.c \ - src/core/lib/security/credentials/ssl/ssl_credentials.c \ - src/core/lib/security/transport/client_auth_filter.c \ - src/core/lib/security/transport/handshake.c \ - src/core/lib/security/transport/secure_endpoint.c \ - src/core/lib/security/transport/security_connector.c \ - src/core/lib/security/transport/server_auth_filter.c \ - src/core/lib/security/transport/tsi_error.c \ - src/core/lib/security/util/b64.c \ - src/core/lib/security/util/json_util.c \ - src/core/lib/surface/init_secure.c \ - src/core/ext/transport/chttp2/alpn/alpn.c \ - src/core/lib/tsi/fake_transport_security.c \ - src/core/lib/tsi/ssl_transport_security.c \ - src/core/lib/tsi/transport_security.c \ src/cpp/codegen/codegen_init.cc \ PUBLIC_HEADERS_CXX += \ @@ -3638,14 +3522,6 @@ PUBLIC_HEADERS_CXX += \ include/grpc/impl/codegen/sync_posix.h \ include/grpc/impl/codegen/sync_windows.h \ include/grpc/impl/codegen/time.h \ - include/grpc/byte_buffer.h \ - include/grpc/byte_buffer_reader.h \ - include/grpc/compression.h \ - include/grpc/grpc.h \ - include/grpc/grpc_posix.h \ - include/grpc/status.h \ - include/grpc/grpc_security.h \ - include/grpc/grpc_security_constants.h \ LIBGRPC++_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(LIBGRPC++_SRC)))) @@ -3682,18 +3558,18 @@ endif ifeq ($(SYSTEM),MINGW32) -$(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/grpc.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/gpr.$(SHARED_EXT) $(OPENSSL_DEP) +$(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/grpc.$(SHARED_EXT) $(OPENSSL_DEP) $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared grpc++.def -Wl,--output-def=$(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).def -Wl,--out-implib=$(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION)-dll.a -o $(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc-imp -lgpr-imp + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared grpc++.def -Wl,--output-def=$(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).def -Wl,--out-implib=$(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION)-dll.a -o $(LIBDIR)/$(CONFIG)/grpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc-imp else -$(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/libgrpc.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgpr.$(SHARED_EXT) $(OPENSSL_DEP) +$(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/libgrpc.$(SHARED_EXT) $(OPENSSL_DEP) $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` ifeq ($(SYSTEM),Darwin) - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -install_name $(SHARED_PREFIX)grpc++$(SHARED_VERSION).$(SHARED_EXT) -dynamiclib -o $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc -lgpr + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -install_name $(SHARED_PREFIX)grpc++$(SHARED_VERSION).$(SHARED_EXT) -dynamiclib -o $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc else - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared -Wl,-soname,libgrpc++.so.0 -o $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc -lgpr + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared -Wl,-soname,libgrpc++.so.0 -o $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgrpc $(Q) ln -sf $(SHARED_PREFIX)grpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).so.0 $(Q) ln -sf $(SHARED_PREFIX)grpc++$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc++$(SHARED_VERSION).so endif @@ -4032,122 +3908,6 @@ LIBGRPC++_UNSECURE_SRC = \ src/cpp/util/status.cc \ src/cpp/util/string_ref.cc \ src/cpp/util/time.cc \ - src/core/lib/channel/channel_args.c \ - src/core/lib/channel/channel_stack.c \ - src/core/lib/channel/channel_stack_builder.c \ - src/core/lib/channel/compress_filter.c \ - src/core/lib/channel/connected_channel.c \ - src/core/lib/channel/http_client_filter.c \ - src/core/lib/channel/http_server_filter.c \ - src/core/lib/compression/compression.c \ - src/core/lib/compression/message_compress.c \ - src/core/lib/debug/trace.c \ - src/core/lib/http/format_request.c \ - src/core/lib/http/httpcli.c \ - src/core/lib/http/parser.c \ - src/core/lib/iomgr/closure.c \ - src/core/lib/iomgr/endpoint.c \ - src/core/lib/iomgr/endpoint_pair_posix.c \ - src/core/lib/iomgr/endpoint_pair_windows.c \ - src/core/lib/iomgr/error.c \ - src/core/lib/iomgr/ev_epoll_linux.c \ - src/core/lib/iomgr/ev_poll_and_epoll_posix.c \ - src/core/lib/iomgr/ev_poll_posix.c \ - src/core/lib/iomgr/ev_posix.c \ - src/core/lib/iomgr/exec_ctx.c \ - src/core/lib/iomgr/executor.c \ - src/core/lib/iomgr/iocp_windows.c \ - src/core/lib/iomgr/iomgr.c \ - src/core/lib/iomgr/iomgr_posix.c \ - src/core/lib/iomgr/iomgr_windows.c \ - src/core/lib/iomgr/load_file.c \ - src/core/lib/iomgr/network_status_tracker.c \ - src/core/lib/iomgr/polling_entity.c \ - src/core/lib/iomgr/pollset_set_windows.c \ - src/core/lib/iomgr/pollset_windows.c \ - src/core/lib/iomgr/resolve_address_posix.c \ - src/core/lib/iomgr/resolve_address_windows.c \ - src/core/lib/iomgr/sockaddr_utils.c \ - src/core/lib/iomgr/socket_utils_common_posix.c \ - src/core/lib/iomgr/socket_utils_linux.c \ - src/core/lib/iomgr/socket_utils_posix.c \ - src/core/lib/iomgr/socket_windows.c \ - src/core/lib/iomgr/tcp_client_posix.c \ - src/core/lib/iomgr/tcp_client_windows.c \ - src/core/lib/iomgr/tcp_posix.c \ - src/core/lib/iomgr/tcp_server_posix.c \ - src/core/lib/iomgr/tcp_server_windows.c \ - src/core/lib/iomgr/tcp_windows.c \ - src/core/lib/iomgr/time_averaged_stats.c \ - src/core/lib/iomgr/timer.c \ - src/core/lib/iomgr/timer_heap.c \ - src/core/lib/iomgr/udp_server.c \ - src/core/lib/iomgr/unix_sockets_posix.c \ - src/core/lib/iomgr/unix_sockets_posix_noop.c \ - src/core/lib/iomgr/wakeup_fd_eventfd.c \ - src/core/lib/iomgr/wakeup_fd_nospecial.c \ - src/core/lib/iomgr/wakeup_fd_pipe.c \ - src/core/lib/iomgr/wakeup_fd_posix.c \ - src/core/lib/iomgr/workqueue_posix.c \ - src/core/lib/iomgr/workqueue_windows.c \ - src/core/lib/json/json.c \ - src/core/lib/json/json_reader.c \ - src/core/lib/json/json_string.c \ - src/core/lib/json/json_writer.c \ - src/core/lib/surface/alarm.c \ - src/core/lib/surface/api_trace.c \ - src/core/lib/surface/byte_buffer.c \ - src/core/lib/surface/byte_buffer_reader.c \ - src/core/lib/surface/call.c \ - src/core/lib/surface/call_details.c \ - src/core/lib/surface/call_log_batch.c \ - src/core/lib/surface/channel.c \ - src/core/lib/surface/channel_init.c \ - src/core/lib/surface/channel_ping.c \ - src/core/lib/surface/channel_stack_type.c \ - src/core/lib/surface/completion_queue.c \ - src/core/lib/surface/event_string.c \ - src/core/lib/surface/lame_client.c \ - src/core/lib/surface/metadata_array.c \ - src/core/lib/surface/server.c \ - src/core/lib/surface/validate_metadata.c \ - src/core/lib/surface/version.c \ - src/core/lib/transport/byte_stream.c \ - src/core/lib/transport/connectivity_state.c \ - src/core/lib/transport/metadata.c \ - src/core/lib/transport/metadata_batch.c \ - src/core/lib/transport/static_metadata.c \ - src/core/lib/transport/transport.c \ - src/core/lib/transport/transport_op_string.c \ - src/core/lib/http/httpcli_security_connector.c \ - src/core/lib/security/context/security_context.c \ - src/core/lib/security/credentials/composite/composite_credentials.c \ - src/core/lib/security/credentials/credentials.c \ - src/core/lib/security/credentials/credentials_metadata.c \ - src/core/lib/security/credentials/fake/fake_credentials.c \ - src/core/lib/security/credentials/google_default/credentials_posix.c \ - src/core/lib/security/credentials/google_default/credentials_windows.c \ - src/core/lib/security/credentials/google_default/google_default_credentials.c \ - src/core/lib/security/credentials/iam/iam_credentials.c \ - src/core/lib/security/credentials/jwt/json_token.c \ - src/core/lib/security/credentials/jwt/jwt_credentials.c \ - src/core/lib/security/credentials/jwt/jwt_verifier.c \ - src/core/lib/security/credentials/oauth2/oauth2_credentials.c \ - src/core/lib/security/credentials/plugin/plugin_credentials.c \ - src/core/lib/security/credentials/ssl/ssl_credentials.c \ - src/core/lib/security/transport/client_auth_filter.c \ - src/core/lib/security/transport/handshake.c \ - src/core/lib/security/transport/secure_endpoint.c \ - src/core/lib/security/transport/security_connector.c \ - src/core/lib/security/transport/server_auth_filter.c \ - src/core/lib/security/transport/tsi_error.c \ - src/core/lib/security/util/b64.c \ - src/core/lib/security/util/json_util.c \ - src/core/lib/surface/init_secure.c \ - src/core/ext/transport/chttp2/alpn/alpn.c \ - src/core/lib/tsi/fake_transport_security.c \ - src/core/lib/tsi/ssl_transport_security.c \ - src/core/lib/tsi/transport_security.c \ src/cpp/codegen/codegen_init.cc \ PUBLIC_HEADERS_CXX += \ @@ -4249,14 +4009,6 @@ PUBLIC_HEADERS_CXX += \ include/grpc/impl/codegen/sync_posix.h \ include/grpc/impl/codegen/sync_windows.h \ include/grpc/impl/codegen/time.h \ - include/grpc/byte_buffer.h \ - include/grpc/byte_buffer_reader.h \ - include/grpc/compression.h \ - include/grpc/grpc.h \ - include/grpc/grpc_posix.h \ - include/grpc/status.h \ - include/grpc/grpc_security.h \ - include/grpc/grpc_security_constants.h \ LIBGRPC++_UNSECURE_OBJS = $(addprefix $(OBJDIR)/$(CONFIG)/, $(addsuffix .o, $(basename $(LIBGRPC++_UNSECURE_SRC)))) @@ -4283,18 +4035,18 @@ endif ifeq ($(SYSTEM),MINGW32) -$(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_UNSECURE_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/gpr.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/grpc_unsecure.$(SHARED_EXT) +$(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_UNSECURE_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/gpr.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/grpc_unsecure.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/grpc.$(SHARED_EXT) $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared grpc++_unsecure.def -Wl,--output-def=$(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).def -Wl,--out-implib=$(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION)-dll.a -o $(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr-imp -lgrpc_unsecure-imp + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared grpc++_unsecure.def -Wl,--output-def=$(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).def -Wl,--out-implib=$(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION)-dll.a -o $(LIBDIR)/$(CONFIG)/grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr-imp -lgrpc_unsecure-imp -lgrpc-imp else -$(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_UNSECURE_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/libgpr.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc_unsecure.$(SHARED_EXT) +$(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT): $(LIBGRPC++_UNSECURE_OBJS) $(ZLIB_DEP) $(PROTOBUF_DEP) $(LIBDIR)/$(CONFIG)/libgpr.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc_unsecure.$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc.$(SHARED_EXT) $(E) "[LD] Linking $@" $(Q) mkdir -p `dirname $@` ifeq ($(SYSTEM),Darwin) - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -install_name $(SHARED_PREFIX)grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) -dynamiclib -o $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr -lgrpc_unsecure + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -install_name $(SHARED_PREFIX)grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) -dynamiclib -o $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr -lgrpc_unsecure -lgrpc else - $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared -Wl,-soname,libgrpc++_unsecure.so.0 -o $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr -lgrpc_unsecure + $(Q) $(LDXX) $(LDFLAGS) -L$(LIBDIR)/$(CONFIG) -shared -Wl,-soname,libgrpc++_unsecure.so.0 -o $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBGRPC++_UNSECURE_OBJS) $(LDLIBS) $(ZLIB_MERGE_LIBS) $(LDLIBSXX) $(LDLIBS_PROTOBUF) -lgpr -lgrpc_unsecure -lgrpc $(Q) ln -sf $(SHARED_PREFIX)grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).so.0 $(Q) ln -sf $(SHARED_PREFIX)grpc++_unsecure$(SHARED_VERSION).$(SHARED_EXT) $(LIBDIR)/$(CONFIG)/libgrpc++_unsecure$(SHARED_VERSION).so endif @@ -15099,6 +14851,34 @@ src/core/ext/transport/chttp2/server/secure/server_secure_chttp2.c: $(OPENSSL_DE src/core/ext/transport/cronet/client/secure/cronet_channel_create.c: $(OPENSSL_DEP) src/core/ext/transport/cronet/transport/cronet_api_dummy.c: $(OPENSSL_DEP) src/core/ext/transport/cronet/transport/cronet_transport.c: $(OPENSSL_DEP) +src/core/lib/http/httpcli_security_connector.c: $(OPENSSL_DEP) +src/core/lib/security/context/security_context.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/composite/composite_credentials.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/credentials.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/credentials_metadata.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/fake/fake_credentials.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/google_default/credentials_posix.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/google_default/credentials_windows.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/google_default/google_default_credentials.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/iam/iam_credentials.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/jwt/json_token.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/jwt/jwt_credentials.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/jwt/jwt_verifier.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/oauth2/oauth2_credentials.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/plugin/plugin_credentials.c: $(OPENSSL_DEP) +src/core/lib/security/credentials/ssl/ssl_credentials.c: $(OPENSSL_DEP) +src/core/lib/security/transport/client_auth_filter.c: $(OPENSSL_DEP) +src/core/lib/security/transport/handshake.c: $(OPENSSL_DEP) +src/core/lib/security/transport/secure_endpoint.c: $(OPENSSL_DEP) +src/core/lib/security/transport/security_connector.c: $(OPENSSL_DEP) +src/core/lib/security/transport/server_auth_filter.c: $(OPENSSL_DEP) +src/core/lib/security/transport/tsi_error.c: $(OPENSSL_DEP) +src/core/lib/security/util/b64.c: $(OPENSSL_DEP) +src/core/lib/security/util/json_util.c: $(OPENSSL_DEP) +src/core/lib/surface/init_secure.c: $(OPENSSL_DEP) +src/core/lib/tsi/fake_transport_security.c: $(OPENSSL_DEP) +src/core/lib/tsi/ssl_transport_security.c: $(OPENSSL_DEP) +src/core/lib/tsi/transport_security.c: $(OPENSSL_DEP) src/core/plugin_registry/grpc_cronet_plugin_registry.c: $(OPENSSL_DEP) src/core/plugin_registry/grpc_plugin_registry.c: $(OPENSSL_DEP) src/cpp/client/secure_credentials.cc: $(OPENSSL_DEP) diff --git a/build.yaml b/build.yaml index dc42a61300e..1b20dde5631 100644 --- a/build.yaml +++ b/build.yaml @@ -712,10 +712,10 @@ filegroups: - src/cpp/util/status.cc - src/cpp/util/string_ref.cc - src/cpp/util/time.cc + deps: + - grpc uses: - grpc++_codegen_base - - grpc_base - - grpc_secure - name: grpc++_codegen_base language: c++ public_headers: diff --git a/tools/doxygen/Doxyfile.c++ b/tools/doxygen/Doxyfile.c++ index db6b36f8c7b..de7acd7777e 100644 --- a/tools/doxygen/Doxyfile.c++ +++ b/tools/doxygen/Doxyfile.c++ @@ -857,15 +857,7 @@ include/grpc/impl/codegen/sync.h \ include/grpc/impl/codegen/sync_generic.h \ include/grpc/impl/codegen/sync_posix.h \ include/grpc/impl/codegen/sync_windows.h \ -include/grpc/impl/codegen/time.h \ -include/grpc/byte_buffer.h \ -include/grpc/byte_buffer_reader.h \ -include/grpc/compression.h \ -include/grpc/grpc.h \ -include/grpc/grpc_posix.h \ -include/grpc/status.h \ -include/grpc/grpc_security.h \ -include/grpc/grpc_security_constants.h +include/grpc/impl/codegen/time.h # This tag can be used to specify the character encoding of the source files # that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses diff --git a/tools/doxygen/Doxyfile.c++.internal b/tools/doxygen/Doxyfile.c++.internal index 660e501d713..76bb3b6c59e 100644 --- a/tools/doxygen/Doxyfile.c++.internal +++ b/tools/doxygen/Doxyfile.c++.internal @@ -858,14 +858,6 @@ include/grpc/impl/codegen/sync_generic.h \ include/grpc/impl/codegen/sync_posix.h \ include/grpc/impl/codegen/sync_windows.h \ include/grpc/impl/codegen/time.h \ -include/grpc/byte_buffer.h \ -include/grpc/byte_buffer_reader.h \ -include/grpc/compression.h \ -include/grpc/grpc.h \ -include/grpc/grpc_posix.h \ -include/grpc/status.h \ -include/grpc/grpc_security.h \ -include/grpc/grpc_security_constants.h \ include/grpc++/impl/codegen/core_codegen.h \ src/cpp/client/secure_credentials.h \ src/cpp/common/secure_auth_context.h \ @@ -873,109 +865,6 @@ src/cpp/server/secure_server_credentials.h \ src/cpp/client/create_channel_internal.h \ src/cpp/server/dynamic_thread_pool.h \ src/cpp/server/thread_pool_interface.h \ -src/core/lib/channel/channel_args.h \ -src/core/lib/channel/channel_stack.h \ -src/core/lib/channel/channel_stack_builder.h \ -src/core/lib/channel/compress_filter.h \ -src/core/lib/channel/connected_channel.h \ -src/core/lib/channel/context.h \ -src/core/lib/channel/http_client_filter.h \ -src/core/lib/channel/http_server_filter.h \ -src/core/lib/compression/algorithm_metadata.h \ -src/core/lib/compression/message_compress.h \ -src/core/lib/debug/trace.h \ -src/core/lib/http/format_request.h \ -src/core/lib/http/httpcli.h \ -src/core/lib/http/parser.h \ -src/core/lib/iomgr/closure.h \ -src/core/lib/iomgr/endpoint.h \ -src/core/lib/iomgr/endpoint_pair.h \ -src/core/lib/iomgr/error.h \ -src/core/lib/iomgr/ev_epoll_linux.h \ -src/core/lib/iomgr/ev_poll_and_epoll_posix.h \ -src/core/lib/iomgr/ev_poll_posix.h \ -src/core/lib/iomgr/ev_posix.h \ -src/core/lib/iomgr/exec_ctx.h \ -src/core/lib/iomgr/executor.h \ -src/core/lib/iomgr/iocp_windows.h \ -src/core/lib/iomgr/iomgr.h \ -src/core/lib/iomgr/iomgr_internal.h \ -src/core/lib/iomgr/iomgr_posix.h \ -src/core/lib/iomgr/load_file.h \ -src/core/lib/iomgr/network_status_tracker.h \ -src/core/lib/iomgr/polling_entity.h \ -src/core/lib/iomgr/pollset.h \ -src/core/lib/iomgr/pollset_set.h \ -src/core/lib/iomgr/pollset_set_windows.h \ -src/core/lib/iomgr/pollset_windows.h \ -src/core/lib/iomgr/resolve_address.h \ -src/core/lib/iomgr/sockaddr.h \ -src/core/lib/iomgr/sockaddr_posix.h \ -src/core/lib/iomgr/sockaddr_utils.h \ -src/core/lib/iomgr/sockaddr_windows.h \ -src/core/lib/iomgr/socket_utils_posix.h \ -src/core/lib/iomgr/socket_windows.h \ -src/core/lib/iomgr/tcp_client.h \ -src/core/lib/iomgr/tcp_posix.h \ -src/core/lib/iomgr/tcp_server.h \ -src/core/lib/iomgr/tcp_windows.h \ -src/core/lib/iomgr/time_averaged_stats.h \ -src/core/lib/iomgr/timer.h \ -src/core/lib/iomgr/timer_heap.h \ -src/core/lib/iomgr/udp_server.h \ -src/core/lib/iomgr/unix_sockets_posix.h \ -src/core/lib/iomgr/wakeup_fd_pipe.h \ -src/core/lib/iomgr/wakeup_fd_posix.h \ -src/core/lib/iomgr/workqueue.h \ -src/core/lib/iomgr/workqueue_posix.h \ -src/core/lib/iomgr/workqueue_windows.h \ -src/core/lib/json/json.h \ -src/core/lib/json/json_common.h \ -src/core/lib/json/json_reader.h \ -src/core/lib/json/json_writer.h \ -src/core/lib/surface/api_trace.h \ -src/core/lib/surface/call.h \ -src/core/lib/surface/call_test_only.h \ -src/core/lib/surface/channel.h \ -src/core/lib/surface/channel_init.h \ -src/core/lib/surface/channel_stack_type.h \ -src/core/lib/surface/completion_queue.h \ -src/core/lib/surface/event_string.h \ -src/core/lib/surface/init.h \ -src/core/lib/surface/lame_client.h \ -src/core/lib/surface/server.h \ -src/core/lib/transport/byte_stream.h \ -src/core/lib/transport/connectivity_state.h \ -src/core/lib/transport/metadata.h \ -src/core/lib/transport/metadata_batch.h \ -src/core/lib/transport/static_metadata.h \ -src/core/lib/transport/transport.h \ -src/core/lib/transport/transport_impl.h \ -src/core/lib/security/context/security_context.h \ -src/core/lib/security/credentials/composite/composite_credentials.h \ -src/core/lib/security/credentials/credentials.h \ -src/core/lib/security/credentials/fake/fake_credentials.h \ -src/core/lib/security/credentials/google_default/google_default_credentials.h \ -src/core/lib/security/credentials/iam/iam_credentials.h \ -src/core/lib/security/credentials/jwt/json_token.h \ -src/core/lib/security/credentials/jwt/jwt_credentials.h \ -src/core/lib/security/credentials/jwt/jwt_verifier.h \ -src/core/lib/security/credentials/oauth2/oauth2_credentials.h \ -src/core/lib/security/credentials/plugin/plugin_credentials.h \ -src/core/lib/security/credentials/ssl/ssl_credentials.h \ -src/core/lib/security/transport/auth_filters.h \ -src/core/lib/security/transport/handshake.h \ -src/core/lib/security/transport/secure_endpoint.h \ -src/core/lib/security/transport/security_connector.h \ -src/core/lib/security/transport/tsi_error.h \ -src/core/lib/security/util/b64.h \ -src/core/lib/security/util/json_util.h \ -src/core/ext/transport/chttp2/alpn/alpn.h \ -src/core/lib/tsi/fake_transport_security.h \ -src/core/lib/tsi/ssl_transport_security.h \ -src/core/lib/tsi/ssl_types.h \ -src/core/lib/tsi/transport_security.h \ -src/core/lib/tsi/transport_security_interface.h \ src/cpp/client/secure_credentials.cc \ src/cpp/common/auth_property_iterator.cc \ src/cpp/common/secure_auth_context.cc \ @@ -1008,122 +897,6 @@ src/cpp/util/slice.cc \ src/cpp/util/status.cc \ src/cpp/util/string_ref.cc \ src/cpp/util/time.cc \ -src/core/lib/channel/channel_args.c \ -src/core/lib/channel/channel_stack.c \ -src/core/lib/channel/channel_stack_builder.c \ -src/core/lib/channel/compress_filter.c \ -src/core/lib/channel/connected_channel.c \ -src/core/lib/channel/http_client_filter.c \ -src/core/lib/channel/http_server_filter.c \ -src/core/lib/compression/compression.c \ -src/core/lib/compression/message_compress.c \ -src/core/lib/debug/trace.c \ -src/core/lib/http/format_request.c \ -src/core/lib/http/httpcli.c \ -src/core/lib/http/parser.c \ -src/core/lib/iomgr/closure.c \ -src/core/lib/iomgr/endpoint.c \ -src/core/lib/iomgr/endpoint_pair_posix.c \ -src/core/lib/iomgr/endpoint_pair_windows.c \ -src/core/lib/iomgr/error.c \ -src/core/lib/iomgr/ev_epoll_linux.c \ -src/core/lib/iomgr/ev_poll_and_epoll_posix.c \ -src/core/lib/iomgr/ev_poll_posix.c \ -src/core/lib/iomgr/ev_posix.c \ -src/core/lib/iomgr/exec_ctx.c \ -src/core/lib/iomgr/executor.c \ -src/core/lib/iomgr/iocp_windows.c \ -src/core/lib/iomgr/iomgr.c \ -src/core/lib/iomgr/iomgr_posix.c \ -src/core/lib/iomgr/iomgr_windows.c \ -src/core/lib/iomgr/load_file.c \ -src/core/lib/iomgr/network_status_tracker.c \ -src/core/lib/iomgr/polling_entity.c \ -src/core/lib/iomgr/pollset_set_windows.c \ -src/core/lib/iomgr/pollset_windows.c \ -src/core/lib/iomgr/resolve_address_posix.c \ -src/core/lib/iomgr/resolve_address_windows.c \ -src/core/lib/iomgr/sockaddr_utils.c \ -src/core/lib/iomgr/socket_utils_common_posix.c \ -src/core/lib/iomgr/socket_utils_linux.c \ -src/core/lib/iomgr/socket_utils_posix.c \ -src/core/lib/iomgr/socket_windows.c \ -src/core/lib/iomgr/tcp_client_posix.c \ -src/core/lib/iomgr/tcp_client_windows.c \ -src/core/lib/iomgr/tcp_posix.c \ -src/core/lib/iomgr/tcp_server_posix.c \ -src/core/lib/iomgr/tcp_server_windows.c \ -src/core/lib/iomgr/tcp_windows.c \ -src/core/lib/iomgr/time_averaged_stats.c \ -src/core/lib/iomgr/timer.c \ -src/core/lib/iomgr/timer_heap.c \ -src/core/lib/iomgr/udp_server.c \ -src/core/lib/iomgr/unix_sockets_posix.c \ -src/core/lib/iomgr/unix_sockets_posix_noop.c \ -src/core/lib/iomgr/wakeup_fd_eventfd.c \ -src/core/lib/iomgr/wakeup_fd_nospecial.c \ -src/core/lib/iomgr/wakeup_fd_pipe.c \ -src/core/lib/iomgr/wakeup_fd_posix.c \ -src/core/lib/iomgr/workqueue_posix.c \ -src/core/lib/iomgr/workqueue_windows.c \ -src/core/lib/json/json.c \ -src/core/lib/json/json_reader.c \ -src/core/lib/json/json_string.c \ -src/core/lib/json/json_writer.c \ -src/core/lib/surface/alarm.c \ -src/core/lib/surface/api_trace.c \ -src/core/lib/surface/byte_buffer.c \ -src/core/lib/surface/byte_buffer_reader.c \ -src/core/lib/surface/call.c \ -src/core/lib/surface/call_details.c \ -src/core/lib/surface/call_log_batch.c \ -src/core/lib/surface/channel.c \ -src/core/lib/surface/channel_init.c \ -src/core/lib/surface/channel_ping.c \ -src/core/lib/surface/channel_stack_type.c \ -src/core/lib/surface/completion_queue.c \ -src/core/lib/surface/event_string.c \ -src/core/lib/surface/lame_client.c \ -src/core/lib/surface/metadata_array.c \ -src/core/lib/surface/server.c \ -src/core/lib/surface/validate_metadata.c \ -src/core/lib/surface/version.c \ -src/core/lib/transport/byte_stream.c \ -src/core/lib/transport/connectivity_state.c \ -src/core/lib/transport/metadata.c \ -src/core/lib/transport/metadata_batch.c \ -src/core/lib/transport/static_metadata.c \ -src/core/lib/transport/transport.c \ -src/core/lib/transport/transport_op_string.c \ -src/core/lib/http/httpcli_security_connector.c \ -src/core/lib/security/context/security_context.c \ -src/core/lib/security/credentials/composite/composite_credentials.c \ -src/core/lib/security/credentials/credentials.c \ -src/core/lib/security/credentials/credentials_metadata.c \ -src/core/lib/security/credentials/fake/fake_credentials.c \ -src/core/lib/security/credentials/google_default/credentials_posix.c \ -src/core/lib/security/credentials/google_default/credentials_windows.c \ -src/core/lib/security/credentials/google_default/google_default_credentials.c \ -src/core/lib/security/credentials/iam/iam_credentials.c \ -src/core/lib/security/credentials/jwt/json_token.c \ -src/core/lib/security/credentials/jwt/jwt_credentials.c \ -src/core/lib/security/credentials/jwt/jwt_verifier.c \ -src/core/lib/security/credentials/oauth2/oauth2_credentials.c \ -src/core/lib/security/credentials/plugin/plugin_credentials.c \ -src/core/lib/security/credentials/ssl/ssl_credentials.c \ -src/core/lib/security/transport/client_auth_filter.c \ -src/core/lib/security/transport/handshake.c \ -src/core/lib/security/transport/secure_endpoint.c \ -src/core/lib/security/transport/security_connector.c \ -src/core/lib/security/transport/server_auth_filter.c \ -src/core/lib/security/transport/tsi_error.c \ -src/core/lib/security/util/b64.c \ -src/core/lib/security/util/json_util.c \ -src/core/lib/surface/init_secure.c \ -src/core/ext/transport/chttp2/alpn/alpn.c \ -src/core/lib/tsi/fake_transport_security.c \ -src/core/lib/tsi/ssl_transport_security.c \ -src/core/lib/tsi/transport_security.c \ src/cpp/codegen/codegen_init.cc # This tag can be used to specify the character encoding of the source files diff --git a/tools/run_tests/sources_and_headers.json b/tools/run_tests/sources_and_headers.json index 3ebb445b8a8..cdbc254f430 100644 --- a/tools/run_tests/sources_and_headers.json +++ b/tools/run_tests/sources_and_headers.json @@ -4345,7 +4345,6 @@ }, { "deps": [ - "gpr", "grpc", "grpc++_base", "grpc++_codegen_base", @@ -4459,6 +4458,7 @@ { "deps": [ "gpr", + "grpc", "grpc++_base", "grpc++_codegen_base", "grpc++_codegen_base_src", @@ -6515,10 +6515,8 @@ }, { "deps": [ - "gpr", - "grpc++_codegen_base", - "grpc_base", - "grpc_secure" + "grpc", + "grpc++_codegen_base" ], "headers": [ "include/grpc++/alarm.h", diff --git a/vsprojects/grpc.sln b/vsprojects/grpc.sln index 8fccc646e5c..84720914b0b 100644 --- a/vsprojects/grpc.sln +++ b/vsprojects/grpc.sln @@ -49,7 +49,6 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc++", "vcxproj\.\grpc++\ EndProjectSection ProjectSection(ProjectDependencies) = postProject {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} - {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc++_reflection", "vcxproj\.\grpc++_reflection\grpc++_reflection.vcxproj", "{5F575402-3F89-5D1A-6910-9DB8BF5D2BAB}" @@ -67,6 +66,7 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc++_unsecure", "vcxproj\ ProjectSection(ProjectDependencies) = postProject {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} = {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} {46CEDFFF-9692-456A-AA24-38B5D6BCF4C5} = {46CEDFFF-9692-456A-AA24-38B5D6BCF4C5} + {29D16885-7228-4C31-81ED-5F9187C7F2A9} = {29D16885-7228-4C31-81ED-5F9187C7F2A9} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "grpc_create_jwt", "vcxproj\.\grpc_create_jwt\grpc_create_jwt.vcxproj", "{77971F8D-F583-3E77-0E3C-6C1FB6B1749C}" diff --git a/vsprojects/vcxproj/grpc++/grpc++.vcxproj b/vsprojects/vcxproj/grpc++/grpc++.vcxproj index a2711ca7a46..cb9e41ea22f 100644 --- a/vsprojects/vcxproj/grpc++/grpc++.vcxproj +++ b/vsprojects/vcxproj/grpc++/grpc++.vcxproj @@ -356,14 +356,6 @@ - - - - - - - - @@ -373,109 +365,6 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @@ -542,238 +431,6 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @@ -781,9 +438,6 @@ {29D16885-7228-4C31-81ED-5F9187C7F2A9} - - {B23D3D1A-9438-4EDA-BEB6-9A0A03D17792} - diff --git a/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters b/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters index f478ac9839c..a9051182b3c 100644 --- a/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters +++ b/vsprojects/vcxproj/grpc++/grpc++.vcxproj.filters @@ -97,354 +97,6 @@ src\cpp\util - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\compression - - - src\core\lib\compression - - - src\core\lib\debug - - - src\core\lib\http - - - src\core\lib\http - - - src\core\lib\http - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\http - - - src\core\lib\security\context - - - src\core\lib\security\credentials\composite - - - src\core\lib\security\credentials - - - src\core\lib\security\credentials - - - src\core\lib\security\credentials\fake - - - src\core\lib\security\credentials\google_default - - - src\core\lib\security\credentials\google_default - - - src\core\lib\security\credentials\google_default - - - src\core\lib\security\credentials\iam - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\oauth2 - - - src\core\lib\security\credentials\plugin - - - src\core\lib\security\credentials\ssl - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\util - - - src\core\lib\security\util - - - src\core\lib\surface - - - src\core\ext\transport\chttp2\alpn - - - src\core\lib\tsi - - - src\core\lib\tsi - - - src\core\lib\tsi - src\cpp\codegen @@ -744,30 +396,6 @@ include\grpc\impl\codegen - - include\grpc - - - include\grpc - - - include\grpc - - - include\grpc - - - include\grpc - - - include\grpc - - - include\grpc - - - include\grpc - @@ -791,315 +419,6 @@ src\cpp\server - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\compression - - - src\core\lib\compression - - - src\core\lib\debug - - - src\core\lib\http - - - src\core\lib\http - - - src\core\lib\http - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\security\context - - - src\core\lib\security\credentials\composite - - - src\core\lib\security\credentials - - - src\core\lib\security\credentials\fake - - - src\core\lib\security\credentials\google_default - - - src\core\lib\security\credentials\iam - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\oauth2 - - - src\core\lib\security\credentials\plugin - - - src\core\lib\security\credentials\ssl - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\util - - - src\core\lib\security\util - - - src\core\ext\transport\chttp2\alpn - - - src\core\lib\tsi - - - src\core\lib\tsi - - - src\core\lib\tsi - - - src\core\lib\tsi - - - src\core\lib\tsi - @@ -1139,90 +458,6 @@ {328ff211-2886-406e-56f9-18ba1686f363} - - {d02f1155-7e7e-3736-3c69-dc9146dc523d} - - - {96d09c4a-59f9-3486-6c2f-cbf695b285d8} - - - {202b1172-189f-afc4-f16c-4ca12677b480} - - - {9de393b8-4b6e-6c34-122a-940419ca9989} - - - {efb6b3e6-8c7b-c2a0-12c6-486c68cdb8ec} - - - {80567a8f-622f-a3ce-c12d-aebb63984b07} - - - {e769265c-8abd-cd64-2cc2-a52da484fe7b} - - - {701b2d46-11c6-3640-b189-45287f00bee3} - - - {ada68fd5-8e51-98cb-71a7-baf7989d8ffa} - - - {e770844e-61d4-555e-59be-81288e21a35f} - - - {04dfa1c8-7ffe-4f06-4a7c-37441dc75764} - - - {a5d5bddf-6f19-b655-a03a-f30ff5c253a5} - - - {dbd8cbb6-6308-d6fe-7a36-06cc7045c037} - - - {ecd2c264-808d-0041-2f69-a5200543de91} - - - {0015e481-7e80-8936-a25c-c3fa260cc095} - - - {fad200df-a5e2-1648-7442-cea0f07edd4d} - - - {397464b3-9bbd-15a5-041b-c7deef1662ec} - - - {567691b4-6a06-cc5a-c6ad-e8c080b89ecf} - - - {d5930113-d396-7a70-d273-d07a1feae0ff} - - - {0f6afb67-4b51-6344-9de7-2b1a18a19e7d} - - - {99faa051-ca9f-cb4f-36d5-95f042fb22bc} - - - {b7a9e7e5-2445-6b0f-4677-5095ca10e760} - - - {436bc65a-0c1b-d85a-2c91-6474588c5cb6} - - - {e6a9bf58-3b0f-0b3d-3a35-3ded80d27695} - - - {b4a1cab8-5c2c-909a-8097-7a5c8f0aa9f7} - - - {fb2276d7-5a11-f1d9-82c3-e7c7f1155523} - - - {4bd7971a-68f7-0d5a-f502-6dea3099caaa} - - - {aa0153b8-c9b6-ae1d-ebdd-89754d8579f1} - {2420a905-e4f1-a5aa-a364-6a112878a39e} diff --git a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj index 84e709611df..03be485b297 100644 --- a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj +++ b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj @@ -356,122 +356,11 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @@ -528,238 +417,6 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @@ -770,6 +427,9 @@ {46CEDFFF-9692-456A-AA24-38B5D6BCF4C5} + + {29D16885-7228-4C31-81ED-5F9187C7F2A9} + diff --git a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters index 1e54e1595d4..ba99bc53c8c 100644 --- a/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters +++ b/vsprojects/vcxproj/grpc++_unsecure/grpc++_unsecure.vcxproj.filters @@ -82,354 +82,6 @@ src\cpp\util - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\compression - - - src\core\lib\compression - - - src\core\lib\debug - - - src\core\lib\http - - - src\core\lib\http - - - src\core\lib\http - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\http - - - src\core\lib\security\context - - - src\core\lib\security\credentials\composite - - - src\core\lib\security\credentials - - - src\core\lib\security\credentials - - - src\core\lib\security\credentials\fake - - - src\core\lib\security\credentials\google_default - - - src\core\lib\security\credentials\google_default - - - src\core\lib\security\credentials\google_default - - - src\core\lib\security\credentials\iam - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\oauth2 - - - src\core\lib\security\credentials\plugin - - - src\core\lib\security\credentials\ssl - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\util - - - src\core\lib\security\util - - - src\core\lib\surface - - - src\core\ext\transport\chttp2\alpn - - - src\core\lib\tsi - - - src\core\lib\tsi - - - src\core\lib\tsi - src\cpp\codegen @@ -729,30 +381,6 @@ include\grpc\impl\codegen - - include\grpc - - - include\grpc - - - include\grpc - - - include\grpc - - - include\grpc - - - include\grpc - - - include\grpc - - - include\grpc - @@ -764,315 +392,6 @@ src\cpp\server - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\channel - - - src\core\lib\compression - - - src\core\lib\compression - - - src\core\lib\debug - - - src\core\lib\http - - - src\core\lib\http - - - src\core\lib\http - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\iomgr - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\json - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\surface - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\transport - - - src\core\lib\security\context - - - src\core\lib\security\credentials\composite - - - src\core\lib\security\credentials - - - src\core\lib\security\credentials\fake - - - src\core\lib\security\credentials\google_default - - - src\core\lib\security\credentials\iam - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\jwt - - - src\core\lib\security\credentials\oauth2 - - - src\core\lib\security\credentials\plugin - - - src\core\lib\security\credentials\ssl - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\transport - - - src\core\lib\security\util - - - src\core\lib\security\util - - - src\core\ext\transport\chttp2\alpn - - - src\core\lib\tsi - - - src\core\lib\tsi - - - src\core\lib\tsi - - - src\core\lib\tsi - - - src\core\lib\tsi - @@ -1112,90 +431,6 @@ {cce6a85d-1111-3834-6825-31e170d93cff} - - {595f2ea0-aafb-87e5-c938-db3ff0b0c69a} - - - {52eca76b-9502-3d96-9064-6415226a860f} - - - {8e70201f-3b54-d3cb-8b30-ebe0d96a9b2a} - - - {d505ab7b-5e44-f307-5361-500128965cdc} - - - {d54bab94-cab9-803d-2737-5120774f1893} - - - {cf8fd5d8-ff54-331d-2d20-36d6cae0e14b} - - - {7e0225af-000b-4873-1c16-caffffbfd084} - - - {0bbdbf56-83ad-bb4b-c4e2-a6d38c342179} - - - {3875f7d7-ff11-c91d-0f98-810260cb554b} - - - {4bd405b9-af65-f0a6-d67a-433f75900668} - - - {f4b146e4-8fba-83a6-1cc1-1262ebb785e8} - - - {b83c8e70-e491-f6f9-a08c-85f632bb61d2} - - - {7e21ce26-45e2-6baf-037d-8ab4374077a9} - - - {613e655a-e5c0-9f0c-2bb4-62310a7329c0} - - - {30bddf3f-0eda-9f2f-8171-d86b1e4896fc} - - - {b34f8fa3-0fb9-4916-be6d-2a14a0794882} - - - {7e11872b-bfbb-7d23-4783-e56909c520e8} - - - {212855e8-b7bc-d5bb-0734-dd28996f28de} - - - {6d3828d0-5e5f-15c2-7d46-5d4039a88aad} - - - {b31e7015-364c-5701-31d0-644b1a8ae8c9} - - - {43e3cb91-4101-1fee-6833-20f77ab7f4e5} - - - {727c0b51-4544-957f-45f2-00bf42ff7db9} - - - {606a441b-0d57-85d8-8079-1e6e502d18f1} - - - {5b0b16ae-a8ad-81c3-afe4-8ac0b9e15311} - - - {56333427-0f81-b88b-bf49-a1b2f462023d} - - - {1d59dcef-3358-d0ab-fa42-64da74065785} - - - {ba865739-5dd9-6731-6772-48c25d45134f} - - - {dd4e4960-5bc8-395b-09c4-f2cbd6f6432b} - {1e5fd68c-bd87-e803-42b0-75a7fa19b91d} From 43ba180c2189127fb51dacf856fbf7ff4bb70a9f Mon Sep 17 00:00:00 2001 From: Stanley Cheung Date: Fri, 8 Jul 2016 12:07:39 -0700 Subject: [PATCH 0478/1039] php: remove gpr_log debug --- src/php/ext/grpc/channel.c | 2 -- src/php/ext/grpc/server.c | 1 - 2 files changed, 3 deletions(-) diff --git a/src/php/ext/grpc/channel.c b/src/php/ext/grpc/channel.c index 9f0431908f9..8d94c59683e 100644 --- a/src/php/ext/grpc/channel.c +++ b/src/php/ext/grpc/channel.c @@ -48,7 +48,6 @@ #include #include -#include #include #include "completion_queue.h" @@ -172,7 +171,6 @@ PHP_METHOD(Channel, __construct) { if (creds == NULL) { channel->wrapped = grpc_insecure_channel_create(target, &args, NULL); } else { - gpr_log(GPR_DEBUG, "Initialized secure channel"); channel->wrapped = grpc_secure_channel_create(creds->wrapped, target, &args, NULL); } diff --git a/src/php/ext/grpc/server.c b/src/php/ext/grpc/server.c index 6df2e4f9782..c13e7cd1f92 100644 --- a/src/php/ext/grpc/server.c +++ b/src/php/ext/grpc/server.c @@ -48,7 +48,6 @@ #include #include -#include #include #include "completion_queue.h" From 77f8da22eefedbb49c13a414ab3ff748137e93d5 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Fri, 8 Jul 2016 15:03:20 -0700 Subject: [PATCH 0479/1039] added a comment just to retrigger tests --- src/compiler/objective_c_plugin.cc | 1 + 1 file changed, 1 insertion(+) diff --git a/src/compiler/objective_c_plugin.cc b/src/compiler/objective_c_plugin.cc index 5026088db1e..be647764024 100644 --- a/src/compiler/objective_c_plugin.cc +++ b/src/compiler/objective_c_plugin.cc @@ -81,6 +81,7 @@ class ObjectiveCGrpcGenerator : public grpc::protobuf::compiler::CodeGenerator { if (IsProtobufLibraryBundledProtoFile(dependency)) { ::grpc::string base_name = header; grpc_generator::StripPrefix(&base_name, "google/protobuf/"); + // create the import code snippet proto_imports += "#if GPB_USE_PROTOBUF_FRAMEWORK_IMPORTS\n" " #import <" + ::grpc::string(ProtobufLibraryFrameworkName) + From 33ab1829a54db3949df483ed44b9c59ab9fb2d84 Mon Sep 17 00:00:00 2001 From: Craig Tiller Date: Fri, 8 Jul 2016 15:19:06 -0700 Subject: [PATCH 0480/1039] Convert time to monotonic internally --- src/core/lib/surface/call.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/core/lib/surface/call.c b/src/core/lib/surface/call.c index e5668be47fe..fc9df76dc18 100644 --- a/src/core/lib/surface/call.c +++ b/src/core/lib/surface/call.c @@ -259,7 +259,8 @@ grpc_call *grpc_call_create( call->metadata_batch[i][j].deadline = gpr_inf_future(GPR_CLOCK_MONOTONIC); } } - call->send_deadline = send_deadline; + call->send_deadline = + gpr_convert_clock_type(send_deadline, GPR_CLOCK_MONOTONIC); GRPC_CHANNEL_INTERNAL_REF(channel, "call"); /* initial refcount dropped by grpc_call_destroy */ grpc_call_stack_init(&exec_ctx, channel_stack, 1, destroy_call, call, From 027ae0b4eba2e65d8b4a6e4ff0005d1ac8836689 Mon Sep 17 00:00:00 2001 From: Makarand Dharmapurikar Date: Fri, 8 Jul 2016 16:39:37 -0700 Subject: [PATCH 0481/1039] removed unneeded import. fixes travis build --- src/objective-c/tests/InteropTests.m | 1 - 1 file changed, 1 deletion(-) diff --git a/src/objective-c/tests/InteropTests.m b/src/objective-c/tests/InteropTests.m index a2f63ac40ca..494743d6041 100644 --- a/src/objective-c/tests/InteropTests.m +++ b/src/objective-c/tests/InteropTests.m @@ -40,7 +40,6 @@ #import #import #import -#import #import #import #import From 7d892fbbc324989ed6d20721aa6efbe442ae731c Mon Sep 17 00:00:00 2001 From: yang-g Date: Fri, 8 Jul 2016 17:06:17 -0700 Subject: [PATCH 0482/1039] another one --- tools/doxygen/Doxyfile.c++.internal.orig | 2505 ---------------------- 1 file changed, 2505 deletions(-) delete mode 100644 tools/doxygen/Doxyfile.c++.internal.orig diff --git a/tools/doxygen/Doxyfile.c++.internal.orig b/tools/doxygen/Doxyfile.c++.internal.orig deleted file mode 100644 index c214b3d3c84..00000000000 --- a/tools/doxygen/Doxyfile.c++.internal.orig +++ /dev/null @@ -1,2505 +0,0 @@ - - -# Doxyfile 1.8.9.1 - -# This file describes the settings to be used by the documentation system -# doxygen (www.doxygen.org) for a project. -# -# All text after a double hash (##) is considered a comment and is placed in -# front of the TAG it is preceding. -# -# All text after a single hash (#) is considered a comment and will be ignored. -# The format is: -# TAG = value [value, ...] -# For lists, items can also be appended using: -# TAG += value [value, ...] -# Values that contain spaces should be placed between quotes (\" \"). - -#--------------------------------------------------------------------------- -# Project related configuration options -#--------------------------------------------------------------------------- - -# This tag specifies the encoding used for all characters in the config file -# that follow. The default is UTF-8 which is also the encoding used for all text -# before the first occurrence of this tag. Doxygen uses libiconv (or the iconv -# built into libc) for the transcoding. See http://www.gnu.org/software/libiconv -# for the list of possible encodings. -# The default value is: UTF-8. - -DOXYFILE_ENCODING = UTF-8 - -# The PROJECT_NAME tag is a single word (or a sequence of words surrounded by -# double-quotes, unless you are using Doxywizard) that should identify the -# project for which the documentation is generated. This name is used in the -# title of most generated pages and in a few other places. -# The default value is: My Project. - -PROJECT_NAME = "GRPC C++" - -# The PROJECT_NUMBER tag can be used to enter a project or revision number. This -# could be handy for archiving the generated documentation or if some version -# control system is used. - -PROJECT_NUMBER = 0.15.0-dev - -# Using the PROJECT_BRIEF tag one can provide an optional one line description -# for a project that appears at the top of each page and should give viewer a -# quick idea about the purpose of the project. Keep the description short. - -PROJECT_BRIEF = - -# With the PROJECT_LOGO tag one can specify a logo or an icon that is included -# in the documentation. The maximum height of the logo should not exceed 55 -# pixels and the maximum width should not exceed 200 pixels. Doxygen will copy -# the logo to the output directory. - -PROJECT_LOGO = - -# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path -# into which the generated documentation will be written. If a relative path is -# entered, it will be relative to the location where doxygen was started. If -# left blank the current directory will be used. - -OUTPUT_DIRECTORY = doc/ref/c++.internal - -# If the CREATE_SUBDIRS tag is set to YES then doxygen will create 4096 sub- -# directories (in 2 levels) under the output directory of each output format and -# will distribute the generated files over these directories. Enabling this -# option can be useful when feeding doxygen a huge amount of source files, where -# putting all generated files in the same directory would otherwise causes -# performance problems for the file system. -# The default value is: NO. - -CREATE_SUBDIRS = NO - -# If the ALLOW_UNICODE_NAMES tag is set to YES, doxygen will allow non-ASCII -# characters to appear in the names of generated files. If set to NO, non-ASCII -# characters will be escaped, for example _xE3_x81_x84 will be used for Unicode -# U+3044. -# The default value is: NO. - -ALLOW_UNICODE_NAMES = NO - -# The OUTPUT_LANGUAGE tag is used to specify the language in which all -# documentation generated by doxygen is written. Doxygen will use this -# information to generate all constant output in the proper language. -# Possible values are: Afrikaans, Arabic, Armenian, Brazilian, Catalan, Chinese, -# Chinese-Traditional, Croatian, Czech, Danish, Dutch, English (United States), -# Esperanto, Farsi (Persian), Finnish, French, German, Greek, Hungarian, -# Indonesian, Italian, Japanese, Japanese-en (Japanese with English messages), -# Korean, Korean-en (Korean with English messages), Latvian, Lithuanian, -# Macedonian, Norwegian, Persian (Farsi), Polish, Portuguese, Romanian, Russian, -# Serbian, Serbian-Cyrillic, Slovak, Slovene, Spanish, Swedish, Turkish, -# Ukrainian and Vietnamese. -# The default value is: English. - -OUTPUT_LANGUAGE = English - -# If the BRIEF_MEMBER_DESC tag is set to YES, doxygen will include brief member -# descriptions after the members that are listed in the file and class -# documentation (similar to Javadoc). Set to NO to disable this. -# The default value is: YES. - -BRIEF_MEMBER_DESC = YES - -# If the REPEAT_BRIEF tag is set to YES, doxygen will prepend the brief -# description of a member or function before the detailed description -# -# Note: If both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the -# brief descriptions will be completely suppressed. -# The default value is: YES. - -REPEAT_BRIEF = YES - -# This tag implements a quasi-intelligent brief description abbreviator that is -# used to form the text in various listings. Each string in this list, if found -# as the leading text of the brief description, will be stripped from the text -# and the result, after processing the whole list, is used as the annotated -# text. Otherwise, the brief description is used as-is. If left blank, the -# following values are used ($name is automatically replaced with the name of -# the entity):The $name class, The $name widget, The $name file, is, provides, -# specifies, contains, represents, a, an and the. - -ABBREVIATE_BRIEF = - -# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then -# doxygen will generate a detailed section even if there is only a brief -# description. -# The default value is: NO. - -ALWAYS_DETAILED_SEC = NO - -# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all -# inherited members of a class in the documentation of that class as if those -# members were ordinary class members. Constructors, destructors and assignment -# operators of the base classes will not be shown. -# The default value is: NO. - -INLINE_INHERITED_MEMB = NO - -# If the FULL_PATH_NAMES tag is set to YES, doxygen will prepend the full path -# before files name in the file list and in the header files. If set to NO the -# shortest path that makes the file name unique will be used -# The default value is: YES. - -FULL_PATH_NAMES = YES - -# The STRIP_FROM_PATH tag can be used to strip a user-defined part of the path. -# Stripping is only done if one of the specified strings matches the left-hand -# part of the path. The tag can be used to show relative paths in the file list. -# If left blank the directory from which doxygen is run is used as the path to -# strip. -# -# Note that you can specify absolute paths here, but also relative paths, which -# will be relative from the directory where doxygen is started. -# This tag requires that the tag FULL_PATH_NAMES is set to YES. - -STRIP_FROM_PATH = - -# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of the -# path mentioned in the documentation of a class, which tells the reader which -# header file to include in order to use a class. If left blank only the name of -# the header file containing the class definition is used. Otherwise one should -# specify the list of include paths that are normally passed to the compiler -# using the -I flag. - -STRIP_FROM_INC_PATH = - -# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter (but -# less readable) file names. This can be useful is your file systems doesn't -# support long names like on DOS, Mac, or CD-ROM. -# The default value is: NO. - -SHORT_NAMES = NO - -# If the JAVADOC_AUTOBRIEF tag is set to YES then doxygen will interpret the -# first line (until the first dot) of a Javadoc-style comment as the brief -# description. If set to NO, the Javadoc-style will behave just like regular Qt- -# style comments (thus requiring an explicit @brief command for a brief -# description.) -# The default value is: NO. - -JAVADOC_AUTOBRIEF = YES - -# If the QT_AUTOBRIEF tag is set to YES then doxygen will interpret the first -# line (until the first dot) of a Qt-style comment as the brief description. If -# set to NO, the Qt-style will behave just like regular Qt-style comments (thus -# requiring an explicit \brief command for a brief description.) -# The default value is: NO. - -QT_AUTOBRIEF = NO - -# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make doxygen treat a -# multi-line C++ special comment block (i.e. a block of //! or /// comments) as -# a brief description. This used to be the default behavior. The new default is -# to treat a multi-line C++ comment block as a detailed description. Set this -# tag to YES if you prefer the old behavior instead. -# -# Note that setting this tag to YES also means that rational rose comments are -# not recognized any more. -# The default value is: NO. - -MULTILINE_CPP_IS_BRIEF = NO - -# If the INHERIT_DOCS tag is set to YES then an undocumented member inherits the -# documentation from any documented member that it re-implements. -# The default value is: YES. - -INHERIT_DOCS = YES - -# If the SEPARATE_MEMBER_PAGES tag is set to YES then doxygen will produce a new -# page for each member. If set to NO, the documentation of a member will be part -# of the file/class/namespace that contains it. -# The default value is: NO. - -SEPARATE_MEMBER_PAGES = NO - -# The TAB_SIZE tag can be used to set the number of spaces in a tab. Doxygen -# uses this value to replace tabs by spaces in code fragments. -# Minimum value: 1, maximum value: 16, default value: 4. - -TAB_SIZE = 2 - -# This tag can be used to specify a number of aliases that act as commands in -# the documentation. An alias has the form: -# name=value -# For example adding -# "sideeffect=@par Side Effects:\n" -# will allow you to put the command \sideeffect (or @sideeffect) in the -# documentation, which will result in a user-defined paragraph with heading -# "Side Effects:". You can put \n's in the value part of an alias to insert -# newlines. - -ALIASES = - -# This tag can be used to specify a number of word-keyword mappings (TCL only). -# A mapping has the form "name=value". For example adding "class=itcl::class" -# will allow you to use the command class in the itcl::class meaning. - -TCL_SUBST = - -# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C sources -# only. Doxygen will then generate output that is more tailored for C. For -# instance, some of the names that are used will be different. The list of all -# members will be omitted, etc. -# The default value is: NO. - -OPTIMIZE_OUTPUT_FOR_C = YES - -# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java or -# Python sources only. Doxygen will then generate output that is more tailored -# for that language. For instance, namespaces will be presented as packages, -# qualified scopes will look different, etc. -# The default value is: NO. - -OPTIMIZE_OUTPUT_JAVA = NO - -# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran -# sources. Doxygen will then generate output that is tailored for Fortran. -# The default value is: NO. - -OPTIMIZE_FOR_FORTRAN = NO - -# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL -# sources. Doxygen will then generate output that is tailored for VHDL. -# The default value is: NO. - -OPTIMIZE_OUTPUT_VHDL = NO - -# Doxygen selects the parser to use depending on the extension of the files it -# parses. With this tag you can assign which parser to use for a given -# extension. Doxygen has a built-in mapping, but you can override or extend it -# using this tag. The format is ext=language, where ext is a file extension, and -# language is one of the parsers supported by doxygen: IDL, Java, Javascript, -# C#, C, C++, D, PHP, Objective-C, Python, Fortran (fixed format Fortran: -# FortranFixed, free formatted Fortran: FortranFree, unknown formatted Fortran: -# Fortran. In the later case the parser tries to guess whether the code is fixed -# or free formatted code, this is the default for Fortran type files), VHDL. For -# instance to make doxygen treat .inc files as Fortran files (default is PHP), -# and .f files as C (default is Fortran), use: inc=Fortran f=C. -# -# Note: For files without extension you can use no_extension as a placeholder. -# -# Note that for custom extensions you also need to set FILE_PATTERNS otherwise -# the files are not read by doxygen. - -EXTENSION_MAPPING = - -# If the MARKDOWN_SUPPORT tag is enabled then doxygen pre-processes all comments -# according to the Markdown format, which allows for more readable -# documentation. See http://daringfireball.net/projects/markdown/ for details. -# The output of markdown processing is further processed by doxygen, so you can -# mix doxygen, HTML, and XML commands with Markdown formatting. Disable only in -# case of backward compatibilities issues. -# The default value is: YES. - -MARKDOWN_SUPPORT = YES - -# When enabled doxygen tries to link words that correspond to documented -# classes, or namespaces to their corresponding documentation. Such a link can -# be prevented in individual cases by putting a % sign in front of the word or -# globally by setting AUTOLINK_SUPPORT to NO. -# The default value is: YES. - -AUTOLINK_SUPPORT = YES - -# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want -# to include (a tag file for) the STL sources as input, then you should set this -# tag to YES in order to let doxygen match functions declarations and -# definitions whose arguments contain STL classes (e.g. func(std::string); -# versus func(std::string) {}). This also make the inheritance and collaboration -# diagrams that involve STL classes more complete and accurate. -# The default value is: NO. - -BUILTIN_STL_SUPPORT = NO - -# If you use Microsoft's C++/CLI language, you should set this option to YES to -# enable parsing support. -# The default value is: NO. - -CPP_CLI_SUPPORT = NO - -# Set the SIP_SUPPORT tag to YES if your project consists of sip (see: -# http://www.riverbankcomputing.co.uk/software/sip/intro) sources only. Doxygen -# will parse them like normal C++ but will assume all classes use public instead -# of private inheritance when no explicit protection keyword is present. -# The default value is: NO. - -SIP_SUPPORT = NO - -# For Microsoft's IDL there are propget and propput attributes to indicate -# getter and setter methods for a property. Setting this option to YES will make -# doxygen to replace the get and set methods by a property in the documentation. -# This will only work if the methods are indeed getting or setting a simple -# type. If this is not the case, or you want to show the methods anyway, you -# should set this option to NO. -# The default value is: YES. - -IDL_PROPERTY_SUPPORT = YES - -# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC -# tag is set to YES then doxygen will reuse the documentation of the first -# member in the group (if any) for the other members of the group. By default -# all members of a group must be documented explicitly. -# The default value is: NO. - -DISTRIBUTE_GROUP_DOC = NO - -# Set the SUBGROUPING tag to YES to allow class member groups of the same type -# (for instance a group of public functions) to be put as a subgroup of that -# type (e.g. under the Public Functions section). Set it to NO to prevent -# subgrouping. Alternatively, this can be done per class using the -# \nosubgrouping command. -# The default value is: YES. - -SUBGROUPING = YES - -# When the INLINE_GROUPED_CLASSES tag is set to YES, classes, structs and unions -# are shown inside the group in which they are included (e.g. using \ingroup) -# instead of on a separate page (for HTML and Man pages) or section (for LaTeX -# and RTF). -# -# Note that this feature does not work in combination with -# SEPARATE_MEMBER_PAGES. -# The default value is: NO. - -INLINE_GROUPED_CLASSES = NO - -# When the INLINE_SIMPLE_STRUCTS tag is set to YES, structs, classes, and unions -# with only public data fields or simple typedef fields will be shown inline in -# the documentation of the scope in which they are defined (i.e. file, -# namespace, or group documentation), provided this scope is documented. If set -# to NO, structs, classes, and unions are shown on a separate page (for HTML and -# Man pages) or section (for LaTeX and RTF). -# The default value is: NO. - -INLINE_SIMPLE_STRUCTS = NO - -# When TYPEDEF_HIDES_STRUCT tag is enabled, a typedef of a struct, union, or -# enum is documented as struct, union, or enum with the name of the typedef. So -# typedef struct TypeS {} TypeT, will appear in the documentation as a struct -# with name TypeT. When disabled the typedef will appear as a member of a file, -# namespace, or class. And the struct will be named TypeS. This can typically be -# useful for C code in case the coding convention dictates that all compound -# types are typedef'ed and only the typedef is referenced, never the tag name. -# The default value is: NO. - -TYPEDEF_HIDES_STRUCT = NO - -# The size of the symbol lookup cache can be set using LOOKUP_CACHE_SIZE. This -# cache is used to resolve symbols given their name and scope. Since this can be -# an expensive process and often the same symbol appears multiple times in the -# code, doxygen keeps a cache of pre-resolved symbols. If the cache is too small -# doxygen will become slower. If the cache is too large, memory is wasted. The -# cache size is given by this formula: 2^(16+LOOKUP_CACHE_SIZE). The valid range -# is 0..9, the default is 0, corresponding to a cache size of 2^16=65536 -# symbols. At the end of a run doxygen will report the cache usage and suggest -# the optimal cache size from a speed point of view. -# Minimum value: 0, maximum value: 9, default value: 0. - -LOOKUP_CACHE_SIZE = 0 - -#--------------------------------------------------------------------------- -# Build related configuration options -#--------------------------------------------------------------------------- - -# If the EXTRACT_ALL tag is set to YES, doxygen will assume all entities in -# documentation are documented, even if no documentation was available. Private -# class members and static file members will be hidden unless the -# EXTRACT_PRIVATE respectively EXTRACT_STATIC tags are set to YES. -# Note: This will also disable the warnings about undocumented members that are -# normally produced when WARNINGS is set to YES. -# The default value is: NO. - -EXTRACT_ALL = YES - -# If the EXTRACT_PRIVATE tag is set to YES, all private members of a class will -# be included in the documentation. -# The default value is: NO. - -EXTRACT_PRIVATE = NO - -# If the EXTRACT_PACKAGE tag is set to YES, all members with package or internal -# scope will be included in the documentation. -# The default value is: NO. - -EXTRACT_PACKAGE = NO - -# If the EXTRACT_STATIC tag is set to YES, all static members of a file will be -# included in the documentation. -# The default value is: NO. - -EXTRACT_STATIC = NO - -# If the EXTRACT_LOCAL_CLASSES tag is set to YES, classes (and structs) defined -# locally in source files will be included in the documentation. If set to NO, -# only classes defined in header files are included. Does not have any effect -# for Java sources. -# The default value is: YES. - -EXTRACT_LOCAL_CLASSES = YES - -# This flag is only useful for Objective-C code. If set to YES, local methods, -# which are defined in the implementation section but not in the interface are -# included in the documentation. If set to NO, only methods in the interface are -# included. -# The default value is: NO. - -EXTRACT_LOCAL_METHODS = NO - -# If this flag is set to YES, the members of anonymous namespaces will be -# extracted and appear in the documentation as a namespace called -# 'anonymous_namespace{file}', where file will be replaced with the base name of -# the file that contains the anonymous namespace. By default anonymous namespace -# are hidden. -# The default value is: NO. - -EXTRACT_ANON_NSPACES = NO - -# If the HIDE_UNDOC_MEMBERS tag is set to YES, doxygen will hide all -# undocumented members inside documented classes or files. If set to NO these -# members will be included in the various overviews, but no documentation -# section is generated. This option has no effect if EXTRACT_ALL is enabled. -# The default value is: NO. - -HIDE_UNDOC_MEMBERS = NO - -# If the HIDE_UNDOC_CLASSES tag is set to YES, doxygen will hide all -# undocumented classes that are normally visible in the class hierarchy. If set -# to NO, these classes will be included in the various overviews. This option -# has no effect if EXTRACT_ALL is enabled. -# The default value is: NO. - -HIDE_UNDOC_CLASSES = NO - -# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, doxygen will hide all friend -# (class|struct|union) declarations. If set to NO, these declarations will be -# included in the documentation. -# The default value is: NO. - -HIDE_FRIEND_COMPOUNDS = NO - -# If the HIDE_IN_BODY_DOCS tag is set to YES, doxygen will hide any -# documentation blocks found inside the body of a function. If set to NO, these -# blocks will be appended to the function's detailed documentation block. -# The default value is: NO. - -HIDE_IN_BODY_DOCS = NO - -# The INTERNAL_DOCS tag determines if documentation that is typed after a -# \internal command is included. If the tag is set to NO then the documentation -# will be excluded. Set it to YES to include the internal documentation. -# The default value is: NO. - -INTERNAL_DOCS = NO - -# If the CASE_SENSE_NAMES tag is set to NO then doxygen will only generate file -# names in lower-case letters. If set to YES, upper-case letters are also -# allowed. This is useful if you have classes or files whose names only differ -# in case and if your file system supports case sensitive file names. Windows -# and Mac users are advised to set this option to NO. -# The default value is: system dependent. - -CASE_SENSE_NAMES = NO - -# If the HIDE_SCOPE_NAMES tag is set to NO then doxygen will show members with -# their full class and namespace scopes in the documentation. If set to YES, the -# scope will be hidden. -# The default value is: NO. - -HIDE_SCOPE_NAMES = NO - -# If the HIDE_COMPOUND_REFERENCE tag is set to NO (default) then doxygen will -# append additional text to a page's title, such as Class Reference. If set to -# YES the compound reference will be hidden. -# The default value is: NO. - -HIDE_COMPOUND_REFERENCE= NO - -# If the SHOW_INCLUDE_FILES tag is set to YES then doxygen will put a list of -# the files that are included by a file in the documentation of that file. -# The default value is: YES. - -SHOW_INCLUDE_FILES = YES - -# If the SHOW_GROUPED_MEMB_INC tag is set to YES then Doxygen will add for each -# grouped member an include statement to the documentation, telling the reader -# which file to include in order to use the member. -# The default value is: NO. - -SHOW_GROUPED_MEMB_INC = NO - -# If the FORCE_LOCAL_INCLUDES tag is set to YES then doxygen will list include -# files with double quotes in the documentation rather than with sharp brackets. -# The default value is: NO. - -FORCE_LOCAL_INCLUDES = NO - -# If the INLINE_INFO tag is set to YES then a tag [inline] is inserted in the -# documentation for inline members. -# The default value is: YES. - -INLINE_INFO = YES - -# If the SORT_MEMBER_DOCS tag is set to YES then doxygen will sort the -# (detailed) documentation of file and class members alphabetically by member -# name. If set to NO, the members will appear in declaration order. -# The default value is: YES. - -SORT_MEMBER_DOCS = YES - -# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the brief -# descriptions of file, namespace and class members alphabetically by member -# name. If set to NO, the members will appear in declaration order. Note that -# this will also influence the order of the classes in the class list. -# The default value is: NO. - -SORT_BRIEF_DOCS = NO - -# If the SORT_MEMBERS_CTORS_1ST tag is set to YES then doxygen will sort the -# (brief and detailed) documentation of class members so that constructors and -# destructors are listed first. If set to NO the constructors will appear in the -# respective orders defined by SORT_BRIEF_DOCS and SORT_MEMBER_DOCS. -# Note: If SORT_BRIEF_DOCS is set to NO this option is ignored for sorting brief -# member documentation. -# Note: If SORT_MEMBER_DOCS is set to NO this option is ignored for sorting -# detailed member documentation. -# The default value is: NO. - -SORT_MEMBERS_CTORS_1ST = NO - -# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the hierarchy -# of group names into alphabetical order. If set to NO the group names will -# appear in their defined order. -# The default value is: NO. - -SORT_GROUP_NAMES = NO - -# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be sorted by -# fully-qualified names, including namespaces. If set to NO, the class list will -# be sorted only by class name, not including the namespace part. -# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES. -# Note: This option applies only to the class list, not to the alphabetical -# list. -# The default value is: NO. - -SORT_BY_SCOPE_NAME = NO - -# If the STRICT_PROTO_MATCHING option is enabled and doxygen fails to do proper -# type resolution of all parameters of a function it will reject a match between -# the prototype and the implementation of a member function even if there is -# only one candidate or it is obvious which candidate to choose by doing a -# simple string match. By disabling STRICT_PROTO_MATCHING doxygen will still -# accept a match between prototype and implementation in such cases. -# The default value is: NO. - -STRICT_PROTO_MATCHING = NO - -# The GENERATE_TODOLIST tag can be used to enable (YES) or disable (NO) the todo -# list. This list is created by putting \todo commands in the documentation. -# The default value is: YES. - -GENERATE_TODOLIST = YES - -# The GENERATE_TESTLIST tag can be used to enable (YES) or disable (NO) the test -# list. This list is created by putting \test commands in the documentation. -# The default value is: YES. - -GENERATE_TESTLIST = YES - -# The GENERATE_BUGLIST tag can be used to enable (YES) or disable (NO) the bug -# list. This list is created by putting \bug commands in the documentation. -# The default value is: YES. - -GENERATE_BUGLIST = YES - -# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or disable (NO) -# the deprecated list. This list is created by putting \deprecated commands in -# the documentation. -# The default value is: YES. - -GENERATE_DEPRECATEDLIST= YES - -# The ENABLED_SECTIONS tag can be used to enable conditional documentation -# sections, marked by \if ... \endif and \cond -# ... \endcond blocks. - -ENABLED_SECTIONS = - -# The MAX_INITIALIZER_LINES tag determines the maximum number of lines that the -# initial value of a variable or macro / define can have for it to appear in the -# documentation. If the initializer consists of more lines than specified here -# it will be hidden. Use a value of 0 to hide initializers completely. The -# appearance of the value of individual variables and macros / defines can be -# controlled using \showinitializer or \hideinitializer command in the -# documentation regardless of this setting. -# Minimum value: 0, maximum value: 10000, default value: 30. - -MAX_INITIALIZER_LINES = 30 - -# Set the SHOW_USED_FILES tag to NO to disable the list of files generated at -# the bottom of the documentation of classes and structs. If set to YES, the -# list will mention the files that were used to generate the documentation. -# The default value is: YES. - -SHOW_USED_FILES = YES - -# Set the SHOW_FILES tag to NO to disable the generation of the Files page. This -# will remove the Files entry from the Quick Index and from the Folder Tree View -# (if specified). -# The default value is: YES. - -SHOW_FILES = YES - -# Set the SHOW_NAMESPACES tag to NO to disable the generation of the Namespaces -# page. This will remove the Namespaces entry from the Quick Index and from the -# Folder Tree View (if specified). -# The default value is: YES. - -SHOW_NAMESPACES = YES - -# The FILE_VERSION_FILTER tag can be used to specify a program or script that -# doxygen should invoke to get the current version for each file (typically from -# the version control system). Doxygen will invoke the program by executing (via -# popen()) the command command input-file, where command is the value of the -# FILE_VERSION_FILTER tag, and input-file is the name of an input file provided -# by doxygen. Whatever the program writes to standard output is used as the file -# version. For an example see the documentation. - -FILE_VERSION_FILTER = - -# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed -# by doxygen. The layout file controls the global structure of the generated -# output files in an output format independent way. To create the layout file -# that represents doxygen's defaults, run doxygen with the -l option. You can -# optionally specify a file name after the option, if omitted DoxygenLayout.xml -# will be used as the name of the layout file. -# -# Note that if you run doxygen from a directory containing a file called -# DoxygenLayout.xml, doxygen will parse it automatically even if the LAYOUT_FILE -# tag is left empty. - -LAYOUT_FILE = - -# The CITE_BIB_FILES tag can be used to specify one or more bib files containing -# the reference definitions. This must be a list of .bib files. The .bib -# extension is automatically appended if omitted. This requires the bibtex tool -# to be installed. See also http://en.wikipedia.org/wiki/BibTeX for more info. -# For LaTeX the style of the bibliography can be controlled using -# LATEX_BIB_STYLE. To use this feature you need bibtex and perl available in the -# search path. See also \cite for info how to create references. - -CITE_BIB_FILES = - -#--------------------------------------------------------------------------- -# Configuration options related to warning and progress messages -#--------------------------------------------------------------------------- - -# The QUIET tag can be used to turn on/off the messages that are generated to -# standard output by doxygen. If QUIET is set to YES this implies that the -# messages are off. -# The default value is: NO. - -QUIET = NO - -# The WARNINGS tag can be used to turn on/off the warning messages that are -# generated to standard error (stderr) by doxygen. If WARNINGS is set to YES -# this implies that the warnings are on. -# -# Tip: Turn warnings on while writing the documentation. -# The default value is: YES. - -WARNINGS = YES - -# If the WARN_IF_UNDOCUMENTED tag is set to YES then doxygen will generate -# warnings for undocumented members. If EXTRACT_ALL is set to YES then this flag -# will automatically be disabled. -# The default value is: YES. - -WARN_IF_UNDOCUMENTED = YES - -# If the WARN_IF_DOC_ERROR tag is set to YES, doxygen will generate warnings for -# potential errors in the documentation, such as not documenting some parameters -# in a documented function, or documenting parameters that don't exist or using -# markup commands wrongly. -# The default value is: YES. - -WARN_IF_DOC_ERROR = YES - -# This WARN_NO_PARAMDOC option can be enabled to get warnings for functions that -# are documented, but have no documentation for their parameters or return -# value. If set to NO, doxygen will only warn about wrong or incomplete -# parameter documentation, but not about the absence of documentation. -# The default value is: NO. - -WARN_NO_PARAMDOC = NO - -# The WARN_FORMAT tag determines the format of the warning messages that doxygen -# can produce. The string should contain the $file, $line, and $text tags, which -# will be replaced by the file and line number from which the warning originated -# and the warning text. Optionally the format may contain $version, which will -# be replaced by the version of the file (if it could be obtained via -# FILE_VERSION_FILTER) -# The default value is: $file:$line: $text. - -WARN_FORMAT = "$file:$line: $text" - -# The WARN_LOGFILE tag can be used to specify a file to which warning and error -# messages should be written. If left blank the output is written to standard -# error (stderr). - -WARN_LOGFILE = - -#--------------------------------------------------------------------------- -# Configuration options related to the input files -#--------------------------------------------------------------------------- - -# The INPUT tag is used to specify the files and/or directories that contain -# documented source files. You may enter file names like myfile.cpp or -# directories like /usr/src/myproject. Separate the files or directories with -# spaces. -# Note: If this tag is empty the current directory is searched. - -INPUT = include/grpc++/alarm.h \ -include/grpc++/channel.h \ -include/grpc++/client_context.h \ -include/grpc++/completion_queue.h \ -include/grpc++/create_channel.h \ -include/grpc++/generic/async_generic_service.h \ -include/grpc++/generic/generic_stub.h \ -include/grpc++/grpc++.h \ -include/grpc++/impl/call.h \ -include/grpc++/impl/client_unary_call.h \ -include/grpc++/impl/codegen/core_codegen.h \ -include/grpc++/impl/grpc_library.h \ -include/grpc++/impl/method_handler_impl.h \ -include/grpc++/impl/rpc_method.h \ -include/grpc++/impl/rpc_service_method.h \ -include/grpc++/impl/serialization_traits.h \ -include/grpc++/impl/server_builder_option.h \ -include/grpc++/impl/server_builder_plugin.h \ -include/grpc++/impl/server_initializer.h \ -include/grpc++/impl/service_type.h \ -include/grpc++/impl/sync.h \ -include/grpc++/impl/sync_cxx11.h \ -include/grpc++/impl/sync_no_cxx11.h \ -include/grpc++/impl/thd.h \ -include/grpc++/impl/thd_cxx11.h \ -include/grpc++/impl/thd_no_cxx11.h \ -include/grpc++/security/auth_context.h \ -include/grpc++/security/auth_metadata_processor.h \ -include/grpc++/security/credentials.h \ -include/grpc++/security/server_credentials.h \ -include/grpc++/server.h \ -include/grpc++/server_builder.h \ -include/grpc++/server_context.h \ -include/grpc++/support/async_stream.h \ -include/grpc++/support/async_unary_call.h \ -include/grpc++/support/byte_buffer.h \ -include/grpc++/support/channel_arguments.h \ -include/grpc++/support/config.h \ -include/grpc++/support/slice.h \ -include/grpc++/support/status.h \ -include/grpc++/support/status_code_enum.h \ -include/grpc++/support/string_ref.h \ -include/grpc++/support/stub_options.h \ -include/grpc++/support/sync_stream.h \ -include/grpc++/support/time.h \ -include/grpc++/impl/codegen/async_stream.h \ -include/grpc++/impl/codegen/async_unary_call.h \ -include/grpc++/impl/codegen/call.h \ -include/grpc++/impl/codegen/call_hook.h \ -include/grpc++/impl/codegen/channel_interface.h \ -include/grpc++/impl/codegen/client_context.h \ -include/grpc++/impl/codegen/client_unary_call.h \ -include/grpc++/impl/codegen/completion_queue.h \ -include/grpc++/impl/codegen/completion_queue_tag.h \ -include/grpc++/impl/codegen/config.h \ -include/grpc++/impl/codegen/core_codegen_interface.h \ -include/grpc++/impl/codegen/create_auth_context.h \ -include/grpc++/impl/codegen/grpc_library.h \ -include/grpc++/impl/codegen/method_handler_impl.h \ -include/grpc++/impl/codegen/rpc_method.h \ -include/grpc++/impl/codegen/rpc_service_method.h \ -include/grpc++/impl/codegen/security/auth_context.h \ -include/grpc++/impl/codegen/serialization_traits.h \ -include/grpc++/impl/codegen/server_context.h \ -include/grpc++/impl/codegen/server_interface.h \ -include/grpc++/impl/codegen/service_type.h \ -include/grpc++/impl/codegen/status.h \ -include/grpc++/impl/codegen/status_code_enum.h \ -include/grpc++/impl/codegen/string_ref.h \ -include/grpc++/impl/codegen/stub_options.h \ -include/grpc++/impl/codegen/sync.h \ -include/grpc++/impl/codegen/sync_cxx11.h \ -include/grpc++/impl/codegen/sync_no_cxx11.h \ -include/grpc++/impl/codegen/sync_stream.h \ -include/grpc++/impl/codegen/time.h \ -include/grpc/impl/codegen/byte_buffer.h \ -include/grpc/impl/codegen/byte_buffer_reader.h \ -include/grpc/impl/codegen/compression_types.h \ -include/grpc/impl/codegen/connectivity_state.h \ -include/grpc/impl/codegen/grpc_types.h \ -include/grpc/impl/codegen/propagation_bits.h \ -include/grpc/impl/codegen/status.h \ -include/grpc/impl/codegen/alloc.h \ -include/grpc/impl/codegen/atm.h \ -include/grpc/impl/codegen/atm_gcc_atomic.h \ -include/grpc/impl/codegen/atm_gcc_sync.h \ -include/grpc/impl/codegen/atm_windows.h \ -include/grpc/impl/codegen/log.h \ -include/grpc/impl/codegen/port_platform.h \ -include/grpc/impl/codegen/slice.h \ -include/grpc/impl/codegen/slice_buffer.h \ -include/grpc/impl/codegen/sync.h \ -include/grpc/impl/codegen/sync_generic.h \ -include/grpc/impl/codegen/sync_posix.h \ -include/grpc/impl/codegen/sync_windows.h \ -include/grpc/impl/codegen/time.h \ -<<<<<<< HEAD -include/grpc++/impl/codegen/config.h \ -include/grpc++/impl/codegen/config_protobuf.h \ -include/grpc++/support/config.h \ -include/grpc++/support/config_protobuf.h \ -include/grpc++/impl/codegen/core_codegen.h \ -======= ->>>>>>> d30d4e279c4a63effaa6e912fc00bd4ad96054c7 -src/cpp/client/secure_credentials.h \ -src/cpp/common/secure_auth_context.h \ -src/cpp/server/secure_server_credentials.h \ -src/cpp/client/create_channel_internal.h \ -src/cpp/server/dynamic_thread_pool.h \ -src/cpp/server/thread_pool_interface.h \ -src/cpp/client/secure_credentials.cc \ -src/cpp/common/auth_property_iterator.cc \ -src/cpp/common/secure_auth_context.cc \ -src/cpp/common/secure_channel_arguments.cc \ -src/cpp/common/secure_create_auth_context.cc \ -src/cpp/server/secure_server_credentials.cc \ -src/cpp/client/channel.cc \ -src/cpp/client/client_context.cc \ -src/cpp/client/create_channel.cc \ -src/cpp/client/create_channel_internal.cc \ -src/cpp/client/credentials.cc \ -src/cpp/client/generic_stub.cc \ -src/cpp/client/insecure_credentials.cc \ -src/cpp/common/channel_arguments.cc \ -src/cpp/common/completion_queue.cc \ -src/cpp/common/core_codegen.cc \ -src/cpp/common/rpc_method.cc \ -src/cpp/server/async_generic_service.cc \ -src/cpp/server/create_default_thread_pool.cc \ -src/cpp/server/dynamic_thread_pool.cc \ -src/cpp/server/insecure_server_credentials.cc \ -src/cpp/server/server.cc \ -src/cpp/server/server_builder.cc \ -src/cpp/server/server_context.cc \ -src/cpp/server/server_credentials.cc \ -src/cpp/util/byte_buffer.cc \ -src/cpp/util/slice.cc \ -src/cpp/util/status.cc \ -src/cpp/util/string_ref.cc \ -src/cpp/util/time.cc \ -src/cpp/codegen/codegen_init.cc - -# This tag can be used to specify the character encoding of the source files -# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses -# libiconv (or the iconv built into libc) for the transcoding. See the libiconv -# documentation (see: http://www.gnu.org/software/libiconv) for the list of -# possible encodings. -# The default value is: UTF-8. - -INPUT_ENCODING = UTF-8 - -# If the value of the INPUT tag contains directories, you can use the -# FILE_PATTERNS tag to specify one or more wildcard patterns (like *.cpp and -# *.h) to filter out the source-files in the directories. If left blank the -# following patterns are tested:*.c, *.cc, *.cxx, *.cpp, *.c++, *.java, *.ii, -# *.ixx, *.ipp, *.i++, *.inl, *.idl, *.ddl, *.odl, *.h, *.hh, *.hxx, *.hpp, -# *.h++, *.cs, *.d, *.php, *.php4, *.php5, *.phtml, *.inc, *.m, *.markdown, -# *.md, *.mm, *.dox, *.py, *.f90, *.f, *.for, *.tcl, *.vhd, *.vhdl, *.ucf, -# *.qsf, *.as and *.js. - -FILE_PATTERNS = - -# The RECURSIVE tag can be used to specify whether or not subdirectories should -# be searched for input files as well. -# The default value is: NO. - -RECURSIVE = NO - -# The EXCLUDE tag can be used to specify files and/or directories that should be -# excluded from the INPUT source files. This way you can easily exclude a -# subdirectory from a directory tree whose root is specified with the INPUT tag. -# -# Note that relative paths are relative to the directory from which doxygen is -# run. - -EXCLUDE = - -# The EXCLUDE_SYMLINKS tag can be used to select whether or not files or -# directories that are symbolic links (a Unix file system feature) are excluded -# from the input. -# The default value is: NO. - -EXCLUDE_SYMLINKS = NO - -# If the value of the INPUT tag contains directories, you can use the -# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude -# certain files from those directories. -# -# Note that the wildcards are matched against the file with absolute path, so to -# exclude all test directories for example use the pattern */test/* - -EXCLUDE_PATTERNS = - -# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names -# (namespaces, classes, functions, etc.) that should be excluded from the -# output. The symbol name can be a fully qualified name, a word, or if the -# wildcard * is used, a substring. Examples: ANamespace, AClass, -# AClass::ANamespace, ANamespace::*Test -# -# Note that the wildcards are matched against the file with absolute path, so to -# exclude all test directories use the pattern */test/* - -EXCLUDE_SYMBOLS = - -# The EXAMPLE_PATH tag can be used to specify one or more files or directories -# that contain example code fragments that are included (see the \include -# command). - -EXAMPLE_PATH = - -# If the value of the EXAMPLE_PATH tag contains directories, you can use the -# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp and -# *.h) to filter out the source-files in the directories. If left blank all -# files are included. - -EXAMPLE_PATTERNS = - -# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be -# searched for input files to be used with the \include or \dontinclude commands -# irrespective of the value of the RECURSIVE tag. -# The default value is: NO. - -EXAMPLE_RECURSIVE = NO - -# The IMAGE_PATH tag can be used to specify one or more files or directories -# that contain images that are to be included in the documentation (see the -# \image command). - -IMAGE_PATH = - -# The INPUT_FILTER tag can be used to specify a program that doxygen should -# invoke to filter for each input file. Doxygen will invoke the filter program -# by executing (via popen()) the command: -# -# -# -# where is the value of the INPUT_FILTER tag, and is the -# name of an input file. Doxygen will then use the output that the filter -# program writes to standard output. If FILTER_PATTERNS is specified, this tag -# will be ignored. -# -# Note that the filter must not add or remove lines; it is applied before the -# code is scanned, but not when the output code is generated. If lines are added -# or removed, the anchors will not be placed correctly. - -INPUT_FILTER = - -# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern -# basis. Doxygen will compare the file name with each pattern and apply the -# filter if there is a match. The filters are a list of the form: pattern=filter -# (like *.cpp=my_cpp_filter). See INPUT_FILTER for further information on how -# filters are used. If the FILTER_PATTERNS tag is empty or if none of the -# patterns match the file name, INPUT_FILTER is applied. - -FILTER_PATTERNS = - -# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using -# INPUT_FILTER) will also be used to filter the input files that are used for -# producing the source files to browse (i.e. when SOURCE_BROWSER is set to YES). -# The default value is: NO. - -FILTER_SOURCE_FILES = NO - -# The FILTER_SOURCE_PATTERNS tag can be used to specify source filters per file -# pattern. A pattern will override the setting for FILTER_PATTERN (if any) and -# it is also possible to disable source filtering for a specific pattern using -# *.ext= (so without naming a filter). -# This tag requires that the tag FILTER_SOURCE_FILES is set to YES. - -FILTER_SOURCE_PATTERNS = - -# If the USE_MDFILE_AS_MAINPAGE tag refers to the name of a markdown file that -# is part of the input, its contents will be placed on the main page -# (index.html). This can be useful if you have a project on for instance GitHub -# and want to reuse the introduction page also for the doxygen output. - -USE_MDFILE_AS_MAINPAGE = - -#--------------------------------------------------------------------------- -# Configuration options related to source browsing -#--------------------------------------------------------------------------- - -# If the SOURCE_BROWSER tag is set to YES then a list of source files will be -# generated. Documented entities will be cross-referenced with these sources. -# -# Note: To get rid of all source code in the generated output, make sure that -# also VERBATIM_HEADERS is set to NO. -# The default value is: NO. - -SOURCE_BROWSER = NO - -# Setting the INLINE_SOURCES tag to YES will include the body of functions, -# classes and enums directly into the documentation. -# The default value is: NO. - -INLINE_SOURCES = NO - -# Setting the STRIP_CODE_COMMENTS tag to YES will instruct doxygen to hide any -# special comment blocks from generated source code fragments. Normal C, C++ and -# Fortran comments will always remain visible. -# The default value is: YES. - -STRIP_CODE_COMMENTS = YES - -# If the REFERENCED_BY_RELATION tag is set to YES then for each documented -# function all documented functions referencing it will be listed. -# The default value is: NO. - -REFERENCED_BY_RELATION = NO - -# If the REFERENCES_RELATION tag is set to YES then for each documented function -# all documented entities called/used by that function will be listed. -# The default value is: NO. - -REFERENCES_RELATION = NO - -# If the REFERENCES_LINK_SOURCE tag is set to YES and SOURCE_BROWSER tag is set -# to YES then the hyperlinks from functions in REFERENCES_RELATION and -# REFERENCED_BY_RELATION lists will link to the source code. Otherwise they will -# link to the documentation. -# The default value is: YES. - -REFERENCES_LINK_SOURCE = YES - -# If SOURCE_TOOLTIPS is enabled (the default) then hovering a hyperlink in the -# source code will show a tooltip with additional information such as prototype, -# brief description and links to the definition and documentation. Since this -# will make the HTML file larger and loading of large files a bit slower, you -# can opt to disable this feature. -# The default value is: YES. -# This tag requires that the tag SOURCE_BROWSER is set to YES. - -SOURCE_TOOLTIPS = YES - -# If the USE_HTAGS tag is set to YES then the references to source code will -# point to the HTML generated by the htags(1) tool instead of doxygen built-in -# source browser. The htags tool is part of GNU's global source tagging system -# (see http://www.gnu.org/software/global/global.html). You will need version -# 4.8.6 or higher. -# -# To use it do the following: -# - Install the latest version of global -# - Enable SOURCE_BROWSER and USE_HTAGS in the config file -# - Make sure the INPUT points to the root of the source tree -# - Run doxygen as normal -# -# Doxygen will invoke htags (and that will in turn invoke gtags), so these -# tools must be available from the command line (i.e. in the search path). -# -# The result: instead of the source browser generated by doxygen, the links to -# source code will now point to the output of htags. -# The default value is: NO. -# This tag requires that the tag SOURCE_BROWSER is set to YES. - -USE_HTAGS = NO - -# If the VERBATIM_HEADERS tag is set the YES then doxygen will generate a -# verbatim copy of the header file for each class for which an include is -# specified. Set to NO to disable this. -# See also: Section \class. -# The default value is: YES. - -VERBATIM_HEADERS = YES - -#--------------------------------------------------------------------------- -# Configuration options related to the alphabetical class index -#--------------------------------------------------------------------------- - -# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index of all -# compounds will be generated. Enable this if the project contains a lot of -# classes, structs, unions or interfaces. -# The default value is: YES. - -ALPHABETICAL_INDEX = YES - -# The COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns in -# which the alphabetical index list will be split. -# Minimum value: 1, maximum value: 20, default value: 5. -# This tag requires that the tag ALPHABETICAL_INDEX is set to YES. - -COLS_IN_ALPHA_INDEX = 5 - -# In case all classes in a project start with a common prefix, all classes will -# be put under the same header in the alphabetical index. The IGNORE_PREFIX tag -# can be used to specify a prefix (or a list of prefixes) that should be ignored -# while generating the index headers. -# This tag requires that the tag ALPHABETICAL_INDEX is set to YES. - -IGNORE_PREFIX = - -#--------------------------------------------------------------------------- -# Configuration options related to the HTML output -#--------------------------------------------------------------------------- - -# If the GENERATE_HTML tag is set to YES, doxygen will generate HTML output -# The default value is: YES. - -GENERATE_HTML = YES - -# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. If a -# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of -# it. -# The default directory is: html. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_OUTPUT = html - -# The HTML_FILE_EXTENSION tag can be used to specify the file extension for each -# generated HTML page (for example: .htm, .php, .asp). -# The default value is: .html. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_FILE_EXTENSION = .html - -# The HTML_HEADER tag can be used to specify a user-defined HTML header file for -# each generated HTML page. If the tag is left blank doxygen will generate a -# standard header. -# -# To get valid HTML the header file that includes any scripts and style sheets -# that doxygen needs, which is dependent on the configuration options used (e.g. -# the setting GENERATE_TREEVIEW). It is highly recommended to start with a -# default header using -# doxygen -w html new_header.html new_footer.html new_stylesheet.css -# YourConfigFile -# and then modify the file new_header.html. See also section "Doxygen usage" -# for information on how to generate the default header that doxygen normally -# uses. -# Note: The header is subject to change so you typically have to regenerate the -# default header when upgrading to a newer version of doxygen. For a description -# of the possible markers and block names see the documentation. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_HEADER = - -# The HTML_FOOTER tag can be used to specify a user-defined HTML footer for each -# generated HTML page. If the tag is left blank doxygen will generate a standard -# footer. See HTML_HEADER for more information on how to generate a default -# footer and what special commands can be used inside the footer. See also -# section "Doxygen usage" for information on how to generate the default footer -# that doxygen normally uses. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_FOOTER = - -# The HTML_STYLESHEET tag can be used to specify a user-defined cascading style -# sheet that is used by each HTML page. It can be used to fine-tune the look of -# the HTML output. If left blank doxygen will generate a default style sheet. -# See also section "Doxygen usage" for information on how to generate the style -# sheet that doxygen normally uses. -# Note: It is recommended to use HTML_EXTRA_STYLESHEET instead of this tag, as -# it is more robust and this tag (HTML_STYLESHEET) will in the future become -# obsolete. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_STYLESHEET = - -# The HTML_EXTRA_STYLESHEET tag can be used to specify additional user-defined -# cascading style sheets that are included after the standard style sheets -# created by doxygen. Using this option one can overrule certain style aspects. -# This is preferred over using HTML_STYLESHEET since it does not replace the -# standard style sheet and is therefore more robust against future updates. -# Doxygen will copy the style sheet files to the output directory. -# Note: The order of the extra style sheet files is of importance (e.g. the last -# style sheet in the list overrules the setting of the previous ones in the -# list). For an example see the documentation. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_EXTRA_STYLESHEET = - -# The HTML_EXTRA_FILES tag can be used to specify one or more extra images or -# other source files which should be copied to the HTML output directory. Note -# that these files will be copied to the base HTML output directory. Use the -# $relpath^ marker in the HTML_HEADER and/or HTML_FOOTER files to load these -# files. In the HTML_STYLESHEET file, use the file name only. Also note that the -# files will be copied as-is; there are no commands or markers available. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_EXTRA_FILES = - -# The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. Doxygen -# will adjust the colors in the style sheet and background images according to -# this color. Hue is specified as an angle on a colorwheel, see -# http://en.wikipedia.org/wiki/Hue for more information. For instance the value -# 0 represents red, 60 is yellow, 120 is green, 180 is cyan, 240 is blue, 300 -# purple, and 360 is red again. -# Minimum value: 0, maximum value: 359, default value: 220. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_COLORSTYLE_HUE = 220 - -# The HTML_COLORSTYLE_SAT tag controls the purity (or saturation) of the colors -# in the HTML output. For a value of 0 the output will use grayscales only. A -# value of 255 will produce the most vivid colors. -# Minimum value: 0, maximum value: 255, default value: 100. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_COLORSTYLE_SAT = 100 - -# The HTML_COLORSTYLE_GAMMA tag controls the gamma correction applied to the -# luminance component of the colors in the HTML output. Values below 100 -# gradually make the output lighter, whereas values above 100 make the output -# darker. The value divided by 100 is the actual gamma applied, so 80 represents -# a gamma of 0.8, The value 220 represents a gamma of 2.2, and 100 does not -# change the gamma. -# Minimum value: 40, maximum value: 240, default value: 80. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_COLORSTYLE_GAMMA = 80 - -# If the HTML_TIMESTAMP tag is set to YES then the footer of each generated HTML -# page will contain the date and time when the page was generated. Setting this -# to NO can help when comparing the output of multiple runs. -# The default value is: YES. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_TIMESTAMP = YES - -# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML -# documentation will contain sections that can be hidden and shown after the -# page has loaded. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_DYNAMIC_SECTIONS = NO - -# With HTML_INDEX_NUM_ENTRIES one can control the preferred number of entries -# shown in the various tree structured indices initially; the user can expand -# and collapse entries dynamically later on. Doxygen will expand the tree to -# such a level that at most the specified number of entries are visible (unless -# a fully collapsed tree already exceeds this amount). So setting the number of -# entries 1 will produce a full collapsed tree by default. 0 is a special value -# representing an infinite number of entries and will result in a full expanded -# tree by default. -# Minimum value: 0, maximum value: 9999, default value: 100. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_INDEX_NUM_ENTRIES = 100 - -# If the GENERATE_DOCSET tag is set to YES, additional index files will be -# generated that can be used as input for Apple's Xcode 3 integrated development -# environment (see: http://developer.apple.com/tools/xcode/), introduced with -# OSX 10.5 (Leopard). To create a documentation set, doxygen will generate a -# Makefile in the HTML output directory. Running make will produce the docset in -# that directory and running make install will install the docset in -# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find it at -# startup. See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html -# for more information. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -GENERATE_DOCSET = NO - -# This tag determines the name of the docset feed. A documentation feed provides -# an umbrella under which multiple documentation sets from a single provider -# (such as a company or product suite) can be grouped. -# The default value is: Doxygen generated docs. -# This tag requires that the tag GENERATE_DOCSET is set to YES. - -DOCSET_FEEDNAME = "Doxygen generated docs" - -# This tag specifies a string that should uniquely identify the documentation -# set bundle. This should be a reverse domain-name style string, e.g. -# com.mycompany.MyDocSet. Doxygen will append .docset to the name. -# The default value is: org.doxygen.Project. -# This tag requires that the tag GENERATE_DOCSET is set to YES. - -DOCSET_BUNDLE_ID = org.doxygen.Project - -# The DOCSET_PUBLISHER_ID tag specifies a string that should uniquely identify -# the documentation publisher. This should be a reverse domain-name style -# string, e.g. com.mycompany.MyDocSet.documentation. -# The default value is: org.doxygen.Publisher. -# This tag requires that the tag GENERATE_DOCSET is set to YES. - -DOCSET_PUBLISHER_ID = org.doxygen.Publisher - -# The DOCSET_PUBLISHER_NAME tag identifies the documentation publisher. -# The default value is: Publisher. -# This tag requires that the tag GENERATE_DOCSET is set to YES. - -DOCSET_PUBLISHER_NAME = Publisher - -# If the GENERATE_HTMLHELP tag is set to YES then doxygen generates three -# additional HTML index files: index.hhp, index.hhc, and index.hhk. The -# index.hhp is a project file that can be read by Microsoft's HTML Help Workshop -# (see: http://www.microsoft.com/en-us/download/details.aspx?id=21138) on -# Windows. -# -# The HTML Help Workshop contains a compiler that can convert all HTML output -# generated by doxygen into a single compiled HTML file (.chm). Compiled HTML -# files are now used as the Windows 98 help format, and will replace the old -# Windows help format (.hlp) on all Windows platforms in the future. Compressed -# HTML files also contain an index, a table of contents, and you can search for -# words in the documentation. The HTML workshop also contains a viewer for -# compressed HTML files. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -GENERATE_HTMLHELP = NO - -# The CHM_FILE tag can be used to specify the file name of the resulting .chm -# file. You can add a path in front of the file if the result should not be -# written to the html output directory. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -CHM_FILE = - -# The HHC_LOCATION tag can be used to specify the location (absolute path -# including file name) of the HTML help compiler (hhc.exe). If non-empty, -# doxygen will try to run the HTML help compiler on the generated index.hhp. -# The file has to be specified with full path. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -HHC_LOCATION = - -# The GENERATE_CHI flag controls if a separate .chi index file is generated -# (YES) or that it should be included in the master .chm file (NO). -# The default value is: NO. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -GENERATE_CHI = NO - -# The CHM_INDEX_ENCODING is used to encode HtmlHelp index (hhk), content (hhc) -# and project file content. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -CHM_INDEX_ENCODING = - -# The BINARY_TOC flag controls whether a binary table of contents is generated -# (YES) or a normal table of contents (NO) in the .chm file. Furthermore it -# enables the Previous and Next buttons. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -BINARY_TOC = NO - -# The TOC_EXPAND flag can be set to YES to add extra items for group members to -# the table of contents of the HTML help documentation and to the tree view. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -TOC_EXPAND = NO - -# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and -# QHP_VIRTUAL_FOLDER are set, an additional index file will be generated that -# can be used as input for Qt's qhelpgenerator to generate a Qt Compressed Help -# (.qch) of the generated HTML documentation. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -GENERATE_QHP = NO - -# If the QHG_LOCATION tag is specified, the QCH_FILE tag can be used to specify -# the file name of the resulting .qch file. The path specified is relative to -# the HTML output folder. -# This tag requires that the tag GENERATE_QHP is set to YES. - -QCH_FILE = - -# The QHP_NAMESPACE tag specifies the namespace to use when generating Qt Help -# Project output. For more information please see Qt Help Project / Namespace -# (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#namespace). -# The default value is: org.doxygen.Project. -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHP_NAMESPACE = org.doxygen.Project - -# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating Qt -# Help Project output. For more information please see Qt Help Project / Virtual -# Folders (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#virtual- -# folders). -# The default value is: doc. -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHP_VIRTUAL_FOLDER = doc - -# If the QHP_CUST_FILTER_NAME tag is set, it specifies the name of a custom -# filter to add. For more information please see Qt Help Project / Custom -# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom- -# filters). -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHP_CUST_FILTER_NAME = - -# The QHP_CUST_FILTER_ATTRS tag specifies the list of the attributes of the -# custom filter to add. For more information please see Qt Help Project / Custom -# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom- -# filters). -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHP_CUST_FILTER_ATTRS = - -# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this -# project's filter section matches. Qt Help Project / Filter Attributes (see: -# http://qt-project.org/doc/qt-4.8/qthelpproject.html#filter-attributes). -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHP_SECT_FILTER_ATTRS = - -# The QHG_LOCATION tag can be used to specify the location of Qt's -# qhelpgenerator. If non-empty doxygen will try to run qhelpgenerator on the -# generated .qhp file. -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHG_LOCATION = - -# If the GENERATE_ECLIPSEHELP tag is set to YES, additional index files will be -# generated, together with the HTML files, they form an Eclipse help plugin. To -# install this plugin and make it available under the help contents menu in -# Eclipse, the contents of the directory containing the HTML and XML files needs -# to be copied into the plugins directory of eclipse. The name of the directory -# within the plugins directory should be the same as the ECLIPSE_DOC_ID value. -# After copying Eclipse needs to be restarted before the help appears. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -GENERATE_ECLIPSEHELP = NO - -# A unique identifier for the Eclipse help plugin. When installing the plugin -# the directory name containing the HTML and XML files should also have this -# name. Each documentation set should have its own identifier. -# The default value is: org.doxygen.Project. -# This tag requires that the tag GENERATE_ECLIPSEHELP is set to YES. - -ECLIPSE_DOC_ID = org.doxygen.Project - -# If you want full control over the layout of the generated HTML pages it might -# be necessary to disable the index and replace it with your own. The -# DISABLE_INDEX tag can be used to turn on/off the condensed index (tabs) at top -# of each HTML page. A value of NO enables the index and the value YES disables -# it. Since the tabs in the index contain the same information as the navigation -# tree, you can set this option to YES if you also set GENERATE_TREEVIEW to YES. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -DISABLE_INDEX = NO - -# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index -# structure should be generated to display hierarchical information. If the tag -# value is set to YES, a side panel will be generated containing a tree-like -# index structure (just like the one that is generated for HTML Help). For this -# to work a browser that supports JavaScript, DHTML, CSS and frames is required -# (i.e. any modern browser). Windows users are probably better off using the -# HTML help feature. Via custom style sheets (see HTML_EXTRA_STYLESHEET) one can -# further fine-tune the look of the index. As an example, the default style -# sheet generated by doxygen has an example that shows how to put an image at -# the root of the tree instead of the PROJECT_NAME. Since the tree basically has -# the same information as the tab index, you could consider setting -# DISABLE_INDEX to YES when enabling this option. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -GENERATE_TREEVIEW = NO - -# The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values that -# doxygen will group on one line in the generated HTML documentation. -# -# Note that a value of 0 will completely suppress the enum values from appearing -# in the overview section. -# Minimum value: 0, maximum value: 20, default value: 4. -# This tag requires that the tag GENERATE_HTML is set to YES. - -ENUM_VALUES_PER_LINE = 4 - -# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be used -# to set the initial width (in pixels) of the frame in which the tree is shown. -# Minimum value: 0, maximum value: 1500, default value: 250. -# This tag requires that the tag GENERATE_HTML is set to YES. - -TREEVIEW_WIDTH = 250 - -# If the EXT_LINKS_IN_WINDOW option is set to YES, doxygen will open links to -# external symbols imported via tag files in a separate window. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -EXT_LINKS_IN_WINDOW = NO - -# Use this tag to change the font size of LaTeX formulas included as images in -# the HTML documentation. When you change the font size after a successful -# doxygen run you need to manually remove any form_*.png images from the HTML -# output directory to force them to be regenerated. -# Minimum value: 8, maximum value: 50, default value: 10. -# This tag requires that the tag GENERATE_HTML is set to YES. - -FORMULA_FONTSIZE = 10 - -# Use the FORMULA_TRANPARENT tag to determine whether or not the images -# generated for formulas are transparent PNGs. Transparent PNGs are not -# supported properly for IE 6.0, but are supported on all modern browsers. -# -# Note that when changing this option you need to delete any form_*.png files in -# the HTML output directory before the changes have effect. -# The default value is: YES. -# This tag requires that the tag GENERATE_HTML is set to YES. - -FORMULA_TRANSPARENT = YES - -# Enable the USE_MATHJAX option to render LaTeX formulas using MathJax (see -# http://www.mathjax.org) which uses client side Javascript for the rendering -# instead of using pre-rendered bitmaps. Use this if you do not have LaTeX -# installed or if you want to formulas look prettier in the HTML output. When -# enabled you may also need to install MathJax separately and configure the path -# to it using the MATHJAX_RELPATH option. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -USE_MATHJAX = NO - -# When MathJax is enabled you can set the default output format to be used for -# the MathJax output. See the MathJax site (see: -# http://docs.mathjax.org/en/latest/output.html) for more details. -# Possible values are: HTML-CSS (which is slower, but has the best -# compatibility), NativeMML (i.e. MathML) and SVG. -# The default value is: HTML-CSS. -# This tag requires that the tag USE_MATHJAX is set to YES. - -MATHJAX_FORMAT = HTML-CSS - -# When MathJax is enabled you need to specify the location relative to the HTML -# output directory using the MATHJAX_RELPATH option. The destination directory -# should contain the MathJax.js script. For instance, if the mathjax directory -# is located at the same level as the HTML output directory, then -# MATHJAX_RELPATH should be ../mathjax. The default value points to the MathJax -# Content Delivery Network so you can quickly see the result without installing -# MathJax. However, it is strongly recommended to install a local copy of -# MathJax from http://www.mathjax.org before deployment. -# The default value is: http://cdn.mathjax.org/mathjax/latest. -# This tag requires that the tag USE_MATHJAX is set to YES. - -MATHJAX_RELPATH = http://cdn.mathjax.org/mathjax/latest - -# The MATHJAX_EXTENSIONS tag can be used to specify one or more MathJax -# extension names that should be enabled during MathJax rendering. For example -# MATHJAX_EXTENSIONS = TeX/AMSmath TeX/AMSsymbols -# This tag requires that the tag USE_MATHJAX is set to YES. - -MATHJAX_EXTENSIONS = - -# The MATHJAX_CODEFILE tag can be used to specify a file with javascript pieces -# of code that will be used on startup of the MathJax code. See the MathJax site -# (see: http://docs.mathjax.org/en/latest/output.html) for more details. For an -# example see the documentation. -# This tag requires that the tag USE_MATHJAX is set to YES. - -MATHJAX_CODEFILE = - -# When the SEARCHENGINE tag is enabled doxygen will generate a search box for -# the HTML output. The underlying search engine uses javascript and DHTML and -# should work on any modern browser. Note that when using HTML help -# (GENERATE_HTMLHELP), Qt help (GENERATE_QHP), or docsets (GENERATE_DOCSET) -# there is already a search function so this one should typically be disabled. -# For large projects the javascript based search engine can be slow, then -# enabling SERVER_BASED_SEARCH may provide a better solution. It is possible to -# search using the keyboard; to jump to the search box use + S -# (what the is depends on the OS and browser, but it is typically -# , /