mirror of https://github.com/grpc/grpc.git
Mark D. Roth
8 years ago
commit
eb35a1d16b
447 changed files with 15381 additions and 6488 deletions
@ -0,0 +1,15 @@ |
||||
There are times when we make changes that include a temporary shim for |
||||
backward-compatibility (e.g., a macro or some other function to preserve |
||||
the original API) to avoid having to bump the major version number in |
||||
the next release. However, when we do eventually want to release a |
||||
feature that does change the API in a non-backward-compatible way, we |
||||
will wind up bumping the major version number anyway, at which point we |
||||
can take the opportunity to clean up any pending backward-compatibility |
||||
shims. |
||||
|
||||
This file lists all pending backward-compatibility changes that should |
||||
be cleaned up the next time we are going to bump the major version |
||||
number: |
||||
|
||||
- remove `ClientContext::set_fail_fast()` method from |
||||
`include/grpc++/impl/codegen/client_context.h` (commit `9477724`) |
||||
@ -0,0 +1,121 @@ |
||||
# `epoll`-based pollset implementation in gRPC |
||||
|
||||
Sree Kuchibhotla (sreek@) [May - 2016] |
||||
(Design input from Craig Tiller and David Klempner) |
||||
|
||||
> Status: As of June 2016, this change is implemented and merged. |
||||
|
||||
> * The bulk of the functionality is in: [ev_poll_linux.c](https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/ev_epoll_linux.c) |
||||
> * Pull request: https://github.com/grpc/grpc/pull/6803 |
||||
|
||||
## 1. Introduction |
||||
The document talks about the proposed changes to `epoll`-based implementation of pollsets in gRPC. Section-2 gives an overview of the current implementation, Section-3 talks about the problems in the current implementation and finally Section-4 talks about the proposed changes. |
||||
|
||||
## 2. Current `epoll`-based implementation in gRPC |
||||
|
||||
 |
||||
|
||||
**Figure 1: Current implementation** |
||||
|
||||
A gRPC client or a server can have more than one completion queue. Each completion queue creates a pollset. |
||||
|
||||
The gRPC core library does not create any threads[^1] on its own and relies on the application using the gRPC core library to provide the threads. A thread starts to poll for events by calling the gRPC core surface APIs `grpc_completion_queue_next()` or `grpc_completion_queue_pluck()`. More than one thread can call `grpc_completion_queue_next()`on the same completion queue[^2]. |
||||
|
||||
A file descriptor can be in more than one completion queue. There are examples in the next section that show how this can happen. |
||||
|
||||
When an event of interest happens in a pollset, multiple threads are woken up and there are no guarantees on which thread actually ends up performing the work i.e executing the callbacks associated with that event. The thread that performs the work finally queues a completion event `grpc_cq_completion` on the appropriate completion queue and "kicks" (i.e wakes ups) the thread that is actually interested in that event (which can be itself - in which case there is no thread hop) |
||||
|
||||
For example, in **Figure 1**, if `fd1` becomes readable, any one of the threads i.e *Threads 1* to *Threads K* or *Thread P*, might be woken up. Let's say *Thread P* was calling a `grpc_completion_queue_pluck()` and was actually interested in the event on `fd1` but *Thread 1* woke up. In this case, *Thread 1* executes the callbacks and finally kicks *Thread P* by signalling `event_fd_P`. *Thread P* wakes up, realizes that there is a new completion event for it and returns from `grpc_completion_queue_pluck()` to its caller. |
||||
|
||||
## 3. Issues in the current architecture |
||||
|
||||
### _Thundering Herds_ |
||||
|
||||
If multiple threads concurrently call `epoll_wait()`, we are guaranteed that only one thread is woken up if one of the `fds` in the set becomes readable/writable. However, in our current implementation, the threads do not directly call a blocking `epoll_wait()`[^3]. Instead, they call `poll()` on the set containing `[event_fd`[^4]`, epoll_fd]`. **(see Figure 1)** |
||||
|
||||
Considering the fact that an `fd` can be in multiple `pollsets` and that each `pollset` might have multiple poller threads, it means that whenever an `fd` becomes readable/writable, all the threads in all the `pollsets` (in which that `fd` is present) are woken up. |
||||
|
||||
The performance impact of this would be more conspicuous on the server side. Here are a two examples of thundering herds on the server side. |
||||
|
||||
Example 1: Listening fds on server |
||||
|
||||
* A gRPC server can have multiple server completion queues (i.e completion queues which are used to listen for incoming channels). |
||||
* A gRPC server can also listen on more than one TCP-port. |
||||
* A listening socket is created for each port the gRPC server would be listening on. |
||||
* Every listening socket's fd is added to all the server completion queues' pollsets. (Currently we do not do any sharding of the listening fds across these pollsets). |
||||
|
||||
This means that for every incoming new channel, all the threads waiting on all the pollsets are woken up. |
||||
|
||||
Example 2: New Incoming-channel fds on server |
||||
|
||||
* Currently, every new incoming channel's `fd` (i.e the socket `fd` that is returned by doing an `accept()` on the new incoming channel) is added to all the server completion queues' pollsets [^5]). |
||||
* Clearly, this would also cause all thundering herd problem for every read onthat fd |
||||
|
||||
There are other scenarios especially on the client side where an fd can end up being on multiple pollsets which would cause thundering herds on the clients. |
||||
|
||||
|
||||
## 4. Proposed changes to the current `epoll`-based polling implementation: |
||||
|
||||
The main idea in this proposal is to group 'related' `fds` into a single epoll-based set. This would ensure that only one thread wakes up in case of an event on one of the `fds` in the epoll set. |
||||
|
||||
To accomplish this, we introduce a new abstraction called `polling_island` which will have an epoll set underneath (See **Figure 2** below). A `polling_island` contains the following: |
||||
|
||||
* `epoll_fd`: The file descriptor of the underlying epoll set |
||||
* `fd_set`: The set of 'fds' in the pollset island i.e in the epoll set (The pollset island merging operation described later requires the list of fds in the pollset island and currently there is no API available to enumerate all the fds in an epoll set) |
||||
* `event_fd`: A level triggered _event fd_ that is used to wake up all the threads waiting on this epoll set (Note: This `event_fd` is added to the underlying epoll set during pollset island creation. This is useful in the pollset island merging operation described later) |
||||
* `merged_to`: The polling island into which this one merged. See section 4.2 (case 2) for more details on this. Also note that if `merged_to` is set, all the other fields in this polling island are not used anymore |
||||
|
||||
In this new model, only one thread wakes up whenever an event of interest happens in an epoll set. |
||||
|
||||
 |
||||
|
||||
**Figure 2: Proposed changes** |
||||
|
||||
### 4.1 Relation between `fd`, `pollset` and `polling_island:` |
||||
|
||||
* An `fd` may belong to multiple `pollsets` but belongs to exactly one `polling_island` |
||||
* A `pollset` belongs to exactly one `polling_island` |
||||
* An `fd` and the `pollset(s`) it belongs to, have same `polling_island` |
||||
|
||||
### 4.2 Algorithm to add an `fd` to a `pollset` |
||||
|
||||
There are two cases to check here: |
||||
|
||||
* **Case 1:** Both `fd` and `pollset` already belong to the same `polling_island` |
||||
* This is straightforward and nothing really needs to be done here |
||||
* **Case 2:** The `fd `and `pollset` point to different `polling_islands`: In this case we _merge_ both the polling islands i.e: |
||||
* Add all the `fds` from the smaller `polling_island `to the larger `polling_island` and update the `merged_to` pointer on the smaller island to point to the larger island. |
||||
* Wake up all the threads waiting on the smaller `polling_island`'s `epoll_fd` (by signalling the `event_fd` on that island) and make them now wait on the larger `polling_island`'s `epoll_fd` |
||||
* Update `fd` and `pollset` to now point to the larger `polling_island` |
||||
|
||||
### 4.3 Directed wakeups: |
||||
|
||||
The new implementation, just like the current implementation, does not provide us any guarantees that the thread that is woken up is the thread that is actually interested in the event. So the thread that woke up executes the callbacks and finally has to 'kick' the appropriate polling thread interested in the event. |
||||
|
||||
In the current implementation, every polling thread also had a `event_fd` on which it was listening to and hence waking it up was as simple as signalling that `event_fd`. However, using an `event_fd` also meant that every thread has to use a `poll()` (on `event_fd` and `epoll_fd`) instead of doing an `epoll_wait()` and this resulted in the thundering herd problems described above. |
||||
|
||||
The proposal here is to use signals and kicking a thread would just be sending a signal to that thread. Unfortunately there are only a few signals available on posix systems and most of them have pre-determined behavior leaving only a few signals `SIGUSR1`, `SIGUSR2` and `SIGRTx (SIGRTMIN to SIGRTMAX)` for custom use. |
||||
|
||||
The calling application might have registered other signal handlers for these signals. `We will provide a new API where the applications can "give a signal number" to gRPC library to use for this purpose. |
||||
|
||||
``` |
||||
void grpc_use_signal(int signal_num) |
||||
``` |
||||
|
||||
If the calling application does not provide a signal number, then the gRPC library will relegate to using a model similar to the current implementation (where every thread does a blocking `poll()` on its `wakeup_fd` and the `epoll_fd`). The function` psi_wait() `in figure 2 implements this logic. |
||||
|
||||
**>> **(**NOTE**: Or alternatively, we can implement a turnstile polling (i.e having only one thread calling `epoll_wait()` on the epoll set at any time - which all other threads call poll on their `wakeup_fds`) |
||||
in case of not getting a signal number from the applications. |
||||
|
||||
|
||||
## Notes |
||||
|
||||
[^1]: Only exception is in case of name-resolution |
||||
|
||||
[^2]: However, a `grpc_completion_queue_next()` and `grpc_completion_queue_pluck()` must not be called in parallel on the same completion queue |
||||
|
||||
[^3]: The threads first do a blocking` poll()` with `[wakeup_fd, epoll_fd]`. If the `poll()` returns due to an event of interest in the epoll set, they then call a non-blocking i.e a zero-timeout `epoll_wait()` on the `epoll_fd` |
||||
|
||||
[^4]: `event_fd` is the linux platform specific implementation of `grpc_wakeup_fd`. A `wakeup_fd` is used to wake up polling threads typically when the event for which the polling thread is waiting is already completed by some other thread. It is also used to wake up the polling threads in case of shutdowns or to re-evaluate the poller's interest in the fds to poll (the last scenario is only in case of `poll`-based (not `epoll`-based) implementation of `pollsets`). |
||||
|
||||
[^5]: See more details about the issue here https://github.com/grpc/grpc/issues/5470 and for a proposed fix here: https://github.com/grpc/grpc/pull/6149 |
||||
|
After Width: | Height: | Size: 52 KiB |
|
After Width: | Height: | Size: 44 KiB |
@ -1,42 +0,0 @@ |
||||
# Copyright 2015, Google Inc. |
||||
# All rights reserved. |
||||
# |
||||
# Redistribution and use in source and binary forms, with or without |
||||
# modification, are permitted provided that the following conditions are |
||||
# met: |
||||
# |
||||
# * Redistributions of source code must retain the above copyright |
||||
# notice, this list of conditions and the following disclaimer. |
||||
# * Redistributions in binary form must reproduce the above |
||||
# copyright notice, this list of conditions and the following disclaimer |
||||
# in the documentation and/or other materials provided with the |
||||
# distribution. |
||||
# * Neither the name of Google Inc. nor the names of its |
||||
# contributors may be used to endorse or promote products derived from |
||||
# this software without specific prior written permission. |
||||
# |
||||
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
|
||||
"""Runs protoc with the gRPC plugin to generate messages and gRPC stubs.""" |
||||
|
||||
from grpc.tools import protoc |
||||
|
||||
protoc.main( |
||||
( |
||||
'', |
||||
'-I../../protos', |
||||
'--python_out=.', |
||||
'--grpc_python_out=.', |
||||
'../../protos/helloworld.proto', |
||||
) |
||||
) |
||||
@ -0,0 +1,67 @@ |
||||
/*
|
||||
* |
||||
* Copyright 2016, Google Inc. |
||||
* All rights reserved. |
||||
* |
||||
* Redistribution and use in source and binary forms, with or without |
||||
* modification, are permitted provided that the following conditions are |
||||
* met: |
||||
* |
||||
* * Redistributions of source code must retain the above copyright |
||||
* notice, this list of conditions and the following disclaimer. |
||||
* * Redistributions in binary form must reproduce the above |
||||
* copyright notice, this list of conditions and the following disclaimer |
||||
* in the documentation and/or other materials provided with the |
||||
* distribution. |
||||
* * Neither the name of Google Inc. nor the names of its |
||||
* contributors may be used to endorse or promote products derived from |
||||
* this software without specific prior written permission. |
||||
* |
||||
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
* |
||||
*/ |
||||
|
||||
#ifndef GRPCXX_TEST_SERVER_CONTEXT_TEST_SPOUSE_H |
||||
#define GRPCXX_TEST_SERVER_CONTEXT_TEST_SPOUSE_H |
||||
|
||||
#include <map> |
||||
|
||||
#include <grpc++/server_context.h> |
||||
|
||||
namespace grpc { |
||||
namespace testing { |
||||
|
||||
// A test-only class to access private members and methods of ServerContext.
|
||||
class ServerContextTestSpouse { |
||||
public: |
||||
explicit ServerContextTestSpouse(ServerContext* ctx) : ctx_(ctx) {} |
||||
|
||||
// Inject client metadata to the ServerContext for the test. The test spouse
|
||||
// must be alive when ServerContext::client_metadata is called.
|
||||
void AddClientMetadata(const grpc::string& key, const grpc::string& value); |
||||
std::multimap<grpc::string, grpc::string> GetInitialMetadata() const { |
||||
return ctx_->initial_metadata_; |
||||
} |
||||
std::multimap<grpc::string, grpc::string> GetTrailingMetadata() const { |
||||
return ctx_->trailing_metadata_; |
||||
} |
||||
|
||||
private: |
||||
ServerContext* ctx_; // not owned
|
||||
std::multimap<grpc::string, grpc::string> client_metadata_storage_; |
||||
}; |
||||
|
||||
} // namespace testing
|
||||
} // namespace grpc
|
||||
|
||||
#endif // GRPCXX_TEST_SERVER_CONTEXT_TEST_SPOUSE_H
|
||||
@ -0,0 +1,4 @@ |
||||
#Overview |
||||
|
||||
This directory contains source code for gRPC protocol buffer compiler (*protoc*) plugins. Along with `protoc`, |
||||
these plugins are used to generate gRPC client and server stubs from `.proto` files. |
||||
@ -0,0 +1,296 @@ |
||||
//
|
||||
// Copyright 2015, Google Inc.
|
||||
// All rights reserved.
|
||||
//
|
||||
// Redistribution and use in source and binary forms, with or without
|
||||
// modification, are permitted provided that the following conditions are
|
||||
// met:
|
||||
//
|
||||
// * Redistributions of source code must retain the above copyright
|
||||
// notice, this list of conditions and the following disclaimer.
|
||||
// * Redistributions in binary form must reproduce the above
|
||||
// copyright notice, this list of conditions and the following disclaimer
|
||||
// in the documentation and/or other materials provided with the
|
||||
// distribution.
|
||||
// * Neither the name of Google Inc. nor the names of its
|
||||
// contributors may be used to endorse or promote products derived from
|
||||
// this software without specific prior written permission.
|
||||
//
|
||||
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
//
|
||||
|
||||
#include "src/core/ext/client_channel/method_config.h" |
||||
|
||||
#include <string.h> |
||||
|
||||
#include <grpc/impl/codegen/grpc_types.h> |
||||
#include <grpc/support/alloc.h> |
||||
#include <grpc/support/log.h> |
||||
#include <grpc/support/string_util.h> |
||||
#include <grpc/support/time.h> |
||||
|
||||
#include "src/core/lib/transport/mdstr_hash_table.h" |
||||
#include "src/core/lib/transport/metadata.h" |
||||
|
||||
//
|
||||
// grpc_method_config
|
||||
//
|
||||
|
||||
// bool vtable
|
||||
|
||||
static void* bool_copy(void* valuep) { |
||||
bool value = *(bool*)valuep; |
||||
bool* new_value = gpr_malloc(sizeof(bool)); |
||||
*new_value = value; |
||||
return new_value; |
||||
} |
||||
|
||||
static int bool_cmp(void* v1, void* v2) { |
||||
bool b1 = *(bool*)v1; |
||||
bool b2 = *(bool*)v2; |
||||
if (!b1 && b2) return -1; |
||||
if (b1 && !b2) return 1; |
||||
return 0; |
||||
} |
||||
|
||||
static grpc_mdstr_hash_table_vtable bool_vtable = {gpr_free, bool_copy, |
||||
bool_cmp}; |
||||
|
||||
// timespec vtable
|
||||
|
||||
static void* timespec_copy(void* valuep) { |
||||
gpr_timespec value = *(gpr_timespec*)valuep; |
||||
gpr_timespec* new_value = gpr_malloc(sizeof(gpr_timespec)); |
||||
*new_value = value; |
||||
return new_value; |
||||
} |
||||
|
||||
static int timespec_cmp(void* v1, void* v2) { |
||||
return gpr_time_cmp(*(gpr_timespec*)v1, *(gpr_timespec*)v2); |
||||
} |
||||
|
||||
static grpc_mdstr_hash_table_vtable timespec_vtable = {gpr_free, timespec_copy, |
||||
timespec_cmp}; |
||||
|
||||
// int32 vtable
|
||||
|
||||
static void* int32_copy(void* valuep) { |
||||
int32_t value = *(int32_t*)valuep; |
||||
int32_t* new_value = gpr_malloc(sizeof(int32_t)); |
||||
*new_value = value; |
||||
return new_value; |
||||
} |
||||
|
||||
static int int32_cmp(void* v1, void* v2) { |
||||
int32_t i1 = *(int32_t*)v1; |
||||
int32_t i2 = *(int32_t*)v2; |
||||
if (i1 < i2) return -1; |
||||
if (i1 > i2) return 1; |
||||
return 0; |
||||
} |
||||
|
||||
static grpc_mdstr_hash_table_vtable int32_vtable = {gpr_free, int32_copy, |
||||
int32_cmp}; |
||||
|
||||
// Hash table keys.
|
||||
#define GRPC_METHOD_CONFIG_WAIT_FOR_READY "grpc.wait_for_ready" // bool
|
||||
#define GRPC_METHOD_CONFIG_TIMEOUT "grpc.timeout" // gpr_timespec
|
||||
#define GRPC_METHOD_CONFIG_MAX_REQUEST_MESSAGE_BYTES \ |
||||
"grpc.max_request_message_bytes" // int32
|
||||
#define GRPC_METHOD_CONFIG_MAX_RESPONSE_MESSAGE_BYTES \ |
||||
"grpc.max_response_message_bytes" // int32
|
||||
|
||||
struct grpc_method_config { |
||||
grpc_mdstr_hash_table* table; |
||||
grpc_mdstr* wait_for_ready_key; |
||||
grpc_mdstr* timeout_key; |
||||
grpc_mdstr* max_request_message_bytes_key; |
||||
grpc_mdstr* max_response_message_bytes_key; |
||||
}; |
||||
|
||||
grpc_method_config* grpc_method_config_create( |
||||
bool* wait_for_ready, gpr_timespec* timeout, |
||||
int32_t* max_request_message_bytes, int32_t* max_response_message_bytes) { |
||||
grpc_method_config* method_config = gpr_malloc(sizeof(grpc_method_config)); |
||||
memset(method_config, 0, sizeof(grpc_method_config)); |
||||
method_config->wait_for_ready_key = |
||||
grpc_mdstr_from_string(GRPC_METHOD_CONFIG_WAIT_FOR_READY); |
||||
method_config->timeout_key = |
||||
grpc_mdstr_from_string(GRPC_METHOD_CONFIG_TIMEOUT); |
||||
method_config->max_request_message_bytes_key = |
||||
grpc_mdstr_from_string(GRPC_METHOD_CONFIG_MAX_REQUEST_MESSAGE_BYTES); |
||||
method_config->max_response_message_bytes_key = |
||||
grpc_mdstr_from_string(GRPC_METHOD_CONFIG_MAX_RESPONSE_MESSAGE_BYTES); |
||||
grpc_mdstr_hash_table_entry entries[4]; |
||||
size_t num_entries = 0; |
||||
if (wait_for_ready != NULL) { |
||||
entries[num_entries].key = method_config->wait_for_ready_key; |
||||
entries[num_entries].value = wait_for_ready; |
||||
entries[num_entries].vtable = &bool_vtable; |
||||
++num_entries; |
||||
} |
||||
if (timeout != NULL) { |
||||
entries[num_entries].key = method_config->timeout_key; |
||||
entries[num_entries].value = timeout; |
||||
entries[num_entries].vtable = ×pec_vtable; |
||||
++num_entries; |
||||
} |
||||
if (max_request_message_bytes != NULL) { |
||||
entries[num_entries].key = method_config->max_request_message_bytes_key; |
||||
entries[num_entries].value = max_request_message_bytes; |
||||
entries[num_entries].vtable = &int32_vtable; |
||||
++num_entries; |
||||
} |
||||
if (max_response_message_bytes != NULL) { |
||||
entries[num_entries].key = method_config->max_response_message_bytes_key; |
||||
entries[num_entries].value = max_response_message_bytes; |
||||
entries[num_entries].vtable = &int32_vtable; |
||||
++num_entries; |
||||
} |
||||
method_config->table = grpc_mdstr_hash_table_create(num_entries, entries); |
||||
return method_config; |
||||
} |
||||
|
||||
grpc_method_config* grpc_method_config_ref(grpc_method_config* method_config) { |
||||
grpc_mdstr_hash_table_ref(method_config->table); |
||||
return method_config; |
||||
} |
||||
|
||||
void grpc_method_config_unref(grpc_method_config* method_config) { |
||||
if (grpc_mdstr_hash_table_unref(method_config->table)) { |
||||
GRPC_MDSTR_UNREF(method_config->wait_for_ready_key); |
||||
GRPC_MDSTR_UNREF(method_config->timeout_key); |
||||
GRPC_MDSTR_UNREF(method_config->max_request_message_bytes_key); |
||||
GRPC_MDSTR_UNREF(method_config->max_response_message_bytes_key); |
||||
gpr_free(method_config); |
||||
} |
||||
} |
||||
|
||||
int grpc_method_config_cmp(const grpc_method_config* method_config1, |
||||
const grpc_method_config* method_config2) { |
||||
return grpc_mdstr_hash_table_cmp(method_config1->table, |
||||
method_config2->table); |
||||
} |
||||
|
||||
const bool* grpc_method_config_get_wait_for_ready( |
||||
const grpc_method_config* method_config) { |
||||
return grpc_mdstr_hash_table_get(method_config->table, |
||||
method_config->wait_for_ready_key); |
||||
} |
||||
|
||||
const gpr_timespec* grpc_method_config_get_timeout( |
||||
const grpc_method_config* method_config) { |
||||
return grpc_mdstr_hash_table_get(method_config->table, |
||||
method_config->timeout_key); |
||||
} |
||||
|
||||
const int32_t* grpc_method_config_get_max_request_message_bytes( |
||||
const grpc_method_config* method_config) { |
||||
return grpc_mdstr_hash_table_get( |
||||
method_config->table, method_config->max_request_message_bytes_key); |
||||
} |
||||
|
||||
const int32_t* grpc_method_config_get_max_response_message_bytes( |
||||
const grpc_method_config* method_config) { |
||||
return grpc_mdstr_hash_table_get( |
||||
method_config->table, method_config->max_response_message_bytes_key); |
||||
} |
||||
|
||||
//
|
||||
// grpc_method_config_table
|
||||
//
|
||||
|
||||
static void method_config_unref(void* valuep) { |
||||
grpc_method_config_unref(valuep); |
||||
} |
||||
|
||||
static void* method_config_ref(void* valuep) { |
||||
return grpc_method_config_ref(valuep); |
||||
} |
||||
|
||||
static int method_config_cmp(void* valuep1, void* valuep2) { |
||||
return grpc_method_config_cmp(valuep1, valuep2); |
||||
} |
||||
|
||||
static const grpc_mdstr_hash_table_vtable method_config_table_vtable = { |
||||
method_config_unref, method_config_ref, method_config_cmp}; |
||||
|
||||
grpc_method_config_table* grpc_method_config_table_create( |
||||
size_t num_entries, grpc_method_config_table_entry* entries) { |
||||
grpc_mdstr_hash_table_entry* hash_table_entries = |
||||
gpr_malloc(sizeof(grpc_mdstr_hash_table_entry) * num_entries); |
||||
for (size_t i = 0; i < num_entries; ++i) { |
||||
hash_table_entries[i].key = entries[i].method_name; |
||||
hash_table_entries[i].value = entries[i].method_config; |
||||
hash_table_entries[i].vtable = &method_config_table_vtable; |
||||
} |
||||
grpc_method_config_table* method_config_table = |
||||
grpc_mdstr_hash_table_create(num_entries, hash_table_entries); |
||||
gpr_free(hash_table_entries); |
||||
return method_config_table; |
||||
} |
||||
|
||||
grpc_method_config_table* grpc_method_config_table_ref( |
||||
grpc_method_config_table* table) { |
||||
return grpc_mdstr_hash_table_ref(table); |
||||
} |
||||
|
||||
void grpc_method_config_table_unref(grpc_method_config_table* table) { |
||||
grpc_mdstr_hash_table_unref(table); |
||||
} |
||||
|
||||
int grpc_method_config_table_cmp(const grpc_method_config_table* table1, |
||||
const grpc_method_config_table* table2) { |
||||
return grpc_mdstr_hash_table_cmp(table1, table2); |
||||
} |
||||
|
||||
grpc_method_config* grpc_method_config_table_get_method_config( |
||||
const grpc_method_config_table* table, const grpc_mdstr* path) { |
||||
grpc_method_config* method_config = grpc_mdstr_hash_table_get(table, path); |
||||
// If we didn't find a match for the path, try looking for a wildcard
|
||||
// entry (i.e., change "/service/method" to "/service/*").
|
||||
if (method_config == NULL) { |
||||
const char* path_str = grpc_mdstr_as_c_string(path); |
||||
const char* sep = strrchr(path_str, '/') + 1; |
||||
const size_t len = (size_t)(sep - path_str); |
||||
char* buf = gpr_malloc(len + 2); // '*' and NUL
|
||||
memcpy(buf, path_str, len); |
||||
buf[len] = '*'; |
||||
buf[len + 1] = '\0'; |
||||
grpc_mdstr* wildcard_path = grpc_mdstr_from_string(buf); |
||||
gpr_free(buf); |
||||
method_config = grpc_mdstr_hash_table_get(table, wildcard_path); |
||||
GRPC_MDSTR_UNREF(wildcard_path); |
||||
} |
||||
return method_config; |
||||
} |
||||
|
||||
static void* copy_arg(void* p) { return grpc_method_config_table_ref(p); } |
||||
|
||||
static void destroy_arg(void* p) { grpc_method_config_table_unref(p); } |
||||
|
||||
static int cmp_arg(void* p1, void* p2) { |
||||
return grpc_method_config_table_cmp(p1, p2); |
||||
} |
||||
|
||||
static grpc_arg_pointer_vtable arg_vtable = {copy_arg, destroy_arg, cmp_arg}; |
||||
|
||||
grpc_arg grpc_method_config_table_create_channel_arg( |
||||
grpc_method_config_table* table) { |
||||
grpc_arg arg; |
||||
arg.type = GRPC_ARG_POINTER; |
||||
arg.key = GRPC_ARG_SERVICE_CONFIG; |
||||
arg.value.pointer.p = table; |
||||
arg.value.pointer.vtable = &arg_vtable; |
||||
return arg; |
||||
} |
||||
@ -0,0 +1,116 @@ |
||||
//
|
||||
// Copyright 2016, Google Inc.
|
||||
// All rights reserved.
|
||||
//
|
||||
// Redistribution and use in source and binary forms, with or without
|
||||
// modification, are permitted provided that the following conditions are
|
||||
// met:
|
||||
//
|
||||
// * Redistributions of source code must retain the above copyright
|
||||
// notice, this list of conditions and the following disclaimer.
|
||||
// * Redistributions in binary form must reproduce the above
|
||||
// copyright notice, this list of conditions and the following disclaimer
|
||||
// in the documentation and/or other materials provided with the
|
||||
// distribution.
|
||||
// * Neither the name of Google Inc. nor the names of its
|
||||
// contributors may be used to endorse or promote products derived from
|
||||
// this software without specific prior written permission.
|
||||
//
|
||||
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
//
|
||||
|
||||
#ifndef GRPC_CORE_EXT_CLIENT_CHANNEL_METHOD_CONFIG_H |
||||
#define GRPC_CORE_EXT_CLIENT_CHANNEL_METHOD_CONFIG_H |
||||
|
||||
#include <stdbool.h> |
||||
|
||||
#include <grpc/impl/codegen/gpr_types.h> |
||||
#include <grpc/impl/codegen/grpc_types.h> |
||||
|
||||
#include "src/core/lib/transport/mdstr_hash_table.h" |
||||
#include "src/core/lib/transport/metadata.h" |
||||
|
||||
/// Per-method configuration.
|
||||
typedef struct grpc_method_config grpc_method_config; |
||||
|
||||
/// Creates a grpc_method_config with the specified parameters.
|
||||
/// Any parameter may be NULL to indicate that the value is unset.
|
||||
///
|
||||
/// \a wait_for_ready indicates whether the client should wait until the
|
||||
/// request deadline for the channel to become ready, even if there is a
|
||||
/// temporary failure before the deadline while attempting to connect.
|
||||
///
|
||||
/// \a timeout indicates the timeout for calls.
|
||||
///
|
||||
/// \a max_request_message_bytes and \a max_response_message_bytes
|
||||
/// indicate the maximum sizes of the request (checked when sending) and
|
||||
/// response (checked when receiving) messages.
|
||||
grpc_method_config* grpc_method_config_create( |
||||
bool* wait_for_ready, gpr_timespec* timeout, |
||||
int32_t* max_request_message_bytes, int32_t* max_response_message_bytes); |
||||
|
||||
grpc_method_config* grpc_method_config_ref(grpc_method_config* method_config); |
||||
void grpc_method_config_unref(grpc_method_config* method_config); |
||||
|
||||
/// Compares two grpc_method_configs.
|
||||
/// The sort order is stable but undefined.
|
||||
int grpc_method_config_cmp(const grpc_method_config* method_config1, |
||||
const grpc_method_config* method_config2); |
||||
|
||||
/// These methods return NULL if the requested field is unset.
|
||||
/// The caller does NOT take ownership of the result.
|
||||
const bool* grpc_method_config_get_wait_for_ready( |
||||
const grpc_method_config* method_config); |
||||
const gpr_timespec* grpc_method_config_get_timeout( |
||||
const grpc_method_config* method_config); |
||||
const int32_t* grpc_method_config_get_max_request_message_bytes( |
||||
const grpc_method_config* method_config); |
||||
const int32_t* grpc_method_config_get_max_response_message_bytes( |
||||
const grpc_method_config* method_config); |
||||
|
||||
/// A table of method configs.
|
||||
typedef grpc_mdstr_hash_table grpc_method_config_table; |
||||
|
||||
typedef struct grpc_method_config_table_entry { |
||||
/// The name is of one of the following forms:
|
||||
/// service/method -- specifies exact service and method name
|
||||
/// service/* -- matches all methods for the specified service
|
||||
grpc_mdstr* method_name; |
||||
grpc_method_config* method_config; |
||||
} grpc_method_config_table_entry; |
||||
|
||||
/// Takes new references to all keys and values in \a entries.
|
||||
grpc_method_config_table* grpc_method_config_table_create( |
||||
size_t num_entries, grpc_method_config_table_entry* entries); |
||||
|
||||
grpc_method_config_table* grpc_method_config_table_ref( |
||||
grpc_method_config_table* table); |
||||
void grpc_method_config_table_unref(grpc_method_config_table* table); |
||||
|
||||
/// Compares two grpc_method_config_tables.
|
||||
/// The sort order is stable but undefined.
|
||||
int grpc_method_config_table_cmp(const grpc_method_config_table* table1, |
||||
const grpc_method_config_table* table2); |
||||
|
||||
/// Gets the method config for the specified \a path, which should be of
|
||||
/// the form "/service/method".
|
||||
/// Returns NULL if the method has no config.
|
||||
/// Caller does NOT own a reference to the result.
|
||||
grpc_method_config* grpc_method_config_table_get_method_config( |
||||
const grpc_method_config_table* table, const grpc_mdstr* path); |
||||
|
||||
/// Returns a channel arg containing \a table.
|
||||
grpc_arg grpc_method_config_table_create_channel_arg( |
||||
grpc_method_config_table* table); |
||||
|
||||
#endif /* GRPC_CORE_EXT_CLIENT_CHANNEL_METHOD_CONFIG_H */ |
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,118 @@ |
||||
/*
|
||||
* |
||||
* Copyright 2016, Google Inc. |
||||
* All rights reserved. |
||||
* |
||||
* Redistribution and use in source and binary forms, with or without |
||||
* modification, are permitted provided that the following conditions are |
||||
* met: |
||||
* |
||||
* * Redistributions of source code must retain the above copyright |
||||
* notice, this list of conditions and the following disclaimer. |
||||
* * Redistributions in binary form must reproduce the above |
||||
* copyright notice, this list of conditions and the following disclaimer |
||||
* in the documentation and/or other materials provided with the |
||||
* distribution. |
||||
* * Neither the name of Google Inc. nor the names of its |
||||
* contributors may be used to endorse or promote products derived from |
||||
* this software without specific prior written permission. |
||||
* |
||||
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
* |
||||
*/ |
||||
|
||||
#include <grpc/support/port_platform.h> |
||||
|
||||
#ifdef GPR_POSIX_WAKEUP_FD |
||||
|
||||
#include "src/core/lib/iomgr/wakeup_fd_cv.h" |
||||
|
||||
#include <errno.h> |
||||
#include <string.h> |
||||
|
||||
#include <grpc/support/alloc.h> |
||||
#include <grpc/support/log.h> |
||||
#include <grpc/support/sync.h> |
||||
#include <grpc/support/thd.h> |
||||
#include <grpc/support/time.h> |
||||
#include <grpc/support/useful.h> |
||||
|
||||
#define MAX_TABLE_RESIZE 256 |
||||
|
||||
extern cv_fd_table g_cvfds; |
||||
|
||||
static grpc_error* cv_fd_init(grpc_wakeup_fd* fd_info) { |
||||
unsigned int i, newsize; |
||||
int idx; |
||||
gpr_mu_lock(&g_cvfds.mu); |
||||
if (!g_cvfds.free_fds) { |
||||
newsize = GPR_MIN(g_cvfds.size * 2, g_cvfds.size + MAX_TABLE_RESIZE); |
||||
g_cvfds.cvfds = gpr_realloc(g_cvfds.cvfds, sizeof(fd_node) * newsize); |
||||
for (i = g_cvfds.size; i < newsize; i++) { |
||||
g_cvfds.cvfds[i].is_set = 0; |
||||
g_cvfds.cvfds[i].cvs = NULL; |
||||
g_cvfds.cvfds[i].next_free = g_cvfds.free_fds; |
||||
g_cvfds.free_fds = &g_cvfds.cvfds[i]; |
||||
} |
||||
g_cvfds.size = newsize; |
||||
} |
||||
|
||||
idx = (int)(g_cvfds.free_fds - g_cvfds.cvfds); |
||||
g_cvfds.free_fds = g_cvfds.free_fds->next_free; |
||||
g_cvfds.cvfds[idx].cvs = NULL; |
||||
g_cvfds.cvfds[idx].is_set = 0; |
||||
fd_info->read_fd = IDX_TO_FD(idx); |
||||
fd_info->write_fd = -1; |
||||
gpr_mu_unlock(&g_cvfds.mu); |
||||
return GRPC_ERROR_NONE; |
||||
} |
||||
|
||||
static grpc_error* cv_fd_wakeup(grpc_wakeup_fd* fd_info) { |
||||
cv_node* cvn; |
||||
gpr_mu_lock(&g_cvfds.mu); |
||||
g_cvfds.cvfds[FD_TO_IDX(fd_info->read_fd)].is_set = 1; |
||||
cvn = g_cvfds.cvfds[FD_TO_IDX(fd_info->read_fd)].cvs; |
||||
while (cvn) { |
||||
gpr_cv_signal(cvn->cv); |
||||
cvn = cvn->next; |
||||
} |
||||
gpr_mu_unlock(&g_cvfds.mu); |
||||
return GRPC_ERROR_NONE; |
||||
} |
||||
|
||||
static grpc_error* cv_fd_consume(grpc_wakeup_fd* fd_info) { |
||||
gpr_mu_lock(&g_cvfds.mu); |
||||
g_cvfds.cvfds[FD_TO_IDX(fd_info->read_fd)].is_set = 0; |
||||
gpr_mu_unlock(&g_cvfds.mu); |
||||
return GRPC_ERROR_NONE; |
||||
} |
||||
|
||||
static void cv_fd_destroy(grpc_wakeup_fd* fd_info) { |
||||
if (fd_info->read_fd == 0) { |
||||
return; |
||||
} |
||||
gpr_mu_lock(&g_cvfds.mu); |
||||
// Assert that there are no active pollers
|
||||
GPR_ASSERT(!g_cvfds.cvfds[FD_TO_IDX(fd_info->read_fd)].cvs); |
||||
g_cvfds.cvfds[FD_TO_IDX(fd_info->read_fd)].next_free = g_cvfds.free_fds; |
||||
g_cvfds.free_fds = &g_cvfds.cvfds[FD_TO_IDX(fd_info->read_fd)]; |
||||
gpr_mu_unlock(&g_cvfds.mu); |
||||
} |
||||
|
||||
static int cv_check_availability(void) { return 1; } |
||||
|
||||
const grpc_wakeup_fd_vtable grpc_cv_wakeup_fd_vtable = { |
||||
cv_fd_init, cv_fd_consume, cv_fd_wakeup, cv_fd_destroy, |
||||
cv_check_availability}; |
||||
|
||||
#endif /* GPR_POSIX_WAKUP_FD */ |
||||
@ -0,0 +1,80 @@ |
||||
/*
|
||||
* |
||||
* Copyright 2016, Google Inc. |
||||
* All rights reserved. |
||||
* |
||||
* Redistribution and use in source and binary forms, with or without |
||||
* modification, are permitted provided that the following conditions are |
||||
* met: |
||||
* |
||||
* * Redistributions of source code must retain the above copyright |
||||
* notice, this list of conditions and the following disclaimer. |
||||
* * Redistributions in binary form must reproduce the above |
||||
* copyright notice, this list of conditions and the following disclaimer |
||||
* in the documentation and/or other materials provided with the |
||||
* distribution. |
||||
* * Neither the name of Google Inc. nor the names of its |
||||
* contributors may be used to endorse or promote products derived from |
||||
* this software without specific prior written permission. |
||||
* |
||||
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
* |
||||
*/ |
||||
|
||||
/*
|
||||
* wakeup_fd_cv uses condition variables to implement wakeup fds. |
||||
* |
||||
* It is intended for use only in cases when eventfd() and pipe() are not |
||||
* available. It can only be used with the "poll" engine. |
||||
* |
||||
* Implementation: |
||||
* A global table of cv wakeup fds is mantained. A cv wakeup fd is a negative |
||||
* file descriptor. poll() is then run in a background thread with only the |
||||
* real socket fds while we wait on a condition variable trigged by either the |
||||
* poll() completion or a wakeup_fd() call. |
||||
* |
||||
*/ |
||||
|
||||
#ifndef GRPC_CORE_LIB_IOMGR_WAKEUP_FD_CV_H |
||||
#define GRPC_CORE_LIB_IOMGR_WAKEUP_FD_CV_H |
||||
|
||||
#include <grpc/support/sync.h> |
||||
|
||||
#include "src/core/lib/iomgr/ev_posix.h" |
||||
|
||||
#define FD_TO_IDX(fd) (-(fd)-1) |
||||
#define IDX_TO_FD(idx) (-(idx)-1) |
||||
|
||||
typedef struct cv_node { |
||||
gpr_cv* cv; |
||||
struct cv_node* next; |
||||
} cv_node; |
||||
|
||||
typedef struct fd_node { |
||||
int is_set; |
||||
cv_node* cvs; |
||||
struct fd_node* next_free; |
||||
} fd_node; |
||||
|
||||
typedef struct cv_fd_table { |
||||
gpr_mu mu; |
||||
int pollcount; |
||||
int shutdown; |
||||
gpr_cv shutdown_complete; |
||||
fd_node* cvfds; |
||||
fd_node* free_fds; |
||||
unsigned int size; |
||||
grpc_poll_function_type poll; |
||||
} cv_fd_table; |
||||
|
||||
#endif /* GRPC_CORE_LIB_IOMGR_WAKEUP_FD_CV_H */ |
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in new issue