Mark D. Roth
8 years ago
commit
14ef5d83e2
46 changed files with 5929 additions and 437 deletions
@ -0,0 +1,158 @@ |
||||
# Combiner Explanation |
||||
## Talk by ctiller, notes by vjpai |
||||
|
||||
Typical way of doing critical section |
||||
|
||||
``` |
||||
mu.lock() |
||||
do_stuff() |
||||
mu.unlock() |
||||
``` |
||||
|
||||
An alternative way of doing it is |
||||
|
||||
``` |
||||
class combiner { |
||||
run(f) { |
||||
mu.lock() |
||||
f() |
||||
mu.unlock() |
||||
} |
||||
mutex mu; |
||||
} |
||||
|
||||
combiner.run(do_stuff) |
||||
``` |
||||
|
||||
If you have two threads calling combiner, there will be some kind of |
||||
queuing in place. It's called `combiner` because you can pass in more |
||||
than one do_stuff at once and they will run under a common `mu`. |
||||
|
||||
The implementation described above has the issue that you're blocking a thread |
||||
for a period of time, and this is considered harmful because it's an application thread that you're blocking. |
||||
|
||||
Instead, get a new property: |
||||
* Keep things running in serial execution |
||||
* Don't ever sleep the thread |
||||
* But maybe allow things to end up running on a different thread from where they were started |
||||
* This means that `do_stuff` doesn't necessarily run to completion when `combiner.run` is invoked |
||||
|
||||
``` |
||||
class combiner { |
||||
mpscq q; // multi-producer single-consumer queue can be made non-blocking |
||||
state s; // is it empty or executing |
||||
|
||||
run(f) { |
||||
if (q.push(f)) { |
||||
// q.push returns true if it's the first thing |
||||
while (q.pop(&f)) { // modulo some extra work to avoid races |
||||
f(); |
||||
} |
||||
} |
||||
} |
||||
} |
||||
``` |
||||
|
||||
The basic idea is that the first one to push onto the combiner |
||||
executes the work and then keeps executing functions from the queue |
||||
until the combiner is drained. |
||||
|
||||
Our combiner does some additional work, with the motivation of write-batching. |
||||
|
||||
We have a second tier of `run` called `run_finally`. Anything queued |
||||
onto `run_finally` runs after we have drained the queue. That means |
||||
that there is essentially a finally-queue. This is not guaranteed to |
||||
be final, but it's best-effort. In the process of running the finally |
||||
item, we might put something onto the main combiner queue and so we'll |
||||
need to re-enter. |
||||
|
||||
`chttp2` runs all ops in the run state except if it sees a write it puts that into a finally. That way anything else that gets put into the combiner can add to that write. |
||||
|
||||
``` |
||||
class combiner { |
||||
mpscq q; // multi-producer single-consumer queue can be made non-blocking |
||||
state s; // is it empty or executing |
||||
queue finally; // you can only do run_finally when you are already running something from the combiner |
||||
|
||||
run(f) { |
||||
if (q.push(f)) { |
||||
// q.push returns true if it's the first thing |
||||
loop: |
||||
while (q.pop(&f)) { // modulo some extra work to avoid races |
||||
f(); |
||||
} |
||||
while (finally.pop(&f)) { |
||||
f(); |
||||
} |
||||
goto loop; |
||||
} |
||||
} |
||||
} |
||||
``` |
||||
|
||||
So that explains how combiners work in general. In gRPC, there is |
||||
`start_batch(..., tag)` and then work only gets activated by somebody |
||||
calling `cq::next` which returns a tag. This gives an API-level |
||||
guarantee that there will be a thread doing polling to actually make |
||||
work happen. However, some operations are not covered by a poller |
||||
thread, such as cancellation that doesn't have a completion. Other |
||||
callbacks that don't have a completion are the internal work that gets |
||||
done before the batch gets completed. We need a condition called |
||||
`covered_by_poller` that means that the item will definitely need some |
||||
thread at some point to call `cq::next` . This includes those |
||||
callbacks that directly cause a completion but also those that are |
||||
indirectly required before getting a completion. If we can't tell for |
||||
sure for a specific path, we have to assumed it is not covered by |
||||
poller. |
||||
|
||||
The above combiner has the problem that it keeps draining for a |
||||
potentially infinite amount of time and that can lead to a huge tail |
||||
latency for some operations. So we can tweak it by returning to the application |
||||
if we know that it is valid to do so: |
||||
|
||||
``` |
||||
while (q.pop(&f)) { |
||||
f(); |
||||
if (control_can_be_returned && some_still_queued_thing_is_covered_by_poller) { |
||||
offload_combiner_work_to_some_other_thread(); |
||||
} |
||||
} |
||||
``` |
||||
|
||||
`offload` is more than `break`; it does `break` but also causes some |
||||
other thread that is currently waiting on a poll to break out of its |
||||
poll. This is done by setting up a per-polling-island work-queue |
||||
(distributor) wakeup FD. The work-queue is the converse of the combiner; it |
||||
tries to spray events onto as many threads as possible to get as much concurrency as possible. |
||||
|
||||
So `offload` really does: |
||||
|
||||
``` |
||||
workqueue.run(continue_from_while_loop); |
||||
break; |
||||
``` |
||||
|
||||
This needs us to add another class variable for a `workqueue` |
||||
(which is really conceptually a distributor). |
||||
|
||||
``` |
||||
workqueue::run(f) { |
||||
q.push(f) |
||||
eventfd.wakeup() |
||||
} |
||||
|
||||
workqueue::readable() { |
||||
eventfd.consume(); |
||||
q.pop(&f); |
||||
f(); |
||||
if (!q.empty()) { |
||||
eventfd.wakeup(); // spray across as many threads as are waiting on this workqueue |
||||
} |
||||
} |
||||
``` |
||||
|
||||
In principle, `run_finally` could get starved, but this hasn't |
||||
happened in practice. If we were concerned about this, we could put a |
||||
limit on how many things come off the regular `q` before the `finally` |
||||
queue gets processed. |
||||
|
||||
@ -0,0 +1,134 @@ |
||||
/*
|
||||
* |
||||
* Copyright 2017, Google Inc. |
||||
* All rights reserved. |
||||
* |
||||
* Redistribution and use in source and binary forms, with or without |
||||
* modification, are permitted provided that the following conditions are |
||||
* met: |
||||
* |
||||
* * Redistributions of source code must retain the above copyright |
||||
* notice, this list of conditions and the following disclaimer. |
||||
* * Redistributions in binary form must reproduce the above |
||||
* copyright notice, this list of conditions and the following disclaimer |
||||
* in the documentation and/or other materials provided with the |
||||
* distribution. |
||||
* * Neither the name of Google Inc. nor the names of its |
||||
* contributors may be used to endorse or promote products derived from |
||||
* this software without specific prior written permission. |
||||
* |
||||
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
* |
||||
*/ |
||||
|
||||
#ifndef GRPC_CORE_LIB_IOMGR_TCP_SERVER_UTILS_POSIX_H |
||||
#define GRPC_CORE_LIB_IOMGR_TCP_SERVER_UTILS_POSIX_H |
||||
|
||||
#include "src/core/lib/iomgr/ev_posix.h" |
||||
#include "src/core/lib/iomgr/resolve_address.h" |
||||
#include "src/core/lib/iomgr/socket_utils_posix.h" |
||||
#include "src/core/lib/iomgr/tcp_server.h" |
||||
|
||||
/* one listening port */ |
||||
typedef struct grpc_tcp_listener { |
||||
int fd; |
||||
grpc_fd *emfd; |
||||
grpc_tcp_server *server; |
||||
grpc_resolved_address addr; |
||||
int port; |
||||
unsigned port_index; |
||||
unsigned fd_index; |
||||
grpc_closure read_closure; |
||||
grpc_closure destroyed_closure; |
||||
struct grpc_tcp_listener *next; |
||||
/* sibling is a linked list of all listeners for a given port. add_port and
|
||||
clone_port place all new listeners in the same sibling list. A member of |
||||
the 'sibling' list is also a member of the 'next' list. The head of each |
||||
sibling list has is_sibling==0, and subsequent members of sibling lists |
||||
have is_sibling==1. is_sibling allows separate sibling lists to be |
||||
identified while iterating through 'next'. */ |
||||
struct grpc_tcp_listener *sibling; |
||||
int is_sibling; |
||||
} grpc_tcp_listener; |
||||
|
||||
/* the overall server */ |
||||
struct grpc_tcp_server { |
||||
gpr_refcount refs; |
||||
/* Called whenever accept() succeeds on a server port. */ |
||||
grpc_tcp_server_cb on_accept_cb; |
||||
void *on_accept_cb_arg; |
||||
|
||||
gpr_mu mu; |
||||
|
||||
/* active port count: how many ports are actually still listening */ |
||||
size_t active_ports; |
||||
/* destroyed port count: how many ports are completely destroyed */ |
||||
size_t destroyed_ports; |
||||
|
||||
/* is this server shutting down? */ |
||||
bool shutdown; |
||||
/* have listeners been shutdown? */ |
||||
bool shutdown_listeners; |
||||
/* use SO_REUSEPORT */ |
||||
bool so_reuseport; |
||||
/* expand wildcard addresses to a list of all local addresses */ |
||||
bool expand_wildcard_addrs; |
||||
|
||||
/* linked list of server ports */ |
||||
grpc_tcp_listener *head; |
||||
grpc_tcp_listener *tail; |
||||
unsigned nports; |
||||
|
||||
/* List of closures passed to shutdown_starting_add(). */ |
||||
grpc_closure_list shutdown_starting; |
||||
|
||||
/* shutdown callback */ |
||||
grpc_closure *shutdown_complete; |
||||
|
||||
/* all pollsets interested in new connections */ |
||||
grpc_pollset **pollsets; |
||||
/* number of pollsets in the pollsets array */ |
||||
size_t pollset_count; |
||||
|
||||
/* next pollset to assign a channel to */ |
||||
gpr_atm next_pollset_to_assign; |
||||
|
||||
grpc_resource_quota *resource_quota; |
||||
}; |
||||
|
||||
/* If successful, add a listener to \a s for \a addr, set \a dsmode for the
|
||||
socket, and return the \a listener. */ |
||||
grpc_error *grpc_tcp_server_add_addr(grpc_tcp_server *s, |
||||
const grpc_resolved_address *addr, |
||||
unsigned port_index, unsigned fd_index, |
||||
grpc_dualstack_mode *dsmode, |
||||
grpc_tcp_listener **listener); |
||||
|
||||
/* Get all addresses assigned to network interfaces on the machine and create a
|
||||
listener for each. requested_port is the port to use for every listener, or 0 |
||||
to select one random port that will be used for every listener. Set *out_port |
||||
to the port selected. Return GRPC_ERROR_NONE only if all listeners were |
||||
added. */ |
||||
grpc_error *grpc_tcp_server_add_all_local_addrs(grpc_tcp_server *s, |
||||
unsigned port_index, |
||||
int requested_port, |
||||
int *out_port); |
||||
|
||||
/* Prepare a recently-created socket for listening. */ |
||||
grpc_error *grpc_tcp_server_prepare_socket(int fd, |
||||
const grpc_resolved_address *addr, |
||||
bool so_reuseport, int *port); |
||||
/* Ruturn true if the platform supports ifaddrs */ |
||||
bool grpc_tcp_server_have_ifaddrs(void); |
||||
|
||||
#endif /* GRPC_CORE_LIB_IOMGR_TCP_SERVER_UTILS_POSIX_H */ |
||||
@ -0,0 +1,220 @@ |
||||
/*
|
||||
* |
||||
* Copyright 2017, Google Inc. |
||||
* All rights reserved. |
||||
* |
||||
* Redistribution and use in source and binary forms, with or without |
||||
* modification, are permitted provided that the following conditions are |
||||
* met: |
||||
* |
||||
* * Redistributions of source code must retain the above copyright |
||||
* notice, this list of conditions and the following disclaimer. |
||||
* * Redistributions in binary form must reproduce the above |
||||
* copyright notice, this list of conditions and the following disclaimer |
||||
* in the documentation and/or other materials provided with the |
||||
* distribution. |
||||
* * Neither the name of Google Inc. nor the names of its |
||||
* contributors may be used to endorse or promote products derived from |
||||
* this software without specific prior written permission. |
||||
* |
||||
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
* |
||||
*/ |
||||
|
||||
#include "src/core/lib/iomgr/port.h" |
||||
|
||||
#ifdef GRPC_HAVE_IFADDRS |
||||
|
||||
#include "src/core/lib/iomgr/tcp_server_utils_posix.h" |
||||
|
||||
#include <errno.h> |
||||
#include <limits.h> |
||||
#include <stdio.h> |
||||
#include <string.h> |
||||
|
||||
#include <grpc/support/alloc.h> |
||||
#include <grpc/support/log.h> |
||||
#include <grpc/support/string_util.h> |
||||
#include <grpc/support/sync.h> |
||||
|
||||
#include "src/core/lib/iomgr/error.h" |
||||
#include "src/core/lib/iomgr/sockaddr.h" |
||||
#include "src/core/lib/iomgr/sockaddr_utils.h" |
||||
#include "src/core/lib/iomgr/unix_sockets_posix.h" |
||||
|
||||
#define MIN_SAFE_ACCEPT_QUEUE_SIZE 100 |
||||
|
||||
static gpr_once s_init_max_accept_queue_size; |
||||
static int s_max_accept_queue_size; |
||||
|
||||
/* get max listen queue size on linux */ |
||||
static void init_max_accept_queue_size(void) { |
||||
int n = SOMAXCONN; |
||||
char buf[64]; |
||||
FILE *fp = fopen("/proc/sys/net/core/somaxconn", "r"); |
||||
if (fp == NULL) { |
||||
/* 2.4 kernel. */ |
||||
s_max_accept_queue_size = SOMAXCONN; |
||||
return; |
||||
} |
||||
if (fgets(buf, sizeof buf, fp)) { |
||||
char *end; |
||||
long i = strtol(buf, &end, 10); |
||||
if (i > 0 && i <= INT_MAX && end && *end == 0) { |
||||
n = (int)i; |
||||
} |
||||
} |
||||
fclose(fp); |
||||
s_max_accept_queue_size = n; |
||||
|
||||
if (s_max_accept_queue_size < MIN_SAFE_ACCEPT_QUEUE_SIZE) { |
||||
gpr_log(GPR_INFO, |
||||
"Suspiciously small accept queue (%d) will probably lead to " |
||||
"connection drops", |
||||
s_max_accept_queue_size); |
||||
} |
||||
} |
||||
|
||||
static int get_max_accept_queue_size(void) { |
||||
gpr_once_init(&s_init_max_accept_queue_size, init_max_accept_queue_size); |
||||
return s_max_accept_queue_size; |
||||
} |
||||
|
||||
static grpc_error *add_socket_to_server(grpc_tcp_server *s, int fd, |
||||
const grpc_resolved_address *addr, |
||||
unsigned port_index, unsigned fd_index, |
||||
grpc_tcp_listener **listener) { |
||||
grpc_tcp_listener *sp = NULL; |
||||
int port = -1; |
||||
char *addr_str; |
||||
char *name; |
||||
|
||||
grpc_error *err = |
||||
grpc_tcp_server_prepare_socket(fd, addr, s->so_reuseport, &port); |
||||
if (err == GRPC_ERROR_NONE) { |
||||
GPR_ASSERT(port > 0); |
||||
grpc_sockaddr_to_string(&addr_str, addr, 1); |
||||
gpr_asprintf(&name, "tcp-server-listener:%s", addr_str); |
||||
gpr_mu_lock(&s->mu); |
||||
s->nports++; |
||||
GPR_ASSERT(!s->on_accept_cb && "must add ports before starting server"); |
||||
sp = gpr_malloc(sizeof(grpc_tcp_listener)); |
||||
sp->next = NULL; |
||||
if (s->head == NULL) { |
||||
s->head = sp; |
||||
} else { |
||||
s->tail->next = sp; |
||||
} |
||||
s->tail = sp; |
||||
sp->server = s; |
||||
sp->fd = fd; |
||||
sp->emfd = grpc_fd_create(fd, name); |
||||
memcpy(&sp->addr, addr, sizeof(grpc_resolved_address)); |
||||
sp->port = port; |
||||
sp->port_index = port_index; |
||||
sp->fd_index = fd_index; |
||||
sp->is_sibling = 0; |
||||
sp->sibling = NULL; |
||||
GPR_ASSERT(sp->emfd); |
||||
gpr_mu_unlock(&s->mu); |
||||
gpr_free(addr_str); |
||||
gpr_free(name); |
||||
} |
||||
|
||||
*listener = sp; |
||||
return err; |
||||
} |
||||
|
||||
/* If successful, add a listener to s for addr, set *dsmode for the socket, and
|
||||
return the *listener. */ |
||||
grpc_error *grpc_tcp_server_add_addr(grpc_tcp_server *s, |
||||
const grpc_resolved_address *addr, |
||||
unsigned port_index, unsigned fd_index, |
||||
grpc_dualstack_mode *dsmode, |
||||
grpc_tcp_listener **listener) { |
||||
grpc_resolved_address addr4_copy; |
||||
int fd; |
||||
grpc_error *err = |
||||
grpc_create_dualstack_socket(addr, SOCK_STREAM, 0, dsmode, &fd); |
||||
if (err != GRPC_ERROR_NONE) { |
||||
return err; |
||||
} |
||||
if (*dsmode == GRPC_DSMODE_IPV4 && |
||||
grpc_sockaddr_is_v4mapped(addr, &addr4_copy)) { |
||||
addr = &addr4_copy; |
||||
} |
||||
return add_socket_to_server(s, fd, addr, port_index, fd_index, listener); |
||||
} |
||||
|
||||
/* Prepare a recently-created socket for listening. */ |
||||
grpc_error *grpc_tcp_server_prepare_socket(int fd, |
||||
const grpc_resolved_address *addr, |
||||
bool so_reuseport, int *port) { |
||||
grpc_resolved_address sockname_temp; |
||||
grpc_error *err = GRPC_ERROR_NONE; |
||||
|
||||
GPR_ASSERT(fd >= 0); |
||||
|
||||
if (so_reuseport && !grpc_is_unix_socket(addr)) { |
||||
err = grpc_set_socket_reuse_port(fd, 1); |
||||
if (err != GRPC_ERROR_NONE) goto error; |
||||
} |
||||
|
||||
err = grpc_set_socket_nonblocking(fd, 1); |
||||
if (err != GRPC_ERROR_NONE) goto error; |
||||
err = grpc_set_socket_cloexec(fd, 1); |
||||
if (err != GRPC_ERROR_NONE) goto error; |
||||
if (!grpc_is_unix_socket(addr)) { |
||||
err = grpc_set_socket_low_latency(fd, 1); |
||||
if (err != GRPC_ERROR_NONE) goto error; |
||||
err = grpc_set_socket_reuse_addr(fd, 1); |
||||
if (err != GRPC_ERROR_NONE) goto error; |
||||
} |
||||
err = grpc_set_socket_no_sigpipe_if_possible(fd); |
||||
if (err != GRPC_ERROR_NONE) goto error; |
||||
|
||||
GPR_ASSERT(addr->len < ~(socklen_t)0); |
||||
if (bind(fd, (struct sockaddr *)addr->addr, (socklen_t)addr->len) < 0) { |
||||
err = GRPC_OS_ERROR(errno, "bind"); |
||||
goto error; |
||||
} |
||||
|
||||
if (listen(fd, get_max_accept_queue_size()) < 0) { |
||||
err = GRPC_OS_ERROR(errno, "listen"); |
||||
goto error; |
||||
} |
||||
|
||||
sockname_temp.len = sizeof(struct sockaddr_storage); |
||||
|
||||
if (getsockname(fd, (struct sockaddr *)sockname_temp.addr, |
||||
(socklen_t *)&sockname_temp.len) < 0) { |
||||
err = GRPC_OS_ERROR(errno, "getsockname"); |
||||
goto error; |
||||
} |
||||
|
||||
*port = grpc_sockaddr_get_port(&sockname_temp); |
||||
return GRPC_ERROR_NONE; |
||||
|
||||
error: |
||||
GPR_ASSERT(err != GRPC_ERROR_NONE); |
||||
if (fd >= 0) { |
||||
close(fd); |
||||
} |
||||
grpc_error *ret = grpc_error_set_int( |
||||
GRPC_ERROR_CREATE_REFERENCING("Unable to configure socket", &err, 1), |
||||
GRPC_ERROR_INT_FD, fd); |
||||
GRPC_ERROR_UNREF(err); |
||||
return ret; |
||||
} |
||||
|
||||
#endif /* GRPC_HAVE_IFADDRS */ |
||||
@ -0,0 +1,195 @@ |
||||
/*
|
||||
* |
||||
* Copyright 2017, Google Inc. |
||||
* All rights reserved. |
||||
* |
||||
* Redistribution and use in source and binary forms, with or without |
||||
* modification, are permitted provided that the following conditions are |
||||
* met: |
||||
* |
||||
* * Redistributions of source code must retain the above copyright |
||||
* notice, this list of conditions and the following disclaimer. |
||||
* * Redistributions in binary form must reproduce the above |
||||
* copyright notice, this list of conditions and the following disclaimer |
||||
* in the documentation and/or other materials provided with the |
||||
* distribution. |
||||
* * Neither the name of Google Inc. nor the names of its |
||||
* contributors may be used to endorse or promote products derived from |
||||
* this software without specific prior written permission. |
||||
* |
||||
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
* |
||||
*/ |
||||
|
||||
#include "src/core/lib/iomgr/port.h" |
||||
|
||||
#ifdef GRPC_HAVE_IFADDRS |
||||
|
||||
#include "src/core/lib/iomgr/tcp_server_utils_posix.h" |
||||
|
||||
#include <errno.h> |
||||
#include <ifaddrs.h> |
||||
#include <stddef.h> |
||||
#include <string.h> |
||||
|
||||
#include <grpc/support/alloc.h> |
||||
#include <grpc/support/log.h> |
||||
#include <grpc/support/string_util.h> |
||||
|
||||
#include "src/core/lib/iomgr/error.h" |
||||
#include "src/core/lib/iomgr/sockaddr.h" |
||||
#include "src/core/lib/iomgr/sockaddr_utils.h" |
||||
|
||||
/* Return the listener in s with address addr or NULL. */ |
||||
static grpc_tcp_listener *find_listener_with_addr(grpc_tcp_server *s, |
||||
grpc_resolved_address *addr) { |
||||
grpc_tcp_listener *l; |
||||
gpr_mu_lock(&s->mu); |
||||
for (l = s->head; l != NULL; l = l->next) { |
||||
if (l->addr.len != addr->len) { |
||||
continue; |
||||
} |
||||
if (memcmp(l->addr.addr, addr->addr, addr->len) == 0) { |
||||
break; |
||||
} |
||||
} |
||||
gpr_mu_unlock(&s->mu); |
||||
return l; |
||||
} |
||||
|
||||
/* Bind to "::" to get a port number not used by any address. */ |
||||
static grpc_error *get_unused_port(int *port) { |
||||
grpc_resolved_address wild; |
||||
grpc_sockaddr_make_wildcard6(0, &wild); |
||||
grpc_dualstack_mode dsmode; |
||||
int fd; |
||||
grpc_error *err = |
||||
grpc_create_dualstack_socket(&wild, SOCK_STREAM, 0, &dsmode, &fd); |
||||
if (err != GRPC_ERROR_NONE) { |
||||
return err; |
||||
} |
||||
if (dsmode == GRPC_DSMODE_IPV4) { |
||||
grpc_sockaddr_make_wildcard4(0, &wild); |
||||
} |
||||
if (bind(fd, (const struct sockaddr *)wild.addr, (socklen_t)wild.len) != 0) { |
||||
err = GRPC_OS_ERROR(errno, "bind"); |
||||
close(fd); |
||||
return err; |
||||
} |
||||
if (getsockname(fd, (struct sockaddr *)wild.addr, (socklen_t *)&wild.len) != |
||||
0) { |
||||
err = GRPC_OS_ERROR(errno, "getsockname"); |
||||
close(fd); |
||||
return err; |
||||
} |
||||
close(fd); |
||||
*port = grpc_sockaddr_get_port(&wild); |
||||
return *port <= 0 ? GRPC_ERROR_CREATE("Bad port") : GRPC_ERROR_NONE; |
||||
} |
||||
|
||||
grpc_error *grpc_tcp_server_add_all_local_addrs(grpc_tcp_server *s, |
||||
unsigned port_index, |
||||
int requested_port, |
||||
int *out_port) { |
||||
struct ifaddrs *ifa = NULL; |
||||
struct ifaddrs *ifa_it; |
||||
unsigned fd_index = 0; |
||||
grpc_tcp_listener *sp = NULL; |
||||
grpc_error *err = GRPC_ERROR_NONE; |
||||
if (requested_port == 0) { |
||||
/* Note: There could be a race where some local addrs can listen on the
|
||||
selected port and some can't. The sane way to handle this would be to |
||||
retry by recreating the whole grpc_tcp_server. Backing out individual |
||||
listeners and orphaning the FDs looks like too much trouble. */ |
||||
if ((err = get_unused_port(&requested_port)) != GRPC_ERROR_NONE) { |
||||
return err; |
||||
} else if (requested_port <= 0) { |
||||
return GRPC_ERROR_CREATE("Bad get_unused_port()"); |
||||
} |
||||
gpr_log(GPR_DEBUG, "Picked unused port %d", requested_port); |
||||
} |
||||
if (getifaddrs(&ifa) != 0 || ifa == NULL) { |
||||
return GRPC_OS_ERROR(errno, "getifaddrs"); |
||||
} |
||||
for (ifa_it = ifa; ifa_it != NULL; ifa_it = ifa_it->ifa_next) { |
||||
grpc_resolved_address addr; |
||||
char *addr_str = NULL; |
||||
grpc_dualstack_mode dsmode; |
||||
grpc_tcp_listener *new_sp = NULL; |
||||
const char *ifa_name = (ifa_it->ifa_name ? ifa_it->ifa_name : "<unknown>"); |
||||
if (ifa_it->ifa_addr == NULL) { |
||||
continue; |
||||
} else if (ifa_it->ifa_addr->sa_family == AF_INET) { |
||||
addr.len = sizeof(struct sockaddr_in); |
||||
} else if (ifa_it->ifa_addr->sa_family == AF_INET6) { |
||||
addr.len = sizeof(struct sockaddr_in6); |
||||
} else { |
||||
continue; |
||||
} |
||||
memcpy(addr.addr, ifa_it->ifa_addr, addr.len); |
||||
if (!grpc_sockaddr_set_port(&addr, requested_port)) { |
||||
/* Should never happen, because we check sa_family above. */ |
||||
err = GRPC_ERROR_CREATE("Failed to set port"); |
||||
break; |
||||
} |
||||
if (grpc_sockaddr_to_string(&addr_str, &addr, 0) < 0) { |
||||
addr_str = gpr_strdup("<error>"); |
||||
} |
||||
gpr_log(GPR_DEBUG, |
||||
"Adding local addr from interface %s flags 0x%x to server: %s", |
||||
ifa_name, ifa_it->ifa_flags, addr_str); |
||||
/* We could have multiple interfaces with the same address (e.g., bonding),
|
||||
so look for duplicates. */ |
||||
if (find_listener_with_addr(s, &addr) != NULL) { |
||||
gpr_log(GPR_DEBUG, "Skipping duplicate addr %s on interface %s", addr_str, |
||||
ifa_name); |
||||
gpr_free(addr_str); |
||||
continue; |
||||
} |
||||
if ((err = grpc_tcp_server_add_addr(s, &addr, port_index, fd_index, &dsmode, |
||||
&new_sp)) != GRPC_ERROR_NONE) { |
||||
char *err_str = NULL; |
||||
grpc_error *root_err; |
||||
if (gpr_asprintf(&err_str, "Failed to add listener: %s", addr_str) < 0) { |
||||
err_str = gpr_strdup("Failed to add listener"); |
||||
} |
||||
root_err = GRPC_ERROR_CREATE(err_str); |
||||
gpr_free(err_str); |
||||
gpr_free(addr_str); |
||||
err = grpc_error_add_child(root_err, err); |
||||
break; |
||||
} else { |
||||
GPR_ASSERT(requested_port == new_sp->port); |
||||
++fd_index; |
||||
if (sp != NULL) { |
||||
new_sp->is_sibling = 1; |
||||
sp->sibling = new_sp; |
||||
} |
||||
sp = new_sp; |
||||
} |
||||
gpr_free(addr_str); |
||||
} |
||||
freeifaddrs(ifa); |
||||
if (err != GRPC_ERROR_NONE) { |
||||
return err; |
||||
} else if (sp == NULL) { |
||||
return GRPC_ERROR_CREATE("No local addresses"); |
||||
} else { |
||||
*out_port = sp->port; |
||||
return GRPC_ERROR_NONE; |
||||
} |
||||
} |
||||
|
||||
bool grpc_tcp_server_have_ifaddrs(void) { return true; } |
||||
|
||||
#endif /* GRPC_HAVE_IFADDRS */ |
||||
@ -0,0 +1,49 @@ |
||||
/*
|
||||
* |
||||
* Copyright 2017, Google Inc. |
||||
* All rights reserved. |
||||
* |
||||
* Redistribution and use in source and binary forms, with or without |
||||
* modification, are permitted provided that the following conditions are |
||||
* met: |
||||
* |
||||
* * Redistributions of source code must retain the above copyright |
||||
* notice, this list of conditions and the following disclaimer. |
||||
* * Redistributions in binary form must reproduce the above |
||||
* copyright notice, this list of conditions and the following disclaimer |
||||
* in the documentation and/or other materials provided with the |
||||
* distribution. |
||||
* * Neither the name of Google Inc. nor the names of its |
||||
* contributors may be used to endorse or promote products derived from |
||||
* this software without specific prior written permission. |
||||
* |
||||
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
* |
||||
*/ |
||||
|
||||
#include "src/core/lib/iomgr/port.h" |
||||
|
||||
#if defined(GRPC_POSIX_SOCKET) && !defined(GRPC_HAVE_IFADDRS) |
||||
|
||||
#include "src/core/lib/iomgr/tcp_server_utils_posix.h" |
||||
|
||||
grpc_error *grpc_tcp_server_add_all_local_addrs(grpc_tcp_server *s, |
||||
unsigned port_index, |
||||
int requested_port, |
||||
int *out_port) { |
||||
return GRPC_ERROR_CREATE("no ifaddrs available"); |
||||
} |
||||
|
||||
bool grpc_tcp_server_have_ifaddrs(void) { return false; } |
||||
|
||||
#endif /* defined(GRPC_POSIX_SOCKET) && !defined(GRPC_HAVE_IFADDRS) */ |
||||
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Loading…
Reference in new issue