mirror of https://github.com/grpc/grpc.git
yang-g
9 years ago
commit
b36a582c96
49 changed files with 696 additions and 146 deletions
@ -0,0 +1,77 @@ |
||||
Connection Backoff Interop Test Descriptions |
||||
=============================================== |
||||
|
||||
This test is to verify the client is reconnecting the server with correct |
||||
backoffs as specified in |
||||
[the spec](http://github.com/grpc/grpc/blob/master/doc/connection-backoff.md). |
||||
The test server has a port (control_port) running a rpc service for controlling |
||||
the server and another port (retry_port) to close any incoming tcp connections. |
||||
The test has the following flow: |
||||
|
||||
1. The server starts listening on control_port. |
||||
2. The client calls Start rpc on server control_port. |
||||
3. The server starts listening on retry_port. |
||||
4. The client connects to server retry_port and retries with backoff for 540s, |
||||
which translates to about 13 retries. |
||||
5. The client calls Stop rpc on server control port. |
||||
6. The client checks the response to see whether the server thinks the backoffs |
||||
are conforming the spec or do its own check on the backoffs in the response. |
||||
|
||||
Client and server use |
||||
[test.proto](https://github.com/grpc/grpc/blob/master/test/proto/test.proto). |
||||
Each language should implement its own client. The C++ server is shared among |
||||
languages. |
||||
|
||||
Client |
||||
------ |
||||
|
||||
Clients should accept these arguments: |
||||
* --server_control_port=PORT |
||||
* The server port to connect to for rpc. For example, "8080" |
||||
* --server_retry_port=PORT |
||||
* The server port to connect to for testing backoffs. For example, "8081" |
||||
|
||||
The client must connect to the control port without TLS. The client should |
||||
either assert on the server returned backoff status or check the returned |
||||
backoffs on its own. |
||||
|
||||
Procedure of client: |
||||
|
||||
1. Calls Start on server control port with a large deadline or no deadline, |
||||
waits for its finish and checks it succeeded. |
||||
2. Initiates a channel connection to server retry port, which should perform |
||||
reconnections with proper backoffs. A convienent way to achieve this is to |
||||
call Start with a deadline of 540s. The rpc should fail with deadline exceeded. |
||||
3. Calls Stop on server control port and checks it succeeded. |
||||
4. Checks the response to see whether the server thinks the backoffs passed the |
||||
test. |
||||
5. Optionally, the client can do its own check on the returned backoffs. |
||||
|
||||
|
||||
Server |
||||
------ |
||||
|
||||
A C++ server can be used for the test. Other languages do NOT need to implement |
||||
a server. To minimize the network delay, the server binary should run on the |
||||
same machine or on a nearby machine (in terms of network distance) with the |
||||
client binary. |
||||
|
||||
A server implements the ReconnectService to its state. It also opens a |
||||
tcp server on the retry_port, which just shuts down all incoming tcp |
||||
connections to simulate connection failures. The server will keep a record of |
||||
all the reconnection timestamps and return the connection backoffs in the |
||||
response in milliseconds. The server also checks the backoffs to see whether |
||||
they conform the spec and returns whether the client passes the test. |
||||
|
||||
If the server receives a Start call when another client is being tested, it |
||||
finishes the call when the other client is done. If some other host connects |
||||
to the server retry_port when a client is being tested, the server will log an |
||||
error but likely would think the client fails the test. |
||||
|
||||
The server accepts these arguments: |
||||
|
||||
* --control_port=PORT |
||||
* The port to listen on for control rpcs. For example, "8080" |
||||
* --retry_port=PORT |
||||
* The tcp server port. For example, "8081" |
||||
|
||||
@ -1 +1,2 @@ |
||||
graft grpc |
||||
include commands.py |
||||
|
||||
@ -0,0 +1,75 @@ |
||||
# Copyright 2015, Google Inc. |
||||
# All rights reserved. |
||||
# |
||||
# Redistribution and use in source and binary forms, with or without |
||||
# modification, are permitted provided that the following conditions are |
||||
# met: |
||||
# |
||||
# * Redistributions of source code must retain the above copyright |
||||
# notice, this list of conditions and the following disclaimer. |
||||
# * Redistributions in binary form must reproduce the above |
||||
# copyright notice, this list of conditions and the following disclaimer |
||||
# in the documentation and/or other materials provided with the |
||||
# distribution. |
||||
# * Neither the name of Google Inc. nor the names of its |
||||
# contributors may be used to endorse or promote products derived from |
||||
# this software without specific prior written permission. |
||||
# |
||||
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
|
||||
"""Provides distutils command classes for the GRPC Python setup process.""" |
||||
|
||||
import os |
||||
import os.path |
||||
import sys |
||||
|
||||
import setuptools |
||||
|
||||
_CONF_PY_ADDENDUM = """ |
||||
extensions.append('sphinx.ext.napoleon') |
||||
napoleon_google_docstring = True |
||||
napoleon_numpy_docstring = True |
||||
|
||||
html_theme = 'sphinx_rtd_theme' |
||||
""" |
||||
|
||||
class SphinxDocumentation(setuptools.Command): |
||||
"""Command to generate documentation via sphinx.""" |
||||
|
||||
description = '' |
||||
user_options = [] |
||||
|
||||
def initialize_options(self): |
||||
pass |
||||
|
||||
def finalize_options(self): |
||||
pass |
||||
|
||||
def run(self): |
||||
# We import here to ensure that setup.py has had a chance to install the |
||||
# relevant package eggs first. |
||||
import sphinx |
||||
import sphinx.apidoc |
||||
metadata = self.distribution.metadata |
||||
src_dir = os.path.join( |
||||
os.getcwd(), self.distribution.package_dir['grpc']) |
||||
sys.path.append(src_dir) |
||||
sphinx.apidoc.main([ |
||||
'', '--force', '--full', '-H', metadata.name, '-A', metadata.author, |
||||
'-V', metadata.version, '-R', metadata.version, |
||||
'-o', os.path.join('doc', 'src'), src_dir]) |
||||
conf_filepath = os.path.join('doc', 'src', 'conf.py') |
||||
with open(conf_filepath, 'a') as conf_file: |
||||
conf_file.write(_CONF_PY_ADDENDUM) |
||||
sphinx.main(['', os.path.join('doc', 'src'), os.path.join('doc', 'build')]) |
||||
|
||||
@ -0,0 +1,2 @@ |
||||
[build_ext] |
||||
inplace=1 |
||||
@ -0,0 +1 @@ |
||||
distrib_virtualenv/ |
||||
@ -0,0 +1,113 @@ |
||||
#!/usr/bin/env python |
||||
# Copyright 2015, Google Inc. |
||||
# All rights reserved. |
||||
# |
||||
# Redistribution and use in source and binary forms, with or without |
||||
# modification, are permitted provided that the following conditions are |
||||
# met: |
||||
# |
||||
# * Redistributions of source code must retain the above copyright |
||||
# notice, this list of conditions and the following disclaimer. |
||||
# * Redistributions in binary form must reproduce the above |
||||
# copyright notice, this list of conditions and the following disclaimer |
||||
# in the documentation and/or other materials provided with the |
||||
# distribution. |
||||
# * Neither the name of Google Inc. nor the names of its |
||||
# contributors may be used to endorse or promote products derived from |
||||
# this software without specific prior written permission. |
||||
# |
||||
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
|
||||
import argparse |
||||
import os |
||||
import os.path |
||||
import shutil |
||||
import subprocess |
||||
import tempfile |
||||
|
||||
parser = argparse.ArgumentParser() |
||||
parser.add_argument('--config', metavar='c', type=str, nargs=1, |
||||
help='GRPC/GPR libraries build configuration', |
||||
default='opt') |
||||
parser.add_argument('--submit', action='store_true') |
||||
parser.add_argument('--gh-user', type=str, help='GitHub user to push as.') |
||||
parser.add_argument('--gh-repo-owner', type=str, |
||||
help=('Owner of the GitHub repository to be pushed; ' |
||||
'defaults to --gh-user.')) |
||||
parser.add_argument('--doc-branch', type=str) |
||||
args = parser.parse_args() |
||||
|
||||
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) |
||||
PROJECT_ROOT = os.path.abspath(os.path.join(SCRIPT_DIR, '..', '..', '..')) |
||||
|
||||
CONFIG = args.config |
||||
SETUP_PATH = os.path.join(PROJECT_ROOT, 'src/python/src/setup.py') |
||||
DOC_PATH = os.path.join(PROJECT_ROOT, 'src/python/src/doc/build') |
||||
INCLUDE_PATH = os.path.join(PROJECT_ROOT, 'include') |
||||
LIBRARY_PATH = os.path.join(PROJECT_ROOT, 'libs/{}'.format(CONFIG)) |
||||
VIRTUALENV_DIR = os.path.join(SCRIPT_DIR, 'distrib_virtualenv') |
||||
VIRTUALENV_PYTHON_PATH = os.path.join(VIRTUALENV_DIR, 'bin', 'python') |
||||
|
||||
environment = os.environ.copy() |
||||
environment.update({ |
||||
'CONFIG': CONFIG, |
||||
'CFLAGS': '-I{}'.format(INCLUDE_PATH), |
||||
'LDFLAGS': '-L{}'.format(LIBRARY_PATH), |
||||
'LD_LIBRARY_PATH': LIBRARY_PATH |
||||
}) |
||||
|
||||
subprocess_arguments_list = [ |
||||
{'args': ['make'], 'cwd': PROJECT_ROOT}, |
||||
{'args': ['virtualenv', VIRTUALENV_DIR], 'env': environment}, |
||||
{'args': [VIRTUALENV_PYTHON_PATH, SETUP_PATH, 'build'], 'env': environment}, |
||||
{'args': [VIRTUALENV_PYTHON_PATH, SETUP_PATH, 'doc'], 'env': environment}, |
||||
] |
||||
|
||||
for subprocess_arguments in subprocess_arguments_list: |
||||
subprocess.check_call(**subprocess_arguments) |
||||
|
||||
if args.submit: |
||||
assert args.gh_user |
||||
assert args.doc_branch |
||||
github_user = args.gh_user |
||||
github_repository_owner = ( |
||||
args.gh_repo_owner if args.gh_repo_owner else gh_user) |
||||
# Create a temporary directory out of tree, checkout gh-pages from the |
||||
# specified repository, edit it, and push it. It's up to the user to then go |
||||
# onto GitHub and make a PR against grpc/grpc:gh-pages. |
||||
repo_parent_dir = tempfile.mkdtemp() |
||||
repo_dir = os.path.join(repo_parent_dir, 'grpc') |
||||
python_doc_dir = os.path.join(repo_dir, 'python') |
||||
doc_branch = args.doc_branch |
||||
|
||||
subprocess.check_call([ |
||||
'git', 'clone', 'https://{}@github.com/{}/grpc'.format( |
||||
github_user, github_repository_owner) |
||||
], cwd=repo_parent_dir) |
||||
subprocess.check_call([ |
||||
'git', 'remote', 'add', 'upstream', 'https://github.com/grpc/grpc' |
||||
], cwd=repo_dir) |
||||
subprocess.check_call(['git', 'fetch', 'upstream'], cwd=repo_dir) |
||||
subprocess.check_call([ |
||||
'git', 'checkout', 'upstream/gh-pages', '-b', doc_branch |
||||
], cwd=repo_dir) |
||||
shutil.rmtree(python_doc_dir, ignore_errors=True) |
||||
shutil.copytree(DOC_PATH, python_doc_dir) |
||||
subprocess.check_call(['git', 'add', '--all'], cwd=repo_dir) |
||||
subprocess.check_call([ |
||||
'git', 'commit', '-m', 'Auto-update Python documentation' |
||||
], cwd=repo_dir) |
||||
subprocess.check_call([ |
||||
'git', 'push', '--set-upstream', 'origin', doc_branch |
||||
], cwd=repo_dir) |
||||
shutil.rmtree(repo_parent_dir) |
||||
@ -0,0 +1,36 @@ |
||||
import argparse |
||||
import xml.etree.cElementTree as ET |
||||
import jobset |
||||
|
||||
argp = argparse.ArgumentParser(description='Run interop tests.') |
||||
argp.add_argument('-l', '--language', |
||||
choices=['build_only', 'c++'], |
||||
nargs='+', |
||||
default=['build_only']) |
||||
args = argp.parse_args() |
||||
|
||||
# build job |
||||
build_steps = 'tools/run_tests/run_interops_build.sh' |
||||
build_job = jobset.JobSpec(cmdline=build_steps, shortname='build') |
||||
|
||||
# test jobs |
||||
_TESTS = ['large_unary', 'empty_unary', 'ping_pong', 'client_streaming', 'server_streaming'] |
||||
jobs = [] |
||||
jobNumber = 0 |
||||
for lang in args.language: |
||||
for test in _TESTS: |
||||
test_job = jobset.JobSpec(cmdline=['tools/run_tests/run_interops_test.sh', '%s' % lang, '%s' % test], shortname=test) |
||||
jobs.append(test_job) |
||||
jobNumber+=1 |
||||
|
||||
root = ET.Element('testsuites') |
||||
testsuite = ET.SubElement(root, 'testsuite', id='1', package='grpc', name='tests') |
||||
|
||||
# always do the build of docker first, and then all the tests can run in parallel |
||||
jobset.run([build_job], maxjobs=1, xml_report=testsuite) |
||||
jobset.run(jobs, maxjobs=jobNumber, xml_report=testsuite) |
||||
|
||||
tree = ET.ElementTree(root) |
||||
tree.write('report.xml', encoding='UTF-8') |
||||
|
||||
|
||||
@ -0,0 +1,47 @@ |
||||
#!/bin/sh |
||||
|
||||
# Copyright 2015, Google Inc. |
||||
# All rights reserved. |
||||
# |
||||
# Redistribution and use in source and binary forms, with or without |
||||
# modification, are permitted provided that the following conditions are |
||||
# met: |
||||
# |
||||
# * Redistributions of source code must retain the above copyright |
||||
# notice, this list of conditions and the following disclaimer. |
||||
# * Redistributions in binary form must reproduce the above |
||||
# copyright notice, this list of conditions and the following disclaimer |
||||
# in the documentation and/or other materials provided with the |
||||
# distribution. |
||||
# * Neither the name of Google Inc. nor the names of its |
||||
# contributors may be used to endorse or promote products derived from |
||||
# this software without specific prior written permission. |
||||
# |
||||
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
|
||||
set -e |
||||
|
||||
#clean up any old docker files and start mirroring repository if not started already |
||||
sudo docker rmi -f grpc/cxx || true |
||||
sudo docker rmi -f grpc/base || true |
||||
sudo docker rmi -f 0.0.0.0:5000/grpc/base || true |
||||
sudo docker run -d -e GCS_BUCKET=docker-interop-images -e STORAGE_PATH=/admin/docker_images -p 5000:5000 google/docker-registry || true |
||||
|
||||
#prepare building by pulling down base images and necessary files |
||||
sudo docker pull 0.0.0.0:5000/grpc/base |
||||
sudo docker tag -f 0.0.0.0:5000/grpc/base grpc/base |
||||
gsutil cp -R gs://docker-interop-images/admin/service_account tools/dockerfile/grpc_cxx |
||||
gsutil cp -R gs://docker-interop-images/admin/cacerts tools/dockerfile/grpc_cxx |
||||
|
||||
#build docker file, add more languages later |
||||
sudo docker build --no-cache -t grpc/cxx tools/dockerfile/grpc_cxx |
||||
@ -0,0 +1,43 @@ |
||||
#!/bin/sh |
||||
|
||||
# Copyright 2015, Google Inc. |
||||
# All rights reserved. |
||||
# |
||||
# Redistribution and use in source and binary forms, with or without |
||||
# modification, are permitted provided that the following conditions are |
||||
# met: |
||||
# |
||||
# * Redistributions of source code must retain the above copyright |
||||
# notice, this list of conditions and the following disclaimer. |
||||
# * Redistributions in binary form must reproduce the above |
||||
# copyright notice, this list of conditions and the following disclaimer |
||||
# in the documentation and/or other materials provided with the |
||||
# distribution. |
||||
# * Neither the name of Google Inc. nor the names of its |
||||
# contributors may be used to endorse or promote products derived from |
||||
# this software without specific prior written permission. |
||||
# |
||||
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
||||
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
||||
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
||||
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
||||
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
||||
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
||||
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
||||
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
||||
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
||||
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
||||
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||
|
||||
language=$1 |
||||
test_case=$2 |
||||
|
||||
set -e |
||||
if [ "$language" = "c++" ] |
||||
then |
||||
sudo docker run grpc/cxx /var/local/git/grpc/bins/opt/interop_client --enable_ssl --use_prod_roots --server_host_override=grpc-test.sandbox.google.com --server_host=grpc-test.sandbox.google.com --server_port=443 --test_case=$test_case |
||||
else |
||||
echo "interop testss not added for $language" |
||||
exit 1 |
||||
fi |
||||
|
||||
Loading…
Reference in new issue