The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#) https://grpc.io/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
Jan Tattermusch 84c74b67f8 build C++ in benchmarks with cmake 6 years ago
..
OWNERS remove matt-kwong from OWNERS 6 years ago
README.md Wait a configurable amount of time for benchmark channels to become ready 6 years ago
__init__.py
bq_upload_result.py correctly set performance job metadata on kokoro 6 years ago
build_performance.sh build C++ in benchmarks with cmake 6 years ago
build_performance_go.sh
build_performance_node.sh Remove added line in node perf build script 6 years ago
build_performance_php7.sh
kill_workers.sh
massage_qps_stats.py format generated code 7 years ago
massage_qps_stats_helpers.py
patch_scenario_results_schema.py
process_local_perf_flamegraphs.sh
process_remote_perf_flamegraphs.sh Fix process_remote_perf_flamegraphs.sh to pass shellcheck 7 years ago
remote_host_build.sh Fix remote_host_build.sh to pass shellcheck 7 years ago
remote_host_prepare.sh make sure port server runs locally too 6 years ago
run_netperf.sh Fix run_netperf.sh to pass shellcheck 7 years ago
run_qps_driver.sh build C++ in benchmarks with cmake 6 years ago
run_worker_csharp.sh Fix run_worker_csharp.sh to pass shellcheck 7 years ago
run_worker_go.sh Fix run_worker_go.sh to pass shellcheck 7 years ago
run_worker_java.sh Fix run_worker_java.sh to pass shellcheck 7 years ago
run_worker_node.sh Make Node perf build script idempotent, update to newer Node version 6 years ago
run_worker_php.sh Fix run_worker_php.sh to pass shellcheck 7 years ago
run_worker_python.sh fix run_python_qps_worker script 7 years ago
run_worker_ruby.sh Fix run_worker_ruby.sh to pass shellcheck 7 years ago
scenario_config.py build C++ in benchmarks with cmake 6 years ago
scenario_result_schema.json Add epollex fd cache 7 years ago

README.md

Overview of performance test suite, with steps for manual runs:

For design of the tests, see https://grpc.io/docs/guides/benchmarking.html.

Pre-reqs for running these manually:

In general the benchmark workers and driver build scripts expect linux_performance_worker_init.sh to have been ran already.

To run benchmarks locally:

On remote machines, to start the driver and workers manually:

The run_performance_test.py top-level runner script can also be used with remote machines, but for e.g., profiling the server, it might be useful to run workers manually.

  1. You'll need a "driver" and separate "worker" machines. For example, you might use one GCE "driver" machine and 3 other GCE "worker" machines that are in the same zone.

  2. Connect to each worker machine and start up a benchmark worker with a "driver_port".

Comands to start workers in different languages:

Running benchmark workers for C-core wrapped languages (C++, Python, C#, Node, Ruby):
  • These are more simple since they all live in the main grpc repo.
$ cd <grpc_repo_root>
$ tools/run_tests/performance/build_performance.sh
$ tools/run_tests/performance/run_worker_<language>.sh
Running benchmark workers for gRPC-Java:
$ cd <grpc-java-repo>
$ ./gradlew -PskipCodegen=true :grpc-benchmarks:installDist
$ benchmarks/build/install/grpc-benchmarks/bin/benchmark_worker --driver_port <driver_port>
Running benchmark workers for gRPC-Go:
$ cd <grpc-go-repo>/benchmark/worker && go install
$ # if profiling, it might be helpful to turn off inlining by building with "-gcflags=-l"
$ $GOPATH/bin/worker --driver_port <driver_port>

Build the driver:

  • Connect to the driver machine (if using a remote driver) and from the grpc repo root:
$ tools/run_tests/performance/build_performance.sh

Run the driver:

  1. Get the 'scenario_json' relevant for the scenario to run. Note that "scenario json" configs are generated from scenario_config.py. The driver takes a list of these configs as a json string of the form: {scenario: <json_list_of_scenarios> } in its --scenarios_json command argument. One quick way to get a valid json string to pass to the driver is by running the run_performance_tests.py locally and copying the logged scenario json command arg.

  2. From the grpc repo root:

  • Set QPS_WORKERS environment variable to a comma separated list of worker machines. Note that the driver will start the "benchmark server" on the first entry in the list, and the rest will be told to run as clients against the benchmark server.

Example running and profiling of go benchmark server:

$ export QPS_WORKERS=<host1>:<10000>,<host2>,10000,<host3>:10000
$ bins/opt/qps_json_driver --scenario_json='<scenario_json_scenario_config_string>'

Example profiling commands

While running the benchmark, a profiler can be attached to the server.

Example to count syscalls in grpc-go server during a benchmark:

  • Connect to server machine and run:
$ netstat -tulpn | grep <driver_port> # to get pid of worker
$ perf stat -p <worker_pid> -e syscalls:sys_enter_write # stop after test complete

Example memory profile of grpc-go server, with go tools pprof:

  • After a run is done on the server, see its alloc profile with:
$ go tool pprof --text --alloc_space http://localhost:<pprof_port>/debug/heap

Configuration environment variables:

  • QPS_WORKER_CHANNEL_CONNECT_TIMEOUT

    Consuming process: qps_worker

    Type: integer (number of seconds)

    This can be used to configure the amount of time that benchmark clients wait for channels to the benchmark server to become ready. This is useful in certain benchmark environments in which the server can take a long time to become ready. Note: if setting this to a high value, then the scenario config under test should probably also have a large "warmup_seconds".

  • QPS_WORKERS

    Consuming process: qps_json_driver

    Type: comma separated list of host:port

    Set this to a comma separated list of QPS worker processes/machines. Each scenario in a scenario config has specifies a certain number of servers, num_servers, and the driver will start "benchmark servers"'s on the first num_server host:port pairs in the comma separated list. The rest will be told to run as clients against the benchmark server.