@ -1,161 +1,14 @@
# Overview of performance test suite, with steps for manual runs:
# Overview of performance test suite
For design of the tests, see https://grpc.io/docs/guides/benchmarking.
For scripts related to the GKE-based performance test suite (in development),
see [gRPC OSS benchmarks ](#grpc-oss-benchmarks ).
This document contains documentation of on how to run gRPC end-to-end benchmarks
using the gRPC OSS benchmarks framework (recommended) or how to run them
manually (for experts only).
## Pre-reqs for running these manually:
## Approach 1: Use gRPC OSS benchmarks framework (Recommended)
In general the benchmark workers and driver build scripts expect
[linux_performance_worker_init.sh ](../../gce/linux_performance_worker_init.sh )
to have been ran already.
### To run benchmarks locally:
- From the grpc repo root, start the
[run_performance_tests.py ](../run_performance_tests.py ) runner script.
### On remote machines, to start the driver and workers manually:
The [run_performance_test.py ](../run_performance_tests.py ) top-level runner
script can also be used with remote machines, but for e.g., profiling the
server, it might be useful to run workers manually.
1. You'll need a "driver" and separate "worker" machines. For example, you might
use one GCE "driver" machine and 3 other GCE "worker" machines that are in
the same zone.
2. Connect to each worker machine and start up a benchmark worker with a
"driver_port".
- For example, to start the grpc-go benchmark worker:
[grpc-go worker main.go ](https://github.com/grpc/grpc-go/blob/master/benchmark/worker/main.go )
--driver_port < driver_port >
#### Commands to start workers in different languages:
- Note that these commands are what the top-level
[run_performance_test.py ](../run_performance_tests.py ) script uses to build
and run different workers through the
[build_performance.sh ](./build_performance.sh ) script and "run worker" scripts
(such as the [run_worker_java.sh ](./run_worker_java.sh )).
##### Running benchmark workers for C-core wrapped languages (C++, Python, C#, Node, Ruby):
- These are more simple since they all live in the main grpc repo.
```
$ cd < grpc_repo_root >
$ tools/run_tests/performance/build_performance.sh
$ tools/run_tests/performance/run_worker_< language > .sh
```
- Note that there is one "run_worker" script per language, e.g.,
[run_worker_csharp.sh ](./run_worker_csharp.sh ) for c#.
##### Running benchmark workers for gRPC-Java:
- You'll need the [grpc-java ](https://github.com/grpc/grpc-java ) repo.
```
$ cd < grpc-java-repo >
$ ./gradlew -PskipCodegen=true -PskipAndroid=true :grpc-benchmarks:installDist
$ benchmarks/build/install/grpc-benchmarks/bin/benchmark_worker --driver_port < driver_port >
```
##### Running benchmark workers for gRPC-Go:
- You'll need the [grpc-go repo ](https://github.com/grpc/grpc-go )
```
$ cd < grpc-go-repo > /benchmark/worker & & go install
$ # if profiling, it might be helpful to turn off inlining by building with "-gcflags=-l"
$ $GOPATH/bin/worker --driver_port < driver_port >
```
#### Build the driver:
- Connect to the driver machine (if using a remote driver) and from the grpc
repo root:
```
$ tools/run_tests/performance/build_performance.sh
```
#### Run the driver:
1. Get the 'scenario_json' relevant for the scenario to run. Note that "scenario
json" configs are generated from [scenario_config.py ](./scenario_config.py ).
The [driver ](../../../test/cpp/qps/qps_json_driver.cc ) takes a list of these
configs as a json string of the form: `{scenario: <json_list_of_scenarios> }`
in its `--scenarios_json` command argument. One quick way to get a valid json
string to pass to the driver is by running the
[run_performance_tests.py ](./run_performance_tests.py ) locally and copying
the logged scenario json command arg.
2. From the grpc repo root:
- Set `QPS_WORKERS` environment variable to a comma separated list of worker
machines. Note that the driver will start the "benchmark server" on the first
entry in the list, and the rest will be told to run as clients against the
benchmark server.
Example running and profiling of go benchmark server:
```
$ export QPS_WORKERS=< host1 > :< 10000 > ,< host2 > ,10000,< host3 > :10000
$ bins/opt/qps_json_driver --scenario_json='< scenario_json_scenario_config_string > '
```
### Example profiling commands
While running the benchmark, a profiler can be attached to the server.
Example to count syscalls in grpc-go server during a benchmark:
- Connect to server machine and run:
```
$ netstat -tulpn | grep < driver_port > # to get pid of worker
$ perf stat -p < worker_pid > -e syscalls:sys_enter_write # stop after test complete
```
Example memory profile of grpc-go server, with `go tools pprof` :
- After a run is done on the server, see its alloc profile with:
```
$ go tool pprof --text --alloc_space http://localhost:< pprof_port > /debug/heap
```
### Configuration environment variables:
- QPS_WORKER_CHANNEL_CONNECT_TIMEOUT
Consuming process: qps_worker
Type: integer (number of seconds)
This can be used to configure the amount of time that benchmark clients wait
for channels to the benchmark server to become ready. This is useful in
certain benchmark environments in which the server can take a long time to
become ready. Note: if setting this to a high value, then the scenario config
under test should probably also have a large "warmup_seconds".
- QPS_WORKERS
Consuming process: qps_json_driver
Type: comma separated list of host:port
Set this to a comma separated list of QPS worker processes/machines. Each
scenario in a scenario config has specifies a certain number of servers,
`num_servers` , and the driver will start "benchmark servers"'s on the first
`num_server` `host:port` pairs in the comma separated list. The rest will be
told to run as clients against the benchmark server.
## gRPC OSS benchmarks
### gRPC OSS benchmarks
The scripts in this section generate LoadTest configurations for the GKE-based
gRPC OSS benchmarks framework. This framework is stored in a separate
@ -165,7 +18,7 @@ These scripts, together with tools defined in [grpc/test-infra], are used in the
continuous integration setup defined in [grpc_e2e_performance_gke.sh] and
[grpc_e2e_performance_v2.sh].
### Generating scenarios
#### Generating scenarios
The benchmarks framework uses the same test scenarios as the legacy one. The
script [scenario_config_exporter.py ](./scenario_config_exporter.py ) can be used
@ -204,7 +57,7 @@ Count Language Client Server Categories
Client and server languages are only set for cross-language scenarios, where the
client or server language do not match the scenario language.
### Generating load test configurations
#### Generating load test configurations
The benchmarks framework uses LoadTest resources configured by YAML files. Each
LoadTest resource specifies a driver, a server, and one or more clients to run
@ -332,7 +185,7 @@ script) but is not indexed and cannot be used to select objects. Scenario name
and uniquifier are added to provide the elements of the LoadTest name uuid in
human-readable form. Additional annotations may be added later for automation.
### Concatenating load test configurations
#### Concatenating load test configurations
The LoadTest configuration generator can process multiple languages at a time,
assuming that they are supported by the template. The convenience script
@ -345,7 +198,7 @@ script can be invoked as follows:
$ loadtest_concat_yaml.py -i infile1.yaml infile2.yaml -o outfile.yaml
```
### Generating load test examples
#### Generating load test examples
The script [loadtest_examples.sh ](./loadtest_examples.sh ) is provided to
generate example load test configurations in all supported languages. This
@ -357,7 +210,7 @@ with prebuilt images.
The [examples ](https://github.com/grpc/test-infra/tree/master/config/samples ) in
the repository [grpc/test-infra] are generated by this script.
### Generating configuration templates
#### Generating configuration templates
The script [loadtest_template.py ](./loadtest_template.py ) generates a load test
configuration template from a set of load test configurations. The source files
@ -430,7 +283,7 @@ may be set to values or to substitution keys in themselves, allowing future
automation scripts to process the tests generated from these configurations in
different ways.
### Running tests
#### Running tests
Collections of tests generated by `loadtest_config.py` are intended to be run
with a test runner. The code for the test runner is stored in a separate
@ -451,3 +304,159 @@ For usage examples, see the continuous integration setup defined in
[grpc/test-infra]: https://github.com/grpc/test-infra
[grpc_e2e_performance_gke.sh]: ../../internal_ci/linux/grpc_e2e_performance_gke.sh
[grpc_e2e_performance_v2.sh]: ../../internal_ci/linux/grpc_e2e_performance_v2.sh
## Approach 2: Running benchmarks locally via legacy tooling (still useful sometimes)
This approach is much more involved than using the gRPC OSS benchmarks framework
(see above), but can still be useful for hands-on low-level experiments
(especially when you know what you are doing).
### Prerequisites for running benchmarks manually:
In general the benchmark workers and driver build scripts expect
[linux_performance_worker_init.sh ](../../gce/linux_performance_worker_init.sh )
to have been ran already.
### To run benchmarks locally:
- From the grpc repo root, start the
[run_performance_tests.py ](../run_performance_tests.py ) runner script.
### On remote machines, to start the driver and workers manually:
The [run_performance_test.py ](../run_performance_tests.py ) top-level runner
script can also be used with remote machines, but for e.g., profiling the
server, it might be useful to run workers manually.
1. You'll need a "driver" and separate "worker" machines. For example, you might
use one GCE "driver" machine and 3 other GCE "worker" machines that are in
the same zone.
2. Connect to each worker machine and start up a benchmark worker with a
"driver_port".
- For example, to start the grpc-go benchmark worker:
[grpc-go worker main.go ](https://github.com/grpc/grpc-go/blob/master/benchmark/worker/main.go )
--driver_port < driver_port >
#### Commands to start workers in different languages:
- Note that these commands are what the top-level
[run_performance_test.py ](../run_performance_tests.py ) script uses to build
and run different workers through the
[build_performance.sh ](./build_performance.sh ) script and "run worker" scripts
(such as the [run_worker_java.sh ](./run_worker_java.sh )).
##### Running benchmark workers for C-core wrapped languages (C++, Python, C#, Node, Ruby):
- These are more simple since they all live in the main grpc repo.
```
$ cd < grpc_repo_root >
$ tools/run_tests/performance/build_performance.sh
$ tools/run_tests/performance/run_worker_< language > .sh
```
- Note that there is one "run_worker" script per language, e.g.,
[run_worker_csharp.sh ](./run_worker_csharp.sh ) for c#.
##### Running benchmark workers for gRPC-Java:
- You'll need the [grpc-java ](https://github.com/grpc/grpc-java ) repo.
```
$ cd < grpc-java-repo >
$ ./gradlew -PskipCodegen=true -PskipAndroid=true :grpc-benchmarks:installDist
$ benchmarks/build/install/grpc-benchmarks/bin/benchmark_worker --driver_port < driver_port >
```
##### Running benchmark workers for gRPC-Go:
- You'll need the [grpc-go repo ](https://github.com/grpc/grpc-go )
```
$ cd < grpc-go-repo > /benchmark/worker & & go install
$ # if profiling, it might be helpful to turn off inlining by building with "-gcflags=-l"
$ $GOPATH/bin/worker --driver_port < driver_port >
```
#### Build the driver:
- Connect to the driver machine (if using a remote driver) and from the grpc
repo root:
```
$ tools/run_tests/performance/build_performance.sh
```
#### Run the driver:
1. Get the 'scenario_json' relevant for the scenario to run. Note that "scenario
json" configs are generated from [scenario_config.py ](./scenario_config.py ).
The [driver ](../../../test/cpp/qps/qps_json_driver.cc ) takes a list of these
configs as a json string of the form: `{scenario: <json_list_of_scenarios> }`
in its `--scenarios_json` command argument. One quick way to get a valid json
string to pass to the driver is by running the
[run_performance_tests.py ](./run_performance_tests.py ) locally and copying
the logged scenario json command arg.
2. From the grpc repo root:
- Set `QPS_WORKERS` environment variable to a comma separated list of worker
machines. Note that the driver will start the "benchmark server" on the first
entry in the list, and the rest will be told to run as clients against the
benchmark server.
Example running and profiling of go benchmark server:
```
$ export QPS_WORKERS=< host1 > :< 10000 > ,< host2 > ,10000,< host3 > :10000
$ bins/opt/qps_json_driver --scenario_json='< scenario_json_scenario_config_string > '
```
### Example profiling commands
While running the benchmark, a profiler can be attached to the server.
Example to count syscalls in grpc-go server during a benchmark:
- Connect to server machine and run:
```
$ netstat -tulpn | grep < driver_port > # to get pid of worker
$ perf stat -p < worker_pid > -e syscalls:sys_enter_write # stop after test complete
```
Example memory profile of grpc-go server, with `go tools pprof` :
- After a run is done on the server, see its alloc profile with:
```
$ go tool pprof --text --alloc_space http://localhost:< pprof_port > /debug/heap
```
### Configuration environment variables:
- QPS_WORKER_CHANNEL_CONNECT_TIMEOUT
Consuming process: qps_worker
Type: integer (number of seconds)
This can be used to configure the amount of time that benchmark clients wait
for channels to the benchmark server to become ready. This is useful in
certain benchmark environments in which the server can take a long time to
become ready. Note: if setting this to a high value, then the scenario config
under test should probably also have a large "warmup_seconds".
- QPS_WORKERS
Consuming process: qps_json_driver
Type: comma separated list of host:port
Set this to a comma separated list of QPS worker processes/machines. Each
scenario in a scenario config has specifies a certain number of servers,
`num_servers` , and the driver will start "benchmark servers"'s on the first
`num_server` `host:port` pairs in the comma separated list. The rest will be
told to run as clients against the benchmark server.
Jan Tattermusch
3 years ago
committed by
GitHub