* Add templates for PSM test.
The commit adds psm related template generated from
loadtest_template.py. This commit also update the
loadtest_example.sh to generate examples based on the templates.
This commit adds the PSM base scenario to each language in
scenario_config.py allowing per language generation of PSM
base scenarios. The PSM base scenarios will be further updated
in loadtest_config.py for number of client channels and the
number of async_server_thread. After update of the two
aforementioned fields, scenarios varying offered_load will
be generated from the base scenarios.
loadtest_config.py is updated to take number of client channels,
the number of async_servers and a list of targeted offered_load.
* GKE benchmarks: add support for benchmarking grpc-dotnet
* add dotnet to loadtest basic templates
* print full path for generated examples
* add grpc-dotnet scenario to loadtest_example.sh generator
* add grpc-dotnet to the experimental kokoro job
* yapf format code
* Run 2to3 on tools directory
* Delete github_stats_tracking
* Re-run 2to3
* Remove unused script
* Remove unused script
* Remove unused line count utility
* Yapf. Isort
* Remove accidentally included file
* Migrate tools/distrib directory to python 3
* Remove unnecessary shebang
* Restore line_count directory
* Immediately convert subprocess.check_output output to string
* Take care of Python 2 shebangs
* Invoke scripts using a Python 3 interpreter
* Yapf. Isort
* Try installing Python 3 first
* See if we have any Python 3 versions installed
* Add Python 3.7 to Windows path
* Try adding a symlink
* Try to symlink differently
* Install six for Python 3
* Run run_interop_tests with python 3
* Try installing six in python3.7 explicitly
* Revert "Try installing six in python3.7 explicitly"
This reverts commit 2cf60d72f3.
* And debug some more
* Fix issue with jobset.py
* Add debug for CI failure
* Revert microbenchmark changes
* make it obvious which gke benchmark script is experimental
* add cfg file for new kokoro jobs with a more fitting name
* update performance readme after renaming _v2 benchmark job
* fix typo
* Reorganize docs for end2end benchmarks
* Update README.md
* Restore heading "gRPC OSS benchmarks" (#27646)
This heading is used in cross-references from other repositories and sites. Also fix formatting.
Co-authored-by: Paulo Castello da Costa <6579971+paulosjca@users.noreply.github.com>
* Add isort_code.sh to sanity tests
* Run tools/distrib/isort_code.sh
* Fine tune the import order for relative imports
* Make pylint and project generation happy
* Fix a few corner cases
* Use --check instead of --diff
* The import order impacts test result somehow
* Make isort print diff and check output at the same time
* Let tools/run_tests/python_utils be firstparty library
* Run isort against latest HEAD
This change ensures that a valid driver configuration is always included in generated load test configurations, and that the driver pod is named with an index (`0`, since there is only one driver), in the same way as client and server pods.
Generated examples can be found in https://github.com/grpc/test-infra/pull/189.
With this change, it is no longer necessary to specify a driver image in order to specify a driver name and pool, so that is removed from the kokoro jobs.
- Support multiline strings in template generation (as already supported in config generation, supporting roundtrip).
- Ignore substitution of variables that are set by the controller at runtime (`DRIVER_PORT`, `KILL_AFTER`, `POD_TIMEOUT`).
Generates performance tests with real names instead of uuids. This allows test names to be up to 253 characters long, which is now supported by the framework.
* Update load test template and config generation.
This change includes the following features and fixes:
* Add a script to generate load test examples.
* Update template generation logic to support round trip from configs to templates (handling of repeated clients and servers for the same language and of named clients and servers in source configs).
* Integrate safe language names from scenario config.
* Update template and config formatting (now that we generate in round trip).
* Fix shellcheck lint warnings.
* Update README.md.
This change fixes the format and location of the images in the driver configuration, so tests run with the proper driver configuration instead of a default.
This change adds the following features and improvements to load test generation:
* Multiple client instances are supported by setting the option `--instances_per_client`.
* `python_asyncio` is supported as a separate language.
* `php` is renamed to `php7` to match the scenario language names.
* Client and server names are shortened by the introduction of an index (`0`, `1`, ... with an optional prefix). This index replaces the uuid added to pod names when no component name is specified.
* Driver pool and image substitution are added to template generation.
* Empty entries are removed from the final configuration.
NOTE: Controller is currently not respecting driver settings from configuration, so driver is still named with a uuid. This will be addressed separately.
This example crashes the driver process, since `bq_upload_result.py` expects
the `big_query_table` argument to have the format `<dataset name>.<table name>`.
The project ID is already encoded in the script.
```
+ python3 /src/code/tools/run_tests/performance/bq_upload_result.py --bq_result_table=grpc-testing.e2e_benchmarks.experimental_results_hork_x5
Traceback (most recent call last):
File "/src/code/tools/run_tests/performance/bq_upload_result.py", line 177, in <module>
dataset_id, table_id = args.bq_result_table.split('.', 2)
ValueError: too many values to unpack (expected 2)
```
* Run all kokoro performance tests on dedicated node pools.
Both official and experimental tests run with kokoro are set to run on separate node pools with the suffix "-ci" (drivers-ci, workers-8core-ci, workers-32core-ci), separate from the default pools used for manual runs.
* Employ prebuilt images in continuous build.
This commits updated Kokoro build job to use prebuilt images to
run tests. The loadtest template was generated using
loadtest_template.py.
This change fixes typos and makes minor changes to the format of the generated yaml files for LoadTest configuration (embedded scenarios end in newline, and element lists are indented). The templates generated earlier are also updated.
Wanlin Du
Jan Tattermusch
Yash Tibrewal
Craig Tiller
Paulo Castello da Costa
Richard Belleville
brandonpaiz
Donghao Qiu
Lidi Zheng
AJ Heller