This adds a new GKE benchmark job, which runs the set of "dashboard"
scenarios for every gRPC experiment configured in the script. Results
are published to BigQuery at
`e2e_benchmarks.ci_cxx_experiment_results_${N}core.${experiment}`
See https://github.com/grpc/grpc/pull/33907 for the scenario config.
Scenarios for language `node` specify the server language as `node`
(instead of leaving it blank), so a flag must be added to
`--allow_server_language=node`.
Scenarios for language `node_purejs` differ in name and in scenario
settings, but otherwise run on identical clients and servers. This
change treats `node_purejs` as `node` for the purpose of generating load
test configurations.
- Switched from yapf to black
- Reconfigure isort for black
- Resolve black/pylint idiosyncrasies
Note: I used `--experimental-string-processing` because black was
producing "implicit string concatenation", similar to what described
here: https://github.com/psf/black/issues/1837. While currently this
feature is experimental, it will be enabled by default:
https://github.com/psf/black/issues/2188. After running black with the
new string processing so that the generated code merges these `"hello" "
world"` strings concatenations, then I removed
`--experimental-string-processing` for stability, and regenerated the
code again.
To the reviewer: don't even try to open "Files Changed" tab 😄 It's
better to review commit-by-commit, and ignore `run black and isort`.
This commit adds the PSM base scenario to each language in
scenario_config.py allowing per language generation of PSM
base scenarios. The PSM base scenarios will be further updated
in loadtest_config.py for number of client channels and the
number of async_server_thread. After update of the two
aforementioned fields, scenarios varying offered_load will
be generated from the base scenarios.
loadtest_config.py is updated to take number of client channels,
the number of async_servers and a list of targeted offered_load.
* Run 2to3 on tools directory
* Delete github_stats_tracking
* Re-run 2to3
* Remove unused script
* Remove unused script
* Remove unused line count utility
* Yapf. Isort
* Remove accidentally included file
* Migrate tools/distrib directory to python 3
* Remove unnecessary shebang
* Restore line_count directory
* Immediately convert subprocess.check_output output to string
* Take care of Python 2 shebangs
* Invoke scripts using a Python 3 interpreter
* Yapf. Isort
* Try installing Python 3 first
* See if we have any Python 3 versions installed
* Add Python 3.7 to Windows path
* Try adding a symlink
* Try to symlink differently
* Install six for Python 3
* Run run_interop_tests with python 3
* Try installing six in python3.7 explicitly
* Revert "Try installing six in python3.7 explicitly"
This reverts commit 2cf60d72f3.
* And debug some more
* Fix issue with jobset.py
* Add debug for CI failure
* Revert microbenchmark changes
* Add isort_code.sh to sanity tests
* Run tools/distrib/isort_code.sh
* Fine tune the import order for relative imports
* Make pylint and project generation happy
* Fix a few corner cases
* Use --check instead of --diff
* The import order impacts test result somehow
* Make isort print diff and check output at the same time
* Let tools/run_tests/python_utils be firstparty library
* Run isort against latest HEAD
This change ensures that a valid driver configuration is always included in generated load test configurations, and that the driver pod is named with an index (`0`, since there is only one driver), in the same way as client and server pods.
Generated examples can be found in https://github.com/grpc/test-infra/pull/189.
With this change, it is no longer necessary to specify a driver image in order to specify a driver name and pool, so that is removed from the kokoro jobs.
- Support multiline strings in template generation (as already supported in config generation, supporting roundtrip).
- Ignore substitution of variables that are set by the controller at runtime (`DRIVER_PORT`, `KILL_AFTER`, `POD_TIMEOUT`).
Generates performance tests with real names instead of uuids. This allows test names to be up to 253 characters long, which is now supported by the framework.
* Update load test template and config generation.
This change includes the following features and fixes:
* Add a script to generate load test examples.
* Update template generation logic to support round trip from configs to templates (handling of repeated clients and servers for the same language and of named clients and servers in source configs).
* Integrate safe language names from scenario config.
* Update template and config formatting (now that we generate in round trip).
* Fix shellcheck lint warnings.
* Update README.md.
This change fixes the format and location of the images in the driver configuration, so tests run with the proper driver configuration instead of a default.
This change adds the following features and improvements to load test generation:
* Multiple client instances are supported by setting the option `--instances_per_client`.
* `python_asyncio` is supported as a separate language.
* `php` is renamed to `php7` to match the scenario language names.
* Client and server names are shortened by the introduction of an index (`0`, `1`, ... with an optional prefix). This index replaces the uuid added to pod names when no component name is specified.
* Driver pool and image substitution are added to template generation.
* Empty entries are removed from the final configuration.
NOTE: Controller is currently not respecting driver settings from configuration, so driver is still named with a uuid. This will be addressed separately.
This change fixes typos and makes minor changes to the format of the generated yaml files for LoadTest configuration (embedded scenarios end in newline, and element lists are indented). The templates generated earlier are also updated.
This commit includes the following changes:
1. A new load test template generator (loadtest_template.py) is added. The template generator combines existing configurations or templates for several languages into a single template that can be used to generate configurations for different languages or combinations of languages.
2. A basic template generated from the example tests in grpc/test-infra (loadtest_template_basic_all_languages.yaml) is added.
3. The load test config generator is updated to use the combined template.
4. An example run consisting of a single test (generated from the combined template) is added and set up to run continuously.
* LoadTest generator for OSS benchmarks framework.
This change adds a LoadTest configuration generator for the OSS
benchmarks framework. The output of the generator is a multipart
YAML file that specifies uniquely named LoadTest resources that
can be applied to a kubernetes cluster.
For the benchmarks framework, see https://github.com/github/test-infra.
Paulo Castello da Costa
AJ Heller
Sergii Tkachenko
Wanlin Du
Paulo Castello da Costa
Richard Belleville
Lidi Zheng
Donghao Qiu