@ -61,9 +61,9 @@ service TestExecutionService {
/ / Unsupported environments will be returned in the state UNSUPPORTED.
/ / A test matrix is limited to use at most 2000 devices in parallel.
/ /
/ / The returned matrix will not yet contain
/ / the executions that will be created for this matrix. That happens later on
/ / and will require a call to GetTestMatrix.
/ / The returned matrix will not yet contain the executions that will be
/ / created for this matrix. Execution creation happens later on and will
/ / require a call to GetTestMatrix.
/ /
/ / May return any of the following canonical error codes :
/ /
@ -83,8 +83,7 @@ service TestExecutionService {
/ / The test matrix will contain the list of test executions to run if and only
/ / if the resultStorage.toolResultsExecution fields have been populated.
/ /
/ / Note : Flaky test executions may still be added to the matrix at a later
/ / stage.
/ / Note : Flaky test executions may be added to the matrix at a later stage.
/ /
/ / May return any of the following canonical error codes :
/ /
@ -552,6 +551,7 @@ message AndroidInstrumentationTest {
/ / Each target must be fully qualified with the package name or class name ,
/ / in one of these formats :
/ /
/ / - "package package_name"
/ / - "class package_name.class_name"
/ / - "class package_name.class_name#method_name"
@ -564,6 +564,7 @@ message AndroidInstrumentationTest {
/ / * * Orchestrator is only compatible with AndroidJUnitRunner version 1.1 or
/ / higher ! * *
/ / Orchestrator offers the following benefits :
/ /
/ / - No shared state
/ / - Crashes are isolated
/ / - Logs are scoped per test
@ -1009,9 +1010,10 @@ enum InvalidMatrixDetails {
/ / OrchestratorOption.
INSTRUMENTATION_ORCHESTRATOR_INCOMPATIBLE = 18 ;
/ / The test APK does not contain the test runner class specified by user or in
/ / the manifest file.
/ / This can be caused by either of the following reasons :
/ / The test APK does not contain the test runner class specified by the user
/ / or in the manifest file. This can be caused by one of the following
/ / reasons :
/ /
/ / - the user provided a runner class name that ' s incorrect , or
/ / - the test runner isn ' t built into the test APK ( might be in the app APK
/ / instead ) .
@ -1184,12 +1186,14 @@ enum OutcomeSummary {
OUTCOME_SUMMARY_UNSPECIFIED = 0 ;
/ / The test matrix run was successful , for instance :
/ /
/ / - All the test cases passed.
/ / - Robo did not detect a crash of the application under test.
SUCCESS = 1 ;
/ / A run failed , for instance :
/ / - One or more test case failed.
/ /
/ / - One or more test cases failed.
/ / - A test timed out.
/ / - The application under test crashed.
FAILURE = 2 ;
@ -1200,6 +1204,7 @@ enum OutcomeSummary {
INCONCLUSIVE = 3 ;
/ / All tests were skipped , for instance :
/ /
/ / - All device configurations were incompatible.
SKIPPED = 4 ;
}
@ -1270,49 +1275,58 @@ message SmartSharding {
/ / The minimum allowed : 120 seconds ( 2 minutes ) .
/ /
/ / The shard count is dynamically set based on time , up to the maximum shard
/ / limit ( described below ) . To guarantee at least one test case for each
/ / limit ( described below ) . To guarantee at least one test case for each
/ / shard , the number of shards will not exceed the number of test cases. Shard
/ / duration will be exceeded if :
/ / - The maximum shard limit is reached and there is more calculated test
/ / time remaining to allocate into shards.
/ / - Any individual test is estimated to be longer than the targeted shard
/ / duration.
/ /
/ / - The maximum shard limit is reached and there is more calculated test time
/ / remaining to allocate into shards.
/ / - Any individual test is estimated to be longer than the targeted shard
/ / duration.
/ /
/ / Shard duration is not guaranteed because smart sharding uses test case
/ / history and default durations which may not be accurate. The rules for
/ / finding the test case timing records are :
/ / - If the service has seen a test case in the last 30 days , the record
/ / of the latest successful one will be used.
/ / - For new test cases , the average duration of other known test cases will
/ /
/ / - If the service has processed a test case in the last 30 days , the record
/ / of the latest successful test case will be used.
/ / - For new test cases , the average duration of other known test cases will
/ / be used.
/ / - If there are no previous test case timing records available , the tes t
/ / case is considered to be 15 seconds long by default .
/ / - If there are no previous test case timing records available , the defaul t
/ / test case duration is 15 seconds.
/ /
/ / Because the actual shard duration can exceed the targeted shard duration ,
/ / we recommend setting the targeted value at least 5 minutes less than the
/ / maximum allowed test timeout ( 45 minutes for physical devices and 60
/ / minutes for virtual ) , or using the custom test timeout value you set. This
/ / approach avoids cancelling the shard before all tests can finish.
/ / we recommend that you set the targeted value at least 5 minutes less than
/ / the maximum allowed test timeout ( 45 minutes for physical devices and 60
/ / minutes for virtual ) , or that you use the custom test timeout value that
/ / you set. This approach avoids cancelling the shard before all tests can
/ / finish.
/ /
/ / Note that there is a limit for maximum number of shards. When you select
/ / one or more physical devices , the number of shards must be < = 50. When you
/ / select one or more ARM virtual devices , it must be < = 100. When you select
/ / only x86 virtual devices , it must be < = 500. To guarantee at least one test
/ / case for per shard , the number of shards will not exceed the number of test
/ / cases. Each shard created will count toward daily test quota.
/ / cases. Each shard created counts toward daily test quota.
google.protobuf.Duration targeted_shard_duration = 1 ;
}
/ / Output only. Details about the shard.
message Shard {
/ / Output only. The index of the shard among all the shards.
int32 shard_index = 1 ;
int32 shard_index = 1 [ ( google.api.field_behavior ) = OUTPUT_ONLY ] ;
/ / Output only. The total number of shards.
int32 num_shards = 2 ;
int32 num_shards = 2 [ ( google.api.field_behavior ) = OUTPUT_ONLY ] ;
/ / Output only. Test targets for each shard. Only set for manual sharding.
TestTargetsForShard test_targets_for_shard = 3 ;
TestTargetsForShard test_targets_for_shard = 3
[ ( google.api.field_behavior ) = OUTPUT_ONLY ] ;
/ / Output only. The estimated shard duration based on previous test case
/ / timing records , if available.
google.protobuf.Duration estimated_shard_duration = 4
[ ( google.api.field_behavior ) = OUTPUT_ONLY ] ;
}
/ / Request to submit a matrix of tests for execution.