mirror of https://github.com/FFmpeg/FFmpeg.git
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
139 lines
5.1 KiB
139 lines
5.1 KiB
FATE Automated Testing Environment |
|
================================== |
|
|
|
FATE is an extended regression suite on the client-side and a means |
|
for results aggregation and presentation on the server-side. |
|
|
|
The first part of this document explains how you can use FATE from |
|
your FFmpeg source directory to test your ffmpeg binary. The second |
|
part describes how you can run FATE to submit the results to FFmpeg's |
|
FATE server. |
|
|
|
In any way you can have a look at the publicly viewable FATE results |
|
by visiting this website: |
|
|
|
http://fate.ffmpeg.org/ |
|
|
|
This is especially recommended for all people contributing source |
|
code to FFmpeg, as it can be seen if some test on some platform broke |
|
with there recent contribution. This usually happens on the platforms |
|
the developers could not test on. |
|
|
|
The second part of this document describes how you can run FATE to |
|
submit your results to FFmpeg's FATE server. If you want to submit your |
|
results be sure to check that your combination of CPU, OS and compiler |
|
is not already listed on the above mentioned website. |
|
|
|
In the third part you can find a comprehensive listing of FATE makefile |
|
targets and variables. |
|
|
|
|
|
1. Using FATE from your FFmpeg source directory |
|
----------------------------------------------- |
|
|
|
If you want to run FATE on your machine you need to have the samples |
|
in place. You can get the samples via the build target fate-rsync. |
|
Use this command from the top-level source directory: |
|
|
|
# make fate-rsync SAMPLES=fate-suite/ |
|
# make fate SAMPLES=fate-suite/ |
|
|
|
The above commands set the samples location by passing a makefile |
|
variable via command line. It is also possible to set the samples |
|
location at source configuration time by invoking configure with |
|
`--samples=<path to the samples directory>'. Afterwards you can |
|
invoke the makefile targets without setting the SAMPLES makefile |
|
variable. This is illustrated by the following commands: |
|
|
|
# ./configure --samples=fate-suite/ |
|
# make fate-rsync |
|
# make fate |
|
|
|
Yet another way to tell FATE about the location of the sample |
|
directory is by making sure the environment variable FATE_SAMPLES |
|
contains the path to your samples directory. This can be achieved |
|
by e.g. putting that variable in your shell profile or by setting |
|
it in your interactive session. |
|
|
|
# FATE_SAMPLES=fate-suite/ make fate |
|
|
|
NOTE: |
|
Do not put a '~' character in the samples path to indicate a home |
|
directory. Because of shell nuances, this will cause FATE to fail. |
|
|
|
|
|
2. Submitting the results to the FFmpeg result aggregation server |
|
----------------------------------------------------------------- |
|
|
|
To submit your results to the server you should run fate through the |
|
shell script tests/fate.sh from the FFmpeg sources. This script needs |
|
to be invoked with a configuration file as its first argument. |
|
|
|
# tests/fate.sh /path/to/fate_config |
|
|
|
A configuration file template with comments describing the individual |
|
configuration variables can be found at tests/fate_config.sh.template . |
|
|
|
Create a configuration that suits your needs, based on the configuration |
|
template. The `slot' configuration variable can be any string that is not |
|
yet used, but it is suggested that you name it adhering to the following |
|
pattern <arch>-<os>-<compiler>-<compiler version>. The configuration file |
|
itself will be sourced in a shell script, therefore all shell features may |
|
be used. This enables you to setup the environment as you need it for your |
|
build. |
|
|
|
For your first test runs the `fate_recv' variable should be empty or |
|
commented out. This will run everything as normal except that it will omit |
|
the submission of the results to the server. The following files should be |
|
present in $workdir as specified in the configuration file: |
|
|
|
- configure.log |
|
- compile.log |
|
- test.log |
|
- report |
|
- version |
|
|
|
When you have everything working properly you can create an SSH key and |
|
send its public part to the FATE server administrator. |
|
|
|
Configure your SSH client to use public key authentication with that key |
|
when connecting to the FATE server. Also do not forget to check the identity |
|
of the server and to accept its host key. This can usually be achieved by |
|
running your SSH client manually and killing it after you accepted the key. |
|
The FATE server's fingerprint is: |
|
|
|
b1:31:c8:79:3f:04:1d:f8:f2:23:26:5a:fd:55:fa:92 |
|
|
|
The only thing left is to automate the execution of the fate.sh script and |
|
the synchronisation of the samples directory. |
|
|
|
|
|
3. FATE makefile targets and variables |
|
-------------------------------------- |
|
|
|
FATE Makefile targets: |
|
|
|
fate-list |
|
Will list all fate/regression test targets. |
|
|
|
fate |
|
Run the FATE test suite (requires the fate-suite dataset). |
|
|
|
FATE Makefile variables: |
|
|
|
V |
|
Verbosity level, can be set to 0, 1 or 2. |
|
* 0: show just the test arguments |
|
* 1: show just the command used in the test |
|
* 2: show everything |
|
|
|
SAMPLES |
|
Specify or override the path to the FATE samples at make time, it has a |
|
meaning only while running the regression tests. |
|
|
|
THREADS |
|
Specify how many threads to use while running regression tests, it is |
|
quite useful to detect thread-related regressions. |
|
|
|
Example: |
|
make V=1 SAMPLES=/var/fate/samples THREADS=2 fate
|
|
|