mirror of https://github.com/FFmpeg/FFmpeg.git
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
168 lines
4.9 KiB
168 lines
4.9 KiB
\input texinfo @c -*- texinfo -*- |
|
|
|
@settitle FATE Automated Testing Environment |
|
@titlepage |
|
@center @titlefont{FATE Automated Testing Environment} |
|
@end titlepage |
|
|
|
@top |
|
|
|
@contents |
|
|
|
@chapter Introduction |
|
|
|
FATE provides a regression test suite embedded within the Libav build system. |
|
It can be run locally and optionally configured to send reports to a web |
|
aggregator and viewer @url{http://fate.libav.org}. |
|
|
|
It is advised to run FATE before submitting patches to the current codebase |
|
and provide new tests when submitting patches to add additional features. |
|
|
|
@chapter Running FATE |
|
|
|
@section Samples and References |
|
In order to run, FATE needs a large amount of data (samples and references) |
|
that is provided separately from the actual source distribution. |
|
|
|
To inform the build system about the test suite location, pass |
|
@option{--samples=<path to the samples>} to @command{configure} or set the |
|
@var{SAMPLES} Make variable or the @var{LIBAV_SAMPLES} environment variable |
|
to a suitable value. |
|
|
|
To use a custom wrapper to run the test, pass @option{--target-exec} to |
|
@command{configure} or set the @var{TARGET_EXEC} Make variable. |
|
|
|
The dataset is available through @command{rsync}, is possible to fetch |
|
the current sample using the straight rsync command or through a specific |
|
@ref{Makefile target}. |
|
|
|
@example |
|
# rsync -aL rsync://fate-suite.libav.org/fate-suite/ fate-suite |
|
@end example |
|
|
|
@example |
|
# make fate-rsync SAMPLES=fate-suite |
|
@end example |
|
|
|
|
|
@chapter Manual Run |
|
FATE regression test can be run through @command{make}. |
|
Specific Makefile targets and Makefile variables are available: |
|
|
|
@anchor{Makefile target} |
|
@section FATE Makefile targets |
|
|
|
@table @option |
|
@item fate-list |
|
List all fate/regression test targets. |
|
|
|
@item fate-rsync |
|
Shortcut to download the fate test samples to the specified test suite location. |
|
|
|
@item fate |
|
Run the FATE test suite (requires the fate-suite dataset). |
|
@end table |
|
|
|
@section FATE Makefile variables |
|
@table @option |
|
@item V |
|
Verbosity level, can be set to 0, 1 or 2. |
|
|
|
@table @option |
|
@item 0 |
|
show just the test arguments |
|
|
|
@item 1 |
|
show just the command used in the test |
|
|
|
@item 2 |
|
show everything |
|
@end table |
|
|
|
@item SAMPLES |
|
Specify or override the path to the FATE samples at make time, it has a |
|
meaning only while running the regression tests. |
|
|
|
@item THREADS |
|
Specify how many threads to use while running regression tests, it is |
|
quite useful to detect thread-related regressions. |
|
|
|
@item THREAD_TYPE |
|
Specify which threading strategy test, either @var{slice} or @var{frame}, |
|
by default @var{slice+frame} |
|
|
|
@item CPUFLAGS |
|
Specify a mask to be applied to autodetected CPU flags. |
|
|
|
@item TARGET_EXEC |
|
Specify or override the wrapper used to run the tests. |
|
|
|
@item GEN |
|
Set to @var{1} to generate the missing or mismatched references. |
|
@end table |
|
|
|
@example |
|
make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate |
|
@end example |
|
|
|
@chapter Automated Tests |
|
In order to automatically testing specific configurations, e.g. multiple |
|
compilers, @command{tests/fate.sh} is provided. |
|
|
|
This shell script builds Libav, runs the regression tests and prepares |
|
a report that can be sent to @url{http://fate.libav.org/} or directly |
|
examined locally. |
|
|
|
@section Testing Profiles |
|
The configuration file passed to @command{fate.sh} is shell scripts as well. |
|
|
|
It must provide at least a @var{slot} identifier, the @var{repo} from |
|
which fetch the sources, the @var{samples} directory, a @var{workdir} with |
|
enough space to build and run all the tests. |
|
Optional submit command @var{fate_recv} and a @var{comment} to describe |
|
the testing profile are available. |
|
|
|
Additional optional parameter to tune the Libav building and reporting process |
|
can be passed. |
|
|
|
@example |
|
slot= # some unique identifier |
|
repo=git://git.libav.org/libav.git # the source repository |
|
#branch=release/10 # the branch to test |
|
samples=/path/to/fate/samples |
|
workdir= # directory in which to do all the work |
|
fate_recv="ssh -T fate@@fate.libav.org" # command to submit report |
|
comment= # optional description |
|
build_only= # set to "yes" for a compile-only instance that skips tests |
|
|
|
# the following are optional and map to configure options |
|
arch= |
|
cpu= |
|
cross_prefix= |
|
as= |
|
cc= |
|
ld= |
|
target_os= |
|
sysroot= |
|
target_exec= |
|
target_path= |
|
target_samples= |
|
extra_cflags= |
|
extra_ldflags= |
|
extra_libs= |
|
extra_conf= # extra configure options not covered above |
|
|
|
#make= # name of GNU make if not 'make' |
|
makeopts= # extra options passed to 'make' |
|
#tar= # command to create a tar archive from its arguments on |
|
# stdout, defaults to 'tar c' |
|
@end example |
|
|
|
@section Special Instances |
|
The @var{TARGET_EXEC} option provides a way to run FATE wrapped in |
|
@command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets |
|
through @command{ssh}. |
|
|
|
@section Submitting Reports |
|
In order to send reports you need to create an @command{ssh} key and send it |
|
to @email{root@@libav.org}.
|
|
|