Benchmarks with START_TIMER indicate that the code is faster with unsigned, (that is
with the patch), there was quite some fluctuation in the numbers so this may be just
random
Fixes: 811/clusterfuzz-testcase-6465493076541440
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The original code left-shifts negative values, which is undefined
in the C99 specification (the one used during normal Libav compilation).
This change multiplies by (1 << shift), which is functionally equivalent,
but has defined behavior.
With this change, fate-idct8x8 compiled with --fsanitize=undefined works.
Bug-Id: 686
omse goes from 0.03060703 (which fails for dct-test) to 0.01663750.
This also actually improve the error of decoding the sample generated
by fate-vsynth3-dnxhd1080i-10bit using simple_idct10 to FAANI, which
goes (when resampled to yuv422p) from:
stddev: 0.06 PSNR: 72.28 MAXDIFF: 1
to identical.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The original code left-shifts negative values, which is undefined
in the C99 specification (the one used during normal FFmpeg compilation).
This change multiplies by (1 << shift), which is functionally equivalent,
but has defined behavior.
With this change, fate-idct8x8 compiled with --fsanitize=undefined works.
Bug-Id: 686
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
The assembly generated by gcc is unchanged by this commit
no changes in either standard deviation or PSNR is seen in any of the changed fate
cases
MSE changes from 0.05012422 to 0.04890000
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Results of IDCT can by far outreach the range of ff_cropTbl[], leading
to overreads and potentially crashes.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
"Copyright (c) 2001 Michael Niedermayer" and "part of Libav" is not likely
not only am i not a libav developer there also was no libav in 2001
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Michael Niedermayer
Diego Biurrun
Vittorio Giovara
Katerina Barone-Adesi
Christophe Gisquet
Michael Niedermayer
Ronald S. Bultje
Mans Rullgard