The original code left-shifts negative values, which is undefined
in the C99 specification (the one used during normal FFmpeg compilation).
This change multiplies by (1 << shift), which is functionally equivalent,
but has defined behavior.
With this change, fate-idct8x8 compiled with --fsanitize=undefined works.
Bug-Id: 686
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
The assembly generated by gcc is unchanged by this commit
no changes in either standard deviation or PSNR is seen in any of the changed fate
cases
MSE changes from 0.05012422 to 0.04890000
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Results of IDCT can by far outreach the range of ff_cropTbl[], leading
to overreads and potentially crashes.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
"Copyright (c) 2001 Michael Niedermayer" and "part of Libav" is not likely
not only am i not a libav developer there also was no libav in 2001
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Katerina Barone-Adesi
Diego Biurrun
Michael Niedermayer
Ronald S. Bultje
Mans Rullgard