Fixes: 673/clusterfuzz-testcase-5948736536576000
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This fixes ubsan warnings in non debug builds by using unsigned operations
in debug builds the correct signed operations are retained so that overflows
(which should not occur in valid files and may indicate problems in the DSP code
or decoder) can be detected.
Alternatively they can be changed to unsigned unconditionally, then its
not possible though to detect overflows easily if someone wants to test
the DSP code for overflows.
The 2nd alternative would be to leave the code as it is and accept that
there are undefined operations in the DSP code and that ubsan output is
full of them in some cases.
Similar changes would be needed in some other DSP routines
Suggested-by: Matt Wolenetz <wolenetz@google.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
benchmarked on sandybridge x86_64:
1358232 decicycles in flac_lpc_32_c
1244575 decicycles in flac_lpc_32_sse4, James Almer's patch
650045 decicycles in flac_lpc_32_sse4, this patch
I haven't tested the edgecases such as odd block lengths
odd block length tested-by: James Almer <jamrial@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Michael Niedermayer
Carl Eugen Hoyos
James Almer
James Darnley
Loren Merritt
Justin Ruggles
Mans Rullgard