avcodec/vp3dsp: Fix multiple signed integer overflow: 46341 * 47523 cannot be represented in type 'int'

Fixes: 664/clusterfuzz-testcase-4917047475568640 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
8 years ago · 2b8b7921c5
parent 8696f25444
commit 2b8b7921c5
1 changed files with 1 additions and 1 deletions
--- a/libavcodec/vp3dsp.c
+++ b/libavcodec/vp3dsp.c
@ -41,7 +41,7 @@
 #define xC6S2 25080
 #define xC7S1 12785

-#define M(a, b) (((a) * (b)) >> 16)
+#define M(a, b) ((int)((SUINT)(a) * (b)) >> 16)

 static av_always_inline void idct(uint8_t *dst, int stride,
                                  int16_t *input, int type)