avcodec/rka: use unsigned for buf0 additions

Fixes: signed integer overflow: -38912000 + -2109276160 cannot be represented in type 'int'
Fixes: 59670/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_RKA_fuzzer-4987563245699072

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit e736238b35)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
release/6.0
Michael Niedermayer 2 years ago
parent 819e9068af
commit 85cb7abb4c
No known key found for this signature in database
GPG Key ID: B18E8928B3948D64
  1. 4
      libavcodec/rka.c

@ -735,10 +735,10 @@ static int decode_filter(RKAContext *s, ChContext *ctx, ACoder *ac, int off, uns
ctx->buf1[off] = (val + (sum >> bits)) * (1U << bits) +
(((1U << bits) - 1U) & ctx->buf1[off + -1]);
}
ctx->buf0[off] = ctx->buf1[off] + ctx->buf0[off + -1];
ctx->buf0[off] = ctx->buf1[off] + (unsigned)ctx->buf0[off + -1];
} else {
val *= 1U << ctx->cmode;
sum += ctx->buf0[off + -1] + val;
sum += ctx->buf0[off + -1] + (unsigned)val;
switch (s->bps) {
case 16: sum = av_clip_int16(sum); break;
case 8: sum = av_clip_int8(sum); break;

Loading…
Cancel
Save