Up until now, the Cook decoder used tables for the lengths of codes and
tables of the codes itself to initialize VLCs; the tables for the codes
were of type uint16_t because the codes were so long. It did not use
explicit symbol tables. This commit instead reorders the tables so that
the code tables are sorted from left to right in the tree. Then the
codes can be easily derived from the lengths and therefore be omitted.
This comes at the price of explicitly coding the symbols, but this is
nevertheless a net win because most of the symbols tables can be coded
on one byte. Furthermore, Cook actually does not use a contiguous range
of symbols for its main VLC tables and the old code compensated for that
by adding holes (codes of length zero) to the tables (that are skipped by
ff_init_vlc_sparse()). This is no longer necessary with the new
approach. All in all, this saves about 1.7KB.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Fixes: off by 1 error
Fixes: index 5 out of bounds for type 'COOKSubpacket [5]'
Fixes: 25772/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_COOK_fuzzer-5762459498184704.fuzz
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: shift exponent -1 is negative
Fixes: out of array read
Fixes: 19028/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_COOK_fuzzer-5759766471376896
Fixes: 19037/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_COOK_fuzzer-5734106625474560
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: signed integer overflow: 2046820356 * 8 cannot be represented in type 'int'
Fixes: 18391/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_COOK_fuzzer-5631674666188800
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: index 25 out of bounds for type 'float [23]'
Fixes: 18355/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_COOK_fuzzer-5641398941908992
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: division by zero
Fixes: 18362/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_COOK_fuzzer-5653727679086592
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes out of array read
Fixes: asan_heap-oob_fb5c50_19_018.rmvb
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>