At least the new videotoolbox decoder does not actually set a frame if
end_frame fails. This causes the API to return success and signals that
a picture was decoded, even though AVFrame->data[0] is NULL.
Fix this by propagating end_frame errors.
h264.h and hevc.h are mutually exclusive due to defining some of the same
names. As such, we need to avoid forcing h264.h to be included if we want
hevc decode acceleration to be possible.
However, some of the pre-hwaccel helper functions need h264.h. To avoid
messy collisions, let's move the declaration of all those helpers to
a separate header which we will exclude for the hevc support (which will
be hwaccel-only).
Signed-off-by: Philip Langdale <philipl@overt.org>
Also move EC ref initialization to where the EC code is called.
Fixes out of array read
Fixes: asan_heap-uaf_143f420_142_20110805_112659_ch0.mkv
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>