This was partially fixed by 233cd89, but it made changes to AVFrame fields
from within end_frame, which doesn't work consistently when multithreading
is enabled. This is what the post_process function is for.
Signed-off-by: Aman Gupta <aman@tmm1.net>
vtctx->cached_hw_frames_ctx is unref'd in videotoolbox_uninit,
but videotoolbox_hevc used ff_videotoolbox_uninit which
doesn't unref cache_hw_frames_ctx.
Signed-off-by: Pavel Koshevoy <pkoshevoy@gmail.com>
Instead of assuming id 0 is used, use the same logic as used for PPS,
where all available entries in the list are emitted.
Signed-off-by: Aman Gupta <aman@tmm1.net>
videotoolbox returns an already cropped stream which led to double
cropping. this issue was introduced with the refactor of the cropping
mechanism in commit 07596e4 for h264 and 000fb61 for HEVC.
to fix this we set the cropping of the frame and the output frame to 0.
Tested-by: ponpon
Fixes ticket #7544.
In a normal hwaccel, the AVHWFramesContext sets AVFrame.hw_frames_ctx
when it initializes a new AVFrame in av_hwframe_get_buffer().
But the VT hwaccel doesn't know what hw_frames_ctx to assign when
the AVFrame is first created, because it depends on the format of
the pixbuf that the decoder eventually decides to return. Thus
newly created AVFrames always have a NULL hw_frames_ctx, and the
hwaccel would only assign the ctx once a frame was done decoding.
This worked fine with the H264 decoder, but with the HEVC decoder
the frame's data may be moved to another empty AVFrame. Since the
empty AVFrame never had hw_frames_ctx set, a frame with a NULL
ctx could be returned to the API user.
This patch works around the issue by moving the derived
hw_frames_ctx from the AVFrame to a new VTHWFrame which now holds
both the CVPixelBufferRef and the AVBuffer. The hw_frames_ctx
is only copied to the AVFrame right before it is about to be
returned to the user in videotoolbox_postproc_frame() (since
in the case of VT, the hw_frames_ctx is only there for the API
user anyway).
Fixes playback on macOS and iOS of some hevc videos like
https://s3.amazonaws.com/tmm1/videotoolbox/germany-hevc-zdf.ts
Signed-off-by: Aman Gupta <aman@tmm1.net>
Older iOS devices don't have a hardware HEVC decoder, but the
software decoder offered by VideoToolbox is well-optimized and
performs much better than the ffmpeg decoder.
Signed-off-by: Aman Gupta <aman@tmm1.net>
AVHWAccel.end_frame can run on a worker thread. The assumption of the
frame threading code is that the worker thread will change the AVFrame
image data, not the AVFrame fields. So the AVFrame fields are not synced
back to the main thread. But this breaks videotoolbox due to its special
requirements (everything else is fine). It actually wants to update
AVFrame fields.
The actual videotoolbox frame is now stored in the dummy AVBufferRef, so
it mimics what happens in non-videotoolbox cases. (Changing the
AVBufferRef contents is a bit like changing the image data.) The
post_process callback copies that reference to the proper AVFrame field.
Based on a patch by wm4.
Signed-off-by: Aman Gupta <aman@tmm1.net>
Previously the codec kept an entire copy of the SPS, and restarted the VT decoder
session whenever it changed. This fixed decoding errors in [1], as
described in 9519983c. On further inspection, that sample features an SPS change
from High/4.0 to High/3.2 while moving from one scene to another.
Yesterday I received [2], which contains minor SPS changes where the
profile and level do not change. These occur frequently and are not associated with
scene changes. After 9519983c, the VT decoder session is recreated unnecessarily when
these are encountered causing visual glitches.
This commit simplifies the state kept in the VTContext to include just the first three
bytes of the SPS, containing the profile and level details. This is populated initially
when the VT decoder session is created, and used to detect changes and force a restart.
This means minor SPS changes are fed directly into the existing decoder, whereas
profile/level changes force the decoder session to be recreated with the new parameters.
After this commit, both samples [1] and [2] playback as expected.
[1] https://s3.amazonaws.com/tmm1/videotoolbox/spschange.ts
[2] https://s3.amazonaws.com/tmm1/videotoolbox/spschange2.ts
Signed-off-by: Aman Gupta <aman@tmm1.net>
This is a partial fix - to fix videotoolbox fully (with multithreaded
hwaccel decoding), much more work would be required.
Before this change, an assertion failure would trigger:
$ ffmpeg -hwaccel videotoolbox -hwaccel_output_format videotoolbox_vld -i h264.ts -f null -y /dev/null
Assertion (frame->private_ref && frame->private_ref->size == sizeof(FrameDecodeData)) || !(avctx->codec->capabilities & (1 << 1)) failed at libavcodec/decode.c:620
Signed-off-by: Aman Gupta <aman@tmm1.net>
This includes the SEI_PREFIX/SEI_SUFFIX NALUs, which can contain
updates like HEVC_SEI_TYPE_ACTIVE_PARAMETER_SETS. Previously, hevc
samples with this SEI present would not playback correctly.
See for example https://github.com/lhc70000/iina/issues/1123
Signed-off-by: Aman Gupta <aman@tmm1.net>
Removes the avctx->extradata_size requirement when creating avcC/hvcC, since
avctx->extradata is only used in the esds code path.
This fixes an issue where the VideoToolbox decoder would not work unless
avformat_find_stream_info() was called.
Signed-off-by: Aman Gupta <aman@tmm1.net>
This fixes decoding of H264 video samples with SPS and PPS changes.
See for example https://s3.amazonaws.com/tmm1/videotoolbox/spschange.ts,
which previously stalled the decoder and failed to produce any new frames
after the SPS change.
Also see https://s3.amazonaws.com/tmm1/videotoolbox/ppschange.ts, which
uses multiple PPS and would previously cause VT decode failures.
If the VideoToolbox session needs to be restarted, and
videotoolbox_start() fails for some reason (for instance, if the video
is interlaced and the decoder is running on iOS), avcodec will return
AVERROR_EXTERNAL. This can be used by the API user to switch to another
decoder.
Signed-off-by: Aman Gupta <aman@tmm1.net>
This allows decode_slice to be invoked multiple times before end_frame,
causing slices to accumulate before being fed into the VT decoder.
An upcoming commit will re-use decode_slice for parameter NALUs, so
they can be propagated into the VT decoder session along with slice
data.
Signed-off-by: Aman Gupta <aman@tmm1.net>
These helpers will be used in later commits to automatically restart
the decoder session when SPS changes are encountered.
Signed-off-by: Aman Gupta <aman@tmm1.net>
With the new decode API, you can't handle errors directly in the API
user - you only know that the hwaccel did not initialize at all.
Add some approximate logging.
This adds tons of code for no other benefit than making VideoToolbox
support conform with the new hwaccel API (using hw_device_ctx and
hw_frames_ctx).
Since VideoToolbox decoding does not actually require the user to
allocate frames, the new code does mostly nothing.
One benefit is that ffmpeg_videotoolbox.c can be dropped once generic
hwaccel support for ffmpeg.c is merged from Libav.
Does not consider VDA or VideoToolbox encoding.
Fun fact: the frame transfer functions are copied from vaapi, as the
mapping makes copying generic boilerplate. Mapping itself is not
exported by the VT code, because I don't know how to test.
The way videotoolbox hooks in as a hwaccel is pretty hacky. The VT decode
API is not invoked until end_frame(), so alloc_frame() returns a dummy
frame with a 1-byte buffer. When end_frame() is eventually called, the
dummy buffer is replaced with the actual decoded data from
VTDecompressionSessionDecodeFrame().
When the VT decoder fails, the frame returned to the h264 decoder from
alloc_frame() remains invalid and should not be used. Before
9747219958, it was accidentally being
returned all the way up to the API user. After that commit, the dummy
frame was unref'd so the user received an error.
However, since that commit, VT hwaccel failures started causing random
segfaults in the h264 decoder. This happened more often on iOS where the
VT implementation is more likely to throw errors on bitstream anomolies.
A recent report of this issue can be see in
http://ffmpeg.org/pipermail/libav-user/2016-November/009831.html
The issue here is that the dummy frame is still referenced internally by the
h264 decoder, as part of the reflist and cur_pic_ptr. Deallocating the
frame causes assertions like this one to trip later on during decoding:
Assertion h->cur_pic_ptr->f->buf[0] failed at src/libavcodec/h264_slice.c:1340
With this commit, we leave the dummy 1-byte frame intact, but avoid returning it
to the user.
This reverts commit 9747219958.
Signed-off-by: wm4 <nfxjfg@googlemail.com>
If AVVideotoolboxContext.cv_pix_fmt_type is set to 0, don't set the
kCVPixelBufferPixelFormatTypeKey value on the VT decoder.
This makes VT output its native format, which can be much faster on
some hardware iterations (if the native format does not match with
the requested format, it will be converted, which is slow).
The default is still forcing nv12.
"When you are done with a decompression session you created, call
VTDecompressionSessionInvalidate to tear it down and then CFRelease to
release your object reference."
If videotoolbox_common_end_frame failed, then the AVFrame was returned
to the API user with the dummy buffer (in AVFrame.buf[0]) still set, and
the decode call indicating success.
These "half-set" AVFrames with dummy buffer are a videotoolbox specific
hack, because the decoder requires an allocated AVFrame for its internal
logic. Videotoolbox on the other hand allocates its frame itself
internally, and outputs it only on end_frame. At this point, the dummy
buffer is replaced with the real frame (unless decoding fails).
This affects Annex B streams (such as demuxed from .ts and others). It
also handles the format change in reinit-large_420_8-to-small_420_8.h264
correctly.
Instead of passing through the extradata, create it on the fly it from
the currently active SPS and PPS. Since reconstructing the PPS and SPS
NALs would be very complicated and verbose, we use the NALs as they
originally appeared in the bitstream.
The code for writing the extradata is somewhat derived from
libavformat/avc.c, but it's small and different enough that sharing it
is not really worth it.
VideoToolbox also implements a software decoder for h264, and will fallback to
using it if the file cannot be decoded on the GPU. In these cases though,
we want the hwaccel to fail so that we can use the libavcodec software decoder
instead of the Apple one.
Signed-off-by: wm4 <nfxjfg@googlemail.com>
Jan Ekström
Steven Liu
Rodger Combs
Akemi
Pavel Koshevoy
fumoboy007
Aman Gupta
Mark Thompson
wm4
Clément Bœsch
wang-bin
Matthieu Bouron
Dan Dennedy
Clément Bœsch
Rick Kern
Clément Bœsch
Stefano Pigozzi