mirror of https://github.com/FFmpeg/FFmpeg.git
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
288 lines
11 KiB
288 lines
11 KiB
Filter design |
|
============= |
|
|
|
This document explains guidelines that should be observed (or ignored with |
|
good reason) when writing filters for libavfilter. |
|
|
|
In this document, the word “frame” indicates either a video frame or a group |
|
of audio samples, as stored in an AVFrame structure. |
|
|
|
|
|
Format negotiation |
|
================== |
|
|
|
The query_formats method should set, for each input and each output links, |
|
the list of supported formats. |
|
|
|
For video links, that means pixel format. For audio links, that means |
|
channel layout, sample format (the sample packing is implied by the sample |
|
format) and sample rate. |
|
|
|
The lists are not just lists, they are references to shared objects. When |
|
the negotiation mechanism computes the intersection of the formats |
|
supported at each end of a link, all references to both lists are replaced |
|
with a reference to the intersection. And when a single format is |
|
eventually chosen for a link amongst the remaining list, again, all |
|
references to the list are updated. |
|
|
|
That means that if a filter requires that its input and output have the |
|
same format amongst a supported list, all it has to do is use a reference |
|
to the same list of formats. |
|
|
|
query_formats can leave some formats unset and return AVERROR(EAGAIN) to |
|
cause the negotiation mechanism to try again later. That can be used by |
|
filters with complex requirements to use the format negotiated on one link |
|
to set the formats supported on another. |
|
|
|
|
|
Frame references ownership and permissions |
|
========================================== |
|
|
|
Principle |
|
--------- |
|
|
|
Audio and video data are voluminous; the frame and frame reference |
|
mechanism is intended to avoid, as much as possible, expensive copies of |
|
that data while still allowing the filters to produce correct results. |
|
|
|
The data is stored in buffers represented by AVFrame structures. |
|
Several references can point to the same frame buffer; the buffer is |
|
automatically deallocated once all corresponding references have been |
|
destroyed. |
|
|
|
The characteristics of the data (resolution, sample rate, etc.) are |
|
stored in the reference; different references for the same buffer can |
|
show different characteristics. In particular, a video reference can |
|
point to only a part of a video buffer. |
|
|
|
A reference is usually obtained as input to the filter_frame method or |
|
requested using the ff_get_video_buffer or ff_get_audio_buffer |
|
functions. A new reference on an existing buffer can be created with |
|
av_frame_ref(). A reference is destroyed using |
|
the av_frame_free() function. |
|
|
|
Reference ownership |
|
------------------- |
|
|
|
At any time, a reference “belongs” to a particular piece of code, |
|
usually a filter. With a few caveats that will be explained below, only |
|
that piece of code is allowed to access it. It is also responsible for |
|
destroying it, although this is sometimes done automatically (see the |
|
section on link reference fields). |
|
|
|
Here are the (fairly obvious) rules for reference ownership: |
|
|
|
* A reference received by the filter_frame method belongs to the |
|
corresponding filter. |
|
|
|
* A reference passed to ff_filter_frame is given away and must no longer |
|
be used. |
|
|
|
* A reference created with av_frame_ref() belongs to the code that |
|
created it. |
|
|
|
* A reference obtained with ff_get_video_buffer or ff_get_audio_buffer |
|
belongs to the code that requested it. |
|
|
|
* A reference given as return value by the get_video_buffer or |
|
get_audio_buffer method is given away and must no longer be used. |
|
|
|
Link reference fields |
|
--------------------- |
|
|
|
The AVFilterLink structure has a few AVFrame fields. |
|
|
|
partial_buf is used by libavfilter internally and must not be accessed |
|
by filters. |
|
|
|
fifo contains frames queued in the filter's input. They belong to the |
|
framework until they are taken by the filter. |
|
|
|
Reference permissions |
|
--------------------- |
|
|
|
Since the same frame data can be shared by several frames, modifying may |
|
have unintended consequences. A frame is considered writable if only one |
|
reference to it exists. The code owning that reference it then allowed |
|
to modify the data. |
|
|
|
A filter can check if a frame is writable by using the |
|
av_frame_is_writable() function. |
|
|
|
A filter can ensure that a frame is writable at some point of the code |
|
by using the ff_inlink_make_frame_writable() function. It will duplicate |
|
the frame if needed. |
|
|
|
A filter can ensure that the frame passed to the filter_frame() callback |
|
is writable by setting the needs_writable flag on the corresponding |
|
input pad. It does not apply to the activate() callback. |
|
|
|
|
|
Frame scheduling |
|
================ |
|
|
|
The purpose of these rules is to ensure that frames flow in the filter |
|
graph without getting stuck and accumulating somewhere. |
|
|
|
Simple filters that output one frame for each input frame should not have |
|
to worry about it. |
|
|
|
There are two design for filters: one using the filter_frame() and |
|
request_frame() callbacks and the other using the activate() callback. |
|
|
|
The design using filter_frame() and request_frame() is legacy, but it is |
|
suitable for filters that have a single input and process one frame at a |
|
time. New filters with several inputs, that treat several frames at a time |
|
or that require a special treatment at EOF should probably use the design |
|
using activate(). |
|
|
|
activate |
|
-------- |
|
|
|
This method is called when something must be done in a filter; the |
|
definition of that "something" depends on the semantic of the filter. |
|
|
|
The callback must examine the status of the filter's links and proceed |
|
accordingly. |
|
|
|
The status of output links is stored in the frame_wanted_out, status_in |
|
and status_out fields and tested by the ff_outlink_frame_wanted() |
|
function. If this function returns true, then the processing requires a |
|
frame on this link and the filter is expected to make efforts in that |
|
direction. |
|
|
|
The status of input links is stored by the status_in, fifo and |
|
status_out fields; they must not be accessed directly. The fifo field |
|
contains the frames that are queued in the input for processing by the |
|
filter. The status_in and status_out fields contains the queued status |
|
(EOF or error) of the link; status_in is a status change that must be |
|
taken into account after all frames in fifo have been processed; |
|
status_out is the status that have been taken into account, it is final |
|
when it is not 0. |
|
|
|
The typical task of an activate callback is to first check the backward |
|
status of output links, and if relevant forward it to the corresponding |
|
input. Then, if relevant, for each input link: test the availability of |
|
frames in fifo and process them; if no frame is available, test and |
|
acknowledge a change of status using ff_inlink_acknowledge_status(); and |
|
forward the result (frame or status change) to the corresponding input. |
|
If nothing is possible, test the status of outputs and forward it to the |
|
corresponding input(s). If still not possible, return FFERROR_NOT_READY. |
|
|
|
If the filters stores internally one or a few frame for some input, it |
|
can consider them to be part of the FIFO and delay acknowledging a |
|
status change accordingly. |
|
|
|
Example code: |
|
|
|
ret = ff_outlink_get_status(outlink); |
|
if (ret) { |
|
ff_inlink_set_status(inlink, ret); |
|
return 0; |
|
} |
|
if (priv->next_frame) { |
|
/* use it */ |
|
return 0; |
|
} |
|
ret = ff_inlink_consume_frame(inlink, &frame); |
|
if (ret < 0) |
|
return ret; |
|
if (ret) { |
|
/* use it */ |
|
return 0; |
|
} |
|
ret = ff_inlink_acknowledge_status(inlink, &status, &pts); |
|
if (ret) { |
|
/* flush */ |
|
ff_outlink_set_status(outlink, status, pts); |
|
return 0; |
|
} |
|
if (ff_outlink_frame_wanted(outlink)) { |
|
ff_inlink_request_frame(inlink); |
|
return 0; |
|
} |
|
return FFERROR_NOT_READY; |
|
|
|
The exact code depends on how similar the /* use it */ blocks are and |
|
how related they are to the /* flush */ block, and needs to apply these |
|
operations to the correct inlink or outlink if there are several. |
|
|
|
Macros are available to factor that when no extra processing is needed: |
|
|
|
FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); |
|
FF_FILTER_FORWARD_STATUS_ALL(outlink, filter); |
|
FF_FILTER_FORWARD_STATUS(inlink, outlink); |
|
FF_FILTER_FORWARD_STATUS_ALL(inlink, filter); |
|
FF_FILTER_FORWARD_WANTED(outlink, inlink); |
|
|
|
filter_frame |
|
------------ |
|
|
|
For filters that do not use the activate() callback, this method is |
|
called when a frame is pushed to the filter's input. It can be called at |
|
any time except in a reentrant way. |
|
|
|
If the input frame is enough to produce output, then the filter should |
|
push the output frames on the output link immediately. |
|
|
|
As an exception to the previous rule, if the input frame is enough to |
|
produce several output frames, then the filter needs output only at |
|
least one per link. The additional frames can be left buffered in the |
|
filter; these buffered frames must be flushed immediately if a new input |
|
produces new output. |
|
|
|
(Example: frame rate-doubling filter: filter_frame must (1) flush the |
|
second copy of the previous frame, if it is still there, (2) push the |
|
first copy of the incoming frame, (3) keep the second copy for later.) |
|
|
|
If the input frame is not enough to produce output, the filter must not |
|
call request_frame to get more. It must just process the frame or queue |
|
it. The task of requesting more frames is left to the filter's |
|
request_frame method or the application. |
|
|
|
If a filter has several inputs, the filter must be ready for frames |
|
arriving randomly on any input. Therefore, any filter with several inputs |
|
will most likely require some kind of queuing mechanism. It is perfectly |
|
acceptable to have a limited queue and to drop frames when the inputs |
|
are too unbalanced. |
|
|
|
request_frame |
|
------------- |
|
|
|
For filters that do not use the activate() callback, this method is |
|
called when a frame is wanted on an output. |
|
|
|
For a source, it should directly call filter_frame on the corresponding |
|
output. |
|
|
|
For a filter, if there are queued frames already ready, one of these |
|
frames should be pushed. If not, the filter should request a frame on |
|
one of its inputs, repeatedly until at least one frame has been pushed. |
|
|
|
Return values: |
|
if request_frame could produce a frame, or at least make progress |
|
towards producing a frame, it should return 0; |
|
if it could not for temporary reasons, it should return AVERROR(EAGAIN); |
|
if it could not because there are no more frames, it should return |
|
AVERROR_EOF. |
|
|
|
The typical implementation of request_frame for a filter with several |
|
inputs will look like that: |
|
|
|
if (frames_queued) { |
|
push_one_frame(); |
|
return 0; |
|
} |
|
input = input_where_a_frame_is_most_needed(); |
|
ret = ff_request_frame(input); |
|
if (ret == AVERROR_EOF) { |
|
process_eof_on_input(); |
|
} else if (ret < 0) { |
|
return ret; |
|
} |
|
return 0; |
|
|
|
Note that, except for filters that can have queued frames and sources, |
|
request_frame does not push frames: it requests them to its input, and |
|
as a reaction, the filter_frame method possibly will be called and do |
|
the work.
|
|
|