Switches to av_frame_side_data_remove_by_props(), covering a number of
cases that we previously ignored. Additionally, stop stripping metadata
when merely changing colorspace or color range, since these do not
affect the actual color volume of the image data, only the encoding.
As discussed in the previous commit, we often need a convenient way of
stripping all side data related to a certain aspect of the frame. This helper
accomplishes just that.
I considered also adding a way to match only side data matching *all*
properties, but I think this is sufficiently useless in practise to not warrant
inclusion in the API.
Many filters modify certain aspects of frame data, e.g. through resizing
(vf_*scale* family), color volume mapping (vf_lut*, vf_tonemap*), or
possibly others.
When this happens, we should strip all frame side data that will no
longer be correct/relevant after the operation. For example, changing
the image size should invalidate AV_FRAME_DATA_PANSCAN because the crop
window (given in pixels) no longer corresponds to the actual image size.
For another example, tone-mapping filters (e.g. from HDR to SDR) should
strip all of the dynamic HDR related metadata.
Since there are a lot of different side data types that are affected by such
operations, it makes sense to establish this information in a common, easily
accessible way. The existing side data properties enum is a perfect fit for
this.
While this one was technically supported on account of the generic options
code allowing "none" as a valid alias for 0, not having it listed here meant
it never showed up in e.g. the -h output, and is also inconsistent with other
places in the code that generally add an explicit alias with appropriate
documentation. Reduces user confusion at negligible cost.
Fixes: #9192
Signed-off-by: Niklas Haas <git@haasn.dev>
Sponsored-by: Sovereign Tech Fund
This leverages the previously introduced color management subsystem in order
to adapt between transfer functions and color spaces, as well as for HDR tone
mapping.
Take special care to handle grayscale formats without a colorspace
gracefully.
This is a lightweight wrapper around the underlying color management system,
whose job it is merely to manage the 3DLUT state and apply them to the frame
data. This is where we might add platform-specific optimizations in the future.
I also plan on adding support for more pixel formats in the future. In
particular, we could support YUV or XYZ input formats directly using only
negligible additional code in the 3DLUT setup functions. This would eliminate
the major source of slowdown, which is currently the roundtrip to RGBA64.
The underlying color mapping logic was ported as straightforwardly as possible
from libplacebo, although the API and glue code has been very heavily
refactored / rewritten. In particular, the generalization of gamut mapping
methods is replaced by a single ICC intent selection, and constants have been
hard-coded.
To minimize the amount of overall operations, this gamut mapping LUT now embeds
a direct end-to-end transformation to the output color space; something that
libplacebo does in shaders, but which is prohibitively expensive in software.
In order to preserve compatibility with dynamic tone mapping without severely
regressing performance, we add the ability to generate a pair of "split" LUTS,
one for encoding the input and output to the perceptual color space, and a
third to embed the tone mapping operation. Additionally, this intermediate
space could be used for additional subjective effect (e.g. changing
saturation or brightness).
The big downside of the new approach is that generating a static color mapping
LUT is now fairly slow, as the chromaticity lobe peaks have to be recomputed
for every single RGB value, since correlated RGB colors are not necessarily
aligned in ICh space. Generating a split 3DLUT significantly alleviates this
problem because the expensive step is done as part of the IPT input LUT, which
can share the same hue peak calculation at least for all input intensities.
This setting can be used to infuence the type of tone and gamut mapping used
internally when color space conversions are required. As discussed at VDD'24,
the default was set to relative colorimetric clipping, which is approximately
associative, surjective and idempotent. As such, it roundtrips well, although
it is strictly speaking not associative on out-of-gamut colors.
Swscale currently handles XYZ by embedding a forced conversion to
BT.709 RGB with a hardcoded matrix. This is not ideal, but to preserve the
status quo and avoid any unexpected changes in behavior, this patch merely
fixes the inferred primaries tag to match the reality.
In the future, I would like to handle XYZ properly, via direct conversion
to the target colorspace (or possibly simply by using a more fitting
RGB intermediate like SMPTE428), but for now just keep the status quo.
Logic is loosely on equivalent decisions in libplacebo. The basic idea is to try
and be a bit conservative by treating AVCOL_*_UNSPECIFIED as a no-op, unless the
other primaries set are non-standard / wide-gamut or HDR. This helps avoid
unintended or unexpected colorspace conversions, while forcing it in cases where
we are almost certain it is needed. The major departure from libplacebo semantics
is that we no default to a 1000:1 contrast ration for SDR displays, instead modelling
them as idealized devices with an infinite contrast ratio.
In either case, setting SWS_STRICT overrides this behavior in favor of always
requiring explicit colorspace metadata.
Logic ported from libplacebo's AVFrame helpers. The basic idea is to use the
provided MaxRGB/MaxSCL values to infer what the actual luminance would have
been, which HDR10+ metadata does not provide directly. It's worth pointing out
that this gives us an *upper* bound on the true maximum luminance, so any
error in the estimation cannot result in clipping.
Only add the condensed values that we actually care about. Group them into
a new struct to make it easier to discard or replace this metadata.
Define a special comparison function that does not choke on undefined/unknown
metadata.
As fixed in the previous commit, this enables semipacked range and
bit depth conversions. Previously these would go through the general
purpose path.
Signed-off-by: Niklas Haas <git@haasn.dev>
Sponsored-by: Sovereign Tech Fund
This fixes multiple bugs with semiplanar formats like NV12. Not only do these
false positive the grayscale format checks (because dst[2] in NULL), but they
also copied an incorrect number of pixels.
Fixes conversions such as nv12 -> nv12, gray8 -> nv12, nv20le -> nv20be, etc.
Fixes: #11239
Signed-off-by: Niklas Haas <git@haasn.dev>
Sponsored-by: Sovereign Tech Fund
We should at least bias towards the nearest integer, instead of always
rounding down, when not dithering. This is a bit more correct.
The FATE changes are only in the cases where sws_dither was explicitly set
to "none", which is exactly as expected.
Signed-off-by: Niklas Haas <git@haasn.dev>
Sponsored-by: Sovereign Tech Fund
This code only checks hcScale. In practice this is not an issue because
the function pointers should always be identical to hyScale for the same
filter size.
Add an assertion just to make sure this assumption never regresses.
Signed-off-by: Niklas Haas <git@haasn.dev>
Sponsored-by: Sovereign Tech Fund
We recently introduced a public field which was a superset
of the queue context we used to have.
Switch to using it entirely.
This also allows us to get rid of the NIH function which was
valid only for video queues.
This code was simply incorrect through and through. It did not
protect what actually has to be protected in a multi-threaded setup.
Perhaps it was used to silence threading errors?
Either way, remove it, and document the correct way to use execution
pools in a threaded environment.
Originally, the decoder had a single execution pool, with one
execution context per thread. Execution pools were always intended
to be thread-safe, as long as there were enough execution contexts
in the pool to satisfy all threads.
Due to synchronization issues, the threading part was removed at some
point, and, for decoding, each thread had its own execution pool.
Having a single execution pool per context is hacky, not to mention
wasteful.
Most importantly, we *cannot* associate single shaders across multiple
execution pools for a single application. This means that we cannot
use shaders to either apply film grain, or use this framework for
software-defined decoders.
The recent commits added threading capabilities back to the execution
pool, and the number of contexts in each pool was increased. This was
done with the assumption that the execution pool was singular, which
it was not. This led to increased parallelism and number of frames
in flight, which is taxing on memory.
This commit finally restores proper threading behaviour.
The validation layer has isses that are reported and addressed in the
earlier commit.
Some drivers are more strict about the size of the reference lists given
(i.e. VAOn12 [1]). The next_prev list is used to handle multiple "L0"
references in AV1 encode. Restrict the size of next_prev based on the
value of ref_l0 when the GOP structure is initialized.
[1] https://github.com/intel/cartwheel-ffmpeg/issues/278
v2: fix indentation issues
These functions were divided into two special cases; one assuming that
uvalpha == 0, and the other assuming that uvalpha == 2048. This worked fine
for simple 2x chroma upscaling but broke for e.g. yuv410p, non-centered chroma,
or other special cases that involved non-aligned chroma filters.
Fix it by instead dividing this check into two cases, a uvalpha==0 fast path
and a uvalpha>0 general path. Instead of (A+B)/2 the general path now multiplies
in the true uvalpha weight.
I tried preserving the old fast path for the case of uvalpha == 2048, but this
was significantly slower in practise versus having just one general path.
However, we still need a uvalpha == 0 path for the unscaled case.
Fixes: ticket #5083
Signed-off-by: Niklas Haas <git@haasn.dev>
Sponsored-by: Sovereign Tech Fund
Niklas Haas
Scott Theisen
Lynne
Benjamin Cheng
Nuo Mi
Peter Ross
Paul B Mahol
sunyuechi
Zhao Zhili