The issue with the old mechanism is that we had to introduce new
API each time we needed a new queue family, and all the queue families
were functionally fixed to a given purpose.
Nvidia's GPUs are able to handle video encoding and compute on the
same queue, which results in a speedup when pre-processing is required.
Also, this enables us to expose optical flow queues for frame interpolation.
struct Foo * declares a new type (namely struct Foo)
if there is no declaration of struct Foo already visible
in the current scope; otherwise it is just a pointer to
an element of the already declared type "struct Foo".
There is a gotcha with the first case:
struct Foo is only declared in its scope; a later declaration
of struct Foo in an enclosing scope declares a different type.
This happens in hwcontext_vulkan.h if it is included before
hwcontext.h, because some declarations of struct AVHWDeviceContext
and struct AVHWFramesContext have function prototype scope.
Compilers warn about this (during checkheaders):
‘struct AVHWDeviceContext’ declared inside parameter list will not
be visible outside of this definition or declaration
Fix this by including hwcontext.h.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
This commit adds proper handling of multiplane images throughout
all of the hwcontext code. To avoid breakage of individual
components, the change is performed as a single commit.
The hack was added to enable exporting of vulkan images to DRM.
On Intel hardware, specifically for DRM images, all planes must be
allocated next to each other, due to hardware limitation, so the hack
used a single large allocation and suballocated all planes from it.
By natively supporting multiplane images, the driver is what decides
the layout, so exporting just works.
It's a hack because it conflicted heavily with image allocation, and
with the whole ecosystem in general, before multiplane images were
supported, which just made it redundant.
This is also the commit which broke the hwcontext hardest and prompted
the entire rewrite in the first place.
This just bumps the required loader library version (libvulkan).
All device-related features, such as video decoding, atomics, etc.
are still optional and the code deals with their loss on a local level
(e.g. the decoder or filter checks for the features it needs, not
the hwcontext).
Bumping the required version essentially packs all maintenance
extensions which correct the spec rather than requiring to enable
them individually.
Always require one semaphore per sw_format plane. This is what
the implementation uses and relies upon throughout. This was
a leftover from an earlier revision that was never needed.
When vulkan image exports to drm, the tilling need to be
VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT. Now add code to create vulkan
image using this format.
Now the following command line works:
ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format \
vaapi -i input_1080p.264 -vf "hwmap=derive_device=vulkan,format=vulkan, \
scale_vulkan=1920:1080,hwmap=derive_device=vaapi,format=vaapi" -c:v h264_vaapi output.264
Signed-off-by: Wenbin Chen <wenbin.chen@intel.com>
Further-modifications-by: Lynne <dev@lynne.ee>
VAAPI on Intel can import external frame, but the planes of the external
frames should be in the same drm object. A new option "contiguous_planes"
is added to device. This flag tells device to allocate places in one
memory. When device is derived from vaapi this flag will be enabled.
A new flag frame_flag is also added to AVVulkanFramesContext. User
can use this flag to force enable or disable this behaviour.
A new variable "offset "is added to AVVKFrame. It describe describe the
offset from the memory currently bound to the VkImage.
Signed-off-by: Wenbin Chen <wenbin.chen@intel.com>
Further-modifications-by: Lynne <dev@lynne.ee>
This simplifies and makes queue family picking simpler and more robust.
The requirements on the device context are relaxed. They made no sense
in the first place.
The video encode/decode extension is still in beta, at least on paper,
but I really doubt they'd change needing a separate queue family.
make checkheaders will get error as follow:
CC libavutil/hwcontext_vulkan.h.o
In file included from libavutil/hwcontext_vulkan.h.c:1:
./libavutil/hwcontext_vulkan.h:130:23: error: ‘AV_NUM_DATA_POINTERS’ undeclared here (not in a function)
130 | void *alloc_pnext[AV_NUM_DATA_POINTERS];
| ^~~~~~~~~~~~~~~~~~~~
./libavutil/hwcontext_vulkan.h:199:43: warning: ‘enum AVPixelFormat’ declared inside parameter list will not be visible outside of this definition or declaration
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
With this, the puzzle of making libplacebo, ffmpeg and any other Vulkan
API users interoperable is complete.
Users of both libraries can initialize one another's contexts without having
to create a new one.
This reverts commit 97b526c192.
It broke the API, and assumed no other APIs used multiple semaphores.
This also disallowed certain optimizations to happen.
Dealing with APIs that give or expect single semaphores is easier when
we use per-image semaphores.
As it turns out, we were already assuming and treating all images as if they had
concurrent access mode. This just changes the flag to CONCURRENT, which has less
restrictions than EXCLUSIVE, and fixed validation messages on machines with
multiple queues.
The validation layer didn't pick this up because the machine I was testing on
had only a single queue.
This solves a huge oversight - it lets users reliably use their own
AVVulkanDeviceContext. Otherwise, the extensions supplied and enabled
are not discoverable by anything outside of hwcontext_vulkan.
Also clarifies that any user-supplied VkInstance must be at least 1.1.
The idea was to allow separate planes to be filtered independently, however,
in hindsight, literaly nothing uses separate per-plane semaphores and it
would only work when each plane is backed by separate device memory.
This commit adds the necessary code to initialize and use a Vulkan device
within the hwcontext libavutil framework.
Currently direct mapping to VAAPI and DRM frames is functional, and
transfers to CUDA and native frames are supported.
Lets hope the future Vulkan video decode extension fits well within this
framework.
Lynne
Andreas Rheinhardt
Wenbin Chen
Timo Rothenpieler
Jun Zhao