V4D offers a way of writing graphical (on- and offscreen) high performance applications with OpenCV. It is light-weight and unencumbered by QT or GTK licenses. It features vector graphics using [NanoVG](https://github.com/inniyah/nanovg) a GUI based on [NanoGUI](https://github.com/mitsuba-renderer/nanogui) and (on supported systems) OpenCL/OpenGL and OpenCL/VAAPI interoperability. It should be included in [OpenCV-contrib](https://github.com/opencv/opencv_contrib) once it is ready.
# Why V4D?
Please refer to the online demos in the \ref v4d_tutorials and \ref v4d_demos section to see at a glance what it can do for you.
Please refer to the online demos in the \ref v4d_tutorials and \ref v4d_demos section to see at a glance what it can do for you.**Note**: The online demos are slower than native builds and are sometimes missing features. If you want full performance (including hardware acceleration) you should really create a native build and test it.
* **OpenGL**: Easy access to OpenGL.
* **GUI**: Simple yet powerful user interfaces through NanoGUI.
* **Vector graphics**: Elegant and fast vector graphics through NanoVG.
* **Font rendering**: Loading of fonts and sophisticated rendering options.
* **Video pipeline**: Through a simple Source/Sink system videos can be efficently read, displayed, edited and saved.
* **Hardware acceleration**: Transparent hardware acceleration usage where possible. (e.g. CL-GL sharing and VAAPI). Actually it is possible to write programs that run almost entirely on the GPU, given driver-features are available.
* **Video pipeline**: Through a simple source/sink system videos can be efficently read, displayed, edited and saved.
* **Hardware acceleration**: Transparent hardware acceleration usage where possible. (e.g. CL-GL interop, VAAPI and CL-VAAPI interop). Actually it is possible to write programs that run almost entirely on the GPU, given driver-features are available.
* **No more highgui** with it's heavy dependencies, licenses and limitations.
* **\ref v4d_webassembly_support**.
# Design Notes
* V4D is not thread safe. Though it is possible to have several V4D objects in one or more threads and synchronize them using ```V4D::makeNonCurrent()``` and ```V4D::makeCurrent()```. This is a limitation of GLFW3. That said, OpenCV algorithms are multi-threaded as usual.
* V4D is not thread safe. Though it is possible to have several V4D objects in one or more threads and synchronize them using ```V4D::makeCurrent()```. This is a limitation of GLFW3/EGL. That said, OpenCV algorithms are multi-threaded as usual.
* V4D uses InputArray/OutputArray/InputOutputArray which gives you the option to work with Mat, std::vector and UMat. Anyway, you should prefer to use UMat whenever possible to automatically use hardware capabilities where available.
* Access to different subsystems (opengl, framebuffer, nanovg and nanogui) is provided through "contexts". A context is simply a function that takes a functor, sets up the subsystem, executes the functor and tears-down the subsystem.
* ```V4D::run``` is not a context. It is an abstraction of a run loop that takes a functor and runs until the application terminates or the functor returns false. This is necessary for portability reasons.
@ -42,25 +42,23 @@ v4d->gl([](const Size sz) {
# Requirements
* C++20 (at the moment)
* OpenGL 3.2 Core (optionally Compat)/OpenGL ES 3.0/OpenGL ES 2.0
* OpenGL 3.2 Core (optionally Compat)/OpenGL ES 3.0/WebGL2
# Optional requirements
* Support for OpenCL 1.2
* Support for cl_khr_gl_sharing and cl_intel_va_api_media_sharing OpenCL extensions.
# Dependencies
* [OpenCV 4.x](https://github.com/opencv/opencv)
* [My OpenCV 4.x fork](https://github.com/kallaballa/opencv) (It works with mainline OpenCV 4.x as well, but will miss some features)
* If you want CL-GL sharing on a recent Intel Platform (Gen8 - Gen12) you currently **need to build** [compute-runtime](https://github.com/intel/compute-runtime) and [my OpenCV 4.x fork](https://github.com/kallaballa/opencv/tree/GCV)
* If you want CL-GL sharing on a recent Intel Platform (Gen8 - Gen12) you currently **need to build** [compute-runtime](https://github.com/intel/compute-runtime)
# Tutorials {#v4d_tutorials}
The tutorials are designed to be read one after the other to give you a good overview over the key concepts of V4D. After that you can move on to the samples.
The tutorials are designed to be read one after the other to give you a good overview over the key concepts of V4D. After that you can move on to the demos.
* \ref v4d_display_image_pipeline
* \ref v4d_display_image_fb
@ -90,16 +88,16 @@ You need to build OpenCV with V4D
This example simply paints the screen blue using OpenGL without shaders for brevity.
This example simply paints the screen blue using OpenGL without shaders for brevity. One important detail of this example is that states are being preserved between invocations of a context type (in this case the ```gl``` context).
Through adding a Source and a Sink v4d becomes capable of video editing. Reads a video, renders text on top and writes the result. Note: Reading and writing of video-data is multi-threaded in the background for performance reasons.
Through adding a source and a sink v4d becomes capable of video editing. Reads a video, renders text on top and writes the result. Note: Reading and writing of video-data is multi-threaded in the background for performance reasons.
| Original author | Amir Hassan (kallaballa) <amir@viel-zu.org> |
| Compatibility | OpenCV >= 4.7 |
## Reading and writing to V4D using custom Sources and Sinks
In the previous tutorial we used a default video source and a video sink to stream a video through V4D which can be manipulated using OpenGL, NanoVG or OpenCV. In this example we are creating a custom Source that generates rainbow frames. For each time the Source is invoked the frame is colored a slightly different color. Additionally the custom Sink saves individual images instead of a video.
## Reading and writing to V4D using custom sources and sinks
In the previous tutorial we used a default video source and a video sink to stream a video through V4D which can be manipulated using OpenGL, NanoVG or OpenCV. In this example we are creating a custom source that generates rainbow frames. For each time the source is invoked the frame is colored a slightly different color. Additionally the custom sink saves individual images instead of a video (only in native builds).
| Original author | Amir Hassan (kallaballa) <amir@viel-zu.org> |
| Compatibility | OpenCV >= 4.7 |
Renders a rainbow cube on white background using OpenGL, applies a glow effect using OpenCV (OpenCL) and encodes on the GPU (VAAPI).
Renders a rainbow cube on blueish background using OpenGL, applies a glow effect using OpenCV (optionally using OpenCL with native builds) and encodes the video (optionally using VAAPI and CL-VA interop with native builds).
| Original author | Amir Hassan (kallaballa) <amir@viel-zu.org> |
| Compatibility | OpenCV >= 4.7 |
Optical flow visualization on top of a video. Uses background subtraction (OpenCV/OpenCL) to isolate areas with motion, detects features to track (OpenCV/OpenCL), calculates the optical flow (OpenCV/OpenCL), uses nanovg for rendering (OpenGL) and post-processes the video (OpenCL). Decodes/encodes on the GPU (VAAPI).
Optical flow visualization on top of a video. Uses background subtraction (OpenCV/OpenCL) to isolate areas with motion, detects features to track (OpenCV/OpenCL), calculates the optical flow (OpenCV/OpenCL), uses nanovg for rendering (OpenGL) and post-processes the video (OpenCV/OpenCL). Decodes/encodes on the video (VAAPI).
kallaballa
2 years ago