mirror of https://github.com/opencv/opencv.git
Merge pull request #12857 from dmatveev:hld
* G-API Documentation: first submission This PR introduces a number of new OpenCV documentation chapters for Graph API module. In particular, the following topics are covered: - Introduction & background information; - High-level design overview; - Kernel API; - Pipeline example. All changes are done in Markdown files, no headers, etc modified. Doxygen references for main API classes will be added later. Also, a tutorial will be introduced soon (in the common Tutorials place) * G-API Documentation - fix warnings & trailing whitespaces * G-API Documentation: address review issues * G-API Documentation: export code snippets to compileable files * gapi: move documentation samplespull/12922/head
parent
ea31c09384
commit
dbed39a931
15 changed files with 755 additions and 8 deletions
@ -0,0 +1,109 @@ |
||||
# Graph API {#gapi} |
||||
|
||||
# Introduction {#gapi_root_intro} |
||||
|
||||
OpenCV Graph API (or G-API) is a new OpenCV module targeted to make |
||||
regular image processing fast and portable. These two goals are |
||||
achieved by introducing a new graph-based model of execution. |
||||
|
||||
G-API is a special module in OpenCV -- in contrast with the majority |
||||
of other main modules, this one acts as a framework rather than some |
||||
specific CV algorithm. G-API provides means to define CV operations, |
||||
construct graphs (in form of expressions) using it, and finally |
||||
implement and run the operations for a particular backend. |
||||
|
||||
# Contents |
||||
|
||||
G-API documentation is organized into the following chapters: |
||||
|
||||
- @subpage gapi_purposes |
||||
|
||||
The motivation behind G-API and its goals. |
||||
|
||||
- @subpage gapi_hld |
||||
|
||||
General overview of G-API architecture and its major internal |
||||
components. |
||||
|
||||
- @subpage gapi_kernel_api |
||||
|
||||
Learn how to introduce new operations in G-API and implement it for |
||||
various backends. |
||||
|
||||
- @subpage gapi_impl |
||||
|
||||
Low-level implementation details of G-API, for those who want to |
||||
contribute. |
||||
|
||||
- API Reference: functions and classes |
||||
|
||||
- @subpage gapi_core |
||||
|
||||
Core G-API operations - arithmetic, boolean, and other matrix |
||||
operations; |
||||
|
||||
- @subpage gapi_imgproc |
||||
|
||||
Image processing functions: color space conversions, various |
||||
filters, etc. |
||||
|
||||
# API Example {#gapi_example} |
||||
|
||||
A very basic example of G-API pipeline is shown below: |
||||
|
||||
@include modules/gapi/samples/api_example.cpp |
||||
|
||||
<!-- TODO align this code with text using marks and itemized list --> |
||||
|
||||
G-API is a separate OpenCV module so its header files have to be |
||||
included explicitly. The first four lines of `main()` create and |
||||
initialize OpenCV's standard video capture object, which fetches |
||||
video frames from either an attached camera or a specified file. |
||||
|
||||
G-API pipelie is constructed next. In fact, it is a series of G-API |
||||
operation calls on cv::GMat data. The important aspect of G-API is |
||||
that this code block is just a declaration of actions, but not the |
||||
actions themselves. No processing happens at this point, G-API only |
||||
tracks which operations form pipeline and how it is connected. G-API |
||||
_Data objects_ (here it is cv::GMat) are used to connect operations |
||||
each other. `in` is an _empty_ cv::GMat signalling that it is a |
||||
beginning of computation. |
||||
|
||||
After G-API code is written, it is captured into a call graph with |
||||
instantiation of cv::GComputation object. This object takes |
||||
input/output data references (in this example, `in` and `out` |
||||
cv::GMat objects, respectively) as parameters and reconstructs the |
||||
call graph based on all the data flow between `in` and `out`. |
||||
|
||||
cv::GComputation is a thin object in sense that it just captures which |
||||
operations form up a computation. However, it can be used to execute |
||||
computations -- in the following processing loop, every captured frame (a |
||||
cv::Mat `input_frame`) is passed to cv::GComputation::apply(). |
||||
|
||||
 |
||||
|
||||
cv::GComputation::apply() is a polimorphic method which accepts a |
||||
variadic number of arguments. Since this computation is defined on one |
||||
input, one output, a special overload of cv::GComputation::apply() is |
||||
used to pass input data and get output data. |
||||
|
||||
Internally, cv::GComputation::apply() compiles the captured graph for |
||||
the given input parameters and executes the compiled graph on data |
||||
immediately. |
||||
|
||||
There is a number important concepts can be outlines with this examle: |
||||
* Graph declaration and graph execution are distinct steps; |
||||
* Graph is built implicitly from a sequence of G-API expressions; |
||||
* G-API supports function-like calls -- e.g. cv::gapi::resize(), and |
||||
operators, e.g operator|() which is used to compute bitwise OR; |
||||
* G-API syntax aims to look pure: every operation call within a graph |
||||
yields a new result, thus forming a directed acyclic graph (DAG); |
||||
* Graph declaration is not bound to any data -- real data objects |
||||
(cv::Mat) come into picture after the graph is already declared. |
||||
|
||||
<!-- FIXME: The above operator|() link links to MatExpr not GAPI --> |
||||
|
||||
See Tutorial[TBD] and Porting examples[TBD] to learn more on various |
||||
G-API features and concepts. |
||||
|
||||
<!-- TODO Add chapter on declaration, compilation, execution --> |
@ -0,0 +1,76 @@ |
||||
# Why Graph API? {#gapi_purposes} |
||||
|
||||
# Motivation behind G-API {#gapi_intro_why} |
||||
|
||||
G-API module brings graph-based model of execution to OpenCV. This |
||||
chapter briefly describes how this new model can help software |
||||
developers in two aspects: optimizing and porting image processing |
||||
algorithms. |
||||
|
||||
## Optimizing with Graph API {#gapi_intro_opt} |
||||
|
||||
Traditionally OpenCV provided a lot of stand-alone image processing |
||||
functions (see modules `core` and `imgproc`). Many of that functions |
||||
are well-optimized (e.g. vectorized for specific CPUs, parallel, etc) |
||||
but still the out-of-box optimization scope has been limited to a |
||||
single function only -- optimizing the whole algorithm built atop of that |
||||
functions was a responsibility of a programmer. |
||||
|
||||
OpenCV 3.0 introduced _Transparent API_ (or _T-API_) which allowed to |
||||
offload OpenCV function calls transparently to OpenCL devices and save |
||||
on Host/Device data transfers with cv::UMat -- and it was a great step |
||||
forward. However, T-API is a dynamic API -- user code still remains |
||||
unconstrained and OpenCL kernels are enqueued in arbitrary order, thus |
||||
eliminating further pipeline-level optimization potential. |
||||
|
||||
G-API brings implicit graph model to OpenCV 4.0. Graph model captures |
||||
all operations and its data dependencies in a pipeline and so provides |
||||
G-API framework with extra information to do pipeline-level |
||||
optimizations. |
||||
|
||||
The cornerstone of graph-based optimizations is _Tiling_. Tiling |
||||
allows to break the processing into smaller parts and reorganize |
||||
operations to enable data parallelism, improve data locality, and save |
||||
memory footprint. Data locality is an especially important aspect of |
||||
software optimization due to diffent costs of memory access on modern |
||||
computer architectures -- the more data is reused in the first level |
||||
cache, the more efficient pipeline is. |
||||
|
||||
Definitely the aforementioned techinques can be applied manually -- |
||||
but it requires extra skills and knowledge of the target platform and |
||||
the algorithm implementation changes irrevocably -- becoming more |
||||
specific, less flexible, and harder to extend and maintain. |
||||
|
||||
G-API takes this responsiblity and complexity from user and does the |
||||
majority of the work by itself, keeping the algorithm code clean from |
||||
device or optimization details. This approach has its own limitations, |
||||
though, as graph model is a _constrained_ model and not every |
||||
algorithm can be represented as a graph, so the G-API scope is limited |
||||
only to regular image processing -- various filters, arithmentic, |
||||
binary operations, and well-defined geometrical transformations. |
||||
|
||||
## Porting with Graph API {#gapi_intro_port} |
||||
|
||||
The essense of G-API is declaring a sequence of operations to run, and |
||||
then executing that sequence. G-API is a constrained API, so it puts a |
||||
number of limitations on which operations can form a pipeline and |
||||
which data these operations may exchange each other. |
||||
|
||||
This formalization in fact helps to make an algorithm portable. G-API |
||||
clearly separates operation _interfaces_ from its _implementations_. |
||||
|
||||
One operation (_kernel_) may have multiple implementations even for a |
||||
single device (e.g., OpenCV-based "reference" implementation and a |
||||
tiled optimized implementation, both running on CPU). Graphs (or |
||||
_Computations_ in G-API terms) are built only using operation |
||||
interfaces, not implementations -- thus the same graph can be executed |
||||
on different devices (and, of course, using different optimization |
||||
techniques) with little-to-no changes in the graph itself. |
||||
|
||||
G-API supports plugins (_Backends_) which aggreate logic and |
||||
intelligence on what is the best way to execute on a particular |
||||
platform. Once a pipeline is built with G-API, it can be parametrized |
||||
to use either of the backends (or a combination of it) and so a graph |
||||
can be ported easily to a new platform. |
||||
|
||||
@sa @ref gapi_hld |
@ -0,0 +1,159 @@ |
||||
# High-level design overview {#gapi_hld} |
||||
|
||||
# G-API High-level design overview |
||||
|
||||
[TOC] |
||||
|
||||
G-API is a heterogeneous framework and provides single API to program |
||||
image processing pipelines with a number of supported backends. |
||||
|
||||
The key design idea is to keep pipeline code itself platform-neutral |
||||
while specifying which kernels to use and which devices to utilize |
||||
using extra parameters at graph compile (configuration) time. This |
||||
requirement has led to the following architecture: |
||||
|
||||
<!-- FIXME: Render from dot directly --> |
||||
|
||||
 |
||||
|
||||
There are three layers in this architecture: |
||||
* **API Layer** -- this is the top layer, which implements G-API |
||||
public interface, its building blocks and semantics. |
||||
When user constructs a pipeline with G-API, he interacts with this |
||||
layer directly, and the entities the user operates on (like cv::GMat |
||||
or cv::GComputation) are provided by this layer. |
||||
* **Graph Compiler Layer** -- this is the intermediate layer which |
||||
unrolls user computation into a graph and then applies a number of |
||||
transformations to it (e.g. optimizations). This layer is built atop |
||||
of [ADE Framework](@ref gapi_detail_ade). |
||||
* **Backends Layer** -- this is the lowest level layer, which lists a |
||||
number of _Backends_. In contrast with the above two layers, |
||||
backends are highly coupled with low-level platform details, with |
||||
every backend standing for every platform. A backend operates on a |
||||
processed graph (coming from the graph compiler) and executes this |
||||
graph optimally for a specific platform or device. |
||||
|
||||
# API layer {#gapi_api_layer} |
||||
|
||||
API layer is what user interacts with when defining and using a |
||||
pipeline (a Computation in G-API terms). API layer defines a set of |
||||
G-API _dynamic_ objects which can be used as inputs, outputs, and |
||||
intermediate data objects within a graph: |
||||
* cv::GMat |
||||
* cv::GScalar |
||||
* cv::GArray (template class) |
||||
|
||||
API layer specifies a list of Operations which are defined on these |
||||
data objects -- so called kernels. See G-API [core](@ref gapi_core) |
||||
and [imgproc](@ref gapi_imgproc) namespaces for details on which |
||||
operations G-API provides by default. |
||||
|
||||
G-API is not limited to these operations only -- users can define |
||||
their own kernels easily using a special macro G_TYPED_KERNEL(). |
||||
|
||||
API layer is also responsible for marshalling and storing operation |
||||
parameters on pipeline creation. In addition to the aforementioned |
||||
G-API dynamic objects, operations may also accept arbitrary |
||||
parameters (more on this [below](@ref gapi_detail_params)), so API |
||||
layer captures its values and stores internally upon the moment of |
||||
execution. |
||||
|
||||
Finally, cv::GComputation and cv::GCompiled are the remaining |
||||
important components of API layer. The former wraps a series of G-API |
||||
expressions into an object (graph), and the latter is a product of |
||||
graph _compilation_ (see [this chapter](@ref gapi_detail_compiler) for |
||||
details). |
||||
|
||||
# Graph compiler layer {#gapi_compiler} |
||||
|
||||
Every G-API computation is compiled before it executes. Compilation |
||||
process is triggered in two ways: |
||||
* _implicitly_, when cv::GComputation::apply() is used. In this case, |
||||
graph compilation is then immediately followed by execution. |
||||
* _explicitly_, when cv::GComputation::compile() is used. In this case, |
||||
a cv::GCompiled object is returned which then can be invoked as a |
||||
C++ functor. |
||||
|
||||
The first way is recommended for cases when input data format is not |
||||
known in advance -- e.g. when it comes from an arbitrary input file. |
||||
The second way is recommended for deployment (production) scenarios |
||||
where input data characteristics are usually predefined. |
||||
|
||||
Graph compilation process is built atop of ADE Framework. Initially, a |
||||
bipartite graph is generated from expressions captured by API layer. |
||||
This graph contains nodes of two types: _Data_ and _Operations_. Graph |
||||
always starts and ends with a Data node(s), with Operations nodes |
||||
in-between. Every Operation node has inputs and outputs, both are Data |
||||
nodes. |
||||
|
||||
After the initial graph is generated, it is actually processed by a |
||||
number of graph transformations, called _passes_. ADE Framework acts |
||||
as a compiler pass management engine, and passes are written |
||||
specifically for G-API. |
||||
|
||||
There are different passes which check graph validity, refine details |
||||
on operations and data, organize nodes into clusters ("Islands") based |
||||
on affinity or user-specified regioning[TBD], and more. Backends also |
||||
are able to inject backend-specific passes into the compilation |
||||
process, see more on this in the [dedicated chapter](@ref gapi_detail_meta). |
||||
|
||||
Result of graph compilation is a compiled object, represented by class |
||||
cv::GCompiled. A new cv::GCompiled object is always created regardless |
||||
if there was an explicit or implicit compilation request (see |
||||
above). Actual graph execution happens within cv::GCompiled and is |
||||
determined by backends which participated in the graph compilation. |
||||
|
||||
@sa cv::GComputation::apply(), cv::GComputation::compile(), cv::GCompiled |
||||
|
||||
# Backends layer {#gapi_backends} |
||||
|
||||
The above diagram lists two backends, _OpenCV_ and _Fluid_. _OpenCV_ |
||||
is so-called "reference backend", which implements G-API operations |
||||
using plain old OpenCV functions. This backend is useful for |
||||
prototyping on a familiar development system. _Fluid_ is a plugin for |
||||
cache-efficient execution on CPU -- it implements a different |
||||
execution policy and operates with its own, special kernels. Fluid |
||||
backend allows to achieve less memory footprint and better memory |
||||
locality when running on CPU. |
||||
|
||||
There may be more backends available, e.g. Halide, OpenCL, etc. -- |
||||
G-API provides an uniform internal API to develop backends so any |
||||
enthusiast or a company are free to scale G-API on a new platform or |
||||
accelerator. In terms of OpenCV infrastructure, every new backend is a |
||||
new distinct OpenCV module, which extends G-API when build as a part |
||||
of OpenCV. |
||||
|
||||
# Graph execution {#gapi_compiled} |
||||
|
||||
The way graph executed is defined by backends selected for |
||||
compilation. In fact, every backend builds its own execution script as |
||||
the final stage of graph compilation process, when an executable |
||||
(compiled) object is being generated. For example, in OpenCV backend, |
||||
this script is just a topologically-sorted sequence of OpenCV |
||||
functions to call; for Fluid backend, it is a similar thing -- a |
||||
topologically sorted list of _Agents_ processing lines of input on |
||||
every iteration. |
||||
|
||||
Graph execution is triggered in two ways: |
||||
* via cv::GComputation::apply(), with graph compiled in-place exactly |
||||
for the given input data; |
||||
* via cv::GCompiled::operator()(), when the graph has been precompiled. |
||||
|
||||
Both methods are polimorphic and take a variadic number of arguments, |
||||
with validity checks performed in runtime. If a number, shapes, and |
||||
formats of passed data objects differ from expected, a run-time |
||||
exception is thrown. G-API also provides _typed_ wrappers to move |
||||
these checks to the compile time -- see cv::GComputationT<>. |
||||
|
||||
G-API graph execution is declared stateless -- it means that a |
||||
compiled functor (cv::GCompiled) acts like a pure C++ function and |
||||
provides the same result for the same set of input arguments. |
||||
|
||||
Both execution methods take \f$N+M\f$ parameters, where \f$N\f$ is a |
||||
number of inputs, and \f$M\f$ is a number of outputs on which a |
||||
cv::GComputation is defined. Note that while G-API types (cv::GMat, |
||||
etc) are used in definition, the execution methods accept OpenCV's |
||||
traditional data types (like cv::Mat) which hold actual data -- see |
||||
table in [parameter marshalling](@#gapi_detail_params). |
||||
|
||||
@sa @ref gapi_impl, @ref gapi_kernel_api |
@ -0,0 +1,170 @@ |
||||
# Kernel API {#gapi_kernel_api} |
||||
|
||||
[TOC] |
||||
|
||||
# G-API Kernel API |
||||
|
||||
The core idea behind G-API is portability -- a pipeline built with |
||||
G-API must be portable (or at least able to be portable). It means |
||||
that either it works out-of-the box when compiled for new platform, |
||||
_or_ G-API provides necessary tools to make it running there, with |
||||
little-to-no changes in the algorithm itself. |
||||
|
||||
This idea can be achieved by separating kernel interface from its |
||||
implementation. Once a pipeline is built using kernel interfaces, it |
||||
becomes implementation-neutral -- the implementation details |
||||
(i.e. which kernels to use) are passed on a separate stage (graph |
||||
compilation). |
||||
|
||||
Kernel-implementation hierarchy may look like: |
||||
|
||||
 |
||||
|
||||
A pipeline itself then can be expressed only in terms of `A`, `B`, and |
||||
so on, and choosing which implementation to use in execution becomes |
||||
an external parameter. |
||||
|
||||
# Defining a kernel {#gapi_defining_kernel} |
||||
|
||||
G-API provides a macro to define a new kernel interface -- |
||||
G_TYPED_KERNEL(): |
||||
|
||||
@snippet modules/gapi/samples/kernel_api_snippets.cpp filter2d_api |
||||
|
||||
This macro is a shortcut to a new type definition. It takes three |
||||
arguments to register a new type, and requires type body to be present |
||||
(see [below](@ref gapi_kernel_supp_info)). The macro arguments are: |
||||
1. Kernel interface name -- also serves as a name of new type defined |
||||
with this macro; |
||||
2. Kernel signature -- an `std::function<>`-like signature which defines |
||||
API of the kernel; |
||||
3. Kernel's unique name -- used to identify kernel when its type |
||||
informattion is stripped within the system. |
||||
|
||||
Kernel declaration may be seen as function declaration -- in both cases |
||||
a new entity must be used then according to the way it was defined. |
||||
|
||||
Kernel signature defines kernel's usage syntax -- which parameters |
||||
it takes during graph construction. Implementations can also use this |
||||
signature to derive it into backend-specific callback signatures (see |
||||
next chapter). |
||||
|
||||
Kernel may accept values of any type, and G-API _dynamic_ types are |
||||
handled in a special way. All other types are opaque to G-API and |
||||
passed to kernel in `outMeta()` or in execution callbacks as-is. |
||||
|
||||
Kernel's return value can _only_ be of G-API dynamic type -- cv::GMat, |
||||
cv::GScalar, or cv::GArray<T>. If an operation has more than one output, |
||||
it should be wrapped into an `std::tuple<>` (which can contain only |
||||
mentioned G-API types). Arbitrary-output-number operations are not |
||||
supported. |
||||
|
||||
Once a kernel is defined, it can be used in pipelines with special, |
||||
G-API-supplied method "::on()". This method has the same signature as |
||||
defined in kernel, so this code: |
||||
|
||||
@snippet modules/gapi/samples/kernel_api_snippets.cpp filter2d_on |
||||
|
||||
is a perfectly legal construction. This example has some verbosity, |
||||
though, so usually a kernel declaration comes with a C++ function |
||||
wrapper ("factory method") which enables optional parameters, more |
||||
compact syntax, Doxygen comments, etc: |
||||
|
||||
@snippet modules/gapi/samples/kernel_api_snippets.cpp filter2d_wrap |
||||
|
||||
so now it can be used like: |
||||
|
||||
@snippet modules/gapi/samples/kernel_api_snippets.cpp filter2d_wrap_call |
||||
|
||||
# Extra information {#gapi_kernel_supp_info} |
||||
|
||||
In the current version, kernel declaration body (everything within the |
||||
curly braces) must contain a static function `outMeta()`. This function |
||||
establishes a functional dependency between operation's input and |
||||
output metadata. |
||||
|
||||
_Metadata_ is an information about data kernel operates on. Since |
||||
non-G-API types are opaque to G-API, G-API cares only about `G*` data |
||||
descriptors (i.e. dimensions and format of cv::GMat, etc). |
||||
|
||||
`outMeta()` is also an example of how kernel's signature can be |
||||
transformed into a derived callback -- note that in this example, |
||||
`outMeta()` signature exactly follows the kernel signature (defined |
||||
within the macro) but is different -- where kernel expects cv::GMat, |
||||
`outMeta()` takes and returns cv::GMatDesc (a G-API structure metadata |
||||
for cv::GMat). |
||||
|
||||
The point of `outMeta()` is to propagate metadata information within |
||||
computation from inputs to outputs and infer metadata of internal |
||||
(intermediate, temporary) data objects. This information is required |
||||
for further pipeline optimizations, memory allocation, and other |
||||
operations done by G-API framework during graph compilation. |
||||
|
||||
<!-- TODO add examples --> |
||||
|
||||
# Implementing a kernel {#gapi_kernel_implementing} |
||||
|
||||
Once a kernel is declared, its interface can be used to implement |
||||
versions of this kernel in different backends. This concept is |
||||
naturally projected from object-oriented programming |
||||
"Interface/Implementation" idiom: an interface can be implemented |
||||
multiple times, and different implementations of a kernel should be |
||||
substitutable with each other without breaking the algorithm |
||||
(pipeline) logic (Liskov Substitution Principle). |
||||
|
||||
Every backend defines its own way to implement a kernel interface. |
||||
This way is regular, though -- whatever plugin is, its kernel |
||||
implementation must be "derived" from a kernel interface type. |
||||
|
||||
Kernel implementation are then organized into _kernel |
||||
packages_. Kernel packages are passed to cv::GComputation::compile() |
||||
as compile arguments, with some hints to G-API on how to select proper |
||||
kernels (see more on this in "Heterogeneity"[TBD]). |
||||
|
||||
For example, the aforementioned `Filter2D` is implemented in |
||||
"reference" CPU (OpenCV) plugin this way (*NOTE* -- this is a |
||||
simplified form with improper border handling): |
||||
|
||||
@snippet modules/gapi/samples/kernel_api_snippets.cpp filter2d_ocv |
||||
|
||||
Note how CPU (OpenCV) plugin has transformed the original kernel |
||||
signature: |
||||
- Input cv::GMat has been substituted with cv::Mat, holding actual input |
||||
data for the underlying OpenCV function call; |
||||
- Output cv::GMat has been transformed into extra output parameter, thus |
||||
`GCPUFilter2D::run()` takes one argument more than the original |
||||
kernel signature. |
||||
|
||||
The basic intuition for kernel developer here is _not to care_ where |
||||
that cv::Mat objects come from instead of the original cv::GMat -- and |
||||
just follow the signature conventions defined by the plugin. G-API |
||||
will call this method during execution and supply all the necessary |
||||
information (and forward the original opaque data as-is). |
||||
|
||||
# Compound kernels |
||||
|
||||
Sometimes kernel is a single thing only on API level. It is convenient |
||||
for users, but on a particular implementation side it would be better to |
||||
have multiple kernels (a subgraph) doing the thing instead. An example |
||||
is goodFeaturesToTrack() -- while in OpenCV backend it may remain a |
||||
single kernel, with Fluid it becomes compound -- Fluid can handle Harris |
||||
response calculation but can't do sparse non-maxima suppression and |
||||
point extraction to an STL vector: |
||||
|
||||
<!-- PIC --> |
||||
|
||||
A compound kernel _implementation_ can be defined using a generic |
||||
macro GAPI_COMPOUND_KERNEL(): |
||||
|
||||
@snippet modules/gapi/samples/kernel_api_snippets.cpp compound |
||||
|
||||
<!-- TODO: ADD on how Compound kernels may simplify dispatching --> |
||||
<!-- TODO: Add details on when expand() is called! --> |
||||
|
||||
It is important to distinguish a compound kernel from G-API high-order |
||||
function, i.e. a C++ function which looks like a kernel but in fact |
||||
generates a subgraph. The core difference is that a compound kernel is |
||||
an _implementation detail_ and a kernel implementation may be either |
||||
compound or not (depending on backend capabilities), while a |
||||
high-order function is a "macro" in terms of G-API and so cannot act as |
||||
an interface which then needs to be implemented by a backend. |
@ -0,0 +1,27 @@ |
||||
# Implementation details {#gapi_impl} |
||||
|
||||
[TOC] |
||||
|
||||
# G-API Implementation details {#gapi_impl_header} |
||||
|
||||
## Api layer details {#gapi_detail_api} |
||||
|
||||
### Expression unrolling {#gapi_detail_expr} |
||||
|
||||
### Parameter marshalling {#gapi_detail_params} |
||||
|
||||
### Operations representation {#gapi_detail_operations} |
||||
|
||||
## Graph compiler details {#gapi_detail_compiler} |
||||
|
||||
### ADE basics {#gapi_detail_ade} |
||||
|
||||
### Graph model representation {#gapi_detail_gmodel} |
||||
|
||||
### G-API metadata and passes {#gapi_detail_meta} |
||||
|
||||
## Backends details {#gapi_detail_backends} |
||||
|
||||
### Backend scope of work {#gapi_backend_scope} |
||||
|
||||
### Graph transformation {#gapi_backend_pass} |
@ -0,0 +1,17 @@ |
||||
digraph { |
||||
rankdir=BT; |
||||
node [shape=record]; |
||||
|
||||
ki_a [label="{<f0> interface\nA}"]; |
||||
ki_b [label="{<f0> interface\nB}"]; |
||||
|
||||
{rank=same; ki_a ki_b}; |
||||
|
||||
"CPU::A" -> ki_a [dir="forward"]; |
||||
"OpenCL::A" -> ki_a [dir="forward"]; |
||||
"Halide::A" -> ki_a [dir="forward"]; |
||||
|
||||
"CPU::B" -> ki_b [dir="forward"]; |
||||
"OpenCL::B" -> ki_b [dir="forward"]; |
||||
"Halide::B" -> ki_b [dir="forward"]; |
||||
} |
@ -1,3 +0,0 @@ |
||||
# Graph API {#gapi} |
||||
|
||||
Introduction to G-API (WIP). |
After Width: | Height: | Size: 64 KiB |
After Width: | Height: | Size: 75 KiB |
After Width: | Height: | Size: 8.6 KiB |
@ -0,0 +1,34 @@ |
||||
#include <opencv2/videoio.hpp> |
||||
#include <opencv2/highgui.hpp> |
||||
#include <opencv2/gapi.hpp> |
||||
#include <opencv2/gapi/core.hpp> |
||||
#include <opencv2/gapi/imgproc.hpp> |
||||
|
||||
int main(int argc, char *argv[]) |
||||
{ |
||||
cv::VideoCapture cap; |
||||
if (argc > 1) cap.open(argv[1]); |
||||
else cap.open(0); |
||||
CV_Assert(cap.isOpened()); |
||||
|
||||
cv::GMat in; |
||||
cv::GMat vga = cv::gapi::resize(in, cv::Size(), 0.5, 0.5); |
||||
cv::GMat gray = cv::gapi::BGR2Gray(vga); |
||||
cv::GMat blurred = cv::gapi::blur(gray, cv::Size(5,5)); |
||||
cv::GMat edges = cv::gapi::Canny(blurred, 32, 128, 3); |
||||
cv::GMat b,g,r; |
||||
std::tie(b,g,r) = cv::gapi::split3(vga); |
||||
cv::GMat out = cv::gapi::merge3(b, g | edges, r); |
||||
cv::GComputation ac(in, out); |
||||
|
||||
cv::Mat input_frame; |
||||
cv::Mat output_frame; |
||||
CV_Assert(cap.read(input_frame)); |
||||
do |
||||
{ |
||||
ac.apply(input_frame, output_frame); |
||||
cv::imshow("output", output_frame); |
||||
} while (cap.read(input_frame) && cv::waitKey(30) < 0); |
||||
|
||||
return 0; |
||||
} |
@ -0,0 +1,157 @@ |
||||
// [filter2d_api]
|
||||
#include <opencv2/gapi.hpp> |
||||
|
||||
G_TYPED_KERNEL(GFilter2D, |
||||
<cv::GMat(cv::GMat,int,cv::Mat,cv::Point,double,int,cv::Scalar)>, |
||||
"org.opencv.imgproc.filters.filter2D") |
||||
{ |
||||
static cv::GMatDesc // outMeta's return value type
|
||||
outMeta(cv::GMatDesc in , // descriptor of input GMat
|
||||
int ddepth , // depth parameter
|
||||
cv::Mat /* coeffs */, // (unused)
|
||||
cv::Point /* anchor */, // (unused)
|
||||
double /* scale */, // (unused)
|
||||
int /* border */, // (unused)
|
||||
cv::Scalar /* bvalue */ ) // (unused)
|
||||
{ |
||||
return in.withDepth(ddepth); |
||||
} |
||||
}; |
||||
// [filter2d_api]
|
||||
|
||||
cv::GMat filter2D(cv::GMat , |
||||
int , |
||||
cv::Mat , |
||||
cv::Point , |
||||
double , |
||||
int , |
||||
cv::Scalar); |
||||
|
||||
// [filter2d_wrap]
|
||||
cv::GMat filter2D(cv::GMat in, |
||||
int ddepth, |
||||
cv::Mat k, |
||||
cv::Point anchor = cv::Point(-1,-1), |
||||
double scale = 0., |
||||
int border = cv::BORDER_DEFAULT, |
||||
cv::Scalar bval = cv::Scalar(0)) |
||||
{ |
||||
return GFilter2D::on(in, ddepth, k, anchor, scale, border, bval); |
||||
} |
||||
// [filter2d_wrap]
|
||||
|
||||
// [compound]
|
||||
#include <opencv2/gapi/gcompoundkernel.hpp> // GAPI_COMPOUND_KERNEL() |
||||
|
||||
using PointArray2f = cv::GArray<cv::Point2f>; |
||||
|
||||
G_TYPED_KERNEL(HarrisCorners, |
||||
<PointArray2f(cv::GMat,int,double,double,int,double)>, |
||||
"org.opencv.imgproc.harris_corner") |
||||
{ |
||||
static cv::GArrayDesc outMeta(const cv::GMatDesc &, |
||||
int, |
||||
double, |
||||
double, |
||||
int, |
||||
double) |
||||
{ |
||||
// No special metadata for arrays in G-API (yet)
|
||||
return cv::empty_array_desc(); |
||||
} |
||||
}; |
||||
|
||||
// Define Fluid-backend-local kernels which form GoodFeatures
|
||||
G_TYPED_KERNEL(HarrisResponse, |
||||
<cv::GMat(cv::GMat,double,int,double)>, |
||||
"org.opencv.fluid.harris_response") |
||||
{ |
||||
static cv::GMatDesc outMeta(const cv::GMatDesc &in, |
||||
double, |
||||
int, |
||||
double) |
||||
{ |
||||
return in.withType(CV_32F, 1); |
||||
} |
||||
}; |
||||
|
||||
G_TYPED_KERNEL(ArrayNMS, |
||||
<PointArray2f(cv::GMat,int,double)>, |
||||
"org.opencv.cpu.nms_array") |
||||
{ |
||||
static cv::GArrayDesc outMeta(const cv::GMatDesc &, |
||||
int, |
||||
double) |
||||
{ |
||||
return cv::empty_array_desc(); |
||||
} |
||||
}; |
||||
|
||||
GAPI_COMPOUND_KERNEL(GFluidHarrisCorners, HarrisCorners) |
||||
{ |
||||
static PointArray2f |
||||
expand(cv::GMat in, |
||||
int maxCorners, |
||||
double quality, |
||||
double minDist, |
||||
int blockSize, |
||||
double k) |
||||
{ |
||||
cv::GMat response = HarrisResponse::on(in, quality, blockSize, k); |
||||
return ArrayNMS::on(response, maxCorners, minDist); |
||||
} |
||||
}; |
||||
|
||||
// Then implement HarrisResponse as Fluid kernel and NMSresponse
|
||||
// as a generic (OpenCV) kernel
|
||||
// [compound]
|
||||
|
||||
// [filter2d_ocv]
|
||||
#include <opencv2/gapi/cpu/gcpukernel.hpp> // GAPI_OCV_KERNEL() |
||||
#include <opencv2/imgproc.hpp> // cv::filter2D() |
||||
|
||||
GAPI_OCV_KERNEL(GCPUFilter2D, GFilter2D) |
||||
{ |
||||
static void |
||||
run(const cv::Mat &in, // in - derived from GMat
|
||||
const int ddepth, // opaque (passed as-is)
|
||||
const cv::Mat &k, // opaque (passed as-is)
|
||||
const cv::Point &anchor, // opaque (passed as-is)
|
||||
const double delta, // opaque (passed as-is)
|
||||
const int border, // opaque (passed as-is)
|
||||
const cv::Scalar &, // opaque (passed as-is)
|
||||
cv::Mat &out) // out - derived from GMat (retval)
|
||||
{ |
||||
cv::filter2D(in, out, ddepth, k, anchor, delta, border); |
||||
} |
||||
}; |
||||
// [filter2d_ocv]
|
||||
|
||||
int main(int, char *[]) |
||||
{ |
||||
std::cout << "This sample is non-complete. It is used as code snippents in documentation." << std::endl; |
||||
|
||||
cv::Mat conv_kernel_mat; |
||||
|
||||
{ |
||||
// [filter2d_on]
|
||||
cv::GMat in; |
||||
cv::GMat out = GFilter2D::on(/* GMat */ in, |
||||
/* int */ -1, |
||||
/* Mat */ conv_kernel_mat, |
||||
/* Point */ cv::Point(-1,-1), |
||||
/* double */ 0., |
||||
/* int */ cv::BORDER_DEFAULT, |
||||
/* Scalar */ cv::Scalar(0)); |
||||
// [filter2d_on]
|
||||
} |
||||
|
||||
{ |
||||
// [filter2d_wrap_call]
|
||||
cv::GMat in; |
||||
cv::GMat out = filter2D(in, -1, conv_kernel_mat); |
||||
// [filter2d_wrap_call]
|
||||
} |
||||
|
||||
return 0; |
||||
} |
Loading…
Reference in new issue