G-API: Introduce a Queue Source #24178
- Added a new IStreamSource class: in fact, a wrapper over a concurrent queue;
- Added minimal example on how it can be used;
- Extended IStreamSource with optional "halt" interface to break the blocking calls in the emitter threads when required to stop.
- Introduced a QueueInput class which allows to pass the whole graph's input vector at once. In fact it is a thin wrapper atop of individual Queue Sources.
There is a hidden trap found with our type system as described in https://github.com/orgs/g-api-org/discussions/2
While it works even in this form, it should be addressed somewhere in the 5.0 timeframe.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
G-API: Fix incorrect OpaqueKind for Kernel outputs #23843
### Pull Request Readiness Checklist
#### Overview
The PR is going to fix several problems:
1. Major: `GKernel` doesn't hold `kind` for its outputs. Since `GModelBuilder` traverse graph from outputs to inputs once it reaches any output of the operation it will use its `kind` to create `Data` meta for all operation outputs. Since it essential for `python` to know `GTypeInfo` (which is `shape` and `kind`) it will be confused.
Consider this operation:
```
@cv.gapi.op('custom.square_mean', in_types=[cv.GArray.Int], out_types=[cv.GOpaque.Float, cv.GArray.Int])
class GSquareMean:
@staticmethod
def outMeta(desc):
return cv.empty_gopaque_desc(), cv.empty_array_desc()
```
Even though `GOpaque` is `Float`, corresponding metadata might have `Int` kind because it might be taken from `cv.GArray.Int`
so it will be a problem if one of the outputs of these operation is graph output because python will cast it to the wrong type based on `Data` meta.
2. Minor: Some of the OpenVINO `IR`'s doesn't any layout information for input. It's usually true only for `IRv10` but since `OpenVINO 2.0` need this information to correctly configure resize we need to put default layout if there no such assigned in `ov::Model`.
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [ ] I agree to contribute to the project under Apache 2 License.
- [ ] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
G-API: Implement InferROI, InferList, InferList2 for OpenVINO backend #23799
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [ ] I agree to contribute to the project under Apache 2 License.
- [ ] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
G-API: Align IE Backend with the latest OpenVINO version #23796
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [ ] I agree to contribute to the project under Apache 2 License.
- [ ] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
G-API: Expose explicit preprocessing for IE Backend #23786
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [ ] I agree to contribute to the project under Apache 2 License.
- [ ] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
[G-API] Implement OpenVINO 2.0 backend #23595
### Pull Request Readiness Checklist
Implemented basic functionality for `OpenVINO` 2.0 G-API backend.
#### Overview
- [x] Implement `Infer` kernel with some of essential configurable parameters + IR/Blob models format support.
- [ ] Implement the rest of kernels: `InferList`, `InferROI`, `Infer2` + other configurable params (e.g reshape)
- [x] Asyncrhonous execution support
- [ ] Remote context support
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
G-API: Integration branch for ONNX & Python-related changes #23597
# Changes overview
## 1. Expose ONNX backend's Normalization and Mean-value parameters in Python
* Since Python G-API bindings rely on `Generic` infer to express Inference, the `Generic` specialization of `onnx::Params` was extended with new methods to control normalization (`/255`) and mean-value; these methods were exposed in the Python bindings
* Found some questionable parts in the existing API which I'd like to review/discuss (see comments)
UPD:
1. Thanks to @TolyaTalamanov normalization inconsistencies have been identified with `squeezenet1.0-9` ONNX model itself; tests using these model were updated to DISABLE normalization and NOT using mean/value.
2. Questionable parts were removed and tests still pass.
### Details (taken from @TolyaTalamanov's comment):
`squeezenet1.0.*onnx` - doesn't require scaling to [0,1] and mean/std because the weights of the first convolution already scaled. ONNX documentation is broken. So the correct approach to use this models is:
1. ONNX: apply preprocessing from the documentation: https://github.com/onnx/models/blob/main/vision/classification/imagenet_preprocess.py#L8-L44 but without normalization step:
```
# DON'T DO IT:
# mean_vec = np.array([0.485, 0.456, 0.406])
# stddev_vec = np.array([0.229, 0.224, 0.225])
# norm_img_data = np.zeros(img_data.shape).astype('float32')
# for i in range(img_data.shape[0]):
# norm_img_data[i,:,:] = (img_data[i,:,:]/255 - mean_vec[i]) / stddev_vec[i]
# # add batch channel
# norm_img_data = norm_img_data.reshape(1, 3, 224, 224).astype('float32')
# return norm_img_data
# INSTEAD
return img_data.reshape(1, 3, 224, 224)
```
2. G-API: Convert image from BGR to RGB and then pass to `apply` as-is with configuring parameters:
```
net = cv.gapi.onnx.params('squeezenet', model_filename)
net.cfgNormalize('data_0', False)
```
**Note**: Results might be difference because `G-API` doesn't apply central crop but just do resize to model resolution.
---
`squeezenet1.1.*onnx` - requires scaling to [0,1] and mean/std - onnx documentation is correct.
1. ONNX: apply preprocessing from the documentation: https://github.com/onnx/models/blob/main/vision/classification/imagenet_preprocess.py#L8-L44
2. G-API: Convert image from BGR to RGB and then pass to `apply` as-is with configuring parameters:
```
net = cv.gapi.onnx.params('squeezenet', model_filename)
net.cfgNormalize('data_0', True) // default
net.cfgMeanStd('data_0', [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
```
**Note**: Results might be difference because `G-API` doesn't apply central crop but just do resize to model resolution.
## 2. Expose Fluid & kernel package-related functionality in Python
* `cv::gapi::combine()`
* `cv::GKernelPackage::size()` (mainly for testing purposes)
* `cv::gapi::imgproc::fluid::kernels()`
Added a test for the above.
## 3. Fixed issues with Python stateful kernel handling
Fixed error message when `outMeta()` of custom python operation fails.
## 4. Fixed various issues in Python tests
1. `test_gapi_streaming.py` - fixed behavior of Desync test to avoid sporadic issues
2. `test_gapi_infer_onnx.py` - fixed model lookup (it was still using the ONNX Zoo layout but was NOT using the proper env var we use to point to one).
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
G-API: replace GAPI_Assert() with 'false' and '0' to GAPI_Error()
* gapi: GAPI_Error() macro
* gapi: replace GAPI_Assert() with 'false' and '0' to GAPI_Error()
* build: eliminate 'unreachable code' after CV_Error() (MSVC 2015)
* build: eliminate 'unreachable code' warning for MSVS 2015/2017
- observed in constructors stubs with throwing exception
Minor refactoring
Partially address review comments
Move DX-related stuff from the sample to a default source
Simplify the default OneVPL config
Address minor review comments
Add class for the default VPL source
WIP: Add initial stub for tests with description
Removing default vpl source and minor refactoring
Refactor default files
Fix build and application crash
Address review comments
Add test on VPL + OCL interaction compared to CPU behavior
Fix test
Fixed handling of new stream, especially for stateful OCV kernels
* Fixed handling of new stream, especially for stateful OCV kernels
* Removed duplication from StateInitOnce tests
* Addressed review comments for PR #21731
- Fixed explanation comments
- Expanded test for stateful OCV kernels in Regular mode
* Addressed review comments for PR #21731
- Moved notification about new stream to the constructor
- Added test on state reset for Regular mode
* Addresed review comments
* Addressed review comments
Co-authored-by: Ruslan Garnov <ruslan.garnov@intel.com>
[G-API] Handle exceptions in streaming executor
* Handle exceptions in streaming executor
* Rethrow exception in non-streaming executor
* Clean up
* Put more tests
* Handle exceptions in IE backend
* Handle exception in IE callbacks
* Handle exception in GExecutor
* Handle all exceptions in IE backend
* Not only (std::exception& e)
* Fix comments to review
* Handle input exception in generic way
* Fix comment
* Clean up
* Apply review comments
* Put more comments
* Fix alignment
* Move test outside of HAVE_NGRAPH
* Fix compilation
G-API: oneVPL DX11 inference
* Draft GPU infer
* Fix incorrect subresource_id for array of textures
* Fix for TheOneSurface in different Frames
* Turn on VPP param configuration
* Add cropIn params
* Remove infer sync sample
* Remove comments
* Remove DX11AllocResource extra init
* Add condition for NV12 processing in giebackend
* Add VPP frames pool param configurable
* -M Remove extra WARN & INFOs, Fix custom MAC
* Remove global vars from example, Fix some comments, Disable blobParam due to OV issue
* Conflict resolving
* Revert back pointer cast for cv::any
GAPI: Add OAK backend
* Initial tests and cmake integration
* Add a public header and change tests
* Stub initial empty template for the OAK backend
* WIP
* WIP
* WIP
* WIP
* Runtime dai hang debug
* Refactoring
* Fix hang and debug frame data
* Fix frame size
* Fix data size issue
* Move test code to sample
* tmp refactoring
* WIP: Code refactoring except for the backend
* WIP: Add non-camera sample
* Fix samples
* Backend refactoring wip
* Backend rework wip
* Backend rework wip
* Remove mat encoder
* Fix namespace
* Minor backend fixes
* Fix hetero sample and refactor backend
* Change linking logic in the backend
* Fix oak sample
* Fix working with ins/outs in OAK island
* Trying to fix nv12 problem
* Make both samples work
* Small refactoring
* Remove meta args
* WIP refactoring kernel API
* Change in/out args API for kernels
* Fix build
* Fix cmake warning
* Partially address review comments
* Partially address review comments
* Address remaining comments
* Add memory ownership
* Change pointer-to-pointer to reference-to-pointer
* Remove unnecessary reference wrappers
* Apply review comments
* Check that graph contains only one OAK island
* Minor refactoring
* Address review comments
G-API: oneVPL merge DX11 acceleration
* Merge DX11 initial
* Fold conditions row in MACRO in utils
* Inject DeviceSelector
* Turn on DeviceSelector in DX11
* Change sharedLock logic & Move FMT checking in FrameAdapter c-tor
* Move out NumSuggestFrame to configure params
* Drain file source fix
* Fix compilation
* Force zero initializetion of SharedLock
* Fix some compiler warnings
* Fix integer comparison warnings
* Fix integers in sample
* Integrate Demux
* Fix compilation
* Add predefined names for some CfgParam
* Trigger CI
* Fix MultithreadCtx bug, Add Dx11 GetBlobParam(), Get rif of ATL CComPtr
* Fix UT: remove unit test with deprecated video from opencv_extra
* Add creators for most usable CfgParam
* Eliminate some warnings
* Fix warning in GAPI_Assert
* Apply comments
* Add VPL wrapped header with MSVC pragma to get rid of global warning masking
[G-API] Fix issue of getting 1D Mat out of RMat::View
* Fix issue of getting 1D Mat out of RMat::View
- added test
- fixed for standalone too (removed Assert(dims.empty()))
* Fixed asVeiw() function for standalone
* Put more detailed comment
G-API: Removing G-API test code that is a reflection of ts module
* gapi: don't hijack testing infrastructure
* Removed initDataPath functionality (ts module exists)
* Removed false for ocv_extra data from findDataFile
Co-authored-by: Alexander Alekhin <alexander.a.alekhin@gmail.com>
Dmitry Matveev
Dmitry Kurtaev
TolyaTalamanov
Dmitry Matveev
Anna Khakimova
Maksim Shabunin
Alexander Alekhin
Alexander Alekhin
Alexey Smirnov
JopKnoppers
SergeyIvanov87
dbudnikov
Kumataro
Sergey
luzpaz
Anastasiya(Asya) Pronina
Sergey Ivanov
Ruslan Garnov
Dmitry Budnikov
Maxim Pashchenkov
Trutnev Aleksei
Orest Chura