VIT track(gsoc realtime object tracking model) #24201
Vit tracker(vision transformer tracker) is a much better model for real-time object tracking. Vit tracker can achieve speeds exceeding nanotrack by 20% in single-threaded mode with ARM chip, and the advantage becomes even more pronounced in multi-threaded mode. In addition, on the dataset, vit tracker demonstrates better performance compared to nanotrack. Moreover, vit trackerprovides confidence values during the tracking process, which can be used to determine if the tracking is currently lost.
opencv_zoo: https://github.com/opencv/opencv_zoo/pull/194
opencv_extra: [https://github.com/opencv/opencv_extra/pull/1088](https://github.com/opencv/opencv_extra/pull/1088)
# Performance comparison is as follows:
NOTE: The speed below is tested by **onnxruntime** because opencv has poor support for the transformer architecture for now.
ONNX speed test on ARM platform(apple M2)(ms):
| thread nums | 1| 2| 3| 4|
|--------|--------|--------|--------|--------|
| nanotrack| 5.25| 4.86| 4.72| 4.49|
| vit tracker| 4.18| 2.41| 1.97| **1.46 (3X)**|
ONNX speed test on x86 platform(intel i3 10105)(ms):
| thread nums | 1| 2| 3| 4|
|--------|--------|--------|--------|--------|
| nanotrack|3.20|2.75|2.46|2.55|
| vit tracker|3.84|2.37|2.10|2.01|
opencv speed test on x86 platform(intel i3 10105)(ms):
| thread nums | 1| 2| 3| 4|
|--------|--------|--------|--------|--------|
| vit tracker|31.3|31.4|31.4|31.4|
preformance test on lasot dataset(AUC is the most important data. Higher AUC means better tracker):
|LASOT | AUC| P| Pnorm|
|--------|--------|--------|--------|
| nanotrack| 46.8| 45.0| 43.3|
| vit tracker| 48.6| 44.8| 54.7|
[https://youtu.be/MJiPnu1ZQRI](https://youtu.be/MJiPnu1ZQRI)
In target tracking tasks, the score is an important indicator that can indicate whether the current target is lost. In the video, vit tracker can track the target and display the current score in the upper left corner of the video. When the target is lost, the score drops significantly. While nanotrack will only return 0.9 score in any situation, so that we cannot determine whether the target is lost.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
[teset data in opencv_extra](https://github.com/opencv/opencv_extra/pull/1016)
NanoTrack is an extremely lightweight and fast object-tracking model.
The total size is **1.1 MB**.
And the FPS on M1 chip is **150**, on Raspberry Pi 4 is about **30**. (Float32 CPU only)
With this model, many users can run object tracking on the edge device.
The author of NanoTrack is @HonglinChu.
The original repo is https://github.com/HonglinChu/NanoTrack.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
there is a recent change, how `std::vector<int>` is wrapped in python,
it used to be a 2d array (requirig that weird `[0]` indexing), now it is only 1d
Avoid `SyntaxWarning` on Python >= 3.8
```
>>> "convolutional" == "convolutional"
True
>>> "convolutional" is "convolutional"
<stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?
True
```
Related to #21121
[GSoC] OpenCV.js: Accelerate OpenCV.js DNN via WebNN
* Add WebNN backend for OpenCV DNN Module
Update dnn.cpp
Update dnn.cpp
Update dnn.cpp
Update dnn.cpp
Add WebNN head files into OpenCV 3rd partiy files
Create webnn.hpp
update cmake
Complete README and add OpenCVDetectWebNN.cmake file
add webnn.cpp
Modify webnn.cpp
Can successfully compile the codes for creating a MLContext
Update webnn.cpp
Update README.md
Update README.md
Update README.md
Update README.md
Update cmake files and
update README.md
Update OpenCVDetectWebNN.cmake and README.md
Update OpenCVDetectWebNN.cmake
Fix OpenCVDetectWebNN.cmake and update README.md
Add source webnn_cpp.cpp and libary libwebnn_proc.so
Update dnn.cpp
Update dnn.cpp
Update dnn.cpp
Update dnn.cpp
update dnn.cpp
update op_webnn
update op_webnn
Update op_webnn.hpp
update op_webnn.cpp & hpp
Update op_webnn.hpp
Update op_webnn
update the skeleton
Update op_webnn.cpp
Update op_webnn
Update op_webnn.cpp
Update op_webnn.cpp
Update op_webnn.hpp
update op_webnn
update op_webnn
Solved the problems of released variables.
Fixed the bugs in op_webnn.cpp
Implement op_webnn
Implement Relu by WebNN API
Update dnn.cpp for better test
Update elementwise_layers.cpp
Implement ReLU6
Update elementwise_layers.cpp
Implement SoftMax using WebNN API
Implement Reshape by WebNN API
Implement PermuteLayer by WebNN API
Implement PoolingLayer using WebNN API
Update pooling_layer.cpp
Update pooling_layer.cpp
Update pooling_layer.cpp
Update pooling_layer.cpp
Update pooling_layer.cpp
Update pooling_layer.cpp
Implement poolingLayer by WebNN API and add more detailed logs
Update dnn.cpp
Update dnn.cpp
Remove redundant codes and add more logs for poolingLayer
Add more logs in the pooling layer implementation
Fix the indent issue and resolve the compiling issue
Fix the build problems
Fix the build issue
FIx the build issue
Update dnn.cpp
Update dnn.cpp
* Fix the build issue
* Implement BatchNorm Layer by WebNN API
* Update convolution_layer.cpp
This is a temporary file for Conv2d layer implementation
* Integrate some general functions into op_webnn.cpp&hpp
* Update const_layer.cpp
* Update convolution_layer.cpp
Still have some bugs that should be fixed.
* Update conv2d layer and fc layer
still have some problems to be fixed.
* update constLayer, conv layer, fc layer
There are still some bugs to be fixed.
* Fix the build issue
* Update concat_layer.cpp
Still have some bugs to be fixed.
* Update conv2d layer, fully connected layer and const layer
* Update convolution_layer.cpp
* Add OpenCV.js DNN module WebNN Backend (both using webnn-polyfill and electron)
* Delete bib19450.aux
* Add WebNN backend for OpenCV DNN Module
Update dnn.cpp
Update dnn.cpp
Update dnn.cpp
Update dnn.cpp
Add WebNN head files into OpenCV 3rd partiy files
Create webnn.hpp
update cmake
Complete README and add OpenCVDetectWebNN.cmake file
add webnn.cpp
Modify webnn.cpp
Can successfully compile the codes for creating a MLContext
Update webnn.cpp
Update README.md
Update README.md
Update README.md
Update README.md
Update cmake files and
update README.md
Update OpenCVDetectWebNN.cmake and README.md
Update OpenCVDetectWebNN.cmake
Fix OpenCVDetectWebNN.cmake and update README.md
Add source webnn_cpp.cpp and libary libwebnn_proc.so
Update dnn.cpp
Update dnn.cpp
Update dnn.cpp
Update dnn.cpp
update dnn.cpp
update op_webnn
update op_webnn
Update op_webnn.hpp
update op_webnn.cpp & hpp
Update op_webnn.hpp
Update op_webnn
update the skeleton
Update op_webnn.cpp
Update op_webnn
Update op_webnn.cpp
Update op_webnn.cpp
Update op_webnn.hpp
update op_webnn
update op_webnn
Solved the problems of released variables.
Fixed the bugs in op_webnn.cpp
Implement op_webnn
Implement Relu by WebNN API
Update dnn.cpp for better test
Update elementwise_layers.cpp
Implement ReLU6
Update elementwise_layers.cpp
Implement SoftMax using WebNN API
Implement Reshape by WebNN API
Implement PermuteLayer by WebNN API
Implement PoolingLayer using WebNN API
Update pooling_layer.cpp
Update pooling_layer.cpp
Update pooling_layer.cpp
Update pooling_layer.cpp
Update pooling_layer.cpp
Update pooling_layer.cpp
Implement poolingLayer by WebNN API and add more detailed logs
Update dnn.cpp
Update dnn.cpp
Remove redundant codes and add more logs for poolingLayer
Add more logs in the pooling layer implementation
Fix the indent issue and resolve the compiling issue
Fix the build problems
Fix the build issue
FIx the build issue
Update dnn.cpp
Update dnn.cpp
* Fix the build issue
* Implement BatchNorm Layer by WebNN API
* Update convolution_layer.cpp
This is a temporary file for Conv2d layer implementation
* Integrate some general functions into op_webnn.cpp&hpp
* Update const_layer.cpp
* Update convolution_layer.cpp
Still have some bugs that should be fixed.
* Update conv2d layer and fc layer
still have some problems to be fixed.
* update constLayer, conv layer, fc layer
There are still some bugs to be fixed.
* Update conv2d layer, fully connected layer and const layer
* Update convolution_layer.cpp
* Add OpenCV.js DNN module WebNN Backend (both using webnn-polyfill and electron)
* Update dnn.cpp
* Fix Error in dnn.cpp
* Resolve duplication in conditions in convolution_layer.cpp
* Fixed the issues in the comments
* Fix building issue
* Update tutorial
* Fixed comments
* Address the comments
* Update CMakeLists.txt
* Offer more accurate perf test on native
* Add better perf tests for both native and web
* Modify per tests for better results
* Use more latest version of Electron
* Support latest WebNN Clamp op
* Add definition of HAVE_WEBNN macro
* Support group convolution
* Implement Scale_layer using WebNN
* Add Softmax option for native classification example
* Fix comments
* Fix comments
Add DNN-based face detection and face recognition into modules/objdetect
* Add DNN-based face detector impl and interface
* Add a sample for DNN-based face detector
* add recog
* add notes
* move samples from samples/cpp to samples/dnn
* add documentation for dnn_face
* add set/get methods for input size, nms & score threshold and topk
* remove the DNN prefix from the face detector and face recognizer
* remove default values in the constructor of impl
* regenerate priors after setting input size
* two filenames for readnet
* Update face.hpp
* Update face_recognize.cpp
* Update face_match.cpp
* Update face.hpp
* Update face_recognize.cpp
* Update face_match.cpp
* Update face_recognize.cpp
* Update dnn_face.markdown
* Update dnn_face.markdown
* Update face.hpp
* Update dnn_face.markdown
* add regression test for face detection
* remove underscore prefix; fix warnings
* add reference & acknowledgement for face detection
* Update dnn_face.markdown
* Update dnn_face.markdown
* Update ts.hpp
* Update test_face.cpp
* Update face_match.cpp
* fix a compile error for python interface; add python examples for face detection and recognition
* Major changes for Vadim's comments:
* Replace class name FaceDetector with FaceDetectorYN in related failes
* Declare local mat before loop in modules/objdetect/src/face_detect.cpp
* Make input image and save flag optional in samples/dnn/face_detect(.cpp, .py)
* Add camera support in samples/dnn/face_detect(.cpp, .py)
* correct file paths for regression test
* fix convertion warnings; remove extra spaces
* update face_recog
* Update dnn_face.markdown
* Fix warnings and errors for the default CI reports:
* Remove trailing white spaces and extra new lines.
* Fix convertion warnings for windows and iOS.
* Add braces around initialization of subobjects.
* Fix warnings and errors for the default CI systems:
* Add prefix 'FR_' for each value name in enum DisType to solve the
redefinition error for iOS compilation; Modify other code accordingly
* Add bookmark '#tutorial_dnn_face' to solve warnings from doxygen
* Correct documentations to solve warnings from doxygen
* update FaceRecognizerSF
* Fix the error for CI to find ONNX models correctly
* add suffix f to float assignments
* add backend & target options for initializing face recognizer
* add checkeq for checking input size and preset size
* update test and threshold
* changes in response to alalek's comments:
* fix typos in samples/dnn/face_match.py
* import numpy before importing cv2
* add documentation to .setInputSize()
* remove extra include in face_recognize.cpp
* fix some bugs
* Update dnn_face.markdown
* update thresholds; remove useless code
* add time suffix to YuNet filename in test
* objdetect: update test code
* Added PaddlePaddle classification model conversion case
* Modify cv2 import as cv
* Modify documents in dnn_conversion/paddlepaddle
* Modify documents in dnn_conversion/paddlepaddle
Add DaSiamRPN tracker sample of c++ version
* add sample dasiamrpn_tracker of c++ version.
* samples(dasiamrpn_tracker.cpp): apply clang-format
- exclude "keys" variable
* samples(dasiamrpn_tracker.cpp): coding style and UX fixes
Scripts for downloading models in DNN samples
* Initial commit. Utility classes and functions for downloading files
* updated download script
* Support YAML parsing, update download script and configs
* Fix problem with archived files
* fix models.yml
* Move download utilities to more appropriate place
* Fix script description
* Update README
* update utilities for broader range of files
* fix loading with no hashsum provided
* remove unnecessary import
* fix for Python2
* Add usage examples for downloadFile function
* Add more secure cache folder selection
* Remove trailing whitespaces
* Fix indentation
* Update function interface
* Change function for temp dir, change entry name in models.yml
* Update getCacheDirectory function call
* Return python implementation for cache directory selection, use more specific env variable
* Fix whitespace
[GSoC] High Level API and Samples for Scene Text Detection and Recognition
* APIs and samples for scene text detection and recognition
* update APIs and tutorial for Text Detection and Recognition
* API updates:
(1) put decodeType into struct Voc
(2) optimize the post-processing of DB
* sample update:
(1) add transformation into scene_text_spotting.cpp
(2) modify text_detection.cpp with API update
* update tutorial
* simplify text recognition API
update tutorial
* update impl usage in recognize() and detect()
* dnn: refactoring public API of TextRecognitionModel/TextDetectionModel
* update provided models
update opencv.bib
* dnn: adjust text rectangle angle
* remove points ordering operation in model.cpp
* update gts of DB test in test_model.cpp
* dnn: ensure to keep text rectangle angle
- avoid 90/180 degree turns
* dnn(text): use quadrangle result in TextDetectionModel API
* dnn: update Text Detection API
(1) keep points' order consistent with (bl, tl, tr, br) in unclip
(2) update contourScore with boundingRect
[GSoC] Add siamrpnpp.py
* Updated base branch with siamrpnpp.py
* Addition of Parsers
* Merged to using few ONNX files, Changes to Parsers, Links to Repo
* Deleted whitespace
* Adjusting flake8 error
* Fixes according to review
* Fix according to review
* Addition of OpenVINO backends and Computation target devices
* Fix on backend after review
* Fixes after review
* Remove extra white space
* Removed Repeated Varaibles
revise default proto to match the filename in documentations
fix a bug
beautify python codes
fix bug
beautify codes
add test samples with larger/smaller size
remove unless code
using bytearray without creating tmp file
remove useless codes
Emmanuel Ferdman
lpylpy0514
alexlyulkov
Alexander Lyulkov
unknown
Alessandro de Oliveira Faria (A.K.A. CABELO)
Zihao Mu
luz paz
Sinitsina Maria
berak
Suleyman TURKMEN
Alexander Alekhin
Sinitsina Maria
Christian Clauss
Hanxi Guo
Chengrui Wang
Yuantao Feng
Shivanshu Tyagi
WJJ1995
Tsukasa Sugiura
rogday
Anna Prigarina
jiangjiajun
Jason
Anastasia M
Alexander Smorkalov
mznw
Sergei Slashchinin
Wenqing Zhang
Omar Alzaibaq
Zhiyuan Chen
jinyup100
Zihao Mu
zhaoyue-zephyrus