mirror of https://github.com/opencv/opencv.git
Custom layers for deep learning networks (#11129)
* Custom deep learning layers support * Stack custom deep learning layerspull/11385/head
Dmitry Kurtaev
7 years ago
committed by
Vadim Pisarevsky
parent
909a25571e
commit
4ec456f0a0
19 changed files with 928 additions and 146 deletions
@ -0,0 +1,192 @@ |
|||||||
|
# Custom deep learning layers support {#tutorial_dnn_custom_layers} |
||||||
|
|
||||||
|
## Introduction |
||||||
|
Deep learning is a fast growing area. The new approaches to build neural networks |
||||||
|
usually introduce new types of layers. They could be modifications of existing |
||||||
|
ones or implement outstanding researching ideas. |
||||||
|
|
||||||
|
OpenCV gives an opportunity to import and run networks from different deep learning |
||||||
|
frameworks. There are a number of the most popular layers. However you can face |
||||||
|
a problem that your network cannot be imported using OpenCV because of unimplemented layers. |
||||||
|
|
||||||
|
The first solution is to create a feature request at https://github.com/opencv/opencv/issues |
||||||
|
mentioning details such a source of model and type of new layer. A new layer could |
||||||
|
be implemented if OpenCV community shares this need. |
||||||
|
|
||||||
|
The second way is to define a **custom layer** so OpenCV's deep learning engine |
||||||
|
will know how to use it. This tutorial is dedicated to show you a process of deep |
||||||
|
learning models import customization. |
||||||
|
|
||||||
|
## Define a custom layer in C++ |
||||||
|
Deep learning layer is a building block of network's pipeline. |
||||||
|
It has connections to **input blobs** and produces results to **output blobs**. |
||||||
|
There are trained **weights** and **hyper-parameters**. |
||||||
|
Layers' names, types, weights and hyper-parameters are stored in files are generated by |
||||||
|
native frameworks during training. If OpenCV mets unknown layer type it throws an |
||||||
|
exception trying to read a model: |
||||||
|
|
||||||
|
``` |
||||||
|
Unspecified error: Can't create layer "layer_name" of type "MyType" in function getLayerInstance |
||||||
|
``` |
||||||
|
|
||||||
|
To import the model correctly you have to derive a class from cv::dnn::Layer with |
||||||
|
the following methods: |
||||||
|
|
||||||
|
@snippet dnn/custom_layers.cpp A custom layer interface |
||||||
|
|
||||||
|
And register it before the import: |
||||||
|
|
||||||
|
@snippet dnn/custom_layers.cpp Register a custom layer |
||||||
|
|
||||||
|
@note `MyType` is a type of unimplemented layer from the thrown exception. |
||||||
|
|
||||||
|
Let's see what all the methods do: |
||||||
|
|
||||||
|
- Constructor |
||||||
|
|
||||||
|
@snippet dnn/custom_layers.cpp MyLayer::MyLayer |
||||||
|
|
||||||
|
Retrieves hyper-parameters from cv::dnn::LayerParams. If your layer has trainable |
||||||
|
weights they will be already stored in the Layer's member cv::dnn::Layer::blobs. |
||||||
|
|
||||||
|
- A static method `create` |
||||||
|
|
||||||
|
@snippet dnn/custom_layers.cpp MyLayer::create |
||||||
|
|
||||||
|
This method should create an instance of you layer and return cv::Ptr with it. |
||||||
|
|
||||||
|
- Output blobs' shape computation |
||||||
|
|
||||||
|
@snippet dnn/custom_layers.cpp MyLayer::getMemoryShapes |
||||||
|
|
||||||
|
Returns layer's output shapes depends on input shapes. You may request an extra |
||||||
|
memory using `internals`. |
||||||
|
|
||||||
|
- Run a layer |
||||||
|
|
||||||
|
@snippet dnn/custom_layers.cpp MyLayer::forward |
||||||
|
|
||||||
|
Implement a layer's logic here. Compute outputs for given inputs. |
||||||
|
|
||||||
|
@note OpenCV manages memory allocated for layers. In the most cases the same memory |
||||||
|
can be reused between layers. So your `forward` implementation should not rely that |
||||||
|
the second invocation of `forward` will has the same data at `outputs` and `internals`. |
||||||
|
|
||||||
|
- Optional `finalize` method |
||||||
|
|
||||||
|
@snippet dnn/custom_layers.cpp MyLayer::finalize |
||||||
|
|
||||||
|
The chain of methods are the following: OpenCV deep learning engine calls `create` |
||||||
|
method once then it calls `getMemoryShapes` for an every created layer then you |
||||||
|
can make some preparations depends on known input dimensions at cv::dnn::Layer::finalize. |
||||||
|
After network was initialized only `forward` method is called for an every network's input. |
||||||
|
|
||||||
|
@note Varying input blobs' sizes such height or width or batch size you make OpenCV |
||||||
|
reallocate all the internal memory. That leads efficiency gaps. Try to initialize |
||||||
|
and deploy models using a fixed batch size and image's dimensions. |
||||||
|
|
||||||
|
## Example: custom layer from Caffe |
||||||
|
Let's create a custom layer `Interp` from https://github.com/cdmh/deeplab-public. |
||||||
|
It's just a simple resize that takes an input blob of size `N x C x Hi x Wi` and returns |
||||||
|
an output blob of size `N x C x Ho x Wo` where `N` is a batch size, `C` is a number of channels, |
||||||
|
`Hi x Wi` and `Ho x Wo` are input and output `height x width` correspondingly. |
||||||
|
This layer has no trainable weights but it has hyper-parameters to specify an output size. |
||||||
|
|
||||||
|
In example, |
||||||
|
~~~~~~~~~~~~~ |
||||||
|
layer { |
||||||
|
name: "output" |
||||||
|
type: "Interp" |
||||||
|
bottom: "input" |
||||||
|
top: "output" |
||||||
|
interp_param { |
||||||
|
height: 9 |
||||||
|
width: 8 |
||||||
|
} |
||||||
|
} |
||||||
|
~~~~~~~~~~~~~ |
||||||
|
|
||||||
|
This way our implementation can look like: |
||||||
|
|
||||||
|
@snippet dnn/custom_layers.cpp InterpLayer |
||||||
|
|
||||||
|
Next we need to register a new layer type and try to import the model. |
||||||
|
|
||||||
|
@snippet dnn/custom_layers.cpp Register InterpLayer |
||||||
|
|
||||||
|
## Example: custom layer from TensorFlow |
||||||
|
This is an example of how to import a network with [tf.image.resize_bilinear](https://www.tensorflow.org/versions/master/api_docs/python/tf/image/resize_bilinear) |
||||||
|
operation. This is also a resize but with an implementation different from OpenCV's or `Interp` above. |
||||||
|
|
||||||
|
Let's create a single layer network: |
||||||
|
~~~~~~~~~~~~~{.py} |
||||||
|
inp = tf.placeholder(tf.float32, [2, 3, 4, 5], 'input') |
||||||
|
resized = tf.image.resize_bilinear(inp, size=[9, 8], name='resize_bilinear') |
||||||
|
~~~~~~~~~~~~~ |
||||||
|
OpenCV sees that TensorFlow's graph in the following way: |
||||||
|
|
||||||
|
``` |
||||||
|
node { |
||||||
|
name: "input" |
||||||
|
op: "Placeholder" |
||||||
|
attr { |
||||||
|
key: "dtype" |
||||||
|
value { |
||||||
|
type: DT_FLOAT |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
node { |
||||||
|
name: "resize_bilinear/size" |
||||||
|
op: "Const" |
||||||
|
attr { |
||||||
|
key: "dtype" |
||||||
|
value { |
||||||
|
type: DT_INT32 |
||||||
|
} |
||||||
|
} |
||||||
|
attr { |
||||||
|
key: "value" |
||||||
|
value { |
||||||
|
tensor { |
||||||
|
dtype: DT_INT32 |
||||||
|
tensor_shape { |
||||||
|
dim { |
||||||
|
size: 2 |
||||||
|
} |
||||||
|
} |
||||||
|
tensor_content: "\t\000\000\000\010\000\000\000" |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
node { |
||||||
|
name: "resize_bilinear" |
||||||
|
op: "ResizeBilinear" |
||||||
|
input: "input:0" |
||||||
|
input: "resize_bilinear/size" |
||||||
|
attr { |
||||||
|
key: "T" |
||||||
|
value { |
||||||
|
type: DT_FLOAT |
||||||
|
} |
||||||
|
} |
||||||
|
attr { |
||||||
|
key: "align_corners" |
||||||
|
value { |
||||||
|
b: false |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
library { |
||||||
|
} |
||||||
|
``` |
||||||
|
Custom layers import from TensorFlow is designed to put all layer's `attr` into |
||||||
|
cv::dnn::LayerParams but input `Const` blobs into cv::dnn::Layer::blobs. |
||||||
|
In our case resize's output shape will be stored in layer's `blobs[0]`. |
||||||
|
|
||||||
|
@snippet dnn/custom_layers.cpp ResizeBilinearLayer |
||||||
|
|
||||||
|
Next we register a layer and try to import the model. |
||||||
|
|
||||||
|
@snippet dnn/custom_layers.cpp Register ResizeBilinearLayer |
||||||
@ -0,0 +1,232 @@ |
|||||||
|
#include <opencv2/dnn.hpp> |
||||||
|
|
||||||
|
//! [A custom layer interface]
|
||||||
|
class MyLayer : public cv::dnn::Layer |
||||||
|
{ |
||||||
|
public: |
||||||
|
//! [MyLayer::MyLayer]
|
||||||
|
MyLayer(const cv::dnn::LayerParams ¶ms); |
||||||
|
//! [MyLayer::MyLayer]
|
||||||
|
|
||||||
|
//! [MyLayer::create]
|
||||||
|
static cv::Ptr<cv::dnn::Layer> create(cv::dnn::LayerParams& params); |
||||||
|
//! [MyLayer::create]
|
||||||
|
|
||||||
|
//! [MyLayer::getMemoryShapes]
|
||||||
|
virtual bool getMemoryShapes(const std::vector<std::vector<int> > &inputs, |
||||||
|
const int requiredOutputs, |
||||||
|
std::vector<std::vector<int> > &outputs, |
||||||
|
std::vector<std::vector<int> > &internals) const; |
||||||
|
//! [MyLayer::getMemoryShapes]
|
||||||
|
|
||||||
|
//! [MyLayer::forward]
|
||||||
|
virtual void forward(std::vector<cv::Mat*> &inputs, std::vector<cv::Mat> &outputs, std::vector<cv::Mat> &internals); |
||||||
|
//! [MyLayer::forward]
|
||||||
|
|
||||||
|
//! [MyLayer::finalize]
|
||||||
|
virtual void finalize(const std::vector<cv::Mat*> &inputs, std::vector<cv::Mat> &outputs); |
||||||
|
//! [MyLayer::finalize]
|
||||||
|
|
||||||
|
virtual void forward(cv::InputArrayOfArrays inputs, cv::OutputArrayOfArrays outputs, cv::OutputArrayOfArrays internals); |
||||||
|
}; |
||||||
|
//! [A custom layer interface]
|
||||||
|
|
||||||
|
//! [InterpLayer]
|
||||||
|
class InterpLayer : public cv::dnn::Layer |
||||||
|
{ |
||||||
|
public: |
||||||
|
InterpLayer(const cv::dnn::LayerParams ¶ms) : Layer(params) |
||||||
|
{ |
||||||
|
outWidth = params.get<int>("width", 0); |
||||||
|
outHeight = params.get<int>("height", 0); |
||||||
|
} |
||||||
|
|
||||||
|
static cv::Ptr<cv::dnn::Layer> create(cv::dnn::LayerParams& params) |
||||||
|
{ |
||||||
|
return cv::Ptr<cv::dnn::Layer>(new InterpLayer(params)); |
||||||
|
} |
||||||
|
|
||||||
|
virtual bool getMemoryShapes(const std::vector<std::vector<int> > &inputs, |
||||||
|
const int requiredOutputs, |
||||||
|
std::vector<std::vector<int> > &outputs, |
||||||
|
std::vector<std::vector<int> > &internals) const |
||||||
|
{ |
||||||
|
CV_UNUSED(requiredOutputs); CV_UNUSED(internals); |
||||||
|
std::vector<int> outShape(4); |
||||||
|
outShape[0] = inputs[0][0]; // batch size
|
||||||
|
outShape[1] = inputs[0][1]; // number of channels
|
||||||
|
outShape[2] = outHeight; |
||||||
|
outShape[3] = outWidth; |
||||||
|
outputs.assign(1, outShape); |
||||||
|
return false; |
||||||
|
} |
||||||
|
|
||||||
|
// Implementation of this custom layer is based on https://github.com/cdmh/deeplab-public/blob/master/src/caffe/layers/interp_layer.cpp
|
||||||
|
virtual void forward(std::vector<cv::Mat*> &inputs, std::vector<cv::Mat> &outputs, std::vector<cv::Mat> &internals) |
||||||
|
{ |
||||||
|
CV_UNUSED(internals); |
||||||
|
cv::Mat& inp = *inputs[0]; |
||||||
|
cv::Mat& out = outputs[0]; |
||||||
|
const float* inpData = (float*)inp.data; |
||||||
|
float* outData = (float*)out.data; |
||||||
|
|
||||||
|
const int batchSize = inp.size[0]; |
||||||
|
const int numChannels = inp.size[1]; |
||||||
|
const int inpHeight = inp.size[2]; |
||||||
|
const int inpWidth = inp.size[3]; |
||||||
|
|
||||||
|
const float rheight = (outHeight > 1) ? static_cast<float>(inpHeight - 1) / (outHeight - 1) : 0.f; |
||||||
|
const float rwidth = (outWidth > 1) ? static_cast<float>(inpWidth - 1) / (outWidth - 1) : 0.f; |
||||||
|
for (int h2 = 0; h2 < outHeight; ++h2) |
||||||
|
{ |
||||||
|
const float h1r = rheight * h2; |
||||||
|
const int h1 = static_cast<int>(h1r); |
||||||
|
const int h1p = (h1 < inpHeight - 1) ? 1 : 0; |
||||||
|
const float h1lambda = h1r - h1; |
||||||
|
const float h0lambda = 1.f - h1lambda; |
||||||
|
for (int w2 = 0; w2 < outWidth; ++w2) |
||||||
|
{ |
||||||
|
const float w1r = rwidth * w2; |
||||||
|
const int w1 = static_cast<int>(w1r); |
||||||
|
const int w1p = (w1 < inpWidth - 1) ? 1 : 0; |
||||||
|
const float w1lambda = w1r - w1; |
||||||
|
const float w0lambda = 1.f - w1lambda; |
||||||
|
const float* pos1 = inpData + h1 * inpWidth + w1; |
||||||
|
float* pos2 = outData + h2 * outWidth + w2; |
||||||
|
for (int c = 0; c < batchSize * numChannels; ++c) |
||||||
|
{ |
||||||
|
pos2[0] = |
||||||
|
h0lambda * (w0lambda * pos1[0] + w1lambda * pos1[w1p]) + |
||||||
|
h1lambda * (w0lambda * pos1[h1p * inpWidth] + w1lambda * pos1[h1p * inpWidth + w1p]); |
||||||
|
pos1 += inpWidth * inpHeight; |
||||||
|
pos2 += outWidth * outHeight; |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
virtual void forward(cv::InputArrayOfArrays, cv::OutputArrayOfArrays, cv::OutputArrayOfArrays) {} |
||||||
|
|
||||||
|
private: |
||||||
|
int outWidth, outHeight; |
||||||
|
}; |
||||||
|
//! [InterpLayer]
|
||||||
|
|
||||||
|
//! [ResizeBilinearLayer]
|
||||||
|
class ResizeBilinearLayer : public cv::dnn::Layer |
||||||
|
{ |
||||||
|
public: |
||||||
|
ResizeBilinearLayer(const cv::dnn::LayerParams ¶ms) : Layer(params) |
||||||
|
{ |
||||||
|
CV_Assert(!params.get<bool>("align_corners", false)); |
||||||
|
CV_Assert(blobs.size() == 1, blobs[0].type() == CV_32SC1); |
||||||
|
outHeight = blobs[0].at<int>(0, 0); |
||||||
|
outWidth = blobs[0].at<int>(0, 1); |
||||||
|
} |
||||||
|
|
||||||
|
static cv::Ptr<cv::dnn::Layer> create(cv::dnn::LayerParams& params) |
||||||
|
{ |
||||||
|
return cv::Ptr<cv::dnn::Layer>(new ResizeBilinearLayer(params)); |
||||||
|
} |
||||||
|
|
||||||
|
virtual bool getMemoryShapes(const std::vector<std::vector<int> > &inputs, |
||||||
|
const int requiredOutputs, |
||||||
|
std::vector<std::vector<int> > &outputs, |
||||||
|
std::vector<std::vector<int> > &internals) const |
||||||
|
{ |
||||||
|
CV_UNUSED(requiredOutputs); CV_UNUSED(internals); |
||||||
|
std::vector<int> outShape(4); |
||||||
|
outShape[0] = inputs[0][0]; // batch size
|
||||||
|
outShape[1] = inputs[0][1]; // number of channels
|
||||||
|
outShape[2] = outHeight; |
||||||
|
outShape[3] = outWidth; |
||||||
|
outputs.assign(1, outShape); |
||||||
|
return false; |
||||||
|
} |
||||||
|
|
||||||
|
// This implementation is based on a reference implementation from
|
||||||
|
// https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/kernels/internal/reference/reference_ops.h
|
||||||
|
virtual void forward(std::vector<cv::Mat*> &inputs, std::vector<cv::Mat> &outputs, std::vector<cv::Mat> &internals) |
||||||
|
{ |
||||||
|
CV_UNUSED(internals); |
||||||
|
cv::Mat& inp = *inputs[0]; |
||||||
|
cv::Mat& out = outputs[0]; |
||||||
|
const float* inpData = (float*)inp.data; |
||||||
|
float* outData = (float*)out.data; |
||||||
|
|
||||||
|
const int batchSize = inp.size[0]; |
||||||
|
const int numChannels = inp.size[1]; |
||||||
|
const int inpHeight = inp.size[2]; |
||||||
|
const int inpWidth = inp.size[3]; |
||||||
|
|
||||||
|
float heightScale = static_cast<float>(inpHeight) / outHeight; |
||||||
|
float widthScale = static_cast<float>(inpWidth) / outWidth; |
||||||
|
for (int b = 0; b < batchSize; ++b) |
||||||
|
{ |
||||||
|
for (int y = 0; y < outHeight; ++y) |
||||||
|
{ |
||||||
|
float input_y = y * heightScale; |
||||||
|
int y0 = static_cast<int>(std::floor(input_y)); |
||||||
|
int y1 = std::min(y0 + 1, inpHeight - 1); |
||||||
|
for (int x = 0; x < outWidth; ++x) |
||||||
|
{ |
||||||
|
float input_x = x * widthScale; |
||||||
|
int x0 = static_cast<int>(std::floor(input_x)); |
||||||
|
int x1 = std::min(x0 + 1, inpWidth - 1); |
||||||
|
for (int c = 0; c < numChannels; ++c) |
||||||
|
{ |
||||||
|
float interpolation = |
||||||
|
inpData[offset(inp.size, c, x0, y0, b)] * (1 - (input_y - y0)) * (1 - (input_x - x0)) + |
||||||
|
inpData[offset(inp.size, c, x0, y1, b)] * (input_y - y0) * (1 - (input_x - x0)) + |
||||||
|
inpData[offset(inp.size, c, x1, y0, b)] * (1 - (input_y - y0)) * (input_x - x0) + |
||||||
|
inpData[offset(inp.size, c, x1, y1, b)] * (input_y - y0) * (input_x - x0); |
||||||
|
outData[offset(out.size, c, x, y, b)] = interpolation; |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
virtual void forward(cv::InputArrayOfArrays, cv::OutputArrayOfArrays, cv::OutputArrayOfArrays) {} |
||||||
|
|
||||||
|
private: |
||||||
|
static inline int offset(const cv::MatSize& size, int c, int x, int y, int b) |
||||||
|
{ |
||||||
|
return x + size[3] * (y + size[2] * (c + size[1] * b)); |
||||||
|
} |
||||||
|
|
||||||
|
int outWidth, outHeight; |
||||||
|
}; |
||||||
|
//! [ResizeBilinearLayer]
|
||||||
|
|
||||||
|
//! [Register a custom layer]
|
||||||
|
#include <opencv2/dnn/layer.details.hpp> // CV_DNN_REGISTER_LAYER_CLASS macro |
||||||
|
|
||||||
|
int main(int argc, char** argv) |
||||||
|
{ |
||||||
|
CV_DNN_REGISTER_LAYER_CLASS(MyType, MyLayer); |
||||||
|
// ...
|
||||||
|
//! [Register a custom layer]
|
||||||
|
CV_UNUSED(argc); CV_UNUSED(argv); |
||||||
|
//! [Register InterpLayer]
|
||||||
|
CV_DNN_REGISTER_LAYER_CLASS(Interp, InterpLayer); |
||||||
|
cv::dnn::Net caffeNet = cv::dnn::readNet("/path/to/config.prototxt", "/path/to/weights.caffemodel"); |
||||||
|
//! [Register InterpLayer]
|
||||||
|
|
||||||
|
//! [Register ResizeBilinearLayer]
|
||||||
|
CV_DNN_REGISTER_LAYER_CLASS(ResizeBilinear, ResizeBilinearLayer); |
||||||
|
cv::dnn::Net tfNet = cv::dnn::readNet("/path/to/graph.pb"); |
||||||
|
//! [Register ResizeBilinearLayer]
|
||||||
|
} |
||||||
|
|
||||||
|
cv::Ptr<cv::dnn::Layer> MyLayer::create(cv::dnn::LayerParams& params) |
||||||
|
{ |
||||||
|
return cv::Ptr<cv::dnn::Layer>(new MyLayer(params)); |
||||||
|
} |
||||||
|
MyLayer::MyLayer(const cv::dnn::LayerParams&) {} |
||||||
|
bool MyLayer::getMemoryShapes(const std::vector<std::vector<int> >&, const int, |
||||||
|
std::vector<std::vector<int> >&, |
||||||
|
std::vector<std::vector<int> >&) const { return false; } |
||||||
|
void MyLayer::forward(std::vector<cv::Mat*>&, std::vector<cv::Mat>&, std::vector<cv::Mat>&) {} |
||||||
|
void MyLayer::finalize(const std::vector<cv::Mat*>&, std::vector<cv::Mat>&) {} |
||||||
|
void MyLayer::forward(cv::InputArrayOfArrays, cv::OutputArrayOfArrays, cv::OutputArrayOfArrays) {} |
||||||
Loading…
Reference in new issue