dnn/Vulkan: skip heavy convolution task

This is a workaround for GPU hang on heavy convolution workload (> 10 GFLOPS).
e.g. ResNet101_DUC_HDC

For the long time task, vkWaitForFences() return without error but next call on
vkQueueSubmit() return -4, i.e. "VK_ERROR_DEVICE_LOST" and driver reports GPU hang.

Need more investigation on root cause of GPU hang and need to optimize convolution shader
to reduce process time.
pull/12985/head
Wu Zhiwen 6 years ago
parent 34e9d1eb3c
commit 33c9d57c6f
  1. 26
      modules/dnn/src/dnn.cpp

@ -1411,6 +1411,32 @@ struct Net::Impl
continue;
}
if (ld.type == "Convolution")
{
std::vector<MatShape> in_shapes;
std::vector<MatShape> out_shapes;
CV_Assert(ld.inputBlobs.size() == ld.outputBlobs.size());
for (int i = 0; i < ld.inputBlobs.size(); i++)
{
in_shapes.push_back(shape(*ld.inputBlobs[i]));
out_shapes.push_back(shape(ld.outputBlobs[i]));
}
int64 flops = layer->getFLOPS(in_shapes, out_shapes);
// FIXME
//
// This is a workaround for GPU hang on heavy convolution workload ( > 10 GFLOPS).
// For the long time task, vkWaitForFences() return without error but next call on
// vkQueueSubmit() return -4, i.e. "VK_ERROR_DEVICE_LOST" and driver reports GPU hang.
//
// Need more investigation on root cause of GPU hang and need to optimize convolution shader
// to reduce process time.
if (flops > CV_BIG_INT(10) * 1000 * 1000 * 1000)
{
continue;
}
}
ld.skip = false;
try

Loading…
Cancel
Save