SparK🔥: "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"

[arXiv] [pdf] [state-of-the-art self-supervised convnet] [bibtex]

Introduction

This is an official implementation of the paper: "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling". We'll be updating frequently these days, so you might consider star ⭐ or watch 👓 this repo to get the latest information. Updates including downstream implementations, Colab tutorial, inference and visualization codes will come soon!

In this work we designed a BERT-style pre-training framework (a.k.a. masked image modeling) for any hierarchical (multi-scale) convnets. As shown above, it gathers all unmasked patches to form a sparse image and uses sparse convolution for encoding. A dense, hierarchical decoder is applied then, to reconstruct all masked pixels. This method is general and powerful: it can be used directly on any convolutional backbones such as classical ResNets (the right) and modern ConvNeXts (left), and can bring a leap in their performance:

See our paper for more analysis, discussion, and evaluation.

Pre-training

See PRETRAIN.md for preparation and pre-training.

ImageNet Fine-tuning

After finishing the preparation in PRETRAIN.md, check downstream_imagenet for subsequent instructions.

Fine-tuning ResNets on COCO

Install detectron2 and see downstream_d2 for more details.

Fine-tuning ConvNeXts on COCO

Install mmcv and mmdetection then see downstream_mmdet for more details.

Acknowledgement

We heavily referred to these useful codebases:

• BEiT
• MAE
• ConvNeXt

We also appreciate these elegant frameworks:

• timm
• MoCoV2
• Detectron2 and MMDetection

License

This project is under the CC-BY 4.0 license. See LICENSE for more details.

Citation

If you found this project useful, please consider adding a star ⭐, or citing us 📖:

@Article{tian2023designing,
  author  = {Keyu Tian and Yi Jiang and Qishuai Diao and Chen Lin and Liwei Wang and Zehuan Yuan},
  title   = {Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling},
  journal = {arXiv:2301.03580},
  year    = {2023},
}