You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
Keyu Tian 1023de5bd5
Update README.md
2 years ago
models [initial commit] 2 years ago
scripts [initial commit] 2 years ago
utils [initial commit] 2 years ago
.gitignore [initial commit] 2 years ago
LICENSE [initial commit] 2 years ago
PRETRAIN.md [upd] readme 2 years ago
README.md Update README.md 2 years ago
decoder.py [initial commit] 2 years ago
dist.py [initial commit] 2 years ago
encoder.py [initial commit] 2 years ago
launch.py [initial commit] 2 years ago
main.py [initial commit] 2 years ago
sampler.py [initial commit] 2 years ago
spark.py [initial commit] 2 years ago

README.md

The Official PyTorch Implementation of SparK🔥 (Sparse and Hierarchical Masked Modeling) arXiv

Introduction

This is an official implementation of the paper: "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling". We'll be updating frequently these days, so you might consider star or watch 👓 this repository to get the latest information.

In this work we designed a BERT-style pre-training framework (a.k.a. masked image modeling) for any hierarchical (multi-scale) convnets. As shown above, it gathers all unmasked patches to form a sparse image and uses sparse convolution for encoding. A dense, hierarchical decoder is applied then, to reconstruct all masked pixels. This method is general and powerful: it can be used directly on any convolutional backbones such as classical ResNets (the right) and modern ConvNeXts (left), and can bring a leap in their performance:

See our paper for more analysis, discussion, and evaluation.

Pre-train

See PRETRAIN.md for preparation and pre-training.

Fine-tune on ImageNet

After finishing the preparation in PRETRAIN.md, see downstream_imagenet for subsequent instructions.

Fine-tune ResNets on COCO

Install Detectron2 and see downstream_d2 for more details.

Fine-tune ConvNeXts on COCO

Install mmcv and mmdetection then see downstream_mmdet for more details.

Acknowledgement

We heavily referred to these useful codebases:

We also appreciate these elegant frameworks:

License

This project is under the CC-BY 4.0 license. See LICENSE for more details.

Citation

If you found this project useful, please consider adding a star , or citing us 📖:

@Article{tian2023designing,
  author  = {Keyu Tian and Yi Jiang and Qishuai Diao and Chen Lin and Liwei Wang and Zehuan Yuan},
  title   = {Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling},
  journal = {arXiv:2301.03580},
  year    = {2023},
}