10 KiB
SparK: the first successful BERT/MAE-style pretraining on any convolutional networks 
This is the official implementation of ICLR paper Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling. We've tried our best to make the codebase clean, short, easy to read, state-of-the-art, and only rely on minimal dependencies.
🔥 News
- On Mar. 22nd (UTC+8 8pm; UTC+0 12am) another livestream would be held at 极市平台-bilibili.
- The share on TechBeat (将门创投) is scheduled on Mar. 16th (UTC+8 8pm) too! [
📹Recorded Video] - We are honored to be invited by Synced ("机器之心机动组 视频号" on WeChat) to give a talk about SparK on Feb. 27th (UTC+0 11am, UTC+8 7pm), welcome! [
📹Recorded Video] - This work got accepted to ICLR 2023 as a Spotlight (notable-top-25%).
- Other articles: [
Synced] [DeepAI] [TheGradient] [Bytedance] [CVers[QbitAI(量子位)] [BAAI(智源)] [机器之心机动组] [极市平台] [ReadPaper笔记]
📺 Video demo (ppt slides attached to slides-animations-demo)
https://user-images.githubusercontent.com/6366788/213662770-5f814de0-cbe8-48d9-8235-e8907fd81e0e.mp4
🕹️ CoLab Visualization Demo
Check pretrain/viz_reconstruction.ipynb for visualizing the reconstruction of SparK pre-trained models, like:
We also provide pretrain/viz_spconv.ipynb that shows the "mask pattern vanishing" issue of dense conv layers.
What's new here?
🔥 On ResNets, generative pre-training surpasses contrastive learning for the first time:
🔥 ConvNeXt gains more from pre-training than Swin-Transformer, up to +3.5 points:
🔥 Larger models benefit more from SparK pre-training, showing a scaling behavior:
🔥 Pre-trained model can make reasonable predictions:
See our paper for more analysis, discussions, and evaluations.
Catalog
- Pre-training code
- Fine-tuning code
- Colab visualization playground (reconstruction, sparse conv)
- Weights & visualization playground on
Huggingface - Weights in
timm
SparK Pre-trained weights
Note: for network definitions, we directly use timm.models.ResNet and official ConvNeXt.
reso.: the image resolution; acc@1: ImageNet-1K fine-tuned acc (top-1)
| arch. | reso. | acc@1 | #params | flops | SparK pre-trained weights (self-supervised) |
|---|---|---|---|---|---|
| ResNet50 | 224 | 80.6 | 26M | 4.1G | resnet50_1kpretrained_timm_style.pth |
| ResNet101 | 224 | 82.2 | 45M | 7.9G | resnet101_1kpretrained_timm_style.pth |
| ResNet152 | 224 | 82.7 | 60M | 11.6G | resnet152_1kpretrained_timm_style.pth |
| ResNet200 | 224 | 83.1 | 65M | 15.1G | resnet200_1kpretrained_timm_style.pth |
| ConvNeXt-S | 224 | 84.1 | 50M | 8.7G | convnextS_1kpretrained_official_style.pth |
| ConvNeXt-B | 224 | 84.8 | 89M | 15.4G | convnextB_1kpretrained_official_style.pth |
| ConvNeXt-L | 224 | 85.4 | 198M | 34.4G | convnextL_1kpretrained_official_style.pth |
| ConvNeXt-L | 384 | 86.0 | 198M | 101.0G | convnextL_384_1kpretrained_official_style.pth |
| L-with-decoder | 384 | 86.0 | 198M | 101.0G | cnxL384_withdecoder_1kpretrained_spark_style.pth |
Installation & Running
We highly recommended you to use torch==1.10.0, torchvision==0.11.1, and timm==0.5.4 for reproduction.
Check INSTALL.md to install all pip dependencies.
-
Pre-training all models on ImageNet-1k: see pretrain/
-
Fine-tuning
- all models on ImageNet: check downstream_imagenet/ for subsequent instructions.
- ResNets on COCO: see downstream_d2/
- ConvNeXts on COCO: see downstream_mmdet/
Acknowledgement
We referred to these useful codebases:
License
This project is under the MIT license. See LICENSE for more details.
Citation
If you found this project useful, you can kindly give us a star ⭐, or cite us in your work 📖:
@Article{tian2023designing,
author = {Keyu Tian and Yi Jiang and Qishuai Diao and Chen Lin and Liwei Wang and Zehuan Yuan},
title = {Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling},
journal = {arXiv:2301.03580},
year = {2023},
}