mae-cnn/downstream_d2/README.md

About code isolation

This downstream_d2 is isolated from pre-training codes. One can treat this downstream_d2 as an independent codebase 🛠️.

Fine-tuned ResNet-50 weights, log files, and performance

[weights (pre-trained by SparK)] [weights (fine-tuned on COCO)] [metrics.json] [log.txt] [tensorboard file]

Installation Detectron2 v0.6 before fine-tuning ResNet on COCO

1. Let you in some python environment, e.g.:

$ conda create -n spark python=3.8 -y
$ conda activate spark

2. Install detectron2==0.6 (e.g., with torch==1.10.0 and cuda11.3):

$ pip install detectron2==0.6 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html

You can also find instructions for different pytorch/cuda versions on this page.

3. Put the COCO dataset folder at downstream_d2/datasets/coco. The folder should follow the directory structure requried by Detectron2, which should look like this:

downstream_d2/datasets/coco:
    annotations/:
        captions_train2017.json  captions_val2017.json
        instances_train2017.json  instances_val2017.json
        person_keypoints_train2017.json  person_keypoints_val2017.json
    train2017/:
        a_lot_images.jpg
    val2017/:
        a_lot_images.jpg

Training from pre-trained checkpoint

The script file for COCO fine-tuning (object detection and instance segmentation) is downstream_d2/train_net.py, which is a modification of Detectron2's tools/train_net.py.

Before fine-tuning a ResNet50 pre-trained by SparK, you should first convert our checkpoint file to Detectron2-style .pkl file:

$ cd /path/to/SparK/downstream_d2
$ python3 convert-timm-to-d2.py /some/path/to/resnet50_1kpretrained_timm_style.pth d2-style.pkl

For a ResNet50, you should see a log reporting len(state)==318:

[convert] .pkl is generated! (from `/some/path/to/resnet50_1kpretrained_timm_style.pth`, to `d2-style.pkl`, len(state)==318)

Then run fine-tuning on single machine with 8 gpus:

$ cd /path/to/SparK/downstream_d2
$ python3 ./train_net.py --resume --num-gpus 8 --config-file ./configs/coco_R_50_FPN_CONV_1x_moco_adam.yaml \
  MODEL.WEIGHTS d2-style.pkl \
  OUTPUT_DIR <your_output_dir>

For multiple machines, plus these args:

--num-machines <total_num> --machine-rank <this_rank> --dist-url <url:port>

In <your_output_dir> you'll see the log files generated by Detectron2.

Details: how we modify the official Detectron2's tools/train_net.py to get our downstream_d2/train_net.py

1. We add two new hyperparameters:

   • str SOLVER.OPTIMIZER: use 'ADAM' (the same as 'ADAMW') or 'SGD' optimizer
   • float SOLVER.LR_DECAY: the decay ratio (from 0. to 1.) of layer-wise learning rate decay trick

2. We implement layer-wise lr decay in downstream_d2/lr_decay.py.

3. We write a script to convert our timm-style pre-trained ResNet weights to Detectron2-style in downstream_d2/convert-timm-to-d2.py.

4. We also add a hook for logging results to cfg.OUTPUT_DIR/d2_coco_log.txt.

All of our modifications to the original are commented with # [modification] ... in downstream_d2/train_net.py or other files.