mae-cnn/downstream_d2/README.md

## About code isolation

This `downstream_d2` is isolated from pre-training codes. One can treat this `downstream_d2` as an independent codebase 🛠️.


## Fine-tuned ResNet-50 weights, log files, and performance

<div align="center">

  [[`fine-tuned weights`](https://drive.google.com/file/d/1Ue7SiQ1E_AwgtYo56Fm-iUlQPZ8vIwYj/view?usp=share_link)]
  [[`metrics.json`](https://drive.google.com/file/d/1wfbUWh4svV8sPWya_0PAhsLHVayDQRCi/view?usp=share_link)]
  [[`log.txt`](https://drive.google.com/file/d/11zVo_87pe9DMAmfNQK9FUfyjQWHTRKxV/view?usp=share_link)]
  [[`tensorboard file`](https://drive.google.com/file/d/1aM1qj8c3-Uka1dZuYmKhgp1lNJpeMDMl/view?usp=share_link)]
</div>

<p align="center">
<img src="https://user-images.githubusercontent.com/39692511/211497479-0563e891-f2ad-4cf1-b682-a21c2be1442d.png" width=80%>
<p>


## Installation [Detectron2 v0.6](https://github.com/facebookresearch/detectron2/releases/tag/v0.6) for fine-tuning ResNet on COCO


1. Let you in some python environment, e.g.:
```shell script
$ conda create -n spark python=3.8 -y
$ conda activate spark
```

2. Install `detectron2==0.6` (e.g., with `torch==1.10.0` and `cuda11.3`):
```shell script
$ pip install detectron2==0.6 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html
```

You can also find instructions for different pytorch/cuda versions on [this page](https://github.com/facebookresearch/detectron2/releases/tag/v0.6).


3. Put the COCO dataset folder at `downstream_d2/datasets/coco`.
The folder should follow the [directory structure](https://github.com/facebookresearch/detectron2/tree/master/datasets) requried by `detectron2`, which should look like this:
```
downstream_d2/datasets/coco:
    annotations/:
        captions_train2017.json  captions_val2017.json
        instances_train2017.json  instances_val2017.json
        person_keypoints_train2017.json  person_keypoints_val2017.json
    train2017/:
        a_lot_images.jpg
    val2017/:
        a_lot_images.jpg
```


## Training from pre-trained checkpoint

The script file for COCO fine-tuning (object detection and instance segmentation) is [downstream_d2/train_net.py](https://github.com/keyu-tian/SparK/blob/main/downstream_d2/train_net.py),
which is a modification of [Detectron2's tools/train_net.py](https://github.com/facebookresearch/detectron2/blob/v0.6/tools/train_net.py).


Before fine-tuning a ResNet50 pre-trained by SparK, you should first convert our checkpoint file to detectron2-style `.pkl` file:

```shell script
$ cd /path/to/SparK/downstream_d2
$ python3 convert-timm-to-d2.py /some/path/to/timm_resnet50_1kpretrained.pth d2-style.pkl
```

For a ResNet50, you should see a log reporting `len(state)==318`:
```text
[convert] .pkl is generated! (from `/some/path/to/timm_resnet50_1kpretrained.pth`, to `d2-style.pkl`, len(state)==318)
```

Then run fine-tuning on single machine with 8 gpus:

```shell script
$ cd /path/to/SparK/downstream_d2
$ python3 ./train_net.py --resume --num-gpus 8 --config-file ./configs/coco_R_50_FPN_CONV_1x_moco_adam.yaml \
  MODEL.WEIGHTS d2-style.pkl \
  OUTPUT_DIR <your_output_dir>
```

For multiple machines, plus these args:
```shell script
--num-machines <total_num> --machine-rank <this_rank> --dist-url <url:port>
```

In `<your_output_dir>` you'll see the log files generated by `detectron2`.


## Details: how we modify the official Detectron2's [tools/train_net.py](https://github.com/facebookresearch/detectron2/blob/v0.6/tools/train_net.py) to get our [downstream_d2/train_net.py](https://github.com/keyu-tian/SparK/blob/main/downstream_d2/train_net.py)

The main modification is adding two new hyperparameters:
- str `SOLVER.OPTIMIZER`: use 'adam' or 'sgd' optimizer
- float `SOLVER.LR_DECAY`: the decay ratio (from 0. to 1.) of layer-wise learning rate decay trick

We also add a hook for logging results to `cfg.OUTPUT_DIR/d2_coco_log.txt`.

All of our modifications to the original `train_net.py` are commented with `# [modification] ...` in [downstream_d2/train_net.py](https://github.com/keyu-tian/SparK/blob/main/downstream_d2/train_net.py).