|
|
## About code isolation |
|
|
|
|
|
This `downstream_imagenet` is isolated from pre-training codes. One can treat this `downstream_imagenet` as an independent codebase 🛠️. |
|
|
|
|
|
|
|
|
## Preparation for ImageNet-1k fine-tuning |
|
|
|
|
|
See [INSTALL.md](https://github.com/keyu-tian/SparK/blob/main/INSTALL.md) to prepare `pip` dependencies and the ImageNet dataset. |
|
|
|
|
|
**Note: for network definitions, we directly use `timm.models.ResNet` and [official ConvNeXt](https://github.com/facebookresearch/ConvNeXt/blob/048efcea897d999aed302f2639b6270aedf8d4c8/models/convnext.py).** |
|
|
|
|
|
|
|
|
## Fine-tuning on ImageNet-1k from pre-trained weights |
|
|
|
|
|
Run [/downstream_imagenet/main.py](/downstream_imagenet/main.py) via `torchrun`. |
|
|
**It is required to specify** the ImageNet data folder (`--data_path`), your experiment name & log dir (`--exp_name` and `--exp_dir`, automatically created if not exists), the model name (`--model`, valid choices see the keys of 'HP_DEFAULT_VALUES' in [/downstream_imagenet/arg.py line14](/downstream_imagenet/arg.py#L14)), and the pretrained weight file `--resume_from` to run fine-tuning. |
|
|
|
|
|
All the other configurations have their default values, listed in [/downstream_imagenet/arg.py#L13](/downstream_imagenet/arg.py#L13). |
|
|
You can overwrite any defaults by `--bs=1024` or something like that. |
|
|
|
|
|
|
|
|
Here is an example to pretrain a ConvNeXt-Small on an 8-GPU single machine: |
|
|
```shell script |
|
|
$ cd /path/to/SparK/downstream_imagenet |
|
|
$ torchrun --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr=localhost --master_port=<some_port> main.py \ |
|
|
--data_path=/path/to/imagenet --exp_name=<your_exp_name> --exp_dir=/path/to/logdir \ |
|
|
--model=convnext_small --resume_from=/some/path/to/convnextS_1kpretrained_official_style.pth |
|
|
``` |
|
|
|
|
|
For multiple machines, change the `--nnodes` and `--master_addr` to your configurations. E.g.: |
|
|
```shell script |
|
|
$ torchrun --nproc_per_node=8 --nnodes=<your_nnodes> --node_rank=<rank_starts_from_0> --master_address=<some_address> --master_port=<some_port> main.py \ |
|
|
... |
|
|
``` |
|
|
|
|
|
|
|
|
## Logging |
|
|
|
|
|
See files under `--exp_dir` to track your experiment: |
|
|
|
|
|
- `<model>_1kfinetuned_last.pth`: the latest model weights |
|
|
- `<model>_1kfinetuned_best.pth`: model weights with the highest acc |
|
|
- `<model>_1kfinetuned_best_ema.pth`: EMA weights with the highest acc |
|
|
- `finetune_log.txt`: records some important information such as: |
|
|
- `git_commit_id`: git version |
|
|
- `cmd`: all arguments passed to the script |
|
|
|
|
|
It also reports training loss/acc, best evaluation acc, and remaining time at each epoch. |
|
|
|
|
|
- `tensorboard_log/`: saves a lot of tensorboard logs, you can visualize accuracies, loss values, learning rates, gradient norms and more things via `tensorboard --logdir /path/to/this/tensorboard_log/ --port 23333`. |
|
|
|
|
|
## Resuming |
|
|
|
|
|
Use `--resume_from` again, like `--resume_from=path/to/<model>_1kfinetuned_last.pth`.
|
|
|
|
