You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
109 lines
4.9 KiB
109 lines
4.9 KiB
--- |
|
comments: true |
|
description: Learn more about MobileSAM, its implementation, comparison with the original SAM, and how to download and test it in the Ultralytics framework. Improve your mobile applications today. |
|
keywords: MobileSAM, Ultralytics, SAM, mobile applications, Arxiv, GPU, API, image encoder, mask decoder, model download, testing method |
|
--- |
|
|
|
 |
|
|
|
# Mobile Segment Anything (MobileSAM) |
|
|
|
The MobileSAM paper is now available on [arXiv](https://arxiv.org/pdf/2306.14289.pdf). |
|
|
|
A demonstration of MobileSAM running on a CPU can be accessed at this [demo link](https://huggingface.co/spaces/dhkim2810/MobileSAM). The performance on a Mac i5 CPU takes approximately 3 seconds. On the Hugging Face demo, the interface and lower-performance CPUs contribute to a slower response, but it continues to function effectively. |
|
|
|
MobileSAM is implemented in various projects including [Grounding-SAM](https://github.com/IDEA-Research/Grounded-Segment-Anything), [AnyLabeling](https://github.com/vietanhdev/anylabeling), and [Segment Anything in 3D](https://github.com/Jumpat/SegmentAnythingin3D). |
|
|
|
MobileSAM is trained on a single GPU with a 100k dataset (1% of the original images) in less than a day. The code for this training will be made available in the future. |
|
|
|
## Adapting from SAM to MobileSAM |
|
|
|
Since MobileSAM retains the same pipeline as the original SAM, we have incorporated the original's pre-processing, post-processing, and all other interfaces. Consequently, those currently using the original SAM can transition to MobileSAM with minimal effort. |
|
|
|
MobileSAM performs comparably to the original SAM and retains the same pipeline except for a change in the image encoder. Specifically, we replace the original heavyweight ViT-H encoder (632M) with a smaller Tiny-ViT (5M). On a single GPU, MobileSAM operates at about 12ms per image: 8ms on the image encoder and 4ms on the mask decoder. |
|
|
|
The following table provides a comparison of ViT-based image encoders: |
|
|
|
| Image Encoder | Original SAM | MobileSAM | |
|
|---------------|--------------|-----------| |
|
| Parameters | 611M | 5M | |
|
| Speed | 452ms | 8ms | |
|
|
|
Both the original SAM and MobileSAM utilize the same prompt-guided mask decoder: |
|
|
|
| Mask Decoder | Original SAM | MobileSAM | |
|
|--------------|--------------|-----------| |
|
| Parameters | 3.876M | 3.876M | |
|
| Speed | 4ms | 4ms | |
|
|
|
Here is the comparison of the whole pipeline: |
|
|
|
| Whole Pipeline (Enc+Dec) | Original SAM | MobileSAM | |
|
|--------------------------|--------------|-----------| |
|
| Parameters | 615M | 9.66M | |
|
| Speed | 456ms | 12ms | |
|
|
|
The performance of MobileSAM and the original SAM are demonstrated using both a point and a box as prompts. |
|
|
|
 |
|
|
|
 |
|
|
|
With its superior performance, MobileSAM is approximately 5 times smaller and 7 times faster than the current FastSAM. More details are available at the [MobileSAM project page](https://github.com/ChaoningZhang/MobileSAM). |
|
|
|
## Testing MobileSAM in Ultralytics |
|
|
|
Just like the original SAM, we offer a straightforward testing method in Ultralytics, including modes for both Point and Box prompts. |
|
|
|
### Model Download |
|
|
|
You can download the model [here](https://github.com/ChaoningZhang/MobileSAM/blob/master/weights/mobile_sam.pt). |
|
|
|
### Point Prompt |
|
|
|
!!! example "" |
|
|
|
=== "Python" |
|
```python |
|
from ultralytics import SAM |
|
|
|
# Load the model |
|
model = SAM('mobile_sam.pt') |
|
|
|
# Predict a segment based on a point prompt |
|
model.predict('ultralytics/assets/zidane.jpg', points=[900, 370], labels=[1]) |
|
``` |
|
|
|
### Box Prompt |
|
|
|
!!! example "" |
|
|
|
=== "Python" |
|
```python |
|
from ultralytics import SAM |
|
|
|
# Load the model |
|
model = SAM('mobile_sam.pt') |
|
|
|
# Predict a segment based on a box prompt |
|
model.predict('ultralytics/assets/zidane.jpg', bboxes=[439, 437, 524, 709]) |
|
``` |
|
|
|
We have implemented `MobileSAM` and `SAM` using the same API. For more usage information, please see the [SAM page](./sam.md). |
|
|
|
## Citations and Acknowledgements |
|
|
|
If you find MobileSAM useful in your research or development work, please consider citing our paper: |
|
|
|
!!! note "" |
|
|
|
=== "BibTeX" |
|
|
|
```bibtex |
|
@article{mobile_sam, |
|
title={Faster Segment Anything: Towards Lightweight SAM for Mobile Applications}, |
|
author={Zhang, Chaoning and Han, Dongshen and Qiao, Yu and Kim, Jung Uk and Bae, Sung Ho and Lee, Seungkyu and Hong, Choong Seon}, |
|
journal={arXiv preprint arXiv:2306.14289}, |
|
year={2023} |
|
} |
|
```
|
|
|