Ultralytics Assistant
4 months ago
committed by
GitHub
commit
1333517e70
49 changed files with 1080 additions and 612 deletions
@ -0,0 +1,49 @@ |
||||
# Ultralytics YOLO 🚀, AGPL-3.0 license |
||||
# Builds ultralytics/ultralytics:jetson-jetpack6 image on DockerHub https://hub.docker.com/r/ultralytics/ultralytics |
||||
# Supports JetPack6.x for YOLOv8 on Jetson AGX Orin, Orin NX and Orin Nano Series |
||||
|
||||
# Start FROM https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-jetpack |
||||
FROM nvcr.io/nvidia/l4t-jetpack:r36.3.0 |
||||
|
||||
# Downloads to user config dir |
||||
ADD https://github.com/ultralytics/assets/releases/download/v0.0.0/Arial.ttf \ |
||||
https://github.com/ultralytics/assets/releases/download/v0.0.0/Arial.Unicode.ttf \ |
||||
/root/.config/Ultralytics/ |
||||
|
||||
# Install dependencies |
||||
RUN apt update && \ |
||||
apt install --no-install-recommends -y git python3-pip libopenmpi-dev libopenblas-base libomp-dev |
||||
|
||||
# Create working directory |
||||
WORKDIR /ultralytics |
||||
|
||||
# Copy contents and assign permissions |
||||
COPY . . |
||||
RUN chown -R root:root . |
||||
ADD https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n.pt . |
||||
|
||||
# Download onnxruntime-gpu 1.18.0 from https://elinux.org/Jetson_Zoo and https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048 |
||||
ADD https://nvidia.box.com/shared/static/48dtuob7meiw6ebgfsfqakc9vse62sg4.whl onnxruntime_gpu-1.18.0-cp310-cp310-linux_aarch64.whl |
||||
|
||||
# Pip install onnxruntime-gpu, torch, torchvision and ultralytics |
||||
RUN python3 -m pip install --upgrade pip wheel |
||||
RUN pip install --no-cache-dir \ |
||||
onnxruntime_gpu-1.18.0-cp310-cp310-linux_aarch64.whl \ |
||||
https://github.com/ultralytics/assets/releases/download/v0.0.0/torch-2.3.0-cp310-cp310-linux_aarch64.whl \ |
||||
https://github.com/ultralytics/assets/releases/download/v0.0.0/torchvision-0.18.0a0+6043bc2-cp310-cp310-linux_aarch64.whl |
||||
RUN pip install --no-cache-dir -e ".[export]" |
||||
RUN rm *.whl |
||||
|
||||
# Usage Examples ------------------------------------------------------------------------------------------------------- |
||||
|
||||
# Build and Push |
||||
# t=ultralytics/ultralytics:latest-jetson-jetpack6 && sudo docker build --platform linux/arm64 -f docker/Dockerfile-jetson-jetpack6 -t $t . && sudo docker push $t |
||||
|
||||
# Run |
||||
# t=ultralytics/ultralytics:latest-jetson-jetpack6 && sudo docker run -it --ipc=host $t |
||||
|
||||
# Pull and Run |
||||
# t=ultralytics/ultralytics:latest-jetson-jetpack6 && sudo docker pull $t && sudo docker run -it --ipc=host $t |
||||
|
||||
# Pull and Run with NVIDIA runtime |
||||
# t=ultralytics/ultralytics:latest-jetson-jetpack6 && sudo docker pull $t && sudo docker run -it --ipc=host --runtime=nvidia $t |
||||
@ -0,0 +1,323 @@ |
||||
--- |
||||
comments: true |
||||
description: Dive into our detailed integration guide on using IBM Watson to train a YOLOv8 model. Uncover key features and step-by-step instructions on model training. |
||||
keywords: IBM Watsonx, IBM Watsonx AI, What is Watson?, IBM Watson Integration, IBM Watson Features, YOLOv8, Ultralytics, Model Training, GPU, TPU, cloud computing |
||||
--- |
||||
|
||||
# A Step-by-Step Guide to Training YOLOv8 Models with IBM Watsonx |
||||
|
||||
Nowadays, scalable [computer vision solutions](../guides/steps-of-a-cv-project.md) are becoming more common and transforming the way we handle visual data. A great example is IBM Watsonx, an advanced AI and data platform that simplifies the development, deployment, and management of AI models. It offers a complete suite for the entire AI lifecycle and seamless integration with IBM Cloud services. |
||||
|
||||
You can train [Ultralytics YOLOv8 models](https://github.com/ultralytics/ultralytics) using IBM Watsonx. It's a good option for enterprises interested in efficient [model training](../modes/train.md), fine-tuning for specific tasks, and improving [model performance](../guides/model-evaluation-insights.md) with robust tools and a user-friendly setup. In this guide, we'll walk you through the process of training YOLOv8 with IBM Watsonx, covering everything from setting up your environment to evaluating your trained models. Let's get started! |
||||
|
||||
## What is IBM Watsonx? |
||||
|
||||
[Watsonx](https://www.ibm.com/watsonx) is IBM's cloud-based platform designed for commercial generative AI and scientific data. IBM Watsonx's three components - watsonx.ai, watsonx.data, and watsonx.governance - come together to create an end-to-end, trustworthy AI platform that can accelerate AI projects aimed at solving business problems. It provides powerful tools for building, training, and [deploying machine learning models](../guides/model-deployment-options.md) and makes it easy to connect with various data sources. |
||||
|
||||
<p align="center"> |
||||
<img width="800" src="https://cdn.stackoverflow.co/images/jo7n4k8s/production/48b67e6aec41f89031a3426cbd1f78322e6776cb-8800x4950.jpg?auto=format" alt="Overview of IBM Watsonx"> |
||||
</p> |
||||
|
||||
Its user-friendly interface and collaborative capabilities streamline the development process and help with efficient model management and deployment. Whether for computer vision, predictive analytics, natural language processing, or other AI applications, IBM Watsonx provides the tools and support needed to drive innovation. |
||||
|
||||
## Key Features of IBM Watsonx |
||||
|
||||
IBM Watsonx is made of three main components: watsonx.ai, watsonx.data, and watsonx.governance. Each component offers features that cater to different aspects of AI and data management. Let's take a closer look at them. |
||||
|
||||
### [Watsonx.ai](https://www.ibm.com/products/watsonx-ai) |
||||
|
||||
Watsonx.ai provides powerful tools for AI development and offers access to IBM-supported custom models, third-party models like [Llama 3](https://www.ultralytics.com/blog/getting-to-know-metas-llama-3), and IBM's own Granite models. It includes the Prompt Lab for experimenting with AI prompts, the Tuning Studio for improving model performance with labeled data, and the Flows Engine for simplifying generative AI application development. Also, it offers comprehensive tools for automating the AI model lifecycle and connecting to various APIs and libraries. |
||||
|
||||
### [Watsonx.data](https://www.ibm.com/products/watsonx-data) |
||||
|
||||
Watsonx.data supports both cloud and on-premises deployments through the IBM Storage Fusion HCI integration. Its user-friendly console provides centralized access to data across environments and makes data exploration easy with common SQL. It optimizes workloads with efficient query engines like Presto and Spark, accelerates data insights with an AI-powered semantic layer, includes a vector database for AI relevance, and supports open data formats for easy sharing of analytics and AI data. |
||||
|
||||
### [Watsonx.governance](https://www.ibm.com/products/watsonx-governance) |
||||
|
||||
Watsonx.governance makes compliance easier by automatically identifying regulatory changes and enforcing policies. It links requirements to internal risk data and provides up-to-date AI factsheets. The platform helps manage risk with alerts and tools to detect issues such as [bias and drift](../guides/model-monitoring-and-maintenance.md). It also automates the monitoring and documentation of the AI lifecycle, organizes AI development with a model inventory, and enhances collaboration with user-friendly dashboards and reporting tools. |
||||
|
||||
## How to Train YOLOv8 Using IBM Watsonx |
||||
|
||||
You can use IBM Watsonx to accelerate your YOLOv8 model training workflow. |
||||
|
||||
### Prerequisites |
||||
|
||||
You need an [IBM Cloud account](https://cloud.ibm.com/registration) to create a [watsonx.ai](https://www.ibm.com/products/watsonx-ai) project, and you'll also need a [Kaggle](./kaggle.md) account to load the data set. |
||||
|
||||
### Step 1: Set Up Your Environment |
||||
|
||||
First, you'll need to set up an IBM account to use a Jupyter Notebook. Log in to [watsonx.ai](https://eu-de.dataplatform.cloud.ibm.com/registration/stepone?preselect_region=true) using your IBM Cloud account. |
||||
|
||||
Then, create a [watsonx.ai project](https://www.ibm.com/docs/en/watsonx/saas?topic=projects-creating-project), and a [Jupyter Notebook](https://www.ibm.com/docs/en/watsonx/saas?topic=editor-creating-managing-notebooks). |
||||
|
||||
Once you do so, a notebook environment will open for you to load your data set. You can use the code from this tutorial to tackle a simple object detection model training task. |
||||
|
||||
### Step 2: Install and Import Relevant Libraries |
||||
|
||||
Next, you can install and import the necessary Python libraries. |
||||
|
||||
!!! Tip "Installation" |
||||
|
||||
=== "CLI" |
||||
|
||||
```bash |
||||
# Install the required packages |
||||
pip install torch torchvision torchaudio |
||||
pip install opencv-contrib-python-headless |
||||
pip install ultralytics==8.0.196 |
||||
``` |
||||
|
||||
For detailed instructions and best practices related to the installation process, check our [Ultralytics Installation guide](../quickstart.md). While installing the required packages for YOLOv8, if you encounter any difficulties, consult our [Common Issues guide](../guides/yolo-common-issues.md) for solutions and tips. |
||||
|
||||
Then, you can import the needed packages. |
||||
|
||||
!!! Example "Import Relevant Libraries" |
||||
|
||||
=== "Python" |
||||
|
||||
```python |
||||
# Import ultralytics |
||||
import ultralytics |
||||
|
||||
ultralytics.checks() |
||||
|
||||
# Import packages to retrieve and display image files |
||||
``` |
||||
|
||||
### Step 3: Load the Data |
||||
|
||||
For this tutorial, we will use a [marine litter dataset](https://www.kaggle.com/datasets/atiqishrak/trash-dataset-icra19) available on Kaggle. With this dataset, we will custom-train a YOLOv8 model to detect and classify litter and biological objects in underwater images. |
||||
|
||||
We can load the dataset directly into the notebook using the Kaggle API. First, create a free Kaggle account. Once you have created an account, you'll need to generate an API key. Directions for generating your key can be found in the [Kaggle API documentation](https://github.com/Kaggle/kaggle-api/blob/main/docs/README.md) under the section "API credentials". |
||||
|
||||
Copy and paste your Kaggle username and API key into the following code. Then run the code to install the API and load the dataset into Watsonx. |
||||
|
||||
!!! Tip "Installation" |
||||
|
||||
=== "CLI" |
||||
|
||||
```bash |
||||
# Install kaggle |
||||
pip install kaggle |
||||
``` |
||||
|
||||
After installing Kaggle, we can load the dataset into Watsonx. |
||||
|
||||
!!! Example "Load the Data" |
||||
|
||||
=== "Python" |
||||
|
||||
```python |
||||
# Replace "username" string with your username |
||||
os.environ["KAGGLE_USERNAME"] = "username" |
||||
# Replace "apiKey" string with your key |
||||
os.environ["KAGGLE_KEY"] = "apiKey" |
||||
|
||||
# Load dataset |
||||
!kaggle datasets download atiqishrak/trash-dataset-icra19 --unzip |
||||
|
||||
# Store working directory path as work_dir |
||||
work_dir = os.getcwd() |
||||
|
||||
# Print work_dir path |
||||
print(os.getcwd()) |
||||
|
||||
# Print work_dir contents |
||||
print(os.listdir(f"{work_dir}")) |
||||
|
||||
# Print trash_ICRA19 subdirectory contents |
||||
print(os.listdir(f"{work_dir}/trash_ICRA19")) |
||||
``` |
||||
|
||||
After loading the dataset, we printed and saved our working directory. We have also printed the contents of our working directory to confirm the "trash_ICRA19" data set was loaded properly. |
||||
|
||||
If you see "trash_ICRA19" among the directory's contents, then it has loaded successfully. You should see three files/folders: a `config.yaml` file, a `videos_for_testing` directory, and a `dataset` directory. We will ignore the `videos_for_testing` directory, so feel free to delete it. |
||||
|
||||
We will use the config.yaml file and the contents of the dataset directory to train our object detection model. Here is a sample image from our marine litter data set. |
||||
|
||||
<p align="center"> |
||||
<img width="400" src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/sQy6asArOJ2weUuQ_POiVA.jpg" alt="Marine Litter with Bounding Box"> |
||||
</p> |
||||
|
||||
### Step 4: Preprocess the Data |
||||
|
||||
Fortunately, all labels in the marine litter data set are already formatted as YOLO .txt files. However, we need to rearrange the structure of the image and label directories in order to help our model process the image and labels. Right now, our loaded data set directory follows this structure: |
||||
|
||||
<p align="center"> |
||||
<img width="400" src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/VfgvRT7vdgkeTQNqVMs_CQ.png" alt="Loaded Dataset Directory"> |
||||
</p> |
||||
|
||||
But, YOLO models by default require separate images and labels in subdirectories within the train/val/test split. We need to reorganize the directory into the following structure: |
||||
|
||||
<p align="center"> |
||||
<img width="400" src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/uUk1YopS94mytGaCav3ZaQ.png" alt="Yolo Directory Structure"> |
||||
</p> |
||||
|
||||
To reorganize the data set directory, we can run the following script: |
||||
|
||||
!!! Example "Preprocess the Data" |
||||
|
||||
=== "Python" |
||||
|
||||
```python |
||||
# Function to reorganize dir |
||||
def organize_files(directory): |
||||
for subdir in ["train", "test", "val"]: |
||||
subdir_path = os.path.join(directory, subdir) |
||||
if not os.path.exists(subdir_path): |
||||
continue |
||||
|
||||
images_dir = os.path.join(subdir_path, "images") |
||||
labels_dir = os.path.join(subdir_path, "labels") |
||||
|
||||
# Create image and label subdirs if non-existent |
||||
os.makedirs(images_dir, exist_ok=True) |
||||
os.makedirs(labels_dir, exist_ok=True) |
||||
|
||||
# Move images and labels to respective subdirs |
||||
for filename in os.listdir(subdir_path): |
||||
if filename.endswith(".txt"): |
||||
shutil.move(os.path.join(subdir_path, filename), os.path.join(labels_dir, filename)) |
||||
elif filename.endswith(".jpg") or filename.endswith(".png") or filename.endswith(".jpeg"): |
||||
shutil.move(os.path.join(subdir_path, filename), os.path.join(images_dir, filename)) |
||||
# Delete .xml files |
||||
elif filename.endswith(".xml"): |
||||
os.remove(os.path.join(subdir_path, filename)) |
||||
|
||||
|
||||
if __name__ == "__main__": |
||||
directory = f"{work_dir}/trash_ICRA19/dataset" |
||||
organize_files(directory) |
||||
``` |
||||
|
||||
Next, we need to modify the .yaml file for the data set. This is the setup we will use in our .yaml file. Class ID numbers start from 0: |
||||
|
||||
```yaml |
||||
path: /path/to/dataset/directory # root directory for dataset |
||||
train: train/images # train images subdirectory |
||||
val: train/images # validation images subdirectory |
||||
test: test/images # test images subdirectory |
||||
|
||||
# Classes |
||||
names: |
||||
0: plastic |
||||
1: bio |
||||
2: rov |
||||
``` |
||||
|
||||
Run the following script to delete the current contents of config.yaml and replace it with the above contents that reflect our new data set directory structure. Be certain to replace the work_dir portion of the root directory path in line 4 with your own working directory path we retrieved earlier. Leave the train, val, and test subdirectory definitions. Also, do not change {work_dir} in line 23 of the code. |
||||
|
||||
!!! Example "Edit the .yaml File" |
||||
|
||||
=== "Python" |
||||
|
||||
```python |
||||
# Contents of new confg.yaml file |
||||
def update_yaml_file(file_path): |
||||
data = { |
||||
"path": "work_dir/trash_ICRA19/dataset", |
||||
"train": "train/images", |
||||
"val": "train/images", |
||||
"test": "test/images", |
||||
"names": {0: "plastic", 1: "bio", 2: "rov"}, |
||||
} |
||||
|
||||
# Ensures the "names" list appears after the sub/directories |
||||
names_data = data.pop("names") |
||||
with open(file_path, "w") as yaml_file: |
||||
yaml.dump(data, yaml_file) |
||||
yaml_file.write("\n") |
||||
yaml.dump({"names": names_data}, yaml_file) |
||||
|
||||
|
||||
if __name__ == "__main__": |
||||
file_path = f"{work_dir}/trash_ICRA19/config.yaml" # .yaml file path |
||||
update_yaml_file(file_path) |
||||
print(f"{file_path} updated successfully.") |
||||
``` |
||||
|
||||
### Step 5: Train the YOLOv8 model |
||||
|
||||
Run the following command-line code to fine tune a pretrained default YOLOv8 model. |
||||
|
||||
!!! Example "Train the YOLOv8 model" |
||||
|
||||
=== "CLI" |
||||
|
||||
```bash |
||||
!yolo task=detect mode=train data={work_dir}/trash_ICRA19/config.yaml model=yolov8s.pt epochs=2 batch=32 lr0=.04 plots=True |
||||
``` |
||||
|
||||
Here's a closer look at the parameters in the model training command: |
||||
|
||||
- **task**: It specifies the computer vision task for which you are using the specified YOLO model and data set. |
||||
- **mode**: Denotes the purpose for which you are loading the specified model and data. Since we are training a model, it is set to "train." Later, when we test our model's performance, we will set it to "predict." |
||||
- **epochs**: This delimits the number of times YOLOv8 will pass through our entire data set. |
||||
- **batch**: The numerical value stipulates the training batch sizes. Batches are the number of images a model processes before it updates its parameters. |
||||
- **lr0**: Specifies the model's initial learning rate. |
||||
- **plots**: Directs YOLO to generate and save plots of our model's training and evaluation metrics. |
||||
|
||||
For a detailed understanding of the model training process and best practices, refer to the [YOLOv8 Model Training guide](../modes/train.md). This guide will help you get the most out of your experiments and ensure you're using YOLOv8 effectively. |
||||
|
||||
### Step 6: Test the Model |
||||
|
||||
We can now run inference to test the performance of our fine-tuned model: |
||||
|
||||
!!! Example "Test the YOLOv8 model" |
||||
|
||||
=== "CLI" |
||||
|
||||
```bash |
||||
!yolo task=detect mode=predict source={work_dir}/trash_ICRA19/dataset/test/images model={work_dir}/runs/detect/train/weights/best.pt conf=0.5 iou=.5 save=True save_txt=True |
||||
``` |
||||
|
||||
This brief script generates predicted labels for each image in our test set, as well as new output image files that overlay the predicted bounding box atop the original image. |
||||
|
||||
Predicted .txt labels for each image are saved via the `save_txt=True` argument and the output images with bounding box overlays are generated through the `save=True` argument. |
||||
The parameter `conf=0.5` informs the model to ignore all predictions with a confidence level of less than 50%. |
||||
|
||||
Lastly, `iou=.5` directs the model to ignore boxes in the same class with an overlap of 50% or greater. It helps to reduce potential duplicate boxes generated for the same object. |
||||
we can load the images with predicted bounding box overlays to view how our model performs on a handful of images. |
||||
|
||||
!!! Example "Display Predictions" |
||||
|
||||
=== "Python" |
||||
|
||||
```python |
||||
# Show the first ten images from the preceding prediction task |
||||
for pred_dir in glob.glob(f"{work_dir}/runs/detect/predict/*.jpg")[:10]: |
||||
img = Image.open(pred_dir) |
||||
display(img) |
||||
``` |
||||
|
||||
The code above displays ten images from the test set with their predicted bounding boxes, accompanied by class name labels and confidence levels. |
||||
|
||||
### Step 7: Evaluate the Model |
||||
|
||||
We can produce visualizations of the model's precision and recall for each class. These visualizations are saved in the home directory, under the train folder. The precision score is displayed in the P_curve.png: |
||||
|
||||
<p align="center"> |
||||
<img width="800" src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/EvQpqt4D6VI2And1T86Fww.png" alt="Precision Confidence Curve"> |
||||
</p> |
||||
|
||||
The graph shows an exponential increase in precision as the model's confidence level for predictions increases. However, the model precision has not yet leveled out at a certain confidence level after two epochs. |
||||
|
||||
The recall graph (R_curve.png) displays an inverse trend: |
||||
|
||||
<p align="center"> |
||||
<img width="800" src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/NS0pQDHuEWM-WlpBpxTydw.png" alt="Recall Confidence Curve"> |
||||
</p> |
||||
|
||||
Unlike precision, recall moves in the opposite direction, showing greater recall with lower confidence instances and lower recall with higher confidence instances. This is an apt example of the trade-off in precision and recall for classification models. |
||||
|
||||
### Step 8: Calculating Intersection Over Union |
||||
|
||||
You can measure the prediction accuracy by calculating the IoU between a predicted bounding box and a ground truth bounding box for the same object. Check out [IBM's tutorial on training YOLOv8](https://developer.ibm.com/tutorials/awb-train-yolo-object-detection-model-in-python/) for more details. |
||||
|
||||
## Summary |
||||
|
||||
We explored IBM Watsonx key features, and how to train a YOLOv8 model using IBM Watsonx. We also saw how IBM Watsonx can enhance your AI workflows with advanced tools for model building, data management, and compliance. |
||||
|
||||
For further details on usage, visit [IBM Watsonx official documentation](https://www.ibm.com/watsonx). |
||||
|
||||
Also, be sure to check out the [Ultralytics integration guide page](./index.md), to learn more about different exciting integrations. |
||||
@ -0,0 +1,110 @@ |
||||
--- |
||||
comments: true |
||||
description: Explore our integration guide that explains how you can use JupyterLab to train a YOLOv8 model. We'll also cover key features and tips for common issues. |
||||
keywords: JupyterLab, What is JupyterLab, How to Use JupyterLab, JupyterLab How to Use, YOLOv8, Ultralytics, Model Training, GPU, TPU, cloud computing |
||||
--- |
||||
|
||||
# A Guide on How to Use JupyterLab to Train Your YOLOv8 Models |
||||
|
||||
Building deep learning models can be tough, especially when you don't have the right tools or environment to work with. If you are facing this issue, JupyterLab might be the right solution for you. JupyterLab is a user-friendly, web-based platform that makes coding more flexible and interactive. You can use it to handle big datasets, create complex models, and even collaborate with others, all in one place. |
||||
|
||||
You can use JupyterLab to [work on projects](../guides/steps-of-a-cv-project.md) related to [Ultralytics YOLOv8 models](https://github.com/ultralytics/ultralytics). JupyterLab is a great option for efficient model development and experimentation. It makes it easy to start experimenting with and [training YOLOv8 models](../modes/train.md) right from your computer. Let's dive deeper into JupyterLab, its key features, and how you can use it to train YOLOv8 models. |
||||
|
||||
## What is JupyterLab? |
||||
|
||||
JupyterLab is an open-source web-based platform designed for working with Jupyter notebooks, code, and data. It's an upgrade from the traditional Jupyter Notebook interface that provides a more versatile and powerful user experience. |
||||
|
||||
JupyterLab allows you to work with notebooks, text editors, terminals, and other tools all in one place. Its flexible design lets you organize your workspace to fit your needs and makes it easier to perform tasks like data analysis, visualization, and machine learning. JupyterLab also supports real-time collaboration, making it ideal for team projects in research and data science. |
||||
|
||||
## Key Features of JupyterLab |
||||
|
||||
Here are some of the key features that make JupyterLab a great option for model development and experimentation: |
||||
|
||||
- **All-in-One Workspace**: JupyterLab is a one-stop shop for all your data science needs. Unlike the classic Jupyter Notebook, which had separate interfaces for text editing, terminal access, and notebooks, JupyterLab integrates all these features into a single, cohesive environment. You can view and edit various file formats, including JPEG, PDF, and CSV, directly within JupyterLab. An all-in-one workspace lets you access everything you need at your fingertips, streamlining your workflow and saving you time. |
||||
- **Flexible Layouts**: One of JupyterLab's standout features is its flexible layout. You can drag, drop, and resize tabs to create a personalized layout that helps you work more efficiently. The collapsible left sidebar keeps essential tabs like the file browser, running kernels, and command palette within easy reach. You can have multiple windows open at once, allowing you to multitask and manage your projects more effectively. |
||||
- **Interactive Code Consoles**: Code consoles in JupyterLab provide an interactive space to test out snippets of code or functions. They also serve as a log of computations made within a notebook. Creating a new console for a notebook and viewing all kernel activity is straightforward. This feature is especially useful when you're experimenting with new ideas or troubleshooting issues in your code. |
||||
- **Markdown Preview**: Working with Markdown files is more efficient in JupyterLab, thanks to its simultaneous preview feature. As you write or edit your Markdown file, you can see the formatted output in real-time. It makes it easier to double-check that your documentation looks perfect, saving you from having to switch back and forth between editing and preview modes. |
||||
- **Run Code from Text Files**: If you're sharing a text file with code, JupyterLab makes it easy to run it directly within the platform. You can highlight the code and press Shift + Enter to execute it. It is great for verifying code snippets quickly and helps guarantee that the code you share is functional and error-free. |
||||
|
||||
## Why Should You Use JupyterLab for Your YOLOv8 Projects? |
||||
|
||||
There are multiple platforms for developing and evaluating machine learning models, so what makes JupyterLab stand out? Let's explore some of the unique aspects that JupyterLab offers for your machine-learning projects: |
||||
|
||||
- **Easy Cell Management**: Managing cells in JupyterLab is a breeze. Instead of the cumbersome cut-and-paste method, you can simply drag and drop cells to rearrange them. |
||||
- **Cross-Notebook Cell Copying**: JupyterLab makes it simple to copy cells between different notebooks. You can drag and drop cells from one notebook to another. |
||||
- **Easy Switch to Classic Notebook View**: For those who miss the classic Jupyter Notebook interface, JupyterLab offers an easy switch back. Simply replace `/lab` in the URL with `/tree` to return to the familiar notebook view. |
||||
- **Multiple Views**: JupyterLab supports multiple views of the same notebook, which is particularly useful for long notebooks. You can open different sections side-by-side for comparison or exploration, and any changes made in one view are reflected in the other. |
||||
- **Customizable Themes**: JupyterLab includes a built-in Dark theme for the notebook, which is perfect for late-night coding sessions. There are also themes available for the text editor and terminal, allowing you to customize the appearance of your entire workspace. |
||||
|
||||
## Common Issues While Working with JupyterLab |
||||
|
||||
When working with Kaggle, you might come across some common issues. Here are some tips to help you navigate the platform smoothly: |
||||
|
||||
- **Managing Kernels**: Kernels are crucial because they manage the connection between the code you write in JupyterLab and the environment where it runs. They can also access and share data between notebooks. When you close a Jupyter Notebook, the kernel might still be running because other notebooks could be using it. If you want to completely shut down a kernel, you can select it, right-click, and choose "Shut Down Kernel" from the pop-up menu. |
||||
- **Installing Python Packages**: Sometimes, you might need additional Python packages that aren't pre-installed on the server. You can easily install these packages in your home directory or a virtual environment by using the command `python -m pip install package-name`. To see all installed packages, use `python -m pip list`. |
||||
- **Deploying Flask/FastAPI API to Posit Connect**: You can deploy your Flask and FastAPI APIs to Posit Connect using the [rsconnect-python](https://docs.posit.co/rsconnect-python/) package from the terminal. Doing so makes it easier to integrate your web applications with JupyterLab and share them with others. |
||||
- **Installing JupyterLab Extensions**: JupyterLab supports various extensions to enhance functionality. You can install and customize these extensions to suit your needs. For detailed instructions, refer to [JupyterLab Extensions Guide](https://jupyterlab.readthedocs.io/en/latest/user/extensions.html) for more information. |
||||
- **Using Multiple Versions of Python**: If you need to work with different versions of Python, you can use Jupyter kernels configured with different Python versions. |
||||
|
||||
## How to Use JupyterLab to Try Out YOLOv8 |
||||
|
||||
JupyterLab makes it easy to experiment with YOLOv8. To get started, follow these simple steps. |
||||
|
||||
### Step 1: Install JupyterLab |
||||
|
||||
First, you need to install JupyterLab. Open your terminal and run the command: |
||||
|
||||
!!! Tip "Installation" |
||||
|
||||
=== "CLI" |
||||
|
||||
```bash |
||||
# Install the required package for JupyterLab |
||||
pip install jupyterlab |
||||
``` |
||||
|
||||
### Step 2: Download the YOLOv8 Tutorial Notebook |
||||
|
||||
Next, download the [tutorial.ipynb](https://github.com/ultralytics/ultralytics/blob/main/examples/tutorial.ipynb) file from the Ultralytics GitHub repository. Save this file to any directory on your local machine. |
||||
|
||||
### Step 3: Launch JupyterLab |
||||
|
||||
Navigate to the directory where you saved the notebook file using your terminal. Then, run the following command to launch JupyterLab: |
||||
|
||||
!!! Example "Usage" |
||||
|
||||
=== "CLI" |
||||
|
||||
```bash |
||||
jupyter lab |
||||
``` |
||||
|
||||
Once you've run this command, it will open JupyterLab in your default web browser, as shown below. |
||||
|
||||
 |
||||
|
||||
### Step 4: Start Experimenting |
||||
|
||||
In JupyterLab, open the tutorial.ipynb notebook. You can now start running the cells to explore and experiment with YOLOv8. |
||||
|
||||
 |
||||
|
||||
JupyterLab's interactive environment allows you to modify code, visualize outputs, and document your findings all in one place. You can try out different configurations and understand how YOLOv8 works. |
||||
|
||||
For a detailed understanding of the model training process and best practices, refer to the [YOLOv8 Model Training guide](../modes/train.md). This guide will help you get the most out of your experiments and ensure you're using YOLOv8 effectively. |
||||
|
||||
## Keep Learning about Jupyterlab |
||||
|
||||
If you're excited to learn more about JupyterLab, here are some great resources to get you started: |
||||
|
||||
- [**JupyterLab Documentation**](https://jupyterlab.readthedocs.io/en/stable/getting_started/starting.html): Dive into the official JupyterLab Documentation to explore its features and capabilities. It's a great way to understand how to use this powerful tool to its fullest potential. |
||||
- [**Try It With Binder**](https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/HEAD?urlpath=lab/tree/demo): Experiment with JupyterLab without installing anything by using Binder, which lets you launch a live JupyterLab instance directly in your browser. It's a great way to start experimenting immediately. |
||||
- [**Installation Guide**](https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html): For a step-by-step guide on installing JupyterLab on your local machine, check out the installation guide. |
||||
|
||||
## Summary |
||||
|
||||
We've explored how JupyterLab can be a powerful tool for experimenting with Ultralytics YOLOv8 models. Using its flexible and interactive environment, you can easily set up JupyterLab on your local machine and start working with YOLOv8. JupyterLab makes it simple to [train](../guides/model-training-tips.md) and [evaluate](../guides/model-testing.md) your models, visualize outputs, and [document your findings](../guides/model-monitoring-and-maintenance.md) all in one place. |
||||
|
||||
For more details, visit the [JupyterLab FAQ Page](https://jupyterlab.readthedocs.io/en/stable/getting_started/faq.html). |
||||
|
||||
Interested in more YOLOv8 integrations? Check out the [Ultralytics integration guide](./index.md) to explore additional tools and capabilities for your machine learning projects. |
||||
@ -0,0 +1,159 @@ |
||||
# Ultralytics YOLO 🚀, AGPL-3.0 license |
||||
|
||||
import concurrent.futures |
||||
import statistics |
||||
import time |
||||
from typing import List, Optional, Tuple |
||||
|
||||
import requests |
||||
|
||||
|
||||
class GCPRegions: |
||||
""" |
||||
A class for managing and analyzing Google Cloud Platform (GCP) regions. |
||||
|
||||
This class provides functionality to initialize, categorize, and analyze GCP regions based on their |
||||
geographical location, tier classification, and network latency. |
||||
|
||||
Attributes: |
||||
regions (Dict[str, Tuple[int, str, str]]): A dictionary of GCP regions with their tier, city, and country. |
||||
|
||||
Methods: |
||||
tier1: Returns a list of tier 1 GCP regions. |
||||
tier2: Returns a list of tier 2 GCP regions. |
||||
lowest_latency: Determines the GCP region(s) with the lowest network latency. |
||||
|
||||
Examples: |
||||
>>> from ultralytics.hub.google import GCPRegions |
||||
>>> regions = GCPRegions() |
||||
>>> lowest_latency_region = regions.lowest_latency(verbose=True, attempts=3) |
||||
>>> print(f"Lowest latency region: {lowest_latency_region[0][0]}") |
||||
""" |
||||
|
||||
def __init__(self): |
||||
"""Initializes the GCPRegions class with predefined Google Cloud Platform regions and their details.""" |
||||
self.regions = { |
||||
"asia-east1": (1, "Taiwan", "China"), |
||||
"asia-east2": (2, "Hong Kong", "China"), |
||||
"asia-northeast1": (1, "Tokyo", "Japan"), |
||||
"asia-northeast2": (1, "Osaka", "Japan"), |
||||
"asia-northeast3": (2, "Seoul", "South Korea"), |
||||
"asia-south1": (2, "Mumbai", "India"), |
||||
"asia-south2": (2, "Delhi", "India"), |
||||
"asia-southeast1": (2, "Jurong West", "Singapore"), |
||||
"asia-southeast2": (2, "Jakarta", "Indonesia"), |
||||
"australia-southeast1": (2, "Sydney", "Australia"), |
||||
"australia-southeast2": (2, "Melbourne", "Australia"), |
||||
"europe-central2": (2, "Warsaw", "Poland"), |
||||
"europe-north1": (1, "Hamina", "Finland"), |
||||
"europe-southwest1": (1, "Madrid", "Spain"), |
||||
"europe-west1": (1, "St. Ghislain", "Belgium"), |
||||
"europe-west10": (2, "Berlin", "Germany"), |
||||
"europe-west12": (2, "Turin", "Italy"), |
||||
"europe-west2": (2, "London", "United Kingdom"), |
||||
"europe-west3": (2, "Frankfurt", "Germany"), |
||||
"europe-west4": (1, "Eemshaven", "Netherlands"), |
||||
"europe-west6": (2, "Zurich", "Switzerland"), |
||||
"europe-west8": (1, "Milan", "Italy"), |
||||
"europe-west9": (1, "Paris", "France"), |
||||
"me-central1": (2, "Doha", "Qatar"), |
||||
"me-west1": (1, "Tel Aviv", "Israel"), |
||||
"northamerica-northeast1": (2, "Montreal", "Canada"), |
||||
"northamerica-northeast2": (2, "Toronto", "Canada"), |
||||
"southamerica-east1": (2, "São Paulo", "Brazil"), |
||||
"southamerica-west1": (2, "Santiago", "Chile"), |
||||
"us-central1": (1, "Iowa", "United States"), |
||||
"us-east1": (1, "South Carolina", "United States"), |
||||
"us-east4": (1, "Northern Virginia", "United States"), |
||||
"us-east5": (1, "Columbus", "United States"), |
||||
"us-south1": (1, "Dallas", "United States"), |
||||
"us-west1": (1, "Oregon", "United States"), |
||||
"us-west2": (2, "Los Angeles", "United States"), |
||||
"us-west3": (2, "Salt Lake City", "United States"), |
||||
"us-west4": (2, "Las Vegas", "United States"), |
||||
} |
||||
|
||||
def tier1(self) -> List[str]: |
||||
"""Returns a list of GCP regions classified as tier 1 based on predefined criteria.""" |
||||
return [region for region, info in self.regions.items() if info[0] == 1] |
||||
|
||||
def tier2(self) -> List[str]: |
||||
"""Returns a list of GCP regions classified as tier 2 based on predefined criteria.""" |
||||
return [region for region, info in self.regions.items() if info[0] == 2] |
||||
|
||||
@staticmethod |
||||
def _ping_region(region: str, attempts: int = 1) -> Tuple[str, float, float, float, float]: |
||||
"""Pings a specified GCP region and returns latency statistics: mean, min, max, and standard deviation.""" |
||||
url = f"https://{region}-docker.pkg.dev" |
||||
latencies = [] |
||||
for _ in range(attempts): |
||||
try: |
||||
start_time = time.time() |
||||
_ = requests.head(url, timeout=5) |
||||
latency = (time.time() - start_time) * 1000 # convert latency to milliseconds |
||||
if latency != float("inf"): |
||||
latencies.append(latency) |
||||
except requests.RequestException: |
||||
pass |
||||
if not latencies: |
||||
return region, float("inf"), float("inf"), float("inf"), float("inf") |
||||
|
||||
std_dev = statistics.stdev(latencies) if len(latencies) > 1 else 0 |
||||
return region, statistics.mean(latencies), std_dev, min(latencies), max(latencies) |
||||
|
||||
def lowest_latency( |
||||
self, |
||||
top: int = 1, |
||||
verbose: bool = False, |
||||
tier: Optional[int] = None, |
||||
attempts: int = 1, |
||||
) -> List[Tuple[str, float, float, float, float]]: |
||||
""" |
||||
Determines the GCP regions with the lowest latency based on ping tests. |
||||
|
||||
Args: |
||||
top (int): Number of top regions to return. |
||||
verbose (bool): If True, prints detailed latency information for all tested regions. |
||||
tier (int | None): Filter regions by tier (1 or 2). If None, all regions are tested. |
||||
attempts (int): Number of ping attempts per region. |
||||
|
||||
Returns: |
||||
(List[Tuple[str, float, float, float, float]]): List of tuples containing region information and |
||||
latency statistics. Each tuple contains (region, mean_latency, std_dev, min_latency, max_latency). |
||||
|
||||
Examples: |
||||
>>> regions = GCPRegions() |
||||
>>> results = regions.lowest_latency(top=3, verbose=True, tier=1, attempts=2) |
||||
>>> print(results[0][0]) # Print the name of the lowest latency region |
||||
""" |
||||
if verbose: |
||||
print(f"Testing GCP regions for latency (with {attempts} {'retry' if attempts == 1 else 'attempts'})...") |
||||
|
||||
regions_to_test = [k for k, v in self.regions.items() if v[0] == tier] if tier else list(self.regions.keys()) |
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=50) as executor: |
||||
results = list(executor.map(lambda r: self._ping_region(r, attempts), regions_to_test)) |
||||
|
||||
sorted_results = sorted(results, key=lambda x: x[1]) |
||||
|
||||
if verbose: |
||||
print(f"{'Region':<25} {'Location':<35} {'Tier':<5} {'Latency (ms)'}") |
||||
for region, mean, std, min_, max_ in sorted_results: |
||||
tier, city, country = self.regions[region] |
||||
location = f"{city}, {country}" |
||||
if mean == float("inf"): |
||||
print(f"{region:<25} {location:<35} {tier:<5} {'Timeout'}") |
||||
else: |
||||
print(f"{region:<25} {location:<35} {tier:<5} {mean:.0f} ± {std:.0f} ({min_:.0f} - {max_:.0f})") |
||||
print(f"\nLowest latency region{'s' if top > 1 else ''}:") |
||||
for region, mean, std, min_, max_ in sorted_results[:top]: |
||||
tier, city, country = self.regions[region] |
||||
location = f"{city}, {country}" |
||||
print(f"{region} ({location}, {mean:.0f} ± {std:.0f} ms ({min_:.0f} - {max_:.0f}))") |
||||
|
||||
return sorted_results[:top] |
||||
|
||||
|
||||
# Usage example |
||||
if __name__ == "__main__": |
||||
regions = GCPRegions() |
||||
top_3_latency_tier1 = regions.lowest_latency(top=3, verbose=True, tier=1, attempts=3) |
||||
@ -1,352 +0,0 @@ |
||||
# Ultralytics YOLO 🚀, AGPL-3.0 license |
||||
|
||||
import os |
||||
from pathlib import Path |
||||
|
||||
import cv2 |
||||
import numpy as np |
||||
import torch |
||||
from PIL import Image |
||||
from torch import Tensor |
||||
|
||||
from ultralytics.utils import TQDM, checks |
||||
|
||||
|
||||
class FastSAMPrompt: |
||||
""" |
||||
Fast Segment Anything Model class for image annotation and visualization. |
||||
|
||||
Attributes: |
||||
device (str): Computing device ('cuda' or 'cpu'). |
||||
results: Object detection or segmentation results. |
||||
source: Source image or image path. |
||||
clip: CLIP model for linear assignment. |
||||
""" |
||||
|
||||
def __init__(self, source, results, device="cuda") -> None: |
||||
"""Initializes FastSAMPrompt with given source, results and device, and assigns clip for linear assignment.""" |
||||
if isinstance(source, (str, Path)) and os.path.isdir(source): |
||||
raise ValueError("FastSAM only accepts image paths and PIL Image sources, not directories.") |
||||
self.device = device |
||||
self.results = results |
||||
self.source = source |
||||
|
||||
# Import and assign clip |
||||
try: |
||||
import clip |
||||
except ImportError: |
||||
checks.check_requirements("git+https://github.com/ultralytics/CLIP.git") |
||||
import clip |
||||
self.clip = clip |
||||
|
||||
@staticmethod |
||||
def _segment_image(image, bbox): |
||||
"""Segments the given image according to the provided bounding box coordinates.""" |
||||
image_array = np.array(image) |
||||
segmented_image_array = np.zeros_like(image_array) |
||||
x1, y1, x2, y2 = bbox |
||||
segmented_image_array[y1:y2, x1:x2] = image_array[y1:y2, x1:x2] |
||||
segmented_image = Image.fromarray(segmented_image_array) |
||||
black_image = Image.new("RGB", image.size, (255, 255, 255)) |
||||
# transparency_mask = np.zeros_like((), dtype=np.uint8) |
||||
transparency_mask = np.zeros((image_array.shape[0], image_array.shape[1]), dtype=np.uint8) |
||||
transparency_mask[y1:y2, x1:x2] = 255 |
||||
transparency_mask_image = Image.fromarray(transparency_mask, mode="L") |
||||
black_image.paste(segmented_image, mask=transparency_mask_image) |
||||
return black_image |
||||
|
||||
@staticmethod |
||||
def _format_results(result, filter=0): |
||||
"""Formats detection results into list of annotations each containing ID, segmentation, bounding box, score and |
||||
area. |
||||
""" |
||||
annotations = [] |
||||
n = len(result.masks.data) if result.masks is not None else 0 |
||||
for i in range(n): |
||||
mask = result.masks.data[i] == 1.0 |
||||
if torch.sum(mask) >= filter: |
||||
annotation = { |
||||
"id": i, |
||||
"segmentation": mask.cpu().numpy(), |
||||
"bbox": result.boxes.data[i], |
||||
"score": result.boxes.conf[i], |
||||
} |
||||
annotation["area"] = annotation["segmentation"].sum() |
||||
annotations.append(annotation) |
||||
return annotations |
||||
|
||||
@staticmethod |
||||
def _get_bbox_from_mask(mask): |
||||
"""Applies morphological transformations to the mask, displays it, and if with_contours is True, draws |
||||
contours. |
||||
""" |
||||
mask = mask.astype(np.uint8) |
||||
contours, hierarchy = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) |
||||
x1, y1, w, h = cv2.boundingRect(contours[0]) |
||||
x2, y2 = x1 + w, y1 + h |
||||
if len(contours) > 1: |
||||
for b in contours: |
||||
x_t, y_t, w_t, h_t = cv2.boundingRect(b) |
||||
x1 = min(x1, x_t) |
||||
y1 = min(y1, y_t) |
||||
x2 = max(x2, x_t + w_t) |
||||
y2 = max(y2, y_t + h_t) |
||||
return [x1, y1, x2, y2] |
||||
|
||||
def plot( |
||||
self, |
||||
annotations, |
||||
output, |
||||
bbox=None, |
||||
points=None, |
||||
point_label=None, |
||||
mask_random_color=True, |
||||
better_quality=True, |
||||
retina=False, |
||||
with_contours=True, |
||||
): |
||||
""" |
||||
Plots annotations, bounding boxes, and points on images and saves the output. |
||||
|
||||
Args: |
||||
annotations (list): Annotations to be plotted. |
||||
output (str or Path): Output directory for saving the plots. |
||||
bbox (list, optional): Bounding box coordinates [x1, y1, x2, y2]. Defaults to None. |
||||
points (list, optional): Points to be plotted. Defaults to None. |
||||
point_label (list, optional): Labels for the points. Defaults to None. |
||||
mask_random_color (bool, optional): Whether to use random color for masks. Defaults to True. |
||||
better_quality (bool, optional): Whether to apply morphological transformations for better mask quality. |
||||
Defaults to True. |
||||
retina (bool, optional): Whether to use retina mask. Defaults to False. |
||||
with_contours (bool, optional): Whether to plot contours. Defaults to True. |
||||
""" |
||||
import matplotlib.pyplot as plt |
||||
|
||||
pbar = TQDM(annotations, total=len(annotations)) |
||||
for ann in pbar: |
||||
result_name = os.path.basename(ann.path) |
||||
image = ann.orig_img[..., ::-1] # BGR to RGB |
||||
original_h, original_w = ann.orig_shape |
||||
# For macOS only |
||||
# plt.switch_backend('TkAgg') |
||||
plt.figure(figsize=(original_w / 100, original_h / 100)) |
||||
# Add subplot with no margin. |
||||
plt.subplots_adjust(top=1, bottom=0, right=1, left=0, hspace=0, wspace=0) |
||||
plt.margins(0, 0) |
||||
plt.gca().xaxis.set_major_locator(plt.NullLocator()) |
||||
plt.gca().yaxis.set_major_locator(plt.NullLocator()) |
||||
plt.imshow(image) |
||||
|
||||
if ann.masks is not None: |
||||
masks = ann.masks.data |
||||
if better_quality: |
||||
if isinstance(masks[0], torch.Tensor): |
||||
masks = np.array(masks.cpu()) |
||||
for i, mask in enumerate(masks): |
||||
mask = cv2.morphologyEx(mask.astype(np.uint8), cv2.MORPH_CLOSE, np.ones((3, 3), np.uint8)) |
||||
masks[i] = cv2.morphologyEx(mask.astype(np.uint8), cv2.MORPH_OPEN, np.ones((8, 8), np.uint8)) |
||||
|
||||
self.fast_show_mask( |
||||
masks, |
||||
plt.gca(), |
||||
random_color=mask_random_color, |
||||
bbox=bbox, |
||||
points=points, |
||||
pointlabel=point_label, |
||||
retinamask=retina, |
||||
target_height=original_h, |
||||
target_width=original_w, |
||||
) |
||||
|
||||
if with_contours: |
||||
contour_all = [] |
||||
temp = np.zeros((original_h, original_w, 1)) |
||||
for i, mask in enumerate(masks): |
||||
mask = mask.astype(np.uint8) |
||||
if not retina: |
||||
mask = cv2.resize(mask, (original_w, original_h), interpolation=cv2.INTER_NEAREST) |
||||
contours, _ = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE) |
||||
contour_all.extend(iter(contours)) |
||||
cv2.drawContours(temp, contour_all, -1, (255, 255, 255), 2) |
||||
color = np.array([0 / 255, 0 / 255, 1.0, 0.8]) |
||||
contour_mask = temp / 255 * color.reshape(1, 1, -1) |
||||
plt.imshow(contour_mask) |
||||
|
||||
# Save the figure |
||||
save_path = Path(output) / result_name |
||||
save_path.parent.mkdir(exist_ok=True, parents=True) |
||||
plt.axis("off") |
||||
plt.savefig(save_path, bbox_inches="tight", pad_inches=0, transparent=True) |
||||
plt.close() |
||||
pbar.set_description(f"Saving {result_name} to {save_path}") |
||||
|
||||
@staticmethod |
||||
def fast_show_mask( |
||||
annotation, |
||||
ax, |
||||
random_color=False, |
||||
bbox=None, |
||||
points=None, |
||||
pointlabel=None, |
||||
retinamask=True, |
||||
target_height=960, |
||||
target_width=960, |
||||
): |
||||
""" |
||||
Quickly shows the mask annotations on the given matplotlib axis. |
||||
|
||||
Args: |
||||
annotation (array-like): Mask annotation. |
||||
ax (matplotlib.axes.Axes): Matplotlib axis. |
||||
random_color (bool, optional): Whether to use random color for masks. Defaults to False. |
||||
bbox (list, optional): Bounding box coordinates [x1, y1, x2, y2]. Defaults to None. |
||||
points (list, optional): Points to be plotted. Defaults to None. |
||||
pointlabel (list, optional): Labels for the points. Defaults to None. |
||||
retinamask (bool, optional): Whether to use retina mask. Defaults to True. |
||||
target_height (int, optional): Target height for resizing. Defaults to 960. |
||||
target_width (int, optional): Target width for resizing. Defaults to 960. |
||||
""" |
||||
import matplotlib.pyplot as plt |
||||
|
||||
n, h, w = annotation.shape # batch, height, width |
||||
|
||||
areas = np.sum(annotation, axis=(1, 2)) |
||||
annotation = annotation[np.argsort(areas)] |
||||
|
||||
index = (annotation != 0).argmax(axis=0) |
||||
if random_color: |
||||
color = np.random.random((n, 1, 1, 3)) |
||||
else: |
||||
color = np.ones((n, 1, 1, 3)) * np.array([30 / 255, 144 / 255, 1.0]) |
||||
transparency = np.ones((n, 1, 1, 1)) * 0.6 |
||||
visual = np.concatenate([color, transparency], axis=-1) |
||||
mask_image = np.expand_dims(annotation, -1) * visual |
||||
|
||||
show = np.zeros((h, w, 4)) |
||||
h_indices, w_indices = np.meshgrid(np.arange(h), np.arange(w), indexing="ij") |
||||
indices = (index[h_indices, w_indices], h_indices, w_indices, slice(None)) |
||||
|
||||
show[h_indices, w_indices, :] = mask_image[indices] |
||||
if bbox is not None: |
||||
x1, y1, x2, y2 = bbox |
||||
ax.add_patch(plt.Rectangle((x1, y1), x2 - x1, y2 - y1, fill=False, edgecolor="b", linewidth=1)) |
||||
# Draw point |
||||
if points is not None: |
||||
plt.scatter( |
||||
[point[0] for i, point in enumerate(points) if pointlabel[i] == 1], |
||||
[point[1] for i, point in enumerate(points) if pointlabel[i] == 1], |
||||
s=20, |
||||
c="y", |
||||
) |
||||
plt.scatter( |
||||
[point[0] for i, point in enumerate(points) if pointlabel[i] == 0], |
||||
[point[1] for i, point in enumerate(points) if pointlabel[i] == 0], |
||||
s=20, |
||||
c="m", |
||||
) |
||||
|
||||
if not retinamask: |
||||
show = cv2.resize(show, (target_width, target_height), interpolation=cv2.INTER_NEAREST) |
||||
ax.imshow(show) |
||||
|
||||
@torch.no_grad() |
||||
def retrieve(self, model, preprocess, elements, search_text: str, device) -> Tensor: |
||||
"""Processes images and text with a model, calculates similarity, and returns softmax score.""" |
||||
preprocessed_images = [preprocess(image).to(device) for image in elements] |
||||
tokenized_text = self.clip.tokenize([search_text]).to(device) |
||||
stacked_images = torch.stack(preprocessed_images) |
||||
image_features = model.encode_image(stacked_images) |
||||
text_features = model.encode_text(tokenized_text) |
||||
image_features /= image_features.norm(dim=-1, keepdim=True) |
||||
text_features /= text_features.norm(dim=-1, keepdim=True) |
||||
probs = 100.0 * image_features @ text_features.T |
||||
return probs[:, 0].softmax(dim=0) |
||||
|
||||
def _crop_image(self, format_results): |
||||
"""Crops an image based on provided annotation format and returns cropped images and related data.""" |
||||
image = Image.fromarray(cv2.cvtColor(self.results[0].orig_img, cv2.COLOR_BGR2RGB)) |
||||
ori_w, ori_h = image.size |
||||
annotations = format_results |
||||
mask_h, mask_w = annotations[0]["segmentation"].shape |
||||
if ori_w != mask_w or ori_h != mask_h: |
||||
image = image.resize((mask_w, mask_h)) |
||||
cropped_images = [] |
||||
filter_id = [] |
||||
for _, mask in enumerate(annotations): |
||||
if np.sum(mask["segmentation"]) <= 100: |
||||
filter_id.append(_) |
||||
continue |
||||
bbox = self._get_bbox_from_mask(mask["segmentation"]) # bbox from mask |
||||
cropped_images.append(self._segment_image(image, bbox)) # save cropped image |
||||
|
||||
return cropped_images, filter_id, annotations |
||||
|
||||
def box_prompt(self, bbox): |
||||
"""Modifies the bounding box properties and calculates IoU between masks and bounding box.""" |
||||
if self.results[0].masks is not None: |
||||
assert bbox[2] != 0 and bbox[3] != 0, "Bounding box width and height should not be zero" |
||||
masks = self.results[0].masks.data |
||||
target_height, target_width = self.results[0].orig_shape |
||||
h = masks.shape[1] |
||||
w = masks.shape[2] |
||||
if h != target_height or w != target_width: |
||||
bbox = [ |
||||
int(bbox[0] * w / target_width), |
||||
int(bbox[1] * h / target_height), |
||||
int(bbox[2] * w / target_width), |
||||
int(bbox[3] * h / target_height), |
||||
] |
||||
bbox[0] = max(round(bbox[0]), 0) |
||||
bbox[1] = max(round(bbox[1]), 0) |
||||
bbox[2] = min(round(bbox[2]), w) |
||||
bbox[3] = min(round(bbox[3]), h) |
||||
|
||||
# IoUs = torch.zeros(len(masks), dtype=torch.float32) |
||||
bbox_area = (bbox[3] - bbox[1]) * (bbox[2] - bbox[0]) |
||||
|
||||
masks_area = torch.sum(masks[:, bbox[1] : bbox[3], bbox[0] : bbox[2]], dim=(1, 2)) |
||||
orig_masks_area = torch.sum(masks, dim=(1, 2)) |
||||
|
||||
union = bbox_area + orig_masks_area - masks_area |
||||
iou = masks_area / union |
||||
max_iou_index = torch.argmax(iou) |
||||
|
||||
self.results[0].masks.data = torch.tensor(np.array([masks[max_iou_index].cpu().numpy()])) |
||||
return self.results |
||||
|
||||
def point_prompt(self, points, pointlabel): # numpy |
||||
"""Adjusts points on detected masks based on user input and returns the modified results.""" |
||||
if self.results[0].masks is not None: |
||||
masks = self._format_results(self.results[0], 0) |
||||
target_height, target_width = self.results[0].orig_shape |
||||
h = masks[0]["segmentation"].shape[0] |
||||
w = masks[0]["segmentation"].shape[1] |
||||
if h != target_height or w != target_width: |
||||
points = [[int(point[0] * w / target_width), int(point[1] * h / target_height)] for point in points] |
||||
onemask = np.zeros((h, w)) |
||||
for annotation in masks: |
||||
mask = annotation["segmentation"] if isinstance(annotation, dict) else annotation |
||||
for i, point in enumerate(points): |
||||
if mask[point[1], point[0]] == 1 and pointlabel[i] == 1: |
||||
onemask += mask |
||||
if mask[point[1], point[0]] == 1 and pointlabel[i] == 0: |
||||
onemask -= mask |
||||
onemask = onemask >= 1 |
||||
self.results[0].masks.data = torch.tensor(np.array([onemask])) |
||||
return self.results |
||||
|
||||
def text_prompt(self, text, clip_download_root=None): |
||||
"""Processes a text prompt, applies it to existing results and returns the updated results.""" |
||||
if self.results[0].masks is not None: |
||||
format_results = self._format_results(self.results[0], 0) |
||||
cropped_images, filter_id, annotations = self._crop_image(format_results) |
||||
clip_model, preprocess = self.clip.load("ViT-B/32", download_root=clip_download_root, device=self.device) |
||||
scores = self.retrieve(clip_model, preprocess, cropped_images, text, device=self.device) |
||||
max_idx = torch.argmax(scores) |
||||
max_idx += sum(np.array(filter_id) <= int(max_idx)) |
||||
self.results[0].masks.data = torch.tensor(np.array([annotations[max_idx]["segmentation"]])) |
||||
return self.results |
||||
|
||||
def everything_prompt(self): |
||||
"""Returns the processed results from the previous methods in the class.""" |
||||
return self.results |
||||
@ -0,0 +1,22 @@ |
||||
# Ultralytics YOLO 🚀, AGPL-3.0 license |
||||
"""Activation modules.""" |
||||
|
||||
import torch |
||||
import torch.nn as nn |
||||
|
||||
|
||||
class AGLU(nn.Module): |
||||
"""Unified activation function module from https://github.com/kostas1515/AGLU.""" |
||||
|
||||
def __init__(self, device=None, dtype=None) -> None: |
||||
"""Initialize the Unified activation function.""" |
||||
super().__init__() |
||||
self.act = nn.Softplus(beta=-1.0) |
||||
self.lambd = nn.Parameter(nn.init.uniform_(torch.empty(1, device=device, dtype=dtype))) # lambda parameter |
||||
self.kappa = nn.Parameter(nn.init.uniform_(torch.empty(1, device=device, dtype=dtype))) # kappa parameter |
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor: |
||||
"""Compute the forward pass of the Unified activation function.""" |
||||
lam = torch.clamp(self.lambd, min=0.0001) |
||||
y = torch.exp((1 / lam) * self.act((self.kappa * x) - torch.log(lam))) |
||||
return y # for AGLU simply return y * input |
||||
Loading…
Reference in new issue