super-gradients/YOLONAS-POSE.md

YOLO-NAS-POSE

A Next-Generation, Pose Estimation Foundational Model generated by Deci’s Neural Architecture Search Technology

Deci is thrilled to announce the release of a new object detection model, YOLO-NAS-POSE - a successor of YOLO-NAS, pose estimation architecture, providing superior real-time object detection capabilities and production-ready performance. Deci's mission is to provide AI teams with tools to remove development barriers and attain efficient inference performance more quickly.

The new YOLO-NAS-POSE delivers state-of-the-art (SOTA) performance with the unparalleled accuracy-speed performance, outperforming other models such as YOLOv8-Pose, DEKR and others.

Deci's proprietary Neural Architecture Search technology, AutoNAC™, generated the architecture of YOLO-NAS-POSE model. The AutoNAC™ engine lets you input any task, data characteristics (access to data is not required), inference environment and performance targets, and then guides you to find the optimal architecture that delivers the best balance between accuracy and inference speed for your specific application. In addition to being data and hardware aware, the AutoNAC engine considers other components in the inference stack, including compilers and quantization.

In terms of pure numbers, YOLO-NAS-POSE is XXX mAP point more accurate and YYY% faster than equivalent variants of YOLOv8.

Model	mAP	Latency (ms)
YOLO-NAS S	TODO	TODO
YOLO-NAS M	TODO	TODO
YOLO-NAS L	TODO	TODO
YOLO-NAS S INT-8	TODO	TODO
YOLO-NAS M INT-8	TODO	TODO
YOLO-NAS L INT-8	TODO	TODO

mAP numbers in table reported for COCO 2017 Val dataset and latency benchmarked for 640x640 images on Nvidia T4 GPU. No flip-TTA was used.

Similary to YOLO-NAS, YOLO-NAS-POSE architecture employs quantization-aware blocks and selective quantization for optimized performance. In fact YOLO-NAS-POSE is a direct successor of YOLO-NAS and uses same backbone and neck as YOLO-NAS. Only the head is different and is optimized by AutoNAC for pose estimation task. That enables us to use transfer learning and fine-tune YOLO-NAS-POSE starting from YOLO-NAS weights.

When converted to its INT8 quantized version, YOLO-NAS-POSE experiences a smaller precision drop (0.51, 0.65, and 0.45 points of mAP for S, M, and L variants) compared to other models that lose 1-2 mAP points during quantization. These techniques culminate in innovative architecture with superior object detection capabilities and top-notch performance.

Quickstart

Extract predicted poses

import super_gradients

yolo_nas = super_gradients.training.models.get("yolo_nas_pose_l", pretrained_weights="coco_pose").cuda()
model_predictions  = yolo_nas.predict("https://deci-pretrained-models.s3.amazonaws.com/sample_images/beatles-abbeyroad.jpg").show()

prediction = model_predictions[0].prediction # One prediction per image - Here we work with 1 image so we get the first.

bboxes = prediction.bboxes_xyxy # [Num Instances, 4] List of predicted bounding boxes for each object 
poses  = prediction.poses       # [Num Instances, Num Joints, 3] list of predicted joints for each detected object (x,y, confidence)
scores = prediction.scores      # [Num Instances] - Confidence value for each predicted instance

Recipes

We provide training recipies for training YOLO-NAS-POSE on COCO, CrowdPose and AnimalPose datasets.

COCO 2017

• super_gradients/recipes/coco_yolo_nas_m.yaml
• super_gradients/recipes/coco_yolo_nas_s.yaml
• super_gradients/recipes/coco_yolo_nas_s_qat.yaml

CrowdPose

• super_gradients/recipes/coco_yolo_nas_m.yaml
• super_gradients/recipes/coco_yolo_nas_s.yaml
• super_gradients/recipes/coco_yolo_nas_s_qat.yaml

AnimalPose

• super_gradients/recipes/coco_yolo_nas_m.yaml
• super_gradients/recipes/coco_yolo_nas_s.yaml
• super_gradients/recipes/coco_yolo_nas_s_qat.yaml

Great fine-tuning potential

We demonstrate great performance of YOLO-NAS on downstream tasks. When fine-tuning on AnimalPose our YOLO-NAS-POSE model achieves higher mAP than our nearest competitors:

TODO: Find good dataset for fine-tuning (CrowdPose does not fit well since it is a subset of COCO and subject for data leakage)

Additional resources

Fine-Tuning Notebook (TODO)

Quantization Aware Training YoloNAS on Custom Dataset Notebook (TODO)

Documentation: YOLO-NAS-POSE Quickstart

Join our Discord Community

LICENSE

TODO: Refine license ?

The YOLO-NAS-POSE model is available under an open-source license with pre-trained weights available for non-commercial use on SuperGradients, Deci's PyTorch-based, open-source, computer vision training library. With SuperGradients, users can train models from scratch or fine-tune existing ones, leveraging advanced built-in training techniques like Distributed Data Parallel, Exponential Moving Average, Automatic mixed precision, and Quantization Aware Training.

License file is available here: YOLO-NAS WEIGHTS LICENSE