zxd
This commit is contained in:
486
README.md
486
README.md
@@ -1,69 +1,44 @@
|
|||||||
# SAM 3: Segment Anything with Concepts
|
# 量迹AI · SAM3「分割一切」视觉分割服务
|
||||||
|
|
||||||
Meta Superintelligence Labs
|
本项目在开源 SAM3(Segment Anything Model 3)能力之上,封装了面向业务的 **“分割一切”** 推理服务:通过 **FastAPI** 提供文本提示词驱动的图像分割接口,并扩展了 **塔罗牌分割/识别**、**人脸与头发分割 + 属性分析** 等场景能力。
|
||||||
|
|
||||||
[Nicolas Carion](https://www.nicolascarion.com/)\*,
|
本仓库定位为:**模型推理 + API 服务** 的可复用工程模板(适合在 MacOS 开发、服务器部署)。
|
||||||
[Laura Gustafson](https://scholar.google.com/citations?user=c8IpF9gAAAAJ&hl=en)\*,
|
|
||||||
[Yuan-Ting Hu](https://scholar.google.com/citations?user=E8DVVYQAAAAJ&hl=en)\*,
|
|
||||||
[Shoubhik Debnath](https://scholar.google.com/citations?user=fb6FOfsAAAAJ&hl=en)\*,
|
|
||||||
[Ronghang Hu](https://ronghanghu.com/)\*,
|
|
||||||
[Didac Suris](https://www.didacsuris.com/)\*,
|
|
||||||
[Chaitanya Ryali](https://scholar.google.com/citations?user=4LWx24UAAAAJ&hl=en)\*,
|
|
||||||
[Kalyan Vasudev Alwala](https://scholar.google.co.in/citations?user=m34oaWEAAAAJ&hl=en)\*,
|
|
||||||
[Haitham Khedr](https://hkhedr.com/)\*, Andrew Huang,
|
|
||||||
[Jie Lei](https://jayleicn.github.io/),
|
|
||||||
[Tengyu Ma](https://scholar.google.com/citations?user=VeTSl0wAAAAJ&hl=en),
|
|
||||||
[Baishan Guo](https://scholar.google.com/citations?user=BC5wDu8AAAAJ&hl=en),
|
|
||||||
Arpit Kalla, [Markus Marks](https://damaggu.github.io/),
|
|
||||||
[Joseph Greer](https://scholar.google.com/citations?user=guL96CkAAAAJ&hl=en),
|
|
||||||
Meng Wang, [Peize Sun](https://peizesun.github.io/),
|
|
||||||
[Roman Rädle](https://scholar.google.com/citations?user=Tpt57v0AAAAJ&hl=en),
|
|
||||||
[Triantafyllos Afouras](https://www.robots.ox.ac.uk/~afourast/),
|
|
||||||
[Effrosyni Mavroudi](https://scholar.google.com/citations?user=vYRzGGEAAAAJ&hl=en),
|
|
||||||
[Katherine Xu](https://k8xu.github.io/)°,
|
|
||||||
[Tsung-Han Wu](https://patrickthwu.com/)°,
|
|
||||||
[Yu Zhou](https://yu-bryan-zhou.github.io/)°,
|
|
||||||
[Liliane Momeni](https://scholar.google.com/citations?user=Lb-KgVYAAAAJ&hl=en)°,
|
|
||||||
[Rishi Hazra](https://rishihazra.github.io/)°,
|
|
||||||
[Shuangrui Ding](https://mark12ding.github.io/)°,
|
|
||||||
[Sagar Vaze](https://sgvaze.github.io/)°,
|
|
||||||
[Francois Porcher](https://scholar.google.com/citations?user=LgHZ8hUAAAAJ&hl=en)°,
|
|
||||||
[Feng Li](https://fengli-ust.github.io/)°,
|
|
||||||
[Siyuan Li](https://siyuanliii.github.io/)°,
|
|
||||||
[Aishwarya Kamath](https://ashkamath.github.io/)°,
|
|
||||||
[Ho Kei Cheng](https://hkchengrex.com/)°,
|
|
||||||
[Piotr Dollar](https://pdollar.github.io/)†,
|
|
||||||
[Nikhila Ravi](https://nikhilaravi.com/)†,
|
|
||||||
[Kate Saenko](https://ai.bu.edu/ksaenko.html)†,
|
|
||||||
[Pengchuan Zhang](https://pzzhang.github.io/pzzhang/)†,
|
|
||||||
[Christoph Feichtenhofer](https://feichtenhofer.github.io/)†
|
|
||||||
|
|
||||||
\* core contributor, ° intern, † project lead, order is random within groups
|
---
|
||||||
|
|
||||||
[[`Paper`](https://ai.meta.com/research/publications/sam-3-segment-anything-with-concepts/)]
|
## 你能用它做什么
|
||||||
[[`Project`](https://ai.meta.com/sam3)]
|
|
||||||
[[`Demo`](https://segment-anything.com/)]
|
|
||||||
[[`Blog`](https://ai.meta.com/blog/segment-anything-model-3/)]
|
|
||||||
[[`BibTeX`](#citing-sam-3)]
|
|
||||||
|
|
||||||
 SAM 3 is a unified foundation model for promptable segmentation in images and videos. It can detect, segment, and track objects using text or visual prompts such as points, boxes, and masks. Compared to its predecessor [SAM 2](https://github.com/facebookresearch/sam2), SAM 3 introduces the ability to exhaustively segment all instances of an open-vocabulary concept specified by a short text phrase or exemplars. Unlike prior work, SAM 3 can handle a vastly larger set of open-vocabulary prompts. It achieves 75-80% of human performance on our new [SA-CO benchmark](https://github.com/facebookresearch/sam3?tab=readme-ov-file#sa-co-dataset) which contains 270K unique concepts, over 50 times more than existing benchmarks.
|
- 通用分割:输入图片 + 文本提示词(中文或英文),返回分割可视化结果图 URL
|
||||||
|
- 分割子图导出:可选返回每个目标的独立裁剪图(支持透明背景抠图)
|
||||||
|
- 塔罗牌:检测指定数量的塔罗牌 → 透视矫正裁剪 →(可选)调用多模态大模型识别牌名与正逆位
|
||||||
|
- 人脸分析:分割人脸/头发区域 → 裁剪保存 →(可选)调用多模态大模型预测性别/年龄等属性
|
||||||
|
|
||||||
This breakthrough is driven by an innovative data engine that has automatically annotated over 4 million unique concepts, creating the largest high-quality open-vocabulary segmentation dataset to date. In addition, SAM 3 introduces a new model architecture featuring a presence token that improves discrimination between closely related text prompts (e.g., “a player in white” vs. “a player in red”), as well as a decoupled detector–tracker design that minimizes task interference and scales efficiently with data.
|
对应实现文件:
|
||||||
|
- API 服务入口:[fastAPI_tarot.py](fastAPI_tarot.py)
|
||||||
|
- 人脸分析流程:[human_analysis_service.py](human_analysis_service.py)
|
||||||
|
|
||||||
<p align="center">
|
---
|
||||||
<img src="assets/dog.gif" width=380 />
|
|
||||||
<img src="assets/player.gif" width=380 />
|
|
||||||
</p>
|
|
||||||
|
|
||||||
## Installation
|
## 项目结构(关键目录)
|
||||||
|
|
||||||
### Prerequisites
|
- sam3/:SAM3 模型核心代码(本仓库已内置,不依赖外部子模块)
|
||||||
|
- static/results/:推理结果落盘目录(接口返回的 URL 指向这里)
|
||||||
|
- assets/:示例素材、演示图片/视频
|
||||||
|
- fastAPI_tarot.py:主要 API 服务(含鉴权、模型加载、推理、可视化与落盘)
|
||||||
|
|
||||||
- Python 3.12 or higher
|
---
|
||||||
- PyTorch 2.7 or higher
|
|
||||||
- CUDA-compatible GPU with CUDA 12.6 or higher
|
|
||||||
|
|
||||||
1. **Create a new Conda environment:**
|
## 环境要求
|
||||||
|
|
||||||
|
- Python:建议 3.10+(SAM3 作为库最低 3.8+)
|
||||||
|
- PyTorch:建议按你的设备(CPU/CUDA)安装匹配版本
|
||||||
|
- 依赖:FastAPI/uvicorn 等(见 requirement.txt,另有 sam3 依赖见 pyproject.toml)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 安装与准备
|
||||||
|
|
||||||
|
### 0) 创建虚拟环境(推荐)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
conda create -n sam3 python=3.12
|
conda create -n sam3 python=3.12
|
||||||
@@ -71,331 +46,172 @@ conda deactivate
|
|||||||
conda activate sam3
|
conda activate sam3
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### 0.1) 安装 PyTorch(按你的设备选择)
|
||||||
|
|
||||||
|
GPU(CUDA)环境请安装与你 CUDA 版本匹配的 PyTorch;CPU 环境可直接安装 CPU 版。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install torch torchvision torchaudio
|
||||||
|
```
|
||||||
|
|
||||||
|
### 1) 安装本仓库与依赖
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e .
|
||||||
|
pip install -r requirement.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
如果需要接口用到的第三方库(如 OpenCV、matplotlib、Pillow、requests、dashscope 等),请按你的实际场景安装(仓库中部分功能会依赖它们)。
|
||||||
|
|
||||||
|
### 2) 下载权重模型(必须)
|
||||||
|
|
||||||
|
> 说明:模型文件较大,建议预留充足磁盘空间。
|
||||||
|
>
|
||||||
|
> 默认将权重保存到 `./dir/`,可按需修改 `--local_dir`。
|
||||||
|
>
|
||||||
|
> 如需通过 Hugging Face 下载,也可以自行改用 `huggingface_hub` 的下载方式。
|
||||||
|
|
||||||
1.1 **下载权重模型:**
|
1.1 **下载权重模型:**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install modelscope
|
pip install modelscope
|
||||||
modelscope download --model facebook/sam3 sam3.pt --local_dir ./dir
|
modelscope download --model facebook/sam3 sam3.pt --local_dir ./dir
|
||||||
```
|
```
|
||||||
|
|
||||||
2. **Install PyTorch with CUDA support:**
|
下载后会得到 `./dir/sam3.pt`。
|
||||||
|
|
||||||
```bash
|
### 3) 配置 SAM3 权重路径(必须)
|
||||||
pip install torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Clone the repository and install the package:**
|
当前 API 服务会在启动时加载 SAM3 权重。你需要确保 `fastAPI_tarot.py` 里构建模型时使用了正确的 `checkpoint_path`。
|
||||||
|
|
||||||
```bash
|
推荐做法:在 [fastAPI_tarot.py](fastAPI_tarot.py) 中将模型构建改为显式传入本地权重路径,例如:
|
||||||
git clone https://github.com/facebookresearch/sam3.git
|
|
||||||
cd sam3
|
|
||||||
pip install -e .
|
|
||||||
```
|
|
||||||
|
|
||||||
4. **Install additional dependencies for example notebooks or development:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# For running example notebooks
|
|
||||||
pip install -e ".[notebooks]"
|
|
||||||
|
|
||||||
# For development
|
|
||||||
pip install -e ".[train,dev]"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Getting Started
|
|
||||||
|
|
||||||
⚠️ Before using SAM 3, please request access to the checkpoints on the SAM 3
|
|
||||||
Hugging Face [repo](https://huggingface.co/facebook/sam3). Once accepted, you
|
|
||||||
need to be authenticated to download the checkpoints. You can do this by running
|
|
||||||
the following [steps](https://huggingface.co/docs/huggingface_hub/en/quick-start#authentication)
|
|
||||||
(e.g. `hf auth login` after generating an access token.)
|
|
||||||
|
|
||||||
### Basic Usage
|
|
||||||
|
|
||||||
```python
|
```python
|
||||||
import torch
|
model = build_sam3_image_model(checkpoint_path="./dir/sam3.pt")
|
||||||
#################################### For Image ####################################
|
|
||||||
from PIL import Image
|
|
||||||
from sam3.model_builder import build_sam3_image_model
|
|
||||||
from sam3.model.sam3_image_processor import Sam3Processor
|
|
||||||
# Load the model
|
|
||||||
model = build_sam3_image_model()
|
|
||||||
processor = Sam3Processor(model)
|
|
||||||
# Load an image
|
|
||||||
image = Image.open("<YOUR_IMAGE_PATH.jpg>")
|
|
||||||
inference_state = processor.set_image(image)
|
|
||||||
# Prompt the model with text
|
|
||||||
output = processor.set_text_prompt(state=inference_state, prompt="<YOUR_TEXT_PROMPT>")
|
|
||||||
|
|
||||||
# Get the masks, bounding boxes, and scores
|
|
||||||
masks, boxes, scores = output["masks"], output["boxes"], output["scores"]
|
|
||||||
|
|
||||||
#################################### For Video ####################################
|
|
||||||
|
|
||||||
from sam3.model_builder import build_sam3_video_predictor
|
|
||||||
|
|
||||||
video_predictor = build_sam3_video_predictor()
|
|
||||||
video_path = "<YOUR_VIDEO_PATH>" # a JPEG folder or an MP4 video file
|
|
||||||
# Start a session
|
|
||||||
response = video_predictor.handle_request(
|
|
||||||
request=dict(
|
|
||||||
type="start_session",
|
|
||||||
resource_path=video_path,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
response = video_predictor.handle_request(
|
|
||||||
request=dict(
|
|
||||||
type="add_prompt",
|
|
||||||
session_id=response["session_id"],
|
|
||||||
frame_index=0, # Arbitrary frame index
|
|
||||||
text="<YOUR_TEXT_PROMPT>",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
output = response["outputs"]
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Examples
|
---
|
||||||
|
|
||||||
The `examples` directory contains notebooks demonstrating how to use SAM3 with
|
## 启动 API 服务(FastAPI)
|
||||||
various types of prompts:
|
|
||||||
|
|
||||||
- [`sam3_image_predictor_example.ipynb`](examples/sam3_image_predictor_example.ipynb)
|
服务定义在 [fastAPI_tarot.py](fastAPI_tarot.py) 中。你可以用 uvicorn 启动(示例命令仅供参考,按需调整 host/port):
|
||||||
: Demonstrates how to prompt SAM 3 with text and visual box prompts on images.
|
|
||||||
- [`sam3_video_predictor_example.ipynb`](examples/sam3_video_predictor_example.ipynb)
|
|
||||||
: Demonstrates how to prompt SAM 3 with text prompts on videos, and doing
|
|
||||||
further interactive refinements with points.
|
|
||||||
- [`sam3_image_batched_inference.ipynb`](examples/sam3_image_batched_inference.ipynb)
|
|
||||||
: Demonstrates how to run batched inference with SAM 3 on images.
|
|
||||||
- [`sam3_agent.ipynb`](examples/sam3_agent.ipynb): Demonsterates the use of SAM
|
|
||||||
3 Agent to segment complex text prompt on images.
|
|
||||||
- [`saco_gold_silver_vis_example.ipynb`](examples/saco_gold_silver_vis_example.ipynb)
|
|
||||||
: Shows a few examples from SA-Co image evaluation set.
|
|
||||||
- [`saco_veval_vis_example.ipynb`](examples/saco_veval_vis_example.ipynb) :
|
|
||||||
Shows a few examples from SA-Co video evaluation set.
|
|
||||||
|
|
||||||
There are additional notebooks in the examples directory that demonstrate how to
|
|
||||||
use SAM 3 for interactive instance segmentation in images and videos (SAM 1/2
|
|
||||||
tasks), or as a tool for an MLLM, and how to run evaluations on the SA-Co
|
|
||||||
dataset.
|
|
||||||
|
|
||||||
To run the Jupyter notebook examples:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Make sure you have the notebooks dependencies installed
|
uvicorn fastAPI_tarot:app --host 127.0.0.1 --port 55600
|
||||||
pip install -e ".[notebooks]"
|
|
||||||
|
|
||||||
# Start Jupyter notebook
|
|
||||||
jupyter notebook examples/sam3_image_predictor_example.ipynb
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Model
|
启动后:
|
||||||
|
- OpenAPI 文档:`http://127.0.0.1:55600/docs`
|
||||||
|
- 静态结果目录:`http://127.0.0.1:55600/static/results/...`
|
||||||
|
|
||||||
SAM 3 consists of a detector and a tracker that share a vision encoder. It has 848M parameters. The
|
---
|
||||||
detector is a DETR-based model conditioned on text, geometry, and image
|
|
||||||
exemplars. The tracker inherits the SAM 2 transformer encoder-decoder
|
|
||||||
architecture, supporting video segmentation and interactive refinement.
|
|
||||||
|
|
||||||
## Image Results
|
## 鉴权说明(API Key)
|
||||||
|
|
||||||
<div align="center">
|
所有接口都要求在请求 Header 中携带 API Key:
|
||||||
<table style="min-width: 80%; border: 2px solid #ddd; border-collapse: collapse">
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th rowspan="3" style="border-right: 2px solid #ddd; padding: 12px 20px">Model</th>
|
|
||||||
<th colspan="3" style="text-align: center; border-right: 2px solid #ddd; padding: 12px 20px">Instance Segmentation</th>
|
|
||||||
<th colspan="5" style="text-align: center; padding: 12px 20px">Box Detection</th>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">LVIS</th>
|
|
||||||
<th style="text-align: center; border-right: 2px solid #ddd; padding: 12px 20px">SA-Co/Gold</th>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">LVIS</th>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">COCO</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">SA-Co/Gold</th>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">AP</th>
|
|
||||||
<th style="text-align: center; border-right: 2px solid #ddd; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">AP</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">AP</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">AP<sub>o</sub>
|
|
||||||
</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
<tr>
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">Human</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 2px solid #ddd; padding: 10px 20px">72.8</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">74.0</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">OWLv2*</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px; color: #999">29.3</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px; color: #999">43.4</td>
|
|
||||||
<td style="text-align: center; border-right: 2px solid #ddd; padding: 10px 20px">24.6</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px; color: #999">30.2</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px; color: #999">45.5</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">46.1</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">23.9</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">24.5</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">DINO-X</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">38.5</td>
|
|
||||||
<td style="text-align: center; border-right: 2px solid #ddd; padding: 10px 20px">21.3</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">52.4</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">56.0</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">22.5</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">Gemini 2.5</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">13.4</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 2px solid #ddd; padding: 10px 20px">13.0</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">16.1</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">14.4</td>
|
|
||||||
</tr>
|
|
||||||
<tr style="border-top: 2px solid #b19c9cff">
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">SAM 3</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">37.2</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">48.5</td>
|
|
||||||
<td style="text-align: center; border-right: 2px solid #ddd; padding: 10px 20px">54.1</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">40.6</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">53.6</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">56.4</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">55.7</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">55.7</td>
|
|
||||||
</tr>
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
|
|
||||||
<p style="text-align: center; margin-top: 10px; font-size: 0.9em; color: #ddd;">* Partially trained on LVIS, AP<sub>o</sub> refers to COCO-O accuracy</p>
|
- Header 名:`X-API-Key`
|
||||||
|
- 值:请使用你服务端配置的 Key(建议用环境变量或配置文件管理,不要在代码里硬编码)
|
||||||
|
|
||||||
</div>
|
---
|
||||||
|
|
||||||
## Video Results
|
## API 接口一览
|
||||||
|
|
||||||
<div align="center">
|
以下接口均在 [fastAPI_tarot.py](fastAPI_tarot.py) 中实现,返回体为 JSON,并通过静态文件服务输出结果图片 URL。
|
||||||
<table style="min-width: 80%; border: 2px solid #ddd; border-collapse: collapse">
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th rowspan="2" style="border-right: 2px solid #ddd; padding: 12px 20px">Model</th>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">SA-V test</th>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">YT-Temporal-1B test</th>
|
|
||||||
<th colspan="2" style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">SmartGlasses test</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">LVVIS test</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">BURST test</th>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">pHOTA</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">pHOTA</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">cgF1</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">pHOTA</th>
|
|
||||||
<th style="text-align: center; border-right: 1px solid #eee; padding: 12px 20px">mAP</th>
|
|
||||||
<th style="text-align: center; padding: 12px 20px">HOTA</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
<tr>
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">Human</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">53.1</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">70.5</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">71.2</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">78.4</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">58.5</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">72.3</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">-</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">-</td>
|
|
||||||
</tr>
|
|
||||||
<tr style="border-top: 2px solid #b19c9cff">
|
|
||||||
<td style="border-right: 2px solid #ddd; padding: 10px 20px">SAM 3</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">30.3</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">58.0</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">50.8</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">69.9</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">36.4</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">63.6</td>
|
|
||||||
<td style="text-align: center; border-right: 1px solid #eee; padding: 10px 20px">36.3</td>
|
|
||||||
<td style="text-align: center; padding: 10px 20px">44.5</td>
|
|
||||||
</tr>
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
## SA-Co Dataset
|
### 1) POST /segment(通用分割)
|
||||||
|
|
||||||
We release 2 image benchmarks, [SA-Co/Gold](scripts/eval/gold/README.md) and
|
功能:对图片做文本提示词分割(中文会自动翻译为英文提示词)。
|
||||||
[SA-Co/Silver](scripts/eval/silver/README.md), and a video benchmark
|
|
||||||
[SA-Co/VEval](scripts/eval/veval/README.md). The datasets contain images (or videos) with annotated noun phrases. Each image/video and noun phrase pair is annotated with instance masks and unique IDs of each object matching the phrase. Phrases that have no matching objects (negative prompts) have no masks, shown in red font in the figure. See the linked READMEs for more details on how to download and run evaluations on the datasets.
|
|
||||||
|
|
||||||
* HuggingFace host: [SA-Co/Gold](https://huggingface.co/datasets/facebook/SACo-Gold), [SA-Co/Silver](https://huggingface.co/datasets/facebook/SACo-Silver) and [SA-Co/VEval](https://huggingface.co/datasets/facebook/SACo-VEval)
|
参数(表单):
|
||||||
* Roboflow host: [SA-Co/Gold](https://universe.roboflow.com/sa-co-gold), [SA-Co/Silver](https://universe.roboflow.com/sa-co-silver) and [SA-Co/VEval](https://universe.roboflow.com/sa-co-veval)
|
- prompt:提示词(必填,例:`cat` / `人` / `白色连衣裙`)
|
||||||
|
- file:上传图片(二选一)
|
||||||
|
- image_url:图片 URL(二选一)
|
||||||
|
- save_segment_images:是否保存并返回每个目标的独立图片(默认 false)
|
||||||
|
- cutout:导出子图时是否透明背景抠图(默认 false)
|
||||||
|
- highlight:是否启用“周边变黑突出主体”的效果(默认 false)
|
||||||
|
- confidence:置信度阈值(0.0-1.0,默认 0.7)
|
||||||
|
|
||||||

|
返回(示例字段):
|
||||||
|
- result_image_url:分割可视化结果图
|
||||||
|
- detected_count:检测到的目标数量
|
||||||
|
- segmented_images:可选的分割子图 URL 列表
|
||||||
|
|
||||||
## Development
|
### 2) POST /segment_tarot(塔罗牌分割)
|
||||||
|
|
||||||
To set up the development environment:
|
功能:检测塔罗牌并对每张牌做透视矫正与裁剪。
|
||||||
|
|
||||||
|
参数(表单):
|
||||||
|
- file / image_url(二选一)
|
||||||
|
- expected_count:期望严格检测到的塔罗牌数量(默认 3)
|
||||||
|
|
||||||
|
返回:
|
||||||
|
- tarot_cards:每张牌的裁剪图 URL、是否被算法旋转矫正等信息
|
||||||
|
- full_visualization:整体分割可视化图 URL(可选)
|
||||||
|
|
||||||
|
### 3) POST /recognize_tarot(塔罗牌全流程:分割 + 识别)
|
||||||
|
|
||||||
|
功能:在 `/segment_tarot` 基础上,进一步调用多模态大模型识别每张牌的名称与正逆位,并尝试识别牌阵。
|
||||||
|
|
||||||
|
说明:
|
||||||
|
- 该流程依赖多模态大模型服务(例如 Qwen-VL / DashScope),请自行配置密钥与可用模型
|
||||||
|
- 强烈建议将密钥通过环境变量注入,避免写入仓库
|
||||||
|
|
||||||
|
### 4) POST /segment_face(人脸/头发分割 + 属性分析)
|
||||||
|
|
||||||
|
功能:分割“face and hair”等区域,并可选调用多模态大模型进行性别/年龄等属性识别。
|
||||||
|
|
||||||
|
参数(表单):
|
||||||
|
- file / image_url(二选一)
|
||||||
|
- prompt:默认 `face and hair`(中文会尝试翻译)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 请求示例(curl)
|
||||||
|
|
||||||
|
### 通用分割(上传图片)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install -e ".[dev,train]"
|
curl -X POST "http://127.0.0.1:55600/segment" \
|
||||||
|
-H "X-API-Key: <YOUR_API_KEY>" \
|
||||||
|
-F "prompt=猫" \
|
||||||
|
-F "file=@/path/to/image.jpg"
|
||||||
```
|
```
|
||||||
|
|
||||||
To format the code:
|
### 通用分割(图片 URL)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ufmt format .
|
curl -X POST "http://127.0.0.1:55600/segment" \
|
||||||
|
-H "X-API-Key: <YOUR_API_KEY>" \
|
||||||
|
-F "prompt=person" \
|
||||||
|
-F "image_url=https://example.com/image.jpg"
|
||||||
```
|
```
|
||||||
|
|
||||||
## Contributing
|
---
|
||||||
|
|
||||||
See [contributing](CONTRIBUTING.md) and the
|
## 结果输出与落盘
|
||||||
[code of conduct](CODE_OF_CONDUCT.md).
|
|
||||||
|
|
||||||
## License
|
- 结果图会写入:`static/results/`
|
||||||
|
- 接口返回的 URL 形如:`/static/results/<filename>` 或 `/static/results/<request_id>/<filename>`
|
||||||
|
- 服务可选开启后台清理任务,定期删除过期结果文件(见 `fastAPI_tarot.py` 末尾的环境变量配置逻辑)
|
||||||
|
|
||||||
This project is licensed under the SAM License - see the [LICENSE](LICENSE) file
|
---
|
||||||
for details.
|
|
||||||
|
|
||||||
## Acknowledgements
|
## 常见问题
|
||||||
|
|
||||||
We would like to thank the following people for their contributions to the SAM 3 project: Alex He, Alexander Kirillov,
|
### 1) 启动时报找不到权重文件?
|
||||||
Alyssa Newcomb, Ana Paula Kirschner Mofarrej, Andrea Madotto, Andrew Westbury, Ashley Gabriel, Azita Shokpour,
|
|
||||||
Ben Samples, Bernie Huang, Carleigh Wood, Ching-Feng Yeh, Christian Puhrsch, Claudette Ward, Daniel Bolya,
|
|
||||||
Daniel Li, Facundo Figueroa, Fazila Vhora, George Orlin, Hanzi Mao, Helen Klein, Hu Xu, Ida Cheng, Jake Kinney,
|
|
||||||
Jiale Zhi, Jo Sampaio, Joel Schlosser, Justin Johnson, Kai Brown, Karen Bergan, Karla Martucci, Kenny Lehmann,
|
|
||||||
Maddie Mintz, Mallika Malhotra, Matt Ward, Michelle Chan, Michelle Restrepo, Miranda Hartley, Muhammad Maaz,
|
|
||||||
Nisha Deo, Peter Park, Phillip Thomas, Raghu Nayani, Rene Martinez Doehner, Robbie Adkins, Ross Girshik, Sasha
|
|
||||||
Mitts, Shashank Jain, Spencer Whitehead, Ty Toledano, Valentin Gabeur, Vincent Cho, Vivian Lee, William Ngan,
|
|
||||||
Xuehai He, Yael Yungster, Ziqi Pang, Ziyi Dou, Zoe Quake.
|
|
||||||
|
|
||||||
## Citing SAM 3
|
请确认:
|
||||||
|
- 已按“下载权重模型”得到 `sam3.pt`
|
||||||
|
- `fastAPI_tarot.py` 中 `build_sam3_image_model(checkpoint_path=...)` 指向正确路径
|
||||||
|
|
||||||
If you use SAM 3 or the SA-Co dataset in your research, please use the following BibTeX entry.
|
### 2) 返回 401/403?
|
||||||
|
|
||||||
```bibtex
|
请确认请求头是否带了 `X-API-Key`,且值与服务端配置一致。
|
||||||
@misc{carion2025sam3segmentconcepts,
|
|
||||||
title={SAM 3: Segment Anything with Concepts},
|
---
|
||||||
author={Nicolas Carion and Laura Gustafson and Yuan-Ting Hu and Shoubhik Debnath and Ronghang Hu and Didac Suris and Chaitanya Ryali and Kalyan Vasudev Alwala and Haitham Khedr and Andrew Huang and Jie Lei and Tengyu Ma and Baishan Guo and Arpit Kalla and Markus Marks and Joseph Greer and Meng Wang and Peize Sun and Roman Rädle and Triantafyllos Afouras and Effrosyni Mavroudi and Katherine Xu and Tsung-Han Wu and Yu Zhou and Liliane Momeni and Rishi Hazra and Shuangrui Ding and Sagar Vaze and Francois Porcher and Feng Li and Siyuan Li and Aishwarya Kamath and Ho Kei Cheng and Piotr Dollár and Nikhila Ravi and Kate Saenko and Pengchuan Zhang and Christoph Feichtenhofer},
|
|
||||||
year={2025},
|
## 免责声明与致谢
|
||||||
eprint={2511.16719},
|
|
||||||
archivePrefix={arXiv},
|
- 本项目为量迹AI内部/业务化工程封装示例,**不是官方 SAM3 仓库**。
|
||||||
primaryClass={cs.CV},
|
- 模型与核心算法能力来自开源的 SAM3 实现与相关依赖,感谢社区贡献者与原作者团队。
|
||||||
url={https://arxiv.org/abs/2511.16719},
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|||||||
Reference in New Issue
Block a user