Tangledup-ai/sam3_local
9
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: Tangledup-ai:1daff5eb927cb37a3fd93879173001e451ab1795
Branches Tags
Tangledup-ai:main
...
compare: Tangledup-ai:8bb00ac9281760946f599b08066429c0b49cbf3b
Branches Tags
Tangledup-ai:main

10 Commits
1daff5eb92 ... 8bb00ac928

Author SHA1 Message Date
quant
8bb00ac928 local 2026-02-15 12:25:24 +08:00
generatedunixname537391475639613
2ec3c0711a fbcode/deeplearning/projects/sam3_release/sam3/train/data 
Reviewed By: JuanBesa

Differential Revision: D91210167

fbshipit-source-id: a563232f4bc82f6f3b99e53df1c88cf0f39747bb
 2026-02-03 13:14:44 -08:00
generatedunixname537391475639613
99d02f28c8 fbcode/deeplearning/projects/sam3_release/sam3/train/data 
Reviewed By: JuanBesa

Differential Revision: D91383480

fbshipit-source-id: 5b98627fb679c7c704c1a2faba9722e3a6f2ec20
 2026-01-27 04:54:06 -08:00
Bowie Chen
11dec2936d apply Black 25.11.0 style in fbcode/deeplearning/projects (21/92) 
Summary:
Formats the covered files with pyfmt.

paintitblack

Reviewed By: itamaro

Differential Revision: D90476315

fbshipit-source-id: ee94c471788b8e7d067813d8b3e0311214d17f3f
 2026-01-11 23:16:49 -08:00
generatedunixname89002005307016
7b89b8fc3f Add missing Pyre mode headers] [batch:11/N] [shard:17/N] 
Differential Revision: D90237984

fbshipit-source-id: 526fd760f303bf31be4f743bdcd77760496de0de
 2026-01-07 05:16:41 -08:00
Manuel López Antequera
5eb25fb54b Pin to numpy 1.26.X not strictly 1.26 (#348) 
Summary: Pull Request resolved: https://github.com/facebookresearch/sam3/pull/348

Reviewed By: haithamkhedr

Differential Revision: D89182040

Pulled By: mlopezantequera

fbshipit-source-id: 73d75c68cb5ec06a645c7c93af44f760bf5b22cc
 2025-12-21 02:34:58 -08:00
generatedunixname537391475639613
962998a167 fbcode/deeplearning/projects/sam3_release/sam3/train/nms_helper.py 
Differential Revision: D88935213

fbshipit-source-id: b0b9cd57858641f7ce398865caef5eed4ad5d8bb
 2025-12-21 01:40:42 -08:00
Matt Le
b26a5f330e Include entire sam3 package instead of just sam3 and sam3.model (#327) 
Summary:
there are several imports within the `sam3.model` package that reference other packages within `sam3` other than `sam3` and `sam3.model` (for example [here](https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_tracker_base.py#L15)).  This fixes the package structure so that you can `pip install` the package and `import sam3`

Pull Request resolved: https://github.com/facebookresearch/sam3/pull/327

Reviewed By: haithamkhedr

Differential Revision: D88950127

Pulled By: lematt1991

fbshipit-source-id: 3554512d304ccdf679a9af8606bbfe1f7f2a1cfb
 2025-12-11 09:23:19 -08:00
Haitham Khedr
757bbb0206 Remove extra args in track_step 
Reviewed By: jayleicn

Differential Revision:
D87886578

Privacy Context Container: L1256182

fbshipit-source-id: 99d47aac7ca76ba8b321716b69d1306581152ac9
 2025-11-29 15:29:00 -08:00
Tengyu Ma
2d1cbaeac7 Update veval README.md for frame shifting alert on sa-co/veval yt1b (#213) 
Summary: Pull Request resolved: https://github.com/facebookresearch/sam3/pull/213

Reviewed By: haithamkhedr

Differential Revision: D87830284

Pulled By: tengyu-ma

fbshipit-source-id: e6cc52f42bfa2de33462f0c26acebcb1bcee0cff
 2025-11-24 18:20:43 -08:00
180 changed files with 1078 additions and 799 deletions
Expand all files Collapse all files

476
examples/sam3_agent.ipynb
View File

@@ -1,242 +1,242 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Copyright (c) Meta Platforms, Inc. and affiliates."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# SAM 3 Agent"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook shows an example of how an MLLM can use SAM 3 as a tool, i.e., \"SAM 3 Agent\", to segment more complex text queries such as \"the leftmost child wearing blue vest\"."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Env Setup"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First install `sam3` in your environment using the [installation instructions](https://github.com/facebookresearch/sam3?tab=readme-ov-file#installation) in the repository."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"# turn on tfloat32 for Ampere GPUs\n",
"# https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices\n",
"torch.backends.cuda.matmul.allow_tf32 = True\n",
"torch.backends.cudnn.allow_tf32 = True\n",
"\n",
"# use bfloat16 for the entire notebook. If your card doesn't support it, try float16 instead\n",
"torch.autocast(\"cuda\", dtype=torch.bfloat16).__enter__()\n",
"\n",
"# inference mode for the whole notebook. Disable if you need gradients\n",
"torch.inference_mode().__enter__()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"SAM3_ROOT = os.path.dirname(os.getcwd())\n",
"os.chdir(SAM3_ROOT)\n",
"\n",
"# setup GPU to use - A single GPU is good with the purpose of this demo\n",
"os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0\"\n",
"_ = os.system(\"nvidia-smi\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Build SAM3 Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sam3\n",
"from sam3 import build_sam3_image_model\n",
"from sam3.model.sam3_image_processor import Sam3Processor\n",
"\n",
"sam3_root = os.path.join(os.path.dirname(sam3.__file__), \"..\")\n",
"bpe_path = f\"{sam3_root}/assets/bpe_simple_vocab_16e6.txt.gz\"\n",
"model = build_sam3_image_model(bpe_path=bpe_path)\n",
"processor = Sam3Processor(model, confidence_threshold=0.5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## LLM Setup\n",
"\n",
"Config which MLLM to use, it can either be a model served by vLLM that you launch from your own machine or a model is served via external API. If you want to using a vLLM model, we also provided insturctions below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"LLM_CONFIGS = {\n",
" # vLLM-served models\n",
" \"qwen3_vl_8b_thinking\": {\n",
" \"provider\": \"vllm\",\n",
" \"model\": \"Qwen/Qwen3-VL-8B-Thinking\",\n",
" }, \n",
" # models served via external APIs\n",
" # add your own\n",
"}\n",
"\n",
"model = \"qwen3_vl_8b_thinking\"\n",
"LLM_API_KEY = \"DUMMY_API_KEY\"\n",
"\n",
"llm_config = LLM_CONFIGS[model]\n",
"llm_config[\"api_key\"] = LLM_API_KEY\n",
"llm_config[\"name\"] = model\n",
"\n",
"# setup API endpoint\n",
"if llm_config[\"provider\"] == \"vllm\":\n",
" LLM_SERVER_URL = \"http://0.0.0.0:8001/v1\" # replace this with your vLLM server address as needed\n",
"else:\n",
" LLM_SERVER_URL = llm_config[\"base_url\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Setup vLLM server \n",
"This step is only required if you are using a model served by vLLM, skip this step if you are calling LLM using an API like Gemini and GPT.\n",
"\n",
"* Install vLLM (in a separate conda env from SAM 3 to avoid dependency conflicts).\n",
" ```bash\n",
" conda create -n vllm python=3.12\n",
" pip install vllm --extra-index-url https://download.pytorch.org/whl/cu128\n",
" ```\n",
"* Start vLLM server on the same machine of this notebook\n",
" ```bash\n",
" # qwen 3 VL 8B thinking\n",
" vllm serve Qwen/Qwen3-VL-8B-Thinking --tensor-parallel-size 4 --allowed-local-media-path / --enforce-eager --port 8001\n",
" ```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run SAM3 Agent Inference"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from functools import partial\n",
"from IPython.display import display, Image\n",
"from sam3.agent.client_llm import send_generate_request as send_generate_request_orig\n",
"from sam3.agent.client_sam3 import call_sam_service as call_sam_service_orig\n",
"from sam3.agent.inference import run_single_image_inference"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"output": {
"id": 689664053567678,
"loadingStatus": "loaded"
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Copyright (c) Meta Platforms, Inc. and affiliates."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# SAM 3 Agent"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook shows an example of how an MLLM can use SAM 3 as a tool, i.e., \"SAM 3 Agent\", to segment more complex text queries such as \"the leftmost child wearing blue vest\"."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Env Setup"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First install `sam3` in your environment using the [installation instructions](https://github.com/facebookresearch/sam3?tab=readme-ov-file#installation) in the repository."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"# turn on tfloat32 for Ampere GPUs\n",
"# https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices\n",
"torch.backends.cuda.matmul.allow_tf32 = True\n",
"torch.backends.cudnn.allow_tf32 = True\n",
"\n",
"# use bfloat16 for the entire notebook. If your card doesn't support it, try float16 instead\n",
"torch.autocast(\"cuda\", dtype=torch.bfloat16).__enter__()\n",
"\n",
"# inference mode for the whole notebook. Disable if you need gradients\n",
"torch.inference_mode().__enter__()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"SAM3_ROOT = os.path.dirname(os.getcwd())\n",
"os.chdir(SAM3_ROOT)\n",
"\n",
"# setup GPU to use - A single GPU is good with the purpose of this demo\n",
"os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0\"\n",
"_ = os.system(\"nvidia-smi\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Build SAM3 Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sam3\n",
"from sam3 import build_sam3_image_model\n",
"from sam3.model.sam3_image_processor import Sam3Processor\n",
"\n",
"sam3_root = os.path.dirname(sam3.__file__)\n",
"bpe_path = f\"{sam3_root}/assets/bpe_simple_vocab_16e6.txt.gz\"\n",
"model = build_sam3_image_model(bpe_path=bpe_path)\n",
"processor = Sam3Processor(model, confidence_threshold=0.5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## LLM Setup\n",
"\n",
"Config which MLLM to use, it can either be a model served by vLLM that you launch from your own machine or a model is served via external API. If you want to using a vLLM model, we also provided insturctions below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"LLM_CONFIGS = {\n",
" # vLLM-served models\n",
" \"qwen3_vl_8b_thinking\": {\n",
" \"provider\": \"vllm\",\n",
" \"model\": \"Qwen/Qwen3-VL-8B-Thinking\",\n",
" },\n",
" # models served via external APIs\n",
" # add your own\n",
"}\n",
"\n",
"model = \"qwen3_vl_8b_thinking\"\n",
"LLM_API_KEY = \"DUMMY_API_KEY\"\n",
"\n",
"llm_config = LLM_CONFIGS[model]\n",
"llm_config[\"api_key\"] = LLM_API_KEY\n",
"llm_config[\"name\"] = model\n",
"\n",
"# setup API endpoint\n",
"if llm_config[\"provider\"] == \"vllm\":\n",
" LLM_SERVER_URL = \"http://0.0.0.0:8001/v1\" # replace this with your vLLM server address as needed\n",
"else:\n",
" LLM_SERVER_URL = llm_config[\"base_url\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Setup vLLM server \n",
"This step is only required if you are using a model served by vLLM, skip this step if you are calling LLM using an API like Gemini and GPT.\n",
"\n",
"* Install vLLM (in a separate conda env from SAM 3 to avoid dependency conflicts).\n",
" ```bash\n",
" conda create -n vllm python=3.12\n",
" pip install vllm --extra-index-url https://download.pytorch.org/whl/cu128\n",
" ```\n",
"* Start vLLM server on the same machine of this notebook\n",
" ```bash\n",
" # qwen 3 VL 8B thinking\n",
" vllm serve Qwen/Qwen3-VL-8B-Thinking --tensor-parallel-size 4 --allowed-local-media-path / --enforce-eager --port 8001\n",
" ```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run SAM3 Agent Inference"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from functools import partial\n",
"from IPython.display import display, Image\n",
"from sam3.agent.client_llm import send_generate_request as send_generate_request_orig\n",
"from sam3.agent.client_sam3 import call_sam_service as call_sam_service_orig\n",
"from sam3.agent.inference import run_single_image_inference"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"output": {
"id": 689664053567678,
"loadingStatus": "loaded"
}
},
"outputs": [],
"source": [
"# prepare input args and run single image inference\n",
"image = \"assets/images/test_image.jpg\"\n",
"prompt = \"the leftmost child wearing blue vest\"\n",
"image = os.path.abspath(image)\n",
"send_generate_request = partial(send_generate_request_orig, server_url=LLM_SERVER_URL, model=llm_config[\"model\"], api_key=llm_config[\"api_key\"])\n",
"call_sam_service = partial(call_sam_service_orig, sam3_processor=processor)\n",
"output_image_path = run_single_image_inference(\n",
" image, prompt, llm_config, send_generate_request, call_sam_service,\n",
" debug=True, output_dir=\"agent_output\"\n",
")\n",
"\n",
"# display output\n",
"if output_image_path is not None:\n",
" display(Image(filename=output_image_path))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"fileHeader": "",
"fileUid": "be59e249-6c09-4634-a9e7-1f06fd233c42",
"isAdHoc": false,
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.11"
}
},
"outputs": [],
"source": [
"# prepare input args and run single image inference\n",
"image = \"assets/images/test_image.jpg\"\n",
"prompt = \"the leftmost child wearing blue vest\"\n",
"image = os.path.abspath(image)\n",
"send_generate_request = partial(send_generate_request_orig, server_url=LLM_SERVER_URL, model=llm_config[\"model\"], api_key=llm_config[\"api_key\"])\n",
"call_sam_service = partial(call_sam_service_orig, sam3_processor=processor)\n",
"output_image_path = run_single_image_inference(\n",
" image, prompt, llm_config, send_generate_request, call_sam_service, \n",
" debug=True, output_dir=\"agent_output\"\n",
")\n",
"\n",
"# display output\n",
"if output_image_path is not None:\n",
" display(Image(filename=output_image_path))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"fileHeader": "",
"fileUid": "be59e249-6c09-4634-a9e7-1f06fd233c42",
"isAdHoc": false,
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.11"
}
},
"nbformat": 4,
"nbformat_minor": 4
"nbformat": 4,
"nbformat_minor": 2
}

10
pyproject.toml
View File

@@ -26,7 +26,7 @@ classifiers = [
]
dependencies = [
"timm>=1.0.17",
"numpy==1.26",
"numpy>=1.26,<2",
"tqdm",
"ftfy==6.1.1",
"regex",
@@ -82,8 +82,12 @@ train = [
"Homepage" = "https://github.com/facebookresearch/sam3"
"Bug Tracker" = "https://github.com/facebookresearch/sam3/issues"
[tool.setuptools]
packages = ["sam3", "sam3.model"]
[tool.setuptools.packages.find]
include = ["sam3*"]
exclude = ["build*", "scripts*", "examples*"]
[tool.setuptools.package-data]
sam3 = ["assets/*.txt.gz"]
[tool.setuptools.dynamic]
version = {attr = "sam3.__version__"}

2
sam3/__init__.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
from .model_builder import build_sam3_image_model
__version__ = "0.1.0"

2
sam3/agent/__init__.py
View File

@@ -1 +1,3 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe

18
sam3/agent/agent_core.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import copy
import json
import os
@@ -294,9 +296,9 @@ def agent_inference(
assert LATEST_SAM3_TEXT_PROMPT != ""
# Make sure that the last message is a image
assert (
messages[-1]["content"][1]["type"] == "image"
), "Second content element should be an image"
assert messages[-1]["content"][1]["type"] == "image", (
"Second content element should be an image"
)
messages.pop() # Remove the last user message
# Add simplified replacement message
simplified_message = {
@@ -316,7 +318,7 @@ def agent_inference(
# MLLM check the mask one by one
for i in range(num_masks):
print(f"🔍 Checking mask {i+1}/{num_masks}...")
print(f"🔍 Checking mask {i + 1}/{num_masks}...")
image_w_mask_i, image_w_zoomed_in_mask_i = visualize(current_outputs, i)
image_w_zoomed_in_mask_i_path = os.path.join(
@@ -361,7 +363,7 @@ def agent_inference(
raise ValueError(
"Generated text is None, which is unexpected. Please check the Qwen server and the input parameters."
)
print(f"Generated text for mask {i+1}: {checking_generated_text}")
print(f"Generated text for mask {i + 1}: {checking_generated_text}")
verdict = (
checking_generated_text.split("<verdict>")[-1]
.split("</verdict>")[0]
@@ -369,11 +371,11 @@ def agent_inference(
)
if "Accept" in verdict:
assert not "Reject" in verdict
print(f"Mask {i+1} accepted, keeping it in the outputs.")
print(f"Mask {i + 1} accepted, keeping it in the outputs.")
masks_to_keep.append(i)
elif "Reject" in verdict:
assert not "Accept" in verdict
print(f"Mask {i+1} rejected, removing it from the outputs.")
print(f"Mask {i + 1} rejected, removing it from the outputs.")
else:
raise ValueError(
f"Unexpected verdict in generated text: {checking_generated_text}. Expected 'Accept' or 'Reject'."
@@ -395,7 +397,7 @@ def agent_inference(
sam_output_dir, rf"{LATEST_SAM3_TEXT_PROMPT}.png"
).replace(
".png",
f"_selected_masks_{'-'.join(map(str, [i+1 for i in masks_to_keep]))}.png".replace(
f"_selected_masks_{'-'.join(map(str, [i + 1 for i in masks_to_keep]))}.png".replace(
"/", "_"
),
)

2
sam3/agent/client_llm.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import base64
import os
from typing import Any, Optional

3
sam3/agent/client_sam3.py
View File

@@ -1,11 +1,12 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import json
import os
import torch
from PIL import Image
from sam3.model.box_ops import box_xyxy_to_xywh
from sam3.train.masks_ops import rle_encode

2
sam3/agent/helpers/__init__.py
View File

@@ -1 +1,3 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe

16
sam3/agent/helpers/boxes.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import math
from enum import IntEnum, unique
from typing import List, Tuple, Union
@@ -82,9 +84,9 @@ class BoxMode(IntEnum):
], "Relative mode not yet supported!"
if from_mode == BoxMode.XYWHA_ABS and to_mode == BoxMode.XYXY_ABS:
assert (
arr.shape[-1] == 5
), "The last dimension of input shape must be 5 for XYWHA format"
assert arr.shape[-1] == 5, (
"The last dimension of input shape must be 5 for XYWHA format"
)
original_dtype = arr.dtype
arr = arr.double()
@@ -242,9 +244,9 @@ class Boxes:
if isinstance(item, int):
return Boxes(self.tensor[item].view(1, -1))
b = self.tensor[item]
assert (
b.dim() == 2
), "Indexing on Boxes with {} failed to return a matrix!".format(item)
assert b.dim() == 2, (
"Indexing on Boxes with {} failed to return a matrix!".format(item)
)
return Boxes(b)
def __len__(self) -> int:
@@ -423,7 +425,7 @@ def matched_pairwise_iou(boxes1: Boxes, boxes2: Boxes) -> torch.Tensor:
Tensor: iou, sized [N].
"""
assert len(boxes1) == len(boxes2), (
"boxlists should have the same" "number of entries, got {}, {}".format(
"boxlists should have the samenumber of entries, got {}, {}".format(
len(boxes1), len(boxes2)
)
)

2
sam3/agent/helpers/color_map.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""
An awesome colormap for really neat visualizations.
Copied from Detectron, and removed gray colors.

2
sam3/agent/helpers/keypoints.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
from typing import Any, List, Tuple, Union
import numpy as np

2
sam3/agent/helpers/mask_overlap_removal.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
from typing import Dict, List
import numpy as np

11
sam3/agent/helpers/masks.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import copy
import itertools
from typing import Any, Iterator, List, Union
@@ -11,7 +13,6 @@ from torch import device
from .boxes import Boxes
from .memory import retry_if_cuda_oom
from .roi_align import ROIAlign
@@ -140,10 +141,10 @@ class BitMasks:
if isinstance(item, int):
return BitMasks(self.tensor[item].unsqueeze(0))
m = self.tensor[item]
assert (
m.dim() == 3
), "Indexing on BitMasks with {} returns a tensor with shape {}!".format(
item, m.shape
assert m.dim() == 3, (
"Indexing on BitMasks with {} returns a tensor with shape {}!".format(
item, m.shape
)
)
return BitMasks(m)

2
sam3/agent/helpers/memory.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import logging
from contextlib import contextmanager
from functools import wraps

2
sam3/agent/helpers/rle.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""Some utilities for RLE encoding that doesn't require downloading the masks to the cpu"""
import numpy as np

2
sam3/agent/helpers/roi_align.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
from torch import nn
from torchvision.ops import roi_align

8
sam3/agent/helpers/rotated_boxes.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
from __future__ import absolute_import, division, print_function, unicode_literals
import math
@@ -361,9 +363,9 @@ class RotatedBoxes(Boxes):
if isinstance(item, int):
return RotatedBoxes(self.tensor[item].view(1, -1))
b = self.tensor[item]
assert (
b.dim() == 2
), "Indexing on RotatedBoxes with {} failed to return a matrix!".format(item)
assert b.dim() == 2, (
"Indexing on RotatedBoxes with {} failed to return a matrix!".format(item)
)
return RotatedBoxes(b)
def __len__(self) -> int:

2
sam3/agent/helpers/som_utils.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import colorsys
from dataclasses import dataclass
from typing import List, Tuple

9
sam3/agent/helpers/visualizer.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import colorsys
import logging
import math
@@ -18,7 +20,6 @@ from matplotlib.backends.backend_agg import FigureCanvasAgg
from PIL import Image
from .boxes import Boxes, BoxMode
from .color_map import random_color
from .keypoints import Keypoints
from .masks import BitMasks, PolygonMasks
@@ -220,9 +221,9 @@ class _PanopticPrediction:
empty_ids.append(id)
if len(empty_ids) == 0:
return np.zeros(self._seg.shape, dtype=np.uint8)
assert (
len(empty_ids) == 1
), ">1 ids corresponds to no labels. This is currently not supported"
assert len(empty_ids) == 1, (
">1 ids corresponds to no labels. This is currently not supported"
)
return (self._seg != empty_ids[0]).numpy().astype(np.bool)
def semantic_masks(self):

2
sam3/agent/helpers/zoom_in.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import io
import math

6
sam3/agent/inference.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import json
import os
@@ -39,7 +41,7 @@ def run_single_image_inference(
print(f"Output JSON {output_json_path} already exists. Skipping.")
return
print(f"{'-'*30} Starting SAM 3 Agent Session... {'-'*30} ")
print(f"{'-' * 30} Starting SAM 3 Agent Session... {'-' * 30} ")
agent_history, final_output_dict, rendered_final_output = agent_inference(
image_path,
text_prompt,
@@ -48,7 +50,7 @@ def run_single_image_inference(
output_dir=output_dir,
debug=debug,
)
print(f"{'-'*30} End of SAM 3 Agent Session... {'-'*30} ")
print(f"{'-' * 30} End of SAM 3 Agent Session... {'-' * 30} ")
final_output_dict["text_prompt"] = text_prompt
final_output_dict["image_path"] = image_path

6
sam3/agent/viz.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import cv2
import numpy as np
import pycocotools.mask as mask_utils
@@ -71,7 +73,9 @@ def visualize(
idx = int(zoom_in_index)
num_masks = len(input_json.get("pred_masks", []))
if idx < 0 or idx >= num_masks:
raise ValueError(f"zoom_in_index {idx} is out of range (0..{num_masks-1}).")
raise ValueError(
f"zoom_in_index {idx} is out of range (0..{num_masks - 1})."
)
# (1) Replicate zoom_in_and_visualize
object_data = {

0
assets/bpe_simple_vocab_16e6.txt.gz → sam3/assets/bpe_simple_vocab_16e6.txt.gz
View File

2
sam3/eval/__init__.py
View File

@@ -1 +1,3 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe

38
sam3/eval/cgf1_eval.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import contextlib
import copy
import json
@@ -124,9 +126,9 @@ class COCOCustom(COCO):
# MODIFICATION: faster and cached subset check
if not hasattr(self, "img_id_set"):
self.img_id_set = set(self.getImgIds())
assert set(annsImgIds).issubset(
self.img_id_set
), "Results do not correspond to current coco set"
assert set(annsImgIds).issubset(self.img_id_set), (
"Results do not correspond to current coco set"
)
# END MODIFICATION
if "caption" in anns[0]:
imgIds = set([img["id"] for img in res.dataset["images"]]) & set(
@@ -299,9 +301,9 @@ class CGF1Eval(COCOeval):
TP = (match_scores >= thresh).sum()
FP = len(dt) - TP
FN = len(gt) - TP
assert (
FP >= 0 and FN >= 0
), f"FP: {FP}, FN: {FN}, TP: {TP}, match_scores: {match_scores}, len(dt): {len(dt)}, len(gt): {len(gt)}, ious: {ious}"
assert FP >= 0 and FN >= 0, (
f"FP: {FP}, FN: {FN}, TP: {TP}, match_scores: {match_scores}, len(dt): {len(dt)}, len(gt): {len(gt)}, ious: {ious}"
)
TPs.append(TP)
FPs.append(FP)
FNs.append(FN)
@@ -597,9 +599,9 @@ class CGF1Evaluator:
"""
assert len(self.coco_gts) > 0, "No ground truth provided for evaluation."
assert len(self.coco_gts) == len(
self.coco_evals
), "Mismatch in number of ground truths and evaluators."
assert len(self.coco_gts) == len(self.coco_evals), (
"Mismatch in number of ground truths and evaluators."
)
if self.verbose:
print(f"Loading predictions from {pred_file}")
@@ -666,17 +668,17 @@ class CGF1Evaluator:
if len(scorings) == 1:
return scorings[0]
assert (
scorings[0].ndim == 3
), f"Expecting results in [numCats, numAreas, numImgs] format, got {scorings[0].shape}"
assert (
scorings[0].shape[0] == 1
), f"Expecting a single category, got {scorings[0].shape[0]}"
assert scorings[0].ndim == 3, (
f"Expecting results in [numCats, numAreas, numImgs] format, got {scorings[0].shape}"
)
assert scorings[0].shape[0] == 1, (
f"Expecting a single category, got {scorings[0].shape[0]}"
)
for scoring in scorings:
assert (
scoring.shape == scorings[0].shape
), f"Shape mismatch: {scoring.shape}, {scorings[0].shape}"
assert scoring.shape == scorings[0].shape, (
f"Shape mismatch: {scoring.shape}, {scorings[0].shape}"
)
selected_imgs = []
for img_id in range(scorings[0].shape[-1]):

12
sam3/eval/coco_eval.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""
COCO evaluator that works in distributed mode.
@@ -16,19 +18,15 @@ import os
import pickle
from collections import defaultdict
from pathlib import Path
from typing import Any, List, Optional
import numpy as np
import pycocotools.mask as mask_utils
import torch
from iopath.common.file_io import g_pathmgr
from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval
from sam3.train.masks_ops import rle_encode
from sam3.train.utils.distributed import (
all_gather,
gather_to_rank_0_via_filesys,
@@ -753,9 +751,9 @@ def loadRes(self, resFile):
anns = resFile
assert type(anns) == list, "results in not an array of objects"
annsImgIds = [ann["image_id"] for ann in anns]
assert set(annsImgIds) == (
set(annsImgIds) & set(self.getImgIds())
), "Results do not correspond to current coco set"
assert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), (
"Results do not correspond to current coco set"
)
if "caption" in anns[0]:
imgIds = set([img["id"] for img in res.dataset["images"]]) & set(
[ann["image_id"] for ann in anns]

2
sam3/eval/coco_eval_offline.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""
This evaluator is meant for regular COCO mAP evaluation, for example on the COCO val set.

2
sam3/eval/coco_reindex.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""
Self-contained COCO JSON re-indexing function that creates temporary files.
"""

8
sam3/eval/coco_writer.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""
COCO prediction dumper for distributed training.
@@ -81,9 +83,9 @@ class PredictionDumper:
self.merge_predictions = merge_predictions
self.pred_file_evaluators = pred_file_evaluators
if self.pred_file_evaluators is not None:
assert (
merge_predictions
), "merge_predictions must be True if pred_file_evaluators are provided"
assert merge_predictions, (
"merge_predictions must be True if pred_file_evaluators are provided"
)
assert self.dump_dir is not None, "dump_dir must be provided"
if is_main_process():

2
sam3/eval/conversion_util.py
View File

@@ -1,4 +1,6 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import json
import os
from collections import defaultdict

28
sam3/eval/demo_eval.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""
This evaluator is based upon COCO evaluation, but evaluates the model in a "demo" setting.
This means that the model's predictions are thresholded and evaluated as "hard" predictions.
@@ -11,11 +13,9 @@ from typing import Optional
import numpy as np
import pycocotools.mask as maskUtils
from pycocotools.cocoeval import COCOeval
from sam3.eval.coco_eval import CocoEvaluator
from sam3.train.masks_ops import compute_F_measure
from sam3.train.utils.distributed import is_main_process
from scipy.optimize import linear_sum_assignment
@@ -154,9 +154,9 @@ class DemoEval(COCOeval):
TP = (match_scores >= thresh).sum()
FP = len(dt) - TP
FN = len(gt) - TP
assert (
FP >= 0 and FN >= 0
), f"FP: {FP}, FN: {FN}, TP: {TP}, match_scores: {match_scores}, len(dt): {len(dt)}, len(gt): {len(gt)}, ious: {ious}"
assert FP >= 0 and FN >= 0, (
f"FP: {FP}, FN: {FN}, TP: {TP}, match_scores: {match_scores}, len(dt): {len(dt)}, len(gt): {len(gt)}, ious: {ious}"
)
TPs.append(TP)
FPs.append(FP)
FNs.append(FN)
@@ -526,17 +526,17 @@ class DemoEvaluator(CocoEvaluator):
if len(scorings) == 1:
return scorings[0]
assert (
scorings[0].ndim == 3
), f"Expecting results in [numCats, numAreas, numImgs] format, got {scorings[0].shape}"
assert (
scorings[0].shape[0] == 1
), f"Expecting a single category, got {scorings[0].shape[0]}"
assert scorings[0].ndim == 3, (
f"Expecting results in [numCats, numAreas, numImgs] format, got {scorings[0].shape}"
)
assert scorings[0].shape[0] == 1, (
f"Expecting a single category, got {scorings[0].shape[0]}"
)
for scoring in scorings:
assert (
scoring.shape == scorings[0].shape
), f"Shape mismatch: {scoring.shape}, {scorings[0].shape}"
assert scoring.shape == scorings[0].shape, (
f"Shape mismatch: {scoring.shape}, {scorings[0].shape}"
)
selected_imgs = []
for img_id in range(scorings[0].shape[-1]):

2
sam3/eval/hota_eval_toolkit/__init__.py
View File

@@ -1 +1,3 @@
# flake8: noqa
# pyre-unsafe

2
sam3/eval/hota_eval_toolkit/run_ytvis_eval.py
View File

@@ -1,5 +1,7 @@
# flake8: noqa
# pyre-unsafe
"""run_youtube_vis.py
Run example:
run_youtube_vis.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL STEm_Seg

2
sam3/eval/hota_eval_toolkit/trackeval/__init__.py
View File

@@ -1,4 +1,6 @@
# flake8: noqa
# pyre-unsafe
from . import datasets, metrics, utils
from .eval import Evaluator

2
sam3/eval/hota_eval_toolkit/trackeval/_timing.py
View File

@@ -1,5 +1,7 @@
# flake8: noqa
# pyre-unsafe
import inspect
from functools import wraps
from time import perf_counter

2
sam3/eval/hota_eval_toolkit/trackeval/datasets/__init__.py
View File

@@ -1,4 +1,6 @@
# flake8: noqa
# pyre-unsafe
from .tao_ow import TAO_OW
from .youtube_vis import YouTubeVIS

2
sam3/eval/hota_eval_toolkit/trackeval/datasets/_base_dataset.py
View File

@@ -1,5 +1,7 @@
# flake8: noqa
# pyre-unsafe
import csv
import io
import os

2
sam3/eval/hota_eval_toolkit/trackeval/datasets/tao_ow.py
View File

@@ -1,5 +1,7 @@
# flake8: noqa
# pyre-unsafe
import itertools
import json
import os

2
sam3/eval/hota_eval_toolkit/trackeval/datasets/youtube_vis.py
View File

@@ -1,5 +1,7 @@
# flake8: noqa
# pyre-unsafe
# note: this file has been modified from its original version in TrackEval in
# https://github.com/JonathonLuiten/TrackEval/blob/master/trackeval/datasets/youtube_vis.py
# to support the following:

9
sam3/eval/hota_eval_toolkit/trackeval/eval.py
View File

@@ -1,5 +1,7 @@
# flake8: noqa
# pyre-unsafe
import os
import time
import traceback
@@ -253,9 +255,10 @@ class Evaluator:
if show_progressbar and TQDM_IMPORTED:
seq_list_sorted = sorted(seq_list)
with Pool(config["NUM_PARALLEL_CORES"]) as pool, tqdm.tqdm(
total=len(seq_list)
) as pbar:
with (
Pool(config["NUM_PARALLEL_CORES"]) as pool,
tqdm.tqdm(total=len(seq_list)) as pbar,
):
_eval_sequence = partial(
eval_sequence,
dataset=dataset,

2
sam3/eval/hota_eval_toolkit/trackeval/metrics/__init__.py
View File

@@ -1,4 +1,6 @@
# flake8: noqa
# pyre-unsafe
from .count import Count
from .hota import HOTA

2
sam3/eval/hota_eval_toolkit/trackeval/metrics/_base_metric.py
View File

@@ -1,5 +1,7 @@
# flake8: noqa
# pyre-unsafe
from abc import ABC, abstractmethod
import numpy as np

2
sam3/eval/hota_eval_toolkit/trackeval/metrics/count.py
View File

@@ -1,5 +1,7 @@
# flake8: noqa
# pyre-unsafe
from .. import _timing
from ._base_metric import _BaseMetric

2
sam3/eval/hota_eval_toolkit/trackeval/metrics/hota.py
View File

@@ -1,5 +1,7 @@
# flake8: noqa
# pyre-unsafe
import os
import numpy as np

2
sam3/eval/hota_eval_toolkit/trackeval/utils.py
View File

@@ -1,5 +1,7 @@
# flake8: noqa
# pyre-unsafe
import argparse
import csv
import os

18
sam3/eval/postprocessors.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""Postprocessors class to transform MDETR output according to the downstream task"""
import dataclasses
@@ -81,9 +83,9 @@ class PostProcessImage(nn.Module):
ret_tensordict: Experimental argument. If true, return a tensordict.TensorDict instead of a list of dictionaries for easier manipulation.
"""
if ret_tensordict:
assert (
consistent is True
), "We don't support returning TensorDict if the outputs have different shapes" # NOTE: It's possible but we don't support it.
assert consistent is True, (
"We don't support returning TensorDict if the outputs have different shapes"
) # NOTE: It's possible but we don't support it.
assert self.detection_threshold <= 0.0, "TODO: implement?"
try:
from tensordict import TensorDict
@@ -116,7 +118,9 @@ class PostProcessImage(nn.Module):
if boxes is None:
assert out_masks is not None
assert not ret_tensordict, "We don't support returning TensorDict if the output does not contain boxes"
assert not ret_tensordict, (
"We don't support returning TensorDict if the output does not contain boxes"
)
B = len(out_masks)
boxes = [None] * B
scores = [None] * B
@@ -416,9 +420,9 @@ class PostProcessAPIVideo(PostProcessImage):
if video_id == -1:
video_id = unique_vid_id.item()
else:
assert (
video_id == unique_vid_id.item()
), "We can only postprocess one video per datapoint"
assert video_id == unique_vid_id.item(), (
"We can only postprocess one video per datapoint"
)
# keeping track of which objects appear in the current frame
obj_ids_per_frame = frame_outs["pred_object_ids"]
assert obj_ids_per_frame.size(-1) == frame_outs["pred_logits"].size(-2)

2
sam3/eval/saco_veval_eval.py
View File

@@ -1,4 +1,6 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import argparse
import json
import os

2
sam3/eval/saco_veval_evaluators.py
View File

@@ -1,4 +1,6 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import json
import os
import tempfile

2
sam3/eval/teta_eval_toolkit/__init__.py
View File

@@ -1,5 +1,7 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
from . import config, datasets, metrics, utils
from .eval import Evaluator

2
sam3/eval/teta_eval_toolkit/_timing.py
View File

@@ -1,6 +1,8 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
import inspect
from functools import wraps
from time import perf_counter

2
sam3/eval/teta_eval_toolkit/config.py
View File

@@ -1,6 +1,8 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
"""Config."""
import argparse
import os

2
sam3/eval/teta_eval_toolkit/datasets/__init__.py
View File

@@ -1,5 +1,7 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
"""Datasets."""
from .coco import COCO
from .tao import TAO

2
sam3/eval/teta_eval_toolkit/datasets/_base_dataset.py
View File

@@ -1,6 +1,8 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
import csv
import io
import os

2
sam3/eval/teta_eval_toolkit/datasets/coco.py
View File

@@ -1,6 +1,8 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
"""COCO Dataset."""
import copy
import itertools

2
sam3/eval/teta_eval_toolkit/datasets/tao.py
View File

@@ -1,6 +1,8 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
"""TAO Dataset."""
import copy
import itertools

2
sam3/eval/teta_eval_toolkit/eval.py
View File

@@ -1,6 +1,8 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
import copy
import os
import pickle

2
sam3/eval/teta_eval_toolkit/metrics/__init__.py
View File

@@ -1,4 +1,6 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
from .teta import TETA

2
sam3/eval/teta_eval_toolkit/metrics/_base_metric.py
View File

@@ -1,6 +1,8 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
from abc import ABC, abstractmethod
import numpy as np

2
sam3/eval/teta_eval_toolkit/metrics/teta.py
View File

@@ -1,6 +1,8 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
"""Track Every Thing Accuracy metric."""
import numpy as np

2
sam3/eval/teta_eval_toolkit/utils.py
View File

@@ -1,6 +1,8 @@
# fmt: off
# flake8: noqa
# pyre-unsafe
import csv
import os
from collections import OrderedDict

8
sam3/eval/ytvis_coco_wrapper.py
View File

@@ -1,6 +1,8 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# (c) Meta Platforms, Inc. and affiliates. Confidential and proprietary.
# pyre-unsafe
import copy
import json
import logging
@@ -93,9 +95,9 @@ class YTVIS(COCO):
anns = resFile
assert type(anns) == list, "results is not an array of objects"
annsImgIds = [ann["image_id"] for ann in anns]
assert set(annsImgIds) == (
set(annsImgIds) & set(self.getImgIds())
), "Results do not correspond to current coco set"
assert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), (
"Results do not correspond to current coco set"
)
if "bboxes" in anns[0] and not anns[0]["bboxes"] == []:
res.dataset["categories"] = copy.deepcopy(self.dataset["categories"])
for id, ann in enumerate(anns):

36
sam3/eval/ytvis_eval.py
View File

@@ -1,4 +1,6 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import copy
import gc
import logging
@@ -107,9 +109,7 @@ class YTVISevalMixin:
) # Num preds x Num GTS x Num frames
inter = inter.sum(-1)
union = union.sum(-1)
assert (
union > 0
).all(), (
assert (union > 0).all(), (
"There exists a tracklet with zero GTs across time. This is suspicious"
)
return inter / union
@@ -134,9 +134,9 @@ class YTVISevalMixin:
iou = inter / union
assert iou >= 0 and iou <= 1, "Encountered an error in IoU computation"
else:
assert np.isclose(inter, 0) and np.isclose(
union, 0
), "Encountered an error in IoU computation"
assert np.isclose(inter, 0) and np.isclose(union, 0), (
"Encountered an error in IoU computation"
)
iou = 1
return iou
@@ -204,16 +204,16 @@ class YTVISResultsWriter:
if len(prediction) == 0:
continue
for k in ["boxes", "scores", "labels"]:
assert (
k in prediction
), f"Expected predictions to have `{k}` key, available keys are {prediction.keys()}"
assert k in prediction, (
f"Expected predictions to have `{k}` key, available keys are {prediction.keys()}"
)
if self.save_per_frame_scores:
assert (
"per_frame_scores" in prediction
), f"Expected predictions to have `per_frame_scores` key, available keys are {prediction.keys()}"
assert xor(
"masks" in prediction, "masks_rle" in prediction
), f"Expected predictions to have either `masks` key or `masks_rle` key, available keys are {prediction.keys()}"
assert "per_frame_scores" in prediction, (
f"Expected predictions to have `per_frame_scores` key, available keys are {prediction.keys()}"
)
assert xor("masks" in prediction, "masks_rle" in prediction), (
f"Expected predictions to have either `masks` key or `masks_rle` key, available keys are {prediction.keys()}"
)
boxes = prediction["boxes"]
boxes = convert_to_xywh(boxes).tolist()
@@ -221,9 +221,9 @@ class YTVISResultsWriter:
labels = prediction["labels"].tolist()
if "masks" in prediction:
masks = prediction["masks"].squeeze(2)
assert (
masks.ndim == 4
), "Expected masks to be of shape(N_preds,T_frames,H,W)"
assert masks.ndim == 4, (
"Expected masks to be of shape(N_preds,T_frames,H,W)"
)
areas = [mask.flatten(1).sum(1).tolist() for mask in masks]
rles = [rle_encode(masklet) for masklet in masks]

8
sam3/logger.py
View File

@@ -1,4 +1,6 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import logging
import os
@@ -40,9 +42,9 @@ def get_logger(name, level=logging.INFO):
"""A command line logger."""
if "LOG_LEVEL" in os.environ:
level = os.environ["LOG_LEVEL"].upper()
assert (
level in LOG_LEVELS
), f"Invalid LOG_LEVEL: {level}, must be one of {list(LOG_LEVELS.keys())}"
assert level in LOG_LEVELS, (
f"Invalid LOG_LEVEL: {level}, must be one of {list(LOG_LEVELS.keys())}"
)
level = LOG_LEVELS[level]
logger = logging.getLogger(name)
logger.setLevel(level)

2
sam3/model/__init__.py
View File

@@ -1 +1,3 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe

2
sam3/model/act_ckpt_utils.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import inspect
from functools import wraps
from typing import Callable, TypeVar, Union

2
sam3/model/box_ops.py
View File

@@ -1,4 +1,6 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""
Utilities for bounding box manipulation and GIoU.
"""

9
sam3/model/data_misc.py
View File

@@ -1,11 +1,12 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""
Misc functions, including distributed helpers.
"""
import collections
import re
from dataclasses import dataclass, field as field_ptr_behaviour, fields, is_dataclass
from typing import Any, get_args, get_origin, List, Mapping, Optional, Sequence, Union
@@ -27,9 +28,9 @@ def interpolate(
input, size, scale_factor, mode, align_corners
)
assert (
input.shape[0] != 0 or input.shape[1] != 0
), "At least one of the two first dimensions must be non zero"
assert input.shape[0] != 0 or input.shape[1] != 0, (
"At least one of the two first dimensions must be non zero"
)
if input.shape[1] == 0:
# Pytorch doesn't support null dimension on the channel dimension, so we transpose to fake a null batch dim

31
sam3/model/decoder.py
View File

@@ -1,4 +1,6 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""
Transformer decoder.
Inspired from Pytorch's version, adds the pre-norm variant
@@ -7,18 +9,13 @@ Inspired from Pytorch's version, adds the pre-norm variant
from typing import Any, Dict, List, Optional
import numpy as np
import torch
from sam3.sam.transformer import RoPEAttention
from torch import nn, Tensor
from torchvision.ops.roi_align import RoIAlign
from .act_ckpt_utils import activation_ckpt_wrapper
from .box_ops import box_cxcywh_to_xyxy
from .model_misc import (
gen_sineembed_for_position,
get_activation_fn,
@@ -442,9 +439,9 @@ class TransformerDecoder(nn.Module):
- valid_ratios/spatial_shapes: bs, nlevel, 2
"""
if memory_mask is not None:
assert (
self.boxRPB == "none"
), "inputting a memory_mask in the presence of boxRPB is unexpected/not implemented"
assert self.boxRPB == "none", (
"inputting a memory_mask in the presence of boxRPB is unexpected/not implemented"
)
apply_dac = apply_dac if apply_dac is not None else self.dac
if apply_dac:
@@ -514,18 +511,18 @@ class TransformerDecoder(nn.Module):
query_pos = self.ref_point_head(query_sine_embed) # nq, bs, d_model
if self.boxRPB != "none" and reference_boxes is not None:
assert (
spatial_shapes.shape[0] == 1
), "only single scale support implemented"
assert spatial_shapes.shape[0] == 1, (
"only single scale support implemented"
)
memory_mask = self._get_rpb_matrix(
reference_boxes,
(spatial_shapes[0, 0], spatial_shapes[0, 1]),
)
memory_mask = memory_mask.flatten(0, 1) # (bs*n_heads, nq, H*W)
if self.training:
assert (
self.use_act_checkpoint
), "Activation checkpointing not enabled in the decoder"
assert self.use_act_checkpoint, (
"Activation checkpointing not enabled in the decoder"
)
output, presence_out = activation_ckpt_wrapper(layer)(
tgt=output,
tgt_query_pos=query_pos,
@@ -674,9 +671,9 @@ class TransformerEncoderCrossAttention(nn.Module):
src_pos[0],
)
assert (
src.shape[1] == prompt.shape[1]
), "Batch size must be the same for src and prompt"
assert src.shape[1] == prompt.shape[1], (
"Batch size must be the same for src and prompt"
)
output = src

2
sam3/model/edt.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""Triton kernel for euclidean distance transform (EDT)"""
import torch

20
sam3/model/encoder.py
View File

@@ -1,6 +1,8 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# Based on https://github.com/IDEA-Research/GroundingDINO
# pyre-unsafe
from typing import Any, Dict, List, Optional, Tuple
import torch
@@ -320,9 +322,9 @@ class TransformerEncoder(nn.Module):
return reference_points
def _prepare_multilevel_features(self, srcs, masks, pos_embeds):
assert (
len(srcs) == self.num_feature_levels
), "mismatch between expected and received # of feature levels"
assert len(srcs) == self.num_feature_levels, (
"mismatch between expected and received # of feature levels"
)
src_flatten = []
mask_flatten = []
@@ -404,9 +406,9 @@ class TransformerEncoder(nn.Module):
- spatial_shapes: Spatial dimensions of each feature level
- valid_ratios: Valid ratios for each feature level
"""
assert (
len(src) == self.num_feature_levels
), "must be equal to num_feature_levels"
assert len(src) == self.num_feature_levels, (
"must be equal to num_feature_levels"
)
if src_key_padding_masks is not None:
assert len(src_key_padding_masks) == self.num_feature_levels
if pos is not None:
@@ -536,9 +538,9 @@ class TransformerEncoderFusion(TransformerEncoder):
else None
)
else:
assert all(
x.dim == 4 for x in src
), "expected list of (bs, c, h, w) tensors"
assert all(x.dim == 4 for x in src), (
"expected list of (bs, c, h, w) tensors"
)
if self.add_pooled_text_to_img_feat:
# Fusion: Add mean pooled text to image features

214
sam3/model/geometry_encoders.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
from typing import Tuple
import torch
@@ -9,7 +11,6 @@ from typing_extensions import override
from .act_ckpt_utils import activation_ckpt_wrapper
from .box_ops import box_cxcywh_to_xyxy
from .model_misc import get_clones
@@ -146,54 +147,42 @@ class Prompt:
)
# Dimension checks
assert (
box_embeddings is not None
and list(box_embeddings.shape[:2])
== [
box_seq_len,
bs,
]
), f"Wrong dimension for box embeddings. Expected [{box_seq_len}, {bs}, *] got {box_embeddings.shape}"
assert (
box_mask is not None
and list(box_mask.shape)
== [
bs,
box_seq_len,
]
), f"Wrong dimension for box mask. Expected [{bs}, {box_seq_len}] got {box_mask.shape}"
assert (
point_embeddings is not None
and list(point_embeddings.shape[:2])
== [
point_seq_len,
bs,
]
), f"Wrong dimension for point embeddings. Expected [{point_seq_len}, {bs}, *] got {point_embeddings.shape}"
assert (
point_mask is not None
and list(point_mask.shape)
== [
bs,
point_seq_len,
]
), f"Wrong dimension for point mask. Expected [{bs}, {point_seq_len}] got {point_mask.shape}"
assert (
box_labels is not None
and list(box_labels.shape)
== [
box_seq_len,
bs,
]
), f"Wrong dimension for box labels. Expected [{box_seq_len}, {bs}] got {box_labels.shape}"
assert (
point_labels is not None
and list(point_labels.shape)
== [
point_seq_len,
bs,
]
), f"Wrong dimension for point labels. Expected [{point_seq_len}, {bs}] got {point_labels.shape}"
assert box_embeddings is not None and list(box_embeddings.shape[:2]) == [
box_seq_len,
bs,
], (
f"Wrong dimension for box embeddings. Expected [{box_seq_len}, {bs}, *] got {box_embeddings.shape}"
)
assert box_mask is not None and list(box_mask.shape) == [
bs,
box_seq_len,
], (
f"Wrong dimension for box mask. Expected [{bs}, {box_seq_len}] got {box_mask.shape}"
)
assert point_embeddings is not None and list(point_embeddings.shape[:2]) == [
point_seq_len,
bs,
], (
f"Wrong dimension for point embeddings. Expected [{point_seq_len}, {bs}, *] got {point_embeddings.shape}"
)
assert point_mask is not None and list(point_mask.shape) == [
bs,
point_seq_len,
], (
f"Wrong dimension for point mask. Expected [{bs}, {point_seq_len}] got {point_mask.shape}"
)
assert box_labels is not None and list(box_labels.shape) == [
box_seq_len,
bs,
], (
f"Wrong dimension for box labels. Expected [{box_seq_len}, {bs}] got {box_labels.shape}"
)
assert point_labels is not None and list(point_labels.shape) == [
point_seq_len,
bs,
], (
f"Wrong dimension for point labels. Expected [{point_seq_len}, {bs}] got {point_labels.shape}"
)
assert (
# Allowed to be None, we leave it to the encoder to check for validity before encoding.
mask_embeddings is None
@@ -202,41 +191,41 @@ class Prompt:
mask_seq_len,
bs,
]
), f"Wrong dimension for mask embeddings. Expected [{mask_seq_len}, {bs}, *] got {mask_embeddings.shape}"
assert (
mask_mask is None
or list(mask_mask.shape)
== [
bs,
mask_seq_len,
]
), f"Wrong dimension for mask attn. mask. Expected [{bs}, {mask_seq_len}] got {mask_mask.shape}"
), (
f"Wrong dimension for mask embeddings. Expected [{mask_seq_len}, {bs}, *] got {mask_embeddings.shape}"
)
assert mask_mask is None or list(mask_mask.shape) == [
bs,
mask_seq_len,
], (
f"Wrong dimension for mask attn. mask. Expected [{bs}, {mask_seq_len}] got {mask_mask.shape}"
)
# Device checks
assert (
box_embeddings is not None and box_embeddings.device == device
), f"Expected box embeddings to be on device {device}, got {box_embeddings.device}"
assert (
box_mask is not None and box_mask.device == device
), f"Expected box mask to be on device {device}, got {box_mask.device}"
assert (
box_labels is not None and box_labels.device == device
), f"Expected box labels to be on device {device}, got {box_labels.device}"
assert (
point_embeddings is not None and point_embeddings.device == device
), f"Expected point embeddings to be on device {device}, got {point_embeddings.device}"
assert (
point_mask is not None and point_mask.device == device
), f"Expected point mask to be on device {device}, got {point_mask.device}"
assert (
point_labels is not None and point_labels.device == device
), f"Expected point labels to be on device {device}, got {point_labels.device}"
assert (
mask_embeddings is None or mask_embeddings.device == device
), f"Expected mask embeddings to be on device {device}, got {mask_embeddings.device}"
assert (
mask_mask is None or mask_mask.device == device
), f"Expected mask attn. mask to be on device {device}, got {mask_mask.device}"
assert box_embeddings is not None and box_embeddings.device == device, (
f"Expected box embeddings to be on device {device}, got {box_embeddings.device}"
)
assert box_mask is not None and box_mask.device == device, (
f"Expected box mask to be on device {device}, got {box_mask.device}"
)
assert box_labels is not None and box_labels.device == device, (
f"Expected box labels to be on device {device}, got {box_labels.device}"
)
assert point_embeddings is not None and point_embeddings.device == device, (
f"Expected point embeddings to be on device {device}, got {point_embeddings.device}"
)
assert point_mask is not None and point_mask.device == device, (
f"Expected point mask to be on device {device}, got {point_mask.device}"
)
assert point_labels is not None and point_labels.device == device, (
f"Expected point labels to be on device {device}, got {point_labels.device}"
)
assert mask_embeddings is None or mask_embeddings.device == device, (
f"Expected mask embeddings to be on device {device}, got {mask_embeddings.device}"
)
assert mask_mask is None or mask_mask.device == device, (
f"Expected mask attn. mask to be on device {device}, got {mask_mask.device}"
)
self.box_embeddings = box_embeddings
self.point_embeddings = point_embeddings
@@ -262,30 +251,30 @@ class Prompt:
if point_embeddings is not None:
point_seq_len = point_embeddings.shape[0]
if bs is not None:
assert (
bs == point_embeddings.shape[1]
), f"Batch size mismatch between box and point embeddings. Got {bs} and {point_embeddings.shape[1]}."
assert bs == point_embeddings.shape[1], (
f"Batch size mismatch between box and point embeddings. Got {bs} and {point_embeddings.shape[1]}."
)
else:
bs = point_embeddings.shape[1]
if device is not None:
assert (
device == point_embeddings.device
), "Device mismatch between box and point embeddings"
assert device == point_embeddings.device, (
"Device mismatch between box and point embeddings"
)
else:
device = point_embeddings.device
if mask_embeddings is not None:
mask_seq_len = mask_embeddings.shape[0]
if bs is not None:
assert (
bs == mask_embeddings.shape[1]
), f"Batch size mismatch between box/point and mask embedding. Got {bs} and {mask_embeddings.shape[1]}"
assert bs == mask_embeddings.shape[1], (
f"Batch size mismatch between box/point and mask embedding. Got {bs} and {mask_embeddings.shape[1]}"
)
else:
bs = mask_embeddings.shape[1]
if device is not None:
assert (
device == mask_embeddings.device
), "Device mismatch between box/point and mask embeddings."
assert device == mask_embeddings.device, (
"Device mismatch between box/point and mask embeddings."
)
else:
device = mask_embeddings.device
@@ -537,9 +526,9 @@ class SequenceGeometryEncoder(nn.Module):
if add_cls:
self.cls_embed = torch.nn.Embedding(1, self.d_model)
assert (
points_direct_project or points_pos_enc or points_pool
), "Error: need at least one way to encode points"
assert points_direct_project or points_pos_enc or points_pool, (
"Error: need at least one way to encode points"
)
assert (
encode_boxes_as_points
or boxes_direct_project
@@ -581,16 +570,16 @@ class SequenceGeometryEncoder(nn.Module):
self.encode = None
if num_layers > 0:
assert (
add_cls
), "It's currently highly recommended to add a CLS when using a transformer"
assert add_cls, (
"It's currently highly recommended to add a CLS when using a transformer"
)
self.encode = get_clones(layer, num_layers)
self.encode_norm = nn.LayerNorm(self.d_model)
if mask_encoder is not None:
assert isinstance(
mask_encoder, MaskEncoder
), f"Expected mask_encoder of type MaskEncoder. Got {type(mask_encoder)}."
assert isinstance(mask_encoder, MaskEncoder), (
f"Expected mask_encoder of type MaskEncoder. Got {type(mask_encoder)}."
)
if add_mask_label:
self.mask_label_embed = torch.nn.Embedding(2, self.d_model)
self.add_mask_label = add_mask_label
@@ -699,16 +688,15 @@ class SequenceGeometryEncoder(nn.Module):
img_feats: torch.Tensor = None,
):
n_masks, bs = masks.shape[:2]
assert (
n_masks == 1
), "We assume one mask per prompt for now. Code should still be functional if this assertion is removed."
assert (
list(attn_mask.shape)
== [
bs,
n_masks,
]
), f"Expected attn_mask to be of shape {bs}x{n_masks}. Got {list(attn_mask.shape)}."
assert n_masks == 1, (
"We assume one mask per prompt for now. Code should still be functional if this assertion is removed."
)
assert list(attn_mask.shape) == [
bs,
n_masks,
], (
f"Expected attn_mask to be of shape {bs}x{n_masks}. Got {list(attn_mask.shape)}."
)
masks, pos = self.mask_encoder(
masks=masks.flatten(0, 1).float(),
pix_feat=img_feats,

4
sam3/model/io_utils.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import contextlib
import os
import queue
@@ -11,9 +13,7 @@ import numpy as np
import torch
import torch.nn.functional as F
import torchvision.transforms.functional as TF
from PIL import Image
from sam3.logger import get_logger
from tqdm import tqdm

6
sam3/model/maskformer_segmentation.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import math
from typing import Dict, List, Optional
@@ -246,7 +248,9 @@ class UniversalSegmentationHead(SegmentationHead):
self.d_model = hidden_dim
if dot_product_scorer is not None:
assert presence_head, "Specifying a dot product scorer without a presence head is likely a mistake"
assert presence_head, (
"Specifying a dot product scorer without a presence head is likely a mistake"
)
self.presence_head = None
if presence_head:

8
sam3/model/memory.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import math
from typing import Tuple
@@ -60,9 +62,9 @@ class SimpleMaskDownSampler(nn.Module):
self.encoder.append(nn.Conv2d(mask_out_chans, embed_dim, kernel_size=1))
self.interpol_size = interpol_size
if self.interpol_size is not None:
assert isinstance(
self.interpol_size, (list, tuple)
), f"Unsupported type {type(self.interpol_size)}. Should be a list or tuple."
assert isinstance(self.interpol_size, (list, tuple)), (
f"Unsupported type {type(self.interpol_size)}. Should be a list or tuple."
)
self.interpol_size = list(interpol_size)
assert len(self.interpol_size) == 2

14
sam3/model/model_misc.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""Various utility models"""
import copy
@@ -328,9 +330,9 @@ class SAM3Output(list):
self.output = output
else:
self.output = []
assert isinstance(
iter_mode, SAM3Output.IterMode
), f"iter_mode shoulf be of enum type 'SAM3Output.IterMode'. Got {type(iter_mode)}"
assert isinstance(iter_mode, SAM3Output.IterMode), (
f"iter_mode shoulf be of enum type 'SAM3Output.IterMode'. Got {type(iter_mode)}"
)
self.iter_mode = iter_mode
# We create a weak reference to self to be used in the lambda functions.
@@ -409,9 +411,9 @@ class SAM3Output(list):
return SAM3Output._IterationMode(model_output=model_output, iter_mode=iter_mode)
def append(self, item: list):
assert isinstance(
item, list
), f"Only list items are supported. Got {type(item)}"
assert isinstance(item, list), (
f"Only list items are supported. Got {type(item)}"
)
self.output.append(item)
def __repr__(self):

3
sam3/model/necks.py
View File

@@ -1,12 +1,13 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""Necks are the interface between a vision backbone and the rest of the detection model"""
from copy import deepcopy
from typing import List, Optional, Tuple
import torch
import torch.nn as nn

2
sam3/model/position_encoding.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import math
from typing import Optional

35
sam3/model/sam1_task_predictor.py
View File

@@ -1,19 +1,18 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# All rights reserved.
# pyre-unsafe
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.
import logging
from typing import List, Optional, Tuple, Union
import numpy as np
import torch
import torch.nn as nn
from PIL.Image import Image
from sam3.model.sam3_tracker_base import Sam3TrackerBase
from sam3.model.utils.sam1_utils import SAM2Transforms
@@ -95,9 +94,9 @@ class SAM3InteractiveImagePredictor(nn.Module):
input_image = self._transforms(image)
input_image = input_image[None, ...].to(self.device)
assert (
len(input_image.shape) == 4 and input_image.shape[1] == 3
), f"input_image must be of size 1x3xHxW, got {input_image.shape}"
assert len(input_image.shape) == 4 and input_image.shape[1] == 3, (
f"input_image must be of size 1x3xHxW, got {input_image.shape}"
)
logging.info("Computing image embeddings for the provided image...")
backbone_out = self.model.forward_image(input_image)
(
@@ -134,17 +133,17 @@ class SAM3InteractiveImagePredictor(nn.Module):
assert isinstance(image_list, list)
self._orig_hw = []
for image in image_list:
assert isinstance(
image, np.ndarray
), "Images are expected to be an np.ndarray in RGB format, and of shape HWC"
assert isinstance(image, np.ndarray), (
"Images are expected to be an np.ndarray in RGB format, and of shape HWC"
)
self._orig_hw.append(image.shape[:2])
# Transform the image to the form expected by the model
img_batch = self._transforms.forward_batch(image_list)
img_batch = img_batch.to(self.device)
batch_size = img_batch.shape[0]
assert (
len(img_batch.shape) == 4 and img_batch.shape[1] == 3
), f"img_batch must be of size Bx3xHxW, got {img_batch.shape}"
assert len(img_batch.shape) == 4 and img_batch.shape[1] == 3, (
f"img_batch must be of size Bx3xHxW, got {img_batch.shape}"
)
logging.info("Computing image embeddings for the provided images...")
backbone_out = self.model.forward_image(img_batch)
(
@@ -300,9 +299,9 @@ class SAM3InteractiveImagePredictor(nn.Module):
):
unnorm_coords, labels, unnorm_box, mask_input = None, None, None, None
if point_coords is not None:
assert (
point_labels is not None
), "point_labels must be supplied if point_coords is supplied."
assert point_labels is not None, (
"point_labels must be supplied if point_coords is supplied."
)
point_coords = torch.as_tensor(
point_coords, dtype=torch.float, device=self.device
)
@@ -439,9 +438,9 @@ class SAM3InteractiveImagePredictor(nn.Module):
raise RuntimeError(
"An image must be set with .set_image(...) to generate an embedding."
)
assert (
self._features is not None
), "Features must exist if an image has been set."
assert self._features is not None, (
"Features must exist if an image has been set."
)
return self._features["image_embed"]
@property

13
sam3/model/sam3_image.py
View File

@@ -1,24 +1,21 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import os
from copy import deepcopy
from typing import Dict, List, Optional, Tuple
import numpy as np
import torch
from sam3.model.model_misc import SAM3Output
from sam3.model.sam1_task_predictor import SAM3InteractiveImagePredictor
from sam3.model.vl_combiner import SAM3VLBackbone
from sam3.perflib.nms import nms_masks
from sam3.train.data.collator import BatchedDatapoint
from .act_ckpt_utils import activation_ckpt_wrapper
from .box_ops import box_cxcywh_to_xyxy
from .geometry_encoders import Prompt
from .model_misc import inverse_sigmoid
@@ -659,9 +656,9 @@ class Sam3Image(torch.nn.Module):
inference_state["original_heights"],
inference_state["original_widths"],
)
assert (
batch_size == len(orig_heights) == len(orig_widths)
), f"Batch size mismatch in predict_inst_batch. Got {batch_size}, {len(orig_heights)}, {len(orig_widths)}"
assert batch_size == len(orig_heights) == len(orig_widths), (
f"Batch size mismatch in predict_inst_batch. Got {batch_size}, {len(orig_heights)}, {len(orig_widths)}"
)
feats = [
feat.permute(1, 2, 0).view(batch_size, -1, *feat_size)
for feat, feat_size in zip(

10
sam3/model/sam3_image_processor.py
View File

@@ -1,12 +1,12 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
from typing import Dict, List
import numpy as np
import PIL
import torch
from sam3.model import box_ops
from sam3.model.data_misc import FindStage, interpolate
from torchvision.transforms import v2
@@ -81,9 +81,9 @@ class Sam3Processor:
if not isinstance(images, list):
raise ValueError("Images must be a list of PIL images or tensors")
assert len(images) > 0, "Images list must not be empty"
assert isinstance(
images[0], PIL.Image.Image
), "Images must be a list of PIL images"
assert isinstance(images[0], PIL.Image.Image), (
"Images must be a list of PIL images"
)
state["original_heights"] = [image.height for image in images]
state["original_widths"] = [image.width for image in images]

7
sam3/model/sam3_tracker_base.py
View File

@@ -1,14 +1,13 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import logging
import torch
import torch.nn.functional as F
from sam3.model.memory import SimpleMaskEncoder
from sam3.model.sam3_tracker_utils import get_1d_sine_pe, select_closest_cond_frames
from sam3.sam.mask_decoder import MaskDecoder, MLP
from sam3.sam.prompt_encoder import PromptEncoder
from sam3.sam.transformer import TwoWayTransformer
@@ -900,8 +899,6 @@ class Sam3TrackerBase(torch.nn.Module):
image=current_image,
point_inputs=backbone_out["point_inputs_per_frame"].get(stage_id, None),
mask_inputs=backbone_out["mask_inputs_per_frame"].get(stage_id, None),
gt_masks=backbone_out["gt_masks_per_frame"].get(stage_id, None),
frames_to_add_correction_pt=frames_to_add_correction_pt,
output_dict=output_dict,
num_frames=num_frames,
)

3
sam3/model/sam3_tracker_utils.py
View File

@@ -1,10 +1,11 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import numpy as np
import torch
import torch.nn.functional as F
from numpy.typing import NDArray
from sam3.model.edt import edt_triton

5
sam3/model/sam3_tracking_predictor.py
View File

@@ -1,10 +1,11 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import logging
from collections import OrderedDict
import torch
from sam3.model.sam3_tracker_base import concat_points, NO_OBJ_SCORE, Sam3TrackerBase
from sam3.model.sam3_tracker_utils import fill_holes_in_mask_scores
from sam3.model.utils.sam2_utils import load_video_frames
@@ -657,8 +658,6 @@ class Sam3TrackerPredictor(Sam3TrackerBase):
image=image,
point_inputs=None,
mask_inputs=mask_inputs,
gt_masks=None,
frames_to_add_correction_pt=[],
output_dict={
"cond_frame_outputs": {},
"non_cond_frame_outputs": {},

39
sam3/model/sam3_video_base.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import datetime
import logging
import math
@@ -14,7 +16,6 @@ import numpy.typing as npt
import torch
import torch.distributed as dist
import torch.nn.functional as F
from sam3 import perflib
from sam3.logger import get_logger
from sam3.model.box_ops import fast_diag_box_iou
@@ -618,9 +619,9 @@ class Sam3VideoBase(nn.Module):
num_obj_dropped_due_to_limit,
trk_id_to_max_iou_high_conf_det,
]
assert (
len(update_plan) == NUM_BROADCAST_ITEMS
), f"Manually update NUM_BROADCAST_ITEMS to be: {len(update_plan)}"
assert len(update_plan) == NUM_BROADCAST_ITEMS, (
f"Manually update NUM_BROADCAST_ITEMS to be: {len(update_plan)}"
)
self.broadcast_python_obj_cpu(update_plan, src=0)
elif self.rank > 0 and self.world_size > 1:
update_plan = [
@@ -840,9 +841,9 @@ class Sam3VideoBase(nn.Module):
binary_tracker_low_res_masks_global = tracker_low_res_masks_global > 0
batch_size = tracker_low_res_masks_global.size(0)
if batch_size > 0:
assert (
len(obj_ids_global) == batch_size
), f"Mismatch in number of objects: {len(obj_ids_global)} vs {batch_size}"
assert len(obj_ids_global) == batch_size, (
f"Mismatch in number of objects: {len(obj_ids_global)} vs {batch_size}"
)
NEVER_OCCLUDED = -1
ALWAYS_OCCLUDED = 100000 # This value should be larger than any possible frame index, indicates that the object was removed by hotstart logic
last_occluded_prev = torch.cat(
@@ -1021,9 +1022,9 @@ class Sam3VideoBase(nn.Module):
reverse: bool = False,
):
# Suppress overlapping masks for objects that were most recently occluded
assert (
binary_low_res_masks.dtype == torch.bool
), f"Expected boolean tensor, got {binary_low_res_masks.dtype}"
assert binary_low_res_masks.dtype == torch.bool, (
f"Expected boolean tensor, got {binary_low_res_masks.dtype}"
)
to_suppress = torch.zeros(
binary_low_res_masks.size(0),
device=binary_low_res_masks.device,
@@ -1128,9 +1129,9 @@ class Sam3VideoBase(nn.Module):
num_frames_propagated += 1
# only 1 frames should be propagated
assert (
num_frames_propagated == 1 and out_frame_idx == frame_idx
), f"num_frames_propagated: {num_frames_propagated}, out_frame_idx: {out_frame_idx}, frame_idx: {frame_idx}"
assert num_frames_propagated == 1 and out_frame_idx == frame_idx, (
f"num_frames_propagated: {num_frames_propagated}, out_frame_idx: {out_frame_idx}, frame_idx: {frame_idx}"
)
assert isinstance(out_obj_ids, list)
obj_ids_local.extend(out_obj_ids)
low_res_masks_list.append(out_low_res_masks.squeeze(1))
@@ -1187,9 +1188,9 @@ class Sam3VideoBase(nn.Module):
assert det_masks.is_floating_point(), "float tensor expected (do not binarize)"
assert trk_masks.is_floating_point(), "float tensor expected (do not binarize)"
assert (
trk_masks.size(0) == len(trk_obj_ids)
), f"trk_masks and trk_obj_ids should have the same length, {trk_masks.size(0)} vs {len(trk_obj_ids)}"
assert trk_masks.size(0) == len(trk_obj_ids), (
f"trk_masks and trk_obj_ids should have the same length, {trk_masks.size(0)} vs {len(trk_obj_ids)}"
)
if trk_masks.size(0) == 0:
# all detections are new
new_det_fa_inds = np.arange(det_masks.size(0))
@@ -1653,9 +1654,9 @@ class Sam3VideoBase(nn.Module):
# a) first, expand "confirmation_data" to include new masklets added in this frame
status_prev = confirmation_data["status"]
consecutive_det_num_prev = confirmation_data["consecutive_det_num"]
assert (
status_prev.shape == obj_ids_all_gpu_prev.shape
), f"Got {status_prev.shape} vs {obj_ids_all_gpu_prev.shape}"
assert status_prev.shape == obj_ids_all_gpu_prev.shape, (
f"Got {status_prev.shape} vs {obj_ids_all_gpu_prev.shape}"
)
obj_id_to_updated_idx = {
obj_id: idx for idx, obj_id in enumerate(obj_ids_all_gpu_updated)

55
sam3/model/sam3_video_inference.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import logging
from collections import defaultdict
@@ -7,7 +9,6 @@ import numpy as np
import torch
import torch.distributed as dist
import torch.nn.functional as F
from sam3 import perflib
from sam3.logger import get_logger
from sam3.model.act_ckpt_utils import clone_output_wrapper
@@ -553,7 +554,9 @@ class Sam3VideoInference(Sam3VideoBase):
assert (
"cached_frame_outputs" in inference_state
and frame_idx in inference_state["cached_frame_outputs"]
), "No cached outputs found. Ensure normal propagation has run first to populate the cache."
), (
"No cached outputs found. Ensure normal propagation has run first to populate the cache."
)
cached_outputs = inference_state["cached_frame_outputs"][frame_idx]
obj_id_to_mask = cached_outputs.copy()
@@ -561,9 +564,9 @@ class Sam3VideoInference(Sam3VideoBase):
# Update with refined masks if provided
if refined_obj_id_to_mask is not None:
for obj_id, refined_mask in refined_obj_id_to_mask.items():
assert (
refined_mask is not None
), f"Refined mask data must be provided for obj_id {obj_id}"
assert refined_mask is not None, (
f"Refined mask data must be provided for obj_id {obj_id}"
)
obj_id_to_mask[obj_id] = refined_mask
return obj_id_to_mask
@@ -658,12 +661,12 @@ class Sam3VideoInference(Sam3VideoBase):
for i, thresh in enumerate(new_det_score_thresh_list):
self.new_det_thresh = thresh
for num_objects in num_objects_list:
logger.info(f"{i+1}/{num_rounds} warming up model compilation")
logger.info(f"{i + 1}/{num_rounds} warming up model compilation")
self.add_prompt(
inference_state, frame_idx=start_frame_idx, text_str="cat"
)
logger.info(
f"{i+1}/{num_rounds} warming up model compilation -- simulating {num_objects}/{self.num_obj_for_compile} objects"
f"{i + 1}/{num_rounds} warming up model compilation -- simulating {num_objects}/{self.num_obj_for_compile} objects"
)
inference_state = self.add_fake_objects_to_inference_state(
inference_state, num_objects, frame_idx=start_frame_idx
@@ -688,7 +691,7 @@ class Sam3VideoInference(Sam3VideoBase):
pass
self.reset_state(inference_state)
logger.info(
f"{i+1}/{num_rounds} warming up model compilation -- completed round {i+1} out of {num_rounds}"
f"{i + 1}/{num_rounds} warming up model compilation -- completed round {i + 1} out of {num_rounds}"
)
# Warm up Tracker memory encoder with varying input shapes
@@ -852,12 +855,12 @@ class Sam3VideoInference(Sam3VideoBase):
logger.debug("Running add_prompt on frame %d", frame_idx)
num_frames = inference_state["num_frames"]
assert (
text_str is not None or boxes_xywh is not None
), "at least one type of prompt (text, boxes) must be provided"
assert (
0 <= frame_idx < num_frames
), f"{frame_idx=} is out of range for a total of {num_frames} frames"
assert text_str is not None or boxes_xywh is not None, (
"at least one type of prompt (text, boxes) must be provided"
)
assert 0 <= frame_idx < num_frames, (
f"{frame_idx=} is out of range for a total of {num_frames} frames"
)
# since it's a semantic prompt, we start over
self.reset_state(inference_state)
@@ -1198,9 +1201,9 @@ class Sam3VideoInferenceWithInstanceInteractivity(Sam3VideoInference):
"propagation_partial",
"propagation_fetch",
]
assert (
action_type in instance_actions + propagation_actions
), f"Invalid action type: {action_type}, must be one of {instance_actions + propagation_actions}"
assert action_type in instance_actions + propagation_actions, (
f"Invalid action type: {action_type}, must be one of {instance_actions + propagation_actions}"
)
action = {
"type": action_type,
"frame_idx": frame_idx,
@@ -1368,12 +1371,12 @@ class Sam3VideoInferenceWithInstanceInteractivity(Sam3VideoInference):
):
if points is not None:
# Tracker instance prompts
assert (
text_str is None and boxes_xywh is None
), "When points are provided, text_str and boxes_xywh must be None."
assert (
obj_id is not None
), "When points are provided, obj_id must be provided."
assert text_str is None and boxes_xywh is None, (
"When points are provided, text_str and boxes_xywh must be None."
)
assert obj_id is not None, (
"When points are provided, obj_id must be provided."
)
return self.add_tracker_new_points(
inference_state,
frame_idx,
@@ -1489,9 +1492,9 @@ class Sam3VideoInferenceWithInstanceInteractivity(Sam3VideoInference):
tracker_states = self._get_tracker_inference_states_by_obj_ids(
inference_state, [obj_id]
)
assert (
len(tracker_states) == 1
), f"[rank={self.rank}] Multiple Tracker inference states found for the same object id."
assert len(tracker_states) == 1, (
f"[rank={self.rank}] Multiple Tracker inference states found for the same object id."
)
tracker_state = tracker_states[0]
# log

5
sam3/model/sam3_video_predictor.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import datetime
import gc
import multiprocessing as mp
@@ -14,7 +16,6 @@ from typing import List, Optional
import psutil
import torch
from sam3.logger import get_logger
logger = get_logger(__name__)
@@ -168,7 +169,7 @@ class Sam3VideoPredictor:
):
"""Remove an object from tracking."""
logger.debug(
f"remove object {obj_id} in session {session_id}: " f"{is_user_action=}"
f"remove object {obj_id} in session {session_id}: {is_user_action=}"
)
session = self._get_session(session_id)
inference_state = session["state"]

8
sam3/model/text_encoder_ve.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
from collections import OrderedDict
from typing import Callable, List, Optional, Tuple, Union
@@ -316,9 +318,9 @@ class VETextEncoder(nn.Module):
# The text is already encoded, use as is.
text_attention_mask, text_memory_resized, tokenized = text
inputs_embeds = tokenized["inputs_embeds"]
assert (
input_boxes is None or len(input_boxes) == 0
), "Can't replace boxes in text if it's already encoded"
assert input_boxes is None or len(input_boxes) == 0, (
"Can't replace boxes in text if it's already encoded"
)
# Note that the input_embeds are returned in pytorch's convention (sequence first)
return (

2
sam3/model/tokenizer_ve.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""
Text Tokenizer.

2
sam3/model/utils/__init__.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# All rights reserved.
# pyre-unsafe
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.

2
sam3/model/utils/misc.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
from collections import defaultdict
from dataclasses import fields, is_dataclass
from typing import Any, Mapping, Protocol, runtime_checkable

2
sam3/model/utils/sam1_utils.py
View File

@@ -1,6 +1,8 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# All rights reserved.
# pyre-unsafe
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.

2
sam3/model/utils/sam2_utils.py
View File

@@ -1,6 +1,8 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# All rights reserved.
# pyre-unsafe
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.

8
sam3/model/vitdet.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""
ViTDet backbone adapted from Detectron2.
This module implements Vision Transformer (ViT) backbone for object detection.
@@ -706,9 +708,9 @@ class ViT(nn.Module):
self.retain_cls_token = retain_cls_token
if self.retain_cls_token:
assert pretrain_use_cls_token
assert (
len(window_block_indexes) == 0
), "windowing not supported with cls token"
assert len(window_block_indexes) == 0, (
"windowing not supported with cls token"
)
assert sum(self.rel_pos_blocks) == 0, "rel pos not supported with cls token"

3
sam3/model/vl_combiner.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
"""Provides utility to combine a vision backbone with a language backbone."""
from copy import copy
@@ -7,7 +9,6 @@ from typing import List, Optional
import torch
import torch.nn as nn
from torch.nn.attention import sdpa_kernel, SDPBackend
from .act_ckpt_utils import activation_ckpt_wrapper

20
sam3/model_builder.py
View File

@@ -1,8 +1,11 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import os
from typing import Optional
import pkg_resources
import torch
import torch.nn as nn
from huggingface_hub import hf_hub_download
@@ -558,8 +561,8 @@ def build_sam3_image_model(
bpe_path=None,
device="cuda" if torch.cuda.is_available() else "cpu",
eval_mode=True,
checkpoint_path=None,
load_from_HF=True,
checkpoint_path="/home/quant/data/dev/sam3/sam3.pt",
load_from_HF=False,
enable_segmentation=True,
enable_inst_interactivity=False,
compile=False,
@@ -580,9 +583,10 @@ def build_sam3_image_model(
A SAM3 image model
"""
if bpe_path is None:
bpe_path = os.path.join(
os.path.dirname(__file__), "..", "assets", "bpe_simple_vocab_16e6.txt.gz"
bpe_path = pkg_resources.resource_filename(
"sam3", "assets/bpe_simple_vocab_16e6.txt.gz"
)
# Create visual components
compile_mode = "default" if compile else None
vision_encoder = _create_vision_backbone(
@@ -647,8 +651,8 @@ def download_ckpt_from_hf():
def build_sam3_video_model(
checkpoint_path: Optional[str] = None,
load_from_HF=True,
checkpoint_path: Optional[str] = "/home/quant/data/dev/sam3/sam3.pt",
load_from_HF=False,
bpe_path: Optional[str] = None,
has_presence_token: bool = True,
geo_encoder_use_img_cross_attn: bool = True,
@@ -668,8 +672,8 @@ def build_sam3_video_model(
Sam3VideoInferenceWithInstanceInteractivity: The instantiated dense tracking model
"""
if bpe_path is None:
bpe_path = os.path.join(
os.path.dirname(__file__), "..", "assets", "bpe_simple_vocab_16e6.txt.gz"
bpe_path = pkg_resources.resource_filename(
"sam3", "assets/bpe_simple_vocab_16e6.txt.gz"
)
# Build Tracker module

2
sam3/perflib/__init__.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import os
is_enabled = False

2
sam3/perflib/associate_det_trk.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
from collections import defaultdict
import torch

2
sam3/perflib/compile.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import torch

14
sam3/perflib/connected_components.py
View File

@@ -1,4 +1,6 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import logging
import torch
@@ -34,9 +36,9 @@ def connected_components_cpu(input_tensor: torch.Tensor):
if input_tensor.dim() == 4 and input_tensor.shape[1] == 1:
input_tensor = input_tensor.squeeze(1)
else:
assert (
input_tensor.dim() == 3
), "Input tensor must be (B, H, W) or (B, 1, H, W)."
assert input_tensor.dim() == 3, (
"Input tensor must be (B, H, W) or (B, 1, H, W)."
)
batch_size = input_tensor.shape[0]
labels_list = []
@@ -65,9 +67,9 @@ def connected_components(input_tensor: torch.Tensor):
if input_tensor.dim() == 3:
input_tensor = input_tensor.unsqueeze(1)
assert (
input_tensor.dim() == 4 and input_tensor.shape[1] == 1
), "Input tensor must be (B, H, W) or (B, 1, H, W)."
assert input_tensor.dim() == 4 and input_tensor.shape[1] == 1, (
"Input tensor must be (B, H, W) or (B, 1, H, W)."
)
if input_tensor.is_cuda:
if HAS_CC_TORCH:

2
sam3/perflib/fa3.py
View File

@@ -1,5 +1,7 @@
# Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved
# pyre-unsafe
import torch

Some files were not shown because too many files have changed in this diff Show More