Comfy-Org/ComfyUI · error · ValueError

SAM3 (non-multiplex) requires initial_mask for video trackin

Error message

SAM3 (non-multiplex) requires initial_mask for video tracking

What it means

For video tracking, the SAM3 detector first tries multiplex-style trackers that can re-detect objects (track_video_with_detection); if the tracker backend lacks that capability (non-multiplex SAM3), it falls back to plain track_video, which needs initial masks for the first frame. The ValueError fires when initial_masks is None on that fallback path — you asked a tracker with no detection ability to discover objects by itself.

Source

Thrown at comfy/ldm/sam3/detector.py:597

            resizer = self.detector.backbone["language_backbone"]["resizer"]
            resized = [(resizer(emb), m.bool() if m is not None else None) for emb, m in text_prompts]
            def detect_fn(trunk_out):
                all_scores, all_masks = [], []
                for emb, mask in resized:
                    det = self.detector.forward_from_trunk(trunk_out, emb, mask)
                    all_scores.append(det["scores"])
                    all_masks.append(det["masks"])
                return {"scores": torch.cat(all_scores, dim=1), "masks": torch.cat(all_masks, dim=1)}

        if hasattr(self.tracker, 'track_video_with_detection'):
            return self.tracker.track_video_with_detection(
                backbone_fn, images, initial_masks, detect_fn,
                new_det_thresh=new_det_thresh, max_objects=max_objects,
                detect_interval=detect_interval, backbone_obj=bb, pbar=pbar,
                target_device=target_device, target_dtype=target_dtype)
        # SAM3 (non-multiplex) — no detection support, requires initial masks
        if initial_masks is None:
            raise ValueError("SAM3 (non-multiplex) requires initial_mask for video tracking")
        return self.tracker.track_video(backbone_fn, images, initial_masks, pbar=pbar, backbone_obj=bb,
                                         target_device=target_device, target_dtype=target_dtype)

View on GitHub (pinned to 1c6d8d45b3)

Solutions

Run SAM3 detection on the first frame (or supply a manually drawn mask) and pass those masks as initial_masks.
If you want tracking with periodic re-detection, use the multiplex-capable SAM3 tracker/checkpoint variant.
Check hasattr(tracker, 'track_video_with_detection') in your own code to decide which API to call before passing None.
Confirm you loaded the full SAM3 video model, not an image-only detector build.

Example fix

# before
result = sam3.track_video(images)
# after
det0 = sam3.detect(images[0], prompt="the cat")
result = sam3.track_video(images, initial_masks=det0["masks"])

Defensive patterns

Strategy: type-guard

Validate before calling

def track_with_fallback(detector, images, prompt=None, initial_masks=None):
    can_redetect = hasattr(detector.tracker, 'track_video_with_detection')
    if not can_redetect and initial_masks is None:
        det = detector.detect(images[0], prompt=prompt)
        initial_masks = det["masks"]
    return detector.tracker.track_video(
        (lambda i, it: detector.image_encoder(it)), images, initial_masks
    )

Type guard

def tracker_supports_detection(detector) -> bool:
    return hasattr(detector.tracker, 'track_video_with_detection')

Try / catch

try:
    out = detector.track_video(images, initial_masks=masks)
except ValueError as e:
    if 'initial_mask' in str(e):
        masks = detector.detect(images[0], prompt=prompt)["masks"]
        out = detector.track_video(images, initial_masks=masks)
    else:
        raise

Prevention

Always run first-frame detection before video tracking on non-multiplex SAM3.
Check hasattr(tracker, 'track_video_with_detection') to pick the tracking API.
Keep a saved first-frame mask alongside video datasets for reproducible tracking.

When it happens

Trigger: Calling the SAM3 detector's video tracking API with initial_masks=None on a non-multiplex tracker build; loading a SAM3 checkpoint variant whose tracker exposes no track_video_with_detection; running 'track from frame 0' without first running single-image detection on frame 0.

Common situations: User runs video segmentation without clicking/generating a starting mask; a checkpoint variant mismatch where the multiplex tracker class was expected; workflow that assumes auto-detection during tracking, which only multiplex SAM3 supports.

Related errors

AI-assisted analysis of Comfy-Org/ComfyUI@1c6d8d45b3 (2026-08-14). Data as JSON: /api/errors/152ee279bbf899d7. Report an issue: GitHub.