{"record":{"id":"152ee279bbf899d7","repo":"Comfy-Org/ComfyUI","slug":"sam3-non-multiplex-requires-initial-mask-for-vid","errorCode":null,"errorMessage":"SAM3 (non-multiplex) requires initial_mask for video tracking","messagePattern":"SAM3 \\(non-multiplex\\) requires initial_mask for video tracking","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"comfy/ldm/sam3/detector.py","lineNumber":597,"sourceCode":"            resizer = self.detector.backbone[\"language_backbone\"][\"resizer\"]\n            resized = [(resizer(emb), m.bool() if m is not None else None) for emb, m in text_prompts]\n            def detect_fn(trunk_out):\n                all_scores, all_masks = [], []\n                for emb, mask in resized:\n                    det = self.detector.forward_from_trunk(trunk_out, emb, mask)\n                    all_scores.append(det[\"scores\"])\n                    all_masks.append(det[\"masks\"])\n                return {\"scores\": torch.cat(all_scores, dim=1), \"masks\": torch.cat(all_masks, dim=1)}\n\n        if hasattr(self.tracker, 'track_video_with_detection'):\n            return self.tracker.track_video_with_detection(\n                backbone_fn, images, initial_masks, detect_fn,\n                new_det_thresh=new_det_thresh, max_objects=max_objects,\n                detect_interval=detect_interval, backbone_obj=bb, pbar=pbar,\n                target_device=target_device, target_dtype=target_dtype)\n        # SAM3 (non-multiplex) — no detection support, requires initial masks\n        if initial_masks is None:\n            raise ValueError(\"SAM3 (non-multiplex) requires initial_mask for video tracking\")\n        return self.tracker.track_video(backbone_fn, images, initial_masks, pbar=pbar, backbone_obj=bb,\n                                         target_device=target_device, target_dtype=target_dtype)\n","sourceCodeStart":579,"sourceCodeEnd":600,"githubUrl":"https://github.com/Comfy-Org/ComfyUI/blob/1c6d8d45b3693bfbb32385b410d813a7fd6be216/comfy/ldm/sam3/detector.py#L579-L600","documentation":"For video tracking, the SAM3 detector first tries multiplex-style trackers that can re-detect objects (track_video_with_detection); if the tracker backend lacks that capability (non-multiplex SAM3), it falls back to plain track_video, which needs initial masks for the first frame. The ValueError fires when initial_masks is None on that fallback path — you asked a tracker with no detection ability to discover objects by itself.","triggerScenarios":"Calling the SAM3 detector's video tracking API with initial_masks=None on a non-multiplex tracker build; loading a SAM3 checkpoint variant whose tracker exposes no track_video_with_detection; running 'track from frame 0' without first running single-image detection on frame 0.","commonSituations":"User runs video segmentation without clicking/generating a starting mask; a checkpoint variant mismatch where the multiplex tracker class was expected; workflow that assumes auto-detection during tracking, which only multiplex SAM3 supports.","solutions":["Run SAM3 detection on the first frame (or supply a manually drawn mask) and pass those masks as initial_masks.","If you want tracking with periodic re-detection, use the multiplex-capable SAM3 tracker/checkpoint variant.","Check hasattr(tracker, 'track_video_with_detection') in your own code to decide which API to call before passing None.","Confirm you loaded the full SAM3 video model, not an image-only detector build."],"exampleFix":"# before\nresult = sam3.track_video(images)\n# after\ndet0 = sam3.detect(images[0], prompt=\"the cat\")\nresult = sam3.track_video(images, initial_masks=det0[\"masks\"])","handlingStrategy":"type-guard","validationCode":"def track_with_fallback(detector, images, prompt=None, initial_masks=None):\n    can_redetect = hasattr(detector.tracker, 'track_video_with_detection')\n    if not can_redetect and initial_masks is None:\n        det = detector.detect(images[0], prompt=prompt)\n        initial_masks = det[\"masks\"]\n    return detector.tracker.track_video(\n        (lambda i, it: detector.image_encoder(it)), images, initial_masks\n    )","typeGuard":"def tracker_supports_detection(detector) -> bool:\n    return hasattr(detector.tracker, 'track_video_with_detection')","tryCatchPattern":"try:\n    out = detector.track_video(images, initial_masks=masks)\nexcept ValueError as e:\n    if 'initial_mask' in str(e):\n        masks = detector.detect(images[0], prompt=prompt)[\"masks\"]\n        out = detector.track_video(images, initial_masks=masks)\n    else:\n        raise","preventionTips":["Always run first-frame detection before video tracking on non-multiplex SAM3.","Check hasattr(tracker, 'track_video_with_detection') to pick the tracking API.","Keep a saved first-frame mask alongside video datasets for reproducible tracking."],"tags":["sam3","video-tracking","segmentation","validation"],"backgroundTag":null,"analyzedSha":"1c6d8d45b3693bfbb32385b410d813a7fd6be216","analyzedAt":"2026-08-14T19:37:18.893Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}