{"record":{"id":"d200bb8899e5bdd8","repo":"sgl-project/sglang","slug":"cosmos3tokenizationstage-requires-a-tokenizer-exp","errorCode":null,"errorMessage":"Cosmos3TokenizationStage requires a tokenizer; expected the Qwen2 tokenizer loaded from the checkpoint's text_tokenizer/ subfolder.","messagePattern":"Cosmos3TokenizationStage requires a tokenizer; expected the Qwen2 tokenizer loaded from the checkpoint's text_tokenizer/ subfolder\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/cosmos3.py","lineNumber":275,"sourceCode":"        cond_indexes = getattr(batch.sampling_params, \"condition_frame_indexes\", None)\n        if not cond_indexes:\n            return [0, 1]\n        return sorted(set(int(i) for i in cond_indexes))\n\n\nclass Cosmos3TokenizationStage(PipelineStage):\n    \"\"\"Tokenization stage for Cosmos3.\n\n    Applies the Qwen2 chat template, appends a duration suffix, and writes\n    ``text_ids`` / ``text_mask`` into ``batch.extra`` for the denoising stage.\n    \"\"\"\n\n    parallelism_type = StageParallelismType.REPLICATED\n\n    def __init__(self, tokenizer):\n        super().__init__()\n        if tokenizer is None:\n            raise ValueError(\n                \"Cosmos3TokenizationStage requires a tokenizer; expected the \"\n                \"Qwen2 tokenizer loaded from the checkpoint's text_tokenizer/ \"\n                \"subfolder.\"\n            )\n        self.tokenizer = tokenizer\n\n    def verify_input(self, batch: Req, server_args: ServerArgs) -> VerificationResult:\n        result = VerificationResult()\n        result.add_check(\"prompt\", batch.prompt, V.string_or_list_strings)\n        return result\n\n    def _tokenize_prompt(\n        self,\n        text: str | list[str],\n        max_sequence_length: int,\n        device: torch.device,\n        use_system_prompt: bool = False,\n        system_prompt: str | None = None,","sourceCodeStart":257,"sourceCodeEnd":293,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/cosmos3.py#L257-L293","documentation":"Cosmos3TokenizationStage needs the Qwen2 text tokenizer (loaded from the checkpoint's text_tokenizer/ subfolder) to build input ids with the chat template. Constructing the stage with tokenizer=None means the checkpoint layout is wrong or the loader skipped the text tokenizer, so __init__ fails fast.","triggerScenarios":"Instantiating Cosmos3TokenizationStage(tokenizer=None) — typically because the checkpoint directory lacks text_tokenizer/, files failed to download, or the loader returned None for the text tokenizer.","commonSituations":"Incomplete checkpoint download/transfer missing the text_tokenizer subfolder; using a Cosmos3 checkpoint converted without the Qwen2 text encoder files; loader version change that renamed or stopped auto-loading the subfolder.","solutions":["Verify the checkpoint contains text_tokenizer/ with Qwen2 tokenizer files (tokenizer.json/tokenizer_config.json) and re-download/copy it if missing","Fix the loader call so it actually loads the text tokenizer from that subfolder and pass the result to the stage","If converting your own checkpoint, include the Qwen2 tokenizer files under text_tokenizer/"],"exampleFix":"# before\nstage = Cosmos3TokenizationStage(tokenizer=load_model(path).text_tokenizer_or_none)\n# after\ntok = AutoTokenizer.from_pretrained(os.path.join(ckpt, \"text_tokenizer\"))\nassert tok is not None and tok.pad_token_id is not None\nstage = Cosmos3TokenizationStage(tokenizer=tok)","handlingStrategy":"type-guard","validationCode":"import os\nfrom transformers import AutoTokenizer\nsub = os.path.join(ckpt_dir, \"text_tokenizer\")\nassert os.path.isdir(sub) and os.listdir(sub), f\"missing {sub}\"","typeGuard":"def usable_tokenizer(tok) -> bool:\n    return tok is not None and getattr(tok, \"pad_token_id\", None) is not None","tryCatchPattern":null,"preventionTips":["Validate checkpoint layout (text_tokenizer/ present) before constructing stages"],"tags":["cosmos3","tokenizer","qwen2","checkpoint","init-validation"],"backgroundTag":"missing-tokenizer-files","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}