{"record":{"id":"8bf6c8eed728892d","repo":"invoke-ai/InvokeAI","slug":"context-len-meta-parameter-s-remain-on-the","errorCode":null,"errorMessage":"{context}: {len(meta)} parameter(s) remain on the meta device after loading (missing or mismatched weights): {meta[:10]}","messagePattern":"(.+?): (.+?) parameter\\(s\\) remain on the meta device after loading \\(missing or mismatched weights\\): (.+?)","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"invokeai/backend/model_manager/load/model_loaders/ideogram4.py","lineNumber":67,"sourceCode":"\ndef _verify_encoder_fully_materialized(model: torch.nn.Module, *, context: str) -> None:\n    \"\"\"Fail if any parameter is still on the meta device after loading the text encoder.\n\n    The encoder is built under ``accelerate.init_empty_weights()`` (every param starts on the meta\n    device) and then filled from the checkpoint. Missing keys are only acceptable for tied weights, which\n    ``transformers`` materializes via ``tie_weights()``; any other missing key leaves a meta tensor that\n    would pass loading but fail later during device movement or encoding. Re-tie, then hard-fail if any\n    meta tensor remains so a bad/mismatched encoder is rejected at load time instead.\n    \"\"\"\n    if hasattr(model, \"tie_weights\"):\n        model.tie_weights()\n    meta = [\n        name\n        for name, tensor in itertools.chain(model.named_parameters(), model.named_buffers())\n        if getattr(tensor, \"is_meta\", False)\n    ]\n    if meta:\n        raise RuntimeError(\n            f\"{context}: {len(meta)} parameter(s) remain on the meta device after loading \"\n            f\"(missing or mismatched weights): {meta[:10]}\"\n        )\n\n\n@ModelLoaderRegistry.register(base=BaseModelType.Ideogram4, type=ModelType.Main, format=ModelFormat.Diffusers)\nclass Ideogram4DiffusersModel(ModelLoader):\n    \"\"\"Loads Ideogram 4 main models (nf4 / fp8) bundled in diffusers layout.\"\"\"\n\n    def _load_model(\n        self,\n        config: AnyModelConfig,\n        submodel_type: Optional[SubModelType] = None,\n    ) -> AnyModel:\n        if not isinstance(config, Main_Diffusers_Ideogram4_Config):\n            raise ValueError(f\"Expected Main_Diffusers_Ideogram4_Config, got {type(config).__name__}.\")\n        if submodel_type is None:\n            raise Exception(\"A submodel type must be provided when loading Ideogram 4 main pipelines.\")","sourceCodeStart":49,"sourceCodeEnd":85,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/model_manager/load/model_loaders/ideogram4.py#L49-L85","documentation":"_verify_encoder_fully_materialized scans a freshly loaded model's parameters and buffers for tensors still on the 'meta' device, which means accelerate's low_cpu_mem_usage loading never assigned real weights to them. Leftover meta tensors indicate missing or mismatched checkpoint weights (e.g. keys not covered, or tied weights not yet resolved). The loader raises RuntimeError listing up to 10 offending names so weight problems fail loudly instead of producing garbage output.","triggerScenarios":"_load_text_encoder loads an Ideogram 4 text encoder with load_state_dict(strict=False, assign=True); the checkpoint is missing keys (or tied weights were not resolved by tie_weights) so some parameters/buffers remain on the meta device, detected by _verify_encoder_fully_materialized.","commonSituations":"Loading a truncated or partially converted checkpoint; a text-encoder checkpoint that omits weights tied to embeddings (lm_head/embed_tokens); mismatched architecture version where checkpoint keys don't match the model class.","solutions":["Inspect the listed parameter names and confirm the checkpoint actually contains those weights; re-download/re-export the checkpoint if truncated.","Ensure the loader calls model.tie_weights() (or equivalent) after load_state_dict when keys are only missing due to weight tying.","Verify the checkpoint key layout matches the expected model architecture; re-map keys if the source used a different naming scheme.","Load without assign=True / with a matching accelerate config if the checkpoint format isn't compatible with meta-device loading."],"exampleFix":"// before\nmodel.load_state_dict(sd, strict=False, assign=True)\n_verify_encoder_fully_materialized(model, context=...)\n// after: resolve tied weights before verification\nmodel.load_state_dict(sd, strict=False, assign=True)\nmodel.tie_weights()\n_verify_encoder_fully_materialized(model, context=...)","handlingStrategy":"try-catch","validationCode":"import itertools\nmeta = [n for n, t in itertools.chain(model.named_parameters(), model.named_buffers()) if getattr(t, \"is_meta\", False)]\nif meta:\n    raise RuntimeError(f\"missing weights before use: {meta[:10]}\")","typeGuard":"def fully_materialized(model) -> bool:\n    import itertools\n    return not any(getattr(t, \"is_meta\", False)\n                   for _, t in itertools.chain(model.named_parameters(), model.named_buffers()))","tryCatchPattern":"try:\n    encoder = loader._load_model(cfg, SubModelType.TextEncoder)\nexcept RuntimeError as e:\n    if \"remain on the meta device\" in str(e):\n        reacquire_checkpoint(cfg.path)  # checkpoint missing weights\n    else:\n        raise","preventionTips":["Verify checkpoint file sizes/checksums against the upstream repo after download.","Call model.tie_weights() after loading state dicts that omit tied weights.","Run the meta-tensor scan once after any low_cpu_mem_usage/assign=True load."],"tags":["weights","meta-device","accelerate","model-loading"],"backgroundTag":"leftover-meta-device-weights","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}