invoke-ai/InvokeAI · critical · RuntimeError

{context}: {len(meta)} parameter(s) remain on the meta devic

Error message

{context}: {len(meta)} parameter(s) remain on the meta device after loading (missing or mismatched weights): {meta[:10]}

What it means

_verify_encoder_fully_materialized scans a freshly loaded model's parameters and buffers for tensors still on the 'meta' device, which means accelerate's low_cpu_mem_usage loading never assigned real weights to them. Leftover meta tensors indicate missing or mismatched checkpoint weights (e.g. keys not covered, or tied weights not yet resolved). The loader raises RuntimeError listing up to 10 offending names so weight problems fail loudly instead of producing garbage output.

Source

Thrown at invokeai/backend/model_manager/load/model_loaders/ideogram4.py:67

def _verify_encoder_fully_materialized(model: torch.nn.Module, *, context: str) -> None:
    """Fail if any parameter is still on the meta device after loading the text encoder.

    The encoder is built under ``accelerate.init_empty_weights()`` (every param starts on the meta
    device) and then filled from the checkpoint. Missing keys are only acceptable for tied weights, which
    ``transformers`` materializes via ``tie_weights()``; any other missing key leaves a meta tensor that
    would pass loading but fail later during device movement or encoding. Re-tie, then hard-fail if any
    meta tensor remains so a bad/mismatched encoder is rejected at load time instead.
    """
    if hasattr(model, "tie_weights"):
        model.tie_weights()
    meta = [
        name
        for name, tensor in itertools.chain(model.named_parameters(), model.named_buffers())
        if getattr(tensor, "is_meta", False)
    ]
    if meta:
        raise RuntimeError(
            f"{context}: {len(meta)} parameter(s) remain on the meta device after loading "
            f"(missing or mismatched weights): {meta[:10]}"
        )


@ModelLoaderRegistry.register(base=BaseModelType.Ideogram4, type=ModelType.Main, format=ModelFormat.Diffusers)
class Ideogram4DiffusersModel(ModelLoader):
    """Loads Ideogram 4 main models (nf4 / fp8) bundled in diffusers layout."""

    def _load_model(
        self,
        config: AnyModelConfig,
        submodel_type: Optional[SubModelType] = None,
    ) -> AnyModel:
        if not isinstance(config, Main_Diffusers_Ideogram4_Config):
            raise ValueError(f"Expected Main_Diffusers_Ideogram4_Config, got {type(config).__name__}.")
        if submodel_type is None:
            raise Exception("A submodel type must be provided when loading Ideogram 4 main pipelines.")

View on GitHub (pinned to 0b6a024f2f)

Solutions

  1. Inspect the listed parameter names and confirm the checkpoint actually contains those weights; re-download/re-export the checkpoint if truncated.
  2. Ensure the loader calls model.tie_weights() (or equivalent) after load_state_dict when keys are only missing due to weight tying.
  3. Verify the checkpoint key layout matches the expected model architecture; re-map keys if the source used a different naming scheme.
  4. Load without assign=True / with a matching accelerate config if the checkpoint format isn't compatible with meta-device loading.

Example fix

// before
model.load_state_dict(sd, strict=False, assign=True)
_verify_encoder_fully_materialized(model, context=...)
// after: resolve tied weights before verification
model.load_state_dict(sd, strict=False, assign=True)
model.tie_weights()
_verify_encoder_fully_materialized(model, context=...)
Defensive patterns

Strategy: try-catch

Validate before calling

import itertools
meta = [n for n, t in itertools.chain(model.named_parameters(), model.named_buffers()) if getattr(t, "is_meta", False)]
if meta:
    raise RuntimeError(f"missing weights before use: {meta[:10]}")

Type guard

def fully_materialized(model) -> bool:
    import itertools
    return not any(getattr(t, "is_meta", False)
                   for _, t in itertools.chain(model.named_parameters(), model.named_buffers()))

Try / catch

try:
    encoder = loader._load_model(cfg, SubModelType.TextEncoder)
except RuntimeError as e:
    if "remain on the meta device" in str(e):
        reacquire_checkpoint(cfg.path)  # checkpoint missing weights
    else:
        raise

Prevention

When it happens

Trigger: _load_text_encoder loads an Ideogram 4 text encoder with load_state_dict(strict=False, assign=True); the checkpoint is missing keys (or tied weights were not resolved by tie_weights) so some parameters/buffers remain on the meta device, detected by _verify_encoder_fully_materialized.

Common situations: Loading a truncated or partially converted checkpoint; a text-encoder checkpoint that omits weights tied to embeddings (lm_head/embed_tokens); mismatched architecture version where checkpoint keys don't match the model class.

Related errors

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/8bf6c8eed728892d. Report an issue: GitHub.