{"record":{"id":"094fb45b4f0a0c2c","repo":"invoke-ai/InvokeAI","slug":"mistral-encoder-returned-only-num-layers-hidden","errorCode":null,"errorMessage":"Mistral encoder returned only {num_layers} hidden layer(s), but FLUX.2 [dev] reads layers {DEV_EXTRACTION_LAYERS} and requires at least {max(DEV_EXTRACTION_LAYERS)}. This is not a supported FLUX.2 [dev] text encoder.","messagePattern":"Mistral encoder returned only (.+?) hidden layer\\(s\\), but FLUX\\.2 \\[dev\\] reads layers (.+?) and requires at least (.+?)\\. This is not a supported FLUX\\.2 \\[dev\\] text encoder\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"invokeai/app/invocations/flux2_dev_text_encoder.py","lineNumber":228,"sourceCode":"            use_cache=False,\n        )\n        if not hasattr(outputs, \"hidden_states\") or outputs.hidden_states is None:\n            raise RuntimeError(\n                \"Mistral encoder did not return hidden_states. \"\n                \"Ensure output_hidden_states=True is supported by this model.\"\n            )\n        num_hidden_states = len(outputs.hidden_states)  # = num_hidden_layers + 1 (embedding output)\n        num_layers = num_hidden_states - 1\n\n        # The standalone Mistral encoder loaders only accept 30-layer cow or 40-layer\n        # Mistral Small 3 weights, so hidden_states[] should always contain the layers\n        # FLUX.2 [dev]'s joint attention was trained to read (10/20/30). A text encoder\n        # extracted from a Main_Diffusers_Flux2 pipeline, however, is loaded via generic\n        # from_pretrained with no layer-count validation — so a nonstandard pipeline with\n        # a <30-layer encoder could reach here. Fail loudly instead of inventing extraction\n        # indices that would silently produce off-distribution (degraded) embeddings.\n        if num_layers < max(DEV_EXTRACTION_LAYERS):\n            raise RuntimeError(\n                f\"Mistral encoder returned only {num_layers} hidden layer(s), but FLUX.2 [dev] reads \"\n                f\"layers {DEV_EXTRACTION_LAYERS} and requires at least {max(DEV_EXTRACTION_LAYERS)}. \"\n                \"This is not a supported FLUX.2 [dev] text encoder.\"\n            )\n        extraction_layers = DEV_EXTRACTION_LAYERS\n\n        # Concatenate the selected layers along the hidden dim: (B, seq, 3 * hidden_size).\n        # This is byte-identical to stack(dim=1).permute(0,2,1,3).reshape(...) but avoids\n        # the two intermediate full copies that stack + permute-reshape would allocate.\n        prompt_embeds = torch.cat([outputs.hidden_states[i] for i in extraction_layers], dim=-1)\n        prompt_embeds = prompt_embeds.to(dtype=text_encoder.dtype, device=device)\n\n        return prompt_embeds\n\n    def _lora_iterator(self, context: InvocationContext) -> Iterator[Tuple[ModelPatchRaw, float]]:\n        \"\"\"Iterate over LoRAs to apply to the Mistral encoder.\"\"\"\n        for lora in self.mistral_encoder.loras:\n            lora_info = context.models.load(lora.lora)","sourceCodeStart":210,"sourceCodeEnd":246,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/app/invocations/flux2_dev_text_encoder.py#L210-L246","documentation":"_encode_prompt throws this RuntimeError when the Mistral encoder has fewer hidden layers than FLUX.2 [dev] requires. FLUX.2 [dev]'s joint attention was trained to read hidden states from layers DEV_EXTRACTION_LAYERS (10/20/30), so an encoder with fewer than 30 layers cannot satisfy the extraction indices. Rather than inventing indices that would yield degraded embeddings, the encoder is rejected loudly.","triggerScenarios":"outputs.hidden_states has length <= 30 (num_layers < max(DEV_EXTRACTION_LAYERS)); possible because Diffusers-pipeline encoders load via generic from_pretrained with no layer validation.","commonSituations":"A nonstandard FLUX.2 Diffusers pipeline with a small/short Mistral-compatible encoder (e.g. a distilled or Klein-family encoder) is supplied as the encoder source and passes earlier checks.","solutions":["Supply the full Mistral Small 3.1 text encoder (with at least 30 hidden layers) as the encoder source.","Extract the encoder from a genuine FLUX.2 [dev] Diffusers pipeline rather than a nonstandard or distilled pipeline.","Verify the encoder config's num_hidden_layers is >= 30 before loading."],"exampleFix":"// before\nencoder = tiny_distilled_mistral_encoder  // 16 layers\n// after\nencoder = mistral_small_3_1_full  // >= 30 layers","handlingStrategy":"validation","validationCode":"cfg = encoder.config\nif getattr(cfg, \"num_hidden_layers\", 0) < max(DEV_EXTRACTION_LAYERS):\n    raise ValueError(f\"Encoder has {cfg.num_hidden_layers} layers; FLUX.2 [dev] needs >= {max(DEV_EXTRACTION_LAYERS)}\")","typeGuard":"def supports_flux2_dev_extraction(encoder) -> bool:\n    return getattr(encoder.config, \"num_hidden_layers\", 0) >= max(DEV_EXTRACTION_LAYERS)","tryCatchPattern":"try:\n    output = text_encoder_invocation.invoke(context)\nexcept RuntimeError as e:\n    if \"hidden layer(s)\" in str(e) and \"not a supported FLUX.2 [dev] text encoder\" in str(e):\n        encoder = load_full_mistral_small_3_1()\n        output = retry_encoding(encoder)\n    else:\n        raise","preventionTips":["Check num_hidden_layers >= 30 in the encoder config before use.","Extract encoders only from official FLUX.2 [dev] Diffusers pipelines.","Avoid distilled/quantized encoder variants for [dev] conditioning."],"tags":["invokeai","model-architecture","text-encoder","layer-count","flux2"],"backgroundTag":"insufficient-hidden-layers","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}