{"record":{"id":"2622af7fe6b6078b","repo":"huggingface/transformers","slug":"flex-attention-does-not-support-dropout-pleas","errorCode":null,"errorMessage":"`flex_attention` does not support `dropout`. Please use it with inference only (`model.eval()`) or turn off the attention dropout in the respective config.","messagePattern":"`flex_attention` does not support `dropout`\\. Please use it with inference only \\(`model\\.eval\\(\\)`\\) or turn off the attention dropout in the respective config\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/flex_attention.py","lineNumber":275,"sourceCode":"        return hidden_states\n    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)\n    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)\n\n\ndef flex_attention_forward(\n    module: torch.nn.Module,\n    query: torch.Tensor,\n    key: torch.Tensor,\n    value: torch.Tensor,\n    attention_mask: Union[torch.Tensor, \"BlockMask\"],\n    scaling: float | None = None,\n    softcap: float | None = None,\n    s_aux: torch.Tensor | None = None,\n    position_bias: torch.Tensor | None = None,\n    **kwargs,\n) -> tuple[torch.Tensor, torch.Tensor | None]:\n    if kwargs.get(\"dropout\", 0.0) > 0:\n        raise ValueError(\n            \"`flex_attention` does not support `dropout`. Please use it with inference\"\n            \" only (`model.eval()`) or turn off the attention dropout in the respective config.\"\n        )\n\n    block_mask = None\n    score_mask = None\n    if isinstance(attention_mask, BlockMask):\n        block_mask = attention_mask\n    else:\n        score_mask = attention_mask\n\n    if score_mask is not None:\n        score_mask = score_mask[:, :, :, : key.shape[-2]]\n\n    def score_mod(score, batch_idx, head_idx, q_idx, kv_idx):\n        if softcap is not None:\n            score = softcap * torch.tanh(score / softcap)\n        if score_mask is not None:","sourceCodeStart":257,"sourceCodeEnd":293,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/flex_attention.py#L257-L293","documentation":"The flex_attention integration does not implement dropout: torch's flex_attention has no dropout parameter (unlike SDPA), so any model config or call that requests attention dropout > 0 while using attn_implementation=\"flex_attention\" is rejected immediately with ValueError. The message points to either inference-only usage (model.eval() typically sets dropout to 0 at call time) or disabling dropout in the config.","triggerScenarios":"Instantiating a model with attn_implementation=\"flex_attention\" whose config has attention_dropout > 0 (or passing dropout via the attention interface), then running a forward while the effective dropout value is > 0 — typically in train mode.","commonSituations":"Fine-tuning a model (train mode) whose config carries attention_dropout=0.1 (very common in Llama/Gemma-family configs) after switching the attention implementation to flex_attention for block-sparse masks; sharing a training config with an inference-only implementation.","solutions":["Set config.attention_dropout = 0.0 before/at model load when using flex_attention","Only use flex_attention for inference (model.eval()), where dropout resolves to 0","For training with dropout, use sdpa or eager attention instead"],"exampleFix":"# before\nconfig = AutoConfig.from_pretrained(model_id)  # attention_dropout=0.1\nmodel = AutoModelForCausalLM.from_pretrained(model_id, config=config, attn_implementation=\"flex_attention\")\nmodel.train(); model(**batch)  # ValueError\n\n# after\nconfig.attention_dropout = 0.0\nmodel = AutoModelForCausalLM.from_pretrained(model_id, config=config, attn_implementation=\"flex_attention\")","handlingStrategy":"validation","validationCode":"config = AutoConfig.from_pretrained(model_id)\nif config.attention_dropout:\n    config.attention_dropout = 0.0  # flex_attention has no dropout\nmodel = AutoModelForCausalLM.from_pretrained(model_id, config=config, attn_implementation=\"flex_attention\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Zero out attention_dropout in the config whenever you select flex_attention","Reserve flex_attention for inference; use sdpa/eager for dropout-enabled training"],"tags":["flex-attention","attention","dropout","config","training-vs-inference"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}