{"record":{"id":"1faf4ef9c2cf8fe1","repo":"huggingface/transformers","slug":"function-activation-string-not-found-in-act2fn-m","errorCode":null,"errorMessage":"function {activation_string} not found in ACT2FN mapping {list(ACT2FN.keys())}","messagePattern":"function (.+?) not found in ACT2FN mapping (.+?)","errorType":"exception","errorClass":"KeyError","httpStatus":null,"severity":"error","filePath":"src/transformers/activations.py","lineNumber":357,"sourceCode":"    \"relu\": nn.ReLU,\n    \"relu2\": ReLUSquaredActivation,\n    \"relu6\": nn.ReLU6,\n    \"sigmoid\": nn.Sigmoid,\n    \"silu\": SiLUActivation,\n    \"sqrtsoftplus\": SqrtSoftplusActivation,\n    \"swish\": nn.SiLU,\n    \"tanh\": nn.Tanh,\n    \"prelu\": nn.PReLU,\n    \"xielu\": XIELUActivation,\n}\nACT2FN = ClassInstantier(ACT2CLS)\n\n\ndef get_activation(activation_string):\n    if activation_string in ACT2FN:\n        return ACT2FN[activation_string]\n    else:\n        raise KeyError(f\"function {activation_string} not found in ACT2FN mapping {list(ACT2FN.keys())}\")\n\n\n# For backwards compatibility with: from activations import gelu_python\ngelu_python = get_activation(\"gelu_python\")\ngelu_new = get_activation(\"gelu_new\")\ngelu = get_activation(\"gelu\")\ngelu_fast = get_activation(\"gelu_fast\")\ngelu_pytorch_tanh = get_activation(\"gelu_pytorch_tanh\")\nquick_gelu = get_activation(\"quick_gelu\")\nsilu = get_activation(\"silu\")\nmish = get_activation(\"mish\")\nlinear_act = get_activation(\"linear\")\n","sourceCodeStart":339,"sourceCodeEnd":370,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/activations.py#L339-L370","documentation":"`deepgemm_bf16_experts_forward` requires bf16 hidden states — the grouped BF16 kernels and the surrounding packing (`per_token_cast`-free path) assume bfloat16 exactly. fp16 or fp32 hidden states are rejected with a ValueError before kernel load.","triggerScenarios":"Running `experts_implementation` that maps to the DeepGEMM BF16 grouped path on a model kept in fp16 (e.g. `torch_dtype=torch.float16`) or fp32; feeding a manually cast fp16 tensor into the experts forward.","commonSituations":"Mixed-precision training where the router/moe block receives fp16 activations; checkpoints loaded in fp16 for A100-era compatibility then moved to DeepGEMM experts on Hopper.","solutions":["Load/run the model in bfloat16 (`torch_dtype=torch.bfloat16`)","Cast hidden states before the experts call: `hidden_states = hidden_states.to(torch.bfloat16)`","If fp16 must be kept, choose a different experts implementation that supports fp16"],"exampleFix":"# before\nmodel = AutoModelForCausalLM.from_pretrained(m, torch_dtype=torch.float16)\nh = h.half()  # -> ValueError: requires bfloat16\n\n# after\nmodel = AutoModelForCausalLM.from_pretrained(m, torch_dtype=torch.bfloat16)","handlingStrategy":"type-guard","validationCode":"if hidden_states.dtype != torch.bfloat16:\n    hidden_states = hidden_states.to(torch.bfloat16)","typeGuard":"def is_bf16(t: torch.Tensor) -> bool:\n    return t.dtype == torch.bfloat16","tryCatchPattern":null,"preventionTips":["Standardize bf16 end-to-end for DeepGEMM BF16 grouped experts","Avoid fp16 loads on Hopper+ MoE pipelines"],"tags":["deepgemm","bfloat16","moe","dtype"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}