{"record":{"id":"68fe7865cf9740a7","repo":"huggingface/transformers","slug":"min-should-be-max-got-min-min-max-max","errorCode":null,"errorMessage":"min should be < max (got min: {min}, max: {max})","messagePattern":"min should be < max \\(got min: (.+?), max: (.+?)\\)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/activations.py","lineNumber":141,"sourceCode":"        return input * torch.sigmoid(1.702 * input)\n\n\nclass ClippedGELUActivation(nn.Module):\n    \"\"\"\n    Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as\n    it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to\n    https://huggingface.co/papers/2004.09602.\n\n    Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when\n    initially created.\n\n    For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 +\n    torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415\n    \"\"\"\n\n    def __init__(self, min: float, max: float):\n        if min > max:\n            raise ValueError(f\"min should be < max (got min: {min}, max: {max})\")\n\n        super().__init__()\n        self.min = min\n        self.max = max\n\n    def forward(self, x: Tensor) -> Tensor:\n        return torch.clip(gelu(x), self.min, self.max)\n\n\nclass AccurateGELUActivation(nn.Module):\n    \"\"\"\n    Applies GELU approximation that is faster than default and more accurate than QuickGELU. See:\n    https://github.com/hendrycks/GELUs\n\n    Implemented along with MEGA (Moving Average Equipped Gated Attention)\n    \"\"\"\n\n    def __init__(self):","sourceCodeStart":123,"sourceCodeEnd":159,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/activations.py#L123-L159","documentation":"`deepgemm_fp8_fp4_linear` requires the input tensor to be bf16 or fp16 because `per_token_cast_to_fp8` quantizes from those precisions; any other dtype (fp32, fp8, int) is rejected up front, before the expensive hub-download/JIT kernel load.","triggerScenarios":"Passing a float32 hidden state (common when a model runs in fp32 or when a layer upcasts before the linear), or an already-quantized fp8 input, into `deepgemm_fp8_fp4_linear`.","commonSituations":"Models loaded with `torch_dtype=torch.float32` for debugging; custom forward code that upcasts activations (`.float()`) before experts/linear layers; autocast disabled so activations stay fp32.","solutions":["Cast the input to bf16/fp16 before the call: `input = input.to(torch.bfloat16)` (or load the model with `torch_dtype=torch.bfloat16`)","Ensure autocast/bf16 training is active so activations reach the linear in half precision","Do not pre-quantize activations yourself — the linear does per-token FP8 casting internally"],"exampleFix":"# before\nout = deepgemm_fp8_fp4_linear(x.float(), w, w_scale)  # fp32 -> ValueError\n\n# after\nout = deepgemm_fp8_fp4_linear(x.to(torch.bfloat16), w, w_scale)","handlingStrategy":"type-guard","validationCode":"if input.dtype not in (torch.bfloat16, torch.float16):\n    input = input.to(torch.bfloat16)","typeGuard":"def is_half_precision(t: torch.Tensor) -> bool:\n    return t.dtype in (torch.bfloat16, torch.float16)","tryCatchPattern":null,"preventionTips":["Load models with torch_dtype=torch.bfloat16 when using DeepGEMM","Normalize activation dtype at block boundaries in custom forward code"],"tags":["deepgemm","dtype","bfloat16","activations"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}