{"record":{"id":"1e92004800b7520d","repo":"huggingface/transformers","slug":"debugunderflowoverflow-inf-nan-detected-aborting","errorCode":null,"errorMessage":"DebugUnderflowOverflow: inf/nan detected, aborting as there is no point running further. Please scroll up above this traceback to see the activation values prior to this event.","messagePattern":"DebugUnderflowOverflow: inf/nan detected, aborting as there is no point running further\\. Please scroll up above this traceback to see the activation values prior to this event\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/debug_utils.py","lineNumber":282,"sourceCode":"            self.batch_number += 1\n            last_frame_of_batch = True\n\n        self.create_frame(module, input, output)\n\n        # if last_frame_of_batch:\n        #     self.batch_end_frame()\n\n        if trace_mode:\n            self.trace_frames()\n\n        if last_frame_of_batch:\n            self.batch_start_frame()\n\n        if self.detected_overflow and not trace_mode:\n            self.dump_saved_frames()\n\n            # now we can abort, as it's pointless to continue running\n            raise ValueError(\n                \"DebugUnderflowOverflow: inf/nan detected, aborting as there is no point running further. \"\n                \"Please scroll up above this traceback to see the activation values prior to this event.\"\n            )\n\n        # abort after certain batch if requested to do so\n        if self.abort_after_batch_num is not None and self.batch_number > self.abort_after_batch_num:\n            raise ValueError(\n                f\"DebugUnderflowOverflow: aborting after {self.batch_number} batches due to\"\n                f\" `abort_after_batch_num={self.abort_after_batch_num}` arg\"\n            )\n\n\ndef get_abs_min_max(var, ctx):\n    abs_var = var.abs()\n    return f\"{abs_var.min():8.2e} {abs_var.max():8.2e} {ctx}\"\n\n\ndef detect_overflow(var, ctx):","sourceCodeStart":264,"sourceCodeEnd":300,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/debug_utils.py#L264-L300","documentation":"DebugUnderflowOverflow is an opt-in tracer that inspects module inputs/outputs each batch looking for inf/nan. When it detects an overflow (and is not in trace-only mode) it dumps the saved frames - per-layer min/max activation stats printed above the traceback - and raises this ValueError to stop training, since weights are already corrupted. The message tells you the diagnostic output is the frame dump printed before the traceback.","triggerScenarios":"Attaching DebugUnderflowOverflow(model) and training; fp16 with a learning rate too high so activations/grads explode; unstable initialization, a broken input pipeline feeding garbage values, or loss functions producing inf (e.g. log(0)).","commonSituations":"Mixed-precision fine-tuning where fp16 overflows in attention/logits layers; a downstream layer NaN-ing and the tracer pinpointing the first frame where inf/nan appears; debugging why loss becomes NaN mid-run.","solutions":["Read the frame dump above the traceback: the layer whose min/max first shows inf/nan (or absurd magnitude) is the origin; fix or stabilize that layer.","Lower the learning rate, enable gradient clipping, or switch fp16 to bf16 if hardware supports it.","Sanitize inputs (no NaN/inf in batches) and add eps to log/softmax/division ops in a custom loss head.","Once fixed, remove DebugUnderflowOverflow - it slows training considerably."],"exampleFix":"# before\nfrom transformers import DebugUnderflowOverflow\ndebug_overflow = DebugUnderflowOverflow(model)  # aborts on first inf/nan\ntrainer.train()\n\n# after: stabilize mixed precision\ntrainer = Trainer(model=model, args=TrainingArguments(\n    fp16=True,\n    bf16=False,\n    learning_rate=2e-5,        # was 1e-3, caused fp16 overflow\n    max_grad_norm=1.0,\n))\ntrainer.train()","handlingStrategy":"try-catch","validationCode":"def batch_is_clean(batch) -> bool:\n    return all(not torch.isnan(t).any() and not torch.isinf(t).any() for t in batch if torch.is_tensor(t))","typeGuard":null,"tryCatchPattern":"try:\n    trainer.train()\nexcept ValueError as e:\n    if \"DebugUnderflowOverflow\" in str(e) and \"inf/nan\" in str(e):\n        # frame dump above the traceback identifies the first bad layer\n        logger.error(\"overflow detected; lowering lr / switching to bf16\")\n        raise\n    raise","preventionTips":["Use bf16 instead of fp16 on supported hardware to reduce overflow risk.","Always set max_grad_norm and a conservative learning rate for fine-tuning.","Treat the frame dump as the real diagnostic: fix the first layer showing inf/nan.","Remove DebugUnderflowOverflow from production training loops."],"tags":["debugging","numerical-stability","mixed-precision","nan"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}