huggingface/transformers · error · ValueError

DebugUnderflowOverflow: inf/nan detected, aborting as there

Error message

DebugUnderflowOverflow: inf/nan detected, aborting as there is no point running further. Please scroll up above this traceback to see the activation values prior to this event.

What it means

DebugUnderflowOverflow is an opt-in tracer that inspects module inputs/outputs each batch looking for inf/nan. When it detects an overflow (and is not in trace-only mode) it dumps the saved frames - per-layer min/max activation stats printed above the traceback - and raises this ValueError to stop training, since weights are already corrupted. The message tells you the diagnostic output is the frame dump printed before the traceback.

Source

Thrown at src/transformers/debug_utils.py:282

            self.batch_number += 1
            last_frame_of_batch = True

        self.create_frame(module, input, output)

        # if last_frame_of_batch:
        #     self.batch_end_frame()

        if trace_mode:
            self.trace_frames()

        if last_frame_of_batch:
            self.batch_start_frame()

        if self.detected_overflow and not trace_mode:
            self.dump_saved_frames()

            # now we can abort, as it's pointless to continue running
            raise ValueError(
                "DebugUnderflowOverflow: inf/nan detected, aborting as there is no point running further. "
                "Please scroll up above this traceback to see the activation values prior to this event."
            )

        # abort after certain batch if requested to do so
        if self.abort_after_batch_num is not None and self.batch_number > self.abort_after_batch_num:
            raise ValueError(
                f"DebugUnderflowOverflow: aborting after {self.batch_number} batches due to"
                f" `abort_after_batch_num={self.abort_after_batch_num}` arg"
            )


def get_abs_min_max(var, ctx):
    abs_var = var.abs()
    return f"{abs_var.min():8.2e} {abs_var.max():8.2e} {ctx}"


def detect_overflow(var, ctx):

View on GitHub (pinned to a597f97485)

Solutions

Read the frame dump above the traceback: the layer whose min/max first shows inf/nan (or absurd magnitude) is the origin; fix or stabilize that layer.
Lower the learning rate, enable gradient clipping, or switch fp16 to bf16 if hardware supports it.
Sanitize inputs (no NaN/inf in batches) and add eps to log/softmax/division ops in a custom loss head.
Once fixed, remove DebugUnderflowOverflow - it slows training considerably.

Example fix

# before
from transformers import DebugUnderflowOverflow
debug_overflow = DebugUnderflowOverflow(model)  # aborts on first inf/nan
trainer.train()

# after: stabilize mixed precision
trainer = Trainer(model=model, args=TrainingArguments(
    fp16=True,
    bf16=False,
    learning_rate=2e-5,        # was 1e-3, caused fp16 overflow
    max_grad_norm=1.0,
))
trainer.train()

Defensive patterns

Strategy: try-catch

Validate before calling

def batch_is_clean(batch) -> bool:
    return all(not torch.isnan(t).any() and not torch.isinf(t).any() for t in batch if torch.is_tensor(t))

Try / catch

try:
    trainer.train()
except ValueError as e:
    if "DebugUnderflowOverflow" in str(e) and "inf/nan" in str(e):
        # frame dump above the traceback identifies the first bad layer
        logger.error("overflow detected; lowering lr / switching to bf16")
        raise
    raise

Prevention

Use bf16 instead of fp16 on supported hardware to reduce overflow risk.
Always set max_grad_norm and a conservative learning rate for fine-tuning.
Treat the frame dump as the real diagnostic: fix the first layer showing inf/nan.
Remove DebugUnderflowOverflow from production training loops.

When it happens

Trigger: Attaching DebugUnderflowOverflow(model) and training; fp16 with a learning rate too high so activations/grads explode; unstable initialization, a broken input pipeline feeding garbage values, or loss functions producing inf (e.g. log(0)).

Common situations: Mixed-precision fine-tuning where fp16 overflows in attention/logits layers; a downstream layer NaN-ing and the tracer pinpointing the first frame where inf/nan appears; debugging why loss becomes NaN mid-run.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/1e92004800b7520d. Report an issue: GitHub.