huggingface/transformers · warning · ValueError

DebugUnderflowOverflow: aborting after

Error message

DebugUnderflowOverflow: aborting after {self.batch_number} batches due to `abort_after_batch_num={self.abort_after_batch_num}` arg

What it means

DebugUnderflowOverflow supports an abort_after_batch_num argument to stop execution after a chosen batch for interactive debugging. When self.batch_number exceeds that limit, it raises this ValueError deliberately. This is a controlled, requested abort - the exception text mirrors the argument you passed - not a fault in the model or data.

Solutions

  1. If the abort was intentional, inspect state in the debugger/REPL as planned; then raise or remove abort_after_batch_num.
  2. For full runs, detach the debugger: do not instantiate DebugUnderflowOverflow (or set abort_after_batch_num=None).
  3. If you still need overflow detection for the whole run, keep the instance but without abort_after_batch_num.

Example fix

# before
debug_overflow = DebugUnderflowOverflow(model, abort_after_batch_num=5)
trainer.train()  # raises at batch 6 by design

# after: full training, no artificial abort
debug_overflow = DebugUnderflowOverflow(model)  # or remove entirely
trainer.train()
Defensive patterns

Strategy: try-catch

Try / catch

try:
    trainer.train()
except ValueError as e:
    if "abort_after_batch_num" in str(e):
        logger.info("debugger abort reached; continuing without debug hook")
    else:
        raise

Prevention

When it happens

Trigger: Constructing DebugUnderflowOverflow(model, abort_after_batch_num=N) and letting training run past batch N; leaving a debug instance attached from an earlier debugging session.

Common situations: Interactive debugging where you drop into a debugger at a specific batch to inspect weights; forgetting to remove the debug hook before launching a real training run.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/e22e5b6d8338dcf3. Report an issue: GitHub.

Appendix: source

Thrown at src/transformers/debug_utils.py:289

        if trace_mode:
            self.trace_frames()

        if last_frame_of_batch:
            self.batch_start_frame()

        if self.detected_overflow and not trace_mode:
            self.dump_saved_frames()

            # now we can abort, as it's pointless to continue running
            raise ValueError(
                "DebugUnderflowOverflow: inf/nan detected, aborting as there is no point running further. "
                "Please scroll up above this traceback to see the activation values prior to this event."
            )

        # abort after certain batch if requested to do so
        if self.abort_after_batch_num is not None and self.batch_number > self.abort_after_batch_num:
            raise ValueError(
                f"DebugUnderflowOverflow: aborting after {self.batch_number} batches due to"
                f" `abort_after_batch_num={self.abort_after_batch_num}` arg"
            )


def get_abs_min_max(var, ctx):
    abs_var = var.abs()
    return f"{abs_var.min():8.2e} {abs_var.max():8.2e} {ctx}"


def detect_overflow(var, ctx):
    """
    Report whether the tensor contains any `nan` or `inf` entries.

    This is useful for detecting overflows/underflows and best to call right after the function that did some math that
    modified the tensor in question.

    This function contains a few other helper features that you can enable and tweak directly if you want to track

View on GitHub (pinned to a597f97485)