hpcaitech/Open-Sora · error · RuntimeError

Activation buffer is full

Error message

Activation buffer is full

What it means

Raised by activation-offload checkpointing when the preallocated CPU activation buffer cannot fit a new tensor. The CheckpointOffloadManager allocates a fixed-size flat buffer up front; each offloaded tensor claims a contiguous slice, so a tensor larger than remaining space (or too many activations) overflows it. It indicates the buffer size was underestimated relative to the model's activation footprint.

Source

Thrown at opensora/acceleration/checkpoint.py:36

    def __init__(self):
        self.enable = False
        self.buffer = None
        self.total_size = 0
        self.avail_offset = 0
        self.tensor_id_queue = []
        self.ignore_tensor_id_set = set()

    def setup_buffer(self, numel: int, dtype: torch.dtype):
        self.buffer = torch.empty(numel, dtype=dtype, pin_memory=True)
        self.total_size = numel
        self.enable = True

    def offload(self, x: torch.Tensor) -> None:
        if not self.enable or id(x) in self.ignore_tensor_id_set:
            return
        size = x.numel()
        if self.avail_offset + size > self.total_size:
            raise RuntimeError("Activation buffer is full")
        assert x.dtype == self.buffer.dtype, f"Wrong dtype of offload tensor"
        cpu_x = self.buffer[self.avail_offset : self.avail_offset + size].view_as(x)
        cpu_x.copy_(x)
        x.data = cpu_x
        self.avail_offset += size
        self.tensor_id_queue.append(id(x))

    def onload(self, x: torch.Tensor) -> None:
        if not self.enable or id(x) in self.ignore_tensor_id_set:
            return
        assert self.tensor_id_queue[-1] == id(x), f"Wrong order of offload/onload"
        # current x is pinned memory
        assert x.data.is_pinned()
        x.data = x.data.to(get_current_device(), non_blocking=True)
        self.tensor_id_queue.pop()
        self.avail_offset -= x.numel()
        if len(self.tensor_id_queue) == 0:
            self.ignore_tensor_id_set.clear()

View on GitHub (pinned to 7ad6a96a13)

Solutions

1. Increase the total buffer size passed when constructing the offload/checkpoint manager so it covers peak activation volume
2. Reduce per-activation memory: smaller batch, shorter video chunks, or more aggressive checkpointing so fewer tensors are offloaded simultaneously
3. Verify the buffer sizing logic accounts for all offloaded activations (sum of numel over one checkpoint segment, not average)
4. Pass the tensors to ignore via ignore_tensor_id_set if some large tensors need not be offloaded

Example fix

# before
manager = CheckpointOffloadManager(buffer_size=1024**3)
# after
manager = CheckpointOffloadManager(buffer_size=4 * 1024**3)  # size for peak activations

size = x.numel()
assert manager.avail_offset + size <= manager.total_size, f'offload buffer overflow: need {manager.avail_offset + size}, have {manager.total_size}'

When it happens

Trigger: Calling offload(x) during checkpointed forward when avail_offset + x.numel() exceeds total_size; e.g. larger batch/spatial dims, longer video sequences, or a mismatch between the buffer size computed at init and actual runtime activation sizes.

Common situations: Scaling up batch size, resolution, or number of frames without resizing the offload buffer; enabling activation checkpoint offload on a bigger model variant than the buffer was sized for.

Related errors

• Unexpected keyword arguments: {kwargs}
• Passing `context_fn` or `debug` is only supported when use_r
• Unsupported input dimension: {x.dim()}
• No chunks were generated. Input shape: {x.shape}
• resize(mode={mode}) not implemented.

AI-assisted analysis of hpcaitech/Open-Sora@7ad6a96a13 (2026-08-28). Data as JSON: /api/errors/2f568650cece6c9e. Report an issue: GitHub.