{"record":{"id":"2f568650cece6c9e","repo":"hpcaitech/Open-Sora","slug":"activation-buffer-is-full","errorCode":null,"errorMessage":"Activation buffer is full","messagePattern":"Activation buffer is full","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"opensora/acceleration/checkpoint.py","lineNumber":36,"sourceCode":"    def __init__(self):\n        self.enable = False\n        self.buffer = None\n        self.total_size = 0\n        self.avail_offset = 0\n        self.tensor_id_queue = []\n        self.ignore_tensor_id_set = set()\n\n    def setup_buffer(self, numel: int, dtype: torch.dtype):\n        self.buffer = torch.empty(numel, dtype=dtype, pin_memory=True)\n        self.total_size = numel\n        self.enable = True\n\n    def offload(self, x: torch.Tensor) -> None:\n        if not self.enable or id(x) in self.ignore_tensor_id_set:\n            return\n        size = x.numel()\n        if self.avail_offset + size > self.total_size:\n            raise RuntimeError(\"Activation buffer is full\")\n        assert x.dtype == self.buffer.dtype, f\"Wrong dtype of offload tensor\"\n        cpu_x = self.buffer[self.avail_offset : self.avail_offset + size].view_as(x)\n        cpu_x.copy_(x)\n        x.data = cpu_x\n        self.avail_offset += size\n        self.tensor_id_queue.append(id(x))\n\n    def onload(self, x: torch.Tensor) -> None:\n        if not self.enable or id(x) in self.ignore_tensor_id_set:\n            return\n        assert self.tensor_id_queue[-1] == id(x), f\"Wrong order of offload/onload\"\n        # current x is pinned memory\n        assert x.data.is_pinned()\n        x.data = x.data.to(get_current_device(), non_blocking=True)\n        self.tensor_id_queue.pop()\n        self.avail_offset -= x.numel()\n        if len(self.tensor_id_queue) == 0:\n            self.ignore_tensor_id_set.clear()","sourceCodeStart":18,"sourceCodeEnd":54,"githubUrl":"https://github.com/hpcaitech/Open-Sora/blob/7ad6a96a135feb81f755c84fb391818718f6beb2/opensora/acceleration/checkpoint.py#L18-L54","documentation":"Raised by activation-offload checkpointing when the preallocated CPU activation buffer cannot fit a new tensor. The CheckpointOffloadManager allocates a fixed-size flat buffer up front; each offloaded tensor claims a contiguous slice, so a tensor larger than remaining space (or too many activations) overflows it. It indicates the buffer size was underestimated relative to the model's activation footprint.","triggerScenarios":"Calling offload(x) during checkpointed forward when avail_offset + x.numel() exceeds total_size; e.g. larger batch/spatial dims, longer video sequences, or a mismatch between the buffer size computed at init and actual runtime activation sizes.","commonSituations":"Scaling up batch size, resolution, or number of frames without resizing the offload buffer; enabling activation checkpoint offload on a bigger model variant than the buffer was sized for.","solutions":["Increase the total buffer size passed when constructing the offload/checkpoint manager so it covers peak activation volume","Reduce per-activation memory: smaller batch, shorter video chunks, or more aggressive checkpointing so fewer tensors are offloaded simultaneously","Verify the buffer sizing logic accounts for all offloaded activations (sum of numel over one checkpoint segment, not average)","Pass the tensors to ignore via ignore_tensor_id_set if some large tensors need not be offloaded"],"exampleFix":"# before\nmanager = CheckpointOffloadManager(buffer_size=1024**3)\n# after\nmanager = CheckpointOffloadManager(buffer_size=4 * 1024**3)  # size for peak activations","handlingStrategy":"validation","validationCode":"size = x.numel()\nassert manager.avail_offset + size <= manager.total_size, f'offload buffer overflow: need {manager.avail_offset + size}, have {manager.total_size}'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Size the offload buffer from peak activation volume (measure with a dry run at max batch/resolution)","Scale buffer size whenever batch size, frame count, or resolution increases","Unit-test the offload path at production sizes before launching long training runs"],"tags":["activation-checkpointing","memory","offload","pytorch"],"backgroundTag":"gpu-out-of-memory","analyzedSha":"7ad6a96a135feb81f755c84fb391818718f6beb2","analyzedAt":"2026-08-28T16:58:37.171Z","schemaVersion":2},"datasetVersion":"2026-08-28T21:17:43.275Z"}