vllm-project/vllm · error · RuntimeError

Assigning / modifying buffers of nn.Module during forward pa

Error message

Assigning / modifying buffers of nn.Module during forward pass is not allowed when using cudagraph inside the compiler because it will cause silent errors. Please use eager mode or fix the code. The following code contains clues about which buffer is being modified (please search for the usage of the function `update`):
{src}

What it means

With the bytecode hook enabled (VLLM_USE_BYTECODE_HOOK, torch<2.8 path), vLLM inspects compiled bytecode of the forward; if 'update' appears in co_names, some nn.Module buffer/dict is being assigned or mutated during forward. Under cudagraph this mutates state that the replaying graph will not re-execute correctly, causing silent numerical errors, so vLLM raises RuntimeError with decompiled source as a clue.

Source

Thrown at vllm/compilation/wrapper.py:264

                    logger.debug("Dynamo transformed code saved to %s", decompiled_file)
                except Exception:
                    pass

        if (
            self.vllm_config.compilation_config.cudagraph_mode != CUDAGraphMode.NONE
            and "update" in new_code.co_names
        ):
            import depyf

            src = depyf.decompile(new_code)
            msg = (
                "Assigning / modifying buffers of nn.Module during forward pass is not "
                "allowed when using cudagraph inside the compiler because it will "
                "cause silent errors. Please use eager mode or fix the code. The "
                "following code contains clues about which buffer is being modified "
                f"(please search for the usage of the function `update`):\n{src}"
            )
            raise RuntimeError(msg)

    def cleanup(self) -> None:
        """Remove the bytecode hook registered by this instance."""
        handle = getattr(self, "_bytecode_hook_handle", None)
        if handle is not None:
            handle.remove()

    @contextmanager
    def _dispatch_to_compiled_code(self) -> Generator[None, None, None]:
        # noqa: E501
        """
        Context manager to dispatch to internally compiled code for torch<2.8.
        Why does this work? Because Dynamo guarantees that the compiled
        bytecode has exactly the same arguments, cell variables, and free
        variables as the original code. Therefore we can directly switch
        the code object in the function and call it.

        See https://dev-discuss.pytorch.org/t/what-is-the-relationship-requirement-among-original-bytecode-transformed-bytecode-and-bytecode-returned-by-hooks-in-dynamo/1693/7 for more details.

View on GitHub (pinned to c794754062)

Solutions

1. Find the `update` call in the decompiled source quoted in the error and remove the buffer mutation from forward (compute it outside or in a non-compiled path).
2. Or run with eager mode / disable cudagraph (enforce_eager=True or cudagraph_mode='NONE') while you fix the model.
3. If the state is genuinely per-step, move it to CPU-side orchestration outside the compiled region.

Example fix

# before
class MyModel(nn.Module):
    def forward(self, x):
        self.step_count.update(x.shape)  # buffer mutation in forward
        return self.proj(x)
# after
class MyModel(nn.Module):
    def forward(self, x):
        return self.proj(x)  # track step_count outside the compiled forward

import ast, inspect

class BufferMutationVisitor(ast.NodeVisitor):
    def visit_Call(self, node):
        if getattr(node.func, 'attr', None) == 'update':
            raise ValueError(f'buffer update at line {node.lineno} mutates state in forward')
        self.generic_visit(node)

def check_no_buffer_updates(model_cls) -> None:
    BufferMutationVisitor().visit(ast.parse(inspect.getsource(model_cls.forward)))

When it happens

Trigger: A model whose forward calls something like self.cache.update(...), self.counters.update(...), or assigns buffers, executed under vLLM compilation with cudagraph enabled and the bytecode hook active (torch<2.8 / VLLM_USE_BYTECODE_HOOK=1).

Common situations: Custom models ported from training code that keep running statistics, sliding windows, or KV-like dicts updated inside forward; MoE or calibration code mutating module state per step.

Related errors

• Attribute {key} not exists in the runnable of cudagraph wrap
• CUDA graph capturing detected at an inappropriate time. This
• cudagraph_capture_sizes not supported in compile_sizes.This
• HTTP request failed: {0}
• JSON error: {0}

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/32f2867cecf97b92. Report an issue: GitHub.