{"record":{"id":"32f2867cecf97b92","repo":"vllm-project/vllm","slug":"assigning-modifying-buffers-of-nn-module-during","errorCode":null,"errorMessage":"Assigning / modifying buffers of nn.Module during forward pass is not allowed when using cudagraph inside the compiler because it will cause silent errors. Please use eager mode or fix the code. The following code contains clues about which buffer is being modified (please search for the usage of the function `update`):\n{src}","messagePattern":"Assigning / modifying buffers of nn\\.Module during forward pass is not allowed when using cudagraph inside the compiler because it will cause silent errors\\. Please use eager mode or fix the code\\. The following code contains clues about which buffer is being modified \\(please search for the usage of the function `update`\\):\n(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"vllm/compilation/wrapper.py","lineNumber":264,"sourceCode":"                    logger.debug(\"Dynamo transformed code saved to %s\", decompiled_file)\n                except Exception:\n                    pass\n\n        if (\n            self.vllm_config.compilation_config.cudagraph_mode != CUDAGraphMode.NONE\n            and \"update\" in new_code.co_names\n        ):\n            import depyf\n\n            src = depyf.decompile(new_code)\n            msg = (\n                \"Assigning / modifying buffers of nn.Module during forward pass is not \"\n                \"allowed when using cudagraph inside the compiler because it will \"\n                \"cause silent errors. Please use eager mode or fix the code. The \"\n                \"following code contains clues about which buffer is being modified \"\n                f\"(please search for the usage of the function `update`):\\n{src}\"\n            )\n            raise RuntimeError(msg)\n\n    def cleanup(self) -> None:\n        \"\"\"Remove the bytecode hook registered by this instance.\"\"\"\n        handle = getattr(self, \"_bytecode_hook_handle\", None)\n        if handle is not None:\n            handle.remove()\n\n    @contextmanager\n    def _dispatch_to_compiled_code(self) -> Generator[None, None, None]:\n        # noqa: E501\n        \"\"\"\n        Context manager to dispatch to internally compiled code for torch<2.8.\n        Why does this work? Because Dynamo guarantees that the compiled\n        bytecode has exactly the same arguments, cell variables, and free\n        variables as the original code. Therefore we can directly switch\n        the code object in the function and call it.\n\n        See https://dev-discuss.pytorch.org/t/what-is-the-relationship-requirement-among-original-bytecode-transformed-bytecode-and-bytecode-returned-by-hooks-in-dynamo/1693/7 for more details.","sourceCodeStart":246,"sourceCodeEnd":282,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/compilation/wrapper.py#L246-L282","documentation":"With the bytecode hook enabled (VLLM_USE_BYTECODE_HOOK, torch<2.8 path), vLLM inspects compiled bytecode of the forward; if 'update' appears in co_names, some nn.Module buffer/dict is being assigned or mutated during forward. Under cudagraph this mutates state that the replaying graph will not re-execute correctly, causing silent numerical errors, so vLLM raises RuntimeError with decompiled source as a clue.","triggerScenarios":"A model whose forward calls something like self.cache.update(...), self.counters.update(...), or assigns buffers, executed under vLLM compilation with cudagraph enabled and the bytecode hook active (torch<2.8 / VLLM_USE_BYTECODE_HOOK=1).","commonSituations":"Custom models ported from training code that keep running statistics, sliding windows, or KV-like dicts updated inside forward; MoE or calibration code mutating module state per step.","solutions":["Find the `update` call in the decompiled source quoted in the error and remove the buffer mutation from forward (compute it outside or in a non-compiled path).","Or run with eager mode / disable cudagraph (enforce_eager=True or cudagraph_mode='NONE') while you fix the model.","If the state is genuinely per-step, move it to CPU-side orchestration outside the compiled region."],"exampleFix":"# before\nclass MyModel(nn.Module):\n    def forward(self, x):\n        self.step_count.update(x.shape)  # buffer mutation in forward\n        return self.proj(x)\n# after\nclass MyModel(nn.Module):\n    def forward(self, x):\n        return self.proj(x)  # track step_count outside the compiled forward","handlingStrategy":"validation","validationCode":"import ast, inspect\n\nclass BufferMutationVisitor(ast.NodeVisitor):\n    def visit_Call(self, node):\n        if getattr(node.func, 'attr', None) == 'update':\n            raise ValueError(f'buffer update at line {node.lineno} mutates state in forward')\n        self.generic_visit(node)\n\ndef check_no_buffer_updates(model_cls) -> None:\n    BufferMutationVisitor().visit(ast.parse(inspect.getsource(model_cls.forward)))","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never mutate module buffers/dicts inside forward of served models","Keep per-step state outside the compiled graph","Run with cudagraph enabled in CI to catch silent mutation issues early"],"tags":["cudagraph","bytecode-hook","silent-corruption","vllm"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}