What causes "caching is not supported" in vllm-project/vllm?

Implement load(handle, graph, example_inputs, graph_index, compile_range) in your CompilerInterface subclass, or return None as the handle from compile() so caching is skipped

vllm-project/vllm · error · NotImplementedError

caching is not supported

Error message

caching is not supported

What it means

CompilerInterface is the base class for vLLM's compilation backends; load() implements cache retrieval and its default body raises NotImplementedError('caching is not supported'). A custom/other compiler backend that returns a handle from compile() without implementing load() will blow up as soon as the compilation cache tries to reuse the artifact.

Source

Thrown at vllm/compilation/compiler_interface.py:114

        `cache_dir/key`.
        """
        return None, None

    def load(
        self,
        handle: Any,
        graph: fx.GraphModule,
        example_inputs: list[Any],
        graph_index: int,
        compile_range: Range,
    ) -> Callable[..., Any]:
        """
        Load the compiled function from the handle.
        Raises an error if the handle is invalid.

        The handle is the second return value of the `compile` function.
        """
        raise NotImplementedError("caching is not supported")


class AlwaysHitShapeEnv:
    """
    Why do we need this class:

    For normal `torch.compile` usage, every compilation will have
    one Dynamo bytecode compilation and one Inductor compilation.
    The Inductor compilation happens under the context of the
    Dynamo bytecode compilation, and that context is used to
    determine the dynamic shape information, etc.

    For our use case, we only run Dynamo bytecode compilation once,
    and run Inductor compilation multiple times with different shapes
    plus a general shape. The compilation for specific shapes happens
    outside of the context of the Dynamo bytecode compilation. At that
    time, we don't have shape environment to provide to Inductor, and
    it will fail the Inductor code cache lookup.

View on GitHub (pinned to c794754062)

Solutions

Implement load(handle, graph, example_inputs, graph_index, compile_range) in your CompilerInterface subclass, or return None as the handle from compile() so caching is skipped
If you do not need caching, disable the compile cache for that backend (VLLM_DISABLE_COMPILE_CACHE=1 or disable_cache in compilation config)
Clear the on-disk compile cache so stale handles from other backends are not loaded

Example fix

# before
class MyCompiler(CompilerInterface):
    def compile(self, *a, **k):
        return fn, "my_handle"   # handle returned, load() not implemented
# after
class MyCompiler(CompilerInterface):
    def compile(self, *a, **k):
        return fn, "my_handle"
    def load(self, handle, graph, example_inputs, graph_index, compile_range):
        return deserialize_artifact(handle)

Defensive patterns

Strategy: validation

Validate before calling

import inspect
assert type(compiler).load is not CompilerInterface.load, "backend must implement load() to use the cache"

Type guard

def backend_supports_cache(compiler) -> bool:
    from vllm.compilation.compiler_interface import CompilerInterface
    return type(compiler).load is not CompilerInterface.load

Try / catch

try:
    fn = compiler.load(handle, graph, inputs, idx, rng)
except NotImplementedError:
    fn = compiler.compile(graph, inputs, cfg, rng, None)[0]  # recompile without cache

Prevention

Return a None handle from compile() if your backend cannot reload artifacts
Implement load() whenever you return a non-None handle

When it happens

Trigger: Using a third-party or custom CompilerInterface subclass that does not override load() (returns the base implementation) while the compile cache attempts load_from_cache with a previously stored handle; also directly calling interface.load(...) on the base class in tests.

Common situations: Plugging a custom inductor-like compiler into vLLM's compilation backend registry; a backend that returns a path string from compile() but forgot the matching load(); version skew where the cache records handles a backend cannot reload.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/0e477cce035a77ed. Report an issue: GitHub.