pathwaycom/pathway · critical · RuntimeError

no persistent storage configured for the disk cache

Error message

no persistent storage configured for the disk cache

What it means

RuntimeError raised lazily (on first cache access, not at construction) when a disk cache needs a persistent directory but the PATHWAY_PERSISTENT_STORAGE environment variable is unset. The cache directory is derived as $PATHWAY_PERSISTENT_STORAGE/runtime_calls/<cache_name>.

Source

Thrown at python/pathway/internals/udfs/caches.py:98

            cache = self._get_cache(func)
            key = self.make_key(args, kwargs)
            if cache is None:
                return func(*args, **kwargs)
            if key not in cache:
                result = func(*args, **kwargs)
                cache[key] = result
            return cache[key]

        return wrapper

    def _get_cache(self, func: Callable) -> diskcache.Cache | None:
        if self._cache is None:
            if self._name is None:
                func = inspect.unwrap(func)
                self._name = f"{func.__module__}_{func.__qualname__}"
            storage_root = os.environ.get("PATHWAY_PERSISTENT_STORAGE")
            if storage_root is None:
                raise RuntimeError(
                    "no persistent storage configured for the disk cache"
                )
            cache_dir = Path(storage_root) / "runtime_calls"
            self._cache = diskcache.Cache(
                cache_dir / self._name, size_limit=self._size_limit
            )
        return self._cache


class DefaultCache(DiskCache):
    """
    The default caching strategy.
    Persistence layer will be used if enabled. Otherwise, cache will be disabled.
    """

    def _get_cache(self, func: Callable) -> diskcache.Cache | None:
        if "PATHWAY_PERSISTENT_STORAGE" not in os.environ:
            return None

View on GitHub (pinned to fa2f74a464)

Solutions

1. Set PATHWAY_PERSISTENT_STORAGE to a writable directory before starting the process: export PATHWAY_PERSISTENT_STORAGE=/path/to/storage
2. Or configure persistence via pw.persistence.Config(...) which sets the storage root
3. In containers/CI, inject the variable in the entrypoint or compose file
4. If persistence is not needed, use an in-memory cache strategy instead of DiskCache

Example fix

// before
PATHWAY_PERSISTENT_STORAGE unset; @pw.udf(cache_strategy=pw.udfs.DiskCache()) -> RuntimeError on first call

// after
export PATHWAY_PERSISTENT_STORAGE=/var/lib/pathway
python run_pipeline.py

import os

def assert_persistent_storage_configured():
    root = os.environ.get('PATHWAY_PERSISTENT_STORAGE')
    if not root:
        raise RuntimeError(
            'Set PATHWAY_PERSISTENT_STORAGE to a writable dir before using disk caches'
        )
    return root

try:
    result = udf(x)
    pw.run()
except RuntimeError as e:
    if 'no persistent storage' in str(e):
        os.environ['PATHWAY_PERSISTENT_STORAGE'] = './persist'
        pw.run()
    else:
        raise

When it happens

Trigger: Using pw.udfs.DiskCache (or DefaultCache) in a UDF without setting PATHWAY_PERSISTENT_STORAGE. The error surfaces on the first UDF call when _get_cache runs, which can be deep inside pipeline execution, far from the configuration mistake.

Common situations: Running a pipeline in a new environment/container/CI without the env var; copying code that worked locally (where the var was set in .env or shell) to a deployment; notebooks where the env var was set in another kernel.

Related errors

• cache name `{name}` used more than once
• Failed to install dependencies
• Column {api.TIME_PSEUDOCOLUMN} cannot contain negative times
• Column {api.DIFF_PSEUDOCOLUMN} can only have 1 and -1 values
• schema does not match given dataframe

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/8fbc9f450da16a21. Report an issue: GitHub.