pola-rs/polars · error · ValueError

Invalid engine argument {engine=}

Error message

Invalid engine argument {engine=}

What it means

`_engine_from_name` (engine_config.py:74) resolves an engine name string to an `Engine` instance and raises ValueError for any name not in the registry. Accepted names are `'auto'`, `'in-memory'`, `'streaming'`, `'gpu'`, plus the legacy alias `'cpu'` (mapped to in-memory). Note that `'remote'` is not a name — the remote engine must be passed as an object, e.g. `pl.RemoteEngine()`. The check applies wherever `engine=` accepts a string (`collect`, `execute`, `sink_*`, scan functions, `pl.collect_all`).

Source

Thrown at py-polars/src/polars/lazyframe/engine_config.py:76

def set_engine_affinity_override(engine: Engine | None) -> None:
    """Set the configured engine override."""
    global _ENGINE_AFFINITY_OVERRIDE
    _ENGINE_AFFINITY_OVERRIDE = engine


def _eager_engine() -> Engine:
    """Return the engine used for internal eager operations."""
    return _IN_MEMORY_ENGINE


def _engine_from_name(engine: EngineTypeName) -> Engine:
    """Resolve an explicit engine name without applying the configured affinity."""
    if engine == "gpu":
        return GPUEngine()

    if (selected := _ENGINE_BY_NAME.get(engine)) is None:
        msg = f"Invalid engine argument {engine=}"
        raise ValueError(msg)
    return selected


def _select_engine(engine: EngineType) -> Engine:
    """
    Resolve an engine argument or configured affinity to an `Engine`.

    An `"auto"` affinity remains unresolved for Rust to select at execution time.
    """
    if isinstance(engine, Engine):
        return engine

    if engine == "auto":
        if _ENGINE_AFFINITY_OVERRIDE is not None:
            return _ENGINE_AFFINITY_OVERRIDE
        engine = get_engine_affinity()

    return _engine_from_name(engine)

View on GitHub (pinned to df599052da)

Solutions

1. Fix the name to one of `'auto' | 'in-memory' | 'streaming' | 'gpu'` (legacy `'cpu'` still accepted)
2. For remote execution pass an instance: `engine=pl.RemoteEngine(...)` — never the string `'remote'`
3. If the value comes from config, validate it against `polars.lazyframe.engine_config.SUPPORTED_ENGINE_NAMES` before use
4. Check for leading/trailing whitespace or underscores in dynamically built names

Example fix

# before
lf.collect(engine='remote')  # ValueError: Invalid engine argument

# after
lf.collect(engine=pl.RemoteEngine())
# or a valid name:
lf.collect(engine='streaming')

from polars.lazyframe.engine_config import SUPPORTED_ENGINE_NAMES

VALID_ENGINE_NAMES = set(SUPPORTED_ENGINE_NAMES) | {'cpu'}  # 'cpu' is a legacy alias

def resolve_engine_name(name: str) -> str:
    if name not in VALID_ENGINE_NAMES:
        raise ValueError(
            f'unknown engine {name!r}; expected one of {sorted(VALID_ENGINE_NAMES)}'
        )
    return name

lf.collect(engine=resolve_engine_name(cfg['engine']))

from typing import TypeGuard
from polars.lazyframe.engine_config import SUPPORTED_ENGINE_NAMES

_VALID = set(SUPPORTED_ENGINE_NAMES) | {'cpu'}

def is_engine_name(name: object) -> TypeGuard[str]:
    return isinstance(name, str) and name in _VALID

try:
    df = lf.collect(engine=engine_name)
except ValueError as e:
    if 'Invalid engine argument' in str(e):
        df = lf.collect(engine='streaming')  # safe default
    else:
        raise

When it happens

Trigger: `lf.collect(engine='remote')`, `engine='in_memory'`/`'streaming-engine'` (typos, underscores), `engine='cpu-force'`, or any string not in `{'auto','in-memory','streaming','gpu','cpu'}`. Also raised when a name read from environment/config is passed through unvalidated.

Common situations: Configuration-driven engine selection (env var, YAML) where a bad or stale value flows into `engine=`; renaming churn across Polars versions (older releases used different engine vocabularies); attempting to select Polars Cloud by string instead of an engine object.

Related errors

• Invalid engine argument {engine=}
• DataFrame `how` must be one of {{{allowed}}}, got {how!r}
• invalid `scaling_mode` {scaling_mode!r}
• distributed options {sorted(kwargs)!r} are not supported wit
• index positions should be smaller than 2^32

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/c56bcb2babb087ca. Report an issue: GitHub.