pola-rs/polars · error · ValueError

distributed options {sorted(kwargs)!r} are not supported wit

Error message

distributed options {sorted(kwargs)!r} are not supported with `scaling_mode='single-node'`

What it means

`RemoteEngine.__init__` (engine_remote.py:129) rejects any extra keyword arguments when `scaling_mode='single-node'`, because those kwargs (`max_workers`, `min_workers`, `shuffle_format`, `partitions_per_worker`, ...) are forwarded to the distributed planner and are meaningless for single-node runs. The message lists the offending option names sorted, and the check runs at construction time ('fail here rather than deep inside a sink').

Source

Thrown at py-polars/src/polars/lazyframe/engine_remote.py:134

        scaling_mode: ScalingMode = "auto",
        engine: EngineTypeName = "auto",
        plan_type: PlanTypePreference = "dot",
        n_retries: int = 0,
        labels: list[str] | str | None = None,
        **kwargs: Any,
    ) -> None:
        if scaling_mode not in _SCALING_MODES:
            msg = f"invalid `scaling_mode` {scaling_mode!r}"
            raise ValueError(msg)
        if engine not in _WORKER_ENGINE_NAMES:
            msg = f"Invalid engine argument {engine=}"
            raise ValueError(msg)
        if scaling_mode == "single-node" and kwargs:
            msg = (
                f"distributed options {sorted(kwargs)!r} are not supported with "
                "`scaling_mode='single-node'`"
            )
            raise ValueError(msg)

        # fail here rather than deep inside a sink
        import_optional(
            "polars_cloud",
            err_prefix="remote engine requested, but required package",
            install_message="Please install using the command `pip install polars-cloud`",
        )

        self.context = context
        self.scaling_mode = scaling_mode
        self.engine = engine
        self.plan_type = plan_type
        self.n_retries = n_retries
        self.labels = [labels] if isinstance(labels, str) else labels
        self.config = kwargs

    @property
    def name(self) -> str:

View on GitHub (pinned to df599052da)

Solutions

1. Remove the distributed options when using `scaling_mode='single-node'`
2. Or switch to `scaling_mode='distributed'` (or `'auto'`) if the options should apply
3. Split engine construction into two presets (single-node vs distributed) instead of one parameterized call

Example fix

# before
engine = pl.RemoteEngine(ctx, scaling_mode='single-node', max_workers=8)  # ValueError

# after
engine = pl.RemoteEngine(ctx, scaling_mode='distributed', max_workers=8)
# or, for one node:
engine = pl.RemoteEngine(ctx, scaling_mode='single-node')

DISTRIBUTED_OPTIONS = {'max_workers', 'min_workers', 'shuffle_format', 'partitions_per_worker'}

def make_remote_engine(ctx, scaling_mode='auto', **opts):
    if scaling_mode == 'single-node' and opts:
        raise ValueError(
            f'options {sorted(opts)} require scaling_mode=\'distributed\', not \'single-node\''
        )
    return pl.RemoteEngine(ctx, scaling_mode=scaling_mode, **opts)

try:
    engine = pl.RemoteEngine(ctx, scaling_mode=mode, **cluster_opts)
except ValueError as e:
    if 'single-node' in str(e):
        engine = pl.RemoteEngine(ctx, scaling_mode='distributed', **cluster_opts)
    else:
        raise

When it happens

Trigger: `pl.RemoteEngine(ctx, scaling_mode='single-node', max_workers=8)`, or any other distributed-planner kwarg combined with `scaling_mode='single-node'`. The kwargs are collected via `**kwargs`, so any unrecognized keyword silently becomes a distributed option and trips this check.

Common situations: A shared engine-construction helper that always passes cluster sizing options, called once with 'single-node' for a small job; typos in option names becoming unintended `**kwargs`; flipping scaling mode in config while leaving sizing options in place.

Related errors

• invalid `scaling_mode` {scaling_mode!r}
• Invalid engine argument {engine=}
• `{name}` is not supported by the remote engine
• DataFrame `how` must be one of {{{allowed}}}, got {how!r}
• Invalid engine argument {engine=}

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/d402609a8876ef9c. Report an issue: GitHub.