pola-rs/polars · error · ValueError

cannot use credential_provider when passing a DeltaTable obj

Error message

cannot use credential_provider when passing a DeltaTable object

What it means

When sink_delta()'s target is an already-constructed DeltaTable object, credentials are managed by that object itself, so polars refuses to also apply a credential_provider. Passing any credential_provider other than None or the sentinel 'auto' alongside a DeltaTable target raises ValueError.

Source

Thrown at py-polars/src/polars/lazyframe/frame.py:3307

        _check_for_unsupported_types(self.collect_schema().dtypes())

        if isinstance(target, (str, Path)):
            target = _resolve_delta_lake_uri(str(target), strict=False)

        from polars.io.cloud.credential_provider._builder import (
            _init_credential_provider_builder,
        )
        from polars.io.cloud.credential_provider._providers import (
            _get_credentials_from_provider_expiry_aware,
        )

        if not isinstance(target, DeltaTable):
            credential_provider_builder = _init_credential_provider_builder(
                credential_provider, target, storage_options, "sink_delta"
            )
        elif credential_provider is not None and credential_provider != "auto":
            msg = "cannot use credential_provider when passing a DeltaTable object"
            raise ValueError(msg)
        else:
            credential_provider_builder = None

        del credential_provider

        credential_provider_creds = {}

        if credential_provider_builder and (
            provider := credential_provider_builder.build_credential_provider()
        ):
            credential_provider_creds = (
                _get_credentials_from_provider_expiry_aware(provider) or {}
            )

        # We aren't calling into polars-native write functions so we just update
        # the storage_options here.
        storage_options = (
            {**(storage_options or {}), **credential_provider_creds}

View on GitHub (pinned to df599052da)

Solutions

1. Drop the credential_provider argument when target is a DeltaTable
2. Instead pass storage_options/credential config to DeltaTable() when constructing it
3. Or switch to a string URI target: lf.sink_delta('s3://bucket/path', credential_provider=...) so polars manages credentials
4. Keep credential_provider='auto' or None if you must pass something alongside a DeltaTable

Example fix

# before
dt = DeltaTable('s3://bucket/path')
lf.sink_delta(dt, credential_provider=credential_provider_obj)

# after
dt = DeltaTable('s3://bucket/path', storage_options=storage_options)
lf.sink_delta(dt)

from deltalake import DeltaTable
if isinstance(target, DeltaTable):
    assert credential_provider in (None, 'auto'), 'DeltaTable target manages its own credentials'
    lf.sink_delta(target)
else:
    lf.sink_delta(target, credential_provider=credential_provider)

def is_delta_table(target) -> bool:
    from deltalake import DeltaTable
    return isinstance(target, DeltaTable)

When it happens

Trigger: dt = DeltaTable('s3://bucket/path', storage_options=...); lf.sink_delta(dt, credential_provider=CredentialProvider...) — mixing object-target with provider-based auth.

Common situations: Cloud storage (S3/Azure/GCS) auth refactorings; code that generically passes credential_provider to all sink calls regardless of target type.

Related errors

• you need to pass delta_merge_options with at least a given p
• did not receive credentials from temporary credentials API f
• cannot {operation}: no storage_location found
• cannot {operation}: no data_source_format found
• unsupported: cannot combine aws_profile with {unhandled_key}

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/0b9ba65460bfc4cc. Report an issue: GitHub.