pola-rs/polars · error · ValueError

unsupported: cannot combine aws_profile with {unhandled_key}

Error message

unsupported: cannot combine aws_profile with {unhandled_key} in storage_options

What it means

When a cloud path is scanned and credentials resolve via the 'auto' builder, the AWS branch (py-polars/src/polars/io/cloud/credential_provider/_builder.py:447-484) inspects storage_options: aws_region/region, aws_default_region/default_region, aws_profile/profile, and endpoint keys are recognized, an internal allow-list is ignored, and every OTHER key is treated as a raw access-key credential. Supplying a profile together with any such unrecognized key is contradictory (profile-based vs key-based auth) and raises ValueError naming the conflicting key.

Source

Thrown at py-polars/src/polars/io/cloud/credential_provider/_builder.py:484

                        "aws_endpoint",
                        "aws_endpoint_url",
                        "endpoint",
                        "endpoint_url",
                    }:
                        has_endpoint_url = True
                    elif k in AUTOINIT_IGNORED_KEYS:
                        continue
                    else:
                        # We assume this is some sort of access key
                        unhandled_key = k

            if unhandled_key is not None:
                if profile is not None:
                    msg = (
                        "unsupported: cannot combine aws_profile with "
                        f"{unhandled_key} in storage_options"
                    )
                    raise ValueError(msg)

            if (
                unhandled_key is None
                and (default := get_default_credential_provider()) is not None
            ):
                return default

            return CredentialProviderBuilder(
                AutoInit(
                    CredentialProviderAWS,
                    profile_name=profile,
                    region_name=region or default_region,
                    _auto_init_unhandled_key=unhandled_key,
                    _storage_options_has_endpoint_url=has_endpoint_url,
                )
            )

        elif _is_gcp_cloud(scheme):

View on GitHub (pinned to df599052da)

Solutions

1. Pick one auth mode: keep aws_profile (plus optional region/endpoint) and delete access-key entries, or drop the profile and keep the keys
2. Move credentials to the standard chain (env vars, ~/.aws/credentials, instance role) and pass no credential keys at all
3. For explicit control pass credential_provider=pl.CredentialProviderAWS(...) or a custom callable

Example fix

# before
lf = pl.scan_parquet(
    "s3://bucket/f.parquet",
    storage_options={"aws_profile": "prod", "aws_access_key_id": "AKIA...", "aws_secret_access_key": "..."},
)
# after
lf = pl.scan_parquet(
    "s3://bucket/f.parquet",
    storage_options={"aws_profile": "prod", "region": "eu-west-1"},
)

AWS_PROFILE_COMPATIBLE = {
    "aws_region", "region", "aws_default_region", "default_region",
    "aws_profile", "profile",
    "aws_endpoint", "aws_endpoint_url", "endpoint", "endpoint_url",
}

def validate_s3_storage_options(opts: dict) -> None:
    has_profile = any(k.lower() in {"aws_profile", "profile"} for k in opts)
    unknown = [k for k in opts if k.lower() not in AWS_PROFILE_COMPATIBLE]
    if has_profile and unknown:
        raise ValueError(f"cannot combine aws_profile with {unknown}; pick one auth mode")

try:
    lf = pl.scan_parquet(path, storage_options=opts)
except ValueError as e:
    if "cannot combine aws_profile" in str(e):
        opts = {k: v for k, v in opts.items() if k.lower() not in {"aws_profile", "profile"}}
        lf = pl.scan_parquet(path, storage_options=opts)
    else:
        raise

When it happens

Trigger: pl.scan_parquet('s3://bucket/f.parquet', storage_options={'aws_profile': 'prod', 'aws_access_key_id': 'AKIA...'}) - profile plus any non-recognized key raises; region/endpoint keys alongside a profile are fine.

Common situations: Templates that merge a shared credentials dict with a per-environment profile; copy-pasted storage_options from other tools; CI that injects key-based credentials while the config also sets a profile.

Related errors

• unsupported: cannot combine token with {unhandled_key} in st
• boto3 must be installed to use `CredentialProviderAWS`
• the provided `credential` object {credential!r} does not hav
• index positions should be smaller than 2^32
• index positions should be greater than or equal to -2^32

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/6d4db93f18ab2dfe. Report an issue: GitHub.