pola-rs/polars · error · ImportError

`fsspec` is required for `storage_options` argument

Error message

`fsspec` is required for `storage_options` argument

What it means

Eager readers (read_csv, read_parquet, ...) route source preparation through process_file, which opens remote paths with fsspec whenever `storage_options` is supplied (py-polars/src/polars/io/_utils.py:166-169). If the dict is non-empty and fsspec is not importable, an ImportError is raised before anything is read. The lazy scan_* APIs use polars' native Rust cloud stack and do not go through this check.

Source

Thrown at py-polars/src/polars/io/_utils.py:169

    An http URL is read into a buffer and returned as a `BytesIO`.

    When `encoding` is not `utf8` or `utf8-lossy`, the whole file is
    first read in Python and decoded using the specified encoding and
    returned as a `BytesIO` (for usage with `read_csv`). If encoding
    ends with "-lossy", characters that can't be decoded are replaced
    with `�`.

    A `bytes` file is returned as a `BytesIO` if `use_pyarrow=True`.

    When fsspec is installed, remote file(s) is (are) opened with
    `fsspec.open(file, **kwargs)` or `fsspec.open_files(file, **kwargs)`.
    If encoding is not `utf8` or `utf8-lossy`, decoding is handled by
    fsspec too.
    """
    storage_options = storage_options.copy() if storage_options else {}
    if storage_options and not _FSSPEC_AVAILABLE:
        msg = "`fsspec` is required for `storage_options` argument"
        raise ImportError(msg)

    # Small helper to use a variable as context
    @contextmanager
    def managed_file(file: Any) -> Iterator[Any]:
        try:
            yield file
        finally:
            pass

    has_utf8_utf8_lossy_encoding = (
        encoding in {"utf8", "utf8-lossy"} if encoding else True
    )
    encoding_str = encoding if encoding else "utf8"
    encoding_str, encoding_errors = (
        (encoding_str[:-6], "replace")
        if encoding_str.endswith("-lossy")
        else (encoding_str, "strict")
    )

View on GitHub (pinned to df599052da)

Solutions

1. Install fsspec plus the protocol extra: pip install fsspec s3fs (for s3://) or gcsfs (for gs://)
2. Switch to the lazy API pl.scan_csv(...)/pl.scan_parquet(...), whose native credential handling needs no fsspec
3. Drop storage_options if environment defaults (env vars, instance profiles) already provide access

Example fix

# before (ImportError: `fsspec` is required)
df = pl.read_csv("s3://bucket/f.csv", storage_options={"anon": "true"})
# after (native cloud path, no fsspec needed)
df = pl.scan_csv("s3://bucket/f.csv").collect()

import importlib.util

def can_use_storage_options() -> bool:
    return importlib.util.find_spec("fsspec") is not None

if opts and not can_use_storage_options():
    raise SystemExit("storage_options requires fsspec: pip install fsspec s3fs")

try:
    df = pl.read_csv(url, storage_options=opts)
except ImportError as e:
    if "fsspec" in str(e):
        raise SystemExit("pip install fsspec (plus s3fs/gcsfs for your scheme)") from e
    raise

When it happens

Trigger: pl.read_csv('s3://bucket/f.csv', storage_options={'anon': 'true'}) with fsspec not installed; passing storage_options even for a local file triggers it, because the check runs on any non-empty dict.

Common situations: Slim Docker/CI images that installed polars without the fssesspec/s3fs/gcsfs extras; code migrated from scan_csv (native cloud support) to read_csv; adding storage_options to an existing job on an environment that never had fsspec.

Related errors

• boto3 must be installed to use `CredentialProviderAWS`
• azure-identity must be installed to use `CredentialProviderA
• google-auth must be installed to use `CredentialProviderGCP`
• expected a file path; {path!r} is a directory
• the graphviz `dot` binary should be on your PATH.(If not ins

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/a756e1abe66a419d. Report an issue: GitHub.