What causes "unsupported encoding {encoding} for hf:// paths" in pola-rs/polars?

Use encoding='utf8' (default) or encoding='utf8-lossy' for hf:// paths; utf8-lossy tolerates invalid UTF-8 bytes

pola-rs/polars · error · ValueError

unsupported encoding {encoding} for hf:// paths

Error message

unsupported encoding {encoding} for hf:// paths

What it means

polars' read_csv routes hf:// (HuggingFace hub) paths to the lazy scan_csv engine, and that engine only understands 'utf8' and 'utf8-lossy' (encoding_supported_in_lazy at functions.py:497). If you pass any other encoding for an hf:// source, the eager read is rejected up front with this ValueError. The same gate applies when POLARS_FORCE_ASYNC=1 forces the lazy path for local files.

Source

Thrown at py-polars/src/polars/io/csv/functions.py:533

            # * The `storage_options` configuration keys are different between
            #   fsspec and object_store (would require a breaking change)
        )
    ):
        source_normalized: str | list[str] | IO[str] | IO[bytes] | bytes | bytearray
        if isinstance(source, (str, Path)):
            source_normalized = normalize_filepath(source, check_not_directory=False)
        elif is_path_or_str_sequence(source, allow_str=False):
            source_normalized = [
                normalize_filepath(source, check_not_directory=False)
                for source in source
            ]
        else:
            source_normalized = source

        if not streaming:
            if not encoding_supported_in_lazy:
                msg = f"unsupported encoding {encoding} for hf:// paths"
                raise ValueError(msg)

        lf = _scan_csv_impl(
            source_normalized,
            has_header=has_header,
            separator=separator,
            comment_prefix=comment_prefix,
            quote_char=quote_char,
            skip_rows=skip_rows,
            skip_lines=skip_lines,
            schema_overrides=schema_overrides,  # type: ignore[arg-type]
            schema=schema,
            null_values=null_values,
            empty_string_is_null=empty_string_is_null,
            ignore_errors=ignore_errors,
            try_parse_dates=try_parse_dates,
            infer_schema_length=infer_schema_length,
            n_rows=n_rows,
            encoding=encoding,  # type: ignore[arg-type]

View on GitHub (pinned to df599052da)

Solutions

Use encoding='utf8' (default) or encoding='utf8-lossy' for hf:// paths; utf8-lossy tolerates invalid UTF-8 bytes
Download the file first (e.g. huggingface_hub.hf_hub_download) and read the local copy with the non-UTF-8 encoding
Re-encode the dataset on the hub to UTF-8 so consumers can use the default

Example fix

# before
pl.read_csv('hf://datasets/acme/data/train.csv', encoding='latin1')

# after
pl.read_csv('hf://datasets/acme/data/train.csv', encoding='utf8-lossy')

# or: download and read locally with any encoding
from huggingface_hub import hf_hub_download
path = hf_hub_download('acme/data', 'train.csv', repo_type='dataset')
pl.read_csv(path, encoding='latin1')

Defensive patterns

Strategy: validation

Validate before calling

LAZY_ENCODINGS = {'utf8', 'utf8-lossy'}
source_str = str(source) if isinstance(source, (str, Path)) else ''
if source_str.startswith('hf://') and encoding not in LAZY_ENCODINGS:
    raise ValueError(
        f'encoding {encoding!r} unsupported for hf:// paths; '
        'use utf8/utf8-lossy or download the file locally'
    )
df = pl.read_csv(source, encoding=encoding)

Try / catch

try:
    df = pl.read_csv(src, encoding=enc)
except ValueError as err:
    if 'unsupported encoding' in str(err) and 'hf://' in str(src):
        df = pl.read_csv(download_locally(src))  # fallback path
    else:
        raise

Prevention

Default to utf8-lossy for hub datasets of unknown provenance - it never raises this error and only substitutes bad bytes
Keep a single LAZY_ENCODINGS = {'utf8', 'utf8-lossy'} constant wherever encodings are configurable
In CI, avoid setting POLARS_FORCE_ASYNC globally unless tests cover the encoding gate

When it happens

Trigger: pl.read_csv('hf://datasets/<org>/<ds>/.../file.csv', encoding='latin1') (any encoding not in {'utf8','utf8-lossy'}); POLARS_FORCE_ASYNC=1 with a non-utf8 encoding and a str/Path source; BytesIO sources are exempt because only str/Path are dispatched.

Common situations: Reading legacy Latin-1/Windows-1252 CSVs hosted on the HuggingFace hub; notebooks copied from local-file workflows that passed encoding='iso-8859-1'; CI environments where POLARS_FORCE_ASYNC=1 is set globally for async testing.

Related errors

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/41a5c6a659986b56. Report an issue: GitHub.