{"record":{"id":"41a5c6a659986b56","repo":"pola-rs/polars","slug":"unsupported-encoding-encoding-for-hf-paths","errorCode":null,"errorMessage":"unsupported encoding {encoding} for hf:// paths","messagePattern":"unsupported encoding (.+?) for hf:// paths","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/io/csv/functions.py","lineNumber":533,"sourceCode":"            # * The `storage_options` configuration keys are different between\n            #   fsspec and object_store (would require a breaking change)\n        )\n    ):\n        source_normalized: str | list[str] | IO[str] | IO[bytes] | bytes | bytearray\n        if isinstance(source, (str, Path)):\n            source_normalized = normalize_filepath(source, check_not_directory=False)\n        elif is_path_or_str_sequence(source, allow_str=False):\n            source_normalized = [\n                normalize_filepath(source, check_not_directory=False)\n                for source in source\n            ]\n        else:\n            source_normalized = source\n\n        if not streaming:\n            if not encoding_supported_in_lazy:\n                msg = f\"unsupported encoding {encoding} for hf:// paths\"\n                raise ValueError(msg)\n\n        lf = _scan_csv_impl(\n            source_normalized,\n            has_header=has_header,\n            separator=separator,\n            comment_prefix=comment_prefix,\n            quote_char=quote_char,\n            skip_rows=skip_rows,\n            skip_lines=skip_lines,\n            schema_overrides=schema_overrides,  # type: ignore[arg-type]\n            schema=schema,\n            null_values=null_values,\n            empty_string_is_null=empty_string_is_null,\n            ignore_errors=ignore_errors,\n            try_parse_dates=try_parse_dates,\n            infer_schema_length=infer_schema_length,\n            n_rows=n_rows,\n            encoding=encoding,  # type: ignore[arg-type]","sourceCodeStart":515,"sourceCodeEnd":551,"githubUrl":"https://github.com/pola-rs/polars/blob/df599052daf96e7a9cc30a3b0c6bd25d6947e3c0/py-polars/src/polars/io/csv/functions.py#L515-L551","documentation":"polars' read_csv routes hf:// (HuggingFace hub) paths to the lazy scan_csv engine, and that engine only understands 'utf8' and 'utf8-lossy' (encoding_supported_in_lazy at functions.py:497). If you pass any other encoding for an hf:// source, the eager read is rejected up front with this ValueError. The same gate applies when POLARS_FORCE_ASYNC=1 forces the lazy path for local files.","triggerScenarios":"pl.read_csv('hf://datasets/<org>/<ds>/.../file.csv', encoding='latin1') (any encoding not in {'utf8','utf8-lossy'}); POLARS_FORCE_ASYNC=1 with a non-utf8 encoding and a str/Path source; BytesIO sources are exempt because only str/Path are dispatched.","commonSituations":"Reading legacy Latin-1/Windows-1252 CSVs hosted on the HuggingFace hub; notebooks copied from local-file workflows that passed encoding='iso-8859-1'; CI environments where POLARS_FORCE_ASYNC=1 is set globally for async testing.","solutions":["Use encoding='utf8' (default) or encoding='utf8-lossy' for hf:// paths; utf8-lossy tolerates invalid UTF-8 bytes","Download the file first (e.g. huggingface_hub.hf_hub_download) and read the local copy with the non-UTF-8 encoding","Re-encode the dataset on the hub to UTF-8 so consumers can use the default"],"exampleFix":"# before\npl.read_csv('hf://datasets/acme/data/train.csv', encoding='latin1')\n\n# after\npl.read_csv('hf://datasets/acme/data/train.csv', encoding='utf8-lossy')\n\n# or: download and read locally with any encoding\nfrom huggingface_hub import hf_hub_download\npath = hf_hub_download('acme/data', 'train.csv', repo_type='dataset')\npl.read_csv(path, encoding='latin1')","handlingStrategy":"validation","validationCode":"LAZY_ENCODINGS = {'utf8', 'utf8-lossy'}\nsource_str = str(source) if isinstance(source, (str, Path)) else ''\nif source_str.startswith('hf://') and encoding not in LAZY_ENCODINGS:\n    raise ValueError(\n        f'encoding {encoding!r} unsupported for hf:// paths; '\n        'use utf8/utf8-lossy or download the file locally'\n    )\ndf = pl.read_csv(source, encoding=encoding)","typeGuard":null,"tryCatchPattern":"try:\n    df = pl.read_csv(src, encoding=enc)\nexcept ValueError as err:\n    if 'unsupported encoding' in str(err) and 'hf://' in str(src):\n        df = pl.read_csv(download_locally(src))  # fallback path\n    else:\n        raise","preventionTips":["Default to utf8-lossy for hub datasets of unknown provenance - it never raises this error and only substitutes bad bytes","Keep a single LAZY_ENCODINGS = {'utf8', 'utf8-lossy'} constant wherever encodings are configurable","In CI, avoid setting POLARS_FORCE_ASYNC globally unless tests cover the encoding gate"],"tags":["polars","csv","huggingface","encoding","valueerror"],"backgroundTag":null,"analyzedSha":"df599052daf96e7a9cc30a3b0c6bd25d6947e3c0","analyzedAt":"2026-08-16T12:10:03.978Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}