{"record":{"id":"1e51bb4672a1b8f4","repo":"pola-rs/polars","slug":"pyarrow-is-required-for-converting-a-pandas-datafr","errorCode":null,"errorMessage":"pyarrow is required for converting a pandas dataframe to Polars, unless each of its columns is a simple numpy-backed one (e.g. 'int64', 'bool', 'float32' - not 'Int64')","messagePattern":"pyarrow is required for converting a pandas dataframe to Polars, unless each of its columns is a simple numpy-backed one \\(e\\.g\\. 'int64', 'bool', 'float32' - not 'Int64'\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/_utils/construction/dataframe.py","lineNumber":1120,"sourceCode":"            return PyDataFrame.empty_with_height(data.shape[0])\n\n        if all(is_simple_numpy_backed_pandas_series(data[col]) for col in data.columns):\n            # Convert via NumPy directly, no PyArrow needed.\n            return pl.DataFrame(\n                {str(col): data[col].to_numpy() for col in data.columns},\n                schema=schema,\n                strict=strict,\n                schema_overrides=schema_overrides,\n                nan_to_null=nan_to_null,\n            )._df\n\n    if not _PYARROW_AVAILABLE:\n        msg = (\n            \"pyarrow is required for converting a pandas dataframe to Polars, \"\n            \"unless each of its columns is a simple numpy-backed one \"\n            \"(e.g. 'int64', 'bool', 'float32' - not 'Int64')\"\n        )\n        raise ImportError(msg)\n    arrow_dict = {}\n    length = data.shape[0]\n\n    if convert_index:\n        for idxcol in data.index.names:\n            arrow_dict[str(idxcol)] = plc.pandas_series_to_arrow(\n                # get_level_values accepts `int | str`\n                # but `index.names` returns `Hashable`\n                data.index.get_level_values(idxcol),  # type: ignore[arg-type, unused-ignore]\n                nan_to_null=nan_to_null,\n                length=length,\n            )\n\n    for col_idx, col_data in data.items():\n        arrow_dict[str(col_idx)] = plc.pandas_series_to_arrow(\n            col_data, nan_to_null=nan_to_null, length=length\n        )\n","sourceCodeStart":1102,"sourceCodeEnd":1138,"githubUrl":"https://github.com/pola-rs/polars/blob/df599052daf96e7a9cc30a3b0c6bd25d6947e3c0/py-polars/src/polars/_utils/construction/dataframe.py#L1102-L1138","documentation":"Converting a pandas DataFrame to Polars goes through pyarrow unless every column is a simple numpy-backed dtype that polars can read directly (int64, bool, float32, ...). Nullable/extension dtypes ('Int64', 'boolean'), 'category', timezone-aware datetimes, and object columns require the arrow path — so when pyarrow is not installed, this ImportError is raised, explicitly noting the numpy-only exception.","triggerScenarios":"`pl.from_pandas(df)` (or DataFrame(df)) without pyarrow installed while df contains pandas nullable dtypes ('Int64', 'boolean', 'string[python]'), 'category', tz-aware datetime64[ns, tz], or object columns; the same call succeeds without pyarrow only if every column is plain numpy-backed.","commonSituations":"Slim production images installing polars without extras; a merge/astype upstream silently changed a column from int64 to nullable Int64, breaking a previously working conversion; pandas code adopting nullable dtypes while the deployment never needed pyarrow before.","solutions":["Install pyarrow: `pip install pyarrow` (or declare the `polars[pandas]` extra)","If pyarrow is truly impossible, cast every non-simple column to a numpy-backed dtype first: `df[\"a\"] = df[\"a\"].astype(\"int64\")` (after fillna), `df[\"cat\"] = df[\"cat\"].astype(str)`, tz-naive via `dt.tz_localize(None)`","Audit dtypes before conversion: `df.dtypes` — anything that is not a plain numpy dtype needs either casting or pyarrow"],"exampleFix":"# before\nimport pandas as pd, polars as pl\npdf = pd.DataFrame({\"a\": pd.array([1, None], dtype=\"Int64\")})\npl.from_pandas(pdf)  # ImportError: pyarrow is required for converting a pandas dataframe to Polars ...\n\n# after (option 1)\n# pip install pyarrow\npl.from_pandas(pdf)\n\n# after (option 2 — cast to simple numpy dtype)\npdf[\"a\"] = pdf[\"a\"].fillna(0).astype(\"int64\")\npl.from_pandas(pdf)","handlingStrategy":"validation","validationCode":"import importlib.util\nimport numpy as np\nimport pandas as pd\n\ndef simple_numpy_backed(df: pd.DataFrame) -> bool:\n    return all(isinstance(df[c].dtype, np.dtype) and df[c].dtype.kind in \"biufcMm\" for c in df.columns)\n\nif not importlib.util.find_spec(\"pyarrow\") and not simple_numpy_backed(df):\n    raise RuntimeError(\"install pyarrow or cast nullable/categorical/object columns to numpy dtypes\")","typeGuard":"import importlib.util\nimport numpy as np\nimport pandas as pd\n\ndef convertible_without_pyarrow(df: pd.DataFrame) -> bool:\n    \"\"\"True if every column is a plain numpy-backed dtype polars can read directly.\"\"\"\n    return all(isinstance(df[c].dtype, np.dtype) for c in df.columns) and not any(\n        isinstance(df[c].dtype, pd.CategoricalDtype) for c in df.columns\n    )\n\ndef pyarrow_available() -> bool:\n    return importlib.util.find_spec(\"pyarrow\") is not None","tryCatchPattern":"try:\n    pldf = pl.from_pandas(df)\nexcept ImportError as e:\n    if \"pyarrow\" not in str(e):\n        raise\n    df = df.convert_dtypes(dtype_backend=\"numpy\") if hasattr(df, \"convert_dtypes\") else df\n    pldf = pl.from_pandas(df.astype({c: \"int64\" for c in df.select_dtypes(\"Int64\")}))","preventionTips":["Install pyarrow (or the polars[pandas] extra) wherever pandas interop is used","Audit df.dtypes before conversion; nullable Int64/boolean, category, and tz-aware dtypes all need pyarrow","Normalize nullable/extension dtypes to numpy dtypes at ingestion time in pyarrow-free environments"],"tags":["pandas-interop","pyarrow","dependencies","dtypes"],"backgroundTag":null,"analyzedSha":"df599052daf96e7a9cc30a3b0c6bd25d6947e3c0","analyzedAt":"2026-08-16T12:10:03.978Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}