pola-rs/polars · error

python function failed

Error message

python function failed

What it means

For lazy scans accepting a with_schema_modify callable, polars invokes the Python lambda with the list of column names and expects it to return the renamed list. If the callable raises any Python exception, call1 returns Err and this expect panics with 'python function failed' (the raised exception is the cause).

Source

Thrown at crates/polars-python/src/lazyframe/general.rs:289

            .with_missing_is_null(empty_string_is_null)
            .with_truncate_ragged_lines(truncate_ragged_lines)
            .with_decimal_comma(decimal_comma)
            .with_glob(glob)
            .with_raise_if_empty(raise_if_empty)
            .with_include_file_paths(include_file_paths.map(|x| x.into()))
            .with_missing_columns_policy(missing_columns.map(|x| x.0));

        if let Some(new_columns) = new_columns {
            r = r.with_column_names_overwrite(new_columns.0);
        }

        if let Some(lambda) = with_schema_modify {
            let f = |schema: Schema| {
                let iter = schema.iter_names().map(|s| s.as_str());
                Python::attach(|py| {
                    let names = PyList::new(py, iter).unwrap();

                    let out = lambda.call1(py, (names,)).expect("python function failed");
                    let new_names = out
                        .extract::<Vec<String>>(py)
                        .expect("python function should return List[str]");
                    polars_ensure!(new_names.len() == schema.len(),
                        ShapeMismatch: "The length of the new names list should be equal to or less than the original column length",
                    );
                    Ok(schema
                        .iter_values()
                        .zip(new_names)
                        .map(|(dtype, name)| Field::new(name.into(), dtype.clone()))
                        .collect())
                })
            };
            r = r.with_schema_modify(f).map_err(PyPolarsErr::from)?
        }

        Ok(r.finish().map_err(PyPolarsErr::from)?.into())
    }

View on GitHub (pinned to df599052da)

Solutions

1. Make the lambda total: never raise for any input list (use .get with a default or fall back to the original name)
2. Log unexpected inputs inside the callback instead of letting exceptions escape
3. Keep the callback pure and simple — mapping only, no IO or parsing
4. Test the callback directly with the current column names before wiring it into the scan

Example fix

# before
rename = lambda names: [prefix + name_map[name] for name in names]  # KeyError escapes -> panic

# after
rename = lambda names: [f"{prefix}{name_map.get(name, name)}" for name in names]

# unit-test the callback against the live schema before wiring it in
names = [f.name for f in lf.collect_schema()]
out = rename(names)
assert isinstance(out, list) and len(out) == len(names) and all(isinstance(n, str) for n in out)

from typing import Callable, List

def is_valid_schema_modifier(f: Callable[[List[str]], List[str]], names: List[str]) -> bool:
    try:
        out = f(list(names))
        return isinstance(out, list) and all(isinstance(n, str) for n in out)
    except Exception:
        return False

When it happens

Trigger: A with_schema_modify lambda that throws — e.g. it calls a dict lookup that KeyErrors on an unexpected name, uses an API that changed, or raises on duplicate columns — during scan construction/first schema resolution.

Common situations: Rename lambdas assuming specific column names that no longer exist after upstream schema changes; exceptions inside helper functions; typed/numpy code that fails on plain lists of str.

Related errors

• python function should return List[str]
• no read method found
• should be hashable
• negative stop is not supported for lazy slices
• cannot describe a LazyFrame that has no columns

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/06dbde92fa730173. Report an issue: GitHub.