What causes "python function failed" in pola-rs/polars?

Fix the lambda so it does not raise: operate per-name, e.g. lambda names: [n.lower() for n in names].

python function failed

Error message

python function failed

What it means

When reading CSV with schema_modify (rename_columns via with_schema_modify), polars calls the user's Python lambda with the list of column names and expects a valid return. `expect("python function failed")` panics if calling the lambda itself raised a Python exception, because the Rust side cannot gracefully convert a failure at this point into an error — it aborts with this message plus the original Python traceback context.

Solutions

Fix the lambda so it does not raise: operate per-name, e.g. lambda names: [n.lower() for n in names].
Test the lambda standalone on a plain list of column names before using it in read/scan.
Wrap risky logic inside the lambda in try/except and return the original names on failure.
Upgrade polars if you need this failure surfaced as a Python exception instead of a panic.

Example fix

# before
pl.scan_csv(path, with_schema_modify=lambda names: names.lower())  # AttributeError -> panic
# after
pl.scan_csv(path, with_schema_modify=lambda names: [n.lower() for n in names])

Defensive patterns

Strategy: try-catch

Validate before calling

fn = lambda names: [n.lower() for n in names]  # test first:
assert isinstance(fn(['A', 'B']), list) and all(isinstance(x, str) for x in fn(['A', 'B']))

Type guard

def safe_renamer(fn):
    def wrapped(names):
        try:
            return list(fn(names))
        except Exception:
            return names
    return wrapped

Try / catch

try:
    lf = pl.scan_csv(path, with_schema_modify=lambda names: [n.lower() for n in names])
    df = lf.collect()
except BaseException as e:
    if 'python function failed' in str(e):
        df = pl.read_csv(path)  # fallback: no renaming
    else:
        raise

Prevention

Unit-test rename lambdas on a plain list before use.
Keep the lambda pure and exception-free.
Return a list of str, one per input name.

When it happens

Trigger: pl.scan_csv(..., with_schema_modify=lambda names: ...) where the lambda raises any exception (NameError, TypeError, index errors, calling unsupported APIs) when invoked with the list of original column names.

Common situations: Lambdas referencing variables not in scope; calling string methods on the list instead of per-name; using functions like `names.lower()` (list has no such method); typos inside the renaming function.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of pola-rs/polars@fe841f959e (2026-09-18). Data as JSON: /api/errors/06dbde92fa730173. Report an issue: GitHub.

Appendix: source

Thrown at crates/polars-python/src/lazyframe/general.rs:288

            .with_truncate_ragged_lines(truncate_ragged_lines)
            .with_decimal_comma(decimal_comma)
            .with_glob(glob)
            .with_raise_if_empty(raise_if_empty)
            .with_include_file_paths(include_file_paths.map(|x| x.into()))
            .with_extra_columns_policy(extra_columns.0)
            .with_missing_columns_policy(missing_columns.map(|x| x.0));

        if let Some(new_columns) = new_columns {
            r = r.with_column_names_overwrite(new_columns.0);
        }

        if let Some(lambda) = with_schema_modify {
            let f = |schema: Schema| {
                let iter = schema.iter_names().map(|s| s.as_str());
                Python::attach(|py| {
                    let names = PyList::new(py, iter).unwrap();

                    let out = lambda.call1(py, (names,)).expect("python function failed");
                    let new_names = out
                        .extract::<Vec<String>>(py)
                        .expect("python function should return List[str]");
                    polars_ensure!(new_names.len() == schema.len(),
                        ShapeMismatch: "The length of the new names list should be equal to or less than the original column length",
                    );
                    Ok(schema
                        .iter_values()
                        .zip(new_names)
                        .map(|(dtype, name)| Field::new(name.into(), dtype.clone()))
                        .collect())
                })
            };
            r = r.with_schema_modify(f).map_err(PyPolarsErr::from)?
        }

        Ok(r.finish().map_err(PyPolarsErr::from)?.into())
    }

View on GitHub (pinned to fe841f959e)