{"record":{"id":"06dbde92fa730173","repo":"pola-rs/polars","slug":"python-function-failed","errorCode":null,"errorMessage":"python function failed","messagePattern":"python function failed","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/polars-python/src/lazyframe/general.rs","lineNumber":289,"sourceCode":"            .with_missing_is_null(empty_string_is_null)\n            .with_truncate_ragged_lines(truncate_ragged_lines)\n            .with_decimal_comma(decimal_comma)\n            .with_glob(glob)\n            .with_raise_if_empty(raise_if_empty)\n            .with_include_file_paths(include_file_paths.map(|x| x.into()))\n            .with_missing_columns_policy(missing_columns.map(|x| x.0));\n\n        if let Some(new_columns) = new_columns {\n            r = r.with_column_names_overwrite(new_columns.0);\n        }\n\n        if let Some(lambda) = with_schema_modify {\n            let f = |schema: Schema| {\n                let iter = schema.iter_names().map(|s| s.as_str());\n                Python::attach(|py| {\n                    let names = PyList::new(py, iter).unwrap();\n\n                    let out = lambda.call1(py, (names,)).expect(\"python function failed\");\n                    let new_names = out\n                        .extract::<Vec<String>>(py)\n                        .expect(\"python function should return List[str]\");\n                    polars_ensure!(new_names.len() == schema.len(),\n                        ShapeMismatch: \"The length of the new names list should be equal to or less than the original column length\",\n                    );\n                    Ok(schema\n                        .iter_values()\n                        .zip(new_names)\n                        .map(|(dtype, name)| Field::new(name.into(), dtype.clone()))\n                        .collect())\n                })\n            };\n            r = r.with_schema_modify(f).map_err(PyPolarsErr::from)?\n        }\n\n        Ok(r.finish().map_err(PyPolarsErr::from)?.into())\n    }","sourceCodeStart":271,"sourceCodeEnd":307,"githubUrl":"https://github.com/pola-rs/polars/blob/df599052daf96e7a9cc30a3b0c6bd25d6947e3c0/crates/polars-python/src/lazyframe/general.rs#L271-L307","documentation":"For lazy scans accepting a with_schema_modify callable, polars invokes the Python lambda with the list of column names and expects it to return the renamed list. If the callable raises any Python exception, call1 returns Err and this expect panics with 'python function failed' (the raised exception is the cause).","triggerScenarios":"A with_schema_modify lambda that throws — e.g. it calls a dict lookup that KeyErrors on an unexpected name, uses an API that changed, or raises on duplicate columns — during scan construction/first schema resolution.","commonSituations":"Rename lambdas assuming specific column names that no longer exist after upstream schema changes; exceptions inside helper functions; typed/numpy code that fails on plain lists of str.","solutions":["Make the lambda total: never raise for any input list (use .get with a default or fall back to the original name)","Log unexpected inputs inside the callback instead of letting exceptions escape","Keep the callback pure and simple — mapping only, no IO or parsing","Test the callback directly with the current column names before wiring it into the scan"],"exampleFix":"# before\nrename = lambda names: [prefix + name_map[name] for name in names]  # KeyError escapes -> panic\n\n# after\nrename = lambda names: [f\"{prefix}{name_map.get(name, name)}\" for name in names]","handlingStrategy":"validation","validationCode":"# unit-test the callback against the live schema before wiring it in\nnames = [f.name for f in lf.collect_schema()]\nout = rename(names)\nassert isinstance(out, list) and len(out) == len(names) and all(isinstance(n, str) for n in out)","typeGuard":"from typing import Callable, List\n\ndef is_valid_schema_modifier(f: Callable[[List[str]], List[str]], names: List[str]) -> bool:\n    try:\n        out = f(list(names))\n        return isinstance(out, list) and all(isinstance(n, str) for n in out)\n    except Exception:\n        return False","tryCatchPattern":null,"preventionTips":["Never let exceptions escape the callback; use .get(name, name) lookups","Test callbacks with real and empty name lists","Keep callbacks pure name-mapping functions"],"tags":["python","lazyframe","schema","callback","panic","ffi"],"backgroundTag":null,"analyzedSha":"df599052daf96e7a9cc30a3b0c6bd25d6947e3c0","analyzedAt":"2026-08-16T12:10:03.978Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}