{"record":{"id":"9697775ab8cb0071","repo":"pola-rs/polars","slug":"the-external-api-has-a-non-utf8-as-format","errorCode":null,"errorMessage":"The external API has a non-utf8 as format","messagePattern":"The external API has a non-utf8 as format","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/polars-arrow/src/ffi/schema.rs","lineNumber":173,"sourceCode":"            n_children: 0,\n            children: ptr::null_mut(),\n            dictionary: std::ptr::null_mut(),\n            release: None,\n            private_data: std::ptr::null_mut(),\n        }\n    }\n\n    pub fn is_null(&self) -> bool {\n        self.private_data.is_null()\n    }\n\n    /// returns the format of this schema.\n    pub(crate) fn format(&self) -> &str {\n        assert!(!self.format.is_null());\n        // safe because the lifetime of `self.format` equals `self`\n        unsafe { CStr::from_ptr(self.format) }\n            .to_str()\n            .expect(\"The external API has a non-utf8 as format\")\n    }\n\n    /// returns the name of this schema.\n    ///\n    /// Since this field is optional, `\"\"` is returned if it is not set (as per the spec).\n    pub(crate) fn name(&self) -> &str {\n        if self.name.is_null() {\n            return \"\";\n        }\n        // safe because the lifetime of `self.name` equals `self`\n        unsafe { CStr::from_ptr(self.name) }.to_str().unwrap()\n    }\n\n    pub(crate) fn child(&self, index: usize) -> &'static Self {\n        assert!(index < self.n_children as usize);\n        unsafe { self.children.add(index).as_ref().unwrap().as_ref().unwrap() }\n    }\n","sourceCodeStart":155,"sourceCodeEnd":191,"githubUrl":"https://github.com/pola-rs/polars/blob/9b5d73fd00236295624374b075d16b1fe6ec6df9/crates/polars-arrow/src/ffi/schema.rs#L155-L191","documentation":"In crates/polars-arrow/src/ffi/schema.rs, `ArrowSchema::format()` reads the C-string `format` field of an imported C Data Interface schema with `CStr::from_ptr(...).to_str().expect(\"The external API has a non-utf8 as format\")`. Arrow format strings ('i', 'u', '+l', 'w:d:16,...') are defined as ASCII, so non-UTF-8 bytes mean the struct is malformed — the crate treats that as a hard invariant breach and panics rather than erroring.","triggerScenarios":"Calling `import_field_c_arrow` (or anything that resolves a field's dtype from an imported ArrowSchema) when the `format` pointer references bytes that are not valid UTF-8: garbage pointer, non-null-terminated buffer, or a struct layout mismatch between the foreign library and polars-arrow's bindings.","commonSituations":"Custom FFI bridges that build ArrowSchema by hand (wrong pointer, freed buffer, forgetting the release callback/owner lifetime); version drift where the C schema ABI changed; passing a schema struct by value after it was released. Note `name()` (a few lines below) has the same exposure via `unwrap()`.","solutions":["Fix the producer: format must be a NUL-terminated UTF-8/ASCII C string owned for the schema's lifetime.","On the Rust side, pre-validate before importing: read the pointer yourself with CStr and check `.to_str().is_ok()`, then call the importer.","Wrap `import_field_c_arrow` in catch_unwind at the interop boundary to degrade the panic into an error.","Verify the Arrow C Data Interface version agreement on both sides of the boundary."],"exampleFix":"// before\nlet field = unsafe { import_field_c_arrow(&schema) }?; // panics on non-utf8 format\n\n// after\nunsafe {\n    assert!(!schema.format.is_null());\n    if std::ffi::CStr::from_ptr(schema.format).to_str().is_err() {\n        polars_bail!(ComputeError: \"foreign ArrowSchema.format is not valid UTF-8\");\n    }\n}\nlet field = unsafe { import_field_c_arrow(&schema) }?;","handlingStrategy":"validation","validationCode":"unsafe {\n    assert!(!schema.format.is_null(), \"ArrowSchema.format must not be null\");\n    if std::ffi::CStr::from_ptr(schema.format).to_str().is_err() {\n        return Err(polars_err!(ComputeError: \"ArrowSchema.format is not valid UTF-8\"));\n    }\n}\nlet field = unsafe { import_field_c_arrow(&schema) }?;","typeGuard":null,"tryCatchPattern":"let field = std::panic::catch_unwind(|| unsafe { import_field_c_arrow(&schema) })\n    .map_err(|_| polars_err!(ComputeError: \"imported ArrowSchema is malformed (format/name not UTF-8)\"))?;","preventionTips":["Keep the format string NUL-terminated and owned for the schema's lifetime on the producer.","Set the release callback and never reuse a released ArrowSchema.","Pre-validate format and name pointers yourself before calling import routines."],"tags":["rust","polars","arrow","ffi","c-data-interface","utf8","schema","panic"],"backgroundTag":"c-data-interface-invalid-metadata","analyzedSha":"9b5d73fd00236295624374b075d16b1fe6ec6df9","analyzedAt":"2026-08-19T12:15:06.350Z","contentChangedAt":"2026-08-19T12:15:06.350Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}