What causes "The external API has a non-utf8 as format" in pola-rs/polars?

Fix the producer: format must be a NUL-terminated UTF-8/ASCII C string owned for the schema's lifetime.

The external API has a non-utf8 as format

Error message

The external API has a non-utf8 as format

What it means

In crates/polars-arrow/src/ffi/schema.rs, `ArrowSchema::format()` reads the C-string `format` field of an imported C Data Interface schema with `CStr::from_ptr(...).to_str().expect("The external API has a non-utf8 as format")`. Arrow format strings ('i', 'u', '+l', 'w:d:16,...') are defined as ASCII, so non-UTF-8 bytes mean the struct is malformed — the crate treats that as a hard invariant breach and panics rather than erroring.

Source

Thrown at crates/polars-arrow/src/ffi/schema.rs:173

            n_children: 0,
            children: ptr::null_mut(),
            dictionary: std::ptr::null_mut(),
            release: None,
            private_data: std::ptr::null_mut(),
        }
    }

    pub fn is_null(&self) -> bool {
        self.private_data.is_null()
    }

    /// returns the format of this schema.
    pub(crate) fn format(&self) -> &str {
        assert!(!self.format.is_null());
        // safe because the lifetime of `self.format` equals `self`
        unsafe { CStr::from_ptr(self.format) }
            .to_str()
            .expect("The external API has a non-utf8 as format")
    }

    /// returns the name of this schema.
    ///
    /// Since this field is optional, `""` is returned if it is not set (as per the spec).
    pub(crate) fn name(&self) -> &str {
        if self.name.is_null() {
            return "";
        }
        // safe because the lifetime of `self.name` equals `self`
        unsafe { CStr::from_ptr(self.name) }.to_str().unwrap()
    }

    pub(crate) fn child(&self, index: usize) -> &'static Self {
        assert!(index < self.n_children as usize);
        unsafe { self.children.add(index).as_ref().unwrap().as_ref().unwrap() }
    }

View on GitHub (pinned to 9b5d73fd00)

Solutions

Fix the producer: format must be a NUL-terminated UTF-8/ASCII C string owned for the schema's lifetime.
On the Rust side, pre-validate before importing: read the pointer yourself with CStr and check `.to_str().is_ok()`, then call the importer.
Wrap `import_field_c_arrow` in catch_unwind at the interop boundary to degrade the panic into an error.
Verify the Arrow C Data Interface version agreement on both sides of the boundary.

Example fix

// before
let field = unsafe { import_field_c_arrow(&schema) }?; // panics on non-utf8 format

// after
unsafe {
    assert!(!schema.format.is_null());
    if std::ffi::CStr::from_ptr(schema.format).to_str().is_err() {
        polars_bail!(ComputeError: "foreign ArrowSchema.format is not valid UTF-8");
    }
}
let field = unsafe { import_field_c_arrow(&schema) }?;

Defensive patterns

Strategy: validation

Validate before calling

unsafe {
    assert!(!schema.format.is_null(), "ArrowSchema.format must not be null");
    if std::ffi::CStr::from_ptr(schema.format).to_str().is_err() {
        return Err(polars_err!(ComputeError: "ArrowSchema.format is not valid UTF-8"));
    }
}
let field = unsafe { import_field_c_arrow(&schema) }?;

Try / catch

let field = std::panic::catch_unwind(|| unsafe { import_field_c_arrow(&schema) })
    .map_err(|_| polars_err!(ComputeError: "imported ArrowSchema is malformed (format/name not UTF-8)"))?;

Prevention

Keep the format string NUL-terminated and owned for the schema's lifetime on the producer.
Set the release callback and never reuse a released ArrowSchema.
Pre-validate format and name pointers yourself before calling import routines.

When it happens

Trigger: Calling `import_field_c_arrow` (or anything that resolves a field's dtype from an imported ArrowSchema) when the `format` pointer references bytes that are not valid UTF-8: garbage pointer, non-null-terminated buffer, or a struct layout mismatch between the foreign library and polars-arrow's bindings.

Common situations: Custom FFI bridges that build ArrowSchema by hand (wrong pointer, freed buffer, forgetting the release callback/owner lifetime); version drift where the C schema ABI changed; passing a schema struct by value after it was released. Note `name()` (a few lines below) has the same exposure via `unwrap()`.

Related errors

AI-assisted analysis of pola-rs/polars@9b5d73fd00 (2026-08-19). Data as JSON: /api/errors/9697775ab8cb0071. Report an issue: GitHub.