pola-rs/polars · error

failed to encode spill file for '{context_id}': {e}

Error message

failed to encode spill file for '{context_id}': {e}

What it means

Panic in the OOC spill path when encoding a DataFrame to the in-memory IPC buffer fails before it can be written to disk (includes the context id and error). Typically caused by an unsupported dtype for IPC serialization.

Source

Thrown at crates/polars-ooc/src/spill_frame.rs:41

        self.estimated_size()
    }

    async fn spill(&self, context_id: &str) -> Self::Spilled {
        let mut df = self.clone();
        let context_id = context_id.to_owned();

        // Encode in the current task (on computational executor).
        let mut buf = Vec::new();

        let mut writer = IpcStreamWriter::new(&mut buf).with_compat_level(CompatLevel::newest());
        let clvl = polars_config::config().ooc_spill_compression_level();
        if clvl > 0 {
            let zstd_lvl = ZstdLevel::try_new(clvl.try_into().unwrap()).unwrap();
            writer = writer.with_compression(Some(IpcCompression::ZSTD(zstd_lvl)));
        }
        writer
            .finish(&mut df)
            .unwrap_or_else(|e| panic!("failed to encode spill file for '{context_id}': {e}",));

        // Do file creation / writing on tokio.
        let file = ASYNC
            .spawn(async move {
                let size = buf.len() as u64;
                let spill_file = SpillFile::new(&context_id, "ipc", size);
                if BYTES_SPILLED_TO_DISK.fetch_add(size).saturating_add(size)
                    > polars_config::config().ooc_disk_budget_bytes()
                {
                    spill_file.creation_aborted();
                    polars_error::abort::polars_abort_ooc_out_of_disk();
                }
                match tokio::fs::write(spill_file.path(), buf).await {
                    Ok(()) => spill_file,
                    Err(e) => {
                        let msg = format!(
                            "failed to create spill file '{}': {e}",
                            spill_file.path().display()

View on GitHub (pinned to 68506541d2)

Solutions

1. Check disk space and write permissions for the spill directory; encoding the spill file failed with the reported I/O error.
2. Reduce memory pressure so spilling is not needed (smaller chunks, streaming engine settings).

When it happens

Trigger: This panic/expect fires when execution reaches an unguarded state described by: "failed to encode spill file for '{context_id}': {e}". Typical triggers: unsupported dtype or feature-gated code path reached at runtime, invalid user input or environment variable value, calling API methods in the wrong order or on mismatched types, or data (lengths, offsets, ranges) violating the function's preconditions.

Common situations: Encountered when input data or configuration does not meet the preconditions of the throwing code path ("failed to encode spill file for '{context_id}': {e}"). Common cases: missing Cargo feature flags, mismatched dtypes/lengths between arrays or series, out-of-range temporal values, malformed environment variables, or operations on unsupported/complex nested types.

AI-assisted analysis of pola-rs/polars@68506541d2 (2026-08-19). Data as JSON: /api/errors/20ff2f7b4a8bd52b. Report an issue: GitHub.