pola-rs/polars · error

not implemented

Error message

not implemented

What it means

The write2 IPC serializer (IpcBatchSerializationContext) emits compressed buffers as little-endian bytes; on a big-endian host it bails with unimplemented!() (crates/polars-arrow/src/io/ipc/write2/array/primitive.rs:45) whenever compression is enabled. Compressed IPC output was only implemented for little-endian machines, so the combination BE host + WriteOptions{compression: Some(_)} panics on the first primitive buffer written.

Source

Thrown at crates/polars-arrow/src/io/ipc/write2/array/primitive.rs:45

        write_bytes(
            ctx,
            BufferOrSlice::Buffer(&buffer.clone().try_transmute().unwrap()),
        )
    } else {
        write_native_type_iter(ctx, buffer.iter().copied())
    }
}

pub(super) fn write_native_type_iter<T: NativeType, I: ExactSizeIterator<Item = T>>(
    ctx: &mut IpcBatchSerializationContext<'_>,
    iter: I,
) -> PolarsResult<()> {
    let start_offset = ctx.arrow_data.len();
    let mut iter = iter.map(|v| T::to_le_bytes(&v));

    if let Some(compression) = ctx.compression {
        if !is_native_little_endian() {
            unimplemented!();
        }

        let bytes_scratch = ctx.bytes_scratch.get();
        bytes_scratch.reserve_exact(std::mem::size_of::<T>() * iter.len());

        iter.for_each(|v| {
            bytes_scratch.extend_from_slice(v.as_ref());
        });

        ctx.arrow_data
            .write_all(&(bytes_scratch.len() as i64).to_le_bytes())?;

        match compression {
            Compression::LZ4 => {
                compression::compress_lz4(bytes_scratch, ctx.arrow_data.as_io_write())?;
            },
            Compression::ZSTD(level) => {
                compression::compress_zstd(bytes_scratch, ctx.arrow_data.as_io_write(), level)?;

View on GitHub (pinned to df599052da)

Solutions

  1. Disable compression on big-endian builds: WriteOptions { compression: None }
  2. Produce the IPC file on a little-endian node and ship it instead
  3. Byte-swap buffers before handing them to the writer if you own the serialization layer
  4. Upstream: implement the swapped compression path or return a PolarsResult error instead of panicking

Example fix

// before
let options = WriteOptions { compression: Some(CompressionCodec::Zstd) }; // panics on s390x

// after
let options = WriteOptions {
    compression: if cfg!(target_endian = "big") { None } else { Some(CompressionCodec::Zstd) },
};
Defensive patterns

Strategy: validation

Validate before calling

if cfg!(target_endian = "big") {
    polars_ensure!(options.compression.is_none(),
        InvalidOperation: "compressed IPC write is not implemented on big-endian hosts; set compression to None");
}

Prevention

When it happens

Trigger: Writing IPC/Feather with compression Some(LZ4/ZSTD) on big-endian targets such as s390x, powerpc or sparc — including emulated runs (qemu-s390x) in CI.

Common situations: Mainframe/POWER deployments; cross-architecture container images; CI build farms that run the test suite under BE emulation and hit the compressed-write tests.

Related errors

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/8be62d0d859ac89e. Report an issue: GitHub.