pola-rs/polars · error

could not read

Error message

could not read

What it means

ReaderBytes::from tries three strategies in order: borrow the bytes (Cursor-like sources), mmap the file, or stream-read into a buffer. The .expect("could not read") fires on the last resort: the source exposed neither bytes nor a File handle (typical for compressed/wrapped readers), and read_to_end then returned an I/O error. So the panic means a real OS-level read failure on a non-mmap-able source, surfaced as a panic instead of an io::Result.

Solutions

1. Verify the source out-of-band first: gzip -t data.csv.gz, and re-stage the file if truncated
2. Read to memory yourself and hand polars bytes so you get a real io::Result: let bytes = std::fs::read(path)?; CsvReadOptions::default().parse(Cursor::new(bytes))?
3. If reading from pipes/fifos, buffer to a temp file or Vec first - the mmap fast path needs a real file anyway
4. Check dmesg/NFS logs for the underlying EIO if the file is local and integrity checks pass

Example fix

// before (panic inside read_to_end on a corrupt/truncated stream)
let df = CsvReadOptions::default().parse(path)?;

// after (io::Error you can actually handle)
let bytes = std::fs::read(path)?;   // or fetch over HTTP and validate first
let df = CsvReadOptions::default().parse(std::io::Cursor::new(bytes))?;

use std::io::{Read, Seek, SeekFrom};
fn preflight_readable(path: &std::path::Path) -> std::io::Result<()> {
    let mut f = std::fs::File::open(path)?;
    let meta = f.metadata()?;
    if meta.len() == 0 { return Ok(()); }
    let mut probe = vec![0u8; 4096.min(meta.len() as usize)];
    f.read_exact(&mut probe)?;   // surfaces permission/EIO errors before polars
    f.seek(SeekFrom::Start(0))?;
    Ok(())
}
// then prefer handing polars bytes you read yourself:
// let bytes = std::fs::read(path)?; CsvReadOptions::default().parse(Cursor::new(bytes))?;

catch_unwind(AssertUnwindSafe(|| CsvReadOptions::default().parse(path.clone()))) to log 'source unreadable: {path}'; the stream is dead so only retry after re-staging the file (re-download, gzip -t).

When it happens

Trigger: read_csv/scan over a source that has no File handle (e.g. a gzip/zstd compressed reader, a pipe, or process substitution like <(cat x.gz)) whose stream fails mid-copy: corrupt archive (unexpected EOF), EPIPE from a closed producer, EIO on failing disks/NFS, or permission changes after open.

Common situations: Partially downloaded .gz files; a plain CSV mislabeled with a .gz extension so the decompressing reader hits garbage; named pipes whose writer exits early; flaky network mounts.

Related errors

• failed to create spill file
• failed to open spill file
• failed to update file last accessed time
• activate 'decompress' feature
• activate dtype

AI-assisted analysis of pola-rs/polars@9b5d73fd00 (2026-08-19). Data as JSON: /api/errors/a530393e4a4ea12e. Report an issue: GitHub.

Appendix: source

Thrown at crates/polars-io/src/mmap.rs:101

impl<'a, T: 'a + MmapBytesReader> From<&'a mut T> for ReaderBytes<'a> {
    fn from(m: &'a mut T) -> Self {
        match m.to_bytes() {
            // , but somehow bchk doesn't see that lifetime is 'a.
            Some(s) => {
                let s = unsafe { std::mem::transmute::<&[u8], &'a [u8]>(s) };
                ReaderBytes::Borrowed(s)
            },
            None => {
                if let Some(f) = m.to_file() {
                    let mmap = MMapSemaphore::new_from_file(f).unwrap();
                    ReaderBytes::Owned(Buffer::from_owner(mmap))
                } else {
                    if verbose() {
                        eprintln!("could not memory map file; read to buffer.")
                    }
                    let mut buf = vec![];
                    m.read_to_end(&mut buf).expect("could not read");
                    ReaderBytes::Owned(Buffer::from_vec(buf))
                }
            },
        }
    }
}

View on GitHub (pinned to 9b5d73fd00)