quickwit-oss/quickwit · error · StorageError

reading file panicked

Error message

reading file panicked

What it means

This error is thrown when the blocking task spawned to read a byte range from a local file panics. `get_slice` runs the read inside `spawn_blocking`; the `.map_err(|_| ...)` discards the panic payload because the JoinError gives no payload access here, and wraps it as an Internal storage error. It means the read thread crashed unexpectedly, not that the file is missing.

Source

Thrown at quickwit/quickwit-storage/src/local_file_storage.rs:235

        let full_path = self.full_path(path)?;
        tokio::task::spawn_blocking(move || {
            use std::io::{Read, Seek};
            // we run these io in a spawn_blocking so there is no scheduling delay between each
            // step, as there would be if using tokio async File.
            let mut file = std::fs::File::open(full_path)?;
            file.seek(SeekFrom::Start(range.start as u64))?;
            let _in_flight_guards = object_storage_get_slice_in_flight_guards(range.len());
            let mut content_bytes: Vec<u8> = Vec::with_capacity(range.len());
            #[allow(clippy::uninit_vec)]
            unsafe {
                content_bytes.set_len(range.len());
            }
            file.read_exact(&mut content_bytes)?;
            Ok(OwnedBytes::new(content_bytes))
        })
        .await
        .map_err(|_| {
            StorageErrorKind::Internal.with_error(anyhow::anyhow!("reading file panicked"))
        })?
    }

    #[tracing::instrument(
        name = "storage.local_file.get_slice_stream",
        skip(self),
        level = "debug"
    )]
    async fn get_slice_stream(
        &self,
        path: &Path,
        range: Range<usize>,
    ) -> StorageResult<Box<dyn AsyncRead + Send + Unpin>> {
        let full_path = self.full_path(path)?;
        let mut file = tokio::fs::File::open(&full_path).await?;
        file.seek(SeekFrom::Start(range.start as u64)).await?;
        Ok(Box::new(file.take(range.len() as u64)))
    }

View on GitHub (pinned to a39730c5cd)

Solutions

1. Check the process logs/stderr for the original panic message and backtrace to find the real cause
2. Verify the file still exists and has not been truncated since the size was read (`ls -l`, compare with split metadata)
3. Re-download or re-index the affected split if it is corrupted
4. Retry the search; if reproducible on a specific split, inspect the split for corruption with `quickwit tool` commands
5. Report a bug with the panic backtrace if it occurs on intact files

let md = tokio::fs::metadata(&file_path).await?;
if !md.is_file() || md.len() == 0 { /* skip or re-fetch split */ }

match storage.get_slice(&path, range).await {
    Ok(bytes) => use(bytes),
    Err(e) if e.kind() == StorageErrorKind::Internal => log::error!("read panicked: {e}"),
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Calling `Storage::get_slice` on a LocalFileStorage when the spawned blocking closure panics — e.g. the file is truncated/deleted between length check and `read_exact` (unexpected EOF in some paths), or a bug in the closure (bad range arithmetic causing a slicing panic, poisoned state).

Common situations: File mutated or removed concurrently by log rotation or another process while Quickwit reads a hotcache/split slice; filesystem errors surfacing as panics; a Quickwit bug in range computation on zero-length or very small files.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• file `{}` is not a regular file, cannot determine its size
• node not found in pending
• OTP logs or traces do not support VRL transforms
• `doc_batch` should not be empty
• `append_records` should be called with `position_opt: None`

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/0876cf097208895c. Report an issue: GitHub.