What causes "failed to run mrecordlog operation" in quickwit-oss/quickwit?

Locate the original poisoning panic in the logs (it precedes this one) and fix its root cause.

failed to run mrecordlog operation

Error message

failed to run mrecordlog operation

What it means

put_slice locks the same internal Mutex as get_slice and `.expect`s it. The lock fails when the mutex was poisoned by a panic in another thread that held it (e.g. inside state.put_slice itself or get_slice's inner state mutation). The library treats a poisoned cache as unrecoverable.

Source

Thrown at quickwit/quickwit-ingest/src/mrecordlog_async.rs:98

        let mut mrecordlog = self.take();

        let join_res: Result<(T, MultiRecordLog), JoinError> =
            tokio::task::spawn_blocking(move || {
                let _entered = inner_span.entered();
                let res = operation(&mut mrecordlog);
                (res, mrecordlog)
            })
            .await;

        match join_res {
            Ok((operation_result, mrecordlog)) => {
                self.mrecordlog_opt = Some(mrecordlog);
                operation_result
            }
            Err(error) => {
                // This could be caused by a panic
                error!(%error, "failed to run mrecordlog operation");
                panic!("failed to run mrecordlog operation");
            }
        }
    }

    #[instrument(name = "mrecordlog.create_queue_async", skip_all, fields(queue))]
    pub async fn create_queue(&mut self, queue: &str) -> Result<(), CreateQueueError> {
        let span = info_span!("mrecordlog.create_queue", queue);
        let queue = queue.to_string();
        self.run_operation(span, move |mrecordlog| {
            mrecordlog
                .create_queue(&queue)
                .inspect(|outcome| {
                    WAL_BYTES_WRITTEN_CREATE_QUEUE.inc_by(outcome.wal_bytes_written);
                })
                .map(|_| ())
        })
        .await
    }

View on GitHub (pinned to a39730c5cd)

Solutions

Locate the original poisoning panic in the logs (it precedes this one) and fix its root cause.
Restart the process — once poisoned, the mutex stays poisoned until restart.
File a report with a backtrace of the original panic; a panic inside the cache's locked region is a bug.
Reduce concurrent pressure (e.g. huge splits, many concurrent leaf searches) if the original panic was resource-related.

Example fix

// before
let mut state = self
    .state
    .lock()
    .expect("file byte range cache mutex is poisoned");
// after
let mut state = self
    .state
    .lock()
    .unwrap_or_else(|poisoned| poisoned.into_inner());

Defensive patterns

Strategy: retry

Try / catch

// surface cache poisoning as a fatal, restartable error
Err(panic) => {
    tracing::error!("byte range cache mutex poisoned: {panic:?}");
    std::process::exit(1); // supervisor restarts
}

Prevention

Fix any bug that panics inside state.put_slice (validate byte ranges first).
Avoid oversubscribing memory; large OwnedBytes copies can OOM-panic inside the lock.
Alert on the first panic per process — later poisoned-mutex panics are secondary.
Restart promptly after any storage-thread panic.

When it happens

Trigger: A panic occurring in any thread while the `state` mutex is held — for example during `state.put_slice` on a malformed byte_range or an allocation failure — then any subsequent put_slice call panics with this message.

Common situations: Corrupt/unexpected split files causing panics during cache insertion; OOM while copying OwnedBytes; a bug in cache state bookkeeping (num_bytes accounting) triggering an assertion inside the locked region.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/beaedc458b533196. Report an issue: GitHub.