{"record":{"id":"beaedc458b533196","repo":"quickwit-oss/quickwit","slug":"failed-to-run-mrecordlog-operation","errorCode":null,"errorMessage":"failed to run mrecordlog operation","messagePattern":"failed to run mrecordlog operation","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-ingest/src/mrecordlog_async.rs","lineNumber":98,"sourceCode":"        let mut mrecordlog = self.take();\n\n        let join_res: Result<(T, MultiRecordLog), JoinError> =\n            tokio::task::spawn_blocking(move || {\n                let _entered = inner_span.entered();\n                let res = operation(&mut mrecordlog);\n                (res, mrecordlog)\n            })\n            .await;\n\n        match join_res {\n            Ok((operation_result, mrecordlog)) => {\n                self.mrecordlog_opt = Some(mrecordlog);\n                operation_result\n            }\n            Err(error) => {\n                // This could be caused by a panic\n                error!(%error, \"failed to run mrecordlog operation\");\n                panic!(\"failed to run mrecordlog operation\");\n            }\n        }\n    }\n\n    #[instrument(name = \"mrecordlog.create_queue_async\", skip_all, fields(queue))]\n    pub async fn create_queue(&mut self, queue: &str) -> Result<(), CreateQueueError> {\n        let span = info_span!(\"mrecordlog.create_queue\", queue);\n        let queue = queue.to_string();\n        self.run_operation(span, move |mrecordlog| {\n            mrecordlog\n                .create_queue(&queue)\n                .inspect(|outcome| {\n                    WAL_BYTES_WRITTEN_CREATE_QUEUE.inc_by(outcome.wal_bytes_written);\n                })\n                .map(|_| ())\n        })\n        .await\n    }","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-ingest/src/mrecordlog_async.rs#L80-L116","documentation":"put_slice locks the same internal Mutex as get_slice and `.expect`s it. The lock fails when the mutex was poisoned by a panic in another thread that held it (e.g. inside state.put_slice itself or get_slice's inner state mutation). The library treats a poisoned cache as unrecoverable.","triggerScenarios":"A panic occurring in any thread while the `state` mutex is held — for example during `state.put_slice` on a malformed byte_range or an allocation failure — then any subsequent put_slice call panics with this message.","commonSituations":"Corrupt/unexpected split files causing panics during cache insertion; OOM while copying OwnedBytes; a bug in cache state bookkeeping (num_bytes accounting) triggering an assertion inside the locked region.","solutions":["Locate the original poisoning panic in the logs (it precedes this one) and fix its root cause.","Restart the process — once poisoned, the mutex stays poisoned until restart.","File a report with a backtrace of the original panic; a panic inside the cache's locked region is a bug.","Reduce concurrent pressure (e.g. huge splits, many concurrent leaf searches) if the original panic was resource-related."],"exampleFix":"// before\nlet mut state = self\n    .state\n    .lock()\n    .expect(\"file byte range cache mutex is poisoned\");\n// after\nlet mut state = self\n    .state\n    .lock()\n    .unwrap_or_else(|poisoned| poisoned.into_inner());","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// surface cache poisoning as a fatal, restartable error\nErr(panic) => {\n    tracing::error!(\"byte range cache mutex poisoned: {panic:?}\");\n    std::process::exit(1); // supervisor restarts\n}","preventionTips":["Fix any bug that panics inside state.put_slice (validate byte ranges first).","Avoid oversubscribing memory; large OwnedBytes copies can OOM-panic inside the lock.","Alert on the first panic per process — later poisoned-mutex panics are secondary.","Restart promptly after any storage-thread panic."],"tags":["rust","mutex","concurrency","cache","storage"],"backgroundTag":"internal-invariant-violation","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}