{"record":{"id":"91d0d0a22e813ad3","repo":"quickwit-oss/quickwit","slug":"append-records-should-be-called-with-position-o","errorCode":null,"errorMessage":"`append_records` should be called with `position_opt: None`","messagePattern":"`append_records` should be called with `position_opt: None`","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-ingest/src/ingest_v2/mrecordlog_utils.rs","lineNumber":96,"sourceCode":"        #[cfg(feature = \"failpoints\")]\n        fail_point!(\"ingester:append_records\", |_| {\n            let io_error = io::Error::from(io::ErrorKind::PermissionDenied);\n            Err(AppendDocBatchError::Io(io_error))\n        });\n\n        mrecordlog\n            .append_records(queue_id, None, encoded_mrecords)\n            .await\n    };\n    match append_result {\n        Ok(Some(offset)) => Ok(Position::offset(offset)),\n        Ok(None) => panic!(\"`doc_batch` should not be empty\"),\n        Err(AppendError::IoError(io_error)) => Err(AppendDocBatchError::Io(io_error)),\n        Err(AppendError::MissingQueue(queue_id)) => {\n            Err(AppendDocBatchError::QueueNotFound(queue_id))\n        }\n        Err(AppendError::Past) => {\n            panic!(\"`append_records` should be called with `position_opt: None`\")\n        }\n    }\n}\n\n/// Error returned when the mrecordlog does not have enough capacity to store some records.\n#[derive(Debug, Clone, Copy, thiserror::Error)]\npub(super) enum NotEnoughCapacityError {\n    #[error(\n        \"write-ahead log is full, capacity: {capacity}, usage: {usage}, requested: {requested}\"\n    )]\n    Disk {\n        usage: ByteSize,\n        capacity: ByteSize,\n        requested: ByteSize,\n    },\n    #[error(\n        \"write-ahead log memory buffer is full: capacity: {capacity}, usage: {usage}, requested: \\\n         {requested}\"","sourceCodeStart":78,"sourceCodeEnd":114,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-ingest/src/ingest_v2/mrecordlog_utils.rs#L78-L114","documentation":"get_slice on the byte-range cache locks an internal std Mutex and `.expect`s the lock result. It fires only when the mutex is poisoned, i.e. another thread panicked while holding the lock. At that point the cache's shared state may be inconsistent, so the code refuses to continue silently.","triggerScenarios":"Any prior panic in put_slice/other methods of the same FileByteRangeCache while holding the `state` lock, followed by a later call to get_slice from any thread.","commonSituations":"A panic inside the caching layer (bug, OOM during slice copy) poisons the mutex; subsequent split downloads reads then panic with this message, often cascading into search failures.","solutions":["Find and fix the original panic that poisoned the mutex — look for the first panic in the logs from the cache/storage threads.","Restart the affected process; poisoning persists for the process lifetime once it occurs.","Report the original panic upstream with a backtrace (RUST_BACKTRACE=1), since a panic inside the cache is a bug.","Check memory limits if the original panic was an allocation failure while copying large byte ranges."],"exampleFix":"// before\nself.state\n    .lock()\n    .expect(\"file byte range cache mutex is poisoned\")\n    .get_slice(byte_range)\n// after\nlet state = self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());\nstate.get_slice(byte_range)","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// this panic cannot be caught selectively by message; treat process crash as fatal:\nmatch std::panic::catch_unwind(AssertUnwindSafe(storage_fetch)) {\n    Ok(v) => v,\n    Err(_) => { tracing::error!(\"byte range cache poisoned; restarting worker\"); restart(); unreachable!() }\n}","preventionTips":["Never panic while holding cache locks; return Result from inner state mutations.","Watch logs for the FIRST panic — the poisoning panic precedes this message.","Keep ample memory headroom to avoid allocation panics in cache code.","Run processes under an auto-restarting supervisor."],"tags":["rust","mutex","concurrency","cache","storage"],"backgroundTag":"internal-invariant-violation","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}