What causes "`append_records` should be called with `position_opt: None`" in quickwit-oss/quickwit?

Find and fix the original panic that poisoned the mutex — look for the first panic in the logs from the cache/storage threads.

quickwit-oss/quickwit · error

`append_records` should be called with `position_opt: None`

Error message

`append_records` should be called with `position_opt: None`

What it means

get_slice on the byte-range cache locks an internal std Mutex and `.expect`s the lock result. It fires only when the mutex is poisoned, i.e. another thread panicked while holding the lock. At that point the cache's shared state may be inconsistent, so the code refuses to continue silently.

Source

Thrown at quickwit/quickwit-ingest/src/ingest_v2/mrecordlog_utils.rs:96

        #[cfg(feature = "failpoints")]
        fail_point!("ingester:append_records", |_| {
            let io_error = io::Error::from(io::ErrorKind::PermissionDenied);
            Err(AppendDocBatchError::Io(io_error))
        });

        mrecordlog
            .append_records(queue_id, None, encoded_mrecords)
            .await
    };
    match append_result {
        Ok(Some(offset)) => Ok(Position::offset(offset)),
        Ok(None) => panic!("`doc_batch` should not be empty"),
        Err(AppendError::IoError(io_error)) => Err(AppendDocBatchError::Io(io_error)),
        Err(AppendError::MissingQueue(queue_id)) => {
            Err(AppendDocBatchError::QueueNotFound(queue_id))
        }
        Err(AppendError::Past) => {
            panic!("`append_records` should be called with `position_opt: None`")
        }
    }
}

/// Error returned when the mrecordlog does not have enough capacity to store some records.
#[derive(Debug, Clone, Copy, thiserror::Error)]
pub(super) enum NotEnoughCapacityError {
    #[error(
        "write-ahead log is full, capacity: {capacity}, usage: {usage}, requested: {requested}"
    )]
    Disk {
        usage: ByteSize,
        capacity: ByteSize,
        requested: ByteSize,
    },
    #[error(
        "write-ahead log memory buffer is full: capacity: {capacity}, usage: {usage}, requested: \
         {requested}"

View on GitHub (pinned to a39730c5cd)

Solutions

Find and fix the original panic that poisoned the mutex — look for the first panic in the logs from the cache/storage threads.
Restart the affected process; poisoning persists for the process lifetime once it occurs.
Report the original panic upstream with a backtrace (RUST_BACKTRACE=1), since a panic inside the cache is a bug.
Check memory limits if the original panic was an allocation failure while copying large byte ranges.

Example fix

// before
self.state
    .lock()
    .expect("file byte range cache mutex is poisoned")
    .get_slice(byte_range)
// after
let state = self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
state.get_slice(byte_range)

Defensive patterns

Strategy: retry

Try / catch

// this panic cannot be caught selectively by message; treat process crash as fatal:
match std::panic::catch_unwind(AssertUnwindSafe(storage_fetch)) {
    Ok(v) => v,
    Err(_) => { tracing::error!("byte range cache poisoned; restarting worker"); restart(); unreachable!() }
}

Prevention

Never panic while holding cache locks; return Result from inner state mutations.
Watch logs for the FIRST panic — the poisoning panic precedes this message.
Keep ample memory headroom to avoid allocation panics in cache code.
Run processes under an auto-restarting supervisor.

When it happens

Trigger: Any prior panic in put_slice/other methods of the same FileByteRangeCache while holding the `state` lock, followed by a later call to get_slice from any thread.

Common situations: A panic inside the caching layer (bug, OOM during slice copy) poisons the mutex; subsequent split downloads reads then panic with this message, often cascading into search failures.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/91d0d0a22e813ad3. Report an issue: GitHub.