quickwit-oss/quickwit · error · StorageError

internal timeout on get_slice

Error message

internal timeout on get_slice

What it means

quickwit-storage wraps storage operations with a per-operation timeout and retry loop (TimeoutAndRetryStorage). When every attempt of a get_slice call times out before completing, the wrapper aborts and returns this Timeout-classified StorageError instead of hanging indefinitely. It signals that the underlying storage (e.g. S3) was reachable-slow or the timeout budget was too small, not that the object is missing.

Source

Thrown at quickwit/quickwit-storage/src/timeout_and_retry_storage.rs:118

            match tokio::time::timeout(timeout_duration, get_slice_fut).await {
                Ok(result) => {
                    match attempt_id {
                        0 => crate::metrics::GET_SLICE_TIMEOUT_SUCCESS_AFTER_0_TIMEOUT.inc(),
                        1 => crate::metrics::GET_SLICE_TIMEOUT_SUCCESS_AFTER_1_TIMEOUT.inc(),
                        _ => crate::metrics::GET_SLICE_TIMEOUT_SUCCESS_AFTER_2_PLUS_TIMEOUT.inc(),
                    }
                    return result;
                }
                Err(_elapsed) => {
                    rate_limited_info!(limit_per_min=60, num_bytes=num_bytes, path=%path.display(), timeout_secs=timeout_duration.as_secs_f32(), "get timeout elapsed");
                    continue;
                }
            }
        }
        rate_limited_warn!(limit_per_min=60, num_bytes=num_bytes, path=%path.display(), "all get_slice attempts timeouted");
        crate::metrics::GET_SLICE_TIMEOUT_ALL_TIMEOUTS.inc();
        return Err(
            StorageErrorKind::Timeout.with_error(anyhow::anyhow!("internal timeout on get_slice"))
        );
    }

    async fn get_slice_stream(
        &self,
        path: &Path,
        range: Range<usize>,
    ) -> StorageResult<Box<dyn AsyncRead + Send + Unpin>> {
        self.underlying.get_slice_stream(path, range).await
    }

    async fn get_all(&self, path: &Path) -> StorageResult<OwnedBytes> {
        self.underlying.get_all(path).await
    }

    async fn delete(&self, path: &Path) -> StorageResult<()> {
        self.underlying.delete(path).await
    }

View on GitHub (pinned to a39730c5cd)

Solutions

1. Check the underlying storage backend health/latency (S3 request metrics, throttling errors) at the time of failure
2. Increase the storage timeout in the node storage configuration and restart the indexer/searcher
3. Retry the operation - the error is transient if caused by throttling or network degradation
4. Reduce slice size or move data closer (same-region bucket) to cut per-request latency

Example fix

// before (quickwit.yaml)
storage:
  timeout: 1s
// after
storage:
  timeout: 30s

// Rust: bound the call and check slice size before issuing
assert!(byte_range.len() <= MAX_SAFE_SLICE_BYTES, "slice too large for configured timeout");

match storage.get_slice(&path, range).await {
    Err(e) if e.kind() == StorageErrorKind::Timeout => schedule_retry_with_backoff(path, range),
    Err(e) => return Err(e.into()),
    Ok(bytes) => Ok(bytes),
}

When it happens

Trigger: Calling Storage::get_slice on a slow storage backend where each retry attempt exceeds the configured timeout; very large slices over high-latency networks; storage backend degradation (S3 throttling, disk contention); timeout configured lower than realistic object-read latency.

Common situations: S3 rate limiting or throttling during heavy query load; reading from a remote-region bucket with high RTT; misconfigured timeout in storage config; transient network degradation on cloud storage.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• `append_records` should be called with `position_opt: None`
• failed to run mrecordlog operation
• stdin cannot be checkpointed
• Unexpected span kind: {}
• no server currently available

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/9387ffedaf3c9342. Report an issue: GitHub.