{"record":{"id":"9387ffedaf3c9342","repo":"quickwit-oss/quickwit","slug":"internal-timeout-on-get-slice","errorCode":null,"errorMessage":"internal timeout on get_slice","messagePattern":"internal timeout on get_slice","errorType":"exception","errorClass":"StorageError","httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-storage/src/timeout_and_retry_storage.rs","lineNumber":118,"sourceCode":"            match tokio::time::timeout(timeout_duration, get_slice_fut).await {\n                Ok(result) => {\n                    match attempt_id {\n                        0 => crate::metrics::GET_SLICE_TIMEOUT_SUCCESS_AFTER_0_TIMEOUT.inc(),\n                        1 => crate::metrics::GET_SLICE_TIMEOUT_SUCCESS_AFTER_1_TIMEOUT.inc(),\n                        _ => crate::metrics::GET_SLICE_TIMEOUT_SUCCESS_AFTER_2_PLUS_TIMEOUT.inc(),\n                    }\n                    return result;\n                }\n                Err(_elapsed) => {\n                    rate_limited_info!(limit_per_min=60, num_bytes=num_bytes, path=%path.display(), timeout_secs=timeout_duration.as_secs_f32(), \"get timeout elapsed\");\n                    continue;\n                }\n            }\n        }\n        rate_limited_warn!(limit_per_min=60, num_bytes=num_bytes, path=%path.display(), \"all get_slice attempts timeouted\");\n        crate::metrics::GET_SLICE_TIMEOUT_ALL_TIMEOUTS.inc();\n        return Err(\n            StorageErrorKind::Timeout.with_error(anyhow::anyhow!(\"internal timeout on get_slice\"))\n        );\n    }\n\n    async fn get_slice_stream(\n        &self,\n        path: &Path,\n        range: Range<usize>,\n    ) -> StorageResult<Box<dyn AsyncRead + Send + Unpin>> {\n        self.underlying.get_slice_stream(path, range).await\n    }\n\n    async fn get_all(&self, path: &Path) -> StorageResult<OwnedBytes> {\n        self.underlying.get_all(path).await\n    }\n\n    async fn delete(&self, path: &Path) -> StorageResult<()> {\n        self.underlying.delete(path).await\n    }","sourceCodeStart":100,"sourceCodeEnd":136,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-storage/src/timeout_and_retry_storage.rs#L100-L136","documentation":"quickwit-storage wraps storage operations with a per-operation timeout and retry loop (TimeoutAndRetryStorage). When every attempt of a get_slice call times out before completing, the wrapper aborts and returns this Timeout-classified StorageError instead of hanging indefinitely. It signals that the underlying storage (e.g. S3) was reachable-slow or the timeout budget was too small, not that the object is missing.","triggerScenarios":"Calling Storage::get_slice on a slow storage backend where each retry attempt exceeds the configured timeout; very large slices over high-latency networks; storage backend degradation (S3 throttling, disk contention); timeout configured lower than realistic object-read latency.","commonSituations":"S3 rate limiting or throttling during heavy query load; reading from a remote-region bucket with high RTT; misconfigured timeout in storage config; transient network degradation on cloud storage.","solutions":["Check the underlying storage backend health/latency (S3 request metrics, throttling errors) at the time of failure","Increase the storage timeout in the node storage configuration and restart the indexer/searcher","Retry the operation - the error is transient if caused by throttling or network degradation","Reduce slice size or move data closer (same-region bucket) to cut per-request latency"],"exampleFix":"// before (quickwit.yaml)\nstorage:\n  timeout: 1s\n// after\nstorage:\n  timeout: 30s","handlingStrategy":"retry","validationCode":"// Rust: bound the call and check slice size before issuing\nassert!(byte_range.len() <= MAX_SAFE_SLICE_BYTES, \"slice too large for configured timeout\");","typeGuard":null,"tryCatchPattern":"match storage.get_slice(&path, range).await {\n    Err(e) if e.kind() == StorageErrorKind::Timeout => schedule_retry_with_backoff(path, range),\n    Err(e) => return Err(e.into()),\n    Ok(bytes) => Ok(bytes),\n}","preventionTips":["Set storage timeout comfortably above worst-case read latency for your backend/region","Monitor GET_SLICE_TIMEOUT metrics and storage throttling alarms","Keep slices small; avoid very large byte ranges in a single call","Retry with exponential backoff on Timeout-kind errors"],"tags":["storage","timeout","network","retry"],"backgroundTag":"request-timeout","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}