{"record":{"id":"1d08504a32fe91d2","repo":"rustfs/rustfs","slug":"timedout-1d0850","errorCode":"TimedOut","errorMessage":"inspect archive generation timed out","messagePattern":"inspect archive generation timed out","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"rustfs/src/admin/handlers/inspect_archive.rs","lineNumber":519,"sourceCode":"    header.extend_from_slice(FORMAT_MAGIC);\n    header.extend_from_slice(&INSPECT_ARCHIVE_VERSION.to_be_bytes());\n    header.extend_from_slice(&u32::try_from(ARCHIVE_CHUNK_SIZE).map_err(io::Error::other)?.to_be_bytes());\n    header.extend_from_slice(&wrapped_len.to_be_bytes());\n    header.extend_from_slice(&nonce_prefix);\n    header.extend_from_slice(&wrapped_key);\n\n    let mut sent = 0;\n    send_bounded(&tx, &mut sent, Bytes::from(header), deadline).await?;\n    let mut digest = Sha256::new();\n    let mut buffer = vec![0_u8; ARCHIVE_CHUNK_SIZE];\n    let mut counter = 0_u32;\n\n    loop {\n        let read = tokio::select! {\n            biased;\n            _ = tx.closed() => return Err(io::Error::new(io::ErrorKind::BrokenPipe, \"inspect archive client disconnected\")),\n            result = tokio::time::timeout_at(deadline, reader.read(&mut buffer)) => {\n                result.map_err(|_| io::Error::new(io::ErrorKind::TimedOut, \"inspect archive generation timed out\"))??\n            }\n        };\n        if read == 0 {\n            break;\n        }\n        digest.update(&buffer[..read]);\n        let framed = encrypt_record(&cipher, &nonce_prefix, RECORD_DATA, counter, &buffer[..read])?;\n        send_bounded(&tx, &mut sent, framed, deadline).await?;\n        counter = counter\n            .checked_add(1)\n            .ok_or_else(|| io::Error::other(\"inspect archive record counter exhausted\"))?;\n    }\n\n    match tokio::time::timeout_at(deadline, completion).await {\n        Err(_) => return Err(io::Error::new(io::ErrorKind::TimedOut, \"inspect archive generation timed out\")),\n        Ok(Ok(Ok(()))) => {}\n        Ok(Ok(Err(error))) => return Err(error),\n        Ok(Err(error)) => return Err(io::Error::other(format!(\"inspect archive producer ended unexpectedly: {error}\"))),","sourceCodeStart":501,"sourceCodeEnd":537,"githubUrl":"https://github.com/rustfs/rustfs/blob/201c653dcd34c2a01b9aec5991ed76176b342118/rustfs/src/admin/handlers/inspect_archive.rs#L501-L537","documentation":"An io::Error with kind TimedOut raised when a single read from the underlying xl.meta/drive reader exceeded the remaining budget of the inspect-archive operation's absolute deadline (INSPECT_ARCHIVE_MAX_DURATION = 30 seconds, set when the request starts). The deadline is enforced with tokio::time::timeout_at around each read, so one stalled drive read is enough to abort the whole encrypted stream. The error is terminal for this request: the design deliberately refuses to let a slow disk stretch an admin diagnostic past its cap.","triggerScenarios":"Invoking the inspect-archive admin endpoint for a bucket/object whose metadata read blocks: an NFS/network drive hanging, a drive in error/recovery state, VM storage stall, or simply an archive large enough that sequential ARCHIVE_CHUNK_SIZE reads cannot finish inside the fixed 30-second window shared with artifact collection and encryption.","commonSituations":"Degraded or ejecting drives during support diagnostics; overloaded nodes where IO latency spikes; inspecting very large objects/metadata sets on slower spinning disks; running under heavy concurrent load so read latency exceeds the fixed budget.","solutions":["Check drive health and latency first (this error usually marks a slow or failing disk, which is exactly what inspect is probing): review dmesg/smart/IO error rates for the drives backing the affected bucket.","Retry the request when the node is less loaded, or narrow the inspection scope (single object, smaller bucket) so reads fit in the 30s budget.","If large-archive inspection legitimately needs longer on your hardware, raise INSPECT_ARCHIVE_MAX_DURATION in rustfs/src/admin/handlers/inspect_archive.rs and rebuild — the constant is the single knob that governs collection, streaming, and encryption.","Verify no client/proxy timeout interplay: the server deadline is independent, so fixing only the client will not stop this error."],"exampleFix":"// before (rustfs/src/admin/handlers/inspect_archive.rs:68)\npub const INSPECT_ARCHIVE_MAX_DURATION: Duration = Duration::from_secs(30);\n// after: allow large archives on slow drives (rebuild rustfs)\npub const INSPECT_ARCHIVE_MAX_DURATION: Duration = Duration::from_secs(120);","handlingStrategy":"retry","validationCode":"// before requesting an archive, sanity-check the target drives respond quickly\nasync fn drive_responsive(path: &str) -> bool {\n    tokio::fs::metadata(path).await.is_ok() && tokio::time::timeout(\n        std::time::Duration::from_secs(2),\n        tokio::fs::read(format!(\"{path}/format.json\")),\n    ).await.is_ok()\n}","typeGuard":"fn is_deadline_timeout(e: &std::io::Error) -> bool {\n    e.kind() == std::io::ErrorKind::TimedOut\n}","tryCatchPattern":"match response {\n    Err(e) if e.kind() == std::io::ErrorKind::TimedOut => {\n        // fixed 30s server budget exceeded: narrow the scope or retry off-peak,\n        // do not blind-loop — a stalled drive will time out again\n        narrow_scope_or_retry_later();\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Keep inspection targets small (one object / small bucket) so reads fit the 30s budget.","Check drive health before invoking inspect on a suspect node — the timeout is often the finding.","Raise INSPECT_ARCHIVE_MAX_DURATION in a custom build only after ruling out failing hardware.","Run inspections off-peak to avoid IO contention consuming the deadline."],"tags":["rust","tokio","timeout","io","admin-api","deadline"],"backgroundTag":"streaming-deadline-timeout","analyzedSha":"201c653dcd34c2a01b9aec5991ed76176b342118","analyzedAt":"2026-08-20T21:57:04.799Z","contentChangedAt":"2026-08-20T21:57:04.799Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}