{"record":{"id":"72b2c28b71167b95","repo":"risingwavelabs/risingwave","slug":"timeout-error-0","errorCode":null,"errorMessage":"Timeout error: {0}","messagePattern":"Timeout error: (.+?)","errorType":"exception","errorClass":"ObjectError","httpStatus":null,"severity":"error","filePath":"src/object_store/src/object/error.rs","lineNumber":53,"sourceCode":"    },\n    #[error(\"disk error: {msg}\")]\n    Disk {\n        msg: String,\n        #[source]\n        inner: io::Error,\n    },\n    #[error(transparent)]\n    Opendal(#[from] opendal::Error),\n    #[error(transparent)]\n    Mem(#[from] crate::object::mem::Error),\n    #[error(\"Internal error: {0}\")]\n    #[construct(skip)]\n    Internal(String),\n    #[cfg(madsim)]\n    #[error(transparent)]\n    Sim(#[from] crate::object::sim::SimError),\n\n    #[error(\"Timeout error: {0}\")]\n    Timeout(String),\n}\n\nimpl ObjectError {\n    pub fn internal(msg: impl ToString) -> Self {\n        ObjectErrorInner::Internal(msg.to_string()).into()\n    }\n\n    /// Tells whether the error indicates the target object is not found.\n    pub fn is_object_not_found_error(&self) -> bool {\n        match self.inner() {\n            ObjectErrorInner::S3 {\n                inner,\n                should_retry: _,\n            } => {\n                if let Some(aws_smithy_runtime_api::client::result::SdkError::ServiceError(err)) =\n                    inner.downcast_ref::<aws_smithy_runtime_api::client::result::SdkError<\n                        GetObjectError,","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/object_store/src/object/error.rs#L35-L71","documentation":"ObjectError::Timeout represents an operation against the object store that exceeded its time limit. The object store layer raises it when a backend interaction (read, write, list) times out, distinguishing it from generic internal errors so callers can retry.","triggerScenarios":"Calls to ObjectStore read/write/list operations whose underlying backend (S3, GCS, OSS, etc.) exceeds the configured request timeout, or madsim simulation timeouts.","commonSituations":"Slow or degraded cloud storage endpoints, oversized requests, network congestion between the compute node and the object store, or too-low timeout configuration.","solutions":["Retry the operation — timeouts are often transient","Check network connectivity and latency to the object store endpoint","Increase the relevant request/operation timeout in the store configuration","Check the storage service status (e.g. S3/GCS health dashboards) for outages"],"exampleFix":"// before: single attempt fails the whole operation\nlet data = store.read(path, range).await?;\n// after: retry on timeout\nlet data = match store.read(path, range).await {\n    Err(e) if e.to_string().starts_with(\"Timeout error:\") => {\n        tokio::time::sleep(Duration::from_secs(1)).await;\n        store.read(path, range).await?\n    }\n    r => r?,\n};","handlingStrategy":"retry","validationCode":"// Ensure the store config has sane timeouts before building\nassert!(config.store_request_timeout_ms.is_none() || config.store_request_timeout_ms.unwrap() > 0);","typeGuard":"fn is_timeout_object_error(e: &ObjectError) -> bool {\n    e.to_string().starts_with(\"Timeout error:\")\n}","tryCatchPattern":"async fn read_with_retry<S: ObjectStore>(store: &S, path: &str, retries: u32) -> ObjectResult<Bytes> {\n    for attempt in 0..=retries {\n        match store.read(path, None).await {\n            Ok(b) => return Ok(b),\n            Err(e) if e.to_string().starts_with(\"Timeout error:\") && attempt < retries => {\n                tokio::time::sleep(Duration::from_millis(200 * 2u64.pow(attempt))).await;\n            }\n            Err(e) => return Err(e),\n        }\n    }\n    unreachable!()\n}","preventionTips":["Configure generous but bounded request timeouts for your object store backend","Monitor network latency between compute nodes and the storage endpoint","Use exponential backoff with jitter when retrying timeouts","Check cloud provider status pages during incident windows"],"tags":["rust","object-store","timeout","network"],"backgroundTag":"request-timeout","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}