risingwavelabs/risingwave · error · ObjectError

Timeout error: {0}

Error message

Timeout error: {0}

What it means

ObjectError::Timeout represents an operation against the object store that exceeded its time limit. The object store layer raises it when a backend interaction (read, write, list) times out, distinguishing it from generic internal errors so callers can retry.

Source

Thrown at src/object_store/src/object/error.rs:53

    },
    #[error("disk error: {msg}")]
    Disk {
        msg: String,
        #[source]
        inner: io::Error,
    },
    #[error(transparent)]
    Opendal(#[from] opendal::Error),
    #[error(transparent)]
    Mem(#[from] crate::object::mem::Error),
    #[error("Internal error: {0}")]
    #[construct(skip)]
    Internal(String),
    #[cfg(madsim)]
    #[error(transparent)]
    Sim(#[from] crate::object::sim::SimError),

    #[error("Timeout error: {0}")]
    Timeout(String),
}

impl ObjectError {
    pub fn internal(msg: impl ToString) -> Self {
        ObjectErrorInner::Internal(msg.to_string()).into()
    }

    /// Tells whether the error indicates the target object is not found.
    pub fn is_object_not_found_error(&self) -> bool {
        match self.inner() {
            ObjectErrorInner::S3 {
                inner,
                should_retry: _,
            } => {
                if let Some(aws_smithy_runtime_api::client::result::SdkError::ServiceError(err)) =
                    inner.downcast_ref::<aws_smithy_runtime_api::client::result::SdkError<
                        GetObjectError,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Retry the operation — timeouts are often transient
2. Check network connectivity and latency to the object store endpoint
3. Increase the relevant request/operation timeout in the store configuration
4. Check the storage service status (e.g. S3/GCS health dashboards) for outages

Example fix

// before: single attempt fails the whole operation
let data = store.read(path, range).await?;
// after: retry on timeout
let data = match store.read(path, range).await {
    Err(e) if e.to_string().starts_with("Timeout error:") => {
        tokio::time::sleep(Duration::from_secs(1)).await;
        store.read(path, range).await?
    }
    r => r?,
};

// Ensure the store config has sane timeouts before building
assert!(config.store_request_timeout_ms.is_none() || config.store_request_timeout_ms.unwrap() > 0);

fn is_timeout_object_error(e: &ObjectError) -> bool {
    e.to_string().starts_with("Timeout error:")
}

async fn read_with_retry<S: ObjectStore>(store: &S, path: &str, retries: u32) -> ObjectResult<Bytes> {
    for attempt in 0..=retries {
        match store.read(path, None).await {
            Ok(b) => return Ok(b),
            Err(e) if e.to_string().starts_with("Timeout error:") && attempt < retries => {
                tokio::time::sleep(Duration::from_millis(200 * 2u64.pow(attempt))).await;
            }
            Err(e) => return Err(e),
        }
    }
    unreachable!()
}

When it happens

Trigger: Calls to ObjectStore read/write/list operations whose underlying backend (S3, GCS, OSS, etc.) exceeds the configured request timeout, or madsim simulation timeouts.

Common situations: Slow or degraded cloud storage endpoints, oversized requests, network congestion between the compute node and the object store, or too-low timeout configuration.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• [Schema history] putObject timed out after {:?}. You can adj
• all request confluent registry all timeout, {context} {}
• {e}
• (dataset guard acquisition error)
• SinkError::SqlServer(anyhow!(err))

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/72b2c28b71167b95. Report an issue: GitHub.