risingwavelabs/risingwave · error · ObjectError

[Schema history] putObject timed out after {:?}. You can adj

Error message

[Schema history] putObject timed out after {:?}. You can adjust it via {} (ms), or set it to 0 to disable.

What it means

putObject wraps the object-store upload in tokio::time::timeout with a 3s default (RW_CDC_SCHEMA_HISTORY_PUT_OBJECT_TIMEOUT_MS). When the upload exceeds this deadline, an ObjectError::timeout is raised with this message telling the operator which env var to tune (or set to 0 to disable).

Source

Thrown at src/jni_core/src/opendal_schema_history.rs:170

        JAVA_BINDING_ASYNC_RUNTIME
            .block_on(async {
                let f = async {
                    let object_store = get_object_store().await;
                    object_store.upload(&object_name, data.into()).await
                };

                match total_timeout {
                    Some(timeout) => match tokio::time::timeout(timeout, f).await {
                        Ok(res) => res,
                        Err(_) => Err(ObjectError::timeout(format!(
                            "[Schema history] putObject timed out after {:?}. You can adjust it via {} (ms), or set it to 0 to disable.",
                            timeout, PUT_OBJECT_TOTAL_TIMEOUT_ENV
                        ))),
                    },
                    None => f.await,
                }
            })
            .map_err(|e| anyhow!(e))?;
        Ok(())
    });
}

#[unsafe(no_mangle)]
pub extern "system" fn Java_com_risingwave_java_binding_Binding_getObject<'a>(
    env: EnvParam<'a>,
    object_name: JString<'a>,
) -> JByteArray<'a> {
    execute_and_catch(env, move |env: &mut EnvParam<'_>| {
        let object_name = env.get_string(&object_name)?;
        let object_name: Cow<'_, str> = (&object_name).into();

        // Security check: validate file extension before any operation
        validate_dat_file_extension(&object_name).map_err(|e| anyhow!(e))?;
        let object_name = prepend_data_directory(&object_name);
        let result = JAVA_BINDING_ASYNC_RUNTIME
            .block_on(async {

View on GitHub (pinned to 6469eb736d)

Solutions

Set RW_CDC_SCHEMA_HISTORY_PUT_OBJECT_TIMEOUT_MS to a larger value (e.g. 30000) in the JVM environment.
Set RW_CDC_SCHEMA_HISTORY_PUT_OBJECT_TIMEOUT_MS=0 to disable the timeout entirely.
Check network connectivity/latency to the object store endpoint and retry the operation.
Reduce payload size or use a closer object store region.

Example fix

// before
// (env) RW_CDC_SCHEMA_HISTORY_PUT_OBJECT_TIMEOUT_MS unset (default 3000)
// after
// (env) RW_CDC_SCHEMA_HISTORY_PUT_OBJECT_TIMEOUT_MS=30000

Defensive patterns

Strategy: retry

Validate before calling

String t = System.getenv("RW_CDC_SCHEMA_HISTORY_PUT_OBJECT_TIMEOUT_MS"); long timeoutMs = (t == null) ? 3000 : Long.parseLong(t.trim());

Try / catch

try { binding.putObject(name, bytes); } catch (RisingWaveException e) { if (e.getMessage().contains("timed out")) { /* retry or raise timeout */ } }

Prevention

Set RW_CDC_SCHEMA_HISTORY_PUT_OBJECT_TIMEOUT_MS generously (e.g. 30000) for high-latency storage
Set it to 0 to disable the deadline when uploads are large
Monitor object store latency and co-locate the compute with the store region

When it happens

Trigger: Calling Binding.putObject when the remote object store (S3/GCS/Azure) takes longer than the configured timeout in milliseconds to complete the upload.

Common situations: Slow or throttled S3 endpoints, large schema-history payloads, high network latency, or the 3000ms default being too tight for a region with high RTT.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/7f313cb17e651beb. Report an issue: GitHub.