risingwavelabs/risingwave · critical

Time went backwards

Error message

Time went backwards

What it means

When building the S3 object path for Snowflake/Redshift sink uploads, the code computes `SystemTime::now().duration_since(UNIX_EPOCH)` and panics with "Time went backwards" if the system clock is earlier than the Unix epoch. This is an expect() panic, not a recoverable error.

Source

Thrown at src/connector/src/sink/snowflake_redshift/mod.rs:233

}

pub async fn build_opendal_writer_path(
    s3_config: &S3Common,
    operator: &Operator,
    dir: Option<&str>,
    target_table_name: &str,
) -> Result<(opendal::Writer, String)> {
    let mut base_path = s3_config.path.clone().unwrap_or("".to_owned());
    if !base_path.ends_with('/') {
        base_path.push('/');
    }
    base_path.push_str(&format!("{}/", target_table_name));
    if let Some(dir) = dir {
        base_path.push_str(&format!("{}/", dir));
    }
    let create_time = SystemTime::now()
        .duration_since(UNIX_EPOCH)
        .expect("Time went backwards");
    let object_name = format!(
        "{}{}_{}.{}",
        base_path,
        Uuid::new_v4(),
        create_time.as_millis(),
        "json",
    );
    let all_path = format!("s3://{}/{}", s3_config.bucket_name, object_name);
    Ok((
        operator.writer_with(&object_name).concurrent(8).await?,
        all_path,
    ))
}

/// Generic JDBC writer for both Redshift and Snowflake sinks
pub struct SnowflakeRedshiftSinkJdbcWriter {
    augmented_row: AugmentedChunk,
    jdbc_sink_writer: CoordinatedRemoteSinkWriter,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Fix the host clock: run NTP sync (e.g. `chronyc makestep` or `systemctl restart chronyd`)
2. Verify the container/VM has a valid clock source and restart it if time is frozen
3. If the panic persists, replace the expect with a fallback (e.g. use 0 or an error) in build_opendal_writer_path

Example fix

// before
let create_time = SystemTime::now().duration_since(UNIX_EPOCH).expect("Time went backwards");
// after
let create_time = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default();

// before starting the sink process, verify host clock
let now = SystemTime::now().duration_since(UNIX_EPOCH);
assert!(now.is_ok(), "system clock is before Unix epoch; fix NTP before running sinks");

fn clock_is_sane() -> bool {
    SystemTime::now().duration_since(UNIX_EPOCH).is_ok()
}

When it happens

Trigger: `build_opendal_writer_path` is invoked (from `write_batch` or `write_manifest_to_s3`) while the host's wall clock is set before 1970-01-01.

Common situations: VMs resuming from suspend with a stale RTC; misconfigured NTP; containers without a proper clock; corrupted system time after reboot.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• S3 configuration is required for S3 sink
• should have meta client
• should get metadata on checkpoint barrier
• newly start epoch {} after update vnode bitmap not matched w
• get none metadata in commit response for coordinated sink wr

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/2683a1f7883d3f8c. Report an issue: GitHub.