databendlabs/databend · error

Time went backwards

Error message

Time went backwards

What it means

`HttpQueryManager::close_query` (invoked by the query final and cancel handlers) computes the current Unix timestamp with `SystemTime::now().duration_since(SystemTime::UNIX_EPOCH).expect("Time went backwards")`. This panics only if the system clock is set before 1970-01-01, which would make query TTL/expiry bookkeeping meaningless. Databend treats a pre-epoch clock as a fatal environment problem and aborts the close operation via panic.

Solutions

1. Fix the host clock: enable/verify NTP (`systemctl status chronyd|ntpd`, `timedatectl set-ntp true`) so system time is current.
2. Check `date`/`timedatectl` on the node; if it shows a pre-1970 date, correct it immediately and restart the Databend process.
3. For VM/container environments, ensure the hypervisor syncs clocks after snapshot restore.
4. Harden the code to fall back to a last-known timestamp or return an error instead of panicking, so a bad clock fails queries gracefully.

Example fix

// before
let now = SystemTime::now()
    .duration_since(SystemTime::UNIX_EPOCH)
    .expect("Time went backwards")
    .as_secs();

// after
let now = SystemTime::now()
    .duration_since(SystemTime::UNIX_EPOCH)
    .map_err(|e| poem::Error::from_status(
        StatusCode::INTERNAL_SERVER_ERROR,
    ))? // log: system clock before UNIX_EPOCH
    .as_secs();

#!/bin/sh
# Pre-check node clock before starting/using Databend HTTP API
now=$(date +%s)
if [ "$now" -lt 0 ]; then echo "System clock before UNIX epoch - fix time sync!"; exit 1; fi

// Server-side hardening
SystemTime::now().duration_since(SystemTime::UNIX_EPOCH)
    .map_err(|_| poem::Error::from_status(StatusCode::INTERNAL_SERVER_ERROR))
    .map(|d| d.as_secs())?

When it happens

Trigger: Calling the query final (`DELETE /v1/query/{id}`) or cancel endpoint while the host clock is before the Unix epoch — caused by RTC reset, VM snapshot restore with wrong clock, missing NTP sync, or clock skew correction rolling the time backwards past epoch.

Common situations: Bare-metal servers with dead CMOS battery; containers/VMs restored from snapshots; freshly provisioned machines before NTP daemon syncs; misconfigured systems with wildly wrong dates.

Related errors

• fail to serialize Scalar
• current_page has taken
• {}
• Temp table id used up
• Invalid temp table desc

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/6dc82ce490796074. Report an issue: GitHub.

Appendix: source

Thrown at src/query/service/src/servers/http/v1/query/http_query_manager.rs:207

                            break;
                        }
                    }
                }
            });
        Ok(query)
    }

    #[async_backtrace::framed]
    pub(crate) async fn close_query(
        self: &Arc<Self>,
        query_id: &str,
        reason: CloseReason,
        client_session_id: &Option<String>,
        check_client_session_id: bool,
    ) -> poem::error::Result<Option<Arc<HttpQuery>>> {
        let now = SystemTime::now()
            .duration_since(SystemTime::UNIX_EPOCH)
            .expect("Time went backwards")
            .as_secs();
        let (query, closed_state) = self.queries.write().close(
            query_id,
            reason,
            now,
            client_session_id,
            check_client_session_id,
        )?;
        if let Some(q) = &query {
            if let Some(st) = closed_state {
                q.kill(st.error_code(q.result_timeout_secs)).await;
            }
        }
        Ok(query)
    }

    #[async_backtrace::framed]
    pub(crate) async fn add_txn(

View on GitHub (pinned to 288d84d76e)