{"record":{"id":"6dc82ce490796074","repo":"databendlabs/databend","slug":"time-went-backwards-6dc82c","errorCode":null,"errorMessage":"Time went backwards","messagePattern":"Time went backwards","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/query/service/src/servers/http/v1/query/http_query_manager.rs","lineNumber":207,"sourceCode":"                            break;\n                        }\n                    }\n                }\n            });\n        Ok(query)\n    }\n\n    #[async_backtrace::framed]\n    pub(crate) async fn close_query(\n        self: &Arc<Self>,\n        query_id: &str,\n        reason: CloseReason,\n        client_session_id: &Option<String>,\n        check_client_session_id: bool,\n    ) -> poem::error::Result<Option<Arc<HttpQuery>>> {\n        let now = SystemTime::now()\n            .duration_since(SystemTime::UNIX_EPOCH)\n            .expect(\"Time went backwards\")\n            .as_secs();\n        let (query, closed_state) = self.queries.write().close(\n            query_id,\n            reason,\n            now,\n            client_session_id,\n            check_client_session_id,\n        )?;\n        if let Some(q) = &query {\n            if let Some(st) = closed_state {\n                q.kill(st.error_code(q.result_timeout_secs)).await;\n            }\n        }\n        Ok(query)\n    }\n\n    #[async_backtrace::framed]\n    pub(crate) async fn add_txn(","sourceCodeStart":189,"sourceCodeEnd":225,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/query/service/src/servers/http/v1/query/http_query_manager.rs#L189-L225","documentation":"`HttpQueryManager::close_query` (invoked by the query final and cancel handlers) computes the current Unix timestamp with `SystemTime::now().duration_since(SystemTime::UNIX_EPOCH).expect(\"Time went backwards\")`. This panics only if the system clock is set before 1970-01-01, which would make query TTL/expiry bookkeeping meaningless. Databend treats a pre-epoch clock as a fatal environment problem and aborts the close operation via panic.","triggerScenarios":"Calling the query final (`DELETE /v1/query/{id}`) or cancel endpoint while the host clock is before the Unix epoch — caused by RTC reset, VM snapshot restore with wrong clock, missing NTP sync, or clock skew correction rolling the time backwards past epoch.","commonSituations":"Bare-metal servers with dead CMOS battery; containers/VMs restored from snapshots; freshly provisioned machines before NTP daemon syncs; misconfigured systems with wildly wrong dates.","solutions":["Fix the host clock: enable/verify NTP (`systemctl status chronyd|ntpd`, `timedatectl set-ntp true`) so system time is current.","Check `date`/`timedatectl` on the node; if it shows a pre-1970 date, correct it immediately and restart the Databend process.","For VM/container environments, ensure the hypervisor syncs clocks after snapshot restore.","Harden the code to fall back to a last-known timestamp or return an error instead of panicking, so a bad clock fails queries gracefully."],"exampleFix":"// before\nlet now = SystemTime::now()\n    .duration_since(SystemTime::UNIX_EPOCH)\n    .expect(\"Time went backwards\")\n    .as_secs();\n\n// after\nlet now = SystemTime::now()\n    .duration_since(SystemTime::UNIX_EPOCH)\n    .map_err(|e| poem::Error::from_status(\n        StatusCode::INTERNAL_SERVER_ERROR,\n    ))? // log: system clock before UNIX_EPOCH\n    .as_secs();","handlingStrategy":"validation","validationCode":"#!/bin/sh\n# Pre-check node clock before starting/using Databend HTTP API\nnow=$(date +%s)\nif [ \"$now\" -lt 0 ]; then echo \"System clock before UNIX epoch - fix time sync!\"; exit 1; fi","typeGuard":null,"tryCatchPattern":"// Server-side hardening\nSystemTime::now().duration_since(SystemTime::UNIX_EPOCH)\n    .map_err(|_| poem::Error::from_status(StatusCode::INTERNAL_SERVER_ERROR))\n    .map(|d| d.as_secs())?","preventionTips":["Enable NTP/chrony on all Databend hosts and verify with timedatectl.","Sync clocks after VM snapshot restore or container migration.","Alert on system time anomalies (pre-epoch dates, large jumps) in monitoring.","Check CMOS battery/clock on bare-metal servers that reboot with wrong time."],"tags":["system-time","clock-skew","panic","http-api"],"backgroundTag":"invalid-date-format","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}