stalwartlabs/stalwart · critical

Invalid system time, panicking to avoid data corruption

Error message

Invalid system time, panicking to avoid data corruption

What it means

Right after setting the snowflake node id, the parser constructs a SnowflakeIdGenerator and calls is_valid(); a false result means the system clock produced an id that fails the generator's validity check (typically a clock set before/outside the epoch or an obviously bogus time). Since snowflake ids derive from wall-clock time, running with an invalid clock risks duplicate or regressing ids, so the server panics at startup to avoid data corruption.

Source

Thrown at crates/common/src/config/inner.rs:58

impl Data {
    pub async fn parse(bp: &mut Bootstrap) -> Self {
        // Parse certificates
        let mut certificates = AHashMap::new();
        let mut subject_names = AHashSet::new();
        parse_certificates(bp, &mut certificates, &mut subject_names).await;
        if subject_names.is_empty() {
            subject_names.insert("localhost".into());
        }

        // Build and test snowflake id generator
        let node_id = bp.node_id();
        if node_id > MAX_NODE_ID {
            panic!("Node id {node_id} exceeds {MAX_NODE_ID}, panicking to avoid data corruption");
        }
        SnowflakeIdGenerator::set_node_id(node_id as u64);
        let id_generator = SnowflakeIdGenerator::new();
        if !id_generator.is_valid() {
            panic!("Invalid system time, panicking to avoid data corruption");
        }

        // Initialize apps
        let applications = WebApplications::new();
        applications.reload(bp).await;

        let blocked_ips = BlockedIps::parse(bp).await;
        let lookup_stores = LookupStores::build(bp).await;

        Data {
            spam_classifier: ArcSwap::from_pointee(SpamClassifier::default()),
            tls_certificates: ArcSwap::from_pointee(certificates),
            tls_self_signed_cert: build_self_signed_cert(
                subject_names
                    .into_iter()
                    .map(Into::into)
                    .collect::<Vec<_>>(),
            )

View on GitHub (pinned to e962003857)

Solutions

1. Fix the system clock before starting the server: enable and verify NTP sync (`timedatectl set-ntp true`, `chronyc tracking`).
2. Manually set a correct current time (`date -s` / hypervisor time sync) if NTP is unavailable, then restart.
3. If running in a container, ensure the host clock is correct since containers share it; restart after host clock repair.

Example fix

// before: container started with host clock at 1970
// after (host):
// $ sudo timedatectl set-ntp true
// $ timedatectl status  # verify 'System clock synchronized: yes'
// then restart the server

// before starting the server, verify the clock:
use std::time::{SystemTime, UNIX_EPOCH};
let now = SystemTime::now().duration_since(UNIX_EPOCH).expect("system clock before Unix epoch");
assert!(now.as_secs() > 1_600_000_000, "system clock looks unset; enable NTP");

fn clock_is_sane() -> bool {
    std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)
        .map(|d| d.as_secs() > 1_600_000_000).unwrap_or(false)
}

// panic at startup is not catchable; run the server under a supervisor that logs the panic and restarts after clock repair

When it happens

Trigger: Starting the server when the host system time is invalid — set before the snowflake epoch, reset to epoch/1970, or otherwise failing SnowflakeIdGenerator::is_valid(). `Config::parse` panics during the id-generator self-test.

Common situations: Fresh VMs or containers whose RTC is unset (clock at 1970), VMs resuming with drifted clocks, misconfigured NTP, or embedded devices without a battery-backed clock.

Related errors

• Node id {node_id} exceeds {MAX_NODE_ID}, panicking to avoid
• Failed to parse calendar template
• Failed to build DNSSEC resolver
• Failed to build DNS resolver
• failed to install aws-lc-rs as the default rustls crypto pro

AI-assisted analysis of stalwartlabs/stalwart@e962003857 (2026-09-06). Data as JSON: /api/errors/8b32d96ab8a10c0b. Report an issue: GitHub.