{"record":{"id":"c8487e2f530d8118","repo":"k3s-io/k3s","slug":"encryption-reload-time-is-incorrectly-ahead-of-cur","errorCode":null,"errorMessage":"encryption reload time is incorrectly ahead of current time","messagePattern":"encryption reload time is incorrectly ahead of current time","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/secretsencrypt/config.go","lineNumber":371,"sourceCode":"\n\t\t// First time, no metrics exist, so return zeros\n\t\tif tsMetric == nil && totalMetrics == nil && initialMetrics {\n\t\t\treturn true, nil\n\t\t}\n\n\t\tif tsMetric == nil {\n\t\t\tlastFailure = \"encryption config time metric not found\"\n\t\t\treturn false, nil\n\t\t}\n\n\t\tif totalMetrics == nil {\n\t\t\tlastFailure = \"encryption config total metric not found\"\n\t\t\treturn false, nil\n\t\t}\n\n\t\tunixUpdateTime = int64(tsMetric.GetMetric()[0].GetGauge().GetValue())\n\t\tif time.Now().Unix() < unixUpdateTime {\n\t\t\treturn true, errors.New(\"encryption reload time is incorrectly ahead of current time\")\n\t\t}\n\n\t\tfor _, totalMetric := range totalMetrics.GetMetric() {\n\t\t\tlogrus.Debugf(\"totalMetric: %+v\", totalMetric)\n\t\t\tfor _, label := range totalMetric.GetLabel() {\n\t\t\t\tif label.GetName() == \"status\" && label.GetValue() == \"success\" {\n\t\t\t\t\treloadSuccessCounter = int64(totalMetric.GetCounter().GetValue())\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\treturn true, nil\n\t})\n\n\tif err != nil {\n\t\terr = fmt.Errorf(\"%w: %s\", err, lastFailure)\n\t}\n\n\treturn unixUpdateTime, reloadSuccessCounter, err","sourceCodeStart":353,"sourceCodeEnd":389,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/secretsencrypt/config.go#L353-L389","documentation":"Before relying on encryption-config reload telemetry, the code reads the apiserver's last-reload unix timestamp from its metrics and sanity-checks it against the local clock. If the metric value is in the future relative to time.Now().Unix(), the check fails: the metric cannot be trusted to order events, usually due to clock skew between the local node and the apiserver's view of time.","triggerScenarios":"NTP steps the observing node's clock backwards after a reload; the apiserver host's clock is ahead of the node running the check (VM live-migration, snapshot restore of a VM, drifted RTC); metric scrape caching serving a stale scrape timestamp paired with a corrected local clock.","commonSituations":"Cloud VMs restored from snapshots resuming with old clocks; heterogeneous chrony/ntp coverage across nodes; laptops/lab hosts suspended and resumed.","solutions":["Synchronize clocks on all involved nodes (systemctl restart chronyd / systemd-timesyncd; verify with timedatectl and cross-node date comparison).","Wait for the next encryption-config reload or metric refresh and retry - once local time passes the metric value the check succeeds.","For VM environments, disable RTC passthrough quirks or use virtio-rtc/PTP sources so guests do not inherit skewed clocks."],"exampleFix":"# before: skewed clocks, validation aborts\ndate; ssh apiserver-host date  # differ by minutes\n# after: sync and retry\nsudo systemctl restart chronyd && timedatectl status\nk3s secrets-encrypt status  # retry once clocks agree","handlingStrategy":"retry","validationCode":"unixUpdateTime := int64(tsMetric.GetMetric()[0].GetGauge().GetValue())\nconst skewTolerance = 5 // seconds\nif time.Now().Unix()+skewTolerance < unixUpdateTime {\n    return errors.New(\"clock skew detected: apiserver reload metric is in the future\")\n}","typeGuard":null,"tryCatchPattern":"if err := wait.Poll(...); err != nil {\n    if strings.Contains(err.Error(), \"incorrectly ahead of current time\") {\n        // transient clock skew: resync NTP and retry after clocks converge\n        time.Sleep(time.Minute)\n    }\n}","preventionTips":["Run chrony/systemd-timesyncd on every node and alert on offset > 1s.","For VMs, use hypervisor clock sync (PTP/virtio) and avoid snapshot-resume without NTP."],"tags":["encryption","metrics","clock-skew","time"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}