{"record":{"id":"96fc9461172e6a08","repo":"kubernetes/kops","slug":"incorrect-timestamp-v","errorCode":null,"errorMessage":"incorrect Timestamp %v","messagePattern":"incorrect Timestamp (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/bootstrap/pkibootstrap/pkiverifier/verifier.go","lineNumber":93,"sourceCode":"\t}\n\n\ttoken := &pkibootstrap.AuthToken{}\n\tif err = json.Unmarshal(tokenBytes, token); err != nil {\n\t\treturn nil, nil, fmt.Errorf(\"unmarshalling authorization token: %w\", err)\n\t}\n\n\ttokenData := &pkibootstrap.AuthTokenData{}\n\tif err := json.Unmarshal(token.Data, tokenData); err != nil {\n\t\treturn nil, nil, fmt.Errorf(\"unmarshalling authorization token data: %w\", err)\n\t}\n\n\t// Guard against replay attacks\n\tif tokenData.Audience != pkibootstrap.AudienceNodeAuthentication {\n\t\treturn nil, nil, fmt.Errorf(\"incorrect Audience\")\n\t}\n\ttimeSkew := math.Abs(time.Since(time.Unix(tokenData.Timestamp, 0)).Seconds())\n\tif timeSkew > float64(v.opt.MaxTimeSkew) {\n\t\treturn nil, nil, fmt.Errorf(\"incorrect Timestamp %v\", tokenData.Timestamp)\n\t}\n\n\t// Verify the token has signed the body content.\n\trequestHash := sha256.Sum256(body)\n\tif !bytes.Equal(requestHash[:], tokenData.RequestHash) {\n\t\treturn nil, nil, fmt.Errorf(\"incorrect RequestHash\")\n\t}\n\n\treturn token, tokenData, nil\n}\n\n// Can generate keys with\n// openssl ecparam -name prime256v1 -genkey -noout -out ec-priv-key.pem\n// openssl ec -in ec-priv-key.pem -pubout > ec-pub-key.pem\n// Note that golang doesn't support secp256k1: https://groups.google.com/g/golang-nuts/c/Mbkug5t3ZYA\n\nfunc (v *verifier) VerifyToken(ctx context.Context, rawRequest *http.Request, authToken string, body []byte) (*bootstrap.VerifyResult, error) {\n\t// Reminder: we shouldn't trust any data we get from the client until we've checked the signature (and even then...)","sourceCodeStart":75,"sourceCodeEnd":111,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/pkg/bootstrap/pkibootstrap/pkiverifier/verifier.go#L75-L111","documentation":"The token's Timestamp claim is more than MaxTimeSkew seconds (default 300) away from kops-controller's clock, so the verifier rejects it as a replay-protection measure. Tokens are single-use within a short window; the message includes the offending unix timestamp. math.Abs is used, so both too-old and too-far-in-the-future timestamps trigger it.","triggerScenarios":"parseTokenData (verifier.go:91-94) raises this when math.Abs(time.Since(time.Unix(tokenData.Timestamp,0)).Seconds()) > v.opt.MaxTimeSkew: node clock skew (NTP not running), a replayed/captured token reused after the window, a clock that jumped after VM migration/suspend, or kops-controller host clock drift in the opposite direction.","commonSituations":"Newly provisioned VM whose clock has not yet synced via NTP/chrony (very common right at node bootstrap time); suspended/resumed nodes; cloud regions or on-prem hosts without a time sync daemon; clock set to UTC vs local confusion in custom token minting; deliberately replayed tokens caught by the guard.","solutions":["Enable and verify NTP/chrony time synchronization on the node (and on the kops-controller host); restart the bootstrap request after the clock converges.","Check the node's clock against the controller: compare `date -u` on both hosts; skew must be under MaxTimeSkew (default 300s).","If legitimate skew is unavoidable, raise the MaxTimeSkew option passed to pkiverifier.NewVerifier (it defaults to 300 when 0).","Ensure a fresh token is minted per request — pkiAuthenticator.CreateToken stamps time.Now().Unix(); never cache or reuse tokens.","If the timestamp is wildly wrong (e.g. epoch 0), fix the client minting code to set Timestamp: time.Now().Unix()."],"exampleFix":"// before: kops-controller options\nopt.MaxTimeSkew = 0 // falls back to 300s default\n// after: tolerate 10 minutes of skew on slow-syncing nodes\nopt.MaxTimeSkew = 600","handlingStrategy":"validation","validationCode":"// Pre-flight check on the node: local clock must be within MaxTimeSkew of a trusted source\nfunc clockWithinSkew(maxSkewSecs int64) bool {\n\tnow := time.Now().Unix()\n\t// compare against NTP/chrony-tracked time or a controller-provided Date header\n\tref := time.Now().Unix() // replace with trusted time source in real use\n\treturn abs64(now-ref) <= maxSkewSecs\n}\n\nfunc abs64(x int64) int64 { if x < 0 { return -x }; return x }","typeGuard":"func timestampWithinSkew(ts int64, maxSkew time.Duration) bool {\n\tskew := time.Since(time.Unix(ts, 0))\n\tif skew < 0 {\n\t\tskew = -skew\n\t}\n\treturn skew <= maxSkew\n}","tryCatchPattern":"result, err := verifier.VerifyToken(ctx, req, authToken, body)\nif err != nil {\n\tvar tsErr timestampError // wrap errors in typed values to enable errors.As\n\tif errors.As(err, &tsErr) {\n\t\t// clock drift: sync time (systemd-timesyncd/chrony) and re-mint a fresh token, then retry once\n\t\treturn retryWithFreshToken(ctx, req)\n\t}\n\treturn nil, err\n}","preventionTips":["Run NTP/chrony on every node and on the kops-controller host; alert on clock drift.","Always mint a fresh token immediately before each bootstrap request; never cache tokens.","Set MaxTimeSkew explicitly in the controller options if your environment has known slow clock convergence.","Monitor for timestamp errors in kops-controller logs as an early sign of NTP failure."],"tags":["go","authentication","pki","clock-skew","ntp"],"backgroundTag":"clock-skew-token-expired","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}