kubernetes/kops · error

incorrect Timestamp %v

Error message

incorrect Timestamp %v

What it means

The token's Timestamp claim is more than MaxTimeSkew seconds (default 300) away from kops-controller's clock, so the verifier rejects it as a replay-protection measure. Tokens are single-use within a short window; the message includes the offending unix timestamp. math.Abs is used, so both too-old and too-far-in-the-future timestamps trigger it.

Source

Thrown at pkg/bootstrap/pkibootstrap/pkiverifier/verifier.go:93

	}

	token := &pkibootstrap.AuthToken{}
	if err = json.Unmarshal(tokenBytes, token); err != nil {
		return nil, nil, fmt.Errorf("unmarshalling authorization token: %w", err)
	}

	tokenData := &pkibootstrap.AuthTokenData{}
	if err := json.Unmarshal(token.Data, tokenData); err != nil {
		return nil, nil, fmt.Errorf("unmarshalling authorization token data: %w", err)
	}

	// Guard against replay attacks
	if tokenData.Audience != pkibootstrap.AudienceNodeAuthentication {
		return nil, nil, fmt.Errorf("incorrect Audience")
	}
	timeSkew := math.Abs(time.Since(time.Unix(tokenData.Timestamp, 0)).Seconds())
	if timeSkew > float64(v.opt.MaxTimeSkew) {
		return nil, nil, fmt.Errorf("incorrect Timestamp %v", tokenData.Timestamp)
	}

	// Verify the token has signed the body content.
	requestHash := sha256.Sum256(body)
	if !bytes.Equal(requestHash[:], tokenData.RequestHash) {
		return nil, nil, fmt.Errorf("incorrect RequestHash")
	}

	return token, tokenData, nil
}

// Can generate keys with
// openssl ecparam -name prime256v1 -genkey -noout -out ec-priv-key.pem
// openssl ec -in ec-priv-key.pem -pubout > ec-pub-key.pem
// Note that golang doesn't support secp256k1: https://groups.google.com/g/golang-nuts/c/Mbkug5t3ZYA

func (v *verifier) VerifyToken(ctx context.Context, rawRequest *http.Request, authToken string, body []byte) (*bootstrap.VerifyResult, error) {
	// Reminder: we shouldn't trust any data we get from the client until we've checked the signature (and even then...)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Enable and verify NTP/chrony time synchronization on the node (and on the kops-controller host); restart the bootstrap request after the clock converges.
2. Check the node's clock against the controller: compare `date -u` on both hosts; skew must be under MaxTimeSkew (default 300s).
3. If legitimate skew is unavoidable, raise the MaxTimeSkew option passed to pkiverifier.NewVerifier (it defaults to 300 when 0).
4. Ensure a fresh token is minted per request — pkiAuthenticator.CreateToken stamps time.Now().Unix(); never cache or reuse tokens.
5. If the timestamp is wildly wrong (e.g. epoch 0), fix the client minting code to set Timestamp: time.Now().Unix().

Example fix

// before: kops-controller options
opt.MaxTimeSkew = 0 // falls back to 300s default
// after: tolerate 10 minutes of skew on slow-syncing nodes
opt.MaxTimeSkew = 600

// Pre-flight check on the node: local clock must be within MaxTimeSkew of a trusted source
func clockWithinSkew(maxSkewSecs int64) bool {
	now := time.Now().Unix()
	// compare against NTP/chrony-tracked time or a controller-provided Date header
	ref := time.Now().Unix() // replace with trusted time source in real use
	return abs64(now-ref) <= maxSkewSecs
}

func abs64(x int64) int64 { if x < 0 { return -x }; return x }

func timestampWithinSkew(ts int64, maxSkew time.Duration) bool {
	skew := time.Since(time.Unix(ts, 0))
	if skew < 0 {
		skew = -skew
	}
	return skew <= maxSkew
}

result, err := verifier.VerifyToken(ctx, req, authToken, body)
if err != nil {
	var tsErr timestampError // wrap errors in typed values to enable errors.As
	if errors.As(err, &tsErr) {
		// clock drift: sync time (systemd-timesyncd/chrony) and re-mint a fresh token, then retry once
		return retryWithFreshToken(ctx, req)
	}
	return nil, err
}

When it happens

Trigger: parseTokenData (verifier.go:91-94) raises this when math.Abs(time.Since(time.Unix(tokenData.Timestamp,0)).Seconds()) > v.opt.MaxTimeSkew: node clock skew (NTP not running), a replayed/captured token reused after the window, a clock that jumped after VM migration/suspend, or kops-controller host clock drift in the opposite direction.

Common situations: Newly provisioned VM whose clock has not yet synced via NTP/chrony (very common right at node bootstrap time); suspended/resumed nodes; cloud regions or on-prem hosts without a time sync daemon; clock set to UTC vs local confusion in custom token minting; deliberately replayed tokens caught by the guard.

Related errors

• unmarshalling authorization token data: %w
• incorrect Audience
• incorrect RequestHash
• failed to verify claim signature for node
• unable to SSH to %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/96fc9461172e6a08. Report an issue: GitHub.