node heartbeat timeout plus reclaim grace must exceed CPA he

Error message

node heartbeat timeout plus reclaim grace must exceed CPA heartbeat timeout plus cancel bound

What it means

The core timing-safety invariant of the credential concurrency lifecycle: nodeHeartbeatTimeout + reclaim-grace must be strictly greater than cpa-heartbeat-timeout + cpa-cancel-bound. It guarantees a node's lease cannot be reclaimed (and its credentials reused) while a CPA-side cancellation is still possibly in flight, preventing double-use of a credential. Violating the inequality rejects the config.

Source

Thrown at internal/config/credential_concurrency.go:184

	}
	return nil
}

// ValidateCredentialConcurrencyLifecycle verifies the Home lifecycle timing safety invariant.
func ValidateCredentialConcurrencyLifecycle(nodeHeartbeatTimeout time.Duration, cfg CredentialConcurrencyConfig) error {
	if nodeHeartbeatTimeout <= 0 {
		return fmt.Errorf("credential concurrency lifecycle durations must be positive")
	}
	if errValidate := ValidateCredentialConcurrency(cfg); errValidate != nil {
		return errValidate
	}
	left, leftOverflow := addCredentialConcurrencyDuration(nodeHeartbeatTimeout, cfg.ReclaimGrace)
	right, rightOverflow := addCredentialConcurrencyDuration(cfg.CPAHeartbeatTimeout, cfg.CPACancelBound)
	if leftOverflow || rightOverflow {
		return fmt.Errorf("credential concurrency lifecycle timing safety invariant overflows")
	}
	if left <= right {
		return fmt.Errorf("node heartbeat timeout plus reclaim grace must exceed CPA heartbeat timeout plus cancel bound")
	}
	return nil
}

func addCredentialConcurrencyDuration(left time.Duration, right time.Duration) (time.Duration, bool) {
	if right > 0 && left > time.Duration(1<<63-1)-right {
		return 0, true
	}
	return left + right, false
}

View on GitHub (pinned to 78f0c4079e)

Solutions

Increase the node heartbeat timeout and/or reclaim-grace so their sum exceeds cpa-heartbeat-timeout + cpa-cancel-bound with margin.
Or reduce cpa-heartbeat-timeout / cpa-cancel-bound.
A safe starting point: node timeout 30s, reclaim-grace 5s, cpa-heartbeat-timeout 3s, cpa-cancel-bound 5s.
Re-run validation after each change until the inequality holds.

Example fix

# before (config.yaml)
credential-concurrency:
  cpa-heartbeat-timeout: 10s
  cpa-cancel-bound: 5s
  reclaim-grace: 1s
# nodeHeartbeatTimeout=5s -> 5+1=6 <= 10+5=15 (fails)

# after
credential-concurrency:
  cpa-heartbeat-timeout: 3s
  cpa-cancel-bound: 5s
  reclaim-grace: 5s
# nodeHeartbeatTimeout=30s -> 30+5=35 > 3+5=8 (passes)

Defensive patterns

Strategy: validation

Validate before calling

// Go: check the invariant before applying config.
func timingInvariantOK(nodeHeartbeatTimeout time.Duration, c config.CredentialConcurrencyConfig) bool {
    return nodeHeartbeatTimeout+c.ReclaimGrace > c.CPAHeartbeatTimeout+c.CPACancelBound
}

Prevention

Whenever you change any of the four timings, recompute both sums and keep left > right with margin.
Start from defaults (node 30s, grace 5s vs CPA 3s + 5s) and change one value at a time.
Add the invariant to your config-change checklist.

When it happens

Trigger: config.yaml where nodeHeartbeatTimeout + reclaim-grace <= cpa-heartbeat-timeout + cpa-cancel-bound, e.g. node timeout 10s + reclaim-grace 5s = 15s vs cpa-heartbeat-timeout 20ms + cpa-cancel-bound 5s = 5.02s passes; but node timeout 2s + grace 1s = 3s vs cpa timeout 2s + cancel bound 5s = 7s fails. Typical failing case: large cpa-cancel-bound with a small node heartbeat timeout.

Common situations: Raising cpa-heartbeat-timeout or cpa-cancel-bound for slow networks without raising the node heartbeat timeout; shrinking reclaim-grace to reclaim credentials faster; embedding the SDK and choosing a small nodeHeartbeatTimeout.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of router-for-me/CLIProxyAPI@78f0c4079e (2026-08-15). Data as JSON: /api/errors/c738591828f30e0b. Report an issue: GitHub.