{"record":{"id":"2d520a833d11f349","repo":"kubernetes/kops","slug":"cluster-did-not-validate-within-a-duration-of-q","errorCode":null,"errorMessage":"cluster did not validate within a duration of %q","messagePattern":"cluster did not validate within a duration of %q","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/instancegroups/instancegroups.go","lineNumber":600,"sourceCode":"\t\t\tfor _, failure := range result.Failures {\n\t\t\t\tmessages = append(messages, failure.Message)\n\t\t\t}\n\t\t\tif ctx.Err() != nil {\n\t\t\t\tklog.Infof(\"Cluster did not pass validation within deadline: %s.\", strings.Join(messages, \", \"))\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tklog.Infof(\"Cluster did not pass validation, will retry in %q: %s.\", c.ValidateTickDuration, strings.Join(messages, \", \"))\n\t\t}\n\n\t\t// Reset the success count; we want N consecutive successful validations\n\t\tsuccessCount = 0\n\n\t\t// Wait before retrying in some cases\n\t\t// TODO: Should we check if we have enough time left before the deadline?\n\t\ttime.Sleep(c.ValidateTickDuration)\n\t}\n\n\treturn fmt.Errorf(\"cluster did not validate within a duration of %q\", c.ValidationTimeout)\n}\n\n// checks if the validation failures returned after cluster validation are relevant to the current\n// instance group whose rolling update is occurring\nfunc hasFailureRelevantToGroup(failures []*validation.ValidationError, group *cloudinstances.CloudInstanceGroup) bool {\n\t// Ignore non critical validation errors in other instance groups like below target size errors\n\tfor _, failure := range failures {\n\t\t// Certain failures like a system-critical-pod failure and dns server related failures\n\t\t// set their InstanceGroup to nil, since we cannot associate the failure to any one group\n\t\tif failure.InstanceGroup == nil {\n\t\t\treturn true\n\t\t}\n\n\t\t// if there is a failure in the same instance group or a failure which has cluster wide impact\n\t\tif (failure.InstanceGroup.IsControlPlane()) || (failure.InstanceGroup == group.InstanceGroup) {\n\t\t\treturn true\n\t\t}\n\t}","sourceCodeStart":582,"sourceCodeEnd":618,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/pkg/instancegroups/instancegroups.go#L582-L618","documentation":"kOps validates cluster health after each rolling-update step, retrying every ValidateTickDuration until ValidationTimeout expires. This error means validation never fully passed (or validation failures relevant to the group persisted) within that window, so the rolling update aborts with a timeout.","triggerScenarios":"maybeValidate loops calling validation until the deadline; every attempt returns failures relevant to the current instance group (hasFailureRelevantToGroup) or the cluster stays unhealthy for longer than ValidationTimeout (default ~15m).","commonSituations":"Nodes not rejoining after replacement (networking/CNI issues); new instances failing to register; DNS/etcd slow to converge; pods stuck Pending due to capacity; too-short --validation-timeout for large clusters.","solutions":["Increase --validation-timeout (e.g. kops rolling-update cluster --validation-timeout 30m)","Check why new nodes are unhealthy: kubectl get nodes, pod events, node up/register logs","Fix validation-blocking components (CNI, DNS, etcd health)","Re-run the rolling update after the cluster stabilizes; kOps resumes safely"],"exampleFix":"// before\nkops rolling-update cluster mycluster --yes            # 15m default timeout\n// after\nkops rolling-update cluster mycluster --yes --validation-timeout 30m","handlingStrategy":"retry","validationCode":"// before the roll, ensure the cluster validates at all\nvr, err := validation.ValidateCluster(ctx, cluster, nil)\nif err != nil || len(vr.Failures) > 0 { /* fix cluster before rolling */ }","typeGuard":"func hasBlockingFailure(failures []*validation.ValidationError, group *cloudinstances.CloudInstanceGroup) bool {\n    return hasFailureRelevantToGroup(failures, group)\n}","tryCatchPattern":"if err := c.maybeValidate(ctx, group, sleepDuration); err != nil {\n    if strings.Contains(err.Error(), \"did not validate within\") {\n        // log state, re-validate manually, then retry the roll\n    }\n    return err\n}","preventionTips":["Size --validation-timeout to cluster size (larger clusters need more)","Fix node-up/CNI/DNS issues before rolling; validation only reports symptoms","Monitor new nodes registering after each replacement","Run validation dry-runs to catch pre-existing unhealthy state"],"tags":["validation","timeout","rolling-update"],"backgroundTag":"cluster-validation-timeout","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}