kubernetes/kops · error

cluster did not validate within a duration of %q

Error message

cluster did not validate within a duration of %q

What it means

kOps validates cluster health after each rolling-update step, retrying every ValidateTickDuration until ValidationTimeout expires. This error means validation never fully passed (or validation failures relevant to the group persisted) within that window, so the rolling update aborts with a timeout.

Source

Thrown at pkg/instancegroups/instancegroups.go:600

			for _, failure := range result.Failures {
				messages = append(messages, failure.Message)
			}
			if ctx.Err() != nil {
				klog.Infof("Cluster did not pass validation within deadline: %s.", strings.Join(messages, ", "))
				break
			}
			klog.Infof("Cluster did not pass validation, will retry in %q: %s.", c.ValidateTickDuration, strings.Join(messages, ", "))
		}

		// Reset the success count; we want N consecutive successful validations
		successCount = 0

		// Wait before retrying in some cases
		// TODO: Should we check if we have enough time left before the deadline?
		time.Sleep(c.ValidateTickDuration)
	}

	return fmt.Errorf("cluster did not validate within a duration of %q", c.ValidationTimeout)
}

// checks if the validation failures returned after cluster validation are relevant to the current
// instance group whose rolling update is occurring
func hasFailureRelevantToGroup(failures []*validation.ValidationError, group *cloudinstances.CloudInstanceGroup) bool {
	// Ignore non critical validation errors in other instance groups like below target size errors
	for _, failure := range failures {
		// Certain failures like a system-critical-pod failure and dns server related failures
		// set their InstanceGroup to nil, since we cannot associate the failure to any one group
		if failure.InstanceGroup == nil {
			return true
		}

		// if there is a failure in the same instance group or a failure which has cluster wide impact
		if (failure.InstanceGroup.IsControlPlane()) || (failure.InstanceGroup == group.InstanceGroup) {
			return true
		}
	}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Increase --validation-timeout (e.g. kops rolling-update cluster --validation-timeout 30m)
2. Check why new nodes are unhealthy: kubectl get nodes, pod events, node up/register logs
3. Fix validation-blocking components (CNI, DNS, etcd health)
4. Re-run the rolling update after the cluster stabilizes; kOps resumes safely

Example fix

// before
kops rolling-update cluster mycluster --yes            # 15m default timeout
// after
kops rolling-update cluster mycluster --yes --validation-timeout 30m

// before the roll, ensure the cluster validates at all
vr, err := validation.ValidateCluster(ctx, cluster, nil)
if err != nil || len(vr.Failures) > 0 { /* fix cluster before rolling */ }

func hasBlockingFailure(failures []*validation.ValidationError, group *cloudinstances.CloudInstanceGroup) bool {
    return hasFailureRelevantToGroup(failures, group)
}

if err := c.maybeValidate(ctx, group, sleepDuration); err != nil {
    if strings.Contains(err.Error(), "did not validate within") {
        // log state, re-validate manually, then retry the roll
    }
    return err
}

When it happens

Trigger: maybeValidate loops calling validation until the deadline; every attempt returns failures relevant to the current instance group (hasFailureRelevantToGroup) or the cluster stays unhealthy for longer than ValidationTimeout (default ~15m).

Common situations: Nodes not rejoining after replacement (networking/CNI issues); new instances failing to register; DNS/etcd slow to converge; pods stuck Pending due to capacity; too-short --validation-timeout for large clusters.

Related errors

• cannot create cluster validator: %v
• waiting for kubernetes API to be served: %w
• wait time exceeded during validation
• cluster passed validation %d consecutive times
• bastion not healthy after update, stopping rolling-update: %

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/2d520a833d11f349. Report an issue: GitHub.