{"record":{"id":"d5f17fcf576bea29","repo":"kubernetes/kops","slug":"error-cordoning-node-v","errorCode":null,"errorMessage":"error cordoning node: %v","messagePattern":"error cordoning node: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/instancegroups/instancegroups.go","lineNumber":704,"sourceCode":"\t\tForce:               true,\n\t\tGracePeriodSeconds:  -1,\n\t\tIgnoreAllDaemonSets: true,\n\t\tOut:                 os.Stdout,\n\t\tErrOut:              os.Stderr,\n\t\tTimeout:             c.DrainTimeout,\n\n\t\t// The zero value would retry evictions without any delay\n\t\tEvictErrorRetryDelay: 5 * time.Second,\n\n\t\t// We want to proceed even when pods are using emptyDir volumes\n\t\tDeleteEmptyDirData: true,\n\t}\n\n\tif err := drain.RunCordonOrUncordon(helper, u.Node, true); err != nil {\n\t\tif apierrors.IsNotFound(err) {\n\t\t\treturn nil\n\t\t}\n\t\treturn fmt.Errorf(\"error cordoning node: %v\", err)\n\t}\n\n\tif err := c.patchExcludeFromLB(ctx, u.Node); err != nil {\n\t\tif apierrors.IsNotFound(err) {\n\t\t\treturn nil\n\t\t}\n\t\treturn fmt.Errorf(\"error excluding node from load balancer: %v\", err)\n\t}\n\n\tshouldDeregister := true\n\tif !c.Options.DeregisterControlPlaneNodes {\n\t\tif u.CloudInstanceGroup != nil && u.CloudInstanceGroup.InstanceGroup != nil {\n\t\t\trole := u.CloudInstanceGroup.InstanceGroup.Spec.Role\n\t\t\tswitch role {\n\t\t\tcase api.InstanceGroupRoleAPIServer, api.InstanceGroupRoleControlPlane:\n\t\t\t\tklog.Infof(\"skipping deregistration of instance %q, as part of instancegroup with role %q\", u.ID, role)\n\t\t\t\tshouldDeregister = false\n\t\t\t}","sourceCodeStart":686,"sourceCodeEnd":722,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/pkg/instancegroups/instancegroups.go#L686-L722","documentation":"After marking the node with a cordon (drain.RunCordonOrUncordon with wantUncordon=false), drainNode wraps any non-NotFound error as 'error cordoning node'. This means the k8s API rejected or failed the PATCH that sets spec.unschedulable=true on the Node.","triggerScenarios":"The cordon PATCH fails with any error other than NotFound: RBAC denial on nodes update, API server unreachable, conflict/timeout on the node object, or a stale Node object that was deleted mid-request.","commonSituations":"kOps service account lacking clusterrole permissions for nodes during `kops rolling-update cluster`; API server briefly unavailable while an LB is being switched; concurrent controllers fighting over node spec.","solutions":["Confirm the credentials used by kOps have RBAC rights to patch nodes (system:node / cluster-admin style binding)","Re-run the rolling update — transient API errors are retriable and NotFound is treated as success","Check `kubectl get node <name> -o yaml` to confirm the node exists and inspect its conditions","Verify connectivity to the cluster API endpoint from the machine running kOps"],"exampleFix":"// before\nkubectl auth can-i patch nodes --as=system:serviceaccount:kube-system:kops\n// after (grant RBAC)\nkubectl create clusterrolebinding kops-rolling --clusterrole=cluster-admin --user=<kops-user>","handlingStrategy":"try-catch","validationCode":"if !rbac.CanIPatch(\"nodes\") {\n    return errors.New(\"credentials cannot patch nodes; cordon will fail\")\n}","typeGuard":null,"tryCatchPattern":"err := c.drainNode(ctx, u)\nvar notFound bool\nif apierrors.IsNotFound(errors.Unwrap(err)) {\n    notFound = true\n}\nif err != nil && !notFound {\n    klog.Warningf(\"cordon failed (%v); retrying once\", err)\n    err = c.drainNode(ctx, u)\n}","preventionTips":["Ensure kOps credentials have node patch RBAC","Confirm API server reachability before updating","Keep node objects free of conflicting controllers"],"tags":["kubernetes","cordon","rbac","rolling-update"],"backgroundTag":"kubernetes-api-request-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}