{"record":{"id":"74f88f3c3342f3c9","repo":"kubernetes/kops","slug":"fail-to-stop-do-instance-v-in-5-mins","errorCode":null,"errorMessage":"fail to stop DO instance %v in 5 mins","messagePattern":"fail to stop DO instance (.+?) in 5 mins","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"upup/pkg/fi/cloudup/do/cloud.go","lineNumber":158,"sourceCode":"\tif err != nil {\n\t\treturn fmt.Errorf(\"error stopping instance %d: %v\", dropletID, err)\n\t}\n\n\t// Wait for 5 min to stop the instance\n\tfor i := 0; i < 5; i++ {\n\t\tdroplet, _, err := c.Client.Droplets.Get(context.TODO(), dropletID)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"error describing instance %d: %v\", dropletID, err)\n\t\t}\n\n\t\tklog.V(8).Infof(\"stopping DO instance %q, current Status: %q\", droplet, droplet.Status)\n\n\t\tif droplet.Status == \"off\" {\n\t\t\tbreak\n\t\t}\n\n\t\tif i == 5 {\n\t\t\treturn fmt.Errorf(\"fail to stop DO instance %v in 5 mins\", dropletID)\n\t\t}\n\n\t\ttime.Sleep(time.Minute * 1)\n\t}\n\n\t_, err = c.Client.Droplets.Delete(context.TODO(), dropletID)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"error stopping instance %d: %v\", dropletID, err)\n\t}\n\n\tklog.V(8).Infof(\"deleted droplet instance %d\", dropletID)\n\n\treturn nil\n}\n\n// DetachInstance is not implemented yet. It needs to cause a cloud instance to no longer be counted against the group's size limits.\nfunc (c *doCloudImplementation) DetachInstance(i *cloudinstances.CloudInstance) error {\n\tklog.V(8).Info(\"digitalocean cloud provider DetachInstance not implemented yet\")","sourceCodeStart":140,"sourceCodeEnd":176,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/upup/pkg/fi/cloudup/do/cloud.go#L140-L176","documentation":"DeleteInstance waits up to 5 one-minute iterations for the droplet's status to become 'off' after issuing Shutdown; if the final iteration still shows the droplet running, kOps returns 'fail to stop DO instance <id> in 5 mins'. The droplet is left running and is not deleted, so the rolling/delete operation aborts.","triggerScenarios":"DropletActions.Shutdown was accepted but the droplet does not reach status 'off' within 5 polling iterations — hung OS shutdown, slow ACPI shutdown on the image, droplet stuck in a transitional state, or the Shutdown action silently no-opped. (Note the loop checks `i == 5` which is never reached since i stops at 4, so termination actually depends on the loop ending without 'off'.)","commonSituations":"Large nodes with slow filesystem unmount taking >5 minutes; guest agent not responding to ACPI power button; droplet with stuck processes; Kubernetes node refusing to drain/shut down cleanly.","solutions":["Manually force power-off the droplet (doctl compute droplet-action power-off <id> or DO console) then re-run the kops operation","Increase patience by deleting in stages: drain the node, then retry delete when the droplet is already off","SSH into the droplet to investigate the stuck shutdown (journalctl, hung NFS mounts)","As a last resort, delete the droplet directly in DO and resync kOps state with `kops update cluster`"],"exampleFix":"// before\n# node stuck shutting down > 5 min\n// after\ndshutdown ... # or:\ndoctl compute droplet-action power-off <droplet-id>\n# wait for status off, then:\nkops rolling-update cluster --name cluster.example.com --yes","handlingStrategy":"fallback","validationCode":"// pre-drain and confirm the node is quiescent before deleting\ncmd := exec.Command(\"kubectl\", \"drain\", nodeName, \"--ignore-daemonsets\", \"--delete-emptydir-data\")\nif err := cmd.Run(); err != nil {\n    return fmt.Errorf(\"drain failed; shutdown would likely time out\")\n}","typeGuard":null,"tryCatchPattern":"if stopped := waitForOff(dropletID, 5*time.Minute); !stopped {\n    // fallback: force power-off then delete\n    client.DropletActions.PowerOff(ctx, dropletID)\n    waitForOff(dropletID, 2*time.Minute)\n    client.Droplets.Delete(ctx, dropletID)\n}","preventionTips":["Drain the Kubernetes node before deleting so shutdown completes fast","Force power-off stuck droplets via doctl and retry","Watch for hung mounts (NFS/EBS-like) that stall guest shutdown","Increase the wait budget for large or heavily loaded nodes"],"tags":["digitalocean","timeout","instance-lifecycle"],"backgroundTag":"instance-shutdown-timeout","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T12:17:11.808Z"}