kubernetes/kops · error

fail to stop DO instance %v in 5 mins

Error message

fail to stop DO instance %v in 5 mins

What it means

DeleteInstance waits up to 5 one-minute iterations for the droplet's status to become 'off' after issuing Shutdown; if the final iteration still shows the droplet running, kOps returns 'fail to stop DO instance <id> in 5 mins'. The droplet is left running and is not deleted, so the rolling/delete operation aborts.

Source

Thrown at upup/pkg/fi/cloudup/do/cloud.go:158

	if err != nil {
		return fmt.Errorf("error stopping instance %d: %v", dropletID, err)
	}

	// Wait for 5 min to stop the instance
	for i := 0; i < 5; i++ {
		droplet, _, err := c.Client.Droplets.Get(context.TODO(), dropletID)
		if err != nil {
			return fmt.Errorf("error describing instance %d: %v", dropletID, err)
		}

		klog.V(8).Infof("stopping DO instance %q, current Status: %q", droplet, droplet.Status)

		if droplet.Status == "off" {
			break
		}

		if i == 5 {
			return fmt.Errorf("fail to stop DO instance %v in 5 mins", dropletID)
		}

		time.Sleep(time.Minute * 1)
	}

	_, err = c.Client.Droplets.Delete(context.TODO(), dropletID)
	if err != nil {
		return fmt.Errorf("error stopping instance %d: %v", dropletID, err)
	}

	klog.V(8).Infof("deleted droplet instance %d", dropletID)

	return nil
}

// DetachInstance is not implemented yet. It needs to cause a cloud instance to no longer be counted against the group's size limits.
func (c *doCloudImplementation) DetachInstance(i *cloudinstances.CloudInstance) error {
	klog.V(8).Info("digitalocean cloud provider DetachInstance not implemented yet")

View on GitHub (pinned to 4c8573c808)

Solutions

1. Manually force power-off the droplet (doctl compute droplet-action power-off <id> or DO console) then re-run the kops operation
2. Increase patience by deleting in stages: drain the node, then retry delete when the droplet is already off
3. SSH into the droplet to investigate the stuck shutdown (journalctl, hung NFS mounts)
4. As a last resort, delete the droplet directly in DO and resync kOps state with `kops update cluster`

Example fix

// before
# node stuck shutting down > 5 min
// after
dshutdown ... # or:
doctl compute droplet-action power-off <droplet-id>
# wait for status off, then:
kops rolling-update cluster --name cluster.example.com --yes

// pre-drain and confirm the node is quiescent before deleting
cmd := exec.Command("kubectl", "drain", nodeName, "--ignore-daemonsets", "--delete-emptydir-data")
if err := cmd.Run(); err != nil {
    return fmt.Errorf("drain failed; shutdown would likely time out")
}

if stopped := waitForOff(dropletID, 5*time.Minute); !stopped {
    // fallback: force power-off then delete
    client.DropletActions.PowerOff(ctx, dropletID)
    waitForOff(dropletID, 2*time.Minute)
    client.Droplets.Delete(ctx, dropletID)
}

When it happens

Trigger: DropletActions.Shutdown was accepted but the droplet does not reach status 'off' within 5 polling iterations — hung OS shutdown, slow ACPI shutdown on the image, droplet stuck in a transitional state, or the Shutdown action silently no-opped. (Note the loop checks `i == 5` which is never reached since i stops at 4, so termination actually depends on the loop ending without 'off'.)

Common situations: Large nodes with slow filesystem unmount taking >5 minutes; guest agent not responding to ACPI power button; droplet with stuck processes; Kubernetes node refusing to drain/shut down cleanly.

Related errors

• error while waiting for volume %s to detach: %s
• error stopping instance %d: %v
• error describing instance %d: %v
• DIGITALOCEAN_ACCESS_TOKEN is required
• DIGITALOCEAN_ACCESS_TOKEN is required

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/74f88f3c3342f3c9. Report an issue: GitHub.