hashicorp/nomad · error

could not detach from node: %w

Error message

could not detach from node: %w

What it means

nodeUnpublishVolumeImpl failed to run the controller detach RPC against the client node and wraps the error as "could not detach from node: <err>". Notably the code deliberately skips this wrap when the error is ErrUnknownNode (checked by string match because RPC breaks error wrapping), treating a garbage-collected node as "already detached". Any other failure reaching the node/plugin during detach surfaces here.

Source

Thrown at nomad/csi_endpoint.go:872

		VolumeID:        vol.ID,
		VolumeNamespace: vol.Namespace,
		ExternalID:      vol.RemoteID(),
		AllocID:         claim.AllocationID,
		NodeID:          claim.NodeID,
		AttachmentMode:  claim.AttachmentMode,
		AccessMode:      claim.AccessMode,
		ReadOnly:        claim.Mode == structs.CSIVolumeClaimRead,
	}
	err := v.srv.RPC("ClientCSI.NodeDetachVolume",
		req, &cstructs.ClientCSINodeDetachVolumeResponse{})
	if err != nil {
		// we should only get this error if the Nomad node disconnects and
		// is garbage-collected, so at this point we don't have any reason
		// to operate as though the volume is attached to it.
		// note: errors.Is cannot be used because the RPC call breaks
		// error wrapping.
		if !strings.Contains(err.Error(), structs.ErrUnknownNode.Error()) {
			return fmt.Errorf("could not detach from node: %w", err)
		}
	}
	return nil
}

// controllerUnpublishVolume handles the sending RPCs to the Controller plugin
// to unpublish the volume (detach it from its host). This function should only
// be called on a copy of the volume.
func (v *CSIVolume) controllerUnpublishVolume(vol *structs.CSIVolume, claim *structs.CSIVolumeClaim) error {
	v.logger.Trace("controller unpublish", "vol", vol.ID)

	if !vol.ControllerRequired {
		claim.State = structs.CSIVolumeClaimStateReadyToFree
		return nil
	}

	// We need a new snapshot after each checkpoint
	snap, err := v.srv.fsm.State().Snapshot()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the client node and controller plugin health: `nomad node status` and `nomad plugin status <plugin-id>`; restart the plugin job if unhealthy.
2. Inspect the wrapped inner error (text after "could not detach from node:") for the plugin's specific failure and address that (credentials, backend API errors, timeouts).
3. Retry the unpublish — controller detach is designed to be re-run; Nomad checkpoints past-claim state so retries are safe.
4. If the node is permanently gone, wait for it to be GC'd: once ErrUnknownNode would be returned, unpublish treats it as detached and succeeds.

Example fix

// before: treating every unpublish error as fatal
if err := client.CSIVolumes().Unpublish(...); err != nil { return err }
// after: retry transient detach failures with backoff
if err := client.CSIVolumes().Unpublish(...); err != nil {
    if strings.Contains(err.Error(), "could not detach from node") {
        return backoff.Retry(unpublish, 3) // node/plugin may recover
    }
    return err
}

node, _, err := client.Nodes().Info(nodeID, nil)
if err == nil && node != nil && node.Status != "ready" {
    // node down: expect detach failures; wait for recovery or GC
}

if err != nil && strings.Contains(err.Error(), "could not detach from node") {
    // transient (node/plugin unreachable): retry with backoff
    // note: ErrUnknownNode cases already return nil server-side
    backoff.Retry(unpublishFunc, expBackoff)
}

When it happens

Trigger: Raised when v.controllerUnpublishVolume / the client RPC fails with an error that does NOT contain structs.ErrUnknownNode: the client node is unreachable, the CSI controller plugin isn't running or errored, the plugin returned a storage-provider failure, or an RPC timeout occurred during ClientCSI.ControllerDetachVolume.

Common situations: Client node crashed or network-partitioned mid-unpublish; the CSI controller plugin task is unhealthy so detach RPCs fail; the storage backend (EBS, Ceph, etc.) rejects the detach because the volume isn't attached or credentials are wrong; timeouts from slow cloud APIs under load.

Related errors

• controller detach volume: %v
• could not validate task driver capabilities: %v
• CSI.ControllerDetachVolume: %w: %v (wraps ErrCSIClientRPCRet
• controller attach volume: %v
• controller validate volume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f4a7561611af38ab. Report an issue: GitHub.