hashicorp/nomad · error

can not set node's scheduling eligibility to eligible while

Error message

can not set node's scheduling eligibility to eligible while it is draining

What it means

updateNodeEligibilityImpl rejects setting a node's SchedulingEligibility to 'eligible' while the node still has a DrainStrategy set — a draining node must not accept new allocations. This is a domain invariant check, not an infrastructure failure.

Source

Thrown at nomad/state/state_store.go:1330

		return fmt.Errorf("node lookup failed: %v", err)
	}
	if existing == nil {
		return fmt.Errorf("node not found")
	}

	// Copy the existing node
	existingNode := existing.(*structs.Node)
	copyNode := existingNode.Copy()
	copyNode.StatusUpdatedAt = updatedAt

	// Add the event if given
	if event != nil {
		appendNodeEvents(index, copyNode, []*structs.NodeEvent{event})
	}

	// Check if this is a valid action
	if copyNode.DrainStrategy != nil && eligibility == structs.NodeSchedulingEligible {
		return fmt.Errorf("can not set node's scheduling eligibility to eligible while it is draining")
	}

	// Update the eligibility in the copy
	copyNode.SchedulingEligibility = eligibility
	copyNode.ModifyIndex = index

	// Insert the node
	if err := txn.Insert("nodes", copyNode); err != nil {
		return fmt.Errorf("node update failed: %v", err)
	}
	if err := txn.Insert("index", &IndexEntry{"nodes", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return nil
}

// UpsertNodeEvents adds the node events to the nodes, rotating events as

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Complete or cancel the drain first (UpdateNodeDrain with drain==nil to remove DrainStrategy), then set eligibility.
2. Wait for the drain to finish — poll the node until DrainStrategy is cleared — before setting eligible.
3. Send a combined drain-stop request with markEligible=true instead of separate calls.
4. Check node.DrainStrategy != nil in the client before issuing the eligibility change.

Example fix

// before
store.UpdateNodeEligibility(idx, nodeID, structs.NodeSchedulingEligible) // fails: still draining
// after
// stop the drain first; markEligible=true re-enables scheduling in one transaction
err := store.UpdateNodeDrain(idx, nodeID, nil, true)

node, err := store.NodeByID(nil, nodeID)
if err != nil {
    return err
}
if node.DrainStrategy != nil && eligibility == structs.NodeSchedulingEligible {
    return fmt.Errorf("node %s is draining; complete or cancel the drain before setting eligible", nodeID)
}

func canSetEligible(n *structs.Node) bool {
    return n != nil && n.DrainStrategy == nil
}

if err := store.UpdateNodeEligibility(idx, nodeID, structs.NodeSchedulingEligible); err != nil {
    if strings.Contains(err.Error(), "while it is draining") {
        // cancel the drain first, then retry
        return store.UpdateNodeDrain(idx, nodeID, nil, true)
    }
    return err
}

When it happens

Trigger: Calling UpdateNodeEligibility(nodeID, NodeSchedulingEligible) on a node whose DrainStrategy is non-nil; a drain-stop request that only marks eligible without clearing the drain strategy (drain==nil but strategy already set); UpsertPlanResults attempting to flip eligibility on a draining node.

Common situations: Operators clicking 'set eligible' on a node mid-drain in the Nomad UI/CLI; automation that sets eligible=true before the drain completes; API users unaware drain must complete or be cancelled first.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Deployment %q has terminal status %q:
• error parsing: root should be an object
• cannot specify Accessor ID
• network already configured but not found in state
• eval not found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c1c3f71835eae047. Report an issue: GitHub.