hashicorp/nomad · error

node secret ID does not match. Not registering node.

Error message

node secret ID does not match. Not registering node.

What it means

Nomad's node Register endpoint rejects registration when the SecretID sent in the request differs from the SecretID already stored for that node ID in state store. This guards against a node ID being hijacked or reused with different credentials. It is a server-side tamper/integrity check on node identity.

Source

Thrown at nomad/node_endpoint.go:184

	snap, err := n.srv.fsm.State().Snapshot()
	if err != nil {
		return err
	}

	ws := memdb.NewWatchSet()
	originalNode, err := snap.NodeByID(ws, args.Node.ID)
	if err != nil {
		return err
	}

	// If the node has an entry in the state store, we perform a check to ensure
	// the secret ID matches the one stored. If there is no entry, we perform a
	// check to ensure the node is allowed to register given the request and the
	// server introduction enforcement configuration.
	if originalNode != nil {
		// Check if the SecretID has been tampered with
		if args.Node.SecretID != originalNode.SecretID && originalNode.SecretID != "" {
			return fmt.Errorf("node secret ID does not match. Not registering node.")
		}

		// Don't allow the Register method to update the node status. Only the
		// UpdateStatus method should be able to do this.
		if originalNode.Status != "" {
			args.Node.Status = originalNode.Status
		}
		// The called function performs all the required logging and metric
		// emitting, so we only need to check the return value.
	} else if !n.newRegistrationAllowed(args, authErr) {
		return structs.ErrPermissionDenied
	}

	// We have a valid node connection, so add the mapping to cache the
	// connection and allow the server to send RPCs to the client. We only cache
	// the connection if it is not being forwarded from another server.
	if n.ctx != nil && n.ctx.NodeID == "" && !args.IsForwarded() {
		n.ctx.NodeID = args.Node.ID

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure the client uses the same SecretID (from its secret file in the data dir) as the registered node — do not wipe the client's secret while keeping the same node ID
2. If the node is genuinely new/replaced, deregister the old node or change the client's node_id
3. Check for config duplication (same node_id on multiple clients) and assign unique node IDs
4. Restart the client with a fresh node_id so the server records a new node entry

Example fix

// before (client hcl): reused node id with fresh state
node {
  id = "abc-123"
}
// after: let Nomad derive/keep the id and secret from the data dir
node {
  # omit id, or keep the id AND preserve data/client/secret
}

// before registering, compare with the server's stored node
stored, _, err := client.Nodes().Info(nodeID, nil)
if err == nil && stored != nil && stored.SecretID != "" && node.SecretID != stored.SecretID {
    return fmt.Errorf("secret mismatch for node %s: regenerate node_id or restore original secret", nodeID)
}

func secretMatches(stored, incoming string) bool {
    return stored == "" || incoming == stored
}

err := client.Nodes().Register(node, nil)
if err != nil && strings.Contains(err.Error(), "node secret ID does not match") {
    // stop retrying; requires manual reconciliation of node_id/secret
}

When it happens

Trigger: Client calls Node.Register (RPC) with args.Node.SecretID different from the persisted node's SecretID while originalNode.SecretID is non-empty; node re-registers after a config change that regenerates its secret; two clients share the same node ID but different secrets.

Common situations: Operator copies a client config (including node ID) to a new machine without resetting state; client data dir wiped but node ID pinned, producing a new secret; stale server state after a restore; misconfigured `node_id` reuse across hosts.

Related errors

• running container as ContainerAdmin is unsafe; change the co
• ACL token not found
• running as user %q is disallowed
• error evaluating symlink: %w
• archive contains symlink that escapes alloc dir

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/715b133f8b99445a. Report an issue: GitHub.