hashicorp/nomad · error

failed to derive Consul token for service %s: %v

Error message

failed to derive Consul token for service %s: %v

What it means

After obtaining the signed service identity, the hook calls getConsulToken to perform the Consul JWT login (DeriveTokenWithJWT) and a token preflight ACL check. Failures are appended to the multierror as 'failed to derive Consul token for service %s'. The wrapped inner error carries the real cause: connectivity, auth method, JWT validity, or ACL scope.

Source

Thrown at client/allocrunner/consul_hook.go:235

					"error getting signed identity for service %s: %v",
					service.Name, err,
				))
				continue
			}

			h.logger.Debug("logging into consul", "name", ti.IdentityName, "type", ti.WorkloadType)
			req := consul.JWTLoginRequest{
				JWT:            swi.JWT,
				AuthMethodName: consulConfig.ServiceIdentityAuthMethod,
				Meta: map[string]string{
					"requested_by": fmt.Sprintf("nomad_service_%s", ti.InterpolatedWorkloadIdentifier),
					"node_id":      h.alloc.NodeID,
				},
			}

			token, err = h.getConsulToken(clusterName, req)
			if err != nil {
				mErr = multierror.Append(mErr, fmt.Errorf(
					"failed to derive Consul token for service %s: %v",
					service.Name, err,
				))
				continue
			}

		}

		// Store token in results.
		if _, ok = tokens[clusterName]; !ok {
			tokens[clusterName] = make(map[string]*consulapi.ACLToken)
		}

		tokens[clusterName][tokenName] = token
	}

	return mErr.ErrorOrNil()
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped inner error from the agent log to classify connectivity vs login vs ACL failure
2. Ensure the Consul auth method and binding rules for Nomad service identities exist (consul acl auth-method list / binding-rule list)
3. Test connectivity from the client to the Consul agent address/port; fix firewalls or consul.address config
4. Reschedule the allocation after fixing; check NTP clock sync so JWTs are not rejected as expired

// pre-validate auth method for service identities on the target Consul
methods, _, err := consulClient.ACL().AuthMethodList(nil)
if err != nil { return err }
found := false
for _, m := range methods {
    if m.Name == consulConfig.ServiceIdentityAuthMethod { found = true }
}
if !found { return fmt.Errorf("auth method %q missing in Consul", consulConfig.ServiceIdentityAuthMethod) }

if err := hook.Prerun(); err != nil {
    if strings.Contains(err.Error(), "failed to derive Consul token for service") {
        // parse wrapped cause; retry transient network failures with backoff,
        // fix Consul ACL/auth-method config for permanent ones
    }
}

When it happens

Trigger: Consul agent unreachable; ServiceIdentityAuthMethod missing/mismatched in Consul; Consul rejects the service JWT (expired, wrong audience); derived token fails TokenPreflightCheck due to missing ACL permissions.

Common situations: Consul auth method or binding rules for service identities not bootstrapped; Consul agent restarted with ACLs reset; network partition between Nomad client and Consul; version incompatibility between Nomad's expected auth method and Consul's.

Related errors

• failed to derive Consul token for task %s: %v
• failed to query Consul services: %w
• [✘] Could not create Consul auth method: %w
• no one-time token returned
• no ACL token returned

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/4dea2f889cd702c4. Report an issue: GitHub.