hashicorp/nomad · error

unable to process ACLToken

Error message

unable to process ACLToken: %w

What it means

decodeACLToken base64-decodes a stored Consul ACL token (from client state / hook resources); if base64.StdEncoding.DecodeString fails, it returns 'unable to process ACLToken: %w'. The stored token blob is not valid base64, indicating corrupted or malformed persisted state.

Solutions

  1. Verify the underlying base64 error in the log to confirm corruption vs encoding variant (url-safe vs std encoding)
  2. Stop the allocation and clear its client state so tokens are re-derived fresh (token loss is safe; Consul re-login will mint new ones)
  3. Check disk health and that the client data dir is not being modified externally
  4. If reproducible, file a Nomad issue with the Nomad version and how the state was produced
Defensive patterns

Strategy: try-catch

Validate before calling

if b64ACLToken != "" {
    if _, err := base64.StdEncoding.DecodeString(b64ACLToken); err != nil {
        return fmt.Errorf("stored ACL token is not valid base64, discarding: %w", err)
    }
}

Type guard

func isBase64(s string) bool {
    _, err := base64.StdEncoding.DecodeString(s)
    return err == nil
}

Try / catch

token := &consulapi.ACLToken{}
if err := decodeACLToken(b64ACLToken, token); err != nil {
    h.logger.Warn("stored consul token unreadable, re-deriving", "error", err)
    token = nil // proceed with fresh JWT login instead of failing
}

Prevention

When it happens

Trigger: loadAllocTokens reads a b64-encoded ACL token from hook resources/state that fails base64 decoding — e.g. hand-edited state, truncated data-dir files, or a token stored by incompatible serialization.

Common situations: Corrupted client data dir after disk full/crash; manual editing or migration of nomad client state files; restoring state from a different Nomad version.

Related errors


AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/51968305695cac3d. Report an issue: GitHub.

Appendix: source

Thrown at client/allocrunner/consul_hook.go:346

type resourcesBackend struct {
	allocID       string
	hookResources *cstructs.AllocHookResources
	db            cstate.StateDB
}

func newResourcesBackend(allocID string, hr *cstructs.AllocHookResources, db cstate.StateDB) *resourcesBackend {
	return &resourcesBackend{
		allocID:       allocID,
		hookResources: hr,
		db:            db,
	}
}

func decodeACLToken(b64ACLToken string, token *consulapi.ACLToken) error {
	decodedBytes, err := base64.StdEncoding.DecodeString(b64ACLToken)
	if err != nil {
		return fmt.Errorf("unable to process ACLToken: %w", err)
	}

	if len(decodedBytes) != 0 {
		if err := json.Unmarshal(decodedBytes, token); err != nil {
			return fmt.Errorf("unable to unmarshal ACLToken: %w", err)
		}
	}

	return nil
}

func encodeACLToken(token *consulapi.ACLToken) (string, error) {
	jsonBytes, err := json.Marshal(token)
	if err != nil {
		return "", fmt.Errorf("unable to marshal ACL token: %w", err)
	}

	return base64.StdEncoding.EncodeToString(jsonBytes), nil

View on GitHub (pinned to 482b49bf1a)