What causes "unable to process ACLToken: %w" in hashicorp/nomad?

Verify the underlying base64 error in the log to confirm corruption vs encoding variant (url-safe vs std encoding)

unable to process ACLToken: %w

Error message

unable to process ACLToken: %w

What it means

decodeACLToken base64-decodes a stored Consul ACL token (from client state / hook resources); if base64.StdEncoding.DecodeString fails, it returns 'unable to process ACLToken: %w'. The stored token blob is not valid base64, indicating corrupted or malformed persisted state.

Source

Thrown at client/allocrunner/consul_hook.go:346

type resourcesBackend struct {
	allocID       string
	hookResources *cstructs.AllocHookResources
	db            cstate.StateDB
}

func newResourcesBackend(allocID string, hr *cstructs.AllocHookResources, db cstate.StateDB) *resourcesBackend {
	return &resourcesBackend{
		allocID:       allocID,
		hookResources: hr,
		db:            db,
	}
}

func decodeACLToken(b64ACLToken string, token *consulapi.ACLToken) error {
	decodedBytes, err := base64.StdEncoding.DecodeString(b64ACLToken)
	if err != nil {
		return fmt.Errorf("unable to process ACLToken: %w", err)
	}

	if len(decodedBytes) != 0 {
		if err := json.Unmarshal(decodedBytes, token); err != nil {
			return fmt.Errorf("unable to unmarshal ACLToken: %w", err)
		}
	}

	return nil
}

func encodeACLToken(token *consulapi.ACLToken) (string, error) {
	jsonBytes, err := json.Marshal(token)
	if err != nil {
		return "", fmt.Errorf("unable to marshal ACL token: %w", err)
	}

	return base64.StdEncoding.EncodeToString(jsonBytes), nil

View on GitHub (pinned to 482b49bf1a)

Solutions

Verify the underlying base64 error in the log to confirm corruption vs encoding variant (url-safe vs std encoding)
Stop the allocation and clear its client state so tokens are re-derived fresh (token loss is safe; Consul re-login will mint new ones)
Check disk health and that the client data dir is not being modified externally
If reproducible, file a Nomad issue with the Nomad version and how the state was produced

Defensive patterns

Strategy: try-catch

Validate before calling

if b64ACLToken != "" {
    if _, err := base64.StdEncoding.DecodeString(b64ACLToken); err != nil {
        return fmt.Errorf("stored ACL token is not valid base64, discarding: %w", err)
    }
}

Type guard

func isBase64(s string) bool {
    _, err := base64.StdEncoding.DecodeString(s)
    return err == nil
}

Try / catch

token := &consulapi.ACLToken{}
if err := decodeACLToken(b64ACLToken, token); err != nil {
    h.logger.Warn("stored consul token unreadable, re-deriving", "error", err)
    token = nil // proceed with fresh JWT login instead of failing
}

Prevention

Never hand-edit Nomad client state files
Protect the client data dir from disk-full and crash corruption; monitor disk health
When migrating clients, let allocations reschedule instead of copying state
On decode failure, fall back to re-deriving tokens rather than failing the hook

When it happens

Trigger: loadAllocTokens reads a b64-encoded ACL token from hook resources/state that fails base64 decoding — e.g. hand-edited state, truncated data-dir files, or a token stored by incompatible serialization.

Common situations: Corrupted client data dir after disk full/crash; manual editing or migration of nomad client state files; restoring state from a different Nomad version.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/51968305695cac3d. Report an issue: GitHub.