hashicorp/nomad · error

failed to load SI token for %s: %w

Error message

failed to load SI token for %s: %w

What it means

maybeLoadSIToken reads the Consul SI (service identity) JWT token file from the task's secrets directory. If os.ReadFile fails with an error other than NotExist, the hook logs it and wraps the error with the task name. A missing file is tolerated (falls back to the Consul agent's token), but any other read error is fatal to Prestart.

Source

Thrown at client/allocrunner/taskrunner/envoy_bootstrap_hook.go:635

		env = append(env, fmt.Sprintf("%s=%s", "CONSUL_HTTP_SSL_VERIFY", v))
	}
	if v := e.namespace; v != "" {
		env = append(env, fmt.Sprintf("%s=%s", "CONSUL_NAMESPACE", v))
	}
	return env
}

// maybeLoadSIToken reads the SI token saved to disk in the secrets directory
// by the service identities prestart hook. This envoy bootstrap hook blocks
// until the sids hook completes, so if the SI token is required to exist (i.e.
// Consul ACLs are enabled), it will be in place by the time we try to read it.
func (h *envoyBootstrapHook) maybeLoadSIToken(task, dir string) (string, error) {
	tokenPath := filepath.Join(dir, sidsTokenFile)
	token, err := os.ReadFile(tokenPath)
	if err != nil {
		if !os.IsNotExist(err) {
			h.logger.Error("failed to load SI token", "task", task, "error", err)
			return "", fmt.Errorf("failed to load SI token for %s: %w", task, err)
		}
		h.logger.Trace("no SI token to load, falling back to agent token", "task", task)
		return h.consulFallbackToken, nil // token file does not exist
	}
	h.logger.Trace("recovered pre-existing SI token", "task", task)
	return string(token), nil
}

func (h *envoyBootstrapHook) servicePreflightCheck(
	ctx context.Context, backoffOpts decay.BackoffOptions, proxyServiceID string) error {

	// keep track of latest error returned from Consul or from missing service
	var apiErr error
	var allocServices *serviceregistration.AllocRegistration

	backoffErr := decay.Backoff(func() (bool, error) {
		// If hook is killed, just stop.
		select {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check ownership/permissions of secrets/ and the SI token file so the Nomad client (task) user can read it.
2. Verify the file at that path is a regular file, not a directory or symlink to nowhere.
3. Check alloc-dir filesystem health (dmesg, mount status).
4. If your workload doesn't need workload identity, ensure the SI token file is simply absent (that path is handled gracefully) rather than malformed.

tokenPath := filepath.Join(secretsDir, "consul_identity")
if fi, err := os.Stat(tokenPath); err != nil && !os.IsNotExist(err) {
    return fmt.Errorf("SI token file unusable: %w", err)
} else if fi != nil && !fi.Mode().IsRegular() {
    return fmt.Errorf("SI token path is not a regular file")
}

token, err := os.ReadFile(tokenPath)
switch {
case err == nil:
    useToken(string(token))
case os.IsNotExist(err):
    useToken(fallbackAgentToken) // tolerated by the hook
default:
    log.Error("SI token read failed", "error", err) // check perms/IO
}

When it happens

Trigger: os.ReadFile(filepath.Join(dir, sidsTokenFile)) returns a permission error, an I/O error, or the path exists but is a directory — anything other than os.IsNotExist.

Common situations: SI token file created by a previous hook has wrong permissions or ownership; secrets dir on a failing filesystem; file replaced by a directory due to a misbehaving template/other hook.

Related errors

• failed to create secrets/envoy_bootstrap.json for envoy: %w
• failed to create alloc/logs/envoy_bootstrap.stderr.0 for env
• failed to open plugin directory %q: %v
• Failed to create keystore tempdir
• must pass non-nil job

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e9c19c9902ae3f79. Report an issue: GitHub.