hashicorp/nomad · critical

could not load key file %s from keystore: %w

Error message

could not load key file %s from keystore: %w

What it means

During keystore load (loadKeyFromStore path), a key file on disk for root key ID <id> could not be read/parsed. The underlying error from loadKeyFromStore (file IO, JSON decode, key unmarshal) is wrapped and stored per key ID; Nomad continues loading other files but will fail keyring init if no file for that key succeeds.

Source

Thrown at nomad/encrypter.go:186

		if !strings.HasSuffix(path, nomadKeystoreExtension) {
			return nil
		}
		idWithIndex := strings.TrimSuffix(filepath.Base(path), nomadKeystoreExtension)
		id, _, _ := strings.Cut(idWithIndex, ".")
		if !helper.IsUUID(id) {
			return nil
		}

		e.keyringLock.RLock()
		_, ok := e.keyring[id]
		e.keyringLock.RUnlock()
		if ok {
			return nil // already loaded this key from another file
		}

		key, err := e.loadKeyFromStore(path)
		if err != nil {
			keyErrors[id] = fmt.Errorf("could not load key file %s from keystore: %w", path, err)
			return nil
		}
		if key.Meta.KeyID != id {
			return fmt.Errorf("root key ID %s must match key file %s", key.Meta.KeyID, path)
		}

		err = e.addCipher(key)
		if err != nil {
			return fmt.Errorf("could not add key file %s to keystore: %w", path, err)
		}

		// we loaded this key from at least one KEK configuration, so clear any
		// error from a previous file that we couldn't read from
		delete(keyErrors, id)
		return nil
	})

	if len(keyErrors) == 0 {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped cause (%w) in the agent log to identify whether it is IO, permissions, or JSON decode
2. Fix file permissions/ownership on the keystore file (should be readable only by the nomad user)
3. Restore the key file from a backup or from another server's keystore directory
4. If the key is intentionally retired, remove its keyring metadata and the corrupt file together (after rotation) so the loader stops looking for it

Example fix

// before: key file unreadable
$ ls -l data/keystore/*.nks.json
-rw------- 1 root root ... abc.nks.json   (nomad runs as 'nomad' user)
// after
$ chown nomad:nomad data/keystore/abc.nks.json && chmod 600 data/keystore/abc.nks.json

// before starting the agent, sanity-check keystore files
import "os"
func checkKeystore(dir string) error {
  files, err := filepath.Glob(filepath.Join(dir, "*.nks.json"))
  if err != nil { return err }
  for _, f := range files {
    if fi, err := os.Stat(f); err != nil || fi.Size() == 0 {
      return fmt.Errorf("keystore file %s missing or empty: %w", f, err)
    }
  }
  return nil
}

When it happens

Trigger: NewEncrypter -> loadKeystore iterates .nks.json files in the keystore directory and calls loadKeyFromStore(path); any IO error (missing file, permissions), JSON parse error, or structs.RootKey decode error for the key matching id produces this error.

Common situations: Keystore directory partially restored from backup, wrong file permissions after running the agent as a different user, manually edited/corrupted key file JSON, stale symlink, or disk issue on the data dir.

Related errors

• root key ID %s must match key file %s
• Failed to query for root keys: %v
• could not add key file %s to keystore: %w
• no such KMS provider %q configured
• failed to fetch key from any peer: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/095db5dd2d1a76c2. Report an issue: GitHub.