hashicorp/nomad · critical

root key ID %s must match key file %s

Error message

root key ID %s must match key file %s

What it means

The key file on disk decoded successfully, but its internal KeyID (key.Meta.KeyID) does not match the root key ID being processed (derived from the filename or keyring metadata). Nomad aborts keystore loading rather than silently binding key material to the wrong key ID.

Source

Thrown at nomad/encrypter.go:190

		id, _, _ := strings.Cut(idWithIndex, ".")
		if !helper.IsUUID(id) {
			return nil
		}

		e.keyringLock.RLock()
		_, ok := e.keyring[id]
		e.keyringLock.RUnlock()
		if ok {
			return nil // already loaded this key from another file
		}

		key, err := e.loadKeyFromStore(path)
		if err != nil {
			keyErrors[id] = fmt.Errorf("could not load key file %s from keystore: %w", path, err)
			return nil
		}
		if key.Meta.KeyID != id {
			return fmt.Errorf("root key ID %s must match key file %s", key.Meta.KeyID, path)
		}

		err = e.addCipher(key)
		if err != nil {
			return fmt.Errorf("could not add key file %s to keystore: %w", path, err)
		}

		// we loaded this key from at least one KEK configuration, so clear any
		// error from a previous file that we couldn't read from
		delete(keyErrors, id)
		return nil
	})

	if len(keyErrors) == 0 {
		return nil
	}

	var mErr multierror.Error

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Compare key.Meta.KeyID in the file (cat the .nks.json) with the expected ID from the filename/error message
2. Restore the correctly-named key file from backup or from a healthy server in the same region
3. Never rename .nks.json files manually; re-run key rotation instead of moving files
4. If the key is obsolete, remove it and the corresponding keyring metadata via nomad keyring rotate/remove

Example fix

// before: file renamed manually, ID mismatch
mv data/keystore/WRONG-ID.nks.json data/keystore/EXPECTED-ID.nks.json
// after: restore the correct key material for EXPECTED-ID from backup
scp healthy-server:/opt/nomad/data/keystore/EXPECTED-ID.nks.json data/keystore/

// verify each key file's embedded ID matches its filename
import "encoding/json"
func keyIDMatches(file string) (bool, error) {
  b, err := os.ReadFile(file)
  if err != nil { return false, err }
  var k struct{ Meta struct{ KeyID string `json:"KeyID"` } `json:"Meta"` }
  if err := json.Unmarshal(b, &k); err != nil { return false, err }
  want := strings.TrimSuffix(filepath.Base(file), ".nks.json")
  return k.Meta.KeyID == want, nil
}

When it happens

Trigger: loadKeystore finds a .nks.json file whose embedded RootKey.Meta.KeyID differs from the id passed to the walker callback — typically after renaming a key file, copying the wrong file into the keystore, or a failed rotation left mismatched metadata.

Common situations: Operator renamed a key file manually, restored an inconsistent mix of keyring-metadata and key files from different snapshots, or copied a key file between clusters.

Related errors

• could not load key file %s from keystore: %w
• could not add key file %s to keystore: %w
• Failed to query for root keys: %v
• invalid signature: %w
• no such KMS provider %q configured

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c7a602c475baf944. Report an issue: GitHub.