hashicorp/nomad · critical

failed to configure keyring: %v

Error message

failed to configure keyring: %v

What it means

setupServer initializes the Serf gossip keyring via setupKeyrings when server.encrypt is configured. If the keyring cannot be created, loaded, or installed (bad key format, missing keyring file, permission issues), the error is wrapped as this message and the agent fails to start.

Source

Thrown at command/agent/agent.go:1195

	if !a.config.Server.Enabled {
		return nil
	}

	// Setup the configuration
	conf, err := a.serverConfig()
	if err != nil {
		return fmt.Errorf("server config setup failed: %s", err)
	}

	// Generate a node ID and persist it if it is the first instance, otherwise
	// read the persisted node ID.
	if err := a.setupNodeID(conf); err != nil {
		return fmt.Errorf("setting up server node ID failed: %s", err)
	}

	// Sets up the keyring for gossip encryption
	if err := a.setupKeyrings(conf); err != nil {
		return fmt.Errorf("failed to configure keyring: %v", err)
	}

	// Create the server
	server, err := nomad.NewServer(conf,
		a.consulCatalog,           // self service discovery
		a.consulConfigEntriesFunc, // writing config entries for gateways
	)
	if err != nil {
		return fmt.Errorf("server setup failed: %v", err)
	}
	a.server = server

	// Consul check addresses default to bind but can be toggled to use advertise
	rpcCheckAddr := a.config.normalizedAddrs.RPC
	serfCheckAddr := a.config.normalizedAddrs.Serf

	defaultConsul := conf.ConsulConfigs[structs.ConsulDefaultCluster]

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Generate the key with `nomad operator keygen` and put exactly that in server.encrypt
2. Check data_dir/server/keystate for stale/corrupt keyring files; remove or fix them (cluster-wide key consistency required)
3. Fix filesystem permissions on the keystate directory
4. Ensure all servers use the same key/number of keys when rekeying

Example fix

// before
server {
  enabled = true
  encrypt = "mysecretkey"
}
// after (shell: nomad operator keygen)
server {
  enabled = true
  encrypt = "pUqJrVyVRj5jsiYEkM/tFQYfiIxD9hP8Bc3Uw3AzwU0="
}

// Validate gossip key before configuring server.encrypt
key := "pUqJrVyVRj5jsiYEkM/tFQYfiIxD9hP8Bc3Uw3AzwU0="
raw, err := base64.StdEncoding.DecodeString(key)
if err != nil || (len(raw) != 16 && len(raw) != 32) {
    log.Fatalf("server.encrypt must be base64 of 16 or 32 bytes: use `nomad operator keygen`")
}

if err := setupKeyrings(conf); err != nil {
    return fmt.Errorf("failed to configure keyring: %w", err)
}

When it happens

Trigger: server.encrypt set to a key that is not a valid 16/32-byte base64 gossip key, a keyring file in data_dir/server/keystate that cannot be parsed or written, or mismatched keys when joining a cluster — during NewAgent.

Common situations: Hand-generating an encrypt key that is not 16 or 32 bytes base64, keyring files left over with different keys after cluster rekey, permission problems on data_dir/server/keystate, or copying data_dir between clusters.

Related errors

• unable to decrypt wrapped key
• failed to get active nomad key: %w
• failed to add key to keyring: %v
• variable error: encrypt: %w
• root key not found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/710babc6f7b9010a. Report an issue: GitHub.