hashicorp/nomad · critical

failed to wait for keyring decryption to complete: %v

Error message

failed to wait for keyring decryption to complete: %v

What it means

At the end of NewServer, the server waits up to startup_timeout for the keyring encrypter to be ready (i.e., for all keyring keys to be decrypted/loaded from peers or local state). If IsReady times out, the server shuts itself down and returns this error listing which keys were not decrypted. Nomad refuses to run with an undecrypted keyring because sensitive data (variables, signing keys) would be unreadable.

Source

Thrown at nomad/server.go:583

	go s.EmitRaftStats(10*time.Second, s.shutdownCh)

	// Start enterprise background workers
	s.startEnterpriseBackground()

	// Enable the keyring replicator on servers; the replicator has to
	// be created before the RPC server and FSM but needs them to
	// exist before it can start.
	s.keyringReplicator = NewKeyringReplicator(s, encrypter)

	// Wait for the keyring to be ready. This is a blocking call and will
	// time out if the keyring takes too long to decrypt its initial set of
	// keys.
	//
	// In the event of a timeout, we shut down the server and return an error to
	// the caller which will include what keys were not decrypted.
	if err := s.encrypter.IsReady(startupTimeout); err != nil {
		_ = s.Shutdown()
		return nil, fmt.Errorf("failed to wait for keyring decryption to complete: %v", err)
	}

	// Done
	return s, nil
}

// startRPCListener starts the server's the RPC listener
func (s *Server) startRPCListener() {
	ctx, cancel := context.WithCancel(context.Background())
	s.rpcCancel = cancel
	go s.listen(ctx)
}

// createRPCListener creates the server's RPC listener
func (s *Server) createRPCListener() (*net.TCPListener, error) {
	s.listenerCh = make(chan struct{})
	listener, err := net.ListenTCP("tcp", s.config.RPCAddr)
	if err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Increase server.startup_timeout (default is generous; slow clusters may need more) and restart.
2. Ensure the local keystore directory (keystore/ under data dir) is intact and copied when migrating servers.
3. Bring the surviving peers that hold keyring keys online so decryption can complete.
4. If keys are permanently lost, follow HashiCorp's keyring recovery procedure (re-encrypt or reset via the documented recovery flow).

Example fix

// before
server {
  startup_timeout = "5s"
}
// after
server {
  startup_timeout = "2m"
}

// preflight: ensure local keystore exists and peers are reachable before start
if keysPending && len(keystoreFiles) == 0 {
    return errors.New("encrypted keyring in use but local keystore directory is empty; copy it from an existing server or restore backup")
}

srv, err := nomad.NewServer(cfg, logger)
if err != nil && strings.Contains(err.Error(), "keyring decryption") {
    // error includes which keys were not decrypted; do not retry blindly
    return fmt.Errorf("fix keystore/peers before restart: %w", err)
}

When it happens

Trigger: NewServer with server.startup_timeout too small and encrypted keyring material present; unreachable peers holding keyring keys; wrong/missing keyring files on disk; gossip encryption key changed so existing keys cannot be decrypted.

Common situations: First boot of a restored server whose keystore was not copied over, cluster rebuild where peers holding keys are down, mismanaged keyrotation leaving undecryptable keys, or very low startup_timeout on slow-starting clusters.

Related errors

• unable to decrypt wrapped key
• failed to configure keyring: %v
• failed to get active nomad key: %w
• rotated key does not exist in keyring: %w
• %w (root key): %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/305e37c1dde6d589. Report an issue: GitHub.