hashicorp/nomad · error

Failed to create keystore tempdir

Error message

Failed to create keystore tempdir

What it means

During server startup NewServer sets up the encryption keyring. In dev mode with no DataDir, it creates a temporary directory for the keystore via os.MkdirTemp; if that OS call fails, the server returns this error. Notably the error discards the underlying cause, so only the OS-level reason (permissions, /tmp full, TMPDIR invalid) explains it.

Source

Thrown at nomad/server.go:436

	// Create the node heartbeater
	s.nodeHeartbeater = newNodeHeartbeater(s)

	// Create the periodic dispatcher for launching periodic jobs.
	s.periodicDispatcher = NewPeriodicDispatch(s.logger, s)

	// Initialize the stats fetcher that autopilot will use.
	s.statsFetcher = NewStatsFetcher(s.logger, s.connPool, s.config.Region)
	s.statsFetcher.SetLocalServer(s)

	// Setup Consul
	s.consulConfigEntries = NewConsulConfigsAPI(consulConfigFunc, s.logger)

	// Set up the keyring
	keystorePath := filepath.Join(s.config.DataDir, "keystore")
	if s.config.DevMode && s.config.DataDir == "" {
		keystorePath, err = os.MkdirTemp("", "nomad-keystore")
		if err != nil {
			return nil, fmt.Errorf("Failed to create keystore tempdir")
		}
	}
	encrypter, err := NewEncrypter(s, keystorePath)
	if err != nil {
		return nil, err
	}
	s.encrypter = encrypter

	// Set up the OIDC discovery configuration required by third parties, such as
	// AWS's IAM OIDC Provider, to authenticate workload identity JWTs.
	if iss := config.OIDCIssuer; iss != "" {
		oidcDisco, err := structs.NewOIDCDiscoveryConfig(iss)
		if err != nil {
			return nil, err
		}
		s.oidcDisco = oidcDisco
		s.logger.Info("issuer set; OIDC Discovery endpoint for workload identities enabled", "issuer", iss)
	} else {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Fix TMPDIR or ensure the default temp location exists and is writable, then restart
2. Set an explicit data_dir so the keystore uses a real path instead of a temp dir
3. Free disk space if the temp filesystem is full
4. Inspect the OS error with `TMPDIR=/path nomad agent -dev` to confirm the cause

Example fix

// before (fails when TMPDIR unwritable)
nomad agent -dev
// after
nomad agent -dev -data-dir /var/lib/nomad

// Ensure a writable temp location before starting a dev-mode server
if cfg.DevMode && cfg.DataDir == "" {
    if d, err := os.MkdirTemp("", "nomad-keystore-test"); err != nil {
        return fmt.Errorf("temp dir unavailable: %w", err)
    } else { os.RemoveAll(d) }
}

When it happens

Trigger: Starting a Nomad server with dev_mode=true and DataDir=="" while os.MkdirTemp("", "nomad-keystore") fails — e.g. TMPDIR points to a nonexistent/unwritable path or the filesystem is full.

Common situations: Containers with read-only or missing temp directories; TMPDIR misconfigured in systemd units or Docker; disk exhaustion on the node running dev-mode servers or local test setups.

Related errors

• failed to create secrets/envoy_bootstrap.json for envoy: %w
• failed to create alloc/logs/envoy_bootstrap.stderr.0 for env
• failed to load SI token for %s: %w
• failed to open plugin directory %q: %v
• could not read path %s from keystore: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/33358dcc385096fd. Report an issue: GitHub.