hashicorp/nomad · critical

failed to create volume watcher: %v

Error message

failed to create volume watcher: %v

What it means

NewServer wraps any error from setupVolumeWatcher, which builds the volume controller/watcher used to reconcile CSI volume claims. A failure here aborts server creation; the error text is just the inner cause wrapped.

Source

Thrown at nomad/server.go:524

		return nil, fmt.Errorf("Failed to start workers: %v", err)
	}

	// Setup the Consul syncer
	if err := s.setupConsulSyncer(); err != nil {
		s.logger.Error("failed to create server consul syncer", "error", err)
		return nil, fmt.Errorf("failed to create server Consul syncer: %v", err)
	}

	// Setup the deployment watcher.
	if err := s.setupDeploymentWatcher(); err != nil {
		s.logger.Error("failed to create deployment watcher", "error", err)
		return nil, fmt.Errorf("failed to create deployment watcher: %v", err)
	}

	// Setup the volume watcher
	if err := s.setupVolumeWatcher(); err != nil {
		s.logger.Error("failed to create volume watcher", "error", err)
		return nil, fmt.Errorf("failed to create volume watcher: %v", err)
	}
	s.volumeControllerFutures = map[string]context.Context{}

	// Start the eval broker notification system so any subscribers can get
	// updates when the processes SetEnabled is triggered.
	go s.evalBroker.enabledNotifier.Run()

	// Setup the node drainer.
	s.setupNodeDrainer()

	// Setup the enterprise state
	if err := s.setupEnterprise(config); err != nil {
		return nil, err
	}

	// Monitor leadership changes
	go s.monitorLeadership()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the 'failed to create volume watcher' log entry immediately above for the root cause.
2. Verify CSI plugin configuration and server config validity.
3. Ensure the server host has adequate resources and restart the agent.
4. If CSI is not used and failure persists, report/upgrade — this path should never fail with valid config.

srv, err := nomad.NewServer(cfg, logger)
if err != nil && strings.Contains(err.Error(), "failed to create volume watcher") {
    logger.Error("volume watcher init failed", "err", err)
    return err
}

When it happens

Trigger: NewServer invoked when setupVolumeWatcher cannot initialize its internal state — typically the CSI volume watcher's eval/notify queue cannot be created or a config-dependent dependency (broker, blockResult) fails.

Common situations: Environments with CSI plugins misconfigured, servers under severe resource pressure during startup, or versions where the CSI controller was partially disabled via feature flags/build tags.

Related errors

• error parsing: root should be an object
• missing policy name
• cannot specify Accessor ID
• missing accessor ID
• missing secret ID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8b00d7ce9cf467e0. Report an issue: GitHub.