hashicorp/nomad · error

failed to store driver state: %v

Error message

failed to store driver state: %v

What it means

After successfully creating a replacement docker logger, RecoverTask persists the updated handle state (new reattach config) via handle.SetDriverState(h.buildState()). If persisting fails, the container is stopped with zero timeout (cleanup) and this error is returned, since recovering a logger whose state cannot be stored would be lost on the next restart.

Source

Thrown at drivers/docker/driver.go:306

	if loggingIsEnabled(d.config, handle.Config) {
		h.dlogger, h.dloggerPluginClient, err = d.reattachToDockerLogger(handleState.ReattachConfig)
		if err != nil {
			d.logger.Warn("failed to reattach to docker logger process", "error", err)

			h.dlogger, h.dloggerPluginClient, err = d.setupNewDockerLogger(container, handle.Config, time.Now())
			if err != nil {
				if _, err := dockerClient.ContainerStop(d.ctx, handleState.ContainerID, stopWithZeroTimeout()); err != nil {
					d.logger.Warn("failed to stop container during cleanup", "container_id", handleState.ContainerID, "error", err)
				}
				return fmt.Errorf("failed to setup replacement docker logger: %v", err)
			}

			if err := handle.SetDriverState(h.buildState()); err != nil {
				if _, err := dockerClient.ContainerStop(d.ctx, handleState.ContainerID, stopWithZeroTimeout()); err != nil {
					d.logger.Warn("failed to stop container during cleanup", "container_id", handleState.ContainerID, "error", err)
				}
				return fmt.Errorf("failed to store driver state: %v", err)
			}
		}
	}

	d.tasks.Set(handle.Config.ID, h)

	// find a pause container?

	go h.run()

	return nil
}

func loggingIsEnabled(driverCfg *DriverConfig, taskCfg *drivers.TaskConfig) bool {
	if driverCfg.DisableLogCollection {
		return false
	}
	if taskCfg.StderrPath == os.DevNull && taskCfg.StdoutPath == os.DevNull {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check disk space and filesystem health on the client state partition (`df -h`, `dmesg` for I/O errors).
2. Inspect the wrapped SetDriverState error in logs — usually 'database is locked', 'no space left on device', or 'read-only file system'.
3. If the state DB is corrupt, restore from backup or stop the client and clear/rebuild the task state (tasks will be rescheduled).
4. Fix permissions/ownership of the client state directory for the user running the agent.

Example fix

// before
if err := handle.SetDriverState(h.buildState()); err != nil {
	return fmt.Errorf("failed to store driver state: %v", err)
}
// after
if err := handle.SetDriverState(h.buildState()); err != nil {
	d.logger.Error("SetDriverState failed; check state dir disk/permissions", "error", err)
	return fmt.Errorf("failed to store driver state: %w", err)
}

// ensure the state dir is writable and has free space before recovery
func stateDirHealthy(dir string) error {
	fi, err := os.Stat(dir)
	if err != nil || !fi.IsDir() {
		return fmt.Errorf("state dir missing: %s", dir)
	}
	probe := filepath.Join(dir, ".write_probe")
	if err := os.WriteFile(probe, []byte("ok"), 0o600); err != nil {
		return fmt.Errorf("state dir not writable: %w", err)
	}
	os.Remove(probe)
	return nil
}

// Go: log the wrapped cause and surface disk/state issues
if err := driver.RecoverTask(handle); err != nil {
	if strings.Contains(err.Error(), "failed to store driver state") {
		cause := fmt.Sprintf("%v", errors.Unwrap(err))
		switch {
		case strings.Contains(cause, "no space left"):
			log.Print("free disk space on the client state partition")
		case strings.Contains(cause, "locked"):
			log.Print("state DB locked; serialize recovery attempts")
		default:
			log.Printf("state persist error: %s", cause)
		}
	}
}

When it happens

Trigger: The underlying state store (task handle state backend, e.g. bolt/state DB) is corrupt, locked, or the disk is full; handle state exceeds size limits; the state directory has wrong permissions; concurrent write contention during recovery.

Common situations: Disk full on the client's data partition (/var/lib/... state dir); state database corrupted after a crash; read-only filesystem after disk errors; permission changes on the state directory following a package upgrade or migration.

Related errors

• failed to fetch docker daemon info: %v
• failed to get docker long operations client: %w
• failed to inspect container for id %q: %v
• failed to setup replacement docker logger: %v
• failed to set driver state: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e1707d4879995405. Report an issue: GitHub.