hashicorp/nomad · error

failed to set driver state: %v

Error message

failed to set driver state: %v

What it means

After a successful launch, StartTask serializes the driver state (PID, task ID, started-at) into the TaskHandle via handle.SetDriverState so the task can be recovered after a client restart. If persisting this state fails, the driver shuts the executor down and aborts the start to avoid an unrecoverable orphaned task.

Source

Thrown at drivers/exec/driver.go:569

		pid:          ps.Pid,
		pluginClient: pluginClient,
		taskConfig:   cfg,
		procState:    drivers.TaskStateRunning,
		startedAt:    time.Now().Round(time.Millisecond),
		logger:       d.logger,
	}

	driverState := TaskState{
		ReattachConfig: pstructs.ReattachConfigFromGoPlugin(pluginClient.ReattachConfig()),
		Pid:            ps.Pid,
		TaskConfig:     cfg,
		StartedAt:      h.startedAt,
	}

	if err := handle.SetDriverState(&driverState); err != nil {
		d.logger.Error("failed to start task, error setting driver state", "error", err)
		_ = exec.Shutdown("", 0)
		return nil, nil, fmt.Errorf("failed to set driver state: %v", err)
	}

	d.tasks.Set(cfg.ID, h)
	go h.run()
	return handle, nil, nil
}

func (d *Driver) WaitTask(ctx context.Context, taskID string) (<-chan *drivers.ExitResult, error) {
	handle, ok := d.tasks.Get(taskID)
	if !ok {
		return nil, drivers.ErrTaskNotFound
	}

	ch := make(chan *drivers.ExitResult)
	go d.handleWait(ctx, handle, ch)

	return ch, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check disk space (df -h) and the health of the Nomad client data dir.
2. Inspect client logs for the underlying state store error to identify I/O vs corruption.
3. Fix data-dir permissions/ownership for the nomad user.
4. If the state DB is corrupt, stop the client and restore or (as last resort) reinitialize the client state, rescheduling allocations.

Example fix

// before: disk full
$ df -h /var/lib/nomad
/dev/sdb 100% used
// after
$ sudo find /var/log -name '*.gz' -delete  # free space
$ sudo systemctl restart nomad
# then reschedule the allocation

// check the client state store can accept writes before starting tasks
statePath := filepath.Join(dataDir, "client", "state.db")
f, err := os.OpenFile(statePath+".probe", os.O_CREATE|os.O_WRONLY, 0o600)
if err != nil {
    return fmt.Errorf("state store not writable: %w", err)
}
f.Close(); os.Remove(statePath + ".probe")

When it happens

Trigger: handle.SetDriverState(&driverState) fails, typically because the underlying client state store (bolt DB) write fails: disk full, state DB corrupted, I/O errors, or permissions on the Nomad data dir.

Common situations: Disk full on the Nomad client; corrupted /var/lib/nomad/client state database; read-only mount of the data dir; permission changes after backups/restores.

Related errors

• Task can't ask for disk resources, they have to be specified
• timed out while opening database, is another Nomad process a
• failed to create state database: %v
• error deleting invalid task state for task %q: %v
• error deleting unexpected task bucket %q: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0b1b075635b68747. Report an issue: GitHub.