hashicorp/nomad · error

failed to set driver state

Error message

failed to set driver state: %v

What it means

After successfully launching the VM process, StartTask persists the qemuDriverState (pid, image path, args) into the task handle via handle.SetDriverState. If this write fails, the driver cannot later restore/stop the task across plugin restarts, so it shuts down the exec'd QEMU process, kills the plugin client, and returns this wrapped error. The VM is never left orphaned, but the task fails to start.

Solutions

1. Inspect the Nomad client logs for the underlying error after 'failed to set driver state' — it names the storage failure.
2. Free disk space / fix permissions on the client's data_dir (host volumes holding alloc state).
3. Retry the job after restoring the client's data_dir; the QEMU process for this attempt is already cleaned up.
4. If persistent, restart the nomad client so its state store reinitializes, then resubmit the job.

// check client storage health before submitting state-heavy workloads
const { execSync } = require("child_process");
function clientDiskHealthy(dataDir) {
  try {
    const out = execSync(`df --output=pcent ${dataDir}`).toString();
    return parseInt(out.match(/(\d+)%/)[1], 10) < 90;
  } catch { return false; }
}

try {
  await nomad.jobs.startTask(cfg);
} catch (e) {
  if (String(e.message).startsWith("failed to set driver state")) {
    console.error("Client failed to persist task state; check disk space/permissions on data_dir, then resubmit. No orphan VM should remain.");
  }
  throw e;
}

When it happens

Trigger: Failure writing handle state to the underlying storage — e.g. disk full, permissions problems in the Nomad client data_dir, the handle's backing store already closed, or serialization failure of qemuDriverState.

Common situations: Client node with a full or failing disk; corrupted or unwritable Nomad data_dir; client being shut down concurrently while the task is starting; I/O errors on the filesystem hosting the alloc/state directories.

Related errors

• cannot destroy running task
• consul tokens for cluster
• consul tokens for cluster
• csi updates already batched
• devices updates already batched

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/730b91074e64c32e. Report an issue: GitHub.

Appendix: source

Thrown at drivers/qemu/driver.go:692

		pluginClient: pluginClient,
		taskConfig:   cfg,
		procState:    drivers.TaskStateRunning,
		startedAt:    time.Now().Round(time.Millisecond),
		logger:       d.logger,
	}

	qemuDriverState := TaskState{
		ReattachConfig: pstructs.ReattachConfigFromGoPlugin(pluginClient.ReattachConfig()),
		Pid:            ps.Pid,
		TaskConfig:     cfg,
		StartedAt:      h.startedAt,
	}

	if err := handle.SetDriverState(&qemuDriverState); err != nil {
		d.logger.Error("failed to start task, error setting driver state", "error", err)
		execImpl.Shutdown("", 0)
		pluginClient.Kill()
		return nil, nil, fmt.Errorf("failed to set driver state: %v", err)
	}

	d.tasks.Set(cfg.ID, h)
	go h.run()

	var driverNetwork *drivers.DriverNetwork
	if len(driverConfig.PortMap) == 1 {
		driverNetwork = &drivers.DriverNetwork{
			PortMap: driverConfig.PortMap,
		}
	}
	return handle, driverNetwork, nil
}

func (d *Driver) WaitTask(ctx context.Context, taskID string) (<-chan *drivers.ExitResult, error) {
	handle, ok := d.tasks.Get(taskID)
	if !ok {
		return nil, drivers.ErrTaskNotFound

View on GitHub (pinned to 482b49bf1a)