hashicorp/nomad · error

failed to set driver state: %v

Error message

failed to set driver state: %v

What it means

After the task launched successfully, StartTask failed to persist the DriverState (pid, task config snapshot) into the task handle via handle.SetDriverState. Because recovery depends on this state, the driver shuts the just-started task down and kills the plugin rather than leak an unrecoverable process.

Source

Thrown at drivers/rawexec/driver.go:476

		taskConfig:   cfg,
		procState:    drivers.TaskStateRunning,
		startedAt:    time.Now().Round(time.Millisecond),
		logger:       d.logger,
		doneCh:       make(chan struct{}),
	}

	driverState := TaskState{
		ReattachConfig: pstructs.ReattachConfigFromGoPlugin(pluginClient.ReattachConfig()),
		Pid:            ps.Pid,
		TaskConfig:     cfg,
		StartedAt:      h.startedAt,
	}

	if err := handle.SetDriverState(&driverState); err != nil {
		d.logger.Error("failed to start task, error setting driver state", "error", err)
		_ = exec.Shutdown("", 0)
		pluginClient.Kill()
		return nil, nil, fmt.Errorf("failed to set driver state: %v", err)
	}

	d.tasks.Set(cfg.ID, h)
	go h.run()
	return handle, nil, nil
}

func (d *Driver) WaitTask(ctx context.Context, taskID string) (<-chan *drivers.ExitResult, error) {
	handle, ok := d.tasks.Get(taskID)
	if !ok {
		return nil, drivers.ErrTaskNotFound
	}

	ch := make(chan *drivers.ExitResult)
	go d.handleWait(ctx, handle, ch)

	return ch, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Capture the wrapped %v and agent logs and report to Nomad if reproducible — this path also shuts down the spawned task automatically
2. Check client disk health/writability of data_dir and restart the agent
3. Retry the allocation (nomad alloc stop) to get a fresh StartTask attempt
4. Confirm Nomad client version consistency to rule out state-encoding schema bugs

// ensure client state store is writable before scheduling
touch <data_dir>/client/.writecheck && rm <data_dir>/client/.writecheck

h, net, err := d.StartTask(cfg)
if err != nil && strings.Contains(err.Error(), "failed to set driver state") {
  // driver already shut the task down; safe to retry fresh
  return retryStartTask(cfg, 1)
}

When it happens

Trigger: handle.SetDriverState returns an error — the handle's state blob cannot be encoded/written (typically a JSON marshal failure of driverState or an internal handle write error) during StartTask.

Common situations: Very rare; indicates a Nomad bug or state-encoding issue (e.g. unexpected values in TaskConfig) rather than user misconfiguration; can appear alongside disk/state store problems on the client.

Related errors

• task with ID %q already started
• only one of cgroups_v1_override and cgroups_v2_override may
• work_dir must be an absolute path
• plugin in use
• post-run hook %q failed: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/93fe3dc91a40b5f9. Report an issue: GitHub.