hashicorp/nomad · error

failed to set driver state: %v

Error message

failed to set driver state: %v

What it means

StartTask successfully launched the Java executor and built the task handle, but persisting the driver state snapshot via handle.SetDriverState failed. Since the task state could not be recorded, the driver cannot reliably recover or track the task, so it shuts the executor down and aborts the start with this wrapped error. The failure reason is always in the wrapped %v detail.

Source

Thrown at drivers/java/driver.go:548

		pid:          ps.Pid,
		pluginClient: pluginClient,
		taskConfig:   cfg,
		procState:    drivers.TaskStateRunning,
		startedAt:    time.Now().Round(time.Millisecond),
		logger:       d.logger,
	}

	driverState := TaskState{
		ReattachConfig: pstructs.ReattachConfigFromGoPlugin(pluginClient.ReattachConfig()),
		Pid:            ps.Pid,
		TaskConfig:     cfg,
		StartedAt:      h.startedAt,
	}

	if err := handle.SetDriverState(&driverState); err != nil {
		d.logger.Error("failed to start task, error setting driver state", "error", err)
		exec.Shutdown("", 0)
		return nil, nil, fmt.Errorf("failed to set driver state: %v", err)
	}

	d.tasks.Set(cfg.ID, h)
	go h.run()
	return handle, nil, nil
}

func javaCmdArgs(driverConfig TaskConfig) []string {
	var args []string

	// Look for jvm options
	if len(driverConfig.JvmOpts) != 0 {
		args = append(args, driverConfig.JvmOpts...)
	}

	// Add the classpath
	if driverConfig.ClassPath != "" {
		args = append(args, "-cp", driverConfig.ClassPath)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped %v error to find the underlying SetDriverState failure cause
2. Verify the Nomad client data_dir exists, is writable, and has free disk space
3. Restart the Nomad client agent to rebuild the driver plugin connection
4. Retry the task start (e.g. reschedule); the failed executor was already shut down so no orphan process remains
5. Check client logs for 'failed to start task, error setting driver state' for the full underlying error

Example fix

// before: no diagnostics on state dir
drivers check
// after: verify state dir is writable before starting tasks
sudo ls -ld /var/lib/nomad/data
sudo -u nomad touch /var/lib/nomad/data/.write_test && rm /var/lib/nomad/data/.write_test

// before starting, ensure the client state dir is writable
if info, err := os.Stat(dataDir); err != nil || !info.IsDir() {
    return fmt.Errorf("client data dir missing: %s", dataDir)
}

handle, _, err := driver.StartTask(cfg)
if err != nil {
    if strings.Contains(err.Error(), "failed to set driver state") {
        // underlying executor was shut down; inspect wrapped error and reschedule
        logger.Error("start aborted by state persistence failure", "cause", err)
        return retryAfterDelay()
    }
    return err
}

When it happens

Trigger: Calling Driver.StartTask when handle.SetDriverState (driver handle state persistence, typically writing state through the plugin/state store) returns an error after the executor has already been spawned. The executor is shut down via exec.Shutdown("", 0) before returning this error.

Common situations: State store or plugin client I/O failures (disk full, permissions on the data dir), corrupted state persistence, serialization issues with the driverState struct, or the underlying plugin connection dropping between executor start and state write.

Related errors

• failed to set driver state: %v
• default_pid_mode must be %q or %q, got %q
• default_ipc_mode must be %q or %q, got %q
• allow_caps configured with capabilities not supported by sys
• pid_mode must be %q or %q, got %q

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0a95261222d651e4. Report an issue: GitHub.