{"record":{"id":"93fe3dc91a40b5f9","repo":"hashicorp/nomad","slug":"failed-to-set-driver-state-v-93fe3d","errorCode":null,"errorMessage":"failed to set driver state: %v","messagePattern":"failed to set driver state: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/rawexec/driver.go","lineNumber":476,"sourceCode":"\t\ttaskConfig:   cfg,\n\t\tprocState:    drivers.TaskStateRunning,\n\t\tstartedAt:    time.Now().Round(time.Millisecond),\n\t\tlogger:       d.logger,\n\t\tdoneCh:       make(chan struct{}),\n\t}\n\n\tdriverState := TaskState{\n\t\tReattachConfig: pstructs.ReattachConfigFromGoPlugin(pluginClient.ReattachConfig()),\n\t\tPid:            ps.Pid,\n\t\tTaskConfig:     cfg,\n\t\tStartedAt:      h.startedAt,\n\t}\n\n\tif err := handle.SetDriverState(&driverState); err != nil {\n\t\td.logger.Error(\"failed to start task, error setting driver state\", \"error\", err)\n\t\t_ = exec.Shutdown(\"\", 0)\n\t\tpluginClient.Kill()\n\t\treturn nil, nil, fmt.Errorf(\"failed to set driver state: %v\", err)\n\t}\n\n\td.tasks.Set(cfg.ID, h)\n\tgo h.run()\n\treturn handle, nil, nil\n}\n\nfunc (d *Driver) WaitTask(ctx context.Context, taskID string) (<-chan *drivers.ExitResult, error) {\n\thandle, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn nil, drivers.ErrTaskNotFound\n\t}\n\n\tch := make(chan *drivers.ExitResult)\n\tgo d.handleWait(ctx, handle, ch)\n\n\treturn ch, nil\n}","sourceCodeStart":458,"sourceCodeEnd":494,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/rawexec/driver.go#L458-L494","documentation":"After the task launched successfully, StartTask failed to persist the DriverState (pid, task config snapshot) into the task handle via handle.SetDriverState. Because recovery depends on this state, the driver shuts the just-started task down and kills the plugin rather than leak an unrecoverable process.","triggerScenarios":"handle.SetDriverState returns an error — the handle's state blob cannot be encoded/written (typically a JSON marshal failure of driverState or an internal handle write error) during StartTask.","commonSituations":"Very rare; indicates a Nomad bug or state-encoding issue (e.g. unexpected values in TaskConfig) rather than user misconfiguration; can appear alongside disk/state store problems on the client.","solutions":["Capture the wrapped %v and agent logs and report to Nomad if reproducible — this path also shuts down the spawned task automatically","Check client disk health/writability of data_dir and restart the agent","Retry the allocation (nomad alloc stop) to get a fresh StartTask attempt","Confirm Nomad client version consistency to rule out state-encoding schema bugs"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// ensure client state store is writable before scheduling\ntouch <data_dir>/client/.writecheck && rm <data_dir>/client/.writecheck","typeGuard":null,"tryCatchPattern":"h, net, err := d.StartTask(cfg)\nif err != nil && strings.Contains(err.Error(), \"failed to set driver state\") {\n  // driver already shut the task down; safe to retry fresh\n  return retryStartTask(cfg, 1)\n}","preventionTips":["Keep data_dir on healthy, writable local storage","Keep Nomad up to date — this path usually indicates a bug","Avoid unclean client kills (SIGKILL) that can leave state tooling in bad shape","Monitor and reschedule allocations that fail StartTask repeatedly"],"tags":["nomad","rawexec","state-persistence","starttask","cleanup"],"backgroundTag":"driver-state-save-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}