{"record":{"id":"0a95261222d651e4","repo":"hashicorp/nomad","slug":"failed-to-set-driver-state-v-0a9526","errorCode":null,"errorMessage":"failed to set driver state: %v","messagePattern":"failed to set driver state: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/java/driver.go","lineNumber":548,"sourceCode":"\t\tpid:          ps.Pid,\n\t\tpluginClient: pluginClient,\n\t\ttaskConfig:   cfg,\n\t\tprocState:    drivers.TaskStateRunning,\n\t\tstartedAt:    time.Now().Round(time.Millisecond),\n\t\tlogger:       d.logger,\n\t}\n\n\tdriverState := TaskState{\n\t\tReattachConfig: pstructs.ReattachConfigFromGoPlugin(pluginClient.ReattachConfig()),\n\t\tPid:            ps.Pid,\n\t\tTaskConfig:     cfg,\n\t\tStartedAt:      h.startedAt,\n\t}\n\n\tif err := handle.SetDriverState(&driverState); err != nil {\n\t\td.logger.Error(\"failed to start task, error setting driver state\", \"error\", err)\n\t\texec.Shutdown(\"\", 0)\n\t\treturn nil, nil, fmt.Errorf(\"failed to set driver state: %v\", err)\n\t}\n\n\td.tasks.Set(cfg.ID, h)\n\tgo h.run()\n\treturn handle, nil, nil\n}\n\nfunc javaCmdArgs(driverConfig TaskConfig) []string {\n\tvar args []string\n\n\t// Look for jvm options\n\tif len(driverConfig.JvmOpts) != 0 {\n\t\targs = append(args, driverConfig.JvmOpts...)\n\t}\n\n\t// Add the classpath\n\tif driverConfig.ClassPath != \"\" {\n\t\targs = append(args, \"-cp\", driverConfig.ClassPath)","sourceCodeStart":530,"sourceCodeEnd":566,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/java/driver.go#L530-L566","documentation":"StartTask successfully launched the Java executor and built the task handle, but persisting the driver state snapshot via handle.SetDriverState failed. Since the task state could not be recorded, the driver cannot reliably recover or track the task, so it shuts the executor down and aborts the start with this wrapped error. The failure reason is always in the wrapped %v detail.","triggerScenarios":"Calling Driver.StartTask when handle.SetDriverState (driver handle state persistence, typically writing state through the plugin/state store) returns an error after the executor has already been spawned. The executor is shut down via exec.Shutdown(\"\", 0) before returning this error.","commonSituations":"State store or plugin client I/O failures (disk full, permissions on the data dir), corrupted state persistence, serialization issues with the driverState struct, or the underlying plugin connection dropping between executor start and state write.","solutions":["Read the wrapped %v error to find the underlying SetDriverState failure cause","Verify the Nomad client data_dir exists, is writable, and has free disk space","Restart the Nomad client agent to rebuild the driver plugin connection","Retry the task start (e.g. reschedule); the failed executor was already shut down so no orphan process remains","Check client logs for 'failed to start task, error setting driver state' for the full underlying error"],"exampleFix":"// before: no diagnostics on state dir\ndrivers check\n// after: verify state dir is writable before starting tasks\nsudo ls -ld /var/lib/nomad/data\nsudo -u nomad touch /var/lib/nomad/data/.write_test && rm /var/lib/nomad/data/.write_test","handlingStrategy":"retry","validationCode":"// before starting, ensure the client state dir is writable\nif info, err := os.Stat(dataDir); err != nil || !info.IsDir() {\n    return fmt.Errorf(\"client data dir missing: %s\", dataDir)\n}","typeGuard":null,"tryCatchPattern":"handle, _, err := driver.StartTask(cfg)\nif err != nil {\n    if strings.Contains(err.Error(), \"failed to set driver state\") {\n        // underlying executor was shut down; inspect wrapped error and reschedule\n        logger.Error(\"start aborted by state persistence failure\", \"cause\", err)\n        return retryAfterDelay()\n    }\n    return err\n}","preventionTips":["Monitor client data_dir disk usage and permissions","Watch Nomad client logs for plugin/state-store errors","Reschedule tasks automatically on this error; no orphan executor remains"],"tags":["nomad","java-driver","state-persistence","task-start"],"backgroundTag":"driver-state-persist-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}