{"record":{"id":"730b91074e64c32e","repo":"hashicorp/nomad","slug":"failed-to-set-driver-state-v-730b91","errorCode":null,"errorMessage":"failed to set driver state: %v","messagePattern":"failed to set driver state: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/qemu/driver.go","lineNumber":692,"sourceCode":"\t\tpluginClient: pluginClient,\n\t\ttaskConfig:   cfg,\n\t\tprocState:    drivers.TaskStateRunning,\n\t\tstartedAt:    time.Now().Round(time.Millisecond),\n\t\tlogger:       d.logger,\n\t}\n\n\tqemuDriverState := TaskState{\n\t\tReattachConfig: pstructs.ReattachConfigFromGoPlugin(pluginClient.ReattachConfig()),\n\t\tPid:            ps.Pid,\n\t\tTaskConfig:     cfg,\n\t\tStartedAt:      h.startedAt,\n\t}\n\n\tif err := handle.SetDriverState(&qemuDriverState); err != nil {\n\t\td.logger.Error(\"failed to start task, error setting driver state\", \"error\", err)\n\t\texecImpl.Shutdown(\"\", 0)\n\t\tpluginClient.Kill()\n\t\treturn nil, nil, fmt.Errorf(\"failed to set driver state: %v\", err)\n\t}\n\n\td.tasks.Set(cfg.ID, h)\n\tgo h.run()\n\n\tvar driverNetwork *drivers.DriverNetwork\n\tif len(driverConfig.PortMap) == 1 {\n\t\tdriverNetwork = &drivers.DriverNetwork{\n\t\t\tPortMap: driverConfig.PortMap,\n\t\t}\n\t}\n\treturn handle, driverNetwork, nil\n}\n\nfunc (d *Driver) WaitTask(ctx context.Context, taskID string) (<-chan *drivers.ExitResult, error) {\n\thandle, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn nil, drivers.ErrTaskNotFound","sourceCodeStart":674,"sourceCodeEnd":710,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/qemu/driver.go#L674-L710","documentation":"After successfully launching the VM process, StartTask persists the qemuDriverState (pid, image path, args) into the task handle via handle.SetDriverState. If this write fails, the driver cannot later restore/stop the task across plugin restarts, so it shuts down the exec'd QEMU process, kills the plugin client, and returns this wrapped error. The VM is never left orphaned, but the task fails to start.","triggerScenarios":"Failure writing handle state to the underlying storage — e.g. disk full, permissions problems in the Nomad client data_dir, the handle's backing store already closed, or serialization failure of qemuDriverState.","commonSituations":"Client node with a full or failing disk; corrupted or unwritable Nomad data_dir; client being shut down concurrently while the task is starting; I/O errors on the filesystem hosting the alloc/state directories.","solutions":["Inspect the Nomad client logs for the underlying error after 'failed to set driver state' — it names the storage failure.","Free disk space / fix permissions on the client's data_dir (host volumes holding alloc state).","Retry the job after restoring the client's data_dir; the QEMU process for this attempt is already cleaned up.","If persistent, restart the nomad client so its state store reinitializes, then resubmit the job."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// check client storage health before submitting state-heavy workloads\nconst { execSync } = require(\"child_process\");\nfunction clientDiskHealthy(dataDir) {\n  try {\n    const out = execSync(`df --output=pcent ${dataDir}`).toString();\n    return parseInt(out.match(/(\\d+)%/)[1], 10) < 90;\n  } catch { return false; }\n}","typeGuard":null,"tryCatchPattern":"try {\n  await nomad.jobs.startTask(cfg);\n} catch (e) {\n  if (String(e.message).startsWith(\"failed to set driver state\")) {\n    console.error(\"Client failed to persist task state; check disk space/permissions on data_dir, then resubmit. No orphan VM should remain.\");\n  }\n  throw e;\n}","preventionTips":["Monitor client node disk usage and health (data_dir full is the usual cause).","Ensure the nomad user can write to data_dir and alloc state directories.","Avoid starting tasks during client shutdown/maintenance windows.","On recurrence, read the nested error in the client log (it precedes this message) and resubmit the job."],"tags":["qemu","state-persistence","io-error","nomad-client"],"backgroundTag":"state-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}