{"record":{"id":"0b1b075635b68747","repo":"hashicorp/nomad","slug":"failed-to-set-driver-state-v","errorCode":null,"errorMessage":"failed to set driver state: %v","messagePattern":"failed to set driver state: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/exec/driver.go","lineNumber":569,"sourceCode":"\t\tpid:          ps.Pid,\n\t\tpluginClient: pluginClient,\n\t\ttaskConfig:   cfg,\n\t\tprocState:    drivers.TaskStateRunning,\n\t\tstartedAt:    time.Now().Round(time.Millisecond),\n\t\tlogger:       d.logger,\n\t}\n\n\tdriverState := TaskState{\n\t\tReattachConfig: pstructs.ReattachConfigFromGoPlugin(pluginClient.ReattachConfig()),\n\t\tPid:            ps.Pid,\n\t\tTaskConfig:     cfg,\n\t\tStartedAt:      h.startedAt,\n\t}\n\n\tif err := handle.SetDriverState(&driverState); err != nil {\n\t\td.logger.Error(\"failed to start task, error setting driver state\", \"error\", err)\n\t\t_ = exec.Shutdown(\"\", 0)\n\t\treturn nil, nil, fmt.Errorf(\"failed to set driver state: %v\", err)\n\t}\n\n\td.tasks.Set(cfg.ID, h)\n\tgo h.run()\n\treturn handle, nil, nil\n}\n\nfunc (d *Driver) WaitTask(ctx context.Context, taskID string) (<-chan *drivers.ExitResult, error) {\n\thandle, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn nil, drivers.ErrTaskNotFound\n\t}\n\n\tch := make(chan *drivers.ExitResult)\n\tgo d.handleWait(ctx, handle, ch)\n\n\treturn ch, nil\n}","sourceCodeStart":551,"sourceCodeEnd":587,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/exec/driver.go#L551-L587","documentation":"After a successful launch, StartTask serializes the driver state (PID, task ID, started-at) into the TaskHandle via handle.SetDriverState so the task can be recovered after a client restart. If persisting this state fails, the driver shuts the executor down and aborts the start to avoid an unrecoverable orphaned task.","triggerScenarios":"handle.SetDriverState(&driverState) fails, typically because the underlying client state store (bolt DB) write fails: disk full, state DB corrupted, I/O errors, or permissions on the Nomad data dir.","commonSituations":"Disk full on the Nomad client; corrupted /var/lib/nomad/client state database; read-only mount of the data dir; permission changes after backups/restores.","solutions":["Check disk space (df -h) and the health of the Nomad client data dir.","Inspect client logs for the underlying state store error to identify I/O vs corruption.","Fix data-dir permissions/ownership for the nomad user.","If the state DB is corrupt, stop the client and restore or (as last resort) reinitialize the client state, rescheduling allocations."],"exampleFix":"// before: disk full\n$ df -h /var/lib/nomad\n/dev/sdb 100% used\n// after\n$ sudo find /var/log -name '*.gz' -delete  # free space\n$ sudo systemctl restart nomad\n# then reschedule the allocation","handlingStrategy":"validation","validationCode":"// check the client state store can accept writes before starting tasks\nstatePath := filepath.Join(dataDir, \"client\", \"state.db\")\nf, err := os.OpenFile(statePath+\".probe\", os.O_CREATE|os.O_WRONLY, 0o600)\nif err != nil {\n    return fmt.Errorf(\"state store not writable: %w\", err)\n}\nf.Close(); os.Remove(statePath + \".probe\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Alert on client disk usage above ~85% to prevent failed state writes.","Keep /var/lib/nomad on reliable, non-read-only storage.","Back up and validate the client state DB before Nomad upgrades.","Fix nomad user ownership of the data dir after restores."],"tags":["nomad","exec-driver","state-persistence","boltdb","disk"],"backgroundTag":"driver-state-save-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}