{"record":{"id":"e1707d4879995405","repo":"hashicorp/nomad","slug":"failed-to-store-driver-state-v","errorCode":null,"errorMessage":"failed to store driver state: %v","messagePattern":"failed to store driver state: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":306,"sourceCode":"\n\tif loggingIsEnabled(d.config, handle.Config) {\n\t\th.dlogger, h.dloggerPluginClient, err = d.reattachToDockerLogger(handleState.ReattachConfig)\n\t\tif err != nil {\n\t\t\td.logger.Warn(\"failed to reattach to docker logger process\", \"error\", err)\n\n\t\t\th.dlogger, h.dloggerPluginClient, err = d.setupNewDockerLogger(container, handle.Config, time.Now())\n\t\t\tif err != nil {\n\t\t\t\tif _, err := dockerClient.ContainerStop(d.ctx, handleState.ContainerID, stopWithZeroTimeout()); err != nil {\n\t\t\t\t\td.logger.Warn(\"failed to stop container during cleanup\", \"container_id\", handleState.ContainerID, \"error\", err)\n\t\t\t\t}\n\t\t\t\treturn fmt.Errorf(\"failed to setup replacement docker logger: %v\", err)\n\t\t\t}\n\n\t\t\tif err := handle.SetDriverState(h.buildState()); err != nil {\n\t\t\t\tif _, err := dockerClient.ContainerStop(d.ctx, handleState.ContainerID, stopWithZeroTimeout()); err != nil {\n\t\t\t\t\td.logger.Warn(\"failed to stop container during cleanup\", \"container_id\", handleState.ContainerID, \"error\", err)\n\t\t\t\t}\n\t\t\t\treturn fmt.Errorf(\"failed to store driver state: %v\", err)\n\t\t\t}\n\t\t}\n\t}\n\n\td.tasks.Set(handle.Config.ID, h)\n\n\t// find a pause container?\n\n\tgo h.run()\n\n\treturn nil\n}\n\nfunc loggingIsEnabled(driverCfg *DriverConfig, taskCfg *drivers.TaskConfig) bool {\n\tif driverCfg.DisableLogCollection {\n\t\treturn false\n\t}\n\tif taskCfg.StderrPath == os.DevNull && taskCfg.StdoutPath == os.DevNull {","sourceCodeStart":288,"sourceCodeEnd":324,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L288-L324","documentation":"After successfully creating a replacement docker logger, RecoverTask persists the updated handle state (new reattach config) via handle.SetDriverState(h.buildState()). If persisting fails, the container is stopped with zero timeout (cleanup) and this error is returned, since recovering a logger whose state cannot be stored would be lost on the next restart.","triggerScenarios":"The underlying state store (task handle state backend, e.g. bolt/state DB) is corrupt, locked, or the disk is full; handle state exceeds size limits; the state directory has wrong permissions; concurrent write contention during recovery.","commonSituations":"Disk full on the client's data partition (/var/lib/... state dir); state database corrupted after a crash; read-only filesystem after disk errors; permission changes on the state directory following a package upgrade or migration.","solutions":["Check disk space and filesystem health on the client state partition (`df -h`, `dmesg` for I/O errors).","Inspect the wrapped SetDriverState error in logs — usually 'database is locked', 'no space left on device', or 'read-only file system'.","If the state DB is corrupt, restore from backup or stop the client and clear/rebuild the task state (tasks will be rescheduled).","Fix permissions/ownership of the client state directory for the user running the agent."],"exampleFix":"// before\nif err := handle.SetDriverState(h.buildState()); err != nil {\n\treturn fmt.Errorf(\"failed to store driver state: %v\", err)\n}\n// after\nif err := handle.SetDriverState(h.buildState()); err != nil {\n\td.logger.Error(\"SetDriverState failed; check state dir disk/permissions\", \"error\", err)\n\treturn fmt.Errorf(\"failed to store driver state: %w\", err)\n}","handlingStrategy":"try-catch","validationCode":"// ensure the state dir is writable and has free space before recovery\nfunc stateDirHealthy(dir string) error {\n\tfi, err := os.Stat(dir)\n\tif err != nil || !fi.IsDir() {\n\t\treturn fmt.Errorf(\"state dir missing: %s\", dir)\n\t}\n\tprobe := filepath.Join(dir, \".write_probe\")\n\tif err := os.WriteFile(probe, []byte(\"ok\"), 0o600); err != nil {\n\t\treturn fmt.Errorf(\"state dir not writable: %w\", err)\n\t}\n\tos.Remove(probe)\n\treturn nil\n}","typeGuard":null,"tryCatchPattern":"// Go: log the wrapped cause and surface disk/state issues\nif err := driver.RecoverTask(handle); err != nil {\n\tif strings.Contains(err.Error(), \"failed to store driver state\") {\n\t\tcause := fmt.Sprintf(\"%v\", errors.Unwrap(err))\n\t\tswitch {\n\t\tcase strings.Contains(cause, \"no space left\"):\n\t\t\tlog.Print(\"free disk space on the client state partition\")\n\t\tcase strings.Contains(cause, \"locked\"):\n\t\t\tlog.Print(\"state DB locked; serialize recovery attempts\")\n\t\tdefault:\n\t\t\tlog.Printf(\"state persist error: %s\", cause)\n\t\t}\n\t}\n}","preventionTips":["Monitor disk space and I/O errors on the agent's state partition.","Avoid concurrent recovery writes; serialize RecoverTask calls per task.","Keep state directory permissions/ownership stable across upgrades.","Back up the state store so a corrupt DB can be restored without data loss."],"tags":["docker","state-persistence","disk","task-recovery"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}