{"record":{"id":"8aa0b75d02531722","repo":"hashicorp/nomad","slug":"failed-to-write-task-runner-state-v","errorCode":null,"errorMessage":"failed to write task_runner state: %v","messagePattern":"failed to write task_runner state: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/state/db_bolt.go","lineNumber":708,"sourceCode":"}\n\n// PutTaskRunnerLocalState stores TaskRunner's LocalState or returns an error.\nfunc (s *BoltStateDB) PutTaskRunnerLocalState(allocID, taskName string, val *trstate.LocalState) error {\n\treturn s.db.Update(func(tx *boltdd.Tx) error {\n\t\treturn putTaskRunnerLocalStateImpl(tx, allocID, taskName, val)\n\t})\n}\n\n// putTaskRunnerLocalStateImpl stores TaskRunner's LocalState in an ongoing\n// transaction or returns an error.\nfunc putTaskRunnerLocalStateImpl(tx *boltdd.Tx, allocID, taskName string, val *trstate.LocalState) error {\n\ttaskBkt, err := getTaskBucket(tx, allocID, taskName)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to retrieve allocation bucket: %v\", err)\n\t}\n\n\tif err := taskBkt.Put(taskLocalStateKey, val); err != nil {\n\t\treturn fmt.Errorf(\"failed to write task_runner state: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// PutTaskState stores a task's state or returns an error.\nfunc (s *BoltStateDB) PutTaskState(allocID, taskName string, state *structs.TaskState) error {\n\treturn s.db.Update(func(tx *boltdd.Tx) error {\n\t\treturn putTaskStateImpl(tx, allocID, taskName, state)\n\t})\n}\n\n// putTaskStateImpl stores a task's state in an ongoing transaction or returns\n// an error.\nfunc putTaskStateImpl(tx *boltdd.Tx, allocID, taskName string, state *structs.TaskState) error {\n\ttaskBkt, err := getTaskBucket(tx, allocID, taskName)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to retrieve allocation bucket: %v\", err)","sourceCodeStart":690,"sourceCodeEnd":726,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/state/db_bolt.go#L690-L726","documentation":"This error wraps a bolt/bbolt Put failure when persisting a TaskRunner's local state (taskLocalStateKey) into the task's bucket inside an ongoing transaction, via putTaskRunnerLocalStateImpl. boltdd.Tx Put returns an error only when the transaction is not writable or an underlying bbolt error (e.g. key too large, db corruption, db closed) occurs. The original bolt error is preserved in the wrapped message.","triggerScenarios":"Calling PutTaskRunnerLocalState (or the upgradeAllocBucket migration path) while the underlying bolt transaction is read-only, the bolt DB file is closed/corrupted, or the disk is full so the write fails.","commonSituations":"Nomad client state database (client state dir) on a full disk or corrupted bolt file; a caller opened a read-only View() transaction but tried to write local state; DB file permissions changed under the Nomad client process.","solutions":["Check disk space and write permissions on the Nomad client data_dir hosting the bolt file","Retry after nomad client restart; if the bolt file is corrupted, stop the client and remove the state db so it is recreated","Ensure callers use Update (writable tx), not View, when writing task runner local state","Inspect the wrapped inner error (%v) for the real bbolt cause"],"exampleFix":"// before\ntaskBkt, err := getTaskBucket(tx, allocID, taskName)\nif err != nil {\n    return fmt.Errorf(\"failed to retrieve allocation bucket: %v\", err)\n}\n// after: fail fast on read-only transactions before Put\nif !tx.Writable() {\n    return fmt.Errorf(\"cannot write task_runner state: transaction is not writable\")\n}\ntaskBkt, err := getTaskBucket(tx, allocID, taskName)\nif err != nil {\n    return fmt.Errorf(\"failed to retrieve allocation bucket: %v\", err)\n}","handlingStrategy":"try-catch","validationCode":"if err := os.Chmod(path.Join(dataDir, \"state\"), 0o700); err != nil {\n    return fmt.Errorf(\"state dir not writable: %w\", err)\n}","typeGuard":"func isWritableTx(tx *boltdd.Tx) bool { return tx.Writable() }","tryCatchPattern":"err := db.PutTaskRunnerLocalState(allocID, taskName, ls)\nif err != nil {\n    var oe * boltsvc.OpError\n    if errors.As(err, &oe) && isDiskFull(oe) {\n        // free space / alert before retrying\n    }\n    return fmt.Errorf(\"persisting task runner local state failed: %w\", err)\n}","preventionTips":["Monitor free disk space on the Nomad client data_dir","Always write state inside Update (writable) transactions, never View","On repeated failures, check bolt file integrity and recreate the state DB","Log the wrapped inner bolt error for root-cause triage"],"tags":["database","bolt","storage","write-failure"],"backgroundTag":"bolt-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}