hashicorp/nomad · error

failed to write task_runner state: %v

Error message

failed to write task_runner state: %v

What it means

This error wraps a bolt/bbolt Put failure when persisting a TaskRunner's local state (taskLocalStateKey) into the task's bucket inside an ongoing transaction, via putTaskRunnerLocalStateImpl. boltdd.Tx Put returns an error only when the transaction is not writable or an underlying bbolt error (e.g. key too large, db corruption, db closed) occurs. The original bolt error is preserved in the wrapped message.

Source

Thrown at client/state/db_bolt.go:708

}

// PutTaskRunnerLocalState stores TaskRunner's LocalState or returns an error.
func (s *BoltStateDB) PutTaskRunnerLocalState(allocID, taskName string, val *trstate.LocalState) error {
	return s.db.Update(func(tx *boltdd.Tx) error {
		return putTaskRunnerLocalStateImpl(tx, allocID, taskName, val)
	})
}

// putTaskRunnerLocalStateImpl stores TaskRunner's LocalState in an ongoing
// transaction or returns an error.
func putTaskRunnerLocalStateImpl(tx *boltdd.Tx, allocID, taskName string, val *trstate.LocalState) error {
	taskBkt, err := getTaskBucket(tx, allocID, taskName)
	if err != nil {
		return fmt.Errorf("failed to retrieve allocation bucket: %v", err)
	}

	if err := taskBkt.Put(taskLocalStateKey, val); err != nil {
		return fmt.Errorf("failed to write task_runner state: %v", err)
	}

	return nil
}

// PutTaskState stores a task's state or returns an error.
func (s *BoltStateDB) PutTaskState(allocID, taskName string, state *structs.TaskState) error {
	return s.db.Update(func(tx *boltdd.Tx) error {
		return putTaskStateImpl(tx, allocID, taskName, state)
	})
}

// putTaskStateImpl stores a task's state in an ongoing transaction or returns
// an error.
func putTaskStateImpl(tx *boltdd.Tx, allocID, taskName string, state *structs.TaskState) error {
	taskBkt, err := getTaskBucket(tx, allocID, taskName)
	if err != nil {
		return fmt.Errorf("failed to retrieve allocation bucket: %v", err)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check disk space and write permissions on the Nomad client data_dir hosting the bolt file
2. Retry after nomad client restart; if the bolt file is corrupted, stop the client and remove the state db so it is recreated
3. Ensure callers use Update (writable tx), not View, when writing task runner local state
4. Inspect the wrapped inner error (%v) for the real bbolt cause

Example fix

// before
taskBkt, err := getTaskBucket(tx, allocID, taskName)
if err != nil {
    return fmt.Errorf("failed to retrieve allocation bucket: %v", err)
}
// after: fail fast on read-only transactions before Put
if !tx.Writable() {
    return fmt.Errorf("cannot write task_runner state: transaction is not writable")
}
taskBkt, err := getTaskBucket(tx, allocID, taskName)
if err != nil {
    return fmt.Errorf("failed to retrieve allocation bucket: %v", err)
}

if err := os.Chmod(path.Join(dataDir, "state"), 0o700); err != nil {
    return fmt.Errorf("state dir not writable: %w", err)
}

func isWritableTx(tx *boltdd.Tx) bool { return tx.Writable() }

err := db.PutTaskRunnerLocalState(allocID, taskName, ls)
if err != nil {
    var oe * boltsvc.OpError
    if errors.As(err, &oe) && isDiskFull(oe) {
        // free space / alert before retrying
    }
    return fmt.Errorf("persisting task runner local state failed: %w", err)
}

When it happens

Trigger: Calling PutTaskRunnerLocalState (or the upgradeAllocBucket migration path) while the underlying bolt transaction is read-only, the bolt DB file is closed/corrupted, or the disk is full so the write fails.

Common situations: Nomad client state database (client state dir) on a full disk or corrupted bolt file; a caller opened a read-only View() transaction but tried to write local state; DB file permissions changed under the Nomad client process.

Related errors

• Allocations bucket doesn't exist and transaction is not writ
• Allocation bucket doesn't exist and transaction is not writa
• Task bucket doesn't exist and transaction is not writable
• failed to write data at key %s: %v
• node update failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8aa0b75d02531722. Report an issue: GitHub.