temporalio/temporal · error

ExecutableTaskTracker encountered unknown task state: %v

Error message

ExecutableTaskTracker encountered unknown task state: %v

What it means

LowWatermark scans the tracker's task queue to compute the ackable low watermark, switching on each task's state (Acked, Nacked, Aborted, Cancelled, Pending). A state outside this set means a TrackableExecutableTask returned an unexpected State() value, so the function panics rather than computing a wrong watermark. It is an exhaustiveness guard on the task-state enum.

Source

Thrown at service/history/replication/executable_task_tracker.go:151

				)
				// unable to save poison pill, retry later
				element = element.Next()
				continue Loop
			}
			nextElement := element.Next()
			t.taskQueue.Remove(element)
			element = nextElement
		case ctasks.TaskStateAborted:
			// noop, do not remove from queue, let it block low watermark
			element = element.Next()
		case ctasks.TaskStateCancelled:
			// noop, do not remove from queue, let it block low watermark
			element = element.Next()
		case ctasks.TaskStatePending:
			// noop, do not remove from queue, let it block low watermark
			element = element.Next()
		default:
			panic(fmt.Sprintf(
				"ExecutableTaskTracker encountered unknown task state: %v",
				taskState,
			))
		}
	}

	if element := t.taskQueue.Front(); element != nil {
		inclusiveLowWatermarkInfo := WatermarkInfo{
			Watermark: element.Value.(TrackableExecutableTask).TaskID(),
			Timestamp: element.Value.(TrackableExecutableTask).TaskCreationTime(),
		}
		return &inclusiveLowWatermarkInfo
	} else if t.exclusiveHighWatermarkInfo != nil {
		inclusiveLowWatermarkInfo := *t.exclusiveHighWatermarkInfo
		return &inclusiveLowWatermarkInfo
	} else {
		return nil
	}

View on GitHub (pinned to bde624efd1)

Solutions

1. Add the missing task state case to LowWatermark's switch, deciding whether it should block the low watermark (like Pending) or be removable (like Acked)
2. Grep ctasks.TaskState* constants and verify every one is handled in the switch
3. Check any custom TrackableExecutableTask implementations return only defined states
4. Run with -race if hit on an unmodified build to rule out a state race

Example fix

// before
case ctasks.TaskStatePending:
  element = element.Next()
default:
  panic(fmt.Sprintf("ExecutableTaskTracker encountered unknown task state: %v", taskState))
// after
case ctasks.TaskStatePending, ctasks.TaskStateNewlyAdded: // handle newly added state
  element = element.Next()
default:
  panic(fmt.Sprintf("ExecutableTaskTracker encountered unknown task state: %v", taskState))

// Validate states are known before tasks enter the tracker:
if !isKnownTaskState(task.State()) {
  return fmt.Errorf("task %v has undefined state %v", task.TaskID(), task.State())
}

func isKnownTaskState(s ctasks.TaskState) bool {
  switch s {
  case ctasks.TaskStateAcked, ctasks.TaskStateNacked, ctasks.TaskStateAborted,
    ctasks.TaskStateCancelled, ctasks.TaskStatePending:
    return true
  }
  return false
}

func safeLowWatermark(t *replication.ExecutableTaskTracker) (wm *replication.WatermarkInfo, err error) {
  defer func() { if r := recover(); r != nil { err = fmt.Errorf("task state panic: %v", r) } }()
  return t.LowWatermark(), nil
}

When it happens

Trigger: A new ctasks.TaskState constant added without updating LowWatermark's switch; a custom/buggy TrackableExecutableTask implementation returning an undefined state; memory corruption of the state field.

Common situations: Forking or upgrading temporal-server where a task state enum was extended on one side only; custom task implementations in internal forks.

Related errors

• upload stream unknown status: %v
• %w: state namespace failover version < ref namespace failove
• %w: state namespace failover version > ref namespace failove
• %w: state transition count < ref transition count: %v < %v
• ErrStaleState

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/82234d38af902fca. Report an issue: GitHub.