temporalio/temporal · critical

ExecutableTaskTracker encountered lower high watermark: %v <

Error message

ExecutableTaskTracker encountered lower high watermark: %v < %v

What it means

ExecutableTaskTracker tracks in-flight replication tasks ordered by task ID under an exclusive high watermark sent by the source cluster. The contract is that every tracked task ID is strictly below the new high watermark; TrackTasks panics if the last task it just accepted has an ID >= the incoming watermark. This means the source sent a watermark inconsistent with its task stream — an ordering/protocol violation between source and target clusters.

Source

Thrown at service/history/replication/executable_task_tracker.go:99

	}

	lastTaskID := int64(-1)
	if item := t.taskQueue.Back(); item != nil {
		lastTaskID = item.Value.(TrackableExecutableTask).TaskID()
	}
Loop:
	for _, task := range tasks {
		if lastTaskID >= task.TaskID() {
			// need to assume source side send replication tasks in order
			continue Loop
		}
		t.taskQueue.PushBack(task)
		filteredTasks = append(filteredTasks, task)
		lastTaskID = task.TaskID()
	}

	if exclusiveHighWatermarkInfo.Watermark <= lastTaskID {
		panic(fmt.Sprintf(
			"ExecutableTaskTracker encountered lower high watermark: %v < %v",
			exclusiveHighWatermarkInfo.Watermark,
			lastTaskID,
		))
	}
	t.exclusiveHighWatermarkInfo = &exclusiveHighWatermarkInfo

	if t.cancelled {
		t.cancelLocked()
	}
	return filteredTasks
}

func (t *ExecutableTaskTrackerImpl) LowWatermark() *WatermarkInfo {
	t.Lock()
	defer t.Unlock()

	element := t.taskQueue.Front()

View on GitHub (pinned to bde624efd1)

Solutions

1. Verify the source cluster version matches the target (upgrade/patch the source if known-buggy)
2. Capture the two watermark values from the panic and compare with the source shard's task IDs to find where ordering broke
3. Check for concurrent writers calling TrackTasks on the same tracker without proper stream-level serialization
4. Report to temporalio/temporal with both cluster versions and the panic values — this indicates a replication protocol bug

Example fix

// before: calling TrackTasks with a locally computed watermark that can lag the task batch
tracker.TrackTasks(WatermarkInfo{Watermark: savedWatermark}, tasks...)
// after: use the watermark carried by the same source batch, which is guaranteed > all task IDs
tracker.TrackTasks(batch.ExclusiveHighWatermark, batch.Tasks...)

// Before calling TrackTasks, assert the batch invariant yourself:
if batch.ExclusiveHighWatermark <= maxTaskID(batch.Tasks) {
  return fmt.Errorf("source batch watermark %v <= max task ID %v; dropping batch",
    batch.ExclusiveHighWatermark, maxTaskID(batch.Tasks))
}

func safeTrack(t *replication.ExecutableTaskTracker, wm replication.WatermarkInfo, tasks ...replication.TrackableExecutableTask) (valid []replication.TrackableExecutableTask, err error) {
  defer func() { if r := recover(); r != nil { err = fmt.Errorf("watermark panic: %v", r) } }()
  return t.TrackTasks(wm, tasks...), nil
}

When it happens

Trigger: Source cluster sending replication tasks after (or with) a high watermark smaller than or equal to those tasks' IDs — e.g. watermark regression after shard restart/failover, out-of-order task delivery across stream shards, or a source-side bug computing the watermark.

Common situations: Multi-cluster replication with a source cluster on a buggy or skewed version; task stream reconnects replaying tasks already accounted for with a stale watermark; clock/shard ownership changes on the source.

Related errors

• %w: state namespace failover version < ref namespace failove
• %w: state namespace failover version > ref namespace failove
• %w: state transition count < ref transition count: %v < %v
• current cluster %q is missing from cluster metadata
• ErrStaleState

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/598faf7eaa455e5a. Report an issue: GitHub.