{"record":{"id":"598faf7eaa455e5a","repo":"temporalio/temporal","slug":"executabletasktracker-encountered-lower-high-water","errorCode":null,"errorMessage":"ExecutableTaskTracker encountered lower high watermark: %v < %v","messagePattern":"ExecutableTaskTracker encountered lower high watermark: (.+?) < (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"service/history/replication/executable_task_tracker.go","lineNumber":99,"sourceCode":"\t}\n\n\tlastTaskID := int64(-1)\n\tif item := t.taskQueue.Back(); item != nil {\n\t\tlastTaskID = item.Value.(TrackableExecutableTask).TaskID()\n\t}\nLoop:\n\tfor _, task := range tasks {\n\t\tif lastTaskID >= task.TaskID() {\n\t\t\t// need to assume source side send replication tasks in order\n\t\t\tcontinue Loop\n\t\t}\n\t\tt.taskQueue.PushBack(task)\n\t\tfilteredTasks = append(filteredTasks, task)\n\t\tlastTaskID = task.TaskID()\n\t}\n\n\tif exclusiveHighWatermarkInfo.Watermark <= lastTaskID {\n\t\tpanic(fmt.Sprintf(\n\t\t\t\"ExecutableTaskTracker encountered lower high watermark: %v < %v\",\n\t\t\texclusiveHighWatermarkInfo.Watermark,\n\t\t\tlastTaskID,\n\t\t))\n\t}\n\tt.exclusiveHighWatermarkInfo = &exclusiveHighWatermarkInfo\n\n\tif t.cancelled {\n\t\tt.cancelLocked()\n\t}\n\treturn filteredTasks\n}\n\nfunc (t *ExecutableTaskTrackerImpl) LowWatermark() *WatermarkInfo {\n\tt.Lock()\n\tdefer t.Unlock()\n\n\telement := t.taskQueue.Front()","sourceCodeStart":81,"sourceCodeEnd":117,"githubUrl":"https://github.com/temporalio/temporal/blob/bde624efd13fbd3843654058db6d9c716166318b/service/history/replication/executable_task_tracker.go#L81-L117","documentation":"ExecutableTaskTracker tracks in-flight replication tasks ordered by task ID under an exclusive high watermark sent by the source cluster. The contract is that every tracked task ID is strictly below the new high watermark; TrackTasks panics if the last task it just accepted has an ID >= the incoming watermark. This means the source sent a watermark inconsistent with its task stream — an ordering/protocol violation between source and target clusters.","triggerScenarios":"Source cluster sending replication tasks after (or with) a high watermark smaller than or equal to those tasks' IDs — e.g. watermark regression after shard restart/failover, out-of-order task delivery across stream shards, or a source-side bug computing the watermark.","commonSituations":"Multi-cluster replication with a source cluster on a buggy or skewed version; task stream reconnects replaying tasks already accounted for with a stale watermark; clock/shard ownership changes on the source.","solutions":["Verify the source cluster version matches the target (upgrade/patch the source if known-buggy)","Capture the two watermark values from the panic and compare with the source shard's task IDs to find where ordering broke","Check for concurrent writers calling TrackTasks on the same tracker without proper stream-level serialization","Report to temporalio/temporal with both cluster versions and the panic values — this indicates a replication protocol bug"],"exampleFix":"// before: calling TrackTasks with a locally computed watermark that can lag the task batch\ntracker.TrackTasks(WatermarkInfo{Watermark: savedWatermark}, tasks...)\n// after: use the watermark carried by the same source batch, which is guaranteed > all task IDs\ntracker.TrackTasks(batch.ExclusiveHighWatermark, batch.Tasks...)","handlingStrategy":"validation","validationCode":"// Before calling TrackTasks, assert the batch invariant yourself:\nif batch.ExclusiveHighWatermark <= maxTaskID(batch.Tasks) {\n  return fmt.Errorf(\"source batch watermark %v <= max task ID %v; dropping batch\",\n    batch.ExclusiveHighWatermark, maxTaskID(batch.Tasks))\n}","typeGuard":null,"tryCatchPattern":"func safeTrack(t *replication.ExecutableTaskTracker, wm replication.WatermarkInfo, tasks ...replication.TrackableExecutableTask) (valid []replication.TrackableExecutableTask, err error) {\n  defer func() { if r := recover(); r != nil { err = fmt.Errorf(\"watermark panic: %v\", r) } }()\n  return t.TrackTasks(wm, tasks...), nil\n}","preventionTips":["Send the watermark carried in the same source batch, never a locally saved one","Serialize TrackTasks calls per stream so watermarks cannot regress","Keep source and target clusters on compatible temporal-server versions","Alert on replication task ordering anomalies (IDs arriving out of order) before they panic"],"tags":["go","replication","watermark","protocol-violation","multi-cluster"],"backgroundTag":"watermark-regression","analyzedSha":"bde624efd13fbd3843654058db6d9c716166318b","analyzedAt":"2026-09-01T07:18:39.080Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}