{"record":{"id":"d23daebf0adf2183","repo":"temporalio/temporal","slug":"found-key-with-non-zero-pending-task-count-but-has","errorCode":null,"errorMessage":"Found key with non-zero pending task count but has no correspoding Slice","messagePattern":"Found key with non-zero pending task count but has no correspoding Slice","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"service/history/queues/action_pending_task_count.go","lineNumber":140,"sourceCode":"\t// order key by # of pending tasks\n\tkeys := make([]any, 0, len(a.tasksPerKey))\n\tfor key, keyPendingTasks := range a.tasksPerKey {\n\t\tcurrentPendingTasks += keyPendingTasks\n\t\tkeys = append(keys, key)\n\t}\n\tpq := collection.NewPriorityQueueWithItems(\n\t\tfunc(this, that any) bool {\n\t\t\treturn a.tasksPerKey[this] > a.tasksPerKey[that]\n\t\t},\n\t\tkeys,\n\t)\n\n\tfor currentPendingTasks > targetPendingTasks && !pq.IsEmpty() {\n\t\tkey := pq.Remove()\n\n\t\tsliceList := a.slicesPerKey[key]\n\t\tif len(sliceList) == 0 {\n\t\t\tpanic(\"Found key with non-zero pending task count but has no correspoding Slice\")\n\t\t}\n\n\t\t// pop the first slice in the list\n\t\tsliceToClear := sliceList[0]\n\t\tsliceList = sliceList[1:]\n\t\ta.slicesPerKey[key] = sliceList\n\n\t\ttasksCleared := a.pendingTasksPerKeyPerSlice[sliceToClear][key]\n\t\ta.tasksPerKey[key] -= tasksCleared\n\t\tcurrentPendingTasks -= tasksCleared\n\t\tif a.tasksPerKey[key] > 0 {\n\t\t\tpq.Add(key)\n\t\t}\n\n\t\ta.keysToClearPerSlice[sliceToClear] = append(a.keysToClearPerSlice[sliceToClear], key)\n\t}\n}\n","sourceCodeStart":122,"sourceCodeEnd":158,"githubUrl":"https://github.com/temporalio/temporal/blob/bde624efd13fbd3843654058db6d9c716166318b/service/history/queues/action_pending_task_count.go#L122-L158","documentation":"This panic fires in findSliceToClear when the priority queue reports a key with a non-zero pending task count, but the internal slicesPerKey map has no Slice registered for that key. It is an internal invariant violation in the queue's bookkeeping: the pending-task counter and the slice registry are out of sync. The typo \"correspoding\" is in the original panic message.","triggerScenarios":"Run calls findSliceToClear to reduce pending tasks; pq.Remove() returns a key whose slicesPerKey entry is empty or missing because slices were cleared/removed without decrementing their pending task counts, or a slice was registered under a different key.","commonSituations":"Custom or modified queue actions that clear slices directly; bugs in Slice removal paths that forget to update slicesPerKey or the pending task counter; race conditions if the action is used outside the single-threaded context it was designed for.","solutions":["Find where slices are removed/cleared from a.slicesPerKey and ensure the corresponding pending task count key is removed from the priority queue at the same time","Verify the key type used for pq and slicesPerKey matches exactly (task key comparison semantics)","Upgrade to a version containing the fix if this is a known temporal history invariant bug","Capture the key, pending count, and pq state in logs before the panic to file a reproducible report"],"exampleFix":"// before\nsliceList := a.slicesPerKey[key]\nif len(sliceList) == 0 {\n    panic(\"Found key with non-zero pending task count but has no correspoding Slice\")\n}\n// after\nsliceList := a.slicesPerKey[key]\nif len(sliceList) == 0 {\n    a.logger.Error(\"invariant violated: key in pending-task queue has no slices\",\n        tag.Key(key), tag.PendingTaskCount(currentPendingTasks))\n    continue\n}","handlingStrategy":"validation","validationCode":"if sliceList, ok := action.slicesPerKey[key]; !ok || len(sliceList) == 0 {\n    // do not invoke the clear path for this key\n    return\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always update pq and slicesPerKey in the same code path","Write an invariant-check unit test that clears slices and asserts counter/registry consistency","Never mutate queue bookkeeping outside the action's Run method"],"tags":["go","panic","invariant-violation","history-service"],"backgroundTag":"internal-state-invariant-violation","analyzedSha":"bde624efd13fbd3843654058db6d9c716166318b","analyzedAt":"2026-09-01T07:18:39.080Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}