temporalio/temporal · error

Queue slice get task from iterator doesn't belong to its ran

Error message

Queue slice get task from iterator doesn't belong to its range, range: %v, task key %v

What it means

Queue slices are history-service task-queue abstractions that own a key Range; every task returned by a persistence iterator must fall inside the slice's scope range. This panic fires in SelectTasks when the DB iterator yields a task whose key lies outside that range — a broken internal invariant, since iterators are constructed from the slice's own range. It indicates stale iterators after scope changes or corrupted range/scope bookkeeping.

Source

Thrown at service/history/queues/slice.go:392

	}()

	executables := make([]Executable, 0, batchSize)
	for len(executables) < batchSize && len(s.iterators) != 0 {
		if s.iterators[0].HasNext() {
			task, err := s.iterators[0].Next()
			if err != nil {
				s.iterators[0] = s.iterators[0].Remaining()
				if len(executables) != 0 {
					// NOTE: we must return the executables here
					// MoreTasks() will return true so queue reader will try to load again
					return executables, nil
				}
				return nil, err
			}

			taskKey := task.GetKey()
			if !s.scope.Range.ContainsKey(taskKey) {
				panic(fmt.Sprintf("Queue slice get task from iterator doesn't belong to its range, range: %v, task key %v",
					s.scope.Range, taskKey))
			}

			if !s.scope.Predicate.Test(task) {
				continue
			}

			executable := s.executableFactory.NewExecutable(task, readerID)
			s.add(executable)
			executables = append(executables, executable)
		} else {
			s.iterators = s.iterators[1:]
		}
	}

	return executables, nil
}

View on GitHub (pinned to bde624efd1)

Solutions

1. Verify all iterators held by the slice are rebuilt whenever scope.Range changes (Clear/ShrinkScope already do this; check custom paths)
2. Ensure slices are not used concurrently from multiple goroutines without the queue's lock
3. Check the persistence iterator (paginationFnProvider) returns only keys within the range it was created for
4. Capture the panic message's range and task key and report to Temporal with shard/task details if reproducible

Example fix

// before: iterator created once at slice construction, reused after scope change
iterators := []Iterator{NewIterator(s.paginationFnProvider, oldRange)}
// after: rebuild iterators whenever the scope range changes (as Clear does)
s.ShrinkScope()
s.iterators = []Iterator{NewIterator(s.paginationFnProvider, s.scope.Range)}

// Before relying on slice behavior, confirm any custom iterator stays in range:
if !slice.Range().ContainsKey(task.GetKey()) {
  return fmt.Errorf("iterator yielded task %v outside slice range %v", task.GetKey(), slice.Range())
}

func taskInRange(r queues.Range, task tasks.Task) bool {
  return r.ContainsKey(task.GetKey())
}

// This is a panic, not an error. Wrap queue operations with recover only at
// goroutine boundaries to convert an invariant bug into a logged failure:
func safeSelect(slice queues.Slice) (executables []tasks.Executable, err error) {
  defer func() { if r := recover(); r != nil { err = fmt.Errorf("queue slice panic: %v", r) } }()
  return slice.SelectTasks()
}

When it happens

Trigger: SliceImpl.SelectTasks iterating after SplitByPredicate/ShrinkScope/Clear mutated the scope while an old iterator with the pre-split range is still in s.iterators; a persistence iterator implementation returning tasks beyond its declared range; race where the slice scope was updated concurrently with SelectTasks.

Common situations: Seen only in temporal-server history service internals; typically after a bug in queue split/merge logic, a version-upgrade mixing old and new slice logic, or a custom tasks.Iterator implementation that ignores its range bounds.

Related errors

• unknown workflow state sync result: %d
• Can not append slice to existing list of slices, incoming sl
• Found overlapping incoming slices, left slice range: %v, rig
• reader with ID %v already exists
• Unable to split scope with range %v at %v

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/0b16882ad91b7ba9. Report an issue: GitHub.