prism error: negative watermark hold count %v for time %v

Error message

prism error: negative watermark hold count %v for time %v

What it means

holdTracker.Drop removes v counts of a watermark hold at a given time. If subtracting yields a negative count, the tracker's bookkeeping is inconsistent — more holds are being dropped than were ever added — so Prism panics to avoid silently corrupting watermark advancement. It is raised from Drop, called by addPending and splitBundle.

Source

Thrown at sdks/go/pkg/beam/runners/prism/internal/engine/holds.go:87

	heap   mtimeHeap
	counts map[mtime.Time]int
}

func newHoldTracker() *holdTracker {
	return &holdTracker{
		counts: map[mtime.Time]int{},
	}
}

// Drop the given hold count. When the count of a hold time reaches zero, it's
// removed from the heap. Drop panics if holds become negative.
func (ht *holdTracker) Drop(hold mtime.Time, v int) {
	n := ht.counts[hold] - v
	if n > 0 {
		ht.counts[hold] = n
		return
	} else if n < 0 {
		panic(fmt.Sprintf("prism error: negative watermark hold count %v for time %v", n, hold))
	}
	delete(ht.counts, hold)
	ht.heap.Remove(hold)
}

// Add a hold a number of times to heap. If the hold time isn't already present in the heap, it is added.
func (ht *holdTracker) Add(hold mtime.Time, v int) {
	// Mark the hold in the heap.
	ht.counts[hold] += v
	if len(ht.counts) != len(ht.heap) {
		// Since there's a difference, the hold should not be in the heap, so we add it.
		heap.Push(&ht.heap, hold)
	}
}

// Min returns the earliest hold in the heap. Returns [mtime.MaxTimestamp] if the heap is empty.
func (ht *holdTracker) Min() mtime.Time {
	minWatermarkHold := mtime.MaxTimestamp

View on GitHub (pinned to 12126d8942)

Solutions

Check whether bundle splitting (splitBundle) is double-dropping holds; disable or reduce splitting (e.g. avoid fractional split attempts) to confirm
Upgrade Beam — hold-tracking bugs around splits have been fixed in past releases
Reproduce with a minimal pipeline using timers/state and event-time holds and report it if it persists
As a local diagnostic, log every Add/Drop pair to identify the unbalanced operation

Example fix

// before: unbalanced drop on split
ht.Drop(holdTime, removedCount) // removedCount includes holds not tracked
// after: clamp/verify before dropping
if tracked := ht.counts[holdTime]; removedCount <= tracked {
    ht.Drop(holdTime, removedCount)
} else {
    slog.Warn("skipping over-drop", "have", tracked, "want", removedCount)
}

Defensive patterns

Strategy: try-catch

Try / catch

defer func() {
    if r := recover(); r != nil {
        if strings.Contains(fmt.Sprint(r), "negative watermark hold count") {
            log.Printf("hold tracker invariant failure: %v", r)
        }
        panic(r)
    }
}()

Prevention

Instrument Add/Drop call sites when using timers with bundle splitting
Keep Beam versions current; hold-tracking around splits has known past fixes
Reproduce with minimal timers+state pipelines before filing bugs

When it happens

Trigger: Calling Drop with a count v exceeding the recorded count for that hold time, e.g. double-dropping the same hold after bundle splitting or element re-pending — an accounting bug in bundle split/hold lifecycle.

Common situations: Dynamic work rebalancing (bundle splitting) combined with stateful/timer holds, or a runner bug where holds are added under one time and dropped under another.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4de264ccbf402f46. Report an issue: GitHub.