What causes "BUG: thread with ID: %v must not access closed Throttler" in vitessio/vitess?

Signal worker threads to stop (context cancel / done channel) and wait for them to exit before calling Close()

BUG: thread with ID: %v must not access closed Throttler

Error message

BUG: thread with ID: %v must not access closed Throttler

What it means

ThrottlerImpl.Throttle() panics if the throttler has been closed (via Close). After close, thread throttlers are torn down and any further Throttle() call from a worker thread is a lifecycle bug on the caller's side.

Source

Thrown at go/vt/throttler/throttler.go:234

	go func() {
		for range rateUpdateChan {
			t.updateMaxRate()
		}
	}()

	return t, nil
}

// Throttle returns a backoff duration which specifies for how long "threadId"
// should wait before it issues the next request.
// If the duration is zero, the thread is not throttled.
// If the duration is not zero, the thread must call Throttle() again after
// the backoff duration elapsed.
// The maximum value for the returned backoff is 1 second since the throttler
// internally operates on a per-second basis.
func (t *ThrottlerImpl) Throttle(threadID int) time.Duration {
	if t.closed {
		panic(fmt.Sprintf("BUG: thread with ID: %v must not access closed Throttler", threadID))
	}
	if t.threadFinished[threadID] {
		panic(fmt.Sprintf("BUG: thread with ID: %v already finished", threadID))
	}
	return t.threadThrottlers[threadID].throttle(t.nowFunc())
}

// MaxLag returns the max of all the last replication lag values seen across all tablets of
// the provided type, excluding ignored tablets.
func (t *ThrottlerImpl) MaxLag(tabletType topodata.TabletType) uint32 {
	cache := t.maxReplicationLagModule.lagCacheByType(tabletType)
	if cache == nil {
		return 0
	}
	return cache.maxLag()
}

// ThreadFinished marks threadID as finished and redistributes the thread's

View on GitHub (pinned to 01a25a7d17)

Solutions

Signal worker threads to stop (context cancel / done channel) and wait for them to exit before calling Close()
Check t.closed or an equivalent guard in the caller's loop before each Throttle call
Ensure Close() is called only after all registered threads have called ThreadFinished()

Example fix

// before
throttler.Close()
wg.Wait()
// after
wg.Wait()
throttler.Close()

Defensive patterns

Strategy: try-catch

Validate before calling

select {
case <-done:
    return
default:
    d := throttler.Throttle(threadID)
    _ = d
}

Type guard

func canThrottle(t *throttler.ThrottlerImpl) func() bool {
    return func() bool { return !t.IsClosed() }
}

Try / catch

defer func() {
    if r := recover(); r != nil {
        if strings.Contains(fmt.Sprint(r), "closed Throttler") {
            return // benign during shutdown
        }
        panic(r)
    }
}()

Prevention

Stop worker loops via context cancel before Close()
wg.Wait() before calling Close()
Call ThreadFinished for every thread before shutdown

When it happens

Trigger: Calling Throttle(threadID) on a ThrottlerImpl after another goroutine invoked Close() — e.g. a worker still running in its loop while the owner shut the throttler down during teardown.

Common situations: Shutdown ordering bugs in vreplication/migration streams: Close called while worker goroutines are mid-Throttle; tests closing the throttler in a defer before all threads finish.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/7ea918cbaa3d486e. Report an issue: GitHub.