What causes "shutdown already in progress" in kubernetes/kops?

Call Stop() only once; check StopRequested() before calling it

shutdown already in progress

Error message

shutdown already in progress

What it means

Stoppable.Stop() is guarded so shutdown happens only once: a mutex plus a shutdown flag. If Stop() is invoked after shutdown has already started (shutdown==true), it returns this error instead of closing the channel a second time (which would panic).

Source

Thrown at dns-controller/pkg/util/stoppable.go:54

// StopChannel gets the stopChannel, initializing it if needed
func (s *Stoppable) StopChannel() <-chan struct{} {
	s.mutex.Lock()
	defer s.mutex.Unlock()

	if s.stopChannel == nil {
		s.stopChannel = make(chan struct{})
	}
	return s.stopChannel
}

// Stop stops the controller.
func (s *Stoppable) Stop() error {
	s.mutex.Lock()
	defer s.mutex.Unlock()

	if s.shutdown {
		return fmt.Errorf("shutdown already in progress")
	}

	// We initialize the channel to avoid a race if we Stop before anyone is watching
	if s.stopChannel == nil {
		s.stopChannel = make(chan struct{})
	}
	close(s.stopChannel)
	klog.Infof("shutting down controller")
	s.shutdown = true

	return nil
}

func (s *Stoppable) StopRequested() bool {
	return s.shutdown
}

View on GitHub (pinned to 4c8573c808)

Solutions

Call Stop() only once; check StopRequested() before calling it
Ignore this error on the second call (it's benign and means shutdown already happened)
Deduplicate signal handling: register the SIGTERM handler once
Refactor so only one component owns shutdown responsibility

Example fix

// before
_ = stoppable.Stop() // may be called twice
// after
if !stoppable.StopRequested() {
    if err := stoppable.Stop(); err != nil {
        klog.V(2).Infof("stop: %v", err)
    }
}

Defensive patterns

Strategy: type-guard

Validate before calling

if s.StopRequested() {
    klog.V(2).Info("shutdown already requested; skipping Stop")
    return
}

Type guard

func canStop(s *util.Stoppable) bool { return !s.StopRequested() }

Try / catch

// Stop is idempotent-intent: treat repeat shutdown as benign
if err := stoppable.Stop(); err != nil && err.Error() != "shutdown already in progress" {
    return fmt.Errorf("stopping dns-controller: %w", err)
}

Prevention

Call Stop from a single shutdown owner (signal handler only)
Check StopRequested() before Stop()
Register signal handlers and HTTP stop endpoints through one deduplicating wrapper

When it happens

Trigger: Calling Stop() twice on the same Stoppable — e.g. SIGTERM handler and an HTTP /stop endpoint both firing, or a deferred Stop plus an explicit Stop.

Common situations: Graceful shutdown triggered by both SIGINT/SIGTERM and healthz/stop HTTP handler; test code stopping controllers in Cleanup after the suite already stopped them; double invocation on reload.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/ae47e485b782482b. Report an issue: GitHub.