What causes "iterator closed" in go-kratos/kratos?

Treat ErrIteratorClosed as a normal end-of-stream signal: break the consumption loop, do not log it as an error

iterator closed

Error message

iterator closed

What it means

ErrIteratorClosed (contrib/registry/kubernetes/registry.go:415) is returned by Iterator.Next (registry.go:306) when its stopCh has been closed. Iterator is the channel-based change stream used by the kubernetes registry's watcher; Stop() closes stopCh (idempotently, via the select/default guard), and any subsequent or in-flight Next() unblocks through the '<-iter.stopCh' case and returns this sentinel. It signals orderly termination, not a fault.

Source

Thrown at contrib/registry/kubernetes/registry.go:415

				}
			}
			addr := protocol + "://" + net.JoinHostPort(podIP, strconv.Itoa(int(port)))
			endpoints = append(endpoints, addr)
		}
	}
	return &registry.ServiceInstance{
		ID:        podLabels[LabelsKeyServiceID],
		Name:      podLabels[LabelsKeyServiceName],
		Version:   podLabels[LabelsKeyServiceVersion],
		Metadata:  metadata,
		Endpoints: endpoints,
	}, nil
}

// //////////// Error Definition ////////////

// ErrIteratorClosed defines the error that the iterator is closed
var ErrIteratorClosed = errors.New("iterator closed")

// ErrorHandleResource defines the error that cannot handle K8S resources normally
type ErrorHandleResource struct {
	Namespace string
	Name      string
	Reason    error
}

// Error implements the error interface
func (err *ErrorHandleResource) Error() string {
	return fmt.Sprintf("failed to handle resource(namespace=%s, name=%s): %s",
		err.Namespace, err.Name, err.Reason)
}

View on GitHub (pinned to 668db92c2c)

Solutions

Treat ErrIteratorClosed as a normal end-of-stream signal: break the consumption loop, do not log it as an error
Ensure only one side owns lifecycle: the goroutine reading Next() should also observe Stop() and exit
Match with errors.Is(err, kubernetes.ErrIteratorClosed) rather than string comparison

Example fix

// before
for {
    ins, err := iter.Next()
    if err != nil { log.Error("watch failed", err) } // noisy on shutdown
}

// after
for {
    ins, err := iter.Next()
    if errors.Is(err, kubernetes.ErrIteratorClosed) {
        return nil // clean shutdown
    }
    if err != nil { return err }
    update(ins)
}

Defensive patterns

Strategy: try-catch

Try / catch

for {
    ins, err := iter.Next()
    if errors.Is(err, kubernetes.ErrIteratorClosed) {
        return nil // clean termination
    }
    if err != nil {
        return err
    }
    handle(ins)
}

Prevention

Model ErrIteratorClosed as end-of-stream, never as a fault to log loudly
Single-owner lifecycle: the goroutine consuming Next() should also react to Stop()
Order shutdown as: stop producer, drain consumer, then close channels

When it happens

Trigger: Calling iter.Next() after iter.Stop(); calling Next concurrently with Stop so the stopCh case wins the select; the owning watcher being stopped by kratos during app shutdown which cascades into the iterator's stopCh. Any Next on a closed iterator yields exactly this error and nothing else.

Common situations: Service shutdown sequences where the consumer goroutine of the instance channel is still looping after the producer was stopped; missing synchronization between 'stop the watcher' and 'drain the loop'; tests that stop the iterator and then assert on one more Next call.

Related errors

AI-assisted analysis of go-kratos/kratos@668db92c2c (2026-08-16). Data as JSON: /api/errors/eabca24c6bf0faa9. Report an issue: GitHub.