What causes "discovery create watcher overtime" in go-kratos/kratos?

Verify the registry endpoints are reachable from the client (ports, DNS, TLS, ACLs)

discovery create watcher overtime

Error message

discovery create watcher overtime

What it means

ErrWatcherCreateTimeout is returned by the gRPC discovery resolver builder when creating the registry watcher does not finish within the builder's timeout (configurable via WithTimeout). Target resolution through the registry fails because the resolver never gets a watcher producing endpoints. It almost always means the registry (etcd/consul/nacos) is slow or unreachable.

Source

Thrown at transport/grpc/resolver/discovery/builder.go:17

package discovery

import (
	"context"
	"errors"
	"strings"
	"time"

	"github.com/google/uuid"
	"google.golang.org/grpc/resolver"

	"github.com/go-kratos/kratos/v3/registry"
)

const name = "discovery"

var ErrWatcherCreateTimeout = errors.New("discovery create watcher overtime")

// Option is builder option.
type Option func(o *builder)

// WithTimeout with timeout option.
func WithTimeout(timeout time.Duration) Option {
	return func(b *builder) {
		b.timeout = timeout
	}
}

// WithInsecure with isSecure option.
func WithInsecure(insecure bool) Option {
	return func(b *builder) {
		b.insecure = insecure
	}
}

View on GitHub (pinned to 668db92c2c)

Solutions

Verify the registry endpoints are reachable from the client (ports, DNS, TLS, ACLs)
Raise the watcher-create timeout with the discovery resolver builder's WithTimeout option
Redial after registry recovery — the rebuilt resolver retries watcher creation
Check registry cluster health/load; slow watch creation usually means an overloaded registry

Example fix

// before
b := discovery.NewBuilder(reg) // default watcher-create timeout, slow registry

// after
b := discovery.NewBuilder(reg, discovery.WithTimeout(30*time.Second))

Defensive patterns

Strategy: retry

Validate before calling

// pre-flight: registry reachable before dialing with the discovery resolver
checkCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()
if _, err := reg.GetService(checkCtx, "health-probe-svc"); err != nil {
    // fix registry connectivity before grpc.Dial
}

Try / catch

if err != nil {
    if errors.Is(err, discovery.ErrWatcherCreateTimeout) {
        time.Sleep(backoff(attempt))
        return dialAgain(ctx) // re-create conn; resolver rebuilds the watcher
    }
    return err
}

Prevention

Point clients at healthy registry endpoints (etcd/consul cluster up)
Set WithTimeout generously on high-latency links
Redial on resolver timeout instead of failing fast at startup

When it happens

Trigger: grpc dialing with the discovery resolver (discovery:///target); the builder runs registry Watch(ctx, name) under a select against the builder timeout; if the registry does not answer in time, ErrWatcherCreateTimeout is returned and the resolver build fails.

Common situations: Registry cluster down or restarting during deploys; firewall/DNS blocking registry endpoints; high-latency cross-DC links exceeding the default timeout; overloaded registry slow to establish watches.

Related errors

AI-assisted analysis of go-kratos/kratos@668db92c2c (2026-08-16). Data as JSON: /api/errors/0c6f84c581ec08b6. Report an issue: GitHub.