go-kratos/kratos · critical

retry after %d times

Error message

retry after %d times

What it means

Returned by the Kratos Eureka client's do() loop after every one of the maxRetry attempts signaled retry=true. retry=true only happens when e.client.Do(request) itself fails (connection refused, DNS failure, timeout, TLS error) at client.go:328-329; each attempt also rotates to the next server URL via pickServer. So this error means every configured Eureka server URL was unreachable at the transport level. Note the final error discards the underlying cause - only the retry count is reported, so check logs/network for the real reason.

Source

Thrown at contrib/registry/eureka/client.go:365

	if resp.StatusCode >= http.StatusBadRequest {
		return false, fmt.Errorf("response Error %d", resp.StatusCode)
	}

	return false, nil
}

func (e *Client) do(ctx context.Context, method string, params []string, input io.Reader, output any) error {
	for i := 0; i < e.maxRetry; i++ {
		retry, err := e.request(ctx, method, params, input, output, i)
		if retry {
			continue
		}
		if err != nil {
			return err
		}
		return nil
	}
	return fmt.Errorf("retry after %d times", e.maxRetry)
}

View on GitHub (pinned to 668db92c2c)

Solutions

1. From the app container, verify reachability of every configured server: curl http://<server>/eureka/apps or a TCP check on the port; fix address/port/DNS/firewall as needed
2. Check whether the Eureka instance(s) are actually running and ready (pod status, health endpoint) - if this fires at startup, add readiness gating or startup retry for the registrar
3. If multiple servers are configured, confirm each entry is individually valid; pickServer cycles through urls[maxRetry] entries, so a bad list wastes all attempts
4. Add caller-level retry with backoff around Register/heartbeat operations, since do() only retries transport errors within one call
5. Increase visibility: capture the underlying transport error (tcp timeout vs refused vs TLS) from logs or a manual request before blaming the client

Example fix

// before: single wrong port; every retry fails -> "retry after 3 times"
cfg := &eureka.Config{Servers: []string{"http://eureka:8760/eureka"}}

// after: correct port plus more fallback servers
cfg := &eureka.Config{Servers: []string{"http://eureka-1:8761/eureka", "http://eureka-2:8761/eureka"}}

// Fail fast if no server URL is reachable before starting the registrar
func reachable(servers []string, path string) error {
	for _, s := range servers {
		resp, err := http.Get(s + path) //nolint:gosec
		if err == nil {
			_ = resp.Body.Close()
			if resp.StatusCode < 500 {
				return nil
			}
		}
	}
	return fmt.Errorf("no eureka server reachable: %v", servers)
}

var err error
for attempt := 0; attempt < 3; attempt++ {
	if err = client.Register(ctx, ins); err == nil {
		break
	}
	if !strings.Contains(err.Error(), "retry after") { // only transport exhaustion is retried
		break
	}
	select {
	case <-time.After(time.Duration(attempt+1) * time.Second):
	case <-ctx.Done():
		return ctx.Err()
	}
}

When it happens

Trigger: All servers in the Config.Servers list are down or unreachable: connection refused (server not listening on that port), DNS name not resolvable, network partition/firewall blocking the port, TLS handshake failure on https URLs, or request context deadline exceeded repeatedly. Also triggered if Servers list is effectively all the same broken host and maxRetry (default 3) attempts all fail.

Common situations: Wrong server address/port in Kubernetes ConfigMap; Eureka pod not ready when the app starts (no startup retry at caller level); DNS entry missing in the environment; security group blocking egress to the Eureka port; https URL with a cert the client cannot validate.

Related errors

• response Error %d
• ErrorCode: %d
• RegisterInstance err %v,%v
• invalid path: %q is not a message
• field already set for oneof %q

AI-assisted analysis of go-kratos/kratos@668db92c2c (2026-08-16). Data as JSON: /api/errors/6a09ff55ad498e47. Report an issue: GitHub.