egametang/ET · error · Exception

service discovery request retry exhausted, request: {request

Error message

service discovery request retry exhausted, request: {requestName}

What it means

Thrown at line 671 in ForwardToDiscoveryByFailoverCore after all 4 failover attempts (maxAttemptCount) failed. Each attempt either threw (and was failover-retried) or hit an endpoint problem. The retry budget is exhausted, so the request is given up as a hard failure.

Source

Thrown at Packages/cn.etetet.servicediscovery/Scripts/Hotfix/Server/ServiceDiscoveryAgentSystem.cs:671

                    bool shouldResolveMaster = self.ShouldResolveMasterAfterFailure(e);
                    if (shouldResolveMaster)
                    {
                        self.InvalidateEndpointAndTriggerBackgroundRegister("call-failure");
                    }
                    else
                    {
                        Log.Warning(
                            $"ServiceDiscoveryAgent call failed without endpoint fallback scene: {self.Root().Name} request: {requestName} target: {targetActorId} error: {e.Message}");
                        throw;
                    }
                }

                int delay = GetMasterResolveRetryDelay(attempt);
                await self.Root().TimerComponent.WaitAsync(delay);
            }

            throw new Exception($"service discovery request retry exhausted, request: {requestName}");
        }

        private static bool ShouldResolveMasterAfterFailure(this ServiceDiscoveryAgent self, Exception e)
        {

            if (e is RpcException rpcException)
            {
                return ServiceDiscoveryErrorHelper.ShouldTriggerFailover(rpcException.Error);
            }

            return true;
        }

        private static int GetMasterResolveRetryDelay(int retry)
        {
            if (retry <= 0)
            {
                return MasterResolveRetryBaseInterval;

View on GitHub (pinned to 5cab01f7a8)

Solutions

1. Stabilize the master scene (resource, network, election) so at least one of 4 attempts succeeds.
2. Increase maxAttemptCount only if the failure mode is genuinely transient and slow to recover (prefer fixing the master).
3. Check logs for the per-attempt errors (the catch block logs warnings) to find the dominant failure code.
4. Verify master lease/election health and DB connectivity.

if (!agent.IsReady())
{
    // do not enter a failover loop destined to exhaust
    return;
}

catch (Exception e) when (e.Message.Contains("retry exhausted"))
{
    Log.Error($"discovery failover exhausted: {e.Message}");
    // escalate: page on-call, circuit-break the operation
}

When it happens

Trigger: Four consecutive attempts failed: each attempt's exception either triggered InvalidateEndpointAndTriggerBackgroundRegister + backoff, or the endpoint could not be resolved. Sustained master instability, repeated follower rejections, or repeated message timeouts across all attempts.

Common situations: Master flapping during a network partition; repeated ERR_MessageTimeout because the master is overloaded; a failover loop where each re-resolve lands on a different (also failing) master; chronic DB unavailability causing every register/response to fail.

Related errors

• ERR_ServiceDiscoveryMasterUnavailable
• service discovery incremental register failed scene: {sceneN
• service discovery incremental unregister failed scene: {scen
• service discovery proxy retryable service call error request
• service discovery endpoint unavailable after resolve, reques

AI-assisted analysis of egametang/ET@5cab01f7a8 (2026-08-13). Data as JSON: /api/errors/182ae9f59ee00591. Report an issue: GitHub.