What causes "failed to req API:{} after all servers({}) tried: {}" in alibaba/nacos?

Check the embedded cause/last error code in the exception message to determine whether it is network (connect refused, timeout) or application (403 auth, 500 server).

alibaba/nacos · critical · NacosException

failed to req API:{} after all servers({}) tried: {}

Error message

failed to req API:{} after all servers({}) tried: {}

What it means

Thrown by NamingHttpClientProxy.reqApi after exhausting all servers in the list (or all maxRetry attempts in domain mode) without a successful response. The exception preserves the last server's error code and message. This is the HTTP proxy's all-servers-failed terminal error.

Source

Thrown at client/src/main/java/com/alibaba/nacos/client/naming/remote/http/NamingHttpClientProxy.java:422

            for (int i = 0; i < servers.size(); i++) {
                String server = servers.get(index);
                try {
                    return callServer(api, params, body, server, method);
                } catch (NacosException e) {
                    exception = e;
                    if (NAMING_LOGGER.isDebugEnabled()) {
                        NAMING_LOGGER.debug("request {} failed.", server, e);
                    }
                }
                index = (index + 1) % servers.size();
            }
        }
        
        NAMING_LOGGER.error("request: {} failed, servers: {}, code: {}, msg: {}", api, servers,
            exception.getErrCode(),
            exception.getErrMsg());
        
        throw new NacosException(exception.getErrCode(),
            "failed to req API:" + api + " after all servers(" + servers + ") tried: "
                + exception.getMessage());
        
    }
    
    /**
     * Call server.
     *
     * @param api       api
     * @param params    parameters
     * @param body      body
     * @param curServer ?
     * @param method    http method
     * @return result
     * @throws NacosException nacos exception
     */
    public String callServer(String api, Map<String, String> params, Map<String, String> body,
        String curServer,

View on GitHub (pinned to 9b989acdf1)

Solutions

Check the embedded cause/last error code in the exception message to determine whether it is network (connect refused, timeout) or application (403 auth, 500 server).
Verify all Nacos server nodes are healthy and reachable on the HTTP port (default 8848).
For auth errors (403), verify the username/password or access key configuration.
Increase client read/connect timeouts if the cause is timeout under load.

Defensive patterns

Strategy: retry

Try / catch

int maxRetries = 3;
for (int attempt = 0; attempt <= maxRetries; attempt++) {
    try {
        namingService.registerInstance(serviceName, instance);
        break;
    } catch (NacosException e) {
        if (e.getErrMsg().contains("failed to req API") && attempt < maxRetries) {
            Thread.sleep(1000L * (attempt + 1));
            continue;
        }
        throw e;
    }
}

Prevention

Monitor Nacos server health and ensure at least one node is always reachable.
Parse the embedded error code in the message to distinguish auth/network/server errors.
Configure appropriate timeouts and retry budgets for the HTTP naming proxy.
Verify auth credentials if the embedded error code is 403.

When it happens

Trigger: Every callServer attempt in the retry loop threw a NacosException — meaning each server in the list returned an error or was unreachable. The final thrown exception carries the last server's errorCode and a descriptive message listing the API, servers, and the last error.

Common situations: All Nacos servers are down or unreachable; all servers return 5xx errors due to an overload or bug; authentication failures across all servers; network partition isolating the client from the cluster.

Related errors

AI-assisted analysis of alibaba/nacos@9b989acdf1 (2026-08-14). Data as JSON: /api/errors/f1c4bd3c02fefc3d. Report an issue: GitHub.