apache/shenyu · critical · ShenyuException

renew master fail

Error message

renew master fail, %s

What it means

ShenyuClusterService.doSelectMaster runs the periodic task where each admin node tries to acquire/renew the distributed master lock. If any exception occurs during master selection, the service closes upstream/instance check services and rethrows as ShenyuException("renew master fail, <cause message>"). The wrapped cause is the real problem.

Solutions

1. Read the wrapped `%s` cause in the log line — fix the underlying zookeeper/etcd error (connectivity, auth, session timeout).
2. Verify the cluster store is reachable and quorum-healthy from every admin node.
3. Increase session/connection timeouts if renewals fail under GC pauses or network latency.
4. Restart the failed admin instance if its check services were closed — it will not process data until it rejoins the cluster.
5. Ensure clocks and versions are consistent across admin nodes to avoid concurrent lock conflicts.

Example fix

// before
shenyu:
  cluster:
    zookeeper:
      sessionTimeout: 3000
// after (tolerate brief GC/network pauses)
shenyu:
  cluster:
    zookeeper:
      sessionTimeout: 60000

if (!clusterClient.isConnected())
    throw new IllegalStateException("cluster store disconnected — master renewal will fail");

try {
    shenyuClusterService.startSelectMasterTask();
} catch (ShenyuException e) {
    LOG.error("master renewal failed, cause: {}", e.getCause(), e);
    // alert: this admin node closed check services and must rejoin
}

When it happens

Trigger: The scheduled startSelectMasterTask fails while acquiring the master lock — zookeeper/etcd connection loss, session expiry, lock node deleted concurrently, or any KeeperException/cluster-store error during doSelectMaster.

Common situations: Zookeeper session expired under load; network blip between admin nodes and the cluster store; multiple admin nodes fighting after a store restart; cluster store (zk/etcd) quorum loss.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• shenyu.jwt.secretKey is not configured. In a multi-instance…
• failed to add curator cache.
• zookeeper url is empty
• failed to add curator cache.
• failed to add tree cache.

AI-assisted analysis of apache/shenyu@567142e072 (2026-09-12). Data as JSON: /api/errors/a6a6e433c2b30ab2. Report an issue: GitHub.

Appendix: source

Thrown at shenyu-admin/src/main/java/org/apache/shenyu/admin/mode/cluster/service/ShenyuClusterService.java:111

            while (renewed) {
                // sleeps selectPeriod seconds then renew the lock
                TimeUnit.SECONDS.sleep(clusterProperties.getSelectPeriod());
                
                renewed = shenyuClusterSelectMasterService.checkMasterStatus();
                if (renewed) {
                    if (LOG.isDebugEnabled()) {
                        LOG.debug("renew master success");
                    }
                }
            }
        } catch (Exception e) {
            LOG.error("select master error", e);
            // close the upstream check service
            upstreamCheckService.close();
            instanceCheckService.close();
            
            String message = String.format("renew master fail, %s", e.getMessage());
            throw new ShenyuException(message);
        } finally {
            try {
                shenyuClusterSelectMasterService.releaseMaster();
            } catch (Exception e) {
                LOG.error("release master error", e);
            }
        }
    }
    
    @Override
    public void start(final String host, final int port, final String contextPath) {
        startSelectMasterTask(host, String.valueOf(port), contextPath);
    }
    
    @Override
    public void shutdown() {
    
    }

View on GitHub (pinned to 567142e072)