{"record":{"id":"f95c891b5928255f","repo":"apache/skywalking","slug":"forward-failed","errorCode":"forward_failed","errorMessage":"no admin channel to forward-target {mainAddress} (peer list out of sync?)","messagePattern":"no admin channel to forward-target (.+?) \\(peer list out of sync\\?\\)","errorType":"http","errorClass":"IllegalStateException","httpStatus":502,"severity":"error","filePath":"oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/cluster/RuntimeRuleClusterClient.java","lineNumber":266,"sourceCode":"     *\n     * <p>Uses a longer deadline than Suspend / Resume because the forwarded workflow on the\n     * main can include compile + DDL + persist which is orders of magnitude slower than a\n     * bookkeeping broadcast. Caller supplies the deadline in ms so admin operations can\n     * tune it independently of cluster-control fan-outs.\n     *\n     * @return the main's response. Never null on success; throws on transport failure so\n     *         the caller can surface a clear diagnostic to the operator.\n     */\n    public ForwardResponse forwardToMain(final String mainAddress,\n                                          final String operation,\n                                          final String catalog, final String name,\n                                          final byte[] body,\n                                          final boolean allowStorageChange,\n                                          final boolean forceReapply,\n                                          final long deadlineMs) {\n        final ManagedChannel channel = findChannelForAddress(mainAddress);\n        if (channel == null) {\n            throw new IllegalStateException(\n                \"no admin channel to forward-target \" + mainAddress + \" (peer list out of sync?)\");\n        }\n        final RuntimeRuleClusterServiceGrpc.RuntimeRuleClusterServiceBlockingStub stub =\n            RuntimeRuleClusterServiceGrpc.newBlockingStub(channel)\n                                         .withDeadlineAfter(deadlineMs, TimeUnit.MILLISECONDS);\n        return stub.forward(ForwardRequest.newBuilder()\n            .setOperation(operation == null ? \"\" : operation)\n            .setCatalog(catalog == null ? \"\" : catalog)\n            .setName(name == null ? \"\" : name)\n            .setBody(body == null ? ByteString.EMPTY : ByteString.copyFrom(body))\n            .setAllowStorageChange(allowStorageChange)\n            .setForceReapply(forceReapply)\n            .setSenderNodeId(selfNodeId)\n            .setIssuedAtMs(System.currentTimeMillis())\n            .build());\n    }\n\n    /**","sourceCodeStart":248,"sourceCodeEnd":284,"githubUrl":"https://github.com/apache/skywalking/blob/102af09b4a56064e22050dded10e2c52e490d040/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/cluster/RuntimeRuleClusterClient.java#L248-L284","documentation":"RuntimeRuleClusterClient.forwardToMain throws IllegalStateException (code forward_failed) when findChannelForAddress(mainAddress) returns null — the client has no cached admin gRPC channel to the node elected as 'main' for this rule operation. The message hints at the root cause: the OAP cluster peer list is out of sync with the address the coordinator handed out, so the forward target is unknown to the channel cache. This is a topology/consistency problem in the OAP cluster, not a rule-content problem.","triggerScenarios":"Calling a runtime-rule operation that must be forwarded to the elected main (install/update/delete with cluster coordination) when findChannelForAddress has no cached channel for mainAddress — e.g. the coordinator returned an address this node never built a channel to, or the channel cache was built before that member joined. Any forwardToMain(...) call in this state throws immediately, before the gRPC stub is even created.","commonSituations":"A node joined/left the cluster recently and stale members still hold the old peer list; Container/k8s environments where a pod's address changed (restart, reschedule) but the cluster register still advertises the old one; Clock or registration races during rolling restarts of the OAP cluster; Network policies blocking admin-port connections so the channel was never established","solutions":["Trigger peer-list reconciliation: wait for the cluster register (ZooKeeper/K8s/Nacos) sync interval, or restart the affected OAP node to force a fresh view","Verify every OAP node advertises a reachable address for the admin/cluster gRPC port (check cluster config: host binding vs container IP, NAT considerations)","Check the target address in the message is actually one of your live OAP members — if it is stale, the coordinator's registration is the thing to fix","Retry the rule operation after topology settles; forwarding is a client-side retryable action once the channel cache refreshes"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"final ManagedChannel ch = client.peekChannelForAddress(mainAddress); // expose a peek variant\nif (ch == null || ch.isShutdown()) { awaitPeerListSync(); then re-check; }","typeGuard":null,"tryCatchPattern":"catch (IllegalStateException e) when 'no admin channel to forward-target': back off one peer-sync interval, then retry the forward — the channel cache repopulates from the reconciled peer list. Give up with an operator alert only after the topology has demonstrably settled.","preventionTips":["Ensure every OAP node registers an externally reachable address/port for the admin gRPC service (container IP vs advertised host)","Size health-check/peer-sync intervals so channel caches converge faster than your rule-apply cadence","Probe cluster membership before batch rule pushes in k8s rolling-restart windows"],"tags":["cluster","grpc","forwarding","topology"],"backgroundTag":null,"analyzedSha":"102af09b4a56064e22050dded10e2c52e490d040","analyzedAt":"2026-08-14T10:47:52.647Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}