apache/seatunnel · error · IllegalStateException

Caller ${callerAddress} cannot get metrics because it is not

Error message

Caller ${callerAddress} cannot get metrics because it is not master. Master is: ${masterAddress}

What it means

GetMetricsOperation.run() restricts metrics retrieval to the cluster master. After confirming the caller is a member, if the caller address does not equal the current master address it throws IllegalStateException — only the master node may request these task metrics.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/task/operation/GetMetricsOperation.java:72

    public GetMetricsOperation(Set<Long> runningJobIds, String[] metricNamePrefixes) {
        this.runningJobIds = runningJobIds;
        this.metricNamePrefixes = metricNamePrefixes;
    }

    @Override
    public void run() {
        ILogger logger = getLogger();

        Address callerAddress = getCallerAddress();

        NodeEngineImpl nodeEngine = (NodeEngineImpl) getNodeEngine();
        if (callerAddress == null
                || nodeEngine.getClusterService().getMember(callerAddress) == null) {
            throw new SecurityException("Caller is not a cluster member: " + callerAddress);
        }
        Address masterAddress = getNodeEngine().getMasterAddress();
        if (!callerAddress.equals(masterAddress)) {
            throw new IllegalStateException(
                    "Caller "
                            + callerAddress
                            + " cannot get metrics"
                            + " because it is not master. Master is: "
                            + masterAddress);
        }
        Predicate<MetricDescriptor> metricDescriptorPredicate =
                dis -> {
                    String jobIdStr = dis.tagValue(JOB_ID);
                    if (jobIdStr == null) {
                        return false;
                    }
                    long jobId;
                    try {
                        jobId = Long.parseLong(jobIdStr);
                    } catch (Exception e) {
                        return false;
                    }

View on GitHub (pinned to cf67b549a7)

Solutions

Ensure only the current master issues metrics queries; refresh master address after failover.
Wait for cluster to stabilize after a master change and re-send the request.
Check split-brain settings and re-run GC/join verification if two masters are present.
Route metrics requests through the master node's REST/API endpoints instead of direct operations.

Defensive patterns

Strategy: validation

Validate before calling

if (!nodeEngine.getMasterAddress().equals(localAddress)) { throw new IllegalStateException("only master may request metrics"); }

Try / catch

try { getMetrics(); } catch (IllegalStateException e) { if (e.getMessage().contains("because it is not master")) { refreshMasterAddressAndRetry(); } else { throw e; } }

Prevention

Always resolve the master address fresh before issuing metrics operations
After failover, wait for cluster stabilization
Watch for split-brain indicators in logs

When it happens

Trigger: A worker (non-master) node sends GetMetricsOperation to another node; or master changed via failover so the caller's view of 'who is master' is stale when run() executes.

Common situations: Master failover mid-job where an old master or worker still issues metrics pulls; split-brain with two nodes believing they are master; custom tooling sending this operation from a worker.

Understand the failure class

Background: "You do not have permission" / 403 Forbidden errors: authenticated but not allowed — causes and fixes across open-source libraries — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/98a5bcca914a150c. Report an issue: GitHub.