apache/seatunnel · error · IllegalStateException

Caller ${callerAddress} cannot get taskGroupLocation metrics

Error message

Caller ${callerAddress} cannot get taskGroupLocation metrics${taskGroupLocations} because it is not master. Master is: ${masterAddress}

What it means

GetTaskGroupMetricsOperation.run() lets the master fetch per-task-group metrics from workers. If the caller address is not the current master address, it throws IllegalStateException naming the caller, requested taskGroupLocations, and the real master — enforcing master-only access to this operation.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/task/operation/GetTaskGroupMetricsOperation.java:57

    private List<TaskGroupLocation> taskGroupLocations;
    private RawJobMetrics response;

    public GetTaskGroupMetricsOperation() {}

    public GetTaskGroupMetricsOperation(List<TaskGroupLocation> taskGroupLocations) {
        this.taskGroupLocations = taskGroupLocations;
    }

    @Override
    public void run() {
        ILogger logger = getLogger();

        Address callerAddress = getCallerAddress();

        NodeEngineImpl nodeEngine = (NodeEngineImpl) getNodeEngine();
        Address masterAddress = getNodeEngine().getMasterAddress();
        if (!callerAddress.equals(masterAddress)) {
            throw new IllegalStateException(
                    "Caller "
                            + callerAddress
                            + " cannot get taskGroupLocation metrics"
                            + taskGroupLocations.toString()
                            + " because it is not master. Master is: "
                            + masterAddress);
        }

        JobMetricsCollector metricsRenderer =
                new JobMetricsCollector(taskGroupLocations, nodeEngine.getLocalMember(), logger);
        nodeEngine.getMetricsRegistry().collect(metricsRenderer);
        response = metricsRenderer.getMetrics();
    }

    @Override
    protected void writeInternal(ObjectDataOutput out) throws IOException {
        super.writeInternal(out);
        out.writeInt(taskGroupLocations.size());

View on GitHub (pinned to cf67b549a7)

Solutions

Verify cluster membership and that exactly one master exists (fix split-brain: same cluster name, proper discovery).
After failover, let the new master re-initiate metric pulls; discard requests from the old master.
Update any custom monitoring code to target/execute from the current master only.
Restart nodes with inconsistent membership lists to rejoin cleanly.

Defensive patterns

Strategy: validation

Validate before calling

if (!nodeEngine.getMasterAddress().equals(callerAddress)) { throw new IllegalStateException("metrics pull restricted to master"); }

Try / catch

try { pullTaskGroupMetrics(groups); } catch (IllegalStateException e) { if (e.getMessage().contains("because it is not master")) { rescheduleOnCurrentMaster(); } else { throw e; } }

Prevention

Ensure a single master after failover (proper quorum/discovery)
Disable stale-master polling loops
Run monitoring through the master's public API

When it happens

Trigger: run() executes on a worker receiving GetTaskGroupMetricsOperation from an address != getNodeEngine().getMasterAddress(); typical when a stale or duplicate master queries metrics, or a non-master node issues the operation.

Common situations: Master failover leaving the previous master still polling workers; split-brain clusters with two masters; misconfigured cluster where an operator tool mimics master behavior from a worker.

Understand the failure class

Background: "You do not have permission" / 403 Forbidden errors: authenticated but not allowed — causes and fixes across open-source libraries — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/05ee6eb418713472. Report an issue: GitHub.