apache/seatunnel · warning

Failed to load finished job metrics for job

Error message

Failed to load finished job metrics for job {}: {}

What it means

getFinishedJobMetricsJson loads a finished job's JobMetrics from the IMap named by Constant.IMAP_FINISHED_JOB_METRICS. Any Throwable during the map access (serialization, network/Hazelcast partition issues, member loss) is caught, logged as this warning, and an empty JobMetrics JSON is returned so the REST endpoint still responds.

Solutions

1. Check cluster member health (all members up, no split-brain) and retry the REST query once the cluster is stable.
2. If deserialization/version errors appear, clear stale entries in the IMap_MARKERS finished-job-metrics map or upgrade all members to the same version.
3. Increase Hazelcast operation timeout if timeouts occur under load.
4. Verify the job actually persisted metrics before finishing; jobs finished before metrics persistence may legitimately have no entry.

Example fix

// before: blind call
// String json = jobInfoService.getJobsByStateJson(...);
// after: tolerate empty metrics response
// String json = getWithRetry(() -> jobInfoService.getJobsByStateJson(...));
// if (json.contains("\"jobId\"") == false) { /* treat as no metrics */ }

// client-side: check cluster health before querying finished job metrics
// boolean clusterHealthy = restGet("/hazelcast/rest/health").isOk();

try {
    String json = jobInfoService.getJobsByStateJson(state);
    // parse json
} catch (Exception e) {
    // treat as empty metrics: JobMetrics.empty().toJsonString()
}

When it happens

Trigger: Calling the jobs-by-state REST endpoint for finished jobs when the finished-job-metrics IMap entry is unreadable: Hazelcast operation timeout, member down holding the partition, deserialization failure of the JobMetrics value, or class-version mismatch of JobDAGInfo/JobMetrics payloads.

Common situations: Node restarted and finished metrics lost/partition repartitioning in progress; cluster membership churn while querying REST; upgraded cluster where old serialized metrics cannot be deserialized.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• Failed to load finished job DAG for job
• Cannot reliably determine vertex assignment for table
• Failed to get HTTP port from member
• Found unassigned metric entries for table ' ' metric ' '…
• Metric array size mismatch for table

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/0b4917f0ecc05a08. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/service/JobInfoService.java:155

                .map(
                        jobState -> {
                            Long jobId = jobState.getJobId();
                            return getJobInfoJson(
                                    jobState,
                                    getFinishedJobMetricsJson(jobId),
                                    getFinishedJobDAGInfo(finishedJobDAGInfo, jobId));
                        })
                .collect(JsonArray::new, JsonArray::add, JsonArray::add);
    }

    private String getFinishedJobMetricsJson(Long jobId) {
        try {
            IMap<Long, JobMetrics> finishedJobMetrics =
                    nodeEngine.getHazelcastInstance().getMap(Constant.IMAP_FINISHED_JOB_METRICS);
            JobMetrics metrics = finishedJobMetrics.getOrDefault(jobId, JobMetrics.empty());
            return metrics == null ? JobMetrics.empty().toJsonString() : metrics.toJsonString();
        } catch (Throwable t) {
            log.warn("Failed to load finished job metrics for job {}: {}", jobId, t.getMessage());
            return JobMetrics.empty().toJsonString();
        }
    }

    private JobDAGInfo getFinishedJobDAGInfo(
            IMap<Long, JobDAGInfo> finishedJobDAGInfo, Long jobId) {
        try {
            return finishedJobDAGInfo.get(jobId);
        } catch (Throwable t) {
            log.warn("Failed to load finished job DAG for job {}: {}", jobId, t.getMessage());
            return null;
        }
    }

    public JsonArray getRunningJobsJson() {
        IMap<Long, JobInfo> values = getRunningJobInfoMap();
        SeaTunnelServer seaTunnelServer = getSeaTunnelServer(true);
        return values.entrySet().stream()

View on GitHub (pinned to cf67b549a7)