{"record":{"id":"0b4917f0ecc05a08","repo":"apache/seatunnel","slug":"failed-to-load-finished-job-metrics-for-job","errorCode":null,"errorMessage":"Failed to load finished job metrics for job {}: {}","messagePattern":"Failed to load finished job metrics for job (.+?): (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/service/JobInfoService.java","lineNumber":155,"sourceCode":"                .map(\n                        jobState -> {\n                            Long jobId = jobState.getJobId();\n                            return getJobInfoJson(\n                                    jobState,\n                                    getFinishedJobMetricsJson(jobId),\n                                    getFinishedJobDAGInfo(finishedJobDAGInfo, jobId));\n                        })\n                .collect(JsonArray::new, JsonArray::add, JsonArray::add);\n    }\n\n    private String getFinishedJobMetricsJson(Long jobId) {\n        try {\n            IMap<Long, JobMetrics> finishedJobMetrics =\n                    nodeEngine.getHazelcastInstance().getMap(Constant.IMAP_FINISHED_JOB_METRICS);\n            JobMetrics metrics = finishedJobMetrics.getOrDefault(jobId, JobMetrics.empty());\n            return metrics == null ? JobMetrics.empty().toJsonString() : metrics.toJsonString();\n        } catch (Throwable t) {\n            log.warn(\"Failed to load finished job metrics for job {}: {}\", jobId, t.getMessage());\n            return JobMetrics.empty().toJsonString();\n        }\n    }\n\n    private JobDAGInfo getFinishedJobDAGInfo(\n            IMap<Long, JobDAGInfo> finishedJobDAGInfo, Long jobId) {\n        try {\n            return finishedJobDAGInfo.get(jobId);\n        } catch (Throwable t) {\n            log.warn(\"Failed to load finished job DAG for job {}: {}\", jobId, t.getMessage());\n            return null;\n        }\n    }\n\n    public JsonArray getRunningJobsJson() {\n        IMap<Long, JobInfo> values = getRunningJobInfoMap();\n        SeaTunnelServer seaTunnelServer = getSeaTunnelServer(true);\n        return values.entrySet().stream()","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/service/JobInfoService.java#L137-L173","documentation":"getFinishedJobMetricsJson loads a finished job's JobMetrics from the IMap named by Constant.IMAP_FINISHED_JOB_METRICS. Any Throwable during the map access (serialization, network/Hazelcast partition issues, member loss) is caught, logged as this warning, and an empty JobMetrics JSON is returned so the REST endpoint still responds.","triggerScenarios":"Calling the jobs-by-state REST endpoint for finished jobs when the finished-job-metrics IMap entry is unreadable: Hazelcast operation timeout, member down holding the partition, deserialization failure of the JobMetrics value, or class-version mismatch of JobDAGInfo/JobMetrics payloads.","commonSituations":"Node restarted and finished metrics lost/partition repartitioning in progress; cluster membership churn while querying REST; upgraded cluster where old serialized metrics cannot be deserialized.","solutions":["Check cluster member health (all members up, no split-brain) and retry the REST query once the cluster is stable.","If deserialization/version errors appear, clear stale entries in the IMap_MARKERS finished-job-metrics map or upgrade all members to the same version.","Increase Hazelcast operation timeout if timeouts occur under load.","Verify the job actually persisted metrics before finishing; jobs finished before metrics persistence may legitimately have no entry."],"exampleFix":"// before: blind call\n// String json = jobInfoService.getJobsByStateJson(...);\n// after: tolerate empty metrics response\n// String json = getWithRetry(() -> jobInfoService.getJobsByStateJson(...));\n// if (json.contains(\"\\\"jobId\\\"\") == false) { /* treat as no metrics */ }","handlingStrategy":"fallback","validationCode":"// client-side: check cluster health before querying finished job metrics\n// boolean clusterHealthy = restGet(\"/hazelcast/rest/health\").isOk();","typeGuard":null,"tryCatchPattern":"try {\n    String json = jobInfoService.getJobsByStateJson(state);\n    // parse json\n} catch (Exception e) {\n    // treat as empty metrics: JobMetrics.empty().toJsonString()\n}","preventionTips":["Keep all members on the same version to avoid deserialization mismatches","Avoid querying finished-job metrics during membership churn","Retain sufficient heap so large metrics entries do not time out"],"tags":["hazelcast","rest-api","metrics","deserialization"],"backgroundTag":"database-query-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}