{"record":{"id":"c308515db47b67a3","repo":"apache/seatunnel","slug":"get-running-jobs-slow-diagnostics-full-costms","errorCode":null,"errorMessage":"GET /running-jobs slow diagnostics: full={} costMs={} thread={} heapUsedMB={} heapTotalMB={} heapMaxMB={}","messagePattern":"GET /running-jobs slow diagnostics: full=(.+?) costMs=(.+?) thread=(.+?) heapUsedMB=(.+?) heapTotalMB=(.+?) heapMaxMB=(.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/servlet/RunningJobsServlet.java","lineNumber":90,"sourceCode":"        long costMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs);\n        long dispatchDelayMs = receivedMs <= 0 ? -1 : Math.max(0, nowMs - receivedMs);\n\n        resp.setHeader(\"X-Dispatch-Delay-Ms\", String.valueOf(dispatchDelayMs));\n        resp.setHeader(\"X-Handler-Cost-Ms\", String.valueOf(costMs));\n\n        writeJsonWithPagination(req, resp, runningJobs);\n\n        if (dispatchDelayMs > 500) {\n            log.warn(\n                    \"GET /running-jobs dispatch delayed: dispatchDelayMs={} thread={}\",\n                    dispatchDelayMs,\n                    Thread.currentThread().getName());\n        }\n        if (costMs > 500) {\n            log.warn(\"GET /running-jobs slow: full={} costMs={}\", full, costMs);\n            Runtime rt = Runtime.getRuntime();\n            long usedBytes = rt.totalMemory() - rt.freeMemory();\n            log.warn(\n                    \"GET /running-jobs slow diagnostics: full={} costMs={} thread={} \"\n                            + \"heapUsedMB={} heapTotalMB={} heapMaxMB={}\",\n                    full,\n                    costMs,\n                    Thread.currentThread().getName(),\n                    usedBytes / 1024 / 1024,\n                    rt.totalMemory() / 1024 / 1024,\n                    rt.maxMemory() / 1024 / 1024);\n        } else {\n            log.debug(\"GET /running-jobs: full={} costMs={}\", full, costMs);\n        }\n    }\n}\n","sourceCodeStart":72,"sourceCodeEnd":104,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/servlet/RunningJobsServlet.java#L72-L104","documentation":"Extended diagnostics warning emitted by RunningJobsServlet.doGet when handling exceeds 500ms: it reports thread name plus heapUsedMB, heapTotalMB, heapMaxMB (from Runtime.totalMemory()-freeMemory()) to pinpoint memory pressure as the slowness cause.","triggerScenarios":"Same trigger as the 'GET /running-jobs slow' warning — costMs > 500 — with a heap snapshot computed at log time; usually seen alongside GC pressure or large job aggregations.","commonSituations":"Heap near max causing full GC pauses; very large running-job sets being collected and serialized; node overloaded by other engine tasks.","solutions":["Compare heapUsedMB to heapMaxMB; if near saturation, raise -Xmx or reduce cluster load","Capture a GC log / heap dump at the time of the warning","Use paginated /running-jobs requests to shrink handler work","Check for other memory-heavy operations running concurrently on the node"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// parse the diagnostics pattern to detect heap saturation\nconst m = logLine.match(/heapUsedMB=(\\d+).*heapMaxMB=(\\d+)/);\nif (m && Number(m[1]) / Number(m[2]) > 0.85) {\n  console.error('Coordinator heap near max — increase -Xmx or scale out');\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep coordinator heap utilization below ~70-80%","Set -Xmx based on job count and REST load","Enable GC logs to catch full-GC pauses early","Run periodic load tests to find the slow-handler threshold of your cluster"],"tags":["performance","jvm","memory","diagnostics"],"backgroundTag":"request-timeout","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}