What causes "GET /running-jobs slow: full={} costMs={}" in apache/seatunnel?

Use pagination query parameters instead of full=true when consuming /running-jobs

GET /running-jobs slow: full=

Error message

GET /running-jobs slow: full={} costMs={}

What it means

RunningJobsServlet.doGet logs this when the GET /running-jobs handler took more than 500ms to execute. `full` indicates whether the response was a full (unpaginated) listing; the companion diagnostics line adds heap stats.

Solutions

Use pagination query parameters instead of full=true when consuming /running-jobs
Reduce number of concurrently running jobs or scale the cluster
Increase node JVM heap if diagnostics show heap pressure
Cache the job list on the client side and poll less often

Example fix

// before
GET /running-jobs?full=true
// after
GET /running-jobs?page=1&pageSize=20

Defensive patterns

Strategy: fallback

Validate before calling

// prefer paginated call unless the job count is small
const useFull = knownJobCount < 50;
const url = useFull ? baseUrl + '/running-jobs?full=true' : baseUrl + '/running-jobs?page=1&pageSize=20';

Try / catch

try {
  const resp = await fetch(url);
  const cost = Number(resp.headers.get('X-Handler-Cost-Ms') || 0);
  if (cost > 500) switchToPaginatedPolling();
} catch (e) { /* fall back to cached last-known list */ }

Prevention

Avoid full=true on large clusters; paginate instead
Cache job lists client-side between polls
Keep running job count within cluster capacity
Monitor X-Handler-Cost-Ms and adapt polling strategy dynamically

When it happens

Trigger: Requesting the full (non-paginated) /running-jobs listing on a cluster with many running jobs, causing long serialization/collection time over 500ms.

Common situations: Large clusters without pagination; dashboards fetching the full job list every few seconds; coordinator under memory/GC pressure while aggregating job statuses.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/1134c7e40e5e4213. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/servlet/RunningJobsServlet.java:87

                        ? jobInfoService.getRunningJobsJson()
                        : jobInfoService.getRunningJobsSummaryJson();

        long costMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs);
        long dispatchDelayMs = receivedMs <= 0 ? -1 : Math.max(0, nowMs - receivedMs);

        resp.setHeader("X-Dispatch-Delay-Ms", String.valueOf(dispatchDelayMs));
        resp.setHeader("X-Handler-Cost-Ms", String.valueOf(costMs));

        writeJsonWithPagination(req, resp, runningJobs);

        if (dispatchDelayMs > 500) {
            log.warn(
                    "GET /running-jobs dispatch delayed: dispatchDelayMs={} thread={}",
                    dispatchDelayMs,
                    Thread.currentThread().getName());
        }
        if (costMs > 500) {
            log.warn("GET /running-jobs slow: full={} costMs={}", full, costMs);
            Runtime rt = Runtime.getRuntime();
            long usedBytes = rt.totalMemory() - rt.freeMemory();
            log.warn(
                    "GET /running-jobs slow diagnostics: full={} costMs={} thread={} "
                            + "heapUsedMB={} heapTotalMB={} heapMaxMB={}",
                    full,
                    costMs,
                    Thread.currentThread().getName(),
                    usedBytes / 1024 / 1024,
                    rt.totalMemory() / 1024 / 1024,
                    rt.maxMemory() / 1024 / 1024);
        } else {
            log.debug("GET /running-jobs: full={} costMs={}", full, costMs);
        }
    }
}

View on GitHub (pinned to cf67b549a7)