apache/seatunnel · warning

Failed to clear scroll ID: ${scrollId}

Error message

Failed to clear scroll ID: ${scrollId}

What it means

A WARN log from ElasticsearchSourceReader.scrollSearchResult: after finishing scroll-based pagination, clearing the scroll context (DELETE /_search/scroll with the scroll ID) failed. Data reading succeeded; the scroll context stays alive on the ES cluster until scroll timeout, consuming heap. The scroll ID is included so it can be cleared manually.

Source

Thrown at seatunnel-connectors-v2/connector-elasticsearch/src/main/java/org/apache/seatunnel/connectors/seatunnel/elasticsearch/source/ElasticsearchSourceReader.java:165

                                    sourceIndexInfo.getSliceId(),
                                    sourceIndexInfo.getSliceMax());
                    scrollId = scrollResult.getScrollId();

                    outputFromScrollResult(scrollResult, sourceIndexInfo, output, deserializer);
                    while (scrollResult.getDocs() != null && !scrollResult.getDocs().isEmpty()) {
                        scrollResult =
                                esRestClient.searchWithScrollId(
                                        scrollResult.getScrollId(),
                                        sourceIndexInfo.getScrollTime());
                        scrollId = scrollResult.getScrollId();
                        outputFromScrollResult(scrollResult, sourceIndexInfo, output, deserializer);
                    }
                } finally {
                    if (StringUtils.isNotEmpty(scrollId)) {
                        try {
                            esRestClient.clearScroll(scrollId);
                        } catch (Exception e) {
                            log.warn("Failed to clear scroll ID: " + scrollId, e);
                        }
                    }
                }
            }
        }
    }

    private void logSliceInfo(ElasticsearchConfig sourceIndexInfo) {
        if (sourceIndexInfo.getSliceMax() > 1) {
            log.info(
                    "Elasticsearch slicing enabled. index={}, slice_id={}, slice_max={}",
                    sourceIndexInfo.getIndex(),
                    sourceIndexInfo.getSliceId(),
                    sourceIndexInfo.getSliceMax());
        }
    }

    /**

View on GitHub (pinned to cf67b549a7)

Solutions

1. Increase the scroll keep-alive (e.g. scroll=10m in config) so the context survives to the final clear
2. Verify ES cluster health and network stability during large scroll reads
3. Manually clear the logged scroll ID: DELETE /_search/scroll {"scroll_id":"..."}
4. If the cluster hits max_open_scroll_context, reduce concurrent SeaTunnel source parallelism or scroll usage

// ensure cluster reachable before job
curl -s "https://es-host:9200/_cluster/health?wait_for_status=yellow&timeout=10s"

// library swallows the exception; operator-side retry:
// curl -X DELETE 'https://es-host:9200/_search/scroll' -H 'Content-Type: application/json' -d '{"scroll_id":"<id from log>"}'

When it happens

Trigger: pollNext -> scrollSearchResult loop finishes, finally block calls esRestClient.clearScroll(scrollId), which throws due to network error, cluster node restart, scroll context already expired/timed out, or ES returning an error for an unknown scroll_id.

Common situations: Long-running reads where the scroll timeout (e.g. 5m) elapsed before the final clear; ES cluster failover losing scroll state; transient HTTP failures; too many open scroll contexts on the cluster (max_open_scroll_context exceeded).

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• DELETE {} response status code={}, body={} for scroll ID: {}
• Failed to clear scroll ID: ${scrollId}
• SCROLL_REQUEST_ERROR
• DELETE_PIT_FAILED
• close elasticsearch connection error

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/f3fd6184b1c6a0f8. Report an issue: GitHub.