apache/seatunnel · warning

Failed to clear scroll ID

Error message

Failed to clear scroll ID: ${scrollId}

What it means

A WARN log from ElasticsearchSourceReader.scrollSearchResult: after finishing scroll-based pagination, clearing the scroll context (DELETE /_search/scroll with the scroll ID) failed. Data reading succeeded; the scroll context stays alive on the ES cluster until scroll timeout, consuming heap. The scroll ID is included so it can be cleared manually.

Solutions

1. Increase the scroll keep-alive (e.g. scroll=10m in config) so the context survives to the final clear
2. Verify ES cluster health and network stability during large scroll reads
3. Manually clear the logged scroll ID: DELETE /_search/scroll {"scroll_id":"..."}
4. If the cluster hits max_open_scroll_context, reduce concurrent SeaTunnel source parallelism or scroll usage

// ensure cluster reachable before job
curl -s "https://es-host:9200/_cluster/health?wait_for_status=yellow&timeout=10s"

// library swallows the exception; operator-side retry:
// curl -X DELETE 'https://es-host:9200/_search/scroll' -H 'Content-Type: application/json' -d '{"scroll_id":"<id from log>"}'

When it happens

Trigger: pollNext -> scrollSearchResult loop finishes, finally block calls esRestClient.clearScroll(scrollId), which throws due to network error, cluster node restart, scroll context already expired/timed out, or ES returning an error for an unknown scroll_id.

Common situations: Long-running reads where the scroll timeout (e.g. 5m) elapsed before the final clear; ES cluster failover losing scroll state; transient HTTP failures; too many open scroll contexts on the cluster (max_open_scroll_context exceeded).

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• DELETE response status code= , body= for scroll ID
• Failed to clear scroll ID
• Attempted to clear scroll with empty scroll ID
• close elasticsearch connection error
• DELETE_PIT_FAILED

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/f3fd6184b1c6a0f8. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-elasticsearch/src/main/java/org/apache/seatunnel/connectors/seatunnel/elasticsearch/source/ElasticsearchSourceReader.java:165

                                    sourceIndexInfo.getSliceId(),
                                    sourceIndexInfo.getSliceMax());
                    scrollId = scrollResult.getScrollId();

                    outputFromScrollResult(scrollResult, sourceIndexInfo, output, deserializer);
                    while (scrollResult.getDocs() != null && !scrollResult.getDocs().isEmpty()) {
                        scrollResult =
                                esRestClient.searchWithScrollId(
                                        scrollResult.getScrollId(),
                                        sourceIndexInfo.getScrollTime());
                        scrollId = scrollResult.getScrollId();
                        outputFromScrollResult(scrollResult, sourceIndexInfo, output, deserializer);
                    }
                } finally {
                    if (StringUtils.isNotEmpty(scrollId)) {
                        try {
                            esRestClient.clearScroll(scrollId);
                        } catch (Exception e) {
                            log.warn("Failed to clear scroll ID: " + scrollId, e);
                        }
                    }
                }
            }
        }
    }

    private void logSliceInfo(ElasticsearchConfig sourceIndexInfo) {
        if (sourceIndexInfo.getSliceMax() > 1) {
            log.info(
                    "Elasticsearch slicing enabled. index={}, slice_id={}, slice_max={}",
                    sourceIndexInfo.getIndex(),
                    sourceIndexInfo.getSliceId(),
                    sourceIndexInfo.getSliceMax());
        }
    }

    /**

View on GitHub (pinned to cf67b549a7)