What causes "Failed to read Arrow batch" in prestodb/presto?

Retry the query; transient storage/network failures often resolve on re-run

prestodb/presto · error · PrestoException

LANCE_ERROR

LANCE_ERROR

Error message

Failed to read Arrow batch

What it means

The Lance connector failed while reading the next Arrow record batch from the Lance scanner, wrapping an IOException into a PrestoException with LANCE_ERROR. This indicates the underlying Arrow/Lance reader (RecordBatchReader) hit an I/O problem mid-scan, so the split cannot produce more pages.

Source

Thrown at presto-lance/src/main/java/com/facebook/presto/lance/LanceArrowToPageScanner.java:88

    public boolean read()
    {
        try {
            boolean hasNext = arrowReader.loadNextBatch();
            if (hasNext) {
                VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
                lastBatchBytes = 0;
                for (FieldVector vector : root.getFieldVectors()) {
                    for (ArrowBuf buf : vector.getFieldBuffers()) {
                        if (buf != null) {
                            lastBatchBytes += buf.capacity();
                        }
                    }
                }
            }
            return hasNext;
        }
        catch (IOException e) {
            throw new PrestoException(LanceErrorCode.LANCE_ERROR, "Failed to read Arrow batch", e);
        }
    }

    public long getLastBatchBytes()
    {
        return lastBatchBytes;
    }

    public Page convert()
    {
        VectorSchemaRoot root;
        try {
            root = arrowReader.getVectorSchemaRoot();
        }
        catch (IOException e) {
            throw new PrestoException(LanceErrorCode.LANCE_ERROR, "Failed to get VectorSchemaRoot", e);
        }

View on GitHub (pinned to 55bb57d202)

Solutions

Retry the query; transient storage/network failures often resolve on re-run
Check that the Lance dataset files still exist and were not compacted/deleted mid-query; avoid concurrent compaction during reads, or pin to a stable dataset version
Inspect the wrapped IOException cause for storage-specific diagnostics (permissions, missing objects, throttling)
Verify Lance library and dataset format version compatibility

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check storage accessibility before starting the scan
try (FileReader reader = new FileReader(path)) {
    if (!Files.exists(path)) throw new IllegalStateException("dataset files missing");
}

Try / catch

try {
    pages = scanner.read(...);
} catch (PrestoException e) {
    if (e.getErrorCode().getName().contains("LANCE")) {
        // inspect e.getCause() (IOException); retry or fail the split
    }
}

Prevention

Avoid compacting/rewriting Lance datasets while queries are running
Use stable dataset version pinning for long scans
Monitor object-store health and set sane retry policies
Keep Lance/Arrow library versions aligned with dataset format

When it happens

Trigger: Calling LanceArrowToPageScanner.read/hasNext when the underlying arrowReader throws IOException while loading the next batch — corrupt fragment files, deleted/compacted data files during the scan, storage (object store/HDFS) read failures, or network interruptions.

Common situations: Lance table files removed or rewritten by a concurrent compaction/write while a long-running query scans them; transient S3/GCS errors; disk or permission issues on local storage; version mismatch between the Lance runtime and dataset format.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/e9951a97ee642dbe. Report an issue: GitHub.