prestodb/presto · error · PrestoException

HIVE_EXCEEDED_SPLIT_BUFFERING_LIMIT

HIVE_EXCEEDED_SPLIT_BUFFERING_LIMIT

Error message

Split buffering for %s.%s exceeded memory limit (%s). %s splits are buffered.

What it means

HiveSplitSource buffers splits in memory up to maxOutstandingSplits / maxOutstandingSplitsBytes. When buffered split bytes exceed the byte limit, it throws HIVE_EXCEEDED_SPLIT_BUFFERING_LIMIT to protect the coordinator from OOM caused by a query whose splits are too numerous or individually too large.

Source

Thrown at presto-hive/src/main/java/com/facebook/presto/hive/HiveSplitSource.java:529

        }

        // The PartitionInfo isn't included in the size of the InternalHiveSplit
        // because it's a shared object. If this is the first InternalHiveSplit
        // for that PartitionInfo, add its cost
        if (split.getPartitionInfo().incrementAndGetReferences() == 1) {
            estimatedSplitSizeInBytes.addAndGet(split.getPartitionInfo().getEstimatedSizeInBytes());
        }

        if (estimatedSplitSizeInBytes.addAndGet(split.getEstimatedSizeInBytes()) > maxOutstandingSplitsBytes) {
            // TODO: investigate alternative split discovery strategies when this error is hit.
            // This limit should never be hit given there is a limit of maxOutstandingSplits.
            // If it's hit, it means individual splits are huge.
            if (loggedHighMemoryWarning.compareAndSet(false, true)) {
                highMemorySplitSourceCounter.update(1);
                log.warn("Split buffering for %s.%s in query %s exceeded memory limit (%s). %s splits are buffered.",
                        databaseName, tableName, queryId, succinctBytes(maxOutstandingSplitsBytes), getBufferedInternalSplitCount());
            }
            throw new PrestoException(HIVE_EXCEEDED_SPLIT_BUFFERING_LIMIT, format(
                    "Split buffering for %s.%s exceeded memory limit (%s). %s splits are buffered.",
                    databaseName, tableName, succinctBytes(maxOutstandingSplitsBytes), getBufferedInternalSplitCount()));
        }
        bufferedInternalSplitCount.incrementAndGet();
        OptionalInt bucketNumber = split.getReadBucketNumber();
        return queues.offer(bucketNumber, split);
    }

    void noMoreSplits()
    {
        if (setIf(stateReference, State.noMoreSplits(), state -> state.getKind() == INITIAL)) {
            // Stop the split loader before finishing the queue.
            // Once the queue is finished, it will always return a completed future to avoid blocking any caller.
            // This could lead to a short period of busy loop in splitLoader (although unlikely in general setup).
            splitLoader.stop();
            queues.noMoreSplits();
        }
    }

View on GitHub (pinned to 55bb57d202)

Solutions

1. Increase max-outstanding-splits-bytes (and max-outstanding-splits) in Hive catalog properties or per-session hive.max_outstanding_splits.
2. Compact small files (e.g., via partition compaction or rewriting with larger target file size) to reduce split count.
3. Add predicate filters / partition pruning so fewer partitions are scanned.
4. Scale out workers so splits are consumed faster, reducing buffered splits.

Example fix

// before: hive.properties
hive.max-outstanding-splits-bytes=64MB
// after
hive.max-outstanding-splits-bytes=512MB

// before the query, estimate buffered split load and tune
long fileCount = countFilesUnderTableLocations(db, table);
if (fileCount * avgFileSizeEstimate > maxOutstandingSplitsBytes) {
    session.setProperty("max_outstanding_splits", largerValue);
}

try { runQuery(sql); } catch (PrestoException e) {
    if (HIVE_EXCEEDED_SPLIT_BUFFERING_LIMIT.toErrorCode().getCode() == e.getErrorCode().getCode()) {
        retryWithProperty(sql, "hive.max_outstanding_splits", "2000000");
    } else { throw e; }
}

When it happens

Trigger: addToQueue detects estimateSize in bytes of buffered internal splits exceeds maxOutstandingSplitsBytes — typically when reading very many small files or a few enormous files, or when the consuming operator is slower than split production.

Common situations: Tables with millions of tiny files; queries with weak partition filters scanning almost the whole table; undersized max-outstanding-splits-bytes session/catalog config; slow worker consumption causing split backlog.

Related errors

• GENERIC_INSUFFICIENT_RESOURCES
• GENERIC_INTERNAL_ERROR
• HIVE_INVALID_ENCRYPTION_METADATA
• HIVE_INVALID_ENCRYPTION_METADATA
• HIVE_INVALID_ENCRYPTION_METADATA

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/5220235e323921ad. Report an issue: GitHub.