{"record":{"id":"5220235e323921ad","repo":"prestodb/presto","slug":"hive-exceeded-split-buffering-limit","errorCode":"HIVE_EXCEEDED_SPLIT_BUFFERING_LIMIT","errorMessage":"Split buffering for %s.%s exceeded memory limit (%s). %s splits are buffered.","messagePattern":"Split buffering for (.+?)\\.(.+?) exceeded memory limit \\((.+?)\\)\\. (.+?) splits are buffered\\.","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-hive/src/main/java/com/facebook/presto/hive/HiveSplitSource.java","lineNumber":529,"sourceCode":"        }\n\n        // The PartitionInfo isn't included in the size of the InternalHiveSplit\n        // because it's a shared object. If this is the first InternalHiveSplit\n        // for that PartitionInfo, add its cost\n        if (split.getPartitionInfo().incrementAndGetReferences() == 1) {\n            estimatedSplitSizeInBytes.addAndGet(split.getPartitionInfo().getEstimatedSizeInBytes());\n        }\n\n        if (estimatedSplitSizeInBytes.addAndGet(split.getEstimatedSizeInBytes()) > maxOutstandingSplitsBytes) {\n            // TODO: investigate alternative split discovery strategies when this error is hit.\n            // This limit should never be hit given there is a limit of maxOutstandingSplits.\n            // If it's hit, it means individual splits are huge.\n            if (loggedHighMemoryWarning.compareAndSet(false, true)) {\n                highMemorySplitSourceCounter.update(1);\n                log.warn(\"Split buffering for %s.%s in query %s exceeded memory limit (%s). %s splits are buffered.\",\n                        databaseName, tableName, queryId, succinctBytes(maxOutstandingSplitsBytes), getBufferedInternalSplitCount());\n            }\n            throw new PrestoException(HIVE_EXCEEDED_SPLIT_BUFFERING_LIMIT, format(\n                    \"Split buffering for %s.%s exceeded memory limit (%s). %s splits are buffered.\",\n                    databaseName, tableName, succinctBytes(maxOutstandingSplitsBytes), getBufferedInternalSplitCount()));\n        }\n        bufferedInternalSplitCount.incrementAndGet();\n        OptionalInt bucketNumber = split.getReadBucketNumber();\n        return queues.offer(bucketNumber, split);\n    }\n\n    void noMoreSplits()\n    {\n        if (setIf(stateReference, State.noMoreSplits(), state -> state.getKind() == INITIAL)) {\n            // Stop the split loader before finishing the queue.\n            // Once the queue is finished, it will always return a completed future to avoid blocking any caller.\n            // This could lead to a short period of busy loop in splitLoader (although unlikely in general setup).\n            splitLoader.stop();\n            queues.noMoreSplits();\n        }\n    }","sourceCodeStart":511,"sourceCodeEnd":547,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-hive/src/main/java/com/facebook/presto/hive/HiveSplitSource.java#L511-L547","documentation":"HiveSplitSource buffers splits in memory up to maxOutstandingSplits / maxOutstandingSplitsBytes. When buffered split bytes exceed the byte limit, it throws HIVE_EXCEEDED_SPLIT_BUFFERING_LIMIT to protect the coordinator from OOM caused by a query whose splits are too numerous or individually too large.","triggerScenarios":"addToQueue detects estimateSize in bytes of buffered internal splits exceeds maxOutstandingSplitsBytes — typically when reading very many small files or a few enormous files, or when the consuming operator is slower than split production.","commonSituations":"Tables with millions of tiny files; queries with weak partition filters scanning almost the whole table; undersized max-outstanding-splits-bytes session/catalog config; slow worker consumption causing split backlog.","solutions":["Increase max-outstanding-splits-bytes (and max-outstanding-splits) in Hive catalog properties or per-session hive.max_outstanding_splits.","Compact small files (e.g., via partition compaction or rewriting with larger target file size) to reduce split count.","Add predicate filters / partition pruning so fewer partitions are scanned.","Scale out workers so splits are consumed faster, reducing buffered splits."],"exampleFix":"// before: hive.properties\nhive.max-outstanding-splits-bytes=64MB\n// after\nhive.max-outstanding-splits-bytes=512MB","handlingStrategy":"validation","validationCode":"// before the query, estimate buffered split load and tune\nlong fileCount = countFilesUnderTableLocations(db, table);\nif (fileCount * avgFileSizeEstimate > maxOutstandingSplitsBytes) {\n    session.setProperty(\"max_outstanding_splits\", largerValue);\n}","typeGuard":null,"tryCatchPattern":"try { runQuery(sql); } catch (PrestoException e) {\n    if (HIVE_EXCEEDED_SPLIT_BUFFERING_LIMIT.toErrorCode().getCode() == e.getErrorCode().getCode()) {\n        retryWithProperty(sql, \"hive.max_outstanding_splits\", \"2000000\");\n    } else { throw e; }\n}","preventionTips":["Compact small files to keep split counts low","Always filter on partition columns when scanning large tables","Size max-outstanding-splits-bytes relative to coordinator heap","Monitor buffered split counts in query metrics"],"tags":["memory","splits","resource-limit","hive"],"backgroundTag":"exceeded-split-buffering-limit","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}