prestodb/presto · error · IllegalArgumentException

Unexpected PrestoSparkMutableRow: 'buffer' and 'array' field

Error message

Unexpected PrestoSparkMutableRow: 'buffer' and 'array' fields are both null

What it means

When PrestoSparkShufflePageInput.getNextPage() decodes shuffle rows into a page, each PrestoSparkMutableRow must carry its payload in either the 'buffer' or 'array' field. If a row has both null, the data layout is invalid, and IllegalArgumentException 'Unexpected PrestoSparkMutableRow: buffer and array fields are both null' is thrown rather than producing a corrupt page.

Source

Thrown at presto-spark-base/src/main/java/com/facebook/presto/spark/execution/shuffle/PrestoSparkShufflePageInput.java:107

                        ByteBuffer buffer = row.getBuffer();
                        verify(buffer.remaining() >= 2, "row data is expected to be at least 2 bytes long");
                        currentIteratorProcessedBytes += buffer.remaining();
                        short entryRowCount = getShortLittleEndian(buffer);
                        rowCount += entryRowCount;
                        currentIteratorProcessedRows += entryRowCount;
                        ((Buffer) buffer).position(buffer.position() + 2);
                        output.writeBytes(buffer.array(), buffer.arrayOffset() + buffer.position(), buffer.remaining());
                    }
                    else if (row.getArray() != null) {
                        verify(row.getLength() >= 2, "row data is expected to be at least 2 bytes long");
                        currentIteratorProcessedBytes += row.getLength();
                        short entryRowCount = getShortLittleEndian(row.getArray(), row.getOffset());
                        rowCount += entryRowCount;
                        currentIteratorProcessedRows += entryRowCount;
                        output.writeBytes(row.getArray(), row.getOffset() + 2, row.getLength() - 2);
                    }
                    else {
                        throw new IllegalArgumentException("Unexpected PrestoSparkMutableRow: 'buffer' and 'array' fields are both null");
                    }
                }
                long end = System.currentTimeMillis();
                shuffleStats.accumulate(
                        currentIteratorProcessedRows,
                        currentIteratorProcessedRowBatches,
                        currentIteratorProcessedBytes,
                        end - start);
                if (!iterator.hasNext()) {
                    shuffleStatsCollector.add(new PrestoSparkShuffleStats(
                            input.getFragmentId(),
                            taskId,
                            READ,
                            shuffleStats.getProcessedRows(),
                            shuffleStats.getProcessedRowBatches(),
                            shuffleStats.getProcessedBytes(),
                            shuffleStats.getElapsedWallTimeMills()));
                    shuffleStats.reset();

View on GitHub (pinned to 55bb57d202)

Solutions

Verify writer and reader use the same shuffle format/version (matching PrestoSparkLocalShuffleInfoTranslator on both sides)
Check the local shuffle files for corruption or truncation (task retries, disk issues) and re-run the failing stage
Confirm no mixed Presto version executors in the cluster (rolling-upgrade skew) writing incompatible rows
Inspect row.getOffset()/getLength() handling — ensure the writer always sets either buffer or array

Example fix

// before
// reading shuffle written by a different Presto version translator
SparkShuffleWriteInfo = translatorV1.toWriteInfo(...)
// after
// use the same translator version on writer and reader
SparkShuffleWriteInfo = sameTranslatorVersion.toWriteInfo(...)

Defensive patterns

Strategy: try-catch

Validate before calling

// before consuming shuffle
if (!row.hasBuffer() && !row.hasArray()) {
    throw new IllegalStateException("Corrupt shuffle row: both buffer and array null");
}

Type guard

boolean isValidRow(PrestoSparkMutableRow row) {
    return row.getBuffer() != null || row.getArray() != null;
}

Try / catch

try {
    page = shufflePageInput.getNextPage();
} catch (IllegalArgumentException e) {
    if (e.getMessage().contains("'buffer' and 'array' fields are both null")) {
        // treat as corrupted/mismatched shuffle; retry stage or fail task cleanly
    }
}

Prevention

Use the same shuffle translator/format version on writer and reader
Avoid mixed Presto versions across executors during rolling upgrades
Verify local shuffle files are not truncated after task retries

When it happens

Trigger: Iterating the shuffle row iterator in getNextPage, a row is neither a buffer-backed nor array-backed row (both fields null) — thrown at PrestoSparkShufflePageInput.java:107 when the writer-side row layout doesn't match the reader's expectations.

Common situations: Mixed Presto/Spark shuffle format versions between writer and reader, a corrupted local shuffle file or truncated shuffle data being replayed, wrong shuffle info translator pairing read/write codecs, or task retry reading a partially written shuffle.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/f3dd073b2e0701b4. Report an issue: GitHub.