prestodb/presto · error · OrcCorruptionException

Write validation failed: %s in %s statistics

Error message

Write validation failed: %s in %s statistics

What it means

This is the generic statistics-value mismatch in write validation: the number of values recorded per column at write time must equal the number of values recomputed from the file. The thrown message embeds a detailed failureMessage comparing the actual and expected ColumnStatistics. A mismatch means data was lost, added, or corrupted between write and read.

Source

Thrown at presto-orc/src/main/java/com/facebook/presto/orc/OrcWriteValidation.java:533

            ColumnStatistics expected = expectedColumnStatistics.get(i);
            validateColumnStatisticsEquivalent(orcDataSourceId, name + " column " + i, actual, expected);
        }
    }

    private static void validateColumnStatisticsEquivalent(
            OrcDataSourceId orcDataSourceId,
            String name,
            ColumnStatistics actualColumnStatistics,
            ColumnStatistics expectedColumnStatistics)
            throws OrcCorruptionException
    {
        requireNonNull(name, "name is null");
        requireNonNull(actualColumnStatistics, "actualColumnStatistics is null");
        requireNonNull(expectedColumnStatistics, "expectedColumnStatistics is null");

        if (actualColumnStatistics.getNumberOfValues() != expectedColumnStatistics.getNumberOfValues()) {
            String failureMessage = format("Actual Values %s does not match expected values %s", actualColumnStatistics, expectedColumnStatistics);
            throw new OrcCorruptionException(orcDataSourceId, "Write validation failed: %s in %s statistics", failureMessage, name);
        }

        if (!Objects.equals(actualColumnStatistics.getBooleanStatistics(), expectedColumnStatistics.getBooleanStatistics())) {
            throw new OrcCorruptionException(orcDataSourceId, "Write validation failed: unexpected boolean counts in %s statistics", name);
        }
        if (!Objects.equals(actualColumnStatistics.getIntegerStatistics(), expectedColumnStatistics.getIntegerStatistics())) {
            IntegerStatistics actualIntegerStatistics = actualColumnStatistics.getIntegerStatistics();
            IntegerStatistics expectedIntegerStatistics = expectedColumnStatistics.getIntegerStatistics();
            // The sum of the integer stats depends on the order of how we merge them.
            // It is possible the sum can overflow with one order but not in another.
            // Ignore the validation of sum if one of the two sums is null.
            if (actualIntegerStatistics == null ||
                    expectedIntegerStatistics == null ||
                    !Objects.equals(actualIntegerStatistics.getMin(), expectedIntegerStatistics.getMin()) ||
                    !Objects.equals(actualIntegerStatistics.getMax(), expectedIntegerStatistics.getMax()) ||
                    (actualIntegerStatistics.getSum() != null &&
                            expectedIntegerStatistics.getSum() != null &&
                            !Objects.equals(actualIntegerStatistics.getSum(), expectedIntegerStatistics.getSum()))) {

View on GitHub (pinned to 55bb57d202)

Solutions

1. Restore the file from a backup or re-run the job that wrote it — the value counts genuinely differ
2. Check for storage-level corruption and enable checksums
3. Upgrade Presto if the mismatch matches a known writer statistics-counting bug
4. Set orc.write-validation=false only to read the file for triage, then re-write it

// Compare read-back non-null counts to expected before validation-sensitive queries
// SELECT count(*) FROM table WHERE col IS NOT NULL; vs recorded stats

try {
    orcBatchReader.nextPage();
} catch (OrcCorruptionException e) {
    if (e.getMessage().contains("Write validation failed")) {
        // file likely truncated/altered; fall back to backup copy
    }
}

When it happens

Trigger: validateColumnStatisticsEquivalent sees actualColumnStatistics.getNumberOfValues() != expectedColumnStatistics.getNumberOfValues() at file, stripe, or row group level — non-null value counts differ from the writer's record.

Common situations: Truncated or partially written ORC files; data modified after write; writer bugs miscounting non-null values for nested/flattened columns.

Related errors

• Unexpected multiple column statistics for node %s in row gro
• Unexpected row group %s in stripe at offset %s
• Write validation failed: unexpected number of columns in %s
• Write validation failed: unexpected boolean counts in %s sta
• Write validation failed: unexpected integer range in %s stat

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/c621d274a12e170f. Report an issue: GitHub.