{"record":{"id":"c621d274a12e170f","repo":"prestodb/presto","slug":"write-validation-failed-s-in-s-statistics","errorCode":null,"errorMessage":"Write validation failed: %s in %s statistics","messagePattern":"Write validation failed: (.+?) in (.+?) statistics","errorType":"exception","errorClass":"OrcCorruptionException","httpStatus":null,"severity":"error","filePath":"presto-orc/src/main/java/com/facebook/presto/orc/OrcWriteValidation.java","lineNumber":533,"sourceCode":"            ColumnStatistics expected = expectedColumnStatistics.get(i);\n            validateColumnStatisticsEquivalent(orcDataSourceId, name + \" column \" + i, actual, expected);\n        }\n    }\n\n    private static void validateColumnStatisticsEquivalent(\n            OrcDataSourceId orcDataSourceId,\n            String name,\n            ColumnStatistics actualColumnStatistics,\n            ColumnStatistics expectedColumnStatistics)\n            throws OrcCorruptionException\n    {\n        requireNonNull(name, \"name is null\");\n        requireNonNull(actualColumnStatistics, \"actualColumnStatistics is null\");\n        requireNonNull(expectedColumnStatistics, \"expectedColumnStatistics is null\");\n\n        if (actualColumnStatistics.getNumberOfValues() != expectedColumnStatistics.getNumberOfValues()) {\n            String failureMessage = format(\"Actual Values %s does not match expected values %s\", actualColumnStatistics, expectedColumnStatistics);\n            throw new OrcCorruptionException(orcDataSourceId, \"Write validation failed: %s in %s statistics\", failureMessage, name);\n        }\n\n        if (!Objects.equals(actualColumnStatistics.getBooleanStatistics(), expectedColumnStatistics.getBooleanStatistics())) {\n            throw new OrcCorruptionException(orcDataSourceId, \"Write validation failed: unexpected boolean counts in %s statistics\", name);\n        }\n        if (!Objects.equals(actualColumnStatistics.getIntegerStatistics(), expectedColumnStatistics.getIntegerStatistics())) {\n            IntegerStatistics actualIntegerStatistics = actualColumnStatistics.getIntegerStatistics();\n            IntegerStatistics expectedIntegerStatistics = expectedColumnStatistics.getIntegerStatistics();\n            // The sum of the integer stats depends on the order of how we merge them.\n            // It is possible the sum can overflow with one order but not in another.\n            // Ignore the validation of sum if one of the two sums is null.\n            if (actualIntegerStatistics == null ||\n                    expectedIntegerStatistics == null ||\n                    !Objects.equals(actualIntegerStatistics.getMin(), expectedIntegerStatistics.getMin()) ||\n                    !Objects.equals(actualIntegerStatistics.getMax(), expectedIntegerStatistics.getMax()) ||\n                    (actualIntegerStatistics.getSum() != null &&\n                            expectedIntegerStatistics.getSum() != null &&\n                            !Objects.equals(actualIntegerStatistics.getSum(), expectedIntegerStatistics.getSum()))) {","sourceCodeStart":515,"sourceCodeEnd":551,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-orc/src/main/java/com/facebook/presto/orc/OrcWriteValidation.java#L515-L551","documentation":"This is the generic statistics-value mismatch in write validation: the number of values recorded per column at write time must equal the number of values recomputed from the file. The thrown message embeds a detailed failureMessage comparing the actual and expected ColumnStatistics. A mismatch means data was lost, added, or corrupted between write and read.","triggerScenarios":"validateColumnStatisticsEquivalent sees actualColumnStatistics.getNumberOfValues() != expectedColumnStatistics.getNumberOfValues() at file, stripe, or row group level — non-null value counts differ from the writer's record.","commonSituations":"Truncated or partially written ORC files; data modified after write; writer bugs miscounting non-null values for nested/flattened columns.","solutions":["Restore the file from a backup or re-run the job that wrote it — the value counts genuinely differ","Check for storage-level corruption and enable checksums","Upgrade Presto if the mismatch matches a known writer statistics-counting bug","Set orc.write-validation=false only to read the file for triage, then re-write it"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Compare read-back non-null counts to expected before validation-sensitive queries\n// SELECT count(*) FROM table WHERE col IS NOT NULL; vs recorded stats","typeGuard":null,"tryCatchPattern":"try {\n    orcBatchReader.nextPage();\n} catch (OrcCorruptionException e) {\n    if (e.getMessage().contains(\"Write validation failed\")) {\n        // file likely truncated/altered; fall back to backup copy\n    }\n}","preventionTips":["Confirm write completion (commit markers) before making files readable","Enable storage checksums to catch truncation","Keep writer/reader Presto versions in sync","Investigate any genuine value-count mismatch — it signals data loss"],"tags":["orc","write-validation","statistics","corruption"],"backgroundTag":"orc-write-validation-mismatch","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}