{"record":{"id":"14b34237fb53cfc7","repo":"prestodb/presto","slug":"column-s-of-type-s-contains-s-offset-position","errorCode":null,"errorMessage":"Column %s, of type %s, contains %s offset positions, but only %s positions were consumed","messagePattern":"Column (.+?), of type (.+?), contains (.+?) offset positions, but only (.+?) positions were consumed","errorType":"exception","errorClass":"InvalidCheckpointException","httpStatus":null,"severity":"error","filePath":"presto-orc/src/main/java/com/facebook/presto/orc/checkpoint/Checkpoints.java","lineNumber":139,"sourceCode":"                case MAP:\n                    checkpoints.putAll(getListOrMapColumnCheckpoints(column, sequence, columnEncoding, compressed, availableStreams, columnPositionsList));\n                    break;\n                case STRUCT:\n                    checkpoints.putAll(getStructColumnCheckpoints(column, sequence, compressed, availableStreams, columnPositionsList));\n                    break;\n                case DECIMAL:\n                    checkpoints.putAll(getDecimalColumnCheckpoints(column, sequence, columnEncoding, compressed, availableStreams, columnPositionsList));\n                    break;\n                default:\n                    throw new IllegalArgumentException(\"Unsupported column type \" + columnType);\n            }\n\n            // The DWRF code is not meticulous in the handling of checkpoints.  It appears that for the first row group\n            // it will write checkpoints for all streams, but in other cases it will write only the streams that exist.\n            // We detect this case by checking that all offsets in the initial position list are zero, and if so, we\n            // clear the extra offsets\n            if (columnPositionsList.hasNextPosition() && !Arrays.stream(positions).allMatch(num -> num == 0)) {\n                throw new InvalidCheckpointException(format(\"Column %s, of type %s, contains %s offset positions, but only %s positions were consumed\",\n                        column,\n                        columnType,\n                        positions.length,\n                        columnPositionsList.getIndex()));\n            }\n        }\n        return checkpoints.build();\n    }\n\n    public static StreamCheckpoint getDictionaryStreamCheckpoint(StreamId streamId, OrcTypeKind columnType, ColumnEncodingKind columnEncoding)\n    {\n        if (streamId.getStreamKind() == DICTIONARY_DATA) {\n            switch (columnType) {\n                case SHORT:\n                case INT:\n                case LONG:\n                    return new LongStreamDwrfCheckpoint(createInputStreamCheckpoint(0, 0));\n                case STRING:","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-orc/src/main/java/com/facebook/presto/orc/checkpoint/Checkpoints.java#L121-L157","documentation":"When reconstructing stream checkpoints from row-group footer position lists, the library consumes positions per stream. If a column's position list still has unconsumed entries after all streams for that column type consumed their positions — and not all extra entries are zero (the DWRF first-row-group quirk) — the position list does not match the expected stream layout and InvalidCheckpointException is thrown.","triggerScenarios":"getStreamCheckpoints processes a ColumnPositionsList whose remaining positions length exceeds what the column's streams consumed, with non-zero leftovers — i.e. mismatched stripe footer metadata vs. the expected stream schema.","commonSituations":"Files written by DWRF/Hive versions with different stream ordering or extra streams, corrupted stripe footers, or files whose column encodings disagree with the position lists.","solutions":["Identify the file/column from the error and validate it with orc-tools/vertex metadata dump.","Rewrite the file with a current writer (CREATE TABLE AS SELECT from the source table).","Check writer-version compatibility between the producing engine and this Presto ORC reader.","If the extra positions are all zero (known DWRF quirk), upgrading the reader version may already tolerate it — try a newer Presto."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { checkpoints = Checkpoints.getStreamCheckpoints(...); }\ncatch (InvalidCheckpointException e) { fallbackToFullStripeScan(e); }","preventionTips":["Keep reader and writer versions aligned","Rewrite legacy files produced by old DWRF writers","Validate stripe footers with tooling before bulk reads"],"tags":["orc","checkpoint","metadata-mismatch"],"backgroundTag":"checkpoint-position-mismatch","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}