prestodb/presto · error · InvalidCheckpointException

Column %s, of type %s, contains %s offset positions, but onl

Error message

Column %s, of type %s, contains %s offset positions, but only %s positions were consumed

What it means

When reconstructing stream checkpoints from row-group footer position lists, the library consumes positions per stream. If a column's position list still has unconsumed entries after all streams for that column type consumed their positions — and not all extra entries are zero (the DWRF first-row-group quirk) — the position list does not match the expected stream layout and InvalidCheckpointException is thrown.

Source

Thrown at presto-orc/src/main/java/com/facebook/presto/orc/checkpoint/Checkpoints.java:139

                case MAP:
                    checkpoints.putAll(getListOrMapColumnCheckpoints(column, sequence, columnEncoding, compressed, availableStreams, columnPositionsList));
                    break;
                case STRUCT:
                    checkpoints.putAll(getStructColumnCheckpoints(column, sequence, compressed, availableStreams, columnPositionsList));
                    break;
                case DECIMAL:
                    checkpoints.putAll(getDecimalColumnCheckpoints(column, sequence, columnEncoding, compressed, availableStreams, columnPositionsList));
                    break;
                default:
                    throw new IllegalArgumentException("Unsupported column type " + columnType);
            }

            // The DWRF code is not meticulous in the handling of checkpoints.  It appears that for the first row group
            // it will write checkpoints for all streams, but in other cases it will write only the streams that exist.
            // We detect this case by checking that all offsets in the initial position list are zero, and if so, we
            // clear the extra offsets
            if (columnPositionsList.hasNextPosition() && !Arrays.stream(positions).allMatch(num -> num == 0)) {
                throw new InvalidCheckpointException(format("Column %s, of type %s, contains %s offset positions, but only %s positions were consumed",
                        column,
                        columnType,
                        positions.length,
                        columnPositionsList.getIndex()));
            }
        }
        return checkpoints.build();
    }

    public static StreamCheckpoint getDictionaryStreamCheckpoint(StreamId streamId, OrcTypeKind columnType, ColumnEncodingKind columnEncoding)
    {
        if (streamId.getStreamKind() == DICTIONARY_DATA) {
            switch (columnType) {
                case SHORT:
                case INT:
                case LONG:
                    return new LongStreamDwrfCheckpoint(createInputStreamCheckpoint(0, 0));
                case STRING:

View on GitHub (pinned to 55bb57d202)

Solutions

1. Identify the file/column from the error and validate it with orc-tools/vertex metadata dump.
2. Rewrite the file with a current writer (CREATE TABLE AS SELECT from the source table).
3. Check writer-version compatibility between the producing engine and this Presto ORC reader.
4. If the extra positions are all zero (known DWRF quirk), upgrading the reader version may already tolerate it — try a newer Presto.

try { checkpoints = Checkpoints.getStreamCheckpoints(...); }
catch (InvalidCheckpointException e) { fallbackToFullStripeScan(e); }

When it happens

Trigger: getStreamCheckpoints processes a ColumnPositionsList whose remaining positions length exceeds what the column's streams consumed, with non-zero leftovers — i.e. mismatched stripe footer metadata vs. the expected stream schema.

Common situations: Files written by DWRF/Hive versions with different stream ordering or extra streams, corrupted stripe footers, or files whose column encodings disagree with the position lists.

Related errors

• Not enough positions for column %s and sequence %s, of type
• Reset stream has a block offset but stream is not compressed
• NOT_SUPPORTED
• HIVE_WRITER_CLOSE_ERROR
• HIVE_WRITER_OPEN_ERROR

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/14b34237fb53cfc7. Report an issue: GitHub.