apache/flink · error · IOException

could not decode the dictionary for {}

Error message

could not decode the dictionary for {}

What it means

IOException from AbstractColumnReader.initFromPages/constructor: the column chunk carried a dictionary page, but dictionaryPage.getEncoding().initDictionary(descriptor, dictionaryPage) threw - the dictionary encoding is unsupported for this physical type or the dictionary bytes are corrupt. The original exception is chained as the cause.

Source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/reader/AbstractColumnReader.java:117

    ByteBufferInputStream dataInputStream;

    /** Dictionary decoder to wrap dictionary ids input stream. */
    private RunLengthDecoder dictionaryIdsDecoder;

    public AbstractColumnReader(ColumnDescriptor descriptor, PageReader pageReader)
            throws IOException {
        this.descriptor = descriptor;
        this.pageReader = pageReader;
        this.maxDefLevel = descriptor.getMaxDefinitionLevel();

        DictionaryPage dictionaryPage = pageReader.readDictionaryPage();
        if (dictionaryPage != null) {
            try {
                this.dictionary =
                        dictionaryPage.getEncoding().initDictionary(descriptor, dictionaryPage);
                this.isCurrentPageDictionaryEncoded = true;
            } catch (IOException e) {
                throw new IOException("could not decode the dictionary for " + descriptor, e);
            }
        } else {
            this.dictionary = null;
            this.isCurrentPageDictionaryEncoded = false;
        }
        /*
         * Total number of values in this column (in this row group).
         */
        long totalValueCount = pageReader.getTotalValueCount();
        if (totalValueCount == 0) {
            throw new IOException("totalValueCount == 0");
        }
    }

    protected void checkTypeName(PrimitiveType.PrimitiveTypeName expectedName) {
        PrimitiveType.PrimitiveTypeName actualName =
                descriptor.getPrimitiveType().getPrimitiveTypeName();
        checkArgument(

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Read the cause exception to identify the failing encoding
2. Rewrite the file with a standard/older writer (PLAIN_DICTIONARY/RLE_DICTIONARY) or disable exotic encodings on the producer
3. Upgrade Flink so its bundled parquet-mr knows the encoding
4. Verify storage-level integrity (checksums, re-download) to rule out corruption

catch (IOException e) { if (e.getMessage() != null && e.getMessage().startsWith("could not decode the dictionary")) { Throwable c = e.getCause(); /* inspect encoding in c, rewrite file with standard encodings or upgrade Flink */ } else throw e; }

When it happens

Trigger: Opening a column reader for a chunk whose dictionary page uses an encoding the reader cannot decode (e.g. a newer/obsolete dictionary encoding for the physical type), or whose dictionary payload is malformed.

Common situations: Parquet files written by writers with encoding extensions (e.g. some Delta/ALPHA encodings) that stock parquet-mr cannot decode; bit-level corruption in dictionary pages from bad storage; version skew between writer-side and Flink's bundled parquet-mr.

Related errors

• expecting more rows but reached last block. Read {} out of {
• totalValueCount == 0
• could not read page {} in col {}
• Error while waiting for job to be initialized
• Translator {} cannot translate the given pipeline {}.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/341a8249be25e77b. Report an issue: GitHub.