apache/flink · error · IllegalArgumentException

Parquet does not support null keys in maps. See https://gith

Error message

Parquet does not support null keys in maps. See https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#maps for more details.

What it means

The parquet map logical type requires keys to be non-null. When writing a Flink MAP column, if any key element is null, ParquetRowDataWriter throws IllegalArgumentException referencing the parquet LogicalTypes spec, because a null key cannot be represented in the MAP key_value encoding.

Source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/row/ParquetRowDataWriter.java:408

        }

        private void writeMapData(MapData mapData) {
            recordConsumer.startGroup();

            if (mapData != null && mapData.size() > 0) {
                recordConsumer.startField(repeatedGroupName, 0);

                ArrayData keyArray = mapData.keyArray();
                ArrayData valueArray = mapData.valueArray();
                for (int i = 0; i < keyArray.size(); i++) {
                    recordConsumer.startGroup();
                    if (!keyArray.isNullAt(i)) {
                        // write key element
                        recordConsumer.startField(keyName, 0);
                        keyWriter.write(keyArray, i);
                        recordConsumer.endField(keyName, 0);
                    } else {
                        throw new IllegalArgumentException(
                                "Parquet does not support null keys in maps. See https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#maps for more details.");
                    }

                    if (!valueArray.isNullAt(i)) {
                        // write value element
                        recordConsumer.startField(valueName, 1);
                        valueWriter.write(valueArray, i);
                        recordConsumer.endField(valueName, 1);
                    }
                    recordConsumer.endGroup();
                }

                recordConsumer.endField(repeatedGroupName, 0);
            }
            recordConsumer.endGroup();
        }
    }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Filter out or replace null keys before the sink: map entries with null keys are dropped or given a sentinel key
2. Fix the producer so null keys never enter the map column
3. If null keys are semantically required, model the data as an ARRAY<ROW<key,value>> instead of MAP

Example fix

// before
Map<String,Integer> in = new HashMap<>();
in.put(null, 1); // null key -> throws at parquet write

// after
in.entrySet().removeIf(e -> e.getKey() == null);
// or use ARRAY<ROW<k STRING, v INT>> to permit null keys

static boolean hasNullKeys(MapData m) {
  ArrayData keys = m.keyArray();
  for (int i = 0; i < keys.size(); i++) { if (keys.isNullAt(i)) return true; }
  return false;
}

try { writer.write(row); } catch (IllegalArgumentException e) { if (e.getMessage().contains("null keys in maps")) { /* sanitize map and re-emit */ } throw e; }

When it happens

Trigger: Writing RowData where mapData.keyArray().isNullAt(i) is true for any entry - i.e. the input map contains a null key.

Common situations: Upstream data (Kafka JSON, JDBC, user functions) producing maps with null keys; Java HashMap允许 null keys flowing into a map-typed column; missing null-key filtering before the parquet sink.

Related errors

• Field {fieldNumber} is null, but expected to hold a key.
• Field {fieldNumber} is null, but expected to hold a key.
• Illegal JSON map data...
• JSON format doesn't support to serialize map data with null
• Expected two fields for the map (key/value), but got: {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/d55402afa35639ae. Report an issue: GitHub.