apache/flink · error · NullKeyFieldException

Field {fieldNumber} is null, but expected to hold a key.

Error message

Field {fieldNumber} is null, but expected to hold a key.

What it means

TupleComparatorBase.compareToReference(TypeComparator other) compares the stored reference records of two comparators field-by-field via each field comparator's compareToReference. If a reference field value is null, the field comparator throws NullPointerException, which is converted to NullKeyFieldException(keyPositions[i]). The record previously stored as reference has a null in a key position, so reference-based comparison cannot proceed.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleComparatorBase.java:181

    //  Comparator Methods
    // --------------------------------------------------------------------------------------------

    @Override
    public int compareToReference(TypeComparator<T> referencedComparator) {
        TupleComparatorBase<T> other = (TupleComparatorBase<T>) referencedComparator;

        int i = 0;
        try {
            for (; i < this.keyPositions.length; i++) {
                @SuppressWarnings("unchecked")
                int cmp = this.comparators[i].compareToReference(other.comparators[i]);
                if (cmp != 0) {
                    return cmp;
                }
            }
            return 0;
        } catch (NullPointerException npex) {
            throw new NullKeyFieldException(keyPositions[i]);
        } catch (IndexOutOfBoundsException iobex) {
            throw new KeyFieldOutOfBoundsException(keyPositions[i]);
        }
    }

    @SuppressWarnings("unchecked")
    @Override
    public int compareSerialized(DataInputView firstSource, DataInputView secondSource)
            throws IOException {
        if (deserializedFields1 == null) {
            instantiateDeserializationUtils();
        }

        int i = 0;
        try {
            for (; i < serializers.length; i++) {
                deserializedFields1[i] =
                        serializers[i].deserialize(deserializedFields1[i], firstSource);

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Eliminate nulls from key fields before records enter the keyed/sorted operator (filter or substitute defaults).
2. Choose key fields that are guaranteed non-null by the schema; validate with a unit test containing edge-case nulls.
3. For fields that legitimately allow null, sort on a null-aware wrapper value instead of raw key positions.

Example fix

// before
env.fromElements(Tuple2.of((Long) null, "a"), Tuple2.of(1L, "b"))
   .groupBy(0).reduce(...); // NullKeyFieldException(0)

// after
env.fromElements(Tuple2.of((Long) null, "a"), Tuple2.of(1L, "b"))
   .filter(t -> t.f0 != null)
   .groupBy(0).reduce(...);

public static <T extends Tuple> boolean referenceSafe(T record, int[] keyPositions) {
    for (int p : keyPositions) {
        if (record.getField(p) == null) return false;
    }
    return true;
}
// only setReference/compareToReference after referenceSafe(record, keyPositions)

try {
    int cmp = a.compareToReference(b);
} catch (NullKeyFieldException e) {
    // reference record has a null key field: route to null-key handling or reject record
}

When it happens

Trigger: A sort/hash-match operator calls setReference on records, later compareToReference runs; one of the reference records had null at a key position -> NPE inside the field comparator -> NullKeyFieldException. Happens with null values in sort/join/group key fields regardless of the record path (reference vs candidate).

Common situations: Null keys from dirty source data, outer-join outputs, optional fields becoming keys. Same data-quality class as error 691 but hit through the reference-comparison path.

Related errors

• Field {fieldNumber} is null, but expected to hold a key.
• Field {fieldNumber} is accessed for a key, but out of bounds
• A NullPointerException occurred while accessing a key field
• {}
• Field {fieldNumber} is accessed for a key, but out of bounds

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/c6f04fedc12bf681. Report an issue: GitHub.