apache/flink · error · NullKeyFieldException

Field {fieldNumber} is null, but expected to hold a key.

Error message

Field {fieldNumber} is null, but expected to hold a key.

What it means

TupleComparator.putNormalizedKey writes the leading key fields' normalized forms into a memory segment for fast sorting. If a key field's value is null, the underlying comparator throws NullPointerException, which this method converts into NullKeyFieldException(keyPositions[i]) with message 'Field {fieldNumber} is null, but expected to hold a key.' Sorting/partitioning requires non-null keys; nulls in key fields are rejected.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleComparator.java:140

    }

    @SuppressWarnings("unchecked")
    @Override
    public void putNormalizedKey(T value, MemorySegment target, int offset, int numBytes) {
        int i = 0;
        try {
            for (; i < this.numLeadingNormalizableKeys && numBytes > 0; i++) {
                int len = this.normalizedKeyLengths[i];
                len = numBytes >= len ? len : numBytes;
                this.comparators[i].putNormalizedKey(
                        value.getFieldNotNull(this.keyPositions[i]), target, offset, len);
                numBytes -= len;
                offset += len;
            }
        } catch (NullFieldException nfex) {
            throw new NullKeyFieldException(nfex);
        } catch (NullPointerException npex) {
            throw new NullKeyFieldException(this.keyPositions[i]);
        }
    }

    @Override
    public int extractKeys(Object record, Object[] target, int index) {
        int localIndex = index;
        for (int i = 0; i < comparators.length; i++) {
            localIndex +=
                    comparators[i].extractKeys(
                            ((Tuple) record).getField(keyPositions[i]), target, localIndex);
        }
        return localIndex - index;
    }

    public TypeComparator<T> duplicate() {
        return new TupleComparator<T>(this);
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Filter or transform null keys before the sort/keyed operation: records with null keys cannot participate in ordering.
2. Replace nulls with a sentinel that sorts as intended (e.g. empty string, Long.MIN_VALUE) at the projection step, and document the sentinel.
3. If nulls are legitimate, sort on a wrapped type with a null-aware comparator instead of raw tuple key positions.

Example fix

// before
DataStream<Tuple2<Long,String>> s = env.fromElements(
    Tuple2.of(1L, (String) null), Tuple2.of(2L, "b"));
s.sortPartition(1, Order.ASCENDING); // NullKeyFieldException(1)

// after
DataStream<Tuple2<Long,String>> s = env.fromElements(
    Tuple2.of(1L, (String) null), Tuple2.of(2L, "b"))
    .filter(t -> t.f1 != null); // or map null -> ""
s.sortPartition(1, Order.ASCENDING);

public static boolean hasNonNullKeys(Tuple t, int[] keyPositions) {
    for (int p : keyPositions) {
        if (t.getField(p) == null) return false;
    }
    return true;
}
// stream.filter(t -> hasNonNullKeys(t, keyPositions)) before sorting/partitioning

public static boolean hasSortSafeKeys(Tuple2<Long,String> t) {
    return t.f1 != null; // key field non-null
}

try {
    comparator.putNormalizedKey(value, segment, off, len);
} catch (org.apache.flink.api.common.typeutils.NullKeyFieldException e) {
    // field index in message: null key encountered; divert record to a null-handling path
}

When it happens

Trigger: Sorting or range-partitioning tuple records where the field at a key position is null, and putNormalizedKey is invoked (normalized-key path of the sort algorithm) -> NPE inside the field comparator -> NullKeyFieldException. Nulls in keys of Tuple1<String> with null element are the classic case.

Common situations: Upstream data contains nulls in the field chosen as sort/group key (dirty source data, left-join misses, JSON with missing attributes). Schema evolution adds nullability to a key column. Unit tests using null placeholders in key fields.

Related errors

• Field {fieldNumber} is null, but expected to hold a key.
• Tuple position is out of range: {fieldPos}
• {}
• Field {fieldIdx} is null, but expected to hold a value.
• Parquet does not support null keys in maps. See https://gith

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/1fd97c1e22fdd068. Report an issue: GitHub.