apache/flink · error · IllegalArgumentException

Number of key fields and comparators differ.

Error message

Number of key fields and comparators differ.

What it means

Thrown by GenericPairComparator's constructor when comparator1.getFlatComparators() and comparator2.getFlatComparators() return arrays of different lengths. The GenericPairComparator is used to compare keys between two sides of a join/co-group; both sides must project to the same number of flat (atomic) key fields for the comparison to be meaningful.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/common/typeutils/GenericPairComparator.java:49

    private final TypeComparator<T1> comparator1;
    private final TypeComparator<T2> comparator2;

    private final TypeComparator<Object>[] comparators1;
    private final TypeComparator<Object>[] comparators2;

    private final Object[] referenceKeyFields;

    private final Object[] candidateKeyFields;

    @SuppressWarnings("unchecked")
    public GenericPairComparator(TypeComparator<T1> comparator1, TypeComparator<T2> comparator2) {
        this.comparator1 = comparator1;
        this.comparator2 = comparator2;
        this.comparators1 = comparator1.getFlatComparators();
        this.comparators2 = comparator2.getFlatComparators();

        if (comparators1.length != comparators2.length) {
            throw new IllegalArgumentException("Number of key fields and comparators differ.");
        }

        int numKeys = comparators1.length;

        for (int i = 0; i < numKeys; i++) {
            this.comparators1[i] = comparators1[i].duplicate();
            this.comparators2[i] = comparators2[i].duplicate();
        }

        this.referenceKeyFields = new Object[numKeys];
        this.candidateKeyFields = new Object[numKeys];
    }

    @Override
    public void setReference(T1 reference) {
        comparator1.extractKeys(reference, referenceKeyFields, 0);
    }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Ensure both sides of the join/co-group use key selectors that produce the same number of flat key fields.
2. If one side needs a composite key, wrap the other side in a Tuple of matching arity.
3. Use a KeySelector that returns a Tuple or POJO with identical field structure on both sides.

Example fix

// before — mismatched key arity on a join
streamA.keyBy(e -> e.getId())              // 1 flat field
      .join(streamB.keyBy(e -> new Tuple2<>(e.getA(), e.getB()))) // 2 flat fields
// throws: Number of key fields and comparators differ.

// after — match key arity on both sides
streamA.keyBy(e -> new Tuple2<>(e.getCategory(), e.getId()))
      .join(streamB.keyBy(e -> new Tuple2<>(e.getA(), e.getB())))

// Verify both comparators have the same number of flat key fields before joining
TypeComparator<?> c1 = typeInfo1.createComparator(new int[]{0}, new boolean[]{true}, 0, config);
TypeComparator<?> c2 = typeInfo2.createComparator(new int[]{0, 1}, new boolean[]{true, true}, 0, config);
if (c1.getFlatComparators().length != c2.getFlatComparators().length) {
    throw new IllegalArgumentException("Key arities differ: "
        + c1.getFlatComparators().length + " vs " + c2.getFlatComparators().length);
}

When it happens

Trigger: Joining or co-grouping two DataStreams/DataSets where the key selectors produce types with a different number of flattened key fields. For example, joining on a single-field key on one side and a two-field composite key on the other. Also triggered when one side's key type is a Tuple2 (2 flat fields) and the other's is a single String (1 flat field).

Common situations: keyBy with different key arities on two streams before a join. Using a POJO key on one side and a Tuple key on the other where the flattened field counts differ. A key selector returning a nested object on one side and a flat scalar on the other.

Related errors

• Could not add a comparator for the logicalkey field index {}
• Cannot create Comparator for {typeClass.getCanonicalName()}.
• The block size parameter must be set and larger than 0.
• Delimiter must not be null
• Line length limit must be at least 1.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/d75ebcf270736d75. Report an issue: GitHub.