pentaho/pentaho-kettle · error · RuntimeException

Stream Lookup comparator error

Error message

Stream Lookup comparator error

What it means

StreamLookupData installs a Comparator over cached KeyValue entries that delegates to cacheKeyMeta.compare(); if the comparison throws KettleValueException it is rethrown as a RuntimeException 'Stream Lookup comparator error'. This happens when two cached key values cannot be compared per the declared value metadata (incompatible or corrupt key data in the cache).

Solutions

1. Normalize the lookup stream key fields to a single consistent data type upstream (Select Values metadata conversion)
2. Filter out or default null/malformed key values before the StreamLookup step
3. Ensure all lookup rows share the same row layout (use 'Field exists in stream?' checks)
4. Clear any stale transformation state / re-run so the cache is rebuilt with consistent metadata

Example fix

// before: raw keys with mixed types go to lookup
// after: Select Values step: 'id' -> type Integer, non-strict, then Filter rows where id IS NULL

// Ensure consistent key types across all lookup rows
Object k = row.getValueMeta(0).convertToNormalStorage(row.getData()[0]);
if (!expectedClass.isInstance(k)) throw new IllegalStateException("Key type drift detected");

boolean comparableKeys(ValueMetaInterface keyMeta, Object a, Object b) {
  return keyMeta != null && a != null && b != null && a.getClass().equals(b.getClass());
}

try { cacheSortOrSearch(); } catch (RuntimeException e) {
  if (e.getMessage().contains("comparator error")) {
    log.error("Incompatible cached key values; rebuild cache with uniform metadata", e);
  }
  throw e;
}

When it happens

Trigger: The comparator's compare(k1,k2) is invoked (e.g. during cache operations/sorting) and cacheKeyMeta.compare throws because cached key objects don't match the cache key metadata types.

Common situations: Mixed-type keys in the lookup stream (e.g. some rows Integer, some String after dynamic metadata); null/malformed key values; row layouts changing mid-stream so cacheKeyMeta no longer matches cached data.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• GPLoadDataOutput.Exception.TypeNotSupported
• The 'A-B%' function only works on numeric data
• The 'A+B%' function only works on numeric data
• The 'A/B in %' function only works on numeric data
• The 'sqrt' function only works on numeric data.

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/c59bb4b05b7fd0c3. Report an issue: GitHub.

Appendix: source

Thrown at engine/src/main/java/org/pentaho/di/trans/steps/streamlookup/StreamLookupData.java:99

  public boolean hasLookupRows;

  public StreamInterface infoStream;

  public StreamLookupData() {
    super();
    look = new HashMap<RowMetaAndData, Object[]>();
    hashIndex = null;
    longIndex = new LongHashIndex();
    list = new ArrayList<KeyValue>();
    metadataVerifiedIntegerPair = false;
    hasLookupRows = false;

    comparator = new Comparator<KeyValue>() {
      public int compare( KeyValue k1, KeyValue k2 ) {
        try {
          return cacheKeyMeta.compare( k1.getKey(), k2.getKey() );
        } catch ( KettleValueException e ) {
          throw new RuntimeException( "Stream Lookup comparator error", e );
        }
      }
    };
  }

}

View on GitHub (pinned to f3058517a1)