{"record":{"id":"c59bb4b05b7fd0c3","repo":"pentaho/pentaho-kettle","slug":"stream-lookup-comparator-error","errorCode":null,"errorMessage":"Stream Lookup comparator error","messagePattern":"Stream Lookup comparator error","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"engine/src/main/java/org/pentaho/di/trans/steps/streamlookup/StreamLookupData.java","lineNumber":99,"sourceCode":"  public boolean hasLookupRows;\n\n  public StreamInterface infoStream;\n\n  public StreamLookupData() {\n    super();\n    look = new HashMap<RowMetaAndData, Object[]>();\n    hashIndex = null;\n    longIndex = new LongHashIndex();\n    list = new ArrayList<KeyValue>();\n    metadataVerifiedIntegerPair = false;\n    hasLookupRows = false;\n\n    comparator = new Comparator<KeyValue>() {\n      public int compare( KeyValue k1, KeyValue k2 ) {\n        try {\n          return cacheKeyMeta.compare( k1.getKey(), k2.getKey() );\n        } catch ( KettleValueException e ) {\n          throw new RuntimeException( \"Stream Lookup comparator error\", e );\n        }\n      }\n    };\n  }\n\n}\n","sourceCodeStart":81,"sourceCodeEnd":106,"githubUrl":"https://github.com/pentaho/pentaho-kettle/blob/f3058517a153da500bf4551f46d79b91bf8ec552/engine/src/main/java/org/pentaho/di/trans/steps/streamlookup/StreamLookupData.java#L81-L106","documentation":"StreamLookupData installs a Comparator over cached KeyValue entries that delegates to cacheKeyMeta.compare(); if the comparison throws KettleValueException it is rethrown as a RuntimeException 'Stream Lookup comparator error'. This happens when two cached key values cannot be compared per the declared value metadata (incompatible or corrupt key data in the cache).","triggerScenarios":"The comparator's compare(k1,k2) is invoked (e.g. during cache operations/sorting) and cacheKeyMeta.compare throws because cached key objects don't match the cache key metadata types.","commonSituations":"Mixed-type keys in the lookup stream (e.g. some rows Integer, some String after dynamic metadata); null/malformed key values; row layouts changing mid-stream so cacheKeyMeta no longer matches cached data.","solutions":["Normalize the lookup stream key fields to a single consistent data type upstream (Select Values metadata conversion)","Filter out or default null/malformed key values before the StreamLookup step","Ensure all lookup rows share the same row layout (use 'Field exists in stream?' checks)","Clear any stale transformation state / re-run so the cache is rebuilt with consistent metadata"],"exampleFix":"// before: raw keys with mixed types go to lookup\n// after: Select Values step: 'id' -> type Integer, non-strict, then Filter rows where id IS NULL","handlingStrategy":"try-catch","validationCode":"// Ensure consistent key types across all lookup rows\nObject k = row.getValueMeta(0).convertToNormalStorage(row.getData()[0]);\nif (!expectedClass.isInstance(k)) throw new IllegalStateException(\"Key type drift detected\");","typeGuard":"boolean comparableKeys(ValueMetaInterface keyMeta, Object a, Object b) {\n  return keyMeta != null && a != null && b != null && a.getClass().equals(b.getClass());\n}","tryCatchPattern":"try { cacheSortOrSearch(); } catch (RuntimeException e) {\n  if (e.getMessage().contains(\"comparator error\")) {\n    log.error(\"Incompatible cached key values; rebuild cache with uniform metadata\", e);\n  }\n  throw e;\n}","preventionTips":["Guarantee one uniform row layout on the lookup stream","Convert key fields to fixed types before the lookup","Handle null keys explicitly (filter or default) before caching"],"tags":["kettle","pdi","comparator","type-mismatch"],"backgroundTag":"type-mismatch","analyzedSha":"f3058517a153da500bf4551f46d79b91bf8ec552","analyzedAt":"2026-09-13T14:04:16.340Z","contentChangedAt":"2026-09-13T14:04:16.340Z","schemaVersion":2},"datasetVersion":"2026-09-20T23:17:15.980Z"}