{"record":{"id":"53df499d38d8b189","repo":"pentaho/pentaho-kettle","slug":"fuzzymatch-error-javaheap","errorCode":null,"errorMessage":"FuzzyMatch.Error.JavaHeap","messagePattern":"FuzzyMatch\\.Error\\.JavaHeap","errorType":"exception","errorClass":"KettleException","httpStatus":null,"severity":"critical","filePath":"engine/src/main/java/org/pentaho/di/trans/steps/fuzzymatch/FuzzyMatch.java","lineNumber":199,"sourceCode":"    Object[] add = null;\n    if ( row[ data.indexOfMainField ] == null ) {\n      add = buildEmptyRow();\n    } else {\n      try {\n        add = getFromCache( row );\n      } catch ( Exception e ) {\n        throw new KettleStepException( e );\n      }\n    }\n    return RowDataUtil.addRowData( row, rowMeta.size(), add );\n  }\n\n  private void addToCache( Object[] value ) throws KettleException {\n    try {\n      data.look.add( value );\n    } catch ( java.lang.OutOfMemoryError o ) {\n      // exception out of memory\n      throw new KettleException( BaseMessages.getString( PKG, \"FuzzyMatch.Error.JavaHeap\", o.toString() ) );\n    }\n  }\n\n  private Object[] getFromCache( Object[] keyRow ) throws KettleValueException {\n    if ( isDebug() ) {\n      logDebug( BaseMessages.getString( PKG, \"FuzzyMatch.Log.ReadingMainStreamRow\", getInputRowMeta().getString(\n        keyRow ) ) );\n    }\n    Object[] retval = null;\n    switch ( meta.getAlgorithmType() ) {\n      case FuzzyMatchMeta.OPERATION_TYPE_LEVENSHTEIN:\n      case FuzzyMatchMeta.OPERATION_TYPE_DAMERAU_LEVENSHTEIN:\n      case FuzzyMatchMeta.OPERATION_TYPE_NEEDLEMAN_WUNSH:\n        retval = doDistance( keyRow );\n        break;\n      case FuzzyMatchMeta.OPERATION_TYPE_DOUBLE_METAPHONE:\n      case FuzzyMatchMeta.OPERATION_TYPE_METAPHONE:\n      case FuzzyMatchMeta.OPERATION_TYPE_SOUNDEX:","sourceCodeStart":181,"sourceCodeEnd":217,"githubUrl":"https://github.com/pentaho/pentaho-kettle/blob/f3058517a153da500bf4551f46d79b91bf8ec552/engine/src/main/java/org/pentaho/di/trans/steps/fuzzymatch/FuzzyMatch.java#L181-L217","documentation":"FuzzyMatch wraps a java.lang.OutOfMemoryError thrown while adding lookup rows to its in-memory cache into a KettleException with message 'FuzzyMatch.Error.JavaHeap'. The step builds the full lookup index in memory (data.look), so it throws when the JVM heap cannot hold more rows. It exists to convert a fatal OOM into a step-level Kettle failure with a clear message.","triggerScenarios":"addToCache() calls data.look.add(value) and the JVM throws OutOfMemoryError; this happens during readLookupValues() while streaming the lookup (second/main) input rows into the cache before transformation processing starts.","commonSituations":"Large lookup tables (millions of rows), JVM started with default or small -Xmx, joining big streams on similarity where the whole lookup side is cached, running transformation on memory-constrained containers.","solutions":["Increase JVM heap: -Xmx (e.g. -Xmx4g) in kettle JVM options / Kitchen/Pan scripts","Reduce lookup input size: filter rows, pre-select only needed fields, or use a database-side fuzzy join","Lower similarity threshold / reduce candidate cache usage if the step allows, or split the lookup into chunks","Use a machine/container with more memory or enable swap cautiously (avoid thrashing)","Consider the 'Merge Join' or streaming steps instead of FuzzyMatch for very large datasets"],"exampleFix":"// before (command line)\nkitchen.sh -file job.kjb\n// after\nkitchen.sh -file job.kjb -level Basic -Xmx4096m   # or export KETTLE_JAVA_OPTS=\"-Xmx4096m\"","handlingStrategy":"try-catch","validationCode":"long freeHeap = Runtime.getRuntime().maxMemory() - (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory());\nif (freeHeap < 512L * 1024 * 1024) {\n  throw new IllegalStateException(\"Insufficient heap for FuzzyMatch lookup cache; start JVM with larger -Xmx\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  transformation.execute(null);\n} catch (KettleException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"FuzzyMatch.Error.JavaHeap\") || e.getCause() instanceof OutOfMemoryError) {\n    // restart JVM with larger -Xmx or reduce lookup data, then retry\n  } else {\n    throw e;\n  }\n}","preventionTips":["Start the Pentaho JVM with generous -Xmx sized to lookup table volume (roughly rows x row size x 3)","Estimate lookup size before running; filter or paginate the lookup stream","Monitor heap usage in Spoon/JMX and alert before full GC storms","Prefer DB-side joins or streaming steps for very large lookup datasets"],"tags":["jvm","out-of-memory","heap","pentaho-kettle"],"backgroundTag":"java-heap-out-of-memory","analyzedSha":"f3058517a153da500bf4551f46d79b91bf8ec552","analyzedAt":"2026-09-13T14:04:16.340Z","contentChangedAt":"2026-09-13T14:04:16.340Z","schemaVersion":2},"datasetVersion":"2026-09-20T23:17:15.980Z"}