pentaho/pentaho-kettle · critical · KettleException

FuzzyMatch.Error.JavaHeap

Error message

FuzzyMatch.Error.JavaHeap

What it means

FuzzyMatch wraps a java.lang.OutOfMemoryError thrown while adding lookup rows to its in-memory cache into a KettleException with message 'FuzzyMatch.Error.JavaHeap'. The step builds the full lookup index in memory (data.look), so it throws when the JVM heap cannot hold more rows. It exists to convert a fatal OOM into a step-level Kettle failure with a clear message.

Solutions

1. Increase JVM heap: -Xmx (e.g. -Xmx4g) in kettle JVM options / Kitchen/Pan scripts
2. Reduce lookup input size: filter rows, pre-select only needed fields, or use a database-side fuzzy join
3. Lower similarity threshold / reduce candidate cache usage if the step allows, or split the lookup into chunks
4. Use a machine/container with more memory or enable swap cautiously (avoid thrashing)
5. Consider the 'Merge Join' or streaming steps instead of FuzzyMatch for very large datasets

Example fix

// before (command line)
kitchen.sh -file job.kjb
// after
kitchen.sh -file job.kjb -level Basic -Xmx4096m   # or export KETTLE_JAVA_OPTS="-Xmx4096m"

long freeHeap = Runtime.getRuntime().maxMemory() - (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory());
if (freeHeap < 512L * 1024 * 1024) {
  throw new IllegalStateException("Insufficient heap for FuzzyMatch lookup cache; start JVM with larger -Xmx");
}

try {
  transformation.execute(null);
} catch (KettleException e) {
  if (e.getMessage() != null && e.getMessage().contains("FuzzyMatch.Error.JavaHeap") || e.getCause() instanceof OutOfMemoryError) {
    // restart JVM with larger -Xmx or reduce lookup data, then retry
  } else {
    throw e;
  }
}

When it happens

Trigger: addToCache() calls data.look.add(value) and the JVM throws OutOfMemoryError; this happens during readLookupValues() while streaming the lookup (second/main) input rows into the cache before transformation processing starts.

Common situations: Large lookup tables (millions of rows), JVM started with default or small -Xmx, joining big streams on similarity where the whole lookup side is cached, running transformation on memory-constrained containers.

Related errors

• Unexpected error during classloader creation
• Bounds of this rule are not numeric: lowerBound=
• ConnectionFileObject.PVFSRoot.UnsupportedOperation
• ConnectionFileObject.UndefinedConnection
• Could not convert to the given local format.

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/53df499d38d8b189. Report an issue: GitHub.

Appendix: source

Thrown at engine/src/main/java/org/pentaho/di/trans/steps/fuzzymatch/FuzzyMatch.java:199

    Object[] add = null;
    if ( row[ data.indexOfMainField ] == null ) {
      add = buildEmptyRow();
    } else {
      try {
        add = getFromCache( row );
      } catch ( Exception e ) {
        throw new KettleStepException( e );
      }
    }
    return RowDataUtil.addRowData( row, rowMeta.size(), add );
  }

  private void addToCache( Object[] value ) throws KettleException {
    try {
      data.look.add( value );
    } catch ( java.lang.OutOfMemoryError o ) {
      // exception out of memory
      throw new KettleException( BaseMessages.getString( PKG, "FuzzyMatch.Error.JavaHeap", o.toString() ) );
    }
  }

  private Object[] getFromCache( Object[] keyRow ) throws KettleValueException {
    if ( isDebug() ) {
      logDebug( BaseMessages.getString( PKG, "FuzzyMatch.Log.ReadingMainStreamRow", getInputRowMeta().getString(
        keyRow ) ) );
    }
    Object[] retval = null;
    switch ( meta.getAlgorithmType() ) {
      case FuzzyMatchMeta.OPERATION_TYPE_LEVENSHTEIN:
      case FuzzyMatchMeta.OPERATION_TYPE_DAMERAU_LEVENSHTEIN:
      case FuzzyMatchMeta.OPERATION_TYPE_NEEDLEMAN_WUNSH:
        retval = doDistance( keyRow );
        break;
      case FuzzyMatchMeta.OPERATION_TYPE_DOUBLE_METAPHONE:
      case FuzzyMatchMeta.OPERATION_TYPE_METAPHONE:
      case FuzzyMatchMeta.OPERATION_TYPE_SOUNDEX:

View on GitHub (pinned to f3058517a1)