pentaho/pentaho-kettle · error · KettleStepException

DimensionLookup.Exception.ErrorDetectedInComparingFields

Error message

DimensionLookup.Exception.ErrorDetectedInComparingFields

What it means

During the comparison of the incoming row's fields with the returned dimension row (to detect if a row changed), the step needs the ValueMeta for each compared column of the return row. If that column does not exist (v2 == null), it concludes the transformation definition was tweaked (stream column matched to a non-existent table column) and throws this exception naming meta.getFieldStream()[i].

Solutions

  1. Open the Dimension Lookup dialog and re-select the comparison fields so each exists in the dimension table
  2. Run 'SQL' button in the dialog to see the actual table layout and ALTER the table to add missing columns
  3. Remove or fix the stale field in the 'Fields to compare' grid
  4. Refresh/re-import the table metadata if the schema changed in the database

Example fix

// before
// fields to compare includes "email_v2" but dim table has "email"
// after
// ALTER TABLE dim_customer ADD COLUMN email_v2 VARCHAR(255);  -- or
// edit dialog: replace email_v2 with email in the fields-to-compare list
Defensive patterns

Strategy: validation

Validate before calling

// before running, verify all compare fields exist in the dimension table
for (String f : fieldsToCompare) {
  if (!dimensionTableColumns.contains(f)) {
    throw new IllegalStateException("Compare field not in dimension table: " + f);
  }
}

Type guard

boolean returnRowHasColumn(RowMetaInterface returnMeta, String field) {
  return returnMeta != null && field != null && returnMeta.indexOfValue(field) >= 0;
}

Try / catch

try {
  lookupValues(rowMeta, row);
} catch (KettleStepException e) {
  if (e.getMessage().contains("ErrorDetectedInComparingFields")) {
    logError("Dimension schema out of sync: " + e.getMessage());
  } else { throw e; }
}

Prevention

When it happens

Trigger: lookupValues compares fields listed in meta.getFieldStream(); when the field at returnRowColNum is absent from data.returnRowMeta, v2 is null and the error fires.

Common situations: Dimension table altered (column dropped/renamed) after the transformation was saved; XML hand-edited so a lookup/compare field points at a column the table doesn't return; changed 'fields to compare' list without updating the table.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/4266b2ea25191ebe. Report an issue: GitHub.

Appendix: source

Thrown at engine/src/main/java/org/pentaho/di/trans/steps/dimensionlookup/DimensionLookup.java:636

                  // through the loop
                  valueData2 = returnRow[ j ]; // get the valueData2 for comparison
                  break; // get outta here.
                } else {
                  // Reset to null because otherwise, we'll get a false finding at the end.
                  // This could be optimized to use a temporary variable to avoid the repeated set if necessary
                  // but it will never be as slow as the database lookup anyway
                  v2 = null;
                }
              }
            } else {
              // We have a value in the columnLookupArray - use the value stored there.
              v2 = data.returnRowMeta.getValueMeta( returnRowColNum );
              valueData2 = returnRow[ returnRowColNum ];
            }
            if ( v2 == null ) {
              // If we made it here, then maybe someone tweaked the XML in the transformation
              // and we're matching a stream column to a column that doesn't really exist. Throw an exception.
              throw new KettleStepException( BaseMessages.getString(
                PKG, "DimensionLookup.Exception.ErrorDetectedInComparingFields", meta.getFieldStream()[ i ] ) );
            }

            try {
              cmp = v1.compare( valueData1, v2, valueData2 );
            } catch ( ClassCastException e ) {
              throw e;
            }

            // Not the same and update = 'N' --> insert
            if ( cmp != 0 ) {
              identical = false;
            }

            // Field flagged for insert: insert
            if ( cmp != 0 && meta.getFieldUpdate()[ i ] == DimensionLookupMeta.TYPE_UPDATE_DIM_INSERT ) {
              insert = true;
            }

View on GitHub (pinned to f3058517a1)