{"record":{"id":"4266b2ea25191ebe","repo":"pentaho/pentaho-kettle","slug":"dimensionlookup-exception-errordetectedincomparingfields","errorCode":null,"errorMessage":"DimensionLookup.Exception.ErrorDetectedInComparingFields","messagePattern":"DimensionLookup\\.Exception\\.ErrorDetectedInComparingFields","errorType":"exception","errorClass":"KettleStepException","httpStatus":null,"severity":"error","filePath":"engine/src/main/java/org/pentaho/di/trans/steps/dimensionlookup/DimensionLookup.java","lineNumber":636,"sourceCode":"                  // through the loop\n                  valueData2 = returnRow[ j ]; // get the valueData2 for comparison\n                  break; // get outta here.\n                } else {\n                  // Reset to null because otherwise, we'll get a false finding at the end.\n                  // This could be optimized to use a temporary variable to avoid the repeated set if necessary\n                  // but it will never be as slow as the database lookup anyway\n                  v2 = null;\n                }\n              }\n            } else {\n              // We have a value in the columnLookupArray - use the value stored there.\n              v2 = data.returnRowMeta.getValueMeta( returnRowColNum );\n              valueData2 = returnRow[ returnRowColNum ];\n            }\n            if ( v2 == null ) {\n              // If we made it here, then maybe someone tweaked the XML in the transformation\n              // and we're matching a stream column to a column that doesn't really exist. Throw an exception.\n              throw new KettleStepException( BaseMessages.getString(\n                PKG, \"DimensionLookup.Exception.ErrorDetectedInComparingFields\", meta.getFieldStream()[ i ] ) );\n            }\n\n            try {\n              cmp = v1.compare( valueData1, v2, valueData2 );\n            } catch ( ClassCastException e ) {\n              throw e;\n            }\n\n            // Not the same and update = 'N' --> insert\n            if ( cmp != 0 ) {\n              identical = false;\n            }\n\n            // Field flagged for insert: insert\n            if ( cmp != 0 && meta.getFieldUpdate()[ i ] == DimensionLookupMeta.TYPE_UPDATE_DIM_INSERT ) {\n              insert = true;\n            }","sourceCodeStart":618,"sourceCodeEnd":654,"githubUrl":"https://github.com/pentaho/pentaho-kettle/blob/f3058517a153da500bf4551f46d79b91bf8ec552/engine/src/main/java/org/pentaho/di/trans/steps/dimensionlookup/DimensionLookup.java#L618-L654","documentation":"During the comparison of the incoming row's fields with the returned dimension row (to detect if a row changed), the step needs the ValueMeta for each compared column of the return row. If that column does not exist (v2 == null), it concludes the transformation definition was tweaked (stream column matched to a non-existent table column) and throws this exception naming meta.getFieldStream()[i].","triggerScenarios":"lookupValues compares fields listed in meta.getFieldStream(); when the field at returnRowColNum is absent from data.returnRowMeta, v2 is null and the error fires.","commonSituations":"Dimension table altered (column dropped/renamed) after the transformation was saved; XML hand-edited so a lookup/compare field points at a column the table doesn't return; changed 'fields to compare' list without updating the table.","solutions":["Open the Dimension Lookup dialog and re-select the comparison fields so each exists in the dimension table","Run 'SQL' button in the dialog to see the actual table layout and ALTER the table to add missing columns","Remove or fix the stale field in the 'Fields to compare' grid","Refresh/re-import the table metadata if the schema changed in the database"],"exampleFix":"// before\n// fields to compare includes \"email_v2\" but dim table has \"email\"\n// after\n// ALTER TABLE dim_customer ADD COLUMN email_v2 VARCHAR(255);  -- or\n// edit dialog: replace email_v2 with email in the fields-to-compare list","handlingStrategy":"validation","validationCode":"// before running, verify all compare fields exist in the dimension table\nfor (String f : fieldsToCompare) {\n  if (!dimensionTableColumns.contains(f)) {\n    throw new IllegalStateException(\"Compare field not in dimension table: \" + f);\n  }\n}","typeGuard":"boolean returnRowHasColumn(RowMetaInterface returnMeta, String field) {\n  return returnMeta != null && field != null && returnMeta.indexOfValue(field) >= 0;\n}","tryCatchPattern":"try {\n  lookupValues(rowMeta, row);\n} catch (KettleStepException e) {\n  if (e.getMessage().contains(\"ErrorDetectedInComparingFields\")) {\n    logError(\"Dimension schema out of sync: \" + e.getMessage());\n  } else { throw e; }\n}","preventionTips":["Whenever the dimension table DDL changes, re-validate the step's field-to-compare grid","Use the dialog's 'SQL' button to detect table/step drift and generate ALTERs","Avoid hand-editing transformation XML; make schema changes through the GUI"],"tags":["etl","schema-mismatch","field-mapping","slowly-changing-dimension"],"backgroundTag":"schema-validation-failed","analyzedSha":"f3058517a153da500bf4551f46d79b91bf8ec552","analyzedAt":"2026-09-13T14:04:16.340Z","contentChangedAt":"2026-09-13T14:04:16.340Z","schemaVersion":2},"datasetVersion":"2026-09-20T23:17:15.980Z"}