pentaho/pentaho-kettle · error · KettleStepException

ElasticSearchBulk.Error.InvalidIdField

ElasticSearchBulk.Error.InvalidIdField

Error message

ElasticSearchBulk.Error.InvalidIdField

What it means

initFieldIndexes also resolves the configured 'id in field' name to an index. If a non-blank idInField is configured but not present in the input row, KettleStepException 'ElasticSearchBulk.Error.InvalidIdField' is thrown. The step cannot use the specified field as the document _id.

Solutions

  1. Ensure the 'id field' configured in the step exists in the incoming stream (case-sensitive)
  2. Verify environment variables in the field name resolve correctly
  3. Re-add the id field upstream or clear the id field setting if unused
  4. Preview the stream to list actual field names

Example fix

// before
// idInField = "uuid" but stream only has "id"
// after
// set id field to "id" or add a Select values step renaming id -> uuid
Defensive patterns

Strategy: validation

Validate before calling

// ensure the id field exists in the stream before this step
int idx = inputRowMeta.indexOfValue(environmentSubstitute(idInField));
if (idx < 0) throw new KettleStepException("Configured id field not in stream: " + idInField);

Try / catch

try { ... } catch (KettleStepException e) { if (e.getMessage().contains("InvalidIdField")) { logError("Check id field setting: " + meta.getIdInField()); } throw e; }

Prevention

When it happens

Trigger: meta.getIdInField() is non-blank but getFieldIdx cannot find that field name in data.inputRowMeta.

Common situations: Typo or renamed id field upstream; variable in the field name not substituted; the id field removed by a Select values step; mismatch between dialog configuration and actual stream fields.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/10181d6b169ff593. Report an issue: GitHub.

Appendix: source

Thrown at plugins/elasticsearch-bulk-insert/core/src/main/java/org/pentaho/di/trans/steps/elasticsearchbulk/ElasticSearchBulk.java:172

                    metaStore );
  }

  private void initFieldIndexes() throws KettleStepException {
    if ( isJsonInsert ) {
      Integer idx = getFieldIdx( data.inputRowMeta, environmentSubstitute( meta.getJsonField() ) );
      if ( idx != null ) {
        jsonFieldIdx = idx.intValue();
      } else {
        throw new KettleStepException( BaseMessages.getString( PKG, "ElasticSearchBulk.Error.NoJsonField" ) );
      }
    }

    idOutFieldName = environmentSubstitute( meta.getIdOutField() );

    if ( StringUtils.isNotBlank( meta.getIdInField() ) ) {
      idFieldIndex = getFieldIdx( data.inputRowMeta, environmentSubstitute( meta.getIdInField() ) );
      if ( idFieldIndex == null ) {
        throw new KettleStepException( BaseMessages.getString( PKG, "ElasticSearchBulk.Error.InvalidIdField" ) );
      }
    } else {
      idFieldIndex = null;
    }
  }

  private static Integer getFieldIdx( RowMetaInterface rowMeta, String fieldName ) {
    if ( fieldName == null ) {
      return null;
    }

    for ( int i = 0; i < rowMeta.size(); i++ ) {
      String name = rowMeta.getValueMeta( i ).getName();
      if ( fieldName.equals( name ) ) {
        return i;
      }
    }
    return null;

View on GitHub (pinned to f3058517a1)