pentaho/pentaho-kettle · error · KettleException

AvroInput.Error.UnexpectedMapValueTypeAtNonExpansionPoint

Error message

AvroInput.Error.UnexpectedMapValueTypeAtNonExpansionPoint

What it means

Thrown when, while resolving a map path, the value's schema type is a map but the reader is not yet at the map-expansion point (it is still extracting a specific key). Normally a non-primitive sub-structure is required here; reaching this state implies inconsistent schema/path assumptions.

Solutions

Align path definitions with each schema version actually in use (define per-schema paths)
Flatten the schema so the map sits at a consistent nesting depth across versions
Enable 'Ignore missing paths' so mismatched rows yield an empty row instead of failing
Avoid top-level union schemas; resolve to a concrete record schema before input

Example fix

// before: single path for both union branches
path: data.items.key
// after: branch-specific handling or enable 'Ignore missing paths'
ignoreMissing = true;

Defensive patterns

Strategy: validation

Validate before calling

// Verify consistent nesting depth across all schema versions in use
for (Schema v : versions) {
  Schema at = resolvePath(v, pathParts);
  if (at.getType() != Schema.Type.RECORD && at.getType() != Schema.Type.ARRAY) {
    throw new IllegalStateException("Path nesting mismatch in schema " + v.getFullName());
  }
}

Try / catch

try {
  return reader.convertMap(...);
} catch (KettleException e) {
  if (e.getMessage().contains("UnexpectedMapValueTypeAtNonExpansionPoint")) {
    return emptyRow(); // with 'Ignore missing paths' semantics
  }
  throw e;
}

Prevention

Keep map fields at the same nesting depth across schema versions
Avoid top-level union schemas; use a concrete record wrapper
Test paths against every schema version fed to the step
Enable 'Ignore missing paths' for heterogeneous inputs

When it happens

Trigger: Per-row schema version switching or a top-level union schema causes the resolved schema at the path position to be a map where the reader expected a record/array to continue traversing, and 'ignore missing paths' is disabled.

Common situations: A union schema whose branches differ in nesting depth (one branch nests the map deeper); switching schema versions per row where one version wraps the map in a record and the other does not.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/6af065ca8bf1d42f. Report an issue: GitHub.

Appendix: source

Thrown at plugins/avro-format/core/src/main/java/org/pentaho/di/trans/steps/avro/input/AvroNestedReader.java:1143

          }
        }

        // what have we got?
        if ( valueType.getType() == Schema.Type.RECORD ) {
          return convertToKettleValues( (GenericData.Record) value, valueType, defaultSchema, space, ignoreMissing );
        } else if ( valueType.getType() == Schema.Type.ARRAY ) {
          return convertToKettleValues( (GenericData.Array) value, valueType, defaultSchema, space, ignoreMissing );
        } else if ( valueType.getType() == Schema.Type.MAP ) {
          return convertToKettleValues( (Map<Utf8, Object>) value, valueType, defaultSchema, space, ignoreMissing );
        } else {
          // we shouldn't have a primitive at this point. If we are
          // extracting a particular key from the map then we're not to the
          // expansion phase,
          // so normally there must be a non-primitive sub-structure. Only if
          // the user is switching schema versions on a per-row basis or the
          // schema is a union at the top level could we end up here
          if ( !ignoreMissing ) {
            throw new KettleException( BaseMessages.getString( PKG,
              "AvroInput.Error.UnexpectedMapValueTypeAtNonExpansionPoint" ) );
          }
          Object[][] result = new Object[ 1 ][ m_outputRowMeta.size() + RowDataUtil.OVER_ALLOCATE_SIZE ];
          return result;
        }
      }
    }

    /**
     * Processes an array at this point in the path.
     *
     * @param array         the array to process
     * @param s             the current schema at this point in the path
     * @param space         environment variables
     * @param ignoreMissing true if null is to be returned for user fields that don't appear in the schema
     * @return an array of Kettle rows corresponding to the expanded map/array and containing all leaf values as defined
     * in the paths
     * @throws KettleException if a problem occurs

View on GitHub (pinned to f3058517a1)