pentaho/pentaho-kettle · error · KettleException

AvroInput.Error.EncounteredAPrimitivePriorToMapExpansion

Error message

AvroInput.Error.EncounteredAPrimitivePriorToMapExpansion

What it means

Thrown by AvroNestedReader while walking an Avro map path: the schema resolved to a primitive type where a map was expected before the map-expansion phase could run. The reader cannot extract a keyed value from a primitive, so it aborts the row unless 'ignore missing paths' is enabled, in which case it returns an empty row.

Solutions

  1. Correct the field path so it does not traverse into a map at a schema location that is a primitive
  2. Verify the Avro schema used by the step matches the data (re-read the schema file / schema registry entry)
  3. Enable 'Ignore missing paths' in the step if a blank row is acceptable for such records
  4. Pin to a single schema version instead of per-row schema switching

Example fix

// before (path assumes map but schema has primitive)
path: orders.total
// after
path: orders            // read scalar directly, no map key traversal
// or fix the schema so 'orders' is map<string,int>
Defensive patterns

Strategy: validation

Validate before calling

// Before configuring the path, check the schema branch at the path position
Schema fieldSchema = schema.getField("orders").schema();
boolean isMap = fieldSchema.getType() == Schema.Type.MAP
    || (fieldSchema.getType() == Schema.Type.UNION
        && fieldSchema.getTypes().stream().anyMatch(t -> t.getType() == Schema.Type.MAP));
if (!isMap) throw new IllegalArgumentException("orders is not a map in this schema");

Type guard

boolean isMapSchema(Schema s) {
  if (s.getType() == Schema.Type.UNION) {
    return s.getTypes().stream().anyMatch(t -> t.getType() == Schema.Type.MAP);
  }
  return s.getType() == Schema.Type.MAP;
}

Try / catch

try {
  Object[][] rows = reader.convertMap(...);
} catch (KettleException e) {
  if (e.getMessage().contains("EncounteredAPrimitivePriorToMapExpansion")) {
    logMinimal("Path targets a primitive; skipping row");
    // return empty row / route to error handling
  } else throw e;
}

Prevention

When it happens

Trigger: A field path in the Avro Input step targets a map (e.g. 'mymap.key') but the actual Avro schema at that point in the (possibly union-resolved) schema is a primitive (string, int, etc.), often because the schema changed between rows or the union branch selected is a primitive.

Common situations: Schema evolution: producer upgraded a field from a map<string,int> to a plain int; per-row schema switching where one version has a map and another a scalar; copy-pasted path definitions from an older schema.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/0f8689a094b0ac18. Report an issue: GitHub.

Appendix: source

Thrown at plugins/avro-format/core/src/main/java/org/pentaho/di/trans/steps/avro/input/AvroNestedReader.java:1119

            // either have a map or primitive here
            if ( value instanceof Map ) {
              // now have to look for the schema of the map
              Schema mapSchema = null;
              for ( Schema ts : valueType.getTypes() ) {
                if ( ts.getType() == Schema.Type.MAP ) {
                  mapSchema = ts;
                  break;
                }
              }
              if ( mapSchema == null ) {
                throw new KettleException( BaseMessages.getString( PKG,
                  "AvroInput.Error.UnableToFindSchemaForUnionMap" ) );
              }
              valueType = mapSchema;
            } else {
              // We shouldn't have a primitive here
              if ( !ignoreMissing ) {
                throw new KettleException( BaseMessages.getString( PKG,
                  "AvroInput.Error.EncounteredAPrimitivePriorToMapExpansion" ) );
              }
              Object[][] result = new Object[ 1 ][ m_outputRowMeta.size() + RowDataUtil.OVER_ALLOCATE_SIZE ];
              return result;
            }
          }
        }

        // what have we got?
        if ( valueType.getType() == Schema.Type.RECORD ) {
          return convertToKettleValues( (GenericData.Record) value, valueType, defaultSchema, space, ignoreMissing );
        } else if ( valueType.getType() == Schema.Type.ARRAY ) {
          return convertToKettleValues( (GenericData.Array) value, valueType, defaultSchema, space, ignoreMissing );
        } else if ( valueType.getType() == Schema.Type.MAP ) {
          return convertToKettleValues( (Map<Utf8, Object>) value, valueType, defaultSchema, space, ignoreMissing );
        } else {
          // we shouldn't have a primitive at this point. If we are
          // extracting a particular key from the map then we're not to the

View on GitHub (pinned to f3058517a1)