apache/beam · error · IllegalArgumentException

Invalid field id for schema

Error message

Invalid field id <id> for schema <schema>

What it means

FieldAccessDescriptor.validateFieldDescriptor checks that any field selected by numeric id is within bounds of the target schema (0 <= id < schema.getFieldCount()). This IllegalArgumentException means a FieldDescriptor referenced a field id that doesn't exist in the schema being resolved against.

Solutions

1. Validate fieldId against schema.getFieldCount() before resolving; clamp or remap stale ids.
2. Prefer withFieldNames (by name) over byId so lookups survive schema evolution.
3. Regenerate descriptors whenever the source Schema changes.
4. Log the schema (included in the message) and align the descriptor to it.

Example fix

// before
FieldAccessDescriptor.withFieldIds(5).resolve(3FieldSchema)

// after
FieldAccessDescriptor.withFieldNames("expectedField").resolve(schema);

if (fieldId < 0 || fieldId >= schema.getFieldCount()) throw new IllegalArgumentException("fieldId out of range: " + fieldId);

static boolean validFieldId(Schema schema, int id) { return id >= 0 && id < schema.getFieldCount(); }

try { accessDescriptor.resolve(schema); }
catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("Invalid field id")) { /* re-derive descriptor from current schema */ } throw e;
}

When it happens

Trigger: Resolving a FieldAccessDescriptor built with withFieldId/FieldDescriptor.byId against a schema with fewer fields — e.g. id 5 on a 3-field schema; typically after schema version drift or resolving one schema's descriptor against a different schema.

Common situations: Caching FieldAccessDescriptor instances across schema versions; programmatic field-id lookups computed off a stale Schema; DoFns selecting fields from an upstream schema that changed shape.

Related errors

• FieldType must be either a row or a container containing…
• Cannot automatically determine a field type from a Row…
• Couldn't find field type for
• Illegal shard number
• Index is out of bounds for obj.__defaults__ in path

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/664a77c46946a0d4. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldAccessDescriptor.java:625

    if (TypeName.ROW.equals(type.getTypeName())) {
      return type.getRowSchema();
    } else if (type.getTypeName().isCollectionType()) {
      return getFieldSchema(type.getCollectionElementType());
    } else if (TypeName.MAP.equals(type.getTypeName())) {
      return getFieldSchema(type.getMapValueType());
    } else if (TypeName.LOGICAL_TYPE.equals(type.getTypeName())) {
      return getFieldSchema(type.getLogicalType().getBaseType());
    } else {
      throw new IllegalArgumentException(
          "FieldType " + type + " must be either a row or a container containing rows");
    }
  }

  private static void validateFieldDescriptor(Schema schema, FieldDescriptor fieldDescriptor) {
    Integer fieldId = fieldDescriptor.getFieldId();
    if (fieldId != null) {
      if (fieldId < 0 || fieldId >= schema.getFieldCount()) {
        throw new IllegalArgumentException("Invalid field id " + fieldId + " for schema " + schema);
      }
    }
    // If qualifiers were specified, validate them.
    // For example, if a selector was a[*][*], then a needs to be a List of a List.
    Field field =
        (fieldId != null)
            ? schema.getField(fieldId)
            : schema.getField(fieldDescriptor.getFieldName());
    FieldType fieldType = field.getType();
    for (Qualifier qualifier : fieldDescriptor.getQualifiers()) {
      switch (qualifier.getKind()) {
        case LIST:
          checkArgument(qualifier.getList().equals(ListQualifier.ALL));
          checkArgument(fieldType.getTypeName().isCollectionType());
          fieldType = fieldType.getCollectionElementType();
          break;
        case MAP:
          checkArgument(qualifier.getMap().equals(MapQualifier.ALL));

View on GitHub (pinned to 12126d8942)