apache/beam · error · IllegalArgumentException

Row Schema does not contain the following specified fields:

Error message

Row Schema does not contain the following specified fields: {notFound}
The following specified fields are not of type Row. Their nested fields could not be reached: {notRowField}

What it means

RowFilter.validateSchemaContainsFields throws IllegalArgumentException when the row Schema does not contain the requested fields, or when a nested path's parent field is not of type Row so its nested fields cannot be reached. The message lists two sets: missing fields (notFound) and non-Row parents (notRowField). keep(), drop(), and only() call this before building the filtered schema.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowFilter.java:292

            notRowField.add(currentFieldName);
            break;
          }
          currentSchema = Preconditions.checkNotNull(nextField.getType().getRowSchema());
        }
      }
    }

    if (!notFound.isEmpty() || !notRowField.isEmpty()) {
      String message = "Validation failed for '" + operation + "'.";
      if (!notFound.isEmpty()) {
        message += "\nRow Schema does not contain the following specified fields: " + notFound;
      }
      if (!notRowField.isEmpty()) {
        message +=
            "\nThe following specified fields are not of type Row. Their nested fields could not be reached: "
                + notRowField;
      }
      throw new IllegalArgumentException(message);
    }
  }

  /**
   * Creates a field tree, separating each top-level field from its (potential) nested fields. E.g.
   * ["foo.bar.baz", "foo.abc", "xyz"] --> {"foo": ["bar.baz", "abc"], "xyz": []}
   */
  @VisibleForTesting
  static Map<String, List<String>> getFieldTree(List<String> fields) {
    Map<String, List<String>> fieldTree = Maps.newHashMap();

    for (String field : fields) {
      List<String> components = Splitter.on(".").splitToList(field);
      String root = components.get(0);
      fieldTree.computeIfAbsent(root, r -> new ArrayList<>());

      if (components.size() > 1) {
        String nestedFields = String.join(".", components.subList(1, components.size()));

View on GitHub (pinned to 12126d8942)

Solutions

1. Compare the message's notFound/notRowField lists against the actual schema and correct the field names.
2. Ensure intermediate path segments of nested fields are of type ROW.
3. Print/verify the schema (pcollection.getSchema()) before filtering.
4. Update code after upstream schema changes and version the schema contract.

Example fix

// before
rowFilter.keep("user.address.city", "zip"); // zip not in schema
// after
rowFilter.keep("user.address.city", "postalCode");

Schema schema = rows.getSchema();
Set<String> names = schema.getFieldNames();
for (String f : requestedFields) {
  String top = f.split("\\.")[0];
  if (!names.contains(top)) {
    throw new IllegalArgumentException("Field not in schema: " + f);
  }
  Field field = schema.getField(top);
  if (f.contains(".") && field.getType().getTypeName() != Schema.TypeName.ROW) {
    throw new IllegalArgumentException("Parent of nested field is not ROW: " + top);
  }
}
rowFilter.keep(requestedFields.toArray(new String[0]));

try {
  PCollection<Row> out = rowFilter.keep(fields);
} catch (IllegalArgumentException e) {
  LOG.error("Field validation failed: {}", e.getMessage());
  throw e;
}

When it happens

Trigger: Calling rowFilter.keep/drop/only with field names (including dotted nested paths like "foo.bar.baz") that don't exist in the schema, or where a path segment resolves to a non-Row field (e.g. "intField.sub").

Common situations: Schema drift: upstream pipeline renamed/removed fields; typos in field names; assuming a field is a nested Row when it's a primitive; PCollection schema inferred differently than expected.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Row expected <fieldCount> fields (<fields>). initialized wit
• Schema can't be empty
• Cannot provide a coder for a Beam Row. Please provide a sche
• Missing required value for group [<group>]. At least one of
• --maxCacheMemoryUsagePercent must be between 0 and 100.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d8a696a28aec7c20. Report an issue: GitHub.