{"record":{"id":"d8a696a28aec7c20","repo":"apache/beam","slug":"row-schema-does-not-contain-the-following-specified-fields","errorCode":null,"errorMessage":"\nRow Schema does not contain the following specified fields: {notFound}\nThe following specified fields are not of type Row. Their nested fields could not be reached: {notRowField}","messagePattern":"\nRow Schema does not contain the following specified fields: (.+?)\nThe following specified fields are not of type Row\\. Their nested fields could not be reached: (.+?)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowFilter.java","lineNumber":292,"sourceCode":"            notRowField.add(currentFieldName);\n            break;\n          }\n          currentSchema = Preconditions.checkNotNull(nextField.getType().getRowSchema());\n        }\n      }\n    }\n\n    if (!notFound.isEmpty() || !notRowField.isEmpty()) {\n      String message = \"Validation failed for '\" + operation + \"'.\";\n      if (!notFound.isEmpty()) {\n        message += \"\\nRow Schema does not contain the following specified fields: \" + notFound;\n      }\n      if (!notRowField.isEmpty()) {\n        message +=\n            \"\\nThe following specified fields are not of type Row. Their nested fields could not be reached: \"\n                + notRowField;\n      }\n      throw new IllegalArgumentException(message);\n    }\n  }\n\n  /**\n   * Creates a field tree, separating each top-level field from its (potential) nested fields. E.g.\n   * [\"foo.bar.baz\", \"foo.abc\", \"xyz\"] --> {\"foo\": [\"bar.baz\", \"abc\"], \"xyz\": []}\n   */\n  @VisibleForTesting\n  static Map<String, List<String>> getFieldTree(List<String> fields) {\n    Map<String, List<String>> fieldTree = Maps.newHashMap();\n\n    for (String field : fields) {\n      List<String> components = Splitter.on(\".\").splitToList(field);\n      String root = components.get(0);\n      fieldTree.computeIfAbsent(root, r -> new ArrayList<>());\n\n      if (components.size() > 1) {\n        String nestedFields = String.join(\".\", components.subList(1, components.size()));","sourceCodeStart":274,"sourceCodeEnd":310,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowFilter.java#L274-L310","documentation":"RowFilter.validateSchemaContainsFields throws IllegalArgumentException when the row Schema does not contain the requested fields, or when a nested path's parent field is not of type Row so its nested fields cannot be reached. The message lists two sets: missing fields (notFound) and non-Row parents (notRowField). keep(), drop(), and only() call this before building the filtered schema.","triggerScenarios":"Calling rowFilter.keep/drop/only with field names (including dotted nested paths like \"foo.bar.baz\") that don't exist in the schema, or where a path segment resolves to a non-Row field (e.g. \"intField.sub\").","commonSituations":"Schema drift: upstream pipeline renamed/removed fields; typos in field names; assuming a field is a nested Row when it's a primitive; PCollection schema inferred differently than expected.","solutions":["Compare the message's notFound/notRowField lists against the actual schema and correct the field names.","Ensure intermediate path segments of nested fields are of type ROW.","Print/verify the schema (pcollection.getSchema()) before filtering.","Update code after upstream schema changes and version the schema contract."],"exampleFix":"// before\nrowFilter.keep(\"user.address.city\", \"zip\"); // zip not in schema\n// after\nrowFilter.keep(\"user.address.city\", \"postalCode\");","handlingStrategy":"validation","validationCode":"Schema schema = rows.getSchema();\nSet<String> names = schema.getFieldNames();\nfor (String f : requestedFields) {\n  String top = f.split(\"\\\\.\")[0];\n  if (!names.contains(top)) {\n    throw new IllegalArgumentException(\"Field not in schema: \" + f);\n  }\n  Field field = schema.getField(top);\n  if (f.contains(\".\") && field.getType().getTypeName() != Schema.TypeName.ROW) {\n    throw new IllegalArgumentException(\"Parent of nested field is not ROW: \" + top);\n  }\n}\nrowFilter.keep(requestedFields.toArray(new String[0]));","typeGuard":null,"tryCatchPattern":"try {\n  PCollection<Row> out = rowFilter.keep(fields);\n} catch (IllegalArgumentException e) {\n  LOG.error(\"Field validation failed: {}\", e.getMessage());\n  throw e;\n}","preventionTips":["Verify field names against pcollection.getSchema() before filtering.","Check that every intermediate segment of a nested path is type ROW.","Pin schema contracts with tests to catch upstream drift.","Beware typo'd field names — the notFound list is authoritative."],"tags":["java","apache-beam","schema","validation"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}