apache/beam · error · IllegalArgumentException

Input schema must contain document id field

Error message

Input schema must contain document id field: {documentIdField}

What it means

When expanding the Firestore write SchemaTransform, the input schema must contain the field used as the Firestore document id (defaults to a configured documentIdField, or a default like '__id__'). The configured field is absent from the input PCollection's schema, so the transform fails fast with this IllegalArgumentException during pipeline construction.

Solutions

Add the configured document id field to the input schema, or change documentIdField to an existing field name
Check exact case and spelling of the field name — schema field lookup is exact
Generate the id in a prior step (e.g. withConstantFields or a MapElements adding an id column)
If relying on the default, either name your id column the default name or explicitly set documentIdField

Example fix

// before
.write().to("books").withDocumentIdField("bookId") // schema has only 'id'
// after
.write().to("books").withDocumentIdField("id") // or rename the column to bookId

Defensive patterns

Strategy: validation

Validate before calling

if (!input.getSchema().hasField(documentIdField)) { throw new IllegalArgumentException("schema missing id field: " + documentIdField); }

Try / catch

try { write.expand(input); } catch (IllegalArgumentException e) { if (e.getMessage().contains("Input schema must contain document id field")) { // fix schema or configuration then rebuild } throw e; }

Prevention

Validate that documentIdField names an existing column before expanding the transform
Keep id-column naming consistent between producers and Firestore config
Print input.getSchema() when debugging schema-transform mismatches

When it happens

Trigger: FirestoreIO.write()/FirestoreWriteSchemaTransformProvider configured with documentIdField="X" where the incoming table's Schema has no field named X; or using the default document id field while the input schema lacks it.

Common situations: Renaming the id column upstream without updating configuration; feeding a schema-transform table whose column names differ (case sensitivity); using the default id field name with custom input schemas.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b9157ad4cfe991fc. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreWriteSchemaTransformProvider.java:102

    private final FirestoreWriteSchemaTransformConfiguration configuration;

    FirestoreWriteSchemaTransform(FirestoreWriteSchemaTransformConfiguration configuration) {
      configuration.validate();
      this.configuration = configuration;
    }

    @Override
    public PCollectionRowTuple expand(PCollectionRowTuple input) {
      PCollection<Row> rows = input.get(INPUT_TAG);
      Schema inputSchema = rows.getSchema();
      String projectId = resolveProjectId(input.getPipeline());
      String databaseId = resolveDatabaseId(input.getPipeline());
      String documentIdField =
          Strings.isNullOrEmpty(configuration.getDocumentIdField())
              ? DEFAULT_DOCUMENT_ID_FIELD
              : configuration.getDocumentIdField();
      if (!inputSchema.hasField(documentIdField)) {
        throw new IllegalArgumentException(
            "Input schema must contain document id field: " + documentIdField);
      }

      boolean handleErrors = ErrorHandling.hasOutput(configuration.getErrorHandling());
      Schema errorSchema = ErrorHandling.errorSchema(inputSchema);

      PCollectionTuple outputTuple =
          rows.apply(
              "ConvertToFirestoreWrite",
              ParDo.of(
                      new RowToWriteFn(
                          inputSchema,
                          projectId,
                          databaseId,
                          configuration.getCollectionId(),
                          documentIdField,
                          handleErrors,
                          errorSchema))

View on GitHub (pinned to 12126d8942)