apache/beam · error · RuntimeException

Failed to convert Firestore document to Beam Row

Error message

Failed to convert Firestore document to Beam Row: {documentName}

What it means

During the read, each Firestore document is converted to a Beam Row via FirestoreUtils.documentToRow. If conversion throws (schema mismatch, unexpected value type, missing/extra fields) and handleErrors is false, the exception is wrapped in a RuntimeException naming the failing document, failing the pipeline. If handleErrors is true, the error is counted and routed to an error output tag instead.

Solutions

Set handleErrors=true to route bad documents to the error PCollection instead of failing the pipeline
Align the JSON schema with the actual document shapes (correct types, make optional fields nullable/optional)
Clean or migrate non-conforming documents in the Firestore collection before running
Inspect the wrapped cause 'e' in the RuntimeException log to identify the exact field/conversion failure

Example fix

// before
.withHandleErrors(false) // pipeline dies on one bad document
// after
.withHandleErrors(true) // bad docs routed to ERROR_TAG output

Defensive patterns

Strategy: try-catch

Try / catch

try {
  Row row = FirestoreUtils.documentToRow(document, schema, documentIdField);
} catch (Exception e) {
  LOG.error("Document {} does not match schema: {}", document.getName(), e.getMessage());
  // count and route to error output, or enable withHandleErrors(true)
}

Prevention

Enable handleErrors(true) in production to dead-letter bad documents
Validate Firestore documents against the JSON schema before pipeline runs (or keep schema permissive with nullable fields)
Avoid unsupported Firestore value types (references, geo points) in schema-mapped collections
Add integration tests covering real document samples from the collection

When it happens

Trigger: A document's field types don't match the JSON-schema-derived Beam Schema (e.g. schema says string, document has number), the configured documentIdField conflicts, or a document contains types documentToRow can't map.

Common situations: Schema drift: Firestore collection documents edited manually or by older app versions no longer matching the declared JSON schema; nullable fields that are absent in some documents; unsupported Firestore value types (e.g. references, geo points) in the schema.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d5559a56097d89a8. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreReadSchemaTransformProvider.java:204

        Schema schema,
        @org.checkerframework.checker.nullness.qual.Nullable String documentIdField,
        boolean handleErrors,
        Schema errorSchema) {
      this.schema = schema;
      this.documentIdField = documentIdField;
      this.handleErrors = handleErrors;
      this.errorSchema = errorSchema;
    }

    @ProcessElement
    public void processElement(@Element Document document, MultiOutputReceiver receiver) {
      try {
        receiver
            .get(OUTPUT_TAG)
            .output(FirestoreUtils.documentToRow(document, schema, documentIdField));
      } catch (Exception e) {
        if (!handleErrors) {
          throw new RuntimeException(
              "Failed to convert Firestore document to Beam Row: " + document.getName(), e);
        }
        errorCounter.inc();
        receiver
            .get(ERROR_TAG)
            .output(
                ErrorHandling.errorRecord(
                    errorSchema, document.getName().getBytes(StandardCharsets.UTF_8), e));
      }
    }
  }
}

View on GitHub (pinned to 12126d8942)