apache/beam · error · RuntimeException

e

Error message

e

What it means

The Firestore write DoFn (FirestoreWriteSchemaTransformProvider) wraps any exception thrown while converting a Beam Row to a Firestore Document (or issuing the write builder) in a RuntimeException and rethrows it, unless error handling ('handleErrors') is enabled. The original cause is preserved as the RuntimeException's cause. This generic wrapper is the control-flow path for any row-to-document failure (bad schema, invalid document id, conversion errors).

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreWriteSchemaTransformProvider.java:194

      this.schema = schema;
      this.projectId = projectId;
      this.databaseId = databaseId;
      this.collectionId = collectionId;
      this.documentIdField = documentIdField;
      this.handleErrors = handleErrors;
      this.errorSchema = errorSchema;
    }

    @ProcessElement
    public void processElement(@Element Row row, MultiOutputReceiver receiver) {
      try {
        Document document =
            FirestoreUtils.rowToDocument(
                row, schema, projectId, databaseId, collectionId, documentIdField);
        receiver.get(OUTPUT_TAG).output(Write.newBuilder().setUpdate(document).build());
      } catch (Exception e) {
        if (!handleErrors) {
          throw new RuntimeException(e);
        }
        errorCounter.inc();
        receiver.get(ERROR_TAG).output(ErrorHandling.errorRecord(errorSchema, row, e));
      }
    }
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the cause of the RuntimeException (e.getCause()) to find the real conversion failure
2. Verify the incoming Row schema exactly matches the schema configured for the Firestore write transform
3. If documentIdField is set, confirm every row has a non-null, non-empty value in that field
4. Enable handleErrors=true to route failing rows to the error output tag instead of failing the pipeline

Example fix

// before
PCollection<Row> written = rows.apply(FirestoreIO.v1().write().bulkWrite()
    .withSchema(rowSchema).withCollectionId("c").buildExpandable());
// after (route bad rows to error output instead of failing)
PCollection<Row> written = rows.apply(FirestoreIO.v1().write().bulkWrite()
    .withSchema(rowSchema).withCollectionId("c")
    .withErrorHandling(true).buildExpandable());

if (row == null || row.getSchema() == null || !row.getSchema().equals(expectedSchema)) {
  throw new IllegalArgumentException("Row schema does not match configured Firestore write schema");
}
if (documentIdField != null && (row.getValue(documentIdField) == null
    || row.getValue(documentIdField).toString().isEmpty())) {
  throw new IllegalArgumentException("documentIdField value missing in row");
}

try {
  Document doc = FirestoreUtils.rowToDocument(row, schema, projectId, databaseId, collectionId, documentIdField);
} catch (RuntimeException e) {
  // e is a wrapper; inspect root cause
  Throwable cause = e.getCause();
  LOG.error("Firestore row conversion failed for row: {}", row, cause);
}

When it happens

Trigger: processElement calls FirestoreUtils.rowToDocument(row, schema, projectId, databaseId, collectionId, documentIdField) and any exception thrown there (or by Write.newBuilder().setUpdate(...)) when handleErrors is false.

Common situations: Input row does not match the configured schema, documentIdField references a missing/null field, or a field value cannot be mapped to a Firestore type. Common in SchemaTransform pipeline configs where the row schema drifted from the FirestoreIO configuration.

Related errors

• Encountered Bad Record:
• Invalid Firestore document name: {documentName}
• Document id field '{documentIdField}' must be set on input r
• Unsupported Firestore value type: {valueTypeCase}
• Collection element type cannot be null.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d7fa70834d8412bb. Report an issue: GitHub.