{"record":{"id":"d5559a56097d89a8","repo":"apache/beam","slug":"failed-to-convert-firestore-document-to-beam-row","errorCode":null,"errorMessage":"Failed to convert Firestore document to Beam Row: {documentName}","messagePattern":"Failed to convert Firestore document to Beam Row: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreReadSchemaTransformProvider.java","lineNumber":204,"sourceCode":"        Schema schema,\n        @org.checkerframework.checker.nullness.qual.Nullable String documentIdField,\n        boolean handleErrors,\n        Schema errorSchema) {\n      this.schema = schema;\n      this.documentIdField = documentIdField;\n      this.handleErrors = handleErrors;\n      this.errorSchema = errorSchema;\n    }\n\n    @ProcessElement\n    public void processElement(@Element Document document, MultiOutputReceiver receiver) {\n      try {\n        receiver\n            .get(OUTPUT_TAG)\n            .output(FirestoreUtils.documentToRow(document, schema, documentIdField));\n      } catch (Exception e) {\n        if (!handleErrors) {\n          throw new RuntimeException(\n              \"Failed to convert Firestore document to Beam Row: \" + document.getName(), e);\n        }\n        errorCounter.inc();\n        receiver\n            .get(ERROR_TAG)\n            .output(\n                ErrorHandling.errorRecord(\n                    errorSchema, document.getName().getBytes(StandardCharsets.UTF_8), e));\n      }\n    }\n  }\n}\n","sourceCodeStart":186,"sourceCodeEnd":217,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreReadSchemaTransformProvider.java#L186-L217","documentation":"During the read, each Firestore document is converted to a Beam Row via FirestoreUtils.documentToRow. If conversion throws (schema mismatch, unexpected value type, missing/extra fields) and handleErrors is false, the exception is wrapped in a RuntimeException naming the failing document, failing the pipeline. If handleErrors is true, the error is counted and routed to an error output tag instead.","triggerScenarios":"A document's field types don't match the JSON-schema-derived Beam Schema (e.g. schema says string, document has number), the configured documentIdField conflicts, or a document contains types documentToRow can't map.","commonSituations":"Schema drift: Firestore collection documents edited manually or by older app versions no longer matching the declared JSON schema; nullable fields that are absent in some documents; unsupported Firestore value types (e.g. references, geo points) in the schema.","solutions":["Set handleErrors=true to route bad documents to the error PCollection instead of failing the pipeline","Align the JSON schema with the actual document shapes (correct types, make optional fields nullable/optional)","Clean or migrate non-conforming documents in the Firestore collection before running","Inspect the wrapped cause 'e' in the RuntimeException log to identify the exact field/conversion failure"],"exampleFix":"// before\n.withHandleErrors(false) // pipeline dies on one bad document\n// after\n.withHandleErrors(true) // bad docs routed to ERROR_TAG output","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  Row row = FirestoreUtils.documentToRow(document, schema, documentIdField);\n} catch (Exception e) {\n  LOG.error(\"Document {} does not match schema: {}\", document.getName(), e.getMessage());\n  // count and route to error output, or enable withHandleErrors(true)\n}","preventionTips":["Enable handleErrors(true) in production to dead-letter bad documents","Validate Firestore documents against the JSON schema before pipeline runs (or keep schema permissive with nullable fields)","Avoid unsupported Firestore value types (references, geo points) in schema-mapped collections","Add integration tests covering real document samples from the collection"],"tags":["java","google-cloud-firestore","schema","data-quality"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}