{"record":{"id":"ea97208edcc74aa1","repo":"apache/beam","slug":"error-while-processing-the-element","errorCode":null,"errorMessage":"Error while processing the element","messagePattern":"Error while processing the element","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordWriteSchemaTransformProvider.java","lineNumber":233,"sourceCode":"        Schema errorSchema,\n        boolean handleErrors) {\n      this.toBytesFn = toBytesFn;\n      this.errorCounter = Metrics.counter(TFRecordReadSchemaTransformProvider.class, name);\n      this.handleErrors = handleErrors;\n      this.errorSchema = errorSchema;\n    }\n\n    @ProcessElement\n    public void process(@DoFn.Element Row row, MultiOutputReceiver receiver) {\n      byte[] output = null;\n      try {\n        output = toBytesFn.apply(row);\n      } catch (Exception e) {\n        if (!handleErrors) {\n          throw new RuntimeException(e);\n        }\n        errorsInBundle += 1;\n        LOG.warn(\"Error while processing the element\", e);\n        receiver.get(ERROR_TAG).output(ErrorHandling.errorRecord(errorSchema, row, e));\n      }\n      if (output != null) {\n        receiver.get(OUTPUT_TAG).output(output);\n      }\n    }\n\n    @FinishBundle\n    public void finish() {\n      errorCounter.inc(errorsInBundle);\n      errorsInBundle = 0L;\n    }\n  }\n}\n","sourceCodeStart":215,"sourceCodeEnd":248,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordWriteSchemaTransformProvider.java#L215-L248","documentation":"TFRecordWriteSchemaTransformProvider's DoFn converts each input Row to bytes via toBytesFn before writing. If the conversion throws and error handling is enabled, the failure is logged at WARN and the element is emitted to the ERROR_TAG as an ErrorHandling record; if error handling is disabled, the exception becomes a RuntimeException that fails the pipeline.","triggerScenarios":"Writing Rows to TFRecord files when toBytesFn throws on an element (null required field, incompatible type, serialization failure) — logged and routed to error output when handleErrors is true, rethrown as RuntimeException otherwise.","commonSituations":"Row schema not matching the expected serialization format (e.g. proto mismatch); nulls in non-nullable proto fields; rows produced upstream with the wrong schema.","solutions":["Inspect records emitted on the error output PCollection to identify the serialization failure.","Validate/normalize the Row schema before writing so it matches the toBytesFn expectation.","Enable error handling (withErrorHandling) to divert bad rows instead of failing the bundle.","Fix the upstream transform producing non-conforming Rows."],"exampleFix":"// before (no error output configured)\n.write() // throws on serialization failure\n// after\n.withErrorHandling()  // routes serialization failures to the error output","handlingStrategy":"try-catch","validationCode":"// Verify a sample Row serializes before writing\nbyte[] bytes = toBytesFn.apply(sampleRow);\nif (bytes == null || bytes.length == 0) {\n  throw new IllegalStateException(\"toBytesFn produced empty output for sample row\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  byte[] out = toBytesFn.apply(row);\n} catch (Exception e) {\n  errorOutput.emit(ErrorHandling.errorRecord(errorSchema, row, e));\n}","preventionTips":["Validate the Row schema matches the serialization target before the write transform.","Reject or null-check fields that cannot be encoded (e.g. nulls in required proto fields) upstream.","Enable error handling when writing user-derived data.","Test toBytesFn against representative rows in unit tests."],"tags":["io","tfrecord","serialization","beam","error-handling"],"backgroundTag":"record-serialization-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}