What causes "Error while parsing the element" in apache/beam?

Inspect the ErrorHandling error records on the error output PCollection; log/fix the offending records.

Error while parsing the element

Error message

Error while parsing the element

What it means

TFRecordReadSchemaTransformProvider's DoFn parses each TFRecord byte[] into a Row via a configured valueMapper. If parsing throws and error handling is enabled (handleErrors), the element is logged at WARN and routed to the ERROR_TAG output as an ErrorHandling record instead of failing the pipeline; if error handling is disabled, the exception is rethrown as a RuntimeException and fails the bundle.

Solutions

Inspect the ErrorHandling error records on the error output PCollection; log/fix the offending records.
Verify the valueMapper/parser matches the actual record encoding in the TFRecord files.
If handleErrors was disabled, enable error handling to divert bad records instead of failing the pipeline.
Regenerate or re-export corrupted source files.

Example fix

// before (no error output configured)
.write() // throws on bad record
// after
.withErrorHandling()  // routes parse failures to the error output instead of crashing

Defensive patterns

Strategy: try-catch

Validate before calling

// Sanity-check the record parses before running the pipeline on a sample
byte[] sample = readFirstRecord(path);
assertDoesNotThrow(() -> valueMapper.apply(sample));

Try / catch

try {
  Row row = valueMapper.apply(bytes);
} catch (Exception e) {
  // mirror the transform: route to dead-letter instead of failing
  errorOutput.emit(ErrorHandling.errorRecord(errorSchema, bytes, e));
}

Prevention

Always configure error handling for reads over potentially corrupt external data.
Validate a sample of the source files against the parser schema before launching.
Monitor the error-output PCollection size and alert on spikes.
Pin schema/proto versions to avoid mismatched parsers.

When it happens

Trigger: Reading TFRecord files with a valueMapper (e.g. protobuf/json parser) that throws on a malformed or incompatible record while the transform is configured with an error output (handleErrors true) — or throwing a RuntimeException when handleErrors is false.

Common situations: Corrupt or truncated TFRecord files; schema/proto mismatch after a schema change; wrong parser configured for the record format.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/70e79df850c382c5. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordReadSchemaTransformProvider.java:187

        Schema errorSchema,
        boolean handleErrors) {
      this.errorCounter = Metrics.counter(TFRecordReadSchemaTransformProvider.class, name);
      this.valueMapper = valueMapper;
      this.handleErrors = handleErrors;
      this.errorSchema = errorSchema;
    }

    @ProcessElement
    public void process(@DoFn.Element byte[] msg, MultiOutputReceiver receiver) {
      Row mappedRow = null;
      try {
        mappedRow = valueMapper.apply(msg);
      } catch (Exception e) {
        if (!handleErrors) {
          throw new RuntimeException(e);
        }
        errorsInBundle += 1;
        LOG.warn("Error while parsing the element", e);
        receiver.get(ERROR_TAG).output(ErrorHandling.errorRecord(errorSchema, msg, e));
      }
      if (mappedRow != null) {
        receiver.get(OUTPUT_TAG).output(mappedRow);
      }
    }

    @FinishBundle
    public void finish(FinishBundleContext c) {
      errorCounter.inc(errorsInBundle);
      errorsInBundle = 0L;
    }
  }
}

View on GitHub (pinned to 12126d8942)