apache/beam · warning

Error while parsing input element

Error message

Error while parsing input element

What it means

FileWriteSchemaTransformFormatProviders' DoFn catches any exception thrown while applying the map function (e.g. a CsvI/O or Json format provider converting a Row to a record) during processElement. It increments the bundle error counter, logs this warning, and routes the failing element plus the exception message to the ERROR_TAG output instead of failing the pipeline. The error means the input Row could not be written/serialized in the requested format.

Source

Thrown at sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformFormatProviders.java:102

    private SerializableFunction<Row, OutputT> mapFn;
    private Counter errorCounter;
    private TupleTag<OutputT> outputTag;
    private long errorsInBundle = 0L;

    public BeamRowMapperWithDlq(
        String name, SerializableFunction<Row, OutputT> mapFn, TupleTag<OutputT> outputTag) {
      errorCounter = Metrics.counter(FileWriteSchemaTransformFormatProvider.class, name);
      this.mapFn = mapFn;
      this.outputTag = outputTag;
    }

    @ProcessElement
    public void process(@DoFn.Element Row row, MultiOutputReceiver receiver) {
      try {
        receiver.get(outputTag).output(mapFn.apply(row));
      } catch (Exception e) {
        errorsInBundle += 1;
        LOG.warn("Error while parsing input element", e);
        receiver
            .get(ERROR_TAG)
            .output(
                Row.withSchema(ERROR_SCHEMA)
                    .addValues(e.toString(), row.toString().getBytes(StandardCharsets.UTF_8))
                    .build());
      }
    }

    @FinishBundle
    public void finish() {
      errorCounter.inc(errorsInBundle);
      errorsInBundle = 0L;
    }
  }

  /**
   * Applies common parameters from {@link FileWriteSchemaTransformConfiguration} to {@link

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the ERROR_TAG PCollection contents; it contains e.toString() and the offending serialized row.
2. Align the read schema with the write format (e.g. make nullable fields compatible with the provider).
3. Choose a different format provider or configure its options (delimiter, quoting) to fit the data.
4. Fix upstream data so fields conform, or filter/transform invalid rows before writing.

Example fix

// before: rows with nulls written via CSV provider fail per-element
FileIO.write().via(new SchemaTransformWrite(...)) // errors routed to ERROR_TAG
// after: sanitize rows before writing
row -> Row.withSchema(schema).addValues(Optional.ofNullable(row.getValue("f")).orElse("")).build()

// pre-check row compatibility with the target format provider
boolean writable = row.getSchema().getFields().stream().allMatch(f -> row.getValue(f.getName()) != null || f.getType().getNullable());

// consume the ERROR_TAG output instead of only the main output
PCollection<Row> errors = result.get(ERROR_TAG);
errors.apply("LogBadRows", ParDo.of(new LogFn<>())).setCoder(ErrorRowCoder.of());

When it happens

Trigger: mapFn.apply(row) throws for a given Row — e.g. malformed field values for the chosen CSV/JSON format provider, null in a non-nullable position, or charset/quote escaping failures.

Common situations: CSV provider given rows containing field values with unescaped delimiters; JSON provider receiving incompatible types; rows produced by FileReadSchemaTransform that don't match the writer's expected schema.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Unable to generate coder for schema {schema}
• Cannot provide SerializableCoder because {} does not impleme
• Java Serialization may be non-deterministic.
• cannot encode a null String
• cannot encode a null Integer

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/681e45565e13dab8. Report an issue: GitHub.