apache/beam · error · RuntimeException

Error parsing JSON

Error message

Error parsing JSON

What it means

During a 'json'-format text table read, each line is converted with jsonToRow. If a line cannot be mapped to the table schema (UnsupportedRowJsonException) and no dead-letter file is configured, the DoFn wraps the cause in a RuntimeException("Error parsing JSON") which fails the pipeline. It exists to surface malformed records instead of silently dropping them.

Solutions

1. Configure a dead-letter file (deadLetterFile) so bad records are routed to DLF_TAG instead of failing the pipeline.
2. Fix the offending JSON records so they conform to the declared schema.
3. Align the table schema with the actual JSON structure (types, required fields).
4. Pre-clean/validate input files before running the pipeline.

Example fix

// before: no dead letter -> pipeline fails on first bad record
// after: route bad records to a dead-letter output instead of throwing
TextTableProvider.JsonToRowFn fn =
    new TextTableProvider.JsonToRowFn(schema, "gs://bucket/dlq/errors");

// validate each line parses before pipeline run (offline precheck)
new ObjectMapper().readValue(line, Map.class);

try {
  pipelineResult.waitUntilFinish();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Error parsing JSON")) {
    // inspect dead-letter / input files for malformed records
  }
}

When it happens

Trigger: Reading a text table with format 'json' where a line is malformed JSON, a JSON array instead of object, fields with mismatched types, missing required columns, or unexpected extra structure — with no deadLetterFile configured.

Common situations: Mixed plain-text and JSON files in the input glob; single-quoted/JSON5 lines; schema changed after files were written; nested fields not matching the declared Beam schema.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

Related errors

• Failed to parse hadoop_config string as JSON
• A 'datagen' table requires either 'rows-per-second' (for…
• ALTER is not supported for table
• Analytics Function [ ] is not supported
• bad struct encoding

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/94c4d818d50da8fa. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java:199

    public static JsonToRow create(Schema schema) {
      return create(schema, null);
    }

    @Override
    public PCollection<Row> expand(PCollection<String> input) {
      PCollectionTuple rows =
          input.apply(
              ParDo.of(
                      new DoFn<String, Row>() {
                        @ProcessElement
                        public void processElement(ProcessContext context) {
                          try {
                            context.output(jsonToRow(getObjectMapper(), context.element()));
                          } catch (UnsupportedRowJsonException jsonException) {
                            if (deadLetterFile() != null) {
                              context.output(DLF_TAG, context.element());
                            } else {
                              throw new RuntimeException("Error parsing JSON", jsonException);
                            }
                          }
                        }
                      })
                  .withOutputTags(
                      MAIN_TAG,
                      deadLetterFile() != null ? TupleTagList.of(DLF_TAG) : TupleTagList.empty()));

      if (deadLetterFile() != null) {
        rows.get(DLF_TAG).setCoder(StringUtf8Coder.of()).apply(writeJsonToDlf());
      }
      return rows.get(MAIN_TAG).setRowSchema(schema());
    }

    private TextIO.Write writeJsonToDlf() {
      return TextIO.write().withDelimiter(new char[] {}).to(deadLetterFile());
    }

View on GitHub (pinned to 12126d8942)