apache/beam · warning

Failed to parse payload

Error message

Failed to parse payload: %s as json at: %s : %s.Dropping resource from batch import.

What it means

FhirIO's batch import body formatting expects each input element to be valid JSON that can be wrapped into an HTTP body part. When Jackson fails to parse/serialize the payload (JsonProcessingException), the element is logged with the character offset of the parse failure, emitted to Write.FAILED_BODY as a HealthcareIOError, and dropped from the batch import — the pipeline continues without it.

Solutions

1. Validate each payload parses as JSON before feeding it to FhirIO (e.g. new ObjectMapper().readTree(payload)).
2. Route and inspect the FAILED_BODY PCollection to find malformed payloads and their offsets.
3. Fix upstream serialization so each element is exactly one complete JSON FHIR resource.
4. If NDJSON, split records into individual JSON strings first.

Example fix

// before
PCollection<String> raw = pipeline.apply(TextIO.read().from("import.ndjson")); // one big blob line
// after
PCollection<String> records = pipeline.apply(TextIO.read().from("import.ndjson")).apply(ParDo.of(new SplitNdJsonFn()));
records.apply("Validate", ParDo.of(new JsonValidationFn())).apply(FhirIO.write().resources());

// Java: validate JSON before feeding FhirIO.write().resources()
private static final com.fasterxml.jackson.databind.ObjectMapper OM = new com.fasterxml.jackson.databind.ObjectMapper();
public static boolean isValidJson(String s) {
  try { OM.readTree(s); return true; } catch (Exception e) { return false; }
}

When it happens

Trigger: Supplying FhirIO.write().resources() with strings that are not valid JSON documents (truncated JSON, NDJSON lines concatenated, HTML error pages, empty strings, or payloads that are valid but not FHIR-shaped enough for the body formatter).

Common situations: Importing from a text source where records were not validated as JSON; encoding issues corrupting payloads; concatenating multiple JSON objects without delimiters; upstream systems emitting partial records.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Error executing GetPatientEverything: FHIR resources…
• Error fetching Fhir resource with ID
• Error search FHIR resources writing to Dead Letter Queue.
• Error fetching HL7v2 message with ID
• Failed to import with error. Moving to deadletter path

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/755fa473430e5dbc. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java:1221

       * @param context the context
       * @throws IOException the io exception
       */
      @ProcessElement
      public void addToFile(ProcessContext context, BoundedWindow window) throws IOException {
        this.window = window;
        String httpBody = context.element();
        try {
          // This will error if not valid JSON an convert Pretty JSON to raw JSON.
          Object data = this.mapper.readValue(httpBody, Object.class);
          String ndJson = this.mapper.writeValueAsString(data) + "\n";
          this.ndJsonChannel.write(ByteBuffer.wrap(ndJson.getBytes(StandardCharsets.UTF_8)));
        } catch (JsonProcessingException e) {
          String resource =
              String.format(
                  "Failed to parse payload: %s as json at: %s : %s."
                      + "Dropping resource from batch import.",
                  httpBody, e.getLocation().getCharOffset(), e.getMessage());
          LOG.warn("{}", resource);
          context.output(
              Write.FAILED_BODY, HealthcareIOError.of(httpBody, new IOException(resource)));
        }
      }

      /**
       * Close file.
       *
       * @param context the context
       * @throws IOException the io exception
       */
      @FinishBundle
      public void closeFile(FinishBundleContext context) throws IOException {
        // Write the file with all elements in this batch to GCS.
        ndJsonChannel.close();
        context.output(resourceId, window.maxTimestamp(), window);
      }
    }

View on GitHub (pinned to 12126d8942)