What causes "Transform with URN %s failed to parse: %s" in apache/beam?

Upgrade/align the Beam SDK versions of all pipeline producers and the translating runner.

apache/beam · error · java.lang.RuntimeException

Transform with URN %s failed to parse: %s

Error message

Transform with URN %s failed to parse: %s

What it means

The writeWithRunnerDeterminedSharding matcher checks WriteFiles transforms by calling WriteFilesTranslation.isRunnerDeterminedSharding, which parses the WriteFilesPayload. IOException while parsing is wrapped in this RuntimeException naming WRITE_FILES_TRANSFORM_URN and the offending transform, indicating a malformed or incompatible WriteFiles payload.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/PTransformMatchers.java:533

      public boolean matchesDuringValidation(AppliedPTransform<?, ?, ?> application) {
        return false;
      }

      @Override
      public String toString() {
        return MoreObjects.toStringHelper("groupWithShardableStatesMatcher").toString();
      }
    };
  }

  public static PTransformMatcher writeWithRunnerDeterminedSharding() {
    return application -> {
      if (PTransformTranslation.WRITE_FILES_TRANSFORM_URN.equals(
          PTransformTranslation.urnForTransformOrNull(application.getTransform()))) {
        try {
          return WriteFilesTranslation.isRunnerDeterminedSharding((AppliedPTransform) application);
        } catch (IOException exc) {
          throw new RuntimeException(
              String.format(
                  "Transform with URN %s failed to parse: %s",
                  PTransformTranslation.WRITE_FILES_TRANSFORM_URN, application.getTransform()),
              exc);
        }
      }
      return false;
    };
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

Upgrade/align the Beam SDK versions of all pipeline producers and the translating runner.
Inspect the failing transform's payload and confirm it parses as RunnerApi.WriteFilesPayload.
Regenerate/re-save the pipeline proto with a consistent SDK version.

Example fix

// before
throw new RuntimeException(String.format("Transform with URN %s failed to parse: %s", URN, application.getTransform()), exc);
// after (same, but ensure producer/consumer SDKs match)
// mvn versions:set -DnewVersion=<same beam version everywhere>

Defensive patterns

Strategy: try-catch

Validate before calling

null

Try / catch

try { matcher.matches(application); } catch (RuntimeException e) { if (e.getCause() instanceof IOException) { /* WriteFilesPayload parse failure */ } throw e; }

Prevention

Match SDK versions that write and read pipeline protos
Validate WriteFilesPayload with the same Beam version that wrote it
Re-save pipelines when upgrading Beam

When it happens

Trigger: Matching an AppliedPTransform with URN beam:transform:WriteFiles whose WriteFilesPayload bytes fail to parse — during runner-determined-sharding handling in pipeline translation.

Common situations: Beam version skew between the pipeline-producer and translating SDK; custom file sinks with nonstandard payloads; corrupted serialized pipelines.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c242ec3b138a217a. Report an issue: GitHub.