What causes "Error parsing deferred artifact payload." in apache/beam?

Regenerate/re-stage the artifacts with the same Beam version used to run the pipeline

apache/beam · error · java.lang.RuntimeException

Error parsing deferred artifact payload.

Error message

Error parsing deferred artifact payload.

What it means

getDeferredArtifacts parses the typePayload of each DEFERRED artifact as a DeferredArtifactPayload proto message. If the bytes are not a valid protobuf (InvalidProtocolBufferException) it throws a RuntimeException. It means the deferred artifact's payload is corrupt or not in the expected proto encoding.

Solutions

Regenerate/re-stage the artifacts with the same Beam version used to run the pipeline
Align SDK versions between the submitting pipeline and the artifact staging service
Inspect the artifact typePayload — only valid DeferredArtifactPayload protobufs are accepted

Defensive patterns

Strategy: try-catch

Try / catch

try { artifacts = getDeferredArtifacts(...); } catch (RuntimeException e) { if (e.getMessage().equals("Error parsing deferred artifact payload.")) { /* re-stage artifacts with matching Beam version */ } }

Prevention

Use the same Beam version for submission and staging
Do not hand-craft typePayload bytes for DEFERRED artifacts
Validate staged artifact metadata after staging

When it happens

Trigger: An artifact registered with the DEFERRED type URN whose typePayload bytes were produced by an incompatible SDK version or hand-crafted/stale staged data, then read during environment creation.

Common situations: Mixing Beam versions between job submission and staging service; custom artifact injection writing raw bytes into typePayload; corrupted staged artifact metadata.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e254e5756bf6a064. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/Environments.java:490

    return artifactsBuilder.build();
  }

  public static List<ArtifactInformation> getDeferredArtifacts(PipelineOptions options) {
    List<String> stagingFiles = options.as(PortablePipelineOptions.class).getFilesToStage();
    if (stagingFiles == null || stagingFiles.isEmpty()) {
      return ImmutableList.of();
    }

    String key = UUID.randomUUID().toString();
    DefaultArtifactResolver.INSTANCE.register(
        (info) -> {
          if (BeamUrns.getUrn(StandardArtifacts.Types.DEFERRED).equals(info.getTypeUrn())) {
            RunnerApi.DeferredArtifactPayload deferredArtifactPayload;
            try {
              deferredArtifactPayload =
                  RunnerApi.DeferredArtifactPayload.parseFrom(info.getTypePayload());
            } catch (InvalidProtocolBufferException e) {
              throw new RuntimeException("Error parsing deferred artifact payload.", e);
            }
            if (key.equals(deferredArtifactPayload.getKey())) {
              return Optional.of(getArtifacts(stagingFiles));
            } else {
              return Optional.empty();
            }
          } else {
            return Optional.empty();
          }
        });

    return ImmutableList.of(
        ArtifactInformation.newBuilder()
            .setTypeUrn(BeamUrns.getUrn(StandardArtifacts.Types.DEFERRED))
            .setTypePayload(
                RunnerApi.DeferredArtifactPayload.newBuilder().setKey(key).build().toByteString())
            .build());
  }

View on GitHub (pinned to 12126d8942)