apache/beam · error · java.lang.RuntimeException

cannot rehydrate PCollection.

Error message

cannot rehydrate PCollection.

What it means

After a successful expansion, External.expand() rehydrates output PCollections from the returned components graph. If fetching a PCollection by id throws IOException, the proto graph is inconsistent or unreadable and the exception is swallowed and replaced by this opaque RuntimeException, losing the underlying cause.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/External.java:316

              .putAllEnvironments(resolveArtifacts(newEnvironmentsWithDependencies, endpoint))
              .build();
      expandedTransform = response.getTransform();
      expandedRequirements = response.getRequirementsList();

      RehydratedComponents rehydratedComponents =
          RehydratedComponents.forComponents(expandedComponents).withPipeline(p);

      ImmutableMap.Builder<TupleTag<?>, PCollection> outputMapBuilder = ImmutableMap.builder();
      expandedTransform
          .getOutputsMap()
          .forEach(
              (localId, pCollectionId) -> {
                try {
                  PCollection col = rehydratedComponents.getPCollection(pCollectionId);
                  externalPCollectionIdMapBuilder.put(col, pCollectionId);
                  outputMapBuilder.put(new TupleTag<>(localId), col);
                } catch (IOException e) {
                  throw new RuntimeException("cannot rehydrate PCollection.");
                }
              });
      externalPCollectionIdMap = externalPCollectionIdMapBuilder.build();

      Map<Coder<?>, String> externalCoderIdMapBuilder = new HashMap<>();
      expandedComponents
          .getPcollectionsMap()
          .forEach(
              (pcolId, pCol) -> {
                try {
                  String coderId = pCol.getCoderId();
                  if (isJavaSDKCompatible(expandedComponents, coderId)) {
                    Coder<?> coder = rehydratedComponents.getCoder(coderId);
                    externalCoderIdMapBuilder.putIfAbsent(coder, coderId);
                  }
                } catch (IOException e) {
                  throw new RuntimeException("cannot rehydrate Coder.");
                }

View on GitHub (pinned to 12126d8942)

Solutions

1. Align the expansion service version with the Beam SDK version used to build the pipeline.
2. Log/attach the expansion request/response (proto) and inspect the referenced pCollection id for presence in components.
3. Restart or upgrade the expansion service; if custom, verify it emits complete RunnerApi.Components.
4. Wrap the cause: modify to `throw new RuntimeException("cannot rehydrate PCollection.", e)` to preserve the IOException for debugging.

Example fix

// before
throw new RuntimeException("cannot rehydrate PCollection.");
// after
throw new RuntimeException("cannot rehydrate PCollection: " + pCollectionId, e);

null

try { external.expand(...); } catch (RuntimeException e) { if ("cannot rehydrate PCollection.".equals(e.getMessage())) { /* inspect expansion response components; version skew likely */ } throw e; }

When it happens

Trigger: Expansion service returns a components graph whose output pCollection ids do not resolve during rehydratedComponents.getPCollection(pCollectionId), typically due to a malformed or version-skewed expansion response.

Common situations: Mismatched Beam SDK versions between pipeline and expansion service producing incompatible proto graphs; custom/broken expansion service implementations returning incomplete components.

Related errors

• Timing number 0b" + timingNumber.toString(2) + " has more th
• No proto encoding for PaneInfoCoder, always part of Windowed
• expansion service error: %s
• cannot rehydrate Coder.
• Unknown %s %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/aef3565ae7d4ae2d. Report an issue: GitHub.