apache/beam · error

pcollection %v not found

Error message

pcollection %v not found

What it means

makeCoderForPCollection looks up the PCollection by id in the pipeline description's Pcollections map. If the referenced id is absent, the builder cannot resolve coders for the node and fails with this error.

Source

Thrown at sdks/go/pkg/beam/core/runtime/exec/translate.go:350

		// TODO[BEAM-6374): Properly handle the multiplex and flatten cases.
		// Right now we just stop datasink collection.
		switch out := n.Out.(type) {
		case *DataSink:
			// We don't remove the PCollection from units here, since we
			// want to ensure it's included in snapshots.
			out.PCol = n
			ret = append(ret, out)
		default:
			ret = append(ret, n)
		}
	}
	return ret, nil
}

func (b *builder) makeCoderForPCollection(id string) (*coder.Coder, *coder.WindowCoder, error) {
	col, ok := b.desc.GetPcollections()[id]
	if !ok {
		return nil, nil, errors.Errorf("pcollection %v not found", id)
	}
	c, err := b.coders.Coder(col.CoderId)
	if err != nil {
		return nil, nil, err
	}
	if coder.IsW(c) {
		// TODO(herohde) 3/16/2018: remove potential WindowedValue from Dataflow.
		// However, windowing strategies are not yet passed through, so the main
		// path always gives us GlobalWindows.

		return coder.SkipW(c), c.Window, nil
	}

	ws, ok := b.desc.GetWindowingStrategies()[col.GetWindowingStrategyId()]
	if !ok {
		return nil, nil, errors.Errorf("windowing strategy %v not found", id)
	}
	wc, err := b.coders.WindowCoder(ws.GetWindowCoderId())

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify every transform input/output in the pipeline proto has a corresponding entry in the Pcollections map
2. Regenerate the pipeline so ids are consistent (re-run pipeline construction)
3. Upgrade/align SDK and runner versions to avoid id mismatches

// Validate all link ids exist in the pcollections map
for _, t := range desc.GetTransforms() {
    for id := range t.GetOutputs() {
        if _, ok := desc.GetPcollections()[id]; !ok {
            return fmt.Errorf("missing pcollection %s", id)
        }
    }
}

if err := exec.UnmarshalPlan(desc); err != nil {
    if strings.Contains(err.Error(), "pcollection") && strings.Contains(err.Error(), "not found") {
        // regenerate pipeline description and retry
    }
}

When it happens

Trigger: A transform input/output link references a pcollection id that doesn't exist in desc.GetPcollections(), during newPCollectionNode or makeLink while unmarshalling a plan.

Common situations: Corrupt or hand-edited pipeline protos; runner bug dropping pcollection entries; version mismatch where ids were re-generated between plan submission and materialization.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• expected one output from DataSource, got %v
• windowing strategy %v not found
• invalid transform payload %v for %v
• Unrecognized state type %v
• unexpected sideinput to combine: got %d, want 1

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5515b6063900e29f. Report an issue: GitHub.