apache/beam · error

tried converting invalid PCollection

Error message

tried converting invalid PCollection

What it means

pCollectionToNode unwraps a beam.PCollection to its internal graph node for cross-language transform mapping, and panics if the PCollection is invalid (its underlying node is unset). An invalid PCollection usually results from using the zero value of PCollection or one produced by a failed/aborted construction path instead of a real pipeline output.

Solutions

1. Call p.IsValid() on every input PCollection before passing it to CrossLanguage/TryCrossLanguage.
2. Ensure all PCollections originate from real transforms on the same pipeline, not zero values.
3. Check earlier in the pipeline construction for ignored errors that produced the invalid PCollection.
4. If shuffling PCollections across pipelines/scopes, rebuild them within the target scope instead of reusing handles.

Example fix

// before
var col beam.PCollection
outs := beam.CrossLanguage(s, urn, payload, addr, map[string]beam.PCollection{"in": col}, nil) // panics
// after
if !col.IsValid() {
    return errors.New("input PCollection is invalid")
}
outs := beam.CrossLanguage(s, urn, payload, addr, map[string]beam.PCollection{"in": col}, nil)

// check every PCollection before cross-language calls
for name, col := range inputs {
    if !col.IsValid() {
        return fmt.Errorf("input %q is an invalid PCollection", name)
    }
}

func validPCollection(p beam.PCollection) bool {
    return p.IsValid()
}

func toNodeSafe(p beam.PCollection) (n *graph.Node, err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("pcollection conversion failed: %v", r)
        }
    }()
    if !p.IsValid() {
        return nil, errors.New("invalid PCollection")
    }
    return pCollectionToNode(p), nil
}

When it happens

Trigger: Calling xlang functions (mapPCollectionToNode via CrossLanguage/TryCrossLanguage) with a PCollection that fails IsValid() — passing a zero-value PCollection, a PCollection from a pipeline that errored during construction, or mixing PCollections between different scope/pipeline instances.

Common situations: Building cross-language transforms with placeholder/empty PCollections; ignoring an earlier error from a transform that returned an invalid PCollection; accidentally declaring PCollection variables without initialization and passing them to xlang helpers.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• attempted to add namespace to missing coder id
• attempted to add namespace to missing windowing strategy id
• failed to add output coder to coder registry
• failed to expand cross language transform for edge
• invalid pcollection

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7de4995c03f83491. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/xlang.go:223

	// Expand the transform into ext.Expanded.
	if err := xlangx.Expand(edge, &ext); err != nil {
		return nil, errors.WithContext(err, "expanding external transform")
	}

	// Ensures the expected named outputs are present
	graphx.VerifyNamedOutputs(&ext)
	// Using the expanded outputs, the graph's counterpart outputs are updated with bounded values
	graphx.ResolveOutputIsBounded(edge, isBoundedUpdater)

	return mapNodeToPCollection(graphx.ExternalOutputs(edge)), nil
}

// Wrapper functions to handle beam <-> graph boundaries

func pCollectionToNode(p PCollection) *graph.Node {
	if !p.IsValid() {
		panic("tried converting invalid PCollection")
	}
	return p.n
}

func nodeToPCollection(n *graph.Node) PCollection {
	if n == nil {
		panic("tried converting invalid Node")
	}
	c := PCollection{n}
	c.SetCoder(NewCoder(c.Type()))
	return c
}

func mapPCollectionToNode(pMap map[string]PCollection) map[string]*graph.Node {
	if pMap == nil {
		return nil
	}

View on GitHub (pinned to 12126d8942)