What causes "invalid pcollection to external: index %v" in apache/beam?

Check that the PCollection at the reported index was assigned the output of a real transform.

invalid pcollection to external: index

Error message

invalid pcollection to external: index %v

What it means

TryExternal validates every input PCollection passed to an external transform (beam.External/TryExternal in sdks/go/pkg/beam/external.go). If any input PCollection is invalid (zero value, not produced by a valid scope, e.g. the PCollection{} zero value), it fails with this message including the offending index. It guards against wiring uninitialized collections into an external transform.

Solutions

Check that the PCollection at the reported index was assigned the output of a real transform.
Ensure the PCollection was produced within the same valid pipeline scope.
Use beam.TryExternal and handle the returned error instead of the panicking beam.External wrapper.
Add col.IsValid() checks before calling External.

Example fix

// before
var input beam.PCollection
beam.External(s, urn, payload, []beam.PCollection{input}, outs, true)
// after
input := beam.ParDo(s, &myFn{}, seed)
if !input.IsValid() { log.Fatal("input pcollection invalid") }
beam.External(s, urn, payload, []beam.PCollection{input}, outs, true)

Defensive patterns

Strategy: validation

Validate before calling

for i, col := range ins {
    if !col.IsValid() {
        return fmt.Errorf("input pcollection %d invalid before External", i)
    }
}

Try / catch

if err := beam.TryExternal(s, urn, payload, ins, outs, true); err != nil {
    return fmt.Errorf("external transform setup failed: %w", err)
}

Prevention

Never use zero-value beam.PCollection variables
Assign every PCollection from a real transform's output
Prefer Try* variants during pipeline development

When it happens

Trigger: Passing a zero-value beam.PCollection{} or a PCollection created outside a valid scope into beam.External's in []PCollection parameter.

Common situations: Declaring `var in beam.PCollection` and forgetting to assign it from a prior transform; constructing PCollections manually instead of via pipeline combinators.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6d34c3e4b9e9a4d4. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/external.go:44

// URN provided to implement the behavior of the operation. Transform
// libraries should expose an API that captures the user's intent and serialize
// the payload as a byte slice that the runner will deserialize.
//
// Use ExternalTagged if the runner will need to associate the PTransforms local PCollection tags
// with values in the payload.
func External(s Scope, urn string, payload []byte, in []PCollection, out []FullType, bounded bool) []PCollection {
	return MustN(TryExternal(s, urn, payload, in, out, bounded))
}

// TryExternal attempts to perform the work of External, returning an error indicating
// why the operation failed.
func TryExternal(s Scope, urn string, payload []byte, in []PCollection, out []FullType, bounded bool) ([]PCollection, error) {
	if !s.IsValid() {
		return nil, errors.New("invalid scope")
	}
	for i, col := range in {
		if !col.IsValid() {
			return nil, errors.Errorf("invalid pcollection to external: index %v", i)
		}
	}

	var ins []*graph.Node
	for _, col := range in {
		ins = append(ins, col.n)
	}
	edge := graph.NewExternal(s.real, s.scope, &graph.Payload{URN: urn, Data: payload}, ins, out, bounded)

	var ret []PCollection
	for _, out := range edge.Output {
		c := PCollection{out.To}
		c.SetCoder(NewCoder(c.Type()))
		ret = append(ret, c)
	}
	return ret, nil
}

View on GitHub (pinned to 12126d8942)