apache/beam · error
invalid main pcollection
Error message
invalid main pcollection
What it means
validate() (used by TryParDo, TryCombinePerKey and related constructors) checks that the main input PCollection is valid before building the transform. This error means col is the zero PCollection — produced when a transform returned an error earlier and the caller ignored it, or when a PCollection variable was never assigned. The pipeline graph cannot reference a non-existent node.
Source
Thrown at sdks/go/pkg/beam/validate.go:53
// ValidateNonCompositeType panics if the type of the PCollection is not a
// composite type. It returns the type.
func ValidateNonCompositeType(col PCollection) typex.FullType {
t := col.Type()
if typex.IsComposite(t.Type()) {
panic(fmt.Sprintf("pcollection must be of non-composite type: %v", col))
}
return t
}
// validate validates and processes the input collection and options. Private convenience
// function.
func validate(s Scope, col PCollection, opts []Option) ([]SideInput, map[string]reflect.Type, error) {
if !s.IsValid() {
return nil, nil, errors.New("invalid scope")
}
if !col.IsValid() {
return nil, nil, errors.New("invalid main pcollection")
}
side, defs := parseOpts(opts)
for i, in := range side {
if !in.Input.IsValid() {
return nil, nil, errors.Errorf("invalid side pcollection: index %v", i)
}
}
typedefs, err := makeTypedefs(defs)
if err != nil {
return nil, nil, err
}
return side, typedefs, nil
}
func makeTypedefs(list []TypeDefinition) (map[string]reflect.Type, error) {
typedefs := make(map[string]reflect.Type)
for _, v := range list {
if !typex.IsUniversal(v.Var) {View on GitHub (pinned to 12126d8942)
Solutions
- Handle errors from every Try* call: `out, err := beam.TryParDo(...); if err != nil { return err }` — the zero PCollection almost always traces back to an earlier ignored error.
- Audit the variable holding the main input and confirm it is the return value of a successful transform, not a zero value.
- Check for shadowed variables (`col :=` vs `col =`) inside if/else blocks.
- Switch to Must-style constructors during debugging so the first construction failure panics immediately instead of propagating an invalid PCollection.
Example fix
// before
col, _ := beam.TryParDo(s, fn, input) // err swallowed; col invalid downstream
beam.ParDo0(s, fn2, col)
// after
col, err := beam.TryParDo(s, fn, input)
if err != nil {
return err
}
beam.ParDo0(s, fn2, col) Defensive patterns
Strategy: validation
Validate before calling
if !col.IsValid() {
return fmt.Errorf("main input PCollection is not initialized; an upstream Try* call likely failed")
} Prevention
- Never discard the error from Try* transform constructors; check it immediately.
- Avoid zero values of beam.PCollection as intermediate variables — always assign from a transform's return.
- Watch for variable shadowing inside branches when building pipeline collections.
When it happens
Trigger: Calling beam.TryParDo(s, dofn, col) / TryCombinePerKey where col is the zero value: typically from `col, _ := someTryTransform(...)` swallowing an earlier error, or `var col beam.PCollection` that was never populated.
Common situations: Ignoring the error from Try* transforms and using the returned zero PCollection downstream; conditional branches that skip populating a PCollection; accidental shadowing of a PCollection variable; using the result of a function that returns PCollection{} on a failure path.
Understand the failure class
Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.
Related errors
AI-assisted analysis of apache/beam@12126d8942 (2026-09-13).
Data as JSON: /api/errors/9bbc1870510b3a8a.
Report an issue: GitHub.