What causes "PCollection is empty, want non-empty collection" in apache/beam?

Inspect the upstream PCollection for emptiness: fix the test input (ptest.Create / datasource) so it actually produces elements.

apache/beam · error

PCollection is empty, want non-empty collection

Error message

PCollection is empty, want non-empty collection

What it means

passert.NonEmpty verifies at runtime (inside the pipeline, via a ParDo over an impulse with the collection as side input) that the asserted PCollection contains at least one element. The DoFn returns this error as soon as it observes that the side-input iterator yields zero elements, failing the test/pipeline. It is the Beam Go testing analog of 'expected at least one record, got none'.

Solutions

Inspect the upstream PCollection for emptiness: fix the test input (ptest.Create / datasource) so it actually produces elements.
Check whether an upstream filter, window trigger, or side-input join is unintentionally discarding all elements.
If emptiness is legitimate for the case being tested, replace passert.NonEmpty with passert.Empty or a size check via passert.Size.
Run the pipeline with logging/beam dump to confirm the number of elements reaching the assertion before changing assertion logic.

Example fix

// before
passert.NonEmpty(s, filtered) // fails when filter drops everything

// after
if expectedCount := countTestRecords(); expectedCount == 0 {
    passert.Empty(s, filtered)
} else {
    passert.NonEmpty(s, filtered)
}

Defensive patterns

Strategy: validation

Validate before calling

if len(testRecords) == 0 {
    // use passert.Empty instead of passert.NonEmpty
}

Prevention

Match the assertion (Empty vs NonEmpty vs Size) to the actual expected record count.
Seed test pipelines with ptest.Create and print intermediate counts when a test first fails.
Review filters/windows upstream of assertions whenever input data changes.

When it happens

Trigger: Calling passert.NonEmpty(s, col) where col resolves to an empty PCollection: the nonEmptyFn.ProcessElement side-input iterator returns false on its first call, and the DoFn returns errors.New("PCollection is empty, want non-empty collection").

Common situations: Test pipelines whose source filter matched no records; reading from an empty test set (e.g. ptest.Create with an empty slice upstream); a transform upstream dropped all elements (buggy filtering/windowing); running against an empty file, empty table or empty Pub/Sub backlog.

Understand the failure class

Background: EmptyResultError / "no results found": when an API or scraper succeeds but returns zero rows — this error's family across 9 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/97383d3095f8199f. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/testing/passert/passert.go:194

	return errors.Errorf(f.Format, fmt.Sprintf("(%v,%v)", x, y))
}

type failGBKFn struct {
	Format string `json:"format"`
}

func (f *failGBKFn) ProcessElement(x beam.X, _ func(*beam.Y) bool) error {
	return errors.Errorf(f.Format, fmt.Sprintf("(%v,*)", x))
}

type nonEmptyFn struct{}

func (n *nonEmptyFn) ProcessElement(_ []byte, iter func(*beam.Z) bool) error {
	var val beam.Z
	for iter(&val) {
		return nil
	}
	return errors.New("PCollection is empty, want non-empty collection")
}

// NonEmpty asserts that the given PCollection has at least one element.
func NonEmpty(s beam.Scope, col beam.PCollection) beam.PCollection {
	s = s.Scope("passert.NonEmpty")
	beam.ParDo0(s, &nonEmptyFn{}, beam.Impulse(s), beam.SideInput{Input: col})
	return col
}

View on GitHub (pinned to 12126d8942)