Processing of an element in transform %v has exceeded the sp

Error message

Processing of an element in transform %v has exceeded the specified timeout of %v without outputting or completing in state %v, SDK harness will be terminated

What it means

The Beam Go SDK's sampler periodically inspects in-flight bundle processing. When an element in a PTransform has been processing longer than the configured restart lull timeout (restartLullTimeout) with no output or state transition, the sampler returns this error to terminate the SDK harness, since the pipeline is presumed hung. It is a deliberate liveness watchdog, not a data-dependent bug.

Source

Thrown at sdks/go/pkg/beam/core/metrics/sampler.go:70

	if v, ok := s.store.stateRegistry[ps.pid]; ok {
		v[ps.state].TotalTime += t
		v[TotalBundle].TotalTime += t

		if s.transitionsAtLastSample != ps.transitions {
			// state change detected
			s.millisSinceLastTransition = 0
			s.transitionsAtLastSample = ps.transitions
			s.nextLogTime = s.logInterval
		} else {
			s.millisSinceLastTransition += t
		}

		if s.millisSinceLastTransition > s.nextLogTime {
			log.Infof(ctx, "Operation ongoing in transform %v for at least %v without outputting or completing in state %v", ps.pid, s.millisSinceLastTransition, getState(ps.state))
			s.nextLogTime += s.logInterval
		}
		if s.restartLullTimeout > 0 && s.millisSinceLastTransition > s.restartLullTimeout {
			return errors.Errorf("Processing of an element in transform %v has exceeded the specified timeout of %v without outputting or completing in state %v, SDK harness will be terminated", ps.pid, s.restartLullTimeout, getState(ps.state))
		}
	}
	return nil
}

// SetLogInterval sets the logging interval for lull reporting.
func (s *StateSampler) SetLogInterval(t time.Duration) {
	s.logInterval = t
}

func loadCurrentState(s *StateSampler) currentStateVal {
	ts := (atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&s.store.bundleState))))
	if ts == nil {
		return currentStateVal{}
	}
	bs := *(*BundleState)(ts)
	return currentStateVal{pid: bs.pid, state: bs.currentState, transitions: atomic.LoadInt64(s.store.transitions)}
}

View on GitHub (pinned to 12126d8942)

Solutions

Fix the underlying stall: profile the DoFn for blocking calls (unbounded waits, missing side input, deadlocked mutexes).
Increase or disable the restart lull timeout via the runner/harness lull-detection options (e.g. set it to 0 to disable) if long processing is legitimate.
Break large elements into smaller ones or add progress logging so the sampler sees state transitions.
Check worker logs around the reported pid/state to identify which transform and state stalled.

Example fix

// before: DoFn blocks forever waiting for external service
resp := <-ch // no timeout, hangs the bundle
// after
select {
case resp := <-ch:
    _ = resp
case <-time.After(30 * time.Second):
    return errors.New("upstream call timed out")
}

Defensive patterns

Strategy: try-catch

Try / catch

// runner-side: handle harness termination and retry the bundle
if err := bundle.Execute(ctx); err != nil {
    if strings.Contains(err.Error(), "exceeded the specified timeout") {
        log.Errorf("lull timeout in %s: %v", transformID, err)
        // alert / restart worker / adjust lull timeout config
    }
    return err
}

Prevention

Never block indefinitely in DoFns; always use timeouts on I/O and channel waits.
Size workers adequately so elements don't stall from CPU/memory starvation.
Set a restart lull timeout appropriate for your slowest legitimate element.
Monitor sampler logs ('Operation ongoing in transform ...') as early warnings before the timeout fires.

When it happens

Trigger: sampler.Sample is called while a transform's state has not transitioned for longer than s.restartLullTimeout; configured via a nonzero restartLullTimeout on the sampler (set when the harness enables lull detection).

Common situations: User DoFns blocked on slow I/O, deadlocks waiting on side inputs, extremely large single elements, or resource starvation (CPU/threads) in the SDK worker container causing processing to stall.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4dc2317a7d31359b. Report an issue: GitHub.