apache/beam · critical

error decoding residual header:

Error message

error decoding residual header:

What it means

When re-materializing residual elements of an unfinished bundle, reElementResiduals decodes a windowed-value header from the residual buffer. A non-EOF decode error means the residual bytes are malformed or produced by an incompatible coder, so prism panics after logging.

Solutions

1. Retry the pipeline; if intermittent, report as a possible prism residual-buffer bug
2. Upgrade Beam to the latest release where residual handling fixes may land
3. Disable/reduce bundle splitting (e.g. via pipeline options) as a workaround to avoid residual paths
4. Capture the logged 'error' field and bundle details to include in a bug report

// pre-check residual buffer plausibility before reprocessing
if len(buf.Bytes()) == 0 { return errors.New("residual buffer empty, skipping re-element") }

func safeReElement(rb Bundle, buf []byte) (ok bool) {
  defer func() { if r := recover(); r != nil { log.Errorf("residual decode panicked: %v", r); ok = false } }()
  reElementResiduals(rb, buf); return true
}

When it happens

Trigger: DecodeWindowedValueHeader returns an error other than io.EOF while decoding a residual bundle's buffer — truncated residual data, coder mismatch between the writer and reader of the residual split, or corrupt on-buffer content.

Common situations: Bundle splitting producing residuals the decoder can't parse; SDK/runner coder version skew; potential prism bug in residual buffer management (noted buffer-invalidation caveat in code).

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• couldn't decode characteristic for variant
• error decoding append bag user state window key
• error decoding watermarks
• error re-encoding characteristic for variant
• generating bundle for stage

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4b331a4daff70615. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:828

			delayed = map[mtime.Time][]Residual{}
		}
		fireAt := emNow.Add(r.Delay)
		delayed[fireAt] = append(delayed[fireAt], r)
	}
	return immediate, delayed
}

func reElementResiduals(residuals []Residual, inputInfo PColInfo, rb RunBundle) []element {
	var unprocessedElements []element
	for _, residual := range residuals {
		buf := bytes.NewBuffer(residual.Element)
		ws, et, pn, err := exec.DecodeWindowedValueHeader(inputInfo.WDec, buf)
		if err != nil {
			if err == io.EOF {
				break
			}
			slog.Error("reElementResiduals: error decoding residual header", "error", err, "bundle", rb)
			panic("error decoding residual header:" + err.Error())
		}
		if len(ws) == 0 {
			slog.Warn("reElementResiduals: sdk provided a windowed value header 0 windows", "bundle", rb)
		}
		// POSSIBLY BAD PATTERN: The buffer is invalidated on the next call, which doesn't always happen.
		// But the decoder won't be mutating the buffer bytes, just reading the data. So the elmBytes
		// should remain pointing to the whole element, and we should have a copy of the key bytes.
		// Ideally, we're simply refering to the key part of the existing buffer.
		elmBytes := buf.Bytes()
		var keyBytes []byte
		if inputInfo.KeyDec != nil {
			keyBytes = inputInfo.KeyDec(buf)
		}

		for _, w := range ws {
			unprocessedElements = append(unprocessedElements,
				element{
					window:    w,

View on GitHub (pinned to 12126d8942)