What causes "empty chunk" in apache/beam?

Verify the staged artifact in GCS is fully uploaded and non-empty; restage the job's artifacts.

empty chunk

Error message

empty chunk

What it means

The GCS artifact proxy's staged-artifact reader wraps fetched chunks; if a fetched message carries a zero-length data payload, Read returns errors.New("empty chunk") rather than serving empty bytes, treating an empty chunk as a protocol violation from the server.

Source

Thrown at sdks/go/pkg/beam/artifact/gcsproxy/staging.go:178

// It also computes the SHA256 of the content.
type reader struct {
	sha256W hash.Hash
	buf     []byte
	stream  jobpb.LegacyArtifactStagingService_PutArtifactServer
}

func (r *reader) Read(buf []byte) (int, error) {
	if len(r.buf) == 0 {
		// Buffer empty. Read from upload stream.

		msg, err := r.stream.Recv()
		if err != nil {
			return 0, err // EOF or real error
		}

		r.buf = msg.GetData().GetData()
		if len(r.buf) == 0 {
			return 0, errors.New("empty chunk")
		}
	}

	// Copy out bytes from non-empty buffer.

	n := len(r.buf)
	if n > len(buf) {
		n = len(buf)
	}
	for i := 0; i < n; i++ {
		buf[i] = r.buf[i]
	}
	if _, err := r.sha256W.Write(r.buf[:n]); err != nil {
		panic(err) // cannot fail
	}
	r.buf = r.buf[n:]
	return n, nil
}

View on GitHub (pinned to 12126d8942)

Solutions

Verify the staged artifact in GCS is fully uploaded and non-empty; restage the job's artifacts.
Check the serving proxy for logic that emits empty data messages and skip/mask them.
Retry the artifact fetch — transient upload races can yield empty chunks.

Example fix

// before
if len(r.buf) == 0 {
	return 0, errors.New("empty chunk")
}
// after
if len(r.buf) == 0 {
	continue // skip empty chunk and fetch next
}

Defensive patterns

Strategy: retry

Validate before calling

info, err := gcsObjectInfo(bucket, path)
if err != nil || info.Size == 0 {
	return fmt.Errorf("artifact %s missing or empty; restage", path)
}

Try / catch

n, err := r.Read(buf)
if err != nil && err.Error() == "empty chunk" {
	// reopen the artifact reader and retry the fetch
}

Prevention

Verify staged artifacts in GCS are fully uploaded (size > 0) before job submission.
Use atomic GCS uploads to avoid workers reading partially written artifacts.
Retry artifact fetches on transient empty-chunk errors.

When it happens

Trigger: Iterating chunks from the artifact staging API during artifact Read when the server (or cache) yields a GetArtifactResponse whose data is empty — e.g. a truncated or mis-encoded artifact in GCS.

Common situations: A zero-byte or partially uploaded artifact file in the staging bucket, or a proxy/server bug producing empty messages between real chunks.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9865ad8ca7f958f0. Report an issue: GitHub.