What causes "short write of data got %d, want %d" in apache/beam?

Inspect the error returned alongside it (write returns the underlying err too).

short write of data got

Error message

short write of data got %d, want %d

What it means

The starcgen write helper detects a partial write: io.Writer returned n < len(data). It signals that the generated output (license header or generated code) was not fully written. Note the condition also requires err != nil, so this usually accompanies an underlying write error which is returned as well.

Solutions

Inspect the error returned alongside it (write returns the underlying err too).
Replace short/custom writers with os.File or bytes.Buffer for starcgen output.
Free disk space or fix the underlying storage error.
If using a custom writer, fix it to return io.ErrShortWrite semantics correctly or loop writes until all bytes are consumed.

Example fix

// before
w := bytes.NewBuffer(make([]byte, 0, 16)) // too small, custom truncating writer
gen.Generate(w, ...)
// after
w := &bytes.Buffer{} // unbounded, correct io.Writer semantics
gen.Generate(w, ...)

Defensive patterns

Strategy: validation

Validate before calling

var probe [1]byte
if _, err := w.Write(probe[:]); err != nil { return fmt.Errorf("writer not ready: %w", err) }

Try / catch

err := gen.Generate(w, ...)
if err != nil && strings.Contains(err.Error(), "short write") {
	log.Printf("underlying write error: %v", err)
}

Prevention

Use standard writers (bytes.Buffer, os.File) with correct io.Writer semantics.
Fix custom writers to loop until all bytes are written or return io.ErrShortWrite.
Monitor disk space on generation output paths.

When it happens

Trigger: Calling Generate (which calls write) with an io.Writer that accepts fewer bytes than requested and returns an error, e.g. a limited buffer, a failing network/file writer, or a custom writer with a short-write bug.

Common situations: Writing generated code to a custom in-memory writer with a fixed-size buffer; disk-full conditions; faulty wrappers around os.File that mishandle n.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/856da1e5edff7ef6. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/cmd/starcgen/starcgen.go:97

		// Always print out the debugging info to the file.
		if _, errw := w.Write(e.Bytes()); errw != nil {
			return fmt.Errorf("error writing debug data to file after err %v:%v", err, errw)
		}
		err = fmt.Errorf("error extracting from asts: %v", err)
		e.Printf("%v", err)
		return err
	}
	data := e.Generate(filename)
	if err := write(w, []byte(license)); err != nil {
		return err
	}
	return write(w, data)
}

func write(w io.Writer, data []byte) error {
	n, err := w.Write(data)
	if err != nil && n < len(data) {
		return fmt.Errorf("short write of data got %d, want %d", n, len(data))
	}
	return err
}

func usage() {
	fmt.Fprintf(os.Stderr, "Usage: %v [options] --inputs=<comma separated of go files>\n", filepath.Base(os.Args[0]))
	flag.PrintDefaults()
}

func main() {
	flag.Usage = usage
	flag.Parse()

	log.SetFlags(log.Lshortfile)
	log.SetPrefix("starcgen: ")

	dir, err := filepath.Abs(filepath.Dir(os.Args[0]))
	if err != nil {

View on GitHub (pinned to 12126d8942)