juicedata/juicefs · error

buffer too short: %d < %d

Error message

buffer too short: %d < %d

What it means

The Noop compressor's Compress copies src into dst unchanged and requires len(dst) >= len(src); if the destination buffer is too small it returns 'buffer too short: %d < %d'. It exists so callers that sized dst via CompressBound (which for Noop equals len(src)) can detect undersized buffers instead of truncating data.

Source

Thrown at pkg/compress/compress.go:57

func NewCompressor(algr string) Compressor {
	algr = strings.ToLower(algr)
	if algr == "zstd" {
		return ZStandard{ZSTD_LEVEL}
	} else if algr == "lz4" {
		return LZ4{}
	} else if algr == "none" || algr == "" {
		return noOp{}
	}
	return nil
}

type noOp struct{}

func (n noOp) Name() string            { return "Noop" }
func (n noOp) CompressBound(l int) int { return l }
func (n noOp) Compress(dst, src []byte) (int, error) {
	if len(dst) < len(src) {
		return 0, fmt.Errorf("buffer too short: %d < %d", len(dst), len(src))
	}
	copy(dst, src)
	return len(src), nil
}
func (n noOp) Decompress(dst, src []byte) (int, error) {
	if len(dst) < len(src) {
		return 0, fmt.Errorf("buffer too short: %d < %d", len(dst), len(src))
	}
	copy(dst, src)
	return len(src), nil
}

// ZStandard implements Compressor interface using zstd library
type ZStandard struct {
	level int
}

// Name returns name of the algorithm Zstd

View on GitHub (pinned to c9a67b23e8)

Solutions

  1. Allocate dst with at least compressor.CompressBound(len(src)) bytes before calling Compress.
  2. For noOp specifically, ensure dst capacity equals len(src) (CompressBound(l) == l).
  3. Check configuration: if volume compression is none, no shrinking occurs, so buffers must be input-sized.

Example fix

// before
dst := make([]byte, len(src)/2)
n, err := comp.Compress(dst, src)

// after
dst := make([]byte, comp.CompressBound(len(src)))
n, err := comp.Compress(dst, src)
Defensive patterns

Strategy: validation

Validate before calling

if len(dst) < comp.CompressBound(len(src)) { dst = make([]byte, comp.CompressBound(len(src))) }

Try / catch

n, err := comp.Compress(dst, src)
if err != nil && strings.HasPrefix(err.Error(), "buffer too short") {
    dst = make([]byte, comp.CompressBound(len(src)))
    n, err = comp.Compress(dst, src)
}

Prevention

When it happens

Trigger: Calling noOp.Compress(dst, src) where len(dst) < len(src) — i.e. the caller allocated dst smaller than CompressBound(len(src)) or passed a nil/empty dst.

Common situations: Configuring compression 'none' in the volume format but writing code (or a caller like the chunk writer) that assumes compressed output is smaller and allocates dst as a fraction of input; tests passing wrong-size buffers.

Related errors

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/b372f7a13d52eb50. Report an issue: GitHub.