dgraph-io/badger · error

Request size offset %d is bigger than maximum offset %d

Error message

Request size offset %d is bigger than maximum offset %d

What it means

validateWrites estimates each incoming request's size and rejects any request whose cumulative value-log offset would exceed maxVlogFileSize (the hard maximum vlog file size). Such a request could never fit in any value log file, so badger fails it before writing anything.

Source

Thrown at value.go:791

	curlf.lock.RUnlock()
	return err
}

func (vlog *valueLog) woffset() uint32 {
	return vlog.writableLogOffset.Load()
}

// validateWrites will check whether the given requests can fit into 4GB vlog file.
// NOTE: 4GB is the maximum size we can create for vlog because value pointer offset is of type
// uint32. If we create more than 4GB, it will overflow uint32. So, limiting the size to 4GB.
func (vlog *valueLog) validateWrites(reqs []*request) error {
	vlogOffset := uint64(vlog.woffset())
	for _, req := range reqs {
		// calculate size of the request.
		size := estimateRequestSize(req)
		estimatedVlogOffset := vlogOffset + size
		if estimatedVlogOffset > uint64(maxVlogFileSize) {
			return fmt.Errorf("Request size offset %d is bigger than maximum offset %d",
				estimatedVlogOffset, maxVlogFileSize)
		}

		if estimatedVlogOffset >= uint64(vlog.opt.ValueLogFileSize) {
			// We'll create a new vlog file if the estimated offset is greater or equal to
			// max vlog size. So, resetting the vlogOffset.
			vlogOffset = 0
			continue
		}
		// Estimated vlog offset will become current vlog offset if the vlog is not rotated.
		vlogOffset = estimatedVlogOffset
	}
	return nil
}

// estimateRequestSize returns the size that needed to be written for the given request.
func estimateRequestSize(req *request) uint64 {
	size := uint64(0)

View on GitHub (pinned to 2a001d466f)

Solutions

1. Shrink or chunk the large value so no single entry exceeds maxVlogFileSize, or store the blob externally and keep a reference key in badger.
2. Check opt.ValueLogFileSize: it must be <= maxVlogFileSize (default max ~2GB); set it within the supported range.
3. Split large batches into smaller requests; estimateRequestSize counts the whole request.
4. For genuinely huge values, use badger's Stream framework or an external store (S3/filesystem) with badger holding metadata.

Example fix

// before
opt.ValueLogFileSize = 4 << 30 // exceeds maxVlogFileSize
db.Set(key, hugeValue) // Request size offset X is bigger than maximum offset Y
// after
opt.ValueLogFileSize = 1 << 30 // within max
if len(hugeValue) > 100<<20 {
    ref := storeExternal(hugeValue)
    db.Set(key, ref)
} else {
    db.Set(key, hugeValue)
}

// Go: validate request size before writing
const maxVlogFileSize = 2 << 30 // badger hard cap
func validateEntrySize(k, v []byte) error {
    est := len(k) + len(v) + 64 // key + value + header overhead
    if uint64(est) > uint64(maxVlogFileSize) {
        return fmt.Errorf("entry too large: %d bytes", est)
    }
    return nil
}

func isRequestTooLarge(err error) bool {
    return err != nil && strings.Contains(err.Error(), "bigger than maximum offset")
}

err := db.Set(key, value)
if err != nil && isRequestTooLarge(err) {
    ref, err2 := storeExternal(value)
    if err2 != nil { return err2 }
    return db.Set(key, ref)
}

When it happens

Trigger: db.Set/db.Write/txn.Commit with a single entry or batch whose estimated size exceeds maxVlogFileSize — commonly when opt.ValueLogFileSize was set above the allowed maximum, or a single value is enormous (hundreds of MB+).

Common situations: Users raising ValueLogFileSize beyond the hard cap and pushing one giant blob; storing large files/blobs in badger instead of object storage; building huge write batches that exceed the limit in aggregate.

Related errors

• ErrValueLogSize
• ErrThresholdZero
• ErrNoRewrite
• ErrNamespaceMode
• ErrZeroBandwidth

AI-assisted analysis of dgraph-io/badger@2a001d466f (2026-09-05). Data as JSON: /api/errors/1b3643a57bf2ab5c. Report an issue: GitHub.