What causes "values length is too large, causing overflow" in openimsdk/open-im-server?

Chunk values into batches well below the 2GB limit and call the function per batch

values length is too large, causing overflow

Error message

values length is too large, causing overflow

What it means

LuaSetBatchWithCommonExpire builds a Lua script argument list from the values slice and guards against exceeding the 2GB (32-bit) allocation limit before appending. If len(values) exceeds maxAllowedLen-1 it returns 'values length is too large, causing overflow' instead of risking an out-of-memory/overflow.

Source

Thrown at pkg/common/storage/cache/redis/lua_script.go:77

	}
	v, err := r.Result()
	if errors.Is(err, redis.Nil) {
		err = nil
	}
	return v, errs.WrapMsg(err, "call lua err", "scriptHash", script.Hash(), "keys", keys, "args", args)
}

func LuaSetBatchWithCommonExpire(ctx context.Context, rdb redis.Scripter, keys []string, values []string, expire int) error {
	// Check if the lengths of keys and values match
	if len(keys) != len(values) {
		return errs.New("keys and values length mismatch").Wrap()
	}

	// Ensure allocation size does not overflow
	maxAllowedLen := (1 << 31) - 1 // 2GB limit (maximum address space for 32-bit systems)

	if len(values) > maxAllowedLen-1 {
		return fmt.Errorf("values length is too large, causing overflow")
	}
	var vals = make([]any, 0, 1+len(values))
	vals = append(vals, expire)
	for _, v := range values {
		vals = append(vals, v)
	}
	_, err := callLua(ctx, rdb, setBatchWithCommonExpireScript, keys, vals)
	return err
}

func LuaSetBatchWithIndividualExpire(ctx context.Context, rdb redis.Scripter, keys []string, values []string, expires []int) error {
	// Check if the lengths of keys, values, and expires match
	if len(keys) != len(values) || len(keys) != len(expires) {
		return errs.New("keys and values length mismatch").Wrap()
	}

	// Ensure the allocation size does not overflow
	maxAllowedLen := (1 << 31) - 1 // 2GB limit (maximum address space for 32-bit systems)

View on GitHub (pinned to 175a7bb067)

Solutions

Chunk values into batches well below the 2GB limit and call the function per batch
Cap batch size at the producer/consumer level
Log and reject oversized batches at the caller boundary

Example fix

// before
LuaSetBatchWithCommonExpire(ctx, rdb, key, expire, allValues)
// after
for i := 0; i < len(allValues); i += 10000 {
    end := min(i+10000, len(allValues))
    LuaSetBatchWithCommonExpire(ctx, rdb, key, expire, allValues[i:end])
}

Defensive patterns

Strategy: validation

Validate before calling

const maxBatch = (1 << 31) - 2
if len(values) > maxBatch {
    return fmt.Errorf("batch too large: %d > %d; chunk before calling", len(values), maxBatch)
}

Try / catch

err := LuaSetBatchWithCommonExpire(ctx, rdb, key, expire, values)
if err != nil && strings.Contains(err.Error(), "values length is too large") {
    // split values and retry per chunk
}

Prevention

Chunk bulk writes to a fixed, small batch size (e.g. 10k entries)
Enforce max batch size at the producer
Load-test cache write pipelines for size limits

When it happens

Trigger: Calling LuaSetBatchWithCommonExpire with a values slice whose total length exceeds (1<<31)-2 bytes/elements.

Common situations: Batching an unbounded set of cache writes into one Lua call; upstream producer dumping very large datasets into a single batch; missing chunking in a bulk-load job.

AI-assisted analysis of openimsdk/open-im-server@175a7bb067 (2026-09-04). Data as JSON: /api/errors/694ca007ed78c2b4. Report an issue: GitHub.