{"record":{"id":"694ca007ed78c2b4","repo":"openimsdk/open-im-server","slug":"values-length-is-too-large-causing-overflow","errorCode":null,"errorMessage":"values length is too large, causing overflow","messagePattern":"values length is too large, causing overflow","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/common/storage/cache/redis/lua_script.go","lineNumber":77,"sourceCode":"\t}\n\tv, err := r.Result()\n\tif errors.Is(err, redis.Nil) {\n\t\terr = nil\n\t}\n\treturn v, errs.WrapMsg(err, \"call lua err\", \"scriptHash\", script.Hash(), \"keys\", keys, \"args\", args)\n}\n\nfunc LuaSetBatchWithCommonExpire(ctx context.Context, rdb redis.Scripter, keys []string, values []string, expire int) error {\n\t// Check if the lengths of keys and values match\n\tif len(keys) != len(values) {\n\t\treturn errs.New(\"keys and values length mismatch\").Wrap()\n\t}\n\n\t// Ensure allocation size does not overflow\n\tmaxAllowedLen := (1 << 31) - 1 // 2GB limit (maximum address space for 32-bit systems)\n\n\tif len(values) > maxAllowedLen-1 {\n\t\treturn fmt.Errorf(\"values length is too large, causing overflow\")\n\t}\n\tvar vals = make([]any, 0, 1+len(values))\n\tvals = append(vals, expire)\n\tfor _, v := range values {\n\t\tvals = append(vals, v)\n\t}\n\t_, err := callLua(ctx, rdb, setBatchWithCommonExpireScript, keys, vals)\n\treturn err\n}\n\nfunc LuaSetBatchWithIndividualExpire(ctx context.Context, rdb redis.Scripter, keys []string, values []string, expires []int) error {\n\t// Check if the lengths of keys, values, and expires match\n\tif len(keys) != len(values) || len(keys) != len(expires) {\n\t\treturn errs.New(\"keys and values length mismatch\").Wrap()\n\t}\n\n\t// Ensure the allocation size does not overflow\n\tmaxAllowedLen := (1 << 31) - 1 // 2GB limit (maximum address space for 32-bit systems)","sourceCodeStart":59,"sourceCodeEnd":95,"githubUrl":"https://github.com/openimsdk/open-im-server/blob/175a7bb0673eca18e9d1b10bff4f728da6b1b513/pkg/common/storage/cache/redis/lua_script.go#L59-L95","documentation":"LuaSetBatchWithCommonExpire builds a Lua script argument list from the values slice and guards against exceeding the 2GB (32-bit) allocation limit before appending. If len(values) exceeds maxAllowedLen-1 it returns 'values length is too large, causing overflow' instead of risking an out-of-memory/overflow.","triggerScenarios":"Calling LuaSetBatchWithCommonExpire with a values slice whose total length exceeds (1<<31)-2 bytes/elements.","commonSituations":"Batching an unbounded set of cache writes into one Lua call; upstream producer dumping very large datasets into a single batch; missing chunking in a bulk-load job.","solutions":["Chunk values into batches well below the 2GB limit and call the function per batch","Cap batch size at the producer/consumer level","Log and reject oversized batches at the caller boundary"],"exampleFix":"// before\nLuaSetBatchWithCommonExpire(ctx, rdb, key, expire, allValues)\n// after\nfor i := 0; i < len(allValues); i += 10000 {\n    end := min(i+10000, len(allValues))\n    LuaSetBatchWithCommonExpire(ctx, rdb, key, expire, allValues[i:end])\n}","handlingStrategy":"validation","validationCode":"const maxBatch = (1 << 31) - 2\nif len(values) > maxBatch {\n    return fmt.Errorf(\"batch too large: %d > %d; chunk before calling\", len(values), maxBatch)\n}","typeGuard":null,"tryCatchPattern":"err := LuaSetBatchWithCommonExpire(ctx, rdb, key, expire, values)\nif err != nil && strings.Contains(err.Error(), \"values length is too large\") {\n    // split values and retry per chunk\n}","preventionTips":["Chunk bulk writes to a fixed, small batch size (e.g. 10k entries)","Enforce max batch size at the producer","Load-test cache write pipelines for size limits"],"tags":["redis","lua","memory-limit","batching"],"backgroundTag":"batch-size-overflow","analyzedSha":"175a7bb0673eca18e9d1b10bff4f728da6b1b513","analyzedAt":"2026-09-04T16:52:56.821Z","contentChangedAt":"2026-09-04T16:52:56.821Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}