{"record":{"id":"142a2cd9e3263e82","repo":"apache/seatunnel","slug":"insert-data-failed-retry-in-smaller-chunks","errorCode":null,"errorMessage":"insert data failed, retry in smaller chunks: {} ","messagePattern":"insert data failed, retry in smaller chunks: (.+?) ","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"seatunnel-connectors-v2/connector-milvus/src/main/java/org/apache/seatunnel/connectors/seatunnel/milvus/sink/MilvusBufferBatchWriter.java","lineNumber":329,"sourceCode":"                        e);\n            }\n        }\n        log.info(\"upsert data success\");\n    }\n\n    private void insertWrite(String partitionName, List<JsonObject> data) {\n        InsertReq insertReq =\n                InsertReq.builder().collectionName(this.collectionName).data(data).build();\n        if (StringUtils.isNotEmpty(partitionName)) {\n            insertReq.setPartitionName(partitionName);\n        }\n        try {\n            milvusClient.insert(insertReq);\n        } catch (Exception e) {\n            if (e.getMessage().contains(\"rate limit exceeded\")\n                    || e.getMessage().contains(\"received message larger than max\")) {\n                if (data.size() > 10) {\n                    log.warn(\"insert data failed, retry in smaller chunks: {} \", data.size() / 2);\n                    // Split the data and retry in smaller chunks\n                    List<JsonObject> firstHalf = data.subList(0, data.size() / 2);\n                    List<JsonObject> secondHalf = data.subList(data.size() / 2, data.size());\n                    this.batchSize = this.batchSize / 2;\n                    insertWrite(partitionName, firstHalf);\n                    insertWrite(partitionName, secondHalf);\n                } else {\n                    // If the data size is 10, throw the exception to avoid infinite recursion\n                    throw new MilvusConnectorException(\n                            MilvusConnectionErrorCode.WRITE_DATA_FAIL, \"insert data failed\", e);\n                }\n            } else {\n                throw new MilvusConnectorException(\n                        MilvusConnectionErrorCode.WRITE_DATA_FAIL,\n                        \"insert data failed with unknown exception\",\n                        e);\n            }\n        }","sourceCodeStart":311,"sourceCodeEnd":347,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-milvus/src/main/java/org/apache/seatunnel/connectors/seatunnel/milvus/sink/MilvusBufferBatchWriter.java#L311-L347","documentation":"An insert to Milvus failed with a rate-limit or message-too-large error. Like the upsert path, the writer halves the batch size, splits the data into two halves, and recursively retries each half — without the 60-second sleep used in upsertWrite.","triggerScenarios":"insertWrite catches an exception whose message contains 'rate limit exceeded' or 'received message larger than max' while data.size() > 10.","commonSituations":"Large initial batches exceeding Milvus message size limits; bursty inserts exceeding Milvus insert rate quotas.","solutions":["Set a smaller batch_size in the Milvus sink config.","Raise Milvus rate limits or proxy max message size on the server.","Throttle source parallelism to reduce insert pressure.","Note this path retries immediately (no backoff sleep); if rate limits persist, add server-side or upstream throttling."],"exampleFix":"// before\nsink {\n  Milvus {\n    batch_size = 10000\n  }\n}\n// after\nsink {\n  Milvus {\n    batch_size = 500\n  }\n}","handlingStrategy":"retry","validationCode":"// keep inserts below max message size\nif (estimateBytes(data) > MAX_GRPC_MSG) { splitAndInsert(data); }","typeGuard":null,"tryCatchPattern":"try { client.insert(req); }\ncatch (Exception e) {\n    if (isRateLimitOrTooLarge(e)) { insertHalves(data); }\n    else throw e;\n}","preventionTips":["Configure batch_size well below Milvus proxy maxMessageSize","Increase server maxMessageSize if batches are legitimately large","Throttle source parallelism on rate-limited Milvus deployments","Remember the insert path retries immediately — add external backoff if needed"],"tags":["milvus","rate-limit","retry","insert"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}