{"record":{"id":"f2346242447bf17c","repo":"apache/seatunnel","slug":"write-data-fail","errorCode":"WRITE_DATA_FAIL","errorMessage":"upsert data failed, size down to 10, break","messagePattern":"upsert data failed, size down to 10, break","errorType":"error_code","errorClass":"MilvusConnectorException","httpStatus":null,"severity":"error","filePath":"seatunnel-connectors-v2/connector-milvus/src/main/java/org/apache/seatunnel/connectors/seatunnel/milvus/sink/MilvusBufferBatchWriter.java","lineNumber":302,"sourceCode":"            milvusClient.upsert(upsertReq);\n        } catch (Exception e) {\n            if (e.getMessage().contains(\"rate limit exceeded\")\n                    || e.getMessage().contains(\"received message larger than max\")) {\n                if (data.size() > 10) {\n                    log.warn(\"upsert data failed, retry in smaller chunks: {} \", data.size() / 2);\n                    this.batchSize = this.batchSize / 2;\n                    log.info(\"sleep 1 minute to avoid rate limit\");\n                    // sleep 1 minute to avoid rate limit\n                    Thread.sleep(60000);\n                    log.info(\"sleep 1 minute success\");\n                    // Split the data and retry in smaller chunks\n                    List<JsonObject> firstHalf = data.subList(0, data.size() / 2);\n                    List<JsonObject> secondHalf = data.subList(data.size() / 2, data.size());\n                    upsertWrite(partitionName, firstHalf);\n                    upsertWrite(partitionName, secondHalf);\n                } else {\n                    // If the data size is 10, throw the exception to avoid infinite recursion\n                    throw new MilvusConnectorException(\n                            MilvusConnectionErrorCode.WRITE_DATA_FAIL,\n                            \"upsert data failed,\" + \" size down to 10, break\",\n                            e);\n                }\n            } else {\n                throw new MilvusConnectorException(\n                        MilvusConnectionErrorCode.WRITE_DATA_FAIL,\n                        \"upsert data failed with unknown exception\",\n                        e);\n            }\n        }\n        log.info(\"upsert data success\");\n    }\n\n    private void insertWrite(String partitionName, List<JsonObject> data) {\n        InsertReq insertReq =\n                InsertReq.builder().collectionName(this.collectionName).data(data).build();\n        if (StringUtils.isNotEmpty(partitionName)) {","sourceCodeStart":284,"sourceCodeEnd":320,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-milvus/src/main/java/org/apache/seatunnel/connectors/seatunnel/milvus/sink/MilvusBufferBatchWriter.java#L284-L320","documentation":"MilvusBufferBatchWriter.upsertWrite retries a failed upsert by splitting the batch in half recursively. If the batch has already shrunk to 10 rows and the upsert still fails, further splitting is pointless, so it throws MilvusConnectorException with WRITE_DATA_FAIL ('size down to 10, break') including the underlying cause e.","triggerScenarios":"A Milvus upsert call fails persistently (bad PK, dimension mismatch, server error) even when the batch is reduced to ~10 rows; the recursive halving path reaches the minimum size and rethrows instead of recursing again.","commonSituations":"Individual bad rows (e.g. primary key duplicates or wrong vector dimension) that fail regardless of batch size; Milvus server-side memory/segment errors on flush; quota or collection state (e.g. loaded/locked) issues causing consistent upsert rejection.","solutions":["Inspect the wrapped cause e for the actual Milvus error (dimension mismatch, PK conflict, collection not loaded).","Validate vector dimensions against the collection schema and deduplicate primary keys in the data.","Check Milvus server health/compaction state and retry the job after fixing server-side issues.","If a few rows are bad, isolate them: write smaller batches or filter/skip failing records upstream."],"exampleFix":"// before\n// rows with dimension 768 written into collection created with dim 1024\n// after\n// align data: generate/store 1024-dim vectors, or recreate collection with dim=768","handlingStrategy":"try-catch","validationCode":"// pre-validate rows before write\nrows.forEach(r -> {\n    if (r.vectorSize() != collectionDim) throw new IllegalArgumentException(\"Vector dim mismatch\");\n});\nList<Object> pks = rows.stream().map(Row::pk).collect(toList());\nif (new HashSet<>(pks).size() != pks.size()) throw new IllegalArgumentException(\"Duplicate primary keys in batch\");","typeGuard":null,"tryCatchPattern":"try {\n    writer.write(rows);\n} catch (MilvusConnectorException e) {\n    if (MilvusConnectionErrorCode.WRITE_DATA_FAIL.equals(e.getSeaTunnelErrorCode())\n        && e.getMessage().contains(\"size down to 10\")) {\n        log.error(\"Persistent upsert failure, inspect cause: {}\", e.getCause());\n        // route to dead-letter queue for row-level triage\n    } else { throw e; }\n}","preventionTips":["Ensure vector dimensions match the collection schema","Deduplicate primary keys before batching","Monitor Milvus server health/segment state for persistent upsert failures"],"tags":["milvus","sink","upsert","batch","write"],"backgroundTag":"database-write-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}