apache/seatunnel · error · MilvusConnectorException

WRITE_DATA_FAIL

WRITE_DATA_FAIL

Error message

upsert data failed, size down to 10, break

What it means

MilvusBufferBatchWriter.upsertWrite retries a failed upsert by splitting the batch in half recursively. If the batch has already shrunk to 10 rows and the upsert still fails, further splitting is pointless, so it throws MilvusConnectorException with WRITE_DATA_FAIL ('size down to 10, break') including the underlying cause e.

Source

Thrown at seatunnel-connectors-v2/connector-milvus/src/main/java/org/apache/seatunnel/connectors/seatunnel/milvus/sink/MilvusBufferBatchWriter.java:302

            milvusClient.upsert(upsertReq);
        } catch (Exception e) {
            if (e.getMessage().contains("rate limit exceeded")
                    || e.getMessage().contains("received message larger than max")) {
                if (data.size() > 10) {
                    log.warn("upsert data failed, retry in smaller chunks: {} ", data.size() / 2);
                    this.batchSize = this.batchSize / 2;
                    log.info("sleep 1 minute to avoid rate limit");
                    // sleep 1 minute to avoid rate limit
                    Thread.sleep(60000);
                    log.info("sleep 1 minute success");
                    // Split the data and retry in smaller chunks
                    List<JsonObject> firstHalf = data.subList(0, data.size() / 2);
                    List<JsonObject> secondHalf = data.subList(data.size() / 2, data.size());
                    upsertWrite(partitionName, firstHalf);
                    upsertWrite(partitionName, secondHalf);
                } else {
                    // If the data size is 10, throw the exception to avoid infinite recursion
                    throw new MilvusConnectorException(
                            MilvusConnectionErrorCode.WRITE_DATA_FAIL,
                            "upsert data failed," + " size down to 10, break",
                            e);
                }
            } else {
                throw new MilvusConnectorException(
                        MilvusConnectionErrorCode.WRITE_DATA_FAIL,
                        "upsert data failed with unknown exception",
                        e);
            }
        }
        log.info("upsert data success");
    }

    private void insertWrite(String partitionName, List<JsonObject> data) {
        InsertReq insertReq =
                InsertReq.builder().collectionName(this.collectionName).data(data).build();
        if (StringUtils.isNotEmpty(partitionName)) {

View on GitHub (pinned to cf67b549a7)

Solutions

  1. Inspect the wrapped cause e for the actual Milvus error (dimension mismatch, PK conflict, collection not loaded).
  2. Validate vector dimensions against the collection schema and deduplicate primary keys in the data.
  3. Check Milvus server health/compaction state and retry the job after fixing server-side issues.
  4. If a few rows are bad, isolate them: write smaller batches or filter/skip failing records upstream.

Example fix

// before
// rows with dimension 768 written into collection created with dim 1024
// after
// align data: generate/store 1024-dim vectors, or recreate collection with dim=768
Defensive patterns

Strategy: try-catch

Validate before calling

// pre-validate rows before write
rows.forEach(r -> {
    if (r.vectorSize() != collectionDim) throw new IllegalArgumentException("Vector dim mismatch");
});
List<Object> pks = rows.stream().map(Row::pk).collect(toList());
if (new HashSet<>(pks).size() != pks.size()) throw new IllegalArgumentException("Duplicate primary keys in batch");

Try / catch

try {
    writer.write(rows);
} catch (MilvusConnectorException e) {
    if (MilvusConnectionErrorCode.WRITE_DATA_FAIL.equals(e.getSeaTunnelErrorCode())
        && e.getMessage().contains("size down to 10")) {
        log.error("Persistent upsert failure, inspect cause: {}", e.getCause());
        // route to dead-letter queue for row-level triage
    } else { throw e; }
}

Prevention

When it happens

Trigger: A Milvus upsert call fails persistently (bad PK, dimension mismatch, server error) even when the batch is reduced to ~10 rows; the recursive halving path reaches the minimum size and rethrows instead of recursing again.

Common situations: Individual bad rows (e.g. primary key duplicates or wrong vector dimension) that fail regardless of batch size; Milvus server-side memory/segment errors on flush; quota or collection state (e.g. loaded/locked) issues causing consistent upsert rejection.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/f2346242447bf17c. Report an issue: GitHub.