What causes "Failed to flush batch: " in apache/seatunnel?

Read the cause (e) message — it identifies whether conversion, commit, or reopen failed; fix the offending row/schema mismatch first.

apache/seatunnel · critical · LanceConnectorException

TABLE_DATASET_WRITE_ST_ROW_EXCEPTION

TABLE_DATASET_WRITE_ST_ROW_EXCEPTION

Error message

Failed to flush batch:

What it means

LanceSinkWriter.flushBatch converts buffered SeaTunnelRows to Arrow fragments (FragmentConverter.reconvert), commits them as an Append transaction, then reopens the dataset. Any failure in row conversion, transaction commit, or reopen is wrapped as LanceConnectorException with code TABLE_DATASET_WRITE_ST_ROW_EXCEPTION. It means writing the buffered batch of rows to the Lance dataset failed.

Source

Thrown at seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/sink/LanceSinkWriter.java:171

                                .operation(Append.builder().fragments(allFragments).build())
                                .build();

                try (Dataset appendedDataset = transaction.commit()) {
                    log.debug(
                            "Flushed {} rows to lance dataset, new version: {}",
                            batchBuffer.size(),
                            appendedDataset.version());
                }

                if (dataset != null) {
                    dataset.close();
                }
                dataset = Dataset.open(config.getDatasetPath(), allocator);
            }

            batchBuffer.clear();
        } catch (Exception e) {
            throw new LanceConnectorException(
                    LanceConnectorErrorCode.TABLE_DATASET_WRITE_ST_ROW_EXCEPTION,
                    "Failed to flush batch: " + e.getMessage(),
                    e);
        }
    }

    @Override
    public void applySchemaChange(SchemaChangeEvent event) throws IOException {
        SinkWriter.super.applySchemaChange(event);
    }

    @Override
    public Optional<LanceCommitInfo> prepareCommit() throws IOException {
        flushBatch();
        return Optional.empty();
    }

    @Override

View on GitHub (pinned to cf67b549a7)

Solutions

Read the cause (e) message — it identifies whether conversion, commit, or reopen failed; fix the offending row/schema mismatch first.
Compare incoming SeaTunnelRowType fields with the dataset schema (field names, types, nullability) and align them.
Ensure only one writer appends to a dataset at a time or reduce concurrency; retry the job after a commit conflict.
Check storage availability/permissions again at flush time; verify the dataset path was not modified by another process.

Example fix

// before
// row column 'age' sent as string while dataset field is int
// after
// cast/validate in the transform stage: cast(age as int) before the Lance sink

Defensive patterns

Strategy: validation

Validate before calling

// before writing, validate row values against dataset schema types/nullability
for (int i = 0; i < rowType.getTotalFields(); i++) {
    Object v = row.getField(i);
    if (v == null && !schema.findField(rowType.getFieldName(i)).getFieldType().isNullable()) throw new IllegalStateException("null in non-nullable field " + rowType.getFieldName(i));
}

Try / catch

try { writer.write(row); } catch (LanceConnectorException e) { if (e.getErrorCode() == LanceConnectorErrorCode.TABLE_DATASET_WRITE_ST_ROW_EXCEPTION) { log.error("batch flush failed: {}", e.getCause(), e); } throw e; }

Prevention

Keep the source SeaTunnelRowType schema in sync with the Lance dataset schema (names, types, nullability).
Avoid multiple concurrent writers appending to the same dataset.
Cast mismatched types (e.g. string dates/numbers) in transforms before the sink.
Monitor for schema-change events from upstream and handle them via applySchemaChange.

When it happens

Trigger: Called from write() when the batch buffer reaches batchSize, from prepareCommit(), or close(). Fails when a row's data does not match the dataset Arrow schema (type/null violations), the Append transaction commit conflicts or fails (e.g. dataset changed concurrently, or dataset handle invalid), or Dataset.open after commit fails.

Common situations: Schema drift between source rows and the Lance dataset (new column, changed type); nulls written to non-nullable fields; concurrent writers appending to the same dataset causing commit conflicts; dataset file deleted/compacted underneath the writer between flushes.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/f02aade2674be99b. Report an issue: GitHub.