What causes "Failed to close dataset: {}" in apache/seatunnel?

Check the logged exception message for the root cause (often a JNI or storage error)

Failed to close dataset: {}

Error message

Failed to close dataset: {}

What it means

LanceSinkWriter.close flushes pending batches and then closes the underlying Lance dataset. If dataset.close() throws (native/JNI resource release failure), the writer logs this warning with the exception message and continues cleanup (nulling the reference and closing the allocator). Data already flushed is unaffected, but the failure may indicate a leaked or corrupted native handle.

Source

Thrown at seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/sink/LanceSinkWriter.java:203

        flushBatch();
        return Optional.empty();
    }

    @Override
    public void abortPrepare() {
        batchBuffer.clear();
    }

    @Override
    public void close() throws IOException {
        try {
            flushBatch();
        } finally {
            if (dataset != null) {
                try {
                    dataset.close();
                } catch (Exception e) {
                    log.warn("Failed to close dataset: {}", e.getMessage());
                }
                dataset = null;
            }

            if (allocator != null) {
                try {
                    allocator.close();
                } catch (Exception e) {
                    log.warn("Failed to close allocator: {}", e.getMessage());
                }
                allocator = null;
            }
        }
    }
}

View on GitHub (pinned to cf67b549a7)

Solutions

Check the logged exception message for the root cause (often a JNI or storage error)
Ensure close() is not called twice on the same writer (task lifecycle guard)
Upgrade the lance JNI library if the message indicates a native crash or leak
Verify the storage backend (S3/OSS/local disk) is healthy and reachable at close time

Example fix

// before
writer.close();
writer.close(); // second close logs 'Failed to close dataset'
// after
if (!closed) {
    writer.close();
    closed = true;
}

Defensive patterns

Strategy: try-catch

Validate before calling

// guard against double close
if (writerClosed) return;

Try / catch

try { writer.close(); } catch (Exception e) { log.warn("Lance writer close issue: {}", e.getMessage()); }

Prevention

Close each writer exactly once per task lifecycle
Ensure flush succeeds before interpreting close warnings as data loss
Monitor storage backend health at teardown

When it happens

Trigger: Closing the sink writer when the Lance JNI dataset handle is already closed, corrupted, or the underlying object store transaction fails during close, after flushBatch has run.

Common situations: Double-close after a task failure/retry; JVM shutdown while native memory is pinned; transient storage backend errors during dataset teardown.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/12dd3c04c1cfaea3. Report an issue: GitHub.