What causes "Failed to create dataset: " in apache/seatunnel?

Check the 'dataset-path' value: it must be a valid Lance URI (absolute local path or s3://...), and the create exception message names the real cause.

apache/seatunnel · critical · LanceConnectorException

TABLE_DATASET_PATH_OPEN_EXCEPTION

TABLE_DATASET_PATH_OPEN_EXCEPTION

Error message

Failed to create dataset:

What it means

LanceSinkWriter.initializeDataset tries Dataset.open on the configured dataset path; if that fails it attempts to create the dataset with Dataset.create and reopen it. If the create/reopen path throws (bad URI, storage auth failure, existing incompatible dataset, invalid write params), the error is wrapped as LanceConnectorException with code TABLE_DATASET_PATH_OPEN_EXCEPTION. It means the sink could not initialize the target Lance dataset at datasetPath.

Source

Thrown at seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/sink/LanceSinkWriter.java:114

        } catch (Exception e) {
            this.schema = SchemaUtils.convertSchema(firstElement, seaTunnelRowType);

            try {
                Dataset.create(
                        allocator,
                        config.getDatasetPath(),
                        schema,
                        new WriteParams.Builder()
                                .withMaxBytesPerFile(config.getMaxBytesPerFile())
                                .withMaxRowsPerFile(config.getMaxRowsPerFile())
                                .withMode(config.getMode())
                                .withStorageOptions(config.getStorageOptions())
                                .build());

                this.dataset = Dataset.open(config.getDatasetPath(), allocator);
                datasetInitialized = true;
            } catch (Exception createEx) {
                throw new LanceConnectorException(
                        LanceConnectorErrorCode.TABLE_DATASET_PATH_OPEN_EXCEPTION,
                        "Failed to create dataset: " + createEx.getMessage(),
                        createEx);
            }
        }
    }

    @Override
    public void write(SeaTunnelRow element) throws IOException {
        if (!datasetInitialized) {
            initializeDataset(element);
        }

        batchBuffer.add(element);

        if (batchBuffer.size() >= batchSize) {
            flushBatch();
        }

View on GitHub (pinned to cf67b549a7)

Solutions

Check the 'dataset-path' value: it must be a valid Lance URI (absolute local path or s3://...), and the create exception message names the real cause.
Add/verify storage options (endpoint, access key, secret, region) in the sink config so Dataset.create can reach the object store.
Verify write permissions on the target location/bucket.
If the dataset exists but open failed, compare its Arrow schema with the incoming SeaTunnelRowType for incompatibilities; recreate the dataset or align schemas.

Example fix

// before
"lance.dataset-path" = "my-bucket/tables/users"  // no scheme, storage options missing
// after
"lance.dataset-path" = "s3://my-bucket/tables/users"
"lance.storage.options" = { "s3.endpoint": "...", "s3.access_key_id": "...", "s3.secret_access_key": "..." }

Defensive patterns

Strategy: try-catch

Validate before calling

// validate config before job submission
assert config.getDatasetPath() != null && (config.getDatasetPath().startsWith("s3://") || new java.io.File(config.getDatasetPath()).isAbsolute());
assert config.getStorageOptions() != null; // when path is remote

Try / catch

try { writer.write(row); } catch (LanceConnectorException e) { if (e.getErrorCode() == LanceConnectorErrorCode.TABLE_DATASET_PATH_OPEN_EXCEPTION) { log.error("dataset init failed at {}: {}", datasetPath, e.getCause(), e); throw e; } }

Prevention

Use fully-qualified URIs (s3://bucket/path or absolute local path) for dataset-path.
Always supply storage options (endpoint, keys, region) for remote paths.
Pre-create the dataset via catalog.createTable so initializeDataset takes the open path.
Verify worker-level access to the object store before launching the job.

When it happens

Trigger: First write() triggers initializeDataset: Dataset.open fails (dataset absent) and the fallback Dataset.create/Dataset.open also fails — e.g. invalid or non-URI datasetPath, missing storage options (S3 credentials/endpoint), object store returning access denied, createMode conflicting with an existing dataset, or corrupt/incompatible existing schema.

Common situations: Typo or missing scheme in the 'lance.dataset-path' config (local path vs s3://); storage options omitted in environments where the Lance namespace required them; job restarted against a dataset created with a different schema; bucket/credentials not configured for the worker pods.

Understand the failure class

Background: "open() failed", "failed to open file", "cannot create file" — what a file open error means and how to fix it — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/7e02c96980e0f581. Report an issue: GitHub.