apache/seatunnel · critical · FileConnectorException

WRITER_OPERATION_FAILED

WRITER_OPERATION_FAILED

Error message

Get parquet writer for file [%s] error

What it means

ParquetWriteStrategy.getOrCreateOutputStream creates/obtains a ParquetWriter for the target file. When constructing the writer throws IOException (disk full, missing parent directory, permission problem, bad HDFS path), it is wrapped as FileConnectorException with WRITER_OPERATION_FAILED and a message naming the file. It indicates the sink could not open a Parquet writer, not a data-format problem.

Source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/ParquetWriteStrategy.java:222

                            ParquetWriter<GenericRecord> newWriter =
                                    AvroParquetWriter.<GenericRecord>builder(outputFile)
                                            .withWriteMode(ParquetFileWriter.Mode.OVERWRITE)
                                            .withDataModel(dataModel)
                                            .withConf(configuration)
                                            // use parquet v1 to improve compatibility
                                            .withWriterVersion(
                                                    ParquetProperties.WriterVersion.PARQUET_1_0)
                                            .withCompressionCodec(
                                                    compressFormat.getParquetCompression())
                                            .withSchema(schema)
                                            .build();
                            this.beingWrittenWriter.put(filePath, newWriter);
                            return newWriter;
                        } catch (IOException e) {
                            String errorMsg =
                                    String.format(
                                            "Get parquet writer for file [%s] error", filePath);
                            throw new FileConnectorException(
                                    CommonErrorCodeDeprecated.WRITER_OPERATION_FAILED, errorMsg, e);
                        }
                    });
        }
        return writer;
    }

    private Object resolveObject(String name, Object data, SeaTunnelDataType<?> seaTunnelDataType) {
        if (data == null) {
            return null;
        }
        switch (seaTunnelDataType.getSqlType()) {
            case ARRAY:
                SeaTunnelDataType<?> elementType =
                        ((ArrayType<?, ?>) seaTunnelDataType).getElementType();
                ArrayList<Object> records = new ArrayList<>(((Object[]) data).length);
                for (Object object : (Object[]) data) {
                    Object resolvedObject = resolveObject(name, object, elementType);

View on GitHub (pinned to cf67b549a7)

Solutions

  1. Check the wrapped cause (e) for the root IOException — permission, space, or connectivity
  2. Verify the sink path exists and the running user has write permission to the target directory
  3. Check disk space / HDFS quota on the worker nodes
  4. Confirm HDFS/S3 connectivity from all worker nodes and retry the job after fixing storage

Example fix

# before (typical failure)
sink.path = hdfs://nn/data/out  # dir not writable
# after
hdfs dfs -chmod -R 775 /data/out   # or fix the configured path/credentials
Defensive patterns

Strategy: try-catch

Validate before calling

Files dirs = new Files(Paths.get(path)); if (!dirs.exists() || !dirs.canWrite()) { fail fast before job submit }

Try / catch

try { sinkWriter.write(row); } catch (FileConnectorException e) { if (e.getErrorCode() == WRITER_OPERATION_FAILED) { log(e.getCause()); /* check storage, then retry job */ } }

Prevention

When it happens

Trigger: Calling writer() on the Parquet write strategy when a new output file must be rolled (file size threshold, checkpoint rollover) and the underlying filesystem fails to create the file or the writer.

Common situations: Disk full on local FS/HDFS; target directory deleted or not writable; HDFS name-node unavailable or in safe mode; incorrect path (permission denied, quota exceeded); transient network errors to remote storage.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/4c07606233fd576a. Report an issue: GitHub.