apache/flink · error · FlinkRuntimeException

Could not create CSV generator.

Error message

Could not create CSV generator.

What it means

Thrown by the CsvBulkWriter constructor when Jackson's CsvMapper fails to create a JsonGenerator over the output stream (mapper.writer(schema).createGenerator(stream, UTF8)). It is a setup-time FlinkRuntimeException: the writer could not be built, so no bulk write can proceed. Almost always the cause is an I/O problem on the target stream (closed/null-underlying FS stream, permissions) rather than CSV-specific logic.

Source

Thrown at flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java:66

            CsvMapper mapper,
            CsvSchema schema,
            Converter<T, R, C> converter,
            @Nullable C converterContext,
            FSDataOutputStream stream) {
        checkNotNull(mapper);
        checkNotNull(schema);

        // Prevent Jackson's writeValue() method calls from closing the stream.
        mapper.getFactory().disable(JsonGenerator.Feature.AUTO_CLOSE_TARGET);
        mapper.disable(SerializationFeature.FLUSH_AFTER_WRITE_VALUE);

        this.converter = checkNotNull(converter);
        this.stream = checkNotNull(stream);
        this.converterContext = converterContext;
        try {
            this.generator = mapper.writer(schema).createGenerator(stream, JsonEncoding.UTF8);
        } catch (IOException e) {
            throw new FlinkRuntimeException("Could not create CSV generator.", e);
        }
    }

    /**
     * Builds a writer with Jackson schema and a type converter.
     *
     * @param mapper The specialized mapper for producing CSV.
     * @param schema The schema that defined the mapping properties.
     * @param converter The type converter that converts incoming elements of type {@code <T>} into
     *     elements of type JsonNode.
     * @param stream The output stream.
     * @param <T> The type of the elements accepted by this writer.
     * @param <C> The type of the converter context.
     * @param <R> The type of the elements produced by this writer.
     */
    static <T, R, C> CsvBulkWriter<T, R, C> forSchema(
            CsvMapper mapper,
            CsvSchema schema,

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Check the attached cause (IOException) — it names the real stream/FS failure.
2. Verify the output location is writable and the filesystem credential/endpoint still valid (e.g. s3a connection test).
3. If the stream is created elsewhere, ensure it is open and not already consumed/closed before CsvBulkWriter construction.
4. Retry the job from checkpoint after fixing the FS issue; this is a transient/environmental failure, not a data error.

// Validate the stream is usable before constructing the writer:
if (stream == null) throw new IllegalStateException("output stream is null");
try { stream.flush(); } catch (IOException e) { /* fail before writer construction with a clear message */ }

catch (FlinkRuntimeException e) { if (e.getCause() instanceof IOException) { /* transient FS failure: restart from last checkpoint after verifying object-store access */ } throw e; }

When it happens

Trigger: Constructing CsvBulkWriter via CsvBulkWriter.forSchema/forFailureHandler on an output stream that is already closed or whose underlying filesystem endpoint failed at open; interceptors wrapping streams that throw on first write-header; charset/encoding issues on exotic platforms.

Common situations: Filesystem connector sinks where the stream handed to BulkWriter.Factory points at an unreachable object store (S3/OSS credentials expired between stream creation and writer creation); test harnesses passing mocked streams that reject writes.

Related errors

• An I/O error occurred while creating temporary file to extra
• Failed to create parent(s) for given base dir: %s
• Compaction file not exist: {path}
• An error occurred while copying the file.
• Wrapped InputStream: cannot search backwards.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/7eebc454c8fddbf2. Report an issue: GitHub.