apache/flink · error · IllegalStateException

A stream against this file was already created.

Error message

A stream against this file was already created.

What it means

StreamOutputFile wraps an FSDataOutputStream and enforces single-use semantics: create() may be called exactly once, guarded by an AtomicBoolean compareAndSet. A second call throws IllegalStateException because the underlying stream cannot be reopened or repositioned.

Source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/StreamOutputFile.java:63

    private final AtomicBoolean used;

    /**
     * Creates a new StreamOutputFile. The first call to {@link #create(long)} or {@link
     * #createOrOverwrite(long)} returns a stream that writes to the given stream.
     *
     * @param stream The stream to write to.
     */
    StreamOutputFile(FSDataOutputStream stream) {
        this.stream = checkNotNull(stream);
        this.used = new AtomicBoolean(false);
    }

    @Override
    public PositionOutputStream create(long blockSizeHint) {
        if (used.compareAndSet(false, true)) {
            return new PositionOutputStreamAdapter(stream);
        } else {
            throw new IllegalStateException("A stream against this file was already created.");
        }
    }

    @Override
    public PositionOutputStream createOrOverwrite(long blockSizeHint) {
        return create(blockSizeHint);
    }

    @Override
    public boolean supportsBlockSize() {
        return false;
    }

    @Override
    public long defaultBlockSize() {
        return DEFAULT_BLOCK_SIZE;
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Create a fresh StreamOutputFile (or a new output stream via the FileSystem) for each write attempt
2. For retry logic, re-resolve the OutputFile from the RecoverableWriter/FileSystem rather than caching it
3. Note createOrOverwrite() has the same restriction here - do not rely on overwrite semantics on a stream-backed file

Example fix

// before
StreamOutputFile f = new StreamOutputFile(stream);
f.create(0);
f.create(0); // throws

// after
try (PositionOutputStream out = new StreamOutputFile(stream).create(0)) {
    // write
}
// need another write? open a NEW stream + StreamOutputFile

// treat OutputFile as one-shot; never call create twice
if (outputFileCreated) { throw new IllegalStateException("reuse"); }

try { out = outputFile.create(blockSize); } catch (IllegalStateException e) { // re-open the file via FileSystem and retry once with a NEW OutputFile }

When it happens

Trigger: Calling create() or createOrOverwrite() (which delegates to create()) twice on the same StreamOutputFile instance.

Common situations: Retries in sink code that reuse the same OutputFile, or a committing/rolling policy that attempts to overwrite the same wrapped stream instead of opening a new file.

Related errors

• Only support seek at first.
• Please use AvroParquetReaders.forSpecificRecord(Class<T>) fo
• Protocol buffer class or descriptor not specified. Please us
• Parquet with case insensitive mode should have no duplicate
• The quality of field type is incompatible with the request s

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/3db575b25204a90c. Report an issue: GitHub.