apache/flink · error · IOException

Compaction file not exist: {path}

Error message

Compaction file not exist: {path}

What it means

Thrown by CompactFileUtils.checkExist when one or more compaction output files cannot be found on the filesystem after the compact writer has committed them. This wraps an IOException indicating that a file the system just wrote is missing, which typically points to a filesystem-level problem rather than a Flink logic bug.

Source

Thrown at flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/utils/CompactFileUtils.java:157

            try (CompactReader<T> reader =
                    readerFactory.create(
                            CompactContext.create(config, fileSystem, partition, path))) {
                T record;
                while ((record = reader.read()) != null) {
                    writer.write(record);
                }
            }
        }

        // commit immediately
        writer.commit();
    }

    private static void checkExist(FileSystem fileSystem, List<Path> candidates)
            throws IOException {
        for (Path path : candidates) {
            if (!fileSystem.exists(path)) {
                throw new IOException("Compaction file not exist: " + path);
            }
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. If using S3, ensure s3a with consistent reads (S3 strong consistency is now default, but older emrfs or custom configs may lag).
2. Verify no external process or concurrent job is deleting compaction output files.
3. Check filesystem permissions and that the output directory is writable and stable.
4. Retry the job from the last successful checkpoint; transient filesystem hiccups often resolve on retry.

try {
    CompactFileUtils.doCompact(fileSystem, candidates, output);
} catch (IOException e) {
    if (e.getMessage().startsWith("Compaction file not exist")) {
        LOG.warn("Compaction output file missing — possible filesystem inconsistency: {}", e.getMessage());
        // retry from checkpoint or alert operations
    }
    throw e;
}

When it happens

Trigger: After CompactFileUtils writes compacted files and calls writer.commit(), the subsequent checkExist loop finds a Path that fileSystem.exists() returns false for.

Common situations: Eventually-consistent object stores (e.g. S3 in older configurations) where a newly written object is not yet visible; concurrent external deletion of the output directory; filesystem permissions or mount issues; race conditions on distributed filesystems.

Related errors

• An I/O error occurred while creating temporary file to extra
• Failed to create parent(s) for given base dir: %s
• Unsupported input message: {value}
• An error occurred while copying the file.
• Wrapped InputStream: cannot search backwards.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/1880c39f91359eb8. Report an issue: GitHub.