{"record":{"id":"1880c39f91359eb8","repo":"apache/flink","slug":"compaction-file-not-exist-path","errorCode":null,"errorMessage":"Compaction file not exist: {path}","messagePattern":"Compaction file not exist: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/utils/CompactFileUtils.java","lineNumber":157,"sourceCode":"            try (CompactReader<T> reader =\n                    readerFactory.create(\n                            CompactContext.create(config, fileSystem, partition, path))) {\n                T record;\n                while ((record = reader.read()) != null) {\n                    writer.write(record);\n                }\n            }\n        }\n\n        // commit immediately\n        writer.commit();\n    }\n\n    private static void checkExist(FileSystem fileSystem, List<Path> candidates)\n            throws IOException {\n        for (Path path : candidates) {\n            if (!fileSystem.exists(path)) {\n                throw new IOException(\"Compaction file not exist: \" + path);\n            }\n        }\n    }\n}\n","sourceCodeStart":139,"sourceCodeEnd":162,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/utils/CompactFileUtils.java#L139-L162","documentation":"Thrown by CompactFileUtils.checkExist when one or more compaction output files cannot be found on the filesystem after the compact writer has committed them. This wraps an IOException indicating that a file the system just wrote is missing, which typically points to a filesystem-level problem rather than a Flink logic bug.","triggerScenarios":"After CompactFileUtils writes compacted files and calls writer.commit(), the subsequent checkExist loop finds a Path that fileSystem.exists() returns false for.","commonSituations":"Eventually-consistent object stores (e.g. S3 in older configurations) where a newly written object is not yet visible; concurrent external deletion of the output directory; filesystem permissions or mount issues; race conditions on distributed filesystems.","solutions":["If using S3, ensure s3a with consistent reads (S3 strong consistency is now default, but older emrfs or custom configs may lag).","Verify no external process or concurrent job is deleting compaction output files.","Check filesystem permissions and that the output directory is writable and stable.","Retry the job from the last successful checkpoint; transient filesystem hiccups often resolve on retry."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    CompactFileUtils.doCompact(fileSystem, candidates, output);\n} catch (IOException e) {\n    if (e.getMessage().startsWith(\"Compaction file not exist\")) {\n        LOG.warn(\"Compaction output file missing — possible filesystem inconsistency: {}\", e.getMessage());\n        // retry from checkpoint or alert operations\n    }\n    throw e;\n}","preventionTips":["Ensure the output filesystem provides strong read-after-write consistency.","Verify no concurrent jobs or external tools delete compaction output directories.","Monitor filesystem health and retry from the last successful checkpoint on transient failures."],"tags":["filesystem-connector","compaction","filesystem","io"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}