{"record":{"id":"1bf92fa83802f8e9","repo":"apache/flink","slug":"the-given-offset-is-not-contained-in-the-any-block","errorCode":null,"errorMessage":"The given offset is not contained in the any block.","messagePattern":"The given offset is not contained in the any block\\.","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java","lineNumber":777,"sourceCode":"     */\n    private int getBlockIndexForPosition(\n            BlockLocation[] blocks, long offset, long halfSplitSize, int startIndex) {\n        // go over all indexes after the startIndex\n        for (int i = startIndex; i < blocks.length; i++) {\n            long blockStart = blocks[i].getOffset();\n            long blockEnd = blockStart + blocks[i].getLength();\n\n            if (offset >= blockStart && offset < blockEnd) {\n                // got the block where the split starts\n                // check if the next block contains more than this one does\n                if (i < blocks.length - 1 && blockEnd - offset < halfSplitSize) {\n                    return i + 1;\n                } else {\n                    return i;\n                }\n            }\n        }\n        throw new IllegalArgumentException(\"The given offset is not contained in the any block.\");\n    }\n\n    // --------------------------------------------------------------------------------------------\n\n    /**\n     * Opens an input stream to the file defined in the input format. The stream is positioned at\n     * the beginning of the given split.\n     *\n     * <p>The stream is actually opened in an asynchronous thread to make sure any interruptions to\n     * the thread working on the input format do not reach the file system.\n     */\n    @Override\n    public void open(FileInputSplit fileSplit) throws IOException {\n\n        this.currentSplit = fileSplit;\n        this.splitStart = fileSplit.getStart();\n        final Path path = fileSplit.getPath();\n        this.splitLength =","sourceCodeStart":759,"sourceCodeEnd":795,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java#L759-L795","documentation":"Thrown internally by FileInputFormat.getBlockIndexForPosition when a computed split start offset does not fall inside any HDFS/filesystem block range returned by getFileBlockLocations. It indicates the split layout no longer matches the physical block layout of the file, so the split-to-block locality mapping is inconsistent.","triggerScenarios":"The file changed size or was rewritten between createInputSplits (which captures block locations) and split assignment/open; a filesystem returns block offsets that do not cover the full file length; splits computed against stale file metadata; a custom FileSystem reports inconsistent BlockLocation offsets/lengths.","commonSituations":"The input file is appended to or overwritten while the job is starting; HDFS under-replication or namemode metadata lag returns partial BlockLocations; reading from an object store (S3) whose 'block locations' are synthetic and inconsistent; a file was truncated between split computation and reading.","solutions":["Ensure input files are stable (not being written/truncated) for the full job run; stage data into an immutable location before submitting the job.","Verify the FileSystem implementation reports correct BlockLocation offsets/lengths; for object stores use the supported S3/Hadoop FS plugin.","Recompute splits by re-running the job once the file is settled; avoid pointing at live/log directories being appended.","If the file is small/unsplittable, set the input format unsplittable so a single split covers the whole file and no block lookup mismatch occurs."],"exampleFix":"// before: pointing at a directory a live process is writing to\nenv.readTextFile(\"hdfs:///logs/today-live/\");\n\n// after: stage an immutable snapshot first\nenv.readTextFile(\"hdfs:///snapshots/today-frozen/\");","handlingStrategy":"validation","validationCode":"// Ensure input files are immutable/stable before submit\nPath input = new Path(\"hdfs:///snapshots/frozen/\");\nFileSystem fs = input.getFileSystem();\nfor (FileStatus f : fs.listStatus(input)) {\n    long len = f.getLen();\n    BlockLocation[] blks = fs.getFileBlockLocations(f, 0, len);\n    long covered = Arrays.stream(blks).mapToLong(BlockLocation::getLength).sum();\n    if (covered < len) {\n        throw new IllegalStateException(\"Block locations do not cover \" + f.getPath());\n    }\n}","typeGuard":null,"tryCatchPattern":"// Catch in the source open path and fail fast with context\ntry {\n    format.open(split);\n} catch (IllegalArgumentException e) {\n    if (e.getMessage().contains(\"not contained in the any block\")) {\n        LOG.error(\"Stale block metadata for {} — file may have changed\", split.getPath());\n    }\n    throw e;\n}","preventionTips":["Stage inputs into an immutable directory before job submission.","Avoid reading from directories a live process is appending to.","For object stores, use the supported FS plugin which reports consistent block locations.","If files are small/unsplittable, mark the format unsplittable to bypass block lookup."],"tags":["flink","file-input","hdfs","block-location","metadata"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}