{"record":{"id":"849eeb67c2f8deed","repo":"prestodb/presto","slug":"hive-missing-data-849eeb","errorCode":"HIVE_MISSING_DATA","errorMessage":"Error opening Hive split %s (offset=%s, length=%s): %s","messagePattern":"Error opening Hive split (.+?) \\(offset=(.+?), length=(.+?)\\): (.+?)","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-hive/src/main/java/com/facebook/presto/hive/rcfile/RcFilePageSourceFactory.java","lineNumber":171,"sourceCode":"                    new DataSize(8, Unit.MEGABYTE));\n\n            return Optional.of(new RcFilePageSource(rcFileReader, columns, typeManager));\n        }\n        catch (Throwable e) {\n            try {\n                inputStream.close();\n            }\n            catch (IOException ignored) {\n            }\n            if (e instanceof PrestoException) {\n                throw (PrestoException) e;\n            }\n            String message = splitError(e, fileSplit);\n            if (e instanceof RcFileCorruptionException) {\n                throw new PrestoException(HIVE_BAD_DATA, message, e);\n            }\n            if (e.getClass().getSimpleName().equals(\"BlockMissingException\")) {\n                throw new PrestoException(HIVE_MISSING_DATA, message, e);\n            }\n            throw new PrestoException(HIVE_CANNOT_OPEN_SPLIT, message, e);\n        }\n    }\n\n    private static String splitError(Throwable t, HiveFileSplit fileSplit)\n    {\n        return format(\"Error opening Hive split %s (offset=%s, length=%s): %s\", fileSplit.getPath(), fileSplit.getStart(), fileSplit.getLength(), t.getMessage());\n    }\n\n    public static TextRcFileEncoding createTextVectorEncoding(Properties schema, DateTimeZone hiveStorageTimeZone)\n    {\n        // separators\n        int nestingLevels;\n        if (!\"true\".equalsIgnoreCase(schema.getProperty(SERIALIZATION_EXTEND_NESTING_LEVELS))) {\n            nestingLevels = TEXT_LEGACY_NESTING_LEVELS;\n        }\n        else {","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-hive/src/main/java/com/facebook/presto/hive/rcfile/RcFilePageSourceFactory.java#L153-L189","documentation":"RcFilePageSourceFactory detects Hadoop BlockMissingException while constructing the RCFile reader and rethrows it as PrestoException HIVE_MISSING_DATA, with the message 'Error opening Hive split <path> (offset=..., length=...): ...'. It means HDFS reported that a block needed by the split is not available on any DataNode — data exists in the namespace but is unreadable.","triggerScenarios":"openFile succeeds but RcFileReader/HdfsRcFileDataSource reads hit a block with no live replicas: DataNodes lost disks, under-replicated blocks after DataNode decommission, blocks deleted by balancer/hdfs fsck issues, or replica corruption marks the block unavailable.","commonSituations":"DataNode failure or disk replacement; hdfs fsck reports MISSING blocks; query after a NameNode restart with stale block reports; under-replicated files after decommissioning nodes too quickly.","solutions":["Run `hdfs fsck <path> -files -blocks -locations` to identify the missing blocks.","Restore replication: check DataNode health, restart failed DataNodes, or `hdfs debug recoverLease -path <file> -retries 3`.","Restore the file from backup/snapshot or re-run the producing job for the affected partition.","Avoid decommissioning/balancer operations while queries scan the affected data.","Drop the affected partition if the data is unrecoverable."],"exampleFix":"// before: fsck shows MISSING blocks for the split's file\n$ hdfs fsck /data/rc/part=1/file.rcfile\n// after: restore replication from backup\nhdfs dfs -rm /data/rc/part=1/file.rcfile;\nhdfs dfs -put /backup/file.rcfile /data/rc/part=1/;\nhdfs dfs -setrep -w 3 /data/rc/part=1/file.rcfile;","handlingStrategy":"retry","validationCode":"// check block health before scheduling heavy scans\n// hdfs fsck <table-path> -files -blocks -locations | grep -i MISSING\nboolean hasMissingBlocks(String path) throws IOException {\n    Process p = new ProcessBuilder(\"hdfs\", \"fsck\", path, \"-files\", \"-blocks\", \"-locations\").start();\n    String out = new String(p.getInputStream().readAllBytes());\n    return out.toUpperCase().contains(\"MISSING\");\n}","typeGuard":"boolean isHiveMissingData(Throwable t) {\n    return t instanceof PrestoException && ((PrestoException) t).getErrorCode().getName().equals(\"HIVE_MISSING_DATA\");\n}","tryCatchPattern":"try {\n    return query(sql);\n} catch (PrestoException e) {\n    if (e.getErrorCode().getName().equals(\"HIVE_MISSING_DATA\")) {\n        // missing blocks rarely self-heal; restore replication first, then retry once\n        recoverLeaseAndReplicate(extractPath(e.getMessage()));\n        return query(sql);\n    }\n    throw e;\n}","preventionTips":["Monitor hdfs fsck for missing/under-replicated blocks on a schedule.","Set adequate replication factor (>=3) and avoid fast decommission of DataNodes.","Take HDFS snapshots of hot table directories for recovery.","Gate large scans on DataNode/disk health alerts."],"tags":["hive","hdfs","missing-block","rcfile","data-loss"],"backgroundTag":"hdfs-block-missing","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}