{"record":{"id":"dd39b0b5ea5e91f7","repo":"prestodb/presto","slug":"hive-missing-data","errorCode":"HIVE_MISSING_DATA","errorMessage":"Error reading from %s at position %s. ","messagePattern":"Error reading from (.+?) at position (.+?)\\. ","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"critical","filePath":"presto-hive/src/main/java/com/facebook/presto/hive/orc/HdfsOrcDataSource.java","lineNumber":74,"sourceCode":"        inputStream.close();\n    }\n\n    @Override\n    protected void readInternal(long position, byte[] buffer, int bufferOffset, int bufferLength)\n    {\n        try {\n            long readStart = System.nanoTime();\n            inputStream.readFully(position, buffer, bufferOffset, bufferLength);\n            stats.readDataBytesPerSecond(bufferLength, System.nanoTime() - readStart);\n        }\n        catch (PrestoException e) {\n            // just in case there is a Presto wrapper or hook\n            throw e;\n        }\n        catch (Exception e) {\n            String message = format(\"Error reading from %s at position %s. \", this, position);\n            if (e.getClass().getSimpleName().equals(\"BlockMissingException\")) {\n                throw new PrestoException(HIVE_MISSING_DATA, message, e);\n            }\n            if (e instanceof IOException) {\n                throw new PrestoException(HIVE_FILESYSTEM_ERROR, message + e.getMessage(), e);\n            }\n            throw new PrestoException(HIVE_UNKNOWN_ERROR, message, e);\n        }\n    }\n}\n","sourceCodeStart":56,"sourceCodeEnd":83,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-hive/src/main/java/com/facebook/presto/hive/orc/HdfsOrcDataSource.java#L56-L83","documentation":"HdfsOrcDataSource.readInternal wraps any exception from the underlying HDFS read. If the exception's simple class name is BlockMissingException, it rethrows as PrestoException(HIVE_MISSING_DATA) — a datanode holding a block replica is unreachable or the block is genuinely missing, so the ORC reader cannot fetch bytes at the requested position.","triggerScenarios":"ORC stripe/footer read at `position` throws an exception whose class name is exactly 'BlockMissingException' (HDFS block has no live replicas).","commonSituations":"Datanode failures or decommissioning during a query; corrupted/lost blocks after disk failure; under-replicated blocks following datanode crashes; files whose blocks were deleted by balancer/cleanup bugs.","solutions":["Run hdfs fsck <file-path> -blocks -locations to identify missing blocks and confirm the damage","Restore the data: re-run the producing job, restore from snapshot/backup, or delete corrupt files and repair the partition","Check datanode health and wait for re-replication (dfs.namenode.replication.work.multiplier-per-progressed-heartbeat) if blocks are merely under-replicated","Retry the query after cluster recovery if the failure was transient (node restart)"],"exampleFix":"// before (operator check)\n-- hdfs fsck /warehouse/t/part-0 -blocks -locations\n-- status: HEALTHY -> corrupt blocks\n\n// after\n-- hadoop fs -rm /warehouse/t/part-0\n-- re-run ETL to regenerate, then ALTER TABLE t ADD PARTITION ...\nSELECT * FROM t;","handlingStrategy":"retry","validationCode":"# preflight replication/health of the files to be read\nhdfs fsck /warehouse/t -files -blocks -locations | grep -E 'MISSING|HEALTHY'","typeGuard":null,"tryCatchPattern":"try {\n    return query(sql);\n} catch (PrestoException e) {\n    if (e.getErrorCode().getName().equals(\"HIVE_MISSING_DATA\")) {\n        // transient node outage: retry with backoff; otherwise repair data\n        return withRetry(sql, 3, exponentialBackoff());\n    }\n    throw e;\n}","preventionTips":["Keep replication factor >= 3 and monitor under-replicated/missing block metrics","Run hdfs fsck on critical table paths before large scheduled queries","Avoid querying during datanode decommission/maintenance windows","Maintain backups/snapshots so lost blocks can be regenerated"],"tags":["hdfs","orc","missing-block","datanode","storage"],"backgroundTag":"hdfs-block-missing","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}