{"record":{"id":"42a7567e3363b791","repo":"dianping/cat","slug":"invalid-index-file-s","errorCode":null,"errorMessage":"Invalid index file: %s","messagePattern":"Invalid index file: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"cat-hadoop/src/main/java/org/unidal/cat/message/storage/hdfs/HdfsBucket.java","lineNumber":245,"sourceCode":"\t\t\t\tint segmentIndex = seq / MESSAGE_PER_SEGMENT;\n\t\t\t\tint segmentOffset = (seq % MESSAGE_PER_SEGMENT) * BYTE_PER_MESSAGE;\n\t\t\t\tInteger segmentId = findSegment(ip, segmentIndex);\n\n\t\t\t\tif (segmentId != null) {\n\t\t\t\t\tlong offset = segmentId.intValue() * SEGMENT_SIZE + segmentOffset;\n\n\t\t\t\t\treturn offset;\n\t\t\t\t} else {\n\t\t\t\t\treturn -1;\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tpublic void load(int headBlockIndex) throws IOException {\n\t\t\t\tSegment segment = new Segment(m_indexSteam, headBlockIndex * ENTRY_PER_SEGMENT * SEGMENT_SIZE);\n\t\t\t\tlong magicCode = segment.readLong();\n\n\t\t\t\tif (magicCode != -1) {\n\t\t\t\t\tthrow new IOException(\"Invalid index file: \" + m_indexSteam);\n\t\t\t\t}\n\n\t\t\t\tm_nextSegment = 1 + ENTRY_PER_SEGMENT * headBlockIndex;\n\n\t\t\t\tint readerIndex = 1;\n\n\t\t\t\twhile (readerIndex < ENTRY_PER_SEGMENT) {\n\t\t\t\t\tint ip = segment.readInt();\n\t\t\t\t\tint index = segment.readInt();\n\n\t\t\t\t\treaderIndex++;\n\n\t\t\t\t\tif (ip != 0) {\n\t\t\t\t\t\tMap<Integer, Integer> map = m_table.get(ip);\n\n\t\t\t\t\t\tif (map == null) {\n\t\t\t\t\t\t\tmap = new HashMap<Integer, Integer>();\n\t\t\t\t\t\t\tm_table.put(ip, map);","sourceCodeStart":227,"sourceCodeEnd":263,"githubUrl":"https://github.com/dianping/cat/blob/e815e74d4c2dd74edac831241f1253fcc7d25381/cat-hadoop/src/main/java/org/unidal/cat/message/storage/hdfs/HdfsBucket.java#L227-L263","documentation":"HdfsBucket's Segment.load throws this IOException when the first long read from an index segment block is not the expected magic code -1. Every segment block in the HDFS index file starts with a magic value; anything else means the file is corrupt, truncated, written by an incompatible version, or the offset math (headBlockIndex * ENTRY_PER_SEGMENT * SEGMENT_SIZE) landed on the wrong position. The check is a data-integrity guard before the segment's ip/index entries are read.","triggerScenarios":"Calling load/head operations on an HDFS index bucket whose data was corrupted (partial write, interrupted upload), whose segment size / entry layout changed between CAT versions, or when headBlockIndex points past the actual file size so reads return garbage or zeros. Also when a non-CAT or wrongly-named file is opened as an index.","commonSituations":"Upgrading CAT storage components where ENTRY_PER_SEGMENT or SEGMENT_SIZE constants changed, making old index files unreadable; HDFS datanode corruption or truncated blocks after a crash; mixing files from different environments in the same storage path.","solutions":["Verify the HDFS index files were produced by the same CAT version as the reader (check SEGMENT_SIZE/ENTRY_PER_SEGMENT constants in HdfsBucket)","Rebuild the affected index from the source data files or restore from a clean HDFS snapshot/backup","Check for truncation: compare file length against expected segment layout (blockIndex bounds) and fsck the HDFS path","If version skew is the cause, read with the old version's code or regenerate the indexes with the new version before querying"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"long expectedLen = (long) (headBlockIndex + 1) * ENTRY_PER_SEGMENT * SEGMENT_SIZE;\nif (indexFile.length() < expectedLen) {\n    // segment beyond file: skip instead of reading garbage\n    return;\n}","typeGuard":null,"tryCatchPattern":"try {\n    segment.load(headBlockIndex);\n} catch (IOException e) {\n    if (e.getMessage() != null && e.getMessage().startsWith(\"Invalid index file\")) {\n        // corrupt/incompatible segment: log, skip this block, continue with remaining segments\n        logger.warn(\"Skipping corrupt index segment \" + headBlockIndex, e);\n    } else {\n        throw e;\n    }\n}","preventionTips":["Pin one CAT version for both writing and reading HDFS index files; version-skew changes segment layout","Verify file length matches the expected segment layout before reading a block index","Keep backups/snapshots of HDFS index paths so corrupted segments can be restored instead of skipped"],"tags":["cat","java","hdfs","storage","data-corruption","index"],"backgroundTag":null,"analyzedSha":"e815e74d4c2dd74edac831241f1253fcc7d25381","analyzedAt":"2026-08-14T14:22:34.512Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}