dianping/cat · error · IOException

Invalid index file: %s

Error message

Invalid index file: %s

What it means

HdfsBucket's Segment.load throws this IOException when the first long read from an index segment block is not the expected magic code -1. Every segment block in the HDFS index file starts with a magic value; anything else means the file is corrupt, truncated, written by an incompatible version, or the offset math (headBlockIndex * ENTRY_PER_SEGMENT * SEGMENT_SIZE) landed on the wrong position. The check is a data-integrity guard before the segment's ip/index entries are read.

Source

Thrown at cat-hadoop/src/main/java/org/unidal/cat/message/storage/hdfs/HdfsBucket.java:245

				int segmentIndex = seq / MESSAGE_PER_SEGMENT;
				int segmentOffset = (seq % MESSAGE_PER_SEGMENT) * BYTE_PER_MESSAGE;
				Integer segmentId = findSegment(ip, segmentIndex);

				if (segmentId != null) {
					long offset = segmentId.intValue() * SEGMENT_SIZE + segmentOffset;

					return offset;
				} else {
					return -1;
				}
			}

			public void load(int headBlockIndex) throws IOException {
				Segment segment = new Segment(m_indexSteam, headBlockIndex * ENTRY_PER_SEGMENT * SEGMENT_SIZE);
				long magicCode = segment.readLong();

				if (magicCode != -1) {
					throw new IOException("Invalid index file: " + m_indexSteam);
				}

				m_nextSegment = 1 + ENTRY_PER_SEGMENT * headBlockIndex;

				int readerIndex = 1;

				while (readerIndex < ENTRY_PER_SEGMENT) {
					int ip = segment.readInt();
					int index = segment.readInt();

					readerIndex++;

					if (ip != 0) {
						Map<Integer, Integer> map = m_table.get(ip);

						if (map == null) {
							map = new HashMap<Integer, Integer>();
							m_table.put(ip, map);

View on GitHub (pinned to e815e74d4c)

Solutions

1. Verify the HDFS index files were produced by the same CAT version as the reader (check SEGMENT_SIZE/ENTRY_PER_SEGMENT constants in HdfsBucket)
2. Rebuild the affected index from the source data files or restore from a clean HDFS snapshot/backup
3. Check for truncation: compare file length against expected segment layout (blockIndex bounds) and fsck the HDFS path
4. If version skew is the cause, read with the old version's code or regenerate the indexes with the new version before querying

long expectedLen = (long) (headBlockIndex + 1) * ENTRY_PER_SEGMENT * SEGMENT_SIZE;
if (indexFile.length() < expectedLen) {
    // segment beyond file: skip instead of reading garbage
    return;
}

try {
    segment.load(headBlockIndex);
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().startsWith("Invalid index file")) {
        // corrupt/incompatible segment: log, skip this block, continue with remaining segments
        logger.warn("Skipping corrupt index segment " + headBlockIndex, e);
    } else {
        throw e;
    }
}

When it happens

Trigger: Calling load/head operations on an HDFS index bucket whose data was corrupted (partial write, interrupted upload), whose segment size / entry layout changed between CAT versions, or when headBlockIndex points past the actual file size so reads return garbage or zeros. Also when a non-CAT or wrongly-named file is opened as an index.

Common situations: Upgrading CAT storage components where ENTRY_PER_SEGMENT or SEGMENT_SIZE constants changed, making old index files unreadable; HDFS datanode corruption or truncated blocks after a crash; mixing files from different environments in the same storage path.

Related errors

• Unsupported message type: %s.
• Invalid message ID format: %s
• Fail to create directory(%s)!
• Only 1 config can be specified while initializing cat.

AI-assisted analysis of dianping/cat@e815e74d4c (2026-08-14). Data as JSON: /api/errors/42a7567e3363b791. Report an issue: GitHub.