apache/hadoop · error · IOException

Cannot have more than 2**25 strings in the fsimage, because

Error message

Cannot have more than 2**25 strings in the fsimage, because of the limitation on the size of string table IDs.

What it means

The reconstructor's string table ran out of IDs: fsimage encodes user/group names as 25-bit string-table IDs inside each inode's 64-bit permission long, capping the table at 2**25 (33,554,432) distinct strings. registerStringId throws when the next ID would reach 0x1ffffff, so the XML references more distinct user/group/mode strings than a single fsimage can hold.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageReconstructor.java:1642

   * This is a simple form of compression which takes advantage of the fact
   * that the same strings tend to occur over and over again.
   * This function will return an ID which we can use to represent the given
   * string.  If the string already exists in the string table, we will use
   * that ID; otherwise, we will allocate a new one.
   *
   * @param str           The string.
   * @return              The ID in the string table.
   * @throws IOException  If we run out of bits in the string table.  We only
   *                      have 25 bits.
   */
  int registerStringId(String str) throws IOException {
    Integer id = stringTable.get(str);
    if (id != null) {
      return id;
    }
    int latestId = latestStringId;
    if (latestId >= 0x1ffffff) {
      throw new IOException("Cannot have more than 2**25 " +
          "strings in the fsimage, because of the limitation on " +
          "the size of string table IDs.");
    }
    stringTable.put(str, latestId);
    latestStringId++;
    return latestId;
  }

  /**
   * Record the length of a section of the FSImage in our FileSummary object.
   * The FileSummary appears at the end of the FSImage and acts as a table of
   * contents for the file.
   *
   * @param sectionNamePb  The name of the section as it should appear in
   *                       the fsimage.  (This is different than the XML
   *                       name.)
   * @throws IOException
   */

View on GitHub (pinned to 2add963021)

Solutions

Reduce the number of distinct user/group names in the XML (consolidate owners, remap rare users to shared accounts) before reconstructing
Split the XML into several images and reconstruct each separately instead of one merged image
If the source was a genuine single fsimage, verify the XML was not corrupted/duplicated - a real image cannot exceed its own table
Stream-count distinct user:group prefixes beforehand and keep the total well under 2**25

Defensive patterns

Strategy: validation

Validate before calling

# python: distinct user:group prefixes must stay under 2**25
import xml.etree.ElementTree as ET
LIMIT = 1 << 25

def string_table_ok(path):
    seen = set()
    for ev, el in ET.iterparse(path, events=('end',)):
        if el.tag == 'permission':
            seen.add(':'.join((el.text or '').split(':')[:2]))
            el.clear()
            if len(seen) >= LIMIT:
                return False
    return True

Try / catch

// catch the 2**25 failure; reduce distinct owners in the XML (or split
// into multiple images), delete partial output, re-run

Prevention

Count distinct user:group pairs before reconstructing merged or synthetic XML
Do not merge multiple fsimage XML dumps into one file
Avoid fuzz tests that give every inode a unique owner
Remember the hard cap is 33,554,432 distinct strings

When it happens

Trigger: Reconstructing XML whose inodes carry more than 33,554,432 distinct user or group names - typically XML hand-merged from several fsimages, bulk-edited with generated tenant names, or synthetically fuzzed with unique owners per file.

Common situations: Merging multiple namespace dumps into one image; test/fuzz harnesses assigning a unique owner to every inode; importing a large external user taxonomy into permission fields.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4e1eeb78d987f97c. Report an issue: GitHub.