apache/hadoop · error · IOException

Invalid line: " + line

Error message

Invalid line: " + line

What it means

TextReader parses each line of the provided-storage alias file by splitting on the configured delimiter (dfs.provided.aliasmap.text.delimiter). A valid FileRegion line must yield 5 fields (blockId, path, offset, length, length-under-management) or 6 with a base64 nonce; any other field count throws IOException naming the offending line.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/blockaliasmap/impl/TextFileRegionAliasMap.java:359

      @Override
      public void remove() {
        throw new UnsupportedOperationException();
      }
    }

    private FileRegion nextInternal(Iterator<FileRegion> i) throws IOException {
      BufferedReader r = iterators.get(i);
      if (null == r) {
        throw new IllegalStateException();
      }
      String line = r.readLine();
      if (null == line) {
        iterators.remove(i);
        return null;
      }
      String[] f = line.split(delim);
      if (f.length != 5 && f.length != 6) {
        throw new IOException("Invalid line: " + line);
      }
      byte[] nonce = new byte[0];
      if (f.length == 6) {
        nonce = Base64.getDecoder().decode(f[5]);
      }
      return new FileRegion(Long.parseLong(f[0]), new Path(f[1]),
          Long.parseLong(f[2]), Long.parseLong(f[3]), Long.parseLong(f[4]),
          nonce);
    }

    public InputStream createStream() throws IOException {
      InputStream i = fs.open(file);
      if (codec != null) {
        i = codec.createInputStream(i);
      }
      return i;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Look at the exact line in the message — count the fields and compare with the expected 'blockId,path,offset,length,length[,nonce]' layout
2. Make writer and reader agree on dfs.provided.aliasmap.text.delimiter, and regenerate the file with TextFileRegionAliasMap's own writer rather than by hand
3. Strip header/comment/blank lines and any malformed trailing fragment

Example fix

# before (delimiter mismatch, file uses tabs, config default is used)
12345,/blocks/1,0,1024,1024,nonce

# after: file fields match configured delimiter, 5 or 6 columns
12345	/blocks/1	0	1024	1024
# with hdfs-site.xml: dfs.provided.aliasmap.text.delimiter = \t

// Validate the alias file before handing it to the reader
try (BufferedReader br = new BufferedReader(new InputStreamReader(fs.open(file)))) {
  String line; int n = 0;
  while ((line = br.readLine()) != null) {
    n++;
    int fields = line.split(Pattern.quote(delim), -1).length;
    if (fields != 5 && fields != 6) {
      throw new IllegalArgumentException(
          "Alias file line " + n + " has " + fields + " fields (need 5 or 6): " + line);
    }
  }
}

try {
  for (FileRegion r : textReader) { /* ... */ }
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Invalid line:")) {
    // fix the file / delimiter config; log the offending line and fail the job
  }
  throw e;
}

When it happens

Trigger: The file's actual delimiter differs from the configured one (commas in file, tabs configured), a required field is missing/extra, the file contains a header, comment, or blank line, or a path itself contains the delimiter and splits a field in two.

Common situations: Hand-authored or externally generated alias files; a generating job written with one delimiter and a reader configured with the default; editing the file and dropping a column; file ending with a stray partial line after a truncated write.

Related errors

• Invalid options " + opts.getClass()
• Refresh not supported by " + getClass()
• InMemoryAliasMap location is null
• Unable to create missing aliasmap location: {levelDBpath}
• Failed to fully delete compressed aliasmap {compressedAliasM

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f9607db8f728e33c. Report an issue: GitHub.