apache/hadoop · error · IllegalArgumentException

Path must be absolute: " + path

Error message

Path must be absolute: " + path

What it means

StreamKeyValUtil.splitKeyVal splits a raw UTF-8 byte array into Hadoop Text key and value at a caller-supplied splitPos, and it validates that splitPos falls inside the half-open range [start, start+length). An IllegalArgumentException with 'splitPos must be in the range' means the computed separator offset lies outside the record buffer, so the split is not merely wrong but impossible. The library throws it to prevent key.set/val.set from reading out of bounds.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BaiduBosFileSystem.java:89

  private BosNativeFileSystemStore store;
  private Path workingDir;
  private long readAhead;
  private int readBufferSize;
  private boolean forceGetNonHierarchyMetadataInListStatus;

  /** Default constructor. */
  public BaiduBosFileSystem() {
  }

  /**
   * Convert a path to a BOS object key.
   *
   * @param path the path to convert
   * @return the BOS object key
   */
  protected String pathToKey(Path path) {
    if (!path.isAbsolute()) {
      throw new IllegalArgumentException(
          "Path must be absolute: " + path);
    }
    checkPath(path);
    return path.toUri().getPath().substring(1);
  }

  private static Path keyToPath(String key) {
    return new Path("/" + key);
  }

  @Override
  public String getScheme() {
    return "bos";
  }

  @Override
  public void close() throws IOException {
    try {

View on GitHub (pinned to 2add963021)

Solutions

1. Recompute the separator position on the same utf buffer with start offset added: splitPos = start + line.indexOf(sep, start), and use -1 handling before calling splitKeyVal.
2. If you derive key length instead, clamp it: splitPos = Math.min(Math.max(splitPos, start), start + length - 1).
3. For 'no separator present' records either skip them or fall back to whole-record-as-key (splitPos = start + length - 1 with separatorLength 0 semantics) instead of passing indexOf result directly.
4. Unit-test with edge cases: empty line, line equal to separator, separator at last byte.

Example fix

// before
int pos = new String(utf, start, length).indexOf('\t');
StreamKeyValUtil.splitKeyVal(utf, start, length, key, val, pos, 1); // pos relative -> throws when start>0

// after
int pos = new String(utf, start, length).indexOf('\t');
if (pos < 0) {
  pos = length - 1; // whole record as key, empty value
} else {
  pos = start + pos; // make buffer-absolute
}
StreamKeyValUtil.splitKeyVal(utf, start, length, key, val, pos, 1);

static int resolveSplitPos(byte[] utf, int start, int length, int sepIdxInRecord, int separatorLength) {
  if (sepIdxInRecord < 0) return start + length - 1; // no separator: whole record as key
  int splitPos = start + sepIdxInRecord;              // make buffer-absolute
  if (splitPos < start || splitPos >= start + length)
    throw new IllegalArgumentException("bad splitPos " + splitPos);
  return splitPos;
}

When it happens

Trigger: Calling any splitKeyVal overload (StreamKeyValUtil.java:65-121) with a splitPos < start, >= start+length, or negative — typically a separator index found by String.indexOf on a different buffer, a separator not found (-1), or a key length >= record length (e.g. separatorLength=1 leaves a zero/negative value half).

Common situations: Custom streaming mappers/reducers using TypedBytesOutput or raw key/value copying (e.g. copyPartition/hadoopStreaming custom IO classes); setting -stream.map.output.field.separator or stream.io.key.length and getting the field index math wrong; records with no separator so the boundary computes past the end of the line; empty input lines.

Related errors

• Append is not supported by BaiduBosFileSystem
• No more entry in " + f
• f + " is a directory"
• f + " already exists"
• parent + " is a file"

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9bdfca6c3c366583. Report an issue: GitHub.