What causes "Expected {} bytes, but read {}" in apache/hadoop?

Retry resolve() a small bounded number of times with a short delay — the window between stat and read is tiny and the next attempt usually sees a consistent file.

apache/hadoop · error · InvalidRecordException

Expected {} bytes, but read {}

Error message

Expected {} bytes, but read {}

What it means

FSRegistryOperationsService.resolve() reads a record file by first sizing it with getFileStatus().getLen(), allocating a byte array of that length, and then issuing a single instream.read(bytes) call. InputStream.read(byte[]) only guarantees at least one byte, not a full buffer, so if the single call returns fewer bytes than the earlier stat reported, resolve() throws InvalidRecordException('Expected N bytes, but read M'). In practice the mismatch almost always means the record file was rewritten or truncated between the getFileStatus call and the read.

Source

Thrown at hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/FSRegistryOperationsService.java:171

      stream.close();
      LOG.info("Bound record to path " + dataPath);
    }
  }

  @Override
  public ServiceRecord resolve(String path) throws PathNotFoundException,
      NoRecordException, InvalidRecordException, IOException {
    // Read the entire file into byte array, should be small metadata

    Long size = fs.getFileStatus(formatDataPath(path)).getLen();
    byte[] bytes = new byte[size.intValue()];

    FSDataInputStream instream = fs.open(formatDataPath(path));
    int bytesRead = instream.read(bytes);
    instream.close();

    if (bytesRead < size) {
      throw new InvalidRecordException(path,
          "Expected " + size + " bytes, but read " + bytesRead);
    }

    // Unmarshal, check, and return
    ServiceRecord record = serviceRecordMarshal.fromBytes(path, bytes);
    RegistryTypeUtils.validateServiceRecord(path, record);
    return record;
  }

  @Override
  public RegistryPathStatus stat(String path)
      throws PathNotFoundException, InvalidPathnameException, IOException {
    FileStatus fstat = fs.getFileStatus(formatDataPath(path));
    int numChildren = fs.listStatus(makePath(path)).length;

    RegistryPathStatus regstat =
        new RegistryPathStatus(fstat.getPath().toString(),
            fstat.getModificationTime(), fstat.getLen(), numChildren);

View on GitHub (pinned to 2add963021)

Solutions

Retry resolve() a small bounded number of times with a short delay — the window between stat and read is tiny and the next attempt usually sees a consistent file.
Eliminate concurrent writers to the same registry path (single publisher per path, complete bind before advertising the path).
If you control the storage layer, patch resolve() to loop until EOF (readFully semantics) instead of a single read() call.

Example fix

// before (FSRegistryOperationsService.resolve)
int bytesRead = instream.read(bytes);
instream.close();
if (bytesRead < size) {
  throw new InvalidRecordException(path, "Expected " + size + " bytes, but read " + bytesRead);
}

// after: read fully; only fail on a genuine size mismatch
long total = 0;
int n;
while (total < bytes.length
    && (n = instream.read(bytes, (int) total, bytes.length - (int) total)) != -1) {
  total += n;
}
instream.close();
if (total != size) {
  throw new InvalidRecordException(path, "Expected " + size + " bytes, but read " + total);
}

Defensive patterns

Strategy: retry

Try / catch

ServiceRecord resolveWithRetry(RegistryOperations ops, String path, int attempts)
    throws IOException {
  IOException last = null;
  for (int i = 0; i < attempts; i++) {
    try {
      return ops.resolve(path);
    } catch (InvalidRecordException e) {
      last = e; // record file changed size between stat and read: transient
    }
  }
  throw last;
}

Prevention

Keep exactly one writer per registry path; complete bind() or delete() before advertising the path to consumers.
Do not treat a single resolve failure as permanent — re-resolve once after a short delay before giving up.
In your own FS code never rely on a single InputStream.read(byte[]) to fill the buffer; use readFully or a loop.

When it happens

Trigger: A concurrent bind()/delete() of the same registry path between the getFileStatus and the fs.open/read sequence; the record file being truncated or replaced mid-resolve; a filesystem input stream that legitimately returns short reads on one call.

Common situations: Multiple writers publishing/unpublishing the same service record at the same time; a resolve racing an unregister; the FS registry backend used under concurrent service discovery traffic. The single-read() pattern is itself a latent defect — readFully would be correct.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a8eee32f6eea22b9. Report an issue: GitHub.