apache/hadoop · warning · RemoteException

Did not find requested id ${id}

Error message

Did not find requested id ${id}

What it means

Thrown by CacheDirectiveIterator when talking to an old NameNode that does not support filtering cache directives by ID. The iterator retries client-side by listing from prevKey = id-1 and scanning the returned batch for the requested id; if the id is not in that window (or the directive no longer exists), it raises RemoteException(InvalidRequestException). It is a fallback-compatibility path, explicitly noted in-code as brittle.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveIterator.java:113

    BatchedEntries<CacheDirectiveEntry> entries;
    try (TraceScope ignored = tracer.newScope("listCacheDirectives")) {
      entries = namenode.listCacheDirectives(prevKey, filter);
    } catch (IOException e) {
      if (e.getMessage().contains("Filtering by ID is unsupported")) {
        // Retry case for old servers, do the filtering client-side
        long id = filter.getId();
        filter = removeIdFromFilter(filter);
        // Using id - 1 as prevId should get us a window containing the id
        // This is somewhat brittle, since it depends on directives being
        // returned in order of ascending ID.
        entries = namenode.listCacheDirectives(id - 1, filter);
        for (int i = 0; i < entries.size(); i++) {
          CacheDirectiveEntry entry = entries.get(i);
          if (entry.getInfo().getId().equals(id)) {
            return new SingleEntry(entry);
          }
        }
        throw new RemoteException(InvalidRequestException.class.getName(),
            "Did not find requested id " + id);
      }
      throw e;
    }
    Preconditions.checkNotNull(entries);
    return entries;
  }

  @Override
  public Long elementToPrevKey(CacheDirectiveEntry entry) {
    return entry.getInfo().getId();
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Treat the exception as 'directive no longer present' if the directive may have been removed; verify with listCacheDirectives without the id filter.
2. Upgrade the NameNode to a version that supports server-side ID filtering, which removes this client-side retry path entirely.
3. Avoid paging through directives while concurrently removing them; snapshot ids first, then list.
4. If the listing is large, page with explicit prevKey instead of relying on the id-filter single-shot path.

Example fix

// before
CacheDirectiveInfo filter = new CacheDirectiveInfo.Builder().setId(42L).build();
RemoteIterator<CacheDirectiveEntry> it = dfs.listCacheDirectives(filter);
CacheDirectiveEntry e = it.next(); // may throw 'Did not find requested id 42' on old NameNodes

// after
try {
  CacheDirectiveEntry e = it.next();
} catch (RemoteException re) {
  if (re.getClassName().equals(InvalidRequestException.class.getName())
      && re.getMessage().contains("Did not find requested id")) {
    // directive 42 was deleted or fell outside the client-side scan window
  } else { throw re; }
}

// Confirm the directive still exists before iterating by id (new NameNodes filter server-side)
boolean exists = false;
RemoteIterator<CacheDirectiveEntry> scan = dfs.listCacheDirectives(null);
while (scan.hasNext()) { if (scan.next().getInfo().getId() == id) { exists = true; break; } }

try {
  return it.next();
} catch (RemoteException re) {
  if (InvalidRequestException.class.getName().equals(re.getClassName())
      && re.getMessage().contains("Did not find requested id")) {
    return null; // directive deleted while listing — treat as absent
  }
  throw re.unwrapRemoteException();
}

When it happens

Trigger: Calling DistributedFileSystem.listCacheDirectives with a CacheDirectiveInfo filter that has setId(...), against a NameNode old enough to answer 'Filtering by ID is unsupported'. The id is not found when: the directive was deleted between listing batches, more than one batch of directives exist after id-1 so the target falls outside the first window, or failover reordered ids.

Common situations: Clusters after a downgrade or with pre-HDFS-7309 NameNodes, tooling that lists a specific directive by id, or tests that remove directives concurrently with iteration.

Related errors

• Mark/reset not supported
• Limit is negative.
• Default Replication is negative
• Max relative expiry is negative.
• Max relative expiry is too big.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/40630428edd51052. Report an issue: GitHub.