apache/hadoop · error · UnsupportedOperationException

{} is expected to be an instance of org.apache.hadoop.mapred

Error message

{} is expected to be an instance of org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage

What it means

refreshLoadedJobCache reloads the loaded-jobs cache but is only meaningful for CachedHistoryStorage, the default HistoryStorage. If a custom storage implementation was configured (JobHistory.createHistoryStorage reads the storage class from configuration), the instanceof check fails and UnsupportedOperationException names the mismatched class. The call is a no-op-with-warning when the service is not STARTED, so this exception specifically means: running service plus non-cached storage.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java:249

    // currently there is 1 to 1 mapping between app and job id
    org.apache.hadoop.mapreduce.JobID oldJobID = TypeConverter.fromYarn(appID);
    Map<JobId, Job> jobs = new HashMap<JobId, Job>();
    JobId jobID = TypeConverter.toYarn(oldJobID);
    jobs.put(jobID, getJob(jobID));
    return jobs;
  }

  @Override
  public Map<JobId, Job> getAllJobs() {
    return storage.getAllPartialJobs();
  }

  public void refreshLoadedJobCache() {
    if (getServiceState() == STATE.STARTED) {
      if (storage instanceof CachedHistoryStorage) {
        ((CachedHistoryStorage) storage).refreshLoadedJobCache();
      } else {
        throw new UnsupportedOperationException(storage.getClass().getName()
            + " is expected to be an instance of "
            + CachedHistoryStorage.class.getName());
      }
    } else {
      LOG.warn("Failed to execute refreshLoadedJobCache: JobHistory service is not started");
    }
  }

  @VisibleForTesting
  HistoryStorage getHistoryStorage() {
    return storage;
  }
  
  /**
   * Look for a set of partial jobs.
   * 
   * @param offset
   *          the offset into the list of jobs.

View on GitHub (pinned to 2add963021)

Solutions

1. Remove the custom mapreduce.jobhistory.storage.class setting to fall back to CachedHistoryStorage if cache refresh is needed.
2. Make the custom class extend org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.
3. Skip invoking refreshLoadedJobCache when a non-cached storage is deployed.

Example fix

// before
jobHistory.refreshLoadedJobCache();

// after
if (jobHistory.getHistoryStorage()
    instanceof org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage) {
  jobHistory.refreshLoadedJobCache();
}

private static boolean isCachedStorage(HistoryStorage storage) {
  return storage instanceof org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage;
}

When it happens

Trigger: Invoking the admin refresh operation (refreshLoadedJobCache) while a custom HistoryStorage class is configured; a storage wrapper that does not extend CachedHistoryStorage.

Common situations: Deployments plugging in custom history storage and then invoking cache refresh; configuration copied from another deployment that sets the storage class.

Related errors

• getShuffleFinishTime() not supported for MapTask
• setShuffleFinishTime() not supported for MapTask
• Name output '{}' has not been defined as multi
• Cannot recover task output for first attempt...
• iterate past last value

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a8cc7c85474867e7. Report an issue: GitHub.