What causes "Index file for the log of " + taskid + " doesn't exist." in apache/hadoop?

Treat it as 'logs unavailable': catch the IOException and surface a friendly message instead of failing the caller.

apache/hadoop · warning · IOException

Index file for the log of " + taskid + " doesn't exist.

Error message

Index file for the log of " + taskid + " doesn't exist.

What it means

TaskLog reads each attempt's log.index file (mapping stdout/stderr/syslog offsets) before serving task logs. If the first readLine() on that index returns null — the file exists but is empty — this IOException is thrown, reporting the taskid. It effectively means the attempt's log index was never written or was truncated, so its logs cannot be located.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java:128

  private static LogFileDetail getLogFileDetail(TaskAttemptID taskid, 
                                                LogName filter,
                                                boolean isCleanup) 
  throws IOException {
    File indexFile = getIndexFile(taskid, isCleanup);
    BufferedReader fis = new BufferedReader(new InputStreamReader(
      SecureIOUtils.openForRead(indexFile, obtainLogDirOwner(taskid), null),
      StandardCharsets.UTF_8));
    //the format of the index file is
    //LOG_DIR: <the dir where the task logs are really stored>
    //stdout:<start-offset in the stdout file> <length>
    //stderr:<start-offset in the stderr file> <length>
    //syslog:<start-offset in the syslog file> <length>
    LogFileDetail l = new LogFileDetail();
    String str = null;
    try {
      str = fis.readLine();
      if (str == null) { // the file doesn't have anything
        throw new IOException("Index file for the log of " + taskid
            + " doesn't exist.");
      }
      l.location = str.substring(str.indexOf(LogFileDetail.LOCATION)
          + LogFileDetail.LOCATION.length());
      // special cases are the debugout and profile.out files. They are
      // guaranteed
      // to be associated with each task attempt since jvm reuse is disabled
      // when profiling/debugging is enabled
      if (filter.equals(LogName.DEBUGOUT) || filter.equals(LogName.PROFILE)) {
        l.length = new File(l.location, filter.toString()).length();
        l.start = 0;
        fis.close();
        return l;
      }
      str = fis.readLine();
      while (str != null) {
        // look for the exact line containing the logname
        if (str.contains(filter.toString())) {

View on GitHub (pinned to 2add963021)

Solutions

Treat it as 'logs unavailable': catch the IOException and surface a friendly message instead of failing the caller.
Verify the attempt actually ran and check whether userlog retention (mapreduce.task.userlog.retain.* / yarn.nodemanager.log-aggregation settings) already removed the logs.
Inspect the attempt's log dir on the node: if log.index is 0 bytes but the log files exist, the attempt died before writing the index — nothing to read.
Increase log retention or enable log aggregation if users routinely need older task logs.

Example fix

// before
LogFileDetail d = TaskLog.readTaskLog(TaskLog.LogName.SYSLOG, taskid, LogFilter.read);

// after
try {
  LogFileDetail d = TaskLog.readTaskLog(TaskLog.LogName.SYSLOG, taskid, LogFilter.read);
} catch (IOException e) {
  // logs unavailable for this attempt; report and continue
}

Defensive patterns

Strategy: try-catch

Validate before calling

// before reading an attempt's logs, check its log.index is non-trivial
File attemptDir = TaskLog.getAttemptDir(taskid);
File idx = new File(attemptDir, TaskLog.LogName.INDEX.toString());
if (!idx.exists() || idx.length() == 0) {
  LOG.warn("No log index for " + taskid + "; logs unavailable");
  return Collections.emptyList();
}

Try / catch

try {
  LogFileDetail detail = TaskLog.readTaskLog(LogName.SYSLOG, taskid, LogFilter.read);
} catch (IOException e) {
  if (e.getMessage().contains("Index file for the log of")) {
    // treat as logs-unavailable; report and continue rather than failing the caller
  }
}

Prevention

Degrade gracefully when task logs are missing — retention may have removed them.
Enable log aggregation or raise retain hours if users need old task logs.
Check that attempts actually started before fetching their logs.

When it happens

Trigger: Serving logs for a task attempt whose JVM was killed before flushing the index (kill -9, OOM), a log dir partially removed by userlog retention/cleanup, or manual deletion of log.index while leaving the attempt dir behind.

Common situations: Viewing logs of long-finished attempts past mapreduce.task.userlog.retain.hours/rm setup; attempts killed by ulimits; reading logs programmatically via TaskLog/TaskLogReader for a task that never really started.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4390199942249c8d. Report an issue: GitHub.