apache/hadoop · error · IOException

Cannot get log path for a in-progress job

Error message

Cannot get log path for a in-progress job

What it means

Thrown by ClientServiceDelegate.getLogFilePath when the job's reported state is not one of SUCCEEDED/FAILED/KILLED/ERROR — i.e. the job is still running or pending. Log file parameters are only computed from a finalized report (container ids and NM locations are stable then); for in-progress jobs the API refuses with IOException by design.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java:534

            taReport.getContainerId().toString(),
            taReport.getContainerId().getApplicationAttemptId()
                .getApplicationId().toString(),
            NodeId.newInstance(taReport.getNodeManagerHost(),
                taReport.getNodeManagerPort()).toString(), report.getUser());
      } else {
        if (report.getAMInfos() == null || report.getAMInfos().size() == 0) {
          throw new IOException("Unable to get log information for job: "
              + oldJobID);
        }
        AMInfo amInfo = report.getAMInfos().get(report.getAMInfos().size() - 1);
        return new LogParams(
            amInfo.getContainerId().toString(),
            amInfo.getAppAttemptId().getApplicationId().toString(),
            NodeId.newInstance(amInfo.getNodeManagerHost(),
                amInfo.getNodeManagerPort()).toString(), report.getUser());
      }
    } else {
      throw new IOException("Cannot get log path for a in-progress job");
    }
  }

  public void close() throws IOException {
    if (rm != null) {
      rm.close();
    }

    if (historyServerProxy != null) {
      RPC.stopProxy(historyServerProxy);
    }

    if (realProxy != null) {
      RPC.stopProxy(realProxy);
      realProxy = null;
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for terminal state first: job.waitForCompletion() or poll JobStatus until isJobComplete(), then call getLogFileParams
2. Use `yarn logs -am <appId>` or `yarn logs -applicationId` to tail in-progress job logs instead
3. Guard the call: only invoke when EnumSet(SUCCEEDED,FAILED,KILLED,ERROR).contains(state)

Example fix

// before
RunningJob r = jc.submitJob(conf);
LogParams lp = client.getLogFileParams(jobId); // throws: still running

// after
Job job = jc.submitJob(conf);
job.waitForCompletion();   // reach terminal state first
LogParams lp = client.getLogFileParams(jobId);

JobStatus st = cluster.getJobStatus(jobId);
boolean terminal = EnumSet.of(JobStatus.State.SUCCEEDED, JobStatus.State.FAILED,
    JobStatus.State.KILLED).contains(st.getState()); // plus RUNTIME_FAILURE analogues
if (!terminal) { /* wait or poll; do not call getLogFileParams yet */ }

boolean isTerminal(JobStatus.State s) {
  return s == JobStatus.State.SUCCEEDED || s == JobStatus.State.FAILED
      || s == JobStatus.State.KILLED;
}

try { lp = cluster.getLogFileParams(jobId); }
catch (IOException e) {
  if (e.getMessage().contains("in-progress job")) {
    job.waitForCompletion(); lp = cluster.getLogFileParams(jobId);
  } else throw e;
}

When it happens

Trigger: Calling Cluster.getLogFileParams/JobClient.getLogFileParams while the job is RUNNING, NEW, or ACCEPTED — before JobMonitor completion (e.g. calling from a thread that did not waitForCompletion).

Common situations: Calling getLogFileParams right after submitJob instead of after job completion; Polling loop that races the job's transition to a terminal state and queries logs on the last RUNNING poll; Reusing sample code that assumed a synchronous job run

Related errors

• Unrecognized job state: {}
• Unable to get log information for task: {oldTaskAttemptID}
• Unable to get log information for job: {oldJobID}
• Unrecognized task type: {}
• Unrecognized State: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4ab72b0e6a3d2ab7. Report an issue: GitHub.