floci-io/floci · critical · IllegalStateException

Could not create build working directory " + containerSrcDi

Error message

Could not create build working directory  " + containerSrcDir + ":  " + workDirResult.errorMessage()

What it means

An IllegalStateException thrown by CodeBuildRunner when the awaited 'mkdir -p <containerSrcDir>' exec inside the just-started build container fails. The mkdir runs from '/' with the build's env before any source copy or phase exec so those cannot race container startup; if it fails the container is unusable and the build cannot proceed. The message embeds the exec's stderr errorMessage().

Source

Thrown at src/main/java/io/github/hectorvent/floci/services/codebuild/CodeBuildRunner.java:244

            containerId = info.containerId();
            runningContainers.put(buildId, containerId);

            logHandle = logStreamer.attach(containerId, logGroup, logStream, region, "codebuild:" + buildId);

            String containerSrcDir = "/codebuild/output/src/src";
            int timeoutMinutes = build.getTimeoutInMinutes() != null ? build.getTimeoutInMinutes() : 60;
            boolean buildFailed = false;

            // createAndStart returns as soon as the container's entrypoint process is
            // running, which can be before any startup command would finish. Create the
            // working directory with an explicit, awaited exec (run from "/", which always
            // exists) so the source copy, the phase execs that chdir into it, and the final
            // artifact copy can never race against container startup.
            PhaseResult workDirResult = runPhase(containerId, "/", envList,
                    List.of("mkdir -p " + containerSrcDir), timeoutMinutes, stopFlag);
            if (workDirResult.stopped()) { finishStopped(build); return; }
            if (workDirResult.failed()) {
                throw new IllegalStateException("Could not create build working directory "
                        + containerSrcDir + ": " + workDirResult.errorMessage());
            }

            // Copy downloaded source files into the container (no-op for NO_SOURCE builds)
            copySourceToContainer(containerId, workspace, containerSrcDir);

            // INSTALL
            if (stopFlag.get()) { finishStopped(build); return; }
            beginPhase(build, "INSTALL");
            build.setCurrentPhase("INSTALL");
            PhaseResult installResult = runPhase(containerId, containerSrcDir, envList,
                    buildspec.installCommands(), timeoutMinutes, stopFlag);
            if (installResult.stopped()) { finishStopped(build); return; }
            if (installResult.failed()) {
                completePhaseWithError(build, "INSTALL", "FAILED", installResult.errorMessage());
                buildFailed = true;
            } else {
                completePhase(build, "INSTALL", "SUCCEEDED");

View on GitHub (pinned to 62ff490619)

Solutions

Inspect the embedded errorMessage() and docker logs for the container to find why mkdir failed
Use an image with a standard shell and coreutils (e.g. aws/codebuild/standard:7.0 or ubuntu-based) as the build environment image
Verify Docker disk space and daemon health (docker system df, docker run --rm <image> mkdir -p /tmp/x)
Check that the workspace/source path configuration does not collide with an existing read-only path in the image

Example fix

# before
environment:
  image: scratch-like/no-shell

# after
environment:
  image: aws/codebuild/standard:7.0

Defensive patterns

Strategy: retry

Validate before calling

# Pre-flight the image can run mkdir
docker run --rm "$IMAGE" sh -c 'mkdir -p /codebuild/tmp' || echo "image lacks shell/mkdir - build will fail"

Try / catch

// Container-level failures can be transient (daemon/disk) - retry only after fixing the image
catch (IllegalStateException e) {
    if (e.getMessage() != null && e.getMessage().contains("working directory")) {
        // inspect docker logs, fix image or daemon, then retry the build
        markBuildForRetryAfterImageFix(buildId);
    } else throw e;
}

Prevention

Use shell-capable images (aws/codebuild/standard:7.0) for builds
Monitor Docker disk space on the Floci host
Test custom images with a trivial mkdir exec before wiring them into projects

When it happens

Trigger: The Docker container exited or crashed immediately after start (bad entrypoint, missing image); the image lacks mkdir or a POSIX shell so the exec fails; the mounted volume or workspace path is invalid inside the container; Docker daemon errors (disk full, IO) cause the exec to return non-zero.

Common situations: Using a custom environment image (e.g. minimal distroless or scratch) that has no mkdir binary; image pull silently produced a wrong/broken tag; Docker host out of disk or overlayfs errors; container runtime (podman/rootless) restrictions on exec.

Related errors

AI-assisted analysis of floci-io/floci@62ff490619 (2026-08-14). Data as JSON: /api/errors/614ac2e12c471120. Report an issue: GitHub.