floci-io/floci · critical · IllegalStateException

Could not create build working directory " +…

Error message

Could not create build working directory  " + containerSrcDir + ":  " + workDirResult.errorMessage()

What it means

An IllegalStateException thrown by CodeBuildRunner when the awaited 'mkdir -p <containerSrcDir>' exec inside the just-started build container fails. The mkdir runs from '/' with the build's env before any source copy or phase exec so those cannot race container startup; if it fails the container is unusable and the build cannot proceed. The message embeds the exec's stderr errorMessage().

Solutions

Inspect the embedded errorMessage() and docker logs for the container to find why mkdir failed
Use an image with a standard shell and coreutils (e.g. aws/codebuild/standard:7.0 or ubuntu-based) as the build environment image
Verify Docker disk space and daemon health (docker system df, docker run --rm <image> mkdir -p /tmp/x)
Check that the workspace/source path configuration does not collide with an existing read-only path in the image

Example fix

# before
environment:
  image: scratch-like/no-shell

# after
environment:
  image: aws/codebuild/standard:7.0

Defensive patterns

Strategy: retry

Validate before calling

# Pre-flight the image can run mkdir
docker run --rm "$IMAGE" sh -c 'mkdir -p /codebuild/tmp' || echo "image lacks shell/mkdir - build will fail"

Try / catch

// Container-level failures can be transient (daemon/disk) - retry only after fixing the image
catch (IllegalStateException e) {
    if (e.getMessage() != null && e.getMessage().contains("working directory")) {
        // inspect docker logs, fix image or daemon, then retry the build
        markBuildForRetryAfterImageFix(buildId);
    } else throw e;
}

Prevention

Use shell-capable images (aws/codebuild/standard:7.0) for builds
Monitor Docker disk space on the Floci host
Test custom images with a trivial mkdir exec before wiring them into projects

When it happens

Trigger: The Docker container exited or crashed immediately after start (bad entrypoint, missing image); the image lacks mkdir or a POSIX shell so the exec fails; the mounted volume or workspace path is invalid inside the container; Docker daemon errors (disk full, IO) cause the exec to return non-zero.

Common situations: Using a custom environment image (e.g. minimal distroless or scratch) that has no mkdir binary; image pull silently produced a wrong/broken tag; Docker host out of disk or overlayfs errors; container runtime (podman/rootless) restrictions on exec.

Related errors

AI-assisted analysis of floci-io/floci@62ff490619 (2026-08-14). Data as JSON: /api/errors/614ac2e12c471120. Report an issue: GitHub.

Appendix: source

Thrown at src/main/java/io/github/hectorvent/floci/services/codebuild/CodeBuildRunner.java:244

            containerId = info.containerId();
            runningContainers.put(buildId, containerId);

            logHandle = logStreamer.attach(containerId, logGroup, logStream, region, "codebuild:" + buildId);

            String containerSrcDir = "/codebuild/output/src/src";
            int timeoutMinutes = build.getTimeoutInMinutes() != null ? build.getTimeoutInMinutes() : 60;
            boolean buildFailed = false;

            // createAndStart returns as soon as the container's entrypoint process is
            // running, which can be before any startup command would finish. Create the
            // working directory with an explicit, awaited exec (run from "/", which always
            // exists) so the source copy, the phase execs that chdir into it, and the final
            // artifact copy can never race against container startup.
            PhaseResult workDirResult = runPhase(containerId, "/", envList,
                    List.of("mkdir -p " + containerSrcDir), timeoutMinutes, stopFlag);
            if (workDirResult.stopped()) { finishStopped(build); return; }
            if (workDirResult.failed()) {
                throw new IllegalStateException("Could not create build working directory "
                        + containerSrcDir + ": " + workDirResult.errorMessage());
            }

            // Copy downloaded source files into the container (no-op for NO_SOURCE builds)
            copySourceToContainer(containerId, workspace, containerSrcDir);

            // INSTALL
            if (stopFlag.get()) { finishStopped(build); return; }
            beginPhase(build, "INSTALL");
            build.setCurrentPhase("INSTALL");
            PhaseResult installResult = runPhase(containerId, containerSrcDir, envList,
                    buildspec.installCommands(), timeoutMinutes, stopFlag);
            if (installResult.stopped()) { finishStopped(build); return; }
            if (installResult.failed()) {
                completePhaseWithError(build, "INSTALL", "FAILED", installResult.errorMessage());
                buildFailed = true;
            } else {
                completePhase(build, "INSTALL", "SUCCEEDED");

View on GitHub (pinned to 62ff490619)