{"record":{"id":"8dbf2fbf7ae07150","repo":"affaan-m/ECC","slug":"artifact-byte-count-exceeded-during-reading","errorCode":null,"errorMessage":"artifact byte count exceeded during reading","messagePattern":"artifact byte count exceeded during reading","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"skills/taste-application/scripts/tasteforge/integration.py","lineNumber":137,"sourceCode":"        flags = os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK\n        parent = _parent_fd(path)\n        before = os.stat(path.name, dir_fd=parent, follow_symlinks=False)\n        if not stat.S_ISREG(before.st_mode) or getattr(before, \"st_flags\", 0) & 0x40000000:\n            raise ValueError(\"artifact must be a resident regular file\")\n        if expected_size is None:\n            expected_size = before.st_size\n        if parse_json and expected_size > _MAX_JSON:\n            raise ValueError(\"JSON artifact exceeds local size limit\")\n        if before.st_size != expected_size:\n            raise ValueError(\"artifact byte count mismatch\")\n        descriptor = os.open(path.name, flags, dir_fd=parent)\n        if _identity(before) != _identity(os.fstat(descriptor)):\n            raise ValueError(\"artifact changed before reading\")\n        digest, chunks, count = hashlib.sha256(), [], 0\n        while data := os.read(descriptor, 65536):\n            count += len(data)\n            if count > expected_size:\n                raise ValueError(\"artifact byte count exceeded during reading\")\n            digest.update(data)\n            if parse_json:\n                chunks.append(data)\n        # Rewalk the named path: a pinned old directory fd can outlive a rename.\n        fresh_parent = _parent_fd(path)\n        try:\n            after = os.stat(path.name, dir_fd=fresh_parent, follow_symlinks=False)\n        finally:\n            os.close(fresh_parent)\n        if (_identity(before) != _identity(os.fstat(descriptor))\n                or _identity(before) != _identity(after)):\n            raise ValueError(\"artifact changed during reading\")\n        if expected_hash is not None and digest.hexdigest() != expected_hash:\n            raise ValueError(\"artifact SHA-256 mismatch\")\n        return _load_json(b\"\".join(chunks)) if parse_json else None\n    except (OSError, AttributeError) as exc:\n        raise ValueError(\"local artifact unavailable or unsafe\") from exc\n    finally:","sourceCodeStart":119,"sourceCodeEnd":155,"githubUrl":"https://github.com/affaan-m/ECC/blob/8321021c54d670126ce3b2969d5deb880b4b0c2a/skills/taste-application/scripts/tasteforge/integration.py#L119-L155","documentation":"While streaming the file in 64KiB chunks, `_read_local` tracks the cumulative byte count and aborts immediately if it exceeds `expected_size`. This catches files that grew after the initial size check or descriptor-level identity checks — a defensive backstop against a file being appended to concurrently while it is being read.","triggerScenarios":"A writer appends to the artifact file between the `os.open` and the read loop, making the actual byte stream longer than the stat-verified `expected_size`; a misreported or sparse size vs. actual readable bytes on an unusual filesystem.","commonSituations":"A logging tool still writing to the artifact; `tee`/append redirection continuing into the output file; a producer that appends a footer/trailer after the hash and size were recorded.","solutions":["Ensure no process appends to the artifact while it is being loaded; wait for the writer to close the file (e.g. check for a `.done` marker).","Regenerate the artifact and its recorded size/hash after all writers finish, then retry the load.","Retry the load once the concurrent appender has stopped — the pre-read checks will then pass.","Change the producer to write atomically (temp file + `os.replace`) instead of appending in place."],"exampleFix":"// before\nsome_tool >> /out/artifact.json   # still appending during load\n// after\nwait_for_done_marker(\"/out/artifact.json.done\")\nload_application_request(\"/out/request.json\")","handlingStrategy":"retry","validationCode":"import os, time\ndef wait_for_quiet(path: str, quiet_secs: float = 1.0, timeout: float = 30.0) -> None:\n    deadline = time.time() + timeout\n    last = os.path.getsize(path)\n    while time.time() < deadline:\n        time.sleep(quiet_secs)\n        cur = os.path.getsize(path)\n        if cur == last:\n            return\n        last = cur\n    raise TimeoutError(f\"file still growing: {path}\")","typeGuard":null,"tryCatchPattern":"try:\n    req = load_application_request(p)\nexcept ValueError as e:\n    if str(e) == \"artifact byte count exceeded during reading\":\n        wait_for_quiet(p)\n        req = load_application_request(p)   # identity + size checks now pass\n    else:\n        raise","preventionTips":["Stop all appenders (log redirections, tee, streaming writers) before loading the artifact.","Use completion markers so consumers only read files the producer has closed.","Switch producers from in-place appends to atomic temp-file + rename publishing.","Record size/hash only after the producer has fully finished writing."],"tags":["race-condition","filesystem","size-limit"],"backgroundTag":"file-changed-during-read","analyzedSha":"8321021c54d670126ce3b2969d5deb880b4b0c2a","analyzedAt":"2026-09-16T10:08:13.343Z","contentChangedAt":"2026-09-16T10:08:13.343Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}