{"record":{"id":"6026a1e7bd981323","repo":"FlowiseAI/Flowise","slug":"failed-to-download-file-key-from-s3-bucket-bu","errorCode":null,"errorMessage":"Failed to download file ${key} from S3 bucket ${bucketName}: ${e.message}","messagePattern":"Failed to download file (.+?) from S3 bucket (.+?): (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/components/nodes/documentloaders/S3Directory/S3Directory.ts","lineNumber":219,"sourceCode":"                        const objectData = await new Promise<Buffer>((resolve, reject) => {\n                            const chunks: Buffer[] = []\n\n                            if (response.Body instanceof Readable) {\n                                response.Body.on('data', (chunk: Buffer) => chunks.push(chunk))\n                                response.Body.on('end', () => resolve(Buffer.concat(chunks)))\n                                response.Body.on('error', reject)\n                            } else {\n                                reject(new Error('Response body is not a readable stream.'))\n                            }\n                        })\n\n                        // create the directory if it doesnt already exist\n                        fsDefault.mkdirSync(path.dirname(filePath), { recursive: true })\n\n                        // write the file to the directory\n                        fsDefault.writeFileSync(filePath, objectData)\n                    } catch (e: any) {\n                        throw new Error(`Failed to download file ${key} from S3 bucket ${bucketName}: ${e.message}`)\n                    }\n                })\n            )\n\n            const loader = new DirectoryLoader(\n                tempDir,\n                {\n                    '.json': (path) => new JSONLoader(path),\n                    '.txt': (path) => new TextLoader(path),\n                    '.csv': (path) => new CSVLoader(path),\n                    '.xls': (path) => new LoadOfSheet(path),\n                    '.xlsx': (path) => new LoadOfSheet(path),\n                    '.xlsm': (path) => new LoadOfSheet(path),\n                    '.xlsb': (path) => new LoadOfSheet(path),\n                    '.docx': (path) => new DocxLoader(path),\n                    '.ppt': (path) => new PowerpointLoader(path),\n                    '.pptx': (path) => new PowerpointLoader(path),\n                    '.pdf': (path) =>","sourceCodeStart":201,"sourceCodeEnd":237,"githubUrl":"https://github.com/FlowiseAI/Flowise/blob/abe4a8601a058047b350c260676826e21dd14101/packages/components/nodes/documentloaders/S3Directory/S3Directory.ts#L201-L237","documentation":"Thrown per-object inside S3Directory's Promise.all when downloading a single key fails. Wraps the underlying S3 SDK / filesystem error (network, NoSuchKey, permissions, stream error, or 'Response body is not a readable stream') and interpolates its message. Because it throws inside Promise.all, the first failing key aborts the whole batch.","triggerScenarios":"GetObject returns NoSuchKey (key deleted between List and Get); access denied (bucket policy/IAM denies s3:GetObject); socket/stream 'error' event fires mid-download; response.Body is not a Readable (SDK v3 streaming shape change); disk full or permission error on fsDefault.writeFileSync.","commonSituations":"Bucket policy tightened mid-run; very large object timing out the stream; IAM role missing s3:GetObject; tempDir filesystem read-only or out of space; AWS SDK v3 version where Body is a Blob/Readable-Native mismatch.","solutions":["Confirm the IAM credential has s3:GetObject on the bucket/prefix.","Check the key still exists (List then Get can race for mutable buckets) — re-list or skip missing keys.","Verify tempDir is writable and has free space.","Make the per-key failure non-fatal by collecting errors instead of throwing inside Promise.all (so one bad key does not abort the batch).","Ensure the AWS SDK v3 version matches the streaming shape expected (Readable check)."],"exampleFix":"// before - one bad key aborts the whole batch\nawait Promise.all(keys.map(async (key) => {\n  try { /* download */ } catch (e) {\n    throw new Error(`Failed to download file ${key} from S3 bucket ${bucketName}: ${e.message}`)\n  }\n}))\n// after - collect failures, continue, surface them at the end\nconst failures: { key: string; message: string }[] = []\nawait Promise.all(keys.map(async (key) => {\n  try { /* download */ }\n  catch (e: any) { failures.push({ key, message: e.message }) }\n}))\nif (failures.length) options.logger.warn(`S3 partial failure: ${JSON.stringify(failures)}`)","handlingStrategy":"try-catch","validationCode":"// Verify IAM can list+get before the batch run\nasync function verifyS3Access(s3Client, bucket) {\n  await s3Client.send(new (require('@aws-sdk/client-s3').ListObjectsV2Command)({ Bucket: bucket, MaxKeys: 1 }))\n}","typeGuard":"const { Readable } = require('stream')\nfunction isReadable(body) { return body instanceof Readable }","tryCatchPattern":"// Make per-key failure non-fatal; collect and continue\nconst failures = []\nawait Promise.all(keys.map(async (key) => {\n  try { /* download key */ }\n  catch (e) { failures.push({ key, message: e.message }) }\n}))\nif (failures.length) throw new Error(`S3 partial download failure: ${JSON.stringify(failures)}`)","preventionTips":["Grant IAM s3:GetObject on the bucket/prefix.","Confirm tempDir is writable with free disk space.","Match the AWS SDK v3 stream shape to the Readable check.","Avoid aborting the whole batch on one bad key — collect failures."],"tags":["s3","aws-sdk","download","stream","iam","batch"],"backgroundTag":null,"analyzedSha":"abe4a8601a058047b350c260676826e21dd14101","analyzedAt":"2026-08-12T16:04:40.823Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}