HumanSignal/label-studio · error · ValidationError

Failed to list storage files

Error message

Failed to list storage files

What it means

The storage file-listing action endpoint catches any exception raised while scanning the storage (listing objects from S3/GCS/Azure) and re-raises it as ValidationError('Failed to list storage files'), logging the real cause. The generic message hides provider-specific failures like auth errors, timeouts, or bad prefixes.

Source

Thrown at label_studio/io_storages/api.py:231

            timeout_seconds = 30

            for object in instance.iter_objects():
                files.append(instance.get_unified_metadata(object))

                # Check if we've reached the file limit
                if len(files) >= limit:
                    files.append({'key': None, 'last_modified': None, 'size': None})
                    break

                # Check if we've exceeded the timeout
                if time.time() - start_time > timeout_seconds:
                    files.append({'key': '... storage scan timeout reached ...', 'last_modified': None, 'size': None})
                    break

            return Response({'files': files})
        except Exception as exc:
            logger.exception('Error listing storage files: %s', exc)
            raise ValidationError('Failed to list storage files')


@extend_schema(exclude=True)
class StorageFormLayoutAPI(generics.RetrieveAPIView):
    permission_required = all_permissions.storages_change
    parser_classes = (JSONParser, FormParser, MultiPartParser)
    storage_type = None

    @extend_schema(exclude=True)
    def get(self, request, *args, **kwargs):
        form_layout_file = os.path.join(os.path.dirname(inspect.getfile(self.__class__)), 'form_layout.yml')
        if not os.path.exists(form_layout_file):
            raise NotFound(f'"form_layout.yml" is not found for {self.__class__.__name__}')

        form_layout = read_yaml(form_layout_file)
        form_layout = self.post_process_form(form_layout)
        return Response(form_layout[self.storage_type])

View on GitHub (pinned to 0b49e9b539)

Solutions

1. Check server logs for the 'Error listing storage files' traceback to see the provider error
2. Re-validate/re-save the storage connection with correct credentials
3. Reduce the prefix to limit the scan scope, or increase timeout settings
4. Retry — transient throttling/network errors resolve on retry
5. Verify the bucket/container still exists and the storage record points to it

Example fix

// before
POST /api/storages/s3/42/list-files  → 400 Failed to list storage files (logs: AccessDenied)
// after
PUT /api/storages/s3/42 {"aws_access_key_id": "<fixed-key>"} then POST list-files → 200 {"files": [...] }

# validate the storage before calling list-files
requests.post(f"{LS_URL}/api/storages/s3/{sid}/validate/", headers=headers).raise_for_status()

def storage_is_valid(resp):
    return resp.status_code == 200

import time
for attempt in range(3):
    try:
        resp = requests.post(f"{LS_URL}/api/storages/s3/{sid}/list-files/", headers=headers, timeout=120)
        resp.raise_for_status()
        break
    except requests.RequestException:
        check_server_logs_for_traceback()  # real cause is logged there
        time.sleep(2 ** attempt)

When it happens

Trigger: POST to /api/storages/<type>/<id>/list-files (sync action returning files) where the underlying SDK call to list blobs/objects raises — invalid credentials, missing bucket, throttling, network timeout, or a scan exceeding the iteration timeout.

Common situations: Very large containers timing out during scan; expired cloud session; bucket renamed after storage was created; rate limiting (S3 SlowDown, Azure 5xx); DNS/egress blocked in self-hosted deployments.

Related errors

• Can't resolve hostname {domain}
• URL resolves to a reserved network address (block: {subnet})
• "file_upload_ids" parameter must be a list of integers
• extract_message(e)
• sample() does not accept arguments.

AI-assisted analysis of HumanSignal/label-studio@0b49e9b539 (2026-08-29). Data as JSON: /api/errors/26ac3f7cd26c828c. Report an issue: GitHub.