# Aniworld Web Application Infrastructure conda activate AniWorld ## Project Structure ``` /home/lukas/Volume/repo/Aniworld/ ├── src/ │ ├── core/ # Core application logic │ │ ├── SeriesApp.py # Main application class with async support │ │ ├── SerieScanner.py # Directory scanner for anime series │ │ ├── entities/ # Domain entities │ │ │ ├── series.py # Serie data model │ │ │ └── SerieList.py # Series list management │ │ ├── interfaces/ # Abstract interfaces │ │ │ └── providers.py # Provider interface definitions │ │ ├── providers/ # Content providers │ │ │ ├── base_provider.py # Base loader interface │ │ │ ├── aniworld_provider.py # Aniworld.to implementation │ │ │ ├── provider_factory.py # Provider factory │ │ │ └── streaming/ # Streaming providers (VOE, etc.) │ │ └── exceptions/ # Custom exceptions │ │ └── Exceptions.py # Exception definitions │ ├── server/ # FastAPI web application │ │ ├── fastapi_app.py # Main FastAPI application (simplified) │ │ ├── main.py # FastAPI application entry point │ │ ├── controllers/ # Route controllers │ │ │ ├── __init__.py # Controllers package │ │ │ ├── health_controller.py # Health check endpoints │ │ │ ├── page_controller.py # HTML page routes │ │ │ └── error_controller.py # Error handling controllers │ │ ├── api/ # API route handlers │ │ │ ├── __init__.py │ │ │ ├── auth.py # Authentication endpoints │ │ │ ├── config.py # Configuration endpoints │ │ │ ├── anime.py # Anime management endpoints │ │ │ ├── download.py # Download queue endpoints │ │ │ ├── websocket.py # WebSocket real-time endpoints │ │ │ └── search.py # Search endpoints │ │ ├── models/ # Pydantic models │ │ │ ├── __init__.py │ │ │ ├── auth.py │ │ │ ├── config.py │ │ │ ├── anime.py │ │ │ ├── download.py │ │ │ └── websocket.py # WebSocket message models │ │ ├── services/ # Business logic services │ │ │ ├── __init__.py │ │ │ ├── auth_service.py │ │ │ ├── config_service.py │ │ │ ├── anime_service.py │ │ │ ├── download_service.py │ │ │ └── websocket_service.py # WebSocket connection management │ │ ├── utils/ # Utility functions │ │ │ ├── __init__.py │ │ │ ├── security.py │ │ │ ├── dependencies.py # Dependency injection │ │ │ ├── templates.py # Shared Jinja2 template config │ │ │ ├── template_helpers.py # Template rendering utilities │ │ │ └── logging.py # Logging utilities │ │ └── web/ # Frontend assets │ │ ├── templates/ # Jinja2 HTML templates │ │ │ ├── index.html # Main application page │ │ │ ├── login.html # Login page │ │ │ ├── setup.html # Initial setup page │ │ │ ├── queue.html # Download queue page │ │ │ └── error.html # Error page │ │ └── static/ # Static web assets │ │ ├── css/ │ │ │ ├── styles.css # Main styles │ │ │ └── ux_features.css # UX enhancements │ │ └── js/ │ │ ├── app.js # Main application logic │ │ ├── queue.js # Queue management │ │ ├── localization.js # i18n support │ │ ├── keyboard_shortcuts.js # Keyboard navigation │ │ ├── user_preferences.js # User settings │ │ ├── undo_redo.js # Undo/redo system │ │ ├── mobile_responsive.js # Mobile support │ │ ├── touch_gestures.js # Touch interactions │ │ ├── accessibility_features.js # A11y features │ │ ├── screen_reader_support.js # Screen reader │ │ ├── color_contrast_compliance.js # WCAG compliance │ │ ├── multi_screen_support.js # Multi-monitor │ │ ├── drag_drop.js # Drag and drop │ │ ├── bulk_operations.js # Bulk actions │ │ └── advanced_search.js # Search filters │ ├── core/ # Existing core functionality │ └── cli/ # Existing CLI application ├── data/ # Application data storage │ ├── config.json # Application configuration │ ├── anime_library.db # SQLite database for anime library │ ├── download_queue.json # Download queue state │ └── cache/ # Temporary cache files ├── logs/ # Application logs │ ├── app.log # Main application log │ ├── download.log # Download-specific logs │ └── error.log # Error logs ├── requirements.txt # Python dependencies ├── docker-compose.yml # Docker deployment configuration └── README.md ``` ## Technology Stack ### Backend - **FastAPI**: Modern Python web framework for building APIs - **Uvicorn**: ASGI server for running FastAPI applications - **SQLite**: Lightweight database for storing anime library and configuration - **Pydantic**: Data validation and serialization - **Jinja2**: Template engine for server-side rendering ### Frontend - **HTML5/CSS3**: Core web technologies - **JavaScript (Vanilla)**: Client-side interactivity - **Bootstrap 5**: CSS framework for responsive design - **HTMX**: Modern approach for dynamic web applications ### Security - **Passlib**: Password hashing and verification - **python-jose**: JWT token handling - **bcrypt**: Secure password hashing ### Authentication Models & Sessions - Authentication request/response Pydantic models live in `src/server/models/auth.py`. - Sessions are represented by `SessionModel` and can be backed by an in-memory store or a persistent table depending on deployment needs. JWTs are used for stateless authentication by default; a persistent session store may be configured in production to enable revocation and long-lived sessions. ## Configuration ### Data Storage - **Configuration**: JSON files in `data/` directory - **Anime Library**: SQLite database with series information - **Download Queue**: JSON file with current download status - **Logs**: Structured logging to files in `logs/` directory ## API Endpoints ### Authentication - `POST /api/auth/login` - Master password authentication - `POST /api/auth/logout` - Logout and invalidate session - `GET /api/auth/status` - Check authentication status ### Configuration - `GET /api/config` - Get current configuration - `PUT /api/config` - Update configuration - `POST /api/setup` - Initial setup ### Configuration API Notes - The configuration endpoints are exposed under `/api/config` and operate primarily on a JSON-serializable `AppConfig` model. They are designed to be lightweight and avoid performing IO during validation (the `/api/config/validate` endpoint runs in-memory checks only). - Persistence of configuration changes is intentionally "best-effort" for now and mirrors fields into the runtime settings object. A follow-up task should add durable storage (file or DB) for configs. ### Anime Management - `GET /api/anime` - List anime with missing episodes - `POST /api/anime/{id}/download` - Add episodes to download queue - `GET /api/anime/{id}` - Get anime details Note: The anime management API has been implemented under `/api/v1/anime` with endpoints for listing series with missing episodes, searching providers, triggering a local rescan, and fetching series details. The implementation delegates to the existing core `SeriesApp` and uses dependency injection for initialization. ### Download Management - `GET /api/queue/status` - Get download queue status and statistics - `POST /api/queue/add` - Add episodes to download queue - `DELETE /api/queue/{id}` - Remove item from queue - `DELETE /api/queue/` - Remove multiple items from queue - `POST /api/queue/start` - Start download queue processing - `POST /api/queue/stop` - Stop download queue processing - `POST /api/queue/pause` - Pause queue processing - `POST /api/queue/resume` - Resume queue processing - `POST /api/queue/reorder` - Reorder pending queue items - `DELETE /api/queue/completed` - Clear completed downloads - `POST /api/queue/retry` - Retry failed downloads ### Search - `GET /api/search?q={query}` - Search for anime - `POST /api/search/add` - Add anime to library ## Logging ### Log Levels - **INFO**: General application information - **WARNING**: Potential issues that don't stop execution - **ERROR**: Errors that affect functionality - **DEBUG**: Detailed debugging information (development only) ### Log Files - `app.log`: General application logs - `download.log`: Download-specific operations - `error.log`: Error and exception logs ## Security Considerations - Master password protection for application access - Secure session management with JWT tokens - Input validation and sanitization - Rate limiting on API endpoints - HTTPS enforcement in production - Secure file path handling to prevent directory traversal ### Authentication Service - A lightweight authentication service is provided by `src/server/services/auth_service.py`. - Uses bcrypt (passlib) to hash the master password and issues JWTs for stateless sessions. Tokens are signed with the `JWT_SECRET_KEY` from configuration and expire based on `SESSION_TIMEOUT_HOURS`. - Failed login attempts are tracked in-memory and a temporary lockout is applied after multiple failures. For multi-process deployments, move this state to a shared store (Redis) and persist the master password hash in a secure config store. ## Core Application Logic ### SeriesApp - Enhanced Core Engine The `SeriesApp` class (`src/core/SeriesApp.py`) is the main application engine for anime series management. Enhanced with async support and web integration capabilities. #### Key Features - **Async Operations**: Support for async download and scan operations - **Progress Callbacks**: Real-time progress reporting via callbacks - **Cancellation Support**: Ability to cancel long-running operations - **Error Handling**: Comprehensive error handling with callback notifications - **Operation Status**: Track current operation status and history #### Core Classes - `SeriesApp`: Main application class - `OperationStatus`: Enum for operation states (IDLE, RUNNING, COMPLETED, CANCELLED, FAILED) - `ProgressInfo`: Dataclass for progress information - `OperationResult`: Dataclass for operation results #### Key Methods - `search(words)`: Search for anime series - `download()`: Download episodes with progress tracking - `ReScan()`: Scan directory for missing episodes - `async_download()`: Async version of download - `async_rescan()`: Async version of rescan - `cancel_operation()`: Cancel current operation - `get_operation_status()`: Get current status - `get_series_list()`: Get series with missing episodes #### Integration Points The SeriesApp integrates with: - Provider system for content downloading - Serie scanner for directory analysis - Series list management for tracking missing episodes - Web layer via async operations and callbacks ## Recent Infrastructure Changes ### Core Logic Enhancement (October 2025) Enhanced `SeriesApp` with async callback support, progress reporting, and cancellation. #### Changes Made 1. **Async Support**: - Added `async_download()` and `async_rescan()` methods - Integrated with asyncio event loop for non-blocking operations - Support for concurrent operations in web environment 2. **Progress Reporting**: - `ProgressInfo` dataclass for structured progress data - Callback system for real-time progress updates - Percentage calculation and status tracking 3. **Cancellation System**: - Internal cancellation flag management - Graceful operation cancellation - Cancellation check during long-running operations 4. **Error Handling**: - `OperationResult` dataclass for operation outcomes - Error callback system for notifications - Specific exception types (IOError, OSError, RuntimeError) - Proper exception propagation and logging 5. **Status Management**: - `OperationStatus` enum for state tracking - Current operation identifier - Status getter methods for monitoring #### Test Coverage Comprehensive test suite (`tests/unit/test_series_app.py`) with 22 tests covering: - Initialization and configuration - Search functionality - Download operations with callbacks - Directory scanning with progress - Async operations - Cancellation handling - Error scenarios - Data model validation ### Template Integration (October 2025) Completed integration of HTML templates with FastAPI Jinja2 system. #### Changes Made 1. **Template Helper Utilities**: - `src/server/utils/template_helpers.py` - Template rendering utilities - Centralized base context for all templates - Template validation and listing functions - DRY principles for template rendering 2. **Enhanced CSS**: - `src/server/web/static/css/ux_features.css` - UX enhancement styles - Drag-and-drop indicators - Bulk selection styling - Keyboard navigation focus indicators - Touch gesture feedback - Mobile responsive enhancements - Accessibility features (high contrast, screen reader support) - Reduced motion support 3. **JavaScript Modules**: - `keyboard_shortcuts.js` - Keyboard navigation (Ctrl+K, Ctrl+R, etc.) - `user_preferences.js` - Settings persistence (localStorage) - `undo_redo.js` - Action history with Ctrl+Z/Ctrl+Y - `mobile_responsive.js` - Mobile detection and layout - `touch_gestures.js` - Swipe gesture handling - `accessibility_features.js` - Focus management and ARIA labels - `screen_reader_support.js` - Live regions for dynamic content - `color_contrast_compliance.js` - WCAG compliance checks - `multi_screen_support.js` - Fullscreen and multi-monitor support - `drag_drop.js` - Drag-and-drop functionality (stub) - `bulk_operations.js` - Bulk selection and actions (stub) - `advanced_search.js` - Advanced filtering (stub) 4. **Updated Controllers**: - Updated `page_controller.py` to use `template_helpers` - Updated `error_controller.py` to use `template_helpers` - Consistent context passing across all templates #### Template Features - **Responsive Design**: Mobile-first approach with viewport meta tags - **Theme Switching**: Light/dark mode with `data-theme` attribute - **Accessibility**: ARIA labels, keyboard navigation, screen reader support - **Internationalization**: Localization support via `localization.js` - **Progressive Enhancement**: Works without JavaScript, enhanced with it #### Verified Templates All HTML templates properly integrated: - `index.html` - Main application page with search and anime list - `login.html` - Master password authentication - `setup.html` - Initial application setup - `queue.html` - Download queue management - `error.html` - Error pages (404, 500) All templates include: - Proper HTML5 structure - Font Awesome icons - Static file references (`/static/css/`, `/static/js/`) - Theme switching support - Responsive viewport configuration ### CSS Integration (October 2025) Integrated existing CSS styling with FastAPI's static file serving system. #### Implementation Details 1. **Static File Configuration**: - Static files mounted at `/static` in `fastapi_app.py` - Directory: `src/server/web/static/` - Files served using FastAPI's `StaticFiles` middleware - All paths use absolute references (`/static/...`) 2. **CSS Architecture**: - `styles.css` (1,840 lines) - Main stylesheet with Fluent UI design system - `ux_features.css` (203 lines) - Enhanced UX features and accessibility 3. **Design System** (`styles.css`): - **Fluent UI Variables**: CSS custom properties for consistent theming - **Light/Dark Themes**: Dynamic theme switching via `[data-theme="dark"]` - **Typography**: Segoe UI font stack with responsive sizing - **Spacing System**: Consistent spacing scale (xs through xxl) - **Color Palette**: Comprehensive color system for both themes - **Border Radius**: Standardized corner radii (sm, md, lg, xl) - **Shadows**: Elevation system with card and elevated variants - **Transitions**: Smooth animations with consistent timing 4. **UX Features** (`ux_features.css`): - Drag-and-drop indicators - Bulk selection styling - Keyboard focus indicators - Touch gesture feedback - Mobile responsive utilities - High contrast mode support (`@media (prefers-contrast: high)`) - Screen reader utilities (`.sr-only`) - Window control components #### CSS Variables **Color System**: ```css /* Light Theme */ --color-bg-primary: #ffffff --color-accent: #0078d4 --color-text-primary: #323130 /* Dark Theme */ --color-bg-primary-dark: #202020 --color-accent-dark: #60cdff --color-text-primary-dark: #ffffff ``` **Spacing & Typography**: ```css --spacing-sm: 8px --spacing-md: 12px --spacing-lg: 16px --font-size-body: 14px --font-size-title: 20px ``` #### Template CSS References All HTML templates correctly reference CSS files: - Index page: Includes both `styles.css` and `ux_features.css` - Other pages: Include `styles.css` - All use absolute paths: `/static/css/styles.css` #### Responsive Design - Mobile-first approach with breakpoints - Media queries for tablet and desktop layouts - Touch-friendly interface elements - Adaptive typography and spacing #### Accessibility Features - WCAG-compliant color contrast - High contrast mode support - Screen reader utilities - Keyboard navigation styling - Focus indicators - Reduced motion support #### Testing Comprehensive test suite in `tests/unit/test_static_files.py`: - CSS file accessibility tests - Theme support verification - Responsive design validation - Accessibility feature checks - Content integrity validation - Path correctness verification All 17 CSS integration tests passing. ### Route Controller Refactoring (October 2025) Restructured the FastAPI application to use a controller-based architecture for better code organization and maintainability. #### Changes Made 1. **Created Controller Structure**: - `src/server/controllers/` - New directory for route controllers - `src/server/controllers/__init__.py` - Controllers package initialization - `src/server/controllers/health_controller.py` - Health check endpoints - `src/server/controllers/page_controller.py` - HTML page routes - `src/server/controllers/error_controller.py` - Error handling controllers 2. **Shared Template Configuration**: - `src/server/utils/templates.py` - Centralized Jinja2 template configuration - Fixed template path resolution for proper template loading 3. **Main Application Updates**: - `src/server/fastapi_app.py` - Refactored to use controller routers - Removed direct route definitions from main file - Added router inclusion using `app.include_router()` - Simplified error handlers to delegate to controller functions 4. **Fixed Import Issues**: - Resolved circular import in `src/core/__init__.py` - Removed non-existent `application` module import #### Controller Architecture ### Anime Service Notes - The new `anime_service` runs the existing blocking `SeriesApp` inside a threadpool (via ThreadPoolExecutor). This keeps the FastAPI event loop responsive while leveraging the existing core logic. - A small in-process LRU cache is used for the frequently-read "missing episodes" list to reduce IO; cache invalidation happens after a rescan. - For multi-worker or multi-host deployments, move cache/state to a shared store (Redis) and ensure the threadpool sizing matches the worker's CPU and IO profile. **Health Controller** (`health_controller.py`): ```python router = APIRouter(prefix="/health", tags=["health"]) @router.get("") - Health check endpoint ``` **Page Controller** (`page_controller.py`): ```python router = APIRouter(tags=["pages"]) @router.get("/") - Main application page @router.get("/setup") - Setup page @router.get("/login") - Login page @router.get("/queue") - Download queue page ``` **Error Controller** (`error_controller.py`): ```python async def not_found_handler() - Custom 404 error handling async def server_error_handler() - Custom 500 error handling ``` #### Benefits of the New Structure - **Separation of Concerns**: Each controller handles specific functionality - **Modularity**: Easy to add new controllers and routes - **Testability**: Controllers can be tested independently - **Maintainability**: Cleaner code organization and easier debugging - **Scalability**: Simple to extend with new features #### Verified Working Endpoints All endpoints tested and confirmed working: - Health: `/health` → Returns `{"status": "healthy", ...}` - Root: `/` → Serves main application page - Setup: `/setup` → Serves setup page - Auth API: `/api/auth/*` → Endpoints for setup, login, logout and status (JWT-based) - Login: `/login` → Serves login page - Queue: `/queue` → Serves download queue page #### File Structure After Refactoring ``` src/server/ ├── fastapi_app.py # Main FastAPI application (simplified) ├── controllers/ # NEW: Route controllers │ ├── __init__.py # Controllers package ├── utils/ │ ├── dependencies.py # Existing dependency injection │ └── templates.py # NEW: Shared Jinja2 template config └── web/ # Existing frontend assets ├── templates/ # HTML templates └── static/ # CSS, JS, images ``` ### Authentication Middleware (October 2025) An authentication middleware component was added to the FastAPI application to centralize token parsing and provide lightweight protection of authentication endpoints: - `src/server/middleware/auth.py` implements: - Bearer JWT parsing and session attachment to `request.state.session` - A simple per-IP in-memory rate limiter applied to `/api/auth/login` and `/api/auth/setup` (default 5 requests/minute) Notes: - This is intentionally simple and designed for single-process deployments. For production use across multiple workers or hosts, replace the in-memory limiter with a distributed store (e.g. Redis) and add a persistent token revocation list if needed. ### API Models and Contracts - Pydantic models living in `src/server/models/` define the canonical API contracts used by FastAPI endpoints. These models are intentionally lightweight and focused on serialization, validation, and OpenAPI documentation generation. - Keep models stable: changes to model shapes are breaking changes for clients. Bump API versioning or provide migration layers when altering public response fields. - Infrastructure considerations: ensure the deployment environment has required libraries (e.g., `pydantic`) installed and that schema validation errors are logged to the centralized logging system. For high-throughput routes, consider response model caching at the application or reverse-proxy layer. ### WebSocket Real-time Communication (October 2025) A comprehensive WebSocket infrastructure was implemented to provide real-time updates for downloads, queue status, and system events: - **File**: `src/server/services/websocket_service.py` - **Models**: `src/server/models/websocket.py` - **Endpoint**: `ws://host:port/ws/connect` #### WebSocket Service Architecture - **ConnectionManager**: Low-level connection lifecycle management - Connection registry with unique connection IDs - Room-based messaging for topic subscriptions - Automatic connection cleanup and health monitoring - Thread-safe operations with asyncio locks - **WebSocketService**: High-level application messaging - Convenient interface for broadcasting application events - Pre-defined message types for downloads, queue, and system events - Singleton pattern via `get_websocket_service()` factory #### Supported Message Types - **Download Events**: `download_progress`, `download_complete`, `download_failed` - **Queue Events**: `queue_status`, `queue_started`, `queue_stopped`, `queue_paused`, `queue_resumed` - **System Events**: `system_info`, `system_warning`, `system_error` - **Connection**: `connected`, `ping`, `pong`, `error` #### Room-Based Messaging Clients can subscribe to specific topics (rooms) to receive targeted updates: - `downloads` room: All download-related events - Custom rooms: Can be added for specific features #### Integration with Download Service - Download service automatically broadcasts progress updates via WebSocket - Broadcast callback registered during service initialization - Updates sent to all clients subscribed to the `downloads` room - No blocking of download operations (async broadcast) #### Client Connection Flow 1. Client connects to `/ws/connect` endpoint 2. Server assigns unique connection ID and sends confirmation 3. Client joins rooms (e.g., `{"action": "join", "room": "downloads"}`) 4. Server broadcasts updates to subscribed rooms 5. Client disconnects (automatic cleanup) #### Infrastructure Notes - **Single-process**: Current implementation uses in-memory connection storage - **Production**: For multi-worker/multi-host deployments: - Move connection registry to Redis or similar shared store - Implement pub/sub for cross-process message broadcasting - Add connection persistence for recovery after restarts - **Monitoring**: WebSocket status available at `/ws/status` endpoint - **Security**: Optional authentication via JWT (user_id tracking) - **Testing**: Comprehensive unit tests in `tests/unit/test_websocket_service.py` ### Download Queue Models - Download queue models in `src/server/models/download.py` define the data structures for the download queue system. - Key models include: - `DownloadItem`: Represents a single queued download with metadata, progress tracking, and error information - `QueueStatus`: Overall queue state with active, pending, completed, and failed downloads - `QueueStats`: Aggregated statistics for monitoring queue performance - `DownloadProgress`: Real-time progress information (percent, speed, ETA) - `DownloadRequest`/`DownloadResponse`: API request/response contracts - Models enforce validation constraints (e.g., positive episode numbers, progress percentage 0-100, non-negative retry counts) and provide clean JSON serialization for API endpoints and WebSocket updates. ### Download Queue Service - The download service (`src/server/services/download_service.py`) manages the complete lifecycle of anime episode downloads. - Core features: - **Priority-based Queue**: Items added with HIGH priority are processed first, NORMAL and LOW follow in FIFO order - **Concurrent Processing**: Configurable max concurrent downloads (default 2) to optimize bandwidth usage - **Persistence**: Queue state is automatically saved to `data/download_queue.json` and recovered on service restart - **Retry Logic**: Failed downloads are automatically retried up to a configurable limit (default 3 attempts) with exponential backoff - **Progress Tracking**: Real-time download progress with speed, percentage, and ETA calculations - **WebSocket Integration**: Broadcasts queue updates, progress, and completion/failure events to connected clients - Operations: - `add_to_queue()`: Add episodes to download queue with priority - `remove_from_queue()`: Cancel pending or active downloads - `reorder_queue()`: Manually adjust queue order for pending items - `pause_queue()`/`resume_queue()`: Control download processing - `retry_failed()`: Retry failed downloads with retry count checks - `get_queue_status()`: Get complete queue state (active, pending, completed, failed) - `get_queue_stats()`: Get aggregated statistics (counts, download size, speed) - Infrastructure notes: - Service uses ThreadPoolExecutor for concurrent download processing - Queue processor runs as async background task with configurable sleep intervals - Progress callbacks are executed in threadpool and broadcast via async WebSocket - For multi-process deployments, move queue state to shared store (Redis/DB) and implement distributed locking for concurrent access control - Singleton instance pattern used via `get_download_service()` factory - Testing: Comprehensive unit tests in `tests/unit/test_download_service.py` cover queue operations, persistence, retry logic, and error handling ### Download Queue API Endpoints (October 2025) Implemented comprehensive REST API endpoints for download queue management: - **File**: `src/server/api/download.py` - **Router Prefix**: `/api/queue` - **Authentication**: All endpoints require JWT authentication via `require_auth` dependency #### Implemented Endpoints 1. **GET /api/queue/status** - Retrieve complete queue status - Returns: `QueueStatusResponse` with status and statistics - Includes: active downloads, pending items, completed/failed items, queue stats 2. **POST /api/queue/add** - Add episodes to download queue - Request: `DownloadRequest` with serie info, episodes, and priority - Returns: `DownloadResponse` with added item IDs - Validates episode list is non-empty - Supports HIGH, NORMAL, and LOW priority levels 3. **DELETE /api/queue/{item_id}** - Remove single item from queue - Returns: 204 No Content on success, 404 if item not found - Cancels active downloads if necessary 4. **DELETE /api/queue/** - Remove multiple items (batch operation) - Request: `QueueOperationRequest` with list of item IDs - Returns: 204 No Content (partial success acceptable) 5. **POST /api/queue/start** - Start queue processor - Idempotent operation (safe to call multiple times) 6. **POST /api/queue/stop** - Stop queue processor - Waits for active downloads to complete (with timeout) 7. **POST /api/queue/pause** - Pause queue processing - Active downloads continue, no new downloads start 8. **POST /api/queue/resume** - Resume queue processing 9. **POST /api/queue/reorder** - Reorder pending queue item - Request: `QueueReorderRequest` with item_id and new_position - Returns: 404 if item not in pending queue 10. **DELETE /api/queue/completed** - Clear completed items from history - Returns count of cleared items 11. **POST /api/queue/retry** - Retry failed downloads - Request: `QueueOperationRequest` with item IDs (empty for all) - Only retries items under max retry limit #### Dependencies - **get_download_service**: Factory function providing singleton DownloadService instance - Automatically initializes AnimeService as dependency - Raises 503 if anime directory not configured - **get_anime_service**: Factory function providing singleton AnimeService instance - Required by DownloadService for anime operations - Both dependencies added to `src/server/utils/dependencies.py` #### Error Handling - All endpoints return structured JSON error responses - HTTP status codes follow REST conventions (200, 201, 204, 400, 401, 404, 500, 503) - Service-level exceptions (DownloadServiceError) mapped to 400 Bad Request - Generic exceptions mapped to 500 Internal Server Error - Authentication errors return 401 Unauthorized #### Testing - Comprehensive test suite in `tests/api/test_download_endpoints.py` - Tests cover: - Successful operations for all endpoints - Authentication requirements - Error conditions (empty lists, not found, service errors) - Priority handling - Batch operations - Uses pytest fixtures for authenticated client and mocked download service #### Integration - Router registered in `src/server/fastapi_app.py` via `app.include_router(download_router)` - Follows same patterns as other API routers (auth, anime, config) - Full OpenAPI documentation available at `/api/docs` ### WebSocket Integration with Core Services (October 2025) Completed comprehensive integration of WebSocket broadcasting with all core services to provide real-time updates for downloads, scans, queue operations, and progress tracking. #### ProgressService **File**: `src/server/services/progress_service.py` A centralized service for tracking and broadcasting real-time progress updates across the application. **Key Features**: - Track multiple concurrent progress operations (downloads, scans, queue changes) - Automatic progress percentage calculation - Progress lifecycle management (start, update, complete, fail, cancel) - WebSocket integration for real-time client updates - Progress history with configurable size limit (default: 50 items) - Thread-safe operations using asyncio locks - Support for progress metadata and custom messages **Progress Types**: - `DOWNLOAD` - File download progress - `SCAN` - Library scan progress - `QUEUE` - Queue operation progress - `SYSTEM` - System-level operations - `ERROR` - Error notifications **Progress Statuses**: - `STARTED` - Operation initiated - `IN_PROGRESS` - Operation in progress - `COMPLETED` - Successfully completed - `FAILED` - Operation failed - `CANCELLED` - Cancelled by user **Core Methods**: - `start_progress()` - Initialize new progress operation - `update_progress()` - Update progress with current/total values - `complete_progress()` - Mark operation as completed - `fail_progress()` - Mark operation as failed - `cancel_progress()` - Cancel ongoing operation - `get_progress()` - Retrieve progress by ID - `get_all_active_progress()` - Get all active operations (optionally filtered by type) **Broadcasting**: - Integrates with WebSocketService via callback - Broadcasts to room-specific channels (e.g., `download_progress`, `scan_progress`) - Configurable broadcast throttling (only on significant changes >1% or forced) - Automatic progress state serialization to JSON **Singleton Pattern**: - Global instance via `get_progress_service()` factory - Initialized during application startup with WebSocket callback #### Integration with Services **DownloadService Integration**: - Progress tracking for each download item - Real-time progress updates during file download - Automatic completion/failure notifications - Progress metadata includes speed, ETA, downloaded bytes **AnimeService Integration**: - Progress tracking for library scans - Scan progress with current/total file counts - Scan completion with statistics - Error notifications on scan failures #### WebSocket Message Models **File**: `src/server/models/websocket.py` Added progress-specific message models: - `ScanProgressMessage` - Scan progress updates - `ScanCompleteMessage` - Scan completion notification - `ScanFailedMessage` - Scan failure notification - `ErrorNotificationMessage` - Critical error notifications - `ProgressUpdateMessage` - Generic progress updates **WebSocket Message Types**: - `SCAN_PROGRESS` - Scan progress updates - `SCAN_COMPLETE` - Scan completion - `SCAN_FAILED` - Scan failure - Extended existing types for downloads and queue updates #### WebSocket Rooms Clients can subscribe to specific progress channels: - `download_progress` - Download progress updates - `scan_progress` - Library scan updates - `queue_progress` - Queue operation updates - `system_progress` - System-level updates Room subscription via client messages: ```json { "action": "join", "room": "download_progress" } ``` #### Application Startup **File**: `src/server/fastapi_app.py` Progress service initialized on application startup: 1. Get ProgressService singleton instance 2. Get WebSocketService singleton instance 3. Register broadcast callback to link progress updates with WebSocket 4. Callback broadcasts progress messages to appropriate rooms #### Testing **File**: `tests/unit/test_progress_service.py` Comprehensive test coverage including: - Progress lifecycle operations (start, update, complete, fail, cancel) - Percentage calculation accuracy - History management and size limits - Broadcast callback invocation - Concurrent progress operations - Metadata handling - Error conditions and edge cases #### Architecture Benefits - **Decoupling**: ProgressService decouples progress tracking from WebSocket broadcasting - **Reusability**: Single service used across all application components - **Scalability**: Supports multiple concurrent operations efficiently - **Observability**: Centralized progress tracking simplifies monitoring - **Real-time UX**: Instant feedback to users on all long-running operations #### Future Enhancements - Persistent progress history (database storage) - Progress rate calculation and trend analysis - Multi-process progress synchronization (Redis/shared store) - Progress event hooks for custom actions - Client-side progress resumption after reconnection ### Core Services WebSocket Integration (October 2025) Completed comprehensive integration of WebSocket broadcasting with all core services (DownloadService, AnimeService, ProgressService) to provide real-time updates to connected clients. #### DownloadService WebSocket Integration **File**: `src/server/services/download_service.py` The download service broadcasts real-time updates for all queue and download operations: **Download Progress Broadcasting**: - `download_progress` - Real-time progress updates during download - Includes: download_id, serie_name, season, episode, progress data (percent, speed, ETA) - Sent via ProgressService which broadcasts to `download_progress` room - Progress callback created for each download item with metadata tracking **Download Completion/Failure Broadcasting**: - `download_complete` - Successful download completion - Includes: download_id, serie_name, season, episode, downloaded_mb - Broadcast to `downloads` room - `download_failed` - Download failure notification - Includes: download_id, serie_name, season, episode, error, retry_count - Broadcast to `downloads` room **Queue Operations Broadcasting**: All queue operations broadcast `queue_status` messages with current queue state: - `items_added` - Items added to queue - Data: added_ids, queue_status (complete queue state) - `items_removed` - Items removed/cancelled - Data: removed_ids, queue_status - `queue_reordered` - Queue order changed - Data: item_id, new_position, queue_status - `items_retried` - Failed items retried - Data: retried_ids, queue_status - `completed_cleared` - Completed items cleared - Data: cleared_count, queue_status **Queue Control Broadcasting**: - `queue_started` - Queue processor started - Data: is_running=True, queue_status - `queue_stopped` - Queue processor stopped - Data: is_running=False, queue_status - `queue_paused` - Queue processing paused - Data: is_paused=True, queue_status - `queue_resumed` - Queue processing resumed - Data: is_paused=False, queue_status **Broadcast Callback Setup**: The download service broadcast callback is registered during dependency injection in `src/server/utils/dependencies.py`: - Maps update types to WebSocket service methods - Routes download_progress, download_complete, download_failed to appropriate rooms - All queue operations broadcast complete queue status for client synchronization #### AnimeService WebSocket Integration **File**: `src/server/services/anime_service.py` The anime service integrates with ProgressService for library scan operations: **Scan Progress Broadcasting**: - Scan operations use ProgressService for progress tracking - Progress updates broadcast to `scan_progress` room - Lifecycle events: - `started` - Scan initialization - `in_progress` - Ongoing scan with current/total file counts - `completed` - Successful scan completion - `failed` - Scan failure with error message **Scan Implementation**: - `rescan()` method wraps SeriesApp.ReScan with progress tracking - Progress callback executed in threadpool updates ProgressService - ProgressService automatically broadcasts to WebSocket clients - Cache invalidation on successful scan completion #### ProgressService WebSocket Integration **File**: `src/server/services/progress_service.py` Central service for tracking and broadcasting all progress operations: **Progress Types**: - `DOWNLOAD` - File download progress - `SCAN` - Library scan progress - `QUEUE` - Queue operation progress - `SYSTEM` - System-level operations - `ERROR` - Error notifications **Progress Lifecycle**: 1. `start_progress()` - Initialize progress operation - Broadcasts to room: `{progress_type}_progress` 2. `update_progress()` - Update progress values - Calculates percentage automatically - Broadcasts only on significant changes (>1% or forced) 3. `complete_progress()` - Mark operation complete - Sets progress to 100% - Moves to history - Broadcasts completion 4. `fail_progress()` - Mark operation failed - Captures error message - Moves to history - Broadcasts failure **Broadcast Callback**: - Callback registered during application startup in `src/server/fastapi_app.py` - Links ProgressService to WebSocketService.manager.broadcast_to_room - All progress updates automatically broadcast to appropriate rooms #### WebSocket Room Structure Clients subscribe to specific rooms to receive targeted updates: **Room Types**: - `downloads` - All download-related events (complete, failed, queue status) - `download_progress` - Real-time download progress updates - `scan_progress` - Library scan progress updates - `queue_progress` - Queue operation progress (future use) - `system_progress` - System-level progress (future use) **Room Subscription**: Clients join rooms by sending WebSocket messages: ```json { "action": "join", "room": "download_progress" } ``` #### Message Format All WebSocket messages follow a consistent structure: ```json { "type": "download_progress" | "download_complete" | "queue_status" | etc., "timestamp": "2025-10-17T12:34:56.789Z", "data": { // Message-specific data } } ``` **Example: Download Progress** ```json { "type": "download_progress", "timestamp": "2025-10-17T12:34:56.789Z", "data": { "download_id": "abc123", "serie_name": "Attack on Titan", "season": 1, "episode": 5, "progress": { "percent": 45.2, "downloaded_mb": 226.0, "total_mb": 500.0, "speed_mbps": 2.5, "eta_seconds": 120 } } } ``` **Example: Queue Status** ```json { "type": "queue_status", "timestamp": "2025-10-17T12:34:56.789Z", "data": { "action": "items_added", "added_ids": ["item1", "item2"], "queue_status": { "is_running": true, "is_paused": false, "active_downloads": [...], "pending_queue": [...], "completed_downloads": [...], "failed_downloads": [...] } } } ``` #### Integration Testing **File**: `tests/integration/test_websocket_integration.py` Comprehensive integration tests verify WebSocket broadcasting: **Test Coverage**: - Download progress broadcasts during active downloads - Queue operation broadcasts (add, remove, reorder, clear, retry) - Queue control broadcasts (start, stop, pause, resume) - Scan progress broadcasts (start, update, complete, fail) - Progress lifecycle broadcasts for all operation types - End-to-end flow with multiple services broadcasting **Test Strategy**: - Mock broadcast callbacks to capture emitted messages - Verify message types, data structure, and content - Test both successful and failure scenarios - Verify proper room routing for different message types #### Architecture Benefits **Decoupling**: - Services use generic broadcast callbacks without WebSocket dependencies - ProgressService provides abstraction layer for progress tracking - Easy to swap WebSocket implementation or add additional broadcast targets **Consistency**: - All services follow same broadcast patterns - Standardized message formats across application - Unified progress tracking via ProgressService **Real-time UX**: - Instant feedback on all long-running operations - Live queue status updates - Progress bars update smoothly without polling - Error notifications delivered immediately **Scalability**: - Room-based messaging enables targeted updates - Multiple concurrent operations supported - Easy to add new progress types and message formats #### Production Considerations **Single-Process Deployment** (Current): - In-memory connection registry in WebSocketService - Works perfectly for single-worker deployments - No additional infrastructure required **Multi-Process/Multi-Host Deployment** (Future): - Move connection registry to Redis or similar shared store - Implement pub/sub for cross-process message broadcasting - Add connection persistence for recovery after restarts - Consider using sticky sessions or connection migration **Performance**: - Progress updates throttled to >1% changes to reduce message volume - Broadcast operations are fire-and-forget (non-blocking) - Failed connections automatically cleaned up - Message serialization cached where possible **Monitoring**: - Structured logging for all broadcast operations - WebSocket status available at `/ws/status` endpoint - Connection count and room membership tracking - Error tracking for failed broadcasts ### Frontend Authentication Integration (October 2025) Completed JWT-based authentication integration between frontend and backend. #### Authentication Token Storage **Files Modified:** - `src/server/web/templates/login.html` - Store JWT token after successful login - `src/server/web/templates/setup.html` - Redirect to login after setup completion - `src/server/web/static/js/app.js` - Include Bearer token in all authenticated requests - `src/server/web/static/js/queue.js` - Include Bearer token in queue API calls **Implementation:** - JWT tokens stored in `localStorage` after successful login - Token expiry stored in `localStorage` for client-side validation - `Authorization: Bearer ` header included in all authenticated requests - Automatic redirect to `/login` on 401 Unauthorized responses - Token cleared from `localStorage` on logout **Key Functions Updated:** - `makeAuthenticatedRequest()` in both `app.js` and `queue.js` - `checkAuthentication()` to verify token and redirect if missing/invalid - `logout()` to clear token and redirect to login ### Frontend API Endpoint Updates (October 2025) Updated frontend JavaScript to match new backend API structure. **Queue Management API Changes:** - `/api/queue/clear` → `/api/queue/completed` for clearing completed downloads - `/api/queue/remove` → `/api/queue/{item_id}` (DELETE) for single item removal - `/api/queue/retry` payload changed to `{item_ids: []}` array format - `/api/download/pause` → `/api/queue/pause` - `/api/download/resume` → `/api/queue/resume` - `/api/download/cancel` → `/api/queue/stop` **Response Format Changes:** - Login returns `{access_token, token_type, expires_at}` instead of `{status: 'success'}` - Setup returns `{status: 'ok'}` instead of `{status: 'success', redirect_url}` - Logout returns `{status: 'ok'}` instead of `{status: 'success'}` - Queue operations return structured responses with counts (e.g., `{cleared_count, retried_count}`) ### Frontend WebSocket Integration (October 2025) WebSocket integration previously completed and verified functional. #### Native WebSocket Implementation **Files:** - `src/server/web/static/js/websocket_client.js` - Native WebSocket wrapper - Templates already updated to use `websocket_client.js` instead of Socket.IO **Event Compatibility:** - Dual event handlers in place for backward compatibility - Old events: `scan_completed`, `scan_error`, `download_completed`, `download_error` - New events: `scan_complete`, `scan_failed`, `download_complete`, `download_failed` - Both event types supported simultaneously **Room Subscriptions:** - `downloads` - Download completion, failures, queue status - `download_progress` - Real-time download progress updates - `scan_progress` - Library scan progress updates ### Frontend Integration Testing (October 2025) Created smoke tests to verify frontend-backend integration. **Test File:** `tests/integration/test_frontend_integration_smoke.py` **Tests:** - JWT token format verification (access_token, token_type, expires_at) - Bearer token authentication on protected endpoints - 401 responses for requests without valid tokens **Test Results:** - Basic authentication flow: ✅ PASSING - Token validation: Functional with rate limiting considerations ### Frontend Integration (October 2025) Completed integration of existing frontend JavaScript with the new FastAPI backend and native WebSocket implementation. #### Native WebSocket Client **File**: `src/server/web/static/js/websocket_client.js` Created a Socket.IO-compatible wrapper using native WebSocket API: **Features**: - Socket.IO-style `.on()` and `.emit()` methods for compatibility - Automatic reconnection with exponential backoff (max 5 attempts) - Room-based subscriptions via `.join()` and `.leave()` methods - Message queueing during disconnection - Proper connection lifecycle management **Usage**: ```javascript const socket = io(); // Creates WebSocket to ws://host:port/ws/connect socket.join('download_progress'); // Subscribe to room socket.on('download_progress', (data) => { ... }); // Handle messages ``` #### WebSocket Message Format All WebSocket messages follow a structured format: ```json { "type": "message_type", "timestamp": "2025-10-17T12:34:56.789Z", "data": {} } ``` **Event Mapping** (Old Socket.IO → New WebSocket): - `scan_completed` / `scan_complete` → Scan finished - `scan_error` / `scan_failed` → Scan error - `download_completed` / `download_complete` → Download finished - `download_error` / `download_failed` → Download error - `queue_updated` / `queue_status` → Queue state changes - `queue_started`, `queue_stopped`, `queue_paused`, `queue_resumed` → Queue control events **Rooms**: - `scan_progress` - Library scan updates - `download_progress` - Real-time download progress - `downloads` - Download completion, failures, queue status #### JavaScript Updates **app.js**: - Added room subscriptions on WebSocket connect - Added dual event handlers for old and new message types - `connected` event handler for initial WebSocket confirmation - Handles both `scan_complete` and legacy `scan_completed` events - Handles both `scan_failed` and legacy `scan_error` events **queue.js**: - Added room subscriptions on WebSocket connect - Added dual event handlers for backward compatibility - Handles both `queue_status` and legacy `queue_updated` events - Handles both `download_complete` and legacy `download_completed` events - Handles both `download_failed` and legacy `download_error` events - Added handlers for `queue_started`, `queue_stopped`, `queue_paused`, `queue_resumed` #### Template Updates **Modified Templates**: - `src/server/web/templates/index.html` - Replaced Socket.IO CDN with websocket_client.js - `src/server/web/templates/queue.html` - Replaced Socket.IO CDN with websocket_client.js **Benefits**: - No external CDN dependency (Socket.IO) - Native browser WebSocket API (faster, smaller) - Full compatibility with existing JavaScript code - Proper integration with backend WebSocket service #### API Router Registration **fastapi_app.py**: - ✅ Added `anime_router` import and registration - All routers now properly included: - `health_router` - Health checks - `page_router` - HTML pages - `auth_router` - Authentication (JWT-based) - `anime_router` - Anime management (NEW) - `download_router` - Download queue - `websocket_router` - WebSocket connection **Anime Endpoints**: - `GET /api/v1/anime` - List anime with missing episodes - `POST /api/v1/anime/rescan` - Trigger library rescan - `POST /api/v1/anime/search` - Search for anime - `GET /api/v1/anime/{anime_id}` - Get anime details #### Authentication Integration JavaScript uses JWT tokens from localStorage for authenticated requests: - Token stored after successful login - Included in `Authorization: Bearer ` header - Automatic redirect to `/login` on 401 responses - Compatible with backend AuthMiddleware #### Testing **Verified Functionality**: - ✅ WebSocket client initialization and connection - ✅ Room subscriptions and message routing - ✅ Event handler compatibility (old and new message types) - ✅ Anime API endpoints (passed pytest tests) - ✅ Download queue API endpoints (existing tests) **Test Command**: ```bash conda run -n AniWorld python -m pytest tests/api/test_anime_endpoints.py -v ``` #### Known Limitations **Legacy Events**: Some Socket.IO events don't have backend implementations: - `scheduled_rescan_*` events - `auto_download_*` events - `download_episode_update` event - `download_series_completed` event **Solution**: These events are kept in JavaScript for future implementation or can be removed if not needed. **Configuration Endpoints**: Many config-related features in app.js don't have backend endpoints: - Scheduler configuration - Logging configuration - Advanced configuration - Config backups **Solution**: These can be implemented later or the UI features removed. #### Documentation **Detailed Documentation**: See `FRONTEND_INTEGRATION.md` for: - Complete API endpoint mapping - WebSocket message format details - Migration guide for developers - Testing strategies - Integration patterns