Aniworld/infrastructure.md

# Aniworld Web Application Infrastructure

conda activate AniWorld

## Project Structure

```
/home/lukas/Volume/repo/Aniworld/
├── src/
│   ├── server/                 # FastAPI web application
│   │   ├── fastapi_app.py     # Main FastAPI application (simplified)
│   │   ├── main.py            # FastAPI application entry point
│   │   ├── controllers/       # Route controllers
│   │   │   ├── __init__.py    # Controllers package
│   │   │   ├── health_controller.py  # Health check endpoints
│   │   │   ├── page_controller.py    # HTML page routes
│   │   │   └── error_controller.py   # Error handling controllers
│   │   ├── api/               # API route handlers
│   │   │   ├── __init__.py
│   │   │   ├── auth.py        # Authentication endpoints
│   │   │   ├── config.py      # Configuration endpoints
│   │   │   ├── anime.py       # Anime management endpoints
│   │   │   ├── download.py    # Download queue endpoints
│   │   │   ├── websocket.py   # WebSocket real-time endpoints
│   │   │   └── search.py      # Search endpoints
│   │   ├── models/            # Pydantic models
│   │   │   ├── __init__.py
│   │   │   ├── auth.py
│   │   │   ├── config.py
│   │   │   ├── anime.py
│   │   │   ├── download.py
│   │   │   └── websocket.py   # WebSocket message models
│   │   ├── services/          # Business logic services
│   │   │   ├── __init__.py
│   │   │   ├── auth_service.py
│   │   │   ├── config_service.py
│   │   │   ├── anime_service.py
│   │   │   ├── download_service.py
│   │   │   └── websocket_service.py  # WebSocket connection management
│   │   ├── utils/             # Utility functions
│   │   │   ├── __init__.py
│   │   │   ├── security.py
│   │   │   ├── dependencies.py  # Dependency injection
│   │   │   └── templates.py     # Shared Jinja2 template config
│   │   └── web/               # Frontend assets
│   │       ├── templates/     # Jinja2 HTML templates
│   │       │   ├── base.html
│   │       │   ├── login.html
│   │       │   ├── setup.html
│   │       │   ├── config.html
│   │       │   ├── anime.html
│   │       │   ├── download.html
│   │       │   └── search.html
│   │       └── static/        # Static web assets
│   │           ├── css/
│   │           ├── js/
│   │           └── images/
│   ├── core/                  # Existing core functionality
│   └── cli/                   # Existing CLI application
├── data/                      # Application data storage
│   ├── config.json           # Application configuration
│   ├── anime_library.db      # SQLite database for anime library
│   ├── download_queue.json   # Download queue state
│   └── cache/                # Temporary cache files
├── logs/                     # Application logs
│   ├── app.log              # Main application log
│   ├── download.log         # Download-specific logs
│   └── error.log            # Error logs
├── requirements.txt         # Python dependencies
├── docker-compose.yml       # Docker deployment configuration
└── README.md
```

## Technology Stack

### Backend

-   **FastAPI**: Modern Python web framework for building APIs
-   **Uvicorn**: ASGI server for running FastAPI applications
-   **SQLite**: Lightweight database for storing anime library and configuration
-   **Pydantic**: Data validation and serialization
-   **Jinja2**: Template engine for server-side rendering

### Frontend

-   **HTML5/CSS3**: Core web technologies
-   **JavaScript (Vanilla)**: Client-side interactivity
-   **Bootstrap 5**: CSS framework for responsive design
-   **HTMX**: Modern approach for dynamic web applications

### Security

-   **Passlib**: Password hashing and verification
-   **python-jose**: JWT token handling
-   **bcrypt**: Secure password hashing

### Authentication Models & Sessions

-   Authentication request/response Pydantic models live in `src/server/models/auth.py`.
-   Sessions are represented by `SessionModel` and can be backed by an in-memory
    store or a persistent table depending on deployment needs. JWTs are used for
    stateless authentication by default; a persistent session store may be
    configured in production to enable revocation and long-lived sessions.

## Configuration

### Data Storage

-   **Configuration**: JSON files in `data/` directory
-   **Anime Library**: SQLite database with series information
-   **Download Queue**: JSON file with current download status
-   **Logs**: Structured logging to files in `logs/` directory

## API Endpoints

### Authentication

-   `POST /api/auth/login` - Master password authentication
-   `POST /api/auth/logout` - Logout and invalidate session
-   `GET /api/auth/status` - Check authentication status

### Configuration

-   `GET /api/config` - Get current configuration
-   `PUT /api/config` - Update configuration
-   `POST /api/setup` - Initial setup

### Configuration API Notes

-   The configuration endpoints are exposed under `/api/config` and
    operate primarily on a JSON-serializable `AppConfig` model. They are
    designed to be lightweight and avoid performing IO during validation
    (the `/api/config/validate` endpoint runs in-memory checks only).
-   Persistence of configuration changes is intentionally "best-effort"
    for now and mirrors fields into the runtime settings object. A
    follow-up task should add durable storage (file or DB) for configs.

### Anime Management

-   `GET /api/anime` - List anime with missing episodes
-   `POST /api/anime/{id}/download` - Add episodes to download queue
-   `GET /api/anime/{id}` - Get anime details

Note: The anime management API has been implemented under `/api/v1/anime` with
endpoints for listing series with missing episodes, searching providers,
triggering a local rescan, and fetching series details. The implementation
delegates to the existing core `SeriesApp` and uses dependency injection for
initialization.

### Download Management

-   `GET /api/queue/status` - Get download queue status and statistics
-   `POST /api/queue/add` - Add episodes to download queue
-   `DELETE /api/queue/{id}` - Remove item from queue
-   `DELETE /api/queue/` - Remove multiple items from queue
-   `POST /api/queue/start` - Start download queue processing
-   `POST /api/queue/stop` - Stop download queue processing
-   `POST /api/queue/pause` - Pause queue processing
-   `POST /api/queue/resume` - Resume queue processing
-   `POST /api/queue/reorder` - Reorder pending queue items
-   `DELETE /api/queue/completed` - Clear completed downloads
-   `POST /api/queue/retry` - Retry failed downloads

### Search

-   `GET /api/search?q={query}` - Search for anime
-   `POST /api/search/add` - Add anime to library

## Logging

### Log Levels

-   **INFO**: General application information
-   **WARNING**: Potential issues that don't stop execution
-   **ERROR**: Errors that affect functionality
-   **DEBUG**: Detailed debugging information (development only)

### Log Files

-   `app.log`: General application logs
-   `download.log`: Download-specific operations
-   `error.log`: Error and exception logs

## Security Considerations

-   Master password protection for application access
-   Secure session management with JWT tokens
-   Input validation and sanitization
-   Rate limiting on API endpoints
-   HTTPS enforcement in production
-   Secure file path handling to prevent directory traversal

### Authentication Service

-   A lightweight authentication service is provided by
    `src/server/services/auth_service.py`.
-   Uses bcrypt (passlib) to hash the master password and issues JWTs for
    stateless sessions. Tokens are signed with the `JWT_SECRET_KEY` from
    configuration and expire based on `SESSION_TIMEOUT_HOURS`.
-   Failed login attempts are tracked in-memory and a temporary lockout is
    applied after multiple failures. For multi-process deployments, move
    this state to a shared store (Redis) and persist the master password
    hash in a secure config store.

## Recent Infrastructure Changes

### Route Controller Refactoring (October 2025)

Restructured the FastAPI application to use a controller-based architecture for better code organization and maintainability.

#### Changes Made

1. **Created Controller Structure**:

    - `src/server/controllers/` - New directory for route controllers
    - `src/server/controllers/__init__.py` - Controllers package initialization
    - `src/server/controllers/health_controller.py` - Health check endpoints
    - `src/server/controllers/page_controller.py` - HTML page routes
    - `src/server/controllers/error_controller.py` - Error handling controllers

2. **Shared Template Configuration**:

    - `src/server/utils/templates.py` - Centralized Jinja2 template configuration
    - Fixed template path resolution for proper template loading

3. **Main Application Updates**:

    - `src/server/fastapi_app.py` - Refactored to use controller routers
    - Removed direct route definitions from main file
    - Added router inclusion using `app.include_router()`
    - Simplified error handlers to delegate to controller functions

4. **Fixed Import Issues**:
    - Resolved circular import in `src/core/__init__.py`
    - Removed non-existent `application` module import

#### Controller Architecture

### Anime Service Notes

-   The new `anime_service` runs the existing blocking `SeriesApp` inside a
    threadpool (via ThreadPoolExecutor). This keeps the FastAPI event loop
    responsive while leveraging the existing core logic.
-   A small in-process LRU cache is used for the frequently-read "missing
    episodes" list to reduce IO; cache invalidation happens after a rescan.
-   For multi-worker or multi-host deployments, move cache/state to a shared
    store (Redis) and ensure the threadpool sizing matches the worker's CPU
    and IO profile.

**Health Controller** (`health_controller.py`):

```python
router = APIRouter(prefix="/health", tags=["health"])
@router.get("") - Health check endpoint
```

**Page Controller** (`page_controller.py`):

```python
router = APIRouter(tags=["pages"])
@router.get("/") - Main application page
@router.get("/setup") - Setup page
@router.get("/login") - Login page
@router.get("/queue") - Download queue page
```

**Error Controller** (`error_controller.py`):

```python
async def not_found_handler() - Custom 404 error handling
async def server_error_handler() - Custom 500 error handling
```

#### Benefits of the New Structure

-   **Separation of Concerns**: Each controller handles specific functionality
-   **Modularity**: Easy to add new controllers and routes
-   **Testability**: Controllers can be tested independently
-   **Maintainability**: Cleaner code organization and easier debugging
-   **Scalability**: Simple to extend with new features

#### Verified Working Endpoints

All endpoints tested and confirmed working:

-   Health: `/health` → Returns `{"status": "healthy", ...}`
-   Root: `/` → Serves main application page
-   Setup: `/setup` → Serves setup page
-   Auth API: `/api/auth/*` → Endpoints for setup, login, logout and status (JWT-based)
-   Login: `/login` → Serves login page
-   Queue: `/queue` → Serves download queue page

#### File Structure After Refactoring

```
src/server/
├── fastapi_app.py                 # Main FastAPI application (simplified)
├── controllers/                   # NEW: Route controllers
│   ├── __init__.py               # Controllers package
├── utils/
│   ├── dependencies.py          # Existing dependency injection
│   └── templates.py              # NEW: Shared Jinja2 template config
└── web/                          # Existing frontend assets
    ├── templates/                # HTML templates
    └── static/                   # CSS, JS, images
```

### Authentication Middleware (October 2025)

An authentication middleware component was added to the FastAPI
application to centralize token parsing and provide lightweight
protection of authentication endpoints:

-   `src/server/middleware/auth.py` implements:
    -   Bearer JWT parsing and session attachment to `request.state.session`
    -   A simple per-IP in-memory rate limiter applied to
        `/api/auth/login` and `/api/auth/setup` (default 5 requests/minute)

Notes:

-   This is intentionally simple and designed for single-process
    deployments. For production use across multiple workers or hosts,
    replace the in-memory limiter with a distributed store (e.g. Redis)
    and add a persistent token revocation list if needed.

### API Models and Contracts

-   Pydantic models living in `src/server/models/` define the canonical
    API contracts used by FastAPI endpoints. These models are intentionally
    lightweight and focused on serialization, validation, and OpenAPI
    documentation generation.
-   Keep models stable: changes to model shapes are breaking changes for
    clients. Bump API versioning or provide migration layers when altering
    public response fields.
-   Infrastructure considerations: ensure the deployment environment has
    required libraries (e.g., `pydantic`) installed and that schema
    validation errors are logged to the centralized logging system. For
    high-throughput routes, consider response model caching at the
    application or reverse-proxy layer.

### WebSocket Real-time Communication (October 2025)

A comprehensive WebSocket infrastructure was implemented to provide real-time
updates for downloads, queue status, and system events:

-   **File**: `src/server/services/websocket_service.py`
-   **Models**: `src/server/models/websocket.py`
-   **Endpoint**: `ws://host:port/ws/connect`

#### WebSocket Service Architecture

-   **ConnectionManager**: Low-level connection lifecycle management

    -   Connection registry with unique connection IDs
    -   Room-based messaging for topic subscriptions
    -   Automatic connection cleanup and health monitoring
    -   Thread-safe operations with asyncio locks

-   **WebSocketService**: High-level application messaging
    -   Convenient interface for broadcasting application events
    -   Pre-defined message types for downloads, queue, and system events
    -   Singleton pattern via `get_websocket_service()` factory

#### Supported Message Types

-   **Download Events**: `download_progress`, `download_complete`, `download_failed`
-   **Queue Events**: `queue_status`, `queue_started`, `queue_stopped`, `queue_paused`, `queue_resumed`
-   **System Events**: `system_info`, `system_warning`, `system_error`
-   **Connection**: `connected`, `ping`, `pong`, `error`

#### Room-Based Messaging

Clients can subscribe to specific topics (rooms) to receive targeted updates:

-   `downloads` room: All download-related events
-   Custom rooms: Can be added for specific features

#### Integration with Download Service

-   Download service automatically broadcasts progress updates via WebSocket
-   Broadcast callback registered during service initialization
-   Updates sent to all clients subscribed to the `downloads` room
-   No blocking of download operations (async broadcast)

#### Client Connection Flow

1. Client connects to `/ws/connect` endpoint
2. Server assigns unique connection ID and sends confirmation
3. Client joins rooms (e.g., `{"action": "join", "room": "downloads"}`)
4. Server broadcasts updates to subscribed rooms
5. Client disconnects (automatic cleanup)

#### Infrastructure Notes

-   **Single-process**: Current implementation uses in-memory connection storage
-   **Production**: For multi-worker/multi-host deployments:
    -   Move connection registry to Redis or similar shared store
    -   Implement pub/sub for cross-process message broadcasting
    -   Add connection persistence for recovery after restarts
-   **Monitoring**: WebSocket status available at `/ws/status` endpoint
-   **Security**: Optional authentication via JWT (user_id tracking)
-   **Testing**: Comprehensive unit tests in `tests/unit/test_websocket_service.py`

### Download Queue Models

-   Download queue models in `src/server/models/download.py` define the data
    structures for the download queue system.
-   Key models include:
    -   `DownloadItem`: Represents a single queued download with metadata,
        progress tracking, and error information
    -   `QueueStatus`: Overall queue state with active, pending, completed,
        and failed downloads
    -   `QueueStats`: Aggregated statistics for monitoring queue performance
    -   `DownloadProgress`: Real-time progress information (percent, speed,
        ETA)
    -   `DownloadRequest`/`DownloadResponse`: API request/response contracts
-   Models enforce validation constraints (e.g., positive episode numbers,
    progress percentage 0-100, non-negative retry counts) and provide
    clean JSON serialization for API endpoints and WebSocket updates.

### Download Queue Service

-   The download service (`src/server/services/download_service.py`) manages
    the complete lifecycle of anime episode downloads.
-   Core features:
    -   **Priority-based Queue**: Items added with HIGH priority are processed
        first, NORMAL and LOW follow in FIFO order
    -   **Concurrent Processing**: Configurable max concurrent downloads (default 2)
        to optimize bandwidth usage
    -   **Persistence**: Queue state is automatically saved to
        `data/download_queue.json` and recovered on service restart
    -   **Retry Logic**: Failed downloads are automatically retried up to a
        configurable limit (default 3 attempts) with exponential backoff
    -   **Progress Tracking**: Real-time download progress with speed,
        percentage, and ETA calculations
    -   **WebSocket Integration**: Broadcasts queue updates, progress, and
        completion/failure events to connected clients
-   Operations:
    -   `add_to_queue()`: Add episodes to download queue with priority
    -   `remove_from_queue()`: Cancel pending or active downloads
    -   `reorder_queue()`: Manually adjust queue order for pending items
    -   `pause_queue()`/`resume_queue()`: Control download processing
    -   `retry_failed()`: Retry failed downloads with retry count checks
    -   `get_queue_status()`: Get complete queue state (active, pending, completed, failed)
    -   `get_queue_stats()`: Get aggregated statistics (counts, download size, speed)
-   Infrastructure notes:
    -   Service uses ThreadPoolExecutor for concurrent download processing
    -   Queue processor runs as async background task with configurable sleep intervals
    -   Progress callbacks are executed in threadpool and broadcast via async WebSocket
    -   For multi-process deployments, move queue state to shared store (Redis/DB)
        and implement distributed locking for concurrent access control
    -   Singleton instance pattern used via `get_download_service()` factory
-   Testing: Comprehensive unit tests in `tests/unit/test_download_service.py`
    cover queue operations, persistence, retry logic, and error handling

### Download Queue API Endpoints (October 2025)

Implemented comprehensive REST API endpoints for download queue management:

-   **File**: `src/server/api/download.py`
-   **Router Prefix**: `/api/queue`
-   **Authentication**: All endpoints require JWT authentication via `require_auth` dependency

#### Implemented Endpoints

1. **GET /api/queue/status** - Retrieve complete queue status

    - Returns: `QueueStatusResponse` with status and statistics
    - Includes: active downloads, pending items, completed/failed items, queue stats

2. **POST /api/queue/add** - Add episodes to download queue

    - Request: `DownloadRequest` with serie info, episodes, and priority
    - Returns: `DownloadResponse` with added item IDs
    - Validates episode list is non-empty
    - Supports HIGH, NORMAL, and LOW priority levels

3. **DELETE /api/queue/{item_id}** - Remove single item from queue

    - Returns: 204 No Content on success, 404 if item not found
    - Cancels active downloads if necessary

4. **DELETE /api/queue/** - Remove multiple items (batch operation)

    - Request: `QueueOperationRequest` with list of item IDs
    - Returns: 204 No Content (partial success acceptable)

5. **POST /api/queue/start** - Start queue processor

    - Idempotent operation (safe to call multiple times)

6. **POST /api/queue/stop** - Stop queue processor

    - Waits for active downloads to complete (with timeout)

7. **POST /api/queue/pause** - Pause queue processing

    - Active downloads continue, no new downloads start

8. **POST /api/queue/resume** - Resume queue processing

9. **POST /api/queue/reorder** - Reorder pending queue item

    - Request: `QueueReorderRequest` with item_id and new_position
    - Returns: 404 if item not in pending queue

10. **DELETE /api/queue/completed** - Clear completed items from history

    - Returns count of cleared items

11. **POST /api/queue/retry** - Retry failed downloads
    - Request: `QueueOperationRequest` with item IDs (empty for all)
    - Only retries items under max retry limit

#### Dependencies

-   **get_download_service**: Factory function providing singleton DownloadService instance
    -   Automatically initializes AnimeService as dependency
    -   Raises 503 if anime directory not configured
-   **get_anime_service**: Factory function providing singleton AnimeService instance
    -   Required by DownloadService for anime operations
-   Both dependencies added to `src/server/utils/dependencies.py`

#### Error Handling

-   All endpoints return structured JSON error responses
-   HTTP status codes follow REST conventions (200, 201, 204, 400, 401, 404, 500, 503)
-   Service-level exceptions (DownloadServiceError) mapped to 400 Bad Request
-   Generic exceptions mapped to 500 Internal Server Error
-   Authentication errors return 401 Unauthorized

#### Testing

-   Comprehensive test suite in `tests/api/test_download_endpoints.py`
-   Tests cover:
    -   Successful operations for all endpoints
    -   Authentication requirements
    -   Error conditions (empty lists, not found, service errors)
    -   Priority handling
    -   Batch operations
-   Uses pytest fixtures for authenticated client and mocked download service

#### Integration

-   Router registered in `src/server/fastapi_app.py` via `app.include_router(download_router)`
-   Follows same patterns as other API routers (auth, anime, config)
-   Full OpenAPI documentation available at `/api/docs`

### WebSocket Real-time Updates (October 2025)

Implemented real-time progress tracking and WebSocket broadcasting for downloads, scans, and system events.

#### ProgressService

**File**: `src/server/services/progress_service.py`

A centralized service for tracking and broadcasting real-time progress updates across the application.

**Key Features**:

-   Track multiple concurrent progress operations (downloads, scans, queue changes)
-   Automatic progress percentage calculation
-   Progress lifecycle management (start, update, complete, fail, cancel)
-   WebSocket integration for real-time client updates
-   Progress history with configurable size limit (default: 50 items)
-   Thread-safe operations using asyncio locks
-   Support for progress metadata and custom messages

**Progress Types**:

-   `DOWNLOAD` - File download progress
-   `SCAN` - Library scan progress
-   `QUEUE` - Queue operation progress
-   `SYSTEM` - System-level operations
-   `ERROR` - Error notifications

**Progress Statuses**:

-   `STARTED` - Operation initiated
-   `IN_PROGRESS` - Operation in progress
-   `COMPLETED` - Successfully completed
-   `FAILED` - Operation failed
-   `CANCELLED` - Cancelled by user

**Core Methods**:

-   `start_progress()` - Initialize new progress operation
-   `update_progress()` - Update progress with current/total values
-   `complete_progress()` - Mark operation as completed
-   `fail_progress()` - Mark operation as failed
-   `cancel_progress()` - Cancel ongoing operation
-   `get_progress()` - Retrieve progress by ID
-   `get_all_active_progress()` - Get all active operations (optionally filtered by type)

**Broadcasting**:

-   Integrates with WebSocketService via callback
-   Broadcasts to room-specific channels (e.g., `download_progress`, `scan_progress`)
-   Configurable broadcast throttling (only on significant changes >1% or forced)
-   Automatic progress state serialization to JSON

**Singleton Pattern**:

-   Global instance via `get_progress_service()` factory
-   Initialized during application startup with WebSocket callback

#### Integration with Services

**DownloadService Integration**:

-   Progress tracking for each download item
-   Real-time progress updates during file download
-   Automatic completion/failure notifications
-   Progress metadata includes speed, ETA, downloaded bytes

**AnimeService Integration**:

-   Progress tracking for library scans
-   Scan progress with current/total file counts
-   Scan completion with statistics
-   Error notifications on scan failures

#### WebSocket Message Models

**File**: `src/server/models/websocket.py`

Added progress-specific message models:

-   `ScanProgressMessage` - Scan progress updates
-   `ScanCompleteMessage` - Scan completion notification
-   `ScanFailedMessage` - Scan failure notification
-   `ErrorNotificationMessage` - Critical error notifications
-   `ProgressUpdateMessage` - Generic progress updates

**WebSocket Message Types**:

-   `SCAN_PROGRESS` - Scan progress updates
-   `SCAN_COMPLETE` - Scan completion
-   `SCAN_FAILED` - Scan failure
-   Extended existing types for downloads and queue updates

#### WebSocket Rooms

Clients can subscribe to specific progress channels:

-   `download_progress` - Download progress updates
-   `scan_progress` - Library scan updates
-   `queue_progress` - Queue operation updates
-   `system_progress` - System-level updates

Room subscription via client messages:

```json
{
    "action": "join",
    "room": "download_progress"
}
```

#### Application Startup

**File**: `src/server/fastapi_app.py`

Progress service initialized on application startup:

1. Get ProgressService singleton instance
2. Get WebSocketService singleton instance
3. Register broadcast callback to link progress updates with WebSocket
4. Callback broadcasts progress messages to appropriate rooms

#### Testing

**File**: `tests/unit/test_progress_service.py`

Comprehensive test coverage including:

-   Progress lifecycle operations (start, update, complete, fail, cancel)
-   Percentage calculation accuracy
-   History management and size limits
-   Broadcast callback invocation
-   Concurrent progress operations
-   Metadata handling
-   Error conditions and edge cases

#### Architecture Benefits

-   **Decoupling**: ProgressService decouples progress tracking from WebSocket broadcasting
-   **Reusability**: Single service used across all application components
-   **Scalability**: Supports multiple concurrent operations efficiently
-   **Observability**: Centralized progress tracking simplifies monitoring
-   **Real-time UX**: Instant feedback to users on all long-running operations

#### Future Enhancements

-   Persistent progress history (database storage)
-   Progress rate calculation and trend analysis
-   Multi-process progress synchronization (Redis/shared store)
-   Progress event hooks for custom actions
-   Client-side progress resumption after reconnection