Lukas fb2cdd4bb6 Task 3.3: Update ProgressService to use key as identifier

- Added optional 'key' and 'folder' fields to ProgressUpdate dataclass
- key: Primary series identifier (provider key, e.g., 'attack-on-titan')
- folder: Optional series folder name for display (e.g., 'Attack on Titan (2013)')
- Updated start_progress() and update_progress() methods to accept key/folder parameters
- Enhanced to_dict() serialization to include key/folder when present
- Updated all docstrings to clarify identifier usage
- Added 5 new comprehensive unit tests for key/folder functionality
- All 25 ProgressService tests passing
- Updated infrastructure.md with series identifier documentation
- Maintains backward compatibility - fields are optional
- Completed Phase 3, Task 3.3 of identifier standardization initiative

2025-11-27 18:36:35 +01:00

81 KiB

Raw Blame History

Aniworld Web Application Infrastructure

conda activate AniWorld

Project Structure

/home/lukas/Volume/repo/Aniworld/
├── src/
│   ├── core/                  # Core application logic
│   │   ├── SeriesApp.py       # Main application class with async support
│   │   ├── SerieScanner.py    # Directory scanner for anime series
│   │   ├── entities/          # Domain entities
│   │   │   ├── series.py      # Serie data model
│   │   │   └── SerieList.py   # Series list management
│   │   ├── interfaces/        # Abstract interfaces
│   │   │   └── providers.py   # Provider interface definitions
│   │   ├── providers/         # Content providers
│   │   │   ├── base_provider.py           # Base loader interface
│   │   │   ├── aniworld_provider.py       # Aniworld.to implementation
│   │   │   ├── provider_factory.py        # Provider factory
│   │   │   ├── provider_config.py         # Provider configuration
│   │   │   ├── health_monitor.py          # Provider health monitoring
│   │   │   ├── failover.py                # Provider failover system
│   │   │   ├── monitored_provider.py      # Performance tracking wrapper
│   │   │   ├── config_manager.py          # Dynamic configuration mgmt
│   │   │   └── streaming/     # Streaming providers (VOE, etc.)
│   │   └── exceptions/        # Custom exceptions
│   │       └── Exceptions.py  # Exception definitions
│   ├── server/                # FastAPI web application
│   │   ├── fastapi_app.py     # Main FastAPI application (simplified)
│   │   ├── main.py            # FastAPI application entry point
│   │   ├── controllers/       # Route controllers
│   │   │   ├── __init__.py    # Controllers package
│   │   │   ├── health_controller.py  # Health check endpoints
│   │   │   ├── page_controller.py    # HTML page routes
│   │   │   └── error_controller.py   # Error handling controllers
│   │   ├── api/               # API route handlers
│   │   │   ├── __init__.py
│   │   │   ├── auth.py        # Authentication endpoints
│   │   │   ├── config.py      # Configuration endpoints
│   │   │   ├── anime.py       # Anime management endpoints
│   │   │   ├── download.py    # Download queue endpoints
│   │   │   ├── scheduler.py   # Scheduler configuration endpoints
│   │   │   └── websocket.py   # WebSocket real-time endpoints
│   │   ├── models/            # Pydantic models
│   │   │   ├── __init__.py
│   │   │   ├── auth.py
│   │   │   ├── config.py
│   │   │   ├── anime.py
│   │   │   ├── download.py
│   │   │   └── websocket.py   # WebSocket message models
│   │   ├── services/          # Business logic services
│   │   │   ├── __init__.py
│   │   │   ├── auth_service.py
│   │   │   ├── config_service.py
│   │   │   ├── anime_service.py
│   │   │   ├── download_service.py
│   │   │   ├── websocket_service.py   # WebSocket connection management
│   │   │   ├── progress_service.py    # Progress tracking
│   │   │   ├── notification_service.py # Notification system
│   │   │   └── cache_service.py       # Caching layer
│   │   ├── database/          # Database layer
│   │   │   ├── __init__.py    # Database package
│   │   │   ├── base.py        # Base models and mixins
│   │   │   ├── models.py      # SQLAlchemy ORM models
│   │   │   └── connection.py  # Database connection management
│   │   ├── utils/             # Utility functions
│   │   │   ├── __init__.py
│   │   │   ├── security.py
│   │   │   ├── dependencies.py    # Dependency injection
│   │   │   ├── templates.py       # Shared Jinja2 template config
│   │   │   ├── template_helpers.py # Template rendering utilities
│   │   │   └── logging.py         # Logging utilities
│   │   └── web/               # Frontend assets
│   │       ├── templates/     # Jinja2 HTML templates
│   │       │   ├── index.html     # Main application page
│   │       │   ├── login.html     # Login page
│   │       │   ├── setup.html     # Initial setup page
│   │       │   ├── queue.html     # Download queue page
│   │       │   └── error.html     # Error page
│   │       └── static/        # Static web assets
│   │           ├── css/
│   │           │   ├── styles.css        # Main styles
│   │           │   └── ux_features.css   # UX enhancements
│   │           └── js/
│   │               ├── app.js                      # Main application logic
│   │               ├── queue.js                    # Queue management
│   │               ├── localization.js             # i18n support
│   │               ├── keyboard_shortcuts.js       # Keyboard navigation
│   │               ├── user_preferences.js         # User settings
│   │               ├── undo_redo.js                # Undo/redo system
│   │               ├── mobile_responsive.js        # Mobile support
│   │               ├── touch_gestures.js           # Touch interactions
│   │               ├── accessibility_features.js   # A11y features
│   │               ├── screen_reader_support.js    # Screen reader
│   │               ├── color_contrast_compliance.js # WCAG compliance
│   │               ├── multi_screen_support.js     # Multi-monitor
│   │               ├── drag_drop.js                # Drag and drop
│   │               ├── bulk_operations.js          # Bulk actions
│   │               └── advanced_search.js          # Search filters
│   ├── core/                  # Existing core functionality
│   └── cli/                   # Existing CLI application
├── data/                      # Application data storage
│   ├── config.json           # Application configuration
│   ├── anime_library.db      # SQLite database for anime library
│   ├── download_queue.json   # Download queue state
│   └── cache/                # Temporary cache files
├── logs/                     # Application logs
│   ├── app.log              # Main application log
│   ├── download.log         # Download-specific logs
│   └── error.log            # Error logs
├── requirements.txt         # Python dependencies
├── docker-compose.yml       # Docker deployment configuration
└── README.md

Technology Stack

Backend

FastAPI: Modern Python web framework for building APIs
Uvicorn: ASGI server for running FastAPI applications
SQLAlchemy: SQL toolkit and ORM for database operations
SQLite: Lightweight database for storing anime library and configuration
Alembic: Database migration tool for schema management
Pydantic: Data validation and serialization
Jinja2: Template engine for server-side rendering

Frontend

HTML5/CSS3: Core web technologies
JavaScript (Vanilla): Client-side interactivity
Bootstrap 5: CSS framework for responsive design
HTMX: Modern approach for dynamic web applications

Security

Passlib: Password hashing and verification
python-jose: JWT token handling
bcrypt: Secure password hashing

Authentication Models & Sessions

Authentication request/response Pydantic models live in src/server/models/auth.py.
Sessions are represented by SessionModel and can be backed by an in-memory store or a persistent table depending on deployment needs. JWTs are used for stateless authentication by default; a persistent session store may be configured in production to enable revocation and long-lived sessions.

Configuration

Data Storage

Configuration: JSON files in data/ directory
Anime Library: SQLite database with series information
Download Queue: JSON file with current download status
Logs: Structured logging to files in logs/ directory

API Endpoints

Authentication

POST /api/auth/login - Master password authentication
POST /api/auth/logout - Logout and invalidate session
GET /api/auth/status - Check authentication status

Configuration

GET /api/config - Get current configuration
PUT /api/config - Update configuration
POST /api/setup - Initial setup

Configuration API Notes

Configuration endpoints are exposed under /api/config
Uses file-based persistence with JSON format for human-readable storage
Automatic backup creation before configuration updates
Configuration validation with detailed error reporting
Backup management with create, restore, list, and delete operations
Configuration schema versioning with migration support
Singleton ConfigService manages all persistence operations
Default configuration location: data/config.json
Backup directory: data/config_backups/
Maximum backups retained: 10 (configurable)
Automatic cleanup of old backups exceeding limit

Key Endpoints:

GET /api/config - Retrieve current configuration
PUT /api/config - Update configuration (creates backup)
POST /api/config/validate - Validate without applying
GET /api/config/backups - List all backups
POST /api/config/backups - Create manual backup
POST /api/config/backups/{name}/restore - Restore from backup
DELETE /api/config/backups/{name} - Delete backup
GET /api/config/section/advanced - Get advanced configuration section
POST /api/config/section/advanced - Update advanced configuration
POST /api/config/directory - Update anime directory
POST /api/config/export - Export configuration to JSON file
POST /api/config/reset - Reset configuration to defaults

Configuration Service Features:

Atomic file writes using temporary files
JSON format with version metadata
Validation before saving
Automatic backup on updates
Migration support for schema changes
Thread-safe singleton pattern
Comprehensive error handling with custom exceptions

Scheduler

GET /api/scheduler/config - Get scheduler configuration
POST /api/scheduler/config - Update scheduler configuration
POST /api/scheduler/trigger-rescan - Manually trigger rescan

Anime Management

GET /api/anime - List anime with missing episodes
POST /api/anime/{id}/download - Add episodes to download queue
GET /api/anime/{id} - Get anime details

Note: The anime management API has been implemented under /api/v1/anime with endpoints for listing series with missing episodes, searching providers, triggering a local rescan, and fetching series details. The implementation delegates to the existing core SeriesApp and uses dependency injection for initialization.

Download Management

GET /api/queue/status - Get download queue status and statistics
POST /api/queue/add - Add episodes to download queue
DELETE /api/queue/{id} - Remove single item from pending queue
POST /api/queue/start - Manually start next download from queue (one at a time)
POST /api/queue/stop - Stop processing new downloads
DELETE /api/queue/completed - Clear completed downloads
DELETE /api/queue/failed - Clear failed downloads
POST /api/queue/retry/{id} - Retry a specific failed download
POST /api/queue/retry - Retry all failed downloads

Manual Download Control:

Queue processing is fully manual - no auto-start
User must click "Start" to begin downloading next item from queue
Only one download active at a time
"Stop" prevents new downloads but allows current to complete
FIFO queue order (first-in, first-out)

Queue Organization:

Pending Queue: Items waiting to be downloaded, displayed in FIFO order
Active Download: Currently downloading item with progress bar (max 1)
Completed Downloads: Successfully downloaded items with completion timestamps
Failed Downloads: Failed items with error messages and retry options

Queue Display Features:

Real-time statistics counters (pending, active, completed, failed)
Empty state messages with helpful hints
Per-section action buttons (clear, retry all)
Start/Stop buttons for manual queue control

WebSocket

WS /api/ws - WebSocket connection for real-time updates
Real-time download progress notifications
Queue status updates
System notifications

Logging

Log Levels

INFO: General application information
WARNING: Potential issues that don't stop execution
ERROR: Errors that affect functionality
DEBUG: Detailed debugging information (development only)

Log Files

app.log: General application logs
download.log: Download-specific operations
error.log: Error and exception logs

Security Considerations

Master password protection for application access
Secure session management with JWT tokens
Input validation and sanitization
Built-in rate limiting in authentication middleware
HTTPS enforcement in production
Secure file path handling to prevent directory traversal

Authentication Service

A lightweight authentication service is provided by src/server/services/auth_service.py.
Uses bcrypt (passlib) to hash the master password and issues JWTs for stateless sessions. Tokens are signed with the JWT_SECRET_KEY from configuration and expire based on SESSION_TIMEOUT_HOURS.
Failed login attempts are tracked in-memory and a temporary lockout is applied after multiple failures. For multi-process deployments, move this state to a shared store (Redis) and persist the master password hash in a secure config store.

Database Layer (October 2025)

A comprehensive SQLAlchemy-based database layer was implemented to provide persistent storage for anime series, episodes, download queue, and user sessions.

Architecture

Location: src/server/database/

Components:

base.py: Base declarative class and mixins (TimestampMixin, SoftDeleteMixin)
models.py: SQLAlchemy ORM models with relationships
connection.py: Database engine, session factory, and dependency injection
__init__.py: Package exports and public API

Database Models

AnimeSeries

Represents anime series with metadata and provider information.

Fields:

id (PK): Auto-incrementing primary key
key: Unique provider identifier (indexed)
name: Series name (indexed)
site: Provider site URL
folder: Local filesystem path
description: Optional series description
status: Series status (ongoing, completed)
total_episodes: Total episode count
cover_url: Cover image URL
episode_dict: JSON field storing episode structure {season: [episodes]}
created_at, updated_at: Audit timestamps (from TimestampMixin)

Relationships:

episodes: One-to-many with Episode (cascade delete)
download_items: One-to-many with DownloadQueueItem (cascade delete)

Episode

Individual episodes linked to anime series.

Fields:

id (PK): Auto-incrementing primary key
series_id (FK): Foreign key to AnimeSeries (indexed)
season: Season number
episode_number: Episode number within season
title: Optional episode title
file_path: Local file path if downloaded
file_size: File size in bytes
is_downloaded: Boolean download status
download_date: Timestamp when downloaded
created_at, updated_at: Audit timestamps

Relationships:

series: Many-to-one with AnimeSeries

DownloadQueueItem

Download queue with status and progress tracking.

Fields:

id (PK): Auto-incrementing primary key
series_id (FK): Foreign key to AnimeSeries (indexed)
season: Season number
episode_number: Episode number
status: Download status enum (indexed)
- Values: PENDING, DOWNLOADING, PAUSED, COMPLETED, FAILED, CANCELLED
priority: Priority enum
- Values: LOW, NORMAL, HIGH
progress_percent: Download progress (0-100)
downloaded_bytes: Bytes downloaded
total_bytes: Total file size
download_speed: Current speed (bytes/sec)
error_message: Error description if failed
retry_count: Number of retry attempts
download_url: Provider download URL
file_destination: Target file path
started_at: Download start timestamp
completed_at: Download completion timestamp
created_at, updated_at: Audit timestamps

Relationships:

series: Many-to-one with AnimeSeries

UserSession

User authentication sessions with JWT tokens.

Fields:

id (PK): Auto-incrementing primary key
session_id: Unique session identifier (indexed)
token_hash: Hashed JWT token
user_id: User identifier (indexed, for multi-user support)
ip_address: Client IP address
user_agent: Client user agent string
expires_at: Session expiration timestamp
is_active: Boolean active status (indexed)
last_activity: Last activity timestamp
created_at, updated_at: Audit timestamps

Methods:

is_expired: Property to check if session has expired
revoke(): Revoke session by setting is_active=False

Mixins

TimestampMixin

Adds automatic timestamp tracking to models.

Fields:

created_at: Automatically set on record creation
updated_at: Automatically updated on record modification

Usage: Inherit in models requiring audit timestamps.

SoftDeleteMixin

Provides soft delete functionality (logical deletion).

Fields:

deleted_at: Timestamp when soft deleted (NULL if active)

Properties:

is_deleted: Check if record is soft deleted

Methods:

soft_delete(): Mark record as deleted
restore(): Restore soft deleted record

Note: Currently not used by models but available for future implementation.

Database Connection Management

Initialization

from src.server.database import init_db, close_db

# Application startup
await init_db()  # Creates engine, session factory, and tables

# Application shutdown
await close_db()  # Closes connections and cleanup

Session Management

Async Sessions (preferred for FastAPI endpoints):

from fastapi import Depends
from sqlalchemy.ext.asyncio import AsyncSession
from src.server.database import get_db_session

@app.get("/anime")
async def get_anime(db: AsyncSession = Depends(get_db_session)):
    result = await db.execute(select(AnimeSeries))
    return result.scalars().all()

Sync Sessions (for non-async operations):

from src.server.database.connection import get_sync_session

session = get_sync_session()
try:
    result = session.execute(select(AnimeSeries))
    return result.scalars().all()
finally:
    session.close()

Database Configuration

Settings (from src/config/settings.py):

DATABASE_URL: Database connection string
- Default: sqlite:///./data/aniworld.db
- Automatically converted to sqlite+aiosqlite:/// for async support
LOG_LEVEL: When set to "DEBUG", enables SQL query logging

Engine Configuration:

SQLite: Uses StaticPool, enables foreign keys and WAL mode
PostgreSQL/MySQL: Uses QueuePool with pre-ping health checks
Connection Pooling: Configured based on database type
Echo: SQL query logging in DEBUG mode

SQLite Optimizations

Foreign Keys: Automatically enabled via PRAGMA
WAL Mode: Write-Ahead Logging for better concurrency
Static Pool: Single connection pool for SQLite
Async Support: aiosqlite driver for async operations

FastAPI Integration

Dependency Injection (in src/server/utils/dependencies.py):

async def get_database_session() -> AsyncGenerator:
    """Dependency to get database session."""
    try:
        from src.server.database import get_db_session

        async with get_db_session() as session:
            yield session
    except ImportError:
        raise HTTPException(status_code=501, detail="Database not installed")
    except RuntimeError as e:
        raise HTTPException(status_code=503, detail=f"Database not available: {str(e)}")

Usage in Endpoints:

from fastapi import Depends
from sqlalchemy.ext.asyncio import AsyncSession
from src.server.utils.dependencies import get_database_session

@router.get("/series/{series_id}")
async def get_series(
    series_id: int,
    db: AsyncSession = Depends(get_database_session)
):
    result = await db.execute(
        select(AnimeSeries).where(AnimeSeries.id == series_id)
    )
    series = result.scalar_one_or_none()
    if not series:
        raise HTTPException(status_code=404, detail="Series not found")
    return series

Testing

Test Suite: tests/unit/test_database_models.py

Coverage:

30+ comprehensive test cases
Model creation and validation
Relationship testing (one-to-many, cascade deletes)
Unique constraint validation
Query operations (filtering, joins)
Session management
Mixin functionality

Test Strategy:

In-memory SQLite database for isolation
Fixtures for engine and session setup
Test all CRUD operations
Verify constraints and relationships
Test edge cases and error conditions

Migration Strategy (Future)

Alembic Integration (planned):

Alembic installed but not yet configured
Will manage schema migrations in production
Auto-generate migrations from model changes
Version control for database schema

Initial Setup:

# Initialize Alembic (future)
alembic init alembic

# Generate initial migration
alembic revision --autogenerate -m "Initial schema"

# Apply migrations
alembic upgrade head

Production Considerations

Single-Process Deployment (current):

SQLite with WAL mode for concurrency
Static pool for single connection
File-based storage at data/aniworld.db

Multi-Process Deployment (future):

Switch to PostgreSQL or MySQL
Configure connection pooling (pool_size, max_overflow)
Use QueuePool for connection management
Consider read replicas for scaling

Performance:

Indexes on frequently queried columns (key, name, status, is_active)
Foreign key constraints for referential integrity
Cascade deletes for cleanup operations
Efficient joins via relationship loading strategies

Monitoring:

SQL query logging in DEBUG mode
Connection pool metrics (when using QueuePool)
Query performance profiling
Database size monitoring

Backup Strategy:

SQLite: File-based backups (copy aniworld.db file)
WAL checkpoint before backup
Automated backup schedule recommended
Store backups in data/config_backups/ or separate location

Integration with Services

AnimeService:

Query series from database
Persist scan results
Update episode metadata

DownloadService:

Load queue from database on startup
Persist queue state continuously
Update download progress in real-time

AuthService:

Store and validate user sessions
Session revocation via database
Query active sessions for monitoring

Benefits of Database Layer

Persistence: Survives application restarts
Relationships: Enforced referential integrity
Queries: Powerful filtering and aggregation
Scalability: Can migrate to PostgreSQL/MySQL
ACID: Atomic transactions for consistency
Migration: Schema versioning with Alembic
Testing: Easy to test with in-memory database

Database Service Layer (October 2025)

Implemented comprehensive service layer for database CRUD operations.

File: src/server/database/service.py

Services:

AnimeSeriesService: CRUD operations for anime series
EpisodeService: Episode management and download tracking
DownloadQueueService: Queue management with priority and status
UserSessionService: Session management and authentication

Key Features:

Repository pattern for clean separation of concerns
Type-safe operations with comprehensive type hints
Async support for all database operations
Transaction management via FastAPI dependency injection
Comprehensive error handling and logging
Search and filtering capabilities
Pagination support for large datasets
Batch operations for performance

AnimeSeriesService Operations:

Create series with metadata and provider information
Retrieve by ID, key, or search query
Update series attributes
Delete series with cascade to episodes and queue items
List all series with pagination and eager loading options

EpisodeService Operations:

Create episodes for series
Retrieve episodes by series, season, or specific episode
Mark episodes as downloaded with file metadata
Delete episodes

DownloadQueueService Operations:

Add items to queue with priority levels (LOW, NORMAL, HIGH)
Retrieve pending, active, or all queue items
Update download status (PENDING, DOWNLOADING, COMPLETED, FAILED, etc.)
Update download progress (percentage, bytes, speed)
Clear completed downloads
Retry failed downloads with max retry limits
Automatic timestamp management (started_at, completed_at)

UserSessionService Operations:

Create authentication sessions with JWT tokens
Retrieve sessions by session ID
Get active sessions with expiry checking
Update last activity timestamp
Revoke sessions for logout
Cleanup expired sessions automatically

Testing:

Comprehensive test suite with 22 test cases
In-memory SQLite for isolated testing
All CRUD operations tested
Edge cases and error conditions covered
100% test pass rate

Integration:

Exported via database package __init__.py
Used by API endpoints via dependency injection
Compatible with existing database models
Follows project coding standards (PEP 8, type hints, docstrings)

Database Migrations (src/server/database/migrations.py):

Simple schema initialization via SQLAlchemy create_all
Schema version checking utility
Documentation for Alembic integration
Production-ready migration strategy outlined

Core Application Logic

SeriesApp - Enhanced Core Engine

The SeriesApp class (src/core/SeriesApp.py) is the main application engine for anime series management. Enhanced with async support and web integration capabilities.

Key Features

Async Operations: Support for async download and scan operations
Progress Callbacks: Real-time progress reporting via callbacks
Cancellation Support: Ability to cancel long-running operations
Error Handling: Comprehensive error handling with callback notifications
Operation Status: Track current operation status and history

Core Classes

SeriesApp: Main application class
OperationStatus: Enum for operation states (IDLE, RUNNING, COMPLETED, CANCELLED, FAILED)
ProgressInfo: Dataclass for progress information
OperationResult: Dataclass for operation results

Key Methods

search(words): Search for anime series
download(): Download episodes with progress tracking
ReScan(): Scan directory for missing episodes
cancel_operation(): Cancel current operation
get_operation_status(): Get current status
get_series_list(): Get series with missing episodes

Integration Points

The SeriesApp integrates with:

Provider system for content downloading
Serie scanner for directory analysis
Series list management for tracking missing episodes
Web layer via async operations and callbacks

Progress Callback System

Overview

A comprehensive callback system for real-time progress reporting, error handling, and operation completion notifications across core operations (scanning, downloading, searching).

Architecture

Interface-based Design: Abstract base classes define callback contracts
Context Objects: Rich context information for each callback type
Callback Manager: Centralized management of multiple callbacks
Thread-safe: Exception handling prevents callback errors from breaking operations

Components

Callback Interfaces (`src/core/interfaces/callbacks.py`)

ProgressCallback: Reports operation progress updates
ErrorCallback: Handles error notifications
CompletionCallback: Notifies operation completion

Context Classes

ProgressContext: Current progress, percentage, phase, and metadata
ErrorContext: Error details, recoverability, retry information
CompletionContext: Success status, results, and statistics

All context dataclasses expose a key field (provider identifier) plus an optional folder field used purely for display metadata. This keeps the callback contract aligned with the broader series identifier standardization work: downstream consumers rely on key for lookups while still showing a user-friendly folder name when needed.

Enums

OperationType: SCAN, DOWNLOAD, SEARCH, INITIALIZATION
ProgressPhase: STARTING, IN_PROGRESS, COMPLETING, COMPLETED, FAILED, CANCELLED

Callback Manager

Register/unregister multiple callbacks per type
Notify all registered callbacks with context
Exception handling for callback errors
Support for clearing all callbacks

Integration

SerieScanner

Reports scanning progress (folder by folder)
Notifies errors for failed folder scans
Reports completion with statistics

SeriesApp

Download progress reporting with percentage
Scan progress through SerieScanner integration
Error notifications for all operations
Completion notifications with results

Usage Example

from src.core.interfaces.callbacks import (
    CallbackManager,
    ProgressCallback,
    ProgressContext
)

class MyProgressCallback(ProgressCallback):
    def on_progress(self, context: ProgressContext):
        print(f"{context.message}: {context.percentage:.1f}%")

# Register callback
manager = CallbackManager()
manager.register_progress_callback(MyProgressCallback())

# Use with SeriesApp
app = SeriesApp(directory, callback_manager=manager)

Recent Infrastructure Changes

Progress Callback System (October 2025)

Implemented a comprehensive progress callback system for real-time operation tracking.

Changes Made

Callback Interfaces:
- Created abstract base classes for progress, error, and completion callbacks
- Defined rich context objects with operation metadata
- Implemented thread-safe callback manager
SerieScanner Integration:
- Added progress reporting for directory scanning
- Implemented per-folder progress updates
- Error callbacks for scan failures
- Completion notifications with statistics
SeriesApp Integration:
- Integrated callback manager into download operations
- Progress updates during episode downloads
- Error handling with callback notifications
- Completion callbacks for all operations
- Backward compatibility with legacy callbacks
Testing:
- 22 comprehensive unit tests
- Coverage for all callback types
- Exception handling verification
- Multiple callback registration tests
Identifier Support (Nov 2025):
- Added key + optional folder fields to every context object
- Docstrings now clarify that key is the canonical lookup identifier
- Tests updated to guarantee both fields serialize correctly

Core Logic Enhancement (October 2025)

Enhanced SeriesApp with async callback support, progress reporting, and cancellation.

Changes Made

Async Support:
- Added async_download() and async_rescan() methods
- Integrated with asyncio event loop for non-blocking operations
- Support for concurrent operations in web environment
Progress Reporting:
- Legacy ProgressInfo dataclass for structured progress data
- New comprehensive callback system with context objects
- Percentage calculation and status tracking
Cancellation System:
- Internal cancellation flag management
- Graceful operation cancellation
- Cancellation check during long-running operations
Error Handling:
- OperationResult dataclass for operation outcomes
- Error callback system for notifications
- Specific exception types (IOError, OSError, RuntimeError)
- Proper exception propagation and logging
Status Management:
- OperationStatus enum for state tracking
- Current operation identifier
- Status getter methods for monitoring

Test Coverage

Comprehensive test suite (tests/unit/test_series_app.py) with 22 tests covering:

Initialization and configuration
Search functionality
Download operations with callbacks
Directory scanning with progress
Async operations
Cancellation handling
Error scenarios
Data model validation

AnimeService Identifier Standardization (November 2025)

Updated AnimeService to consistently use key as the primary series identifier, aligning with the broader identifier standardization initiative.

Changes Made

Documentation Updates:
- Enhanced class docstring to clarify key vs folder usage
- Updated all method docstrings to document identifier roles
- key: Primary identifier for series lookups (provider-assigned, URL-safe)
- folder: Metadata only, used for display and filesystem operations
Event Handler Clarification:
- _on_download_status(): Documents that events include both key and serie_folder
- _on_scan_status(): Documents that events include both key and folder
- Event handlers properly forward both identifiers to progress service
Method Documentation:
- list_missing(): Returns series dicts with key as primary identifier
- search(): Returns results with key as identifier
- rescan(): Clarifies all series identified by key
- download(): Detailed documentation of parameter roles
Code Quality Improvements:
- Updated type hints to use modern Python 3.9+ style (list[dict] vs List[dict])
- Fixed line length violations for PEP 8 compliance
- Improved type safety with explicit type annotations

Implementation Status

✅ All methods use key for series identification
✅ Event handlers properly receive and forward key field
✅ Docstrings clearly document identifier usage
✅ All anime service tests pass (18/18 passing)
✅ Code follows project standards (PEP 8, type hints, docstrings)

Task: Phase 3, Task 3.2 - Update AnimeService to Use Key
Completion Date: November 23, 2025

Template Integration (October 2025)

Completed integration of HTML templates with FastAPI Jinja2 system.

Changes Made

Template Helper Utilities:
- src/server/utils/template_helpers.py - Template rendering utilities
- Centralized base context for all templates
- Template validation and listing functions
- DRY principles for template rendering
Enhanced CSS:
- src/server/web/static/css/ux_features.css - UX enhancement styles
- Drag-and-drop indicators
- Bulk selection styling
- Keyboard navigation focus indicators
- Touch gesture feedback
- Mobile responsive enhancements
- Accessibility features (high contrast, screen reader support)
- Reduced motion support
JavaScript Modules:
- keyboard_shortcuts.js - Keyboard navigation (Ctrl+K, Ctrl+R, etc.)
- user_preferences.js - Settings persistence (localStorage)
- undo_redo.js - Action history with Ctrl+Z/Ctrl+Y
- mobile_responsive.js - Mobile detection and layout
- touch_gestures.js - Swipe gesture handling
- accessibility_features.js - Focus management and ARIA labels
- screen_reader_support.js - Live regions for dynamic content
- color_contrast_compliance.js - WCAG compliance checks
- multi_screen_support.js - Fullscreen and multi-monitor support
- drag_drop.js - Drag-and-drop functionality (stub)
- bulk_operations.js - Bulk selection and actions (stub)
- advanced_search.js - Advanced filtering (stub)
Updated Controllers:
- Updated page_controller.py to use template_helpers
- Updated error_controller.py to use template_helpers
- Consistent context passing across all templates

Template Features

Responsive Design: Mobile-first approach with viewport meta tags
Theme Switching: Light/dark mode with data-theme attribute
Accessibility: ARIA labels, keyboard navigation, screen reader support
Internationalization: Localization support via localization.js
Progressive Enhancement: Works without JavaScript, enhanced with it

Verified Templates

All HTML templates properly integrated:

index.html - Main application page with search and anime list
login.html - Master password authentication
setup.html - Initial application setup
queue.html - Download queue management
error.html - Error pages (404, 500)

All templates include:

Proper HTML5 structure
Font Awesome icons
Static file references (/static/css/, /static/js/)
Theme switching support
Responsive viewport configuration

CSS Integration (October 2025)

Integrated existing CSS styling with FastAPI's static file serving system.

Implementation Details

Static File Configuration:
- Static files mounted at /static in fastapi_app.py
- Directory: src/server/web/static/
- Files served using FastAPI's StaticFiles middleware
- All paths use absolute references (/static/...)
CSS Architecture:
- styles.css (1,840 lines) - Main stylesheet with Fluent UI design system
- ux_features.css (203 lines) - Enhanced UX features and accessibility
Design System (styles.css):
- Fluent UI Variables: CSS custom properties for consistent theming
- Light/Dark Themes: Dynamic theme switching via [data-theme="dark"]
- Typography: Segoe UI font stack with responsive sizing
- Spacing System: Consistent spacing scale (xs through xxl)
- Color Palette: Comprehensive color system for both themes
- Border Radius: Standardized corner radii (sm, md, lg, xl)
- Shadows: Elevation system with card and elevated variants
- Transitions: Smooth animations with consistent timing
UX Features (ux_features.css):
- Drag-and-drop indicators
- Bulk selection styling
- Keyboard focus indicators
- Touch gesture feedback
- Mobile responsive utilities
- High contrast mode support (@media (prefers-contrast: high))
- Screen reader utilities (.sr-only)
- Window control components

CSS Variables

Color System:

/* Light Theme */
--color-bg-primary: #ffffff
--color-accent: #0078d4
--color-text-primary: #323130

/* Dark Theme */
--color-bg-primary-dark: #202020
--color-accent-dark: #60cdff
--color-text-primary-dark: #ffffff

Spacing & Typography:

--spacing-sm: 8px
--spacing-md: 12px
--spacing-lg: 16px
--font-size-body: 14px
--font-size-title: 20px

Template CSS References

All HTML templates correctly reference CSS files:

Index page: Includes both styles.css and ux_features.css
Other pages: Include styles.css
All use absolute paths: /static/css/styles.css

Responsive Design

Mobile-first approach with breakpoints
Media queries for tablet and desktop layouts
Touch-friendly interface elements
Adaptive typography and spacing

Accessibility Features

WCAG-compliant color contrast
High contrast mode support
Screen reader utilities
Keyboard navigation styling
Focus indicators
Reduced motion support

Testing

Comprehensive test suite in tests/unit/test_static_files.py:

CSS file accessibility tests
Theme support verification
Responsive design validation
Accessibility feature checks
Content integrity validation
Path correctness verification

All 17 CSS integration tests passing.

Route Controller Refactoring (October 2025)

Restructured the FastAPI application to use a controller-based architecture for better code organization and maintainability.

Changes Made

Created Controller Structure:
- src/server/controllers/ - New directory for route controllers
- src/server/controllers/__init__.py - Controllers package initialization
- src/server/controllers/health_controller.py - Health check endpoints
- src/server/controllers/page_controller.py - HTML page routes
- src/server/controllers/error_controller.py - Error handling controllers
Shared Template Configuration:
- src/server/utils/templates.py - Centralized Jinja2 template configuration
- Fixed template path resolution for proper template loading
Main Application Updates:
- src/server/fastapi_app.py - Refactored to use controller routers
- Removed direct route definitions from main file
- Added router inclusion using app.include_router()
- Simplified error handlers to delegate to controller functions
Fixed Import Issues:
- Resolved circular import in src/core/__init__.py
- Removed non-existent application module import

Controller Architecture

Anime Service Notes

The new anime_service runs the existing blocking SeriesApp inside a threadpool (via ThreadPoolExecutor). This keeps the FastAPI event loop responsive while leveraging the existing core logic.
A small in-process LRU cache is used for the frequently-read "missing episodes" list to reduce IO; cache invalidation happens after a rescan.
For multi-worker or multi-host deployments, move cache/state to a shared store (Redis) and ensure the threadpool sizing matches the worker's CPU and IO profile.

Health Controller (health_controller.py):

router = APIRouter(prefix="/health", tags=["health"])
@router.get("") - Health check endpoint

Page Controller (page_controller.py):

router = APIRouter(tags=["pages"])
@router.get("/") - Main application page
@router.get("/setup") - Setup page
@router.get("/login") - Login page
@router.get("/queue") - Download queue page

Error Controller (error_controller.py):

async def not_found_handler() - Custom 404 error handling
async def server_error_handler() - Custom 500 error handling

Benefits of the New Structure

Separation of Concerns: Each controller handles specific functionality
Modularity: Easy to add new controllers and routes
Testability: Controllers can be tested independently
Maintainability: Cleaner code organization and easier debugging
Scalability: Simple to extend with new features

Verified Working Endpoints

All endpoints tested and confirmed working:

Health: /health → Returns {"status": "healthy", ...}
Root: / → Serves main application page
Setup: /setup → Serves setup page
Auth API: /api/auth/* → Endpoints for setup, login, logout and status (JWT-based)
Login: /login → Serves login page
Queue: /queue → Serves download queue page

File Structure After Refactoring

src/server/
├── fastapi_app.py                 # Main FastAPI application (simplified)
├── controllers/                   # NEW: Route controllers
│   ├── __init__.py               # Controllers package
├── utils/
│   ├── dependencies.py          # Existing dependency injection
│   └── templates.py              # NEW: Shared Jinja2 template config
└── web/                          # Existing frontend assets
    ├── templates/                # HTML templates
    └── static/                   # CSS, JS, images

Authentication Middleware (October 2025)

An authentication middleware component was added to the FastAPI application to centralize token parsing and provide lightweight protection of authentication endpoints:

src/server/middleware/auth.py implements:
- Bearer JWT parsing and session attachment to request.state.session
- A simple per-IP in-memory rate limiter applied to /api/auth/login and /api/auth/setup (default 5 requests/minute)

Notes:

This is intentionally simple and designed for single-process deployments. For production use across multiple workers or hosts, replace the in-memory limiter with a distributed store (e.g. Redis) and add a persistent token revocation list if needed.

API Models and Contracts

Pydantic models living in src/server/models/ define the canonical API contracts used by FastAPI endpoints. These models are intentionally lightweight and focused on serialization, validation, and OpenAPI documentation generation.
Keep models stable: changes to model shapes are breaking changes for clients. Bump API versioning or provide migration layers when altering public response fields.
Infrastructure considerations: ensure the deployment environment has required libraries (e.g., pydantic) installed and that schema validation errors are logged to the centralized logging system. For high-throughput routes, consider response model caching at the application or reverse-proxy layer.

WebSocket Real-time Communication (October 2025)

A comprehensive WebSocket infrastructure was implemented to provide real-time updates for downloads, queue status, and system events:

File: src/server/services/websocket_service.py
Models: src/server/models/websocket.py
Endpoint: ws://host:port/ws/connect

WebSocket Service Architecture

ConnectionManager: Low-level connection lifecycle management
- Connection registry with unique connection IDs
- Room-based messaging for topic subscriptions
- Automatic connection cleanup and health monitoring
- Thread-safe operations with asyncio locks
WebSocketService: High-level application messaging
- Convenient interface for broadcasting application events
- Pre-defined message types for downloads, queue, and system events
- Singleton pattern via get_websocket_service() factory

Supported Message Types

Download Events: download_progress, download_complete, download_failed
Queue Events: queue_status, queue_started, queue_stopped, queue_paused, queue_resumed
System Events: system_info, system_warning, system_error
Connection: connected, ping, pong, error

Room-Based Messaging

Clients can subscribe to specific topics (rooms) to receive targeted updates:

downloads room: All download-related events
Custom rooms: Can be added for specific features

Integration with Download Service

Download service automatically broadcasts progress updates via WebSocket
Broadcast callback registered during service initialization
Updates sent to all clients subscribed to the downloads room
No blocking of download operations (async broadcast)

Client Connection Flow

Client connects to /ws/connect endpoint
Server assigns unique connection ID and sends confirmation
Client joins rooms (e.g., {"action": "join", "room": "downloads"})
Server broadcasts updates to subscribed rooms
Client disconnects (automatic cleanup)

Infrastructure Notes

Single-process: Current implementation uses in-memory connection storage
Production: For multi-worker/multi-host deployments:
- Move connection registry to Redis or similar shared store
- Implement pub/sub for cross-process message broadcasting
- Add connection persistence for recovery after restarts
Monitoring: WebSocket status available at /ws/status endpoint
Security: Optional authentication via JWT (user_id tracking)
Testing: Comprehensive unit tests in tests/unit/test_websocket_service.py

Download Queue Models

Download queue models in src/server/models/download.py define the data structures for the download queue system.
Key models include:
- DownloadItem: Represents a single queued download with metadata, progress tracking, and error information
- QueueStatus: Overall queue state with active, pending, completed, and failed downloads
- QueueStats: Aggregated statistics for monitoring queue performance
- DownloadProgress: Real-time progress information (percent, speed, ETA)
- DownloadRequest/DownloadResponse: API request/response contracts
Models enforce validation constraints (e.g., positive episode numbers, progress percentage 0-100, non-negative retry counts) and provide clean JSON serialization for API endpoints and WebSocket updates.

Download Queue Service

The download service (src/server/services/download_service.py) manages the complete lifecycle of anime episode downloads.
Core features:
- Priority-based Queue: Items added with HIGH priority are processed first, NORMAL and LOW follow in FIFO order
- Concurrent Processing: Configurable max concurrent downloads (default 2) to optimize bandwidth usage
- Persistence: Queue state is automatically saved to data/download_queue.json and recovered on service restart
- Retry Logic: Failed downloads are automatically retried up to a configurable limit (default 3 attempts) with exponential backoff
- Progress Tracking: Real-time download progress with speed, percentage, and ETA calculations
- WebSocket Integration: Broadcasts queue updates, progress, and completion/failure events to connected clients
Operations:
- add_to_queue(): Add episodes to download queue with priority
- remove_from_queue(): Cancel pending or active downloads
- reorder_queue(): Manually adjust queue order for pending items
- pause_queue()/resume_queue(): Control download processing
- retry_failed(): Retry failed downloads with retry count checks
- get_queue_status(): Get complete queue state (active, pending, completed, failed)
- get_queue_stats(): Get aggregated statistics (counts, download size, speed)
Infrastructure notes:
- Service uses ThreadPoolExecutor for concurrent download processing
- Queue processor runs as async background task with configurable sleep intervals
- Progress callbacks are executed in threadpool and broadcast via async WebSocket
- For multi-process deployments, move queue state to shared store (Redis/DB) and implement distributed locking for concurrent access control
- Singleton instance pattern used via get_download_service() factory
Testing: Comprehensive unit tests in tests/unit/test_download_service.py cover queue operations, persistence, retry logic, and error handling

Series Identifier Standardization (November 2025)

Task 3.1 Completed: Updated DownloadService to use standardized series identifiers.

Changes Made:

serie_id Field: Now explicitly documented as the provider key (e.g., "attack-on-titan"). This is the unique, URL-safe identifier used for all series lookups and identification throughout the system.
serie_folder Field: Changed from Optional to required. This field contains the filesystem folder name (e.g., "Attack on Titan (2013)") and is used exclusively for filesystem operations.
Separation of Concerns: Removed incorrect fallback logic that used serie_id as a substitute for serie_folder. These fields now serve distinct purposes and must both be provided.
Enhanced Documentation: Updated docstrings in add_to_queue() method and all Pydantic models to clarify the purpose and usage of each identifier field.
Improved Logging: Updated log statements to reference serie_key for identification, making it clear which identifier is being used.

Models Updated:

DownloadItem (src/server/models/download.py):
- serie_id: Required field with provider key
- serie_folder: Changed from Optional to required
- Both fields have enhanced field descriptions
DownloadRequest (src/server/models/download.py):
- serie_folder: Changed from Optional to required
- Enhanced field descriptions for both identifiers

Service Changes:

DownloadService.add_to_queue(): Updated to validate that serie_folder is always provided and raises DownloadServiceError if missing
Removed fallback logic: folder = item.serie_folder if item.serie_folder else item.serie_id
Added validation check before download execution
Updated logging to use serie_key parameter name

Testing:

Updated all test fixtures to include required serie_folder field
All 25 download service tests passing
All 47 download model tests passing

Benefits:

Clear separation between provider identifier (key) and filesystem path (folder)
Prevents confusion and potential bugs from mixing identifiers
Consistent with broader series identifier standardization effort
Better error messages when required fields are missing

Download Queue API Endpoints (October 2025)

Implemented comprehensive REST API endpoints for download queue management:

File: src/server/api/download.py
Router Prefix: /api/queue
Authentication: All endpoints require JWT authentication via require_auth dependency

Implemented Endpoints

GET /api/queue/status - Retrieve complete queue status
- Returns: QueueStatusResponse with status and statistics
- Includes: active downloads, pending items, completed/failed items, queue stats
POST /api/queue/add - Add episodes to download queue
- Request: DownloadRequest with serie info, episodes, and priority
- Returns: DownloadResponse with added item IDs
- Validates episode list is non-empty
- Supports HIGH, NORMAL, and LOW priority levels
DELETE /api/queue/{item_id} - Remove single item from queue
- Returns: 204 No Content on success, 404 if item not found
- Cancels active downloads if necessary
DELETE /api/queue/ - Remove multiple items (batch operation)
- Request: QueueOperationRequest with list of item IDs
- Returns: 204 No Content (partial success acceptable)
POST /api/queue/start - Start queue processor
- Idempotent operation (safe to call multiple times)
POST /api/queue/stop - Stop queue processor
- Waits for active downloads to complete (with timeout)
POST /api/queue/pause - Pause queue processing
- Active downloads continue, no new downloads start
POST /api/queue/resume - Resume queue processing
POST /api/queue/reorder - Reorder pending queue item
- Request: QueueReorderRequest with item_id and new_position
- Returns: 404 if item not in pending queue
DELETE /api/queue/completed - Clear completed items from history
- Returns count of cleared items
POST /api/queue/retry - Retry failed downloads
- Request: QueueOperationRequest with item IDs (empty for all)
- Only retries items under max retry limit

Dependencies

get_download_service: Factory function providing singleton DownloadService instance
- Automatically initializes AnimeService as dependency
- Raises 503 if anime directory not configured
get_anime_service: Factory function providing singleton AnimeService instance
- Required by DownloadService for anime operations
Both dependencies added to src/server/utils/dependencies.py

Error Handling

All endpoints return structured JSON error responses
HTTP status codes follow REST conventions (200, 201, 204, 400, 401, 404, 500, 503)
Service-level exceptions (DownloadServiceError) mapped to 400 Bad Request
Generic exceptions mapped to 500 Internal Server Error
Authentication errors return 401 Unauthorized

Testing

Comprehensive test suite in tests/api/test_download_endpoints.py
Tests cover:
- Successful operations for all endpoints
- Authentication requirements
- Error conditions (empty lists, not found, service errors)
- Priority handling
- Batch operations
Uses pytest fixtures for authenticated client and mocked download service

Integration

Router registered in src/server/fastapi_app.py via app.include_router(download_router)
Follows same patterns as other API routers (auth, anime, config)
Full OpenAPI documentation available at /api/docs

WebSocket Integration with Core Services (October 2025)

Completed comprehensive integration of WebSocket broadcasting with all core services to provide real-time updates for downloads, scans, queue operations, and progress tracking.

ProgressService

File: src/server/services/progress_service.py

A centralized service for tracking and broadcasting real-time progress updates across the application.

Key Features:

Track multiple concurrent progress operations (downloads, scans, queue changes)
Automatic progress percentage calculation
Progress lifecycle management (start, update, complete, fail, cancel)
WebSocket integration for real-time client updates
Progress history with configurable size limit (default: 50 items)
Thread-safe operations using asyncio locks
Support for progress metadata and custom messages

Progress Types:

DOWNLOAD - File download progress
SCAN - Library scan progress
QUEUE - Queue operation progress
SYSTEM - System-level operations
ERROR - Error notifications

Progress Statuses:

STARTED - Operation initiated
IN_PROGRESS - Operation in progress
COMPLETED - Successfully completed
FAILED - Operation failed
CANCELLED - Cancelled by user

Core Methods:

start_progress() - Initialize new progress operation
update_progress() - Update progress with current/total values
complete_progress() - Mark operation as completed
fail_progress() - Mark operation as failed
cancel_progress() - Cancel ongoing operation
get_progress() - Retrieve progress by ID
get_all_active_progress() - Get all active operations (optionally filtered by type)

Broadcasting:

Integrates with WebSocketService via callback
Broadcasts to room-specific channels (e.g., download_progress, scan_progress)
Configurable broadcast throttling (only on significant changes >1% or forced)
Automatic progress state serialization to JSON

Singleton Pattern:

Global instance via get_progress_service() factory
Initialized during application startup with WebSocket callback

Series Identifier Support (November 2025):

Added optional key and folder fields to ProgressUpdate dataclass
key: Primary series identifier (provider key, e.g., 'attack-on-titan')
folder: Optional series folder name (e.g., 'Attack on Titan (2013)')
Both fields are included in progress events when series-related
to_dict() serialization includes key/folder when present
start_progress() and update_progress() accept key/folder parameters
Maintains backward compatibility - fields are optional

Usage Example:

# Start progress for a series download
await progress_service.start_progress(
    progress_id="download-123",
    progress_type=ProgressType.DOWNLOAD,
    title="Downloading Attack on Titan",
    key="attack-on-titan",
    folder="Attack on Titan (2013)",
    total=100
)

# Update progress
await progress_service.update_progress(
    progress_id="download-123",
    current=50,
    message="Downloaded 50 MB"
)

Test Coverage:

25 comprehensive unit tests
Tests verify key/folder serialization
Tests verify key/folder preservation during updates
Tests verify optional nature of fields
All tests passing

Integration with Services

DownloadService Integration:

Progress tracking for each download item
Real-time progress updates during file download
Automatic completion/failure notifications
Progress metadata includes speed, ETA, downloaded bytes

AnimeService Integration:

Progress tracking for library scans
Scan progress with current/total file counts
Scan completion with statistics
Error notifications on scan failures

WebSocket Message Models

File: src/server/models/websocket.py

Added progress-specific message models:

ScanProgressMessage - Scan progress updates
ScanCompleteMessage - Scan completion notification
ScanFailedMessage - Scan failure notification
ErrorNotificationMessage - Critical error notifications
ProgressUpdateMessage - Generic progress updates

WebSocket Message Types:

SCAN_PROGRESS - Scan progress updates
SCAN_COMPLETE - Scan completion
SCAN_FAILED - Scan failure
Extended existing types for downloads and queue updates

WebSocket Rooms

Clients can subscribe to specific progress channels:

download_progress - Download progress updates
scan_progress - Library scan updates
queue_progress - Queue operation updates
system_progress - System-level updates

Room subscription via client messages:

{
    "action": "join",
    "room": "download_progress"
}

Application Startup

File: src/server/fastapi_app.py

Progress service initialized on application startup:

Get ProgressService singleton instance
Get WebSocketService singleton instance
Register broadcast callback to link progress updates with WebSocket
Callback broadcasts progress messages to appropriate rooms

Testing

File: tests/unit/test_progress_service.py

Comprehensive test coverage including:

Progress lifecycle operations (start, update, complete, fail, cancel)
Percentage calculation accuracy
History management and size limits
Broadcast callback invocation
Concurrent progress operations
Metadata handling
Error conditions and edge cases

Architecture Benefits

Decoupling: ProgressService decouples progress tracking from WebSocket broadcasting
Reusability: Single service used across all application components
Scalability: Supports multiple concurrent operations efficiently
Observability: Centralized progress tracking simplifies monitoring
Real-time UX: Instant feedback to users on all long-running operations

Future Enhancements

Persistent progress history (database storage)
Progress rate calculation and trend analysis
Multi-process progress synchronization (Redis/shared store)
Progress event hooks for custom actions
Client-side progress resumption after reconnection

Core Services WebSocket Integration (October 2025)

Completed comprehensive integration of WebSocket broadcasting with all core services (DownloadService, AnimeService, ProgressService) to provide real-time updates to connected clients.

DownloadService WebSocket Integration

File: src/server/services/download_service.py

The download service broadcasts real-time updates for all queue and download operations:

Download Progress Broadcasting:

download_progress - Real-time progress updates during download
- Includes: download_id, serie_name, season, episode, progress data (percent, speed, ETA)
- Sent via ProgressService which broadcasts to download_progress room
- Progress callback created for each download item with metadata tracking

Download Completion/Failure Broadcasting:

download_complete - Successful download completion
- Includes: download_id, serie_name, season, episode, downloaded_mb
- Broadcast to downloads room
download_failed - Download failure notification
- Includes: download_id, serie_name, season, episode, error, retry_count
- Broadcast to downloads room

Queue Operations Broadcasting: All queue operations broadcast queue_status messages with current queue state:

items_added - Items added to queue
- Data: added_ids, queue_status (complete queue state)
items_removed - Items removed/cancelled
- Data: removed_ids, queue_status
queue_reordered - Queue order changed
- Data: item_id, new_position, queue_status
items_retried - Failed items retried
- Data: retried_ids, queue_status
completed_cleared - Completed items cleared
- Data: cleared_count, queue_status

Queue Control Broadcasting:

queue_started - Queue processor started
- Data: is_running=True, queue_status
queue_stopped - Queue processor stopped
- Data: is_running=False, queue_status
queue_paused - Queue processing paused
- Data: is_paused=True, queue_status
queue_resumed - Queue processing resumed
- Data: is_paused=False, queue_status

Broadcast Callback Setup: The download service broadcast callback is registered during dependency injection in src/server/utils/dependencies.py:

Maps update types to WebSocket service methods
Routes download_progress, download_complete, download_failed to appropriate rooms
All queue operations broadcast complete queue status for client synchronization

AnimeService WebSocket Integration

File: src/server/services/anime_service.py

The anime service integrates with ProgressService for library scan operations:

Scan Progress Broadcasting:

Scan operations use ProgressService for progress tracking
Progress updates broadcast to scan_progress room
Lifecycle events:
- started - Scan initialization
- in_progress - Ongoing scan with current/total file counts
- completed - Successful scan completion
- failed - Scan failure with error message

Scan Implementation:

rescan() method wraps SeriesApp.ReScan with progress tracking
Progress callback executed in threadpool updates ProgressService
ProgressService automatically broadcasts to WebSocket clients
Cache invalidation on successful scan completion

ProgressService WebSocket Integration

File: src/server/services/progress_service.py

Central service for tracking and broadcasting all progress operations:

Progress Types:

DOWNLOAD - File download progress
SCAN - Library scan progress
QUEUE - Queue operation progress
SYSTEM - System-level operations
ERROR - Error notifications

Progress Lifecycle:

start_progress() - Initialize progress operation
- Broadcasts to room: {progress_type}_progress
update_progress() - Update progress values
- Calculates percentage automatically
- Broadcasts only on significant changes (>1% or forced)
complete_progress() - Mark operation complete
- Sets progress to 100%
- Moves to history
- Broadcasts completion
fail_progress() - Mark operation failed
- Captures error message
- Moves to history
- Broadcasts failure

Broadcast Callback:

Callback registered during application startup in src/server/fastapi_app.py
Links ProgressService to WebSocketService.manager.broadcast_to_room
All progress updates automatically broadcast to appropriate rooms

WebSocket Room Structure

Clients subscribe to specific rooms to receive targeted updates:

Room Types:

downloads - All download-related events (complete, failed, queue status)
download_progress - Real-time download progress updates
scan_progress - Library scan progress updates
queue_progress - Queue operation progress (future use)
system_progress - System-level progress (future use)

Room Subscription: Clients join rooms by sending WebSocket messages:

{
    "action": "join",
    "room": "download_progress"
}

Message Format

All WebSocket messages follow a consistent structure:

{
  "type": "download_progress" | "download_complete" | "queue_status" | etc.,
  "timestamp": "2025-10-17T12:34:56.789Z",
  "data": {
    // Message-specific data
  }
}

Example: Download Progress

{
    "type": "download_progress",
    "timestamp": "2025-10-17T12:34:56.789Z",
    "data": {
        "download_id": "abc123",
        "serie_name": "Attack on Titan",
        "season": 1,
        "episode": 5,
        "progress": {
            "percent": 45.2,
            "downloaded_mb": 226.0,
            "total_mb": 500.0,
            "speed_mbps": 2.5,
            "eta_seconds": 120
        }
    }
}

Example: Queue Status

{
  "type": "queue_status",
  "timestamp": "2025-10-17T12:34:56.789Z",
  "data": {
    "action": "items_added",
    "added_ids": ["item1", "item2"],
    "queue_status": {
      "is_running": true,
      "is_paused": false,
      "active_downloads": [...],
      "pending_queue": [...],
      "completed_downloads": [...],
      "failed_downloads": [...]
    }
  }
}

Integration Testing

File: tests/integration/test_websocket_integration.py

Comprehensive integration tests verify WebSocket broadcasting:

Test Coverage:

Download progress broadcasts during active downloads
Queue operation broadcasts (add, remove, reorder, clear, retry)
Queue control broadcasts (start, stop, pause, resume)
Scan progress broadcasts (start, update, complete, fail)
Progress lifecycle broadcasts for all operation types
End-to-end flow with multiple services broadcasting

Test Strategy:

Mock broadcast callbacks to capture emitted messages
Verify message types, data structure, and content
Test both successful and failure scenarios
Verify proper room routing for different message types

Architecture Benefits

Decoupling:

Services use generic broadcast callbacks without WebSocket dependencies
ProgressService provides abstraction layer for progress tracking
Easy to swap WebSocket implementation or add additional broadcast targets

Consistency:

All services follow same broadcast patterns
Standardized message formats across application
Unified progress tracking via ProgressService

Real-time UX:

Instant feedback on all long-running operations
Live queue status updates
Progress bars update smoothly without polling
Error notifications delivered immediately

Scalability:

Room-based messaging enables targeted updates
Multiple concurrent operations supported
Easy to add new progress types and message formats

Production Considerations

Single-Process Deployment (Current):

In-memory connection registry in WebSocketService
Works perfectly for single-worker deployments
No additional infrastructure required

Multi-Process/Multi-Host Deployment (Future):

Move connection registry to Redis or similar shared store
Implement pub/sub for cross-process message broadcasting
Add connection persistence for recovery after restarts
Consider using sticky sessions or connection migration

Performance:

Progress updates throttled to >1% changes to reduce message volume
Broadcast operations are fire-and-forget (non-blocking)
Failed connections automatically cleaned up
Message serialization cached where possible

Monitoring:

Structured logging for all broadcast operations
WebSocket status available at /ws/status endpoint
Connection count and room membership tracking
Error tracking for failed broadcasts

Frontend Authentication Integration (October 2025)

Completed JWT-based authentication integration between frontend and backend.

Authentication Token Storage

Files Modified:

src/server/web/templates/login.html - Store JWT token after successful login
src/server/web/templates/setup.html - Redirect to login after setup completion
src/server/web/static/js/app.js - Include Bearer token in all authenticated requests
src/server/web/static/js/queue.js - Include Bearer token in queue API calls

Implementation:

JWT tokens stored in localStorage after successful login
Token expiry stored in localStorage for client-side validation
Authorization: Bearer <token> header included in all authenticated requests
Automatic redirect to /login on 401 Unauthorized responses
Token cleared from localStorage on logout

Key Functions Updated:

makeAuthenticatedRequest() in both app.js and queue.js
checkAuthentication() to verify token and redirect if missing/invalid
logout() to clear token and redirect to login

Frontend API Endpoint Updates (October 2025)

Updated frontend JavaScript to match new backend API structure.

Queue Management API Changes:

/api/queue/clear → /api/queue/completed for clearing completed downloads
/api/queue/remove → /api/queue/{item_id} (DELETE) for single item removal
/api/queue/retry payload changed to {item_ids: []} array format
/api/download/pause → /api/queue/pause
/api/download/resume → /api/queue/resume
/api/download/cancel → /api/queue/stop

Response Format Changes:

Login returns {access_token, token_type, expires_at} instead of {status: 'success'}
Setup returns {status: 'ok'} instead of {status: 'success', redirect_url}
Logout returns {status: 'ok'} instead of {status: 'success'}
Queue operations return structured responses with counts (e.g., {cleared_count, retried_count})

Frontend WebSocket Integration (October 2025)

WebSocket integration previously completed and verified functional.

Native WebSocket Implementation

Files:

src/server/web/static/js/websocket_client.js - Native WebSocket wrapper
Templates already updated to use websocket_client.js instead of Socket.IO

Event Compatibility:

Dual event handlers in place for backward compatibility
Old events: scan_completed, scan_error, download_completed, download_error
New events: scan_complete, scan_failed, download_complete, download_failed
Both event types supported simultaneously

Room Subscriptions:

downloads - Download completion, failures, queue status
download_progress - Real-time download progress updates
scan_progress - Library scan progress updates

Frontend Integration Testing (October 2025)

Created smoke tests to verify frontend-backend integration.

Test File: tests/integration/test_frontend_integration_smoke.py

Tests:

JWT token format verification (access_token, token_type, expires_at)
Bearer token authentication on protected endpoints
401 responses for requests without valid tokens

Test Results:

Basic authentication flow: ✅ PASSING
Token validation: Functional with rate limiting considerations

Frontend Integration (October 2025)

Completed integration of existing frontend JavaScript with the new FastAPI backend and native WebSocket implementation.

Native WebSocket Client

File: src/server/web/static/js/websocket_client.js

Created a Socket.IO-compatible wrapper using native WebSocket API:

Features:

Socket.IO-style .on() and .emit() methods for compatibility
Automatic reconnection with exponential backoff (max 5 attempts)
Room-based subscriptions via .join() and .leave() methods
Message queueing during disconnection
Proper connection lifecycle management

Usage:

const socket = io();  // Creates WebSocket to ws://host:port/ws/connect
socket.join('download_progress');  // Subscribe to room
socket.on('download_progress', (data) => { ... });  // Handle messages

WebSocket Message Format

All WebSocket messages follow a structured format:

{
    "type": "message_type",
    "timestamp": "2025-10-17T12:34:56.789Z",
    "data": {}
}

Event Mapping (Old Socket.IO → New WebSocket):

scan_completed / scan_complete → Scan finished
scan_error / scan_failed → Scan error
download_completed / download_complete → Download finished
download_error / download_failed → Download error
queue_updated / queue_status → Queue state changes
queue_started, queue_stopped, queue_paused, queue_resumed → Queue control events

Rooms:

scan_progress - Library scan updates
download_progress - Real-time download progress
downloads - Download completion, failures, queue status

JavaScript Updates

app.js:

Added room subscriptions on WebSocket connect
Added dual event handlers for old and new message types
connected event handler for initial WebSocket confirmation
Handles both scan_complete and legacy scan_completed events
Handles both scan_failed and legacy scan_error events

queue.js:

Added room subscriptions on WebSocket connect
Added dual event handlers for backward compatibility
Handles both queue_status and legacy queue_updated events
Handles both download_complete and legacy download_completed events
Handles both download_failed and legacy download_error events
Added handlers for queue_started, queue_stopped, queue_paused, queue_resumed

Template Updates

Modified Templates:

src/server/web/templates/index.html - Replaced Socket.IO CDN with websocket_client.js
src/server/web/templates/queue.html - Replaced Socket.IO CDN with websocket_client.js

Benefits:

No external CDN dependency (Socket.IO)
Native browser WebSocket API (faster, smaller)
Full compatibility with existing JavaScript code
Proper integration with backend WebSocket service

API Router Registration

fastapi_app.py:

✅ Added anime_router import and registration
All routers now properly included:
- health_router - Health checks
- page_router - HTML pages
- auth_router - Authentication (JWT-based)
- anime_router - Anime management (NEW)
- download_router - Download queue
- websocket_router - WebSocket connection

Anime Endpoints:

GET /api/v1/anime - List anime with missing episodes
POST /api/v1/anime/rescan - Trigger library rescan
POST /api/v1/anime/search - Search for anime
GET /api/v1/anime/{anime_id} - Get anime details

Authentication Integration

JavaScript uses JWT tokens from localStorage for authenticated requests:

Token stored after successful login
Included in Authorization: Bearer <token> header
Automatic redirect to /login on 401 responses
Compatible with backend AuthMiddleware

Testing

Verified Functionality:

✅ WebSocket client initialization and connection
✅ Room subscriptions and message routing
✅ Event handler compatibility (old and new message types)
✅ Anime API endpoints (passed pytest tests)
✅ Download queue API endpoints (existing tests)
✅ Frontend integration tests (comprehensive)

Frontend Integration Test Suite: tests/frontend/test_existing_ui_integration.py

Coverage:

Authentication flow with JWT tokens
API endpoint compatibility (anime, download, config)
WebSocket real-time updates
Data format validation
Error handling (401, 400/422)
Multiple client broadcast scenarios

Test Classes:

TestFrontendAuthentication: JWT login, logout, auth status
TestFrontendAnimeAPI: Anime list, search, rescan operations
TestFrontendDownloadAPI: Queue management, start/pause/stop
TestFrontendWebSocketIntegration: Connection, broadcasts, progress
TestFrontendConfigAPI: Configuration get/update
TestFrontendJavaScriptIntegration: Bearer token patterns
TestFrontendErrorHandling: JSON errors, validation
TestFrontendRealTimeUpdates: Download events, notifications
TestFrontendDataFormats: Response format validation

Test Commands:

# Run all frontend integration tests
conda run -n AniWorld python -m pytest tests/frontend/test_existing_ui_integration.py -v

# Run specific test class
conda run -n AniWorld python -m pytest tests/frontend/test_existing_ui_integration.py::TestFrontendAuthentication -v

# Run all API tests
conda run -n AniWorld python -m pytest tests/api/ -v

# Run all tests
conda run -n AniWorld python -m pytest tests/ -v

Note: Some tests require auth service state isolation. The test suite uses fixtures to reset authentication state before each test. If you encounter auth-related test failures, they may be due to shared state across test runs.

Known Limitations

Legacy Events: Some Socket.IO events don't have backend implementations:

scheduled_rescan_* events
auto_download_* events
download_episode_update event
download_series_completed event

Solution: These events are kept in JavaScript for future implementation or can be removed if not needed.

Configuration Endpoints: Many config-related features in app.js don't have backend endpoints:

Scheduler configuration
Logging configuration
Advanced configuration
Config backups

Solution: These can be implemented later or the UI features removed.

Documentation

Detailed Documentation: See FRONTEND_INTEGRATION.md for:

Complete API endpoint mapping
WebSocket message format details
Migration guide for developers
Testing strategies
Integration patterns

81 KiB Raw Blame History

Aniworld Web Application Infrastructure

Project Structure

Technology Stack

Backend

Frontend

Security

Authentication Models & Sessions

Configuration

Data Storage

API Endpoints

Authentication

Configuration

Configuration API Notes

Scheduler

Anime Management

Download Management

WebSocket

Logging

Log Levels

Log Files

Security Considerations

Authentication Service

Database Layer (October 2025)

Architecture

Database Models

AnimeSeries

Episode

DownloadQueueItem

UserSession

Mixins

TimestampMixin

SoftDeleteMixin

Database Connection Management

Initialization

Session Management

Database Configuration

SQLite Optimizations

FastAPI Integration

Testing

Migration Strategy (Future)

Production Considerations

Integration with Services

Benefits of Database Layer

Database Service Layer (October 2025)

Core Application Logic

SeriesApp - Enhanced Core Engine

Key Features

Core Classes

Key Methods

Integration Points

Progress Callback System

Overview

Architecture

Components

Callback Interfaces (src/core/interfaces/callbacks.py)

Context Classes

Enums

Callback Manager

Integration

SerieScanner

SeriesApp

Usage Example

Recent Infrastructure Changes

Progress Callback System (October 2025)

Changes Made

Core Logic Enhancement (October 2025)

Changes Made

Test Coverage

AnimeService Identifier Standardization (November 2025)

Changes Made

Implementation Status

Template Integration (October 2025)

Changes Made

Template Features

Verified Templates

CSS Integration (October 2025)

Implementation Details

CSS Variables

Template CSS References

81 KiB

Raw Blame History

Callback Interfaces (`src/core/interfaces/callbacks.py`)