lukas.pupkalipinski/Aniworld
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Aniworld/infrastructure.md
Lukas 727486795c fix download
2025-10-30 21:13:08 +01:00

2175 lines
76 KiB
Markdown
Raw Blame History

# Aniworld Web Application Infrastructure
conda activate AniWorld
## Project Structure
```
/home/lukas/Volume/repo/Aniworld/
├── src/
│ ├── core/ # Core application logic
│ │ ├── SeriesApp.py # Main application class with async support
│ │ ├── SerieScanner.py # Directory scanner for anime series
│ │ ├── entities/ # Domain entities
│ │ │ ├── series.py # Serie data model
│ │ │ └── SerieList.py # Series list management
│ │ ├── interfaces/ # Abstract interfaces
│ │ │ └── providers.py # Provider interface definitions
│ │ ├── providers/ # Content providers
│ │ │ ├── base_provider.py # Base loader interface
│ │ │ ├── aniworld_provider.py # Aniworld.to implementation
│ │ │ ├── provider_factory.py # Provider factory
│ │ │ ├── provider_config.py # Provider configuration
│ │ │ ├── health_monitor.py # Provider health monitoring
│ │ │ ├── failover.py # Provider failover system
│ │ │ ├── monitored_provider.py # Performance tracking wrapper
│ │ │ ├── config_manager.py # Dynamic configuration mgmt
│ │ │ └── streaming/ # Streaming providers (VOE, etc.)
│ │ └── exceptions/ # Custom exceptions
│ │ └── Exceptions.py # Exception definitions
│ ├── server/ # FastAPI web application
│ │ ├── fastapi_app.py # Main FastAPI application (simplified)
│ │ ├── main.py # FastAPI application entry point
│ │ ├── controllers/ # Route controllers
│ │ │ ├── __init__.py # Controllers package
│ │ │ ├── health_controller.py # Health check endpoints
│ │ │ ├── page_controller.py # HTML page routes
│ │ │ └── error_controller.py # Error handling controllers
│ │ ├── api/ # API route handlers
│ │ │ ├── __init__.py
│ │ │ ├── auth.py # Authentication endpoints
│ │ │ ├── config.py # Configuration endpoints
│ │ │ ├── anime.py # Anime management endpoints
│ │ │ ├── download.py # Download queue endpoints
│ │ │ ├── scheduler.py # Scheduler configuration endpoints
│ │ │ └── websocket.py # WebSocket real-time endpoints
│ │ ├── models/ # Pydantic models
│ │ │ ├── __init__.py
│ │ │ ├── auth.py
│ │ │ ├── config.py
│ │ │ ├── anime.py
│ │ │ ├── download.py
│ │ │ └── websocket.py # WebSocket message models
│ │ ├── services/ # Business logic services
│ │ │ ├── __init__.py
│ │ │ ├── auth_service.py
│ │ │ ├── config_service.py
│ │ │ ├── anime_service.py
│ │ │ ├── download_service.py
│ │ │ ├── websocket_service.py # WebSocket connection management
│ │ │ ├── progress_service.py # Progress tracking
│ │ │ ├── notification_service.py # Notification system
│ │ │ └── cache_service.py # Caching layer
│ │ ├── database/ # Database layer
│ │ │ ├── __init__.py # Database package
│ │ │ ├── base.py # Base models and mixins
│ │ │ ├── models.py # SQLAlchemy ORM models
│ │ │ └── connection.py # Database connection management
│ │ ├── utils/ # Utility functions
│ │ │ ├── __init__.py
│ │ │ ├── security.py
│ │ │ ├── dependencies.py # Dependency injection
│ │ │ ├── templates.py # Shared Jinja2 template config
│ │ │ ├── template_helpers.py # Template rendering utilities
│ │ │ └── logging.py # Logging utilities
│ │ └── web/ # Frontend assets
│ │ ├── templates/ # Jinja2 HTML templates
│ │ │ ├── index.html # Main application page
│ │ │ ├── login.html # Login page
│ │ │ ├── setup.html # Initial setup page
│ │ │ ├── queue.html # Download queue page
│ │ │ └── error.html # Error page
│ │ └── static/ # Static web assets
│ │ ├── css/
│ │ │ ├── styles.css # Main styles
│ │ │ └── ux_features.css # UX enhancements
│ │ └── js/
│ │ ├── app.js # Main application logic
│ │ ├── queue.js # Queue management
│ │ ├── localization.js # i18n support
│ │ ├── keyboard_shortcuts.js # Keyboard navigation
│ │ ├── user_preferences.js # User settings
│ │ ├── undo_redo.js # Undo/redo system
│ │ ├── mobile_responsive.js # Mobile support
│ │ ├── touch_gestures.js # Touch interactions
│ │ ├── accessibility_features.js # A11y features
│ │ ├── screen_reader_support.js # Screen reader
│ │ ├── color_contrast_compliance.js # WCAG compliance
│ │ ├── multi_screen_support.js # Multi-monitor
│ │ ├── drag_drop.js # Drag and drop
│ │ ├── bulk_operations.js # Bulk actions
│ │ └── advanced_search.js # Search filters
│ ├── core/ # Existing core functionality
│ └── cli/ # Existing CLI application
├── data/ # Application data storage
│ ├── config.json # Application configuration
│ ├── anime_library.db # SQLite database for anime library
│ ├── download_queue.json # Download queue state
│ └── cache/ # Temporary cache files
├── logs/ # Application logs
│ ├── app.log # Main application log
│ ├── download.log # Download-specific logs
│ └── error.log # Error logs
├── requirements.txt # Python dependencies
├── docker-compose.yml # Docker deployment configuration
└── README.md
```
## Technology Stack
### Backend
- **FastAPI**: Modern Python web framework for building APIs
- **Uvicorn**: ASGI server for running FastAPI applications
- **SQLAlchemy**: SQL toolkit and ORM for database operations
- **SQLite**: Lightweight database for storing anime library and configuration
- **Alembic**: Database migration tool for schema management
- **Pydantic**: Data validation and serialization
- **Jinja2**: Template engine for server-side rendering
### Frontend
- **HTML5/CSS3**: Core web technologies
- **JavaScript (Vanilla)**: Client-side interactivity
- **Bootstrap 5**: CSS framework for responsive design
- **HTMX**: Modern approach for dynamic web applications
### Security
- **Passlib**: Password hashing and verification
- **python-jose**: JWT token handling
- **bcrypt**: Secure password hashing
### Authentication Models & Sessions
- Authentication request/response Pydantic models live in `src/server/models/auth.py`.
- Sessions are represented by `SessionModel` and can be backed by an in-memory
store or a persistent table depending on deployment needs. JWTs are used for
stateless authentication by default; a persistent session store may be
configured in production to enable revocation and long-lived sessions.
## Configuration
### Data Storage
- **Configuration**: JSON files in `data/` directory
- **Anime Library**: SQLite database with series information
- **Download Queue**: JSON file with current download status
- **Logs**: Structured logging to files in `logs/` directory
## API Endpoints
### Authentication
- `POST /api/auth/login` - Master password authentication
- `POST /api/auth/logout` - Logout and invalidate session
- `GET /api/auth/status` - Check authentication status
### Configuration
- `GET /api/config` - Get current configuration
- `PUT /api/config` - Update configuration
- `POST /api/setup` - Initial setup
### Configuration API Notes
- Configuration endpoints are exposed under `/api/config`
- Uses file-based persistence with JSON format for human-readable storage
- Automatic backup creation before configuration updates
- Configuration validation with detailed error reporting
- Backup management with create, restore, list, and delete operations
- Configuration schema versioning with migration support
- Singleton ConfigService manages all persistence operations
- Default configuration location: `data/config.json`
- Backup directory: `data/config_backups/`
- Maximum backups retained: 10 (configurable)
- Automatic cleanup of old backups exceeding limit
**Key Endpoints:**
- `GET /api/config` - Retrieve current configuration
- `PUT /api/config` - Update configuration (creates backup)
- `POST /api/config/validate` - Validate without applying
- `GET /api/config/backups` - List all backups
- `POST /api/config/backups` - Create manual backup
- `POST /api/config/backups/{name}/restore` - Restore from backup
- `DELETE /api/config/backups/{name}` - Delete backup
- `GET /api/config/section/advanced` - Get advanced configuration section
- `POST /api/config/section/advanced` - Update advanced configuration
- `POST /api/config/directory` - Update anime directory
- `POST /api/config/export` - Export configuration to JSON file
- `POST /api/config/reset` - Reset configuration to defaults
**Configuration Service Features:**
- Atomic file writes using temporary files
- JSON format with version metadata
- Validation before saving
- Automatic backup on updates
- Migration support for schema changes
- Thread-safe singleton pattern
- Comprehensive error handling with custom exceptions
### Scheduler
- `GET /api/scheduler/config` - Get scheduler configuration
- `POST /api/scheduler/config` - Update scheduler configuration
- `POST /api/scheduler/trigger-rescan` - Manually trigger rescan
### Anime Management
- `GET /api/anime` - List anime with missing episodes
- `POST /api/anime/{id}/download` - Add episodes to download queue
- `GET /api/anime/{id}` - Get anime details
Note: The anime management API has been implemented under `/api/v1/anime` with
endpoints for listing series with missing episodes, searching providers,
triggering a local rescan, and fetching series details. The implementation
delegates to the existing core `SeriesApp` and uses dependency injection for
initialization.
### Download Management
- `GET /api/queue/status` - Get download queue status and statistics
- `POST /api/queue/add` - Add episodes to download queue
- `DELETE /api/queue/{id}` - Remove item from queue
- `DELETE /api/queue/` - Remove multiple items from queue
- `POST /api/queue/start` - Start download queue processing
- `POST /api/queue/stop` - Stop download queue processing
- `POST /api/queue/pause` - Pause queue processing
- `POST /api/queue/resume` - Resume queue processing
- `POST /api/queue/reorder` - Reorder pending queue items (bulk or single)
- `DELETE /api/queue/completed` - Clear completed downloads
- `DELETE /api/queue/failed` - Clear failed downloads
- `POST /api/queue/retry` - Retry failed downloads
**Queue Reordering:**
- Supports bulk reordering with `{"item_ids": ["id1", "id2", ...]}` payload
- Items are reordered in the exact order provided in the array
- Only affects pending (non-active) downloads
- Real-time drag-and-drop UI with visual feedback
**Queue Organization:**
- **Pending Queue**: Items waiting to be downloaded, displayed in order with drag handles
- **Active Downloads**: Currently downloading items with progress bars
- **Completed Downloads**: Successfully downloaded items with completion timestamps
- **Failed Downloads**: Failed items with error messages and retry options
**Queue Display Features:**
- Numbered position indicators for pending items
- Drag handle icons for visual reordering cues
- Real-time statistics counters (pending, active, completed, failed)
- Empty state messages with helpful hints
- Per-section action buttons (clear, retry all)
### WebSocket
- `WS /api/ws` - WebSocket connection for real-time updates
- Real-time download progress notifications
- Queue status updates
- System notifications
## Logging
### Log Levels
- **INFO**: General application information
- **WARNING**: Potential issues that don't stop execution
- **ERROR**: Errors that affect functionality
- **DEBUG**: Detailed debugging information (development only)
### Log Files
- `app.log`: General application logs
- `download.log`: Download-specific operations
- `error.log`: Error and exception logs
## Security Considerations
- Master password protection for application access
- Secure session management with JWT tokens
- Input validation and sanitization
- Built-in rate limiting in authentication middleware
- HTTPS enforcement in production
- Secure file path handling to prevent directory traversal
### Authentication Service
- A lightweight authentication service is provided by
`src/server/services/auth_service.py`.
- Uses bcrypt (passlib) to hash the master password and issues JWTs for
stateless sessions. Tokens are signed with the `JWT_SECRET_KEY` from
configuration and expire based on `SESSION_TIMEOUT_HOURS`.
- Failed login attempts are tracked in-memory and a temporary lockout is
applied after multiple failures. For multi-process deployments, move
this state to a shared store (Redis) and persist the master password
hash in a secure config store.
## Database Layer (October 2025)
A comprehensive SQLAlchemy-based database layer was implemented to provide
persistent storage for anime series, episodes, download queue, and user sessions.
### Architecture
**Location**: `src/server/database/`
**Components**:
- `base.py`: Base declarative class and mixins (TimestampMixin, SoftDeleteMixin)
- `models.py`: SQLAlchemy ORM models with relationships
- `connection.py`: Database engine, session factory, and dependency injection
- `__init__.py`: Package exports and public API
### Database Models
#### AnimeSeries
Represents anime series with metadata and provider information.
**Fields**:
- `id` (PK): Auto-incrementing primary key
- `key`: Unique provider identifier (indexed)
- `name`: Series name (indexed)
- `site`: Provider site URL
- `folder`: Local filesystem path
- `description`: Optional series description
- `status`: Series status (ongoing, completed)
- `total_episodes`: Total episode count
- `cover_url`: Cover image URL
- `episode_dict`: JSON field storing episode structure {season: [episodes]}
- `created_at`, `updated_at`: Audit timestamps (from TimestampMixin)
**Relationships**:
- `episodes`: One-to-many with Episode (cascade delete)
- `download_items`: One-to-many with DownloadQueueItem (cascade delete)
#### Episode
Individual episodes linked to anime series.
**Fields**:
- `id` (PK): Auto-incrementing primary key
- `series_id` (FK): Foreign key to AnimeSeries (indexed)
- `season`: Season number
- `episode_number`: Episode number within season
- `title`: Optional episode title
- `file_path`: Local file path if downloaded
- `file_size`: File size in bytes
- `is_downloaded`: Boolean download status
- `download_date`: Timestamp when downloaded
- `created_at`, `updated_at`: Audit timestamps
**Relationships**:
- `series`: Many-to-one with AnimeSeries
#### DownloadQueueItem
Download queue with status and progress tracking.
**Fields**:
- `id` (PK): Auto-incrementing primary key
- `series_id` (FK): Foreign key to AnimeSeries (indexed)
- `season`: Season number
- `episode_number`: Episode number
- `status`: Download status enum (indexed)
- Values: PENDING, DOWNLOADING, PAUSED, COMPLETED, FAILED, CANCELLED
- `priority`: Priority enum
- Values: LOW, NORMAL, HIGH
- `progress_percent`: Download progress (0-100)
- `downloaded_bytes`: Bytes downloaded
- `total_bytes`: Total file size
- `download_speed`: Current speed (bytes/sec)
- `error_message`: Error description if failed
- `retry_count`: Number of retry attempts
- `download_url`: Provider download URL
- `file_destination`: Target file path
- `started_at`: Download start timestamp
- `completed_at`: Download completion timestamp
- `created_at`, `updated_at`: Audit timestamps
**Relationships**:
- `series`: Many-to-one with AnimeSeries
#### UserSession
User authentication sessions with JWT tokens.
**Fields**:
- `id` (PK): Auto-incrementing primary key
- `session_id`: Unique session identifier (indexed)
- `token_hash`: Hashed JWT token
- `user_id`: User identifier (indexed, for multi-user support)
- `ip_address`: Client IP address
- `user_agent`: Client user agent string
- `expires_at`: Session expiration timestamp
- `is_active`: Boolean active status (indexed)
- `last_activity`: Last activity timestamp
- `created_at`, `updated_at`: Audit timestamps
**Methods**:
- `is_expired`: Property to check if session has expired
- `revoke()`: Revoke session by setting is_active=False
### Mixins
#### TimestampMixin
Adds automatic timestamp tracking to models.
**Fields**:
- `created_at`: Automatically set on record creation
- `updated_at`: Automatically updated on record modification
**Usage**: Inherit in models requiring audit timestamps.
#### SoftDeleteMixin
Provides soft delete functionality (logical deletion).
**Fields**:
- `deleted_at`: Timestamp when soft deleted (NULL if active)
**Properties**:
- `is_deleted`: Check if record is soft deleted
**Methods**:
- `soft_delete()`: Mark record as deleted
- `restore()`: Restore soft deleted record
**Note**: Currently not used by models but available for future implementation.
### Database Connection Management
#### Initialization
```python
from src.server.database import init_db, close_db
# Application startup
await init_db() # Creates engine, session factory, and tables
# Application shutdown
await close_db() # Closes connections and cleanup
```
#### Session Management
**Async Sessions** (preferred for FastAPI endpoints):
```python
from fastapi import Depends
from sqlalchemy.ext.asyncio import AsyncSession
from src.server.database import get_db_session
@app.get("/anime")
async def get_anime(db: AsyncSession = Depends(get_db_session)):
result = await db.execute(select(AnimeSeries))
return result.scalars().all()
```
**Sync Sessions** (for non-async operations):
```python
from src.server.database.connection import get_sync_session
session = get_sync_session()
try:
result = session.execute(select(AnimeSeries))
return result.scalars().all()
finally:
session.close()
```
### Database Configuration
**Settings** (from `src/config/settings.py`):
- `DATABASE_URL`: Database connection string
- Default: `sqlite:///./data/aniworld.db`
- Automatically converted to `sqlite+aiosqlite:///` for async support
- `LOG_LEVEL`: When set to "DEBUG", enables SQL query logging
**Engine Configuration**:
- **SQLite**: Uses StaticPool, enables foreign keys and WAL mode
- **PostgreSQL/MySQL**: Uses QueuePool with pre-ping health checks
- **Connection Pooling**: Configured based on database type
- **Echo**: SQL query logging in DEBUG mode
### SQLite Optimizations
- **Foreign Keys**: Automatically enabled via PRAGMA
- **WAL Mode**: Write-Ahead Logging for better concurrency
- **Static Pool**: Single connection pool for SQLite
- **Async Support**: aiosqlite driver for async operations
### FastAPI Integration
**Dependency Injection** (in `src/server/utils/dependencies.py`):
```python
async def get_database_session() -> AsyncGenerator:
"""Dependency to get database session."""
try:
from src.server.database import get_db_session
async with get_db_session() as session:
yield session
except ImportError:
raise HTTPException(status_code=501, detail="Database not installed")
except RuntimeError as e:
raise HTTPException(status_code=503, detail=f"Database not available: {str(e)}")
```
**Usage in Endpoints**:
```python
from fastapi import Depends
from sqlalchemy.ext.asyncio import AsyncSession
from src.server.utils.dependencies import get_database_session
@router.get("/series/{series_id}")
async def get_series(
series_id: int,
db: AsyncSession = Depends(get_database_session)
):
result = await db.execute(
select(AnimeSeries).where(AnimeSeries.id == series_id)
)
series = result.scalar_one_or_none()
if not series:
raise HTTPException(status_code=404, detail="Series not found")
return series
```
### Testing
**Test Suite**: `tests/unit/test_database_models.py`
**Coverage**:
- 30+ comprehensive test cases
- Model creation and validation
- Relationship testing (one-to-many, cascade deletes)
- Unique constraint validation
- Query operations (filtering, joins)
- Session management
- Mixin functionality
**Test Strategy**:
- In-memory SQLite database for isolation
- Fixtures for engine and session setup
- Test all CRUD operations
- Verify constraints and relationships
- Test edge cases and error conditions
### Migration Strategy (Future)
**Alembic Integration** (planned):
- Alembic installed but not yet configured
- Will manage schema migrations in production
- Auto-generate migrations from model changes
- Version control for database schema
**Initial Setup**:
```bash
# Initialize Alembic (future)
alembic init alembic
# Generate initial migration
alembic revision --autogenerate -m "Initial schema"
# Apply migrations
alembic upgrade head
```
### Production Considerations
**Single-Process Deployment** (current):
- SQLite with WAL mode for concurrency
- Static pool for single connection
- File-based storage at `data/aniworld.db`
**Multi-Process Deployment** (future):
- Switch to PostgreSQL or MySQL
- Configure connection pooling (pool_size, max_overflow)
- Use QueuePool for connection management
- Consider read replicas for scaling
**Performance**:
- Indexes on frequently queried columns (key, name, status, is_active)
- Foreign key constraints for referential integrity
- Cascade deletes for cleanup operations
- Efficient joins via relationship loading strategies
**Monitoring**:
- SQL query logging in DEBUG mode
- Connection pool metrics (when using QueuePool)
- Query performance profiling
- Database size monitoring
**Backup Strategy**:
- SQLite: File-based backups (copy `aniworld.db` file)
- WAL checkpoint before backup
- Automated backup schedule recommended
- Store backups in `data/config_backups/` or separate location
### Integration with Services
**AnimeService**:
- Query series from database
- Persist scan results
- Update episode metadata
**DownloadService**:
- Load queue from database on startup
- Persist queue state continuously
- Update download progress in real-time
**AuthService**:
- Store and validate user sessions
- Session revocation via database
- Query active sessions for monitoring
### Benefits of Database Layer
- **Persistence**: Survives application restarts
- **Relationships**: Enforced referential integrity
- **Queries**: Powerful filtering and aggregation
- **Scalability**: Can migrate to PostgreSQL/MySQL
- **ACID**: Atomic transactions for consistency
- **Migration**: Schema versioning with Alembic
- **Testing**: Easy to test with in-memory database
### Database Service Layer (October 2025)
Implemented comprehensive service layer for database CRUD operations.
**File**: `src/server/database/service.py`
**Services**:
- `AnimeSeriesService`: CRUD operations for anime series
- `EpisodeService`: Episode management and download tracking
- `DownloadQueueService`: Queue management with priority and status
- `UserSessionService`: Session management and authentication
**Key Features**:
- Repository pattern for clean separation of concerns
- Type-safe operations with comprehensive type hints
- Async support for all database operations
- Transaction management via FastAPI dependency injection
- Comprehensive error handling and logging
- Search and filtering capabilities
- Pagination support for large datasets
- Batch operations for performance
**AnimeSeriesService Operations**:
- Create series with metadata and provider information
- Retrieve by ID, key, or search query
- Update series attributes
- Delete series with cascade to episodes and queue items
- List all series with pagination and eager loading options
**EpisodeService Operations**:
- Create episodes for series
- Retrieve episodes by series, season, or specific episode
- Mark episodes as downloaded with file metadata
- Delete episodes
**DownloadQueueService Operations**:
- Add items to queue with priority levels (LOW, NORMAL, HIGH)
- Retrieve pending, active, or all queue items
- Update download status (PENDING, DOWNLOADING, COMPLETED, FAILED, etc.)
- Update download progress (percentage, bytes, speed)
- Clear completed downloads
- Retry failed downloads with max retry limits
- Automatic timestamp management (started_at, completed_at)
**UserSessionService Operations**:
- Create authentication sessions with JWT tokens
- Retrieve sessions by session ID
- Get active sessions with expiry checking
- Update last activity timestamp
- Revoke sessions for logout
- Cleanup expired sessions automatically
**Testing**:
- Comprehensive test suite with 22 test cases
- In-memory SQLite for isolated testing
- All CRUD operations tested
- Edge cases and error conditions covered
- 100% test pass rate
**Integration**:
- Exported via database package `__init__.py`
- Used by API endpoints via dependency injection
- Compatible with existing database models
- Follows project coding standards (PEP 8, type hints, docstrings)
**Database Migrations** (`src/server/database/migrations.py`):
- Simple schema initialization via SQLAlchemy create_all
- Schema version checking utility
- Documentation for Alembic integration
- Production-ready migration strategy outlined
## Core Application Logic
### SeriesApp - Enhanced Core Engine
The `SeriesApp` class (`src/core/SeriesApp.py`) is the main application engine for anime series management. Enhanced with async support and web integration capabilities.
#### Key Features
- **Async Operations**: Support for async download and scan operations
- **Progress Callbacks**: Real-time progress reporting via callbacks
- **Cancellation Support**: Ability to cancel long-running operations
- **Error Handling**: Comprehensive error handling with callback notifications
- **Operation Status**: Track current operation status and history
#### Core Classes
- `SeriesApp`: Main application class
- `OperationStatus`: Enum for operation states (IDLE, RUNNING, COMPLETED, CANCELLED, FAILED)
- `ProgressInfo`: Dataclass for progress information
- `OperationResult`: Dataclass for operation results
#### Key Methods
- `search(words)`: Search for anime series
- `download()`: Download episodes with progress tracking
- `ReScan()`: Scan directory for missing episodes
- `async_download()`: Async version of download
- `async_rescan()`: Async version of rescan
- `cancel_operation()`: Cancel current operation
- `get_operation_status()`: Get current status
- `get_series_list()`: Get series with missing episodes
#### Integration Points
The SeriesApp integrates with:
- Provider system for content downloading
- Serie scanner for directory analysis
- Series list management for tracking missing episodes
- Web layer via async operations and callbacks
## Progress Callback System
### Overview
A comprehensive callback system for real-time progress reporting, error handling, and operation completion notifications across core operations (scanning, downloading, searching).
### Architecture
- **Interface-based Design**: Abstract base classes define callback contracts
- **Context Objects**: Rich context information for each callback type
- **Callback Manager**: Centralized management of multiple callbacks
- **Thread-safe**: Exception handling prevents callback errors from breaking operations
### Components
#### Callback Interfaces (`src/core/interfaces/callbacks.py`)
- `ProgressCallback`: Reports operation progress updates
- `ErrorCallback`: Handles error notifications
- `CompletionCallback`: Notifies operation completion
#### Context Classes
- `ProgressContext`: Current progress, percentage, phase, and metadata
- `ErrorContext`: Error details, recoverability, retry information
- `CompletionContext`: Success status, results, and statistics
#### Enums
- `OperationType`: SCAN, DOWNLOAD, SEARCH, INITIALIZATION
- `ProgressPhase`: STARTING, IN_PROGRESS, COMPLETING, COMPLETED, FAILED, CANCELLED
#### Callback Manager
- Register/unregister multiple callbacks per type
- Notify all registered callbacks with context
- Exception handling for callback errors
- Support for clearing all callbacks
### Integration
#### SerieScanner
- Reports scanning progress (folder by folder)
- Notifies errors for failed folder scans
- Reports completion with statistics
#### SeriesApp
- Download progress reporting with percentage
- Scan progress through SerieScanner integration
- Error notifications for all operations
- Completion notifications with results
### Usage Example
```python
from src.core.interfaces.callbacks import (
CallbackManager,
ProgressCallback,
ProgressContext
)
class MyProgressCallback(ProgressCallback):
def on_progress(self, context: ProgressContext):
print(f"{context.message}: {context.percentage:.1f}%")
# Register callback
manager = CallbackManager()
manager.register_progress_callback(MyProgressCallback())
# Use with SeriesApp
app = SeriesApp(directory, callback_manager=manager)
```
## Recent Infrastructure Changes
### Progress Callback System (October 2025)
Implemented a comprehensive progress callback system for real-time operation tracking.
#### Changes Made
1. **Callback Interfaces**:
- Created abstract base classes for progress, error, and completion callbacks
- Defined rich context objects with operation metadata
- Implemented thread-safe callback manager
2. **SerieScanner Integration**:
- Added progress reporting for directory scanning
- Implemented per-folder progress updates
- Error callbacks for scan failures
- Completion notifications with statistics
3. **SeriesApp Integration**:
- Integrated callback manager into download operations
- Progress updates during episode downloads
- Error handling with callback notifications
- Completion callbacks for all operations
- Backward compatibility with legacy callbacks
4. **Testing**:
- 22 comprehensive unit tests
- Coverage for all callback types
- Exception handling verification
- Multiple callback registration tests
### Core Logic Enhancement (October 2025)
Enhanced `SeriesApp` with async callback support, progress reporting, and cancellation.
#### Changes Made
1. **Async Support**:
- Added `async_download()` and `async_rescan()` methods
- Integrated with asyncio event loop for non-blocking operations
- Support for concurrent operations in web environment
2. **Progress Reporting**:
- Legacy `ProgressInfo` dataclass for structured progress data
- New comprehensive callback system with context objects
- Percentage calculation and status tracking
3. **Cancellation System**:
- Internal cancellation flag management
- Graceful operation cancellation
- Cancellation check during long-running operations
4. **Error Handling**:
- `OperationResult` dataclass for operation outcomes
- Error callback system for notifications
- Specific exception types (IOError, OSError, RuntimeError)
- Proper exception propagation and logging
5. **Status Management**:
- `OperationStatus` enum for state tracking
- Current operation identifier
- Status getter methods for monitoring
#### Test Coverage
Comprehensive test suite (`tests/unit/test_series_app.py`) with 22 tests covering:
- Initialization and configuration
- Search functionality
- Download operations with callbacks
- Directory scanning with progress
- Async operations
- Cancellation handling
- Error scenarios
- Data model validation
### Template Integration (October 2025)
Completed integration of HTML templates with FastAPI Jinja2 system.
#### Changes Made
1. **Template Helper Utilities**:
- `src/server/utils/template_helpers.py` - Template rendering utilities
- Centralized base context for all templates
- Template validation and listing functions
- DRY principles for template rendering
2. **Enhanced CSS**:
- `src/server/web/static/css/ux_features.css` - UX enhancement styles
- Drag-and-drop indicators
- Bulk selection styling
- Keyboard navigation focus indicators
- Touch gesture feedback
- Mobile responsive enhancements
- Accessibility features (high contrast, screen reader support)
- Reduced motion support
3. **JavaScript Modules**:
- `keyboard_shortcuts.js` - Keyboard navigation (Ctrl+K, Ctrl+R, etc.)
- `user_preferences.js` - Settings persistence (localStorage)
- `undo_redo.js` - Action history with Ctrl+Z/Ctrl+Y
- `mobile_responsive.js` - Mobile detection and layout
- `touch_gestures.js` - Swipe gesture handling
- `accessibility_features.js` - Focus management and ARIA labels
- `screen_reader_support.js` - Live regions for dynamic content
- `color_contrast_compliance.js` - WCAG compliance checks
- `multi_screen_support.js` - Fullscreen and multi-monitor support
- `drag_drop.js` - Drag-and-drop functionality (stub)
- `bulk_operations.js` - Bulk selection and actions (stub)
- `advanced_search.js` - Advanced filtering (stub)
4. **Updated Controllers**:
- Updated `page_controller.py` to use `template_helpers`
- Updated `error_controller.py` to use `template_helpers`
- Consistent context passing across all templates
#### Template Features
- **Responsive Design**: Mobile-first approach with viewport meta tags
- **Theme Switching**: Light/dark mode with `data-theme` attribute
- **Accessibility**: ARIA labels, keyboard navigation, screen reader support
- **Internationalization**: Localization support via `localization.js`
- **Progressive Enhancement**: Works without JavaScript, enhanced with it
#### Verified Templates
All HTML templates properly integrated:
- `index.html` - Main application page with search and anime list
- `login.html` - Master password authentication
- `setup.html` - Initial application setup
- `queue.html` - Download queue management
- `error.html` - Error pages (404, 500)
All templates include:
- Proper HTML5 structure
- Font Awesome icons
- Static file references (`/static/css/`, `/static/js/`)
- Theme switching support
- Responsive viewport configuration
### CSS Integration (October 2025)
Integrated existing CSS styling with FastAPI's static file serving system.
#### Implementation Details
1. **Static File Configuration**:
- Static files mounted at `/static` in `fastapi_app.py`
- Directory: `src/server/web/static/`
- Files served using FastAPI's `StaticFiles` middleware
- All paths use absolute references (`/static/...`)
2. **CSS Architecture**:
- `styles.css` (1,840 lines) - Main stylesheet with Fluent UI design system
- `ux_features.css` (203 lines) - Enhanced UX features and accessibility
3. **Design System** (`styles.css`):
- **Fluent UI Variables**: CSS custom properties for consistent theming
- **Light/Dark Themes**: Dynamic theme switching via `[data-theme="dark"]`
- **Typography**: Segoe UI font stack with responsive sizing
- **Spacing System**: Consistent spacing scale (xs through xxl)
- **Color Palette**: Comprehensive color system for both themes
- **Border Radius**: Standardized corner radii (sm, md, lg, xl)
- **Shadows**: Elevation system with card and elevated variants
- **Transitions**: Smooth animations with consistent timing
4. **UX Features** (`ux_features.css`):
- Drag-and-drop indicators
- Bulk selection styling
- Keyboard focus indicators
- Touch gesture feedback
- Mobile responsive utilities
- High contrast mode support (`@media (prefers-contrast: high)`)
- Screen reader utilities (`.sr-only`)
- Window control components
#### CSS Variables
**Color System**:
```css
/* Light Theme */
--color-bg-primary: #ffffff
--color-accent: #0078d4
--color-text-primary: #323130
/* Dark Theme */
--color-bg-primary-dark: #202020
--color-accent-dark: #60cdff
--color-text-primary-dark: #ffffff
```
**Spacing & Typography**:
```css
--spacing-sm: 8px
--spacing-md: 12px
--spacing-lg: 16px
--font-size-body: 14px
--font-size-title: 20px
```
#### Template CSS References
All HTML templates correctly reference CSS files:
- Index page: Includes both `styles.css` and `ux_features.css`
- Other pages: Include `styles.css`
- All use absolute paths: `/static/css/styles.css`
#### Responsive Design
- Mobile-first approach with breakpoints
- Media queries for tablet and desktop layouts
- Touch-friendly interface elements
- Adaptive typography and spacing
#### Accessibility Features
- WCAG-compliant color contrast
- High contrast mode support
- Screen reader utilities
- Keyboard navigation styling
- Focus indicators
- Reduced motion support
#### Testing
Comprehensive test suite in `tests/unit/test_static_files.py`:
- CSS file accessibility tests
- Theme support verification
- Responsive design validation
- Accessibility feature checks
- Content integrity validation
- Path correctness verification
All 17 CSS integration tests passing.
### Route Controller Refactoring (October 2025)
Restructured the FastAPI application to use a controller-based architecture for better code organization and maintainability.
#### Changes Made
1. **Created Controller Structure**:
- `src/server/controllers/` - New directory for route controllers
- `src/server/controllers/__init__.py` - Controllers package initialization
- `src/server/controllers/health_controller.py` - Health check endpoints
- `src/server/controllers/page_controller.py` - HTML page routes
- `src/server/controllers/error_controller.py` - Error handling controllers
2. **Shared Template Configuration**:
- `src/server/utils/templates.py` - Centralized Jinja2 template configuration
- Fixed template path resolution for proper template loading
3. **Main Application Updates**:
- `src/server/fastapi_app.py` - Refactored to use controller routers
- Removed direct route definitions from main file
- Added router inclusion using `app.include_router()`
- Simplified error handlers to delegate to controller functions
4. **Fixed Import Issues**:
- Resolved circular import in `src/core/__init__.py`
- Removed non-existent `application` module import
#### Controller Architecture
### Anime Service Notes
- The new `anime_service` runs the existing blocking `SeriesApp` inside a
threadpool (via ThreadPoolExecutor). This keeps the FastAPI event loop
responsive while leveraging the existing core logic.
- A small in-process LRU cache is used for the frequently-read "missing
episodes" list to reduce IO; cache invalidation happens after a rescan.
- For multi-worker or multi-host deployments, move cache/state to a shared
store (Redis) and ensure the threadpool sizing matches the worker's CPU
and IO profile.
**Health Controller** (`health_controller.py`):
```python
router = APIRouter(prefix="/health", tags=["health"])
@router.get("") - Health check endpoint
```
**Page Controller** (`page_controller.py`):
```python
router = APIRouter(tags=["pages"])
@router.get("/") - Main application page
@router.get("/setup") - Setup page
@router.get("/login") - Login page
@router.get("/queue") - Download queue page
```
**Error Controller** (`error_controller.py`):
```python
async def not_found_handler() - Custom 404 error handling
async def server_error_handler() - Custom 500 error handling
```
#### Benefits of the New Structure
- **Separation of Concerns**: Each controller handles specific functionality
- **Modularity**: Easy to add new controllers and routes
- **Testability**: Controllers can be tested independently
- **Maintainability**: Cleaner code organization and easier debugging
- **Scalability**: Simple to extend with new features
#### Verified Working Endpoints
All endpoints tested and confirmed working:
- Health: `/health` → Returns `{"status": "healthy", ...}`
- Root: `/` → Serves main application page
- Setup: `/setup` → Serves setup page
- Auth API: `/api/auth/*` → Endpoints for setup, login, logout and status (JWT-based)
- Login: `/login` → Serves login page
- Queue: `/queue` → Serves download queue page
#### File Structure After Refactoring
```
src/server/
├── fastapi_app.py # Main FastAPI application (simplified)
├── controllers/ # NEW: Route controllers
│ ├── __init__.py # Controllers package
├── utils/
│ ├── dependencies.py # Existing dependency injection
│ └── templates.py # NEW: Shared Jinja2 template config
└── web/ # Existing frontend assets
├── templates/ # HTML templates
└── static/ # CSS, JS, images
```
### Authentication Middleware (October 2025)
An authentication middleware component was added to the FastAPI
application to centralize token parsing and provide lightweight
protection of authentication endpoints:
- `src/server/middleware/auth.py` implements:
- Bearer JWT parsing and session attachment to `request.state.session`
- A simple per-IP in-memory rate limiter applied to
`/api/auth/login` and `/api/auth/setup` (default 5 requests/minute)
Notes:
- This is intentionally simple and designed for single-process
deployments. For production use across multiple workers or hosts,
replace the in-memory limiter with a distributed store (e.g. Redis)
and add a persistent token revocation list if needed.
### API Models and Contracts
- Pydantic models living in `src/server/models/` define the canonical
API contracts used by FastAPI endpoints. These models are intentionally
lightweight and focused on serialization, validation, and OpenAPI
documentation generation.
- Keep models stable: changes to model shapes are breaking changes for
clients. Bump API versioning or provide migration layers when altering
public response fields.
- Infrastructure considerations: ensure the deployment environment has
required libraries (e.g., `pydantic`) installed and that schema
validation errors are logged to the centralized logging system. For
high-throughput routes, consider response model caching at the
application or reverse-proxy layer.
### WebSocket Real-time Communication (October 2025)
A comprehensive WebSocket infrastructure was implemented to provide real-time
updates for downloads, queue status, and system events:
- **File**: `src/server/services/websocket_service.py`
- **Models**: `src/server/models/websocket.py`
- **Endpoint**: `ws://host:port/ws/connect`
#### WebSocket Service Architecture
- **ConnectionManager**: Low-level connection lifecycle management
- Connection registry with unique connection IDs
- Room-based messaging for topic subscriptions
- Automatic connection cleanup and health monitoring
- Thread-safe operations with asyncio locks
- **WebSocketService**: High-level application messaging
- Convenient interface for broadcasting application events
- Pre-defined message types for downloads, queue, and system events
- Singleton pattern via `get_websocket_service()` factory
#### Supported Message Types
- **Download Events**: `download_progress`, `download_complete`, `download_failed`
- **Queue Events**: `queue_status`, `queue_started`, `queue_stopped`, `queue_paused`, `queue_resumed`
- **System Events**: `system_info`, `system_warning`, `system_error`
- **Connection**: `connected`, `ping`, `pong`, `error`
#### Room-Based Messaging
Clients can subscribe to specific topics (rooms) to receive targeted updates:
- `downloads` room: All download-related events
- Custom rooms: Can be added for specific features
#### Integration with Download Service
- Download service automatically broadcasts progress updates via WebSocket
- Broadcast callback registered during service initialization
- Updates sent to all clients subscribed to the `downloads` room
- No blocking of download operations (async broadcast)
#### Client Connection Flow
1. Client connects to `/ws/connect` endpoint
2. Server assigns unique connection ID and sends confirmation
3. Client joins rooms (e.g., `{"action": "join", "room": "downloads"}`)
4. Server broadcasts updates to subscribed rooms
5. Client disconnects (automatic cleanup)
#### Infrastructure Notes
- **Single-process**: Current implementation uses in-memory connection storage
- **Production**: For multi-worker/multi-host deployments:
- Move connection registry to Redis or similar shared store
- Implement pub/sub for cross-process message broadcasting
- Add connection persistence for recovery after restarts
- **Monitoring**: WebSocket status available at `/ws/status` endpoint
- **Security**: Optional authentication via JWT (user_id tracking)
- **Testing**: Comprehensive unit tests in `tests/unit/test_websocket_service.py`
### Download Queue Models
- Download queue models in `src/server/models/download.py` define the data
structures for the download queue system.
- Key models include:
- `DownloadItem`: Represents a single queued download with metadata,
progress tracking, and error information
- `QueueStatus`: Overall queue state with active, pending, completed,
and failed downloads
- `QueueStats`: Aggregated statistics for monitoring queue performance
- `DownloadProgress`: Real-time progress information (percent, speed,
ETA)
- `DownloadRequest`/`DownloadResponse`: API request/response contracts
- Models enforce validation constraints (e.g., positive episode numbers,
progress percentage 0-100, non-negative retry counts) and provide
clean JSON serialization for API endpoints and WebSocket updates.
### Download Queue Service
- The download service (`src/server/services/download_service.py`) manages
the complete lifecycle of anime episode downloads.
- Core features:
- **Priority-based Queue**: Items added with HIGH priority are processed
first, NORMAL and LOW follow in FIFO order
- **Concurrent Processing**: Configurable max concurrent downloads (default 2)
to optimize bandwidth usage
- **Persistence**: Queue state is automatically saved to
`data/download_queue.json` and recovered on service restart
- **Retry Logic**: Failed downloads are automatically retried up to a
configurable limit (default 3 attempts) with exponential backoff
- **Progress Tracking**: Real-time download progress with speed,
percentage, and ETA calculations
- **WebSocket Integration**: Broadcasts queue updates, progress, and
completion/failure events to connected clients
- Operations:
- `add_to_queue()`: Add episodes to download queue with priority
- `remove_from_queue()`: Cancel pending or active downloads
- `reorder_queue()`: Manually adjust queue order for pending items
- `pause_queue()`/`resume_queue()`: Control download processing
- `retry_failed()`: Retry failed downloads with retry count checks
- `get_queue_status()`: Get complete queue state (active, pending, completed, failed)
- `get_queue_stats()`: Get aggregated statistics (counts, download size, speed)
- Infrastructure notes:
- Service uses ThreadPoolExecutor for concurrent download processing
- Queue processor runs as async background task with configurable sleep intervals
- Progress callbacks are executed in threadpool and broadcast via async WebSocket
- For multi-process deployments, move queue state to shared store (Redis/DB)
and implement distributed locking for concurrent access control
- Singleton instance pattern used via `get_download_service()` factory
- Testing: Comprehensive unit tests in `tests/unit/test_download_service.py`
cover queue operations, persistence, retry logic, and error handling
### Download Queue API Endpoints (October 2025)
Implemented comprehensive REST API endpoints for download queue management:
- **File**: `src/server/api/download.py`
- **Router Prefix**: `/api/queue`
- **Authentication**: All endpoints require JWT authentication via `require_auth` dependency
#### Implemented Endpoints
1. **GET /api/queue/status** - Retrieve complete queue status
- Returns: `QueueStatusResponse` with status and statistics
- Includes: active downloads, pending items, completed/failed items, queue stats
2. **POST /api/queue/add** - Add episodes to download queue
- Request: `DownloadRequest` with serie info, episodes, and priority
- Returns: `DownloadResponse` with added item IDs
- Validates episode list is non-empty
- Supports HIGH, NORMAL, and LOW priority levels
3. **DELETE /api/queue/{item_id}** - Remove single item from queue
- Returns: 204 No Content on success, 404 if item not found
- Cancels active downloads if necessary
4. **DELETE /api/queue/** - Remove multiple items (batch operation)
- Request: `QueueOperationRequest` with list of item IDs
- Returns: 204 No Content (partial success acceptable)
5. **POST /api/queue/start** - Start queue processor
- Idempotent operation (safe to call multiple times)
6. **POST /api/queue/stop** - Stop queue processor
- Waits for active downloads to complete (with timeout)
7. **POST /api/queue/pause** - Pause queue processing
- Active downloads continue, no new downloads start
8. **POST /api/queue/resume** - Resume queue processing
9. **POST /api/queue/reorder** - Reorder pending queue item
- Request: `QueueReorderRequest` with item_id and new_position
- Returns: 404 if item not in pending queue
10. **DELETE /api/queue/completed** - Clear completed items from history
- Returns count of cleared items
11. **POST /api/queue/retry** - Retry failed downloads
- Request: `QueueOperationRequest` with item IDs (empty for all)
- Only retries items under max retry limit
#### Dependencies
- **get_download_service**: Factory function providing singleton DownloadService instance
- Automatically initializes AnimeService as dependency
- Raises 503 if anime directory not configured
- **get_anime_service**: Factory function providing singleton AnimeService instance
- Required by DownloadService for anime operations
- Both dependencies added to `src/server/utils/dependencies.py`
#### Error Handling
- All endpoints return structured JSON error responses
- HTTP status codes follow REST conventions (200, 201, 204, 400, 401, 404, 500, 503)
- Service-level exceptions (DownloadServiceError) mapped to 400 Bad Request
- Generic exceptions mapped to 500 Internal Server Error
- Authentication errors return 401 Unauthorized
#### Testing
- Comprehensive test suite in `tests/api/test_download_endpoints.py`
- Tests cover:
- Successful operations for all endpoints
- Authentication requirements
- Error conditions (empty lists, not found, service errors)
- Priority handling
- Batch operations
- Uses pytest fixtures for authenticated client and mocked download service
#### Integration
- Router registered in `src/server/fastapi_app.py` via `app.include_router(download_router)`
- Follows same patterns as other API routers (auth, anime, config)
- Full OpenAPI documentation available at `/api/docs`
### WebSocket Integration with Core Services (October 2025)
Completed comprehensive integration of WebSocket broadcasting with all core services to provide real-time updates for downloads, scans, queue operations, and progress tracking.
#### ProgressService
**File**: `src/server/services/progress_service.py`
A centralized service for tracking and broadcasting real-time progress updates across the application.
**Key Features**:
- Track multiple concurrent progress operations (downloads, scans, queue changes)
- Automatic progress percentage calculation
- Progress lifecycle management (start, update, complete, fail, cancel)
- WebSocket integration for real-time client updates
- Progress history with configurable size limit (default: 50 items)
- Thread-safe operations using asyncio locks
- Support for progress metadata and custom messages
**Progress Types**:
- `DOWNLOAD` - File download progress
- `SCAN` - Library scan progress
- `QUEUE` - Queue operation progress
- `SYSTEM` - System-level operations
- `ERROR` - Error notifications
**Progress Statuses**:
- `STARTED` - Operation initiated
- `IN_PROGRESS` - Operation in progress
- `COMPLETED` - Successfully completed
- `FAILED` - Operation failed
- `CANCELLED` - Cancelled by user
**Core Methods**:
- `start_progress()` - Initialize new progress operation
- `update_progress()` - Update progress with current/total values
- `complete_progress()` - Mark operation as completed
- `fail_progress()` - Mark operation as failed
- `cancel_progress()` - Cancel ongoing operation
- `get_progress()` - Retrieve progress by ID
- `get_all_active_progress()` - Get all active operations (optionally filtered by type)
**Broadcasting**:
- Integrates with WebSocketService via callback
- Broadcasts to room-specific channels (e.g., `download_progress`, `scan_progress`)
- Configurable broadcast throttling (only on significant changes >1% or forced)
- Automatic progress state serialization to JSON
**Singleton Pattern**:
- Global instance via `get_progress_service()` factory
- Initialized during application startup with WebSocket callback
#### Integration with Services
**DownloadService Integration**:
- Progress tracking for each download item
- Real-time progress updates during file download
- Automatic completion/failure notifications
- Progress metadata includes speed, ETA, downloaded bytes
**AnimeService Integration**:
- Progress tracking for library scans
- Scan progress with current/total file counts
- Scan completion with statistics
- Error notifications on scan failures
#### WebSocket Message Models
**File**: `src/server/models/websocket.py`
Added progress-specific message models:
- `ScanProgressMessage` - Scan progress updates
- `ScanCompleteMessage` - Scan completion notification
- `ScanFailedMessage` - Scan failure notification
- `ErrorNotificationMessage` - Critical error notifications
- `ProgressUpdateMessage` - Generic progress updates
**WebSocket Message Types**:
- `SCAN_PROGRESS` - Scan progress updates
- `SCAN_COMPLETE` - Scan completion
- `SCAN_FAILED` - Scan failure
- Extended existing types for downloads and queue updates
#### WebSocket Rooms
Clients can subscribe to specific progress channels:
- `download_progress` - Download progress updates
- `scan_progress` - Library scan updates
- `queue_progress` - Queue operation updates
- `system_progress` - System-level updates
Room subscription via client messages:
```json
{
"action": "join",
"room": "download_progress"
}
```
#### Application Startup
**File**: `src/server/fastapi_app.py`
Progress service initialized on application startup:
1. Get ProgressService singleton instance
2. Get WebSocketService singleton instance
3. Register broadcast callback to link progress updates with WebSocket
4. Callback broadcasts progress messages to appropriate rooms
#### Testing
**File**: `tests/unit/test_progress_service.py`
Comprehensive test coverage including:
- Progress lifecycle operations (start, update, complete, fail, cancel)
- Percentage calculation accuracy
- History management and size limits
- Broadcast callback invocation
- Concurrent progress operations
- Metadata handling
- Error conditions and edge cases
#### Architecture Benefits
- **Decoupling**: ProgressService decouples progress tracking from WebSocket broadcasting
- **Reusability**: Single service used across all application components
- **Scalability**: Supports multiple concurrent operations efficiently
- **Observability**: Centralized progress tracking simplifies monitoring
- **Real-time UX**: Instant feedback to users on all long-running operations
#### Future Enhancements
- Persistent progress history (database storage)
- Progress rate calculation and trend analysis
- Multi-process progress synchronization (Redis/shared store)
- Progress event hooks for custom actions
- Client-side progress resumption after reconnection
### Core Services WebSocket Integration (October 2025)
Completed comprehensive integration of WebSocket broadcasting with all core services (DownloadService, AnimeService, ProgressService) to provide real-time updates to connected clients.
#### DownloadService WebSocket Integration
**File**: `src/server/services/download_service.py`
The download service broadcasts real-time updates for all queue and download operations:
**Download Progress Broadcasting**:
- `download_progress` - Real-time progress updates during download
- Includes: download_id, serie_name, season, episode, progress data (percent, speed, ETA)
- Sent via ProgressService which broadcasts to `download_progress` room
- Progress callback created for each download item with metadata tracking
**Download Completion/Failure Broadcasting**:
- `download_complete` - Successful download completion
- Includes: download_id, serie_name, season, episode, downloaded_mb
- Broadcast to `downloads` room
- `download_failed` - Download failure notification
- Includes: download_id, serie_name, season, episode, error, retry_count
- Broadcast to `downloads` room
**Queue Operations Broadcasting**:
All queue operations broadcast `queue_status` messages with current queue state:
- `items_added` - Items added to queue
- Data: added_ids, queue_status (complete queue state)
- `items_removed` - Items removed/cancelled
- Data: removed_ids, queue_status
- `queue_reordered` - Queue order changed
- Data: item_id, new_position, queue_status
- `items_retried` - Failed items retried
- Data: retried_ids, queue_status
- `completed_cleared` - Completed items cleared
- Data: cleared_count, queue_status
**Queue Control Broadcasting**:
- `queue_started` - Queue processor started
- Data: is_running=True, queue_status
- `queue_stopped` - Queue processor stopped
- Data: is_running=False, queue_status
- `queue_paused` - Queue processing paused
- Data: is_paused=True, queue_status
- `queue_resumed` - Queue processing resumed
- Data: is_paused=False, queue_status
**Broadcast Callback Setup**:
The download service broadcast callback is registered during dependency injection in `src/server/utils/dependencies.py`:
- Maps update types to WebSocket service methods
- Routes download_progress, download_complete, download_failed to appropriate rooms
- All queue operations broadcast complete queue status for client synchronization
#### AnimeService WebSocket Integration
**File**: `src/server/services/anime_service.py`
The anime service integrates with ProgressService for library scan operations:
**Scan Progress Broadcasting**:
- Scan operations use ProgressService for progress tracking
- Progress updates broadcast to `scan_progress` room
- Lifecycle events:
- `started` - Scan initialization
- `in_progress` - Ongoing scan with current/total file counts
- `completed` - Successful scan completion
- `failed` - Scan failure with error message
**Scan Implementation**:
- `rescan()` method wraps SeriesApp.ReScan with progress tracking
- Progress callback executed in threadpool updates ProgressService
- ProgressService automatically broadcasts to WebSocket clients
- Cache invalidation on successful scan completion
#### ProgressService WebSocket Integration
**File**: `src/server/services/progress_service.py`
Central service for tracking and broadcasting all progress operations:
**Progress Types**:
- `DOWNLOAD` - File download progress
- `SCAN` - Library scan progress
- `QUEUE` - Queue operation progress
- `SYSTEM` - System-level operations
- `ERROR` - Error notifications
**Progress Lifecycle**:
1. `start_progress()` - Initialize progress operation
- Broadcasts to room: `{progress_type}_progress`
2. `update_progress()` - Update progress values
- Calculates percentage automatically
- Broadcasts only on significant changes (>1% or forced)
3. `complete_progress()` - Mark operation complete
- Sets progress to 100%
- Moves to history
- Broadcasts completion
4. `fail_progress()` - Mark operation failed
- Captures error message
- Moves to history
- Broadcasts failure
**Broadcast Callback**:
- Callback registered during application startup in `src/server/fastapi_app.py`
- Links ProgressService to WebSocketService.manager.broadcast_to_room
- All progress updates automatically broadcast to appropriate rooms
#### WebSocket Room Structure
Clients subscribe to specific rooms to receive targeted updates:
**Room Types**:
- `downloads` - All download-related events (complete, failed, queue status)
- `download_progress` - Real-time download progress updates
- `scan_progress` - Library scan progress updates
- `queue_progress` - Queue operation progress (future use)
- `system_progress` - System-level progress (future use)
**Room Subscription**:
Clients join rooms by sending WebSocket messages:
```json
{
"action": "join",
"room": "download_progress"
}
```
#### Message Format
All WebSocket messages follow a consistent structure:
```json
{
"type": "download_progress" | "download_complete" | "queue_status" | etc.,
"timestamp": "2025-10-17T12:34:56.789Z",
"data": {
// Message-specific data
}
}
```
**Example: Download Progress**
```json
{
"type": "download_progress",
"timestamp": "2025-10-17T12:34:56.789Z",
"data": {
"download_id": "abc123",
"serie_name": "Attack on Titan",
"season": 1,
"episode": 5,
"progress": {
"percent": 45.2,
"downloaded_mb": 226.0,
"total_mb": 500.0,
"speed_mbps": 2.5,
"eta_seconds": 120
}
}
}
```
**Example: Queue Status**
```json
{
"type": "queue_status",
"timestamp": "2025-10-17T12:34:56.789Z",
"data": {
"action": "items_added",
"added_ids": ["item1", "item2"],
"queue_status": {
"is_running": true,
"is_paused": false,
"active_downloads": [...],
"pending_queue": [...],
"completed_downloads": [...],
"failed_downloads": [...]
}
}
}
```
#### Integration Testing
**File**: `tests/integration/test_websocket_integration.py`
Comprehensive integration tests verify WebSocket broadcasting:
**Test Coverage**:
- Download progress broadcasts during active downloads
- Queue operation broadcasts (add, remove, reorder, clear, retry)
- Queue control broadcasts (start, stop, pause, resume)
- Scan progress broadcasts (start, update, complete, fail)
- Progress lifecycle broadcasts for all operation types
- End-to-end flow with multiple services broadcasting
**Test Strategy**:
- Mock broadcast callbacks to capture emitted messages
- Verify message types, data structure, and content
- Test both successful and failure scenarios
- Verify proper room routing for different message types
#### Architecture Benefits
**Decoupling**:
- Services use generic broadcast callbacks without WebSocket dependencies
- ProgressService provides abstraction layer for progress tracking
- Easy to swap WebSocket implementation or add additional broadcast targets
**Consistency**:
- All services follow same broadcast patterns
- Standardized message formats across application
- Unified progress tracking via ProgressService
**Real-time UX**:
- Instant feedback on all long-running operations
- Live queue status updates
- Progress bars update smoothly without polling
- Error notifications delivered immediately
**Scalability**:
- Room-based messaging enables targeted updates
- Multiple concurrent operations supported
- Easy to add new progress types and message formats
#### Production Considerations
**Single-Process Deployment** (Current):
- In-memory connection registry in WebSocketService
- Works perfectly for single-worker deployments
- No additional infrastructure required
**Multi-Process/Multi-Host Deployment** (Future):
- Move connection registry to Redis or similar shared store
- Implement pub/sub for cross-process message broadcasting
- Add connection persistence for recovery after restarts
- Consider using sticky sessions or connection migration
**Performance**:
- Progress updates throttled to >1% changes to reduce message volume
- Broadcast operations are fire-and-forget (non-blocking)
- Failed connections automatically cleaned up
- Message serialization cached where possible
**Monitoring**:
- Structured logging for all broadcast operations
- WebSocket status available at `/ws/status` endpoint
- Connection count and room membership tracking
- Error tracking for failed broadcasts
### Frontend Authentication Integration (October 2025)
Completed JWT-based authentication integration between frontend and backend.
#### Authentication Token Storage
**Files Modified:**
- `src/server/web/templates/login.html` - Store JWT token after successful login
- `src/server/web/templates/setup.html` - Redirect to login after setup completion
- `src/server/web/static/js/app.js` - Include Bearer token in all authenticated requests
- `src/server/web/static/js/queue.js` - Include Bearer token in queue API calls
**Implementation:**
- JWT tokens stored in `localStorage` after successful login
- Token expiry stored in `localStorage` for client-side validation
- `Authorization: Bearer <token>` header included in all authenticated requests
- Automatic redirect to `/login` on 401 Unauthorized responses
- Token cleared from `localStorage` on logout
**Key Functions Updated:**
- `makeAuthenticatedRequest()` in both `app.js` and `queue.js`
- `checkAuthentication()` to verify token and redirect if missing/invalid
- `logout()` to clear token and redirect to login
### Frontend API Endpoint Updates (October 2025)
Updated frontend JavaScript to match new backend API structure.
**Queue Management API Changes:**
- `/api/queue/clear` → `/api/queue/completed` for clearing completed downloads
- `/api/queue/remove` → `/api/queue/{item_id}` (DELETE) for single item removal
- `/api/queue/retry` payload changed to `{item_ids: []}` array format
- `/api/download/pause` → `/api/queue/pause`
- `/api/download/resume` → `/api/queue/resume`
- `/api/download/cancel` → `/api/queue/stop`
**Response Format Changes:**
- Login returns `{access_token, token_type, expires_at}` instead of `{status: 'success'}`
- Setup returns `{status: 'ok'}` instead of `{status: 'success', redirect_url}`
- Logout returns `{status: 'ok'}` instead of `{status: 'success'}`
- Queue operations return structured responses with counts (e.g., `{cleared_count, retried_count}`)
### Frontend WebSocket Integration (October 2025)
WebSocket integration previously completed and verified functional.
#### Native WebSocket Implementation
**Files:**
- `src/server/web/static/js/websocket_client.js` - Native WebSocket wrapper
- Templates already updated to use `websocket_client.js` instead of Socket.IO
**Event Compatibility:**
- Dual event handlers in place for backward compatibility
- Old events: `scan_completed`, `scan_error`, `download_completed`, `download_error`
- New events: `scan_complete`, `scan_failed`, `download_complete`, `download_failed`
- Both event types supported simultaneously
**Room Subscriptions:**
- `downloads` - Download completion, failures, queue status
- `download_progress` - Real-time download progress updates
- `scan_progress` - Library scan progress updates
### Frontend Integration Testing (October 2025)
Created smoke tests to verify frontend-backend integration.
**Test File:** `tests/integration/test_frontend_integration_smoke.py`
**Tests:**
- JWT token format verification (access_token, token_type, expires_at)
- Bearer token authentication on protected endpoints
- 401 responses for requests without valid tokens
**Test Results:**
- Basic authentication flow: ✅ PASSING
- Token validation: Functional with rate limiting considerations
### Frontend Integration (October 2025)
Completed integration of existing frontend JavaScript with the new FastAPI backend and native WebSocket implementation.
#### Native WebSocket Client
**File**: `src/server/web/static/js/websocket_client.js`
Created a Socket.IO-compatible wrapper using native WebSocket API:
**Features**:
- Socket.IO-style `.on()` and `.emit()` methods for compatibility
- Automatic reconnection with exponential backoff (max 5 attempts)
- Room-based subscriptions via `.join()` and `.leave()` methods
- Message queueing during disconnection
- Proper connection lifecycle management
**Usage**:
```javascript
const socket = io(); // Creates WebSocket to ws://host:port/ws/connect
socket.join('download_progress'); // Subscribe to room
socket.on('download_progress', (data) => { ... }); // Handle messages
```
#### WebSocket Message Format
All WebSocket messages follow a structured format:
```json
{
"type": "message_type",
"timestamp": "2025-10-17T12:34:56.789Z",
"data": {}
}
```
**Event Mapping** (Old Socket.IO → New WebSocket):
- `scan_completed` / `scan_complete` → Scan finished
- `scan_error` / `scan_failed` → Scan error
- `download_completed` / `download_complete` → Download finished
- `download_error` / `download_failed` → Download error
- `queue_updated` / `queue_status` → Queue state changes
- `queue_started`, `queue_stopped`, `queue_paused`, `queue_resumed` → Queue control events
**Rooms**:
- `scan_progress` - Library scan updates
- `download_progress` - Real-time download progress
- `downloads` - Download completion, failures, queue status
#### JavaScript Updates
**app.js**:
- Added room subscriptions on WebSocket connect
- Added dual event handlers for old and new message types
- `connected` event handler for initial WebSocket confirmation
- Handles both `scan_complete` and legacy `scan_completed` events
- Handles both `scan_failed` and legacy `scan_error` events
**queue.js**:
- Added room subscriptions on WebSocket connect
- Added dual event handlers for backward compatibility
- Handles both `queue_status` and legacy `queue_updated` events
- Handles both `download_complete` and legacy `download_completed` events
- Handles both `download_failed` and legacy `download_error` events
- Added handlers for `queue_started`, `queue_stopped`, `queue_paused`, `queue_resumed`
#### Template Updates
**Modified Templates**:
- `src/server/web/templates/index.html` - Replaced Socket.IO CDN with websocket_client.js
- `src/server/web/templates/queue.html` - Replaced Socket.IO CDN with websocket_client.js
**Benefits**:
- No external CDN dependency (Socket.IO)
- Native browser WebSocket API (faster, smaller)
- Full compatibility with existing JavaScript code
- Proper integration with backend WebSocket service
#### API Router Registration
**fastapi_app.py**:
- ✅ Added `anime_router` import and registration
- All routers now properly included:
- `health_router` - Health checks
- `page_router` - HTML pages
- `auth_router` - Authentication (JWT-based)
- `anime_router` - Anime management (NEW)
- `download_router` - Download queue
- `websocket_router` - WebSocket connection
**Anime Endpoints**:
- `GET /api/v1/anime` - List anime with missing episodes
- `POST /api/v1/anime/rescan` - Trigger library rescan
- `POST /api/v1/anime/search` - Search for anime
- `GET /api/v1/anime/{anime_id}` - Get anime details
#### Authentication Integration
JavaScript uses JWT tokens from localStorage for authenticated requests:
- Token stored after successful login
- Included in `Authorization: Bearer <token>` header
- Automatic redirect to `/login` on 401 responses
- Compatible with backend AuthMiddleware
#### Testing
**Verified Functionality**:
- ✅ WebSocket client initialization and connection
- ✅ Room subscriptions and message routing
- ✅ Event handler compatibility (old and new message types)
- ✅ Anime API endpoints (passed pytest tests)
- ✅ Download queue API endpoints (existing tests)
- ✅ Frontend integration tests (comprehensive)
**Frontend Integration Test Suite**: `tests/frontend/test_existing_ui_integration.py`
**Coverage**:
- Authentication flow with JWT tokens
- API endpoint compatibility (anime, download, config)
- WebSocket real-time updates
- Data format validation
- Error handling (401, 400/422)
- Multiple client broadcast scenarios
**Test Classes**:
- `TestFrontendAuthentication`: JWT login, logout, auth status
- `TestFrontendAnimeAPI`: Anime list, search, rescan operations
- `TestFrontendDownloadAPI`: Queue management, start/pause/stop
- `TestFrontendWebSocketIntegration`: Connection, broadcasts, progress
- `TestFrontendConfigAPI`: Configuration get/update
- `TestFrontendJavaScriptIntegration`: Bearer token patterns
- `TestFrontendErrorHandling`: JSON errors, validation
- `TestFrontendRealTimeUpdates`: Download events, notifications
- `TestFrontendDataFormats`: Response format validation
**Test Commands**:
```bash
# Run all frontend integration tests
conda run -n AniWorld python -m pytest tests/frontend/test_existing_ui_integration.py -v
# Run specific test class
conda run -n AniWorld python -m pytest tests/frontend/test_existing_ui_integration.py::TestFrontendAuthentication -v
# Run all API tests
conda run -n AniWorld python -m pytest tests/api/ -v
# Run all tests
conda run -n AniWorld python -m pytest tests/ -v
```
**Note**: Some tests require auth service state isolation. The test suite uses fixtures to reset authentication state before each test. If you encounter auth-related test failures, they may be due to shared state across test runs.
#### Known Limitations
**Legacy Events**: Some Socket.IO events don't have backend implementations:
- `scheduled_rescan_*` events
- `auto_download_*` events
- `download_episode_update` event
- `download_series_completed` event
**Solution**: These events are kept in JavaScript for future implementation or can be removed if not needed.
**Configuration Endpoints**: Many config-related features in app.js don't have backend endpoints:
- Scheduler configuration
- Logging configuration
- Advanced configuration
- Config backups
**Solution**: These can be implemented later or the UI features removed.
#### Documentation
**Detailed Documentation**: See `FRONTEND_INTEGRATION.md` for:
- Complete API endpoint mapping
- WebSocket message format details
- Migration guide for developers
- Testing strategies
- Integration patterns
Reference in New Issue View Git Blame Copy Permalink