2180 lines
76 KiB
Markdown
2180 lines
76 KiB
Markdown
# Aniworld Web Application Infrastructure
|
|
|
|
conda activate AniWorld
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
/home/lukas/Volume/repo/Aniworld/
|
|
├── src/
|
|
│ ├── core/ # Core application logic
|
|
│ │ ├── SeriesApp.py # Main application class with async support
|
|
│ │ ├── SerieScanner.py # Directory scanner for anime series
|
|
│ │ ├── entities/ # Domain entities
|
|
│ │ │ ├── series.py # Serie data model
|
|
│ │ │ └── SerieList.py # Series list management
|
|
│ │ ├── interfaces/ # Abstract interfaces
|
|
│ │ │ └── providers.py # Provider interface definitions
|
|
│ │ ├── providers/ # Content providers
|
|
│ │ │ ├── base_provider.py # Base loader interface
|
|
│ │ │ ├── aniworld_provider.py # Aniworld.to implementation
|
|
│ │ │ ├── provider_factory.py # Provider factory
|
|
│ │ │ ├── provider_config.py # Provider configuration
|
|
│ │ │ ├── health_monitor.py # Provider health monitoring
|
|
│ │ │ ├── failover.py # Provider failover system
|
|
│ │ │ ├── monitored_provider.py # Performance tracking wrapper
|
|
│ │ │ ├── config_manager.py # Dynamic configuration mgmt
|
|
│ │ │ └── streaming/ # Streaming providers (VOE, etc.)
|
|
│ │ └── exceptions/ # Custom exceptions
|
|
│ │ └── Exceptions.py # Exception definitions
|
|
│ ├── server/ # FastAPI web application
|
|
│ │ ├── fastapi_app.py # Main FastAPI application (simplified)
|
|
│ │ ├── main.py # FastAPI application entry point
|
|
│ │ ├── controllers/ # Route controllers
|
|
│ │ │ ├── __init__.py # Controllers package
|
|
│ │ │ ├── health_controller.py # Health check endpoints
|
|
│ │ │ ├── page_controller.py # HTML page routes
|
|
│ │ │ └── error_controller.py # Error handling controllers
|
|
│ │ ├── api/ # API route handlers
|
|
│ │ │ ├── __init__.py
|
|
│ │ │ ├── auth.py # Authentication endpoints
|
|
│ │ │ ├── config.py # Configuration endpoints
|
|
│ │ │ ├── anime.py # Anime management endpoints
|
|
│ │ │ ├── download.py # Download queue endpoints
|
|
│ │ │ ├── scheduler.py # Scheduler configuration endpoints
|
|
│ │ │ └── websocket.py # WebSocket real-time endpoints
|
|
│ │ ├── models/ # Pydantic models
|
|
│ │ │ ├── __init__.py
|
|
│ │ │ ├── auth.py
|
|
│ │ │ ├── config.py
|
|
│ │ │ ├── anime.py
|
|
│ │ │ ├── download.py
|
|
│ │ │ └── websocket.py # WebSocket message models
|
|
│ │ ├── services/ # Business logic services
|
|
│ │ │ ├── __init__.py
|
|
│ │ │ ├── auth_service.py
|
|
│ │ │ ├── config_service.py
|
|
│ │ │ ├── anime_service.py
|
|
│ │ │ ├── download_service.py
|
|
│ │ │ ├── websocket_service.py # WebSocket connection management
|
|
│ │ │ ├── progress_service.py # Progress tracking
|
|
│ │ │ ├── notification_service.py # Notification system
|
|
│ │ │ └── cache_service.py # Caching layer
|
|
│ │ ├── database/ # Database layer
|
|
│ │ │ ├── __init__.py # Database package
|
|
│ │ │ ├── base.py # Base models and mixins
|
|
│ │ │ ├── models.py # SQLAlchemy ORM models
|
|
│ │ │ └── connection.py # Database connection management
|
|
│ │ ├── utils/ # Utility functions
|
|
│ │ │ ├── __init__.py
|
|
│ │ │ ├── security.py
|
|
│ │ │ ├── dependencies.py # Dependency injection
|
|
│ │ │ ├── templates.py # Shared Jinja2 template config
|
|
│ │ │ ├── template_helpers.py # Template rendering utilities
|
|
│ │ │ └── logging.py # Logging utilities
|
|
│ │ └── web/ # Frontend assets
|
|
│ │ ├── templates/ # Jinja2 HTML templates
|
|
│ │ │ ├── index.html # Main application page
|
|
│ │ │ ├── login.html # Login page
|
|
│ │ │ ├── setup.html # Initial setup page
|
|
│ │ │ ├── queue.html # Download queue page
|
|
│ │ │ └── error.html # Error page
|
|
│ │ └── static/ # Static web assets
|
|
│ │ ├── css/
|
|
│ │ │ ├── styles.css # Main styles
|
|
│ │ │ └── ux_features.css # UX enhancements
|
|
│ │ └── js/
|
|
│ │ ├── app.js # Main application logic
|
|
│ │ ├── queue.js # Queue management
|
|
│ │ ├── localization.js # i18n support
|
|
│ │ ├── keyboard_shortcuts.js # Keyboard navigation
|
|
│ │ ├── user_preferences.js # User settings
|
|
│ │ ├── undo_redo.js # Undo/redo system
|
|
│ │ ├── mobile_responsive.js # Mobile support
|
|
│ │ ├── touch_gestures.js # Touch interactions
|
|
│ │ ├── accessibility_features.js # A11y features
|
|
│ │ ├── screen_reader_support.js # Screen reader
|
|
│ │ ├── color_contrast_compliance.js # WCAG compliance
|
|
│ │ ├── multi_screen_support.js # Multi-monitor
|
|
│ │ ├── drag_drop.js # Drag and drop
|
|
│ │ ├── bulk_operations.js # Bulk actions
|
|
│ │ └── advanced_search.js # Search filters
|
|
│ ├── core/ # Existing core functionality
|
|
│ └── cli/ # Existing CLI application
|
|
├── data/ # Application data storage
|
|
│ ├── config.json # Application configuration
|
|
│ ├── anime_library.db # SQLite database for anime library
|
|
│ ├── download_queue.json # Download queue state
|
|
│ └── cache/ # Temporary cache files
|
|
├── logs/ # Application logs
|
|
│ ├── app.log # Main application log
|
|
│ ├── download.log # Download-specific logs
|
|
│ └── error.log # Error logs
|
|
├── requirements.txt # Python dependencies
|
|
├── docker-compose.yml # Docker deployment configuration
|
|
└── README.md
|
|
```
|
|
|
|
## Technology Stack
|
|
|
|
### Backend
|
|
|
|
- **FastAPI**: Modern Python web framework for building APIs
|
|
- **Uvicorn**: ASGI server for running FastAPI applications
|
|
- **SQLAlchemy**: SQL toolkit and ORM for database operations
|
|
- **SQLite**: Lightweight database for storing anime library and configuration
|
|
- **Alembic**: Database migration tool for schema management
|
|
- **Pydantic**: Data validation and serialization
|
|
- **Jinja2**: Template engine for server-side rendering
|
|
|
|
### Frontend
|
|
|
|
- **HTML5/CSS3**: Core web technologies
|
|
- **JavaScript (Vanilla)**: Client-side interactivity
|
|
- **Bootstrap 5**: CSS framework for responsive design
|
|
- **HTMX**: Modern approach for dynamic web applications
|
|
|
|
### Security
|
|
|
|
- **Passlib**: Password hashing and verification
|
|
- **python-jose**: JWT token handling
|
|
- **bcrypt**: Secure password hashing
|
|
|
|
### Authentication Models & Sessions
|
|
|
|
- Authentication request/response Pydantic models live in `src/server/models/auth.py`.
|
|
- Sessions are represented by `SessionModel` and can be backed by an in-memory
|
|
store or a persistent table depending on deployment needs. JWTs are used for
|
|
stateless authentication by default; a persistent session store may be
|
|
configured in production to enable revocation and long-lived sessions.
|
|
|
|
## Configuration
|
|
|
|
### Data Storage
|
|
|
|
- **Configuration**: JSON files in `data/` directory
|
|
- **Anime Library**: SQLite database with series information
|
|
- **Download Queue**: JSON file with current download status
|
|
- **Logs**: Structured logging to files in `logs/` directory
|
|
|
|
## API Endpoints
|
|
|
|
### Authentication
|
|
|
|
- `POST /api/auth/login` - Master password authentication
|
|
- `POST /api/auth/logout` - Logout and invalidate session
|
|
- `GET /api/auth/status` - Check authentication status
|
|
|
|
### Configuration
|
|
|
|
- `GET /api/config` - Get current configuration
|
|
- `PUT /api/config` - Update configuration
|
|
- `POST /api/setup` - Initial setup
|
|
|
|
### Configuration API Notes
|
|
|
|
- Configuration endpoints are exposed under `/api/config`
|
|
- Uses file-based persistence with JSON format for human-readable storage
|
|
- Automatic backup creation before configuration updates
|
|
- Configuration validation with detailed error reporting
|
|
- Backup management with create, restore, list, and delete operations
|
|
- Configuration schema versioning with migration support
|
|
- Singleton ConfigService manages all persistence operations
|
|
- Default configuration location: `data/config.json`
|
|
- Backup directory: `data/config_backups/`
|
|
- Maximum backups retained: 10 (configurable)
|
|
- Automatic cleanup of old backups exceeding limit
|
|
|
|
**Key Endpoints:**
|
|
|
|
- `GET /api/config` - Retrieve current configuration
|
|
- `PUT /api/config` - Update configuration (creates backup)
|
|
- `POST /api/config/validate` - Validate without applying
|
|
- `GET /api/config/backups` - List all backups
|
|
- `POST /api/config/backups` - Create manual backup
|
|
- `POST /api/config/backups/{name}/restore` - Restore from backup
|
|
- `DELETE /api/config/backups/{name}` - Delete backup
|
|
- `GET /api/config/section/advanced` - Get advanced configuration section
|
|
- `POST /api/config/section/advanced` - Update advanced configuration
|
|
- `POST /api/config/directory` - Update anime directory
|
|
- `POST /api/config/export` - Export configuration to JSON file
|
|
- `POST /api/config/reset` - Reset configuration to defaults
|
|
|
|
**Configuration Service Features:**
|
|
|
|
- Atomic file writes using temporary files
|
|
- JSON format with version metadata
|
|
- Validation before saving
|
|
- Automatic backup on updates
|
|
- Migration support for schema changes
|
|
- Thread-safe singleton pattern
|
|
- Comprehensive error handling with custom exceptions
|
|
|
|
### Scheduler
|
|
|
|
- `GET /api/scheduler/config` - Get scheduler configuration
|
|
- `POST /api/scheduler/config` - Update scheduler configuration
|
|
- `POST /api/scheduler/trigger-rescan` - Manually trigger rescan
|
|
|
|
### Anime Management
|
|
|
|
- `GET /api/anime` - List anime with missing episodes
|
|
- `POST /api/anime/{id}/download` - Add episodes to download queue
|
|
- `GET /api/anime/{id}` - Get anime details
|
|
|
|
Note: The anime management API has been implemented under `/api/v1/anime` with
|
|
endpoints for listing series with missing episodes, searching providers,
|
|
triggering a local rescan, and fetching series details. The implementation
|
|
delegates to the existing core `SeriesApp` and uses dependency injection for
|
|
initialization.
|
|
|
|
### Download Management
|
|
|
|
- `GET /api/queue/status` - Get download queue status and statistics
|
|
- `POST /api/queue/add` - Add episodes to download queue
|
|
- `DELETE /api/queue/{id}` - Remove single item from pending queue
|
|
- `POST /api/queue/start` - Manually start next download from queue (one at a time)
|
|
- `POST /api/queue/stop` - Stop processing new downloads
|
|
- `DELETE /api/queue/completed` - Clear completed downloads
|
|
- `DELETE /api/queue/failed` - Clear failed downloads
|
|
- `POST /api/queue/retry/{id}` - Retry a specific failed download
|
|
- `POST /api/queue/retry` - Retry all failed downloads
|
|
|
|
**Manual Download Control:**
|
|
|
|
- Queue processing is fully manual - no auto-start
|
|
- User must click "Start" to begin downloading next item from queue
|
|
- Only one download active at a time
|
|
- "Stop" prevents new downloads but allows current to complete
|
|
- FIFO queue order (first-in, first-out)
|
|
|
|
**Queue Organization:**
|
|
|
|
- **Pending Queue**: Items waiting to be downloaded, displayed in FIFO order
|
|
- **Active Download**: Currently downloading item with progress bar (max 1)
|
|
- **Completed Downloads**: Successfully downloaded items with completion timestamps
|
|
- **Failed Downloads**: Failed items with error messages and retry options
|
|
|
|
**Queue Display Features:**
|
|
|
|
- Real-time statistics counters (pending, active, completed, failed)
|
|
- Empty state messages with helpful hints
|
|
- Per-section action buttons (clear, retry all)
|
|
- Start/Stop buttons for manual queue control
|
|
|
|
### WebSocket
|
|
|
|
- `WS /api/ws` - WebSocket connection for real-time updates
|
|
- Real-time download progress notifications
|
|
- Queue status updates
|
|
- System notifications
|
|
|
|
## Logging
|
|
|
|
### Log Levels
|
|
|
|
- **INFO**: General application information
|
|
- **WARNING**: Potential issues that don't stop execution
|
|
- **ERROR**: Errors that affect functionality
|
|
- **DEBUG**: Detailed debugging information (development only)
|
|
|
|
### Log Files
|
|
|
|
- `app.log`: General application logs
|
|
- `download.log`: Download-specific operations
|
|
- `error.log`: Error and exception logs
|
|
|
|
## Security Considerations
|
|
|
|
- Master password protection for application access
|
|
- Secure session management with JWT tokens
|
|
- Input validation and sanitization
|
|
- Built-in rate limiting in authentication middleware
|
|
- HTTPS enforcement in production
|
|
- Secure file path handling to prevent directory traversal
|
|
|
|
### Authentication Service
|
|
|
|
- A lightweight authentication service is provided by
|
|
`src/server/services/auth_service.py`.
|
|
- Uses bcrypt (passlib) to hash the master password and issues JWTs for
|
|
stateless sessions. Tokens are signed with the `JWT_SECRET_KEY` from
|
|
configuration and expire based on `SESSION_TIMEOUT_HOURS`.
|
|
- Failed login attempts are tracked in-memory and a temporary lockout is
|
|
applied after multiple failures. For multi-process deployments, move
|
|
this state to a shared store (Redis) and persist the master password
|
|
hash in a secure config store.
|
|
|
|
## Database Layer (October 2025)
|
|
|
|
A comprehensive SQLAlchemy-based database layer was implemented to provide
|
|
persistent storage for anime series, episodes, download queue, and user sessions.
|
|
|
|
### Architecture
|
|
|
|
**Location**: `src/server/database/`
|
|
|
|
**Components**:
|
|
|
|
- `base.py`: Base declarative class and mixins (TimestampMixin, SoftDeleteMixin)
|
|
- `models.py`: SQLAlchemy ORM models with relationships
|
|
- `connection.py`: Database engine, session factory, and dependency injection
|
|
- `__init__.py`: Package exports and public API
|
|
|
|
### Database Models
|
|
|
|
#### AnimeSeries
|
|
|
|
Represents anime series with metadata and provider information.
|
|
|
|
**Fields**:
|
|
|
|
- `id` (PK): Auto-incrementing primary key
|
|
- `key`: Unique provider identifier (indexed)
|
|
- `name`: Series name (indexed)
|
|
- `site`: Provider site URL
|
|
- `folder`: Local filesystem path
|
|
- `description`: Optional series description
|
|
- `status`: Series status (ongoing, completed)
|
|
- `total_episodes`: Total episode count
|
|
- `cover_url`: Cover image URL
|
|
- `episode_dict`: JSON field storing episode structure {season: [episodes]}
|
|
- `created_at`, `updated_at`: Audit timestamps (from TimestampMixin)
|
|
|
|
**Relationships**:
|
|
|
|
- `episodes`: One-to-many with Episode (cascade delete)
|
|
- `download_items`: One-to-many with DownloadQueueItem (cascade delete)
|
|
|
|
#### Episode
|
|
|
|
Individual episodes linked to anime series.
|
|
|
|
**Fields**:
|
|
|
|
- `id` (PK): Auto-incrementing primary key
|
|
- `series_id` (FK): Foreign key to AnimeSeries (indexed)
|
|
- `season`: Season number
|
|
- `episode_number`: Episode number within season
|
|
- `title`: Optional episode title
|
|
- `file_path`: Local file path if downloaded
|
|
- `file_size`: File size in bytes
|
|
- `is_downloaded`: Boolean download status
|
|
- `download_date`: Timestamp when downloaded
|
|
- `created_at`, `updated_at`: Audit timestamps
|
|
|
|
**Relationships**:
|
|
|
|
- `series`: Many-to-one with AnimeSeries
|
|
|
|
#### DownloadQueueItem
|
|
|
|
Download queue with status and progress tracking.
|
|
|
|
**Fields**:
|
|
|
|
- `id` (PK): Auto-incrementing primary key
|
|
- `series_id` (FK): Foreign key to AnimeSeries (indexed)
|
|
- `season`: Season number
|
|
- `episode_number`: Episode number
|
|
- `status`: Download status enum (indexed)
|
|
- Values: PENDING, DOWNLOADING, PAUSED, COMPLETED, FAILED, CANCELLED
|
|
- `priority`: Priority enum
|
|
- Values: LOW, NORMAL, HIGH
|
|
- `progress_percent`: Download progress (0-100)
|
|
- `downloaded_bytes`: Bytes downloaded
|
|
- `total_bytes`: Total file size
|
|
- `download_speed`: Current speed (bytes/sec)
|
|
- `error_message`: Error description if failed
|
|
- `retry_count`: Number of retry attempts
|
|
- `download_url`: Provider download URL
|
|
- `file_destination`: Target file path
|
|
- `started_at`: Download start timestamp
|
|
- `completed_at`: Download completion timestamp
|
|
- `created_at`, `updated_at`: Audit timestamps
|
|
|
|
**Relationships**:
|
|
|
|
- `series`: Many-to-one with AnimeSeries
|
|
|
|
#### UserSession
|
|
|
|
User authentication sessions with JWT tokens.
|
|
|
|
**Fields**:
|
|
|
|
- `id` (PK): Auto-incrementing primary key
|
|
- `session_id`: Unique session identifier (indexed)
|
|
- `token_hash`: Hashed JWT token
|
|
- `user_id`: User identifier (indexed, for multi-user support)
|
|
- `ip_address`: Client IP address
|
|
- `user_agent`: Client user agent string
|
|
- `expires_at`: Session expiration timestamp
|
|
- `is_active`: Boolean active status (indexed)
|
|
- `last_activity`: Last activity timestamp
|
|
- `created_at`, `updated_at`: Audit timestamps
|
|
|
|
**Methods**:
|
|
|
|
- `is_expired`: Property to check if session has expired
|
|
- `revoke()`: Revoke session by setting is_active=False
|
|
|
|
### Mixins
|
|
|
|
#### TimestampMixin
|
|
|
|
Adds automatic timestamp tracking to models.
|
|
|
|
**Fields**:
|
|
|
|
- `created_at`: Automatically set on record creation
|
|
- `updated_at`: Automatically updated on record modification
|
|
|
|
**Usage**: Inherit in models requiring audit timestamps.
|
|
|
|
#### SoftDeleteMixin
|
|
|
|
Provides soft delete functionality (logical deletion).
|
|
|
|
**Fields**:
|
|
|
|
- `deleted_at`: Timestamp when soft deleted (NULL if active)
|
|
|
|
**Properties**:
|
|
|
|
- `is_deleted`: Check if record is soft deleted
|
|
|
|
**Methods**:
|
|
|
|
- `soft_delete()`: Mark record as deleted
|
|
- `restore()`: Restore soft deleted record
|
|
|
|
**Note**: Currently not used by models but available for future implementation.
|
|
|
|
### Database Connection Management
|
|
|
|
#### Initialization
|
|
|
|
```python
|
|
from src.server.database import init_db, close_db
|
|
|
|
# Application startup
|
|
await init_db() # Creates engine, session factory, and tables
|
|
|
|
# Application shutdown
|
|
await close_db() # Closes connections and cleanup
|
|
```
|
|
|
|
#### Session Management
|
|
|
|
**Async Sessions** (preferred for FastAPI endpoints):
|
|
|
|
```python
|
|
from fastapi import Depends
|
|
from sqlalchemy.ext.asyncio import AsyncSession
|
|
from src.server.database import get_db_session
|
|
|
|
@app.get("/anime")
|
|
async def get_anime(db: AsyncSession = Depends(get_db_session)):
|
|
result = await db.execute(select(AnimeSeries))
|
|
return result.scalars().all()
|
|
```
|
|
|
|
**Sync Sessions** (for non-async operations):
|
|
|
|
```python
|
|
from src.server.database.connection import get_sync_session
|
|
|
|
session = get_sync_session()
|
|
try:
|
|
result = session.execute(select(AnimeSeries))
|
|
return result.scalars().all()
|
|
finally:
|
|
session.close()
|
|
```
|
|
|
|
### Database Configuration
|
|
|
|
**Settings** (from `src/config/settings.py`):
|
|
|
|
- `DATABASE_URL`: Database connection string
|
|
- Default: `sqlite:///./data/aniworld.db`
|
|
- Automatically converted to `sqlite+aiosqlite:///` for async support
|
|
- `LOG_LEVEL`: When set to "DEBUG", enables SQL query logging
|
|
|
|
**Engine Configuration**:
|
|
|
|
- **SQLite**: Uses StaticPool, enables foreign keys and WAL mode
|
|
- **PostgreSQL/MySQL**: Uses QueuePool with pre-ping health checks
|
|
- **Connection Pooling**: Configured based on database type
|
|
- **Echo**: SQL query logging in DEBUG mode
|
|
|
|
### SQLite Optimizations
|
|
|
|
- **Foreign Keys**: Automatically enabled via PRAGMA
|
|
- **WAL Mode**: Write-Ahead Logging for better concurrency
|
|
- **Static Pool**: Single connection pool for SQLite
|
|
- **Async Support**: aiosqlite driver for async operations
|
|
|
|
### FastAPI Integration
|
|
|
|
**Dependency Injection** (in `src/server/utils/dependencies.py`):
|
|
|
|
```python
|
|
async def get_database_session() -> AsyncGenerator:
|
|
"""Dependency to get database session."""
|
|
try:
|
|
from src.server.database import get_db_session
|
|
|
|
async with get_db_session() as session:
|
|
yield session
|
|
except ImportError:
|
|
raise HTTPException(status_code=501, detail="Database not installed")
|
|
except RuntimeError as e:
|
|
raise HTTPException(status_code=503, detail=f"Database not available: {str(e)}")
|
|
```
|
|
|
|
**Usage in Endpoints**:
|
|
|
|
```python
|
|
from fastapi import Depends
|
|
from sqlalchemy.ext.asyncio import AsyncSession
|
|
from src.server.utils.dependencies import get_database_session
|
|
|
|
@router.get("/series/{series_id}")
|
|
async def get_series(
|
|
series_id: int,
|
|
db: AsyncSession = Depends(get_database_session)
|
|
):
|
|
result = await db.execute(
|
|
select(AnimeSeries).where(AnimeSeries.id == series_id)
|
|
)
|
|
series = result.scalar_one_or_none()
|
|
if not series:
|
|
raise HTTPException(status_code=404, detail="Series not found")
|
|
return series
|
|
```
|
|
|
|
### Testing
|
|
|
|
**Test Suite**: `tests/unit/test_database_models.py`
|
|
|
|
**Coverage**:
|
|
|
|
- 30+ comprehensive test cases
|
|
- Model creation and validation
|
|
- Relationship testing (one-to-many, cascade deletes)
|
|
- Unique constraint validation
|
|
- Query operations (filtering, joins)
|
|
- Session management
|
|
- Mixin functionality
|
|
|
|
**Test Strategy**:
|
|
|
|
- In-memory SQLite database for isolation
|
|
- Fixtures for engine and session setup
|
|
- Test all CRUD operations
|
|
- Verify constraints and relationships
|
|
- Test edge cases and error conditions
|
|
|
|
### Migration Strategy (Future)
|
|
|
|
**Alembic Integration** (planned):
|
|
|
|
- Alembic installed but not yet configured
|
|
- Will manage schema migrations in production
|
|
- Auto-generate migrations from model changes
|
|
- Version control for database schema
|
|
|
|
**Initial Setup**:
|
|
|
|
```bash
|
|
# Initialize Alembic (future)
|
|
alembic init alembic
|
|
|
|
# Generate initial migration
|
|
alembic revision --autogenerate -m "Initial schema"
|
|
|
|
# Apply migrations
|
|
alembic upgrade head
|
|
```
|
|
|
|
### Production Considerations
|
|
|
|
**Single-Process Deployment** (current):
|
|
|
|
- SQLite with WAL mode for concurrency
|
|
- Static pool for single connection
|
|
- File-based storage at `data/aniworld.db`
|
|
|
|
**Multi-Process Deployment** (future):
|
|
|
|
- Switch to PostgreSQL or MySQL
|
|
- Configure connection pooling (pool_size, max_overflow)
|
|
- Use QueuePool for connection management
|
|
- Consider read replicas for scaling
|
|
|
|
**Performance**:
|
|
|
|
- Indexes on frequently queried columns (key, name, status, is_active)
|
|
- Foreign key constraints for referential integrity
|
|
- Cascade deletes for cleanup operations
|
|
- Efficient joins via relationship loading strategies
|
|
|
|
**Monitoring**:
|
|
|
|
- SQL query logging in DEBUG mode
|
|
- Connection pool metrics (when using QueuePool)
|
|
- Query performance profiling
|
|
- Database size monitoring
|
|
|
|
**Backup Strategy**:
|
|
|
|
- SQLite: File-based backups (copy `aniworld.db` file)
|
|
- WAL checkpoint before backup
|
|
- Automated backup schedule recommended
|
|
- Store backups in `data/config_backups/` or separate location
|
|
|
|
### Integration with Services
|
|
|
|
**AnimeService**:
|
|
|
|
- Query series from database
|
|
- Persist scan results
|
|
- Update episode metadata
|
|
|
|
**DownloadService**:
|
|
|
|
- Load queue from database on startup
|
|
- Persist queue state continuously
|
|
- Update download progress in real-time
|
|
|
|
**AuthService**:
|
|
|
|
- Store and validate user sessions
|
|
- Session revocation via database
|
|
- Query active sessions for monitoring
|
|
|
|
### Benefits of Database Layer
|
|
|
|
- **Persistence**: Survives application restarts
|
|
- **Relationships**: Enforced referential integrity
|
|
- **Queries**: Powerful filtering and aggregation
|
|
- **Scalability**: Can migrate to PostgreSQL/MySQL
|
|
- **ACID**: Atomic transactions for consistency
|
|
- **Migration**: Schema versioning with Alembic
|
|
- **Testing**: Easy to test with in-memory database
|
|
|
|
### Database Service Layer (October 2025)
|
|
|
|
Implemented comprehensive service layer for database CRUD operations.
|
|
|
|
**File**: `src/server/database/service.py`
|
|
|
|
**Services**:
|
|
|
|
- `AnimeSeriesService`: CRUD operations for anime series
|
|
- `EpisodeService`: Episode management and download tracking
|
|
- `DownloadQueueService`: Queue management with priority and status
|
|
- `UserSessionService`: Session management and authentication
|
|
|
|
**Key Features**:
|
|
|
|
- Repository pattern for clean separation of concerns
|
|
- Type-safe operations with comprehensive type hints
|
|
- Async support for all database operations
|
|
- Transaction management via FastAPI dependency injection
|
|
- Comprehensive error handling and logging
|
|
- Search and filtering capabilities
|
|
- Pagination support for large datasets
|
|
- Batch operations for performance
|
|
|
|
**AnimeSeriesService Operations**:
|
|
|
|
- Create series with metadata and provider information
|
|
- Retrieve by ID, key, or search query
|
|
- Update series attributes
|
|
- Delete series with cascade to episodes and queue items
|
|
- List all series with pagination and eager loading options
|
|
|
|
**EpisodeService Operations**:
|
|
|
|
- Create episodes for series
|
|
- Retrieve episodes by series, season, or specific episode
|
|
- Mark episodes as downloaded with file metadata
|
|
- Delete episodes
|
|
|
|
**DownloadQueueService Operations**:
|
|
|
|
- Add items to queue with priority levels (LOW, NORMAL, HIGH)
|
|
- Retrieve pending, active, or all queue items
|
|
- Update download status (PENDING, DOWNLOADING, COMPLETED, FAILED, etc.)
|
|
- Update download progress (percentage, bytes, speed)
|
|
- Clear completed downloads
|
|
- Retry failed downloads with max retry limits
|
|
- Automatic timestamp management (started_at, completed_at)
|
|
|
|
**UserSessionService Operations**:
|
|
|
|
- Create authentication sessions with JWT tokens
|
|
- Retrieve sessions by session ID
|
|
- Get active sessions with expiry checking
|
|
- Update last activity timestamp
|
|
- Revoke sessions for logout
|
|
- Cleanup expired sessions automatically
|
|
|
|
**Testing**:
|
|
|
|
- Comprehensive test suite with 22 test cases
|
|
- In-memory SQLite for isolated testing
|
|
- All CRUD operations tested
|
|
- Edge cases and error conditions covered
|
|
- 100% test pass rate
|
|
|
|
**Integration**:
|
|
|
|
- Exported via database package `__init__.py`
|
|
- Used by API endpoints via dependency injection
|
|
- Compatible with existing database models
|
|
- Follows project coding standards (PEP 8, type hints, docstrings)
|
|
|
|
**Database Migrations** (`src/server/database/migrations.py`):
|
|
|
|
- Simple schema initialization via SQLAlchemy create_all
|
|
- Schema version checking utility
|
|
- Documentation for Alembic integration
|
|
- Production-ready migration strategy outlined
|
|
|
|
## Core Application Logic
|
|
|
|
### SeriesApp - Enhanced Core Engine
|
|
|
|
The `SeriesApp` class (`src/core/SeriesApp.py`) is the main application engine for anime series management. Enhanced with async support and web integration capabilities.
|
|
|
|
#### Key Features
|
|
|
|
- **Async Operations**: Support for async download and scan operations
|
|
- **Progress Callbacks**: Real-time progress reporting via callbacks
|
|
- **Cancellation Support**: Ability to cancel long-running operations
|
|
- **Error Handling**: Comprehensive error handling with callback notifications
|
|
- **Operation Status**: Track current operation status and history
|
|
|
|
#### Core Classes
|
|
|
|
- `SeriesApp`: Main application class
|
|
- `OperationStatus`: Enum for operation states (IDLE, RUNNING, COMPLETED, CANCELLED, FAILED)
|
|
- `ProgressInfo`: Dataclass for progress information
|
|
- `OperationResult`: Dataclass for operation results
|
|
|
|
#### Key Methods
|
|
|
|
- `search(words)`: Search for anime series
|
|
- `download()`: Download episodes with progress tracking
|
|
- `ReScan()`: Scan directory for missing episodes
|
|
- `cancel_operation()`: Cancel current operation
|
|
- `get_operation_status()`: Get current status
|
|
- `get_series_list()`: Get series with missing episodes
|
|
|
|
#### Integration Points
|
|
|
|
The SeriesApp integrates with:
|
|
|
|
- Provider system for content downloading
|
|
- Serie scanner for directory analysis
|
|
- Series list management for tracking missing episodes
|
|
- Web layer via async operations and callbacks
|
|
|
|
## Progress Callback System
|
|
|
|
### Overview
|
|
|
|
A comprehensive callback system for real-time progress reporting, error handling, and operation completion notifications across core operations (scanning, downloading, searching).
|
|
|
|
### Architecture
|
|
|
|
- **Interface-based Design**: Abstract base classes define callback contracts
|
|
- **Context Objects**: Rich context information for each callback type
|
|
- **Callback Manager**: Centralized management of multiple callbacks
|
|
- **Thread-safe**: Exception handling prevents callback errors from breaking operations
|
|
|
|
### Components
|
|
|
|
#### Callback Interfaces (`src/core/interfaces/callbacks.py`)
|
|
|
|
- `ProgressCallback`: Reports operation progress updates
|
|
- `ErrorCallback`: Handles error notifications
|
|
- `CompletionCallback`: Notifies operation completion
|
|
|
|
#### Context Classes
|
|
|
|
- `ProgressContext`: Current progress, percentage, phase, and metadata
|
|
- `ErrorContext`: Error details, recoverability, retry information
|
|
- `CompletionContext`: Success status, results, and statistics
|
|
|
|
All context dataclasses expose a `key` field (provider identifier) plus an
|
|
optional `folder` field used purely for display metadata. This keeps the
|
|
callback contract aligned with the broader series identifier standardization
|
|
work: downstream consumers rely on `key` for lookups while still showing a
|
|
user-friendly folder name when needed.
|
|
|
|
#### Enums
|
|
|
|
- `OperationType`: SCAN, DOWNLOAD, SEARCH, INITIALIZATION
|
|
- `ProgressPhase`: STARTING, IN_PROGRESS, COMPLETING, COMPLETED, FAILED, CANCELLED
|
|
|
|
#### Callback Manager
|
|
|
|
- Register/unregister multiple callbacks per type
|
|
- Notify all registered callbacks with context
|
|
- Exception handling for callback errors
|
|
- Support for clearing all callbacks
|
|
|
|
### Integration
|
|
|
|
#### SerieScanner
|
|
|
|
- Reports scanning progress (folder by folder)
|
|
- Notifies errors for failed folder scans
|
|
- Reports completion with statistics
|
|
|
|
#### SeriesApp
|
|
|
|
- Download progress reporting with percentage
|
|
- Scan progress through SerieScanner integration
|
|
- Error notifications for all operations
|
|
- Completion notifications with results
|
|
|
|
### Usage Example
|
|
|
|
```python
|
|
from src.core.interfaces.callbacks import (
|
|
CallbackManager,
|
|
ProgressCallback,
|
|
ProgressContext
|
|
)
|
|
|
|
class MyProgressCallback(ProgressCallback):
|
|
def on_progress(self, context: ProgressContext):
|
|
print(f"{context.message}: {context.percentage:.1f}%")
|
|
|
|
# Register callback
|
|
manager = CallbackManager()
|
|
manager.register_progress_callback(MyProgressCallback())
|
|
|
|
# Use with SeriesApp
|
|
app = SeriesApp(directory, callback_manager=manager)
|
|
```
|
|
|
|
## Recent Infrastructure Changes
|
|
|
|
### Progress Callback System (October 2025)
|
|
|
|
Implemented a comprehensive progress callback system for real-time operation tracking.
|
|
|
|
#### Changes Made
|
|
|
|
1. **Callback Interfaces**:
|
|
|
|
- Created abstract base classes for progress, error, and completion callbacks
|
|
- Defined rich context objects with operation metadata
|
|
- Implemented thread-safe callback manager
|
|
|
|
2. **SerieScanner Integration**:
|
|
|
|
- Added progress reporting for directory scanning
|
|
- Implemented per-folder progress updates
|
|
- Error callbacks for scan failures
|
|
- Completion notifications with statistics
|
|
|
|
3. **SeriesApp Integration**:
|
|
|
|
- Integrated callback manager into download operations
|
|
- Progress updates during episode downloads
|
|
- Error handling with callback notifications
|
|
- Completion callbacks for all operations
|
|
- Backward compatibility with legacy callbacks
|
|
|
|
4. **Testing**:
|
|
- 22 comprehensive unit tests
|
|
- Coverage for all callback types
|
|
- Exception handling verification
|
|
- Multiple callback registration tests
|
|
5. **Identifier Support (Nov 2025)**:
|
|
- Added `key` + optional `folder` fields to every context object
|
|
- Docstrings now clarify that `key` is the canonical lookup identifier
|
|
- Tests updated to guarantee both fields serialize correctly
|
|
|
|
### Core Logic Enhancement (October 2025)
|
|
|
|
Enhanced `SeriesApp` with async callback support, progress reporting, and cancellation.
|
|
|
|
#### Changes Made
|
|
|
|
1. **Async Support**:
|
|
|
|
- Added `async_download()` and `async_rescan()` methods
|
|
- Integrated with asyncio event loop for non-blocking operations
|
|
- Support for concurrent operations in web environment
|
|
|
|
2. **Progress Reporting**:
|
|
|
|
- Legacy `ProgressInfo` dataclass for structured progress data
|
|
- New comprehensive callback system with context objects
|
|
- Percentage calculation and status tracking
|
|
|
|
3. **Cancellation System**:
|
|
|
|
- Internal cancellation flag management
|
|
- Graceful operation cancellation
|
|
- Cancellation check during long-running operations
|
|
|
|
4. **Error Handling**:
|
|
|
|
- `OperationResult` dataclass for operation outcomes
|
|
- Error callback system for notifications
|
|
- Specific exception types (IOError, OSError, RuntimeError)
|
|
- Proper exception propagation and logging
|
|
|
|
5. **Status Management**:
|
|
- `OperationStatus` enum for state tracking
|
|
- Current operation identifier
|
|
- Status getter methods for monitoring
|
|
|
|
#### Test Coverage
|
|
|
|
Comprehensive test suite (`tests/unit/test_series_app.py`) with 22 tests covering:
|
|
|
|
- Initialization and configuration
|
|
- Search functionality
|
|
- Download operations with callbacks
|
|
- Directory scanning with progress
|
|
- Async operations
|
|
- Cancellation handling
|
|
- Error scenarios
|
|
- Data model validation
|
|
|
|
### Template Integration (October 2025)
|
|
|
|
Completed integration of HTML templates with FastAPI Jinja2 system.
|
|
|
|
#### Changes Made
|
|
|
|
1. **Template Helper Utilities**:
|
|
|
|
- `src/server/utils/template_helpers.py` - Template rendering utilities
|
|
- Centralized base context for all templates
|
|
- Template validation and listing functions
|
|
- DRY principles for template rendering
|
|
|
|
2. **Enhanced CSS**:
|
|
|
|
- `src/server/web/static/css/ux_features.css` - UX enhancement styles
|
|
- Drag-and-drop indicators
|
|
- Bulk selection styling
|
|
- Keyboard navigation focus indicators
|
|
- Touch gesture feedback
|
|
- Mobile responsive enhancements
|
|
- Accessibility features (high contrast, screen reader support)
|
|
- Reduced motion support
|
|
|
|
3. **JavaScript Modules**:
|
|
|
|
- `keyboard_shortcuts.js` - Keyboard navigation (Ctrl+K, Ctrl+R, etc.)
|
|
- `user_preferences.js` - Settings persistence (localStorage)
|
|
- `undo_redo.js` - Action history with Ctrl+Z/Ctrl+Y
|
|
- `mobile_responsive.js` - Mobile detection and layout
|
|
- `touch_gestures.js` - Swipe gesture handling
|
|
- `accessibility_features.js` - Focus management and ARIA labels
|
|
- `screen_reader_support.js` - Live regions for dynamic content
|
|
- `color_contrast_compliance.js` - WCAG compliance checks
|
|
- `multi_screen_support.js` - Fullscreen and multi-monitor support
|
|
- `drag_drop.js` - Drag-and-drop functionality (stub)
|
|
- `bulk_operations.js` - Bulk selection and actions (stub)
|
|
- `advanced_search.js` - Advanced filtering (stub)
|
|
|
|
4. **Updated Controllers**:
|
|
- Updated `page_controller.py` to use `template_helpers`
|
|
- Updated `error_controller.py` to use `template_helpers`
|
|
- Consistent context passing across all templates
|
|
|
|
#### Template Features
|
|
|
|
- **Responsive Design**: Mobile-first approach with viewport meta tags
|
|
- **Theme Switching**: Light/dark mode with `data-theme` attribute
|
|
- **Accessibility**: ARIA labels, keyboard navigation, screen reader support
|
|
- **Internationalization**: Localization support via `localization.js`
|
|
- **Progressive Enhancement**: Works without JavaScript, enhanced with it
|
|
|
|
#### Verified Templates
|
|
|
|
All HTML templates properly integrated:
|
|
|
|
- `index.html` - Main application page with search and anime list
|
|
- `login.html` - Master password authentication
|
|
- `setup.html` - Initial application setup
|
|
- `queue.html` - Download queue management
|
|
- `error.html` - Error pages (404, 500)
|
|
|
|
All templates include:
|
|
|
|
- Proper HTML5 structure
|
|
- Font Awesome icons
|
|
- Static file references (`/static/css/`, `/static/js/`)
|
|
- Theme switching support
|
|
- Responsive viewport configuration
|
|
|
|
### CSS Integration (October 2025)
|
|
|
|
Integrated existing CSS styling with FastAPI's static file serving system.
|
|
|
|
#### Implementation Details
|
|
|
|
1. **Static File Configuration**:
|
|
|
|
- Static files mounted at `/static` in `fastapi_app.py`
|
|
- Directory: `src/server/web/static/`
|
|
- Files served using FastAPI's `StaticFiles` middleware
|
|
- All paths use absolute references (`/static/...`)
|
|
|
|
2. **CSS Architecture**:
|
|
|
|
- `styles.css` (1,840 lines) - Main stylesheet with Fluent UI design system
|
|
- `ux_features.css` (203 lines) - Enhanced UX features and accessibility
|
|
|
|
3. **Design System** (`styles.css`):
|
|
|
|
- **Fluent UI Variables**: CSS custom properties for consistent theming
|
|
- **Light/Dark Themes**: Dynamic theme switching via `[data-theme="dark"]`
|
|
- **Typography**: Segoe UI font stack with responsive sizing
|
|
- **Spacing System**: Consistent spacing scale (xs through xxl)
|
|
- **Color Palette**: Comprehensive color system for both themes
|
|
- **Border Radius**: Standardized corner radii (sm, md, lg, xl)
|
|
- **Shadows**: Elevation system with card and elevated variants
|
|
- **Transitions**: Smooth animations with consistent timing
|
|
|
|
4. **UX Features** (`ux_features.css`):
|
|
- Drag-and-drop indicators
|
|
- Bulk selection styling
|
|
- Keyboard focus indicators
|
|
- Touch gesture feedback
|
|
- Mobile responsive utilities
|
|
- High contrast mode support (`@media (prefers-contrast: high)`)
|
|
- Screen reader utilities (`.sr-only`)
|
|
- Window control components
|
|
|
|
#### CSS Variables
|
|
|
|
**Color System**:
|
|
|
|
```css
|
|
/* Light Theme */
|
|
--color-bg-primary: #ffffff
|
|
--color-accent: #0078d4
|
|
--color-text-primary: #323130
|
|
|
|
/* Dark Theme */
|
|
--color-bg-primary-dark: #202020
|
|
--color-accent-dark: #60cdff
|
|
--color-text-primary-dark: #ffffff
|
|
```
|
|
|
|
**Spacing & Typography**:
|
|
|
|
```css
|
|
--spacing-sm: 8px
|
|
--spacing-md: 12px
|
|
--spacing-lg: 16px
|
|
--font-size-body: 14px
|
|
--font-size-title: 20px
|
|
```
|
|
|
|
#### Template CSS References
|
|
|
|
All HTML templates correctly reference CSS files:
|
|
|
|
- Index page: Includes both `styles.css` and `ux_features.css`
|
|
- Other pages: Include `styles.css`
|
|
- All use absolute paths: `/static/css/styles.css`
|
|
|
|
#### Responsive Design
|
|
|
|
- Mobile-first approach with breakpoints
|
|
- Media queries for tablet and desktop layouts
|
|
- Touch-friendly interface elements
|
|
- Adaptive typography and spacing
|
|
|
|
#### Accessibility Features
|
|
|
|
- WCAG-compliant color contrast
|
|
- High contrast mode support
|
|
- Screen reader utilities
|
|
- Keyboard navigation styling
|
|
- Focus indicators
|
|
- Reduced motion support
|
|
|
|
#### Testing
|
|
|
|
Comprehensive test suite in `tests/unit/test_static_files.py`:
|
|
|
|
- CSS file accessibility tests
|
|
- Theme support verification
|
|
- Responsive design validation
|
|
- Accessibility feature checks
|
|
- Content integrity validation
|
|
- Path correctness verification
|
|
|
|
All 17 CSS integration tests passing.
|
|
|
|
### Route Controller Refactoring (October 2025)
|
|
|
|
Restructured the FastAPI application to use a controller-based architecture for better code organization and maintainability.
|
|
|
|
#### Changes Made
|
|
|
|
1. **Created Controller Structure**:
|
|
|
|
- `src/server/controllers/` - New directory for route controllers
|
|
- `src/server/controllers/__init__.py` - Controllers package initialization
|
|
- `src/server/controllers/health_controller.py` - Health check endpoints
|
|
- `src/server/controllers/page_controller.py` - HTML page routes
|
|
- `src/server/controllers/error_controller.py` - Error handling controllers
|
|
|
|
2. **Shared Template Configuration**:
|
|
|
|
- `src/server/utils/templates.py` - Centralized Jinja2 template configuration
|
|
- Fixed template path resolution for proper template loading
|
|
|
|
3. **Main Application Updates**:
|
|
|
|
- `src/server/fastapi_app.py` - Refactored to use controller routers
|
|
- Removed direct route definitions from main file
|
|
- Added router inclusion using `app.include_router()`
|
|
- Simplified error handlers to delegate to controller functions
|
|
|
|
4. **Fixed Import Issues**:
|
|
- Resolved circular import in `src/core/__init__.py`
|
|
- Removed non-existent `application` module import
|
|
|
|
#### Controller Architecture
|
|
|
|
### Anime Service Notes
|
|
|
|
- The new `anime_service` runs the existing blocking `SeriesApp` inside a
|
|
threadpool (via ThreadPoolExecutor). This keeps the FastAPI event loop
|
|
responsive while leveraging the existing core logic.
|
|
- A small in-process LRU cache is used for the frequently-read "missing
|
|
episodes" list to reduce IO; cache invalidation happens after a rescan.
|
|
- For multi-worker or multi-host deployments, move cache/state to a shared
|
|
store (Redis) and ensure the threadpool sizing matches the worker's CPU
|
|
and IO profile.
|
|
|
|
**Health Controller** (`health_controller.py`):
|
|
|
|
```python
|
|
router = APIRouter(prefix="/health", tags=["health"])
|
|
@router.get("") - Health check endpoint
|
|
```
|
|
|
|
**Page Controller** (`page_controller.py`):
|
|
|
|
```python
|
|
router = APIRouter(tags=["pages"])
|
|
@router.get("/") - Main application page
|
|
@router.get("/setup") - Setup page
|
|
@router.get("/login") - Login page
|
|
@router.get("/queue") - Download queue page
|
|
```
|
|
|
|
**Error Controller** (`error_controller.py`):
|
|
|
|
```python
|
|
async def not_found_handler() - Custom 404 error handling
|
|
async def server_error_handler() - Custom 500 error handling
|
|
```
|
|
|
|
#### Benefits of the New Structure
|
|
|
|
- **Separation of Concerns**: Each controller handles specific functionality
|
|
- **Modularity**: Easy to add new controllers and routes
|
|
- **Testability**: Controllers can be tested independently
|
|
- **Maintainability**: Cleaner code organization and easier debugging
|
|
- **Scalability**: Simple to extend with new features
|
|
|
|
#### Verified Working Endpoints
|
|
|
|
All endpoints tested and confirmed working:
|
|
|
|
- Health: `/health` → Returns `{"status": "healthy", ...}`
|
|
- Root: `/` → Serves main application page
|
|
- Setup: `/setup` → Serves setup page
|
|
- Auth API: `/api/auth/*` → Endpoints for setup, login, logout and status (JWT-based)
|
|
- Login: `/login` → Serves login page
|
|
- Queue: `/queue` → Serves download queue page
|
|
|
|
#### File Structure After Refactoring
|
|
|
|
```
|
|
src/server/
|
|
├── fastapi_app.py # Main FastAPI application (simplified)
|
|
├── controllers/ # NEW: Route controllers
|
|
│ ├── __init__.py # Controllers package
|
|
├── utils/
|
|
│ ├── dependencies.py # Existing dependency injection
|
|
│ └── templates.py # NEW: Shared Jinja2 template config
|
|
└── web/ # Existing frontend assets
|
|
├── templates/ # HTML templates
|
|
└── static/ # CSS, JS, images
|
|
```
|
|
|
|
### Authentication Middleware (October 2025)
|
|
|
|
An authentication middleware component was added to the FastAPI
|
|
application to centralize token parsing and provide lightweight
|
|
protection of authentication endpoints:
|
|
|
|
- `src/server/middleware/auth.py` implements:
|
|
- Bearer JWT parsing and session attachment to `request.state.session`
|
|
- A simple per-IP in-memory rate limiter applied to
|
|
`/api/auth/login` and `/api/auth/setup` (default 5 requests/minute)
|
|
|
|
Notes:
|
|
|
|
- This is intentionally simple and designed for single-process
|
|
deployments. For production use across multiple workers or hosts,
|
|
replace the in-memory limiter with a distributed store (e.g. Redis)
|
|
and add a persistent token revocation list if needed.
|
|
|
|
### API Models and Contracts
|
|
|
|
- Pydantic models living in `src/server/models/` define the canonical
|
|
API contracts used by FastAPI endpoints. These models are intentionally
|
|
lightweight and focused on serialization, validation, and OpenAPI
|
|
documentation generation.
|
|
- Keep models stable: changes to model shapes are breaking changes for
|
|
clients. Bump API versioning or provide migration layers when altering
|
|
public response fields.
|
|
- Infrastructure considerations: ensure the deployment environment has
|
|
required libraries (e.g., `pydantic`) installed and that schema
|
|
validation errors are logged to the centralized logging system. For
|
|
high-throughput routes, consider response model caching at the
|
|
application or reverse-proxy layer.
|
|
|
|
### WebSocket Real-time Communication (October 2025)
|
|
|
|
A comprehensive WebSocket infrastructure was implemented to provide real-time
|
|
updates for downloads, queue status, and system events:
|
|
|
|
- **File**: `src/server/services/websocket_service.py`
|
|
- **Models**: `src/server/models/websocket.py`
|
|
- **Endpoint**: `ws://host:port/ws/connect`
|
|
|
|
#### WebSocket Service Architecture
|
|
|
|
- **ConnectionManager**: Low-level connection lifecycle management
|
|
|
|
- Connection registry with unique connection IDs
|
|
- Room-based messaging for topic subscriptions
|
|
- Automatic connection cleanup and health monitoring
|
|
- Thread-safe operations with asyncio locks
|
|
|
|
- **WebSocketService**: High-level application messaging
|
|
- Convenient interface for broadcasting application events
|
|
- Pre-defined message types for downloads, queue, and system events
|
|
- Singleton pattern via `get_websocket_service()` factory
|
|
|
|
#### Supported Message Types
|
|
|
|
- **Download Events**: `download_progress`, `download_complete`, `download_failed`
|
|
- **Queue Events**: `queue_status`, `queue_started`, `queue_stopped`, `queue_paused`, `queue_resumed`
|
|
- **System Events**: `system_info`, `system_warning`, `system_error`
|
|
- **Connection**: `connected`, `ping`, `pong`, `error`
|
|
|
|
#### Room-Based Messaging
|
|
|
|
Clients can subscribe to specific topics (rooms) to receive targeted updates:
|
|
|
|
- `downloads` room: All download-related events
|
|
- Custom rooms: Can be added for specific features
|
|
|
|
#### Integration with Download Service
|
|
|
|
- Download service automatically broadcasts progress updates via WebSocket
|
|
- Broadcast callback registered during service initialization
|
|
- Updates sent to all clients subscribed to the `downloads` room
|
|
- No blocking of download operations (async broadcast)
|
|
|
|
#### Client Connection Flow
|
|
|
|
1. Client connects to `/ws/connect` endpoint
|
|
2. Server assigns unique connection ID and sends confirmation
|
|
3. Client joins rooms (e.g., `{"action": "join", "room": "downloads"}`)
|
|
4. Server broadcasts updates to subscribed rooms
|
|
5. Client disconnects (automatic cleanup)
|
|
|
|
#### Infrastructure Notes
|
|
|
|
- **Single-process**: Current implementation uses in-memory connection storage
|
|
- **Production**: For multi-worker/multi-host deployments:
|
|
- Move connection registry to Redis or similar shared store
|
|
- Implement pub/sub for cross-process message broadcasting
|
|
- Add connection persistence for recovery after restarts
|
|
- **Monitoring**: WebSocket status available at `/ws/status` endpoint
|
|
- **Security**: Optional authentication via JWT (user_id tracking)
|
|
- **Testing**: Comprehensive unit tests in `tests/unit/test_websocket_service.py`
|
|
|
|
### Download Queue Models
|
|
|
|
- Download queue models in `src/server/models/download.py` define the data
|
|
structures for the download queue system.
|
|
- Key models include:
|
|
- `DownloadItem`: Represents a single queued download with metadata,
|
|
progress tracking, and error information
|
|
- `QueueStatus`: Overall queue state with active, pending, completed,
|
|
and failed downloads
|
|
- `QueueStats`: Aggregated statistics for monitoring queue performance
|
|
- `DownloadProgress`: Real-time progress information (percent, speed,
|
|
ETA)
|
|
- `DownloadRequest`/`DownloadResponse`: API request/response contracts
|
|
- Models enforce validation constraints (e.g., positive episode numbers,
|
|
progress percentage 0-100, non-negative retry counts) and provide
|
|
clean JSON serialization for API endpoints and WebSocket updates.
|
|
|
|
### Download Queue Service
|
|
|
|
- The download service (`src/server/services/download_service.py`) manages
|
|
the complete lifecycle of anime episode downloads.
|
|
- Core features:
|
|
- **Priority-based Queue**: Items added with HIGH priority are processed
|
|
first, NORMAL and LOW follow in FIFO order
|
|
- **Concurrent Processing**: Configurable max concurrent downloads (default 2)
|
|
to optimize bandwidth usage
|
|
- **Persistence**: Queue state is automatically saved to
|
|
`data/download_queue.json` and recovered on service restart
|
|
- **Retry Logic**: Failed downloads are automatically retried up to a
|
|
configurable limit (default 3 attempts) with exponential backoff
|
|
- **Progress Tracking**: Real-time download progress with speed,
|
|
percentage, and ETA calculations
|
|
- **WebSocket Integration**: Broadcasts queue updates, progress, and
|
|
completion/failure events to connected clients
|
|
- Operations:
|
|
- `add_to_queue()`: Add episodes to download queue with priority
|
|
- `remove_from_queue()`: Cancel pending or active downloads
|
|
- `reorder_queue()`: Manually adjust queue order for pending items
|
|
- `pause_queue()`/`resume_queue()`: Control download processing
|
|
- `retry_failed()`: Retry failed downloads with retry count checks
|
|
- `get_queue_status()`: Get complete queue state (active, pending, completed, failed)
|
|
- `get_queue_stats()`: Get aggregated statistics (counts, download size, speed)
|
|
- Infrastructure notes:
|
|
- Service uses ThreadPoolExecutor for concurrent download processing
|
|
- Queue processor runs as async background task with configurable sleep intervals
|
|
- Progress callbacks are executed in threadpool and broadcast via async WebSocket
|
|
- For multi-process deployments, move queue state to shared store (Redis/DB)
|
|
and implement distributed locking for concurrent access control
|
|
- Singleton instance pattern used via `get_download_service()` factory
|
|
- Testing: Comprehensive unit tests in `tests/unit/test_download_service.py`
|
|
cover queue operations, persistence, retry logic, and error handling
|
|
|
|
### Download Queue API Endpoints (October 2025)
|
|
|
|
Implemented comprehensive REST API endpoints for download queue management:
|
|
|
|
- **File**: `src/server/api/download.py`
|
|
- **Router Prefix**: `/api/queue`
|
|
- **Authentication**: All endpoints require JWT authentication via `require_auth` dependency
|
|
|
|
#### Implemented Endpoints
|
|
|
|
1. **GET /api/queue/status** - Retrieve complete queue status
|
|
|
|
- Returns: `QueueStatusResponse` with status and statistics
|
|
- Includes: active downloads, pending items, completed/failed items, queue stats
|
|
|
|
2. **POST /api/queue/add** - Add episodes to download queue
|
|
|
|
- Request: `DownloadRequest` with serie info, episodes, and priority
|
|
- Returns: `DownloadResponse` with added item IDs
|
|
- Validates episode list is non-empty
|
|
- Supports HIGH, NORMAL, and LOW priority levels
|
|
|
|
3. **DELETE /api/queue/{item_id}** - Remove single item from queue
|
|
|
|
- Returns: 204 No Content on success, 404 if item not found
|
|
- Cancels active downloads if necessary
|
|
|
|
4. **DELETE /api/queue/** - Remove multiple items (batch operation)
|
|
|
|
- Request: `QueueOperationRequest` with list of item IDs
|
|
- Returns: 204 No Content (partial success acceptable)
|
|
|
|
5. **POST /api/queue/start** - Start queue processor
|
|
|
|
- Idempotent operation (safe to call multiple times)
|
|
|
|
6. **POST /api/queue/stop** - Stop queue processor
|
|
|
|
- Waits for active downloads to complete (with timeout)
|
|
|
|
7. **POST /api/queue/pause** - Pause queue processing
|
|
|
|
- Active downloads continue, no new downloads start
|
|
|
|
8. **POST /api/queue/resume** - Resume queue processing
|
|
|
|
9. **POST /api/queue/reorder** - Reorder pending queue item
|
|
|
|
- Request: `QueueReorderRequest` with item_id and new_position
|
|
- Returns: 404 if item not in pending queue
|
|
|
|
10. **DELETE /api/queue/completed** - Clear completed items from history
|
|
|
|
- Returns count of cleared items
|
|
|
|
11. **POST /api/queue/retry** - Retry failed downloads
|
|
- Request: `QueueOperationRequest` with item IDs (empty for all)
|
|
- Only retries items under max retry limit
|
|
|
|
#### Dependencies
|
|
|
|
- **get_download_service**: Factory function providing singleton DownloadService instance
|
|
- Automatically initializes AnimeService as dependency
|
|
- Raises 503 if anime directory not configured
|
|
- **get_anime_service**: Factory function providing singleton AnimeService instance
|
|
- Required by DownloadService for anime operations
|
|
- Both dependencies added to `src/server/utils/dependencies.py`
|
|
|
|
#### Error Handling
|
|
|
|
- All endpoints return structured JSON error responses
|
|
- HTTP status codes follow REST conventions (200, 201, 204, 400, 401, 404, 500, 503)
|
|
- Service-level exceptions (DownloadServiceError) mapped to 400 Bad Request
|
|
- Generic exceptions mapped to 500 Internal Server Error
|
|
- Authentication errors return 401 Unauthorized
|
|
|
|
#### Testing
|
|
|
|
- Comprehensive test suite in `tests/api/test_download_endpoints.py`
|
|
- Tests cover:
|
|
- Successful operations for all endpoints
|
|
- Authentication requirements
|
|
- Error conditions (empty lists, not found, service errors)
|
|
- Priority handling
|
|
- Batch operations
|
|
- Uses pytest fixtures for authenticated client and mocked download service
|
|
|
|
#### Integration
|
|
|
|
- Router registered in `src/server/fastapi_app.py` via `app.include_router(download_router)`
|
|
- Follows same patterns as other API routers (auth, anime, config)
|
|
- Full OpenAPI documentation available at `/api/docs`
|
|
|
|
### WebSocket Integration with Core Services (October 2025)
|
|
|
|
Completed comprehensive integration of WebSocket broadcasting with all core services to provide real-time updates for downloads, scans, queue operations, and progress tracking.
|
|
|
|
#### ProgressService
|
|
|
|
**File**: `src/server/services/progress_service.py`
|
|
|
|
A centralized service for tracking and broadcasting real-time progress updates across the application.
|
|
|
|
**Key Features**:
|
|
|
|
- Track multiple concurrent progress operations (downloads, scans, queue changes)
|
|
- Automatic progress percentage calculation
|
|
- Progress lifecycle management (start, update, complete, fail, cancel)
|
|
- WebSocket integration for real-time client updates
|
|
- Progress history with configurable size limit (default: 50 items)
|
|
- Thread-safe operations using asyncio locks
|
|
- Support for progress metadata and custom messages
|
|
|
|
**Progress Types**:
|
|
|
|
- `DOWNLOAD` - File download progress
|
|
- `SCAN` - Library scan progress
|
|
- `QUEUE` - Queue operation progress
|
|
- `SYSTEM` - System-level operations
|
|
- `ERROR` - Error notifications
|
|
|
|
**Progress Statuses**:
|
|
|
|
- `STARTED` - Operation initiated
|
|
- `IN_PROGRESS` - Operation in progress
|
|
- `COMPLETED` - Successfully completed
|
|
- `FAILED` - Operation failed
|
|
- `CANCELLED` - Cancelled by user
|
|
|
|
**Core Methods**:
|
|
|
|
- `start_progress()` - Initialize new progress operation
|
|
- `update_progress()` - Update progress with current/total values
|
|
- `complete_progress()` - Mark operation as completed
|
|
- `fail_progress()` - Mark operation as failed
|
|
- `cancel_progress()` - Cancel ongoing operation
|
|
- `get_progress()` - Retrieve progress by ID
|
|
- `get_all_active_progress()` - Get all active operations (optionally filtered by type)
|
|
|
|
**Broadcasting**:
|
|
|
|
- Integrates with WebSocketService via callback
|
|
- Broadcasts to room-specific channels (e.g., `download_progress`, `scan_progress`)
|
|
- Configurable broadcast throttling (only on significant changes >1% or forced)
|
|
- Automatic progress state serialization to JSON
|
|
|
|
**Singleton Pattern**:
|
|
|
|
- Global instance via `get_progress_service()` factory
|
|
- Initialized during application startup with WebSocket callback
|
|
|
|
#### Integration with Services
|
|
|
|
**DownloadService Integration**:
|
|
|
|
- Progress tracking for each download item
|
|
- Real-time progress updates during file download
|
|
- Automatic completion/failure notifications
|
|
- Progress metadata includes speed, ETA, downloaded bytes
|
|
|
|
**AnimeService Integration**:
|
|
|
|
- Progress tracking for library scans
|
|
- Scan progress with current/total file counts
|
|
- Scan completion with statistics
|
|
- Error notifications on scan failures
|
|
|
|
#### WebSocket Message Models
|
|
|
|
**File**: `src/server/models/websocket.py`
|
|
|
|
Added progress-specific message models:
|
|
|
|
- `ScanProgressMessage` - Scan progress updates
|
|
- `ScanCompleteMessage` - Scan completion notification
|
|
- `ScanFailedMessage` - Scan failure notification
|
|
- `ErrorNotificationMessage` - Critical error notifications
|
|
- `ProgressUpdateMessage` - Generic progress updates
|
|
|
|
**WebSocket Message Types**:
|
|
|
|
- `SCAN_PROGRESS` - Scan progress updates
|
|
- `SCAN_COMPLETE` - Scan completion
|
|
- `SCAN_FAILED` - Scan failure
|
|
- Extended existing types for downloads and queue updates
|
|
|
|
#### WebSocket Rooms
|
|
|
|
Clients can subscribe to specific progress channels:
|
|
|
|
- `download_progress` - Download progress updates
|
|
- `scan_progress` - Library scan updates
|
|
- `queue_progress` - Queue operation updates
|
|
- `system_progress` - System-level updates
|
|
|
|
Room subscription via client messages:
|
|
|
|
```json
|
|
{
|
|
"action": "join",
|
|
"room": "download_progress"
|
|
}
|
|
```
|
|
|
|
#### Application Startup
|
|
|
|
**File**: `src/server/fastapi_app.py`
|
|
|
|
Progress service initialized on application startup:
|
|
|
|
1. Get ProgressService singleton instance
|
|
2. Get WebSocketService singleton instance
|
|
3. Register broadcast callback to link progress updates with WebSocket
|
|
4. Callback broadcasts progress messages to appropriate rooms
|
|
|
|
#### Testing
|
|
|
|
**File**: `tests/unit/test_progress_service.py`
|
|
|
|
Comprehensive test coverage including:
|
|
|
|
- Progress lifecycle operations (start, update, complete, fail, cancel)
|
|
- Percentage calculation accuracy
|
|
- History management and size limits
|
|
- Broadcast callback invocation
|
|
- Concurrent progress operations
|
|
- Metadata handling
|
|
- Error conditions and edge cases
|
|
|
|
#### Architecture Benefits
|
|
|
|
- **Decoupling**: ProgressService decouples progress tracking from WebSocket broadcasting
|
|
- **Reusability**: Single service used across all application components
|
|
- **Scalability**: Supports multiple concurrent operations efficiently
|
|
- **Observability**: Centralized progress tracking simplifies monitoring
|
|
- **Real-time UX**: Instant feedback to users on all long-running operations
|
|
|
|
#### Future Enhancements
|
|
|
|
- Persistent progress history (database storage)
|
|
- Progress rate calculation and trend analysis
|
|
- Multi-process progress synchronization (Redis/shared store)
|
|
- Progress event hooks for custom actions
|
|
- Client-side progress resumption after reconnection
|
|
|
|
### Core Services WebSocket Integration (October 2025)
|
|
|
|
Completed comprehensive integration of WebSocket broadcasting with all core services (DownloadService, AnimeService, ProgressService) to provide real-time updates to connected clients.
|
|
|
|
#### DownloadService WebSocket Integration
|
|
|
|
**File**: `src/server/services/download_service.py`
|
|
|
|
The download service broadcasts real-time updates for all queue and download operations:
|
|
|
|
**Download Progress Broadcasting**:
|
|
|
|
- `download_progress` - Real-time progress updates during download
|
|
- Includes: download_id, serie_name, season, episode, progress data (percent, speed, ETA)
|
|
- Sent via ProgressService which broadcasts to `download_progress` room
|
|
- Progress callback created for each download item with metadata tracking
|
|
|
|
**Download Completion/Failure Broadcasting**:
|
|
|
|
- `download_complete` - Successful download completion
|
|
- Includes: download_id, serie_name, season, episode, downloaded_mb
|
|
- Broadcast to `downloads` room
|
|
- `download_failed` - Download failure notification
|
|
- Includes: download_id, serie_name, season, episode, error, retry_count
|
|
- Broadcast to `downloads` room
|
|
|
|
**Queue Operations Broadcasting**:
|
|
All queue operations broadcast `queue_status` messages with current queue state:
|
|
|
|
- `items_added` - Items added to queue
|
|
- Data: added_ids, queue_status (complete queue state)
|
|
- `items_removed` - Items removed/cancelled
|
|
- Data: removed_ids, queue_status
|
|
- `queue_reordered` - Queue order changed
|
|
- Data: item_id, new_position, queue_status
|
|
- `items_retried` - Failed items retried
|
|
- Data: retried_ids, queue_status
|
|
- `completed_cleared` - Completed items cleared
|
|
- Data: cleared_count, queue_status
|
|
|
|
**Queue Control Broadcasting**:
|
|
|
|
- `queue_started` - Queue processor started
|
|
- Data: is_running=True, queue_status
|
|
- `queue_stopped` - Queue processor stopped
|
|
- Data: is_running=False, queue_status
|
|
- `queue_paused` - Queue processing paused
|
|
- Data: is_paused=True, queue_status
|
|
- `queue_resumed` - Queue processing resumed
|
|
- Data: is_paused=False, queue_status
|
|
|
|
**Broadcast Callback Setup**:
|
|
The download service broadcast callback is registered during dependency injection in `src/server/utils/dependencies.py`:
|
|
|
|
- Maps update types to WebSocket service methods
|
|
- Routes download_progress, download_complete, download_failed to appropriate rooms
|
|
- All queue operations broadcast complete queue status for client synchronization
|
|
|
|
#### AnimeService WebSocket Integration
|
|
|
|
**File**: `src/server/services/anime_service.py`
|
|
|
|
The anime service integrates with ProgressService for library scan operations:
|
|
|
|
**Scan Progress Broadcasting**:
|
|
|
|
- Scan operations use ProgressService for progress tracking
|
|
- Progress updates broadcast to `scan_progress` room
|
|
- Lifecycle events:
|
|
- `started` - Scan initialization
|
|
- `in_progress` - Ongoing scan with current/total file counts
|
|
- `completed` - Successful scan completion
|
|
- `failed` - Scan failure with error message
|
|
|
|
**Scan Implementation**:
|
|
|
|
- `rescan()` method wraps SeriesApp.ReScan with progress tracking
|
|
- Progress callback executed in threadpool updates ProgressService
|
|
- ProgressService automatically broadcasts to WebSocket clients
|
|
- Cache invalidation on successful scan completion
|
|
|
|
#### ProgressService WebSocket Integration
|
|
|
|
**File**: `src/server/services/progress_service.py`
|
|
|
|
Central service for tracking and broadcasting all progress operations:
|
|
|
|
**Progress Types**:
|
|
|
|
- `DOWNLOAD` - File download progress
|
|
- `SCAN` - Library scan progress
|
|
- `QUEUE` - Queue operation progress
|
|
- `SYSTEM` - System-level operations
|
|
- `ERROR` - Error notifications
|
|
|
|
**Progress Lifecycle**:
|
|
|
|
1. `start_progress()` - Initialize progress operation
|
|
- Broadcasts to room: `{progress_type}_progress`
|
|
2. `update_progress()` - Update progress values
|
|
- Calculates percentage automatically
|
|
- Broadcasts only on significant changes (>1% or forced)
|
|
3. `complete_progress()` - Mark operation complete
|
|
- Sets progress to 100%
|
|
- Moves to history
|
|
- Broadcasts completion
|
|
4. `fail_progress()` - Mark operation failed
|
|
- Captures error message
|
|
- Moves to history
|
|
- Broadcasts failure
|
|
|
|
**Broadcast Callback**:
|
|
|
|
- Callback registered during application startup in `src/server/fastapi_app.py`
|
|
- Links ProgressService to WebSocketService.manager.broadcast_to_room
|
|
- All progress updates automatically broadcast to appropriate rooms
|
|
|
|
#### WebSocket Room Structure
|
|
|
|
Clients subscribe to specific rooms to receive targeted updates:
|
|
|
|
**Room Types**:
|
|
|
|
- `downloads` - All download-related events (complete, failed, queue status)
|
|
- `download_progress` - Real-time download progress updates
|
|
- `scan_progress` - Library scan progress updates
|
|
- `queue_progress` - Queue operation progress (future use)
|
|
- `system_progress` - System-level progress (future use)
|
|
|
|
**Room Subscription**:
|
|
Clients join rooms by sending WebSocket messages:
|
|
|
|
```json
|
|
{
|
|
"action": "join",
|
|
"room": "download_progress"
|
|
}
|
|
```
|
|
|
|
#### Message Format
|
|
|
|
All WebSocket messages follow a consistent structure:
|
|
|
|
```json
|
|
{
|
|
"type": "download_progress" | "download_complete" | "queue_status" | etc.,
|
|
"timestamp": "2025-10-17T12:34:56.789Z",
|
|
"data": {
|
|
// Message-specific data
|
|
}
|
|
}
|
|
```
|
|
|
|
**Example: Download Progress**
|
|
|
|
```json
|
|
{
|
|
"type": "download_progress",
|
|
"timestamp": "2025-10-17T12:34:56.789Z",
|
|
"data": {
|
|
"download_id": "abc123",
|
|
"serie_name": "Attack on Titan",
|
|
"season": 1,
|
|
"episode": 5,
|
|
"progress": {
|
|
"percent": 45.2,
|
|
"downloaded_mb": 226.0,
|
|
"total_mb": 500.0,
|
|
"speed_mbps": 2.5,
|
|
"eta_seconds": 120
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**Example: Queue Status**
|
|
|
|
```json
|
|
{
|
|
"type": "queue_status",
|
|
"timestamp": "2025-10-17T12:34:56.789Z",
|
|
"data": {
|
|
"action": "items_added",
|
|
"added_ids": ["item1", "item2"],
|
|
"queue_status": {
|
|
"is_running": true,
|
|
"is_paused": false,
|
|
"active_downloads": [...],
|
|
"pending_queue": [...],
|
|
"completed_downloads": [...],
|
|
"failed_downloads": [...]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### Integration Testing
|
|
|
|
**File**: `tests/integration/test_websocket_integration.py`
|
|
|
|
Comprehensive integration tests verify WebSocket broadcasting:
|
|
|
|
**Test Coverage**:
|
|
|
|
- Download progress broadcasts during active downloads
|
|
- Queue operation broadcasts (add, remove, reorder, clear, retry)
|
|
- Queue control broadcasts (start, stop, pause, resume)
|
|
- Scan progress broadcasts (start, update, complete, fail)
|
|
- Progress lifecycle broadcasts for all operation types
|
|
- End-to-end flow with multiple services broadcasting
|
|
|
|
**Test Strategy**:
|
|
|
|
- Mock broadcast callbacks to capture emitted messages
|
|
- Verify message types, data structure, and content
|
|
- Test both successful and failure scenarios
|
|
- Verify proper room routing for different message types
|
|
|
|
#### Architecture Benefits
|
|
|
|
**Decoupling**:
|
|
|
|
- Services use generic broadcast callbacks without WebSocket dependencies
|
|
- ProgressService provides abstraction layer for progress tracking
|
|
- Easy to swap WebSocket implementation or add additional broadcast targets
|
|
|
|
**Consistency**:
|
|
|
|
- All services follow same broadcast patterns
|
|
- Standardized message formats across application
|
|
- Unified progress tracking via ProgressService
|
|
|
|
**Real-time UX**:
|
|
|
|
- Instant feedback on all long-running operations
|
|
- Live queue status updates
|
|
- Progress bars update smoothly without polling
|
|
- Error notifications delivered immediately
|
|
|
|
**Scalability**:
|
|
|
|
- Room-based messaging enables targeted updates
|
|
- Multiple concurrent operations supported
|
|
- Easy to add new progress types and message formats
|
|
|
|
#### Production Considerations
|
|
|
|
**Single-Process Deployment** (Current):
|
|
|
|
- In-memory connection registry in WebSocketService
|
|
- Works perfectly for single-worker deployments
|
|
- No additional infrastructure required
|
|
|
|
**Multi-Process/Multi-Host Deployment** (Future):
|
|
|
|
- Move connection registry to Redis or similar shared store
|
|
- Implement pub/sub for cross-process message broadcasting
|
|
- Add connection persistence for recovery after restarts
|
|
- Consider using sticky sessions or connection migration
|
|
|
|
**Performance**:
|
|
|
|
- Progress updates throttled to >1% changes to reduce message volume
|
|
- Broadcast operations are fire-and-forget (non-blocking)
|
|
- Failed connections automatically cleaned up
|
|
- Message serialization cached where possible
|
|
|
|
**Monitoring**:
|
|
|
|
- Structured logging for all broadcast operations
|
|
- WebSocket status available at `/ws/status` endpoint
|
|
- Connection count and room membership tracking
|
|
- Error tracking for failed broadcasts
|
|
|
|
### Frontend Authentication Integration (October 2025)
|
|
|
|
Completed JWT-based authentication integration between frontend and backend.
|
|
|
|
#### Authentication Token Storage
|
|
|
|
**Files Modified:**
|
|
|
|
- `src/server/web/templates/login.html` - Store JWT token after successful login
|
|
- `src/server/web/templates/setup.html` - Redirect to login after setup completion
|
|
- `src/server/web/static/js/app.js` - Include Bearer token in all authenticated requests
|
|
- `src/server/web/static/js/queue.js` - Include Bearer token in queue API calls
|
|
|
|
**Implementation:**
|
|
|
|
- JWT tokens stored in `localStorage` after successful login
|
|
- Token expiry stored in `localStorage` for client-side validation
|
|
- `Authorization: Bearer <token>` header included in all authenticated requests
|
|
- Automatic redirect to `/login` on 401 Unauthorized responses
|
|
- Token cleared from `localStorage` on logout
|
|
|
|
**Key Functions Updated:**
|
|
|
|
- `makeAuthenticatedRequest()` in both `app.js` and `queue.js`
|
|
- `checkAuthentication()` to verify token and redirect if missing/invalid
|
|
- `logout()` to clear token and redirect to login
|
|
|
|
### Frontend API Endpoint Updates (October 2025)
|
|
|
|
Updated frontend JavaScript to match new backend API structure.
|
|
|
|
**Queue Management API Changes:**
|
|
|
|
- `/api/queue/clear` → `/api/queue/completed` for clearing completed downloads
|
|
- `/api/queue/remove` → `/api/queue/{item_id}` (DELETE) for single item removal
|
|
- `/api/queue/retry` payload changed to `{item_ids: []}` array format
|
|
- `/api/download/pause` → `/api/queue/pause`
|
|
- `/api/download/resume` → `/api/queue/resume`
|
|
- `/api/download/cancel` → `/api/queue/stop`
|
|
|
|
**Response Format Changes:**
|
|
|
|
- Login returns `{access_token, token_type, expires_at}` instead of `{status: 'success'}`
|
|
- Setup returns `{status: 'ok'}` instead of `{status: 'success', redirect_url}`
|
|
- Logout returns `{status: 'ok'}` instead of `{status: 'success'}`
|
|
- Queue operations return structured responses with counts (e.g., `{cleared_count, retried_count}`)
|
|
|
|
### Frontend WebSocket Integration (October 2025)
|
|
|
|
WebSocket integration previously completed and verified functional.
|
|
|
|
#### Native WebSocket Implementation
|
|
|
|
**Files:**
|
|
|
|
- `src/server/web/static/js/websocket_client.js` - Native WebSocket wrapper
|
|
- Templates already updated to use `websocket_client.js` instead of Socket.IO
|
|
|
|
**Event Compatibility:**
|
|
|
|
- Dual event handlers in place for backward compatibility
|
|
- Old events: `scan_completed`, `scan_error`, `download_completed`, `download_error`
|
|
- New events: `scan_complete`, `scan_failed`, `download_complete`, `download_failed`
|
|
- Both event types supported simultaneously
|
|
|
|
**Room Subscriptions:**
|
|
|
|
- `downloads` - Download completion, failures, queue status
|
|
- `download_progress` - Real-time download progress updates
|
|
- `scan_progress` - Library scan progress updates
|
|
|
|
### Frontend Integration Testing (October 2025)
|
|
|
|
Created smoke tests to verify frontend-backend integration.
|
|
|
|
**Test File:** `tests/integration/test_frontend_integration_smoke.py`
|
|
|
|
**Tests:**
|
|
|
|
- JWT token format verification (access_token, token_type, expires_at)
|
|
- Bearer token authentication on protected endpoints
|
|
- 401 responses for requests without valid tokens
|
|
|
|
**Test Results:**
|
|
|
|
- Basic authentication flow: ✅ PASSING
|
|
- Token validation: Functional with rate limiting considerations
|
|
|
|
### Frontend Integration (October 2025)
|
|
|
|
Completed integration of existing frontend JavaScript with the new FastAPI backend and native WebSocket implementation.
|
|
|
|
#### Native WebSocket Client
|
|
|
|
**File**: `src/server/web/static/js/websocket_client.js`
|
|
|
|
Created a Socket.IO-compatible wrapper using native WebSocket API:
|
|
|
|
**Features**:
|
|
|
|
- Socket.IO-style `.on()` and `.emit()` methods for compatibility
|
|
- Automatic reconnection with exponential backoff (max 5 attempts)
|
|
- Room-based subscriptions via `.join()` and `.leave()` methods
|
|
- Message queueing during disconnection
|
|
- Proper connection lifecycle management
|
|
|
|
**Usage**:
|
|
|
|
```javascript
|
|
const socket = io(); // Creates WebSocket to ws://host:port/ws/connect
|
|
socket.join('download_progress'); // Subscribe to room
|
|
socket.on('download_progress', (data) => { ... }); // Handle messages
|
|
```
|
|
|
|
#### WebSocket Message Format
|
|
|
|
All WebSocket messages follow a structured format:
|
|
|
|
```json
|
|
{
|
|
"type": "message_type",
|
|
"timestamp": "2025-10-17T12:34:56.789Z",
|
|
"data": {}
|
|
}
|
|
```
|
|
|
|
**Event Mapping** (Old Socket.IO → New WebSocket):
|
|
|
|
- `scan_completed` / `scan_complete` → Scan finished
|
|
- `scan_error` / `scan_failed` → Scan error
|
|
- `download_completed` / `download_complete` → Download finished
|
|
- `download_error` / `download_failed` → Download error
|
|
- `queue_updated` / `queue_status` → Queue state changes
|
|
- `queue_started`, `queue_stopped`, `queue_paused`, `queue_resumed` → Queue control events
|
|
|
|
**Rooms**:
|
|
|
|
- `scan_progress` - Library scan updates
|
|
- `download_progress` - Real-time download progress
|
|
- `downloads` - Download completion, failures, queue status
|
|
|
|
#### JavaScript Updates
|
|
|
|
**app.js**:
|
|
|
|
- Added room subscriptions on WebSocket connect
|
|
- Added dual event handlers for old and new message types
|
|
- `connected` event handler for initial WebSocket confirmation
|
|
- Handles both `scan_complete` and legacy `scan_completed` events
|
|
- Handles both `scan_failed` and legacy `scan_error` events
|
|
|
|
**queue.js**:
|
|
|
|
- Added room subscriptions on WebSocket connect
|
|
- Added dual event handlers for backward compatibility
|
|
- Handles both `queue_status` and legacy `queue_updated` events
|
|
- Handles both `download_complete` and legacy `download_completed` events
|
|
- Handles both `download_failed` and legacy `download_error` events
|
|
- Added handlers for `queue_started`, `queue_stopped`, `queue_paused`, `queue_resumed`
|
|
|
|
#### Template Updates
|
|
|
|
**Modified Templates**:
|
|
|
|
- `src/server/web/templates/index.html` - Replaced Socket.IO CDN with websocket_client.js
|
|
- `src/server/web/templates/queue.html` - Replaced Socket.IO CDN with websocket_client.js
|
|
|
|
**Benefits**:
|
|
|
|
- No external CDN dependency (Socket.IO)
|
|
- Native browser WebSocket API (faster, smaller)
|
|
- Full compatibility with existing JavaScript code
|
|
- Proper integration with backend WebSocket service
|
|
|
|
#### API Router Registration
|
|
|
|
**fastapi_app.py**:
|
|
|
|
- ✅ Added `anime_router` import and registration
|
|
- All routers now properly included:
|
|
- `health_router` - Health checks
|
|
- `page_router` - HTML pages
|
|
- `auth_router` - Authentication (JWT-based)
|
|
- `anime_router` - Anime management (NEW)
|
|
- `download_router` - Download queue
|
|
- `websocket_router` - WebSocket connection
|
|
|
|
**Anime Endpoints**:
|
|
|
|
- `GET /api/v1/anime` - List anime with missing episodes
|
|
- `POST /api/v1/anime/rescan` - Trigger library rescan
|
|
- `POST /api/v1/anime/search` - Search for anime
|
|
- `GET /api/v1/anime/{anime_id}` - Get anime details
|
|
|
|
#### Authentication Integration
|
|
|
|
JavaScript uses JWT tokens from localStorage for authenticated requests:
|
|
|
|
- Token stored after successful login
|
|
- Included in `Authorization: Bearer <token>` header
|
|
- Automatic redirect to `/login` on 401 responses
|
|
- Compatible with backend AuthMiddleware
|
|
|
|
#### Testing
|
|
|
|
**Verified Functionality**:
|
|
|
|
- ✅ WebSocket client initialization and connection
|
|
- ✅ Room subscriptions and message routing
|
|
- ✅ Event handler compatibility (old and new message types)
|
|
- ✅ Anime API endpoints (passed pytest tests)
|
|
- ✅ Download queue API endpoints (existing tests)
|
|
- ✅ Frontend integration tests (comprehensive)
|
|
|
|
**Frontend Integration Test Suite**: `tests/frontend/test_existing_ui_integration.py`
|
|
|
|
**Coverage**:
|
|
|
|
- Authentication flow with JWT tokens
|
|
- API endpoint compatibility (anime, download, config)
|
|
- WebSocket real-time updates
|
|
- Data format validation
|
|
- Error handling (401, 400/422)
|
|
- Multiple client broadcast scenarios
|
|
|
|
**Test Classes**:
|
|
|
|
- `TestFrontendAuthentication`: JWT login, logout, auth status
|
|
- `TestFrontendAnimeAPI`: Anime list, search, rescan operations
|
|
- `TestFrontendDownloadAPI`: Queue management, start/pause/stop
|
|
- `TestFrontendWebSocketIntegration`: Connection, broadcasts, progress
|
|
- `TestFrontendConfigAPI`: Configuration get/update
|
|
- `TestFrontendJavaScriptIntegration`: Bearer token patterns
|
|
- `TestFrontendErrorHandling`: JSON errors, validation
|
|
- `TestFrontendRealTimeUpdates`: Download events, notifications
|
|
- `TestFrontendDataFormats`: Response format validation
|
|
|
|
**Test Commands**:
|
|
|
|
```bash
|
|
# Run all frontend integration tests
|
|
conda run -n AniWorld python -m pytest tests/frontend/test_existing_ui_integration.py -v
|
|
|
|
# Run specific test class
|
|
conda run -n AniWorld python -m pytest tests/frontend/test_existing_ui_integration.py::TestFrontendAuthentication -v
|
|
|
|
# Run all API tests
|
|
conda run -n AniWorld python -m pytest tests/api/ -v
|
|
|
|
# Run all tests
|
|
conda run -n AniWorld python -m pytest tests/ -v
|
|
```
|
|
|
|
**Note**: Some tests require auth service state isolation. The test suite uses fixtures to reset authentication state before each test. If you encounter auth-related test failures, they may be due to shared state across test runs.
|
|
|
|
#### Known Limitations
|
|
|
|
**Legacy Events**: Some Socket.IO events don't have backend implementations:
|
|
|
|
- `scheduled_rescan_*` events
|
|
- `auto_download_*` events
|
|
- `download_episode_update` event
|
|
- `download_series_completed` event
|
|
|
|
**Solution**: These events are kept in JavaScript for future implementation or can be removed if not needed.
|
|
|
|
**Configuration Endpoints**: Many config-related features in app.js don't have backend endpoints:
|
|
|
|
- Scheduler configuration
|
|
- Logging configuration
|
|
- Advanced configuration
|
|
- Config backups
|
|
|
|
**Solution**: These can be implemented later or the UI features removed.
|
|
|
|
#### Documentation
|
|
|
|
**Detailed Documentation**: See `FRONTEND_INTEGRATION.md` for:
|
|
|
|
- Complete API endpoint mapping
|
|
- WebSocket message format details
|
|
- Migration guide for developers
|
|
- Testing strategies
|
|
- Integration patterns
|