lukas.pupkalipinski/Aniworld
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: queue issue

Browse Source
This commit is contained in:
Lukas
2026-07-31 07:33:12 +02:00
parent e7628ac44c
commit 10ef590242
10 changed files with 283 additions and 352 deletions
Download Patch File Download Diff File Expand all files Collapse all files

113
Docs/ARCHITECTURE.md
View File

@@ -80,9 +80,12 @@ src/server/
| +-- progress_service.py # Progress tracking
| +-- websocket_service.py# WebSocket broadcasting
| +-- queue_repository.py # Database persistence
| +-- nfo_service.py # NFO metadata management
| +-- setup_service.py # Series key resolution from folder names
| +-- folder_scan_service.py # Daily folder maintenance scan
| +-- nfo_scan_service.py # NFO creation, update, and regeneration from TMDB
| +-- scan_service.py # Library rescan (episodes, missing files)
| +-- folder_naming_service.py # Folder rename to Title (YYYY) convention
| +-- scheduler/ # Scheduled tasks
| | +-- scheduler_service.py # Cron-based library rescans
+-- models/ # Pydantic models
| +-- auth.py # Auth request/response models
| +-- config.py # Configuration models
@@ -166,11 +169,42 @@ src/server/web/static/js/
| +-- socket-handler.js # WebSocket event handlers
| +-- app-init.js # Application initialization
+-- queue/ # Queue page modules
+-- queue-api.js # Queue API interactions
+-- queue-renderer.js # Queue list rendering
+-- progress-handler.js # Download progress updates
+-- queue-socket-handler.js # WebSocket events for queue
+-- queue-init.js # Queue page initialization
| +-- queue-api.js # Queue API wrapper (uses ApiClient internally)
| +-- queue-renderer.js # Queue DOM rendering
| +-- progress-handler.js # Download progress updates
| +-- queue-socket-handler.js # WebSocket events for queue
| +-- queue-init.js # Queue page initialisation and orchestration
```
**Queue Module Public APIs:**
```javascript
// queue-api.js — wraps /api/queue/* endpoints via AniWorld.ApiClient
AniWorld.QueueAPI.loadQueueData() // GET /api/queue/status → queue data
AniWorld.QueueAPI.startQueue() // POST /api/queue/start
AniWorld.QueueAPI.stopQueue() // POST /api/queue/stop
AniWorld.QueueAPI.removeFromQueue(itemId) // DELETE /api/queue/{item_id}
AniWorld.QueueAPI.retryDownloads(itemIds) // POST /api/queue/retry
AniWorld.QueueAPI.clearCompleted() // DELETE /api/queue/completed
AniWorld.QueueAPI.clearFailed() // DELETE /api/queue/failed
AniWorld.QueueAPI.clearPending() // DELETE /api/queue/pending
// queue-init.js — page orchestration (imports QueueAPI internally)
AniWorld.QueueApp.init() // Bootstrap; calls loadQueueData()
AniWorld.QueueApp.loadQueueData() // Fetch queue data and render
AniWorld.QueueApp.startDownload() // Start queue processing
AniWorld.QueueApp.stopDownloads() // Stop queue processing
AniWorld.QueueApp.removeFromQueue(id) // Remove single item
AniWorld.QueueApp.retryDownload(id) // Retry failed item
AniWorld.QueueApp.retryAllFailed() // Retry all failed items
AniWorld.QueueApp.clearQueue(type) // Clear completed|failed|pending
```
> **Module dependency rule**: Queue page modules must access API endpoints via
> `AniWorld.QueueAPI.<method>()`. The `API` object (`AniWorld.Constants.API`) is
> private to each module's IIFE closure and is NOT a global. Do NOT use bare
> `fetch(API.QUEUE_STATUS, ...)` in `queue-init.js` — use
> `AniWorld.QueueAPI.loadQueueData()` instead.
```
#### Module Pattern
@@ -195,51 +229,48 @@ AniWorld.ModuleName = (function () {
Source: [src/server/web/static/](../src/server/web/static/)
### 2.3 Core Layer (`src/core/`)
### 2.3 Core Layer (`src/server/`)
Domain logic for anime series management.
Domain logic for anime series management, NFO metadata, and episode downloads.
```
src/core/
+-- SeriesApp.py # Main application facade
src/server/
+-- SerieScanner.py # Directory scanning, targeted single-series scan
+-- entities/ # Domain entities
| +-- series.py # Serie class with sanitized_folder property
| +-- SerieList.py # SerieList collection with sanitized folder support
| +-- nfo_models.py # Pydantic models for tvshow.nfo (TVShowNFO, ActorInfo…)
+-- services/ # Domain services
| +-- nfo_service.py # NFO lifecycle: create / update tvshow.nfo
| +-- nfo_repair_service.py # Detect & repair incomplete tvshow.nfo files
| | # (parse_nfo_tags, find_missing_tags, NfoRepairService)
| +-- tmdb_client.py # Async TMDB API client
+-- utils/ # Utility helpers (no side-effects)
| +-- nfo_generator.py # TVShowNFO → XML serialiser
+-- SerieList.py # Series collection (stub; see src/server/database/SerieList.py)
+-- nfo/ # NFO metadata generation and mapping
| +-- nfo_generator.py # TVShowNFO → XML serialiser (generate_tvshow_nfo)
| +-- nfo_mapper.py # TMDB API dict → TVShowNFO (tmdb_to_nfo_model,
| | # _extract_rating_by_country, _extract_fsk_rating)
| +-- image_downloader.py # TMDB image downloader
| +-- nfo_models.py # Pydantic models for NFO XML (TVShowNFO, ActorInfo…)
| +-- tmdb_client.py # Async TMDB API client
+-- providers/ # External provider adapters
| +-- base_provider.py # Loader interface
| +-- provider_factory.py # Provider registry
+-- interfaces/ # Abstract interfaces
| +-- aniworld_provider.py # AniWorld scraper
| +-- enhanced_provider.py # Multi-provider with failover
| +-- provider_config.py # Provider preference configuration
| +-- streaming/ # Provider-specific extractors (VOE, Doodstream, etc.)
+-- entities/
| +-- nfo_models.py # Domain entities for NFO (aligns with nfo/nfo_models.py)
+-- interfaces/
| +-- callbacks.py # Progress callback system
+-- exceptions/ # Domain exceptions
+-- Exceptions.py # Custom exceptions
| +-- providers.py # Provider interface definitions
+-- exceptions/
+-- Exceptions.py # Custom exceptions
```
**Key Components:**
| Component | Purpose |
| -------------- | -------------------------------------------------------------------------- |
| `SeriesApp` | Main application facade for anime operations |
| `SerieScanner` | Scans directories for anime; `scan_single_series()` for targeted scans |
| `Serie` | Domain entity with `sanitized_folder` property for filesystem-safe names |
| `SerieList` | Collection management with automatic folder creation using sanitized names |
|| Component | Purpose |
| --- | --- |
| `SerieScanner` | Scans directories for anime; `scan_single_series()` for targeted scans |
| `tmdb_client.py` | Async TMDB API client |
| `nfo_generator.py` | Serialises `TVShowNFO` to XML |
| `nfo_mapper.py` | Maps TMDB API response to `TVShowNFO` domain model |
| `enhanced_provider.py` | Multi-provider downloader with failover chain |
**Initialization:**
`SeriesApp` is initialized with `skip_load=True` passed to `SerieList`, preventing automatic loading of series from data files on every instantiation. Series data is loaded once during application setup via `sync_series_from_data_files()` in the FastAPI lifespan, which reads data files and syncs them to the database. Subsequent operations load series from the database through the service layer.
Source: [src/core/](../src/core/)
> **Note:** The `src/core/` directory was an earlier architectural proposal and is
> currently empty. All domain logic lives under `src/server/`.
### 2.4 Infrastructure Layer (`src/infrastructure/`)
@@ -428,8 +459,8 @@ Source: [src/server/middleware/auth.py](../src/server/middleware/auth.py#L1-L209
| Exception / cancellation | Temp + `.part` fragments removed in `except` block |
Source: [src/server/services/download_service.py](../src/server/services/download_service.py#L1-L150),
[src/core/providers/aniworld_provider.py](../src/core/providers/aniworld_provider.py),
[src/core/providers/enhanced_provider.py](../src/core/providers/enhanced_provider.py)
[src/server/providers/aniworld_provider.py](../src/server/providers/aniworld_provider.py),
[src/server/providers/enhanced_provider.py](../src/server/providers/enhanced_provider.py)
### 3.3 WebSocket Event Flow
@@ -731,7 +762,7 @@ class Loader(ABC):
def get_episodes(self, serie: Serie) -> Dict[int, List[int]]: ...
```
Source: [src/core/providers/base_provider.py](../src/core/providers/base_provider.py)
Source: [src/server/providers/base_provider.py](../src/server/providers/base_provider.py)
### 8.2 Filesystem Integration
@@ -745,7 +776,7 @@ SerieScanner(
)
```
Source: [src/core/SerieScanner.py](../src/core/SerieScanner.py#L59-L96)
Source: [src/server/SerieScanner.py](../src/server/SerieScanner.py#L59-L96)
---

60
Docs/CHANGELOG.md
View File

@@ -83,6 +83,16 @@ This changelog follows [Keep a Changelog](https://keepachangelog.com/) principle
### Fixed
- **Queue view blank after adding items**: `queue-init.js`'s `loadQueueData()` called
`API.QUEUE_STATUS` directly, but `API` is a local variable inside
`AniWorld.QueueAPI`'s IIFE — not accessible globally. Items added to the
queue were persisted server-side but the queue page could not fetch them,
leaving the view empty with an `API is not defined` console error. Fixed by
replacing the inline `fetch` with `AniWorld.QueueAPI.loadQueueData()`, which
already exists and correctly accesses the endpoint through its own closure.
The same file already uses `AniWorld.QueueAPI.*` for all other queue
operations (`startQueue`, `stopQueue`, `removeFromQueue`, etc.).
- **Bug**: `src/server/api/nfo.py` called the non-existent
`anime_service.update_series_nfo_status(...)` method, which would
raise `AttributeError` after a successful NFO repair. Renamed the
@@ -118,17 +128,14 @@ This changelog follows [Keep a Changelog](https://keepachangelog.com/) principle
### Added
- **Encoding detection for HTML parsing** (`src/core/providers/aniworld_provider.py`):
- **Encoding detection for HTML parsing** (`src/server/providers/aniworld_provider.py`):
Added `_decode_html_content()` function that uses `chardet` to detect the actual
encoding of HTML content before parsing. Falls back to UTF-8 with `errors='replace'`
to handle pages with mismatched encoding declarations. Applied to all BeautifulSoup
parsing calls to prevent "Some characters could not be decoded" warnings.
- **chardet dependency**: Added `chardet>=5.2.0` to `requirements.txt` for encoding detection.
### Added
- **Temp file cleanup after every download** (`src/core/providers/aniworld_provider.py`,
`src/core/providers/enhanced_provider.py`): Module-level helper
- **Temp file cleanup after every download** (`src/server/providers/aniworld_provider.py`,
`src/server/providers/enhanced_provider.py`): Module-level helper
`_cleanup_temp_file()` removes the working temp file and any yt-dlp `.part`
fragments after each download attempt — on success, on failure, and on
exceptions (including `BrokenPipeError` and cancellation). Ensures that no
@@ -145,37 +152,34 @@ This changelog follows [Keep a Changelog](https://keepachangelog.com/) principle
### Added
- **NFO tag completeness (`nfo_mapper.py`)**: All 17 required NFO tags are now
- **NFO tag completeness (`src/server/nfo/nfo_mapper.py`)**: All 17 required NFO tags are now
explicitly populated during creation: `originaltitle`, `sorttitle`, `year`,
`plot`, `outline`, `tagline`, `runtime`, `premiered`, `status`, `imdbid`,
`genre`, `studio`, `country`, `actor`, `watched`, `dateadded`, `mpaa`.
- **`src/core/utils/nfo_mapper.py`**: New module containing
- **`src/server/nfo/nfo_mapper.py`**: New module containing
`tmdb_to_nfo_model()`, `_extract_rating_by_country()`, and
`_extract_fsk_rating()`. Extracted from `NFOService` to keep files under
500 lines and isolate pure mapping logic.
`_extract_fsk_rating()`. Extracted to keep files under 500 lines and isolate
pure mapping logic.
- **`src/server/nfo/nfo_generator.py`**: XML serialiser for NFO files
(`generate_tvshow_nfo`).
- **US MPAA rating**: `_extract_rating_by_country(ratings, "US")` now maps the
US TMDB content rating to the `<mpaa>` NFO tag.
- **`NfoRepairService` (`src/core/services/nfo_repair_service.py`)**: New service
that detects incomplete `tvshow.nfo` files and triggers TMDB re-fetch.
Provides `parse_nfo_tags()`, `find_missing_tags()`, `nfo_needs_repair()`, and
`NfoRepairService.repair_series()`. 13 required tags are checked.
- **`perform_nfo_repair_scan()`
(`src/server/services/folder_scan_service.py`)**: New async function
that iterates every series directory, checks whether `tvshow.nfo` is missing
required tags using `nfo_needs_repair()`, and queues the series for background
reload via `asyncio.create_task`. Skips gracefully when `tmdb_api_key` or
`anime_directory` is not configured.
- **NFO repair wired into scheduled folder scan (`src/server/services/folder_scan_service.py`)**:
`perform_nfo_repair_scan(background_loader=None)` is called during the
scheduled daily folder scan, keeping startup fast while ensuring regular
maintenance.
- **`NfoScanService` (`src/server/services/nfo_scan_service.py`)**: New service
that detects incomplete `tvshow.nfo` files and regenerates them from TMDB.
Provides `scan_all()`, `_scan_series()`, `_create_nfo()`,
`_update_nfo_if_needed()`, and `_regenerate_nfo()`. 17 NFO tags are written.
- **`ScanService` (`src/server/services/scan_service.py`)**: New service for
library rescans — detects new and removed episode files and syncs the
`episodes` table accordingly.
- **`FolderNamingService` (`src/server/services/folder_naming_service.py`)**:
Renames series folders to the `Title (YYYY)` convention using the year from
`tvshow.nfo`. Prevents double-year accumulation on repeated runs.
### Changed
- `NFOService._tmdb_to_nfo_model()` and `NFOService._extract_fsk_rating()` moved
to `src/core/utils/nfo_mapper.py` as module-level functions
`tmdb_to_nfo_model()` and `_extract_fsk_rating()`.
- `src/core/services/nfo_service.py` reduced from 640 → 471 lines.
- `src/server/nfo/nfo_mapper.py` and `src/server/nfo/nfo_generator.py`
replaced the monolithic NFO logic from the previous service.
- NFO generation moved to `src/server/nfo/nfo_generator.py`.
---

2
Docs/CONFIGURATION.md
View File

@@ -128,7 +128,7 @@ Location: `data/config.json`
},
"backup": {
"enabled": false,
"path": "data/backups",
"path": "data/config_backups",
"keep_days": 30
},
"nfo": {

90
Docs/NFO_GUIDE.md
View File

@@ -728,11 +728,11 @@ Every poster check action is logged:
### 8.1 Custom NFO Templates
You can customize NFO generation by modifying the NFO service:
You can customise NFO generation by modifying `src/server/nfo/nfo_generator.py`:
```python
# src/core/services/nfo_creator.py
def generate_tvshow_nfo(self, metadata: dict) -> str:
# src/server/nfo/nfo_generator.py
def generate_tvshow_nfo(metadata: TVShowNFO) -> str:
# Add custom fields or modify structure
pass
```
@@ -811,78 +811,64 @@ updated via `create_tvshow_nfo()` / `update_tvshow_nfo()`.
| `watched` | Always `false` on creation | ✅ |
| `dateadded` | System clock at creation time (`YYYY-MM-DD HH:MM:SS`) | ✅ |
The mapping logic lives in `src/core/utils/nfo_mapper.py` (`tmdb_to_nfo_model`).
The XML serialisation lives in `src/core/utils/nfo_generator.py`
The mapping logic lives in `src/server/nfo/nfo_mapper.py` (`tmdb_to_nfo_model`).
The XML serialisation lives in `src/server/nfo/nfo_generator.py`
(`generate_tvshow_nfo`).
---
## 11. Automatic NFO Repair
NFO repair now runs as part of the scheduled daily folder scan rather than on every
startup. When the scheduler triggers `FolderScanService.run_folder_scan()`, the first
step is `perform_nfo_repair_scan(background_loader=None)`. Each incomplete NFO is
queued as a background `asyncio` task, so the scan returns quickly while repairs
continue asynchronously.
NFO repair runs as part of the scheduled daily scan via ``SchedulerService``.
When the scheduler fires, it calls ``_run_nfo_scan()`` which delegates to
``NfoScanService.scan_all()``. This detects series whose ``tvshow.nfo`` is
missing required tags and regenerates them from TMDB.
### How It Works
1. **Scan** — `perform_nfo_repair_scan()` in
`src/server/services/initialization_service.py` is called from
`FolderScanService.run_folder_scan()` (`src/server/services/folder_scan_service.py`).
2. **Detect** — `nfo_needs_repair(nfo_path)` from
`src/core/services/nfo_repair_service.py` parses each `tvshow.nfo` with
`lxml` and checks for the 13 required tags listed below.
3. **Repair** — Series whose NFO is incomplete are queued for background reload
via `asyncio.create_task`. Each task creates its own isolated
:class:`NFOService` / :class:`TMDBClient` so concurrent tasks never share an
``aiohttp`` session — this prevents "Connector is closed" errors when many repairs
run in parallel. A semaphore caps TMDB concurrency at 3 to stay within rate limits.
1. **Scheduler** fires the daily job (``SchedulerService._run_nfo_scan()``)
2. **Detect** — ``NfoScanService._scan_series()`` parses each ``tvshow.nfo``
and calls ``_create_nfo()`` / ``_update_nfo_if_needed()`` /
``_regenerate_nfo()`` to fill missing tags from TMDB
3. **Repair** — If TMDB lookup succeeds, the NFO is overwritten with complete
data; if it fails, the original is kept and the failure is logged
### Tags Checked (13 required)
### Tags Written / Updated
| XPath | Tag name |
| ----------------- | --------------- |
| `./title` | `title` |
| `./originaltitle` | `originaltitle` |
| `./year` | `year` |
| `./plot` | `plot` |
| `./runtime` | `runtime` |
| `./premiered` | `premiered` |
| `./status` | `status` |
| `./imdbid` | `imdbid` |
| `./genre` | `genre` |
| `./studio` | `studio` |
| `./country` | `country` |
| `./actor/name` | `actor/name` |
| `./watched` | `watched` |
The NFO scan writes all 17 tags listed in the
[Tag Reference](#10-tag-reference) above. Missing or empty tags trigger a
regeneration from TMDB.
### Log Messages
| Message | Meaning |
| ----------------------------------------------------------- | ------------------------------------------------- |
| `NFO repair scan complete: 0 of N series queued for repair` | All NFOs are complete — no action needed |
| `NFO repair scan complete: X of N series queued for repair` | X series had incomplete NFOs and have been queued |
| `NFO repair scan skipped: TMDB API key not configured` | Set `tmdb_api_key` in `data/config.json` |
| `NFO repair scan skipped: anime directory not configured` | Set `anime_directory` in `data/config.json` |
|| Message | Meaning |
| --- | --- |
| `NFO scan complete: N series processed` | Scan finished normally |
| `NFO scan skipped: TMDB API key not configured` | ``tmdb_api_key`` is empty — set it in ``data/config.json`` |
| `NFO scan skipped: anime directory not configured` | ``anime_directory`` is not set |
### Triggering a Manual Repair
### Manual Repair
You can also repair a single series on demand via the API:
You can repair a single series on demand via the API:
```http
POST /api/nfo/update/{series_key}
POST /api/nfo/{series_key}/create
```
This calls `NFOService.update_tvshow_nfo()` directly and overwrites the existing
`tvshow.nfo` with fresh data from TMDB.
or update with fresh TMDB data:
```http
POST /api/nfo/{series_key}/update
```
### Source Files
| File | Purpose |
| ----------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `src/core/services/nfo_repair_service.py` | `REQUIRED_TAGS`, `parse_nfo_tags`, `find_missing_tags`, `nfo_needs_repair`, `NfoRepairService` |
| `src/server/services/folder_scan_service.py` | `perform_nfo_repair_scan` — invoked during the scheduled daily folder scan |
|| File | Purpose |
| --- | --- |
| ``src/server/services/scheduler/scheduler_service.py`` | ``SchedulerService._run_nfo_scan()`` — entry point called by the scheduler |
| ``src/server/services/nfo_scan_service.py`` | ``NfoScanService.scan_all()`` — detects incomplete NFOs and regenerates them |
| ``src/server/services/scan_service.py`` | ``ScanService`` — library rescan (episodes, missing files) |
| ``src/server/services/folder_naming_service.py`` | ``FolderNamingService`` — renames folders to ``Title (YYYY)`` format |
---

2
Docs/features.md
View File

@@ -90,7 +90,7 @@ The application now features a comprehensive configuration system that allows us
- **Queue Organization**: Displays downloads organized by status (pending, active, completed, failed)
- **NFO Integration**: Automatic NFO and media file creation before episode downloads
- **Manual Start/Stop Control**: User manually starts downloads one at a time with Start/Stop buttons
- **FIFO Queue Processing**: First-in, first-out queue order (no priority or reordering)
- **Queue Processing Order**: Items processed in user-defined order via drag-and-drop reordering (`POST /api/queue/reorder`)
- **Single Download Mode**: Only one download active at a time, new downloads must be manually started
- **Download Status Display**: Real-time status updates and progress of current download
- **Queue Operations**: Add and remove items from the pending queue