Fix stale data file updates on download completion

When episodes are downloaded successfully, the in-memory Serie.episodeDict
is updated, but the deprecated data file was not being synced. This caused
UI to show episodes as missing when already downloaded.

Changes:
- Update data file in _remove_episode_from_memory when download completes
- DB is authoritative; data file is optional backup (deprecated)
- Gracefully skip update if data file doesn't exist

New integration tests for episode download sync:
- Verify episode removed from missing list after download
- Verify in-memory cache updated after download
- Verify data file updated after download (when it exists)
- Verify downloads work without data file
This commit is contained in:
2026-05-26 18:57:04 +02:00
parent ceb6a2aeb4
commit 6d30747f25
7 changed files with 550 additions and 4 deletions

View File

@@ -139,6 +139,10 @@ This changelog follows [Keep a Changelog](https://keepachangelog.com/) principle
- Modified `src/server/api/anime.py` to save scanned episodes to database - Modified `src/server/api/anime.py` to save scanned episodes to database
- Episodes table properly tracks missing episodes with automatic cleanup - Episodes table properly tracks missing episodes with automatic cleanup
### Deprecated
- **Legacy Series Files (key/data)**: File-based series storage is deprecated. `key` and `data` files in anime folders will be removed in v3.0.0. Database storage is now the primary method. See [docs/MIGRATION_GUIDE.md](docs/MIGRATION_GUIDE.md) for details.
--- ---
## Sections for Each Release ## Sections for Each Release

View File

@@ -588,7 +588,52 @@ EpisodeService.create(is_downloaded=False)
--- ---
## 13. Database Location ## 13. Series Persistence
### Schema
**AnimeSeries Table**: Stores series metadata (key, name, site, folder, year)
| Column | Type | Constraints | Description |
|-----------|--------------|---------------------------|----------------------|
| `id` | INTEGER | PRIMARY KEY | Auto-increment |
| `key` | VARCHAR(255) | UNIQUE, NOT NULL | Series provider key |
| `name` | VARCHAR(500) | NOT NULL | Display name |
| `site` | VARCHAR(500) | | Provider site URL |
| `folder` | VARCHAR(1000)| | Filesystem folder |
**Episode Table**: Stores per-episode metadata (season, episode_number, is_downloaded)
| Column | Type | Constraints | Description |
|-----------------|--------------|---------------------------|----------------------|
| `id` | INTEGER | PRIMARY KEY | Auto-increment |
| `series_id` | INTEGER | FOREIGN KEY → anime_series| Parent series |
| `season` | INTEGER | NOT NULL | Season number |
| `episode_number`| INTEGER | NOT NULL | Episode number |
| `is_downloaded` | BOOLEAN | DEFAULT FALSE | Download status |
### Relationships
- `AnimeSeries.episodes` → List of Episode objects (one-to-many)
- `Episode.series` → Parent AnimeSeries (many-to-one)
- Cascade delete: Deleting a series removes all its episodes
### Queries
```python
# Get all series with episodes
AnimeSeriesService.get_all(db, with_episodes=True)
# Get by provider key
AnimeSeriesService.get_by_key(db, key)
# Get by folder path
AnimeSeriesService.get_by_folder(db, folder)
```
---
## 14. Database Location
| Environment | Default Location | | Environment | Default Location |
| ----------- | ------------------------------------------------- | | ----------- | ------------------------------------------------- |

View File

@@ -398,3 +398,29 @@ forward this to `notification_service.notify_download_failed()` so users see
a HIGH-priority alert. The loader keeps the failure detail in a HIGH-priority alert. The loader keeps the failure detail in
`logs/download_errors.log` for post-mortem. `logs/download_errors.log` for post-mortem.
## Series Storage
### Overview
Series metadata now stored in the database (SQLAlchemy ORM).
Legacy files (`key` and `data` per folder) are deprecated but preserved
for backward compatibility.
### Architecture
- **Database**: Single source of truth for all series metadata
- **In-Memory Cache**: SeriesApp maintains a cache for performance
- **Filesystem**: Only used for episode files themselves, not metadata
### Migration
First startup after upgrade automatically imports any legacy
series files into the database.
### Legacy Files
- `key` file: Contains series provider key (deprecated)
- `data` file: Contains Serie JSON object (deprecated)
Both are safe to delete after migration; not needed for normal operation.

111
docs/MIGRATION_GUIDE.md Normal file
View File

@@ -0,0 +1,111 @@
# Migration Guide: File-Based to Database Storage
## Overview
This guide covers the transition from file-based series metadata storage to the new database-backed system introduced in v2.0.
## What Changed
**Before v2.0**: Series metadata stored in `key` and `data` files alongside anime folders.
**After v2.0**: All metadata stored in SQLite database (`aniworld.db`). Files are deprecated but still supported for backward compatibility during migration.
## Automated Migration
The application automatically migrates on first startup:
1. Scans anime directory for `key` and `data` files
2. Parses legacy files into `AnimeSeries` and `Episode` records
3. Loads series into in-memory cache
4. Logs migration results
**No manual action required.**
## Manual Verification
After first startup with the new version:
1. **Check logs** for: `"Migrated X series from files to DB"`
2. **Verify series count**: UI shows same number of series as before
3. **Confirm episodes**: Episode counts match expected totals
```bash
# Check migration log
grep "Migrated" logs/app.log
# Verify series via API
curl http://localhost:8000/api/anime | jq '.total'
```
## After Migration
### Safe to Delete
Once verified, these files can be removed:
```
<anime_folder>/
├── Attack on Titan (2013)/
│ ├── key # ❌ Can delete
│ ├── data # ❌ Can delete
│ └── Season 1/
│ └── ...
```
**Deleting these files does not affect the database.** The metadata now lives in `aniworld.db`.
### Backup (Recommended)
Before deleting, backup the files:
```bash
# Create backup directory
mkdir -p backup/legacy_series_files
# Copy all key and data files
find /path/to/anime -name "key" -o -name "data" | while read f; do
cp "$f" "backup/legacy_series_files/"
done
```
## Reverting (Not Recommended)
If you must revert to file-based storage:
1. **Restore from database backup** (if available)
2. **Export manually** (no export script exists)
**Warning**: File-based storage is deprecated and will be removed in v3.0.0.
## Troubleshooting
### Series Not Appearing After Migration
1. Check logs for migration errors: `grep -i error logs/app.log`
2. Verify `key` and `data` files exist and are readable
3. Manually trigger rescan: `POST /api/scheduler/trigger-rescan`
### Duplicate Series
1. Check for duplicate `key` files (same series in multiple folders)
2. Verify series key uniqueness in database:
```bash
sqlite3 aniworld.db "SELECT key, COUNT(*) FROM anime_series GROUP BY key HAVING COUNT(*) > 1;"
```
### Missing Episodes
1. Trigger targeted scan for affected series
2. Check episode sync logs
3. Verify file permissions on anime directory
## Deprecation Timeline
| Version | Status |
|---------|--------|
| v2.0.x | Legacy files supported, migration automated |
| v2.1.x | Legacy files still supported, warnings in logs |
| v3.0.0 | **Legacy files removed** - database only |
Upgrade to v3.0.0 before legacy file support ends.

View File

@@ -31,14 +31,16 @@ flowchart TB
subgraph Core["Core Layer"] subgraph Core["Core Layer"]
SeriesApp["SeriesApp"] SeriesApp["SeriesApp"]
SeriesCache["SeriesCache<br/>(In-Memory)"]
SerieScanner["SerieScanner"] SerieScanner["SerieScanner"]
SerieList["SerieList"] SerieList["SerieList"]
end end
subgraph Data["Data Layer"] subgraph Data["Data Layer"]
SQLite[(SQLite<br/>aniworld.db)] SQLite[("SQLite<br/>aniworld.db")]
ConfigJSON[(config.json)] ConfigJSON[(config.json)]
FileSystem[(File System<br/>Anime Directory)] FileSystem[(File System<br/>Anime Episodes)]
LegacyFiles[("Legacy Files<br/>key/data<br/>(Deprecated)")]
end end
subgraph External["External"] subgraph External["External"]
@@ -71,9 +73,13 @@ flowchart TB
AnimeService --> SQLite AnimeService --> SQLite
%% Core to Data %% Core to Data
SeriesApp --> SeriesCache
SeriesCache -.->|Cached Series| SQLite
SeriesApp --> SerieScanner SeriesApp --> SerieScanner
SeriesApp --> SerieList SeriesApp --> SerieList
SerieScanner --> FileSystem SerieScanner -->|Scan Episodes| FileSystem
SerieScanner -->|Detect Series| SQLite
SerieScanner -->|Migrate Legacy| LegacyFiles
SerieScanner --> Provider SerieScanner --> Provider
%% Event flow %% Event flow

View File

@@ -466,6 +466,27 @@ class DownloadService:
"missing episodes remaining", "missing episodes remaining",
len(app.series_list), len(app.series_list),
) )
# Update deprecated data file if it exists
# DB is authoritative; data file is optional backup
serie_folder = serie.folder
data_path = Path(self._directory) / serie_folder / "data"
if data_path.exists():
try:
import warnings
with warnings.catch_warnings():
warnings.simplefilter("ignore", DeprecationWarning)
serie.save_to_file(str(data_path))
logger.debug(
"Updated data file after download: %s",
data_path,
)
except Exception as e:
logger.warning(
"Failed to update data file %s: %s",
data_path,
e,
)
else: else:
logger.debug( logger.debug(
"Episode %d not in season %d for %s, " "Episode %d not in season %d for %s, "

View File

@@ -0,0 +1,333 @@
"""Integration tests for episode download sync with data file updates.
Tests verify that when episodes are downloaded successfully:
- In-memory Serie.episodeDict is updated
- Deprecated data file is updated (if it exists)
- Missing episode list reflects the change immediately
"""
import asyncio
import json
import os
import tempfile
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from src.core.entities.series import Serie
from src.core.SeriesApp import SeriesApp
from src.server.models.download import DownloadItem, DownloadPriority, DownloadStatus
from src.server.services.download_service import DownloadService
class TestEpisodeRemovedFromMissingListAfterDownload:
"""Verify episode no longer appears in missing list after download completes."""
@pytest.fixture
def temp_dir(self):
"""Create temp directory for test data files."""
with tempfile.TemporaryDirectory() as tmp:
yield Path(tmp)
@pytest.fixture
def mock_anime_service(self, temp_dir):
"""Create mock anime service with app."""
anime_service = MagicMock()
anime_service._directory = str(temp_dir)
# Create mock app withSerie with missing episodes
serie = Serie(
key="test-series",
name="Test Series",
site="https://example.com",
folder="Test Series",
episodeDict={1: [1, 2, 3]},
)
mock_app = MagicMock()
mock_app.list.keyDict = {"test-series": serie}
mock_app.list.GetMissingEpisode.return_value = [serie]
mock_app.series_list = [serie]
anime_service._app = mock_app
anime_service._cached_list_missing = MagicMock()
anime_service._broadcast_series_updated = AsyncMock()
return anime_service
@pytest.fixture
def mock_download_service(self, mock_anime_service):
"""Create download service with mocked dependencies."""
with tempfile.TemporaryDirectory() as tmp:
service = DownloadService(
anime_service=mock_anime_service,
queue_repository=MagicMock(),
max_retries=3,
)
service._directory = tmp
yield service
@pytest.mark.asyncio
async def test_episode_removed_from_missing_list_after_download(
self, mock_download_service, mock_anime_service
):
"""Verify episode no longer appears in missing list after download completes."""
serie = mock_anime_service._app.list.keyDict["test-series"]
# Verify episode starts in missing list
assert 2 in serie.episodeDict[1], "Episode should start in missing list"
# Simulate download completion by calling _remove_episode_from_memory
mock_download_service._remove_episode_from_memory("test-series", 1, 2)
# Episode should be removed from episodeDict
assert 2 not in serie.episodeDict[1], "Episode should be removed from missing list"
assert serie.episodeDict[1] == [1, 3]
# series_list should be refreshed
mock_anime_service._app.list.GetMissingEpisode.assert_called()
class TestDownloadUpdatesInMemoryCache:
"""Verify in-memory Serie.episodeDict is updated after download."""
@pytest.fixture
def mock_anime_service(self):
"""Create mock anime service with app."""
anime_service = MagicMock()
anime_service._directory = "/tmp/test"
# Create mock app with series having multiple seasons and episodes
serie = Serie(
key="multi-season-series",
name="Multi Season Series",
site="https://example.com",
folder="Multi Season Series",
episodeDict={
1: [1, 2, 3, 4, 5],
2: [1, 2, 3],
},
)
mock_app = MagicMock()
mock_app.list.keyDict = {"multi-season-series": serie}
mock_app.list.GetMissingEpisode.return_value = [serie]
mock_app.series_list = [serie]
anime_service._app = mock_app
anime_service._cached_list_missing = MagicMock()
anime_service._broadcast_series_updated = AsyncMock()
return anime_service
@pytest.fixture
def mock_download_service(self, mock_anime_service):
"""Create download service with mocked dependencies."""
with tempfile.TemporaryDirectory() as tmp:
service = DownloadService(
anime_service=mock_anime_service,
queue_repository=MagicMock(),
max_retries=3,
)
service._directory = tmp
yield service
@pytest.mark.asyncio
async def test_download_updates_in_memory_cache(
self, mock_download_service, mock_anime_service
):
"""Verify in-memory Serie.episodeDict is updated after download."""
# First reset to known state (remove the defaults first call might have set)
serie = mock_anime_service._app.list.keyDict["multi-season-series"]
# Put back episodes after the fixture setup
serie.episodeDict = {1: [1, 2, 3, 4, 5], 2: [1, 2, 3]}
# Verify preconditions
assert 1 in serie.episodeDict[1]
assert 3 in serie.episodeDict[2]
# Simulate downloading multiple episodes
mock_download_service._remove_episode_from_memory("multi-season-series", 1, 1)
mock_download_service._remove_episode_from_memory("multi-season-series", 1, 3)
mock_download_service._remove_episode_from_memory("multi-season-series", 2, 2)
# Verify episodes removed
assert 1 not in serie.episodeDict[1], "Episode 1 of season 1 should be removed"
assert 3 not in serie.episodeDict[1], "Episode 3 of season 1 should be removed"
assert 2 in serie.episodeDict[1], "Episode 2 of season 1 should remain"
assert 3 in serie.episodeDict[2], "Episode 3 of season 2 should remain"
assert 2 not in serie.episodeDict[2], "Episode 2 of season 2 should be removed"
# Verify seasons with no episodes are cleaned up
assert 2 in serie.episodeDict, "Season 2 should still exist (has episode 1, 3)"
@pytest.mark.asyncio
async def test_last_episode_removes_season(
self, mock_download_service, mock_anime_service
):
"""Verify that removing last episode in a season removes the season key."""
# Modify the series so season 1 only has episode 2 left
serie = mock_anime_service._app.list.keyDict["multi-season-series"]
# Reset and set to proper test state
serie.episodeDict = {1: [2], 2: [1, 2, 3]} # Season 1 only has episode 2
# Verify initial state
assert 2 in serie.episodeDict[1]
assert 2 in serie.episodeDict[2]
# Remove last episode of season 1 (episode 2)
mock_download_service._remove_episode_from_memory("multi-season-series", 1, 2)
# Season 1 should be completely removed
assert 1 not in serie.episodeDict, "Season 1 should be removed"
# Season 2 should still exist
assert 2 in serie.episodeDict, "Season 2 should still exist"
class TestDataFileUpdatedAfterDownload:
"""Verify data file is updated after download (when it exists)."""
@pytest.fixture
def temp_dir(self):
"""Create temp directory for test data files."""
with tempfile.TemporaryDirectory() as tmp:
yield Path(tmp)
@pytest.fixture
def mock_anime_service(self, temp_dir):
"""Create mock anime service with app."""
anime_service = MagicMock()
anime_service._directory = str(temp_dir)
# Create series folder with data file
series_folder = temp_dir / "Test Series"
series_folder.mkdir()
data_path = series_folder / "data"
serie = Serie(
key="test-series-with-data",
name="Test Series",
site="https://example.com",
folder="Test Series",
episodeDict={1: [1, 2, 3]},
)
# Save data file to disk
import warnings
with warnings.catch_warnings():
warnings.simplefilter("ignore", DeprecationWarning)
serie.save_to_file(str(data_path))
# Update episodeDict to simulate in-progress download state
# (episodeDict still has all episodes; will be updated after download)
mock_app = MagicMock()
mock_app.list.keyDict = {"test-series-with-data": serie}
mock_app.list.GetMissingEpisode.return_value = [serie]
mock_app.series_list = [serie]
anime_service._app = mock_app
anime_service._cached_list_missing = MagicMock()
anime_service._broadcast_series_updated = AsyncMock()
return anime_service
@pytest.fixture
def mock_download_service(self, mock_anime_service):
"""Create download service with mocked dependencies."""
service = DownloadService(
anime_service=mock_anime_service,
queue_repository=MagicMock(),
max_retries=3,
)
service._directory = str(mock_anime_service._directory)
yield service
@pytest.mark.asyncio
async def test_data_file_updated_after_download(
self, mock_download_service, mock_anime_service, temp_dir
):
"""Verify data file is updated after download when data file exists."""
serie = mock_anime_service._app.list.keyDict["test-series-with-data"]
data_path = temp_dir / "Test Series" / "data"
# Verify data file exists before test
assert data_path.exists(), "Data file should exist before test"
# Read original data file
with open(data_path) as f:
original_data = json.load(f)
assert 2 in original_data["episodeDict"]["1"], "Episode should be in original data"
# Simulate download completion
mock_download_service._remove_episode_from_memory("test-series-with-data", 1, 2)
# Read updated data file
with open(data_path) as f:
updated_data = json.load(f)
# Verify episode 2 was removed from data file
assert 2 not in updated_data["episodeDict"]["1"], "Episode should be removed from data file"
assert updated_data["episodeDict"]["1"] == [1, 3]
class TestDataFileNotRequiredForDownload:
"""Verify downloads work even when data file doesn't exist."""
@pytest.fixture
def temp_dir(self):
"""Create temp directory without data files."""
with tempfile.TemporaryDirectory() as tmp:
yield Path(tmp)
@pytest.fixture
def mock_anime_service(self, temp_dir):
"""Create mock anime service with app but no data file."""
anime_service = MagicMock()
anime_service._directory = str(temp_dir)
# Create series with NO data file on disk (only in memory)
serie = Serie(
key="memory-only-series",
name="Memory Only Series",
site="https://example.com",
folder="Memory Only Series",
episodeDict={1: [1, 2, 3]},
)
mock_app = MagicMock()
mock_app.list.keyDict = {"memory-only-series": serie}
mock_app.list.GetMissingEpisode.return_value = [serie]
mock_app.series_list = [serie]
anime_service._app = mock_app
anime_service._cached_list_missing = MagicMock()
anime_service._broadcast_series_updated = AsyncMock()
return anime_service
@pytest.fixture
def mock_download_service(self, mock_anime_service):
"""Create download service with mocked dependencies."""
service = DownloadService(
anime_service=mock_anime_service,
queue_repository=MagicMock(),
max_retries=3,
)
service._directory = str(mock_anime_service._directory)
yield service
@pytest.mark.asyncio
async def test_download_works_without_data_file(
self, mock_download_service, mock_anime_service
):
"""Verify downloads work even when no data file exists on disk."""
serie = mock_anime_service._app.list.keyDict["memory-only-series"]
data_path = Path(mock_anime_service._directory) / "Memory Only Series" / "data"
# Verify no data file exists
assert not data_path.exists(), "No data file should exist"
# Simulate download completion
# This should NOT raise even without data file
mock_download_service._remove_episode_from_memory("memory-only-series", 1, 2)
# Episode should be removed from in-memory state
assert 2 not in serie.episodeDict[1], "Episode should be removed from memory"
# Data file should still not exist (no file created)
assert not data_path.exists(), "No data file should be created"