diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index bfd7259..7622c43 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -139,6 +139,10 @@ This changelog follows [Keep a Changelog](https://keepachangelog.com/) principle - Modified `src/server/api/anime.py` to save scanned episodes to database - Episodes table properly tracks missing episodes with automatic cleanup +### Deprecated + +- **Legacy Series Files (key/data)**: File-based series storage is deprecated. `key` and `data` files in anime folders will be removed in v3.0.0. Database storage is now the primary method. See [docs/MIGRATION_GUIDE.md](docs/MIGRATION_GUIDE.md) for details. + --- ## Sections for Each Release diff --git a/docs/DATABASE.md b/docs/DATABASE.md index 7e85693..9b2b9c6 100644 --- a/docs/DATABASE.md +++ b/docs/DATABASE.md @@ -588,7 +588,52 @@ EpisodeService.create(is_downloaded=False) --- -## 13. Database Location +## 13. Series Persistence + +### Schema + +**AnimeSeries Table**: Stores series metadata (key, name, site, folder, year) + +| Column | Type | Constraints | Description | +|-----------|--------------|---------------------------|----------------------| +| `id` | INTEGER | PRIMARY KEY | Auto-increment | +| `key` | VARCHAR(255) | UNIQUE, NOT NULL | Series provider key | +| `name` | VARCHAR(500) | NOT NULL | Display name | +| `site` | VARCHAR(500) | | Provider site URL | +| `folder` | VARCHAR(1000)| | Filesystem folder | + +**Episode Table**: Stores per-episode metadata (season, episode_number, is_downloaded) + +| Column | Type | Constraints | Description | +|-----------------|--------------|---------------------------|----------------------| +| `id` | INTEGER | PRIMARY KEY | Auto-increment | +| `series_id` | INTEGER | FOREIGN KEY → anime_series| Parent series | +| `season` | INTEGER | NOT NULL | Season number | +| `episode_number`| INTEGER | NOT NULL | Episode number | +| `is_downloaded` | BOOLEAN | DEFAULT FALSE | Download status | + +### Relationships + +- `AnimeSeries.episodes` → List of Episode objects (one-to-many) +- `Episode.series` → Parent AnimeSeries (many-to-one) +- Cascade delete: Deleting a series removes all its episodes + +### Queries + +```python +# Get all series with episodes +AnimeSeriesService.get_all(db, with_episodes=True) + +# Get by provider key +AnimeSeriesService.get_by_key(db, key) + +# Get by folder path +AnimeSeriesService.get_by_folder(db, folder) +``` + +--- + +## 14. Database Location | Environment | Default Location | | ----------- | ------------------------------------------------- | diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 5561396..ac41d05 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -398,3 +398,29 @@ forward this to `notification_service.notify_download_failed()` so users see a HIGH-priority alert. The loader keeps the failure detail in `logs/download_errors.log` for post-mortem. +## Series Storage + +### Overview + +Series metadata now stored in the database (SQLAlchemy ORM). +Legacy files (`key` and `data` per folder) are deprecated but preserved +for backward compatibility. + +### Architecture + +- **Database**: Single source of truth for all series metadata +- **In-Memory Cache**: SeriesApp maintains a cache for performance +- **Filesystem**: Only used for episode files themselves, not metadata + +### Migration + +First startup after upgrade automatically imports any legacy +series files into the database. + +### Legacy Files + +- `key` file: Contains series provider key (deprecated) +- `data` file: Contains Serie JSON object (deprecated) + +Both are safe to delete after migration; not needed for normal operation. + diff --git a/docs/MIGRATION_GUIDE.md b/docs/MIGRATION_GUIDE.md new file mode 100644 index 0000000..5a5e91b --- /dev/null +++ b/docs/MIGRATION_GUIDE.md @@ -0,0 +1,111 @@ +# Migration Guide: File-Based to Database Storage + +## Overview + +This guide covers the transition from file-based series metadata storage to the new database-backed system introduced in v2.0. + +## What Changed + +**Before v2.0**: Series metadata stored in `key` and `data` files alongside anime folders. + +**After v2.0**: All metadata stored in SQLite database (`aniworld.db`). Files are deprecated but still supported for backward compatibility during migration. + +## Automated Migration + +The application automatically migrates on first startup: + +1. Scans anime directory for `key` and `data` files +2. Parses legacy files into `AnimeSeries` and `Episode` records +3. Loads series into in-memory cache +4. Logs migration results + +**No manual action required.** + +## Manual Verification + +After first startup with the new version: + +1. **Check logs** for: `"Migrated X series from files to DB"` +2. **Verify series count**: UI shows same number of series as before +3. **Confirm episodes**: Episode counts match expected totals + +```bash +# Check migration log +grep "Migrated" logs/app.log + +# Verify series via API +curl http://localhost:8000/api/anime | jq '.total' +``` + +## After Migration + +### Safe to Delete + +Once verified, these files can be removed: + +``` +/ +├── Attack on Titan (2013)/ +│ ├── key # ❌ Can delete +│ ├── data # ❌ Can delete +│ └── Season 1/ +│ └── ... +``` + +**Deleting these files does not affect the database.** The metadata now lives in `aniworld.db`. + +### Backup (Recommended) + +Before deleting, backup the files: + +```bash +# Create backup directory +mkdir -p backup/legacy_series_files + +# Copy all key and data files +find /path/to/anime -name "key" -o -name "data" | while read f; do + cp "$f" "backup/legacy_series_files/" +done +``` + +## Reverting (Not Recommended) + +If you must revert to file-based storage: + +1. **Restore from database backup** (if available) +2. **Export manually** (no export script exists) + +**Warning**: File-based storage is deprecated and will be removed in v3.0.0. + +## Troubleshooting + +### Series Not Appearing After Migration + +1. Check logs for migration errors: `grep -i error logs/app.log` +2. Verify `key` and `data` files exist and are readable +3. Manually trigger rescan: `POST /api/scheduler/trigger-rescan` + +### Duplicate Series + +1. Check for duplicate `key` files (same series in multiple folders) +2. Verify series key uniqueness in database: + +```bash +sqlite3 aniworld.db "SELECT key, COUNT(*) FROM anime_series GROUP BY key HAVING COUNT(*) > 1;" +``` + +### Missing Episodes + +1. Trigger targeted scan for affected series +2. Check episode sync logs +3. Verify file permissions on anime directory + +## Deprecation Timeline + +| Version | Status | +|---------|--------| +| v2.0.x | Legacy files supported, migration automated | +| v2.1.x | Legacy files still supported, warnings in logs | +| v3.0.0 | **Legacy files removed** - database only | + +Upgrade to v3.0.0 before legacy file support ends. \ No newline at end of file diff --git a/docs/diagrams/system-architecture.mmd b/docs/diagrams/system-architecture.mmd index 6445d57..83ca2ce 100644 --- a/docs/diagrams/system-architecture.mmd +++ b/docs/diagrams/system-architecture.mmd @@ -31,14 +31,16 @@ flowchart TB subgraph Core["Core Layer"] SeriesApp["SeriesApp"] + SeriesCache["SeriesCache
(In-Memory)"] SerieScanner["SerieScanner"] SerieList["SerieList"] end subgraph Data["Data Layer"] - SQLite[(SQLite
aniworld.db)] + SQLite[("SQLite
aniworld.db")] ConfigJSON[(config.json)] - FileSystem[(File System
Anime Directory)] + FileSystem[(File System
Anime Episodes)] + LegacyFiles[("Legacy Files
key/data
(Deprecated)")] end subgraph External["External"] @@ -71,9 +73,13 @@ flowchart TB AnimeService --> SQLite %% Core to Data + SeriesApp --> SeriesCache + SeriesCache -.->|Cached Series| SQLite SeriesApp --> SerieScanner SeriesApp --> SerieList - SerieScanner --> FileSystem + SerieScanner -->|Scan Episodes| FileSystem + SerieScanner -->|Detect Series| SQLite + SerieScanner -->|Migrate Legacy| LegacyFiles SerieScanner --> Provider %% Event flow diff --git a/src/server/services/download_service.py b/src/server/services/download_service.py index 3fa48ab..62bae3f 100644 --- a/src/server/services/download_service.py +++ b/src/server/services/download_service.py @@ -466,6 +466,27 @@ class DownloadService: "missing episodes remaining", len(app.series_list), ) + + # Update deprecated data file if it exists + # DB is authoritative; data file is optional backup + serie_folder = serie.folder + data_path = Path(self._directory) / serie_folder / "data" + if data_path.exists(): + try: + import warnings + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + serie.save_to_file(str(data_path)) + logger.debug( + "Updated data file after download: %s", + data_path, + ) + except Exception as e: + logger.warning( + "Failed to update data file %s: %s", + data_path, + e, + ) else: logger.debug( "Episode %d not in season %d for %s, " diff --git a/tests/integration/test_episode_download_sync.py b/tests/integration/test_episode_download_sync.py new file mode 100644 index 0000000..3eb6b19 --- /dev/null +++ b/tests/integration/test_episode_download_sync.py @@ -0,0 +1,333 @@ +"""Integration tests for episode download sync with data file updates. + +Tests verify that when episodes are downloaded successfully: +- In-memory Serie.episodeDict is updated +- Deprecated data file is updated (if it exists) +- Missing episode list reflects the change immediately +""" +import asyncio +import json +import os +import tempfile +from pathlib import Path +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from src.core.entities.series import Serie +from src.core.SeriesApp import SeriesApp +from src.server.models.download import DownloadItem, DownloadPriority, DownloadStatus +from src.server.services.download_service import DownloadService + + +class TestEpisodeRemovedFromMissingListAfterDownload: + """Verify episode no longer appears in missing list after download completes.""" + + @pytest.fixture + def temp_dir(self): + """Create temp directory for test data files.""" + with tempfile.TemporaryDirectory() as tmp: + yield Path(tmp) + + @pytest.fixture + def mock_anime_service(self, temp_dir): + """Create mock anime service with app.""" + anime_service = MagicMock() + anime_service._directory = str(temp_dir) + + # Create mock app withSerie with missing episodes + serie = Serie( + key="test-series", + name="Test Series", + site="https://example.com", + folder="Test Series", + episodeDict={1: [1, 2, 3]}, + ) + mock_app = MagicMock() + mock_app.list.keyDict = {"test-series": serie} + mock_app.list.GetMissingEpisode.return_value = [serie] + mock_app.series_list = [serie] + anime_service._app = mock_app + anime_service._cached_list_missing = MagicMock() + anime_service._broadcast_series_updated = AsyncMock() + + return anime_service + + @pytest.fixture + def mock_download_service(self, mock_anime_service): + """Create download service with mocked dependencies.""" + with tempfile.TemporaryDirectory() as tmp: + service = DownloadService( + anime_service=mock_anime_service, + queue_repository=MagicMock(), + max_retries=3, + ) + service._directory = tmp + yield service + + @pytest.mark.asyncio + async def test_episode_removed_from_missing_list_after_download( + self, mock_download_service, mock_anime_service + ): + """Verify episode no longer appears in missing list after download completes.""" + serie = mock_anime_service._app.list.keyDict["test-series"] + + # Verify episode starts in missing list + assert 2 in serie.episodeDict[1], "Episode should start in missing list" + + # Simulate download completion by calling _remove_episode_from_memory + mock_download_service._remove_episode_from_memory("test-series", 1, 2) + + # Episode should be removed from episodeDict + assert 2 not in serie.episodeDict[1], "Episode should be removed from missing list" + assert serie.episodeDict[1] == [1, 3] + + # series_list should be refreshed + mock_anime_service._app.list.GetMissingEpisode.assert_called() + + +class TestDownloadUpdatesInMemoryCache: + """Verify in-memory Serie.episodeDict is updated after download.""" + + @pytest.fixture + def mock_anime_service(self): + """Create mock anime service with app.""" + anime_service = MagicMock() + anime_service._directory = "/tmp/test" + + # Create mock app with series having multiple seasons and episodes + serie = Serie( + key="multi-season-series", + name="Multi Season Series", + site="https://example.com", + folder="Multi Season Series", + episodeDict={ + 1: [1, 2, 3, 4, 5], + 2: [1, 2, 3], + }, + ) + mock_app = MagicMock() + mock_app.list.keyDict = {"multi-season-series": serie} + mock_app.list.GetMissingEpisode.return_value = [serie] + mock_app.series_list = [serie] + anime_service._app = mock_app + anime_service._cached_list_missing = MagicMock() + anime_service._broadcast_series_updated = AsyncMock() + + return anime_service + + @pytest.fixture + def mock_download_service(self, mock_anime_service): + """Create download service with mocked dependencies.""" + with tempfile.TemporaryDirectory() as tmp: + service = DownloadService( + anime_service=mock_anime_service, + queue_repository=MagicMock(), + max_retries=3, + ) + service._directory = tmp + yield service + + @pytest.mark.asyncio + async def test_download_updates_in_memory_cache( + self, mock_download_service, mock_anime_service + ): + """Verify in-memory Serie.episodeDict is updated after download.""" + # First reset to known state (remove the defaults first call might have set) + serie = mock_anime_service._app.list.keyDict["multi-season-series"] + + # Put back episodes after the fixture setup + serie.episodeDict = {1: [1, 2, 3, 4, 5], 2: [1, 2, 3]} + + # Verify preconditions + assert 1 in serie.episodeDict[1] + assert 3 in serie.episodeDict[2] + + # Simulate downloading multiple episodes + mock_download_service._remove_episode_from_memory("multi-season-series", 1, 1) + mock_download_service._remove_episode_from_memory("multi-season-series", 1, 3) + mock_download_service._remove_episode_from_memory("multi-season-series", 2, 2) + + # Verify episodes removed + assert 1 not in serie.episodeDict[1], "Episode 1 of season 1 should be removed" + assert 3 not in serie.episodeDict[1], "Episode 3 of season 1 should be removed" + assert 2 in serie.episodeDict[1], "Episode 2 of season 1 should remain" + assert 3 in serie.episodeDict[2], "Episode 3 of season 2 should remain" + assert 2 not in serie.episodeDict[2], "Episode 2 of season 2 should be removed" + + # Verify seasons with no episodes are cleaned up + assert 2 in serie.episodeDict, "Season 2 should still exist (has episode 1, 3)" + + @pytest.mark.asyncio + async def test_last_episode_removes_season( + self, mock_download_service, mock_anime_service + ): + """Verify that removing last episode in a season removes the season key.""" + # Modify the series so season 1 only has episode 2 left + serie = mock_anime_service._app.list.keyDict["multi-season-series"] + # Reset and set to proper test state + serie.episodeDict = {1: [2], 2: [1, 2, 3]} # Season 1 only has episode 2 + + # Verify initial state + assert 2 in serie.episodeDict[1] + assert 2 in serie.episodeDict[2] + + # Remove last episode of season 1 (episode 2) + mock_download_service._remove_episode_from_memory("multi-season-series", 1, 2) + + # Season 1 should be completely removed + assert 1 not in serie.episodeDict, "Season 1 should be removed" + # Season 2 should still exist + assert 2 in serie.episodeDict, "Season 2 should still exist" + + +class TestDataFileUpdatedAfterDownload: + """Verify data file is updated after download (when it exists).""" + + @pytest.fixture + def temp_dir(self): + """Create temp directory for test data files.""" + with tempfile.TemporaryDirectory() as tmp: + yield Path(tmp) + + @pytest.fixture + def mock_anime_service(self, temp_dir): + """Create mock anime service with app.""" + anime_service = MagicMock() + anime_service._directory = str(temp_dir) + + # Create series folder with data file + series_folder = temp_dir / "Test Series" + series_folder.mkdir() + data_path = series_folder / "data" + + serie = Serie( + key="test-series-with-data", + name="Test Series", + site="https://example.com", + folder="Test Series", + episodeDict={1: [1, 2, 3]}, + ) + + # Save data file to disk + import warnings + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + serie.save_to_file(str(data_path)) + + # Update episodeDict to simulate in-progress download state + # (episodeDict still has all episodes; will be updated after download) + mock_app = MagicMock() + mock_app.list.keyDict = {"test-series-with-data": serie} + mock_app.list.GetMissingEpisode.return_value = [serie] + mock_app.series_list = [serie] + anime_service._app = mock_app + anime_service._cached_list_missing = MagicMock() + anime_service._broadcast_series_updated = AsyncMock() + + return anime_service + + @pytest.fixture + def mock_download_service(self, mock_anime_service): + """Create download service with mocked dependencies.""" + service = DownloadService( + anime_service=mock_anime_service, + queue_repository=MagicMock(), + max_retries=3, + ) + service._directory = str(mock_anime_service._directory) + yield service + + @pytest.mark.asyncio + async def test_data_file_updated_after_download( + self, mock_download_service, mock_anime_service, temp_dir + ): + """Verify data file is updated after download when data file exists.""" + serie = mock_anime_service._app.list.keyDict["test-series-with-data"] + data_path = temp_dir / "Test Series" / "data" + + # Verify data file exists before test + assert data_path.exists(), "Data file should exist before test" + + # Read original data file + with open(data_path) as f: + original_data = json.load(f) + assert 2 in original_data["episodeDict"]["1"], "Episode should be in original data" + + # Simulate download completion + mock_download_service._remove_episode_from_memory("test-series-with-data", 1, 2) + + # Read updated data file + with open(data_path) as f: + updated_data = json.load(f) + + # Verify episode 2 was removed from data file + assert 2 not in updated_data["episodeDict"]["1"], "Episode should be removed from data file" + assert updated_data["episodeDict"]["1"] == [1, 3] + + +class TestDataFileNotRequiredForDownload: + """Verify downloads work even when data file doesn't exist.""" + + @pytest.fixture + def temp_dir(self): + """Create temp directory without data files.""" + with tempfile.TemporaryDirectory() as tmp: + yield Path(tmp) + + @pytest.fixture + def mock_anime_service(self, temp_dir): + """Create mock anime service with app but no data file.""" + anime_service = MagicMock() + anime_service._directory = str(temp_dir) + + # Create series with NO data file on disk (only in memory) + serie = Serie( + key="memory-only-series", + name="Memory Only Series", + site="https://example.com", + folder="Memory Only Series", + episodeDict={1: [1, 2, 3]}, + ) + + mock_app = MagicMock() + mock_app.list.keyDict = {"memory-only-series": serie} + mock_app.list.GetMissingEpisode.return_value = [serie] + mock_app.series_list = [serie] + anime_service._app = mock_app + anime_service._cached_list_missing = MagicMock() + anime_service._broadcast_series_updated = AsyncMock() + + return anime_service + + @pytest.fixture + def mock_download_service(self, mock_anime_service): + """Create download service with mocked dependencies.""" + service = DownloadService( + anime_service=mock_anime_service, + queue_repository=MagicMock(), + max_retries=3, + ) + service._directory = str(mock_anime_service._directory) + yield service + + @pytest.mark.asyncio + async def test_download_works_without_data_file( + self, mock_download_service, mock_anime_service + ): + """Verify downloads work even when no data file exists on disk.""" + serie = mock_anime_service._app.list.keyDict["memory-only-series"] + data_path = Path(mock_anime_service._directory) / "Memory Only Series" / "data" + + # Verify no data file exists + assert not data_path.exists(), "No data file should exist" + + # Simulate download completion + # This should NOT raise even without data file + mock_download_service._remove_episode_from_memory("memory-only-series", 1, 2) + + # Episode should be removed from in-memory state + assert 2 not in serie.episodeDict[1], "Episode should be removed from memory" + + # Data file should still not exist (no file created) + assert not data_path.exists(), "No data file should be created"