instructions.md

@@ -39,6 +39,363 @@ If you encounter:
 
 ---
 
+## 🎯 Current Task: Migrate Series Data from File Storage to Database
+
+### Background
+
+The current implementation stores anime series metadata in `data` files (JSON format without `.json` extension) located in each series folder (e.g., `{anime_directory}/{series_folder}/data`). This task migrates that storage to the SQLite database using the existing `AnimeSeries` model.
+
+### Files Involved
+
+**Current Data File Storage:**
+
+-   `src/core/entities/series.py` - `Serie` class with `save_to_file()` and `load_from_file()` methods
+-   `src/core/entities/SerieList.py` - `SerieList` class that loads/saves data files
+-   `src/core/SerieScanner.py` - Scanner that creates data files during scan
+
+**Database Components (Already Exist):**
+
+-   `src/server/database/models.py` - `AnimeSeries` model (already defined)
+-   `src/server/database/service.py` - `AnimeSeriesService` with CRUD operations
+-   `src/server/database/init.py` - Database initialization
+
+**API Endpoints:**
+
+-   `src/server/api/anime.py` - `/api/anime/add` endpoint that adds new series
+
+---
+
+### Task 1: Create Data File Migration Service ⬜
+
+**File:** `src/server/services/data_migration_service.py`
+
+**Description:** Create a service that migrates existing `data` files to the database.
+
+**Requirements:**
+
+1. Create `DataMigrationService` class with the following methods:
+
+    - `scan_for_data_files(anime_directory: str) -> List[Path]` - Find all `data` files in the anime directory
+    - `migrate_data_file(data_path: Path, db: AsyncSession) -> bool` - Migrate single data file to DB
+    - `migrate_all(anime_directory: str, db: AsyncSession) -> MigrationResult` - Migrate all found data files
+    - `is_migration_needed(anime_directory: str) -> bool` - Check if there are data files to migrate
+
+2. Migration logic:
+
+    - Read the `data` file using `Serie.load_from_file()`
+    - Check if series with same `key` already exists in DB using `AnimeSeriesService.get_by_key()`
+    - If not exists, create new `AnimeSeries` record using `AnimeSeriesService.create()`
+    - If exists, optionally update the `episode_dict` if it has changed
+    - Log all operations with appropriate log levels
+
+3. Create `MigrationResult` dataclass:
+
+    ```python
+    @dataclass
+    class MigrationResult:
+        total_found: int
+        migrated: int
+        skipped: int
+        failed: int
+        errors: List[str]
+    ```
+
+4. Handle errors gracefully - don't stop migration on individual file failures
+
+**Testing Requirements:**
+
+-   Unit tests in `tests/unit/test_data_migration_service.py`
+-   Test data file scanning
+-   Test single file migration
+-   Test migration when series already exists
+-   Test error handling for corrupted files
+
+---
+
+### Task 2: Create Startup Migration Script ⬜
+
+**File:** `src/server/services/startup_migration.py`
+
+**Description:** Create a migration runner that executes on every application startup.
+
+**Requirements:**
+
+1. Create `run_startup_migration(anime_directory: str) -> MigrationResult` async function
+2. This function should:
+
+    - Check if migration is needed using `DataMigrationService.is_migration_needed()`
+    - If needed, run `DataMigrationService.migrate_all()`
+    - Log migration results
+    - Return the `MigrationResult`
+
+3. Create `ensure_migration_on_startup()` async function that:
+    - Gets the anime directory from settings/config
+    - Runs `run_startup_migration()` if directory is configured
+    - Handles cases where directory is not yet configured (first run)
+
+**Testing Requirements:**
+
+-   Unit tests in `tests/unit/test_startup_migration.py`
+-   Test migration runs when data files exist
+-   Test migration skips when no data files exist
+-   Test handling of unconfigured anime directory
+
+---
+
+### Task 3: Integrate Migration into FastAPI Lifespan ⬜
+
+**File:** `src/server/fastapi_app.py`
+
+**Description:** Add the startup migration to the FastAPI application lifespan.
+
+**Requirements:**
+
+1. Import `ensure_migration_on_startup` from startup migration service
+2. Call `await ensure_migration_on_startup()` in the `lifespan` function after config is loaded
+3. Log migration results
+4. Do NOT block application startup on migration failure - log error and continue
+
+**Example Integration:**
+
+```python
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    # ... existing startup code ...
+
+    # Run data file to database migration
+    try:
+        from src.server.services.startup_migration import ensure_migration_on_startup
+        migration_result = await ensure_migration_on_startup()
+        if migration_result:
+            logger.info(
+                "Data migration complete: %d migrated, %d skipped, %d failed",
+                migration_result.migrated,
+                migration_result.skipped,
+                migration_result.failed
+            )
+    except Exception as e:
+        logger.error("Data migration failed: %s", e, exc_info=True)
+        # Continue startup - migration failure should not block app
+
+    # ... rest of startup ...
+```
+
+**Testing Requirements:**
+
+-   Integration test that verifies migration runs on startup
+-   Test that app starts even if migration fails
+
+---
+
+### Task 4: Update SerieList to Use Database ⬜
+
+**File:** `src/core/entities/SerieList.py`
+
+**Description:** Modify `SerieList` class to read from database instead of data files.
+
+**Requirements:**
+
+1. Add optional `db_session` parameter to `__init__`
+2. Modify `load_series()` method:
+
+    - If `db_session` is provided, load from database using `AnimeSeriesService.get_all()`
+    - Convert `AnimeSeries` models to `Serie` objects
+    - Fall back to file-based loading if no `db_session` (for backward compatibility)
+
+3. Modify `add()` method:
+
+    - If `db_session` is provided, save to database using `AnimeSeriesService.create()`
+    - Do NOT create data files anymore
+    - Fall back to file-based saving if no `db_session`
+
+4. Add new method `load_series_from_db(db: AsyncSession) -> None`
+5. Add new method `add_to_db(serie: Serie, db: AsyncSession) -> None`
+
+**Important:** Keep backward compatibility - file-based operations should still work when no `db_session` is provided.
+
+**Testing Requirements:**
+
+-   Unit tests for database-based loading
+-   Unit tests for database-based adding
+-   Test backward compatibility with file-based operations
+-   Test conversion between `AnimeSeries` model and `Serie` entity
+
+---
+
+### Task 5: Update SerieScanner to Use Database ⬜
+
+**File:** `src/core/SerieScanner.py`
+
+**Description:** Modify `SerieScanner` to save scan results to database instead of data files.
+
+**Requirements:**
+
+1. Add optional `db_session` parameter to `__init__`
+2. Modify scanning logic (around line 185-188):
+
+    - If `db_session` is provided, save to database instead of file
+    - Use `AnimeSeriesService.create()` or `AnimeSeriesService.update()` for upserting
+    - Do NOT create data files anymore when using database
+
+3. Create helper method `_save_serie_to_db(serie: Serie, db: AsyncSession) -> None`
+4. Create helper method `_update_serie_in_db(serie: Serie, db: AsyncSession) -> None`
+
+**Important:** Keep backward compatibility for CLI usage without database.
+
+**Testing Requirements:**
+
+-   Unit tests for database-based saving during scan
+-   Test that scan results persist to database
+-   Test upsert behavior (update existing series)
+
+---
+
+### Task 6: Update Anime API Endpoints ⬜
+
+**File:** `src/server/api/anime.py`
+
+**Description:** Update the `/api/anime/add` endpoint to save to database instead of file.
+
+**Requirements:**
+
+1. Modify `add_series()` endpoint:
+
+    - Get database session using dependency injection
+    - Use `AnimeSeriesService.create()` to save new series
+    - Remove or replace file-based `series_app.list.add(serie)` call
+    - Return the created series info including database ID
+
+2. Add database session dependency:
+
+    ```python
+    from src.server.database import get_db_session
+
+    @router.post("/add")
+    async def add_series(
+        request: AddSeriesRequest,
+        _auth: dict = Depends(require_auth),
+        series_app: Any = Depends(get_series_app),
+        db: AsyncSession = Depends(get_db_session),
+    ) -> dict:
+    ```
+
+3. Update list/get endpoints to optionally read from database
+
+**Testing Requirements:**
+
+-   API test for adding series via database
+-   Test that added series appears in database
+-   Test duplicate key handling
+
+---
+
+### Task 7: Update Dependencies and SeriesApp ⬜
+
+**File:** `src/server/utils/dependencies.py` and `src/core/SeriesApp.py`
+
+**Description:** Update dependency injection to provide database sessions to core components.
+
+**Requirements:**
+
+1. Update `get_series_app()` dependency:
+
+    - Initialize `SerieList` with database session when available
+    - Pass database session to `SerieScanner` when available
+
+2. Create `get_series_app_with_db()` dependency that provides database-aware `SeriesApp`
+
+3. Update `SeriesApp.__init__()`:
+    - Add optional `db_session` parameter
+    - Pass to `SerieList` and `SerieScanner`
+
+**Testing Requirements:**
+
+-   Test `SeriesApp` initialization with database session
+-   Test dependency injection provides correct sessions
+
+---
+
+### Task 8: Write Integration Tests ⬜
+
+**File:** `tests/integration/test_data_file_migration.py`
+
+**Description:** Create comprehensive integration tests for the migration workflow.
+
+**Test Cases:**
+
+1. `test_migration_on_fresh_start` - No data files, no database entries
+2. `test_migration_with_existing_data_files` - Data files exist, migrate to DB
+3. `test_migration_skips_existing_db_entries` - Series already in DB, skip migration
+4. `test_add_series_saves_to_database` - New series via API saves to DB
+5. `test_scan_saves_to_database` - Scan results save to DB
+6. `test_list_reads_from_database` - Series list reads from DB
+7. `test_search_and_add_workflow` - Search -> Add -> Verify in DB
+
+**Setup:**
+
+-   Use pytest fixtures with temporary directories
+-   Use test database (in-memory SQLite)
+-   Create sample data files for migration tests
+
+---
+
+### Task 9: Clean Up Legacy Code ⬜
+
+**Description:** Remove or deprecate file-based storage code after database migration is stable.
+
+**Requirements:**
+
+1. Add deprecation warnings to file-based methods:
+
+    - `Serie.save_to_file()` - Add `warnings.warn()` with deprecation notice
+    - `Serie.load_from_file()` - Add `warnings.warn()` with deprecation notice
+    - `SerieList.add()` file path - Log deprecation when creating data files
+
+2. Update documentation:
+
+    - Document that data files are deprecated
+    - Document database storage as the primary method
+    - Update `infrastructure.md` with new architecture
+
+3. Do NOT remove file-based code yet - keep for backward compatibility
+
+**Testing Requirements:**
+
+-   Test that deprecation warnings are raised
+-   Verify existing file-based tests still pass
+
+---
+
+### Task 10: Final Validation ⬜
+
+**Description:** Validate the complete migration implementation.
+
+**Validation Checklist:**
+
+-   [ ] All unit tests pass: `conda run -n AniWorld python -m pytest tests/unit/ -v`
+-   [ ] All integration tests pass: `conda run -n AniWorld python -m pytest tests/integration/ -v`
+-   [ ] All API tests pass: `conda run -n AniWorld python -m pytest tests/api/ -v`
+-   [ ] Migration runs automatically on server startup
+-   [ ] New series added via API are saved to database
+-   [ ] Scan results are saved to database
+-   [ ] Series list is read from database
+-   [ ] Existing data files are migrated to database on first run
+-   [ ] Application starts successfully even with no data files
+-   [ ] Application starts successfully even with no anime directory configured
+-   [ ] Deprecation warnings appear in logs when file-based methods are used
+-   [ ] No new data files are created after migration
+
+**Manual Testing:**
+
+1. Start fresh server: `conda run -n AniWorld python -m uvicorn src.server.fastapi_app:app --host 127.0.0.1 --port 8000 --reload`
+2. Login and configure anime directory
+3. Run rescan - verify series appear in database
+4. Search and add new series - verify saved to database
+5. Restart server - verify migration detects no new files to migrate
+6. Check database for all series entries
+
+---
+
 ## 📚 Helpful Commands
 
 ```bash