diff --git a/instructions.md b/instructions.md index 6f5248e..248d642 100644 --- a/instructions.md +++ b/instructions.md @@ -39,6 +39,363 @@ If you encounter: --- +## 🎯 Current Task: Migrate Series Data from File Storage to Database + +### Background + +The current implementation stores anime series metadata in `data` files (JSON format without `.json` extension) located in each series folder (e.g., `{anime_directory}/{series_folder}/data`). This task migrates that storage to the SQLite database using the existing `AnimeSeries` model. + +### Files Involved + +**Current Data File Storage:** + +- `src/core/entities/series.py` - `Serie` class with `save_to_file()` and `load_from_file()` methods +- `src/core/entities/SerieList.py` - `SerieList` class that loads/saves data files +- `src/core/SerieScanner.py` - Scanner that creates data files during scan + +**Database Components (Already Exist):** + +- `src/server/database/models.py` - `AnimeSeries` model (already defined) +- `src/server/database/service.py` - `AnimeSeriesService` with CRUD operations +- `src/server/database/init.py` - Database initialization + +**API Endpoints:** + +- `src/server/api/anime.py` - `/api/anime/add` endpoint that adds new series + +--- + +### Task 1: Create Data File Migration Service ⬜ + +**File:** `src/server/services/data_migration_service.py` + +**Description:** Create a service that migrates existing `data` files to the database. + +**Requirements:** + +1. Create `DataMigrationService` class with the following methods: + + - `scan_for_data_files(anime_directory: str) -> List[Path]` - Find all `data` files in the anime directory + - `migrate_data_file(data_path: Path, db: AsyncSession) -> bool` - Migrate single data file to DB + - `migrate_all(anime_directory: str, db: AsyncSession) -> MigrationResult` - Migrate all found data files + - `is_migration_needed(anime_directory: str) -> bool` - Check if there are data files to migrate + +2. Migration logic: + + - Read the `data` file using `Serie.load_from_file()` + - Check if series with same `key` already exists in DB using `AnimeSeriesService.get_by_key()` + - If not exists, create new `AnimeSeries` record using `AnimeSeriesService.create()` + - If exists, optionally update the `episode_dict` if it has changed + - Log all operations with appropriate log levels + +3. Create `MigrationResult` dataclass: + + ```python + @dataclass + class MigrationResult: + total_found: int + migrated: int + skipped: int + failed: int + errors: List[str] + ``` + +4. Handle errors gracefully - don't stop migration on individual file failures + +**Testing Requirements:** + +- Unit tests in `tests/unit/test_data_migration_service.py` +- Test data file scanning +- Test single file migration +- Test migration when series already exists +- Test error handling for corrupted files + +--- + +### Task 2: Create Startup Migration Script ⬜ + +**File:** `src/server/services/startup_migration.py` + +**Description:** Create a migration runner that executes on every application startup. + +**Requirements:** + +1. Create `run_startup_migration(anime_directory: str) -> MigrationResult` async function +2. This function should: + + - Check if migration is needed using `DataMigrationService.is_migration_needed()` + - If needed, run `DataMigrationService.migrate_all()` + - Log migration results + - Return the `MigrationResult` + +3. Create `ensure_migration_on_startup()` async function that: + - Gets the anime directory from settings/config + - Runs `run_startup_migration()` if directory is configured + - Handles cases where directory is not yet configured (first run) + +**Testing Requirements:** + +- Unit tests in `tests/unit/test_startup_migration.py` +- Test migration runs when data files exist +- Test migration skips when no data files exist +- Test handling of unconfigured anime directory + +--- + +### Task 3: Integrate Migration into FastAPI Lifespan ⬜ + +**File:** `src/server/fastapi_app.py` + +**Description:** Add the startup migration to the FastAPI application lifespan. + +**Requirements:** + +1. Import `ensure_migration_on_startup` from startup migration service +2. Call `await ensure_migration_on_startup()` in the `lifespan` function after config is loaded +3. Log migration results +4. Do NOT block application startup on migration failure - log error and continue + +**Example Integration:** + +```python +@asynccontextmanager +async def lifespan(app: FastAPI): + # ... existing startup code ... + + # Run data file to database migration + try: + from src.server.services.startup_migration import ensure_migration_on_startup + migration_result = await ensure_migration_on_startup() + if migration_result: + logger.info( + "Data migration complete: %d migrated, %d skipped, %d failed", + migration_result.migrated, + migration_result.skipped, + migration_result.failed + ) + except Exception as e: + logger.error("Data migration failed: %s", e, exc_info=True) + # Continue startup - migration failure should not block app + + # ... rest of startup ... +``` + +**Testing Requirements:** + +- Integration test that verifies migration runs on startup +- Test that app starts even if migration fails + +--- + +### Task 4: Update SerieList to Use Database ⬜ + +**File:** `src/core/entities/SerieList.py` + +**Description:** Modify `SerieList` class to read from database instead of data files. + +**Requirements:** + +1. Add optional `db_session` parameter to `__init__` +2. Modify `load_series()` method: + + - If `db_session` is provided, load from database using `AnimeSeriesService.get_all()` + - Convert `AnimeSeries` models to `Serie` objects + - Fall back to file-based loading if no `db_session` (for backward compatibility) + +3. Modify `add()` method: + + - If `db_session` is provided, save to database using `AnimeSeriesService.create()` + - Do NOT create data files anymore + - Fall back to file-based saving if no `db_session` + +4. Add new method `load_series_from_db(db: AsyncSession) -> None` +5. Add new method `add_to_db(serie: Serie, db: AsyncSession) -> None` + +**Important:** Keep backward compatibility - file-based operations should still work when no `db_session` is provided. + +**Testing Requirements:** + +- Unit tests for database-based loading +- Unit tests for database-based adding +- Test backward compatibility with file-based operations +- Test conversion between `AnimeSeries` model and `Serie` entity + +--- + +### Task 5: Update SerieScanner to Use Database ⬜ + +**File:** `src/core/SerieScanner.py` + +**Description:** Modify `SerieScanner` to save scan results to database instead of data files. + +**Requirements:** + +1. Add optional `db_session` parameter to `__init__` +2. Modify scanning logic (around line 185-188): + + - If `db_session` is provided, save to database instead of file + - Use `AnimeSeriesService.create()` or `AnimeSeriesService.update()` for upserting + - Do NOT create data files anymore when using database + +3. Create helper method `_save_serie_to_db(serie: Serie, db: AsyncSession) -> None` +4. Create helper method `_update_serie_in_db(serie: Serie, db: AsyncSession) -> None` + +**Important:** Keep backward compatibility for CLI usage without database. + +**Testing Requirements:** + +- Unit tests for database-based saving during scan +- Test that scan results persist to database +- Test upsert behavior (update existing series) + +--- + +### Task 6: Update Anime API Endpoints ⬜ + +**File:** `src/server/api/anime.py` + +**Description:** Update the `/api/anime/add` endpoint to save to database instead of file. + +**Requirements:** + +1. Modify `add_series()` endpoint: + + - Get database session using dependency injection + - Use `AnimeSeriesService.create()` to save new series + - Remove or replace file-based `series_app.list.add(serie)` call + - Return the created series info including database ID + +2. Add database session dependency: + + ```python + from src.server.database import get_db_session + + @router.post("/add") + async def add_series( + request: AddSeriesRequest, + _auth: dict = Depends(require_auth), + series_app: Any = Depends(get_series_app), + db: AsyncSession = Depends(get_db_session), + ) -> dict: + ``` + +3. Update list/get endpoints to optionally read from database + +**Testing Requirements:** + +- API test for adding series via database +- Test that added series appears in database +- Test duplicate key handling + +--- + +### Task 7: Update Dependencies and SeriesApp ⬜ + +**File:** `src/server/utils/dependencies.py` and `src/core/SeriesApp.py` + +**Description:** Update dependency injection to provide database sessions to core components. + +**Requirements:** + +1. Update `get_series_app()` dependency: + + - Initialize `SerieList` with database session when available + - Pass database session to `SerieScanner` when available + +2. Create `get_series_app_with_db()` dependency that provides database-aware `SeriesApp` + +3. Update `SeriesApp.__init__()`: + - Add optional `db_session` parameter + - Pass to `SerieList` and `SerieScanner` + +**Testing Requirements:** + +- Test `SeriesApp` initialization with database session +- Test dependency injection provides correct sessions + +--- + +### Task 8: Write Integration Tests ⬜ + +**File:** `tests/integration/test_data_file_migration.py` + +**Description:** Create comprehensive integration tests for the migration workflow. + +**Test Cases:** + +1. `test_migration_on_fresh_start` - No data files, no database entries +2. `test_migration_with_existing_data_files` - Data files exist, migrate to DB +3. `test_migration_skips_existing_db_entries` - Series already in DB, skip migration +4. `test_add_series_saves_to_database` - New series via API saves to DB +5. `test_scan_saves_to_database` - Scan results save to DB +6. `test_list_reads_from_database` - Series list reads from DB +7. `test_search_and_add_workflow` - Search -> Add -> Verify in DB + +**Setup:** + +- Use pytest fixtures with temporary directories +- Use test database (in-memory SQLite) +- Create sample data files for migration tests + +--- + +### Task 9: Clean Up Legacy Code ⬜ + +**Description:** Remove or deprecate file-based storage code after database migration is stable. + +**Requirements:** + +1. Add deprecation warnings to file-based methods: + + - `Serie.save_to_file()` - Add `warnings.warn()` with deprecation notice + - `Serie.load_from_file()` - Add `warnings.warn()` with deprecation notice + - `SerieList.add()` file path - Log deprecation when creating data files + +2. Update documentation: + + - Document that data files are deprecated + - Document database storage as the primary method + - Update `infrastructure.md` with new architecture + +3. Do NOT remove file-based code yet - keep for backward compatibility + +**Testing Requirements:** + +- Test that deprecation warnings are raised +- Verify existing file-based tests still pass + +--- + +### Task 10: Final Validation ⬜ + +**Description:** Validate the complete migration implementation. + +**Validation Checklist:** + +- [ ] All unit tests pass: `conda run -n AniWorld python -m pytest tests/unit/ -v` +- [ ] All integration tests pass: `conda run -n AniWorld python -m pytest tests/integration/ -v` +- [ ] All API tests pass: `conda run -n AniWorld python -m pytest tests/api/ -v` +- [ ] Migration runs automatically on server startup +- [ ] New series added via API are saved to database +- [ ] Scan results are saved to database +- [ ] Series list is read from database +- [ ] Existing data files are migrated to database on first run +- [ ] Application starts successfully even with no data files +- [ ] Application starts successfully even with no anime directory configured +- [ ] Deprecation warnings appear in logs when file-based methods are used +- [ ] No new data files are created after migration + +**Manual Testing:** + +1. Start fresh server: `conda run -n AniWorld python -m uvicorn src.server.fastapi_app:app --host 127.0.0.1 --port 8000 --reload` +2. Login and configure anime directory +3. Run rescan - verify series appear in database +4. Search and add new series - verify saved to database +5. Restart server - verify migration detects no new files to migrate +6. Check database for all series entries + +--- + ## 📚 Helpful Commands ```bash