lukas.pupkalipinski/Aniworld
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

new tasks

Browse Source
This commit is contained in:
Lukas 2025-12-01 18:04:49 +01:00
parent 338e3feb4a
commit 0222262f8f
1 changed files with 357 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

357
instructions.md
View File

@ -39,6 +39,363 @@ If you encounter:
--- ---
## 🎯 Current Task: Migrate Series Data from File Storage to Database
### Background
The current implementation stores anime series metadata in `data` files (JSON format without `.json` extension) located in each series folder (e.g., `{anime_directory}/{series_folder}/data`). This task migrates that storage to the SQLite database using the existing `AnimeSeries` model.
### Files Involved
**Current Data File Storage:**
- `src/core/entities/series.py` - `Serie` class with `save_to_file()` and `load_from_file()` methods
- `src/core/entities/SerieList.py` - `SerieList` class that loads/saves data files
- `src/core/SerieScanner.py` - Scanner that creates data files during scan
**Database Components (Already Exist):**
- `src/server/database/models.py` - `AnimeSeries` model (already defined)
- `src/server/database/service.py` - `AnimeSeriesService` with CRUD operations
- `src/server/database/init.py` - Database initialization
**API Endpoints:**
- `src/server/api/anime.py` - `/api/anime/add` endpoint that adds new series
---
### Task 1: Create Data File Migration Service ⬜
**File:** `src/server/services/data_migration_service.py`
**Description:** Create a service that migrates existing `data` files to the database.
**Requirements:**
1. Create `DataMigrationService` class with the following methods:
- `scan_for_data_files(anime_directory: str) -> List[Path]` - Find all `data` files in the anime directory
- `migrate_data_file(data_path: Path, db: AsyncSession) -> bool` - Migrate single data file to DB
- `migrate_all(anime_directory: str, db: AsyncSession) -> MigrationResult` - Migrate all found data files
- `is_migration_needed(anime_directory: str) -> bool` - Check if there are data files to migrate
2. Migration logic:
- Read the `data` file using `Serie.load_from_file()`
- Check if series with same `key` already exists in DB using `AnimeSeriesService.get_by_key()`
- If not exists, create new `AnimeSeries` record using `AnimeSeriesService.create()`
- If exists, optionally update the `episode_dict` if it has changed
- Log all operations with appropriate log levels
3. Create `MigrationResult` dataclass:
```python
@dataclass
class MigrationResult:
total_found: int
migrated: int
skipped: int
failed: int
errors: List[str]
```
4. Handle errors gracefully - don't stop migration on individual file failures
**Testing Requirements:**
- Unit tests in `tests/unit/test_data_migration_service.py`
- Test data file scanning
- Test single file migration
- Test migration when series already exists
- Test error handling for corrupted files
---
### Task 2: Create Startup Migration Script ⬜
**File:** `src/server/services/startup_migration.py`
**Description:** Create a migration runner that executes on every application startup.
**Requirements:**
1. Create `run_startup_migration(anime_directory: str) -> MigrationResult` async function
2. This function should:
- Check if migration is needed using `DataMigrationService.is_migration_needed()`
- If needed, run `DataMigrationService.migrate_all()`
- Log migration results
- Return the `MigrationResult`
3. Create `ensure_migration_on_startup()` async function that:
- Gets the anime directory from settings/config
- Runs `run_startup_migration()` if directory is configured
- Handles cases where directory is not yet configured (first run)
**Testing Requirements:**
- Unit tests in `tests/unit/test_startup_migration.py`
- Test migration runs when data files exist
- Test migration skips when no data files exist
- Test handling of unconfigured anime directory
---
### Task 3: Integrate Migration into FastAPI Lifespan ⬜
**File:** `src/server/fastapi_app.py`
**Description:** Add the startup migration to the FastAPI application lifespan.
**Requirements:**
1. Import `ensure_migration_on_startup` from startup migration service
2. Call `await ensure_migration_on_startup()` in the `lifespan` function after config is loaded
3. Log migration results
4. Do NOT block application startup on migration failure - log error and continue
**Example Integration:**
```python
@asynccontextmanager
async def lifespan(app: FastAPI):
# ... existing startup code ...
# Run data file to database migration
try:
from src.server.services.startup_migration import ensure_migration_on_startup
migration_result = await ensure_migration_on_startup()
if migration_result:
logger.info(
"Data migration complete: %d migrated, %d skipped, %d failed",
migration_result.migrated,
migration_result.skipped,
migration_result.failed
)
except Exception as e:
logger.error("Data migration failed: %s", e, exc_info=True)
# Continue startup - migration failure should not block app
# ... rest of startup ...
```
**Testing Requirements:**
- Integration test that verifies migration runs on startup
- Test that app starts even if migration fails
---
### Task 4: Update SerieList to Use Database ⬜
**File:** `src/core/entities/SerieList.py`
**Description:** Modify `SerieList` class to read from database instead of data files.
**Requirements:**
1. Add optional `db_session` parameter to `__init__`
2. Modify `load_series()` method:
- If `db_session` is provided, load from database using `AnimeSeriesService.get_all()`
- Convert `AnimeSeries` models to `Serie` objects
- Fall back to file-based loading if no `db_session` (for backward compatibility)
3. Modify `add()` method:
- If `db_session` is provided, save to database using `AnimeSeriesService.create()`
- Do NOT create data files anymore
- Fall back to file-based saving if no `db_session`
4. Add new method `load_series_from_db(db: AsyncSession) -> None`
5. Add new method `add_to_db(serie: Serie, db: AsyncSession) -> None`
**Important:** Keep backward compatibility - file-based operations should still work when no `db_session` is provided.
**Testing Requirements:**
- Unit tests for database-based loading
- Unit tests for database-based adding
- Test backward compatibility with file-based operations
- Test conversion between `AnimeSeries` model and `Serie` entity
---
### Task 5: Update SerieScanner to Use Database ⬜
**File:** `src/core/SerieScanner.py`
**Description:** Modify `SerieScanner` to save scan results to database instead of data files.
**Requirements:**
1. Add optional `db_session` parameter to `__init__`
2. Modify scanning logic (around line 185-188):
- If `db_session` is provided, save to database instead of file
- Use `AnimeSeriesService.create()` or `AnimeSeriesService.update()` for upserting
- Do NOT create data files anymore when using database
3. Create helper method `_save_serie_to_db(serie: Serie, db: AsyncSession) -> None`
4. Create helper method `_update_serie_in_db(serie: Serie, db: AsyncSession) -> None`
**Important:** Keep backward compatibility for CLI usage without database.
**Testing Requirements:**
- Unit tests for database-based saving during scan
- Test that scan results persist to database
- Test upsert behavior (update existing series)
---
### Task 6: Update Anime API Endpoints ⬜
**File:** `src/server/api/anime.py`
**Description:** Update the `/api/anime/add` endpoint to save to database instead of file.
**Requirements:**
1. Modify `add_series()` endpoint:
- Get database session using dependency injection
- Use `AnimeSeriesService.create()` to save new series
- Remove or replace file-based `series_app.list.add(serie)` call
- Return the created series info including database ID
2. Add database session dependency:
```python
from src.server.database import get_db_session
@router.post("/add")
async def add_series(
request: AddSeriesRequest,
_auth: dict = Depends(require_auth),
series_app: Any = Depends(get_series_app),
db: AsyncSession = Depends(get_db_session),
) -> dict:
```
3. Update list/get endpoints to optionally read from database
**Testing Requirements:**
- API test for adding series via database
- Test that added series appears in database
- Test duplicate key handling
---
### Task 7: Update Dependencies and SeriesApp ⬜
**File:** `src/server/utils/dependencies.py` and `src/core/SeriesApp.py`
**Description:** Update dependency injection to provide database sessions to core components.
**Requirements:**
1. Update `get_series_app()` dependency:
- Initialize `SerieList` with database session when available
- Pass database session to `SerieScanner` when available
2. Create `get_series_app_with_db()` dependency that provides database-aware `SeriesApp`
3. Update `SeriesApp.__init__()`:
- Add optional `db_session` parameter
- Pass to `SerieList` and `SerieScanner`
**Testing Requirements:**
- Test `SeriesApp` initialization with database session
- Test dependency injection provides correct sessions
---
### Task 8: Write Integration Tests ⬜
**File:** `tests/integration/test_data_file_migration.py`
**Description:** Create comprehensive integration tests for the migration workflow.
**Test Cases:**
1. `test_migration_on_fresh_start` - No data files, no database entries
2. `test_migration_with_existing_data_files` - Data files exist, migrate to DB
3. `test_migration_skips_existing_db_entries` - Series already in DB, skip migration
4. `test_add_series_saves_to_database` - New series via API saves to DB
5. `test_scan_saves_to_database` - Scan results save to DB
6. `test_list_reads_from_database` - Series list reads from DB
7. `test_search_and_add_workflow` - Search -> Add -> Verify in DB
**Setup:**
- Use pytest fixtures with temporary directories
- Use test database (in-memory SQLite)
- Create sample data files for migration tests
---
### Task 9: Clean Up Legacy Code ⬜
**Description:** Remove or deprecate file-based storage code after database migration is stable.
**Requirements:**
1. Add deprecation warnings to file-based methods:
- `Serie.save_to_file()` - Add `warnings.warn()` with deprecation notice
- `Serie.load_from_file()` - Add `warnings.warn()` with deprecation notice
- `SerieList.add()` file path - Log deprecation when creating data files
2. Update documentation:
- Document that data files are deprecated
- Document database storage as the primary method
- Update `infrastructure.md` with new architecture
3. Do NOT remove file-based code yet - keep for backward compatibility
**Testing Requirements:**
- Test that deprecation warnings are raised
- Verify existing file-based tests still pass
---
### Task 10: Final Validation ⬜
**Description:** Validate the complete migration implementation.
**Validation Checklist:**
- [ ] All unit tests pass: `conda run -n AniWorld python -m pytest tests/unit/ -v`
- [ ] All integration tests pass: `conda run -n AniWorld python -m pytest tests/integration/ -v`
- [ ] All API tests pass: `conda run -n AniWorld python -m pytest tests/api/ -v`
- [ ] Migration runs automatically on server startup
- [ ] New series added via API are saved to database
- [ ] Scan results are saved to database
- [ ] Series list is read from database
- [ ] Existing data files are migrated to database on first run
- [ ] Application starts successfully even with no data files
- [ ] Application starts successfully even with no anime directory configured
- [ ] Deprecation warnings appear in logs when file-based methods are used
- [ ] No new data files are created after migration
**Manual Testing:**
1. Start fresh server: `conda run -n AniWorld python -m uvicorn src.server.fastapi_app:app --host 127.0.0.1 --port 8000 --reload`
2. Login and configure anime directory
3. Run rescan - verify series appear in database
4. Search and add new series - verify saved to database
5. Restart server - verify migration detects no new files to migrate
6. Check database for all series entries
---
## 📚 Helpful Commands ## 📚 Helpful Commands
```bash ```bash