Add database migration from legacy data files

- Create DataMigrationService for migrating data files to SQLite
- Add sync database methods to AnimeSeriesService
- Update SerieScanner to save to database with file fallback
- Update anime API endpoints to use database with fallback
- Add delete endpoint for anime series
- Add automatic migration on startup in fastapi_app.py lifespan
- Add 28 unit tests for migration service
- Add 14 integration tests for migration flow
- Update infrastructure.md and database README docs

Migration runs automatically on startup, legacy data files preserved.
This commit is contained in:
2025-12-01 17:42:09 +01:00
parent 338e3feb4a
commit 17754a86f0
9 changed files with 2209 additions and 45 deletions

View File

@@ -162,7 +162,83 @@ All series-related WebSocket events include `key` as the primary identifier in t
- `AnimeSeriesService.get_by_key(key)` - **Primary lookup method**
- `AnimeSeriesService.get_by_id(id)` - Internal lookup by database ID
- No `get_by_folder()` method exists - folder is never used for lookups
- `AnimeSeriesService.get_all(db)` - Get all series from database
- `AnimeSeriesService.create(db, key, name, site, folder, episode_dict)` - Create new series
- `AnimeSeriesService.update(db, id, **kwargs)` - Update existing series
- `AnimeSeriesService.delete(db, id)` - Delete series by ID
- `AnimeSeriesService.upsert_sync(db, key, name, site, folder, episode_dict)` - Sync upsert for scanner
No `get_by_folder()` method exists - folder is never used for lookups.
## Data Storage Migration
### Background
The application has migrated from file-based storage to SQLite database storage for anime series metadata.
**Previous Storage (Deprecated)**:
- Individual `data` files (no extension) in each anime folder
- Example: `/anime-directory/Attack on Titan (2013)/data`
**Current Storage (Database)**:
- SQLite database at `data/aniworld.db`
- Managed by `AnimeSeriesService` using SQLAlchemy
### Migration Service
The `DataMigrationService` handles automatic migration of legacy data files to the database:
```python
from src.server.services.data_migration_service import DataMigrationService
# Check for legacy files
service = DataMigrationService()
files = await service.check_for_legacy_data_files(anime_directory)
# Migrate all to database
result = await service.migrate_all_legacy_data(anime_directory, db_session)
print(result) # Migration Result: 10 migrated, 2 skipped, 0 failed
# Optional: cleanup old files with backup
await service.cleanup_migrated_files(files, backup=True)
```
### Automatic Migration on Startup
Migration runs automatically during application startup:
1. Database is initialized (`init_db()`)
2. Legacy data files are detected
3. Files are migrated to database
4. Results are logged (no files are deleted automatically)
### Migration Result
```python
@dataclass
class MigrationResult:
total_found: int # Total legacy files found
migrated: int # Successfully migrated
failed: int # Failed to migrate
skipped: int # Already in database
errors: List[str] # Error messages
```
### Deprecation Notes
- **Legacy file-based storage is deprecated** - Do not create new `data` files
- **SerieScanner**: Updated to save to database (with file fallback for CLI)
- **API endpoints**: Now use database as primary source
- **CLI**: Still uses file-based storage for backward compatibility
### Related Files
| File | Purpose |
| ---- | ------- |
| `src/server/services/data_migration_service.py` | Migration service |
| `src/server/database/service.py` | Database CRUD operations |
| `src/server/database/models.py` | SQLAlchemy models |
| `src/core/SerieScanner.py` | Scanner with DB support |
## Core Services