Add database migration from legacy data files
- Create DataMigrationService for migrating data files to SQLite - Add sync database methods to AnimeSeriesService - Update SerieScanner to save to database with file fallback - Update anime API endpoints to use database with fallback - Add delete endpoint for anime series - Add automatic migration on startup in fastapi_app.py lifespan - Add 28 unit tests for migration service - Add 14 integration tests for migration flow - Update infrastructure.md and database README docs Migration runs automatically on startup, legacy data files preserved.
This commit is contained in:
@@ -162,7 +162,83 @@ All series-related WebSocket events include `key` as the primary identifier in t
|
||||
|
||||
- `AnimeSeriesService.get_by_key(key)` - **Primary lookup method**
|
||||
- `AnimeSeriesService.get_by_id(id)` - Internal lookup by database ID
|
||||
- No `get_by_folder()` method exists - folder is never used for lookups
|
||||
- `AnimeSeriesService.get_all(db)` - Get all series from database
|
||||
- `AnimeSeriesService.create(db, key, name, site, folder, episode_dict)` - Create new series
|
||||
- `AnimeSeriesService.update(db, id, **kwargs)` - Update existing series
|
||||
- `AnimeSeriesService.delete(db, id)` - Delete series by ID
|
||||
- `AnimeSeriesService.upsert_sync(db, key, name, site, folder, episode_dict)` - Sync upsert for scanner
|
||||
|
||||
No `get_by_folder()` method exists - folder is never used for lookups.
|
||||
|
||||
## Data Storage Migration
|
||||
|
||||
### Background
|
||||
|
||||
The application has migrated from file-based storage to SQLite database storage for anime series metadata.
|
||||
|
||||
**Previous Storage (Deprecated)**:
|
||||
- Individual `data` files (no extension) in each anime folder
|
||||
- Example: `/anime-directory/Attack on Titan (2013)/data`
|
||||
|
||||
**Current Storage (Database)**:
|
||||
- SQLite database at `data/aniworld.db`
|
||||
- Managed by `AnimeSeriesService` using SQLAlchemy
|
||||
|
||||
### Migration Service
|
||||
|
||||
The `DataMigrationService` handles automatic migration of legacy data files to the database:
|
||||
|
||||
```python
|
||||
from src.server.services.data_migration_service import DataMigrationService
|
||||
|
||||
# Check for legacy files
|
||||
service = DataMigrationService()
|
||||
files = await service.check_for_legacy_data_files(anime_directory)
|
||||
|
||||
# Migrate all to database
|
||||
result = await service.migrate_all_legacy_data(anime_directory, db_session)
|
||||
print(result) # Migration Result: 10 migrated, 2 skipped, 0 failed
|
||||
|
||||
# Optional: cleanup old files with backup
|
||||
await service.cleanup_migrated_files(files, backup=True)
|
||||
```
|
||||
|
||||
### Automatic Migration on Startup
|
||||
|
||||
Migration runs automatically during application startup:
|
||||
|
||||
1. Database is initialized (`init_db()`)
|
||||
2. Legacy data files are detected
|
||||
3. Files are migrated to database
|
||||
4. Results are logged (no files are deleted automatically)
|
||||
|
||||
### Migration Result
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class MigrationResult:
|
||||
total_found: int # Total legacy files found
|
||||
migrated: int # Successfully migrated
|
||||
failed: int # Failed to migrate
|
||||
skipped: int # Already in database
|
||||
errors: List[str] # Error messages
|
||||
```
|
||||
|
||||
### Deprecation Notes
|
||||
|
||||
- **Legacy file-based storage is deprecated** - Do not create new `data` files
|
||||
- **SerieScanner**: Updated to save to database (with file fallback for CLI)
|
||||
- **API endpoints**: Now use database as primary source
|
||||
- **CLI**: Still uses file-based storage for backward compatibility
|
||||
|
||||
### Related Files
|
||||
|
||||
| File | Purpose |
|
||||
| ---- | ------- |
|
||||
| `src/server/services/data_migration_service.py` | Migration service |
|
||||
| `src/server/database/service.py` | Database CRUD operations |
|
||||
| `src/server/database/models.py` | SQLAlchemy models |
|
||||
| `src/core/SerieScanner.py` | Scanner with DB support |
|
||||
|
||||
## Core Services
|
||||
|
||||
|
||||
Reference in New Issue
Block a user