lukas.pupkalipinski/Aniworld
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

NoDataFile #1

Merged
lukas.pupkalipinski merged 70 commits from NoDataFile into main 2026-01-09 18:42:18 +01:00
Conversation 0 Commits 70 Files Changed 161 +20606 -7422
lukas.pupkalipinski commented 2026-01-09 18:42:05 +01:00
Owner
Copy Link

Pull Request Template

Description

Brief description of the changes in this PR.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Code refactoring
  • Performance improvement
  • Test improvement

Changes Made

  • List the main changes
  • Include any new files added
  • Include any files removed or renamed

Testing

  • Unit tests pass
  • Integration tests pass
  • Manual testing completed
  • Performance testing (if applicable)

Screenshots (if applicable)

Add screenshots of UI changes or new features.

Checklist

  • My code follows the project's coding standards
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published

Related Issues

Fixes #(issue number)
Related to #(issue number)

Additional Notes

Any additional information, deployment notes, or context for reviewers.

# Pull Request Template ## Description Brief description of the changes in this PR. ## Type of Change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update - [ ] Code refactoring - [ ] Performance improvement - [ ] Test improvement ## Changes Made - List the main changes - Include any new files added - Include any files removed or renamed ## Testing - [ ] Unit tests pass - [ ] Integration tests pass - [ ] Manual testing completed - [ ] Performance testing (if applicable) ## Screenshots (if applicable) Add screenshots of UI changes or new features. ## Checklist - [ ] My code follows the project's coding standards - [ ] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published ## Related Issues Fixes #(issue number) Related to #(issue number) ## Additional Notes Any additional information, deployment notes, or context for reviewers.
lukas.pupkalipinski added 70 commits 2026-01-09 18:42:06 +01:00 
- Add db_session and skip_load parameters to SerieList.__init__
- Add async load_series_from_db() method for database loading
- Add async add_to_db() method for database storage
- Add async contains_in_db() method for database checks
- Add _convert_from_db() and _convert_to_db_dict() helper methods
- Add deprecation warnings to file-based add() method
- Maintain backward compatibility for file-based operations
- Add comprehensive unit tests (29 tests, all passing)
- Update instructions.md to mark Task 4 complete
- Add db_session parameter to SerieScanner.__init__
- Add async scan_async() method for database-backed scanning
- Add _save_serie_to_db() helper for creating/updating series
- Add _update_serie_in_db() helper for updating existing series
- Add deprecation warning to file-based scan() method
- Maintain backward compatibility for CLI usage
- Add comprehensive unit tests (15 tests, all passing)
- Update instructions.md to mark Task 5 complete
Task 6: Update Anime API endpoints to use database
- Modified add_series endpoint to save series to database when available
- Added get_optional_database_session dependency for graceful fallback
- Falls back to file-based storage when database unavailable
- All 55 API tests and 809 unit tests pass
- Added db_session parameter to SeriesApp.__init__()
- Added db_session property and set_db_session() method
- Added init_from_db_async() for async database initialization
- Pass db_session to SerieList and SerieScanner during construction
- Added get_series_app_with_db() dependency for FastAPI endpoints
- All 815 unit tests and 55 API tests pass
Task 8: Write integration tests for data file migration
- Added test_migration_on_fresh_start_no_data_files test
- Added test_add_series_saves_to_database test
- Added test_scan_async_saves_to_database test
- Added test_load_series_from_db test
- Added test_search_and_add_workflow test
- All 11 migration integration tests pass
- All 870 tests pass (815 unit + 55 API)
Task 9: Clean up legacy code
- Added deprecation warnings to Serie.save_to_file() and load_from_file()
- Updated infrastructure.md with Data Storage section documenting:
  - SQLite database as primary storage
  - Legacy file storage as deprecated
  - Data migration process
- Added deprecation warning tests for Serie class
- Updated existing tests to handle new warnings
- All 1012 tests pass (872 unit + 55 API + 85 integration)
Task 10: Final Validation - All checks passed
- All 817 unit tests pass
- All 140 integration tests pass
- All 55 API tests pass
- Total: 1012 tests passing

All 10 migration tasks completed:
1. ✅ Create Data File Migration Service
2. ✅ Create Startup Migration Script
3. ✅ Integrate Migration into FastAPI Lifespan
4. ✅ Update SerieList to Use Database
5. ✅ Update SerieScanner to Use Database
6. ✅ Update Anime API Endpoints
7. ✅ Update Dependencies and SeriesApp
8. ✅ Write Integration Tests
9. ✅ Clean Up Legacy Code
10. ✅ Final Validation
- Created QueueRepository adapter in src/server/services/queue_repository.py
- Refactored DownloadService to use repository pattern instead of JSON
- Updated application startup to initialize download service from database
- Updated all test fixtures to use MockQueueRepository
- All 1104 tests passing
- Updated infrastructure.md with queue database schema and storage details
- Updated instructions.md to mark migration task as completed
- No deprecated JSON code remains in codebase
- Add get_all_series_from_data_files() to SeriesApp
- Sync series from data files to DB on startup
- Add unit tests for new SeriesApp method
- Add integration tests for sync functionality
- Update documentation
- Moved _sync_series_to_database from fastapi_app.py to anime_service.py
- Renamed to sync_series_from_data_files for better clarity
- Updated all imports and test references
- Removed completed TODO tasks from instructions.md
- Call sync_series_from_data_files after initial setup completes
- Call sync_series_from_data_files when anime directory is updated
- Return synced_series count in directory update response
- Use folder name as fallback when series name is empty
- Skip series with both empty name and folder
- Add try/catch for individual series to prevent one failure
  from stopping the entire sync
- Fix return type annotation in SetupRedirectMiddleware.dispatch() to use Response instead of RedirectResponse
- Replace broad 'except Exception' with specific exception types (FileNotFoundError, ValueError, OSError, etc.)
- Rename AppConfig.validate() to validate_config() to avoid shadowing BaseModel.validate()
- Fix ValidationResult.errors field to use List[str] with default_factory
- Add pylint disable comments for intentional broad exception catches during shutdown
- Rename lifespan parameter to _application to indicate unused variable
- Update all callers to use new validate_config() method name
- Update SeriesApp.rescan() to use database storage by default (use_database=True)
- Use SerieScanner.scan_async() for database mode, which saves directly to DB
- Fall back to legacy file-based scan() when use_database=False (for CLI compatibility)
- Reinitialize SerieList from database after scan when in database mode
- Update unit tests to use use_database=False for mocked tests
- Add parameter to control storage mode for backward compatibility
- Add documentation warnings for in-memory rate limiting and failed login attempts
- Consolidate duplicate health endpoints into api/health.py
- Fix CLI to use correct async rescan method names
- Update download.py and anime.py to use custom exception classes
- Add WebSocket room validation and rate limiting
- Remove db_session parameter from SeriesApp, SerieList, SerieScanner
- Move all database operations to AnimeService (service layer)
- Add add_series_to_db, contains_in_db methods to AnimeService
- Update sync_series_from_data_files to use inline DB operations
- Remove obsolete test classes for removed DB methods
- Fix pylint issues: add broad-except comments, fix line lengths
- Core layer (src/core/) now has zero database imports

722 unit tests pass
- Update _update_series_in_db to sync missing episodes bidirectionally
- Add delete_by_series_and_episode method to EpisodeService
- Remove downloaded episodes from DB after successful download
- Clear anime service cache when episodes are removed
- Fix tests to use 'message' instead of 'detail' in API responses
- Mock DB operations in rescan tests
- Fixed room name mismatch: ProgressService was broadcasting to
  'download_progress' but JS clients join 'downloads' room
- Added _get_room_for_progress_type() mapping function
- Updated all progress methods to use correct room names
- Added 13 new tests for room name mapping and broadcast verification
- Updated existing tests to expect correct room names
- Fixed JS clients to join valid rooms (downloads, queue, scan)
- Add broadcast_scan_started, broadcast_scan_progress, broadcast_scan_completed to WebSocketService
- Inject WebSocketService into AnimeService for real-time scan progress broadcasts
- Add CSS styles for scan progress overlay with spinner, stats, and completion state
- Update app.js to handle scan events and display progress overlay
- Add unit tests for new WebSocket broadcast methods
- All 1022 tests passing
- Add total_items parameter to broadcast_scan_started and broadcast_scan_progress
- Pass total from SeriesApp to WebSocket broadcasts in AnimeService
- Update JS overlay to show progress bar and current/total count
- Add CSS for progress bar styling
- Add unit tests for new total_items parameter
- All 1024 tests passing
- Add is_scanning state tracking in AnimeService
- Add get_scan_status method to AnimeService
- Add /api/anime/scan/status endpoint to check scan state
- Add checkActiveScanStatus in JS to restore overlay on reconnect
- All 1024 tests passing
- Add _scan_lock asyncio.Lock to AnimeService
- Check if lock is held before starting rescan
- Use async with to ensure lock is released on completion or exception
- All 1024 tests passing
- Show overlay immediately when rescan is clicked (before API response)
- Add click-outside-to-close on overlay background
- Add click on rescan-status indicator to reopen overlay
- Add cursor pointer to rescan-status for clickability feedback
- All 1024 tests passing
- Added defensive check for rescan-status element before adding event listener
- Added e.stopPropagation() to prevent click event bubbling issues
- Added console logging for debugging click events
- Call checkActiveScanStatus() directly in init() method, not just on socket connect
  This ensures scan status is checked immediately on page load even if WebSocket
  connection is delayed
- Add debug logging to checkActiveScanStatus() for better tracing
- Update status indicator before showing overlay for faster feedback
- Add warning logs when DOM elements are not found
- Ensure idle state is explicitly set when no scan is running
- Add debug logging to AnimeService.get_scan_status()
- Create transaction.py with @transactional decorator, atomic() context manager
- Add TransactionPropagation modes: REQUIRED, REQUIRES_NEW, NESTED
- Add savepoint support for nested transactions with partial rollback
- Update connection.py with TransactionManager, get_transactional_session
- Update service.py with bulk operations (bulk_mark_downloaded, bulk_delete)
- Wrap QueueRepository.save_item() and clear_all() in atomic transactions
- Add comprehensive tests (66 transaction tests, 90% coverage)
- All 1090 tests passing
- Add WebSocket shutdown() with client notification and graceful close
- Enhance download service stop() with pending state persistence
- Expand FastAPI lifespan shutdown with proper cleanup sequence
- Add SQLite WAL checkpoint before database close
- Update stop_server.sh to use SIGTERM with timeout fallback
- Configure uvicorn timeout_graceful_shutdown=30s
- Update ARCHITECTURE.md with shutdown documentation
- Add sanitize_folder_name utility for filesystem-safe folder names
- Add sanitized_folder property to Serie entity
- Update SerieList.add() to use sanitized display names for folders
- Add scan_single_series() method for targeted episode scanning
- Enhance add_series endpoint: DB save -> folder create -> targeted scan
- Update response to include missing_episodes and total_missing
- Add comprehensive unit tests for new functionality
- Update API tests with proper mock support
- Add cancellation flag to AniworldLoader with request_cancel/reset_cancel/is_cancelled methods
- Update base_provider.Loader interface with cancellation abstract methods
- Integrate cancellation check in YT-DLP progress hooks
- Add request_download_cancel method to SeriesApp and AnimeService
- Update DownloadService.stop() to request cancellation before shutdown
- Clean up temp files on cancellation
- Import and use DownloadCancelled exception which YT-DLP properly handles
- Add InterruptedError handling throughout the call chain
- Fire 'cancelled' status event when download is cancelled
- Handle InterruptedError in DownloadService to set CANCELLED status
- Replace callback system with events library in SerieScanner
- Update SeriesApp to subscribe to loader and scanner events
- Refactor ScanService to use Events instead of CallbackManager
- Remove CallbackManager imports and callback classes
- Add safe event calling with error handling in SerieScanner
- Update AniworldLoader to use Events for download progress
- Remove progress_callback parameter from download methods
- Update all affected tests for Events pattern
- Fix test_series_app.py for new event subscription model
- Comment out obsolete callback tests in test_scan_service.py

All core tests passing. Events provide cleaner event-driven architecture.
- Add ThreadPoolExecutor with 3 max workers to SeriesApp
- Replace all asyncio.to_thread calls with loop.run_in_executor
- Add shutdown() method to properly cleanup executor
- Integrate SeriesApp.shutdown() into FastAPI shutdown sequence
- Ensures proper resource cleanup on Ctrl+C (SIGINT/SIGTERM)
- Remove None assignment for download_progress event in AniworldLoader
- Remove None assignments for download_status and scan_status events in SeriesApp
- Events library requires events to not be initialized to None
- Verified logging configuration is properly set to INFO level
- Update fastapi_app.py to use INFO level instead of DEBUG
- Update development.py config to default to INFO instead of DEBUG
- Update uvicorn log_level from debug to info
- Prevents debug messages from appearing in logs
lukas.pupkalipinski merged commit 489c37357e into main 2026-01-09 18:42:18 +01:00
lukas.pupkalipinski deleted branch NoDataFile 2026-01-09 18:42:18 +01:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
lukas.pupkalipinski
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lukas.pupkalipinski/Aniworld#1
No description provided.
Delete Branch "NoDataFile"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?