Add database transaction support with atomic operations
- Create transaction.py with @transactional decorator, atomic() context manager - Add TransactionPropagation modes: REQUIRED, REQUIRES_NEW, NESTED - Add savepoint support for nested transactions with partial rollback - Update connection.py with TransactionManager, get_transactional_session - Update service.py with bulk operations (bulk_mark_downloaded, bulk_delete) - Wrap QueueRepository.save_item() and clear_all() in atomic transactions - Add comprehensive tests (66 transaction tests, 90% coverage) - All 1090 tests passing
This commit is contained in:
@@ -121,147 +121,202 @@ For each task completed:
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Current Task: Make MP4 Scanning Progress Visible in UI
|
||||
---
|
||||
|
||||
### Problem Statement
|
||||
## Task: Add Database Transaction Support
|
||||
|
||||
When users trigger a library rescan (via the "Rescan Library" button on the anime page), the MP4 file scanning happens silently in the background. Users only see a brief toast message, but there's no visual feedback showing:
|
||||
### Objective
|
||||
|
||||
1. That scanning is actively happening
|
||||
2. How many files/directories have been scanned
|
||||
3. The progress through the scan operation
|
||||
4. When scanning is complete with results
|
||||
Implement proper transaction handling across all database write operations using SQLAlchemy's transaction support. This ensures data consistency and prevents partial writes during compound operations.
|
||||
|
||||
Currently, the only indication is in server logs:
|
||||
### Background
|
||||
|
||||
```
|
||||
INFO: Starting directory rescan
|
||||
INFO: Scanning for .mp4 files
|
||||
Currently, the application uses SQLAlchemy sessions with auto-commit behavior through the `get_db_session()` generator. While individual operations are atomic, compound operations (multiple writes) can result in partial commits if an error occurs mid-operation.
|
||||
|
||||
### Requirements
|
||||
|
||||
1. **All database write operations must be wrapped in explicit transactions**
|
||||
2. **Compound operations must be atomic** - either all writes succeed or all fail
|
||||
3. **Nested operations should use savepoints** for partial rollback capability
|
||||
4. **Existing functionality must not break** - backward compatible changes only
|
||||
5. **All tests must pass after implementation**
|
||||
|
||||
---
|
||||
|
||||
### Step 1: Create Transaction Utilities Module
|
||||
|
||||
**File**: `src/server/database/transaction.py`
|
||||
|
||||
Create a new module providing transaction management utilities:
|
||||
|
||||
1. **`@transactional` decorator** - Wraps a function in a transaction boundary
|
||||
|
||||
- Accepts a session parameter or retrieves one via dependency injection
|
||||
- Commits on success, rolls back on exception
|
||||
- Re-raises exceptions after rollback
|
||||
- Logs transaction start, commit, and rollback events
|
||||
|
||||
2. **`TransactionContext` class** - Context manager for explicit transaction control
|
||||
|
||||
- Supports `with` statement usage
|
||||
- Provides `savepoint()` method for nested transactions using `begin_nested()`
|
||||
- Handles commit/rollback automatically
|
||||
|
||||
3. **`atomic()` function** - Async context manager for async operations
|
||||
- Same behavior as `TransactionContext` but for async code
|
||||
|
||||
**Interface Requirements**:
|
||||
|
||||
- Decorator must work with both sync and async functions
|
||||
- Must handle the case where session is already in a transaction
|
||||
- Must support optional `propagation` parameter (REQUIRED, REQUIRES_NEW, NESTED)
|
||||
|
||||
---
|
||||
|
||||
### Step 2: Update Connection Module
|
||||
|
||||
**File**: `src/server/database/connection.py`
|
||||
|
||||
Modify the existing session management:
|
||||
|
||||
1. Add `get_transactional_session()` generator that does NOT auto-commit
|
||||
2. Add `TransactionManager` class for manual transaction control
|
||||
3. Keep `get_db_session()` unchanged for backward compatibility
|
||||
4. Add session state inspection utilities (`is_in_transaction()`, `get_transaction_depth()`)
|
||||
|
||||
---
|
||||
|
||||
### Step 3: Wrap Service Layer Operations
|
||||
|
||||
**File**: `src/server/database/service.py`
|
||||
|
||||
Apply transaction handling to all compound write operations:
|
||||
|
||||
**AnimeService**:
|
||||
|
||||
- `create_anime_with_episodes()` - if exists, wrap in transaction
|
||||
- Any method that calls multiple repository methods
|
||||
|
||||
**EpisodeService**:
|
||||
|
||||
- `bulk_update_episodes()` - if exists
|
||||
- `mark_episodes_downloaded()` - if handles multiple episodes
|
||||
|
||||
**DownloadQueueService**:
|
||||
|
||||
- `add_batch_to_queue()` - if exists
|
||||
- `clear_and_repopulate()` - if exists
|
||||
- Any method performing multiple writes
|
||||
|
||||
**SessionService**:
|
||||
|
||||
- `rotate_session()` - delete old + create new must be atomic
|
||||
- `cleanup_expired_sessions()` - bulk delete operation
|
||||
|
||||
**Pattern to follow**:
|
||||
|
||||
```python
|
||||
@transactional
|
||||
def compound_operation(self, session: Session, data: SomeModel) -> Result:
|
||||
# Multiple write operations here
|
||||
# All succeed or all fail
|
||||
```
|
||||
|
||||
### Desired Outcome
|
||||
---
|
||||
|
||||
Users should see real-time progress in the UI during library scanning with:
|
||||
### Step 4: Update Queue Repository
|
||||
|
||||
1. **Progress overlay** showing scan is active with a spinner animation
|
||||
2. **Live counters** showing directories scanned and files found
|
||||
3. **Current directory display** showing which folder is being scanned (truncated if too long)
|
||||
4. **Completion summary** showing total files found, directories scanned, and elapsed time
|
||||
5. **Auto-dismiss** the overlay after showing completion summary
|
||||
**File**: `src/server/services/queue_repository.py`
|
||||
|
||||
Ensure atomic operations for:
|
||||
|
||||
1. `save_item()` - check existence + insert/update must be atomic
|
||||
2. `remove_item()` - if involves multiple deletes
|
||||
3. `clear_all_items()` - bulk delete should be transactional
|
||||
4. `reorder_queue()` - multiple position updates must be atomic
|
||||
|
||||
---
|
||||
|
||||
### Step 5: Update API Endpoints
|
||||
|
||||
**Files**: `src/server/api/anime.py`, `src/server/api/downloads.py`, `src/server/api/auth.py`
|
||||
|
||||
Review and update endpoints that perform multiple database operations:
|
||||
|
||||
1. Identify endpoints calling multiple service methods
|
||||
2. Wrap in transaction boundary at the endpoint level OR ensure services handle it
|
||||
3. Prefer service-level transactions over endpoint-level for reusability
|
||||
|
||||
---
|
||||
|
||||
### Step 6: Add Unit Tests
|
||||
|
||||
**File**: `tests/unit/test_transactions.py`
|
||||
|
||||
Create comprehensive tests:
|
||||
|
||||
1. **Test successful transaction commit** - verify all changes persisted
|
||||
2. **Test rollback on exception** - verify no partial writes
|
||||
3. **Test nested transaction with savepoint** - verify partial rollback works
|
||||
4. **Test decorator with sync function**
|
||||
5. **Test decorator with async function**
|
||||
6. **Test context manager usage**
|
||||
7. **Test transaction propagation modes**
|
||||
|
||||
**File**: `tests/unit/test_service_transactions.py`
|
||||
|
||||
1. Test each service's compound operations for atomicity
|
||||
2. Mock exceptions mid-operation to verify rollback
|
||||
3. Verify no orphaned data after failed operations
|
||||
|
||||
---
|
||||
|
||||
### Step 7: Update Integration Tests
|
||||
|
||||
**File**: `tests/integration/test_db_transactions.py`
|
||||
|
||||
1. Test real database transaction behavior
|
||||
2. Test concurrent transaction handling
|
||||
3. Test transaction isolation levels if applicable
|
||||
|
||||
---
|
||||
|
||||
### Step 7: Update Dokumentation
|
||||
|
||||
1. Check Docs folder and updated the needed files
|
||||
|
||||
---
|
||||
|
||||
### Implementation Notes
|
||||
|
||||
- **SQLAlchemy Pattern**: Use `session.begin_nested()` for savepoints
|
||||
- **Error Handling**: Always log transaction failures with full context
|
||||
- **Performance**: Transactions have overhead - don't wrap single operations unnecessarily
|
||||
- **Testing**: Use `session.rollback()` in test fixtures to ensure clean state
|
||||
|
||||
### Files to Modify
|
||||
|
||||
#### 1. `src/server/services/websocket_service.py`
|
||||
|
||||
Add three new broadcast methods for scan events:
|
||||
|
||||
- **broadcast_scan_started**: Notify clients that a scan has begun, include the root directory path
|
||||
- **broadcast_scan_progress**: Send periodic updates with directories scanned count, files found count, and current directory name
|
||||
- **broadcast_scan_completed**: Send final summary with total directories, total files, and elapsed time in seconds
|
||||
|
||||
Follow the existing pattern used by `broadcast_download_progress` for message structure consistency.
|
||||
|
||||
#### 2. `src/server/services/scanner_service.py`
|
||||
|
||||
Modify the scanning logic to emit progress via WebSocket:
|
||||
|
||||
- Inject `WebSocketService` dependency into the scanner service
|
||||
- At scan start, call `broadcast_scan_started`
|
||||
- During directory traversal, track directories scanned and files found
|
||||
- Every 10 directories (to avoid WebSocket spam), call `broadcast_scan_progress`
|
||||
- Track elapsed time using `time.time()`
|
||||
- At scan completion, call `broadcast_scan_completed` with summary statistics
|
||||
- Ensure the scan still works correctly even if WebSocket broadcast fails (wrap in try/except)
|
||||
|
||||
#### 3. `src/server/static/css/style.css`
|
||||
|
||||
Add styles for the scan progress overlay:
|
||||
|
||||
- Full-screen semi-transparent overlay (z-index high enough to be on top)
|
||||
- Centered container with background matching theme (use CSS variables)
|
||||
- Spinner animation using CSS keyframes
|
||||
- Styling for current directory text (truncated with ellipsis)
|
||||
- Styling for statistics display
|
||||
- Success state styling for completion
|
||||
- Ensure it works in both light and dark mode themes
|
||||
|
||||
#### 4. `src/server/static/js/anime.js`
|
||||
|
||||
Add WebSocket message handlers and UI functions:
|
||||
|
||||
- Handle `scan_started` message: Create and show progress overlay with spinner
|
||||
- Handle `scan_progress` message: Update directory count, file count, and current directory text
|
||||
- Handle `scan_completed` message: Show completion summary, then auto-remove overlay after 3 seconds
|
||||
- Ensure overlay is properly cleaned up if page navigates away
|
||||
- Update the existing rescan button handler to work with the new progress system
|
||||
|
||||
### WebSocket Message Types
|
||||
|
||||
Define three new message types following the existing project patterns:
|
||||
|
||||
1. **scan_started**: type, directory path, timestamp
|
||||
2. **scan_progress**: type, directories_scanned, files_found, current_directory, timestamp
|
||||
3. **scan_completed**: type, total_directories, total_files, elapsed_seconds, timestamp
|
||||
|
||||
### Implementation Steps
|
||||
|
||||
1. First modify `websocket_service.py` to add the three new broadcast methods
|
||||
2. Add unit tests for the new WebSocket methods
|
||||
3. Modify `scanner_service.py` to use the new broadcast methods during scanning
|
||||
4. Add CSS styles to `style.css` for the progress overlay
|
||||
5. Update `anime.js` to handle the new WebSocket messages and display the UI
|
||||
6. Test the complete flow manually
|
||||
7. Verify all existing tests still pass
|
||||
|
||||
### Testing Requirements
|
||||
|
||||
**Unit Tests:**
|
||||
|
||||
- Test each new WebSocket broadcast method
|
||||
- Test that scanner service calls WebSocket methods at appropriate times
|
||||
- Mock WebSocket service in scanner tests
|
||||
|
||||
**Manual Testing:**
|
||||
|
||||
- Start server and login
|
||||
- Navigate to anime page
|
||||
- Click "Rescan Library" button
|
||||
- Verify overlay appears immediately with spinner
|
||||
- Verify counters update during scan
|
||||
- Verify current directory updates
|
||||
- Verify completion summary appears
|
||||
- Verify overlay auto-dismisses after 3 seconds
|
||||
- Test in both light and dark mode
|
||||
- Verify no JavaScript console errors
|
||||
| File | Action |
|
||||
| ------------------------------------------- | ------------------------------------------ |
|
||||
| `src/server/database/transaction.py` | CREATE - New transaction utilities |
|
||||
| `src/server/database/connection.py` | MODIFY - Add transactional session support |
|
||||
| `src/server/database/service.py` | MODIFY - Apply @transactional decorator |
|
||||
| `src/server/services/queue_repository.py` | MODIFY - Ensure atomic operations |
|
||||
| `src/server/api/anime.py` | REVIEW - Check for multi-write endpoints |
|
||||
| `src/server/api/downloads.py` | REVIEW - Check for multi-write endpoints |
|
||||
| `src/server/api/auth.py` | REVIEW - Check for multi-write endpoints |
|
||||
| `tests/unit/test_transactions.py` | CREATE - Transaction unit tests |
|
||||
| `tests/unit/test_service_transactions.py` | CREATE - Service transaction tests |
|
||||
| `tests/integration/test_db_transactions.py` | CREATE - Integration tests |
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- [x] Progress overlay appears immediately when scan starts
|
||||
- [x] Spinner animation is visible during scanning
|
||||
- [x] Directory counter updates periodically (every ~10 directories)
|
||||
- [x] Files found counter updates as MP4 files are discovered
|
||||
- [x] Current directory name is displayed (truncated if path is too long)
|
||||
- [x] Scan completion shows total directories, files, and elapsed time
|
||||
- [x] Overlay auto-dismisses 3 seconds after completion
|
||||
- [x] Works correctly in both light and dark mode
|
||||
- [x] No JavaScript errors in browser console
|
||||
- [x] All existing tests continue to pass
|
||||
- [x] New unit tests added and passing
|
||||
- [x] All database write operations use explicit transactions
|
||||
- [x] Compound operations are atomic (all-or-nothing)
|
||||
- [x] Exceptions trigger proper rollback
|
||||
- [x] No partial writes occur on failures
|
||||
- [x] All existing tests pass (1090 tests passing)
|
||||
- [x] New transaction tests pass with >90% coverage (90% achieved)
|
||||
- [x] Logging captures transaction lifecycle events
|
||||
- [x] Documentation updated in DATABASE.md
|
||||
- [x] Code follows project coding standards
|
||||
|
||||
### Edge Cases to Handle
|
||||
|
||||
- Empty directory with no MP4 files
|
||||
- Very large directory structure (ensure UI remains responsive)
|
||||
- WebSocket connection lost during scan (scan should still complete)
|
||||
- User navigates away during scan (cleanup overlay properly)
|
||||
- Rapid consecutive scan requests (debounce or queue)
|
||||
|
||||
### Notes
|
||||
|
||||
- Keep progress updates throttled to avoid overwhelming the WebSocket connection
|
||||
- Use existing CSS variables for colors to maintain theme consistency
|
||||
- Follow existing JavaScript patterns in the codebase
|
||||
- The scan functionality must continue to work even if WebSocket fails
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user