Add queue deduplication to prevent duplicate entries
- In-memory dedup in add_to_queue() using _pending_by_episode dict - Batch-local dedup via seen_in_batch set (handles duplicates within single call) - Database unique index on episode_id via __table_args__ - 5-minute cooldown in _auto_download_missing() to prevent rapid re-triggers - Updated _add_to_pending_queue() and _remove_from_pending_queue() to track episode keys - Added TestQueueDeduplication with 4 test cases - Updated DEVELOPMENT.md and TESTING.md with queue dedup docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@@ -61,4 +61,56 @@ This document provides guidance for developers working on the Aniworld project.
|
||||
- Commit message format
|
||||
- Pull request process
|
||||
8. Common Development Tasks
|
||||
|
||||
### Adding Queue Deduplication
|
||||
|
||||
The download queue prevents duplicate entries at two levels:
|
||||
|
||||
**In-Memory Deduplication** (`src/server/services/download_service.py`):
|
||||
- `_pending_by_episode` dict tracks pending episodes: key = `(serie_id, season, episode)`
|
||||
- `_add_to_pending_queue()` updates the dict when adding items
|
||||
- `add_to_queue()` checks this dict before adding episodes (includes batch-local dedup)
|
||||
- `_remove_from_pending_queue()` cleans up the dict when items are removed
|
||||
|
||||
**Database Constraint** (`src/server/models.py`):
|
||||
- `DownloadQueueItem` has a unique index on `episode_id` via `__table_args__`
|
||||
- Prevents duplicate queue entries at the database level
|
||||
- Unique constraint: `Index("ix_download_queue_episode_pending", "episode_id", unique=True)`
|
||||
|
||||
**Scheduler Cooldown** (`src/server/services/scheduler_service.py`):
|
||||
- `_last_auto_download_time` tracks when auto-download last ran
|
||||
- 5-minute cooldown prevents rapid re-triggers
|
||||
- Checked at start of `_auto_download_missing()`
|
||||
|
||||
### Mocking the Download Queue
|
||||
|
||||
When testing components that use the download queue:
|
||||
|
||||
```python
|
||||
# Mock repository for unit tests
|
||||
class MockQueueRepository:
|
||||
def __init__(self):
|
||||
self._items: Dict[str, DownloadItem] = {}
|
||||
|
||||
async def save_item(self, item: DownloadItem) -> DownloadItem:
|
||||
self._items[item.id] = item
|
||||
return item
|
||||
|
||||
async def get_all_items(self) -> List[DownloadItem]:
|
||||
return list(self._items.values())
|
||||
|
||||
# Use in fixture
|
||||
@pytest.fixture
|
||||
def mock_queue_repository():
|
||||
return MockQueueRepository()
|
||||
|
||||
@pytest.fixture
|
||||
def download_service(mock_anime_service, mock_queue_repository):
|
||||
return DownloadService(
|
||||
anime_service=mock_anime_service,
|
||||
queue_repository=mock_queue_repository,
|
||||
max_retries=3,
|
||||
)
|
||||
```
|
||||
|
||||
9. Troubleshooting Development Issues
|
||||
|
||||
Reference in New Issue
Block a user