Replace check-then-insert race condition with INSERT ON CONFLICT.
- upsert_pending uses RETURNING id for atomic upsert
- UNIQUE(source_id, content_hash) constraint from migration 6
- blocklist_import_workflow updated to use upsert_pending
- test_import_source_success fixed for async mock patterns
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit addresses race conditions in multi-step database operations by:
1. Wrap write operations in BEGIN IMMEDIATE ... COMMIT transactions:
- import_run_repo: create_pending, mark_completed, mark_failed
- geo_cache_repo: all upsert_*_and_commit functions
- geo_cache_repo: bulk_upsert_entries_and_neg_entries_and_commit
2. Handle concurrent write collisions gracefully:
- import_run_repo.create_pending can now raise IntegrityError
- blocklist_import_workflow catches IntegrityError and retries lookup
- Logs 'blocklist_import_lost_race' event when another request wins the race
3. Add comprehensive documentation:
- Backend-Development.md § 6.3 Database Transactions
- Explains when to use BEGIN IMMEDIATE
- Shows transaction pattern with try-except-rollback
- Documents race condition error handling pattern
The solution leverages SQLite's UNIQUE constraint for data integrity while
handling the concurrent case gracefully in application logic. This is more
efficient than using BEGIN EXCLUSIVE which would serialize all writers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CRITICAL FIX: Background tasks (especially blocklist_import) crashed mid-execution,
leaving partial state. On retry, the same bans were applied again, causing duplicates.
Solution: Content-hash based operation tracking for blocklist imports:
- Added import_runs table (migration 6) to track operations by source + content hash
- Before banning, check if this exact content has already been imported
- If completed: skip banning (already done), optionally re-warm cache
- If new or failed: proceed with ban and mark as completed or failed
Changes:
- Database: Migration 6 adds import_runs table with operation state tracking
- Model: Added ImportRunEntry for import run records
- Repository: New import_run_repo module with CRUD operations
- Workflow: Updated blocklist_import_workflow to check operation history before banning
- Dependencies: Registered import_run_repo for dependency injection
- Tests: Added test_import_source_idempotent_on_retry and test_import_source_different_content_not_reused
- Documentation: Added Task Idempotency section to Backend-Development.md
Verification:
- All 7 import tests pass (5 existing + 2 new idempotency tests)
- Type checking: mypy --strict ✅
- Linting: ruff ✅
- No API changes, backwards compatible via automatic migration
Fixes: Background tasks not idempotent #CRITICAL
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
