Replace check-then-insert race condition with INSERT ON CONFLICT.
- upsert_pending uses RETURNING id for atomic upsert
- UNIQUE(source_id, content_hash) constraint from migration 6
- blocklist_import_workflow updated to use upsert_pending
- test_import_source_success fixed for async mock patterns
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The API pagination infrastructure was already correctly implemented with:
- PaginatedListResponse base model containing 'items' and 'pagination' fields
- PaginationMetadata object with all required fields (page, page_size, total, total_pages, has_next_page, has_prev_page)
- All services correctly calling create_pagination_metadata()
However, there were two bugs preventing tests from passing:
1. IMPORT BUG: time_utils.py was importing TIME_RANGE_SECONDS from app.models.ban
when it's actually defined in app.models._common. This caused import errors
in tests that exercise time-range filtering.
2. TEST BUG: Test assertions were using outdated API structure, accessing
.total, .page, .page_size directly on paginated responses instead of
through the .pagination object.
Fixed locations:
- test_mappers/test_ban_mappers.py: 3 assertions updated to use .pagination.*
- test_services/test_blocklist_service.py: 6 assertions updated
- test_services/test_history_service.py: 14 assertions updated
All paginated API endpoints now correctly return pagination metadata:
- GET /api/history
- GET /api/history/archive
- GET /api/dashboard/bans
- GET /api/jails/{name}/banned
- GET /api/blocklists/log
Verified with 24 passing pagination tests demonstrating correct behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CRITICAL FIX: Background tasks (especially blocklist_import) crashed mid-execution,
leaving partial state. On retry, the same bans were applied again, causing duplicates.
Solution: Content-hash based operation tracking for blocklist imports:
- Added import_runs table (migration 6) to track operations by source + content hash
- Before banning, check if this exact content has already been imported
- If completed: skip banning (already done), optionally re-warm cache
- If new or failed: proceed with ban and mark as completed or failed
Changes:
- Database: Migration 6 adds import_runs table with operation state tracking
- Model: Added ImportRunEntry for import run records
- Repository: New import_run_repo module with CRUD operations
- Workflow: Updated blocklist_import_workflow to check operation history before banning
- Dependencies: Registered import_run_repo for dependency injection
- Tests: Added test_import_source_idempotent_on_retry and test_import_source_different_content_not_reused
- Documentation: Added Task Idempotency section to Backend-Development.md
Verification:
- All 7 import tests pass (5 existing + 2 new idempotency tests)
- Type checking: mypy --strict ✅
- Linting: ruff ✅
- No API changes, backwards compatible via automatic migration
Fixes: Background tasks not idempotent #CRITICAL
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Task 4 (Better Jail Configuration) implementation:
- Add fail2ban_config_dir setting to app/config.py
- New file_config_service: list/view/edit/create jail.d, filter.d, action.d files
with path-traversal prevention and 512 KB content size limit
- New file_config router: GET/PUT/POST endpoints for jail files, filter files,
and action files; PUT .../enabled for toggle on/off
- Extend config_service with delete_log_path() and add_log_path()
- Add DELETE /api/config/jails/{name}/logpath and POST /api/config/jails/{name}/logpath
- Extend geo router with re-resolve endpoint; add geo_re_resolve background task
- Update blocklist_service with revised scheduling helpers
- Update Docker compose files with BANGUI_FAIL2BAN_CONFIG_DIR env var and
rw volume mount for the fail2ban config directory
- Frontend: new Jail Files, Filters, Actions tabs in ConfigPage; file editor
with accordion-per-file, editable textarea, save/create; add/delete log paths
- Frontend: types in types/config.ts; API calls in api/config.ts and api/endpoints.ts
- 63 new backend tests (test_file_config_service, test_file_config, test_geo_re_resolve)
- 6 new frontend tests in ConfigPageLogPath.test.tsx
- ruff, mypy --strict, tsc --noEmit, eslint: all clean; 617 backend tests pass
When the most recent scheduled import completed with errors, surface the
failure in the persistent app shell:
- A warning MessageBar appears at top of main content area
- An amber badge is rendered on the Blocklists sidebar nav item
Backend: add last_run_errors: bool | None to ScheduleInfo model and
populate it in get_schedule_info() from the latest import_log row.
Frontend: extend ScheduleInfo type, add useBlocklistStatus polling hook,
wire both indicators into MainLayout.
Tests: 3 new service tests + 1 new router test (433 total, all pass).
_is_not_found_error in jail_service did not match the concatenated form
'unknownjailexception' that fail2ban produces when it serialises
UnknownJailException, so JailOperationError was raised instead of
JailNotFoundError and every ban attempt in the import loop failed
individually, skipping all 27 840 IPs before returning an error.
Two changes:
- Add 'unknownjail' to the phrase list in _is_not_found_error so that
UnknownJailException is correctly mapped to JailNotFoundError.
- In blocklist_service.import_source, catch JailNotFoundError explicitly
and break out of the loop immediately with a warning log instead of
retrying on every IP.
