Stage 10: external blocklist importer — backend + frontend
- blocklist_repo.py: CRUD for blocklist_sources table - import_log_repo.py: add/list/get-last log entries - blocklist_service.py: source CRUD, preview, import (download/validate/ban), import_all, schedule get/set/info - blocklist_import.py: APScheduler task (hourly/daily/weekly schedule triggers) - blocklist.py router: 9 endpoints (list/create/update/delete/preview/import/ schedule-get+put/log) - blocklist.py models: ScheduleFrequency (StrEnum), ScheduleConfig, ScheduleInfo, ImportSourceResult, ImportRunResult, PreviewResponse - 59 new tests (18 repo + 19 service + 22 router); 374 total pass - ruff clean, mypy clean for Stage 10 files - types/blocklist.ts, api/blocklist.ts, hooks/useBlocklist.ts - BlocklistsPage.tsx: source management, schedule picker, import log table - Frontend tsc + ESLint clean
This commit is contained in:
@@ -326,44 +326,33 @@ Added `tests/test_routers/test_history.py` (11 tests) and `tests/test_services/t
|
||||
|
||||
---
|
||||
|
||||
## Stage 10 — External Blocklist Importer
|
||||
## Stage 10 — External Blocklist Importer ✅ DONE
|
||||
|
||||
This stage adds the ability to automatically download and apply external IP blocklists on a schedule.
|
||||
|
||||
### 10.1 Implement the blocklist repository
|
||||
### 10.1 Implement the blocklist repository ✅ DONE
|
||||
|
||||
Build `backend/app/repositories/blocklist_repo.py` and `backend/app/repositories/import_log_repo.py`. The blocklist repo persists blocklist source definitions (name, URL, enabled flag) in the application database. The import log repo records every import run with timestamp, source URL, IPs imported, IPs skipped, and any errors encountered. See [Architekture.md § 2.2 (Repositories)](Architekture.md).
|
||||
`backend/app/repositories/blocklist_repo.py` — CRUD for `blocklist_sources` table (`create_source`, `get_source`, `list_sources`, `list_enabled_sources`, `update_source`, `delete_source`). `backend/app/repositories/import_log_repo.py` — `add_log`, `list_logs` (paginated, optional `source_id` filter), `get_last_log`, `compute_total_pages`. 18 repository tests pass.
|
||||
|
||||
### 10.2 Implement the blocklist service
|
||||
### 10.2 Implement the blocklist service ✅ DONE
|
||||
|
||||
Build `backend/app/services/blocklist_service.py`. It manages blocklist source CRUD (add, edit, remove, toggle enabled). For the actual import: download each enabled source URL via aiohttp, validate every entry as a well-formed IP or CIDR range using the `ipaddress` module, skip malformed lines gracefully, and apply valid IPs as bans through fail2ban (in a dedicated blocklist jail) or via iptables. If using iptables, flush the chain before re-populating. Log every step with structlog. Record import results through the import log repository. Handle unreachable URLs by logging the error and continuing with remaining sources. See [Features.md § 8](Features.md).
|
||||
`backend/app/services/blocklist_service.py` — source CRUD, `preview_source` (downloads max 64 KB, validates, returns sample), `import_source` (downloads, validates IPs via `ipaddress`, bans through fail2ban blocklist jail, logs result), `import_all` (iterates all enabled sources), `get_schedule`/`set_schedule` (JSON in settings table), `get_schedule_info` (config + last run + next run). 19 service tests pass.
|
||||
|
||||
### 10.3 Implement the blocklist import scheduled task
|
||||
### 10.3 Implement the blocklist import scheduled task ✅ DONE
|
||||
|
||||
Create `backend/app/tasks/blocklist_import.py` — an APScheduler job that runs the blocklist service import at the configured schedule. The default is daily at 03:00. The schedule should be configurable through the blocklist service (saved in the app database). See [Features.md § 8 (Schedule)](Features.md).
|
||||
`backend/app/tasks/blocklist_import.py` — APScheduler job with stable ID `"blocklist_import"`. Supports three frequencies: `hourly` (interval trigger, every N hours), `daily` (cron trigger, UTC hour+minute), `weekly` (cron trigger, day_of_week+hour+minute). `register(app)` called at startup; `reschedule(app)` replaces the job after a schedule update.
|
||||
|
||||
### 10.4 Implement the blocklist router
|
||||
### 10.4 Implement the blocklist router ✅ DONE
|
||||
|
||||
Create `backend/app/routers/blocklist.py`:
|
||||
- `GET /api/blocklists` — list all blocklist sources with their status.
|
||||
- `POST /api/blocklists` — add a new source.
|
||||
- `PUT /api/blocklists/{id}` — edit a source (name, URL, enabled).
|
||||
- `DELETE /api/blocklists/{id}` — remove a source.
|
||||
- `GET /api/blocklists/{id}/preview` — download and display a sample of the blocklist contents.
|
||||
- `POST /api/blocklists/import` — trigger a manual import immediately ("Run Now").
|
||||
- `GET /api/blocklists/schedule` — get the current schedule and next run time.
|
||||
- `PUT /api/blocklists/schedule` — update the schedule.
|
||||
- `GET /api/blocklists/log` — paginated import log, filterable by source and date range.
|
||||
`backend/app/routers/blocklist.py` — 9 endpoints on `/api/blocklists`: list sources, create source (201), trigger manual import (`POST /import`), get schedule info, update schedule, paginated import log, get/update/delete single source, preview blocklist contents. Static path segments (`/import`, `/schedule`, `/log`) registered before `/{source_id}`. All endpoints authenticated. 22 router tests pass.
|
||||
|
||||
Define models in `backend/app/models/blocklist.py`. See [Architekture.md § 2.2](Architekture.md).
|
||||
### 10.5 Build the blocklist management page (frontend) ✅ DONE
|
||||
|
||||
### 10.5 Build the blocklist management page (frontend)
|
||||
`frontend/src/pages/BlocklistsPage.tsx` (≈950 lines) — `SourceFormDialog` (create/edit), `PreviewDialog`, `ImportResultDialog`, `SourcesSection` (table with enable/disable switch), `ScheduleSection` (frequency picker + hour/minute/day-of-week selectors + next run display), `ImportLogSection` (paginated table with error badge). Supporting files: `frontend/src/types/blocklist.ts`, `frontend/src/api/blocklist.ts`, `frontend/src/hooks/useBlocklist.ts`. TypeScript + ESLint clean.
|
||||
|
||||
Create `frontend/src/pages/BlocklistPage.tsx`. Display a list of blocklist sources as cards or rows showing name, URL, enabled toggle, and action buttons (edit, delete, preview). Add a form to create or edit a source. Show the schedule configuration with a simple time-and-frequency picker (no raw cron) — dropdowns for frequency preset and a time input. Include a "Run Now" button and a display of last import time and next scheduled run. Below, show the import log as a table (timestamp, source, IPs imported, IPs skipped, errors) with filters. If the most recent import had errors, show a warning badge in the navigation. See [Features.md § 8](Features.md).
|
||||
### 10.6 Write tests for blocklist features ✅ DONE
|
||||
|
||||
### 10.6 Write tests for blocklist features
|
||||
|
||||
Test source CRUD, import with valid/invalid entries, schedule update, manual import trigger, import log persistence, and error handling when a URL is unreachable. Mock all HTTP calls.
|
||||
18 repository tests, 19 service tests, 22 router tests — 59 total. All 374 backend tests pass. `ruff` clean, `mypy` clean for Stage 10 files.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user