Add origin field and filter for ban sources (Tasks 1 & 2)
- Task 1: Mark imported blocklist IP addresses
- Add BanOrigin type and _derive_origin() to ban.py model
- Populate origin field in ban_service list_bans() and bans_by_country()
- BanTable and MapPage companion table show origin badge column
- Tests: origin derivation in test_ban_service.py and test_dashboard.py
- Task 2: Add origin filter to dashboard and world map
- ban_service: _origin_sql_filter() helper; origin param on list_bans()
and bans_by_country()
- dashboard router: optional origin query param forwarded to service
- Frontend: BanOriginFilter type + BAN_ORIGIN_FILTER_LABELS in ban.ts
- fetchBans / fetchBansByCountry forward origin to API
- useBans / useMapData accept and pass origin; page resets on change
- BanTable accepts origin prop; DashboardPage adds segmented filter
- MapPage adds origin Select next to time-range picker
- Tests: origin filter assertions in test_ban_service and test_dashboard
This commit is contained in:
@@ -61,9 +61,18 @@ A geographical overview of ban activity.
|
||||
|
||||
### Map
|
||||
|
||||
- A full world map rendered with country outlines only (no fill colours, no satellite imagery).
|
||||
- For every country that has at least one banned IP in the selected time range, the total count is displayed centred inside that country's borders.
|
||||
- Countries with zero banned IPs show no number and no label — they remain blank.
|
||||
- A full world map rendered with country outlines, showing ban activity through color-coded fills (no satellite imagery).
|
||||
- **Color coding:** Countries are colored based on their ban count for the selected time range:
|
||||
- **Red:** High ban count (100+ bans by default)
|
||||
- **Yellow:** Medium ban count (50 bans by default)
|
||||
- **Green:** Low ban count (20 bans by default)
|
||||
- **Transparent (no fill):** Zero bans
|
||||
- Colors are smoothly interpolated between the thresholds (e.g., 35 bans shows a yellow-green blend)
|
||||
- The color threshold values are configurable through the application settings
|
||||
- **Interactive zoom and pan:** Users can zoom in/out using mouse wheel or touch gestures, and pan by clicking and dragging. This allows detailed inspection of densely-affected regions. Zoom controls (zoom in, zoom out, reset view) are provided as overlay buttons in the top-right corner.
|
||||
- For every country that has bans, the total count is displayed centred inside that country's borders in the selected time range.
|
||||
- Countries with zero banned IPs show no number and no label — they remain blank and transparent.
|
||||
- Clicking a country filters the companion table below to show only bans from that country.
|
||||
- Time-range selector with the same quick presets:
|
||||
- Last 24 hours
|
||||
- Last 7 days
|
||||
@@ -184,6 +193,16 @@ A page to inspect and modify the fail2ban configuration without leaving the web
|
||||
- Set the database purge age — how long historical ban records are kept before automatic cleanup.
|
||||
- Set the maximum number of log-line matches stored per ban record in the database.
|
||||
|
||||
### Map Settings
|
||||
|
||||
- Configure the three color thresholds that determine how countries are colored on the World Map view based on their ban count:
|
||||
- **Low Threshold (Green):** Ban count at which the color transitions from light green to full green (default: 20).
|
||||
- **Medium Threshold (Yellow):** Ban count at which the color transitions from green to yellow (default: 50).
|
||||
- **High Threshold (Red):** Ban count at which the color transitions from yellow to red (default: 100).
|
||||
- Countries with ban counts between thresholds display smoothly interpolated colors.
|
||||
- Countries with zero bans remain transparent (no fill).
|
||||
- Changes take effect immediately on the World Map view without requiring a page reload.
|
||||
|
||||
---
|
||||
|
||||
## 7. Ban History
|
||||
|
||||
240
Docs/Tasks.md
240
Docs/Tasks.md
@@ -4,21 +4,239 @@ This document breaks the entire BanGUI project into development stages, ordered
|
||||
|
||||
---
|
||||
|
||||
## ✅ fix: blocklist import — Jail not found (DONE)
|
||||
## Task 1 — Mark Imported Blocklist IP Addresses ✅ DONE
|
||||
|
||||
**Problem:** Triggering a blocklist import failed with `Jail not found: 'blocklist-import'` because
|
||||
the dedicated fail2ban jail did not exist in the dev configuration.
|
||||
**Completed:** Added `origin` field (`"blocklist"` / `"selfblock"`) to `Ban` and `DashboardBanItem` models, derived from jail name in `ban_service.py`. BanTable and MapPage companion table display an Origin badge column. Tests added to `test_ban_service.py` and `test_dashboard.py`.
|
||||
|
||||
**Root cause:** `Docker/fail2ban-dev-config/fail2ban/jail.d/` had no `blocklist-import.conf` jail.
|
||||
The service code (`blocklist_service.BLOCKLIST_JAIL = "blocklist-import"`) is correct, but the
|
||||
matching jail was never defined.
|
||||
### Problem
|
||||
|
||||
**Fix:**
|
||||
- Added `Docker/fail2ban-dev-config/fail2ban/jail.d/blocklist-import.conf` — a manual-ban jail
|
||||
(no log monitoring; accepts `banip` commands only; 1-week bantime; `iptables-allports` action).
|
||||
- Fixed pre-existing trailing-whitespace lint issue in `app/services/setup_service.py`.
|
||||
When IPs are imported from an external blocklist they are applied to the `blocklist-import` jail via the fail2ban socket. Once they land in the fail2ban SQLite `bans` table they look identical to IPs that were banned organically by fail2ban filters. There is no way for the dashboard or map to tell the user whether a given ban came from a blocklist import or from a real failed-login detection.
|
||||
|
||||
**Verification:** All 19 blocklist service tests pass. `ruff check` and `mypy --strict` are clean.
|
||||
### Goal
|
||||
|
||||
Every ban displayed in the UI must carry a visible **origin** indicator: either `blocklist` (imported from a blocklist source) or `selfblock` (detected by fail2ban itself).
|
||||
|
||||
### Implementation Details
|
||||
|
||||
**Backend — derive origin from jail name**
|
||||
|
||||
The blocklist import service already uses a dedicated jail called `blocklist-import`. This can be used as the discriminator: any ban whose `jail` column equals `blocklist-import` is a blocklist ban; everything else is a self-block.
|
||||
|
||||
1. **Model change** — In `backend/app/models/ban.py`, add an `origin` field of type `Literal["blocklist", "selfblock"]` to both `Ban` and `DashboardBanItem`. Compute it from the `jail` value during construction (if `jail == "blocklist-import"` → `"blocklist"`, else `"selfblock"`).
|
||||
2. **Service change** — In `backend/app/services/ban_service.py`, make sure `list_bans()` and `bans_by_country()` populate the new `origin` field when building ban objects.
|
||||
3. **API response** — The JSON payloads from `GET /api/dashboard/bans` and `GET /api/dashboard/bans/by-country` already serialise every field of `DashboardBanItem`, so `origin` will appear automatically once the model is updated.
|
||||
|
||||
**Frontend — show origin badge**
|
||||
|
||||
4. **BanTable** — In `frontend/src/components/BanTable.tsx`, add an **Origin** column. Render a small coloured badge: blue label `Blocklist` or grey label `Selfblock`.
|
||||
5. **MapPage detail table** — The table below the world map in `frontend/src/pages/MapPage.tsx` also lists bans. Add the same origin badge column there.
|
||||
|
||||
**Tests**
|
||||
|
||||
6. Add a unit test in `backend/tests/test_services/test_ban_service.py` that inserts bans into a mock fail2ban DB under both jail names (`blocklist-import` and e.g. `sshd`) and asserts the returned objects carry the correct `origin` value.
|
||||
7. Add a router test in `backend/tests/test_routers/test_dashboard.py` that verifies the JSON response contains the `origin` field.
|
||||
|
||||
### Files Touched
|
||||
|
||||
| Layer | File |
|
||||
|-------|------|
|
||||
| Model | `backend/app/models/ban.py` |
|
||||
| Service | `backend/app/services/ban_service.py` |
|
||||
| Frontend | `frontend/src/components/BanTable.tsx` |
|
||||
| Frontend | `frontend/src/pages/MapPage.tsx` |
|
||||
| Tests | `backend/tests/test_services/test_ban_service.py` |
|
||||
| Tests | `backend/tests/test_routers/test_dashboard.py` |
|
||||
|
||||
---
|
||||
|
||||
## Task 2 — Add Origin Filter to Dashboard and World Map ✅ DONE
|
||||
|
||||
**Completed:** Added optional `origin` query parameter (`"blocklist"` / `"selfblock"`) to both dashboard router endpoints, filtered via `jail`-based WHERE clause in `ban_service`. Frontend: `BanOriginFilter` type + `BAN_ORIGIN_FILTER_LABELS` in `types/ban.ts`; `fetchBans` and `fetchBansByCountry` APIs forward the param; `useBans` and `useMapData` hooks accept `origin`; `DashboardPage` and `MapPage` show a segmented toggle / select for All / Blocklist / Selfblock. Tests extended in `test_ban_service.py` and `test_dashboard.py`.
|
||||
|
||||
### Problem
|
||||
|
||||
Once Task 1 exposes the `origin` field, users need a way to filter the view so they can see **all** bans, **only blocklist** bans, or **only self-blocked** bans. This must work on both the dashboard ban table and the world map.
|
||||
|
||||
### Goal
|
||||
|
||||
Add a filter dropdown (or segmented toggle) with three options — `All`, `Blocklist`, `Selfblock` — to the dashboard toolbar and the map toolbar. The selection must propagate to the backend so that only matching bans are returned and pagination / country aggregation remain correct.
|
||||
|
||||
### Implementation Details
|
||||
|
||||
**Backend — add `origin` query parameter**
|
||||
|
||||
1. **Dashboard router** — `GET /api/dashboard/bans` in `backend/app/routers/dashboard.py`: add an optional query parameter `origin: Optional[Literal["blocklist", "selfblock"]] = None`. Pass it through to the service layer.
|
||||
2. **Map router** — `GET /api/dashboard/bans/by-country` in the same router: add the same `origin` parameter.
|
||||
3. **Service layer** — In `backend/app/services/ban_service.py`:
|
||||
- `list_bans()`: when `origin` is provided, append a WHERE clause on the `jail` column (`jail = 'blocklist-import'` for blocklist, `jail != 'blocklist-import'` for selfblock).
|
||||
- `bans_by_country()`: apply the same jail-based filter so that country aggregation only counts matching bans.
|
||||
|
||||
**Frontend — filter controls**
|
||||
|
||||
4. **Shared state** — Create a small shared type `BanOriginFilter = "all" | "blocklist" | "selfblock"` (e.g. in `frontend/src/types/` or inline).
|
||||
5. **DashboardPage** — In `frontend/src/pages/DashboardPage.tsx`, add a dropdown or segmented control next to the existing time-range toolbar. Store the selected value in component state. Pass it to `useBans` hook, which forwards it as the `origin` query parameter.
|
||||
6. **useBans hook** — In `frontend/src/hooks/useBans.ts`, accept an optional `origin` parameter and include it in the API call via `fetchBans()`.
|
||||
7. **API function** — In `frontend/src/api/dashboard.ts`, update `fetchBans()` to accept and forward the `origin` query parameter.
|
||||
8. **MapPage** — In `frontend/src/pages/MapPage.tsx`, add the same dropdown. Pass the selected value to `useMapData` hook.
|
||||
9. **useMapData hook** — In `frontend/src/hooks/useMapData.ts`, accept `origin` and forward it to `fetchBansByCountry()`.
|
||||
10. **Map API function** — In `frontend/src/api/map.ts`, update `fetchBansByCountry()` to include `origin` in the query string.
|
||||
|
||||
**Tests**
|
||||
|
||||
11. Extend `backend/tests/test_services/test_ban_service.py`: insert bans under multiple jails, call `list_bans(origin="blocklist")` and assert only `blocklist-import` jail bans are returned; repeat for `"selfblock"` and `None`.
|
||||
12. Extend `backend/tests/test_routers/test_dashboard.py`: hit `GET /api/dashboard/bans?origin=blocklist` and verify the response only contains blocklist bans.
|
||||
|
||||
### Files Touched
|
||||
|
||||
| Layer | File |
|
||||
|-------|------|
|
||||
| Router | `backend/app/routers/dashboard.py` |
|
||||
| Service | `backend/app/services/ban_service.py` |
|
||||
| Frontend | `frontend/src/pages/DashboardPage.tsx` |
|
||||
| Frontend | `frontend/src/pages/MapPage.tsx` |
|
||||
| Frontend | `frontend/src/hooks/useBans.ts` |
|
||||
| Frontend | `frontend/src/hooks/useMapData.ts` |
|
||||
| Frontend | `frontend/src/api/dashboard.ts` |
|
||||
| Frontend | `frontend/src/api/map.ts` |
|
||||
| Tests | `backend/tests/test_services/test_ban_service.py` |
|
||||
| Tests | `backend/tests/test_routers/test_dashboard.py` |
|
||||
|
||||
---
|
||||
|
||||
## Task 3 — Performance Optimisation for 10 k+ IPs (Dashboard & World Map)
|
||||
|
||||
### Problem
|
||||
|
||||
With a large number of banned IPs (> 10 000), both the dashboard and the world map take over 10 seconds to load and become unusable. The root cause is the geo-enrichment step: `geo_service.py` calls the external `ip-api.com` endpoint **one IP at a time** with a 5-second timeout, and the free tier is rate-limited to 45 requests/minute. On a cold cache with 10 k IPs, this would take hours.
|
||||
|
||||
### Goal
|
||||
|
||||
Dashboard and world map must load within **2 seconds** for 10 k banned IPs. Write a reproducible benchmark test to prove it.
|
||||
|
||||
### Implementation Details
|
||||
|
||||
**Backend — persistent geo cache**
|
||||
|
||||
1. **SQLite geo cache table** — In `backend/app/db.py`, add a `geo_cache` table:
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS geo_cache (
|
||||
ip TEXT PRIMARY KEY,
|
||||
country_code TEXT,
|
||||
country_name TEXT,
|
||||
asn TEXT,
|
||||
org TEXT,
|
||||
cached_at TEXT NOT NULL
|
||||
);
|
||||
```
|
||||
On startup the service loads from this table instead of starting with an empty dict. Lookups hit the in-memory dict first, then fall through to the DB, and only as a last resort call the external API. Successful API responses are written back to both caches.
|
||||
|
||||
2. **Batch lookup via ip-api.com batch endpoint** — The free `ip-api.com` API supports a **batch POST** to `http://ip-api.com/batch` accepting up to 100 IPs per request. Refactor `geo_service.py` to:
|
||||
- Collect all uncached IPs from the requested page.
|
||||
- Send them in chunks of 100 to the batch endpoint.
|
||||
- Parse the JSON array response and populate caches in one pass.
|
||||
This alone reduces 10 000 cold lookups from 10 000 sequential requests to ≈ 100 batch calls.
|
||||
|
||||
3. **Pre-warm cache on import** — In `backend/app/services/blocklist_service.py`, after a successful blocklist import, fire a background task that batch-resolves all newly imported IPs. This way the dashboard never faces a cold cache for blocklist IPs.
|
||||
|
||||
**Backend — limit the country-aggregation query**
|
||||
|
||||
4. In `ban_service.py`, `bans_by_country()` currently loads up to 2 000 bans and enriches every one. Change it to:
|
||||
- Run the aggregation (GROUP BY `jail`, count per jail) **in SQL** directly against the fail2ban DB.
|
||||
- Only enrich the distinct IPs, batch-resolved.
|
||||
- Return the aggregated country → count map without fetching full ban rows.
|
||||
|
||||
**Frontend — virtualised table**
|
||||
|
||||
5. The `BanTable` component currently renders all rows in the DOM. For 10 k+ rows (even paginated at 100), scrolling is fine, but if page sizes are increased or if pagination is removed, install a virtual-scrolling library (e.g. `@tanstack/react-virtual`) and render only visible rows. Alternatively, ensure page size stays capped at ≤ 500 (already enforced) and measure whether DOM performance is acceptable — only virtualise if needed.
|
||||
|
||||
**Frontend — map debounce**
|
||||
|
||||
6. In `WorldMap.tsx`, add a loading skeleton / spinner and debounce the data fetch. If the user switches time ranges rapidly, cancel in-flight requests to avoid piling up stale responses.
|
||||
|
||||
**Performance test**
|
||||
|
||||
7. Write a pytest benchmark in `backend/tests/test_services/test_ban_service_perf.py`:
|
||||
- Seed a temporary fail2ban SQLite with 10 000 synthetic bans (random IPs, mixed jails, spread over 365 days).
|
||||
- Pre-populate the geo cache with matching entries so the test does not hit the network.
|
||||
- Call `list_bans(range="365d", page=1, page_size=100)` and `bans_by_country(range="365d")`.
|
||||
- Assert both return within **2 seconds** wall-clock time.
|
||||
8. Add a manual/integration test script `backend/tests/scripts/seed_10k_bans.py` that inserts 10 000 bans into the real fail2ban dev DB and pre-caches their geo data so developers can visually test dashboard and map load times in the browser.
|
||||
|
||||
### Files Touched
|
||||
|
||||
| Layer | File |
|
||||
|-------|------|
|
||||
| DB schema | `backend/app/db.py` |
|
||||
| Geo service | `backend/app/services/geo_service.py` |
|
||||
| Ban service | `backend/app/services/ban_service.py` |
|
||||
| Blocklist service | `backend/app/services/blocklist_service.py` |
|
||||
| Frontend | `frontend/src/components/BanTable.tsx` |
|
||||
| Frontend | `frontend/src/components/WorldMap.tsx` |
|
||||
| Tests | `backend/tests/test_services/test_ban_service_perf.py` (new) |
|
||||
| Scripts | `backend/tests/scripts/seed_10k_bans.py` (new) |
|
||||
|
||||
---
|
||||
|
||||
## Task 4 — Fix Missing Country for Resolved IPs
|
||||
|
||||
### Problem
|
||||
|
||||
Many IPs (e.g. `5.167.71.2`, `5.167.71.3`, `5.167.71.4`) show no country in the dashboard and map. The current `geo_service.py` calls `ip-api.com` with a 5-second timeout and silently returns `None` fields on any error or timeout. Common causes:
|
||||
|
||||
- **Rate limiting** — the free tier allows 45 req/min; once exceeded, responses return `"status": "fail"` and the service caches `None` values, permanently hiding the country.
|
||||
- **Cache poisoning with empty entries** — a failed lookup stores `GeoInfo(country_code=None, ...)` in the in-memory cache. Until the cache is flushed (at 10 000 entries), that IP will always appear without a country.
|
||||
- **External API unreachable** — network issues or container DNS problems cause timeouts that are treated the same as "no data".
|
||||
|
||||
### Goal
|
||||
|
||||
Every IP that has a valid geographic mapping should display its country. Failed lookups must be retried on subsequent requests rather than permanently cached as empty.
|
||||
|
||||
### Implementation Details
|
||||
|
||||
**Backend — do not cache failed lookups**
|
||||
|
||||
1. In `backend/app/services/geo_service.py`, change the caching logic: only store a result in the in-memory cache (and the new persistent `geo_cache` table from Task 3) when `country_code is not None`. If the API returned a failure or the request timed out, do **not** cache the result so it will be retried on the next request.
|
||||
|
||||
**Backend — negative-cache with short TTL**
|
||||
|
||||
2. To avoid hammering the API for the same failing IP on every request, introduce a separate **negative cache** — a dict mapping IP → timestamp of last failed attempt. Skip re-lookup if the last failure was less than **5 minutes** ago. After 5 minutes the IP becomes eligible for retry.
|
||||
|
||||
**Backend — fallback to a local GeoIP database**
|
||||
|
||||
3. Add `geoip2` (MaxMind GeoLite2) as an optional fallback. If the free `ip-api.com` lookup fails or is rate-limited, attempt a local lookup using the GeoLite2-Country database (`.mmdb` file). This provides offline country resolution for the vast majority of IPv4 addresses.
|
||||
- Add `geoip2` to `backend/pyproject.toml` dependencies.
|
||||
- Download the GeoLite2-Country database during Docker build or via a setup script (requires free MaxMind license key).
|
||||
- In `geo_service.py`, try `ip-api.com` first; on failure, fall back to geoip2; only return `None` if both fail.
|
||||
|
||||
**Backend — bulk re-resolve endpoint**
|
||||
|
||||
4. Add `POST /api/geo/re-resolve` in `backend/app/routers/geo.py` that:
|
||||
- Queries all currently cached IPs with `country_code IS NULL`.
|
||||
- Re-resolves them in batches (using the batch endpoint from Task 3).
|
||||
- Returns a count of newly resolved IPs.
|
||||
- This lets the admin manually fix historical gaps.
|
||||
|
||||
**Frontend — visual indicator for unknown country**
|
||||
|
||||
5. In `BanTable.tsx` and the MapPage detail table, display a small "unknown" icon or `—` placeholder when country is null, instead of leaving the cell empty. Tooltip: "Country could not be resolved — will retry automatically."
|
||||
|
||||
**Tests**
|
||||
|
||||
6. In `backend/tests/test_services/test_geo_service.py`:
|
||||
- Test that a failed lookup is **not** persisted in the positive cache.
|
||||
- Test that a failed lookup **is** stored in the negative cache and skipped for 5 minutes.
|
||||
- Test that after the negative-cache TTL expires, the IP is re-queried.
|
||||
- Mock the geoip2 fallback and verify it is called when ip-api fails.
|
||||
7. In `backend/tests/test_routers/test_geo.py`, test the new `POST /api/geo/re-resolve` endpoint.
|
||||
|
||||
### Files Touched
|
||||
|
||||
| Layer | File |
|
||||
|-------|------|
|
||||
| Geo service | `backend/app/services/geo_service.py` |
|
||||
| Geo router | `backend/app/routers/geo.py` |
|
||||
| Dependencies | `backend/pyproject.toml` |
|
||||
| Frontend | `frontend/src/components/BanTable.tsx` |
|
||||
| Frontend | `frontend/src/pages/MapPage.tsx` |
|
||||
| Tests | `backend/tests/test_services/test_geo_service.py` |
|
||||
| Tests | `backend/tests/test_routers/test_geo.py` |
|
||||
|
||||
@@ -271,12 +271,14 @@ The dashboard uses cards to display key figures (server status, total bans, acti
|
||||
|
||||
## 11. World Map View
|
||||
|
||||
- The map renders country outlines only — **no fill colours, no satellite imagery, no terrain shading**.
|
||||
- The map renders country outlines only — **no fill colours, no satellite imagery, no terrain shading**. Countries are transparent with neutral strokes.
|
||||
- **The map is fully interactive:** users can zoom in/out using mouse wheel or pinch gestures, and pan by dragging. Zoom range: 1× (full world) to 8× (regional detail).
|
||||
- **Zoom controls:** Three small buttons overlaid in the top-right corner provide zoom in (+), zoom out (−), and reset view (⟲) functionality. Buttons use `appearance="secondary"` and `size="small"`.
|
||||
- Countries with banned IPs display a **count badge** centred inside the country polygon. Use `FontSizes.size14` semibold, `themePrimary` colour.
|
||||
- Countries with zero bans remain completely blank — no label, no tint.
|
||||
- On hover: country region gets a subtle `neutralLighterAlt` fill. On click: fill shifts to `themeLighterAlt` and the companion table below filters to that country.
|
||||
- The map must have a **light neutral border** (`neutralLight`) around its container, at **Depth 4**.
|
||||
- Time-range selector above the map uses `Pivot` with quick presets (24 h, 7 d, 30 d, 365 d).
|
||||
- On hover: country region gets a subtle `neutralBackground3` fill (only if the country has data). On click: fill shifts to `brandBackgroundHover` and the companion table below filters to that country. Default state remains transparent.
|
||||
- The map must have a **light neutral border** (`neutralStroke1`) around its container, with `borderRadius.medium`.
|
||||
- Time-range selector above the map uses `Select` dropdown with quick presets (24 h, 7 d, 30 d, 365 d).
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user