BanGUI/Docs/Tasks.md

# BanGUI — Task List

This document breaks the entire BanGUI project into development stages, ordered so that each stage builds on the previous one. Every task is described in prose with enough detail for a developer to begin work. References point to the relevant documentation.

---

## Task 1 — Mark Imported Blocklist IP Addresses ✅ DONE

**Completed:** Added `origin` field (`"blocklist"` / `"selfblock"`) to `Ban` and `DashboardBanItem` models, derived from jail name in `ban_service.py`. BanTable and MapPage companion table display an Origin badge column. Tests added to `test_ban_service.py` and `test_dashboard.py`.

### Problem

When IPs are imported from an external blocklist they are applied to the `blocklist-import` jail via the fail2ban socket. Once they land in the fail2ban SQLite `bans` table they look identical to IPs that were banned organically by fail2ban filters. There is no way for the dashboard or map to tell the user whether a given ban came from a blocklist import or from a real failed-login detection.

### Goal

Every ban displayed in the UI must carry a visible **origin** indicator: either `blocklist` (imported from a blocklist source) or `selfblock` (detected by fail2ban itself).

### Implementation Details

**Backend — derive origin from jail name**

The blocklist import service already uses a dedicated jail called `blocklist-import`. This can be used as the discriminator: any ban whose `jail` column equals `blocklist-import` is a blocklist ban; everything else is a self-block.

1. **Model change** — In `backend/app/models/ban.py`, add an `origin` field of type `Literal["blocklist", "selfblock"]` to both `Ban` and `DashboardBanItem`. Compute it from the `jail` value during construction (if `jail == "blocklist-import"` → `"blocklist"`, else `"selfblock"`).
2. **Service change** — In `backend/app/services/ban_service.py`, make sure `list_bans()` and `bans_by_country()` populate the new `origin` field when building ban objects.
3. **API response** — The JSON payloads from `GET /api/dashboard/bans` and `GET /api/dashboard/bans/by-country` already serialise every field of `DashboardBanItem`, so `origin` will appear automatically once the model is updated.

**Frontend — show origin badge**

4. **BanTable** — In `frontend/src/components/BanTable.tsx`, add an **Origin** column. Render a small coloured badge: blue label `Blocklist` or grey label `Selfblock`.
5. **MapPage detail table** — The table below the world map in `frontend/src/pages/MapPage.tsx` also lists bans. Add the same origin badge column there.

**Tests**

6. Add a unit test in `backend/tests/test_services/test_ban_service.py` that inserts bans into a mock fail2ban DB under both jail names (`blocklist-import` and e.g. `sshd`) and asserts the returned objects carry the correct `origin` value.
7. Add a router test in `backend/tests/test_routers/test_dashboard.py` that verifies the JSON response contains the `origin` field.

### Files Touched

| Layer | File |
|-------|------|
| Model | `backend/app/models/ban.py` |
| Service | `backend/app/services/ban_service.py` |
| Frontend | `frontend/src/components/BanTable.tsx` |
| Frontend | `frontend/src/pages/MapPage.tsx` |
| Tests | `backend/tests/test_services/test_ban_service.py` |
| Tests | `backend/tests/test_routers/test_dashboard.py` |

---

## Task 2 — Add Origin Filter to Dashboard and World Map ✅ DONE

**Completed:** Added optional `origin` query parameter (`"blocklist"` / `"selfblock"`) to both dashboard router endpoints, filtered via `jail`-based WHERE clause in `ban_service`. Frontend: `BanOriginFilter` type + `BAN_ORIGIN_FILTER_LABELS` in `types/ban.ts`; `fetchBans` and `fetchBansByCountry` APIs forward the param; `useBans` and `useMapData` hooks accept `origin`; `DashboardPage` and `MapPage` show a segmented toggle / select for All / Blocklist / Selfblock. Tests extended in `test_ban_service.py` and `test_dashboard.py`.

### Problem

Once Task 1 exposes the `origin` field, users need a way to filter the view so they can see **all** bans, **only blocklist** bans, or **only self-blocked** bans. This must work on both the dashboard ban table and the world map.

### Goal

Add a filter dropdown (or segmented toggle) with three options — `All`, `Blocklist`, `Selfblock` — to the dashboard toolbar and the map toolbar. The selection must propagate to the backend so that only matching bans are returned and pagination / country aggregation remain correct.

### Implementation Details

**Backend — add `origin` query parameter**

1. **Dashboard router** — `GET /api/dashboard/bans` in `backend/app/routers/dashboard.py`: add an optional query parameter `origin: Optional[Literal["blocklist", "selfblock"]] = None`. Pass it through to the service layer.
2. **Map router** — `GET /api/dashboard/bans/by-country` in the same router: add the same `origin` parameter.
3. **Service layer** — In `backend/app/services/ban_service.py`:
   - `list_bans()`: when `origin` is provided, append a WHERE clause on the `jail` column (`jail = 'blocklist-import'` for blocklist, `jail != 'blocklist-import'` for selfblock).
   - `bans_by_country()`: apply the same jail-based filter so that country aggregation only counts matching bans.

**Frontend — filter controls**

4. **Shared state** — Create a small shared type `BanOriginFilter = "all" | "blocklist" | "selfblock"` (e.g. in `frontend/src/types/` or inline).
5. **DashboardPage** — In `frontend/src/pages/DashboardPage.tsx`, add a dropdown or segmented control next to the existing time-range toolbar. Store the selected value in component state. Pass it to `useBans` hook, which forwards it as the `origin` query parameter.
6. **useBans hook** — In `frontend/src/hooks/useBans.ts`, accept an optional `origin` parameter and include it in the API call via `fetchBans()`.
7. **API function** — In `frontend/src/api/dashboard.ts`, update `fetchBans()` to accept and forward the `origin` query parameter.
8. **MapPage** — In `frontend/src/pages/MapPage.tsx`, add the same dropdown. Pass the selected value to `useMapData` hook.
9. **useMapData hook** — In `frontend/src/hooks/useMapData.ts`, accept `origin` and forward it to `fetchBansByCountry()`.
10. **Map API function** — In `frontend/src/api/map.ts`, update `fetchBansByCountry()` to include `origin` in the query string.

**Tests**

11. Extend `backend/tests/test_services/test_ban_service.py`: insert bans under multiple jails, call `list_bans(origin="blocklist")` and assert only `blocklist-import` jail bans are returned; repeat for `"selfblock"` and `None`.
12. Extend `backend/tests/test_routers/test_dashboard.py`: hit `GET /api/dashboard/bans?origin=blocklist` and verify the response only contains blocklist bans.

### Files Touched

| Layer | File |
|-------|------|
| Router | `backend/app/routers/dashboard.py` |
| Service | `backend/app/services/ban_service.py` |
| Frontend | `frontend/src/pages/DashboardPage.tsx` |
| Frontend | `frontend/src/pages/MapPage.tsx` |
| Frontend | `frontend/src/hooks/useBans.ts` |
| Frontend | `frontend/src/hooks/useMapData.ts` |
| Frontend | `frontend/src/api/dashboard.ts` |
| Frontend | `frontend/src/api/map.ts` |
| Tests | `backend/tests/test_services/test_ban_service.py` |
| Tests | `backend/tests/test_routers/test_dashboard.py` |

---

## Task 3 — Performance Optimisation for 10 k+ IPs (Dashboard & World Map) ✅ DONE

**Completed:**
- Added persistent `geo_cache` SQLite table in `db.py`; loaded into in-memory cache at startup via `geo_service.load_cache_from_db()`.
- Rewrote `geo_service.py`: added `lookup_batch()` using `ip-api.com/batch` (100 IPs/call); failed lookups no longer cached so they are retried; only successful resolutions written to persistent store.
- Rewrote `bans_by_country()` in `ban_service.py`: SQL `GROUP BY ip` aggregation instead of loading 2 000 raw rows, batch geo-resolution via `lookup_batch()`, companion table limited to 200 rows (already geo-cached).
- Updated `dashboard.py` router `GET /bans/by-country` to pass `http_session` + `app_db` directly to `bans_by_country()`.
- Added geo cache pre-warm in `blocklist_service.import_source()`: after import, newly banned IPs are batch-resolved and persisted.
- Added debounce (300 ms) to `useMapData` hook to cancel stale in-flight requests when filters change rapidly; sets loading=true immediately for instant skeleton feedback.
- BanTable: page size capped at 100 per page with next/prev pagination — DOM perf not an issue, no virtualisation needed.
- Performance benchmark `tests/test_services/test_ban_service_perf.py`: seeds 10 000 bans in a temp DB, pre-warms geo cache, asserts `list_bans` and `bans_by_country` both complete in < 2 seconds.
- Seed script `tests/scripts/seed_10k_bans.py`: inserts 10 000 synthetic bans + pre-caches geo data for browser-level load-time verification.

### Problem

With a large number of banned IPs (> 10 000), both the dashboard and the world map take over 10 seconds to load and become unusable. The root cause is the geo-enrichment step: `geo_service.py` calls the external `ip-api.com` endpoint **one IP at a time** with a 5-second timeout, and the free tier is rate-limited to 45 requests/minute. On a cold cache with 10 k IPs, this would take hours.

### Goal

Dashboard and world map must load within **2 seconds** for 10 k banned IPs. Write a reproducible benchmark test to prove it.

### Implementation Details

**Backend — persistent geo cache**

1. **SQLite geo cache table** — In `backend/app/db.py`, add a `geo_cache` table:
   ```sql
   CREATE TABLE IF NOT EXISTS geo_cache (
       ip TEXT PRIMARY KEY,
       country_code TEXT,
       country_name TEXT,
       asn TEXT,
       org TEXT,
       cached_at TEXT NOT NULL
   );
   ```
   On startup the service loads from this table instead of starting with an empty dict. Lookups hit the in-memory dict first, then fall through to the DB, and only as a last resort call the external API. Successful API responses are written back to both caches.

2. **Batch lookup via ip-api.com batch endpoint** — The free `ip-api.com` API supports a **batch POST** to `http://ip-api.com/batch` accepting up to 100 IPs per request. Refactor `geo_service.py` to:
   - Collect all uncached IPs from the requested page.
   - Send them in chunks of 100 to the batch endpoint.
   - Parse the JSON array response and populate caches in one pass.
   This alone reduces 10 000 cold lookups from 10 000 sequential requests to ≈ 100 batch calls.

3. **Pre-warm cache on import** — In `backend/app/services/blocklist_service.py`, after a successful blocklist import, fire a background task that batch-resolves all newly imported IPs. This way the dashboard never faces a cold cache for blocklist IPs.

**Backend — limit the country-aggregation query**

4. In `ban_service.py`, `bans_by_country()` currently loads up to 2 000 bans and enriches every one. Change it to:
   - Run the aggregation (GROUP BY `jail`, count per jail) **in SQL** directly against the fail2ban DB.
   - Only enrich the distinct IPs, batch-resolved.
   - Return the aggregated country → count map without fetching full ban rows.

**Frontend — virtualised table**

5. The `BanTable` component currently renders all rows in the DOM. For 10 k+ rows (even paginated at 100), scrolling is fine, but if page sizes are increased or if pagination is removed, install a virtual-scrolling library (e.g. `@tanstack/react-virtual`) and render only visible rows. Alternatively, ensure page size stays capped at ≤ 500 (already enforced) and measure whether DOM performance is acceptable — only virtualise if needed.

**Frontend — map debounce**

6. In `WorldMap.tsx`, add a loading skeleton / spinner and debounce the data fetch. If the user switches time ranges rapidly, cancel in-flight requests to avoid piling up stale responses.

**Performance test**

7. Write a pytest benchmark in `backend/tests/test_services/test_ban_service_perf.py`:
   - Seed a temporary fail2ban SQLite with 10 000 synthetic bans (random IPs, mixed jails, spread over 365 days).
   - Pre-populate the geo cache with matching entries so the test does not hit the network.
   - Call `list_bans(range="365d", page=1, page_size=100)` and `bans_by_country(range="365d")`.
   - Assert both return within **2 seconds** wall-clock time.
8. Add a manual/integration test script `backend/tests/scripts/seed_10k_bans.py` that inserts 10 000 bans into the real fail2ban dev DB and pre-caches their geo data so developers can visually test dashboard and map load times in the browser.

### Files Touched

| Layer | File |
|-------|------|
| DB schema | `backend/app/db.py` |
| Geo service | `backend/app/services/geo_service.py` |
| Ban service | `backend/app/services/ban_service.py` |
| Blocklist service | `backend/app/services/blocklist_service.py` |
| Frontend | `frontend/src/components/BanTable.tsx` |
| Frontend | `frontend/src/components/WorldMap.tsx` |
| Tests | `backend/tests/test_services/test_ban_service_perf.py` (new) |
| Scripts | `backend/tests/scripts/seed_10k_bans.py` (new) |

---

## Task 4 — Fix Missing Country for Resolved IPs ✅ DONE

**Completed:**
- `geo_service.py`: Added `_neg_cache: dict[str, float]` with 5-minute TTL (`_NEG_CACHE_TTL = 300`). Failed lookups (any cause) are written to the neg cache and returned immediately without querying the API until the TTL expires. `clear_neg_cache()` flushes it (used by the re-resolve endpoint).
- `geo_service.py`: Added `init_geoip(mmdb_path)` + `_geoip_lookup(ip)` using `geoip2.database.Reader`. When ip-api fails, the local GeoLite2-Country `.mmdb` is tried as fallback. Only fires if `BANGUI_GEOIP_DB_PATH` is set and the file exists; otherwise silently skipped.
- `geo_service.py`: Fixed `lookup_batch()` bug where failed API results were stored in the positive in-memory cache (`_store` was called unconditionally). Now only positive results go into `_cache`; failures try geoip2 fallback then go into `_neg_cache`.
- `geo_service.py`: Added `_persist_neg_entry(db, ip)` — `INSERT OR IGNORE` into `geo_cache` with `country_code=NULL` so the re-resolve endpoint can find previously failed IPs without overwriting existing positive entries.
- `config.py`: Added `geoip_db_path: str | None` setting (env `BANGUI_GEOIP_DB_PATH`).
- `pyproject.toml`: Added `geoip2>=4.8.0` dependency.
- `main.py`: Calls `geo_service.init_geoip(settings.geoip_db_path)` during lifespan startup.
- `routers/geo.py`: Added `POST /api/geo/re-resolve` — queries `geo_cache WHERE country_code IS NULL`, clears neg cache, batch-re-resolves all those IPs, returns `{"resolved": N, "total": M}`.
- `BanTable.tsx`: Country cell now wraps the `—` placeholder in a Fluent UI Tooltip with message "Country could not be resolved — will retry automatically."
- `MapPage.tsx`: Same Tooltip treatment for the `—` placeholder in the companion table.
- Tests: Updated `test_geo_service.py` — removed outdated `result is None` assertions (lookup now always returns GeoInfo), updated neg-cache test, added `TestNegativeCache` (4 tests) and `TestGeoipFallback` (4 tests). Added `TestReResolve` (4 tests) in `test_geo.py`. **430 total tests pass.**

### Problem

Many IPs (e.g. `5.167.71.2`, `5.167.71.3`, `5.167.71.4`) show no country in the dashboard and map. The current `geo_service.py` calls `ip-api.com` with a 5-second timeout and silently returns `None` fields on any error or timeout. Common causes:

- **Rate limiting** — the free tier allows 45 req/min; once exceeded, responses return `"status": "fail"` and the service caches `None` values, permanently hiding the country.
- **Cache poisoning with empty entries** — a failed lookup stores `GeoInfo(country_code=None, ...)` in the in-memory cache. Until the cache is flushed (at 10 000 entries), that IP will always appear without a country.
- **External API unreachable** — network issues or container DNS problems cause timeouts that are treated the same as "no data".

### Goal

Every IP that has a valid geographic mapping should display its country. Failed lookups must be retried on subsequent requests rather than permanently cached as empty.

### Implementation Details

**Backend — do not cache failed lookups**

1. In `backend/app/services/geo_service.py`, change the caching logic: only store a result in the in-memory cache (and the new persistent `geo_cache` table from Task 3) when `country_code is not None`. If the API returned a failure or the request timed out, do **not** cache the result so it will be retried on the next request.

**Backend — negative-cache with short TTL**

2. To avoid hammering the API for the same failing IP on every request, introduce a separate **negative cache** — a dict mapping IP → timestamp of last failed attempt. Skip re-lookup if the last failure was less than **5 minutes** ago. After 5 minutes the IP becomes eligible for retry.

**Backend — fallback to a local GeoIP database**

3. Add `geoip2` (MaxMind GeoLite2) as an optional fallback. If the free `ip-api.com` lookup fails or is rate-limited, attempt a local lookup using the GeoLite2-Country database (`.mmdb` file). This provides offline country resolution for the vast majority of IPv4 addresses.
   - Add `geoip2` to `backend/pyproject.toml` dependencies.
   - Download the GeoLite2-Country database during Docker build or via a setup script (requires free MaxMind license key).
   - In `geo_service.py`, try `ip-api.com` first; on failure, fall back to geoip2; only return `None` if both fail.

**Backend — bulk re-resolve endpoint**

4. Add `POST /api/geo/re-resolve` in `backend/app/routers/geo.py` that:
   - Queries all currently cached IPs with `country_code IS NULL`.
   - Re-resolves them in batches (using the batch endpoint from Task 3).
   - Returns a count of newly resolved IPs.
   - This lets the admin manually fix historical gaps.

**Frontend — visual indicator for unknown country**

5. In `BanTable.tsx` and the MapPage detail table, display a small "unknown" icon or `—` placeholder when country is null, instead of leaving the cell empty. Tooltip: "Country could not be resolved — will retry automatically."

**Tests**

6. In `backend/tests/test_services/test_geo_service.py`:
   - Test that a failed lookup is **not** persisted in the positive cache.
   - Test that a failed lookup **is** stored in the negative cache and skipped for 5 minutes.
   - Test that after the negative-cache TTL expires, the IP is re-queried.
   - Mock the geoip2 fallback and verify it is called when ip-api fails.
7. In `backend/tests/test_routers/test_geo.py`, test the new `POST /api/geo/re-resolve` endpoint.

### Files Touched

| Layer | File |
|-------|------|
| Geo service | `backend/app/services/geo_service.py` |
| Geo router | `backend/app/routers/geo.py` |
| Dependencies | `backend/pyproject.toml` |
| Frontend | `frontend/src/components/BanTable.tsx` |
| Frontend | `frontend/src/pages/MapPage.tsx` |
| Tests | `backend/tests/test_services/test_geo_service.py` |
| Tests | `backend/tests/test_routers/test_geo.py` |

---

## Task 5 — Blocklist Import Error Badge in Navigation ✅ DONE

**Completed:** Added `last_run_errors: bool | None` to `ScheduleInfo` model and updated `get_schedule_info()` to derive it from the last import log entry. Frontend: added `last_run_errors` to `ScheduleInfo` type; added `useBlocklistStatus` hook that polls `GET /api/blocklists/schedule` every 60 s; `MainLayout` renders a warning `MessageBar` and an amber badge on the Blocklists nav item when `hasErrors` is `true`. Tests: 3 new service tests + 1 new router test; 433 tests pass.

### Problem

[Features.md § 8](Features.md) requires: *"Show a warning badge in the navigation if the most recent import encountered errors"* and *"Notify the user (via the UI status bar) when a scheduled import fails so it does not go unnoticed."*

Currently `ScheduleInfo` (returned by `GET /api/blocklists/schedule`) contains `last_run_at` but no indicator of whether the last run had errors. The `MainLayout` sidebar only warns about fail2ban being offline; there is no blocklist-import failure indicator anywhere in the shell.

### Goal

When the most recent blocklist import run completed with errors, a warning indicator must be visible in the persistent app shell until the condition clears (i.e. a successful import runs). Concretely:

1. A warning `MessageBar` appears at the top of the main content area (alongside the existing fail2ban-offline bar).
2. A small warning badge is rendered on the **Blocklists** navigation item in the sidebar.

### Implementation Details

**Backend — expose error flag in `ScheduleInfo`**

1. **`app/models/blocklist.py`** — Add `last_run_errors: bool | None = None` to `ScheduleInfo`. `True` means the most recent run's `errors` column was non-null; `None` means no run has ever completed.
2. **`app/services/blocklist_service.py`** — In `get_schedule_info()`, after fetching `last_log`, set `last_run_errors = last_log["errors"] is not None` when `last_log` is not `None`, else leave it as `None`.

**Frontend — poll and display**

3. **`frontend/src/types/blocklist.ts`** — Add `last_run_errors: boolean | null` to `ScheduleInfo`.
4. **`frontend/src/hooks/useBlocklist.ts`** — Add a new exported hook `useBlocklistStatus` that polls `GET /api/blocklists/schedule` every 60 seconds (plus on mount) and returns `{ hasErrors: boolean }`. Errors from the poll itself should not surface to the user — silently treat as "unknown".
5. **`frontend/src/layouts/MainLayout.tsx`**:
   - Import and call `useBlocklistStatus`.
   - When `hasErrors` is `true`, render a second `MessageBar` (intent `"warning"`) in the warning-bar slot describing the blocklist import failure.
   - Add a small amber `Badge` (number `!` or just the dot shape) to the Blocklists `NavLink` entry so users see the indicator even when they're on another page.

**Tests**

6. **`backend/tests/test_services/test_blocklist_service.py`** — Three new tests in `TestSchedule`:
   - `test_get_schedule_info_no_errors_when_log_has_no_errors` — inserts a successful import log entry (errors=None), asserts `info.last_run_errors is False`.
   - `test_get_schedule_info_errors_when_log_has_errors` — inserts a log entry with a non-null `errors` string, asserts `info.last_run_errors is True`.
   - `test_get_schedule_info_none_when_no_log` — already covered by the existing test; verify it now also asserts `info.last_run_errors is None`.
7. **`backend/tests/test_routers/test_blocklist.py`** — One new test in `TestGetSchedule`:
   - `test_schedule_response_includes_last_run_errors` — patches `get_schedule_info` to return a `ScheduleInfo` with `last_run_errors=True`, confirms the JSON field is present and `True`.

### Files Touched

| Layer | File |
|-------|------|
| Model | `backend/app/models/blocklist.py` |
| Service | `backend/app/services/blocklist_service.py` |
| Frontend type | `frontend/src/types/blocklist.ts` |
| Frontend hook | `frontend/src/hooks/useBlocklist.ts` |
| Frontend layout | `frontend/src/layouts/MainLayout.tsx` |
| Tests | `backend/tests/test_services/test_blocklist_service.py` |
| Tests | `backend/tests/test_routers/test_blocklist.py` |