BanGUI/Docs/Tasks.md

BanGUI — Task List

This document breaks the entire BanGUI project into development stages, ordered so that each stage builds on the previous one. Every task is described in prose with enough detail for a developer to begin work. References point to the relevant documentation.

---

Task 1 — Mark Imported Blocklist IP Addresses ✅ DONE

Completed: Added origin field ("blocklist" / "selfblock") to Ban and DashboardBanItem models, derived from jail name in ban_service.py. BanTable and MapPage companion table display an Origin badge column. Tests added to test_ban_service.py and test_dashboard.py.

Problem

When IPs are imported from an external blocklist they are applied to the blocklist-import jail via the fail2ban socket. Once they land in the fail2ban SQLite bans table they look identical to IPs that were banned organically by fail2ban filters. There is no way for the dashboard or map to tell the user whether a given ban came from a blocklist import or from a real failed-login detection.

Goal

Every ban displayed in the UI must carry a visible origin indicator: either blocklist (imported from a blocklist source) or selfblock (detected by fail2ban itself).

Implementation Details

Backend — derive origin from jail name

The blocklist import service already uses a dedicated jail called blocklist-import. This can be used as the discriminator: any ban whose jail column equals blocklist-import is a blocklist ban; everything else is a self-block.

1. Model change — In backend/app/models/ban.py, add an origin field of type Literal["blocklist", "selfblock"] to both Ban and DashboardBanItem. Compute it from the jail value during construction (if jail == "blocklist-import" → "blocklist", else "selfblock").
2. Service change — In backend/app/services/ban_service.py, make sure list_bans() and bans_by_country() populate the new origin field when building ban objects.
3. API response — The JSON payloads from GET /api/dashboard/bans and GET /api/dashboard/bans/by-country already serialise every field of DashboardBanItem, so origin will appear automatically once the model is updated.

Frontend — show origin badge

4. BanTable — In frontend/src/components/BanTable.tsx, add an Origin column. Render a small coloured badge: blue label Blocklist or grey label Selfblock.
5. MapPage detail table — The table below the world map in frontend/src/pages/MapPage.tsx also lists bans. Add the same origin badge column there.

Tests

6. Add a unit test in backend/tests/test_services/test_ban_service.py that inserts bans into a mock fail2ban DB under both jail names (blocklist-import and e.g. sshd) and asserts the returned objects carry the correct origin value.
7. Add a router test in backend/tests/test_routers/test_dashboard.py that verifies the JSON response contains the origin field.

Files Touched

Layer	File
Model	backend/app/models/ban.py
Service	backend/app/services/ban_service.py
Frontend	frontend/src/components/BanTable.tsx
Frontend	frontend/src/pages/MapPage.tsx
Tests	backend/tests/test_services/test_ban_service.py
Tests	backend/tests/test_routers/test_dashboard.py

---

Task 2 — Add Origin Filter to Dashboard and World Map ✅ DONE

Completed: Added optional origin query parameter ("blocklist" / "selfblock") to both dashboard router endpoints, filtered via jail-based WHERE clause in ban_service. Frontend: BanOriginFilter type + BAN_ORIGIN_FILTER_LABELS in types/ban.ts; fetchBans and fetchBansByCountry APIs forward the param; useBans and useMapData hooks accept origin; DashboardPage and MapPage show a segmented toggle / select for All / Blocklist / Selfblock. Tests extended in test_ban_service.py and test_dashboard.py.

Problem

Once Task 1 exposes the origin field, users need a way to filter the view so they can see all bans, only blocklist bans, or only self-blocked bans. This must work on both the dashboard ban table and the world map.

Goal

Add a filter dropdown (or segmented toggle) with three options — All, Blocklist, Selfblock — to the dashboard toolbar and the map toolbar. The selection must propagate to the backend so that only matching bans are returned and pagination / country aggregation remain correct.

Implementation Details

Backend — add origin query parameter

1. Dashboard router — GET /api/dashboard/bans in backend/app/routers/dashboard.py: add an optional query parameter origin: Optional[Literal["blocklist", "selfblock"]] = None. Pass it through to the service layer.
2. Map router — GET /api/dashboard/bans/by-country in the same router: add the same origin parameter.
3. Service layer — In backend/app/services/ban_service.py:
   • list_bans(): when origin is provided, append a WHERE clause on the jail column (jail = 'blocklist-import' for blocklist, jail != 'blocklist-import' for selfblock).
   • bans_by_country(): apply the same jail-based filter so that country aggregation only counts matching bans.

Frontend — filter controls

4. Shared state — Create a small shared type BanOriginFilter = "all" | "blocklist" | "selfblock" (e.g. in frontend/src/types/ or inline).
5. DashboardPage — In frontend/src/pages/DashboardPage.tsx, add a dropdown or segmented control next to the existing time-range toolbar. Store the selected value in component state. Pass it to useBans hook, which forwards it as the origin query parameter.
6. useBans hook — In frontend/src/hooks/useBans.ts, accept an optional origin parameter and include it in the API call via fetchBans().
7. API function — In frontend/src/api/dashboard.ts, update fetchBans() to accept and forward the origin query parameter.
8. MapPage — In frontend/src/pages/MapPage.tsx, add the same dropdown. Pass the selected value to useMapData hook.
9. useMapData hook — In frontend/src/hooks/useMapData.ts, accept origin and forward it to fetchBansByCountry().
10. Map API function — In frontend/src/api/map.ts, update fetchBansByCountry() to include origin in the query string.

Tests

11. Extend backend/tests/test_services/test_ban_service.py: insert bans under multiple jails, call list_bans(origin="blocklist") and assert only blocklist-import jail bans are returned; repeat for "selfblock" and None.
12. Extend backend/tests/test_routers/test_dashboard.py: hit GET /api/dashboard/bans?origin=blocklist and verify the response only contains blocklist bans.

Files Touched

Layer	File
Router	backend/app/routers/dashboard.py
Service	backend/app/services/ban_service.py
Frontend	frontend/src/pages/DashboardPage.tsx
Frontend	frontend/src/pages/MapPage.tsx
Frontend	frontend/src/hooks/useBans.ts
Frontend	frontend/src/hooks/useMapData.ts
Frontend	frontend/src/api/dashboard.ts
Frontend	frontend/src/api/map.ts
Tests	backend/tests/test_services/test_ban_service.py
Tests	backend/tests/test_routers/test_dashboard.py

---

Task 3 — Performance Optimisation for 10 k+ IPs (Dashboard & World Map)

Problem

With a large number of banned IPs (> 10 000), both the dashboard and the world map take over 10 seconds to load and become unusable. The root cause is the geo-enrichment step: geo_service.py calls the external ip-api.com endpoint one IP at a time with a 5-second timeout, and the free tier is rate-limited to 45 requests/minute. On a cold cache with 10 k IPs, this would take hours.

Goal

Dashboard and world map must load within 2 seconds for 10 k banned IPs. Write a reproducible benchmark test to prove it.

Implementation Details

Backend — persistent geo cache

1. SQLite geo cache table — In backend/app/db.py, add a geo_cache table:

   CREATE TABLE IF NOT EXISTS geo_cache (
       ip TEXT PRIMARY KEY,
       country_code TEXT,
       country_name TEXT,
       asn TEXT,
       org TEXT,
       cached_at TEXT NOT NULL
   );
   

   On startup the service loads from this table instead of starting with an empty dict. Lookups hit the in-memory dict first, then fall through to the DB, and only as a last resort call the external API. Successful API responses are written back to both caches.

2. Batch lookup via ip-api.com batch endpoint — The free ip-api.com API supports a batch POST to http://ip-api.com/batch accepting up to 100 IPs per request. Refactor geo_service.py to:

   • Collect all uncached IPs from the requested page.
   • Send them in chunks of 100 to the batch endpoint.
   • Parse the JSON array response and populate caches in one pass. This alone reduces 10 000 cold lookups from 10 000 sequential requests to ≈ 100 batch calls.

3. Pre-warm cache on import — In backend/app/services/blocklist_service.py, after a successful blocklist import, fire a background task that batch-resolves all newly imported IPs. This way the dashboard never faces a cold cache for blocklist IPs.

Backend — limit the country-aggregation query

4. In ban_service.py, bans_by_country() currently loads up to 2 000 bans and enriches every one. Change it to:
   • Run the aggregation (GROUP BY jail, count per jail) in SQL directly against the fail2ban DB.
   • Only enrich the distinct IPs, batch-resolved.
   • Return the aggregated country → count map without fetching full ban rows.

Frontend — virtualised table

5. The BanTable component currently renders all rows in the DOM. For 10 k+ rows (even paginated at 100), scrolling is fine, but if page sizes are increased or if pagination is removed, install a virtual-scrolling library (e.g. @tanstack/react-virtual) and render only visible rows. Alternatively, ensure page size stays capped at ≤ 500 (already enforced) and measure whether DOM performance is acceptable — only virtualise if needed.

Frontend — map debounce

6. In WorldMap.tsx, add a loading skeleton / spinner and debounce the data fetch. If the user switches time ranges rapidly, cancel in-flight requests to avoid piling up stale responses.

Performance test

7. Write a pytest benchmark in backend/tests/test_services/test_ban_service_perf.py:
   • Seed a temporary fail2ban SQLite with 10 000 synthetic bans (random IPs, mixed jails, spread over 365 days).
   • Pre-populate the geo cache with matching entries so the test does not hit the network.
   • Call list_bans(range="365d", page=1, page_size=100) and bans_by_country(range="365d").
   • Assert both return within 2 seconds wall-clock time.
8. Add a manual/integration test script backend/tests/scripts/seed_10k_bans.py that inserts 10 000 bans into the real fail2ban dev DB and pre-caches their geo data so developers can visually test dashboard and map load times in the browser.

Files Touched

Layer	File
DB schema	backend/app/db.py
Geo service	backend/app/services/geo_service.py
Ban service	backend/app/services/ban_service.py
Blocklist service	backend/app/services/blocklist_service.py
Frontend	frontend/src/components/BanTable.tsx
Frontend	frontend/src/components/WorldMap.tsx
Tests	backend/tests/test_services/test_ban_service_perf.py (new)
Scripts	backend/tests/scripts/seed_10k_bans.py (new)

---

Task 4 — Fix Missing Country for Resolved IPs

Problem

Many IPs (e.g. 5.167.71.2, 5.167.71.3, 5.167.71.4) show no country in the dashboard and map. The current geo_service.py calls ip-api.com with a 5-second timeout and silently returns None fields on any error or timeout. Common causes:

• Rate limiting — the free tier allows 45 req/min; once exceeded, responses return "status": "fail" and the service caches None values, permanently hiding the country.
• Cache poisoning with empty entries — a failed lookup stores GeoInfo(country_code=None, ...) in the in-memory cache. Until the cache is flushed (at 10 000 entries), that IP will always appear without a country.
• External API unreachable — network issues or container DNS problems cause timeouts that are treated the same as "no data".

Goal

Every IP that has a valid geographic mapping should display its country. Failed lookups must be retried on subsequent requests rather than permanently cached as empty.

Implementation Details

Backend — do not cache failed lookups

1. In backend/app/services/geo_service.py, change the caching logic: only store a result in the in-memory cache (and the new persistent geo_cache table from Task 3) when country_code is not None. If the API returned a failure or the request timed out, do not cache the result so it will be retried on the next request.

Backend — negative-cache with short TTL

2. To avoid hammering the API for the same failing IP on every request, introduce a separate negative cache — a dict mapping IP → timestamp of last failed attempt. Skip re-lookup if the last failure was less than 5 minutes ago. After 5 minutes the IP becomes eligible for retry.

Backend — fallback to a local GeoIP database

3. Add geoip2 (MaxMind GeoLite2) as an optional fallback. If the free ip-api.com lookup fails or is rate-limited, attempt a local lookup using the GeoLite2-Country database (.mmdb file). This provides offline country resolution for the vast majority of IPv4 addresses.
   • Add geoip2 to backend/pyproject.toml dependencies.
   • Download the GeoLite2-Country database during Docker build or via a setup script (requires free MaxMind license key).
   • In geo_service.py, try ip-api.com first; on failure, fall back to geoip2; only return None if both fail.

Backend — bulk re-resolve endpoint

4. Add POST /api/geo/re-resolve in backend/app/routers/geo.py that:
   • Queries all currently cached IPs with country_code IS NULL.
   • Re-resolves them in batches (using the batch endpoint from Task 3).
   • Returns a count of newly resolved IPs.
   • This lets the admin manually fix historical gaps.

Frontend — visual indicator for unknown country

5. In BanTable.tsx and the MapPage detail table, display a small "unknown" icon or — placeholder when country is null, instead of leaving the cell empty. Tooltip: "Country could not be resolved — will retry automatically."

Tests

6. In backend/tests/test_services/test_geo_service.py:
   • Test that a failed lookup is not persisted in the positive cache.
   • Test that a failed lookup is stored in the negative cache and skipped for 5 minutes.
   • Test that after the negative-cache TTL expires, the IP is re-queried.
   • Mock the geoip2 fallback and verify it is called when ip-api fails.
7. In backend/tests/test_routers/test_geo.py, test the new POST /api/geo/re-resolve endpoint.

Files Touched

Layer	File
Geo service	backend/app/services/geo_service.py
Geo router	backend/app/routers/geo.py
Dependencies	backend/pyproject.toml
Frontend	frontend/src/components/BanTable.tsx
Frontend	frontend/src/pages/MapPage.tsx
Tests	backend/tests/test_services/test_geo_service.py
Tests	backend/tests/test_routers/test_geo.py