Add dashboard country charts (Stages 1–3)
- Install Recharts v3 as the project charting library - Add chartTheme utility with Fluent UI v9 token resolution helper and a 5-colour categorical palette (resolves CSS vars at runtime) - Add TopCountriesPieChart: top-4 + Other slice, Tooltip, Legend - Add TopCountriesBarChart: horizontal top-20 bar chart - Add useDashboardCountryData hook (wraps /api/dashboard/bans/by-country) - Integrate both charts into DashboardPage in a responsive chartsRow (side-by-side on wide screens, stacked on narrow) - All tsc --noEmit and eslint checks pass with zero warnings
This commit is contained in:
583
Docs/Tasks.md
583
Docs/Tasks.md
@@ -4,224 +4,479 @@ This document breaks the entire BanGUI project into development stages, ordered
|
||||
|
||||
---
|
||||
|
||||
## Issue: World Map Loading Time — Architecture Fix
|
||||
## Stage 1 — Dashboard Charts Foundation
|
||||
|
||||
### Problem Summary
|
||||
### Task 1.1 — Install and configure a charting library
|
||||
|
||||
The `GET /api/dashboard/bans/by-country` endpoint is extremely slow on first load. A single request with ~5,200 unique IPs produces **10,400 SQLite commits** and **6,000 INSERT statements** against the app database — all during a read-only GET request. The log shows 21,000+ lines of SQL trace for just 18 HTTP requests.
|
||||
**Status:** `done`
|
||||
|
||||
Root causes (ordered by impact):
|
||||
The frontend currently has no charting library. Install **Recharts** (`recharts`) as the project charting library. Recharts is React-native, composable, and integrates cleanly with Fluent UI v9 theming.
|
||||
|
||||
1. **Per-IP commit during geo cache writes** — `geo_service._persist_entry()` and `_persist_neg_entry()` each call `await db.commit()` after every single INSERT. With 5,200 uncached IPs this means 5,200+ individual commits, each forcing an `fsync`. This is the dominant bottleneck.
|
||||
2. **DB writes on a GET request** — The bans/by-country endpoint passes `app_db` to `geo_service.lookup_batch()`, which triggers INSERT+COMMIT for every resolved IP. A GET request should never produce database writes/commits. Users do not expect loading a map page to mutate the database.
|
||||
3. **Same pattern exists in other endpoints** — The following GET endpoints also trigger geo cache commits: `/api/dashboard/bans`, `/api/bans/active`, `/api/history`, `/api/history/{ip}`, `/api/geo/lookup/{ip}`.
|
||||
**Steps:**
|
||||
|
||||
### Evidence from `log.log`
|
||||
1. Run `npm install recharts` in the `frontend/` directory.
|
||||
2. Verify the dependency appears in `package.json` under `dependencies`.
|
||||
3. Confirm the build still succeeds with `npm run build` (no type errors, no warnings).
|
||||
|
||||
- Log line count: **21,117 lines** for 18 HTTP requests
|
||||
- `INSERT INTO geo_cache`: **6,000** executions
|
||||
- `db.commit()`: **10,400** calls (each INSERT + its commit = 2 ops per IP)
|
||||
- `geo_batch_lookup_start`: reports `total=5200` uncached IPs
|
||||
- The bans/by-country response is at line **21,086** out of 21,117 — the entire log is essentially one request's geo persist work
|
||||
- Other requests (`/api/dashboard/status`, `/api/blocklists/schedule`, `/api/config/map-color-thresholds`) interleave with the geo persist loop because they share the same single async DB connection
|
||||
No wrapper or configuration file is needed — Recharts components are imported directly where used.
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- `recharts` is listed in `frontend/package.json`.
|
||||
- `npm run build` succeeds with zero errors or warnings.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Batch geo cache writes — eliminate per-IP commits ✅ DONE
|
||||
### Task 1.2 — Create a shared chart theme utility
|
||||
|
||||
**File:** `backend/app/services/geo_service.py`
|
||||
**Status:** `done`
|
||||
|
||||
**What to change:**
|
||||
Create a small utility at `frontend/src/utils/chartTheme.ts` that exports a function (or constant object) mapping Fluent UI v9 design tokens to Recharts-compatible colour values. The charts must respect the current Fluent theme (light and dark mode). At minimum export:
|
||||
|
||||
The functions `_persist_entry()` and `_persist_neg_entry()` each call `await db.commit()` after every INSERT. Instead, the commit should happen once after the entire batch is processed.
|
||||
- A palette of 5+ distinct categorical colours for pie/bar slices, derived from Fluent token aliases (e.g. `colorPaletteBlueBorderActive`, `colorPaletteRedBorderActive`, etc.).
|
||||
- Axis/grid/tooltip colours derived from `colorNeutralForeground2`, `colorNeutralStroke2`, `colorNeutralBackground1`, etc.
|
||||
- A helper that returns the CSS value of a Fluent token at runtime (since Recharts needs literal CSS colour strings, not CSS custom properties).
|
||||
|
||||
1. Remove `await db.commit()` from both `_persist_entry()` and `_persist_neg_entry()`.
|
||||
2. In `lookup_batch()`, after the loop over all chunks is complete and all `_persist_entry()` / `_persist_neg_entry()` calls have been made, issue a single `await db.commit()` if `db is not None`.
|
||||
3. Wrap the single commit in a try/except to handle any errors gracefully.
|
||||
Keep the file under 60 lines. No React components here — pure utility.
|
||||
|
||||
**Expected impact:** Reduces commits from ~5,200 to **1** per request. This alone should cut the endpoint response time dramatically.
|
||||
**References:** [Web-Design.md](Web-Design.md) § colour tokens.
|
||||
|
||||
**Testing:** Existing tests in `test_services/test_ban_service.py` and `test_services/test_geo_service.py` should continue to pass. Verify the geo_cache table still gets populated after a batch lookup by checking the DB contents in an integration test.
|
||||
**Acceptance criteria:**
|
||||
|
||||
- The exported palette contains at least 5 distinct colours.
|
||||
- Colours change correctly between light and dark mode.
|
||||
- `tsc --noEmit` and `eslint` pass with zero warnings.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Do not write geo cache during GET requests ✅ DONE
|
||||
## Stage 2 — Country Pie Chart (Top 4 + Other)
|
||||
|
||||
**Files:** `backend/app/routers/dashboard.py`, `backend/app/routers/bans.py`, `backend/app/routers/history.py`, `backend/app/routers/geo.py`
|
||||
### Task 2.1 — Create the `TopCountriesPieChart` component
|
||||
|
||||
**What to change:**
|
||||
**Status:** `done`
|
||||
|
||||
GET endpoints should not pass `app_db` (or equivalent) to geo_service functions. The geo resolution should still populate the in-memory cache (which is fast, free, and ephemeral), but should NOT write to SQLite during a read request.
|
||||
Create `frontend/src/components/TopCountriesPieChart.tsx`. This component renders a **pie chart (Kuchendiagramm)** showing the **top 4 countries by ban count** plus an **"Other"** slice that aggregates every remaining country.
|
||||
|
||||
For each of these GET endpoints:
|
||||
- `GET /api/dashboard/bans/by-country` in `dashboard.py` — stop passing `app_db=db` to `bans_by_country()`; pass `app_db=None` instead.
|
||||
- `GET /api/dashboard/bans` in `dashboard.py` — stop passing `app_db=db` to `list_bans()`; pass `app_db=None` instead.
|
||||
- `GET /api/bans/active` in `bans.py` — the enricher callback should not pass `db` to `geo_service.lookup()`.
|
||||
- `GET /api/history` and `GET /api/history/{ip}` in `history.py` — same: enricher should not pass `db`.
|
||||
- `GET /api/geo/lookup/{ip}` in `geo.py` — do not pass `db` to `geo_service.lookup()`.
|
||||
**Data source:** The component receives the `countries` map (`Record<string, number>`) and `country_names` map (`Record<string, string>`) from the existing `/api/dashboard/bans/by-country` endpoint response (`BansByCountryResponse`). No new API endpoint is needed.
|
||||
|
||||
The persistent geo cache should only be written during explicit write operations:
|
||||
- `POST /api/geo/re-resolve` (already a POST — this is correct)
|
||||
- Blocklist import tasks (`blocklist_service.py`)
|
||||
- Application startup via `load_cache_from_db()`
|
||||
**Aggregation logic (frontend):**
|
||||
|
||||
**Expected impact:** GET requests become truly read-only. No commits, no `fsync`, no write contention on the app DB during map loads.
|
||||
1. Sort the `countries` entries descending by ban count.
|
||||
2. Take the top 4 entries.
|
||||
3. Sum all remaining entries into a single `"Other"` bucket.
|
||||
4. The result is exactly 5 slices (or fewer if fewer than 5 countries exist).
|
||||
|
||||
**Testing:** Run the full test suite. Verify that:
|
||||
1. The bans/by-country endpoint still returns correct country data (from in-memory cache).
|
||||
2. The `geo_cache` table is still populated when `POST /api/geo/re-resolve` is called or after blocklist import.
|
||||
3. After a server restart, geo data is still available (because `load_cache_from_db()` warms memory from previously persisted data).
|
||||
**Visual requirements:**
|
||||
|
||||
---
|
||||
- Use `<PieChart>` and `<Pie>` from Recharts with `<Cell>` for per-slice colours from the chart theme palette (Task 1.2).
|
||||
- Display a `<Tooltip>` on hover showing the country name and ban count.
|
||||
- Display a `<Legend>` listing each slice with its country name (full name from `country_names`, not just the code) and percentage.
|
||||
- Label each slice with the percentage (use Recharts `label` prop or `<Label>`).
|
||||
- Use `makeStyles` for any layout styling. Follow [Web-Design.md](Web-Design.md) spacing and card conventions.
|
||||
- Wrap the chart in a responsive container so it scales with its parent.
|
||||
|
||||
### Task 3: Periodically persist the in-memory geo cache (background task) ✅ DONE
|
||||
**Props interface:**
|
||||
|
||||
**Files:** `backend/app/services/geo_service.py`, `backend/app/tasks/` (new task file)
|
||||
|
||||
**What to change:**
|
||||
|
||||
After Task 2, GET requests no longer write to the DB. But newly resolved IPs during GET requests only live in the in-memory cache and would be lost on restart. To avoid this, add a background task that periodically flushes new in-memory entries to the `geo_cache` table.
|
||||
|
||||
1. In `geo_service.py`, add a module-level `_dirty: set[str]` that tracks IPs added to `_cache` but not yet persisted. When `_store()` adds an entry, also add the IP to `_dirty`.
|
||||
2. Add a new function `flush_dirty(db: aiosqlite.Connection) -> int` that:
|
||||
- Takes the current `_dirty` set and clears it atomically.
|
||||
- Uses `executemany()` to batch-INSERT all dirty entries in one transaction.
|
||||
- Calls `db.commit()` once.
|
||||
- Returns the count of flushed entries.
|
||||
3. Register a periodic task (using the existing APScheduler setup) that calls `flush_dirty()` every 60 seconds (configurable). This is similar to how other background tasks already run.
|
||||
|
||||
**Expected impact:** Geo data is persisted without blocking any request. If the server restarts, at most 60 seconds of new geo data is lost (and it will simply be re-fetched from ip-api.com on the next request).
|
||||
|
||||
**Testing:** Write a test that:
|
||||
- Calls `lookup_batch()` without a DB reference.
|
||||
- Verifies IPs are in `_dirty`.
|
||||
- Calls `flush_dirty(db)`.
|
||||
- Verifies the geo_cache table contains the entries and `_dirty` is empty.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Reduce redundant SQL queries per request (settings / auth) ✅ DONE
|
||||
|
||||
**Files:** `backend/app/dependencies.py`, `backend/app/main.py`, `backend/app/repositories/settings_repo.py`
|
||||
|
||||
**What to change:**
|
||||
|
||||
The log shows that every single HTTP request executes at least 2 separate SQL queries before reaching the actual endpoint logic:
|
||||
1. `SELECT value FROM settings WHERE key = 'setup_completed'` (SetupRedirectMiddleware)
|
||||
2. `SELECT id, token, ... FROM sessions WHERE token = ?` (require_auth dependency)
|
||||
|
||||
When multiple requests arrive concurrently (as seen in the log — 3 parallel requests trigger 3× setup_completed + 3× session token queries), this adds unnecessary DB contention.
|
||||
|
||||
Options (implement one or both):
|
||||
- **Cache `setup_completed` in memory:** Once setup is complete, it never goes back to incomplete. Cache the result in `app.state` and skip the DB query on subsequent requests. Set it on first `True` read and clear it only if the app restarts.
|
||||
- **Cache session validation briefly:** Use a short TTL in-memory cache (e.g., 5–10 seconds) for validated session tokens. This reduces repeated DB lookups for the same token across near-simultaneous requests.
|
||||
|
||||
**Expected impact:** Reduces per-request overhead from 2+ SQL queries to 0 for the common case (setup done, session recently validated).
|
||||
|
||||
**Testing:** Existing auth and setup tests must continue to pass. Add a test that validates the cached path (second request skips DB).
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Audit and verify — run full test suite ✅ DONE
|
||||
|
||||
After tasks 1–4 are implemented, run:
|
||||
|
||||
```bash
|
||||
cd backend && python -m pytest tests/ -x -q
|
||||
```ts
|
||||
interface TopCountriesPieChartProps {
|
||||
countries: Record<string, number>;
|
||||
countryNames: Record<string, string>;
|
||||
}
|
||||
```
|
||||
|
||||
Verify:
|
||||
- All tests pass (460 passing, up from 443 baseline).
|
||||
- `ruff check backend/app/` passes.
|
||||
- `mypy --strict` passes on all changed files.
|
||||
- 83% overall coverage (above the 80% threshold).
|
||||
**Acceptance criteria:**
|
||||
|
||||
- Always renders exactly 5 slices (or fewer when data has < 5 countries).
|
||||
- The "Other" slice correctly sums all countries outside the top 4.
|
||||
- Tooltip displays country name + count on hover.
|
||||
- Legend shows country name + percentage.
|
||||
- Responsive — no horizontal overflow on narrow viewports.
|
||||
- `tsc --noEmit` passes. No `any` types. ESLint clean.
|
||||
|
||||
---
|
||||
|
||||
## Developer Notes — Learnings & Gotchas
|
||||
### Task 2.2 — Create a `useDashboardCountryData` hook
|
||||
|
||||
These notes capture non-obvious findings from the investigation. Read them before you start coding.
|
||||
**Status:** `done`
|
||||
|
||||
### Architecture Overview
|
||||
Create `frontend/src/hooks/useDashboardCountryData.ts`. This hook wraps the existing `GET /api/dashboard/bans/by-country` call and returns the data the dashboard charts need. The existing `useMapData` hook is designed for the map page and should not be reused because it is coupled to map-specific debouncing and state.
|
||||
|
||||
BanGUI has **two separate SQLite databases**:
|
||||
**Signature:**
|
||||
|
||||
1. **fail2ban DB** — owned by fail2ban, opened read-only (`?mode=ro`) via `aiosqlite.connect(f"file:{path}?mode=ro", uri=True)`. Path is discovered at runtime by asking the fail2ban daemon (`get dbfile` via Unix socket). Contains the `bans` table.
|
||||
2. **App DB** (`bangui.db`) — BanGUI's own database. Holds `settings`, `sessions`, `geo_cache`, `blocklist_sources`, `import_log`. This is the one being hammered by commits during GET requests.
|
||||
```ts
|
||||
function useDashboardCountryData(
|
||||
timeRange: TimeRange,
|
||||
origin: BanOriginFilter,
|
||||
): {
|
||||
countries: Record<string, number>;
|
||||
countryNames: Record<string, string>;
|
||||
bans: DashboardBanItem[];
|
||||
total: number;
|
||||
isLoading: boolean;
|
||||
error: string | null;
|
||||
};
|
||||
```
|
||||
|
||||
There is a **single shared app DB connection** living at `request.app.state.db`. All concurrent requests share it. This means long-running writes (like 5,200 sequential INSERT+COMMIT loops) block other requests that need the same connection. The log confirms this: `setup_completed` checks and session lookups from parallel requests interleave with the geo persist loop.
|
||||
**Behaviour:**
|
||||
|
||||
### The Geo Resolution Pipeline
|
||||
- Call `GET /api/dashboard/bans/by-country?range={timeRange}` with optional `origin` query param (omit when `"all"`).
|
||||
- Use the typed API client from `api/client.ts`.
|
||||
- Set `isLoading` while fetching, populate `error` on failure.
|
||||
- Re-fetch when `timeRange` or `origin` changes.
|
||||
- Mirror the data-fetching patterns used by `useBans` / `useMapData`.
|
||||
|
||||
`geo_service.py` implements a two-tier cache:
|
||||
**Acceptance criteria:**
|
||||
|
||||
1. **In-memory dict** (`_cache: dict[str, GeoInfo]`) — module-level, lives for the process lifetime. Fast, no I/O.
|
||||
2. **SQLite `geo_cache` table** — survives restarts. Loaded into `_cache` at startup via `load_cache_from_db()`.
|
||||
- Returns typed data matching `BansByCountryResponse`.
|
||||
- Re-fetches on param change.
|
||||
- `tsc --noEmit` and ESLint pass.
|
||||
|
||||
There is also a **negative cache** (`_neg_cache: dict[str, float]`) for failed lookups with a 5-minute TTL. Failed IPs are not retried within that window.
|
||||
---
|
||||
|
||||
The batch resolution flow in `lookup_batch()`:
|
||||
1. Check `_cache` and `_neg_cache` for each IP → split into cached vs uncached.
|
||||
2. Send uncached IPs to `ip-api.com/batch` in chunks of 100.
|
||||
3. For each resolved IP: update `_cache` (fast) AND call `_persist_entry(db, ip, info)` (slow — INSERT + COMMIT).
|
||||
4. For failed IPs: try MaxMind GeoLite2 local DB fallback (`_geoip_lookup()`). If that also fails, add to `_neg_cache` and call `_persist_neg_entry()`.
|
||||
### Task 2.3 — Integrate the pie chart into `DashboardPage`
|
||||
|
||||
**Critical insight:** Step 3 is where the bottleneck lives. The `_persist_entry` function issues a separate `await db.commit()` after each INSERT. With 5,200 IPs, that's 5,200 `fsync` calls — each one waits for the disk.
|
||||
**Status:** `done`
|
||||
|
||||
### Specific File Locations You Need
|
||||
Add the `TopCountriesPieChart` below the `ServerStatusBar` and above the "Ban List" section on the `DashboardPage`. The chart must share the same `timeRange` and `originFilter` state that already exists on the page.
|
||||
|
||||
| File | Key functions | Notes |
|
||||
|------|--------------|-------|
|
||||
| `backend/app/services/geo_service.py` L231–260 | `_persist_entry()` | The INSERT + COMMIT per IP — **this is the hot path** |
|
||||
| `backend/app/services/geo_service.py` L262–280 | `_persist_neg_entry()` | Same pattern for failed lookups |
|
||||
| `backend/app/services/geo_service.py` L374–460 | `lookup_batch()` | Main batch function — calls `_persist_entry` in a loop |
|
||||
| `backend/app/services/geo_service.py` L130–145 | `_store()` | Updates the in-memory `_cache` dict — fast, no I/O |
|
||||
| `backend/app/services/geo_service.py` L202–230 | `load_cache_from_db()` | Startup warm-up, reads entire `geo_cache` table into memory |
|
||||
| `backend/app/services/ban_service.py` L326–430 | `bans_by_country()` | Calls `lookup_batch()` with `db=app_db` |
|
||||
| `backend/app/services/ban_service.py` L130–210 | `list_bans()` | Also calls `lookup_batch()` with `app_db` |
|
||||
| `backend/app/routers/dashboard.py` | `get_bans_by_country()` | Passes `app_db=db` — this is where db gets threaded through |
|
||||
| `backend/app/routers/bans.py` | `get_active_bans()` | Uses single-IP `lookup()` via enricher callback with `db` |
|
||||
| `backend/app/routers/history.py` | `get_history()`, `get_ip_history()` | Same enricher-with-db pattern |
|
||||
| `backend/app/routers/geo.py` | `lookup_ip()` | Single IP lookup, passes `db` |
|
||||
| `backend/app/main.py` L268–306 | `SetupRedirectMiddleware` | Runs `get_setting(db, "setup_completed")` on every request |
|
||||
| `backend/app/dependencies.py` L54–100 | `require_auth()` | Runs session token SELECT on every authenticated request |
|
||||
| `backend/app/repositories/settings_repo.py` | `get_setting()` | Individual SELECT per key; `get_all_settings()` exists but is unused in middleware |
|
||||
**Layout:**
|
||||
|
||||
### Endpoints That Commit During GET Requests
|
||||
- Place the pie chart inside a new section card (reuse the `section` / `sectionHeader` pattern from the existing ban-list section).
|
||||
- Section title: **"Top Countries"**.
|
||||
- The pie chart card sits in a future row of chart cards (see Task 3.3). For now, render it full-width. Use a CSS class name like `chartsRow` so the bar chart can be added beside it later.
|
||||
|
||||
All of these GET endpoints currently write to the app DB via geo_service:
|
||||
**Acceptance criteria:**
|
||||
|
||||
| Endpoint | How | Commit count per request |
|
||||
|----------|-----|--------------------------|
|
||||
| `GET /api/dashboard/bans/by-country` | `bans_by_country()` → `lookup_batch()` → `_persist_entry()` per IP | Up to N (N = uncached IPs, can be thousands) |
|
||||
| `GET /api/dashboard/bans` | `list_bans()` → `lookup_batch()` → `_persist_entry()` per IP | Up to page_size (max 500) |
|
||||
| `GET /api/bans/active` | enricher → `lookup()` → `_persist_entry()` per IP | 1 per ban in response |
|
||||
| `GET /api/history` | enricher → `lookup()` → `_persist_entry()` per IP | 1 per row |
|
||||
| `GET /api/history/{ip}` | enricher → `lookup()` → `_persist_entry()` | 1 |
|
||||
| `GET /api/geo/lookup/{ip}` | `lookup()` → `_persist_entry()` | 1 |
|
||||
- The pie chart renders on the dashboard, respecting the selected time range and origin filter.
|
||||
- Changing the time range or origin filter re-renders the chart with new data.
|
||||
- The loading and error states from the hook are handled (show `<Spinner>` while loading, `<MessageBar>` on error).
|
||||
- `tsc --noEmit` and ESLint pass.
|
||||
|
||||
The only endpoint that **should** write geo data is `POST /api/geo/re-resolve` (already a POST).
|
||||
---
|
||||
|
||||
### Concurrency / Connection Sharing Issue
|
||||
## Stage 3 — Country Bar Chart (Top 20)
|
||||
|
||||
The app DB connection (`app.state.db`) is a single `aiosqlite.Connection`. aiosqlite serialises operations through a background thread, so concurrent `await db.execute()` calls from different request handlers are queued. This is visible in the log: while the geo persist loop runs its 5,200 INSERT+COMMITs, other requests' `setup_completed` and session-token queries get interleaved between commits. They all complete, but everything is slower because they wait in the queue.
|
||||
### Task 3.1 — Create the `TopCountriesBarChart` component
|
||||
|
||||
This is not a bug to fix right now, but keep it in mind: if you batch the commits (Task 1) and stop writing on GETs (Task 2), the contention problem largely goes away because the long-running write loop no longer exists.
|
||||
**Status:** `done`
|
||||
|
||||
### Test Infrastructure
|
||||
Create `frontend/src/components/TopCountriesBarChart.tsx`. This component renders a **horizontal bar chart (Balkendiagramm)** showing the **top 20 countries by ban count**.
|
||||
|
||||
- **443 tests** currently passing, **82% coverage**.
|
||||
- Tests use `pytest` + `pytest-asyncio` + `httpx.AsyncClient`.
|
||||
- External dependencies (fail2ban socket, ip-api.com) are fully mocked in tests.
|
||||
- Run with: `cd backend && python -m pytest tests/ -x -q`
|
||||
- Lint: `ruff check backend/app/`
|
||||
- Types: `mypy --strict` on changed files
|
||||
- All code must follow rules in `Docs/Backend-Development.md`.
|
||||
**Data source:** Same `countries` and `country_names` maps from `BansByCountryResponse` — passed as props identical to the pie chart.
|
||||
|
||||
### What NOT to Do
|
||||
**Aggregation logic (frontend):**
|
||||
|
||||
1. **Do not add a second DB connection** to "fix" the concurrency issue. The single-connection model is intentional for SQLite (WAL mode notwithstanding). Batching commits is the correct fix.
|
||||
2. **Do not remove the SQLite geo_cache entirely.** It serves a real purpose: surviving process restarts without re-fetching thousands of IPs from ip-api.com.
|
||||
3. **Do not cache geo data in Redis or add a new dependency.** The two-tier cache (in-memory dict + SQLite) is the right architecture for this app's scale. The problem is purely commit frequency.
|
||||
4. **Do not change the `_cache` dict to an LRU or TTL cache.** The current eviction strategy (flush at 50,000 entries) is fine. The issue is the persistent layer, not the in-memory layer.
|
||||
5. **Do not skip writing test cases.** The project enforces >80% coverage. Every change needs tests.
|
||||
1. Sort the `countries` entries descending by ban count.
|
||||
2. Take the top 20 entries.
|
||||
3. No "Other" bucket — the bar chart is detail-focused.
|
||||
|
||||
**Visual requirements:**
|
||||
|
||||
- Use `<BarChart>` (horizontal via `layout="vertical"`) from Recharts with `<Bar>`, `<XAxis>`, `<YAxis>`, `<CartesianGrid>`, and `<Tooltip>`.
|
||||
- Y-axis shows country names (full name from `country_names`, truncated to ~20 chars with ellipsis if needed).
|
||||
- X-axis shows ban count (numeric).
|
||||
- Bars are coloured with the primary colour from the chart theme palette.
|
||||
- Tooltip shows the full country name and exact ban count.
|
||||
- Chart height should be dynamic based on the number of bars (e.g. `barCount * 36px` min), with a reasonable minimum height.
|
||||
- Wrap in a `<ResponsiveContainer>` for width.
|
||||
|
||||
**Props interface:**
|
||||
|
||||
```ts
|
||||
interface TopCountriesBarChartProps {
|
||||
countries: Record<string, number>;
|
||||
countryNames: Record<string, string>;
|
||||
}
|
||||
```
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- Renders up to 20 bars, sorted descending.
|
||||
- Country names readable on the Y-axis; tooltip provides full detail.
|
||||
- Responsive width, dynamic height.
|
||||
- `tsc --noEmit` passes. No `any`. ESLint clean.
|
||||
|
||||
---
|
||||
|
||||
### Task 3.2 — Integrate the bar chart into `DashboardPage`
|
||||
|
||||
**Status:** `done`
|
||||
|
||||
Add the `TopCountriesBarChart` to the dashboard alongside the pie chart.
|
||||
|
||||
**Layout:**
|
||||
|
||||
- The charts section now contains two cards side-by-side in a responsive grid row (the `chartsRow` class from Task 2.3):
|
||||
- Left: **Top Countries** pie chart (Task 2.1).
|
||||
- Right: **Top 20 Countries** bar chart (Task 3.1).
|
||||
- On narrow screens (< 768 px viewport width) the cards should stack vertically.
|
||||
- Both charts consume data from the **same** `useDashboardCountryData` hook call — do not fetch twice.
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- Both charts render side by side on wide screens, stacked on narrow screens.
|
||||
- A single API call feeds both charts.
|
||||
- Time range / origin filter controls affect both charts.
|
||||
- Loading / error states handled for both.
|
||||
- `tsc --noEmit` and ESLint pass.
|
||||
|
||||
---
|
||||
|
||||
## Stage 4 — Bans-Over-Time Trend Chart
|
||||
|
||||
### Task 4.1 — Add a backend endpoint for time-series ban aggregation
|
||||
|
||||
**Status:** `not started`
|
||||
|
||||
The existing endpoints return flat lists or country-aggregated counts but **no time-bucketed series**. A dashboard trend chart needs data grouped into time buckets.
|
||||
|
||||
Create a new endpoint: **`GET /api/dashboard/bans/trend`**.
|
||||
|
||||
**Query params:**
|
||||
|
||||
| Param | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `range` | `TimeRange` | `"24h"` | Time-range preset. |
|
||||
| `origin` | `BanOrigin \| null` | `null` | Optional filter by ban origin. |
|
||||
|
||||
**Response model** (`BanTrendResponse`):
|
||||
|
||||
```python
|
||||
class BanTrendBucket(BaseModel):
|
||||
timestamp: str # ISO 8601 UTC start of the bucket
|
||||
count: int # Number of bans in this bucket
|
||||
|
||||
class BanTrendResponse(BaseModel):
|
||||
buckets: list[BanTrendBucket]
|
||||
bucket_size: str # Human-readable label: "1h", "6h", "1d", "7d"
|
||||
```
|
||||
|
||||
**Bucket strategy:**
|
||||
|
||||
| Range | Bucket size | Example buckets |
|
||||
|---|---|---|
|
||||
| `24h` | 1 hour | 24 buckets |
|
||||
| `7d` | 6 hours | 28 buckets |
|
||||
| `30d` | 1 day | 30 buckets |
|
||||
| `365d` | 7 days | ~52 buckets |
|
||||
|
||||
**Implementation:**
|
||||
|
||||
- Add the Pydantic models to `backend/app/models/ban.py`.
|
||||
- Add the service function in `backend/app/services/ban_service.py`. Query the fail2ban database (`bans` table), group rows by the computed bucket. Use SQL `CAST((banned_at - ?) / bucket_seconds AS INTEGER)` style bucketing.
|
||||
- Add the route in `backend/app/routers/dashboard.py`.
|
||||
- Follow the existing layering: router → service → repository.
|
||||
- Write tests for the new endpoint in `backend/tests/test_routers/` and `backend/tests/test_services/`.
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- `GET /api/dashboard/bans/trend?range=24h` returns 24 hourly buckets.
|
||||
- Each bucket has a correct ISO 8601 timestamp and count.
|
||||
- Origin filter is applied correctly.
|
||||
- Empty buckets (zero bans) are included so the frontend has a continuous series.
|
||||
- Tests pass and cover happy path + empty data + origin filter.
|
||||
- `ruff check` and `mypy --strict` pass.
|
||||
|
||||
---
|
||||
|
||||
### Task 4.2 — Create the `BanTrendChart` component
|
||||
|
||||
**Status:** `not started`
|
||||
|
||||
Create `frontend/src/components/BanTrendChart.tsx`. This component renders an **area/line chart** showing the number of bans over time.
|
||||
|
||||
**Data source:** A new `useBanTrend` hook that calls `GET /api/dashboard/bans/trend`.
|
||||
|
||||
**Visual requirements:**
|
||||
|
||||
- Use `<AreaChart>` (or `<LineChart>`) from Recharts with `<Area>`, `<XAxis>`, `<YAxis>`, `<CartesianGrid>`, `<Tooltip>`.
|
||||
- X-axis: time labels formatted human-readably (e.g. "Mon 14:00", "Mar 5").
|
||||
- Y-axis: ban count.
|
||||
- Area fill with a semi-transparent version of the primary chart colour.
|
||||
- Tooltip shows exact timestamp + count.
|
||||
- Responsive via `<ResponsiveContainer>`.
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- Displays a continuous time-series line with the correct number of data points for each range.
|
||||
- Readable axis labels for all four time ranges.
|
||||
- Responsive.
|
||||
- `tsc --noEmit`, ESLint clean.
|
||||
|
||||
---
|
||||
|
||||
### Task 4.3 — Integrate the trend chart into `DashboardPage`
|
||||
|
||||
**Status:** `not started`
|
||||
|
||||
Add the `BanTrendChart` to the dashboard page **above** the two country charts and **below** the `ServerStatusBar`.
|
||||
|
||||
**Layout:**
|
||||
|
||||
- Full-width section card.
|
||||
- Section title: **"Ban Trend"**.
|
||||
- Shares the same `timeRange` and `originFilter` state.
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- The trend chart renders on the dashboard showing bans over time.
|
||||
- Responds to time-range and origin-filter changes.
|
||||
- Loading/error states handled.
|
||||
- `tsc --noEmit` and ESLint pass.
|
||||
|
||||
---
|
||||
|
||||
## Stage 5 — Jail Distribution Chart
|
||||
|
||||
### Task 5.1 — Add a backend endpoint for ban counts per jail
|
||||
|
||||
**Status:** `not started`
|
||||
|
||||
The existing `GET /api/jails` endpoint returns jail metadata with `status.currently_banned` — but this counts **currently active** bans, not historical bans in the selected time window. The dashboard needs historical ban counts per jail within the selected time range.
|
||||
|
||||
Create a new endpoint: **`GET /api/dashboard/bans/by-jail`**.
|
||||
|
||||
**Query params:**
|
||||
|
||||
| Param | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `range` | `TimeRange` | `"24h"` | Time-range preset. |
|
||||
| `origin` | `BanOrigin \| null` | `null` | Optional origin filter. |
|
||||
|
||||
**Response model** (`BansByJailResponse`):
|
||||
|
||||
```python
|
||||
class JailBanCount(BaseModel):
|
||||
jail: str
|
||||
count: int
|
||||
|
||||
class BansByJailResponse(BaseModel):
|
||||
jails: list[JailBanCount]
|
||||
total: int
|
||||
```
|
||||
|
||||
**Implementation:**
|
||||
|
||||
- Query the `bans` table: `SELECT jail, COUNT(*) FROM bans WHERE timestart >= ? GROUP BY jail ORDER BY COUNT(*) DESC`.
|
||||
- Apply origin filter by checking whether `jail == 'blocklist-import'`.
|
||||
- Add models, service function, route, and tests following existing patterns.
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- Returns jail names with ban counts descending, within the selected time window.
|
||||
- Origin filter works correctly.
|
||||
- Tests covering happy path, empty data, and filter.
|
||||
- `ruff check` and `mypy --strict` pass.
|
||||
|
||||
---
|
||||
|
||||
### Task 5.2 — Create the `JailDistributionChart` component
|
||||
|
||||
**Status:** `not started`
|
||||
|
||||
Create `frontend/src/components/JailDistributionChart.tsx`. This component renders a **horizontal bar chart** showing the distribution of bans across jails.
|
||||
|
||||
**Why this is useful and not covered by existing views:** The current Jails page shows configuration details and live counters per jail, but **does not** provide a visual comparison of which jails are catching the most threats within a selectable time window. An admin reviewing the dashboard benefits from an at-a-glance answer to: *"Which services are being attacked most frequently right now?"* — this is fundamentally different from the country-based charts (which answer *"where"*) and from the ban trend (which answers *"when"*). The jail distribution answers **"what service is targeted"** and helps prioritise hardening efforts.
|
||||
|
||||
**Data source:** A new `useJailDistribution` hook calling `GET /api/dashboard/bans/by-jail`.
|
||||
|
||||
**Visual requirements:**
|
||||
|
||||
- Horizontal `<BarChart>` from Recharts.
|
||||
- Y-axis: jail names.
|
||||
- X-axis: ban count.
|
||||
- Colour-coded bars from the chart theme.
|
||||
- Tooltip with jail name and exact count.
|
||||
- Responsive.
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- Renders one bar per jail, sorted descending.
|
||||
- Responsive.
|
||||
- `tsc --noEmit`, ESLint clean.
|
||||
|
||||
---
|
||||
|
||||
### Task 5.3 — Integrate the jail distribution chart into `DashboardPage`
|
||||
|
||||
**Status:** `not started`
|
||||
|
||||
Add the `JailDistributionChart` as a third chart card alongside the two country charts, or in a second chart row below them if space is constrained.
|
||||
|
||||
**Layout decision:**
|
||||
|
||||
- If three cards fit side-by-side at the standard breakpoint, place all three in one row.
|
||||
- Otherwise, use a 2-column + 1-column stacked layout (pie + bar on row 1, jail chart full-width on row 2). Choose whichever looks cleaner.
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- The jail distribution chart renders on the dashboard.
|
||||
- Shares time-range and origin-filter controls with the other charts.
|
||||
- Loading/error states handled.
|
||||
- Responsive layout.
|
||||
- `tsc --noEmit` and ESLint pass.
|
||||
|
||||
---
|
||||
|
||||
## Stage 6 — Polish and Final Review
|
||||
|
||||
### Task 6.1 — Ensure consistent loading, error, and empty states across all charts
|
||||
|
||||
**Status:** `not started`
|
||||
|
||||
Review all four chart components and ensure:
|
||||
|
||||
1. **Loading state**: Each shows a Fluent UI `<Spinner>` centred in its card while data is fetching.
|
||||
2. **Error state**: Each shows a Fluent UI `<MessageBar intent="error">` with a retry button.
|
||||
3. **Empty state**: When the data set has zero bans, each chart shows a friendly message (e.g. "No bans in this time range") instead of an empty or broken chart.
|
||||
|
||||
Extract a small shared wrapper if three or more charts duplicate the same loading/error/empty pattern (e.g. `ChartCard` or `ChartStateWrapper`).
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- All charts handle loading, error, and empty states consistently.
|
||||
- No broken or blank chart renders when data is unavailable.
|
||||
- `tsc --noEmit` and ESLint pass.
|
||||
|
||||
---
|
||||
|
||||
### Task 6.2 — Write frontend tests for chart components
|
||||
|
||||
**Status:** `not started`
|
||||
|
||||
Add tests for each chart component to confirm:
|
||||
|
||||
- Correct number of rendered slices/bars given known test data.
|
||||
- "Other" aggregation logic in the pie chart.
|
||||
- Top-N truncation in the bar chart.
|
||||
- Hook re-fetch on prop change.
|
||||
- Loading and error states render the expected UI.
|
||||
|
||||
Follow the project's existing frontend test setup and conventions.
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- Each chart component has at least one happy-path and one edge-case test.
|
||||
- Tests pass.
|
||||
- ESLint clean.
|
||||
|
||||
---
|
||||
|
||||
### Task 6.3 — Full build and lint check
|
||||
|
||||
**Status:** `not started`
|
||||
|
||||
Run the complete quality-assurance pipeline:
|
||||
|
||||
1. Backend: `ruff check`, `mypy --strict`, `pytest` with coverage.
|
||||
2. Frontend: `tsc --noEmit`, `eslint`, `npm run build`.
|
||||
3. Fix any warnings or errors introduced during stages 1–6.
|
||||
4. Verify overall test coverage remains ≥ 80 %.
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- Zero lint warnings/errors on both backend and frontend.
|
||||
- All tests pass.
|
||||
- Build artifacts generated successfully.
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user