BanGUI/Docs/Tasks.md

# BanGUI — Task List

This document breaks the entire BanGUI project into development stages, ordered so that each stage builds on the previous one. Every task is described in prose with enough detail for a developer to begin work. References point to the relevant documentation.

---

## Issue: World Map Loading Time — Architecture Fix

### Problem Summary

The `GET /api/dashboard/bans/by-country` endpoint is extremely slow on first load. A single request with ~5,200 unique IPs produces **10,400 SQLite commits** and **6,000 INSERT statements** against the app database — all during a read-only GET request. The log shows 21,000+ lines of SQL trace for just 18 HTTP requests.

Root causes (ordered by impact):

1. **Per-IP commit during geo cache writes** — `geo_service._persist_entry()` and `_persist_neg_entry()` each call `await db.commit()` after every single INSERT. With 5,200 uncached IPs this means 5,200+ individual commits, each forcing an `fsync`. This is the dominant bottleneck.
2. **DB writes on a GET request** — The bans/by-country endpoint passes `app_db` to `geo_service.lookup_batch()`, which triggers INSERT+COMMIT for every resolved IP. A GET request should never produce database writes/commits. Users do not expect loading a map page to mutate the database.
3. **Same pattern exists in other endpoints** — The following GET endpoints also trigger geo cache commits: `/api/dashboard/bans`, `/api/bans/active`, `/api/history`, `/api/history/{ip}`, `/api/geo/lookup/{ip}`.

### Evidence from `log.log`

- Log line count: **21,117 lines** for 18 HTTP requests
- `INSERT INTO geo_cache`: **6,000** executions
- `db.commit()`: **10,400** calls (each INSERT + its commit = 2 ops per IP)
- `geo_batch_lookup_start`: reports `total=5200` uncached IPs
- The bans/by-country response is at line **21,086** out of 21,117 — the entire log is essentially one request's geo persist work
- Other requests (`/api/dashboard/status`, `/api/blocklists/schedule`, `/api/config/map-color-thresholds`) interleave with the geo persist loop because they share the same single async DB connection

---

### Task 1: Batch geo cache writes — eliminate per-IP commits ✅ DONE

**Summary:** Removed `await db.commit()` from `_persist_entry()` and `_persist_neg_entry()`. Added a single `await db.commit()` (wrapped in try/except) at the end of `lookup_batch()` after all chunk processing, and after each `_persist_entry` / `_persist_neg_entry` call in `lookup()`. Reduces commits from ~5,200 to **1** per batch request.

**File:** `backend/app/services/geo_service.py`

**What to change:**

The functions `_persist_entry()` and `_persist_neg_entry()` each call `await db.commit()` after every INSERT. Instead, the commit should happen once after the entire batch is processed.

1. Remove `await db.commit()` from both `_persist_entry()` and `_persist_neg_entry()`.
2. In `lookup_batch()`, after the loop over all chunks is complete and all `_persist_entry()` / `_persist_neg_entry()` calls have been made, issue a single `await db.commit()` if `db is not None`.
3. Wrap the single commit in a try/except to handle any errors gracefully.

**Expected impact:** Reduces commits from ~5,200 to **1** per request. This alone should cut the endpoint response time dramatically.

**Testing:** Existing tests in `test_services/test_ban_service.py` and `test_services/test_geo_service.py` should continue to pass. Verify the geo_cache table still gets populated after a batch lookup by checking the DB contents in an integration test.

---

### Task 2: Do not write geo cache during GET requests ✅ DONE

**Summary:** Removed `db` dependency injection from `GET /api/dashboard/bans` and `GET /api/dashboard/bans/by-country` in `dashboard.py`. Both now pass `app_db=None` to their respective service calls. The other GET endpoints (`/api/bans/active`, `/api/history`, `/api/history/{ip}`, `/api/geo/lookup/{ip}`) already did not pass `db` to geo lookups — confirmed correct.

**Files:** `backend/app/routers/dashboard.py`, `backend/app/routers/bans.py`, `backend/app/routers/history.py`, `backend/app/routers/geo.py`

**What to change:**

GET endpoints should not pass `app_db` (or equivalent) to geo_service functions. The geo resolution should still populate the in-memory cache (which is fast, free, and ephemeral), but should NOT write to SQLite during a read request.

For each of these GET endpoints:
- `GET /api/dashboard/bans/by-country` in `dashboard.py` — stop passing `app_db=db` to `bans_by_country()`; pass `app_db=None` instead.
- `GET /api/dashboard/bans` in `dashboard.py` — stop passing `app_db=db` to `list_bans()`; pass `app_db=None` instead.
- `GET /api/bans/active` in `bans.py` — the enricher callback should not pass `db` to `geo_service.lookup()`.
- `GET /api/history` and `GET /api/history/{ip}` in `history.py` — same: enricher should not pass `db`.
- `GET /api/geo/lookup/{ip}` in `geo.py` — do not pass `db` to `geo_service.lookup()`.

The persistent geo cache should only be written during explicit write operations:
- `POST /api/geo/re-resolve` (already a POST — this is correct)
- Blocklist import tasks (`blocklist_service.py`)
- Application startup via `load_cache_from_db()`

**Expected impact:** GET requests become truly read-only. No commits, no `fsync`, no write contention on the app DB during map loads.

**Testing:** Run the full test suite. Verify that:
1. The bans/by-country endpoint still returns correct country data (from in-memory cache).
2. The `geo_cache` table is still populated when `POST /api/geo/re-resolve` is called or after blocklist import.
3. After a server restart, geo data is still available (because `load_cache_from_db()` warms memory from previously persisted data).

---

### Task 3: Periodically persist the in-memory geo cache (background task) ✅ DONE

**Summary:** Added `_dirty: set[str]` to `geo_service.py`. `_store()` now adds IPs with a non-null `country_code` to `_dirty`; `clear_cache()` clears it. Added `flush_dirty(db)` which atomically snapshots/clears `_dirty`, batch-upserts all rows via `executemany()`, commits once, and re-adds entries on failure. Created `backend/app/tasks/geo_cache_flush.py` with a 60-second APScheduler job, registered in `main.py`.

**Files:** `backend/app/services/geo_service.py`, `backend/app/tasks/` (new task file)

**What to change:**

After Task 2, GET requests no longer write to the DB. But newly resolved IPs during GET requests only live in the in-memory cache and would be lost on restart. To avoid this, add a background task that periodically flushes new in-memory entries to the `geo_cache` table.

1. In `geo_service.py`, add a module-level `_dirty: set[str]` that tracks IPs added to `_cache` but not yet persisted. When `_store()` adds an entry, also add the IP to `_dirty`.
2. Add a new function `flush_dirty(db: aiosqlite.Connection) -> int` that:
   - Takes the current `_dirty` set and clears it atomically.
   - Uses `executemany()` to batch-INSERT all dirty entries in one transaction.
   - Calls `db.commit()` once.
   - Returns the count of flushed entries.
3. Register a periodic task (using the existing APScheduler setup) that calls `flush_dirty()` every 60 seconds (configurable). This is similar to how other background tasks already run.

**Expected impact:** Geo data is persisted without blocking any request. If the server restarts, at most 60 seconds of new geo data is lost (and it will simply be re-fetched from ip-api.com on the next request).

**Testing:** Write a test that:
- Calls `lookup_batch()` without a DB reference.
- Verifies IPs are in `_dirty`.
- Calls `flush_dirty(db)`.
- Verifies the geo_cache table contains the entries and `_dirty` is empty.

---

### Task 4: Reduce redundant SQL queries per request (settings / auth)

**Files:** `backend/app/dependencies.py`, `backend/app/main.py`, `backend/app/repositories/settings_repo.py`

**What to change:**

The log shows that every single HTTP request executes at least 2 separate SQL queries before reaching the actual endpoint logic:
1. `SELECT value FROM settings WHERE key = 'setup_completed'` (SetupRedirectMiddleware)
2. `SELECT id, token, ... FROM sessions WHERE token = ?` (require_auth dependency)

When multiple requests arrive concurrently (as seen in the log — 3 parallel requests trigger 3× setup_completed + 3× session token queries), this adds unnecessary DB contention.

Options (implement one or both):
- **Cache `setup_completed` in memory:** Once setup is complete, it never goes back to incomplete. Cache the result in `app.state` and skip the DB query on subsequent requests. Set it on first `True` read and clear it only if the app restarts.
- **Cache session validation briefly:** Use a short TTL in-memory cache (e.g., 5–10 seconds) for validated session tokens. This reduces repeated DB lookups for the same token across near-simultaneous requests.

**Expected impact:** Reduces per-request overhead from 2+ SQL queries to 0 for the common case (setup done, session recently validated).

**Testing:** Existing auth and setup tests must continue to pass. Add a test that validates the cached path (second request skips DB).

---

### Task 5: Audit and verify — run full test suite

After tasks 1–4 are implemented, run:

```bash
cd backend && python -m pytest tests/ -x -q
```

Verify:
- All tests pass (currently 443).
- `ruff check backend/app/` passes.
- `mypy --strict backend/app/` passes on changed files.
- Manual smoke test: load the world map page and verify it renders quickly with correct country data.