lukas.pupkalipinski/BanGUI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Optimise geo lookup and aggregation for 10k+ IPs

Browse Source 
- Add persistent geo_cache SQLite table (db.py)
- Rewrite geo_service: batch API (100 IPs/call), two-tier cache,
  no caching of failed lookups so they are retried
- Pre-warm geo cache from DB on startup (main.py lifespan)
- Rewrite bans_by_country: SQL GROUP BY ip aggregation + lookup_batch
  instead of 2000-row fetch + asyncio.gather individual calls
- Pre-warm geo cache after blocklist import (blocklist_service)
- Add 300ms debounce to useMapData hook to cancel stale requests
- Add perf benchmark asserting <2s for 10k bans
- Add seed_10k_bans.py script for manual perf testing
This commit is contained in:
Lukas
2026-03-07 20:28:51 +01:00
parent 53d664de4f
commit ddfc8a0b02
13 changed files with 917 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
Docs/Tasks.md
View File

@@ -103,7 +103,18 @@ Add a filter dropdown (or segmented toggle) with three options — `All`, `Block
---
## Task 3 — Performance Optimisation for 10 k+ IPs (Dashboard & World Map)
## Task 3 — Performance Optimisation for 10 k+ IPs (Dashboard & World Map) ✅ DONE
**Completed:**
- Added persistent `geo_cache` SQLite table in `db.py`; loaded into in-memory cache at startup via `geo_service.load_cache_from_db()`.
- Rewrote `geo_service.py`: added `lookup_batch()` using `ip-api.com/batch` (100 IPs/call); failed lookups no longer cached so they are retried; only successful resolutions written to persistent store.
- Rewrote `bans_by_country()` in `ban_service.py`: SQL `GROUP BY ip` aggregation instead of loading 2 000 raw rows, batch geo-resolution via `lookup_batch()`, companion table limited to 200 rows (already geo-cached).
- Updated `dashboard.py` router `GET /bans/by-country` to pass `http_session` + `app_db` directly to `bans_by_country()`.
- Added geo cache pre-warm in `blocklist_service.import_source()`: after import, newly banned IPs are batch-resolved and persisted.
- Added debounce (300 ms) to `useMapData` hook to cancel stale in-flight requests when filters change rapidly; sets loading=true immediately for instant skeleton feedback.
- BanTable: page size capped at 100 per page with next/prev pagination — DOM perf not an issue, no virtualisation needed.
- Performance benchmark `tests/test_services/test_ban_service_perf.py`: seeds 10 000 bans in a temp DB, pre-warms geo cache, asserts `list_bans` and `bans_by_country` both complete in < 2 seconds.
- Seed script `tests/scripts/seed_10k_bans.py`: inserts 10 000 synthetic bans + pre-caches geo data for browser-level load-time verification.
### Problem