BanGUI/Docs/Tasks.md

BanGUI — Task List

This document breaks the entire BanGUI project into development stages, ordered so that each stage builds on the previous one. Every task is described in prose with enough detail for a developer to begin work. References point to the relevant documentation.

---

Task 1 — Make Geo-Cache Persistent ✅ DONE

Goal: Minimise calls to the external geo-IP lookup service by caching results in the database.

Details:

• Currently geo-IP results may only live in memory and are lost on restart. Persist every successful geo-lookup result into the database so the external service is called as rarely as possible.
• On each geo-lookup request, first query the database for a cached entry for that IP. Only call the external service if no cached entry exists (or the entry has expired, if a TTL policy is desired).
• After a successful external lookup, write the result back to the database immediately.
• Review the existing implementation in app/services/geo_service.py and the related repository/model code. Verify that:
  • The DB table/model for geo-cache entries exists and has the correct schema (IP, country, city, latitude, longitude, looked-up timestamp, etc.).
  • The repository layer exposes get_by_ip and upsert (or equivalent) methods.
  • The service checks the cache before calling the external API.
  • Bulk inserts are used where multiple IPs need to be resolved at once (see Task 3).

---

Task 2 — Fix geo_lookup_request_failed Warnings ✅ DONE

Goal: Investigate and fix the frequent geo_lookup_request_failed log warnings that occur with an empty error field.

Resolution: The root cause was str(exc) returning "" for aiohttp exceptions with no message (e.g. ServerDisconnectedError). Fixed by:

• Replacing error=str(exc) with error=repr(exc) in both lookup() and _batch_api_call() so the exception class name is always present in the log.
• Adding exc_type=type(exc).__name__ field to every network-error log event for easy filtering.
• Moving import aiohttp from the TYPE_CHECKING block to a regular runtime import and replacing the raw-float timeout arguments with aiohttp.ClientTimeout(total=...), removing the # type: ignore[arg-type] workarounds.
• Three new tests in TestErrorLogging verify empty-message exceptions are correctly captured.

Observed behaviour (from container logs):

{"ip": "197.221.98.153", "error": "", "event": "geo_lookup_request_failed", ...}
{"ip": "197.231.178.38", "error": "", "event": "geo_lookup_request_failed", ...}
{"ip": "197.234.201.154", "error": "", "event": "geo_lookup_request_failed", ...}
{"ip": "197.234.206.108", "error": "", "event": "geo_lookup_request_failed", ...}

Details:

• Open app/services/geo_service.py and trace the code path that emits the geo_lookup_request_failed event.
• The error field is empty, which suggests the request may silently fail (e.g. the external service returns a non-200 status, an empty body, or the response parsing swallows the real error).
• Ensure the actual HTTP status code and response body (or exception message) are captured and logged in the error field so failures are diagnosable.
• Check whether the external geo-IP service has rate-limiting or IP-range restrictions that could explain the failures.
• Add proper error handling: distinguish between transient errors (timeout, 429, 5xx) and permanent ones (invalid IP, 404) so retries can be applied only when appropriate.

---

Task 3 — Non-Blocking Web Requests & Bulk DB Operations ✅ DONE

Goal: Ensure the web UI remains responsive while geo-IP lookups and database writes are in progress.

Resolution:

• Bulk DB writes: geo_service.lookup_batch now collects resolved IPs into pos_rows / neg_ips lists across the chunk loop and flushes them with two executemany calls per chunk instead of one execute per IP.
• lookup_cached_only: New function that returns (geo_map, uncached) immediately from the in-memory + SQLite cache with no API calls. Used by bans_by_country for its hot path.
• Background geo resolution: bans_by_country calls lookup_cached_only for an instant response, then fires asyncio.create_task(geo_service.lookup_batch(uncached, …)) to populate the cache in the background for subsequent requests.
• Batch enrichment for get_active_bans: jail_service.get_active_bans now accepts http_session / app_db and resolves all banned IPs in a single lookup_batch call (chunked 100-IP batches) instead of firing one coroutine per IP through asyncio.gather.
• 12 new tests across test_geo_service.py, test_jail_service.py, and test_ban_service.py; ruff and mypy --strict clean; 145 tests pass.

Details:

• After the geo-IP service was integrated, web UI requests became slow or appeared to hang because geo lookups and individual DB writes block the async event loop.
• Bulk DB operations: When multiple IPs need geo data at once (e.g. loading the ban list), collect all uncached IPs and resolve them in a single batch. Use bulk INSERT … ON CONFLICT (or equivalent) to write results to the DB in one round-trip instead of one query per IP.
• Non-blocking external calls: Make sure all HTTP calls to the external geo-IP service use an async HTTP client (httpx.AsyncClient or similar) so the event loop is never blocked by network I/O.
• Non-blocking DB access: Ensure all database operations use the async SQLAlchemy session (or are off-loaded to a thread) so they do not block request handling.
• Background processing: Consider moving bulk geo-lookups into a background task (e.g. the existing task infrastructure in app/tasks/) so the API endpoint returns immediately and the UI is updated once results are ready.

---

Task 4 — Better Jail Configuration

Goal: Expose the full fail2ban configuration surface (jails, filters, actions) in the web UI.

Reference config directory: /home/lukas/Volume/repo/BanGUI/Docker/fail2ban-dev-config/fail2ban/

4a — Activate / Deactivate Jail Configs

• List all .conf and .local files in the jail config folder.
• Allow the user to toggle inactive jails to active (and vice-versa) from the UI.

4b — Editable Log Paths

• Each jail has a logpath setting. Expose this in the UI as an editable text field so the user can point a jail at a different log file without SSH access.

4c — Audit Missing Config Options

• Open every jail .conf/.local file in the dev-config directory and compare the available options with what the web UI currently exposes.
• Write down all options that are not yet implemented in the UI (e.g. findtime, bantime.increment, ignoreip, ignorecommand, maxretry, backend, usedns, destemail, sender, action, etc.).
• Document the findings in this task or a separate reference file so they can be implemented incrementally.

4d — Filter Configuration (filter.d)

• List all filter files in filter.d/.
• Allow the user to activate, deactivate, view, and edit filter definitions from the UI.
• Provide an option to create a brand-new filter file.

4e — Action Configuration (action.d)

• List all action files in action.d/.
• Allow the user to activate, deactivate, view, and edit action definitions from the UI.
• Provide an option to create a brand-new action file.

4f — Create New Configuration Files

• Add a UI flow to create a new jail, filter, or action configuration file from scratch (or from a template).
• Validate the new file before writing it to the config directory.