111 lines
7.4 KiB
Markdown
111 lines
7.4 KiB
Markdown
# BanGUI — Task List
|
|
|
|
This document breaks the entire BanGUI project into development stages, ordered so that each stage builds on the previous one. Every task is described in prose with enough detail for a developer to begin work. References point to the relevant documentation.
|
|
|
|
---
|
|
|
|
## Task 1 — Make Geo-Cache Persistent ✅ DONE
|
|
|
|
**Goal:** Minimise calls to the external geo-IP lookup service by caching results in the database.
|
|
|
|
**Details:**
|
|
|
|
- Currently geo-IP results may only live in memory and are lost on restart. Persist every successful geo-lookup result into the database so the external service is called as rarely as possible.
|
|
- On each geo-lookup request, first query the database for a cached entry for that IP. Only call the external service if no cached entry exists (or the entry has expired, if a TTL policy is desired).
|
|
- After a successful external lookup, write the result back to the database immediately.
|
|
- Review the existing implementation in `app/services/geo_service.py` and the related repository/model code. Verify that:
|
|
- The DB table/model for geo-cache entries exists and has the correct schema (IP, country, city, latitude, longitude, looked-up timestamp, etc.).
|
|
- The repository layer exposes `get_by_ip` and `upsert` (or equivalent) methods.
|
|
- The service checks the cache before calling the external API.
|
|
- Bulk inserts are used where multiple IPs need to be resolved at once (see Task 3).
|
|
|
|
---
|
|
|
|
## Task 2 — Fix `geo_lookup_request_failed` Warnings ✅ DONE
|
|
|
|
**Goal:** Investigate and fix the frequent `geo_lookup_request_failed` log warnings that occur with an empty `error` field.
|
|
|
|
**Resolution:** The root cause was `str(exc)` returning `""` for aiohttp exceptions with no message (e.g. `ServerDisconnectedError`). Fixed by:
|
|
- Replacing `error=str(exc)` with `error=repr(exc)` in both `lookup()` and `_batch_api_call()` so the exception class name is always present in the log.
|
|
- Adding `exc_type=type(exc).__name__` field to every network-error log event for easy filtering.
|
|
- Moving `import aiohttp` from the `TYPE_CHECKING` block to a regular runtime import and replacing the raw-float `timeout` arguments with `aiohttp.ClientTimeout(total=...)`, removing the `# type: ignore[arg-type]` workarounds.
|
|
- Three new tests in `TestErrorLogging` verify empty-message exceptions are correctly captured.
|
|
|
|
**Observed behaviour (from container logs):**
|
|
|
|
```
|
|
{"ip": "197.221.98.153", "error": "", "event": "geo_lookup_request_failed", ...}
|
|
{"ip": "197.231.178.38", "error": "", "event": "geo_lookup_request_failed", ...}
|
|
{"ip": "197.234.201.154", "error": "", "event": "geo_lookup_request_failed", ...}
|
|
{"ip": "197.234.206.108", "error": "", "event": "geo_lookup_request_failed", ...}
|
|
```
|
|
|
|
**Details:**
|
|
|
|
- Open `app/services/geo_service.py` and trace the code path that emits the `geo_lookup_request_failed` event.
|
|
- The `error` field is empty, which suggests the request may silently fail (e.g. the external service returns a non-200 status, an empty body, or the response parsing swallows the real error).
|
|
- Ensure the actual HTTP status code and response body (or exception message) are captured and logged in the `error` field so failures are diagnosable.
|
|
- Check whether the external geo-IP service has rate-limiting or IP-range restrictions that could explain the failures.
|
|
- Add proper error handling: distinguish between transient errors (timeout, 429, 5xx) and permanent ones (invalid IP, 404) so retries can be applied only when appropriate.
|
|
|
|
---
|
|
|
|
## Task 3 — Non-Blocking Web Requests & Bulk DB Operations ✅ DONE
|
|
|
|
**Goal:** Ensure the web UI remains responsive while geo-IP lookups and database writes are in progress.
|
|
|
|
**Resolution:**
|
|
- **Bulk DB writes:** `geo_service.lookup_batch` now collects resolved IPs into `pos_rows` / `neg_ips` lists across the chunk loop and flushes them with two `executemany` calls per chunk instead of one `execute` per IP.
|
|
- **`lookup_cached_only`:** New function that returns `(geo_map, uncached)` immediately from the in-memory + SQLite cache with no API calls. Used by `bans_by_country` for its hot path.
|
|
- **Background geo resolution:** `bans_by_country` calls `lookup_cached_only` for an instant response, then fires `asyncio.create_task(geo_service.lookup_batch(uncached, …))` to populate the cache in the background for subsequent requests.
|
|
- **Batch enrichment for `get_active_bans`:** `jail_service.get_active_bans` now accepts `http_session` / `app_db` and resolves all banned IPs in a single `lookup_batch` call (chunked 100-IP batches) instead of firing one coroutine per IP through `asyncio.gather`.
|
|
- 12 new tests across `test_geo_service.py`, `test_jail_service.py`, and `test_ban_service.py`; `ruff` and `mypy --strict` clean; 145 tests pass.
|
|
|
|
**Details:**
|
|
|
|
- After the geo-IP service was integrated, web UI requests became slow or appeared to hang because geo lookups and individual DB writes block the async event loop.
|
|
- **Bulk DB operations:** When multiple IPs need geo data at once (e.g. loading the ban list), collect all uncached IPs and resolve them in a single batch. Use bulk `INSERT … ON CONFLICT` (or equivalent) to write results to the DB in one round-trip instead of one query per IP.
|
|
- **Non-blocking external calls:** Make sure all HTTP calls to the external geo-IP service use an async HTTP client (`httpx.AsyncClient` or similar) so the event loop is never blocked by network I/O.
|
|
- **Non-blocking DB access:** Ensure all database operations use the async SQLAlchemy session (or are off-loaded to a thread) so they do not block request handling.
|
|
- **Background processing:** Consider moving bulk geo-lookups into a background task (e.g. the existing task infrastructure in `app/tasks/`) so the API endpoint returns immediately and the UI is updated once results are ready.
|
|
|
|
---
|
|
|
|
## Task 4 — Better Jail Configuration
|
|
|
|
**Goal:** Expose the full fail2ban configuration surface (jails, filters, actions) in the web UI.
|
|
|
|
Reference config directory: `/home/lukas/Volume/repo/BanGUI/Docker/fail2ban-dev-config/fail2ban/`
|
|
|
|
### 4a — Activate / Deactivate Jail Configs
|
|
|
|
- List all `.conf` and `.local` files in the jail config folder.
|
|
- Allow the user to toggle inactive jails to active (and vice-versa) from the UI.
|
|
|
|
### 4b — Editable Log Paths
|
|
|
|
- Each jail has a `logpath` setting. Expose this in the UI as an editable text field so the user can point a jail at a different log file without SSH access.
|
|
|
|
### 4c — Audit Missing Config Options
|
|
|
|
- Open every jail `.conf`/`.local` file in the dev-config directory and compare the available options with what the web UI currently exposes.
|
|
- Write down all options that are **not yet implemented** in the UI (e.g. `findtime`, `bantime.increment`, `ignoreip`, `ignorecommand`, `maxretry`, `backend`, `usedns`, `destemail`, `sender`, `action`, etc.).
|
|
- Document the findings in this task or a separate reference file so they can be implemented incrementally.
|
|
|
|
### 4d — Filter Configuration (`filter.d`)
|
|
|
|
- List all filter files in `filter.d/`.
|
|
- Allow the user to activate, deactivate, view, and edit filter definitions from the UI.
|
|
- Provide an option to create a brand-new filter file.
|
|
|
|
### 4e — Action Configuration (`action.d`)
|
|
|
|
- List all action files in `action.d/`.
|
|
- Allow the user to activate, deactivate, view, and edit action definitions from the UI.
|
|
- Provide an option to create a brand-new action file.
|
|
|
|
### 4f — Create New Configuration Files
|
|
|
|
- Add a UI flow to create a new jail, filter, or action configuration file from scratch (or from a template).
|
|
- Validate the new file before writing it to the config directory.
|