BanGUI/Docs/Tasks.md

# BanGUI — Task List

This document breaks the entire BanGUI project into development stages, ordered so that each stage builds on the previous one. Every task is described in prose with enough detail for a developer to begin work. References point to the relevant documentation.

---

## Task 1 — Make Geo-Cache Persistent ✅ DONE

**Goal:** Minimise calls to the external geo-IP lookup service by caching results in the database.

**Details:**

- Currently geo-IP results may only live in memory and are lost on restart. Persist every successful geo-lookup result into the database so the external service is called as rarely as possible.
- On each geo-lookup request, first query the database for a cached entry for that IP. Only call the external service if no cached entry exists (or the entry has expired, if a TTL policy is desired).
- After a successful external lookup, write the result back to the database immediately.
- Review the existing implementation in `app/services/geo_service.py` and the related repository/model code. Verify that:
  - The DB table/model for geo-cache entries exists and has the correct schema (IP, country, city, latitude, longitude, looked-up timestamp, etc.).
  - The repository layer exposes `get_by_ip` and `upsert` (or equivalent) methods.
  - The service checks the cache before calling the external API.
  - Bulk inserts are used where multiple IPs need to be resolved at once (see Task 3).

---

## Task 2 — Fix `geo_lookup_request_failed` Warnings ✅ DONE

**Goal:** Investigate and fix the frequent `geo_lookup_request_failed` log warnings that occur with an empty `error` field.

**Resolution:** The root cause was `str(exc)` returning `""` for aiohttp exceptions with no message (e.g. `ServerDisconnectedError`). Fixed by:
- Replacing `error=str(exc)` with `error=repr(exc)` in both `lookup()` and `_batch_api_call()` so the exception class name is always present in the log.
- Adding `exc_type=type(exc).__name__` field to every network-error log event for easy filtering.
- Moving `import aiohttp` from the `TYPE_CHECKING` block to a regular runtime import and replacing the raw-float `timeout` arguments with `aiohttp.ClientTimeout(total=...)`, removing the `# type: ignore[arg-type]` workarounds.
- Three new tests in `TestErrorLogging` verify empty-message exceptions are correctly captured.

**Observed behaviour (from container logs):**

```
{"ip": "197.221.98.153", "error": "", "event": "geo_lookup_request_failed", ...}
{"ip": "197.231.178.38", "error": "", "event": "geo_lookup_request_failed", ...}
{"ip": "197.234.201.154", "error": "", "event": "geo_lookup_request_failed", ...}
{"ip": "197.234.206.108", "error": "", "event": "geo_lookup_request_failed", ...}
```

**Details:**

- Open `app/services/geo_service.py` and trace the code path that emits the `geo_lookup_request_failed` event.
- The `error` field is empty, which suggests the request may silently fail (e.g. the external service returns a non-200 status, an empty body, or the response parsing swallows the real error).
- Ensure the actual HTTP status code and response body (or exception message) are captured and logged in the `error` field so failures are diagnosable.
- Check whether the external geo-IP service has rate-limiting or IP-range restrictions that could explain the failures.
- Add proper error handling: distinguish between transient errors (timeout, 429, 5xx) and permanent ones (invalid IP, 404) so retries can be applied only when appropriate.

---

## Task 3 — Non-Blocking Web Requests & Bulk DB Operations ✅ DONE

**Goal:** Ensure the web UI remains responsive while geo-IP lookups and database writes are in progress.

**Resolution:**
- **Bulk DB writes:** `geo_service.lookup_batch` now collects resolved IPs into `pos_rows` / `neg_ips` lists across the chunk loop and flushes them with two `executemany` calls per chunk instead of one `execute` per IP.
- **`lookup_cached_only`:** New function that returns `(geo_map, uncached)` immediately from the in-memory + SQLite cache with no API calls. Used by `bans_by_country` for its hot path.
- **Background geo resolution:** `bans_by_country` calls `lookup_cached_only` for an instant response, then fires `asyncio.create_task(geo_service.lookup_batch(uncached, …))` to populate the cache in the background for subsequent requests.
- **Batch enrichment for `get_active_bans`:** `jail_service.get_active_bans` now accepts `http_session` / `app_db` and resolves all banned IPs in a single `lookup_batch` call (chunked 100-IP batches) instead of firing one coroutine per IP through `asyncio.gather`.
- 12 new tests across `test_geo_service.py`, `test_jail_service.py`, and `test_ban_service.py`; `ruff` and `mypy --strict` clean; 145 tests pass.

**Details:**

- After the geo-IP service was integrated, web UI requests became slow or appeared to hang because geo lookups and individual DB writes block the async event loop.
- **Bulk DB operations:** When multiple IPs need geo data at once (e.g. loading the ban list), collect all uncached IPs and resolve them in a single batch. Use bulk `INSERT … ON CONFLICT` (or equivalent) to write results to the DB in one round-trip instead of one query per IP.
- **Non-blocking external calls:** Make sure all HTTP calls to the external geo-IP service use an async HTTP client (`httpx.AsyncClient` or similar) so the event loop is never blocked by network I/O.
- **Non-blocking DB access:** Ensure all database operations use the async SQLAlchemy session (or are off-loaded to a thread) so they do not block request handling.
- **Background processing:** Consider moving bulk geo-lookups into a background task (e.g. the existing task infrastructure in `app/tasks/`) so the API endpoint returns immediately and the UI is updated once results are ready.

---

## Task 4 — Better Jail Configuration ✅ DONE

**Goal:** Expose the full fail2ban configuration surface (jails, filters, actions) in the web UI.

Reference config directory: `/home/lukas/Volume/repo/BanGUI/Docker/fail2ban-dev-config/fail2ban/`

**Implementation summary:**

- **Backend:** New `app/models/file_config.py`, `app/services/file_config_service.py`, and `app/routers/file_config.py` with full CRUD for `jail.d/`, `filter.d/`, `action.d/` files. Path-traversal prevention via `_assert_within()` + `_validate_new_name()`. `app/config.py` extended with `fail2ban_config_dir` setting.
- **Backend (socket):** Added `delete_log_path()` to `config_service.py` + `DELETE /api/config/jails/{name}/logpath` endpoint.
- **Docker:** Both compose files updated with `BANGUI_FAIL2BAN_CONFIG_DIR` env var; volume mount changed `:ro` → `:rw`.
- **Frontend:** New `Jail Files`, `Filters`, `Actions` tabs in `ConfigPage.tsx`. Delete buttons for log paths in jail accordion. Full API call layer in `api/config.ts` + new types in `types/config.ts`.
- **Tests:** 44 service unit tests + 19 router integration tests; all pass; ruff clean.

**Task 4c audit findings — options not yet exposed in the UI:**
- Per-jail: `ignoreip`, `bantime.increment`, `bantime.rndtime`, `bantime.maxtime`, `bantime.factor`, `bantime.formula`, `bantime.multipliers`, `bantime.overalljails`, `ignorecommand`, `prefregex`, `timezone`, `journalmatch`, `usedns`, `backend` (read-only shown), `destemail`, `sender`, `action` override
- Global: `allowipv6`, `before` includes

### 4a — Activate / Deactivate Jail Configs ✅ DONE

- Listed all `.conf` and `.local` files in `jail.d/` via `GET /api/config/jail-files`.
- Toggle enabled/disabled via `PUT /api/config/jail-files/{filename}/enabled` which patches the `enabled = true/false` line in the config file, preserving all comments.
- Frontend: **Jail Files** tab with enabled `Switch` per file and read-only content viewer.

### 4b — Editable Log Paths ✅ DONE

- Added `DELETE /api/config/jails/{name}/logpath?log_path=…` endpoint (uses fail2ban socket `set <jail> dellogpath`).
- Frontend: each log path in the Jails accordion now has a dismiss button to remove it.

### 4c — Audit Missing Config Options ✅ DONE

- Audit findings documented above.

### 4d — Filter Configuration (`filter.d`) ✅ DONE

- Listed all filter files via `GET /api/config/filters`.
- View and edit individual filters via `GET/PUT /api/config/filters/{name}`.
- Create new filter via `POST /api/config/filters`.
- Frontend: **Filters** tab with accordion-per-file, editable textarea, save button, and create-new form.

### 4e — Action Configuration (`action.d`) ✅ DONE

- Listed all action files via `GET /api/config/actions`.
- View and edit individual actions via `GET/PUT /api/config/actions/{name}`.
- Create new action via `POST /api/config/actions`.
- Frontend: **Actions** tab with identical structure to Filters tab.

### 4f — Create New Configuration Files ✅ DONE

- Create filter and action files via `POST /api/config/filters` and `POST /api/config/actions` with name validation (`_SAFE_NAME_RE`) and 512 KB content size limit.
- Frontend: "New Filter/Action File" section at the bottom of each tab with name input, content textarea, and create button.

---

## Task 5 — Add Log Path to Jail (Config UI) ✅ DONE

**Goal:** Allow users to add new log file paths to an existing fail2ban jail directly from the Configuration → Jails tab, completing the "Add Log Observation" feature from [Features.md § 6.3](Features.md).

**Implementation summary:**

- `ConfigPage.tsx` `JailAccordionPanel`:
  - Added `addLogPath` and `AddLogPathRequest` imports.
  - Added state: `newLogPath`, `newLogPathTail` (default `true`), `addingLogPath`.
  - Added `handleAddLogPath` callback: calls `addLogPath(jail.name, { log_path, tail })`, appends path to `logPaths` state, clears input, shows success/error feedback.
  - Added inline "Add Log Path" form below the existing log-path list — an `Input` for the file path, a `Switch` for tail/head selection, and an "Add" button with `aria-label="Add log path"`.
- 6 new frontend tests in `src/components/__tests__/ConfigPageLogPath.test.tsx` covering: rendering, disabled state, enabled state, successful add, success message, and API error surfacing.
- `tsc --noEmit`, `eslint`: zero errors.