Files

Lukas e1a6491ac2 docs: add API reference and database schema docs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

2026-05-03 19:50:36 +02:00

11 KiB

Raw Blame History

Database Schema Documentation

BanGUI uses two SQLite databases:

Database	Purpose	Location
BanGUI app DB	Own configuration, sessions, blocklist sources, import logs, geo cache	`bangui.db`
fail2ban DB	fail2ban's internal ban/jail data (read-only)	Configured via `FAIL2BAN_DB` env var

1. BanGUI Application Schema

Single source of truth: backend/app/db.py.

1.1 `settings`

Key-value store for application configuration.

Column	Type	Constraints
`id`	INTEGER	PRIMARY KEY AUTOINCREMENT
`key`	TEXT	NOT NULL UNIQUE
`value`	TEXT	NOT NULL
`created_at`	TEXT	NOT NULL DEFAULT ISO 8601
`updated_at`	TEXT	NOT NULL DEFAULT ISO 8601

Indexes: PK only.

Purpose: Stores app-wide settings (e.g., timezone, UI preferences). All settings access goes through settings_repo / settings_service.

1.2 `sessions`

Session tokens for web authentication.

Column	Type	Constraints
`id`	INTEGER	PRIMARY KEY AUTOINCREMENT
`token_hash`	TEXT	NOT NULL UNIQUE
`created_at`	TEXT	NOT NULL DEFAULT ISO 8601
`expires_at`	TEXT	NOT NULL

Indexes: idx_sessions_token_hash (UNIQUE) on token_hash.

Purpose: Web session management. Tokens are SHA-256 hashed before storage. Sessions expire and are cleaned up by session_cleanup task. See auth_service.py.

1.3 `blocklist_sources`

Blocklist source definitions for the import pipeline.

Column	Type	Constraints
`id`	INTEGER	PRIMARY KEY AUTOINCREMENT
`name`	TEXT	NOT NULL
`url`	TEXT	NOT NULL UNIQUE
`enabled`	INTEGER	NOT NULL DEFAULT 1 (boolean)
`created_at`	TEXT	NOT NULL DEFAULT ISO 8601
`updated_at`	TEXT	NOT NULL DEFAULT ISO 8601

Indexes: PK only.

Purpose: Defines sources for blocklist imports. See blocklist_repo, blocklist_service, blocklist_import_workflow.

1.4 `import_log`

Audit log of individual blocklist import operations.

Column	Type	Constraints
`id`	INTEGER	PRIMARY KEY AUTOINCREMENT
`source_id`	INTEGER	REFERENCES `blocklist_sources(id)` ON DELETE RESTRICT
`source_url`	TEXT	NOT NULL
`timestamp`	INTEGER	NOT NULL (UNIX epoch)
`ips_imported`	INTEGER	NOT NULL DEFAULT 0
`ips_skipped`	INTEGER	NOT NULL DEFAULT 0
`errors`	TEXT

Indexes:

idx_import_log_id_desc on (id DESC) — cursor pagination
idx_import_log_source_id_desc on (source_id, id DESC) — filtered pagination

Purpose: Audit trail for imports. source_id RESTRICT prevents source deletion when logs exist. See migration 9.

Migration 8: timestamp migrated from TEXT ISO 8601 to INTEGER UNIX epoch.

1.5 `geo_cache`

Geo-IP lookup cache for ban IP metadata.

Column	Type	Constraints
`ip`	TEXT	PRIMARY KEY
`country_code`	TEXT
`country_name`	TEXT
`asn`	TEXT
`org`	TEXT
`cached_at`	TEXT	NOT NULL DEFAULT ISO 8601

Additional (migration 3):

Column	Type	Constraints
`last_seen`	TEXT	NOT NULL DEFAULT ISO 8601

Indexes: PK only.

Purpose: Caches GeoIP results to reduce third-party API calls. TTL managed by geo_cache_cleanup task. See geo_cache_repo, geo_service.

1.6 `history_archive`

Archived ban/unban history mirrored from fail2ban DB.

Column	Type	Constraints
`id`	INTEGER	PRIMARY KEY AUTOINCREMENT
`jail`	TEXT	NOT NULL
`ip`	TEXT	NOT NULL
`timeofban`	INTEGER	NOT NULL (UNIX epoch)
`bancount`	INTEGER	NOT NULL
`data`	TEXT	NOT NULL (JSON)
`action`	TEXT	NOT NULL CHECK IN ('ban', 'unban')
`created_at`	TEXT	NOT NULL DEFAULT ISO 8601

Constraints: UNIQUE(ip, jail, action, timeofban) prevents duplicate archive rows.

Indexes:

idx_history_archive_jail_timeofban on (jail, timeofban DESC) — dashboard filter by jail + time ordering
idx_history_archive_timeofban_jail_action on (timeofban DESC, jail, action) — timeline filters
idx_history_archive_ip on (ip) — IP prefix/exact searches
idx_history_archive_action on (action) — ban/unban filtering

Purpose: Long-term ban history. Synced from fail2ban DB by history_sync task. See history_archive_repo, history_service.

1.7 `scheduler_lock`

Database-backed mutex for multi-worker scheduler safety.

Column	Type	Constraints
`id`	INTEGER	PRIMARY KEY CHECK (id = 1) — singleton row
`pid`	INTEGER	NOT NULL
`hostname`	TEXT	NOT NULL
`created_at`	REAL	NOT NULL (UNIX epoch)
`heartbeat_at`	REAL	NOT NULL (UNIX epoch)

Indexes: PK only (singleton constraint).

Purpose: Only one worker process holds the scheduler lock at a time. Lock is heartbeat-renewed by scheduler_lock_heartbeat task. Uses BEGIN IMMEDIATE transaction to acquire atomically. See scheduler_lock.py.

1.8 `import_runs`

Tracks unique blocklist imports for idempotent retries.

Column	Type	Constraints
`id`	INTEGER	PRIMARY KEY AUTOINCREMENT
`source_id`	INTEGER	NOT NULL REFERENCES `blocklist_sources(id)` ON DELETE CASCADE
`content_hash`	TEXT	NOT NULL
`status`	TEXT	NOT NULL CHECK IN ('pending', 'completed', 'failed')
`imported_count`	INTEGER	NOT NULL DEFAULT 0
`skipped_count`	INTEGER	NOT NULL DEFAULT 0
`error_message`	TEXT
`created_at`	TEXT	NOT NULL DEFAULT ISO 8601
`updated_at`	TEXT	NOT NULL DEFAULT ISO 8601

Constraints: UNIQUE(source_id, content_hash) — same source + content = same import run.

Indexes: idx_import_runs_source_status on (source_id, status) — lookup completed imports by source.

Purpose: Prevents duplicate IP bans on import crash/retry. See migration 6 and blocklist_import_workflow.

1.9 `schema_migrations`

Tracks applied schema versions.

Column	Type	Constraints
`version`	INTEGER	PRIMARY KEY
`migrated_at`	TEXT	NOT NULL DEFAULT ISO 8601

Indexes: PK only.

Purpose: Idempotent schema migration tracker. Records each applied version number. See init_db() and _migrate_schema().

2. Fail2ban Database Schema

Read-only access via fail2ban_db_repo. Fail2ban manages this DB; BanGUI mirrors data into history_archive.

2.1 `fail2banDb`

Column	Type	Constraints
`version`	INTEGER

Single row tracking DB schema version.

2.2 `jails`

Column	Type	Constraints
`name`	TEXT	NOT NULL UNIQUE
`enabled`	INTEGER	NOT NULL DEFAULT 1

Indexes: jails_name on (name).

2.3 `logs`

Column	Type	Constraints
`jail`	TEXT	NOT NULL FK → `jails(name)` ON DELETE CASCADE
`path`	TEXT
`firstlinemd5`	TEXT
`lastfilepos`	INTEGER	DEFAULT 0
`UNIQUE(jail, path)`
`UNIQUE(jail, path, firstlinemd5)`

Indexes: logs_path on (path), logs_jail_path on (jail, path).

2.4 `bans`

Column	Type	Constraints
`jail`	TEXT	NOT NULL FK → `jails(name)`
`ip`	TEXT
`timeofban`	INTEGER	NOT NULL
`bantime`	INTEGER	NOT NULL
`bancount`	INTEGER	NOT NULL DEFAULT 1
`data`	JSON

Indexes:

bans_jail_timeofban_ip on (jail, timeofban)
bans_jail_ip on (jail, ip)
bans_ip on (ip)

2.5 `bips`

Backup IPs table (ban backup).

Column	Type	Constraints
`ip`	TEXT	NOT NULL
`jail`	TEXT	NOT NULL FK → `jails(name)`
`timeofban`	INTEGER	NOT NULL
`bantime`	INTEGER	NOT NULL
`bancount`	INTEGER	NOT NULL DEFAULT 1
`data`	JSON
PRIMARY KEY	`(ip, jail)`

Indexes: bips_timeofban on (timeofban), bips_ip on (ip).

3. Relationships and Constraints

blocklist_sources (1) ──(id)──→ import_log.source_id  [RESTRICT on delete]
                         └──→ import_runs.source_id  [CASCADE on delete]

settings: standalone (key-value, no FK)
sessions: standalone (token hash, no FK)
geo_cache: standalone (IP → geo data, no FK)
history_archive: standalone (archived ban history, no FK)
scheduler_lock: singleton row (id=1), no FK
schema_migrations: standalone (migration tracking, no FK)

Fail2ban tables are separate and read-only from BanGUI's perspective.

4. Indexes Summary

Table	Index	Columns
`sessions`	`idx_sessions_token_hash`	`token_hash` UNIQUE
`import_log`	`idx_import_log_id_desc`	`id DESC`
`import_log`	`idx_import_log_source_id_desc`	`source_id, id DESC`
`import_runs`	`idx_import_runs_source_status`	`source_id, status`
`history_archive`	`idx_history_archive_jail_timeofban`	`jail, timeofban DESC`
`history_archive`	`idx_history_archive_timeofban_jail_action`	`timeofban DESC, jail, action`
`history_archive`	`idx_history_archive_ip`	`ip`
`history_archive`	`idx_history_archive_action`	`action`
`jails`	`jails_name`	`name`
`logs`	`logs_path`	`path`
`logs`	`logs_jail_path`	`jail, path`
`bans`	`bans_jail_timeofban_ip`	`jail, timeofban`
`bans`	`bans_jail_ip`	`jail, ip`
`bans`	`bans_ip`	`ip`
`bips`	`bips_timeofban`	`timeofban`
`bips`	`bips_ip`	`ip`

5. Migration History

Version	Description
1	Initial schema: `settings`, `sessions`, `blocklist_sources`, `import_log`, `geo_cache`, `history_archive`, `schema_migrations`
2	Hash session tokens (`token_hash` column). Invalidates all existing sessions.
3	Add `last_seen` to `geo_cache` for retention policy.
4	Add `scheduler_lock` table for multi-worker scheduler mutex.
5	Add indexes to `history_archive` for query performance (4 indexes).
6	Add `import_runs` table for idempotent import tracking.
7	Add indexes to `import_log` for cursor-based pagination.
8	Migrate `import_log.timestamp` from TEXT ISO 8601 → INTEGER UNIX epoch.
9	Change `import_log.source_id` FK to `ON DELETE RESTRICT` (prevents orphaned logs). Recreate table with new FK semantics.

Current schema version: 9 (_CURRENT_SCHEMA_VERSION in db.py).

6. Performance Notes

WAL mode (PRAGMA journal_mode=WAL) — concurrent reads allowed, better write performance under concurrency.
Foreign keys enforced (PRAGMA foreign_keys=ON) — data integrity at DB level.
Busy timeout 5000 ms — prevents "database is locked" errors under contention.
history_archive indexes — tuned for dashboard filter + time ordering + pagination. See migration 5 and PERFORMANCE.md.
import_log indexes — tuned for cursor-based pagination (newest-first by id). See migration 7.
geo_cache PK on ip — O(1) lookup for geo enrichment on ban events.
scheduler_lock singleton (CHECK (id = 1)) — trivial lock existence check.

For detailed query patterns and benchmarks, see Docs/PERFORMANCE.md.

11 KiB Raw Blame History

Database Schema Documentation

1. BanGUI Application Schema

1.1 settings

1.2 sessions

1.3 blocklist_sources

1.4 import_log

1.5 geo_cache

1.6 history_archive

1.7 scheduler_lock

1.8 import_runs

1.9 schema_migrations

2. Fail2ban Database Schema

2.1 fail2banDb

2.2 jails

2.3 logs

2.4 bans

2.5 bips

3. Relationships and Constraints

4. Indexes Summary

5. Migration History

6. Performance Notes

11 KiB

Raw Blame History

1.1 `settings`

1.2 `sessions`

1.3 `blocklist_sources`

1.4 `import_log`

1.5 `geo_cache`

1.6 `history_archive`

1.7 `scheduler_lock`

1.8 `import_runs`

1.9 `schema_migrations`

2.1 `fail2banDb`

2.2 `jails`

2.3 `logs`

2.4 `bans`

2.5 `bips`