BanGUI/Docs/DATABASE_MIGRATIONS.md

Database Migrations

BanGUI uses SQLite with a versioned migration system. Migrations are applied automatically on startup.

Schema Version Table

The schema_migrations table tracks applied migrations:

CREATE TABLE IF NOT EXISTS schema_migrations (
    version     INTEGER PRIMARY KEY,
    migrated_at TEXT    NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
);

How Migrations Work

On startup (init_db()):

1. Current schema version is read from schema_migrations
2. If version < latest, each missing migration is applied in order
3. Each migration runs inside a BEGIN IMMEDIATE ... COMMIT transaction
4. On failure, ROLLBACK restores database to pre-migration state

Transactional Guarantees

Every migration is atomic. If any statement fails:

• All DDL changes are rolled back
• schema_migrations table is NOT updated
• Next startup re-applies the same migration from scratch

try:
    await db.execute("BEGIN IMMEDIATE;")
    for statement in statements:
        await db.execute(statement)
    await db.execute("INSERT INTO schema_migrations (version) VALUES (?);", (version,))
    await db.commit()
except Exception:
    await db.rollback()
    raise

Idempotency

Migrations use CREATE TABLE IF NOT EXISTS and CREATE INDEX IF NOT EXISTS where possible. Re-running a failed or partial migration is safe.

WAL Mode and Crash Safety

BanGUI uses SQLite WAL mode (PRAGMA journal_mode=WAL). After a crash:

• SQLite auto-recovers using the WAL file
• .wal file may contain uncommitted changes that are rolled back
• Orphaned .wal files from previous crashes are detected and cleaned up on startup

Detecting Orphaned WAL Files

On startup, if the database is in WAL mode but no WAL file exists:

async def _cleanup_orphaned_wal_files(db: aiosqlite.Connection, db_path: Path) -> None:
    """Remove orphaned WAL files after crashes."""
    wal_path = Path(str(db_path) + "-wal")
    if wal_path.exists() and db_path.exists():
        # Check if WAL file is stale (database was opened since)
        pass  # SQLite handles this automatically

Migration Failure Recovery

If a migration fails mid-way:

1. Startup fails — application refuses to start
2. Rollback occurs — database returns to pre-migration state
3. Logs show error — exception with full traceback

Manual Recovery Steps

1. Check current schema version:

   sqlite3 bangui.db "SELECT MAX(version) FROM schema_migrations;"
   

2. Check which tables exist:

   sqlite3 bangui.db "SELECT name FROM sqlite_master WHERE type='table';"
   

3. Manually apply the failed migration:

   sqlite3 bangui.db "BEGIN IMMEDIATE;"
   # Run your migration SQL here
   sqlite3 bangui.db "INSERT INTO schema_migrations (version) VALUES (?);"
   sqlite3 bangui.db "COMMIT;"
   

4. Or roll back to a known state:

   sqlite3 bangui.db "DELETE FROM schema_migrations WHERE version > ?;"
   

Complete Database Reset (Development Only)

If the database is unrecoverable:

rm bangui.db bangui.db-wal bangui.db-shm
# Restart application - schema will be recreated from migration 1

Migration Version History

Version	Description
1	Initial schema (settings, sessions, blocklist_sources, import_log, geo_cache, history_archive)
2	Hash session tokens (DROP + recreate sessions)
3	Add last_seen to geo_cache
4	Add scheduler_lock table
5	Add indexes to history_archive
6	Add import_runs table for idempotent imports
7	Add indexes to import_log
8	Migrate import_log.timestamp TEXT→INTEGER UNIX
9	Change import_log.source_id FK to ON DELETE RESTRICT

Adding New Migrations

1. Increment _CURRENT_SCHEMA_VERSION in backend/app/db.py
2. Add migration script to _MIGRATIONS dict with new version key
3. Write migration as CREATE IF NOT EXISTS or ALTER TABLE ADD COLUMN to ensure idempotency
4. Test with test_apply_migration_is_atomic_rollback pattern
5. Update this document with migration description

Long-Running Migrations

For migrations that modify large tables:

• Use ALTER TABLE ADD COLUMN (instant on SQLite)
• Avoid CREATE INDEX CONCURRENTLY (SQLite does not support this)
• For table rebuilds, split into phases with explicit progress tracking

Disaster Recovery Checklist

If database is corrupted after migration failure:

• Stop all BanGUI instances
• Backup bangui.db, bangui.db-wal, bangui.db-shm
• Run PRAGMA integrity_check;
• Identify last successful migration version
• Delete schema_migrations rows for failed migrations
• Either: manually fix migration, or restore from backup
• Restart application