BanGUI/Docs/DATABASE_MIGRATIONS.md

# Database Migrations

BanGUI uses SQLite with a versioned migration system. Migrations are applied automatically on startup.

## Schema Version Table

The `schema_migrations` table tracks applied migrations:

```sql
CREATE TABLE IF NOT EXISTS schema_migrations (
    version     INTEGER PRIMARY KEY,
    migrated_at TEXT    NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
);
```

## How Migrations Work

On startup (`init_db()`):

1. Current schema version is read from `schema_migrations`
2. If version < latest, each missing migration is applied in order
3. Each migration runs inside a `BEGIN IMMEDIATE ... COMMIT` transaction
4. On failure, `ROLLBACK` restores database to pre-migration state

## Transactional Guarantees

Every migration is **atomic**. If any statement fails:

- All DDL changes are rolled back
- `schema_migrations` table is NOT updated
- Next startup re-applies the same migration from scratch

```python
try:
    await db.execute("BEGIN IMMEDIATE;")
    for statement in statements:
        await db.execute(statement)
    await db.execute("INSERT INTO schema_migrations (version) VALUES (?);", (version,))
    await db.commit()
except Exception:
    await db.rollback()
    raise
```

## Idempotency

Migrations use `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` where possible. Re-running a failed or partial migration is safe.

## WAL Mode and Crash Safety

BanGUI uses SQLite WAL mode (`PRAGMA journal_mode=WAL`). After a crash:

- SQLite auto-recovers using the WAL file
- `.wal` file may contain uncommitted changes that are rolled back
- Orphaned `.wal` files from previous crashes are detected and cleaned up on startup

### Detecting Orphaned WAL Files

On startup, if the database is in WAL mode but no WAL file exists:

```python
async def _cleanup_orphaned_wal_files(db: aiosqlite.Connection, db_path: Path) -> None:
    """Remove orphaned WAL files after crashes."""
    wal_path = Path(str(db_path) + "-wal")
    if wal_path.exists() and db_path.exists():
        # Check if WAL file is stale (database was opened since)
        pass  # SQLite handles this automatically
```

## Migration Failure Recovery

If a migration fails mid-way:

1. **Startup fails** — application refuses to start
2. **Rollback occurs** — database returns to pre-migration state
3. **Logs show error** — exception with full traceback

### Manual Recovery Steps

1. **Check current schema version:**
   ```bash
   sqlite3 bangui.db "SELECT MAX(version) FROM schema_migrations;"
   ```

2. **Check which tables exist:**
   ```bash
   sqlite3 bangui.db "SELECT name FROM sqlite_master WHERE type='table';"
   ```

3. **Manually apply the failed migration:**
   ```bash
   sqlite3 bangui.db "BEGIN IMMEDIATE;"
   # Run your migration SQL here
   sqlite3 bangui.db "INSERT INTO schema_migrations (version) VALUES (?);"
   sqlite3 bangui.db "COMMIT;"
   ```

4. **Or roll back to a known state:**
   ```bash
   sqlite3 bangui.db "DELETE FROM schema_migrations WHERE version > ?;"
   ```

### Complete Database Reset (Development Only)

If the database is unrecoverable:

```bash
rm bangui.db bangui.db-wal bangui.db-shm
# Restart application - schema will be recreated from migration 1
```

## Migration Version History

| Version | Description |
|---------|-------------|
| 1 | Initial schema (settings, sessions, blocklist_sources, import_log, geo_cache, history_archive) |
| 2 | Hash session tokens (DROP + recreate sessions) |
| 3 | Add last_seen to geo_cache |
| 4 | Add scheduler_lock table |
| 5 | Add indexes to history_archive |
| 6 | Add import_runs table for idempotent imports |
| 7 | Add indexes to import_log |
| 8 | Migrate import_log.timestamp TEXT→INTEGER UNIX |
| 9 | Change import_log.source_id FK to ON DELETE RESTRICT |

## Adding New Migrations

1. Increment `_CURRENT_SCHEMA_VERSION` in `backend/app/db.py`
2. Add migration script to `_MIGRATIONS` dict with new version key
3. Write migration as `CREATE IF NOT EXISTS` or `ALTER TABLE ADD COLUMN` to ensure idempotency
4. Test with `test_apply_migration_is_atomic_rollback` pattern
5. Update this document with migration description

## Long-Running Migrations

For migrations that modify large tables:

- Use `ALTER TABLE ADD COLUMN` (instant on SQLite)
- Avoid `CREATE INDEX CONCURRENTLY` (SQLite does not support this)
- For table rebuilds, split into phases with explicit progress tracking

## Disaster Recovery Checklist

If database is corrupted after migration failure:

- [ ] Stop all BanGUI instances
- [ ] Backup `bangui.db`, `bangui.db-wal`, `bangui.db-shm`
- [ ] Run `PRAGMA integrity_check;`
- [ ] Identify last successful migration version
- [ ] Delete `schema_migrations` rows for failed migrations
- [ ] Either: manually fix migration, or restore from backup
- [ ] Restart application