Move X-BanGUI-Request header name/value to backend/app/utils/constants.py as single source of truth. Add GET /api/v1/config/security-headers endpoint. Update csrf middleware, frontend api client, and docs to use shared constants.
- Add _acquire_readonly_connection() that applies PRAGMA query_only=ON after connect
- Verify PRAGMA value back to catch URI flag bypasses
- Wrap in async context manager _readonly_connection() used by all repo methods
- Replace hardcoded '_SESSION_COOKIE_NAME' in CSRF middleware with import from
app.utils.constants
- Remove completed Issues #45 and #46 from Docs/Tasks.md (Issue #46 now fixed,
#45 cache invalidation deferred to auth refactor branch)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Stale sessions from a stolen device could be reused up to the cache
TTL after a legitimate user re-logs in, because login never cleared
the existing cache entry.
Changes:
- Add invalidate_by_user(user_id) to SessionCache protocol
- InMemorySessionCache maintains a user_id -> set[token] index to
support O(1) invalidation of all sessions for a given user
- NoOpSessionCache stub updated for API compatibility
- auth_service.login() now returns the Session object alongside
signed_token and expires_at
- login router calls session_cache.invalidate_by_user(session.id)
immediately after successful authentication
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fail with RuntimeError when WEB_CONCURRENCY or BANGUI_WORKERS > 1.
In-memory session cache, rate-limit windows, and runtime state are
process-local. Multi-worker silently causes stale limits, ghost sessions,
inconsistent status.
Skipped when TESTING=1.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add GitHub Actions CI pipeline with pytest, ruff, mypy
- Expand Tasks.md with implementation tracking and testing criteria
- Update CONTRIBUTING.md with CI requirements
- Add Testing-Requirements.md with coverage thresholds and PR checks
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Extract jail status/processing to helper functions
- Add error_handling.py service for centralized error handling
- Update config.py with validation and defaults
- Update .env.example with all config options
- Remove obsolete Tasks.md, add Service-Development.md
- Minor fixes across routers and services
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace contextlib.suppress with try/except + warning log
- Add test for fail2ban client
- Remove stale Issue #21 from Tasks.md (indexes)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- New get_fail2ban_db_path() in setup_service resolves DB path from configured socket path
- New ensure_fail2ban_indexes() creates missing performance indexes on bans table
- Call ensure_fail2ban_indexes on every startup before first ban query
- Remove completed tasks from Docs/Tasks.md
- Update Docs/PERFORMANCE.md with index findings
- Move config loading to dedicated ConfigLoader class with validation
- Add DATABASE_MIGRATIONS.md content to TROUBLESHOOTING.md
- Add API_STATUS_CODES.md documenting all API response codes
- Update runner.csx to use new config structure
- Add check_responses.py validation script
- Update config tests for new structure
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add explicit HTTP status code documentation to every endpoint
across 15 router files. Each endpoint now declares all possible
response codes (200/201/204/400/401/404/409/429/502/503) with
descriptions so frontend can distinguish error types.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Establishes shared type conventions to prevent runtime type mismatches
between TS frontend and Python backend. Covers snake_case JSON field
names, null vs empty string handling, timestamp formats, and validation
patterns for country codes, bans, and jail configuration.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Detect catastrophic backtracking patterns before regex compilation
using regexploit library. Add ReDoSDetectedError exception and
_MINIMUM_STARRINESS threshold (>=3) to catch dangerous patterns
like (a+)+b. Update pyproject.toml deps, add tests for detection.
Replace check-then-insert race condition with INSERT ON CONFLICT.
- upsert_pending uses RETURNING id for atomic upsert
- UNIQUE(source_id, content_hash) constraint from migration 6
- blocklist_import_workflow updated to use upsert_pending
- test_import_source_success fixed for async mock patterns
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Enhance response model with additional fields and validation
- Update health and server router implementations
- Improve frontend type definitions and API integration
- Clean up documentation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- All backend routers moved to /api/v1/ prefix
- Frontend BASE_URL updated to /api/v1
- Setup redirect middleware updated to redirect to /api/v1/setup
- Health router path fixed: prefix=/api/v1/health, @router.get('')
- conftest.py: set server_status=online for test fixture
- Created Docs/API_VERSIONING.md with deprecation policy
- Updated Docs/Backend-Development.md with versioning section
- Updated Instructions.md curl examples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add optional requestKey parameter to UseFetchDataOptions
- Implement module-level cache (inFlightRequests) to track in-flight requests
- When requestKey is provided, multiple hook instances with same key share in-flight requests
- Prevents duplicate API calls when multiple components fetch same data or rapid refresh calls
- Cache entries are automatically cleared when response arrives (success or error)
- Maintains backward compatibility: without requestKey, behaves as before
- Adds comprehensive tests for deduplication scenarios
This reduces bandwidth waste and prevents race conditions caused by concurrent requests for identical data.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add custom comparison function to React.memo for TopCountriesPieChart
- Add custom comparison function to React.memo for TopCountriesBarChart
- Use JSON.stringify for deep equality comparison of countries and countryNames
- Prevents unnecessary re-renders when parent updates with same data
- Avoids Recharts reprocessing 5000+ data points on each parent re-render
All tests passing. No linting issues.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit adds support for shipping logs to external centralized logging platforms, addressing the MEDIUM priority task for structured logging infrastructure.
## Key Changes:
### 1. New Documentation: Docs/Observability.md
- Comprehensive guide to logging architecture and configuration
- Covers all three supported platforms (Datadog, Papertrail, Elasticsearch)
- Includes best practices, security considerations, and troubleshooting
- Documents sensitive data handling and compliance requirements
### 2. Core Implementation: app/utils/external_logging.py
- ExternalLogHandler: Abstract base class for non-blocking log delivery
- DatadogLogHandler: HTTP API integration with JSON payloads
- PapertrailLogHandler: Syslog protocol over TCP
- ElasticsearchLogHandler: Bulk API integration with NDJSON format
- Features:
- Async buffering with configurable batch size and flush interval
- Exponential backoff retry logic
- Non-blocking delivery (never blocks application logic)
- Proper error handling and internal logging
- Lifecycle management (start/shutdown)
### 3. Configuration: app/config.py
- New Settings fields for external logging:
- external_logging_enabled (default: False)
- external_logging_provider (datadog/papertrail/elasticsearch)
- external_logging_buffer_size (default: 1000)
- external_logging_flush_interval_seconds (default: 5.0)
- Provider-specific configuration (API keys, hosts, batch sizes)
- All fields have sensible defaults
- Full field validation and normalization
### 4. Integration: app/main.py
- Global _external_log_handler for application lifecycle
- _external_logging_processor: structlog processor for handler integration
- Updated _configure_logging(): Add handler to processor chain when enabled
- Updated _lifespan(): Initialize handler before startup, shutdown on termination
### 5. Tests: backend/tests/test_external_logging.py
- 20 comprehensive tests covering all handlers and factory
- Configuration validation tests
- All tests passing
## Design Decisions:
1. **Non-blocking Delivery**: External logging never blocks request handling.
Failures are logged locally but don't impact application.
2. **Buffering Strategy**: In-memory buffer with configurable size prevents
unbounded memory growth. When buffer fills, oldest logs are dropped with
a warning.
3. **Retry Logic**: Transient failures (timeouts, 5xx errors) are retried
with exponential backoff. Permanent failures (bad credentials) are logged
and skipped.
4. **Disabled by Default**: External logging is opt-in via environment
variables, maintaining backward compatibility with existing deployments.
5. **Provider Flexibility**: Support for multiple platforms allows users to
choose based on their infrastructure (cloud-native, on-premise, etc).
## Backward Compatibility:
- All new configuration fields have defaults
- External logging disabled by default
- No changes to existing logging behavior unless explicitly configured
- No new required dependencies
## Testing:
- All 20 new tests passing
- Existing tests unaffected (same count of passing tests)
- Configuration validation tested
- Handler creation and lifecycle management tested
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds support for gradual session secret rotation without forcing logout:
- Add BANGUI_SESSION_SECRET_PREVIOUS config field for rotation window
- Implement unwrap_session_token_with_rotation() to accept tokens signed with
either current or previous secret
- Update validate_session() to transparently accept old tokens during rotation
- Update logout() to accept tokens from both secrets
- Add comprehensive logging for rotation events and metrics
- Add 8 new tests covering all rotation scenarios
- Update documentation with step-by-step rotation strategy
- Update .env.example with previous secret field
Key features:
- No forced logout: old tokens continue working during rotation window
- Transparent validation: old tokens are automatically logged for monitoring
- Production-safe: can rotate secrets without service interruption
- Metrics-ready: logs track token rotation for observability
Rotation workflow:
1. Generate new secret and set BANGUI_SESSION_SECRET
2. Set BANGUI_SESSION_SECRET_PREVIOUS to old secret
3. Wait for old tokens to expire (≥ session_duration_minutes)
4. Unset BANGUI_SESSION_SECRET_PREVIOUS to complete rotation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Implement cursor-based pagination in pagination.py
- Update response models to standardize pagination structure
- Add cursor pagination utilities for repositories
- Update HistoryArchiveRepository and ImportLogRepository with new pagination
- Add comprehensive tests for cursor pagination
- Update documentation for backend development and task tracking
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add AbortController refs and abort signal checks to prevent race conditions
and memory leaks when components unmount or new requests are initiated.
Components fixed:
- JailsTab.tsx: validation handler with AbortController pattern
- JailInfoSection.tsx: handle function with useCallback wrapper
- RawConfigSection.tsx: fetch handler with abort checks
- ConfFilesTab.tsx: file fetch handler with abort signal verification
- IgnoreListSection.tsx: three handlers (add, remove, toggle) with callbacks
All handlers now:
1. Abort previous requests before initiating new ones
2. Create and store new AbortController instances
3. Check abort status before state updates in .then()/.catch()
4. Include cleanup effects that abort on unmount
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The API pagination infrastructure was already correctly implemented with:
- PaginatedListResponse base model containing 'items' and 'pagination' fields
- PaginationMetadata object with all required fields (page, page_size, total, total_pages, has_next_page, has_prev_page)
- All services correctly calling create_pagination_metadata()
However, there were two bugs preventing tests from passing:
1. IMPORT BUG: time_utils.py was importing TIME_RANGE_SECONDS from app.models.ban
when it's actually defined in app.models._common. This caused import errors
in tests that exercise time-range filtering.
2. TEST BUG: Test assertions were using outdated API structure, accessing
.total, .page, .page_size directly on paginated responses instead of
through the .pagination object.
Fixed locations:
- test_mappers/test_ban_mappers.py: 3 assertions updated to use .pagination.*
- test_services/test_blocklist_service.py: 6 assertions updated
- test_services/test_history_service.py: 14 assertions updated
All paginated API endpoints now correctly return pagination metadata:
- GET /api/history
- GET /api/history/archive
- GET /api/dashboard/bans
- GET /api/jails/{name}/banned
- GET /api/blocklists/log
Verified with 24 passing pagination tests demonstrating correct behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Create PaginationMetadata model with computed derived fields (total_pages, has_next_page, has_prev_page)
- Update PaginatedListResponse to embed pagination metadata in a separate 'pagination' object
- Add create_pagination_metadata() factory function in utils/pagination.py for consistent computation
- Update all paginated service functions to use new structure:
- history_service.list_history()
- blocklist_service.get_import_logs()
- jail_service.get_jail_banned_ips()
- ban_mappers.map_domain_dashboard_ban_list_to_response()
- Update response model docstrings with new structure examples
- Update Backend-Development.md documentation with new pagination patterns
- Update test fixtures to work with new response structure
Response shape changes from:
{"items": [...], "total": 100, "page": 1, "page_size": 50}
To:
{"items": [...], "pagination": {"page": 1, "page_size": 50, "total": 100, "total_pages": 2, "has_next_page": true, "has_prev_page": false}}
Benefits:
- Frontend receives all pagination state needed for UI controls
- No need for frontend to calculate total_pages or page navigation logic
- Consolidated pagination metadata reduces field sprawl
- OpenAPI schema automatically reflects changes
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit addresses race conditions in multi-step database operations by:
1. Wrap write operations in BEGIN IMMEDIATE ... COMMIT transactions:
- import_run_repo: create_pending, mark_completed, mark_failed
- geo_cache_repo: all upsert_*_and_commit functions
- geo_cache_repo: bulk_upsert_entries_and_neg_entries_and_commit
2. Handle concurrent write collisions gracefully:
- import_run_repo.create_pending can now raise IntegrityError
- blocklist_import_workflow catches IntegrityError and retries lookup
- Logs 'blocklist_import_lost_race' event when another request wins the race
3. Add comprehensive documentation:
- Backend-Development.md § 6.3 Database Transactions
- Explains when to use BEGIN IMMEDIATE
- Shows transaction pattern with try-except-rollback
- Documents race condition error handling pattern
The solution leverages SQLite's UNIQUE constraint for data integrity while
handling the concurrent case gracefully in application logic. This is more
efficient than using BEGIN EXCLUSIVE which would serialize all writers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The health check endpoint now properly indicates service unavailability:
- Returns HTTP 200 when fail2ban is online
- Returns HTTP 503 when fail2ban is offline
This allows Docker and other orchestration tools to correctly detect when
fail2ban is unreachable and automatically restart the backend container,
preventing the situation where Docker treats the container as healthy
despite fail2ban being down.
Changes:
- Update GET /api/health to return 503 on fail2ban offline
- Return appropriate JSON response bodies for each state
- Update tests to verify both online (200) and offline (503) scenarios
- Update Dockerfile HEALTHCHECK documentation
- Add Health Checks section to Deployment.md documentation
All tests pass with 100% coverage on health.py.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
