- Extract jail status/processing to helper functions
- Add error_handling.py service for centralized error handling
- Update config.py with validation and defaults
- Update .env.example with all config options
- Remove obsolete Tasks.md, add Service-Development.md
- Minor fixes across routers and services
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace contextlib.suppress with try/except + warning log
- Add test for fail2ban client
- Remove stale Issue #21 from Tasks.md (indexes)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- New get_fail2ban_db_path() in setup_service resolves DB path from configured socket path
- New ensure_fail2ban_indexes() creates missing performance indexes on bans table
- Call ensure_fail2ban_indexes on every startup before first ban query
- Remove completed tasks from Docs/Tasks.md
- Update Docs/PERFORMANCE.md with index findings
Detect catastrophic backtracking patterns before regex compilation
using regexploit library. Add ReDoSDetectedError exception and
_MINIMUM_STARRINESS threshold (>=3) to catch dangerous patterns
like (a+)+b. Update pyproject.toml deps, add tests for detection.
- All backend routers moved to /api/v1/ prefix
- Frontend BASE_URL updated to /api/v1
- Setup redirect middleware updated to redirect to /api/v1/setup
- Health router path fixed: prefix=/api/v1/health, @router.get('')
- conftest.py: set server_status=online for test fixture
- Created Docs/API_VERSIONING.md with deprecation policy
- Updated Docs/Backend-Development.md with versioning section
- Updated Instructions.md curl examples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit adds support for shipping logs to external centralized logging platforms, addressing the MEDIUM priority task for structured logging infrastructure.
## Key Changes:
### 1. New Documentation: Docs/Observability.md
- Comprehensive guide to logging architecture and configuration
- Covers all three supported platforms (Datadog, Papertrail, Elasticsearch)
- Includes best practices, security considerations, and troubleshooting
- Documents sensitive data handling and compliance requirements
### 2. Core Implementation: app/utils/external_logging.py
- ExternalLogHandler: Abstract base class for non-blocking log delivery
- DatadogLogHandler: HTTP API integration with JSON payloads
- PapertrailLogHandler: Syslog protocol over TCP
- ElasticsearchLogHandler: Bulk API integration with NDJSON format
- Features:
- Async buffering with configurable batch size and flush interval
- Exponential backoff retry logic
- Non-blocking delivery (never blocks application logic)
- Proper error handling and internal logging
- Lifecycle management (start/shutdown)
### 3. Configuration: app/config.py
- New Settings fields for external logging:
- external_logging_enabled (default: False)
- external_logging_provider (datadog/papertrail/elasticsearch)
- external_logging_buffer_size (default: 1000)
- external_logging_flush_interval_seconds (default: 5.0)
- Provider-specific configuration (API keys, hosts, batch sizes)
- All fields have sensible defaults
- Full field validation and normalization
### 4. Integration: app/main.py
- Global _external_log_handler for application lifecycle
- _external_logging_processor: structlog processor for handler integration
- Updated _configure_logging(): Add handler to processor chain when enabled
- Updated _lifespan(): Initialize handler before startup, shutdown on termination
### 5. Tests: backend/tests/test_external_logging.py
- 20 comprehensive tests covering all handlers and factory
- Configuration validation tests
- All tests passing
## Design Decisions:
1. **Non-blocking Delivery**: External logging never blocks request handling.
Failures are logged locally but don't impact application.
2. **Buffering Strategy**: In-memory buffer with configurable size prevents
unbounded memory growth. When buffer fills, oldest logs are dropped with
a warning.
3. **Retry Logic**: Transient failures (timeouts, 5xx errors) are retried
with exponential backoff. Permanent failures (bad credentials) are logged
and skipped.
4. **Disabled by Default**: External logging is opt-in via environment
variables, maintaining backward compatibility with existing deployments.
5. **Provider Flexibility**: Support for multiple platforms allows users to
choose based on their infrastructure (cloud-native, on-premise, etc).
## Backward Compatibility:
- All new configuration fields have defaults
- External logging disabled by default
- No changes to existing logging behavior unless explicitly configured
- No new required dependencies
## Testing:
- All 20 new tests passing
- Existing tests unaffected (same count of passing tests)
- Configuration validation tested
- Handler creation and lifecycle management tested
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Implement cursor-based pagination in pagination.py
- Update response models to standardize pagination structure
- Add cursor pagination utilities for repositories
- Update HistoryArchiveRepository and ImportLogRepository with new pagination
- Add comprehensive tests for cursor pagination
- Update documentation for backend development and task tracking
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The API pagination infrastructure was already correctly implemented with:
- PaginatedListResponse base model containing 'items' and 'pagination' fields
- PaginationMetadata object with all required fields (page, page_size, total, total_pages, has_next_page, has_prev_page)
- All services correctly calling create_pagination_metadata()
However, there were two bugs preventing tests from passing:
1. IMPORT BUG: time_utils.py was importing TIME_RANGE_SECONDS from app.models.ban
when it's actually defined in app.models._common. This caused import errors
in tests that exercise time-range filtering.
2. TEST BUG: Test assertions were using outdated API structure, accessing
.total, .page, .page_size directly on paginated responses instead of
through the .pagination object.
Fixed locations:
- test_mappers/test_ban_mappers.py: 3 assertions updated to use .pagination.*
- test_services/test_blocklist_service.py: 6 assertions updated
- test_services/test_history_service.py: 14 assertions updated
All paginated API endpoints now correctly return pagination metadata:
- GET /api/history
- GET /api/history/archive
- GET /api/dashboard/bans
- GET /api/jails/{name}/banned
- GET /api/blocklists/log
Verified with 24 passing pagination tests demonstrating correct behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Create PaginationMetadata model with computed derived fields (total_pages, has_next_page, has_prev_page)
- Update PaginatedListResponse to embed pagination metadata in a separate 'pagination' object
- Add create_pagination_metadata() factory function in utils/pagination.py for consistent computation
- Update all paginated service functions to use new structure:
- history_service.list_history()
- blocklist_service.get_import_logs()
- jail_service.get_jail_banned_ips()
- ban_mappers.map_domain_dashboard_ban_list_to_response()
- Update response model docstrings with new structure examples
- Update Backend-Development.md documentation with new pagination patterns
- Update test fixtures to work with new response structure
Response shape changes from:
{"items": [...], "total": 100, "page": 1, "page_size": 50}
To:
{"items": [...], "pagination": {"page": 1, "page_size": 50, "total": 100, "total_pages": 2, "has_next_page": true, "has_prev_page": false}}
Benefits:
- Frontend receives all pagination state needed for UI controls
- No need for frontend to calculate total_pages or page navigation logic
- Consolidated pagination metadata reduces field sprawl
- OpenAPI schema automatically reflects changes
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add global rate limiter utility with configurable limits and cleanup
- Move rate limiting logic to middleware for consistent application
- Update auth routes to use new rate limiter
- Add comprehensive tests for rate limiter functionality
- Update documentation with backend development guidelines and tasks
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses: Backend session cache not cluster-safe (multi-worker issue)
Problem:
- Session cache is process-local (InMemorySessionCache)
- Multi-worker deployments (uvicorn --workers N) create separate processes
- Each process has its own independent session cache
- Sessions cached in Worker A are invisible to Workers B, C, D
- Users randomly logged out when requests land on different workers
- Also affects RuntimeState, rate limiter, and background jobs
Solution (Option A - Strict single-worker enforcement):
- Enhance startup validation with clearer error messages
- Update error messages to explain the problem and how to fix it
- Document single-worker requirement prominently in Docker configs
- Update module docstrings to clarify constraints
Changes:
1. app/startup.py:
- Enhanced _check_single_worker_mode() error message with troubleshooting
- Enhanced _stage_check_worker_mode_and_acquire_lock() error message
- Removed unused import
2. app/utils/session_cache.py:
- Updated module docstring to explain constraints more clearly
- Added references to deployment documentation
- Clarified multi-worker solution for future implementation
3. app/utils/runtime_state.py:
- Updated module docstring with deployment constraint references
- Aligned messaging with session_cache.py
4. Docker/Dockerfile.backend:
- Added comprehensive comments about single-worker requirement
- Explained impact in multi-worker deployments
- Referenced deployment constraints documentation
5. Docker/docker-compose.yml, compose.prod.yml, compose.debug.yml:
- Added documentation comments about BANGUI_WORKERS constraint
- Explained why single-worker is required
6. backend/tests/test_startup_integration.py:
- Fixed test unpacking to match function return signature (3 values, not 2)
This ensures multi-worker deployments fail loudly at startup with clear
guidance on what went wrong and how to fix it. The database-backed scheduler
lock provides defense-in-depth for container orchestration scenarios.
For future multi-worker support, implement:
- Redis or database-backed session cache
- Shared RuntimeState coordination
- Distributed APScheduler backend
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Update rate limiter to use exponential backoff instead of fixed limit
- Implement progressive delays for failed login attempts (0.5s, 1s, 2s, 4s, 5s max)
- Update auth router documentation and endpoint docs
- Refactor test suite to match new rate limiting behavior
- Update backend development documentation
- Clean up unused tasks documentation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add comprehensive documentation for backend development
- Improve client IP detection with utility functions and tests
- Update auth router with better error handling
- Refactor config module with environment-based settings
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement periodic cleanup of expired rate-limiter entries to prevent
unbounded memory growth during long runtimes.
Changes:
- Create rate_limiter_cleanup task that calls cleanup_expired() every 30 minutes
- Register the task in the startup DAG alongside other background jobs
- Update rate_limiter module documentation with operational notes about the
cleanup lifecycle and memory management strategy
The cleanup is conservative and only removes IPs with no recent attempts
(all timestamps outside the rate-limit window), so active IPs are preserved.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Problem
The blocklist URL validation at create/update time has a TOCTOU (time-of-check-to-time-of-use) window.
An attacker can perform a DNS-rebinding attack where:
1. User adds blocklist URL pointing to attacker.com
2. At create time, attacker.com resolves to a public IP → validation passes
3. Later, when fetching, attacker.com resolves to 192.168.1.1 (internal network)
4. HTTP client connects to the private IP, potentially accessing internal services
## Solution
Add runtime destination IP validation at connection time via a custom socket factory:
- Created 'dns_validated_connector.py' with create_dns_validated_socket_factory() that validates
all resolved IPs before socket creation
- HTTP session now uses the validated socket factory, protecting all blocklist imports globally
- Rejects connections to RFC 1918 private ranges, loopback, link-local, ULA, multicast, and
reserved addresses (IPv4 and IPv6)
- Added comprehensive test coverage with 13 test cases
## Changes
- backend/app/services/dns_validated_connector.py: Custom socket factory with IP validation
- backend/app/startup.py: Use DNS-validated socket factory in HTTP session creation
- backend/app/utils/ip_utils.py: Updated docstring explaining runtime validation
- backend/app/services/blocklist_downloader.py: Updated module docstring
- backend/app/services/blocklist_service.py: Updated docstrings explaining two-layer protection
- backend/tests/test_services/test_dns_validated_connector.py: Test suite for socket factory
- Docs/Architekture.md: Added detailed section on DNS-rebinding protection
## Testing
- All 13 DNS validation tests pass
- All blocklist downloader tests pass (unaffected by changes)
- Linting: ruff, mypy pass with --strict
- Test coverage: 90% line coverage on dns_validated_connector.py
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TASK-004: Replace module-level mutable runtime flags in service layer with
injected state holder, eliminating hidden global state and improving testability
and synchronization boundaries.
Changes:
- Create JailServiceState dataclass in app/utils/runtime_state.py to hold
backend capability cache and synchronization lock
- Add JailServiceState as a field in RuntimeState (with default_factory)
- Remove module-level _backend_cmd_supported and _backend_cmd_lock from
jail_service.py
- Refactor _check_backend_cmd_supported() to accept state parameter
- Inject JailServiceState into list_jails() and _fetch_jail_summary() via
parameters
- Add get_jail_service_state() dependency provider in app/dependencies.py
- Add JailServiceStateDep type alias for router injection
- Update jails router to receive and pass state to service functions
- Update all tests to use jail_service_state fixture and pass state to functions
- Remove duplicate _MAX_PAGE_SIZE constant definition
- Document mutable state management in Backend-Development.md
- Update Architecture.md to describe JailServiceState and state nesting pattern
Benefits:
- Eliminates global mutable state and associated race conditions
- Makes state visible to callers (not hidden in module scope)
- Enables test isolation (each test gets fresh state)
- Prepares codebase for multi-worker deployments (state can be extracted to
shared backend)
- Synchronization boundaries are now explicit (state.get_backend_cmd_lock())
Compliance:
- All tests pass (17 passed in TestListJails, TestGetJail, TestLockInitialization)
- No ruff linting errors
- Type-safe: JailServiceState properly typed with asyncio.Lock, bool | None
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Create logged_task() helper in backend/app/utils/async_utils.py to wrap
fire-and-forget coroutines with exception logging
- Ensures unhandled task exceptions are always logged to structlog instead of
silently discarded (Python 3.11+ RuntimeWarning)
- Update ban_service.py to use logged_task() for geo_cache.lookup_batch()
background resolution
- Add comprehensive tests for logged_task() in test_async_utils.py
- Document fire-and-forget task conventions in Backend-Development.md
The logged_task() wrapper catches any exception raised in a background task,
logs it with full traceback context and task name, and never re-raises.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
**Issue:**
- log_target accepted arbitrary paths, allowing authenticated users to write
files as root via fail2ban (e.g., /etc/cron.d/bangui-pwned)
- fail2ban runs as root and opens files specified in log_target
**Solution:**
1. **Model layer validation:** Already existed in GlobalConfigUpdate, prevents
invalid paths before reaching service
2. **Service layer validation:** Added defensive check in update_global_config()
that validates log_target even if model validation is bypassed
3. **New validation helper:** Added validate_log_target() utility that accepts
special values (STDOUT, STDERR, SYSLOG) or paths within allowed directories
**Changes:**
- app/utils/path_utils.py: Added validate_log_target() helper
- app/services/config_service.py: Added service-layer validation before
sending command to fail2ban
- backend/tests: Fixed session_secret length issues in fixtures (min 32 chars)
- backend/tests: Added tests for valid special log targets
- Docs/Backend-Development.md: Documented log_target security requirements
**Test Coverage:**
- Model validation rejects /etc/passwd (existing test)
- Model validation accepts STDOUT, STDERR, SYSLOG special values
- Model validation accepts paths in allowed directories
- Service layer validation tested with special values
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add escape_like() helper to escape % and _ wildcards in LIKE queries
- Update fail2ban_db_repo.get_history_page() to use escaping
- Update history_archive_repo.get_archived_history() to use escaping
- Add ESCAPE clause to all LIKE queries
- Add comprehensive unit tests for escape_like function
- Add integration tests for LIKE wildcard handling
- Document LIKE escaping best practices in Backend-Development.md
Fixes TASK-017: Prevent unintended LIKE matches when IP filter contains
special characters like underscore or percent sign.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Extract path validation logic into shared helper function in
backend/app/utils/path_utils.py (validate_log_path)
- Refactor AddLogPathRequest to use the helper function
- Apply the same validation to DELETE /api/config/jails/{name}/logpath
endpoint by validating the log_path query parameter
- Return HTTP 422 with descriptive error if validation fails
- Add comprehensive unit tests for path validation
- Update Backend-Development.md with usage examples
This prevents path-traversal attacks on the delete_log_path endpoint
by ensuring all log paths are within allowlisted directories.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add in-memory rate limiter with per-IP deque tracking of attempt timestamps
- Limit login attempts to 5 per 60 seconds per IP, return 429 on excess
- Add Retry-After header to rate limit responses
- Implement IP extraction utility with proxy trust validation (prevent X-Forwarded-For spoofing)
- Integrate rate limiter into auth router and dependencies
- Add 10-second asyncio.sleep on failed login attempts to further slow brute-force
- Add comprehensive tests for rate limiting (9 new tests, all passing)
- Update Features.md to document login rate limiting
- Update Backend-Development.md with rate limiting conventions and design patterns
- Fix test infrastructure issues: update password to meet complexity requirements
- Fix TestValidateSession tests to use Bearer token authentication
- All tests passing: 23 auth tests + full test suite coverage
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add comprehensive docstring to runtime_state.py explaining single-process
constraint, impacts in multi-worker deployments, and solution approach
- Add comprehensive docstring to session_cache.py explaining process-local
cache limitation, security implications, and Redis/database alternatives
- Update Architecture.md to clarify session cache is process-local and
describe single-worker enforcement via TASK-002
- Update Architecture.md runtime state section with detailed explanation of
per-process state and multi-worker impacts
- Add Backend-Development.md section 13.7.2 documenting session cache
pluggability pattern with example Redis implementation
- All tests pass; linting passes; type checking has pre-existing errors
This is the short-term fix for TASK-003: enforce single-worker deployment
(TASK-002) and document the constraint clearly. The long-term fix (Redis
backend) is deferred as a follow-up.
Consolidate the two divergent implementations of _since_unix from ban_service.py
and history_service.py into a single shared utility function in time_utils.py.
Changes:
- Move _since_unix to app/utils/time_utils.py with consistent time.time() approach
- Move TIME_RANGE_SLACK_SECONDS constant to app/utils/constants.py
- Update ban_service.py to import since_unix from time_utils
- Update history_service.py to import since_unix from time_utils
- Both services now use the same window boundary calculation with 60-second slack
- Add comprehensive tests for the shared since_unix function
- Document timestamp handling rationale in Backend-Development.md
This ensures dashboard and history queries return consistent row counts for the
same time range by using the same timestamp calculation and slack window across
all services.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Refactor action_config_service, filter_config_service, jail_config_service, and jail_service
- Add jail_socket utility module for socket communication
- Update test_jail_service with new test cases
- Update architecture and task documentation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move session cache initialization from per-request _build_app_context to
startup lifespan handler. The session cache type is now decided once at app
startup based on settings, making _build_app_context pure (read-only).
Changes:
- Move cache initialization logic to new _update_session_cache() in main.py
- Call _update_session_cache() during lifespan startup to initialize cache
- Remove three if/elif/elif branches mutating state.session_cache from _build_app_context
- Add cache swap logic to set_runtime_settings() in runtime_state.py to handle
runtime settings changes (e.g., setup wizard updates)
- Keep app.state.session_cache initialization in create_app() for test compatibility
This ensures:
- _build_app_context is pure and doesn't mutate app state on each request
- Session cache configuration decisions are centralized at startup
- Settings changes during runtime (via setup wizard) also trigger cache swap
- Cache initialization logic is isolated in one place
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Reorganized dashboard router with improved structure
- Enhanced ban_service with better separation of concerns
- Updated history service with cleaner logic
- Improved constants and configuration handling
- Updated documentation of completed tasks
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Consolidate duplicate _ok(), _to_dict(), ensure_list(), and is_not_found_error()
functions from 6 service modules into a single canonical implementation at
backend/app/utils/fail2ban_response.py.
Changes:
- Create fail2ban_response.py with canonical implementations
- Remove local duplicates from: ban_service, jail_service, config_service,
health_service, server_service, config_file_utils
- Update all imports to use shared module
- Add comprehensive docstrings and examples
- Update Architecture.md and Backend-Development.md documentation
Benefits:
- Single source of truth for response parsing logic
- Eliminates code duplication across service layer
- Improves maintainability and consistency
- Enables centralized bug fixes and improvements
Tests: All 228 service tests passing, no regressions
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use NoOpSessionCache in backend/app/main.py and dynamically switch cache implementation in backend/app/dependencies.py so disabled cache mode remains safe while get_session_cache always returns a valid object.
Wire DEFAULT_BLOCKING_EXECUTOR as the default executor in backend/app/utils/async_utils.py, preserving custom executors and marking Task 22 completed in Docs/Tasks.md.
