Commit Graph

166 Commits

Author SHA1 Message Date
Copilot
22db607875 Add fail2ban DB index management and socket-based path resolution
- New get_fail2ban_db_path() in setup_service resolves DB path from configured socket path
- New ensure_fail2ban_indexes() creates missing performance indexes on bans table
- Call ensure_fail2ban_indexes on every startup before first ban query
- Remove completed tasks from Docs/Tasks.md
- Update Docs/PERFORMANCE.md with index findings
2026-05-03 12:17:31 +02:00
0133489920 Update observability docs and task utilities
- Add Observability.md documentation
- Standardize task logging with correlation_id support
- Add log_sanitizer utility for PII masking
- Update Tasks.md tracking
- Update geo_cache tasks and other task modules with correlation_id

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 11:52:09 +02:00
7ad885d276 refactor: separate config service from jail config service
- Split config_service.py into config_service.py and jail_config_service.py
- Update Docs/Tasks.md, Security.md, TROUBLESHOOTING.md
2026-05-03 01:05:18 +02:00
881cfbdd71 fix: replace broad except Exception with specific exception types
- jail_service: catch ValueError (fail2ban protocol error) instead of Exception
- health.py: catch AttributeError (not OSError/TypeError) for defensive checks
- ban_service: re-raise programming errors in geo lookup handlers
- server_service: catch Fail2BanConnectionError, Fail2BanProtocolError, ValueError
- config_writer: catch OSError instead of Exception

Programming errors now bubble to global handler instead of being silently caught.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 00:54:44 +02:00
bd6170722a feat(geo): add cache hit/miss metrics and prewarm support
- Add _hits/_misses counters to GeoCache for cache hit/miss ratio tracking
- Reset counters on clear()
- Count hits before misses in lookup_batch() to avoid interleaving
- Add synchronous prewarm() using asyncio.create_task for fire-and-forget
- Add hits/misses fields to GeoCacheStatsResponse model
- Add TestCacheMetrics (5 tests), TestPrewarm (3 tests), TestLargeBanList (2 tests)
- Fix _make_async_db() mock: db.execute is not async, returns ctx manager
- Move collections.abc to TYPE_CHECKING block (TC003)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 00:35:47 +02:00
e436727942 fix: atomic upsert for import runs (Issue #12)
Replace check-then-insert race condition with INSERT ON CONFLICT.
- upsert_pending uses RETURNING id for atomic upsert
- UNIQUE(source_id, content_hash) constraint from migration 6
- blocklist_import_workflow updated to use upsert_pending
- test_import_source_success fixed for async mock patterns

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 23:39:43 +02:00
cc6dbcf3f0 feat: implement API versioning /api/v1/
- All backend routers moved to /api/v1/ prefix
- Frontend BASE_URL updated to /api/v1
- Setup redirect middleware updated to redirect to /api/v1/setup
- Health router path fixed: prefix=/api/v1/health, @router.get('')
- conftest.py: set server_status=online for test fixture
- Created Docs/API_VERSIONING.md with deprecation policy
- Updated Docs/Backend-Development.md with versioning section
- Updated Instructions.md curl examples

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 21:29:30 +02:00
0d5882b32f Fix HIGH priority issues: unbounded queries, rate limiting, health checks
Issue #3 - Unbounded Query Results (OOM):
- get_all_archived_history() now uses keyset pagination with bounded max_rows (50k default)
- Added 'id' field to records from get_archived_history() and get_archived_history_keyset()
- Protocol signature updated with page_size, max_rows, last_ban_id params

Issue #7 - Docker Health Check Fails:
- Added curl to Dockerfile.backend runtime image
- HEALTHCHECK now uses 'curl -f http://localhost:8000/api/health'
- compose.prod.yml: increased start_period to 40s, timeout to 10s
- Frontend healthcheck proxies to backend /api/health

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-01 21:47:36 +02:00
60d9c5b340 Refactor filter configuration with regex validation
- Add regex validation utility for query strings
- Update filter_config_service to use regex validation
- Add comprehensive test coverage for regex validator
- Update exception handling for validation errors
- Update documentation for tasks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-01 18:17:12 +02:00
8138857ee1 feat: Implement session secret rotation support
Adds support for gradual session secret rotation without forcing logout:

- Add BANGUI_SESSION_SECRET_PREVIOUS config field for rotation window
- Implement unwrap_session_token_with_rotation() to accept tokens signed with
  either current or previous secret
- Update validate_session() to transparently accept old tokens during rotation
- Update logout() to accept tokens from both secrets
- Add comprehensive logging for rotation events and metrics
- Add 8 new tests covering all rotation scenarios
- Update documentation with step-by-step rotation strategy
- Update .env.example with previous secret field

Key features:
- No forced logout: old tokens continue working during rotation window
- Transparent validation: old tokens are automatically logged for monitoring
- Production-safe: can rotate secrets without service interruption
- Metrics-ready: logs track token rotation for observability

Rotation workflow:
1. Generate new secret and set BANGUI_SESSION_SECRET
2. Set BANGUI_SESSION_SECRET_PREVIOUS to old secret
3. Wait for old tokens to expire (≥ session_duration_minutes)
4. Unset BANGUI_SESSION_SECRET_PREVIOUS to complete rotation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-01 18:01:11 +02:00
73021429f7 refactor: restructure API pagination metadata for better frontend usability
- Create PaginationMetadata model with computed derived fields (total_pages, has_next_page, has_prev_page)
- Update PaginatedListResponse to embed pagination metadata in a separate 'pagination' object
- Add create_pagination_metadata() factory function in utils/pagination.py for consistent computation
- Update all paginated service functions to use new structure:
  - history_service.list_history()
  - blocklist_service.get_import_logs()
  - jail_service.get_jail_banned_ips()
  - ban_mappers.map_domain_dashboard_ban_list_to_response()
- Update response model docstrings with new structure examples
- Update Backend-Development.md documentation with new pagination patterns
- Update test fixtures to work with new response structure

Response shape changes from:
  {"items": [...], "total": 100, "page": 1, "page_size": 50}
To:
  {"items": [...], "pagination": {"page": 1, "page_size": 50, "total": 100, "total_pages": 2, "has_next_page": true, "has_prev_page": false}}

Benefits:
- Frontend receives all pagination state needed for UI controls
- No need for frontend to calculate total_pages or page navigation logic
- Consolidated pagination metadata reduces field sprawl
- OpenAPI schema automatically reflects changes

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-30 22:24:42 +02:00
f9e283541b Add explicit database transaction isolation to multi-step operations
This commit addresses race conditions in multi-step database operations by:

1. Wrap write operations in BEGIN IMMEDIATE ... COMMIT transactions:
   - import_run_repo: create_pending, mark_completed, mark_failed
   - geo_cache_repo: all upsert_*_and_commit functions
   - geo_cache_repo: bulk_upsert_entries_and_neg_entries_and_commit

2. Handle concurrent write collisions gracefully:
   - import_run_repo.create_pending can now raise IntegrityError
   - blocklist_import_workflow catches IntegrityError and retries lookup
   - Logs 'blocklist_import_lost_race' event when another request wins the race

3. Add comprehensive documentation:
   - Backend-Development.md § 6.3 Database Transactions
   - Explains when to use BEGIN IMMEDIATE
   - Shows transaction pattern with try-except-rollback
   - Documents race condition error handling pattern

The solution leverages SQLite's UNIQUE constraint for data integrity while
handling the concurrent case gracefully in application logic. This is more
efficient than using BEGIN EXCLUSIVE which would serialize all writers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-30 22:04:15 +02:00
52f237d5d4 Make background tasks idempotent - prevent duplicate bans on retry
CRITICAL FIX: Background tasks (especially blocklist_import) crashed mid-execution,
leaving partial state. On retry, the same bans were applied again, causing duplicates.

Solution: Content-hash based operation tracking for blocklist imports:
- Added import_runs table (migration 6) to track operations by source + content hash
- Before banning, check if this exact content has already been imported
- If completed: skip banning (already done), optionally re-warm cache
- If new or failed: proceed with ban and mark as completed or failed

Changes:
- Database: Migration 6 adds import_runs table with operation state tracking
- Model: Added ImportRunEntry for import run records
- Repository: New import_run_repo module with CRUD operations
- Workflow: Updated blocklist_import_workflow to check operation history before banning
- Dependencies: Registered import_run_repo for dependency injection
- Tests: Added test_import_source_idempotent_on_retry and test_import_source_different_content_not_reused
- Documentation: Added Task Idempotency section to Backend-Development.md

Verification:
- All 7 import tests pass (5 existing + 2 new idempotency tests)
- Type checking: mypy --strict 
- Linting: ruff 
- No API changes, backwards compatible via automatic migration

Fixes: Background tasks not idempotent #CRITICAL

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-30 21:54:14 +02:00
9afdbe2852 Refactor auth and setup services
- Updated auth_service.py to improve authentication logic
- Modified setup_service.py for better configuration handling
- Added comprehensive tests for setup_service
- Updated documentation in Tasks.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-30 20:10:00 +02:00
100fd47c4b Refactor: Make model packages true leaf nodes - remove app-layer dependencies
Models in app/models/ are now pure data classes with no cross-layer dependencies.
This ensures the models layer remains a true leaf node in the dependency graph.

Changes:
- Create app/models/_common.py with shared types (TimeRange, bucket_count, constants)
- Move TimeRange and time-range constants from ban.py to _common.py
- Update history.py, routers, and services to import from _common.py
- Remove imports from app.config and app.utils from config.py models
- Move field validators from models to router layer:
  - Add log_target validation in config_misc router
  - Add log_path validation in jail_config router
- Update test_models.py to reflect validators moved to router layer
- Update documentation (Architekture.md, Backend-Development.md) with model layering rules
- Fix import ordering and type annotations in affected files

Model layering rule: Models may only import from:
✓ Standard library and third-party packages (Pydantic, typing)
✓ Other models in app/models/ (sibling models)
✓ app.models.response (response envelopes)
✗ app.services, app.config, app.utils, or any application layer

Validation requiring app-level state (settings, allowed directories) now happens
at the router or service layer, not in model validators.

Fixes: Models were not true leaf nodes due to circular imports and app-layer dependencies

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-30 19:31:11 +02:00
1302ac821f Fix non-atomic setup persistence across DB contexts (Issue #30)
Implement transactional setup with explicit state machine and crash-safety
to prevent partial commits from leaving inconsistent state.

## Changes

### Core Implementation
1. **settings_repo.py**: Add atomic batch settings write
   - New set_settings_batch() method: writes multiple settings in single
     transaction (BEGIN IMMEDIATE ... COMMIT). Either all settings persist
     or none do, preventing partial state if crash occurs mid-batch.

2. **setup_service.py**: Refactor run_setup() with transactional phases
   - Phase 0: Compute password hash early (before any DB writes) to ensure
     idempotency. Same hash is used throughout retries, preventing divergent
     hashes from bcrypt's random salt.
   - Phase 1 (Bootstrap DB transaction): Set setup_state=in_progress and
     database_path, then commit. First checkpoint for crash detection.
   - Phase 2 (Filesystem): Initialize runtime database (idempotent)
   - Phase 3 (Runtime DB transaction): Batch-write all settings atomically
   - Phase 4 (Bootstrap DB transaction): Set setup_state=complete and
     setup_completed=1. Final commit point.

3. **protocols.py**: Add set_settings_batch to SettingsRepository protocol

### Testing
- Added 6 new transactionality tests covering:
  - State machine transitions (None → in_progress → complete)
  - Password hash idempotency across retries
  - Atomic batch writes (all-or-nothing persistence)
  - Bootstrap DB state tracking
  - Database path propagation to both DBs
  - Recovery on partial failure
- All 18 tests pass (12 existing + 6 new)

### Documentation
- Updated Docs/Architekture.md with new section 6:
  - Setup state machine with state transitions
  - Transaction boundary documentation
  - Password hash idempotency rationale
  - Backward compatibility notes

## Design Decisions

### Why This Approach
- Current code already idempotent via INSERT OR REPLACE, but password
  hash non-idempotency created silent inconsistency risk
- Simpler than multi-state machine: 2 states sufficient for detection
- Maintains backward compatibility (setup_completed key still written)
- Explicit transactions make crash-safety obvious to future maintainers

### Crash Scenarios Now Handled
1. Crash after Phase 1 → detected by setup_state=in_progress on retry
2. Crash after Phase 2 → runtime DB may be partial, safe to retry
3. Crash after Phase 3 → runtime DB rolls back on next connection
4. Crash after Phase 4 → setup_completed detected, skipped

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-29 19:19:53 +02:00
cc4370c50d feat: Add runtime DNS-rebinding protection for blocklist HTTP connections
## Problem
The blocklist URL validation at create/update time has a TOCTOU (time-of-check-to-time-of-use) window.
An attacker can perform a DNS-rebinding attack where:
1. User adds blocklist URL pointing to attacker.com
2. At create time, attacker.com resolves to a public IP → validation passes
3. Later, when fetching, attacker.com resolves to 192.168.1.1 (internal network)
4. HTTP client connects to the private IP, potentially accessing internal services

## Solution
Add runtime destination IP validation at connection time via a custom socket factory:

- Created 'dns_validated_connector.py' with create_dns_validated_socket_factory() that validates
  all resolved IPs before socket creation
- HTTP session now uses the validated socket factory, protecting all blocklist imports globally
- Rejects connections to RFC 1918 private ranges, loopback, link-local, ULA, multicast, and
  reserved addresses (IPv4 and IPv6)
- Added comprehensive test coverage with 13 test cases

## Changes
- backend/app/services/dns_validated_connector.py: Custom socket factory with IP validation
- backend/app/startup.py: Use DNS-validated socket factory in HTTP session creation
- backend/app/utils/ip_utils.py: Updated docstring explaining runtime validation
- backend/app/services/blocklist_downloader.py: Updated module docstring
- backend/app/services/blocklist_service.py: Updated docstrings explaining two-layer protection
- backend/tests/test_services/test_dns_validated_connector.py: Test suite for socket factory
- Docs/Architekture.md: Added detailed section on DNS-rebinding protection

## Testing
- All 13 DNS validation tests pass
- All blocklist downloader tests pass (unaffected by changes)
- Linting: ruff, mypy pass with --strict
- Test coverage: 90% line coverage on dns_validated_connector.py

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-29 19:10:51 +02:00
a2129bb9bd Pagination contract is not standardized across endpoints 2026-04-28 21:40:22 +02:00
b27765928a Standardize API response envelopes: use items for collection responses and update tests 2026-04-28 20:48:00 +02:00
1c673d600c Standardize API response envelope shapes across all endpoints
This commit standardizes how API responses are wrapped, solving issue #24.

Problem:
- Inconsistent response envelopes (jails vs items vs bans vs no wrapper)
- Frontend required multiple field name variants
- Integration bugs from branching logic
- No clear pattern for different response types

Solution:
- Created response.py with base classes: PaginatedListResponse,
  CollectionResponse, CommandResponse
- Standardized all list/collection responses to use 'items' field
- Domain-specific field names for detail and aggregation responses
- Updated all backends routers and mappers
- Updated frontend types and hooks to match

Changes:
Backend:
- backend/app/models/response.py (new): Base response models
- backend/app/models/ban.py: Updated responses to inherit from bases
- backend/app/models/jail.py: Updated JailListResponse, JailCommandResponse
- backend/app/models/config.py: Updated collection responses
- backend/app/services/jail_service.py: Updated return statements
- backend/app/mappers/ban_mappers.py: Updated 'bans' to 'items'
- backend/tests/test_mappers/test_ban_mappers.py: Updated tests

Frontend:
- frontend/src/types/jail.ts: Updated response interfaces
- frontend/src/types/config.ts: Updated response interfaces
- frontend/src/hooks/useActiveBans.ts: Updated selector
- frontend/src/hooks/useJailList.ts: Updated selector
- frontend/src/hooks/useJailConfigs.ts: Updated selector
- frontend/src/hooks/useConfigActiveStatus.ts: Updated field access
- frontend/src/hooks/useJailAdmin.ts: Updated field access

Documentation:
- Docs/Backend-Development.md: Added § 4.1 API Response Envelope Policy

The policy defines:
1. Paginated lists use PaginatedListResponse (items, total, page, page_size)
2. Non-paginated collections use CollectionResponse (items, total)
3. Detail responses use entity-specific field names (jail, status, settings)
4. Command responses use CommandResponse (message, success, optional target)
5. Aggregations use domain-specific fields (jails, countries, buckets, bans)

All responses now follow one of these patterns, reducing frontend complexity.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-28 10:12:55 +02:00
52a4d04d92 Task 8: Standardize modeling style (TypedDict vs Pydantic)
Convert inconsistent modeling style to standardized Pydantic models for all
external-facing data structures while maintaining TypedDict compatibility where
appropriate for internal layer-private structures.

Changes:
- Converted IpLookupResult TypedDict to use IpLookupResponse Pydantic model
  in jail_service.lookup_ip() for consistency with routers
- Added GeoCacheEntry Pydantic model for geo cache repository rows
- Converted GeoCacheRow TypedDict to use GeoCacheEntry alias
- Converted ImportLogRow TypedDict to use ImportLogEntry alias
- Updated routers and services to work with Pydantic models
- Updated all tests to use Pydantic model field access (attributes)
  instead of dict subscripting

Documentation:
- Added 'Model Type Usage by Layer' section to Backend-Development.md
- Defines when TypedDict is allowed (internal structures) vs Pydantic
  (external-facing, cross-boundary data)
- Provides clear guidance on modeling conventions per layer

Benefits:
- Consistent validation and serialization behavior
- Better IDE support and type checking
- Clearer separation of concerns by layer
- Reduced maintenance cost from mixed validation approaches

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-28 07:53:30 +02:00
3888c5eb3f Refactor ban management with domain models and mappers
- Add ban domain model for core business logic separation
- Implement mapper pattern for DTO/domain conversions
- Update ban service with new domain-driven approach
- Refactor router endpoints to use new architecture
- Add comprehensive mapper tests
- Update documentation with architecture changes

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-28 07:46:02 +02:00
2e221f6852 Refactor: Move module-level mutable flags to JailServiceState
TASK-004: Replace module-level mutable runtime flags in service layer with
injected state holder, eliminating hidden global state and improving testability
and synchronization boundaries.

Changes:
- Create JailServiceState dataclass in app/utils/runtime_state.py to hold
  backend capability cache and synchronization lock
- Add JailServiceState as a field in RuntimeState (with default_factory)
- Remove module-level _backend_cmd_supported and _backend_cmd_lock from
  jail_service.py
- Refactor _check_backend_cmd_supported() to accept state parameter
- Inject JailServiceState into list_jails() and _fetch_jail_summary() via
  parameters
- Add get_jail_service_state() dependency provider in app/dependencies.py
- Add JailServiceStateDep type alias for router injection
- Update jails router to receive and pass state to service functions
- Update all tests to use jail_service_state fixture and pass state to functions
- Remove duplicate _MAX_PAGE_SIZE constant definition
- Document mutable state management in Backend-Development.md
- Update Architecture.md to describe JailServiceState and state nesting pattern

Benefits:
- Eliminates global mutable state and associated race conditions
- Makes state visible to callers (not hidden in module scope)
- Enables test isolation (each test gets fresh state)
- Prepares codebase for multi-worker deployments (state can be extracted to
  shared backend)
- Synchronization boundaries are now explicit (state.get_backend_cmd_lock())

Compliance:
- All tests pass (17 passed in TestListJails, TestGetJail, TestLockInitialization)
- No ruff linting errors
- Type-safe: JailServiceState properly typed with asyncio.Lock, bool | None

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-27 18:42:52 +02:00
e08a16c7dd Refactor: Split blocklist import flow into focused components
Extracted the monolithic import_source() function (776 lines) into focused,
testable components with clear single responsibilities:

- BlocklistDownloader: HTTP download with exponential backoff retry logic
  * Handles transient failures (429, 5xx errors, timeouts)
  * Configurable retry attempts and backoff strategy
  * 93% test coverage

- BlocklistParser: Parse and validate IP addresses
  * Extract valid IPv4/IPv6 addresses from text
  * Skip CIDRs and malformed entries gracefully
  * Separate parsing from validation concerns
  * 100% test coverage

- BanExecutor: Ban execution with error handling
  * Ban IPs via fail2ban socket
  * Stop on JailNotFoundError (jail doesn't exist)
  * Continue on JailOperationError (individual ban failures)
  * 100% test coverage

- BlocklistImportWorkflow: Thin orchestrator
  * Coordinates the download → parse → ban → log flow
  * Pre-warms geo cache with newly banned IPs
  * 96% test coverage

- blocklist_service.py: Maintains public API
  * Source CRUD (create, read, update, delete)
  * URL validation and preview functionality
  * Scheduling configuration and import triggers
  * 92% test coverage

Benefits:
* Each component is independently testable with mock dependencies
* Error handling is explicit and localized
* Components can evolve independently
* Logging is contextual and clear
* Retry and transient error handling are isolated

Testing:
* All 36 existing blocklist_service tests pass
* All 13 blocklist import task tests pass
* Added 17 comprehensive component unit tests
* Combined 96%+ coverage on new modules
* Zero type errors in new code

Documentation:
* Updated Refactoring.md with detailed architecture notes
* Added component architecture diagram to Architekture.md
* Documented ownership and responsibilities of each component

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-27 18:34:11 +02:00
3bbf413c55 refactor: Make service dependencies explicit and injectable
Remove hidden cross-service coupling by making dependencies explicit through
dependency injection while maintaining backward compatibility via lazy imports.

Key changes:
- history_service and ban_service: Removed direct module-level imports of
  fail2ban_metadata_service, added optional service parameters to functions
- Added get_fail2ban_metadata_service() provider to dependencies.py
- Updated history router to inject Fail2BanMetadataService dependency
- history_service functions now use lazy imports in fallback paths for
  backward compatibility when service is not explicitly injected
- All test patches updated to use internal _get_fail2ban_db_path() helper
- jail_config_service and jail_service already follow best practices

This pattern prevents circular imports, makes services testable via explicit
mocking, and documents service dependencies clearly.

Fixes: Instructions.md #2 - Hidden cross-service coupling

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-27 18:26:08 +02:00
bc315b936b Refactor services and update documentation
- Refactor ban_service.py with improved error handling
- Refactor blocklist_service.py for better code organization
- Update geo_cache.py with performance improvements
- Update backend development guide and task documentation
- Update runner.csx script

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-26 20:27:04 +02:00
1d91e24a88 TASK-030: Secure IP geolocation with MMDB-primary resolver
Make MaxMind GeoLite2-Country MMDB the primary IP resolver (local, encrypted)
and demote ip-api.com to optional fallback only (disabled by default).

Changes:
- Add geoip_allow_http_fallback config flag (default False) to Settings
- Refactor GeoCache.lookup() and lookup_batch() to try MMDB first
- Update startup.py to pass config flag and log security warning when HTTP enabled
- Update all 49 tests to reflect new MMDB-primary strategy
- Add comprehensive geoip configuration section to Backend-Development.md
- Update Architekture.md to show MMDB + optional HTTP in system dependencies
- Update .env.example with BANGUI_GEOIP_DB_PATH and HTTP fallback flag

Security impact:
- 99% of IP addresses (successful MMDB lookups) now stay local, encrypted
- HTTP-only IPs are cached for 5 minutes to minimize external calls
- Operators must explicitly enable HTTP fallback (security-conscious default)
- GDPR/CCPA compliance: no PII sent over unencrypted networks by default

Fixes TASK-030: Resolved plaintext IP transmission to ip-api.com

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-26 15:31:39 +02:00
5d24780c63 TASK-028: Add exception logging to fire-and-forget asyncio.create_task()
- Create logged_task() helper in backend/app/utils/async_utils.py to wrap
  fire-and-forget coroutines with exception logging
- Ensures unhandled task exceptions are always logged to structlog instead of
  silently discarded (Python 3.11+ RuntimeWarning)
- Update ban_service.py to use logged_task() for geo_cache.lookup_batch()
  background resolution
- Add comprehensive tests for logged_task() in test_async_utils.py
- Document fire-and-forget task conventions in Backend-Development.md

The logged_task() wrapper catches any exception raised in a background task,
logs it with full traceback context and task name, and never re-raises.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-26 15:17:30 +02:00
a768a2d303 TASK-025: Remove HMAC bypass in unwrap_session_token
- Remove the early-return branch that skipped HMAC verification for unsigned tokens
- Raise ValueError if the signature separator is absent
- Update unwrap_session_token docstring to reflect mandatory signing requirement
- Add comprehensive session token signing documentation to Backend-Development.md
- Document the session token format, signing/verification pattern, and security rationale

All tokens must now carry a valid HMAC-SHA256 signature. Tokens without a
signature are rejected immediately. This removes the vulnerability where an
attacker with database access could bypass the HMAC layer by using raw tokens.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-26 15:02:02 +02:00
ec253d9b7a TASK-021: Implement atomic writes for set_jail_config_enabled and write_jail_config_file 2026-04-26 14:27:33 +02:00
d476e9d611 TASK-020: Fix log_target security vulnerability (defense in depth)
**Issue:**
- log_target accepted arbitrary paths, allowing authenticated users to write
  files as root via fail2ban (e.g., /etc/cron.d/bangui-pwned)
- fail2ban runs as root and opens files specified in log_target

**Solution:**
1. **Model layer validation:** Already existed in GlobalConfigUpdate, prevents
   invalid paths before reaching service
2. **Service layer validation:** Added defensive check in update_global_config()
   that validates log_target even if model validation is bypassed
3. **New validation helper:** Added validate_log_target() utility that accepts
   special values (STDOUT, STDERR, SYSLOG) or paths within allowed directories

**Changes:**
- app/utils/path_utils.py: Added validate_log_target() helper
- app/services/config_service.py: Added service-layer validation before
  sending command to fail2ban
- backend/tests: Fixed session_secret length issues in fixtures (min 32 chars)
- backend/tests: Added tests for valid special log targets
- Docs/Backend-Development.md: Documented log_target security requirements

**Test Coverage:**
- Model validation rejects /etc/passwd (existing test)
- Model validation accepts STDOUT, STDERR, SYSLOG special values
- Model validation accepts paths in allowed directories
- Service layer validation tested with special values

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-26 14:23:56 +02:00
4ceb11a4e3 TASK-018: Make config file writes atomic using write-to-temp + rename
- Replace Path.write_text() with tempfile.NamedTemporaryFile + os.replace()
  in _write_conf_file() and _create_conf_file()
- Ensures atomic operations on same filesystem (temp file in target.parent)
- Prevents config corruption if process is killed mid-write
- Follows existing pattern in jail_config_service.py
- Add proper cleanup of temp files on error with contextlib.suppress()
- Document atomic file write conventions in Backend-Development.md

This prevents fail2ban config files (especially jail.d/*.conf) from being
left in a truncated or corrupt state, which could disable active protection.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-26 14:11:18 +02:00
b6e8e3f5ff Clean up unused imports and remove completed task
- Remove TASK-016 from Docs/Tasks.md (completed)
- Remove unused imports from protocols.py (Iterable, BanIpCount)
- Remove unused imports from raw_config_io_service.py (asyncio, ConfigDirError, ConfigFileExistsError, ConfFileEntry)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-26 14:08:43 +02:00
5b24a9c142 TASK-011: Remove session token prefix from log output
Replace sensitive token fragments in structured logs with:
- login(): Use session_id=session.id (database row ID) instead of token_prefix
- logout(): Use token_hash (SHA256 one-way hash, first 12 chars) instead of token_prefix

This prevents partial token material leakage into log aggregation systems while
maintaining useful session correlation via hashed tokens or database IDs.

Also updated Backend-Development.md to clarify logging conventions for
sensitive data handling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-26 13:19:26 +02:00
4ab767e3d4 TASK-009: Mitigate SSRF vulnerability in blocklist URL validation
- Change BlocklistSourceCreate.url from str to AnyHttpUrl (Pydantic type)
  - Rejects non-http schemes (file://, ftp://, etc.) at model boundary

- Add is_private_ip() utility to detect RFC 1918 private ranges:
  - 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 (RFC 1918)
  - 127.0.0.0/8, ::1/128 (loopback)
  - 169.254.0.0/16, fe80::/10 (link-local)
  - IPv6 site-local, multicast, and reserved ranges

- Add async validate_blocklist_url() function:
  - Resolves hostname via DNS using loop.run_in_executor()
  - Rejects if hostname resolves to private/reserved IP
  - Raises ValueError on validation failure

- Integrate validation into service layer:
  - create_source() calls validate_blocklist_url() before persist
  - update_source() conditionally validates if url provided
  - Both raise ValueError on failure

- Update router endpoints with error handling:
  - create_blocklist() and update_blocklist() catch ValueError
  - Return HTTP 400 Bad Request with descriptive error message

- Add comprehensive test coverage (9 new SSRF tests):
  - file://, ftp://, localhost, 127.0.0.1, 192.168.x.x
  - 10.x.x.x, 172.16.x.x, 169.254.x.x (link-local)
  - Valid public URLs (passes validation)
  - All 36 service tests passing

- Update documentation:
  - Features.md: Document URL validation constraints
  - Backend-Development.md: Add SSRF prevention pattern section

Fixes SSRF vulnerability where authenticated users could supply
file://, ftp://, or private IP URLs and the backend would fetch them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-26 12:57:23 +02:00
4b8af1d43a Fix import formatting and sorting
Ruff formatting fixes for import organization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-25 18:54:57 +02:00
1a3401f418 T-10: Fix get_geo_batch_lookup for proper injection with GeoCache instance
Instead of returning a bound method (geo_cache.lookup_batch), now inject
the GeoCache instance directly into routers and services. This provides
proper runtime isolation since T-04 made GeoCache a proper object.

Changes:
- Remove get_geo_batch_lookup() dependency provider
- Add GeoCacheDep type alias for injecting GeoCache instances
- Update all routers (bans, blocklist, dashboard, jails) to use GeoCacheDep
- Update ban_service, blocklist_service, jail_service to accept GeoCache
- Update service protocols to match new signatures
- Update docstrings to reference GeoCache methods instead of module functions

All callers now call geo_cache.lookup_batch(...) directly instead of
geo_batch_lookup(...), providing real dependency injection with proper
testing isolation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-25 18:53:47 +02:00
ac2028e1c2 Fix: Consolidate divergent _since_unix implementations (T-09)
Consolidate the two divergent implementations of _since_unix from ban_service.py
and history_service.py into a single shared utility function in time_utils.py.

Changes:
- Move _since_unix to app/utils/time_utils.py with consistent time.time() approach
- Move TIME_RANGE_SLACK_SECONDS constant to app/utils/constants.py
- Update ban_service.py to import since_unix from time_utils
- Update history_service.py to import since_unix from time_utils
- Both services now use the same window boundary calculation with 60-second slack
- Add comprehensive tests for the shared since_unix function
- Document timestamp handling rationale in Backend-Development.md

This ensures dashboard and history queries return consistent row counts for the
same time range by using the same timestamp calculation and slack window across
all services.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-25 18:44:59 +02:00
420ea18fd9 Refactor backend services and utilities
- Update service layer implementations
- Improve configuration handling utilities
- Update documentation tasks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-25 18:39:30 +02:00
83452ffc23 Refactor backend services and jail configuration
- Refactor action_config_service, filter_config_service, jail_config_service, and jail_service
- Add jail_socket utility module for socket communication
- Update test_jail_service with new test cases
- Update architecture and task documentation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-25 18:34:03 +02:00
d467190eb1 Refactor geo caching and service layer tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-25 18:15:31 +02:00
654dbdb000 T-04: Encapsulate geo_service module-level mutable state in GeoCache class
Create GeoCache class with all mutable state as instance attributes:
- _cache, _neg_cache, _dirty, _geoip_reader, _geoip_initialized, _cache_lock
- All public methods: lookup(), lookup_batch(), lookup_cached_only(), flush_dirty(), load_from_db(), clear(), etc.

Initialization & Dependency Injection:
- Instantiate GeoCache in startup.py and store on app.state.geo_cache
- Add get_geo_cache() dependency function in dependencies.py
- Inject into routes and tasks via FastAPI's dependency system

Backward Compatibility:
- Maintain module-level functions in geo_service.py as deprecated wrappers
- All old callers continue to work through _default_geo_cache instance
- Remove test-escape-hatch functions (clear_cache, clear_neg_cache moved to methods)

Background Tasks:
- Update geo_cache_flush.py and geo_re_resolve.py to receive GeoCache instance
- Tasks now operate on injected instance rather than module globals

Tests:
- Refactor test_geo_service.py with geo_cache fixture providing fresh instances
- Update patch paths to target GeoCache methods correctly
- Fix internal state assertions to access instance attributes

Documentation:
- Update Architekture.md to document GeoCache as managed stateful service
- Describe cache lifecycle (load on startup, flush periodically, re-resolve stale)
- Note process-local limitations for multi-worker deployments

Fixes violation of Single Responsibility Principle: module no longer owns both
lookup logic and cache lifecycle management. Cache is now a first-class
injectable service with transparent lifecycle.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 16:18:09 +02:00
fdfd24508f Refactor backend services and routers
- Reorganized dashboard router with improved structure
- Enhanced ban_service with better separation of concerns
- Updated history service with cleaner logic
- Improved constants and configuration handling
- Updated documentation of completed tasks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 16:06:10 +02:00
b634ce876a refactor: Extract fail2ban response utilities into shared module
Consolidate duplicate _ok(), _to_dict(), ensure_list(), and is_not_found_error()
functions from 6 service modules into a single canonical implementation at
backend/app/utils/fail2ban_response.py.

Changes:
- Create fail2ban_response.py with canonical implementations
- Remove local duplicates from: ban_service, jail_service, config_service,
  health_service, server_service, config_file_utils
- Update all imports to use shared module
- Add comprehensive docstrings and examples
- Update Architecture.md and Backend-Development.md documentation

Benefits:
- Single source of truth for response parsing logic
- Eliminates code duplication across service layer
- Improves maintainability and consistency
- Enables centralized bug fixes and improvements

Tests: All 228 service tests passing, no regressions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 15:11:21 +02:00
01f2e07921 Extract shared raw config file helpers and simplify raw_config_io_service 2026-04-18 20:18:54 +02:00
c1f188643c Move geo cache commit handling into repository layer 2026-04-18 20:10:05 +02:00
52e08e17a4 Guard geo_service.init_geoip against repeated initialization 2026-04-18 19:54:05 +02:00
db5b4cb77e Add settings and history archive repository protocols and DI support 2026-04-17 20:54:08 +02:00
7055971163 Harden preview_log path validation and add regression test 2026-04-17 20:37:14 +02:00
4754f1407e Mark task 19 done and centralize map-color thresholds in settings_service 2026-04-17 17:04:09 +02:00