CRITICAL FIX: Background tasks (especially blocklist_import) crashed mid-execution,
leaving partial state. On retry, the same bans were applied again, causing duplicates.
Solution: Content-hash based operation tracking for blocklist imports:
- Added import_runs table (migration 6) to track operations by source + content hash
- Before banning, check if this exact content has already been imported
- If completed: skip banning (already done), optionally re-warm cache
- If new or failed: proceed with ban and mark as completed or failed
Changes:
- Database: Migration 6 adds import_runs table with operation state tracking
- Model: Added ImportRunEntry for import run records
- Repository: New import_run_repo module with CRUD operations
- Workflow: Updated blocklist_import_workflow to check operation history before banning
- Dependencies: Registered import_run_repo for dependency injection
- Tests: Added test_import_source_idempotent_on_retry and test_import_source_different_content_not_reused
- Documentation: Added Task Idempotency section to Backend-Development.md
Verification:
- All 7 import tests pass (5 existing + 2 new idempotency tests)
- Type checking: mypy --strict ✅
- Linting: ruff ✅
- No API changes, backwards compatible via automatic migration
Fixes: Background tasks not idempotent #CRITICAL
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Models in app/models/ are now pure data classes with no cross-layer dependencies.
This ensures the models layer remains a true leaf node in the dependency graph.
Changes:
- Create app/models/_common.py with shared types (TimeRange, bucket_count, constants)
- Move TimeRange and time-range constants from ban.py to _common.py
- Update history.py, routers, and services to import from _common.py
- Remove imports from app.config and app.utils from config.py models
- Move field validators from models to router layer:
- Add log_target validation in config_misc router
- Add log_path validation in jail_config router
- Update test_models.py to reflect validators moved to router layer
- Update documentation (Architekture.md, Backend-Development.md) with model layering rules
- Fix import ordering and type annotations in affected files
Model layering rule: Models may only import from:
✓ Standard library and third-party packages (Pydantic, typing)
✓ Other models in app/models/ (sibling models)
✓ app.models.response (response envelopes)
✗ app.services, app.config, app.utils, or any application layer
Validation requiring app-level state (settings, allowed directories) now happens
at the router or service layer, not in model validators.
Fixes: Models were not true leaf nodes due to circular imports and app-layer dependencies
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Align frontend and backend error observability with correlation IDs and
structured telemetry for distributed tracing across systems.
Backend changes:
- Add CorrelationIdMiddleware to generate/extract correlation IDs
- Include correlation_id in all ErrorResponse objects
- Store correlation ID in structlog contextvars for automatic inclusion in logs
- Add correlation ID to response headers (X-Correlation-ID)
Frontend changes:
- API client automatically generates session-scoped UUID4 and includes
X-Correlation-ID header in all requests
- Extract correlation ID from API error responses
- Update error handlers to use telemetry with correlation IDs
- Add telemetry logging to ErrorBoundary, PageErrorBoundary, SectionErrorBoundary
- Implement redaction utilities for privacy-safe logging of sensitive data
Documentation:
- Add observability guidelines to Web-Development.md
* Correlation ID usage patterns
* Privacy & security best practices
* Telemetry event structure
* Redaction utilities for sensitive data
- Add distributed tracing architecture section to Architecture.md
* Correlation ID flow across frontend/backend
* Example troubleshooting scenario
* Implementation details for future enhancements
Testing:
- Add comprehensive tests for correlation middleware
- Update error boundary tests to verify telemetry integration
- Verify TypeScript and ESLint pass with no warnings
Fixes: Issue #40 - Frontend and backend observability are not aligned
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit standardizes how API responses are wrapped, solving issue #24.
Problem:
- Inconsistent response envelopes (jails vs items vs bans vs no wrapper)
- Frontend required multiple field name variants
- Integration bugs from branching logic
- No clear pattern for different response types
Solution:
- Created response.py with base classes: PaginatedListResponse,
CollectionResponse, CommandResponse
- Standardized all list/collection responses to use 'items' field
- Domain-specific field names for detail and aggregation responses
- Updated all backends routers and mappers
- Updated frontend types and hooks to match
Changes:
Backend:
- backend/app/models/response.py (new): Base response models
- backend/app/models/ban.py: Updated responses to inherit from bases
- backend/app/models/jail.py: Updated JailListResponse, JailCommandResponse
- backend/app/models/config.py: Updated collection responses
- backend/app/services/jail_service.py: Updated return statements
- backend/app/mappers/ban_mappers.py: Updated 'bans' to 'items'
- backend/tests/test_mappers/test_ban_mappers.py: Updated tests
Frontend:
- frontend/src/types/jail.ts: Updated response interfaces
- frontend/src/types/config.ts: Updated response interfaces
- frontend/src/hooks/useActiveBans.ts: Updated selector
- frontend/src/hooks/useJailList.ts: Updated selector
- frontend/src/hooks/useJailConfigs.ts: Updated selector
- frontend/src/hooks/useConfigActiveStatus.ts: Updated field access
- frontend/src/hooks/useJailAdmin.ts: Updated field access
Documentation:
- Docs/Backend-Development.md: Added § 4.1 API Response Envelope Policy
The policy defines:
1. Paginated lists use PaginatedListResponse (items, total, page, page_size)
2. Non-paginated collections use CollectionResponse (items, total)
3. Detail responses use entity-specific field names (jail, status, settings)
4. Command responses use CommandResponse (message, success, optional target)
5. Aggregations use domain-specific fields (jails, countries, buckets, bans)
All responses now follow one of these patterns, reducing frontend complexity.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Convert inconsistent modeling style to standardized Pydantic models for all
external-facing data structures while maintaining TypedDict compatibility where
appropriate for internal layer-private structures.
Changes:
- Converted IpLookupResult TypedDict to use IpLookupResponse Pydantic model
in jail_service.lookup_ip() for consistency with routers
- Added GeoCacheEntry Pydantic model for geo cache repository rows
- Converted GeoCacheRow TypedDict to use GeoCacheEntry alias
- Converted ImportLogRow TypedDict to use ImportLogEntry alias
- Updated routers and services to work with Pydantic models
- Updated all tests to use Pydantic model field access (attributes)
instead of dict subscripting
Documentation:
- Added 'Model Type Usage by Layer' section to Backend-Development.md
- Defines when TypedDict is allowed (internal structures) vs Pydantic
(external-facing, cross-boundary data)
- Provides clear guidance on modeling conventions per layer
Benefits:
- Consistent validation and serialization behavior
- Better IDE support and type checking
- Clearer separation of concerns by layer
- Reduced maintenance cost from mixed validation approaches
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add ban domain model for core business logic separation
- Implement mapper pattern for DTO/domain conversions
- Update ban service with new domain-driven approach
- Refactor router endpoints to use new architecture
- Add comprehensive mapper tests
- Update documentation with architecture changes
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fixes a critical security vulnerability where the session token was
being returned in the JSON response body of POST /api/auth/login.
This exposed the token to JavaScript, allowing malicious scripts to
steal it and bypass the HttpOnly cookie protection.
Changes:
- Backend: Remove 'token' field from LoginResponse model (auth.py)
- Backend: Update login() endpoint to return only 'expires_at'
- Frontend: Update LoginResponse type to exclude 'token' field
- Backend: Update test helper _login() to extract token from cookie
- Backend: Update test cases to verify token is NOT in response body
- Documentation: Add section 'Authentication Endpoints' in Backend-Development.md
- Documentation: Update Web-Development.md to explain HttpOnly cookie benefits
Security benefit: Session tokens are now only accessible via HttpOnly
cookies, protected from JavaScript access, XSS attacks, and malicious
third-party scripts. The frontend continues to use only the cookie for
authentication.
All auth tests pass (23 tests). Type checking and linting pass with
zero errors.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bcrypt silently truncates passwords at 72 bytes, so passwords longer than 72
characters provide no additional security. This commit enforces the 72-byte
maximum across the authentication and setup flows.
Changes:
- Add max_length=72 to LoginRequest.password and SetupRequest.master_password
- Update field validator in SetupRequest to explicitly check max_length
- Add comprehensive tests for password length validation (6 new test cases)
- Document the 72-byte limitation in Features.md (master password options)
- Add new section 12 'Password Hashing' in Backend-Development.md explaining:
- The bcrypt truncation behavior
- Why the limit is enforced
- The validation flow from frontend to backend
- What happens when passwords exceed the limit
All existing tests pass, no regressions introduced.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Extract path validation logic into shared helper function in
backend/app/utils/path_utils.py (validate_log_path)
- Refactor AddLogPathRequest to use the helper function
- Apply the same validation to DELETE /api/config/jails/{name}/logpath
endpoint by validating the log_path query parameter
- Return HTTP 422 with descriptive error if validation fails
- Add comprehensive unit tests for path validation
- Update Backend-Development.md with usage examples
This prevents path-traversal attacks on the delete_log_path endpoint
by ensuring all log paths are within allowlisted directories.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add LogLevel Literal type: CRITICAL, ERROR, WARNING, NOTICE, INFO, DEBUG
- Add log_target validation to accept special values (STDOUT, STDERR, SYSLOG)
or validated file paths within allowed directories
- Update GlobalConfigResponse to use LogLevel type
- Add field_validator for log_target in both GlobalConfigUpdate and
GlobalConfigResponse following the same pattern as AddLogPathRequest
- Add @autouse fixture to test_config_service.py to mock get_settings
- Update existing tests to use uppercase log level values
- Add 12 comprehensive tests for new validation in test_models.py
- Update Features.md to document valid log_target and log_level values
- Add section to Backend-Development.md documenting Literal types and
field_validator patterns with examples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restrict monitored log paths to a configurable allowlist of safe directories
to prevent authenticated users from instructing fail2ban to monitor arbitrary
files on the system, which could leak contents via fail2ban logging.
Changes:
- Add 'allowed_log_dirs' setting to Settings (defaults to /var/log, /config/log)
- Add @field_validator to AddLogPathRequest to validate log paths at request time
- Validator resolves paths to canonical form and checks against allowed prefixes
- Use Path.is_relative_to() to prevent prefix bypass attacks like /var/log_evil
- Add comprehensive tests for valid/invalid paths and symlink handling
- Update Features.md and Backend-Development.md with security documentation
Security improvements:
- Blocks access to sensitive files (/etc/shadow, /etc/passwd, etc.)
- Resolves symlinks before validation to prevent escape routes
- Uses proper path comparison instead of string prefix matching
- Configurable via BANGUI_ALLOWED_LOG_DIRS environment variable
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add has_local_override field to InactiveJail model
- Update _build_inactive_jail and list_inactive_jails to compute the field
- Add delete_jail_local_override() service function
- Add DELETE /api/config/jails/{name}/local router endpoint
- Surface has_local_override in frontend InactiveJail type
- Show Deactivate Jail button in JailsTab when has_local_override is true
- Add tests: TestBuildInactiveJail, TestListInactiveJails, TestDeleteJailLocalOverride
Task 1: Move Configuration to last position in sidebar NAV_ITEMS
Task 2: Add automatic rollback when jail activation fails
- Back up .local override file before writing
- Restore original file (or delete) on reload failure, health-check
failure, or jail not appearing post-reload
- Return recovered=True/False in JailActivationResponse
- Show warning/critical banner in ActivateJailDialog based on recovery
- Add _restore_local_file_sync and _rollback_activation_async helpers
- Add 3 new tests: rollback on reload failure, health-check failure,
and double failure (recovered=False)
Task 3: Color pie chart legend labels to match their slice color
- legendFormatter now returns ReactNode with span style={{ color }}
- Import LegendPayload from recharts/types/component/DefaultLegendContent
Task 2: adds a new Log tab to the Configuration page.
Backend:
- New Pydantic models: Fail2BanLogResponse, ServiceStatusResponse
(backend/app/models/config.py)
- New service methods in config_service.py:
read_fail2ban_log() — queries socket for log target/level, validates the
resolved path against a safe-prefix allowlist (/var/log) to prevent
path traversal, then reads the tail of the file via the existing
_read_tail_lines() helper; optional substring filter applied server-side.
get_service_status() — delegates to health_service.probe() and appends
log level/target from the socket.
- New endpoints in routers/config.py:
GET /api/config/fail2ban-log?lines=200&filter=...
GET /api/config/service-status
Both require authentication; log endpoint returns 400 for non-file log
targets or path-traversal attempts, 502 when fail2ban is unreachable.
Frontend:
- New LogTab.tsx component:
Service Health panel (Running/Offline badge, version, jail count, bans,
failures, log level/target, offline warning banner).
Log viewer with color-coded lines (error=red, warning=yellow,
debug=grey), toolbar (filter input + debounce, lines selector, manual
refresh, auto-refresh with interval selector), truncation notice, and
auto-scroll to bottom on data updates.
fetchData uses Promise.allSettled so a log-read failure never hides the
service-health panel.
- Types: Fail2BanLogResponse, ServiceStatusResponse (types/config.ts)
- API functions: fetchFail2BanLog, fetchServiceStatus (api/config.ts)
- Endpoint constants (api/endpoints.ts)
- ConfigPage.tsx: Log tab added after existing tabs
Tests:
- Backend service tests: TestReadFail2BanLog (6), TestGetServiceStatus (2)
- Backend router tests: TestGetFail2BanLog (8), TestGetServiceStatus (3)
- Frontend: LogTab.test.tsx (8 tests)
Docs:
- Features.md: Log section added under Configuration View
- Architekture.md: config.py router and config_service.py descriptions updated
- Tasks.md: Task 2 marked done
Task 1: Backend/LogEncoding/DatePattern dropdowns in JailConfigDetail
- Added BACKENDS, LOG_ENCODINGS, DATE_PATTERN_PRESETS constants
- Backend and Log Encoding: <Input readOnly> → <Select> (editable, auto-saves)
- Date Pattern: <Input> → <Combobox freeform> with presets
- Extended JailConfigUpdate model (backend, log_encoding) and service
- Added readOnly prop to JailConfigDetail (all fields, toggles, buttons)
- Extended RegexList with readOnly prop
Task 2: Fix raw action/filter config always blank
- Added key={selectedAction.name} to ActionDetail in ActionsTab
- Added key={selectedFilter.name} to FilterDetail in FiltersTab
Task 3: Inactive jail full GUI same as active jails
- Extended InactiveJail Pydantic model with all config fields
- Added _parse_time_to_seconds helper to config_file_service
- Updated _build_inactive_jail to populate all extended fields
- Extended InactiveJail TypeScript type to match
- Rewrote InactiveJailDetail to reuse JailConfigDetail (readOnly=true)
Task 4: Fix banaction interpolation error when activating jails
- _write_local_override_sync now includes banaction=iptables-multiport
and banaction_allports=iptables-allports in every .local file
- PUT /api/config/filters/{name}: updates failregex/ignoreregex/datepattern/
journalmatch in filter.d/{name}.local; validates regex via re.compile();
supports ?reload=true
- POST /api/config/filters: creates filter.d/{name}.local from FilterCreateRequest;
returns 409 if file already exists
- DELETE /api/config/filters/{name}: deletes .local only; returns 409 for
conf-only (readonly) filters
- POST /api/config/jails/{name}/filter: assigns filter to jail by writing
'filter = {name}' to jail.d/{jail}.local; supports ?reload=true
- New models: FilterUpdateRequest, FilterCreateRequest, AssignFilterRequest
- New service helpers: _safe_filter_name, _validate_regex_patterns,
_write_filter_local_sync, _set_jail_local_key_sync
- Fixed .local-only filter discovery in _parse_filters_sync (5-tuple return)
- Fixed get_filter extension stripping (.local is 6 chars not 5)
- Renamed file_config.py raw-write routes to /raw suffix
(PUT /filters/{name}/raw, POST /filters/raw) to avoid routing conflicts
- Full service + router tests; all 930 tests pass
- Add list_filters() and get_filter() to config_file_service.py:
scans filter.d/, parses [Definition] + [Init] sections, merges .local
overrides, and cross-references running jails to set active/used_by_jails
- Add FilterConfig.active, used_by_jails, source_file, has_local_override
fields to the Pydantic model; add FilterListResponse and FilterNotFoundError
- Add GET /api/config/filters and GET /api/config/filters/{name} to config.py
- Remove the shadowed GET /api/config/filters list route from file_config.py;
rename GET /api/config/filters/{name} raw variant to /filters/{name}/raw
- Update frontend: fetchFilterFiles() adapts FilterListResponse -> ConfFilesResponse;
add fetchFilters() and fetchFilter() to api/config.ts; remove unused
fetchFilterFiles/fetchActionFiles calls from useConfigActiveStatus
- Fix ConfigPageLogPath test mock to include fetchInactiveJails and related
exports introduced by Stage 1
- Backend: 169 tests pass, mypy --strict clean, ruff clean
- Frontend: 63 tests pass, tsc --noEmit clean, eslint clean
- Backend: config_file_service.py parses jail.conf/jail.local/jail.d/*
following fail2ban merge order; discovers jails not running in fail2ban
- Backend: 3 new API endpoints (GET /jails/inactive, POST /jails/{name}/activate,
POST /jails/{name}/deactivate); moved /jails/inactive before /jails/{name}
to fix route-ordering conflict
- Frontend: ActivateJailDialog component with optional parameter overrides
- Frontend: JailsTab extended with inactive jail list and InactiveJailDetail pane
- Frontend: JailsPage JailOverviewSection shows inactive jails with toggle
- Tests: 57 service tests + 16 router tests for all new endpoints (all pass)
- Docs: Features.md, Architekture.md, Tasks.md updated; Tasks 1.1-1.5 marked done
- Add conffile_parser.py: reads, writes and manipulates fail2ban .conf
files while preserving comments and section structure
- Extend config models with ActionConfig, FilterConfig, ConfFileContent,
and related Pydantic schemas for jails, actions, and filters
- Add use_dns and prefregex fields to JailConfig model (backend + frontend types)
- Add prefregex to JailConfigUpdate; validate as regex before writing
- Fetch usedns and prefregex in get_jail_config via asyncio.gather
- Write usedns and prefregex in update_jail_config
- ConfigPage JailAccordionPanel: editable date_pattern input, dns_mode
Select dropdown (yes/warn/no/raw), and prefregex input
- 8 new service unit tests + 3 new router integration tests
- 628 tests pass; 85% line coverage; ruff/mypy/tsc/eslint clean
- Backend: Add BantimeEscalation + BantimeEscalationUpdate Pydantic models
to app/models/config.py; add bantime_escalation field to Jail in jail.py
- Backend: jail_service.get_jail_detail() fetches 7 bantime.* socket commands
(increment, factor, formula, multipliers, maxtime, rndtime, overalljails)
and populates bantime_escalation on the returned Jail object
- Backend: config_service.get_jail_config() fetches same 7 commands;
update_jail_config() writes escalation fields when provided
- Frontend: Add BantimeEscalation + BantimeEscalationUpdate interfaces to
types/config.ts; extend JailConfig + JailConfigUpdate; extend Jail in
types/jail.ts
- Frontend: JailDetailPage.tsx adds BantimeEscalationSection component that
renders only when increment is enabled (shows factor, formula, multipliers,
max_time, rnd_time, overall_jails)
- Frontend: ConfigPage.tsx JailAccordionPanel adds full escalation edit form
(Switch for enable/disable, number inputs for factor/max_time/rnd_time,
text inputs for formula/multipliers, Switch for overall_jails);
handleSave includes bantime_escalation in the JailConfigUpdate payload
- Tests: Update ConfigPageLogPath.test.tsx mock to include bantime_escalation:null
- Docs: Mark Task 6 as DONE in Tasks.md
Task 4 (Better Jail Configuration) implementation:
- Add fail2ban_config_dir setting to app/config.py
- New file_config_service: list/view/edit/create jail.d, filter.d, action.d files
with path-traversal prevention and 512 KB content size limit
- New file_config router: GET/PUT/POST endpoints for jail files, filter files,
and action files; PUT .../enabled for toggle on/off
- Extend config_service with delete_log_path() and add_log_path()
- Add DELETE /api/config/jails/{name}/logpath and POST /api/config/jails/{name}/logpath
- Extend geo router with re-resolve endpoint; add geo_re_resolve background task
- Update blocklist_service with revised scheduling helpers
- Update Docker compose files with BANGUI_FAIL2BAN_CONFIG_DIR env var and
rw volume mount for the fail2ban config directory
- Frontend: new Jail Files, Filters, Actions tabs in ConfigPage; file editor
with accordion-per-file, editable textarea, save/create; add/delete log paths
- Frontend: types in types/config.ts; API calls in api/config.ts and api/endpoints.ts
- 63 new backend tests (test_file_config_service, test_file_config, test_geo_re_resolve)
- 6 new frontend tests in ConfigPageLogPath.test.tsx
- ruff, mypy --strict, tsc --noEmit, eslint: all clean; 617 backend tests pass
- backend: GET /api/dashboard/bans/by-jail endpoint
- JailBanCount + BansByJailResponse Pydantic models in ban.py
- bans_by_jail() service function with origin filter support
- Route added to dashboard router
- 17 new tests (7 service, 10 router); full suite 497 passed, 83% coverage
- frontend: JailDistributionChart component
- JailBanCount / BansByJailResponse types in types/ban.ts
- dashboardBansByJail endpoint constant in api/endpoints.ts
- fetchBansByJail() in api/dashboard.ts
- useJailDistribution hook in hooks/useJailDistribution.ts
- JailDistributionChart component (horizontal bar chart, Recharts)
- DashboardPage: full-width Jail Distribution section below Top Countries
- Send fail2ban's `unban --all` command via new `unban_all_ips()` service
function; returns the count of unbanned IPs
- Add `UnbanAllResponse` Pydantic model (message + count)
- Add `DELETE /api/bans/all` router endpoint; handles 502 on socket error
- Frontend: `bansAll` endpoint constant, `unbanAllBans()` API call,
`UnbanAllResponse` type, `unbanAll` action in `useActiveBans` hook
- JailsPage: "Clear All Bans" button (visible when bans > 0) with a
Fluent UI confirmation Dialog before executing the operation
- 7 new tests (3 service, 4 router); 440 total pass, 82% coverage
When the most recent scheduled import completed with errors, surface the
failure in the persistent app shell:
- A warning MessageBar appears at top of main content area
- An amber badge is rendered on the Blocklists sidebar nav item
Backend: add last_run_errors: bool | None to ScheduleInfo model and
populate it in get_schedule_info() from the latest import_log row.
Frontend: extend ScheduleInfo type, add useBlocklistStatus polling hook,
wire both indicators into MainLayout.
Tests: 3 new service tests + 1 new router test (433 total, all pass).
- Task 1: Mark imported blocklist IP addresses
- Add BanOrigin type and _derive_origin() to ban.py model
- Populate origin field in ban_service list_bans() and bans_by_country()
- BanTable and MapPage companion table show origin badge column
- Tests: origin derivation in test_ban_service.py and test_dashboard.py
- Task 2: Add origin filter to dashboard and world map
- ban_service: _origin_sql_filter() helper; origin param on list_bans()
and bans_by_country()
- dashboard router: optional origin query param forwarded to service
- Frontend: BanOriginFilter type + BAN_ORIGIN_FILTER_LABELS in ban.ts
- fetchBans / fetchBansByCountry forward origin to API
- useBans / useMapData accept and pass origin; page resets on change
- BanTable accepts origin prop; DashboardPage adds segmented filter
- MapPage adds origin Select next to time-range picker
- Tests: origin filter assertions in test_ban_service and test_dashboard
- Implement ban model, service, and router endpoints in backend
- Add ban table component and dashboard integration in frontend
- Update ban-related types and API endpoints
- Add comprehensive tests for ban service and dashboard router
- Update documentation (Features, Tasks, Architecture, Web-Design)
- Clean up old fail2ban configuration files
- Update Makefile with new commands
