Commit Graph

601 Commits

Author SHA1 Message Date
23c3a0d9e6 feat: add e2e test suite with Robot Framework
Add e2e/ dir with Robot Framework tests for page loading, ban records,
blocklist import, config edit. Add requirements.txt. Update Makefile with
test commands. Update .gitignore, backend docs, testing requirements docs.
2026-05-04 08:29:12 +02:00
5fa67d3288 backup
Co-authored-by: Copilot <copilot@github.com>
2026-05-04 08:16:20 +02:00
744275d17f backup 2026-05-04 07:20:20 +02:00
58173bd6a9 backup 2026-05-04 07:20:16 +02:00
69e1726045 Refactor data fetching hooks, add page size lint test
- Simplify useFetchData: remove unused URL building logic
- Add usePolledData initial implementation
- Add router page_size param validation test
- Update API reference docs
- Clean up tasks doc
2026-05-04 06:48:24 +02:00
0a3f9c6c16 refactor(backend): external logging metrics, required mode, health checks
- Add external_logging_init_failures counter
- Add external_log_required flag, raise if init fails and required
- Health endpoint: add external_logging status check
- Blocklist service: enrich with metadata fields, update import logic
- Health check task: add runtime_state dependency, fix return typing
- Metrics: add Histogram for request latencies
- Frontend: align BlocklistImportLogSection props
- Docs: update deployment guide, remove stale tasks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-04 03:45:13 +02:00
42e177e6ea feat(frontend): add ignoreCancellation option for background tasks
Allow useNavigationAbortSignal to opt out of navigation-based abort
for long-lived background tasks like polling. Set ignoreCancellation: true
to keep requests alive across route changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-04 02:57:56 +02:00
eb339efcfd Add Kubernetes liveness/readiness probes and middleware order validation
- Split /health into /health/live (liveness) and /health/ready (readiness)
  following Kubernetes conventions. Combined /health retained for backward
  compatibility with existing Docker HEALTHCHECK definitions.
- Add ReadyCheck and ReadyResponse models for structured readiness output.
- Add _assert_middleware_order() startup check enforcing:
  RateLimit → Csrf → CorrelationId middleware chain.
- Register CorrelationIdMiddleware, CsrfMiddleware, RateLimitMiddleware
  in create_app() with documented required order (reverse of processing).
- Add correlation.py, csrf.py, rate_limit.py middleware modules.
- Add health probe tests in test_health_probes.py.
- Update test_main.py with middleware order assertion tests.
- Update frontend useFetchData hook tests.
- Docs: update Deployment.md with Kubernetes probe config examples.
2026-05-04 02:42:09 +02:00
65fe747cba feat(backend): add deprecation middleware and API versioning support
- Add deprecation middleware for warning headers on sunset endpoints
- Add jails_v2 router for API v2 migration path
- Update CI workflow with new test coverage
- Update API versioning documentation
- Remove completed tasks from Tasks.md
2026-05-04 00:03:52 +02:00
c8b48b5b65 fix(api): correlation ID survives HMR; fix endpoint template literal typos
- client.ts: store correlation ID in sessionStorage so HMR (module re-eval)
  does not generate a new ID mid-session; add clearSessionCorrelationId()
- endpoints.ts: fix 3 template literal trailing-quote bugs (missing ')' chars);
  replace template literals with string concat for encodeURIComponent calls
- AuthProvider.tsx: call clearSessionCorrelationId() on logout
- App.tsx: reorder ThemeProvider import before AuthProvider per PROVIDER_ORDER.md;
  indent Routes inside AuthProvider to match expected tree structure
- Tasks.md: update task status
- providerTreeOrder.test.tsx: add integration tests for provider nesting order

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 23:35:18 +02:00
fc57c83f79 refactor: split pagination logic from response models
- Extract pagination logic to separate util module
- Update response models to use new pagination util
- Fix pagination calculation edge cases

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 22:57:21 +02:00
b2747381ec Remove Issue #51 from Tasks.md (type-enforcement tracked elsewhere) 2026-05-03 22:47:24 +02:00
edebf1a339 feat(services): add ErrorContract enum and PartialResult type
Add typed wrappers for error handling patterns in error_handling.py:

- ErrorContract(enum): machine-checkable pattern selector with
  from_value() helper and string constants matching the existing
  ABORT_ON_ERROR/RETURN_DEFAULT/PARTIAL_RESULT module-level values
- ErrorEntry: typed error container for PARTIAL_RESULT (context + cause)
- PartialResult[T]: typed result wrapper for PARTIAL_RESULT operations

Existing string constants preserved for backward compat.
Updated module docstring with type annotation table and examples.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 22:46:47 +02:00
a2afec2d1e Remove completed Issue #50 navigation abort signal task
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 22:43:19 +02:00
52a70c3eea Add import-linter boundary to forbid routers importing app.dependencies
Issue #51: Enforce repository boundary at CI level using import-linter.
Contract 'forbid_router_db_import' checks that app.routers never imports
app.dependencies directly, keeping the DB access path through service
contexts only.

- Add import-linter>=2.0.0 to dev dependencies (backend/pyproject.toml)
- Configure [tool.importlinter] with package_root and root_packages
- Add [[tool.importlinter.contracts]] with type='forbidden', source
  app.routers, forbidden app.dependencies
- Add 'Import Boundary' CI job (import-linter)
- Document import-linter in CONTRIBUTING.md code quality table

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 22:42:46 +02:00
3376009903 fix(nav): move AbortController creation synchronously in render
Previously the AbortController was created inside useEffect, which runs
after render. This meant requests initiated during render received
the previous cycle's (possibly aborted) signal and were cancelled
immediately instead of completing.

Now the controller is created synchronously when pathname changes, before
any request can be initiated on the new route. The old controller is
aborted in the same conditional block, before the new one is created.

Side effect: removed resolved Issue #49 from Tasks.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 22:37:07 +02:00
7fcfc14199 fix(auth): dedupe handler + error utils refactor
- Add 401/403 dedup guard to API client to prevent double logout
- Extract fetchError util: isAuthError + getErrorMessage
- AuthProvider uses new error utils, removes duplicate logic
- Remove completed task docs from Tasks.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 22:13:12 +02:00
dafe8d61e2 feat(security): add CSRF header constants and security-headers endpoint
Move X-BanGUI-Request header name/value to backend/app/utils/constants.py as single source of truth. Add GET /api/v1/config/security-headers endpoint. Update csrf middleware, frontend api client, and docs to use shared constants.
2026-05-03 22:06:43 +02:00
cee3daffc1 fix: enforce PRAGMA query_only on fail2ban DB and refactor CSRF cookie name
- Add _acquire_readonly_connection() that applies PRAGMA query_only=ON after connect
- Verify PRAGMA value back to catch URI flag bypasses
- Wrap in async context manager _readonly_connection() used by all repo methods
- Replace hardcoded '_SESSION_COOKIE_NAME' in CSRF middleware with import from
  app.utils.constants
- Remove completed Issues #45 and #46 from Docs/Tasks.md (Issue #46 now fixed,
  #45 cache invalidation deferred to auth refactor branch)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 21:47:42 +02:00
1c3dff31e8 feat(rate-limiting): add per-bucket limits and startup validation
- Add per-bucket rate limit config (ban, unban, import, config, jail, filter, action)
- Add process-local warning at startup for multi-worker deployments
- Document Redis migration path for shared state across workers
- Remove Issue #42 from Tasks.md (resolved)
2026-05-03 20:53:21 +02:00
c3cd1574dc fix(auth): invalidate session cache on login
Stale sessions from a stolen device could be reused up to the cache
TTL after a legitimate user re-logs in, because login never cleared
the existing cache entry.

Changes:
- Add invalidate_by_user(user_id) to SessionCache protocol
- InMemorySessionCache maintains a user_id -> set[token] index to
  support O(1) invalidation of all sessions for a given user
- NoOpSessionCache stub updated for API compatibility
- auth_service.login() now returns the Session object alongside
  signed_token and expires_at
- login router calls session_cache.invalidate_by_user(session.id)
  immediately after successful authentication

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 20:51:51 +02:00
ae9313568e feat: enforce single-worker at startup
Fail with RuntimeError when WEB_CONCURRENCY or BANGUI_WORKERS > 1.

In-memory session cache, rate-limit windows, and runtime state are
process-local. Multi-worker silently causes stale limits, ghost sessions,
inconsistent status.

Skipped when TESTING=1.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 20:33:23 +02:00
e1a6491ac2 docs: add API reference and database schema docs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 19:50:36 +02:00
4d09d2538d docs: Add security best practices to Deployment.md
- Secrets management via environment variables
- Container security hardening (non-root user, filesystem permissions, capabilities)
- Network security and TLS termination guidance
- Prune obsolete task tracking from Tasks.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 19:48:52 +02:00
624f869f5b docs: add CI workflow and testing requirements documentation
- Add GitHub Actions CI pipeline with pytest, ruff, mypy
- Expand Tasks.md with implementation tracking and testing criteria
- Update CONTRIBUTING.md with CI requirements
- Add Testing-Requirements.md with coverage thresholds and PR checks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 19:42:50 +02:00
497d7cab41 docs: clean up completed accessibility issue, expand WCAG guidelines
- Remove Issue #31 (weak password validation) from Tasks.md
- Mark Issue #32 (accessibility) done in Tasks.md
- Expand Web-Development.md §14 with WCAG compliance rules, ARIA guide, keyboard nav, form accessibility, testing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 18:53:58 +02:00
c96b87ee8b feat: reject common passwords in SetupRequest
- Add ~75 common plaintext passwords to setup.py validator
- Check case-insensitively; passes complexity but blocked
- Add tests: reject common, accept unique, short common fail on length
- Update Security.md docs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 18:25:17 +02:00
96525573fa Normalise IP addresses across backend
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 18:19:41 +02:00
85d05ee582 docs: add editorconfig setup and remove completed task
- Add EditorConfig section to CONTRIBUTING.md with IDE plugins table
- Remove Issue #28 from Tasks.md (pre-commit hooks now documented)
2026-05-03 18:07:33 +02:00
5f0ab40816 refactor(backend): clean up models setup, improve ip utils, add adr docs
- Extract ADR documents for architectural decisions (SQLite, FastAPI, React, APScheduler, Scheduler)
- Refactor setup.py: improve code structure and readability
- Add IP validation utilities with test coverage
- Update frontend components (BanTable, HistoryPage)
- Add pre-commit hooks and CONTRIBUTING.md
- Add .editorconfig for consistent coding standards
2026-05-03 18:04:45 +02:00
2f9fc8076d refactor(backend): clean up jail service, add error handling service
- Extract jail status/processing to helper functions
- Add error_handling.py service for centralized error handling
- Update config.py with validation and defaults
- Update .env.example with all config options
- Remove obsolete Tasks.md, add Service-Development.md
- Minor fixes across routers and services

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 17:40:37 +02:00
2df029f7e8 refactor(ban_service): extract _bans_by_country_load_data helper
Break up long function into focused helper. Load data logic separate from aggregation.
2026-05-03 17:00:34 +02:00
5058a50143 Refactor backend: fix geo cache cleanup, scheduler heartbeat, correlation middleware; update docs 2026-05-03 16:02:40 +02:00
896751ada9 fix: handle socket close errors properly in PapertrailLogHandler
- Replace contextlib.suppress with try/except + warning log
- Add test for fail2ban client
- Remove stale Issue #21 from Tasks.md (indexes)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 12:25:14 +02:00
Copilot
22db607875 Add fail2ban DB index management and socket-based path resolution
- New get_fail2ban_db_path() in setup_service resolves DB path from configured socket path
- New ensure_fail2ban_indexes() creates missing performance indexes on bans table
- Call ensure_fail2ban_indexes on every startup before first ban query
- Remove completed tasks from Docs/Tasks.md
- Update Docs/PERFORMANCE.md with index findings
2026-05-03 12:17:31 +02:00
0133489920 Update observability docs and task utilities
- Add Observability.md documentation
- Standardize task logging with correlation_id support
- Add log_sanitizer utility for PII masking
- Update Tasks.md tracking
- Update geo_cache tasks and other task modules with correlation_id

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 11:52:09 +02:00
7b93499551 Refactor config loading and add status code docs
- Move config loading to dedicated ConfigLoader class with validation
- Add DATABASE_MIGRATIONS.md content to TROUBLESHOOTING.md
- Add API_STATUS_CODES.md documenting all API response codes
- Update runner.csx to use new config structure
- Add check_responses.py validation script
- Update config tests for new structure

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 11:52:01 +02:00
8f26776bb3 docs: add OpenAPI responses={} to all router endpoints
Add explicit HTTP status code documentation to every endpoint
across 15 router files. Each endpoint now declares all possible
response codes (200/201/204/400/401/404/409/429/502/503) with
descriptions so frontend can distinguish error types.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 01:12:08 +02:00
7ad885d276 refactor: separate config service from jail config service
- Split config_service.py into config_service.py and jail_config_service.py
- Update Docs/Tasks.md, Security.md, TROUBLESHOOTING.md
2026-05-03 01:05:18 +02:00
881cfbdd71 fix: replace broad except Exception with specific exception types
- jail_service: catch ValueError (fail2ban protocol error) instead of Exception
- health.py: catch AttributeError (not OSError/TypeError) for defensive checks
- ban_service: re-raise programming errors in geo lookup handlers
- server_service: catch Fail2BanConnectionError, Fail2BanProtocolError, ValueError
- config_writer: catch OSError instead of Exception

Programming errors now bubble to global handler instead of being silently caught.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 00:54:44 +02:00
bd6170722a feat(geo): add cache hit/miss metrics and prewarm support
- Add _hits/_misses counters to GeoCache for cache hit/miss ratio tracking
- Reset counters on clear()
- Count hits before misses in lookup_batch() to avoid interleaving
- Add synchronous prewarm() using asyncio.create_task for fire-and-forget
- Add hits/misses fields to GeoCacheStatsResponse model
- Add TestCacheMetrics (5 tests), TestPrewarm (3 tests), TestLargeBanList (2 tests)
- Fix _make_async_db() mock: db.execute is not async, returns ctx manager
- Move collections.abc to TYPE_CHECKING block (TC003)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 00:35:47 +02:00
b587c6e850 docs: add TYPE_SAFETY.md documenting frontend/backend type conventions
Establishes shared type conventions to prevent runtime type mismatches
between TS frontend and Python backend. Covers snake_case JSON field
names, null vs empty string handling, timestamp formats, and validation
patterns for country codes, bans, and jail configuration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 00:12:44 +02:00
0817a4cb47 fix(regex_validator): add ReDoS detection via regexploit
Detect catastrophic backtracking patterns before regex compilation
using regexploit library. Add ReDoSDetectedError exception and
_MINIMUM_STARRINESS threshold (>=3) to catch dangerous patterns
like (a+)+b. Update pyproject.toml deps, add tests for detection.
2026-05-03 00:05:33 +02:00
e436727942 fix: atomic upsert for import runs (Issue #12)
Replace check-then-insert race condition with INSERT ON CONFLICT.
- upsert_pending uses RETURNING id for atomic upsert
- UNIQUE(source_id, content_hash) constraint from migration 6
- blocklist_import_workflow updated to use upsert_pending
- test_import_source_success fixed for async mock patterns

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 23:39:43 +02:00
1285bc8571 feat: comprehensive health check with DB, scheduler, cache
- Add /api/v1/health endpoint with component-level checks
- Verify DB connectivity, fail2ban socket, scheduler, session cache
- Add SQLite WAL cleanup on startup (orphan crash files)
- Migration 8: import_log.timestamp → INTEGER UNIX epoch
- Align import_log timestamps with history_archive (already UNIX int)
- Add unit tests for DB cleanup and health router

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 23:03:57 +02:00
b631c1c546 feat(backend): implement graceful shutdown for container stop
Graceful shutdown ensures in-flight operations complete before process exits:
- Lifespan shutdown handler drains pending tasks with 25s timeout
- Scheduler stops accepting new jobs immediately
- HTTP session, external logging, scheduler lock, DB conn closed cleanly
- 25s Python timeout leaves 5s margin before Docker's 30s SIGKILL

Files changed:
- backend/app/main.py: enhanced _lifespan shutdown with task drain
- Docker/Dockerfile.backend: documented signal handling in header
- Docker/docker-compose.yml: added stop_grace_period: 30s
- Docker/compose.prod.yml: added stop_grace_period: 30s
- Docs/Deployment.md: new Graceful Shutdown section with sequence table
- Docs/TROUBLESHOOTING.md: new Graceful Shutdown Issues section

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 22:47:10 +02:00
f6c3c02183 Refactor response handling and health check endpoints
- Enhance response model with additional fields and validation
- Update health and server router implementations
- Improve frontend type definitions and API integration
- Clean up documentation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 21:57:00 +02:00
cc6dbcf3f0 feat: implement API versioning /api/v1/
- All backend routers moved to /api/v1/ prefix
- Frontend BASE_URL updated to /api/v1
- Setup redirect middleware updated to redirect to /api/v1/setup
- Health router path fixed: prefix=/api/v1/health, @router.get('')
- conftest.py: set server_status=online for test fixture
- Created Docs/API_VERSIONING.md with deprecation policy
- Updated Docs/Backend-Development.md with versioning section
- Updated Instructions.md curl examples

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 21:29:30 +02:00
0d5882b32f Fix HIGH priority issues: unbounded queries, rate limiting, health checks
Issue #3 - Unbounded Query Results (OOM):
- get_all_archived_history() now uses keyset pagination with bounded max_rows (50k default)
- Added 'id' field to records from get_archived_history() and get_archived_history_keyset()
- Protocol signature updated with page_size, max_rows, last_ban_id params

Issue #7 - Docker Health Check Fails:
- Added curl to Dockerfile.backend runtime image
- HEALTHCHECK now uses 'curl -f http://localhost:8000/api/health'
- compose.prod.yml: increased start_period to 40s, timeout to 10s
- Frontend healthcheck proxies to backend /api/health

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-01 21:47:36 +02:00
1830da496d backup
Co-authored-by: Copilot <copilot@github.com>
2026-05-01 20:23:54 +02:00