Commit Graph

468 Commits

Author SHA1 Message Date
48ef85bec5 backup 2026-05-05 19:51:14 +02:00
481f32bb85 backup 2026-05-05 18:47:56 +02:00
d25b56e7e1 backup 2026-05-04 13:13:01 +02:00
48d57c31e1 backup 2026-05-04 13:12:57 +02:00
e41831447f docs: update documentation and e2e tests
- Add configuration docs for database and rate limiting
- Remove completed tasks from tracking list
- Update testing requirements with new test patterns
- Enhance web development docs with frontend guidelines
- Expand page loading and ban records e2e test coverage
2026-05-04 08:34:18 +02:00
23c3a0d9e6 feat: add e2e test suite with Robot Framework
Add e2e/ dir with Robot Framework tests for page loading, ban records,
blocklist import, config edit. Add requirements.txt. Update Makefile with
test commands. Update .gitignore, backend docs, testing requirements docs.
2026-05-04 08:29:12 +02:00
5fa67d3288 backup
Co-authored-by: Copilot <copilot@github.com>
2026-05-04 08:16:20 +02:00
744275d17f backup 2026-05-04 07:20:20 +02:00
58173bd6a9 backup 2026-05-04 07:20:16 +02:00
69e1726045 Refactor data fetching hooks, add page size lint test
- Simplify useFetchData: remove unused URL building logic
- Add usePolledData initial implementation
- Add router page_size param validation test
- Update API reference docs
- Clean up tasks doc
2026-05-04 06:48:24 +02:00
0a3f9c6c16 refactor(backend): external logging metrics, required mode, health checks
- Add external_logging_init_failures counter
- Add external_log_required flag, raise if init fails and required
- Health endpoint: add external_logging status check
- Blocklist service: enrich with metadata fields, update import logic
- Health check task: add runtime_state dependency, fix return typing
- Metrics: add Histogram for request latencies
- Frontend: align BlocklistImportLogSection props
- Docs: update deployment guide, remove stale tasks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-04 03:45:13 +02:00
42e177e6ea feat(frontend): add ignoreCancellation option for background tasks
Allow useNavigationAbortSignal to opt out of navigation-based abort
for long-lived background tasks like polling. Set ignoreCancellation: true
to keep requests alive across route changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-04 02:57:56 +02:00
eb339efcfd Add Kubernetes liveness/readiness probes and middleware order validation
- Split /health into /health/live (liveness) and /health/ready (readiness)
  following Kubernetes conventions. Combined /health retained for backward
  compatibility with existing Docker HEALTHCHECK definitions.
- Add ReadyCheck and ReadyResponse models for structured readiness output.
- Add _assert_middleware_order() startup check enforcing:
  RateLimit → Csrf → CorrelationId middleware chain.
- Register CorrelationIdMiddleware, CsrfMiddleware, RateLimitMiddleware
  in create_app() with documented required order (reverse of processing).
- Add correlation.py, csrf.py, rate_limit.py middleware modules.
- Add health probe tests in test_health_probes.py.
- Update test_main.py with middleware order assertion tests.
- Update frontend useFetchData hook tests.
- Docs: update Deployment.md with Kubernetes probe config examples.
2026-05-04 02:42:09 +02:00
65fe747cba feat(backend): add deprecation middleware and API versioning support
- Add deprecation middleware for warning headers on sunset endpoints
- Add jails_v2 router for API v2 migration path
- Update CI workflow with new test coverage
- Update API versioning documentation
- Remove completed tasks from Tasks.md
2026-05-04 00:03:52 +02:00
c8b48b5b65 fix(api): correlation ID survives HMR; fix endpoint template literal typos
- client.ts: store correlation ID in sessionStorage so HMR (module re-eval)
  does not generate a new ID mid-session; add clearSessionCorrelationId()
- endpoints.ts: fix 3 template literal trailing-quote bugs (missing ')' chars);
  replace template literals with string concat for encodeURIComponent calls
- AuthProvider.tsx: call clearSessionCorrelationId() on logout
- App.tsx: reorder ThemeProvider import before AuthProvider per PROVIDER_ORDER.md;
  indent Routes inside AuthProvider to match expected tree structure
- Tasks.md: update task status
- providerTreeOrder.test.tsx: add integration tests for provider nesting order

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 23:35:18 +02:00
fc57c83f79 refactor: split pagination logic from response models
- Extract pagination logic to separate util module
- Update response models to use new pagination util
- Fix pagination calculation edge cases

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 22:57:21 +02:00
b2747381ec Remove Issue #51 from Tasks.md (type-enforcement tracked elsewhere) 2026-05-03 22:47:24 +02:00
a2afec2d1e Remove completed Issue #50 navigation abort signal task
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 22:43:19 +02:00
3376009903 fix(nav): move AbortController creation synchronously in render
Previously the AbortController was created inside useEffect, which runs
after render. This meant requests initiated during render received
the previous cycle's (possibly aborted) signal and were cancelled
immediately instead of completing.

Now the controller is created synchronously when pathname changes, before
any request can be initiated on the new route. The old controller is
aborted in the same conditional block, before the new one is created.

Side effect: removed resolved Issue #49 from Tasks.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 22:37:07 +02:00
7fcfc14199 fix(auth): dedupe handler + error utils refactor
- Add 401/403 dedup guard to API client to prevent double logout
- Extract fetchError util: isAuthError + getErrorMessage
- AuthProvider uses new error utils, removes duplicate logic
- Remove completed task docs from Tasks.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 22:13:12 +02:00
dafe8d61e2 feat(security): add CSRF header constants and security-headers endpoint
Move X-BanGUI-Request header name/value to backend/app/utils/constants.py as single source of truth. Add GET /api/v1/config/security-headers endpoint. Update csrf middleware, frontend api client, and docs to use shared constants.
2026-05-03 22:06:43 +02:00
cee3daffc1 fix: enforce PRAGMA query_only on fail2ban DB and refactor CSRF cookie name
- Add _acquire_readonly_connection() that applies PRAGMA query_only=ON after connect
- Verify PRAGMA value back to catch URI flag bypasses
- Wrap in async context manager _readonly_connection() used by all repo methods
- Replace hardcoded '_SESSION_COOKIE_NAME' in CSRF middleware with import from
  app.utils.constants
- Remove completed Issues #45 and #46 from Docs/Tasks.md (Issue #46 now fixed,
  #45 cache invalidation deferred to auth refactor branch)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 21:47:42 +02:00
1c3dff31e8 feat(rate-limiting): add per-bucket limits and startup validation
- Add per-bucket rate limit config (ban, unban, import, config, jail, filter, action)
- Add process-local warning at startup for multi-worker deployments
- Document Redis migration path for shared state across workers
- Remove Issue #42 from Tasks.md (resolved)
2026-05-03 20:53:21 +02:00
ae9313568e feat: enforce single-worker at startup
Fail with RuntimeError when WEB_CONCURRENCY or BANGUI_WORKERS > 1.

In-memory session cache, rate-limit windows, and runtime state are
process-local. Multi-worker silently causes stale limits, ghost sessions,
inconsistent status.

Skipped when TESTING=1.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 20:33:23 +02:00
e1a6491ac2 docs: add API reference and database schema docs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 19:50:36 +02:00
4d09d2538d docs: Add security best practices to Deployment.md
- Secrets management via environment variables
- Container security hardening (non-root user, filesystem permissions, capabilities)
- Network security and TLS termination guidance
- Prune obsolete task tracking from Tasks.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 19:48:52 +02:00
624f869f5b docs: add CI workflow and testing requirements documentation
- Add GitHub Actions CI pipeline with pytest, ruff, mypy
- Expand Tasks.md with implementation tracking and testing criteria
- Update CONTRIBUTING.md with CI requirements
- Add Testing-Requirements.md with coverage thresholds and PR checks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 19:42:50 +02:00
497d7cab41 docs: clean up completed accessibility issue, expand WCAG guidelines
- Remove Issue #31 (weak password validation) from Tasks.md
- Mark Issue #32 (accessibility) done in Tasks.md
- Expand Web-Development.md §14 with WCAG compliance rules, ARIA guide, keyboard nav, form accessibility, testing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 18:53:58 +02:00
c96b87ee8b feat: reject common passwords in SetupRequest
- Add ~75 common plaintext passwords to setup.py validator
- Check case-insensitively; passes complexity but blocked
- Add tests: reject common, accept unique, short common fail on length
- Update Security.md docs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 18:25:17 +02:00
96525573fa Normalise IP addresses across backend
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 18:19:41 +02:00
85d05ee582 docs: add editorconfig setup and remove completed task
- Add EditorConfig section to CONTRIBUTING.md with IDE plugins table
- Remove Issue #28 from Tasks.md (pre-commit hooks now documented)
2026-05-03 18:07:33 +02:00
5f0ab40816 refactor(backend): clean up models setup, improve ip utils, add adr docs
- Extract ADR documents for architectural decisions (SQLite, FastAPI, React, APScheduler, Scheduler)
- Refactor setup.py: improve code structure and readability
- Add IP validation utilities with test coverage
- Update frontend components (BanTable, HistoryPage)
- Add pre-commit hooks and CONTRIBUTING.md
- Add .editorconfig for consistent coding standards
2026-05-03 18:04:45 +02:00
2f9fc8076d refactor(backend): clean up jail service, add error handling service
- Extract jail status/processing to helper functions
- Add error_handling.py service for centralized error handling
- Update config.py with validation and defaults
- Update .env.example with all config options
- Remove obsolete Tasks.md, add Service-Development.md
- Minor fixes across routers and services

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 17:40:37 +02:00
2df029f7e8 refactor(ban_service): extract _bans_by_country_load_data helper
Break up long function into focused helper. Load data logic separate from aggregation.
2026-05-03 17:00:34 +02:00
5058a50143 Refactor backend: fix geo cache cleanup, scheduler heartbeat, correlation middleware; update docs 2026-05-03 16:02:40 +02:00
896751ada9 fix: handle socket close errors properly in PapertrailLogHandler
- Replace contextlib.suppress with try/except + warning log
- Add test for fail2ban client
- Remove stale Issue #21 from Tasks.md (indexes)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 12:25:14 +02:00
Copilot
22db607875 Add fail2ban DB index management and socket-based path resolution
- New get_fail2ban_db_path() in setup_service resolves DB path from configured socket path
- New ensure_fail2ban_indexes() creates missing performance indexes on bans table
- Call ensure_fail2ban_indexes on every startup before first ban query
- Remove completed tasks from Docs/Tasks.md
- Update Docs/PERFORMANCE.md with index findings
2026-05-03 12:17:31 +02:00
0133489920 Update observability docs and task utilities
- Add Observability.md documentation
- Standardize task logging with correlation_id support
- Add log_sanitizer utility for PII masking
- Update Tasks.md tracking
- Update geo_cache tasks and other task modules with correlation_id

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 11:52:09 +02:00
7b93499551 Refactor config loading and add status code docs
- Move config loading to dedicated ConfigLoader class with validation
- Add DATABASE_MIGRATIONS.md content to TROUBLESHOOTING.md
- Add API_STATUS_CODES.md documenting all API response codes
- Update runner.csx to use new config structure
- Add check_responses.py validation script
- Update config tests for new structure

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 11:52:01 +02:00
7ad885d276 refactor: separate config service from jail config service
- Split config_service.py into config_service.py and jail_config_service.py
- Update Docs/Tasks.md, Security.md, TROUBLESHOOTING.md
2026-05-03 01:05:18 +02:00
bd6170722a feat(geo): add cache hit/miss metrics and prewarm support
- Add _hits/_misses counters to GeoCache for cache hit/miss ratio tracking
- Reset counters on clear()
- Count hits before misses in lookup_batch() to avoid interleaving
- Add synchronous prewarm() using asyncio.create_task for fire-and-forget
- Add hits/misses fields to GeoCacheStatsResponse model
- Add TestCacheMetrics (5 tests), TestPrewarm (3 tests), TestLargeBanList (2 tests)
- Fix _make_async_db() mock: db.execute is not async, returns ctx manager
- Move collections.abc to TYPE_CHECKING block (TC003)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 00:35:47 +02:00
b587c6e850 docs: add TYPE_SAFETY.md documenting frontend/backend type conventions
Establishes shared type conventions to prevent runtime type mismatches
between TS frontend and Python backend. Covers snake_case JSON field
names, null vs empty string handling, timestamp formats, and validation
patterns for country codes, bans, and jail configuration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-03 00:12:44 +02:00
e436727942 fix: atomic upsert for import runs (Issue #12)
Replace check-then-insert race condition with INSERT ON CONFLICT.
- upsert_pending uses RETURNING id for atomic upsert
- UNIQUE(source_id, content_hash) constraint from migration 6
- blocklist_import_workflow updated to use upsert_pending
- test_import_source_success fixed for async mock patterns

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 23:39:43 +02:00
1285bc8571 feat: comprehensive health check with DB, scheduler, cache
- Add /api/v1/health endpoint with component-level checks
- Verify DB connectivity, fail2ban socket, scheduler, session cache
- Add SQLite WAL cleanup on startup (orphan crash files)
- Migration 8: import_log.timestamp → INTEGER UNIX epoch
- Align import_log timestamps with history_archive (already UNIX int)
- Add unit tests for DB cleanup and health router

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 23:03:57 +02:00
b631c1c546 feat(backend): implement graceful shutdown for container stop
Graceful shutdown ensures in-flight operations complete before process exits:
- Lifespan shutdown handler drains pending tasks with 25s timeout
- Scheduler stops accepting new jobs immediately
- HTTP session, external logging, scheduler lock, DB conn closed cleanly
- 25s Python timeout leaves 5s margin before Docker's 30s SIGKILL

Files changed:
- backend/app/main.py: enhanced _lifespan shutdown with task drain
- Docker/Dockerfile.backend: documented signal handling in header
- Docker/docker-compose.yml: added stop_grace_period: 30s
- Docker/compose.prod.yml: added stop_grace_period: 30s
- Docs/Deployment.md: new Graceful Shutdown section with sequence table
- Docs/TROUBLESHOOTING.md: new Graceful Shutdown Issues section

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 22:47:10 +02:00
f6c3c02183 Refactor response handling and health check endpoints
- Enhance response model with additional fields and validation
- Update health and server router implementations
- Improve frontend type definitions and API integration
- Clean up documentation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 21:57:00 +02:00
cc6dbcf3f0 feat: implement API versioning /api/v1/
- All backend routers moved to /api/v1/ prefix
- Frontend BASE_URL updated to /api/v1
- Setup redirect middleware updated to redirect to /api/v1/setup
- Health router path fixed: prefix=/api/v1/health, @router.get('')
- conftest.py: set server_status=online for test fixture
- Created Docs/API_VERSIONING.md with deprecation policy
- Updated Docs/Backend-Development.md with versioning section
- Updated Instructions.md curl examples

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-02 21:29:30 +02:00
0d5882b32f Fix HIGH priority issues: unbounded queries, rate limiting, health checks
Issue #3 - Unbounded Query Results (OOM):
- get_all_archived_history() now uses keyset pagination with bounded max_rows (50k default)
- Added 'id' field to records from get_archived_history() and get_archived_history_keyset()
- Protocol signature updated with page_size, max_rows, last_ban_id params

Issue #7 - Docker Health Check Fails:
- Added curl to Dockerfile.backend runtime image
- HEALTHCHECK now uses 'curl -f http://localhost:8000/api/health'
- compose.prod.yml: increased start_period to 40s, timeout to 10s
- Frontend healthcheck proxies to backend /api/health

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-01 21:47:36 +02:00
1830da496d backup
Co-authored-by: Copilot <copilot@github.com>
2026-05-01 20:23:54 +02:00
3b3728c58d feat: implement request deduplication in useFetchData
- Add optional requestKey parameter to UseFetchDataOptions
- Implement module-level cache (inFlightRequests) to track in-flight requests
- When requestKey is provided, multiple hook instances with same key share in-flight requests
- Prevents duplicate API calls when multiple components fetch same data or rapid refresh calls
- Cache entries are automatically cleared when response arrives (success or error)
- Maintains backward compatibility: without requestKey, behaves as before
- Adds comprehensive tests for deduplication scenarios

This reduces bandwidth waste and prevents race conditions caused by concurrent requests for identical data.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-01 18:44:46 +02:00