TASK-032: Implement geo_cache retention policy and cleanup
Add automatic cleanup of stale geolocation cache entries to prevent unbounded database growth. Resolves the issue where unique IP addresses accumulated indefinitely in the geo_cache table, degrading query performance. ## Changes ### Database Schema (Migration 3) - Add 'last_seen' column to geo_cache table tracking last reference time - Existing entries default to current timestamp ### Repository Layer (geo_cache_repo.py) - Update upsert_entry() to set/refresh last_seen on insert/update - Update upsert_neg_entry() to set/refresh last_seen on negative cache hits - Update bulk_upsert_entries() to set/refresh last_seen in batch operations - Add delete_stale_entries(db, cutoff_iso) -> int for purging old entries ### Background Task (geo_cache_cleanup.py) - New APScheduler task that runs nightly (24-hour interval) - Calculates cutoff as 90 days ago from current time (UTC) - Deletes all entries with last_seen older than cutoff - Logs operation results (info when deleted > 0, debug when 0 deleted) - Configurable retention period via GEO_CACHE_RETENTION_DAYS constant ### Application Startup (startup.py) - Register geo_cache_cleanup task in scheduler during app startup - Placed after geo_cache_flush in task registration order ### Tests - Add delete_stale_entries test cases covering: * Removal of old entries beyond cutoff * No deletion when all entries are recent * Empty table edge case - Update existing test fixtures to include last_seen column - Add full test suite for cleanup task registration and execution ### Documentation - Architekture.md: Document cleanup task, update schema/diagram - Backend-Development.md: Add retention policy documentation ## Behavior When an IP is accessed, its last_seen is refreshed. After 90 days of no access, an IP is purged by the nightly cleanup. On next encounter, the IP is re-resolved from MaxMind MMDB or ip-api.com (if configured). This is acceptable because: 1. Stale geolocation data may become inaccurate over time 2. Re-resolution cost is minimal compared to unbounded storage growth 3. Active IPs maintain fresh data through their last_seen updates Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@@ -98,7 +98,8 @@ async def upsert_entry(
|
||||
country_name = excluded.country_name,
|
||||
asn = excluded.asn,
|
||||
org = excluded.org,
|
||||
cached_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now')
|
||||
cached_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now'),
|
||||
last_seen = strftime('%Y-%m-%dT%H:%M:%fZ', 'now')
|
||||
""",
|
||||
(ip, country_code, country_name, asn, org),
|
||||
)
|
||||
@@ -120,7 +121,11 @@ async def upsert_entry_and_commit(
|
||||
async def upsert_neg_entry(db: aiosqlite.Connection, ip: str) -> None:
|
||||
"""Record a failed lookup attempt as a negative entry."""
|
||||
await db.execute(
|
||||
"INSERT OR IGNORE INTO geo_cache (ip) VALUES (?)",
|
||||
"""
|
||||
INSERT INTO geo_cache (ip) VALUES (?)
|
||||
ON CONFLICT(ip) DO UPDATE SET
|
||||
last_seen = strftime('%Y-%m-%dT%H:%M:%fZ', 'now')
|
||||
""",
|
||||
(ip,),
|
||||
)
|
||||
|
||||
@@ -148,7 +153,8 @@ async def bulk_upsert_entries(
|
||||
country_name = excluded.country_name,
|
||||
asn = excluded.asn,
|
||||
org = excluded.org,
|
||||
cached_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now')
|
||||
cached_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now'),
|
||||
last_seen = strftime('%Y-%m-%dT%H:%M:%fZ', 'now')
|
||||
""",
|
||||
rows,
|
||||
)
|
||||
@@ -202,3 +208,21 @@ async def bulk_upsert_entries_and_neg_entries_and_commit(
|
||||
await db.commit()
|
||||
|
||||
return positive_count, negative_count
|
||||
|
||||
|
||||
async def delete_stale_entries(db: aiosqlite.Connection, cutoff_iso: str) -> int:
|
||||
"""Delete geo cache entries not referenced since the cutoff timestamp.
|
||||
|
||||
Args:
|
||||
db: Open BanGUI application database connection.
|
||||
cutoff_iso: ISO 8601 timestamp (e.g., '2024-01-01T00:00:00Z'). Entries with
|
||||
``last_seen`` before this time will be deleted.
|
||||
|
||||
Returns:
|
||||
The number of rows deleted.
|
||||
"""
|
||||
async with db.execute(
|
||||
"DELETE FROM geo_cache WHERE last_seen < ?",
|
||||
(cutoff_iso,),
|
||||
) as cur:
|
||||
return cur.rowcount if cur.rowcount is not None else 0
|
||||
|
||||
Reference in New Issue
Block a user