refactor: restructure API pagination metadata for better frontend usability
- Create PaginationMetadata model with computed derived fields (total_pages, has_next_page, has_prev_page)
- Update PaginatedListResponse to embed pagination metadata in a separate 'pagination' object
- Add create_pagination_metadata() factory function in utils/pagination.py for consistent computation
- Update all paginated service functions to use new structure:
- history_service.list_history()
- blocklist_service.get_import_logs()
- jail_service.get_jail_banned_ips()
- ban_mappers.map_domain_dashboard_ban_list_to_response()
- Update response model docstrings with new structure examples
- Update Backend-Development.md documentation with new pagination patterns
- Update test fixtures to work with new response structure
Response shape changes from:
{"items": [...], "total": 100, "page": 1, "page_size": 50}
To:
{"items": [...], "pagination": {"page": 1, "page_size": 50, "total": 100, "total_pages": 2, "has_next_page": true, "has_prev_page": false}}
Benefits:
- Frontend receives all pagination state needed for UI controls
- No need for frontend to calculate total_pages or page navigation logic
- Consolidated pagination metadata reduces field sprawl
- OpenAPI schema automatically reflects changes
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@@ -417,9 +417,14 @@ class JailListResponse(PaginatedListResponse[JailSummary]):
|
||||
# Returns:
|
||||
{
|
||||
"items": [...], # T[]
|
||||
"total": 100, # int: total items across all pages
|
||||
"page": 2, # int: current page (1-based)
|
||||
"page_size": 20 # int: items per page
|
||||
"pagination": {
|
||||
"page": 2, # int: current page (1-based)
|
||||
"page_size": 20, # int: items per page
|
||||
"total": 100, # int: total items across all pages
|
||||
"total_pages": 5, # int: computed total number of pages
|
||||
"has_next_page": true, # bool: whether more pages exist
|
||||
"has_prev_page": true # bool: whether previous pages exist
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -526,7 +531,7 @@ class BansByCountryResponse(BaseModel):
|
||||
|
||||
| Pattern | Used for | Field Names | Example |
|
||||
|---------|----------|---|---|
|
||||
| **PaginatedListResponse** | Paginated collections | `items`, `total`, `page`, `page_size` | `GET /api/dashboard/bans` |
|
||||
| **PaginatedListResponse** | Paginated collections | `items`, `pagination` (page, page_size, total, total_pages, has_next_page, has_prev_page) | `GET /api/dashboard/bans` |
|
||||
| **CollectionResponse** | Complete collections | `items`, `total` | `GET /api/config/jails` |
|
||||
| **Detail Response** | Single entity + metadata | Entity name + descriptors | `GET /api/jails/{name}` |
|
||||
| **CommandResponse** | Action results | `message`, `success` + optional identifiers | `POST /api/jails/{name}/start` |
|
||||
@@ -561,6 +566,7 @@ All paginated endpoints follow a consistent query parameter contract:
|
||||
```python
|
||||
from fastapi import Query
|
||||
from app.utils.constants import DEFAULT_PAGE_SIZE
|
||||
from app.utils.pagination import create_pagination_metadata
|
||||
|
||||
@router.get("/items")
|
||||
async def get_items(
|
||||
@@ -577,24 +583,29 @@ async def get_items(
|
||||
items = await db.fetch("SELECT * FROM items LIMIT ? OFFSET ?", page_size, offset)
|
||||
total = await db.fetchval("SELECT COUNT(*) FROM items")
|
||||
|
||||
# Create pagination metadata with computed fields
|
||||
pagination = create_pagination_metadata(total, page, page_size)
|
||||
|
||||
return PaginatedListResponse(
|
||||
items=items,
|
||||
total=total,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
pagination=pagination,
|
||||
)
|
||||
```
|
||||
|
||||
**Helper functions** are available in `app.utils.pagination`:
|
||||
|
||||
```python
|
||||
from app.utils.pagination import get_offset, compute_total_pages
|
||||
from app.utils.pagination import get_offset, compute_total_pages, create_pagination_metadata
|
||||
|
||||
# Calculate database offset from page and page_size
|
||||
offset = get_offset(page, page_size) # Equivalent to (page - 1) * page_size
|
||||
|
||||
# Calculate total pages for rendering pagination UI (optional)
|
||||
total_pages = compute_total_pages(total, page_size)
|
||||
|
||||
# Create complete pagination metadata with all computed fields
|
||||
pagination = create_pagination_metadata(total, page, page_size)
|
||||
# Returns PaginationMetadata with: page, page_size, total, total_pages, has_next_page, has_prev_page
|
||||
```
|
||||
|
||||
**Rules:**
|
||||
@@ -602,8 +613,8 @@ total_pages = compute_total_pages(total, page_size)
|
||||
1. **Use 1-based pages** — Not 0-based offsets. Page 1 is always the first page.
|
||||
2. **Always provide defaults** — Use `DEFAULT_PAGE_SIZE` (100) and initial page 1.
|
||||
3. **Cap maximum page_size at 500** — Prevents accidental DoS from enormous requests.
|
||||
4. **Respond with `PaginatedListResponse[T]`** — Must include `items`, `total`, `page`, `page_size`.
|
||||
5. **Never include `total_pages` in responses** — The frontend can calculate it as `Math.ceil(total / page_size)`.
|
||||
4. **Use `create_pagination_metadata()`** — Factory function computes derived fields (total_pages, has_next_page, has_prev_page) consistently.
|
||||
5. **Respond with `PaginatedListResponse[T]`** — Must include `items` and `pagination` metadata object.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,51 +1,3 @@
|
||||
## [IMPORTANT] Scheduler lock race condition
|
||||
|
||||
**Where found**
|
||||
|
||||
- `backend/app/utils/scheduler_lock.py:56-58` — heartbeat interval 10 seconds
|
||||
|
||||
**Why this is needed**
|
||||
|
||||
Current design: Process A acquires lock, heartbeat misses, lock expires, Process B acquires lock, both running simultaneously → duplicate work, data corruption.
|
||||
|
||||
**Goal**
|
||||
|
||||
Implement robust distributed locking that prevents concurrent execution.
|
||||
|
||||
**What to do**
|
||||
|
||||
**Option A (Strengthen heartbeat):**
|
||||
- Reduce interval to 5s (half of timeout)
|
||||
- Use database advisory locks
|
||||
- Monitor heartbeat failures
|
||||
|
||||
**Option B (Migrate to Redis):**
|
||||
- Use `redlock-py` or `aioredis`
|
||||
- Simpler, more reliable than database-backed
|
||||
|
||||
**Current code improvements:**
|
||||
- Log when heartbeat fails
|
||||
- Add metric for lock contention
|
||||
- Test multi-process scenario
|
||||
|
||||
**Possible traps and issues**
|
||||
|
||||
- Database locks don't scale under high contention
|
||||
- Redis adds new dependency
|
||||
- Clock skew breaks timestamp-based expiry
|
||||
|
||||
**Docs changes needed**
|
||||
|
||||
- Update `Docs/Deployment.md` § Scheduler Lock
|
||||
- Add troubleshooting: "Blocklist import runs twice"
|
||||
|
||||
**Doc references**
|
||||
|
||||
- `Docs/Deployment.md` (scheduler)
|
||||
- `backend/app/utils/scheduler_lock.py` (lock implementation)
|
||||
|
||||
---
|
||||
|
||||
## [IMPORTANT] API pagination doesn't return metadata
|
||||
|
||||
**Where found**
|
||||
|
||||
Reference in New Issue
Block a user