docs: add logging instruction reference to tasks

2026-03-17 11:38:57 +01:00
parent 151a08e033
commit e5fae0a0a2
2 changed files with 451 additions and 0 deletions
--- a/docs/InstructionsLogging.md
+++ b/docs/InstructionsLogging.md
@@ -0,0 +1,90 @@
+# Logging Instructions
+
+This document describes how to write and refactor logging across the AniWorld codebase to make logs **human-readable**, **debug-friendly**, and **noise-free**.
+
+> ✅ Goal: Logs should help a developer understand what happened, why it happened, and what to inspect next — without overwhelming them with duplicates or irrelevant details.
+
+---
+
+## 1. Principles for Great Logs
+
+### 1.1 Use the Right Log Level
+
+- `DEBUG`: Detailed internal state useful when debugging a specific issue (e.g., decision points, returned values, request/response payloads). Not for normal operation.
+- `INFO`: High-level events that represent what the system is doing (e.g., "Import started", "New series added", "Config reloaded"). Use sparingly.
+- `WARNING`: Something unexpected happened, but the system can continue (e.g., missing optional file, fallback behavior).
+- `ERROR`: An operation failed and needs attention (e.g., exception caught, failed database write).
+- `CRITICAL`: The system is in an unusable state (e.g., config corruption, failed startup).
+
+### 1.2 Keep Logs Human-Readable
+
+- Write messages in a clear, descriptive sentence-style format.
+- Avoid cryptic codes or single-word log messages.
+- Prefer `logger.debug("... %s", value)`-style formatting over f-strings to avoid unnecessary work when the log level is disabled.
+
+### 1.3 Avoid Log Spam
+
+- Don’t log inside hot loops unless you explicitly aggregate and log a summary (e.g., "Processed 124 files, 3 failures").
+- Avoid repeated/logging the same event at the same level (e.g., do not log "Retrying" 10 times at INFO; log once at INFO and then use DEBUG for each retry).
+- Use rate limiting or debounce patterns for logs that can fire rapidly (e.g., external service health checks).
+- Prefer a single higher-level log with context rather than many low-level logs that clutter output.
+
+### 1.4 Log Objects Usefully
+
+- When logging objects, log the minimal useful representation (e.g., ID, name, status) rather than the full object or its memory address.
+- If an object has a `.dict()`, `.to_dict()`, or `.as_dict()` helper (common in Pydantic models), log that rather than relying on `repr()`.
+- Add a `__repr__` or `__str__` implementation to domain models that returns a helpful, concise string with key identifiers.
+- Use structured logging (e.g., `logger.info("Series added", extra={"series_id": series.id, "title": series.title})`) where supported.
+- For exceptions, prefer `logger.exception("Failed to ...")` to capture stack traces.
+
+---
+
+## 2. Refactoring Existing Logs
+
+When improving or refactoring existing log statements, aim to make them:
+
+- **Actionable**: A developer reading the log should know what happened and what to check next.
+- **Non-redundant**: Remove duplicates and ensure only one log records the same high-level event at a given level.
+- **Context-rich**: Include identifiers (e.g., `series_id`, `file_path`, `user_id`) and key state that explains why a decision was made.
+- **Level-appropriate**: Downgrade noisy INFO logs to DEBUG, and elevate critical failures to ERROR/CRITICAL.
+
+### 2.1 Refactor Checklist
+
+1. **Locate noisy logs**: Search for repeated messages (e.g., "Start", "Done") and determine whether they should be DEBUG or removed.
+2. **Replace ad-hoc prints**: Remove `print()` statements or `print(obj)` and replace with `logger.*` calls.
+3. **Use structured context**: If a function logs multiple related messages, include the same context in each (e.g., `extra={"series_id": series.id}`) or use a context manager that attaches it.
+4. **Validate object output**: Ensure any logged object produces a useful representation (add methods or translate to dict). If not, log the key fields explicitly.
+5. **Batch repetitive events**: If a loop logs per item, consider collecting stats and logging a summary at the end.
+
+## 3. Adding New Logs
+
+When adding logs to new code paths:
+
+- Log **important state transitions** (e.g., "Queue started", "Download completed", "Config reloaded").
+- For error paths, include what failed and why (e.g., "Could not load config from X: {exc}").
+- Prefer logging at the boundaries of operations, not deep inside utility functions unless it aids debugging.
+- Write logs in full sentences, with a clear subject, verb, and object.
+
+---
+
+## 4. Example Patterns
+
+```python
+logger.info("Import completed", extra={"series_id": series.id, "count": len(imported)})
+
+logger.debug(
+    "Fetched feed items",
+    extra={"feed_url": feed.url, "item_count": len(items)},
+)
+
+try:
+    result = download_episode(episode)
+except Exception:
+    logger.exception("Failed to download episode %s", episode.id)
+```
+
+> 💡 When in doubt, favor **fewer, richer logs** over many noisy logs.
+
+# importent
+
+note which file is done.
--- a/tasks.md
+++ b/tasks.md
@@ -0,0 +1,361 @@
+# Logging Audit & Improvement Tasks
+
+This tasks list is written for an AI agent to systematically rework and add logging across the AniWorld codebase. Each task includes a specific file and the intended logging improvement.
+
+> 📌 **Goal:** Improve log clarity, reduce noise, ensure consistent structured context, and add missing logs for important events and failures.
+
+### How to execute these tasks
+
+1. **For each file** in the list, open it and locate existing `logger.*` calls.
+2. **Ask:**
+    - Is the message actionable and human-readable?
+    - Does it include key context (ids, names, paths, operation results)?
+    - Is the level correct (DEBUG/INFO/WARNING/ERROR/CRITICAL)?
+    - Are exceptions logged using `logger.exception(...)` or `exc_info=True`?
+    - Are we avoiding log spam (e.g., loops, repeated messages)?
+3. **Add or adjust logs** by:
+    - Moving noisy INFO logs to DEBUG unless they represent important state changes.
+    - Adding INFO-level logs at boundary points (entry/exit of operations, request endpoints).
+    - Including structured `extra={...}` fields where helpful.
+    - Logging minimal useful object representations (e.g., `series.id`, `file_path`).
+4. **Validate** that changes do not expose secrets or PII in logs.
+5. **Follow the project logging style** in `docs/InstructionsLogging.md` when drafting or refactoring log calls.
+
+---
+
+## 1) Core Logging Infrastructure
+
+- **`src/infrastructure/logging/logger.py`**
+    - Ensure the project has a single, consistent logging entry point for all modules.
+    - Validate that logger names are consistent and allow per-module filtering.
+    - Confirm structured logging support (extra/context fields) is correctly implemented.
+
+- **`src/server/config/logging_config.py`**
+    - Verify log format templates are clear, consistent, and include timestamp + level + module.
+    - Make sure log level configuration is reliable and respects env/config.
+    - Confirm file rotation, log file creation, and log handlers are configured properly.
+
+- **`src/server/utils/logging.py`**
+    - Confirm helper utilities for creating loggers and configuring log levels are correct.
+    - Ensure default logger levels (download, security, performance) are clear and non-spammy.
+
+- **`src/server/utils/log_manager.py`**
+    - Review any dynamic log level changes and confirm they are safe and properly logged.
+
+---
+
+## 2) Startup & Runtime Lifecycle Logging
+
+- **`src/server/fastapi_app.py`**
+    - Refactor startup/shutdown logs to ensure each major step (db init, service initialization, background tasks) is logged at an appropriate level.
+    - Add missing context (e.g., which service or component is starting/stopping) in the messages.
+    - Ensure exceptions are logged via `logger.exception(...)` where appropriate.
+
+- **`src/server/services/initialization_service.py`**
+    - Add logs for key initialization steps and failures.
+
+- **`src/server/services/cache_service.py`**
+    - Ensure cache hits/misses (if relevant) are logged at DEBUG and errors are logged at WARNING/ERROR.
+
+- **`src/server/services/notification_service.py`**
+    - Add logs for notification events and failures.
+
+---
+
+## 3) API / Controller Logging
+
+- **`src/server/api/anime.py`**
+- **`src/server/api/nfo.py`**
+- **`src/server/api/config.py`**
+- **`src/server/api/health.py`**
+- **`src/server/api/scheduler.py`**
+- **`src/server/api/logging.py`**
+- **`src/server/api/websocket.py`**
+
+For each API module:
+
+- Ensure each endpoint logs at INFO when invoked (at least the route and key identifiers).
+- Add DEBUG logs for important steps inside handlers (data validation, service calls, responses) without spamming.
+- Ensure errors/exceptions are logged with stack traces.
+
+---
+
+## 4) Middleware and Error Handling
+
+- **`src/server/middleware/error_handler.py`**
+    - Make sure handled exceptions are logged appropriately.
+    - Avoid logging the same error multiple times at the same level.
+
+- **`src/server/middleware/security.py`**
+    - Ensure security-related events (auth failures, suspicious activity) are logged at WARNING/ERROR with context.
+
+---
+
+## 5) Database & Transaction Logging
+
+- **`src/server/database/connection.py`**
+    - Log key connection lifecycle events and failures.
+
+- **`src/server/database/init.py`**
+    - Ensure DB initialization steps are logged.
+
+- **`src/server/database/service.py`**
+    - Review logs for CRUD operations on series and episodes.
+    - Confirm that success logs include ids and key values.
+    - Ensure exceptions are logged with full context.
+
+- **`src/server/database/system_settings_service.py`**
+    - Confirm logs for scan completion/reset are accurate and not noisy.
+
+- **`src/server/database/transaction.py`**
+    - Ensure transaction start/commit/rollback logs are DEBUG, and warnings/errors for failures are logged.
+    - Confirm savepoint logs include savepoint names.
+
+---
+
+## 6) Background Services & Worker Logging
+
+- **`src/server/services/background_loader_service.py`**
+    - Add/info logs for job start/completion, and any batch summaries.
+
+- **`src/server/services/scheduler_service.py`**
+    - Ensure job scheduling, execution, and failure are logged.
+
+- **`src/server/services/download_service.py`**
+    - Add informative logs for downloads (start, success, failure, retry).
+
+- **`src/server/services/queue_repository.py`**
+    - Ensure queue operations are logged at suitable levels (DEBUG for enqueue/dequeue, WARNING for errors).
+
+- **`src/server/services/progress_service.py`**
+    - Add logs for progress updates and any errors.
+
+- **`src/server/services/scan_service.py`**
+    - Add logs for scan start/completion, skipped items, and failures.
+
+- **`src/server/services/anime_service.py`**
+    - Ensure high-level operations (create/update/delete) are logged with identifiers.
+
+- **`src/server/services/websocket_service.py`**
+    - Add logs for connection/disconnection events and key message paths.
+
+---
+
+## 7) Server Utilities & Helpers
+
+- **`src/server/utils/dependencies.py`**
+    - Ensure dependency injections and resource retrievals are logged at DEBUG when there is failure.
+
+- **`src/server/utils/error_tracking.py`**
+    - Verify errors are logged with context and stack trace.
+
+- **`src/server/utils/metrics.py`**
+    - Ensure metrics collection failures are logged appropriately.
+
+- **`src/server/utils/system.py`**
+    - Add logs for system-level operations (memory, disk checks) if missing.
+
+- **`src/server/utils/template_helpers.py`**
+    - Add logs for template rendering errors.
+
+---
+
+## 8) Core Application Logging
+
+- **`src/core/SeriesApp.py`**
+- **`src/core/SerieScanner.py`**
+
+For core application logic:
+
+- Ensure key events (scan start/end, errors, decisions) are logged at correct level, and logs are not too noisy when scanning large directories.
+
+---
+
+## 9) Providers / Integration Logging
+
+- **`src/core/providers/aniworld_provider.py`**
+- **`src/core/providers/enhanced_provider.py`**
+- **`src/core/providers/monitored_provider.py`**
+- **`src/core/providers/failover.py`**
+- **`src/core/providers/config_manager.py`**
+- **`src/core/providers/health_monitor.py`**
+
+For provider modules:
+
+- Ensure failures / retries / fallbacks are logged at WARNING or ERROR with context.
+- Log important state changes at INFO (e.g., provider enabled/disabled).
+
+---
+
+## 10) Core Services / Utility Logging
+
+- **`src/core/services/nfo_service.py`**
+- **`src/core/services/nfo_factory.py`**
+- **`src/core/services/nfo_repair_service.py`**
+- **`src/core/services/series_manager_service.py`**
+- **`src/core/services/tmdb_client.py`**
+
+Ensure all operations that can fail or produce important output are logged with enough context (series id/name, operation result, duration).
+
+- **`src/core/utils/nfo_generator.py`**
+- **`src/core/utils/nfo_mapper.py`**
+- **`src/core/utils/image_downloader.py`**
+
+Ensure any external calls (HTTP, file I/O) and failures are logged with reasons.
+
+---
+
+## 11) CLI & Scripts
+
+- **`src/cli/nfo_cli.py`**
+    - Ensure startup logs are concise; reduce repetitive output; group statistics.
+
+- **`Docker/test_vpn.py`**
+    - Ensure logging is clear and not too verbose (especially in loops).
+
+---
+
+## 12) Tests (Verify Logging Behavior)
+
+- **`tests/unit/test_infrastructure_logger.py`**
+- **`tests/unit/test_logging.py`**
+- **`tests/unit/test_nfo_update_parsing.py`**
+- **`tests/unit/test_parallel_anime_add.py`**
+- **`tests/api/test_concurrent_anime_add.py`**
+- **`tests/performance/test_nfo_batch_performance.py`**
+- **`tests/performance/test_websocket_load.py`**
+- **`tests/security/test_encryption_security.py`**
+
+Tasks:
+
+- Ensure log capture/formatting tests are up to date.
+- Add tests to validate that key operations log at expected levels (INFO/DEBUG/ERROR) and include required context.
+
+---
+
+## 13) Follow-up Tasks (Post-Refactor)
+
+1. Run the full test suite to confirm logging changes do not break assertions.
+2. Verify logs during normal runs are not overly noisy; tune log levels accordingly.
+3. Ensure deployment/production log behavior matches expectations (rotate logs, preserve errors, and hide debug output).
+
+---
+
+## 14) Full Python File Inventory (excluding tests)
+
+Below is the complete list of all `.py` files in the repo (excluding anything under `tests/`).
+
+### 14a) Mutable logging tasks (edit these files)
+
+These are files where logging should be added/updated as part of the audit.
+
+- **`Docker/test_vpn.py`** – Review logging clarity, avoid verbose loops, and ensure errors are logged with context.
+- **`run_server.py`** – Ensure server startup logs (and any error paths) have clear context and use proper log levels.
+- **`src/cli/nfo_cli.py`** – Confirm CLI output is logged appropriately with summary reporting.
+- **`src/config/settings.py`** – Ensure config parsing failures are logged and invalid values are handled or reported.
+- **`src/core/entities/nfo_models.py`** – Add logs for model validation errors or unexpected data.
+- **`src/core/entities/SerieList.py`** – Ensure list operations and conversions log useful identifiers.
+- **`src/core/entities/series.py`** – Log key series state changes and validation failures.
+- **`src/core/error_handler.py`** – Ensure caught errors are logged and propagated correctly.
+- **`src/core/exceptions/Exceptions.py`** – Validate exception definitions support useful logging messages.
+- **`src/core/interfaces/callbacks.py`** – Log callback errors or unexpected callback payloads.
+- **`src/core/interfaces/providers.py`** – Ensure provider interface errors are logged clearly.
+- **`src/core/providers/aniworld_provider.py`** – Add logs for API calls, retries, and failure reasons.
+- **`src/core/providers/base_provider.py`** – Ensure base provider logs setup and teardown.
+- **`src/core/providers/config_manager.py`** – Log config load/refresh and validation errors.
+- **`src/core/providers/enhanced_provider.py`** – Review logging for error reporting and rate limiting.
+- **`src/core/providers/failover.py`** – Ensure failover decisions and switchovers are logged.
+- **`src/core/providers/health_monitor.py`** – Add logs for health-check results and degraded state.
+- **`src/core/providers/monitored_provider.py`** – Ensure monitored provider metrics and errors are logged.
+- **`src/core/providers/provider_config.py`** – Log config validation and defaulting behavior.
+- **`src/core/providers/provider_factory.py`** – Ensure factory creation errors are logged with context.
+- **`src/core/providers/streaming/Provider.py`** – Add logs for stream start/stop and errors.
+- **`src/core/providers/streaming/voe.py`** – Log VOE fetch attempts and failures.
+- **`src/core/SeriesApp.py`** – Confirm app lifecycle events and errors are logged.
+- **`src/core/SerieScanner.py`** – Ensure scan start/end, skipped items, and errors are logged.
+- **`src/core/services/nfo_factory.py`** – Log NFO generation steps and failures.
+- **`src/core/services/nfo_repair_service.py`** – Ensure repair actions and failures are logged.
+- **`src/core/services/nfo_service.py`** – Add logs for NFO operations and exceptions.
+- **`src/core/services/series_manager_service.py`** – Log series management actions and key identifiers.
+- **`src/core/services/tmdb_client.py`** – Ensure external API calls and failures are logged.
+- **`src/core/utils/image_downloader.py`** – Add logs for download attempts, successes, and errors.
+- **`src/core/utils/nfo_generator.py`** – Ensure generation steps and failures are logged.
+- **`src/core/utils/nfo_mapper.py`** – Add logs for mapping failures and questionable data.
+- **`src/infrastructure/logging/logger.py`** – Validate centralized logger behavior and structured logging support.
+- **`src/infrastructure/logging/uvicorn_config.py`** – Ensure Uvicorn logging config is correct and includes context.
+- **`src/infrastructure/security/config_encryption.py`** – Log encryption/decryption failures and warnings.
+- **`src/infrastructure/security/database_integrity.py`** – Add logs for integrity check results and failures.
+- **`src/infrastructure/security/file_integrity.py`** – Ensure file integrity checks log results and errors.
+- **`src/server/api/anime.py`** – Confirm endpoints log requests, key params, and errors.
+- **`src/server/api/auth.py`** – Ensure auth failures and token operations are logged.
+- **`src/server/api/config.py`** – Add logs for config read/write and validation.
+- **`src/server/api/download.py`** – Log download request lifecycle and errors.
+- **`src/server/api/health.py`** – Ensure health checks log status and failure reasons.
+- **`src/server/api/logging.py`** – Ensure log endpoint behavior is clear and properly logged.
+- **`src/server/api/nfo.py`** – Add logs for NFO endpoint actions and errors.
+- **`src/server/api/scheduler.py`** – Log scheduling actions, triggers, and failures.
+- **`src/server/api/websocket.py`** – Ensure websocket connect/disconnect and message errors are logged.
+- **`src/server/controllers/error_controller.py`** – Add logs for handled errors.
+- **`src/server/controllers/page_controller.py`** – Add logs for page rendering errors.
+- **`src/server/database/base.py`** – Ensure base DB operations log unexpected issues.
+- **`src/server/database/connection.py`** – Add connection lifecycle and error logs.
+- **`src/server/database/init.py`** – Ensure DB init steps are logged.
+- **`src/server/database/models.py`** – Validate model validation errors are logged.
+- **`src/server/database/service.py`** – Confirm CRUD operations log with context.
+- **`src/server/database/system_settings_service.py`** – Ensure system settings actions log correctly.
+- **`src/server/database/transaction.py`** – Verify transaction start/commit/rollback logging.
+- **`src/server/fastapi_app.py`** – Review startup/shutdown and error logging.
+- **`src/server/middleware/auth.py`** – Ensure auth middleware logs auth events and failures.
+- **`src/server/middleware/error_handler.py`** – Confirm error handling logs exceptions clearly.
+- **`src/server/middleware/security.py`** – Ensure security events are logged with context.
+- **`src/server/middleware/setup_redirect.py`** – Add logs for redirect behavior and errors.
+- **`src/server/models/anime.py`** – Ensure model validation errors are logged.
+- **`src/server/models/auth.py`** – Confirm auth model logging for invalid data.
+- **`src/server/models/config.py`** – Add logs for config model validation.
+- **`src/server/models/download.py`** – Ensure download model failures are logged.
+- **`src/server/models/nfo.py`** – Ensure NFO model issues are logged.
+- **`src/server/models/websocket.py`** – Confirm websocket model errors are logged.
+- **`src/server/services/anime_service.py`** – Ensure operations log relevant ids and results.
+- **`src/server/services/auth_service.py`** – Add logs for auth operations and failures.
+- **`src/server/services/background_loader_service.py`** – Ensure background jobs log start/stop and errors.
+- **`src/server/services/cache_service.py`** – Confirm caching logs are sane and useful.
+- **`src/server/services/config_service.py`** – Ensure config updates and errors are logged.
+- **`src/server/services/download_service.py`** – Add download lifecycle logs.
+- **`src/server/services/initialization_service.py`** – Confirm initialization steps are logged.
+- **`src/server/services/notification_service.py`** – Ensure notification dispatch/failure logs exist.
+- **`src/server/services/progress_service.py`** – Add progress update logs.
+- **`src/server/services/queue_repository.py`** – Ensure queue operations are logged.
+- **`src/server/services/scan_service.py`** – Confirm scan lifecycle and failure logging.
+- **`src/server/services/scheduler_service.py`** – Ensure scheduler actions and failures are logged.
+- **`src/server/services/websocket_service.py`** – Confirm websocket lifecycle and message errors are logged.
+- **`src/server/utils/dependencies.py`** – Ensure dependency errors are logged.
+- **`src/server/utils/error_tracking.py`** – Verify error tracking logs contain context.
+- **`src/server/utils/filesystem.py`** – Add filesystem operation logs and errors.
+- **`src/server/utils/logging.py`** – Ensure helper logging utilities behave as expected.
+- **`src/server/utils/log_manager.py`** – Review dynamic log-level changes and logs.
+- **`src/server/utils/media.py`** – Add logs for media operations and failures.
+- **`src/server/utils/metrics.py`** – Ensure metric collection/logging failures are captured.
+- **`src/server/utils/system.py`** – Add system-level operation logs and error handling.
+- **`src/server/utils/template_helpers.py`** – Ensure template rendering errors are logged.
+- **`src/server/utils/templates.py`** – Add logs for template loading and render errors.
+- **`src/server/utils/validators.py`** – Ensure validation failures are logged appropriately.
+- **`src/server/web/middleware/__init__.py`** – Verify middleware initialization logging.
+
+### 14b) Reference / low-touch files (review only)
+
+- **`src/core/__init__.py`** – Verify any module-level logging is appropriate.
+- **`src/core/providers/__init__.py`** – Verify module-level logging is appropriate.
+- **`src/infrastructure/logging/__init__.py`** – Confirm module exports and initialization logs are appropriate.
+- **`src/infrastructure/security/__init__.py`** – Verify module-level logging is appropriate.
+- **`src/server/api/__init__.py`** – Verify any module setup logging.
+- **`src/server/config/__init__.py`** – Verify config initialization logging.
+- **`src/server/controllers/__init__.py`** – Verify module import logging is non-spammy.
+- **`src/server/database/__init__.py`** – Confirm any initialization logs.
+- **`src/server/exceptions/__init__.py`** – Confirm module-level logging is appropriate.
+- **`src/server/models/__init__.py`** – Verify any module init logs.
+- **`src/server/utils/__init__.py`** – Confirm initialization logging is appropriate.
+- **`src/server/web/middleware/__init__.py`** – Verify middleware initialization logging.
+
+---
+
+> ✅ **Done:** This `tasks.md` file enumerates each relevant source file and a clear instruction for how to improve or add logging in that file.