diff --git a/docs/InstructionsLogging.md b/docs/InstructionsLogging.md new file mode 100644 index 0000000..1eff714 --- /dev/null +++ b/docs/InstructionsLogging.md @@ -0,0 +1,90 @@ +# Logging Instructions + +This document describes how to write and refactor logging across the AniWorld codebase to make logs **human-readable**, **debug-friendly**, and **noise-free**. + +> βœ… Goal: Logs should help a developer understand what happened, why it happened, and what to inspect next β€” without overwhelming them with duplicates or irrelevant details. + +--- + +## 1. Principles for Great Logs + +### 1.1 Use the Right Log Level + +- `DEBUG`: Detailed internal state useful when debugging a specific issue (e.g., decision points, returned values, request/response payloads). Not for normal operation. +- `INFO`: High-level events that represent what the system is doing (e.g., "Import started", "New series added", "Config reloaded"). Use sparingly. +- `WARNING`: Something unexpected happened, but the system can continue (e.g., missing optional file, fallback behavior). +- `ERROR`: An operation failed and needs attention (e.g., exception caught, failed database write). +- `CRITICAL`: The system is in an unusable state (e.g., config corruption, failed startup). + +### 1.2 Keep Logs Human-Readable + +- Write messages in a clear, descriptive sentence-style format. +- Avoid cryptic codes or single-word log messages. +- Prefer `logger.debug("... %s", value)`-style formatting over f-strings to avoid unnecessary work when the log level is disabled. + +### 1.3 Avoid Log Spam + +- Don’t log inside hot loops unless you explicitly aggregate and log a summary (e.g., "Processed 124 files, 3 failures"). +- Avoid repeated/logging the same event at the same level (e.g., do not log "Retrying" 10 times at INFO; log once at INFO and then use DEBUG for each retry). +- Use rate limiting or debounce patterns for logs that can fire rapidly (e.g., external service health checks). +- Prefer a single higher-level log with context rather than many low-level logs that clutter output. + +### 1.4 Log Objects Usefully + +- When logging objects, log the minimal useful representation (e.g., ID, name, status) rather than the full object or its memory address. +- If an object has a `.dict()`, `.to_dict()`, or `.as_dict()` helper (common in Pydantic models), log that rather than relying on `repr()`. +- Add a `__repr__` or `__str__` implementation to domain models that returns a helpful, concise string with key identifiers. +- Use structured logging (e.g., `logger.info("Series added", extra={"series_id": series.id, "title": series.title})`) where supported. +- For exceptions, prefer `logger.exception("Failed to ...")` to capture stack traces. + +--- + +## 2. Refactoring Existing Logs + +When improving or refactoring existing log statements, aim to make them: + +- **Actionable**: A developer reading the log should know what happened and what to check next. +- **Non-redundant**: Remove duplicates and ensure only one log records the same high-level event at a given level. +- **Context-rich**: Include identifiers (e.g., `series_id`, `file_path`, `user_id`) and key state that explains why a decision was made. +- **Level-appropriate**: Downgrade noisy INFO logs to DEBUG, and elevate critical failures to ERROR/CRITICAL. + +### 2.1 Refactor Checklist + +1. **Locate noisy logs**: Search for repeated messages (e.g., "Start", "Done") and determine whether they should be DEBUG or removed. +2. **Replace ad-hoc prints**: Remove `print()` statements or `print(obj)` and replace with `logger.*` calls. +3. **Use structured context**: If a function logs multiple related messages, include the same context in each (e.g., `extra={"series_id": series.id}`) or use a context manager that attaches it. +4. **Validate object output**: Ensure any logged object produces a useful representation (add methods or translate to dict). If not, log the key fields explicitly. +5. **Batch repetitive events**: If a loop logs per item, consider collecting stats and logging a summary at the end. + +## 3. Adding New Logs + +When adding logs to new code paths: + +- Log **important state transitions** (e.g., "Queue started", "Download completed", "Config reloaded"). +- For error paths, include what failed and why (e.g., "Could not load config from X: {exc}"). +- Prefer logging at the boundaries of operations, not deep inside utility functions unless it aids debugging. +- Write logs in full sentences, with a clear subject, verb, and object. + +--- + +## 4. Example Patterns + +```python +logger.info("Import completed", extra={"series_id": series.id, "count": len(imported)}) + +logger.debug( + "Fetched feed items", + extra={"feed_url": feed.url, "item_count": len(items)}, +) + +try: + result = download_episode(episode) +except Exception: + logger.exception("Failed to download episode %s", episode.id) +``` + +> πŸ’‘ When in doubt, favor **fewer, richer logs** over many noisy logs. + +# importent + +note which file is done. diff --git a/tasks.md b/tasks.md new file mode 100644 index 0000000..fc9c018 --- /dev/null +++ b/tasks.md @@ -0,0 +1,361 @@ +# Logging Audit & Improvement Tasks + +This tasks list is written for an AI agent to systematically rework and add logging across the AniWorld codebase. Each task includes a specific file and the intended logging improvement. + +> πŸ“Œ **Goal:** Improve log clarity, reduce noise, ensure consistent structured context, and add missing logs for important events and failures. + +### How to execute these tasks + +1. **For each file** in the list, open it and locate existing `logger.*` calls. +2. **Ask:** + - Is the message actionable and human-readable? + - Does it include key context (ids, names, paths, operation results)? + - Is the level correct (DEBUG/INFO/WARNING/ERROR/CRITICAL)? + - Are exceptions logged using `logger.exception(...)` or `exc_info=True`? + - Are we avoiding log spam (e.g., loops, repeated messages)? +3. **Add or adjust logs** by: + - Moving noisy INFO logs to DEBUG unless they represent important state changes. + - Adding INFO-level logs at boundary points (entry/exit of operations, request endpoints). + - Including structured `extra={...}` fields where helpful. + - Logging minimal useful object representations (e.g., `series.id`, `file_path`). +4. **Validate** that changes do not expose secrets or PII in logs. +5. **Follow the project logging style** in `docs/InstructionsLogging.md` when drafting or refactoring log calls. + +--- + +## 1) Core Logging Infrastructure + +- **`src/infrastructure/logging/logger.py`** + - Ensure the project has a single, consistent logging entry point for all modules. + - Validate that logger names are consistent and allow per-module filtering. + - Confirm structured logging support (extra/context fields) is correctly implemented. + +- **`src/server/config/logging_config.py`** + - Verify log format templates are clear, consistent, and include timestamp + level + module. + - Make sure log level configuration is reliable and respects env/config. + - Confirm file rotation, log file creation, and log handlers are configured properly. + +- **`src/server/utils/logging.py`** + - Confirm helper utilities for creating loggers and configuring log levels are correct. + - Ensure default logger levels (download, security, performance) are clear and non-spammy. + +- **`src/server/utils/log_manager.py`** + - Review any dynamic log level changes and confirm they are safe and properly logged. + +--- + +## 2) Startup & Runtime Lifecycle Logging + +- **`src/server/fastapi_app.py`** + - Refactor startup/shutdown logs to ensure each major step (db init, service initialization, background tasks) is logged at an appropriate level. + - Add missing context (e.g., which service or component is starting/stopping) in the messages. + - Ensure exceptions are logged via `logger.exception(...)` where appropriate. + +- **`src/server/services/initialization_service.py`** + - Add logs for key initialization steps and failures. + +- **`src/server/services/cache_service.py`** + - Ensure cache hits/misses (if relevant) are logged at DEBUG and errors are logged at WARNING/ERROR. + +- **`src/server/services/notification_service.py`** + - Add logs for notification events and failures. + +--- + +## 3) API / Controller Logging + +- **`src/server/api/anime.py`** +- **`src/server/api/nfo.py`** +- **`src/server/api/config.py`** +- **`src/server/api/health.py`** +- **`src/server/api/scheduler.py`** +- **`src/server/api/logging.py`** +- **`src/server/api/websocket.py`** + +For each API module: + +- Ensure each endpoint logs at INFO when invoked (at least the route and key identifiers). +- Add DEBUG logs for important steps inside handlers (data validation, service calls, responses) without spamming. +- Ensure errors/exceptions are logged with stack traces. + +--- + +## 4) Middleware and Error Handling + +- **`src/server/middleware/error_handler.py`** + - Make sure handled exceptions are logged appropriately. + - Avoid logging the same error multiple times at the same level. + +- **`src/server/middleware/security.py`** + - Ensure security-related events (auth failures, suspicious activity) are logged at WARNING/ERROR with context. + +--- + +## 5) Database & Transaction Logging + +- **`src/server/database/connection.py`** + - Log key connection lifecycle events and failures. + +- **`src/server/database/init.py`** + - Ensure DB initialization steps are logged. + +- **`src/server/database/service.py`** + - Review logs for CRUD operations on series and episodes. + - Confirm that success logs include ids and key values. + - Ensure exceptions are logged with full context. + +- **`src/server/database/system_settings_service.py`** + - Confirm logs for scan completion/reset are accurate and not noisy. + +- **`src/server/database/transaction.py`** + - Ensure transaction start/commit/rollback logs are DEBUG, and warnings/errors for failures are logged. + - Confirm savepoint logs include savepoint names. + +--- + +## 6) Background Services & Worker Logging + +- **`src/server/services/background_loader_service.py`** + - Add/info logs for job start/completion, and any batch summaries. + +- **`src/server/services/scheduler_service.py`** + - Ensure job scheduling, execution, and failure are logged. + +- **`src/server/services/download_service.py`** + - Add informative logs for downloads (start, success, failure, retry). + +- **`src/server/services/queue_repository.py`** + - Ensure queue operations are logged at suitable levels (DEBUG for enqueue/dequeue, WARNING for errors). + +- **`src/server/services/progress_service.py`** + - Add logs for progress updates and any errors. + +- **`src/server/services/scan_service.py`** + - Add logs for scan start/completion, skipped items, and failures. + +- **`src/server/services/anime_service.py`** + - Ensure high-level operations (create/update/delete) are logged with identifiers. + +- **`src/server/services/websocket_service.py`** + - Add logs for connection/disconnection events and key message paths. + +--- + +## 7) Server Utilities & Helpers + +- **`src/server/utils/dependencies.py`** + - Ensure dependency injections and resource retrievals are logged at DEBUG when there is failure. + +- **`src/server/utils/error_tracking.py`** + - Verify errors are logged with context and stack trace. + +- **`src/server/utils/metrics.py`** + - Ensure metrics collection failures are logged appropriately. + +- **`src/server/utils/system.py`** + - Add logs for system-level operations (memory, disk checks) if missing. + +- **`src/server/utils/template_helpers.py`** + - Add logs for template rendering errors. + +--- + +## 8) Core Application Logging + +- **`src/core/SeriesApp.py`** +- **`src/core/SerieScanner.py`** + +For core application logic: + +- Ensure key events (scan start/end, errors, decisions) are logged at correct level, and logs are not too noisy when scanning large directories. + +--- + +## 9) Providers / Integration Logging + +- **`src/core/providers/aniworld_provider.py`** +- **`src/core/providers/enhanced_provider.py`** +- **`src/core/providers/monitored_provider.py`** +- **`src/core/providers/failover.py`** +- **`src/core/providers/config_manager.py`** +- **`src/core/providers/health_monitor.py`** + +For provider modules: + +- Ensure failures / retries / fallbacks are logged at WARNING or ERROR with context. +- Log important state changes at INFO (e.g., provider enabled/disabled). + +--- + +## 10) Core Services / Utility Logging + +- **`src/core/services/nfo_service.py`** +- **`src/core/services/nfo_factory.py`** +- **`src/core/services/nfo_repair_service.py`** +- **`src/core/services/series_manager_service.py`** +- **`src/core/services/tmdb_client.py`** + +Ensure all operations that can fail or produce important output are logged with enough context (series id/name, operation result, duration). + +- **`src/core/utils/nfo_generator.py`** +- **`src/core/utils/nfo_mapper.py`** +- **`src/core/utils/image_downloader.py`** + +Ensure any external calls (HTTP, file I/O) and failures are logged with reasons. + +--- + +## 11) CLI & Scripts + +- **`src/cli/nfo_cli.py`** + - Ensure startup logs are concise; reduce repetitive output; group statistics. + +- **`Docker/test_vpn.py`** + - Ensure logging is clear and not too verbose (especially in loops). + +--- + +## 12) Tests (Verify Logging Behavior) + +- **`tests/unit/test_infrastructure_logger.py`** +- **`tests/unit/test_logging.py`** +- **`tests/unit/test_nfo_update_parsing.py`** +- **`tests/unit/test_parallel_anime_add.py`** +- **`tests/api/test_concurrent_anime_add.py`** +- **`tests/performance/test_nfo_batch_performance.py`** +- **`tests/performance/test_websocket_load.py`** +- **`tests/security/test_encryption_security.py`** + +Tasks: + +- Ensure log capture/formatting tests are up to date. +- Add tests to validate that key operations log at expected levels (INFO/DEBUG/ERROR) and include required context. + +--- + +## 13) Follow-up Tasks (Post-Refactor) + +1. Run the full test suite to confirm logging changes do not break assertions. +2. Verify logs during normal runs are not overly noisy; tune log levels accordingly. +3. Ensure deployment/production log behavior matches expectations (rotate logs, preserve errors, and hide debug output). + +--- + +## 14) Full Python File Inventory (excluding tests) + +Below is the complete list of all `.py` files in the repo (excluding anything under `tests/`). + +### 14a) Mutable logging tasks (edit these files) + +These are files where logging should be added/updated as part of the audit. + +- **`Docker/test_vpn.py`** – Review logging clarity, avoid verbose loops, and ensure errors are logged with context. +- **`run_server.py`** – Ensure server startup logs (and any error paths) have clear context and use proper log levels. +- **`src/cli/nfo_cli.py`** – Confirm CLI output is logged appropriately with summary reporting. +- **`src/config/settings.py`** – Ensure config parsing failures are logged and invalid values are handled or reported. +- **`src/core/entities/nfo_models.py`** – Add logs for model validation errors or unexpected data. +- **`src/core/entities/SerieList.py`** – Ensure list operations and conversions log useful identifiers. +- **`src/core/entities/series.py`** – Log key series state changes and validation failures. +- **`src/core/error_handler.py`** – Ensure caught errors are logged and propagated correctly. +- **`src/core/exceptions/Exceptions.py`** – Validate exception definitions support useful logging messages. +- **`src/core/interfaces/callbacks.py`** – Log callback errors or unexpected callback payloads. +- **`src/core/interfaces/providers.py`** – Ensure provider interface errors are logged clearly. +- **`src/core/providers/aniworld_provider.py`** – Add logs for API calls, retries, and failure reasons. +- **`src/core/providers/base_provider.py`** – Ensure base provider logs setup and teardown. +- **`src/core/providers/config_manager.py`** – Log config load/refresh and validation errors. +- **`src/core/providers/enhanced_provider.py`** – Review logging for error reporting and rate limiting. +- **`src/core/providers/failover.py`** – Ensure failover decisions and switchovers are logged. +- **`src/core/providers/health_monitor.py`** – Add logs for health-check results and degraded state. +- **`src/core/providers/monitored_provider.py`** – Ensure monitored provider metrics and errors are logged. +- **`src/core/providers/provider_config.py`** – Log config validation and defaulting behavior. +- **`src/core/providers/provider_factory.py`** – Ensure factory creation errors are logged with context. +- **`src/core/providers/streaming/Provider.py`** – Add logs for stream start/stop and errors. +- **`src/core/providers/streaming/voe.py`** – Log VOE fetch attempts and failures. +- **`src/core/SeriesApp.py`** – Confirm app lifecycle events and errors are logged. +- **`src/core/SerieScanner.py`** – Ensure scan start/end, skipped items, and errors are logged. +- **`src/core/services/nfo_factory.py`** – Log NFO generation steps and failures. +- **`src/core/services/nfo_repair_service.py`** – Ensure repair actions and failures are logged. +- **`src/core/services/nfo_service.py`** – Add logs for NFO operations and exceptions. +- **`src/core/services/series_manager_service.py`** – Log series management actions and key identifiers. +- **`src/core/services/tmdb_client.py`** – Ensure external API calls and failures are logged. +- **`src/core/utils/image_downloader.py`** – Add logs for download attempts, successes, and errors. +- **`src/core/utils/nfo_generator.py`** – Ensure generation steps and failures are logged. +- **`src/core/utils/nfo_mapper.py`** – Add logs for mapping failures and questionable data. +- **`src/infrastructure/logging/logger.py`** – Validate centralized logger behavior and structured logging support. +- **`src/infrastructure/logging/uvicorn_config.py`** – Ensure Uvicorn logging config is correct and includes context. +- **`src/infrastructure/security/config_encryption.py`** – Log encryption/decryption failures and warnings. +- **`src/infrastructure/security/database_integrity.py`** – Add logs for integrity check results and failures. +- **`src/infrastructure/security/file_integrity.py`** – Ensure file integrity checks log results and errors. +- **`src/server/api/anime.py`** – Confirm endpoints log requests, key params, and errors. +- **`src/server/api/auth.py`** – Ensure auth failures and token operations are logged. +- **`src/server/api/config.py`** – Add logs for config read/write and validation. +- **`src/server/api/download.py`** – Log download request lifecycle and errors. +- **`src/server/api/health.py`** – Ensure health checks log status and failure reasons. +- **`src/server/api/logging.py`** – Ensure log endpoint behavior is clear and properly logged. +- **`src/server/api/nfo.py`** – Add logs for NFO endpoint actions and errors. +- **`src/server/api/scheduler.py`** – Log scheduling actions, triggers, and failures. +- **`src/server/api/websocket.py`** – Ensure websocket connect/disconnect and message errors are logged. +- **`src/server/controllers/error_controller.py`** – Add logs for handled errors. +- **`src/server/controllers/page_controller.py`** – Add logs for page rendering errors. +- **`src/server/database/base.py`** – Ensure base DB operations log unexpected issues. +- **`src/server/database/connection.py`** – Add connection lifecycle and error logs. +- **`src/server/database/init.py`** – Ensure DB init steps are logged. +- **`src/server/database/models.py`** – Validate model validation errors are logged. +- **`src/server/database/service.py`** – Confirm CRUD operations log with context. +- **`src/server/database/system_settings_service.py`** – Ensure system settings actions log correctly. +- **`src/server/database/transaction.py`** – Verify transaction start/commit/rollback logging. +- **`src/server/fastapi_app.py`** – Review startup/shutdown and error logging. +- **`src/server/middleware/auth.py`** – Ensure auth middleware logs auth events and failures. +- **`src/server/middleware/error_handler.py`** – Confirm error handling logs exceptions clearly. +- **`src/server/middleware/security.py`** – Ensure security events are logged with context. +- **`src/server/middleware/setup_redirect.py`** – Add logs for redirect behavior and errors. +- **`src/server/models/anime.py`** – Ensure model validation errors are logged. +- **`src/server/models/auth.py`** – Confirm auth model logging for invalid data. +- **`src/server/models/config.py`** – Add logs for config model validation. +- **`src/server/models/download.py`** – Ensure download model failures are logged. +- **`src/server/models/nfo.py`** – Ensure NFO model issues are logged. +- **`src/server/models/websocket.py`** – Confirm websocket model errors are logged. +- **`src/server/services/anime_service.py`** – Ensure operations log relevant ids and results. +- **`src/server/services/auth_service.py`** – Add logs for auth operations and failures. +- **`src/server/services/background_loader_service.py`** – Ensure background jobs log start/stop and errors. +- **`src/server/services/cache_service.py`** – Confirm caching logs are sane and useful. +- **`src/server/services/config_service.py`** – Ensure config updates and errors are logged. +- **`src/server/services/download_service.py`** – Add download lifecycle logs. +- **`src/server/services/initialization_service.py`** – Confirm initialization steps are logged. +- **`src/server/services/notification_service.py`** – Ensure notification dispatch/failure logs exist. +- **`src/server/services/progress_service.py`** – Add progress update logs. +- **`src/server/services/queue_repository.py`** – Ensure queue operations are logged. +- **`src/server/services/scan_service.py`** – Confirm scan lifecycle and failure logging. +- **`src/server/services/scheduler_service.py`** – Ensure scheduler actions and failures are logged. +- **`src/server/services/websocket_service.py`** – Confirm websocket lifecycle and message errors are logged. +- **`src/server/utils/dependencies.py`** – Ensure dependency errors are logged. +- **`src/server/utils/error_tracking.py`** – Verify error tracking logs contain context. +- **`src/server/utils/filesystem.py`** – Add filesystem operation logs and errors. +- **`src/server/utils/logging.py`** – Ensure helper logging utilities behave as expected. +- **`src/server/utils/log_manager.py`** – Review dynamic log-level changes and logs. +- **`src/server/utils/media.py`** – Add logs for media operations and failures. +- **`src/server/utils/metrics.py`** – Ensure metric collection/logging failures are captured. +- **`src/server/utils/system.py`** – Add system-level operation logs and error handling. +- **`src/server/utils/template_helpers.py`** – Ensure template rendering errors are logged. +- **`src/server/utils/templates.py`** – Add logs for template loading and render errors. +- **`src/server/utils/validators.py`** – Ensure validation failures are logged appropriately. +- **`src/server/web/middleware/__init__.py`** – Verify middleware initialization logging. + +### 14b) Reference / low-touch files (review only) + +- **`src/core/__init__.py`** – Verify any module-level logging is appropriate. +- **`src/core/providers/__init__.py`** – Verify module-level logging is appropriate. +- **`src/infrastructure/logging/__init__.py`** – Confirm module exports and initialization logs are appropriate. +- **`src/infrastructure/security/__init__.py`** – Verify module-level logging is appropriate. +- **`src/server/api/__init__.py`** – Verify any module setup logging. +- **`src/server/config/__init__.py`** – Verify config initialization logging. +- **`src/server/controllers/__init__.py`** – Verify module import logging is non-spammy. +- **`src/server/database/__init__.py`** – Confirm any initialization logs. +- **`src/server/exceptions/__init__.py`** – Confirm module-level logging is appropriate. +- **`src/server/models/__init__.py`** – Verify any module init logs. +- **`src/server/utils/__init__.py`** – Confirm initialization logging is appropriate. +- **`src/server/web/middleware/__init__.py`** – Verify middleware initialization logging. + +--- + +> βœ… **Done:** This `tasks.md` file enumerates each relevant source file and a clear instruction for how to improve or add logging in that file.