2025-10-19 19:57:42 +02:00

19 KiB

Raw Blame History

Test Fixing Instructions for AniWorld Project

📋 General Instructions

Overview

This document lists all failed tests identified during the test run on October 19, 2025. Each test failure needs to be investigated and resolved. The failures are categorized by module/area for easier resolution.

Important Guidelines

Double-Check Before Fixing
- Always verify whether the test is wrong or the code is wrong
- Read the test implementation carefully
- Review the code being tested
- Check if the expected behavior in the test matches the actual requirements
- Consider if the test expectations are outdated or incorrect
Root Cause Analysis
- Understand why the test is failing before making changes
- Check if it's a:
  - Logic error in production code
  - Incorrect test expectations
  - Mock/fixture setup issue
  - Async/await issue
  - Authentication/authorization issue
  - Missing dependency or service
Fix Strategy
- Fix production code if the business logic is wrong
- Fix test code if the expectations are incorrect
- Update both if requirements have changed
- Document why you chose to fix test vs code
Testing Process
- Run the specific test after each fix to verify
- Run related tests to ensure no regression
- Run all tests after batch fixes to verify overall system health
Code Quality Standards
- Follow PEP8 and project coding standards
- Use type hints where applicable
- Write clear, self-documenting code
- Add comments for complex logic
- Update docstrings if behavior changes

📊 Test Failure Summary

Total Statistics:

✅ Passed: 409 tests
❌ Failed: 106 tests
⚠️ Errors: 68 tests
📝 Warnings: 638 warnings

🔴 Critical Issues to Address First

1. Async/Await Issues in Frontend Integration Tests

Priority: HIGH

Files Affected:

tests/integration/test_frontend_auth_integration.py

Symptoms:

RuntimeWarning: coroutine was never awaited
Multiple instances in test methods

Tests Failing:

test_login_returns_access_token
test_login_with_wrong_password
test_logout_clears_session
test_authenticated_request_without_token_returns_401
test_authenticated_request_with_invalid_token_returns_401
test_remember_me_extends_token_expiry
test_setup_fails_if_already_configured
test_weak_password_validation_in_setup
test_full_authentication_workflow
test_token_included_in_all_authenticated_requests

Root Cause: The tests are calling client.post() without awaiting the async operation.

Investigation Required:

Check if test methods are marked as async
Verify that await keyword is used for async client calls
Ensure proper pytest-asyncio setup

Fix Approach:

# WRONG:
client.post("/api/auth/setup", json={"master_password": "StrongP@ss123"})

# CORRECT:
await client.post("/api/auth/setup", json={"master_password": "StrongP@ss123"})

2. WebSocket Service Broadcast Failures

Priority: HIGH

Files Affected:

tests/unit/test_websocket_service.py

Tests Failing:

TestConnectionManager::test_send_personal_message
TestConnectionManager::test_broadcast_to_room
TestWebSocketService::test_broadcast_download_progress
TestWebSocketService::test_broadcast_download_complete
TestWebSocketService::test_broadcast_download_failed
TestWebSocketService::test_broadcast_queue_status
TestWebSocketService::test_send_error

Symptoms:

AssertionError: assert False
 +  where False = <AsyncMock name='mock.send_json' id='...'>.called

Root Cause: Mock WebSocket's send_json method is not being called as expected. Possible issues:

The broadcast is happening but to a different websocket instance
The websocket is disconnected or inactive before broadcast
The mock is not configured correctly

Investigation Required:

Review WebSocket service broadcast implementation
Check connection lifecycle management
Verify mock setup captures the actual call
Test if messages are being sent but to wrong connection

Fix Approach:

Review src/server/services/websocket_service.py broadcast methods
Check if connection is marked as active
Verify room membership before broadcast
Ensure mock WebSocket is properly registered with ConnectionManager

3. Authentication/Authorization Failures

Priority: HIGH

Files Affected:

tests/api/test_config_endpoints.py
tests/api/test_download_endpoints.py
tests/integration/test_auth_flow.py

Tests Failing:

Config Endpoints (7 failures)

test_validate_invalid_config
test_update_config_unauthorized
test_list_backups (403 instead of 200)
test_create_backup (403 instead of expected)
test_restore_backup (403 instead of expected)
test_delete_backup (403 instead of expected)
test_config_persistence (403 instead of expected)

Download Endpoints (2 failures)

test_get_queue_status_unauthorized
test_queue_endpoints_require_auth

Auth Flow Integration (43 failures)

Multiple test classes affected - all authentication flow tests

Symptoms:

403 Forbidden responses when 200 expected
Authentication/authorization not working as expected
Tokens not being validated correctly

Root Cause Options:

Middleware not properly checking authentication
Test fixtures not setting up auth correctly
Token generation/validation broken
Session management issues

Investigation Required:

Review src/server/middleware/auth.py
Check src/server/services/auth_service.py
Verify test fixtures for authentication setup
Check if routes are properly protected

4. Frontend Integration Test Errors

Priority: HIGH

Files Affected:

tests/frontend/test_existing_ui_integration.py

Tests Failing:

TestFrontendAuthentication::test_auth_status_endpoint_not_configured
TestFrontendAuthentication::test_auth_status_configured_not_authenticated
TestFrontendAuthentication::test_login_returns_jwt_token
TestFrontendAuthentication::test_unauthorized_request_returns_401
TestFrontendJavaScriptIntegration::test_frontend_handles_401_gracefully

Additional Test Errors (with assertion errors):

37 more tests in this file with ERROR status

Investigation Required:

Check test setup and fixtures
Verify client initialization
Review async handling

5. WebSocket Integration Test Failures

Priority: MEDIUM

Files Affected:

tests/integration/test_websocket.py

Tests Failing (48 total):

Connection Management (7 failures)

TestWebSocketConnection::test_websocket_endpoint_exists (ERROR)
TestWebSocketConnection::test_connection_manager_tracks_connections
TestWebSocketConnection::test_disconnect_removes_connection
TestWebSocketConnection::test_room_assignment_on_connection
TestWebSocketConnection::test_multiple_rooms_support

Message Broadcasting (4 failures)

TestMessageBroadcasting::test_broadcast_to_all_connections
TestMessageBroadcasting::test_broadcast_to_specific_room
TestMessageBroadcasting::test_broadcast_with_json_message
TestMessageBroadcasting::test_broadcast_handles_disconnected_clients

Progress Integration (3 failures)

TestProgressIntegration::test_download_progress_broadcasts_to_websocket
TestProgressIntegration::test_download_complete_notification
TestProgressIntegration::test_download_failed_notification

Queue Status Broadcasting (3 failures)

TestQueueStatusBroadcasting::test_queue_status_update_broadcast
TestQueueStatusBroadcasting::test_queue_item_added_notification
TestQueueStatusBroadcasting::test_queue_item_removed_notification

System Messaging (2 failures)

TestSystemMessaging::test_system_notification_broadcast
TestSystemMessaging::test_error_message_broadcast

Concurrent Connections (2 failures)

TestConcurrentConnections::test_multiple_clients_in_same_room
TestConcurrentConnections::test_concurrent_broadcasts_to_different_rooms

Connection Error Handling (3 failures)

TestConnectionErrorHandling::test_handle_send_failure
TestConnectionErrorHandling::test_handle_multiple_send_failures
TestConnectionErrorHandling::test_cleanup_after_disconnect

Message Formatting (2 failures)

TestMessageFormatting::test_message_structure_validation
TestMessageFormatting::test_different_message_types

Room Management (3 failures)

TestRoomManagement::test_room_creation_on_first_connection
TestRoomManagement::test_room_cleanup_when_empty
TestRoomManagement::test_client_can_be_in_one_room

Complete Workflow (2 failures)

TestCompleteWebSocketWorkflow::test_full_download_notification_workflow
TestCompleteWebSocketWorkflow::test_multi_room_workflow

Investigation Required:

WebSocket endpoint routing
Connection manager implementation
Room management logic
Broadcast mechanism

6. Download Flow Integration Test Errors

Priority: HIGH

Files Affected:

tests/integration/test_download_flow.py

All Tests Have ERROR Status (22 tests):

Authentication Requirements (4 errors)

TestAuthenticationRequirements::test_queue_status_requires_auth
TestAuthenticationRequirements::test_add_to_queue_requires_auth
TestAuthenticationRequirements::test_queue_control_requires_auth
TestAuthenticationRequirements::test_item_operations_require_auth

Download Flow End-to-End (5 errors)

TestDownloadFlowEndToEnd::test_add_episodes_to_queue
TestDownloadFlowEndToEnd::test_queue_status_after_adding_items
TestDownloadFlowEndToEnd::test_add_with_different_priorities
TestDownloadFlowEndToEnd::test_validation_error_for_empty_episodes
TestDownloadFlowEndToEnd::test_validation_error_for_invalid_priority

Queue Control Operations (4 errors)

TestQueueControlOperations::test_start_queue_processing
TestQueueControlOperations::test_pause_queue_processing
TestQueueControlOperations::test_resume_queue_processing
TestQueueControlOperations::test_clear_completed_downloads

Queue Item Operations (3 errors)

TestQueueItemOperations::test_remove_item_from_queue
TestQueueItemOperations::test_retry_failed_item
TestQueueItemOperations::test_reorder_queue_items

Progress Tracking (2 errors)

TestDownloadProgressTracking::test_queue_status_includes_progress
TestDownloadProgressTracking::test_queue_statistics

Error Handling (2 errors)

TestErrorHandlingAndRetries::test_handle_download_failure
TestErrorHandlingAndRetries::test_retry_count_increments

Concurrent Operations (2 errors)

TestConcurrentOperations::test_multiple_concurrent_downloads
TestConcurrentOperations::test_concurrent_status_requests

Plus Additional Tests:

Queue Persistence (2 errors)
WebSocket Integration (1 error)
Complete Download Workflow (2 errors)

Investigation Required:

Check test setup/teardown
Verify authentication setup in fixtures
Review async test handling

7. Download Endpoints API Test Errors

Priority: HIGH

Files Affected:

tests/api/test_download_endpoints.py

All Tests Have ERROR Status (18 tests):

test_get_queue_status
test_add_to_queue
test_add_to_queue_with_high_priority
test_add_to_queue_empty_episodes
test_add_to_queue_service_error
test_remove_from_queue_single
test_remove_from_queue_not_found
test_remove_multiple_from_queue
test_remove_multiple_empty_list
test_start_queue
test_stop_queue
test_pause_queue
test_resume_queue
test_reorder_queue
test_reorder_queue_not_found
test_clear_completed
test_retry_failed
test_retry_all_failed

Investigation Required:

Check test fixtures
Review authentication setup
Verify download service availability

8. Template Integration Test Failures

Priority: MEDIUM

Files Affected:

tests/unit/test_template_integration.py

Tests Failing:

TestTemplateIntegration::test_error_template_404
TestTemplateIntegration::test_queue_template_has_websocket_script
TestTemplateIntegration::test_templates_accessibility_features

Investigation Required:

Check template rendering
Verify template file locations
Review Jinja2 template configuration

9. Frontend Integration Smoke Tests

Priority: MEDIUM

Files Affected:

tests/integration/test_frontend_integration_smoke.py

Tests Failing:

TestFrontendIntegration::test_login_returns_jwt_token
TestFrontendIntegration::test_authenticated_endpoints_require_bearer_token
TestFrontendIntegration::test_queue_endpoints_accessible_with_token

Investigation Required:

Check authentication flow
Verify token generation
Review endpoint protection

⚠️ Non-Critical Warnings to Address

Deprecated Pydantic V1 API Usage

Count: ~20 warnings

Location: src/config/settings.py

Issue:

# Deprecated usage
Field(default="value", env="ENV_VAR")

# Should be:
Field(default="value", json_schema_extra={"env": "ENV_VAR"})

Files to Update:

src/config/settings.py - Update Field definitions
src/server/models/config.py - Update validators to @field_validator

Deprecated datetime.utcnow() Usage

Count: ~100+ warnings

Issue:

# Deprecated
datetime.utcnow()

# Should use
datetime.now(datetime.UTC)

Files to Update:

src/server/services/auth_service.py
src/server/services/download_service.py
src/server/services/progress_service.py
src/server/services/websocket_service.py
src/server/database/service.py
src/server/database/models.py
All test files using datetime.utcnow()

Deprecated FastAPI on_event

Count: 4 warnings

Location: src/server/fastapi_app.py

Issue:

# Deprecated
@app.on_event("startup")
@app.on_event("shutdown")

# Should use lifespan
from contextlib import asynccontextmanager

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Startup
    yield
    # Shutdown

Deprecated Pydantic .dict() Method

Count: ~5 warnings

Issue:

# Deprecated
session.dict()

# Should be
session.model_dump()

Files to Update:

src/server/middleware/auth.py
src/server/utils/dependencies.py

📝 Task Checklist for AI Agent

Phase 1: Critical Async Issues

Fix all async/await issues in test_frontend_auth_integration.py (10 tests)
Verify test methods are properly marked as async
Run and verify: pytest tests/integration/test_frontend_auth_integration.py -v

Phase 2: WebSocket Broadcast Issues

Investigate WebSocket service broadcast implementation
Fix mock configuration in test_websocket_service.py (7 tests)
Fix connection lifecycle management
Run and verify: pytest tests/unit/test_websocket_service.py -v

Phase 3: Authentication System

Debug auth middleware and service
Fix auth flow integration tests (43 tests)
Fix config endpoint auth issues (7 tests)
Fix download endpoint auth issues (2 tests)
Run and verify: pytest tests/integration/test_auth_flow.py -v
Run and verify: pytest tests/api/test_config_endpoints.py -v

Phase 4: Frontend Integration

Fix frontend auth integration tests (5 failures + 37 errors)
Fix frontend integration smoke tests (3 failures)
Run and verify: pytest tests/frontend/ -v

Phase 5: WebSocket Integration

Fix websocket integration tests (48 failures)
Test connection management
Test broadcasting mechanism
Run and verify: pytest tests/integration/test_websocket.py -v

Phase 6: Download Flow

Fix download flow integration tests (22+ errors)
Fix download endpoint API tests (18 errors)
Run and verify: pytest tests/integration/test_download_flow.py -v
Run and verify: pytest tests/api/test_download_endpoints.py -v

Phase 7: Template Integration

Fix template integration tests (3 failures)
Run and verify: pytest tests/unit/test_template_integration.py -v

Phase 8: Deprecation Warnings

Update Pydantic V2 Field definitions
Replace datetime.utcnow() with datetime.now(datetime.UTC)
Update FastAPI to use lifespan instead of on_event
Replace .dict() with .model_dump()
Run and verify: pytest tests/ -v --tb=short

Phase 9: Final Verification

Run all tests: pytest tests/ -v
Verify all tests pass
Verify warnings are reduced
Document any remaining issues

🎯 Success Criteria

All tests passing: 0 failures, 0 errors
Warnings reduced: Aim for < 50 warnings (mostly from dependencies)
Code quality maintained: No shortcuts or hacks
Documentation updated: Any behavior changes documented
Git commits: Logical, atomic commits with clear messages

📞 Escalation

If you encounter:

Architecture issues requiring design decisions
Tests that conflict with documented requirements
Breaking changes needed
Unclear requirements or expectations

Document the issue and escalate rather than guessing.

📚 Helpful Commands

# Run all tests
conda run -n AniWorld python -m pytest tests/ -v --tb=short

# Run specific test file
conda run -n AniWorld python -m pytest tests/unit/test_websocket_service.py -v

# Run specific test class
conda run -n AniWorld python -m pytest tests/unit/test_websocket_service.py::TestWebSocketService -v

# Run specific test
conda run -n AniWorld python -m pytest tests/unit/test_websocket_service.py::TestWebSocketService::test_broadcast_download_progress -v

# Run with extra verbosity
conda run -n AniWorld python -m pytest tests/ -vv

# Run with full traceback
conda run -n AniWorld python -m pytest tests/ -v --tb=long

# Run and stop at first failure
conda run -n AniWorld python -m pytest tests/ -v -x

# Run tests matching pattern
conda run -n AniWorld python -m pytest tests/ -v -k "auth"

# Show all print statements
conda run -n AniWorld python -m pytest tests/ -v -s

📖 Additional Notes

This document was generated on: October 19, 2025
Test run took: 6.63 seconds
Python environment: AniWorld (conda)
Framework: pytest with FastAPI TestClient

Remember: The goal is not just to make tests pass, but to ensure the system works correctly and reliably!

19 KiB Raw Blame History

Test Fixing Instructions for AniWorld Project

📋 General Instructions

Overview

Important Guidelines

📊 Test Failure Summary

🔴 Critical Issues to Address First

1. Async/Await Issues in Frontend Integration Tests

2. WebSocket Service Broadcast Failures

3. Authentication/Authorization Failures

Config Endpoints (7 failures)

Download Endpoints (2 failures)

Auth Flow Integration (43 failures)

4. Frontend Integration Test Errors

5. WebSocket Integration Test Failures

Connection Management (7 failures)

Message Broadcasting (4 failures)

Progress Integration (3 failures)

Queue Status Broadcasting (3 failures)

System Messaging (2 failures)

Concurrent Connections (2 failures)

Connection Error Handling (3 failures)

Message Formatting (2 failures)

Room Management (3 failures)

Complete Workflow (2 failures)

6. Download Flow Integration Test Errors

Authentication Requirements (4 errors)

Download Flow End-to-End (5 errors)

Queue Control Operations (4 errors)

Queue Item Operations (3 errors)

Progress Tracking (2 errors)

Error Handling (2 errors)

Concurrent Operations (2 errors)

7. Download Endpoints API Test Errors

8. Template Integration Test Failures

9. Frontend Integration Smoke Tests

⚠️ Non-Critical Warnings to Address

Deprecated Pydantic V1 API Usage

Deprecated datetime.utcnow() Usage

Deprecated FastAPI on_event

Deprecated Pydantic .dict() Method

📝 Task Checklist for AI Agent

Phase 1: Critical Async Issues

Phase 2: WebSocket Broadcast Issues

Phase 3: Authentication System

Phase 4: Frontend Integration

Phase 5: WebSocket Integration

Phase 6: Download Flow

Phase 7: Template Integration

Phase 8: Deprecation Warnings

Phase 9: Final Verification

🎯 Success Criteria

📞 Escalation

📚 Helpful Commands

📖 Additional Notes

19 KiB

Raw Blame History