lukas.pupkalipinski/Aniworld
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Aniworld/instructions.md
Lukas 028d91283e feat: implement download queue service with persistence, priority, and retry logic 
- Added comprehensive download queue service (download_service.py)
  - Priority-based queue management (HIGH, NORMAL, LOW)
  - Concurrent download processing with configurable limits
  - Automatic queue persistence to JSON file
  - Retry logic for failed downloads with max retry limits
  - Real-time progress tracking and WebSocket broadcasting
  - Queue operations: add, remove, reorder, pause, resume
  - Statistics tracking: download speeds, sizes, ETA calculations

- Created comprehensive unit tests (test_download_service.py)
  - 23 tests covering all service functionality
  - Tests for queue management, persistence, retry logic
  - Broadcast callbacks, error handling, and lifecycle

- Added structlog dependency for structured logging
- Updated infrastructure.md with download service documentation
- Removed completed task from instructions.md

All tests passing (23/23)
2025-10-17 10:07:16 +02:00

659 lines
20 KiB
Markdown
Raw Blame History

# Aniworld Web Application Development Instructions
This document provides detailed tasks for AI agents to implement a modern web application for the Aniworld anime download manager. All tasks should follow the coding guidelines specified in the project's copilot instructions.
## Project Overview
The goal is to create a FastAPI-based web application that provides a modern interface for the existing Aniworld anime download functionality. The core anime logic should remain in `SeriesApp.py` while the web layer provides REST API endpoints and a responsive UI.
## Architecture Principles
- **Single Responsibility**: Each file/class has one clear purpose
- **Dependency Injection**: Use FastAPI's dependency system
- **Clean Separation**: Web layer calls core logic, never the reverse
- **File Size Limit**: Maximum 500 lines per file
- **Type Hints**: Use comprehensive type annotations
- **Error Handling**: Proper exception handling and logging
## Implementation Order
The tasks should be completed in the following order to ensure proper dependencies and logical progression:
1. **Project Structure Setup** - Foundation and dependencies
2. **Authentication System** - Security layer implementation
3. **Configuration Management** - Settings and config handling
4. **Anime Management Integration** - Core functionality wrapper
5. **Download Queue Management** - Queue handling and persistence
6. **WebSocket Real-time Updates** - Real-time communication
7. **Frontend Integration** - Integrate existing frontend assets
8. **Core Logic Integration** - Enhance existing core functionality
9. **Database Layer** - Data persistence and management
10. **Testing** - Comprehensive test coverage
11. **Deployment and Configuration** - Production setup
12. **Documentation and Error Handling** - Final documentation and error handling
# make the following steps for each task or subtask. make sure you do not miss one
1. Task the next task
2. Process the task
3. Make Tests.
4. Remove task from instructions.md.
5. Update infrastructure.md, but only add text that belongs to a infrastructure doc. make sure to summarize text or delete text that do not belog to infrastructure.md. Keep it clear and short.
6. Commit in git
## Core Tasks
### 5. Download Queue Management
#### [] Implement download API endpoints
- []Create `src/server/api/download.py`
- []Add GET `/api/queue/status` - get queue status
- []Add POST `/api/queue/add` - add to queue
- []Add DELETE `/api/queue/{id}` - remove from queue
- []Add POST `/api/queue/start` - start downloads
- []Add POST `/api/queue/stop` - stop downloads
### 6. WebSocket Real-time Updates
#### [] Implement WebSocket manager
- []Create `src/server/services/websocket_service.py`
- []Add connection management
- []Implement broadcast functionality
- []Include room-based messaging
- []Add connection cleanup
#### [] Add real-time progress updates
- []Create `src/server/services/progress_service.py`
- []Implement download progress broadcasting
- []Add scan progress updates
- []Include queue status changes
- []Add error notifications
#### [] Integrate WebSocket with core services
- []Update download service to emit progress
- []Add scan progress notifications
- []Include queue change broadcasts
- []Add error/completion notifications
### 7. Frontend Integration
#### [] Integrate existing HTML templates
- []Review and integrate existing HTML templates in `src/server/web/templates/`
- []Ensure templates work with FastAPI Jinja2 setup
- []Update template paths and static file references if needed
- []Maintain existing responsive layout and theme switching
#### [] Integrate existing JavaScript functionality
- []Review existing JavaScript files in `src/server/web/static/js/`
- []Update API endpoint URLs to match FastAPI routes
- []Ensure WebSocket connections work with new backend
- []Maintain existing functionality for app.js and queue.js
#### [] Integrate existing CSS styling
- []Review and integrate existing CSS files in `src/server/web/static/css/`
- []Ensure styling works with FastAPI static file serving
- []Maintain existing responsive design and theme support
- []Update any hardcoded paths if necessary
#### [] Update frontend-backend integration
- []Ensure existing JavaScript calls match new API endpoints
- []Update authentication flow to work with new auth system
- []Verify WebSocket events match new service implementations
- []Test all existing UI functionality with new backend
### 8. Core Logic Integration
#### [] Enhance SeriesApp for web integration
- []Update `src/core/SeriesApp.py`
- []Add async callback support
- []Implement progress reporting
- []Include better error handling
- []Add cancellation support
#### [] Create progress callback system
- []Add progress callback interface
- []Implement scan progress reporting
- []Add download progress tracking
- []Include error/completion callbacks
#### [] Add configuration persistence
- []Implement configuration file management
- []Add settings validation
- []Include backup/restore functionality
- []Add migration support for config updates
### 9. Database Layer
#### [] Implement database models
- []Create `src/server/database/models.py`
- []Add SQLAlchemy models for anime series
- []Implement download queue persistence
- []Include user session storage
#### [] Create database service
- []Create `src/server/database/service.py`
- []Add CRUD operations for anime data
- []Implement queue persistence
- []Include database migration support
#### [] Add database initialization
- []Create `src/server/database/init.py`
- []Implement database setup
- []Add initial data migration
- []Include schema validation
### 10. Testing
#### [] Create unit tests for services
- []Create `tests/unit/test_auth_service.py`
- []Create `tests/unit/test_anime_service.py`
- []Create `tests/unit/test_download_service.py`
- []Create `tests/unit/test_config_service.py`
#### [] Create API endpoint tests
- []Create `tests/api/test_auth_endpoints.py`
- []Create `tests/api/test_anime_endpoints.py`
- []Create `tests/api/test_download_endpoints.py`
- []Create `tests/api/test_config_endpoints.py`
#### [] Create integration tests
- []Create `tests/integration/test_download_flow.py`
- []Create `tests/integration/test_auth_flow.py`
- []Create `tests/integration/test_websocket.py`
#### [] Create frontend integration tests
- []Create `tests/frontend/test_existing_ui_integration.py`
- []Test existing JavaScript functionality with new backend
- []Verify WebSocket connections and real-time updates
- []Test authentication flow with existing frontend
### 11. Deployment and Configuration
#### [] Create Docker configuration
- []Create `Dockerfile`
- []Create `docker-compose.yml`
- []Add environment configuration
- []Include volume mappings for existing web assets
#### [] Create production configuration
- []Create `src/server/config/production.py`
- []Add environment variable handling
- []Include security settings
- []Add performance optimizations
#### [] Create startup scripts
- []Create `scripts/start.sh`
- []Create `scripts/setup.py`
- []Add dependency installation
- []Include database initialization
### 12. Documentation and Error Handling
#### [] Create API documentation
- []Add OpenAPI/Swagger documentation
- []Include endpoint descriptions
- []Add request/response examples
- []Include authentication details
#### [] Implement comprehensive error handling
- []Create custom exception classes
- []Add error logging and tracking
- []Implement user-friendly error messages
- []Include error recovery mechanisms
#### [] Create user documentation
- []Create `docs/user_guide.md`
- []Add installation instructions
- []Include configuration guide
- []Add troubleshooting section
## File Size Guidelines
- []**Models**: Max 200 lines each
- []**Services**: Max 450 lines each
- []**API Endpoints**: Max 350 lines each
- []**Templates**: Max 400 lines each
- []**JavaScript**: Max 500 lines each
- []**CSS**: Max 500 lines each
- []**Tests**: Max 400 lines each
## Existing Frontend Assets
The following frontend assets already exist and should be integrated:
- []**Templates**: Located in `src/server/web/templates/`
- []**JavaScript**: Located in `src/server/web/static/js/` (app.js, queue.js, etc.)
- []**CSS**: Located in `src/server/web/static/css/`
- []**Static Assets**: Images and other assets in `src/server/web/static/`
When working with these files:
- []Review existing functionality before making changes
- []Maintain existing UI/UX patterns and design
- []Update API calls to match new FastAPI endpoints
- []Preserve existing WebSocket event handling
- []Keep existing theme and responsive design features
## Quality Assurance
#### [] Code quality checks
- []Run linting with flake8/pylint
- []Check type hints with mypy
- []Validate formatting with black
- []Run security checks with bandit
#### [] Performance testing
- []Load test API endpoints
- []Test WebSocket connection limits
- []Validate download performance
- []Check memory usage patterns
#### [] Security validation
- []Test authentication bypass attempts
- []Validate input sanitization
- []Check for injection vulnerabilities
- []Test session management security
Each task should be implemented with proper error handling, logging, and type hints according to the project's coding standards.
## Additional Implementation Guidelines
### Code Style and Standards
- **Type Hints**: Use comprehensive type annotations throughout all modules
- **Docstrings**: Follow PEP 257 for function and class documentation
- **Error Handling**: Implement custom exception classes with meaningful messages
- **Logging**: Use structured logging with appropriate log levels
- **Security**: Validate all inputs and sanitize outputs
- **Performance**: Use async/await patterns for I/O operations
### Monitoring and Health Checks
#### [] Implement health check endpoints
- []Create `src/server/api/health.py`
- []Add GET `/health` - basic health check
- []Add GET `/health/detailed` - comprehensive system status
- []Include dependency checks (database, file system)
- []Add performance metrics
#### [] Create monitoring service
- []Create `src/server/services/monitoring_service.py`
- []Implement system resource monitoring
- []Add download queue metrics
- []Include error rate tracking
- []Add performance benchmarking
#### [] Add metrics collection
- []Create `src/server/utils/metrics.py`
- []Implement Prometheus metrics export
- []Add custom business metrics
- []Include request timing and counts
- []Add download success/failure rates
### Advanced Features
#### [] Implement backup and restore
- []Create `src/server/services/backup_service.py`
- []Add configuration backup/restore
- []Implement anime data export/import
- []Include download history preservation
- []Add scheduled backup functionality
#### [] Create notification system
- []Create `src/server/services/notification_service.py`
- []Implement email notifications for completed downloads
- []Add webhook support for external integrations
- []Include in-app notification system
- []Add notification preference management
#### [] Add analytics and reporting
- []Create `src/server/services/analytics_service.py`
- []Implement download statistics
- []Add series popularity tracking
- []Include storage usage analysis
- []Add performance reports
### Maintenance and Operations
#### [] Create maintenance endpoints
- []Create `src/server/api/maintenance.py`
- []Add POST `/api/maintenance/cleanup` - cleanup temporary files
- []Add POST `/api/maintenance/rebuild-index` - rebuild search index
- []Add GET `/api/maintenance/stats` - system statistics
- []Add POST `/api/maintenance/vacuum` - database maintenance
#### [] Implement log management
- []Create `src/server/utils/log_manager.py`
- []Add log rotation and archival
- []Implement log level management
- []Include log search and filtering
- []Add log export functionality
#### [] Create system utilities
- []Create `src/server/utils/system.py`
- []Add disk space monitoring
- []Implement file system cleanup
- []Include process management utilities
- []Add system information gathering
### Security Enhancements
#### [] Implement rate limiting
- []Create `src/server/middleware/rate_limit.py`
- []Add endpoint-specific rate limits
- []Implement IP-based limiting
- []Include user-based rate limiting
- []Add bypass mechanisms for authenticated users
#### [] Add security headers
- []Create `src/server/middleware/security.py`
- []Implement CORS headers
- []Add CSP headers
- []Include security headers (HSTS, X-Frame-Options)
- []Add request sanitization
#### [] Create audit logging
- []Create `src/server/services/audit_service.py`
- []Log all authentication attempts
- []Track configuration changes
- []Monitor download activities
- []Include user action tracking
### Data Management
#### [] Implement data validation
- []Create `src/server/utils/validators.py`
- []Add Pydantic custom validators
- []Implement business rule validation
- []Include data integrity checks
- []Add format validation utilities
#### [] Create data migration tools
- []Create `src/server/database/migrations/`
- []Add database schema migration scripts
- []Implement data transformation tools
- []Include rollback mechanisms
- []Add migration validation
#### [] Add caching layer
- []Create `src/server/services/cache_service.py`
- []Implement Redis caching
- []Add in-memory caching for frequent data
- []Include cache invalidation strategies
- []Add cache performance monitoring
### Integration Enhancements
#### [] Extend provider system
- []Enhance `src/core/providers/` for better web integration
- []Add provider health monitoring
- []Implement provider failover mechanisms
- []Include provider performance tracking
- []Add dynamic provider configuration
#### [] Create plugin system
- []Create `src/server/plugins/`
- []Add plugin loading and management
- []Implement plugin API
- []Include plugin configuration
- []Add plugin security validation
#### [] Add external API integrations
- []Create `src/server/integrations/`
- []Add anime database API connections
- []Implement metadata enrichment services
- []Include content recommendation systems
- []Add external notification services
### Advanced Testing
#### [] Performance testing
- []Create `tests/performance/`
- []Add load testing for API endpoints
- []Implement stress testing for download system
- []Include memory leak detection
- []Add concurrency testing
#### [] Security testing
- []Create `tests/security/`
- []Add penetration testing scripts
- []Implement vulnerability scanning
- []Include authentication bypass testing
- []Add input validation testing
#### [] End-to-end testing
- []Create `tests/e2e/`
- []Add full workflow testing
- []Implement UI automation tests
- []Include cross-browser testing
- []Add mobile responsiveness testing
### Deployment Strategies
#### [] Container orchestration
- []Create `kubernetes/` directory
- []Add Kubernetes deployment manifests
- []Implement service discovery
- []Include load balancing configuration
- []Add auto-scaling policies
#### [] CI/CD pipeline
- []Create `.github/workflows/`
- []Add automated testing pipeline
- []Implement deployment automation
- []Include security scanning
- []Add performance benchmarking
#### [] Environment management
- []Create environment-specific configurations
- []Add secrets management
- []Implement feature flags
- []Include environment validation
- []Add rollback mechanisms
## Implementation Best Practices
### Error Handling Patterns
```python
# Custom exception hierarchy
class AniWorldException(Exception):
"""Base exception for AniWorld application"""
pass
class AuthenticationError(AniWorldException):
"""Authentication related errors"""
pass
class DownloadError(AniWorldException):
"""Download related errors"""
pass
# Service-level error handling
async def download_episode(episode_id: str) -> DownloadResult:
try:
result = await downloader.download(episode_id)
return result
except ProviderError as e:
logger.error(f"Provider error downloading {episode_id}: {e}")
raise DownloadError(f"Failed to download episode: {e}")
except Exception as e:
logger.exception(f"Unexpected error downloading {episode_id}")
raise DownloadError("Unexpected download error")
```
### Logging Standards
```python
import logging
import structlog
# Configure structured logging
structlog.configure(
processors=[
structlog.stdlib.add_log_level,
structlog.stdlib.add_logger_name,
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.JSONRenderer()
],
wrapper_class=structlog.stdlib.BoundLogger,
logger_factory=structlog.stdlib.LoggerFactory(),
cache_logger_on_first_use=True,
)
logger = structlog.get_logger(__name__)
# Usage examples
logger.info("Download started", episode_id=episode_id, user_id=user_id)
logger.error("Download failed", episode_id=episode_id, error=str(e))
```
### API Response Patterns
```python
from pydantic import BaseModel
from typing import Optional, List, Any
class APIResponse(BaseModel):
success: bool
message: Optional[str] = None
data: Optional[Any] = None
errors: Optional[List[str]] = None
class PaginatedResponse(APIResponse):
total: int
page: int
per_page: int
pages: int
# Usage in endpoints
@router.get("/anime", response_model=PaginatedResponse)
async def list_anime(page: int = 1, per_page: int = 20):
try:
anime_list, total = await anime_service.list_anime(page, per_page)
return PaginatedResponse(
success=True,
data=anime_list,
total=total,
page=page,
per_page=per_page,
pages=(total + per_page - 1) // per_page
)
except Exception as e:
logger.exception("Failed to list anime")
return APIResponse(
success=False,
message="Failed to retrieve anime list",
errors=[str(e)]
)
```
### Dependency Injection Patterns
```python
from fastapi import Depends
from typing import Annotated
# Service dependencies
def get_anime_service() -> AnimeService:
return AnimeService()
def get_download_service() -> DownloadService:
return DownloadService()
# Dependency annotations
AnimeServiceDep = Annotated[AnimeService, Depends(get_anime_service)]
DownloadServiceDep = Annotated[DownloadService, Depends(get_download_service)]
# Usage in endpoints
@router.post("/download")
async def start_download(
request: DownloadRequest,
download_service: DownloadServiceDep,
anime_service: AnimeServiceDep
):
# Implementation
pass
```
## Final Implementation Notes
1. **Incremental Development**: Implement features incrementally, testing each component thoroughly before moving to the next
2. **Code Review**: Review all generated code for adherence to project standards
3. **Documentation**: Document all public APIs and complex logic
4. **Testing**: Maintain test coverage above 80% for all new code
5. **Performance**: Profile and optimize critical paths, especially download and streaming operations
6. **Security**: Regular security audits and dependency updates
7. **Monitoring**: Implement comprehensive monitoring and alerting
8. **Maintenance**: Plan for regular maintenance and updates
## Task Completion Checklist
For each task completed:
- [ ] Implementation follows coding standards
- [ ] Unit tests written and passing
- [ ] Integration tests passing
- [ ] Documentation updated
- [ ] Error handling implemented
- [ ] Logging added
- [ ] Security considerations addressed
- [ ] Performance validated
- [ ] Code reviewed
- [ ] Task marked as complete in instructions.md
- [ ] Infrastructure.md updated
- [ ] Changes committed to git
This comprehensive guide ensures a robust, maintainable, and scalable anime download management system with modern web capabilities.
Reference in New Issue View Git Blame Copy Permalink