Followup to commit c84f968 (read-boundary dedup) and commit f75d591
(cleanup CLI). The read boundary filters duplicates out of the
in-memory episodeDict and the CLI cleans up historical duplicates
in the DB, but the underlying pathology — duplicate rows being
created in the first place — was still active on every rescan.
Two layered prevention fixes:
1. Schema-level guard: add UNIQUE(series_id, season, episode_number)
to the episodes table. SQLite's CREATE UNIQUE INDEX requires
no existing duplicates, but the cleanup CLI from f75d591 has
already been run (or is a one-shot prerequisite for users on
older DBs). Future duplicate rows are rejected at the DB layer.
2. Write-site guard: SerieScanner.scan_single_series used to
`extend` the in-memory episodeDict on every rescan of a
series already in keyDict — across N rescans, the same missing
list was appended N times, growing the dict with duplicates that
then flowed through _update_series_in_db into the episodes
table. The fix replaces the cache with the latest scan result
instead of extending, and dedupes within a single call as
defense in depth against a buggy upstream loader.
Defensive dedup is layered three deep:
- schema constraint (this commit, primary)
- scan_single_series replace-not-extend (this commit, secondary)
- episodeDict property read-boundary dedup (commit c84f968,
tertiary — covers legacy DBs that predate the constraint)
Tests:
- Updated test_serie_scanner.test_scan_single_series_existing_entry
to assert the new replace-not-merge behavior (the old assertion
encoded the buggy extend behavior).
- New test_serie_scanner_scan_dedup.py covers the regression
directly: two rescans of the same series with the same missing
list must yield a canonical dict, not an accumulated one.
- test_database_models and test_clean_duplicate_episodes_cli now
use a legacy_engine fixture that drops the UNIQUE constraint,
so the duplicate-row scenarios they exercise (the read-boundary
dedup and the cleanup tool, both meant to defend against
pre-migration state) can still be tested under the new schema.
Verified manually: clean_duplicate_episodes --apply on the user's
backup DB still removes all 633 duplicate rows under the new
schema (the CLI doesn't depend on the UNIQUE constraint — it
operates on whatever rows already exist).
The 'episodes' table has no UNIQUE constraint on
(series_id, season, episode_number), so historical scans can leave
duplicate rows behind. Commit c84f968 added a read-boundary dedup
in AnimeSeries.episodeDict so the rest of the stack never sees
duplicates — but the duplicate rows themselves still bloat the DB
and confuse direct SQL queries.
This commit adds a standalone CLI to find and (with --apply)
delete those duplicate rows. The cleanup keeps the lowest 'id'
per tuple (the oldest insert, which is most likely to have
populated title / file_path fields) and is idempotent.
Usage:
python -m src.cli.clean_duplicate_episodes # dry-run report
python -m src.cli.clean_duplicate_episodes --apply # actually delete
python -m src.cli.clean_duplicate_episodes --max-series 5 # limit report
Override the target DB with DATABASE_URL=sqlite:///path/to.db.
Verified against the user's backup DB: 633 duplicate rows across
231 (series, season, episode) tuples removed cleanly, leaving
1221 unique rows. Re-running reports no duplicates. The cleanup
does not affect the read-boundary dedup — both layers are
defensive in depth.
The 'episodes added to download queue never shown' symptom has two
layered causes that masked each other:
1. The 'episodes' table has no UNIQUE constraint on
(series_id, season, episode_number), so historical scans can leave
duplicate rows behind. AnimeSeries.episodeDict iterated the
SQLAlchemy 'episodes' relationship without deduping, so the dict
exposed duplicate entries to list_missing() and the queue UI.
The frontend forwarded the duplicated episode list verbatim to
POST /api/queue/add; the backend's pending-episode dedup then
rejected every duplicate as 'already pending' and the user saw
'Skipped 44 duplicate episodes, Added 0'.
2. selection-manager.downloadSelected counted 'episodes.length'
(the input array) instead of data.added_items.length (the
server-confirmed count). Combined with the backend returning
success on an empty add, the user saw a misleading 'Added 44
episode(s)' toast for an empty queue.
Fix: dedupe at the read boundary. The episodeDict property now
filters duplicate (season, episode_number) pairs from both the
DB-loaded relationship and the legacy _episode_dict_cache path.
Existing duplicate rows in the user's DB are inert — the read
filter makes them invisible to the rest of the stack. The
frontend now trusts the server response, logs a console warning
when an input list shrinks to zero added items, and shows an
accurate toast.
Tests:
- TestEpisodeDictDedup class with 4 regression tests covering:
* duplicate relationship rows deduped
* is_downloaded rows still filtered out
* _episode_dict_cache path also deduped (set by scanners/loaders
that may store duplicates)
* dedup is per-(season, ep_num), preserving legitimate
same-ep-num-across-different-seasons entries
Verified manually against the user's backup DB: 'erased' has 44
duplicate rows in the episodes table; episodeDict now returns
{1: [1..12]} instead of {1: [1,1,2,2,3,3,3,3,...]}, matching the
12-episode canonical list.
A finished download marks the Episode row with is_downloaded=True and
populates file_path (commit 0ba2587). The intent was to preserve
download history, but the row stayed in the DB forever because two
sibling sync methods refused to delete it:
- AnimeService._update_series_in_db: `downloaded_set` guard skipped
deletion if the row was marked downloaded.
- SerieScanner._sync_episodes_to_db: `if ep.is_downloaded: continue`
kept downloaded rows.
The user-visible bug: the missing-list UI is correct (it filters by
is_downloaded and the broadcast rebuilds from the DB), but a finished
download left a stale entry in the DB that the user could see when
querying the database directly. Worse, this stale row accumulated
indefinitely across rescans.
Fix: drop the is_downloaded preservation guards. Once the scanner
confirms the file is on disk and the episode is no longer in the
missing set, the Episode row has no further purpose and is deleted to
keep the DB in sync with the filesystem. The UI derives "missing"
from row presence, so deleting the row is the correct way to make the
episode stop appearing as missing in *all* views (UI and DB queries).
Tests:
- test_update_series_deletes_downloaded_episodes_when_no_longer_missing
(RED): regression for _update_series_in_db.
- test_update_series_keeps_still_missing_episodes: sanity sibling
to ensure the fix does not over-reach and delete still-missing
episodes.
- test_deletes_downloaded_episodes_when_no_longer_missing
(RED): regression for SerieScanner._sync_episodes_to_db.
- Replaced test_preserves_downloaded_episodes (which asserted the
old buggy behavior) with the deletion-asserting variant.
delete_series() removed the row from the database and the folder
from disk, but never evicted the entry from SerieList.keyDict — the
in-memory cache that list_series_with_filters() reads from via
SeriesApp.list.GetList(). As a result /api/anime kept returning the
deleted series on every page reload (the 'Beyblade Burst still listed
after delete' bug).
Changes:
* Add SerieList.remove(key) so the cache has a proper eviction API.
* In delete_series(), call it (or fall back to keyDict.pop) after a
successful DB delete.
* Broadcast a broader series_list_changed event so any client that
missed the specific series_deleted event can re-sync by re-fetching
/api/anime. Front-end: new SERIES_LIST_CHANGED constant, handler
that triggers SeriesManager.reloadSeries().
* Two new regression tests: one asserting the in-memory cache is
pruned, one asserting the broader broadcast fires.
When the year-suffixed target folder already exists on disk, both
FolderNamingService and AnimeService.rename_folder_if_needed used to
silently bail out. The bare folder (e.g. 'Ultraman') was left next to
the year-suffixed one ('Ultraman (2019)'), producing the symptom
'reports series like Ultraman as added twice' — the DB has one row but
the filesystem has two folders holding the same content.
Fix: when the target already exists, merge the source's contents into
the target (target version wins on file conflicts; source copies are
removed so cleanup succeeds), remove the now-empty source directory,
update DB row + in-memory cache. Plain rename path is unchanged.
Also fixes a latent TypeError in rename_folder_if_needed where
self._directory (a str) was used with the '/' operator. Production
behavior was that any rename through that method raised and was
swallowed by the caller's try/except, leaving the bare folder
untouched. The new path builds Path objects from the string base.
Tests:
- Replaced test_skips_when_target_folder_already_exists (which
codified the bug) with three tests that cover the new merge
contract: clean merge, no-overwrite, and empty-source removal.
- Added tests/unit/test_rename_folder_if_needed.py covering the
same scenarios plus the str-directory regression. All seven go
red on the unfixed code and green with the fix.
Fixes the 'Ultraman' / 'Ultraman (2019)' duplicate-folder bug.
Two follow-ups to the previous fix that resolved the CWD-relative-path
bug in delete_series:
1. Reorder deletion: filesystem first, database second.
When both delete_database=True and delete_folder=True were set and
the folder delete failed (any reason — path math bug, permission
error, missing volume, etc.), the database row was already gone by
the time the folder delete was attempted. This left an orphan
folder on disk that the user had no normal way to clean up — the
delete UI returns 'Series not found' when the row is gone.
Doing folder delete first means a folder-side failure preserves
the DB row, so the user can retry the delete once the underlying
issue is resolved.
2. Orphan-folder recovery: when the user requests delete_folder=True
for a key whose DB row no longer exists, scan the configured
anime directory for a folder that uniquely matches the key
(normalized: strip trailing (YYYY), lowercase, remove
hyphens/underscores/non-alphanumerics) and delete it. This
recovers the Beyblade Burst scenario: a previous delete attempt
removed the row but the folder stayed on disk; retrying the
delete now cleans up the orphan.
Safety: recovery only triggers when exactly one folder matches
the normalized key. Zero matches → clear 'no folder matching'
error. Multiple matches (ambiguous) → refuses to delete anything.
Adds four regression tests:
- test_delete_series_db_preserved_when_folder_fails
- test_delete_series_orphan_folder_recovery (the Beyblade Burst case)
- test_delete_series_orphan_folder_no_match
- test_delete_series_orphan_folder_ambiguous
The folder column in anime_series stores the relative folder name
(e.g. 'Beyblade Burst (2016)'), not an absolute path. The old
delete_series code called os.path.abspath(folder_path) directly,
which joins a relative path against the process current working
directory. Inside the container the FastAPI app runs with CWD=/app
while the anime directory is /data, so 'Beyblade Burst (2016)'
resolved to '/app/Beyblade Burst (2016)' and the is_safe_path check
correctly (but unhelpfully) flagged it as outside the /data base,
skipping the folder delete while still removing the row.
Fix: resolve relative folder paths against the configured anime
directory before validating against the base. Absolute paths still
work unchanged.
Also harden is_safe_path() the same way so a relative target is
treated as relative to base_path, not to the process CWD. Path
traversal ('../etc/passwd') is still rejected.
Adds two regression tests:
- is_safe_path with chdir to '/' and relative target resolves
against the base
- delete_series with chdir to '/', relative folder in DB,
succeeds and removes the folder
Runs after NFO refresh during scheduled rescans. Renames folders that
are missing a year (e.g. 'Naruto' → 'Naruto (1999)') using the year
from the database record.
Safety: _build_target_folder() always strips any existing year suffix
first, preventing double/triple year accumulation like
'Naruto (1999) (1999) (1999)'.
Changes:
- New FolderNamingService (folder_naming_service.py) with safe target
name construction, DB update, and in-memory cache update
- New SchedulerConfig field: folder_naming_after_nfo_scan (default True)
- Integrated as step 3 in scheduler _perform_rescan() after NFO scan
- Runtime UI: existing 'folder-scan-enabled' checkbox in index.html
now wired to toggle the feature (app.js + scheduler-config.js)
- Setup screen: new checkbox in setup.html Scheduler Settings section
- API: scheduler config endpoint returns all scan toggles
- Tests: 39 unit tests covering static helpers, rename logic, safety
guard, and integration cases (folder_naming_service.py)
- Docs: testing guide updated with FolderNamingService examples
- Add 'system_progress' event type to loading page redirect condition
- Add checkUnresolvedAndRedirect() for phase=initial to handle race condition
where backend initialization completes before WebSocket connects
- Backend now emits series_sync progress events during initial setup
- Loading page checks /api/setup/unresolved immediately on load for phase=initial
- Fixes users getting stuck on loading page after setup
- Create ImageLoadingService that downloads poster.jpg, fanart.jpg, and
logo.png from TMDB when anime is added or during scheduler rescan
- Integrate into BackgroundLoaderService._load_nfo_and_images() to trigger
image downloads when new anime is added
- Add image_scan_after_rescan config option to scheduler (default: true)
- Add _run_image_scan() to scheduler rescan flow, processing series in
batches of 10 to respect TMDB rate limits
- Fix SearchResult model missing folder, snippet, and score fields
- Update background_loader tests to match new image loading behavior
- Call _load_series_into_memory() before NFO scan phases to sync DB
to SeriesApp memory, fixing missing NFO for recently resolved folders
- Add TMDB lookup for series without cached tmdb_id during NFO creation
- Add get_tmdb_client() factory and get_tmdb_image_base_url() helpers
- Fix: use get_tv_show_details instead of deprecated get_series_details
- Fix tests: mock _load_series_into_memory in NFO scan tests
- Removed TestSyncSeriesFromDataFiles class from test_anime_service.py
- Updated TestSyncAnimeFolders tests to expect sync_count=0
- Removed TestSyncSeriesToDatabase class from test_data_file_db_sync.py
Prevent scheduler from triggering immediate rescan when:
- No previous scan recorded AND initial setup not yet completed
- Anime directory doesn't exist during initial sync
The setup flow will trigger rescan when ready. Also adds graceful
handling when anime directory is missing during data file sync.
Fixes: 503 error on setup when scheduler triggers rescan before
anime directory is configured
_execute_nfo_scan() was importing get_anime_service from anime_service.py
which is a factory requiring series_app argument. Changed to import from
dependencies.py which handles series_app internally and provides proper
dependency injection with caching.
- Add check for existing series by key in SetupService.run to skip duplicates
- Fix Path construction in initialization_service.py cleanup function
- Update unit tests to mock get_by_key and get_series_app
When SetupService cannot auto-resolve a provider key for an anime folder,
the folder is now tracked in the new 'unresolved_folders' table instead of
being silently skipped. Users can then resolve these via the new API:
- GET /api/setup/unresolved - list unresolved folders with search suggestions
- POST /api/setup/unresolved/{folder}/resolve - provide key to resolve folder
The SetupService.run() now:
- Tracks unresolved folders instead of skipping them
- Re-creates AnimeSeries for previously unresolved folders that are now resolved
- Includes unresolved count in logs
New files:
- src/server/api/setup_endpoints.py - API endpoints for unresolved management
- tests/unit/test_unresolved_folder_service.py - service and model tests
Modified:
- src/server/database/models.py - add UnresolvedFolder model
- src/server/database/service.py - add UnresolvedFolderService
- src/server/services/setup_service.py - track unresolved folders
- src/server/fastapi_app.py - include setup router
- Fix _resolve_key_via_search to use 'title' instead of 'name'
- Extract key from 'link' field URL (e.g., /anime/stream/naruto -> naruto)
- Skip folders with unresolved keys instead of crashing with 'Series key cannot be empty'
- Update tests to use correct field names (title/link)
SetupService.run() now checks each anime folder for tvshow.nfo,
logo.png, and poster/fanart images instead of using hardcoded
defaults. Provider key resolution via search is unchanged.
Extract SetupService class from initialization_service to handle:
- Scan data/ folder subdirectories
- Extract title and year from folder names (YYYY pattern)
- Create AnimeSeries records in database
- Resolve provider keys via search (single exact match)
Updates _scan_folders_to_database() to delegate to SetupService.run().
Adds comprehensive unit tests for SetupService.
- Add nfo_scan_after_rescan config option (default: true)
- Implement year caching in AniworldLoader and EnhancedAniWorldLoader
- Make get_year abstract method in base provider
- Run NFO validation/creation after scheduled rescan completes
- Add _YearDict cache to avoid re-extracting year from HTML
Delete folder_rename_service.py. Stub out get_duplicate_folders API to return
empty response. Update folder_scan_service and tests to skip rename step.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Convert handle_network_failure and handle_download_failure from instance methods to static methods. Hardcode retry params (max_retries, delays) instead of using instance state. Improves testability and removes implicit dependencies.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously, the method created a SerieList instance which only loads
from database, not from data files. Now reads data files directly
and parses JSON to create AnimeSeries objects.
Also added _load_data_file helper function and fixed logger.warning
calls to use proper format strings instead of keyword arguments.
Updated unit tests to use real temp directories instead of mocks.
- Add _scan_folders_to_database() - iterates anime_directory subdirs
- Extract title/year from folder names via (YYYY) pattern
- Resolve provider key via search when single match found
- Create AnimeSeries records for new folders only
- Add corresponding unit tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move nfo_models, tmdb_client, nfo_generator, nfo_mapper from
scattered temp directories into single src/server/nfo/ package.
Update all imports to reflect new structure.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Move src/core/ → src/server/
- Split SerieList.py (531 lines) and series.py (414 lines) into src/server/database/
- Add database/models.py for SQLAlchemy models
- Update all test imports to reflect new structure
- Remove deprecated test files (test_serie_class.py, test_serie_folder_with_year.py)
- Extract rescan logic into new RescanService (src/server/services/rescan_service.py)
- SchedulerService now only handles APScheduler cron scheduling
- Move scheduler sub-services (folder_rename, folder_scan, key_resolution) to scheduler/ folder
- Keep RescanOrchestrator as backward-compatible alias
- Update all imports across api/, server/, and test files
folder_rename_service depends on clean NFO files but repair tasks
were fire-and-forget. Now collect all repair tasks and await them
with asyncio.gather before validate_and_rename_series_folders runs.
Also update tests that mock asyncio.create_task to also mock
asyncio.gather since perform_nfo_repair_scan now awaits tasks.
- Add key_resolution_service.py to resolve provider keys for folders without key/data files
- Search anime provider and match folder names (case-insensitive, exact match required)
- Only save to DB if exactly one match found; otherwise skip
- Add comprehensive unit tests (28 tests)
- Integrate into scheduler_service after nfo_repair scan
- Update ARCHITECTURE.md documentation
Add configurable folder rename patterns via settings with anime_folder_rename_regex and custom_pattern options. Integrate into SerieScanner and SeriesApp for consistent episode organization.
- SerieScanner: Remove key file fallback, keep data file fallback
- SystemSettings: Add legacy_key_cleanup_completed flag
- initialization_service: Add cleanup task to remove key files from folders with DB entries
- Tests updated to reflect key file removal from legacy path
Key files caused duplicate key errors on folder rename. DB is now sole source of truth.
- Add _cleanup_orphaned_folder() to delete/move old folder contents after rename
- Empty folders: delete directly via rmdir()
- Non-empty folders: move contents to new path, then delete old folder
- Handle PermissionError and OSError gracefully with logging
- Add dry_run parameter to preview changes without applying them
- Add --dry-run support to validate_and_rename_series_folders()
- Add unit tests for _cleanup_orphaned_folder and dry-run mode
- All 66 related tests pass
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SchedulerConfig.__init__ maps legacy auto_download/folder_scan keys to the
primary auto_download_after_rescan/folder_scan_enabled fields. However,
model_dump() was including auto_download=null and folder_scan=null in
serialised output. When this was written to config.json and reloaded,
those keys were present (albeit null), so the alias mapping was skipped
and the primary fields retained default False values instead of the
configured True values.
Fix:
- Override SchedulerConfig.model_dump() to drop None-valued alias fields
before returning the serialised dict.
- ConfigService.save_config() re-serialises the scheduler field through
its overridden model_dump() so the fix applies when writing to disk.
Tests added:
- test_roundtrip_excludes_none_alias_fields: verifies model_dump omits
null auto_download/folder_scan keys.
- test_save_and_load_scheduler_flags_roundtrip: end-to-end roundtrip
through ConfigService confirms raw JSON and loaded values match.
Pre-existing failure in test_core_error_handler.py is unrelated.
- SerieScanner: generate key from folder when no key/data files exist
- Handle edge cases: non-Latin characters, special symbols in folder names
- anime_service: expose loading_status and loading_error fields
- Update tests to match new fallback behavior
- Add NFO_FOLDER_IGNORE_PATTERNS setting to skip TV shows like
The Last of Us, Loki, Chernobyl, Star Trek Discovery
- Update SerieScanner.__find_mp4_files() to skip ignored folders
- Update SerieList.load_series() to skip ignored folders
- Add should_ignore_folder() method for pattern matching
- Add folder_ignore_patterns property for pattern parsing
- Add comprehensive tests for ignore pattern functionality
- Update NFO_GUIDE.md with ignore patterns documentation
- Update CONFIGURATION.md with new setting
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scheduler used a separate SQLite file (scheduler.db) only to persist one
cron job. This was originally required because APScheduler's
SQLAlchemyJobStore is sync-only, creating an async/sync driver conflict
when accessing the same file.
The job is rebuilt from config.json on every startup regardless
(replace_existing=True), so the persisted state only served misfire
detection. Moved misfire detection into the app layer by querying
system_settings.last_scan_timestamp on startup: if the last scan is
>23h but <25h ago, an immediate rescan is triggered.
Change summary:
- Remove SQLAlchemyJobStore; use default MemoryJobStore instead
- Add _check_missed_run() that reads last_scan_timestamp from aniworld.db
- Update docs/DEVELOPMENT.md scheduler troubleshooting section
- Update the scheduler unit test that verified SQLAlchemyJobStore
Check existing tvshow.nfo for TMDB ID before querying TMDB API.
If found, fetch details directly using cached ID instead of searching.
Reduces API calls and improves performance for already-indexed series.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- SerieScanner.scan() now detects running event loop and uses create_task()
when already in async context, avoids RuntimeError
- NFOService.close() now also closes image_downloader to prevent resource leaks
- Add integration tests for TMDBClient lifecycle management
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
