Followup to commit c84f968 (read-boundary dedup) and commit f75d591
(cleanup CLI). The read boundary filters duplicates out of the
in-memory episodeDict and the CLI cleans up historical duplicates
in the DB, but the underlying pathology — duplicate rows being
created in the first place — was still active on every rescan.
Two layered prevention fixes:
1. Schema-level guard: add UNIQUE(series_id, season, episode_number)
to the episodes table. SQLite's CREATE UNIQUE INDEX requires
no existing duplicates, but the cleanup CLI from f75d591 has
already been run (or is a one-shot prerequisite for users on
older DBs). Future duplicate rows are rejected at the DB layer.
2. Write-site guard: SerieScanner.scan_single_series used to
`extend` the in-memory episodeDict on every rescan of a
series already in keyDict — across N rescans, the same missing
list was appended N times, growing the dict with duplicates that
then flowed through _update_series_in_db into the episodes
table. The fix replaces the cache with the latest scan result
instead of extending, and dedupes within a single call as
defense in depth against a buggy upstream loader.
Defensive dedup is layered three deep:
- schema constraint (this commit, primary)
- scan_single_series replace-not-extend (this commit, secondary)
- episodeDict property read-boundary dedup (commit c84f968,
tertiary — covers legacy DBs that predate the constraint)
Tests:
- Updated test_serie_scanner.test_scan_single_series_existing_entry
to assert the new replace-not-merge behavior (the old assertion
encoded the buggy extend behavior).
- New test_serie_scanner_scan_dedup.py covers the regression
directly: two rescans of the same series with the same missing
list must yield a canonical dict, not an accumulated one.
- test_database_models and test_clean_duplicate_episodes_cli now
use a legacy_engine fixture that drops the UNIQUE constraint,
so the duplicate-row scenarios they exercise (the read-boundary
dedup and the cleanup tool, both meant to defend against
pre-migration state) can still be tested under the new schema.
Verified manually: clean_duplicate_episodes --apply on the user's
backup DB still removes all 633 duplicate rows under the new
schema (the CLI doesn't depend on the UNIQUE constraint — it
operates on whatever rows already exist).
The 'episodes' table has no UNIQUE constraint on
(series_id, season, episode_number), so historical scans can leave
duplicate rows behind. Commit c84f968 added a read-boundary dedup
in AnimeSeries.episodeDict so the rest of the stack never sees
duplicates — but the duplicate rows themselves still bloat the DB
and confuse direct SQL queries.
This commit adds a standalone CLI to find and (with --apply)
delete those duplicate rows. The cleanup keeps the lowest 'id'
per tuple (the oldest insert, which is most likely to have
populated title / file_path fields) and is idempotent.
Usage:
python -m src.cli.clean_duplicate_episodes # dry-run report
python -m src.cli.clean_duplicate_episodes --apply # actually delete
python -m src.cli.clean_duplicate_episodes --max-series 5 # limit report
Override the target DB with DATABASE_URL=sqlite:///path/to.db.
Verified against the user's backup DB: 633 duplicate rows across
231 (series, season, episode) tuples removed cleanly, leaving
1221 unique rows. Re-running reports no duplicates. The cleanup
does not affect the read-boundary dedup — both layers are
defensive in depth.
The 'episodes added to download queue never shown' symptom has two
layered causes that masked each other:
1. The 'episodes' table has no UNIQUE constraint on
(series_id, season, episode_number), so historical scans can leave
duplicate rows behind. AnimeSeries.episodeDict iterated the
SQLAlchemy 'episodes' relationship without deduping, so the dict
exposed duplicate entries to list_missing() and the queue UI.
The frontend forwarded the duplicated episode list verbatim to
POST /api/queue/add; the backend's pending-episode dedup then
rejected every duplicate as 'already pending' and the user saw
'Skipped 44 duplicate episodes, Added 0'.
2. selection-manager.downloadSelected counted 'episodes.length'
(the input array) instead of data.added_items.length (the
server-confirmed count). Combined with the backend returning
success on an empty add, the user saw a misleading 'Added 44
episode(s)' toast for an empty queue.
Fix: dedupe at the read boundary. The episodeDict property now
filters duplicate (season, episode_number) pairs from both the
DB-loaded relationship and the legacy _episode_dict_cache path.
Existing duplicate rows in the user's DB are inert — the read
filter makes them invisible to the rest of the stack. The
frontend now trusts the server response, logs a console warning
when an input list shrinks to zero added items, and shows an
accurate toast.
Tests:
- TestEpisodeDictDedup class with 4 regression tests covering:
* duplicate relationship rows deduped
* is_downloaded rows still filtered out
* _episode_dict_cache path also deduped (set by scanners/loaders
that may store duplicates)
* dedup is per-(season, ep_num), preserving legitimate
same-ep-num-across-different-seasons entries
Verified manually against the user's backup DB: 'erased' has 44
duplicate rows in the episodes table; episodeDict now returns
{1: [1..12]} instead of {1: [1,1,2,2,3,3,3,3,...]}, matching the
12-episode canonical list.
A finished download marks the Episode row with is_downloaded=True and
populates file_path (commit 0ba2587). The intent was to preserve
download history, but the row stayed in the DB forever because two
sibling sync methods refused to delete it:
- AnimeService._update_series_in_db: `downloaded_set` guard skipped
deletion if the row was marked downloaded.
- SerieScanner._sync_episodes_to_db: `if ep.is_downloaded: continue`
kept downloaded rows.
The user-visible bug: the missing-list UI is correct (it filters by
is_downloaded and the broadcast rebuilds from the DB), but a finished
download left a stale entry in the DB that the user could see when
querying the database directly. Worse, this stale row accumulated
indefinitely across rescans.
Fix: drop the is_downloaded preservation guards. Once the scanner
confirms the file is on disk and the episode is no longer in the
missing set, the Episode row has no further purpose and is deleted to
keep the DB in sync with the filesystem. The UI derives "missing"
from row presence, so deleting the row is the correct way to make the
episode stop appearing as missing in *all* views (UI and DB queries).
Tests:
- test_update_series_deletes_downloaded_episodes_when_no_longer_missing
(RED): regression for _update_series_in_db.
- test_update_series_keeps_still_missing_episodes: sanity sibling
to ensure the fix does not over-reach and delete still-missing
episodes.
- test_deletes_downloaded_episodes_when_no_longer_missing
(RED): regression for SerieScanner._sync_episodes_to_db.
- Replaced test_preserves_downloaded_episodes (which asserted the
old buggy behavior) with the deletion-asserting variant.
delete_series() removed the row from the database and the folder
from disk, but never evicted the entry from SerieList.keyDict — the
in-memory cache that list_series_with_filters() reads from via
SeriesApp.list.GetList(). As a result /api/anime kept returning the
deleted series on every page reload (the 'Beyblade Burst still listed
after delete' bug).
Changes:
* Add SerieList.remove(key) so the cache has a proper eviction API.
* In delete_series(), call it (or fall back to keyDict.pop) after a
successful DB delete.
* Broadcast a broader series_list_changed event so any client that
missed the specific series_deleted event can re-sync by re-fetching
/api/anime. Front-end: new SERIES_LIST_CHANGED constant, handler
that triggers SeriesManager.reloadSeries().
* Two new regression tests: one asserting the in-memory cache is
pruned, one asserting the broader broadcast fires.
POST /api/anime/{key}/regenerate-nfo and PUT /api/anime/{key}/settings
(both of which can regenerate NFO when apply_to_nfo=True) were calling
_create_or_update_nfo without the required anime_service argument,
producing HTTP 500 with 'missing 1 required positional argument:
anime_service' on the anime-settings page.
Also drops the unused series_data parameter from
_create_or_update_nfo's signature — the value was constructed at every
call site but never read inside the function body. All four callers
in anime.py and nfo.py are updated to match.
The Anime Settings page (src/server/web/static/js/pages/anime-settings.js)
calls GET /api/nfo/{key}/content from its 'View NFO XML' button, but that
endpoint was removed during the NFO refactor (commits 21af502, a8e5487).
The frontend was never updated, so every click on the button 404'd.
Fix:
- Add NfoContentResponse model (key, folder, content, file_size,
last_modified) to src/server/models/nfo.py.
- Add GET /api/nfo/{key}/content handler to src/server/api/nfo.py that
reads <anime_directory>/<folder>/tvshow.nfo and returns it as
{"content": "<xml>", ...} — matching what anime-settings.js
viewNfoContent() already expects (data.content).
- Expose viewNfoContent on AniWorld.AnimeSettingsManager so it is
consistent with the other public methods and directly callable from
tests / other modules.
Tests:
- tests/api/test_nfo_endpoints.py: 4 new tests (auth-required, happy
path returning XML, 404 on unknown series, 404 on missing tvshow.nfo).
Also remove the file-local autouse 'reset_auth' fixture that wiped
the conftest's master-password setup and made any login-based test
fail with a stale-hash 'invalid credentials' error — that fixture
was pre-existing and is a no-op now that conftest.py handles reset.
- tests/frontend/unit/anime_settings.test.js: 3 new tests for
viewNfoContent (URL + auth header, writes <pre>, error toast on
404) and an assertion in the public-API surface test.
The success path of handleConfirm() never reset the isSubmitting flag,
so hide()'s early-return guard (line 209: 'if (isSubmitting) return')
kept the modal visible after a successful delete — leaving the user
looking at a stuck 'Deleting...' dialog while the card had already been
removed by the WebSocket SERIES_DELETED event.
Reset isSubmitting and the confirm button text before calling hide(),
and capture currentKey into a local before hide() nulls it so the
follow-up removeSeries() call receives the right key.
When the year-suffixed target folder already exists on disk, both
FolderNamingService and AnimeService.rename_folder_if_needed used to
silently bail out. The bare folder (e.g. 'Ultraman') was left next to
the year-suffixed one ('Ultraman (2019)'), producing the symptom
'reports series like Ultraman as added twice' — the DB has one row but
the filesystem has two folders holding the same content.
Fix: when the target already exists, merge the source's contents into
the target (target version wins on file conflicts; source copies are
removed so cleanup succeeds), remove the now-empty source directory,
update DB row + in-memory cache. Plain rename path is unchanged.
Also fixes a latent TypeError in rename_folder_if_needed where
self._directory (a str) was used with the '/' operator. Production
behavior was that any rename through that method raised and was
swallowed by the caller's try/except, leaving the bare folder
untouched. The new path builds Path objects from the string base.
Tests:
- Replaced test_skips_when_target_folder_already_exists (which
codified the bug) with three tests that cover the new merge
contract: clean merge, no-overwrite, and empty-source removal.
- Added tests/unit/test_rename_folder_if_needed.py covering the
same scenarios plus the str-directory regression. All seven go
red on the unfixed code and green with the fix.
Fixes the 'Ultraman' / 'Ultraman (2019)' duplicate-folder bug.
Two follow-ups to the previous fix that resolved the CWD-relative-path
bug in delete_series:
1. Reorder deletion: filesystem first, database second.
When both delete_database=True and delete_folder=True were set and
the folder delete failed (any reason — path math bug, permission
error, missing volume, etc.), the database row was already gone by
the time the folder delete was attempted. This left an orphan
folder on disk that the user had no normal way to clean up — the
delete UI returns 'Series not found' when the row is gone.
Doing folder delete first means a folder-side failure preserves
the DB row, so the user can retry the delete once the underlying
issue is resolved.
2. Orphan-folder recovery: when the user requests delete_folder=True
for a key whose DB row no longer exists, scan the configured
anime directory for a folder that uniquely matches the key
(normalized: strip trailing (YYYY), lowercase, remove
hyphens/underscores/non-alphanumerics) and delete it. This
recovers the Beyblade Burst scenario: a previous delete attempt
removed the row but the folder stayed on disk; retrying the
delete now cleans up the orphan.
Safety: recovery only triggers when exactly one folder matches
the normalized key. Zero matches → clear 'no folder matching'
error. Multiple matches (ambiguous) → refuses to delete anything.
Adds four regression tests:
- test_delete_series_db_preserved_when_folder_fails
- test_delete_series_orphan_folder_recovery (the Beyblade Burst case)
- test_delete_series_orphan_folder_no_match
- test_delete_series_orphan_folder_ambiguous
The folder column in anime_series stores the relative folder name
(e.g. 'Beyblade Burst (2016)'), not an absolute path. The old
delete_series code called os.path.abspath(folder_path) directly,
which joins a relative path against the process current working
directory. Inside the container the FastAPI app runs with CWD=/app
while the anime directory is /data, so 'Beyblade Burst (2016)'
resolved to '/app/Beyblade Burst (2016)' and the is_safe_path check
correctly (but unhelpfully) flagged it as outside the /data base,
skipping the folder delete while still removing the row.
Fix: resolve relative folder paths against the configured anime
directory before validating against the base. Absolute paths still
work unchanged.
Also harden is_safe_path() the same way so a relative target is
treated as relative to base_path, not to the process CWD. Path
traversal ('../etc/passwd') is still rejected.
Adds two regression tests:
- is_safe_path with chdir to '/' and relative target resolves
against the base
- delete_series with chdir to '/', relative folder in DB,
succeeds and removes the folder
Add null guards and element re-caching in delete-modal.js so the modal
recovers gracefully if its DOM is replaced (e.g. by an HTMX swap) between
init and show().
Also fix the two tests that broke in this environment:
- test_delete_modal.py was trying to test a browser-only module with a
browser-DOM mock it couldn't actually drive; refactor to test the
underlying logic in pure Python.
- test_delete_anime_security.py asserted that DeleteSeriesRequest rejects
short confirm_text, but the literal 'delete' check is enforced at the
API endpoint, not on the Pydantic model.
Prevent fully-downloaded series from being queued by auto-download:
- SerieList.GetMissingEpisode(): filter by is_downloaded instead of
checking if episodeDict is non-empty. episodeDict from the DB
relationship includes all episodes (including downloaded), so a
series with only downloaded episodes still had a truthy episodeDict.
- AnimeSeries.episodeDict property: skip episodes where is_downloaded=True
when building the dict from the DB relationship. This makes the
property consistent with the is_downloaded filtering already done
manually in list_series_with_filters(), and ensures that calling
code anywhere in the codebase gets the correct missing-episode view.
Fixes hana-kimi (and any other fully-downloaded series) incorrectly
appearing in the auto-download queue after a rescan.
SQLAlchemy async sessions are not safe for concurrent operations. load_series_images_batch was using asyncio.gather to run multiple load_series_images calls concurrently, all sharing the same db session. This caused 'session is provisioning a new connection; concurrent operations are not permitted' errors for every series in the batch.
Fix by processing each batch sequentially instead of concurrently. BATCH_SIZE=10 still paces TMDB requests as intended.
When _try_direct_stream() succeeded, it streamed chunks directly via
requests.get() without firing any progress events. This caused the WebSocket
frontend to only see 'started' (0%) and 'completed' (100%) — no incremental
updates.
Add per-chunk progress events to _try_direct_stream that mirror yt-dlp's
hook format (downloaded_bytes, total_bytes, speed, eta, status), emitted at
~1% intervals. These flow through the existing download_progress handler chain
(SeriesApp → anime_service → progress_service → WebSocket → frontend).
The websocket-client strips the outer {type, data, ...} wrapper before
emitting to handlers, so handlers receive the inner series data object
directly (e.g. {key, name, missing_episodes}) — not {type, data: {...}}.
The series_updated handler was checking data.data which always failed,
causing every update to fall back to a full loadSeries() call instead of
calling updateSingleSeries() directly. This prevented the missing
episodes count from updating in real-time after a download completed.
Fix: check data.key directly and pass data (not data.data) to
updateSingleSeries().
Runs after NFO refresh during scheduled rescans. Renames folders that
are missing a year (e.g. 'Naruto' → 'Naruto (1999)') using the year
from the database record.
Safety: _build_target_folder() always strips any existing year suffix
first, preventing double/triple year accumulation like
'Naruto (1999) (1999) (1999)'.
Changes:
- New FolderNamingService (folder_naming_service.py) with safe target
name construction, DB update, and in-memory cache update
- New SchedulerConfig field: folder_naming_after_nfo_scan (default True)
- Integrated as step 3 in scheduler _perform_rescan() after NFO scan
- Runtime UI: existing 'folder-scan-enabled' checkbox in index.html
now wired to toggle the feature (app.js + scheduler-config.js)
- Setup screen: new checkbox in setup.html Scheduler Settings section
- API: scheduler config endpoint returns all scan toggles
- Tests: 39 unit tests covering static helpers, rename logic, safety
guard, and integration cases (folder_naming_service.py)
- Docs: testing guide updated with FolderNamingService examples
- Add 'system_progress' event type to loading page redirect condition
- Add checkUnresolvedAndRedirect() for phase=initial to handle race condition
where backend initialization completes before WebSocket connects
- Backend now emits series_sync progress events during initial setup
- Loading page checks /api/setup/unresolved immediately on load for phase=initial
- Fixes users getting stuck on loading page after setup
Reset was not putting app in unconfigured state. _needs_setup()
was checking is_configured() but not verifying master_password_hash
exists in saved config file. Added explicit check for this key so
reset properly triggers setup form.
Also removed completed Task 26 from docs.
- Add /api/config/admin/reset unauthenticated endpoint for tests
- Add auth_service.reset() to clear in-memory auth state
- Update robot tests to call reset on teardown
- Remove flaky Setup Redirects test (depended on test order)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add data-testid='toast' to toast element in ui-utils.js
- Update toast selector in ui_keywords.resource to use [data-testid='toast']
- Update login rate limit test to check for 'invalid' instead of 'lockout'
(testing mode disables lockout, so test verifies proper error message)
- Remove completed task docs from tasks.md
Robot Framework's Create Dictionary converts nested dicts like {'enabled': False} to string representation. Parse these strings before Pydantic validation. Also handle Pydantic models passed directly from unit tests.
Robot Framework's Create Dictionary converts ['mon', 'tue'] into a
string. Add _parse_schedule_days to handle JSON/Python-literal parsing
before Pydantic type validation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Use anime_service.rescan() instead of direct trigger_rescan import
- Change Optional[Any] to Optional[Dict[str, Any]] in ConfigUpdate for better type hints
- Add ValidationError handling in ConfigUpdate.apply_to
- Update tests to match new implementation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace TMDBProvider with tmdb_client in nfo.py
- Add anime_service param to repair functions
- Add null guard for empty series name lookup
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Task 24 fix. Robot tests now use Get Element States to check button
enabled state, which works reliably. Also added 100ms delay after auth
to ensure token availability, and error handling in loadQueueData.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- AniWorld.UiUtils → AniWorld.UI for toast notifications
- Update robot tests to use actual element IDs (settings-section, save-db-btn, field-folder)
- Fix anime key format in test URLs (Attack on Titan - Season 1)
- Remove completed Task 22 from Docs/tasks.md
- Remove completed Task 17 and 18 from Docs/tasks.md
- Remove ANIWORLD_TESTING bypass in _check_locked (testing env now uses test auth service)
- Remove redundant lock expiry cleanup (handled by _get_fail_record)
- Fix login.html whitespace formatting
- Increase setup retry wait from 6s to 45s for rate-limited CI environments
