lukas.pupkalipinski/Aniworld
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(episodes): prevent duplicate-row accumulation at schema + write sites

Browse Source 
Followup to commit c84f968 (read-boundary dedup) and commit f75d591
(cleanup CLI). The read boundary filters duplicates out of the
in-memory episodeDict and the CLI cleans up historical duplicates
in the DB, but the underlying pathology — duplicate rows being
created in the first place — was still active on every rescan.

Two layered prevention fixes:

1. Schema-level guard: add UNIQUE(series_id, season, episode_number)
   to the episodes table. SQLite's CREATE UNIQUE INDEX requires
   no existing duplicates, but the cleanup CLI from f75d591 has
   already been run (or is a one-shot prerequisite for users on
   older DBs). Future duplicate rows are rejected at the DB layer.

2. Write-site guard: SerieScanner.scan_single_series used to
   `extend` the in-memory episodeDict on every rescan of a
   series already in keyDict — across N rescans, the same missing
   list was appended N times, growing the dict with duplicates that
   then flowed through _update_series_in_db into the episodes
   table. The fix replaces the cache with the latest scan result
   instead of extending, and dedupes within a single call as
   defense in depth against a buggy upstream loader.

Defensive dedup is layered three deep:
  - schema constraint (this commit, primary)
  - scan_single_series replace-not-extend (this commit, secondary)
  - episodeDict property read-boundary dedup (commit c84f968,
    tertiary — covers legacy DBs that predate the constraint)

Tests:
  - Updated test_serie_scanner.test_scan_single_series_existing_entry
    to assert the new replace-not-merge behavior (the old assertion
    encoded the buggy extend behavior).
  - New test_serie_scanner_scan_dedup.py covers the regression
    directly: two rescans of the same series with the same missing
    list must yield a canonical dict, not an accumulated one.
  - test_database_models and test_clean_duplicate_episodes_cli now
    use a legacy_engine fixture that drops the UNIQUE constraint,
    so the duplicate-row scenarios they exercise (the read-boundary
    dedup and the cleanup tool, both meant to defend against
    pre-migration state) can still be tested under the new schema.

Verified manually: clean_duplicate_episodes --apply on the user's
backup DB still removes all 633 duplicate rows under the new
schema (the CLI doesn't depend on the UNIQUE constraint — it
operates on whatever rows already exist).
This commit is contained in:
Lukas
2026-09-15 20:58:30 +02:00
parent f75d591453
commit c3aca9217d
6 changed files with 417 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files

87
tests/unit/test_clean_duplicate_episodes_cli.py
View File

@@ -10,7 +10,7 @@ from __future__ import annotations
from typing import List
import pytest
from sqlalchemy import select
from sqlalchemy import select, text
from sqlalchemy.ext.asyncio import (
AsyncEngine,
AsyncSession,
@@ -29,14 +29,55 @@ from src.server.database.models import AnimeSeries, Episode
async def in_memory_engine():
"""Provide an in-memory async SQLite engine with the schema
already created, and patch the connection module's session
factory to use it for the duration of the test."""
factory to use it for the duration of the test.
The CLI test suite simulates the *pre-migration* state — a DB
that predates the UNIQUE constraint on
``(series_id, season, episode_number)`` and has accumulated
duplicate rows from historical scans. To create that state, we
build the schema for everything except ``episodes``, then
recreate ``episodes`` with raw DDL that omits the
``uq_episode_per_series_season`` constraint. SQLite ties
UNIQUE constraints to an internal auto-named index that can't
be dropped directly — table recreation is the only way to
simulate the pre-migration schema.
"""
engine: AsyncEngine = create_async_engine(
"sqlite+aiosqlite:///:memory:",
echo=False,
poolclass=StaticPool,
)
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
# Create everything except episodes.
await conn.run_sync(
lambda sync_conn: [
t.create(sync_conn)
for t in Base.metadata.sorted_tables
if t.name != "episodes"
]
)
# Recreate episodes without the UNIQUE constraint.
await conn.execute(
text(
"""
CREATE TABLE episodes (
id INTEGER NOT NULL PRIMARY KEY,
series_id INTEGER NOT NULL
REFERENCES anime_series(id) ON DELETE CASCADE,
season INTEGER NOT NULL,
episode_number INTEGER NOT NULL,
title VARCHAR(500),
file_path VARCHAR(1000),
is_downloaded BOOLEAN NOT NULL DEFAULT 0,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP NOT NULL,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP NOT NULL
)
"""
)
)
await conn.execute(
text("CREATE INDEX ix_episodes_series_id ON episodes (series_id)")
)
factory = async_sessionmaker(
bind=engine,
@@ -75,6 +116,10 @@ async def _add_episode(
is_downloaded: bool = False,
title: str | None = None,
) -> int:
"""Insert an Episode row. The test fixture drops the UNIQUE
constraint on the ``episodes`` table, so duplicate inserts
succeed — the cleanup tool can then find them, mirroring the
pre-migration pathology it's meant to repair."""
ep = Episode(
series_id=series_id,
season=season,
@@ -120,8 +165,11 @@ async def test_find_groups_duplicates_per_tuple(in_memory_engine):
count=3 and keep_id = the lowest id."""
async with AsyncSession(in_memory_engine) as session:
sid = await _add_series(session, "triple")
e1 = await _add_episode(session, sid, 1, 1)
# Three rows for (S1, E2): keep_id should be e2 (lowest)
await _add_episode(session, sid, 1, 1)
# Three rows for (S1, E2): the first via the normal path,
# the next two via raw SQL to bypass the UNIQUE constraint.
# Mirrors the pre-migration pathology the cleanup tool exists
# to repair.
e2 = await _add_episode(session, sid, 1, 2)
e3 = await _add_episode(session, sid, 1, 2)
e4 = await _add_episode(session, sid, 1, 2)
@@ -147,8 +195,12 @@ async def test_delete_keeps_lowest_id_per_tuple(in_memory_engine):
sid = await _add_series(session, "keep-lowest")
e1 = await _add_episode(session, sid, 1, 1) # unique
e2 = await _add_episode(session, sid, 1, 2) # lowest of dupes
await _add_episode(session, sid, 1, 2) # e3 - duplicate of e2
await _add_episode(session, sid, 1, 2) # e4 - duplicate of e2
await _add_episode(
session, sid, 1, 2
) # e3 - duplicate of e2
await _add_episode(
session, sid, 1, 2
) # e4 - duplicate of e2
e5 = await _add_episode(session, sid, 1, 3) # unique
duplicates = await cli.find_duplicate_episodes()
@@ -179,7 +231,11 @@ async def test_delete_keeps_lowest_even_with_title_metadata(in_memory_engine):
sid = await _add_series(session, "metadata")
e1 = await _add_episode(session, sid, 1, 1) # no title
e2 = await _add_episode(
session, sid, 1, 1, title="Better Episode 1"
session,
sid,
1,
1,
title="Better Episode 1",
)
duplicates = await cli.find_duplicate_episodes()
@@ -203,8 +259,13 @@ async def test_full_workflow_is_idempotent(in_memory_engine):
async with AsyncSession(in_memory_engine) as session:
sid = await _add_series(session, "idempotent")
for ep_num in (1, 2, 3):
for _ in range(3): # 3 copies of each
await _add_episode(session, sid, 1, ep_num)
await _add_episode(session, sid, 1, ep_num)
await _add_episode(
session, sid, 1, ep_num
)
await _add_episode(
session, sid, 1, ep_num
)
first = await cli.find_duplicate_episodes()
assert len(first) == 3
@@ -225,8 +286,10 @@ async def test_max_series_filter_limits_report(in_memory_engine):
s3 = await _add_series(session, "gamma")
for sid in (s1, s2, s3):
for ep_num in (1, 2):
for _ in range(2): # duplicates
await _add_episode(session, sid, 1, ep_num)
await _add_episode(session, sid, 1, ep_num)
await _add_episode(
session, sid, 1, ep_num
)
# Without filter: all three series have duplicates.
full = await cli.find_duplicate_episodes()