fix(episodes): prevent duplicate-row accumulation at schema + write sites

Followup to commit c84f968 (read-boundary dedup) and commit f75d591
(cleanup CLI). The read boundary filters duplicates out of the
in-memory episodeDict and the CLI cleans up historical duplicates
in the DB, but the underlying pathology — duplicate rows being
created in the first place — was still active on every rescan.

Two layered prevention fixes:

1. Schema-level guard: add UNIQUE(series_id, season, episode_number)
   to the episodes table. SQLite's CREATE UNIQUE INDEX requires
   no existing duplicates, but the cleanup CLI from f75d591 has
   already been run (or is a one-shot prerequisite for users on
   older DBs). Future duplicate rows are rejected at the DB layer.

2. Write-site guard: SerieScanner.scan_single_series used to
   `extend` the in-memory episodeDict on every rescan of a
   series already in keyDict — across N rescans, the same missing
   list was appended N times, growing the dict with duplicates that
   then flowed through _update_series_in_db into the episodes
   table. The fix replaces the cache with the latest scan result
   instead of extending, and dedupes within a single call as
   defense in depth against a buggy upstream loader.

Defensive dedup is layered three deep:
  - schema constraint (this commit, primary)
  - scan_single_series replace-not-extend (this commit, secondary)
  - episodeDict property read-boundary dedup (commit c84f968,
    tertiary — covers legacy DBs that predate the constraint)

Tests:
  - Updated test_serie_scanner.test_scan_single_series_existing_entry
    to assert the new replace-not-merge behavior (the old assertion
    encoded the buggy extend behavior).
  - New test_serie_scanner_scan_dedup.py covers the regression
    directly: two rescans of the same series with the same missing
    list must yield a canonical dict, not an accumulated one.
  - test_database_models and test_clean_duplicate_episodes_cli now
    use a legacy_engine fixture that drops the UNIQUE constraint,
    so the duplicate-row scenarios they exercise (the read-boundary
    dedup and the cleanup tool, both meant to defend against
    pre-migration state) can still be tested under the new schema.

Verified manually: clean_duplicate_episodes --apply on the user's
backup DB still removes all 633 duplicate rows under the new
schema (the CLI doesn't depend on the UNIQUE constraint — it
operates on whatever rows already exist).
This commit is contained in:
2026-09-15 20:58:30 +02:00
parent f75d591453
commit c3aca9217d
6 changed files with 417 additions and 57 deletions

View File

@@ -776,18 +776,35 @@ class SerieScanner:
# Create or update AnimeSeries in keyDict
if key in self.keyDict:
# Update existing anime - rebuild episodeDict from episodes
# Update existing anime - rebuild episodeDict from the
# latest scan results. The previous implementation
# extended the existing list with ``missing_episodes``,
# which accumulated duplicates across rescans of the
# same series; the in-memory cache then propagated
# duplicates through ``_update_series_in_db`` and into
# the ``episodes`` table until the UNIQUE constraint
# was added. Replace, don't extend.
existing = self.keyDict[key]
existing_ep_dict = existing.episodeDict
# Merge missing episodes
# Use ``dict.fromkeys`` to dedupe within a season, in
# case ``missing_episodes`` itself contains duplicate
# episode numbers from a buggy loader upstream.
rebuilt: dict = {}
for season, eps in missing_episodes.items():
if season not in existing_ep_dict:
existing_ep_dict[season] = []
existing_ep_dict[season].extend(eps)
seen: set = set()
cleaned: list = []
for ep_num in eps:
if ep_num in seen:
continue
seen.add(ep_num)
cleaned.append(ep_num)
if cleaned:
rebuilt[season] = cleaned
existing.episodeDict = rebuilt
existing.folder = folder
logger.debug(
"Updated existing series %s with %d missing episodes",
key,
sum(len(eps) for eps in missing_episodes.values())
sum(len(eps) for eps in rebuilt.values()),
)
else:
# Extract year from folder name if present, otherwise leave as None

View File

@@ -15,7 +15,17 @@ from datetime import datetime, timezone
from enum import Enum
from typing import Any, Dict, List, Optional
from sqlalchemy import Boolean, DateTime, ForeignKey, Index, Integer, String, Text, func
from sqlalchemy import (
Boolean,
DateTime,
ForeignKey,
Index,
Integer,
String,
Text,
UniqueConstraint,
func,
)
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
from src.server.database.base import Base, TimestampMixin
@@ -328,7 +338,25 @@ class Episode(Base, TimestampMixin):
updated_at: Last update timestamp (from TimestampMixin)
"""
__tablename__ = "episodes"
# Table-level constraints. The UNIQUE constraint on
# (series_id, season, episode_number) is the schema-level guard
# against the duplicate-row pathology: every (series, season,
# episode) tuple can have at most one row. Rescans that try to
# create a duplicate row will fail at the DB layer rather than
# silently accumulating rows. Defense-in-depth on top of the
# write-side dedup in SerieScanner._sync_episodes_to_db and
# AnimeService._update_series_in_db, and the read-boundary dedup
# in AnimeSeries.episodeDict.
__table_args__ = (
UniqueConstraint(
"series_id",
"season",
"episode_number",
name="uq_episode_per_series_season",
),
)
# Primary key
id: Mapped[int] = mapped_column(
Integer, primary_key=True, autoincrement=True