fix(episodes): prevent duplicate-row accumulation at schema + write sites
Followup to commitc84f968(read-boundary dedup) and commitf75d591(cleanup CLI). The read boundary filters duplicates out of the in-memory episodeDict and the CLI cleans up historical duplicates in the DB, but the underlying pathology — duplicate rows being created in the first place — was still active on every rescan. Two layered prevention fixes: 1. Schema-level guard: add UNIQUE(series_id, season, episode_number) to the episodes table. SQLite's CREATE UNIQUE INDEX requires no existing duplicates, but the cleanup CLI fromf75d591has already been run (or is a one-shot prerequisite for users on older DBs). Future duplicate rows are rejected at the DB layer. 2. Write-site guard: SerieScanner.scan_single_series used to `extend` the in-memory episodeDict on every rescan of a series already in keyDict — across N rescans, the same missing list was appended N times, growing the dict with duplicates that then flowed through _update_series_in_db into the episodes table. The fix replaces the cache with the latest scan result instead of extending, and dedupes within a single call as defense in depth against a buggy upstream loader. Defensive dedup is layered three deep: - schema constraint (this commit, primary) - scan_single_series replace-not-extend (this commit, secondary) - episodeDict property read-boundary dedup (commitc84f968, tertiary — covers legacy DBs that predate the constraint) Tests: - Updated test_serie_scanner.test_scan_single_series_existing_entry to assert the new replace-not-merge behavior (the old assertion encoded the buggy extend behavior). - New test_serie_scanner_scan_dedup.py covers the regression directly: two rescans of the same series with the same missing list must yield a canonical dict, not an accumulated one. - test_database_models and test_clean_duplicate_episodes_cli now use a legacy_engine fixture that drops the UNIQUE constraint, so the duplicate-row scenarios they exercise (the read-boundary dedup and the cleanup tool, both meant to defend against pre-migration state) can still be tested under the new schema. Verified manually: clean_duplicate_episodes --apply on the user's backup DB still removes all 633 duplicate rows under the new schema (the CLI doesn't depend on the UNIQUE constraint — it operates on whatever rows already exist).
This commit is contained in:
@@ -776,18 +776,35 @@ class SerieScanner:
|
||||
|
||||
# Create or update AnimeSeries in keyDict
|
||||
if key in self.keyDict:
|
||||
# Update existing anime - rebuild episodeDict from episodes
|
||||
# Update existing anime - rebuild episodeDict from the
|
||||
# latest scan results. The previous implementation
|
||||
# extended the existing list with ``missing_episodes``,
|
||||
# which accumulated duplicates across rescans of the
|
||||
# same series; the in-memory cache then propagated
|
||||
# duplicates through ``_update_series_in_db`` and into
|
||||
# the ``episodes`` table until the UNIQUE constraint
|
||||
# was added. Replace, don't extend.
|
||||
existing = self.keyDict[key]
|
||||
existing_ep_dict = existing.episodeDict
|
||||
# Merge missing episodes
|
||||
# Use ``dict.fromkeys`` to dedupe within a season, in
|
||||
# case ``missing_episodes`` itself contains duplicate
|
||||
# episode numbers from a buggy loader upstream.
|
||||
rebuilt: dict = {}
|
||||
for season, eps in missing_episodes.items():
|
||||
if season not in existing_ep_dict:
|
||||
existing_ep_dict[season] = []
|
||||
existing_ep_dict[season].extend(eps)
|
||||
seen: set = set()
|
||||
cleaned: list = []
|
||||
for ep_num in eps:
|
||||
if ep_num in seen:
|
||||
continue
|
||||
seen.add(ep_num)
|
||||
cleaned.append(ep_num)
|
||||
if cleaned:
|
||||
rebuilt[season] = cleaned
|
||||
existing.episodeDict = rebuilt
|
||||
existing.folder = folder
|
||||
logger.debug(
|
||||
"Updated existing series %s with %d missing episodes",
|
||||
key,
|
||||
sum(len(eps) for eps in missing_episodes.values())
|
||||
sum(len(eps) for eps in rebuilt.values()),
|
||||
)
|
||||
else:
|
||||
# Extract year from folder name if present, otherwise leave as None
|
||||
|
||||
@@ -15,7 +15,17 @@ from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from sqlalchemy import Boolean, DateTime, ForeignKey, Index, Integer, String, Text, func
|
||||
from sqlalchemy import (
|
||||
Boolean,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Index,
|
||||
Integer,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
func,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
|
||||
|
||||
from src.server.database.base import Base, TimestampMixin
|
||||
@@ -328,7 +338,25 @@ class Episode(Base, TimestampMixin):
|
||||
updated_at: Last update timestamp (from TimestampMixin)
|
||||
"""
|
||||
__tablename__ = "episodes"
|
||||
|
||||
|
||||
# Table-level constraints. The UNIQUE constraint on
|
||||
# (series_id, season, episode_number) is the schema-level guard
|
||||
# against the duplicate-row pathology: every (series, season,
|
||||
# episode) tuple can have at most one row. Rescans that try to
|
||||
# create a duplicate row will fail at the DB layer rather than
|
||||
# silently accumulating rows. Defense-in-depth on top of the
|
||||
# write-side dedup in SerieScanner._sync_episodes_to_db and
|
||||
# AnimeService._update_series_in_db, and the read-boundary dedup
|
||||
# in AnimeSeries.episodeDict.
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"series_id",
|
||||
"season",
|
||||
"episode_number",
|
||||
name="uq_episode_per_series_season",
|
||||
),
|
||||
)
|
||||
|
||||
# Primary key
|
||||
id: Mapped[int] = mapped_column(
|
||||
Integer, primary_key=True, autoincrement=True
|
||||
|
||||
Reference in New Issue
Block a user