Repository navigation
fix: remove duplicate images across pools and guard against new ones - #48
Merged
Merged
Conversation
pytest writes a <testsuites> root without counts, so the check always saw 0 tests and passed. Also fail when the report has skips or no tests.
The fp copy silently overwrote the wildfire image and its label. Collisions are checked before the previous outputs are deleted.
A folder registered in the other pool, or holding an image already in either pool by content, is skipped. Overlapping platform alerts share frames, which put the same image in train and test.
Smoke alerts annotated industrial or other carry no box. They were planned, rejected by add_data, and still counted in the FP quota. The planner now applies add_data's label rule, shared from ingest.py.
Name-based checks missed the same JPEGs filed under two folder names in train and test. The leakage stage now depends on both pools.
- drop fp brison_226, registered as wildfire with human labels - drop fp donon-sarrebourg-02_276 (train), 9 images shared with a test folder - every image now lives in one folder: shared frames are kept by test > val > train, wildfire > fp, larger folder; 79 images trimmed from 26 folders, 13 more folders dropped as full copies or left with too few labels - remove 4 dropped test fp folders from the lockfile - remove 152 planned wildfire alerts that had no box and were never registered - rebuild every stage; leakage report: 21 passed
add_data refuses them, but the plan and ledger were already written: the folder was staged on every run and an fp kept its object's slot.
donon-sarrebourg-02 276 and 317 and bischwiller-02 14-19-18 left the pools as duplicates. Their recurring objects get their slot back; the planner now skips them since they share images with planned folders.
An alert repeating an image from a folder no longer in the export was still planned and took its object's slot. The planner now also compares against the raw pools, and hashes only the frames materialisation writes: one per capture, so a dropped same-second detection no longer blocks an alert.
They were invisible to the builders and checks but still compared against by add_data, and one repeated 29 images of a registered folder. A test now requires each pool's folders to be exactly its registry. Rebuilt outputs are unchanged.
The temporal model needs context. MIN_SEQUENCE_IMAGES applies to wildfire and fp in every split; the freeze skips short test fps so the lockfile still matches the build. Short sequences stay in the pools and in YOLO.
Removes 4 short fps from the test lockfile. Test is now 216 wildfire and 869 fp sequences. Leakage report: 25 passed.
Every guard downstream — the sequential build, the freeze, the planner's content check and the leakage tests — globs *.jpg. A .png or .jpeg accepted here would be invisible to all of them: not counted, not hashed, not deduplicated.
An alert sharing an image with the pools was skipped after select_fp had already picked it, so its recurring object lost its slot and the next-best candidate was never tried. The content check now runs on the usable alerts before selection; only clashes between two new alerts are still resolved in the planning loop, smoke first. Alerts with fewer than MIN_SEQUENCE_IMAGES frames are no longer planned: add_data would take them, the sequential build would drop them, and a false positive would spend its object's slot on YOLO alone.
The quota counts test FPs with at least MIN_SEQUENCE_IMAGES images, so trimming a frozen folder under that shrinks the quota beneath the lockfile and surfaces as a baffling 'exceeds the quota'. Name the cause instead: a frozen folder's content is part of the baseline and must be restored.
The report is committed because CI has no data, so a stale one passed as long as it was green. The check now also compares the inputs dvc.lock recorded for test_data_leakage with what the repository points at: the .dvc pointer for a raw pool, the producing stage's output for a processed dataset, the file itself for code and the git-tracked state files.
The lockfile shrank by 8 folders in this release, which the spec, the runbook and CLAUDE.md all forbid: say so, call v4.6.0 a new baseline not comparable with v4.5.x, and keep the rule for what follows. The runbook also gains the raw-pool prerequisite for the planner, its new warnings, add_data's DUPLICATES section and the matching error-table rows. README no longer claims 50% FP in test.
Narrowing the accepted extensions only made .png and .JPG uncounted: the folder still validated and add_data copied it whole, leaving images every consumer's *.jpg glob would never see. A foreign file is now a structural issue, so the folder is neither copied nor registered.
Two new alerts sharing a frame were resolved after select_fp, so with the quota filled by the dropped one its object got no sequence until a rerun. All new alerts are now deduplicated ahead of selection, smoke first, then by score, and the planning loop no longer has a second check.
Two alerts on one camera view in the same second share a folder name. The content check dropped by name, so when the pair also shared a frame the first survived the loop and the second's drop removed both. Collapse them first, keeping the latter as the materialiser does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
.jpgimage is refused outright, since every downstream guard globs*.jpg; the planner deduplicates new alerts against the pools and against each other before selection, so a clash no longer costs a recurring object its slot or needs a rerun, and leaves out alerts under 4 frames; the freeze names a frozen folder that lost images instead of reporting a quota overflow; CI fails when the committed leakage report predates the inputs recorded indvc.lock.sequential_testv4.6.0 is not a superset of v4.5.x: 8 frozen folders left the pool with the clean-up, so this release is a new baseline and metrics are not comparable across it. Documented in the spec, the runbook and CLAUDE.md. The 4-image rule also drops about 7% of the wildfire sequences from sequential train.