Time series event detection models

[samueljackson92]

Adds four new time series event detection models to toktagger:

• DTW motifs - use user generated annotations as motifs and use Dynamic Time Warping to detect similar events within the time series trace.
• Stumpy - use user generated annotations as motifs for the matrix profile to find similar motifs in the data.
• Minirocket - use minirocket and a linear probe to classify time series windows.
• Shapelets - use a shapelet transform to classify time series windows (using sktime's default RotationForest estimator, an ensemble of decision trees).

Each model supports either single or multi-channel inputs.

To Test:

• open a project with a time series view.
• click on the train models button.
• train one of the models
• once trained, close the training window
• open the model predictions interface
• run the model
• annotations should be generated for this shot and plotted.

Additionally, check the new tests & CI are passing

[abdullah-ukaea]

Known issue: multiple signal inputs

As described in #330, the parameter description says that users can provide multiple signals, but the current schema form only renders one input and provides no way to add another. The Pydantic schema supports multiple values, the problem is in the UI’s list handling.

@wk9874 is looking into this. I have also prototyped a possible fix in https://github.com/ukaea/toktagger/blob/slj/ts-event-detection-models-schemaFix/toktagger/ui/src/app/components/ui/schemaForm.tsx

[abdullah-ukaea]

The new models trained and predicted successfully in my functional testing, but the training forms contain too much text and require excessive scrolling. Overall it is is not a good UI experience.

Could we show only a short model summary and brief parameter descriptions in the form? The detailed explanations could perhaps live in the Zensical documentation, with a link from the UI if needed.

The user currently sees:

I think the model summaries could be closer to:

• DTW: “Detects events that have a similar shape to labelled examples using Dynamic Time Warping.”
• STUMPY: “Detects events that are similar to labelled examples using STUMPY.”
• MiniRocket: “Learns to detect labelled events using MiniRocket time-series features.”
• Shapelet: “Learns to detect events using characteristic shapes found in the training signals.”

Parameter descriptions should be similarly concise.

The form text is generated from the model docstrings in these files, so make the changes here:

• toktagger/api/models/shapelet.py
• toktagger/api/models/stumpy_motif.py
• toktagger/api/models/dtw_motif.py
• toktagger/api/models/minirocket.py

[samueljackson92]

@abdullah-ukaea this should be ready for re-review now.

[abdullah-ukaea]

As discussed, could we make these two usability improvements before I start the next phase of code review? @samueljackson92

1. In the training form, make Signal Names a dropdown containing the signals available for the current sample, rather than requiring users to type them manually.

2. In the prediction tool, let users select a specific trained model by name. Currently, selecting a model type such as MiniRocket automatically uses its latest version. If users have trained models named test1 and test2, there is no way to choose between them.

[abdullah-ukaea]

Side Note

In similar spirit to #387 (review) I think this PR includes a few changes beyond just adding the time-series event-detection models. Changes such as multi-file tabular loading could have been handled in a separate PR. Smaller PRs make the intended behaviour clearer, are quicker to review and test, and make things easier for both authors and reviewers.

Summary of review

Thanks for addressing the earlier feedback.

The main areas I think still need attention before merging are:

• I really think model training should probably remain on the project samples page. From an individual shot page, most users will not realise that they are training on all validated project samples. See #304 (comment)
• There are still some time series model correctness edge cases around window sizes, background-only shots, MiniRocket’s minimum window length etc
• Model names are not unique but are used to identify predictions and existing predictions are deleted before replacement predictions succeed.
• I am not convinced the multi-file tabular/Parquet loader combines files correctly. Have you functionally tested it?
• There are many verbose multi-line comments that add unnecessary noise to our codebase. Claude in general loves adding comments so we need to tighten claude.md to combat this.
• This branch is quite stale, I recommend solving the minor merge conflict and pulling the latest dev branch.

The core functionality is more or less done with some bugs, but we really need to do some tidying up before this can be merged.

[abdullah-ukaea]

Users can give multiple models the same name, but we use that name to identify and delete their predictions. This means predictions from different models could be mixed up or removed.

Could we use the unique model ID for prediction tracking and keep the model name for display only?

[abdullah-ukaea]

The previous successful predictions are removed as soon as a new job is submitted. If something fails, the user loses the old results and gets no replacement.

Maybe keep the previous predictions until the new inference has completed successfully. Then you can replace the old predictions as part of the successful write, ideally in one database operation

[abdullah-ukaea]

I think the shared alignment helper needs more consideration.

This aligns every signal to the longest signal’s full time range, so missing sections are filled with repeated values that could affect training and prediction. It also assumes the time arrays are valid and match their value arrays.

Example: Say you have one signal "Ip" between 0 - 10 seconds and another signal "dalpha" between 2-8 seconds, np.interp fills the missing edges with repeated endpoint values.

Could we validate each signal and align them only over their shared time range, returning a clear error when the data is incompatible?

Examples of potential errors cases:

• Return a clear error for an empty overlap
• Return a clear error for non-increasing or non-finite times
• Return a clear error for mismatched time/value lengths

[abdullah-ukaea]

Are you sure other labels should affect the selected class’s window size? This doesn't look right.

The window size is calculated before filtering by class_label, so annotations from other labels can incorrectly affect it. For example, long H-mode annotations could make the window for an ELM model much larger.

Could we calculate the window using only annotations that match class_label, and use all annotations only when class_label is blank?

[abdullah-ukaea]

Similar issue to comment https://github.com/ukaea/toktagger/pull/304/changes#r3960474554

To summarise I am concerned that other non selected class labels affect the selected class’s window size.

I think we should filter ann_time_pairs by params.class_label before calculating the window size.

[abdullah-ukaea]

Should model training really be available from the individual sample page?

Opening the training form from one shot suggests that only this shot will be used, but the model actually trains on all validated samples in the project. Nothing in the modal explains that the current shot is not the only training input, so users can easily train on different data from what they expect.

This misleads users when doing model training. Adding training here is a non-trivial UI decision that needs some clear functional justification.

Could we keep training on the samples page like before, or if you insist on keeping it here clearly explain in the modal that all validated project samples will be used?

Personally, I think model training should remain on the samples page like before unless we add a feature to support training on the current shot only.