MSc Dissertation — Kautum Krishnan Panjalaraja (K25020148) MSc Data Science, Department of Informatics, King's College London Supervisor: Prof. Hannah Cao · Submitted 6 August 2026
Report (PDF) · Presentation Video · Research Journey · Declaration of Sole Authorship
Published IoT intrusion detection results routinely report accuracy or F1 at or above 98%. Almost all of them are measured by shuffling network traffic at random into a training set and a test set — which, because the attack traffic in these datasets comes from a handful of physical machines, puts the same attacking machine on both sides of the split. A model can then score well by recognising a machine it has already met rather than by recognising an attack. A deployed detector never gets that advantage, since the machine attacking it tomorrow was not in its training data.
This project rebuilds the evaluation around that constraint and re-measures the field's two headline claims. First: how much of the reported score survives when whole attacker machines are held out of training? The same detector scores macro-F1 0.9002 under the standard random split and 0.6059 once attacker machines are held out — and the ranking between two competing model families reverses between the two protocols. Second: does predicting a flow's future packets, the mechanism two recent papers propose but never test end to end, actually improve early detection? Built and measured under the same leakage-controlled protocol, it does not: 18 of 21 predict-then-classify configurations score worse than classifying the observed packets alone, and the underlying "plausibility" premise those papers rely on turns out to run backwards.
This repository contains everything needed to reproduce both results, the three further negative results that follow from them, and the evaluation instrument itself.
Same detector, same data, two evaluation protocols — the ranking of models reverses along with the score.
| Finding | Evidence |
|---|---|
| Evaluation protocol, not the model, accounts for roughly a third of reported performance, and it reverses which model wins | macro-F1 0.9002 (random split) → 0.6059 (attacker-machine holdout); model ranking flips |
| The holdout penalty falls on naming the attack, not detecting it | attack-vs-benign separation holds at AUC 0.9716 even as multi-class F1 falls |
| Predicting a flow's future packets does not improve detection, and the field's "plausibility" premise is inverted | 18 of 21 predict-then-classify configurations score below the five-packet baseline; the plausibility signal's AUC falls below 0.5 across all three predictors tested |
| No language-model component beat its simplest sensible alternative, in any of three roles tried | fine-tuned DistilBERT finishes last of ten model families; a 1.5B-parameter packet predictor loses to copying the previous packet forward |
| Detection and prediction both work when aimed correctly | device-level escalation accuracy 0.9853 under holdout; next-event forecasting reaches 0.9261 top-3 accuracy on botnets never seen in training |
Full detail, per-class breakdowns, and the reasoning behind each result are in
the report.
docs/research-journey.md covers how the project's
evaluation methodology took shape, including a dataset survey of roughly 25
candidates, four real engineering failures, and a scoring-metric bug that
reversed a methodological conclusion mid-project.
| File | What it is |
|---|---|
..._FinalReport_2025-26.pdf |
The report as submitted, 39 pages |
Sixteen notebooks, numbered in the order to read them. Each result file in
Results/ carries the number of the notebook that produced it, so
03_detector_randomforest.ipynb writes 03_detector_randomforest.json. All
notebooks are saved with their output cells intact, so results can be read
without executing anything.
| File | What it does |
|---|---|
00_all_results_presentation.ipynb |
Loads every result file and prints the numbers quoted in the report |
01_verification_proof.ipynb |
Read-only check. Reloads the saved models and data, retrains the baseline live, and confirms the stored numbers |
02_data_preparation_toniot.ipynb |
Builds the 109,203-flow corpus from the raw ToN_IoT captures |
03_detector_randomforest.ipynb |
The locked detector under both protocols. Produces the headline 0.9002 and 0.6059 |
04_detector_lightgbm.ipynb |
The alternative design on an identical protocol, where the ranking inversion appears |
05_host_escalation.ipynb |
Aggregates flow verdicts up to devices, binary accuracy 0.9853 |
06_forecaster_markov_lstm.ipynb |
Next-event forecasting on CTU-13 and on Gotham |
07_augmentation_qwen25.ipynb |
LoRA fine-tune, synthetic flow generation, augmentation probe |
08_augmentation_smollm2.ipynb |
The same study with a second generator |
09_augmentation_glmedge.ipynb |
The same study with a third generator |
10_augmentation_chatglm2_not_run.ipynb |
Never run — needed a CUDA GPU that was not available. No results exist and none are claimed anywhere in the report |
11_core_predict_then_classify.ipynb |
The core experiment. Predicts future packets, classifies, and compares against observed-only. 18 of 21 configurations lose |
12_figure_generation.ipynb |
Regenerates the eleven data-driven report figures as vector output |
13_packet_count_k3.ipynb, _k5.ipynb, _k10.ipynb |
Confirmatory reruns at three packet-count settings; k5 is the setting used throughout the report |
Seventeen JSON files. Every number quoted in the report traces to one of these, and the report names the relevant file in each table caption.
| File | Backs which claim |
|---|---|
03_detector_randomforest.json |
Detection under both protocols, and the per-class F1 breakdown |
04_detector_lightgbm.json |
The LightGBM comparison and the ranking inversion |
05_host_escalation.json |
Host escalation, 8 of 9 machines caught, 0 false alarms on 59 benign hosts |
06_forecaster_ctu13.json, 06_forecaster_gotham.json |
Next-event forecasting on CTU-13 held-out botnets and its replication on Gotham |
07_augmentation_qwen25.json, 08_augmentation_smollm2.json, 09_augmentation_glmedge.json |
The three generator-model augmentation probes |
11_core_predict_then_classify.json |
The core experiment, all 24 rows |
bakeoff_model_families.json, bakeoff_generation.json |
The earlier-phase model and generation bake-offs |
13_packet_count_k3/k5/k10.json |
The packet-count decision |
robustness_screening.json, robustness_full_scale.json, robustness_regularisation.json |
The robustness search against the locked detector |
One note on reading these files: several contain a field named honest,
which stores the per-fold mean macro-F1. The report quotes pooled
out-of-fold macro-F1 throughout — the mean of the per_class_honest values,
not the honest field. Section 5.5 of the report explains why the pooled
figure is the reliable one.
Eleven data-driven figures, each as a vector PDF and a PNG. The report contains thirteen figures in total; the remaining two are protocol diagrams drawn directly in LaTeX.
Prepared model inputs, roughly 49 MB. The raw corpora are not included — see Datasets below.
The recorded project presentation video.
- Open
01_verification_proof.ipynb. It reloads the saved data and models, rebuilds the fixed holdout from the raw group labels, retrains the baseline RandomForest live, and checks what it computes against what is stored. It reproduces holdout 0.7621 and random split 0.8979 to four decimal places. - For the headline detection numbers, run
03_detector_randomforest.ipynb, which writes03_detector_randomforest.json. 00_all_results_presentation.ipynbprints every number the report quotes, in report order, without retraining anything.
No raw packet captures are included. The five corpora total roughly 194 GB — far beyond a reasonable repository size — and two of them ship live malware binaries that should not be redistributed. All five are publicly available from their originating institutions; full citations are in the report's References section.
| Corpus | Role in this project | Source institution |
|---|---|---|
| ToN_IoT | Training. Every detection number in the report comes from here: 109,203 flows, nine classes, 68 hosts, nine of them attacker machines | UNSW Canberra |
| CTU-13 | Testing. Real botnet captures — the primary external test and the corpus the next-event forecaster is trained and tested on | Stratosphere Laboratory, CTU |
| Gotham | Validation. An independent testbed confirming the forecasting result is not specific to CTU-13 | Public research testbed |
| IoT-23 | Stress test, included deliberately as the boundary case where the forecaster fails — and that failure is reported, not omitted | Stratosphere Laboratory, CTU |
| CICIoT2023 | Replication corpus for the two baseline papers this work responds to. Contributes no training data; nothing in the report depends on it | Canadian Institute for Cybersecurity, UNB |
CTU-13 and IoT-23 ship live malware binaries alongside their captures. They
were held on a read-only volume throughout this project and nothing in them
was ever executed — only derived event logs and flow statistics left that
volume. Data/ (roughly 49 MB) is what the notebooks actually read, and
every number in the report reproduces from these files plus the notebooks,
without the raw captures. Raw captures are turned into those extracts by
Notebooks/02_data_preparation_toniot.ipynb; paths inside it point at the
original working tree and would need adjusting to re-run from raw. Re-running
the three language-model notebooks additionally needs a CUDA GPU of compute
capability 7.0 or above.
Result files and notebooks were renamed for this repository so each result
carries the number of the notebook that produced it. Their stored output
cells were not — those are the record of the runs as they actually
happened, so a few print the original working names
(K02_detector_metrics.json, kchoice_k5.json, and Kaggle output paths for
the three generator studies). Editing them would mean altering evidence after
the fact. The referenced content is the same as the files now in Results/
under their new names.
Python 3.11. Detection and forecasting run on CPU. The three LoRA
fine-tuning notebooks ran on Kaggle's free-tier T4 GPUs. Third-party
libraries used are listed in DECLARATION.txt and are not
included in this repository.
Sole author: Kautum Krishnan Panjalaraja. See
DECLARATION.txt for the full authorship declaration and
third-party component disclosure. The report's Section 6 discloses the use
of AI-assisted tooling for grammar and consistency checking during
manuscript preparation, confirmed with the supervisor; all experiments were
designed, run, and verified by the author.
Code in this repository is released under the MIT License. The report, figures, and video are the author's academic work, shared here for reference; copyright is retained by the author.