An implementation of the SORT tracking algorithm using a custom Kalman Filter for prediction of objects' positions. Detections are provided by YOLOv11 and tracked objects are updated by associating predictions with new observations.
- The multi-object tracking system monitors multiple objects (e.g., vehicles) within video sequences.
- A Kalman Filter is employed to estimate an object's position, even when the object is not detected in a given frame.
- The Hungarian Algorithm constructs a cost matrix (often based on IoU) to optimally match new detections with predictions. If no satisfactory match is found, a new tracker is initiated.
- Bounding boxes are annotated with identification numbers for verification of the tracking process.
The Kalman Filter code can be found on: https://github.com/ManuelZ/Kalman-Filter
MOT16-13-annotated.mp4
The project is an installable package. Environments and dependencies are managed with uv:
uv venv
uv pip install -e .
Optional dependency groups:
uv pip install -e ".[deep]"— PyTorch + torchvision + rerun-sdk (DeepSORT re-ID)uv pip install -e ".[eval]"— trackeval (benchmark scoring)uv pip install -e ".[deep,eval]"— both
The scripts/ directory contains entry points for running, benchmarking, and tuning the tracker:
- scripts/mot16_sort.py — Runs SORT, DeepSORT, Ultralytics ByteTrack, or Ultralytics BoT-SORT on a single MOT16 sequence and writes tracking results in MOT CSV format. Detections come either from the dataset's
det.txt(--det) or from a YOLO model (--yolo-path); the tracker is selected with--tracker sort|deepsort|ul_bytetrack|ul_botsort. - scripts/mot16_benchmark.py — Runs SORT, DeepSORT, Ultralytics ByteTrack, or Ultralytics BoT-SORT over every sequence under
--mot16-root, arranges the outputs in the layout expected by TrackEval, and invokes it to report the HOTA metric.
Usage examples for each script are shown in the sections below.
MOT16 is a multi-object tracking benchmark focused on pedestrian tracking in crowded scenes, released as part of the MOTChallenge. It contains 14 video sequences (7 for training with public ground truth, 7 for testing) captured from static and moving cameras, at various resolutions and frame rates, under different lighting and viewpoint conditions.
Layout on disk:
MOT16/
├── train/ # ground truth is provided
│ ├── MOT16-02/
│ │ ├── img1/ # frames as 000001.jpg, 000002.jpg, ...
│ │ ├── det/det.txt # public detections (DPM)
│ │ ├── gt/gt.txt # ground-truth trajectories
│ │ └── seqinfo.ini # name, imDir, frameRate, seqLength, imWidth, imHeight, imExt
│ ├── MOT16-04/...
│ ├── MOT16-05/...
│ ├── MOT16-09/...
│ ├── MOT16-10/...
│ ├── MOT16-11/...
│ └── MOT16-13/...
└── test/ # no gt/ folder; submit results to MOTChallenge
├── MOT16-01/
├── MOT16-03/...
├── MOT16-06/...
├── MOT16-07/...
├── MOT16-08/...
├── MOT16-12/...
└── MOT16-14/...
- Each line of
gt.txtcontains 9 values and follow the convention:frame, id, x, y, w, h, conf/valid, class, visibility, whereclassidentifies the object category andvisibility∈ [0, 1]. - Each line of
det.txtcontains 10 values and follows the convention:frame, id, x, y, w, h, conf, x, y, z, whereidis always-1and x, y, z are always-1.
Sequences vary in resolution (e.g. 1920×1080 for sequences 01-05,07-14, 640×480 for 05,06) and frame rate (14, 25, or 30 fps).
scripts/mot16_benchmark.py runs the tracker on every sequence under --mot16-root, writes MOT-format results in TrackEval's expected layout, and invokes TrackEval to report the HOTA metric.
Throughout this section, "dataset detections" means the public detections shipped with MOT16 (det/det.txt inside each sequence), while "YOLO detections" means detections produced on-the-fly by a YOLO model passed via --yolo-path.
python scripts/mot16_benchmark.py ^
--mot16-root /path/to/MOT16/train ^
--tracker <tracker> ^
[--yolo-path yolo11x.pt]
--tracker |
Detections |
|---|---|
sort (default) |
dataset det.txt, or --yolo-path |
deepsort |
dataset det.txt, or --yolo-path |
ul_bytetrack |
--yolo-path required |
ul_botsort |
--yolo-path required |
The ul_bytetrack and ul_botsort options delegate detection and tracking to Ultralytics' built-in model.track() pipeline (bytetrack.yaml / botsort.yaml) and require --yolo-path. They're included for side-by-side comparison against this repo's SORT/DeepSORT.
Per-sequence results and TrackEval's CSV summaries land under trackeval_results/ by default (override with --results-dir).
scripts/mot16_sort.py runs SORT or DeepSORT on a single MOT16 sequence and writes results in MOT CSV format.
Detections can come from the dataset's own detection files (--det) or from a YOLO model (--yolo-path).
The tracker is selected with --tracker sort (default) or --tracker deepsort.
python scripts/mot16_sort.py ^
--frames /path/to/MOT16/train/MOT16-13/img1 ^
--tracker <tracker> ^
[--det /path/to/MOT16/train/MOT16-13/det/det.txt | --yolo-path yolo11x.pt] ^
--output results/seq_13_<tracker>.txt
--tracker |
Detections |
|---|---|
sort (default) |
--det or --yolo-path |
deepsort |
--det or --yolo-path |
ul_bytetrack |
--yolo-path only |
ul_botsort |
--yolo-path only |
tracking/sort.py also implements its own ByteTrack-style two-stage association (enabled via the bytetrack section in tracking/sort.yaml) as an alternative to the classic SORT matching. High-score detections are matched first, then low-score detections are given a second chance against the remaining unmatched predictions with a looser IoU threshold.
Work in progress: this is separate from the ul_bytetrack option (which delegates to Ultralytics' own implementation). The current logic does not implement track "rebirth" as described in [5], lost trackers are dropped after max_cycles_without_update cycles rather than being kept around for potential re-association with a later detection.
Ultralytics ByteTrack's two-stage association beats this SORT implementation on every MOT16 sequence (+4.9 HOTA combined), mainly via better ID association.
SORT with YOLO11x detections (ByteTrack disabled)
HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0) HOTALocA(0)
MOT16-02 27.223 23.926 31.41 24.798 78.351 32.542 86.268 83.21 27.783 32.667 78.188 25.542
MOT16-04 31.267 23.909 40.989 24.602 83.454 42.49 90.208 86.405 31.739 37.677 81.915 30.863
MOT16-05 39.069 40.813 38.319 47.914 59.288 41.746 71.951 74.783 42.635 59.749 63.942 38.204
MOT16-09 45.071 54.011 37.992 59.764 74.362 40.825 79.325 81.479 47.592 60.033 74.825 44.92
MOT16-10 33.057 37.005 29.736 39.698 71.606 31.324 77.245 78.992 34.325 44.902 72.253 32.443
MOT16-11 48.385 49.788 47.406 57.463 70.355 50.819 81.357 81.931 52.152 64.896 74.327 48.235
MOT16-13 27.815 25.701 31.117 27.172 68.786 32.879 77.809 77.531 28.749 36.576 70.314 25.718
COMBINED 34.003 30.442 38.449 32.429 74.214 40.543 84.484 81.983 35.197 43.673 75.31 32.89
Ultralytics ByteTrack with YOLO11x detections
HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0) HOTALocA(0)
MOT16-02 32.83 26.268 41.29 27.282 79.018 44.038 81.608 83.584 33.512 40.015 78.713 31.497
MOT16-04 37.23 27.98 49.596 28.934 83.826 51.484 89.642 86.925 37.877 44.755 82.288 36.828
MOT16-05 44.014 44.795 43.718 52.47 63.306 50.422 65.407 77.434 47.829 64.543 68.593 44.272
MOT16-09 49.366 56.208 43.673 62.069 76.077 47.847 76.387 82.664 52.031 64.259 76.872 49.397
MOT16-10 35.418 37.848 33.288 40.619 72.619 37.887 71.809 79.785 36.755 48.206 73.33 35.35
MOT16-11 50.663 52.457 49.245 60.091 73.278 54.709 79.767 83.756 54.369 66.007 77.128 50.91
MOT16-13 35.324 27.772 45.306 29.214 74.218 48.264 78.93 81.231 36.304 45.209 76.055 34.383
COMBINED 38.91 33.438 45.599 35.62 76.463 49.19 81.91 83.392 40.244 49.251 77.511 38.175
This implementation reflects my own learning journey, drawing on insights from the structure, code, and techniques found in:
-
[1] A. Milan, L. Leal-Taixe, I. Reid, S. Roth, and K. Schindler, “MOT16: A Benchmark for Multi-Object Tracking,” 2016, arXiv. doi: 10.48550/ARXIV.1603.00831.
-
[2] A. Bewley, abewley/sort. (Jul. 30, 2026). Python. Accessed: Jul. 31, 2026. [Online]. Available: https://github.com/abewley/sort
-
[3] PacktPublishing/OpenCV-4-with-Python-Blueprints-Second-Edition. (Jul. 10, 2026). Python. Packt. Accessed: Jul. 31, 2026. [Online]. Available: https://github.com/PacktPublishing/OpenCV-4-with-Python-Blueprints-Second-Edition/blob/master/chapter10/sort.py
-
[4] J. Cohen, Jeremy26/tracking_course. (Nov. 26, 2025). Jupyter Notebook. Accessed: Jul. 31, 2026. [Online]. Available: https://github.com/Jeremy26/tracking_course
-
[5] X. Zhou, V. Koltun, and P. Krähenbühl, “Tracking Objects as Points,” 2020, arXiv. doi: 10.48550/ARXIV.2004.01177.