Deep Knowledge Tracing with RNN and LSTM - ASSISTments 2017

Overview

This project implements a Knowledge Tracing system designed to predict whether a student will answer their next educational interaction correctly or incorrectly. The models are trained and evaluated on the ASSISTments 2017 dataset, utilizing various Deep Learning architectures implemented in PyTorch.

The codebase has been refactored into a highly modular pipeline. Each core stage—data preprocessing, feature engineering, sequence padding, model definition, training, and evaluation—is isolated in its own dedicated module. The versioned execution scripts act as orchestrators to run specific configurations of the pipeline.

---

Project Structure

Deep-Knowledge-Tracing/
├── Data/
│   └── assistments2017.csv
├── Src/
│   ├── preprocessing.py          # Data loading, time processing, skill encoding, and scaling
│   ├── features.py               # Feature list configurations for each model version
│   ├── sequences.py              # Student-level sequence generation and padding
│   ├── datasets.py               # TensorDataset and DataLoader construction
│   ├── models.py                 # RNN and LSTM PyTorch architectures
│   ├── training.py               # Training and validation loops with Early Stopping
│   ├── evaluation.py             # Testing, metrics calculation, JSON logging, and plotting
│   ├── test_saved_model.py       # Inference script for pre-trained models on test data
│   ├── Task3_RNN_V3.py           # Execution script for RNN V3
│   ├── Task3_LSTM_V1.py          # Execution script for LSTM V1
│   └── Archive/
│       ├── Task3_RNN_V1.py       # Execution script for RNN V1
│       ├── Task3_RNN_V2.py       # Execution script for RNN V2
│       └── Draft/Task3.ipynb
├── Models/                       # Saved weights and JSON metric logs (Ignored in git)
├── Plots/                        # EDA and evaluation visualizations
├── test_output/                  # Inference CSV outputs
├── requirements.txt
└── README.md

---

Model Versions

RNN V1

Execution Script: Src/Archive/Task3_RNN_V1.py

• Model Architecture: KT_RNN_V1
• Skill Embedding: Excluded
• Features Used: hintCount, consecutiveErrorsInRow, attemptCount, month, year
• Outputs: Stored in Models/RNN_V1/ and Plots/RNN_V1/.

RNN V2

Execution Script: Src/Archive/Task3_RNN_V2.py

• Model Architecture: KT_RNN_WithSkills
• Skill Embedding: Included
• Features Used: hintCount, consecutiveErrorsInRow, attemptCount, month, year, day, timeTaken, is_weekend
• Outputs: Stored in Models/RNN_V2/ and Plots/RNN_V2/.

RNN V3

Execution Script: Src/Task3_RNN_V3.py

• Model Architecture: KT_RNN_WithSkills
• Features Used: Same as V2, with a mathematical optimization: a log1p transformation is applied to timeTaken prior to standard scaling to handle extreme outliers.
• Outputs: Stored in Models/RNN_V3/ and Plots/RNN_V3/.

LSTM V1

Execution Script: Src/Task3_LSTM_V1.py

• Model Architecture: KT_LSTM_WithSkills
• Features Used: Inherits the feature set and log1p(timeTaken) transformation from RNN V3.
• Outputs: Stored in Models/LSTM_V1/ and Plots/LSTM_V1/.

---

Performance Comparison

The following table summarizes the test metrics derived from the Models/<version>/V<n>.json log files. The reported Accuracy, Precision, Recall, and AUC were evaluated strictly on the test set.

Model Family	Version	Primary Modification vs. Previous Version	MAX_SEQ_LEN	HIDDEN_DIM	NUM_LAYERS	DROPOUT	LR	EPOCHS	Accuracy	Precision	Recall	AUC
RNN_V1	V1	Baseline without skill embedding	100	64	2	0.2	0.001	20	0.8199	0.5857	0.3481	0.8682
RNN_V1	V2	Increased learning rate (0.001 → 0.003)	100	64	2	0.2	0.003	20	0.8198	0.5807	0.3646	0.8760
RNN_V1	V3	Increased sequence length (100 → 110)	110	64	2	0.2	0.003	20	0.8371	0.5929	0.3393	0.8896
RNN_V1	V4	Increased hidden dimension (64 → 128)	110	128	2	0.2	0.003	20	0.8371	0.5802	0.3853	0.8890
RNN_V1	V5	Increased epochs (20 → 30)	110	128	2	0.2	0.003	30	0.8415	0.5850	0.4499	0.8927
RNN_V2	V1	Added skill embedding and behavioral/temporal features	110	128	2	0.2	0.003	30	0.8381	0.5829	0.3931	0.8913
RNN_V2	V2	Increased dropout (0.2 → 0.4)	110	128	2	0.4	0.003	30	0.8390	0.5793	0.4269	0.8927
RNN_V2	V3	Increased sequence length (110 → 140)	140	128	2	0.4	0.003	30	0.8728	0.5690	0.4622	0.9181
RNN_V2	V4	Increased sequence length (140 → 150)	150	128	2	0.4	0.003	30	0.8777	0.5662	0.3644	0.9208
RNN_V3	V1	Applied log1p(timeTaken) transformation vs. RNN_V2	140	128	2	0.4	0.003	30	0.8698	0.5762	0.3448	0.9143
RNN_V3	V2	Decreased dropout (0.4 → 0.2)	140	128	2	0.2	0.003	30	0.8749	0.5818	0.4507	0.9194
RNN_V3	V3	Increased sequence length (140 → 150)	150	128	2	0.2	0.003	30	0.8839	0.6055	0.3776	0.9254
RNN_V3	V4	Increased sequence length (150 → 200)	200	128	2	0.2	0.003	30	0.9111	0.5695	0.4668	0.9456
RNN_V3	V5	Increased sequence length (200 → 300)	300	128	2	0.2	0.003	30	0.9422	0.5896	0.4430	0.9659
RNN_V3	V6	Increased sequence length (300 → 400)	400	128	2	0.2	0.003	30	0.9566	0.5848	0.4604	0.9751
RNN_V3	V7	Increased sequence length (400 → 500)	500	128	2	0.2	0.003	30	0.9649	0.5847	0.4294	0.9799
LSTM_V1	V1	Replaced RNN with LSTM, using RNN_V3 features	150	128	2	0.2	0.003	30	0.8823	0.5842	0.4140	0.9247
LSTM_V1	V2	Decreased number of layers (2 → 1)	150	128	1	0.2	0.003	30	0.8825	0.5839	0.4218	0.9255
LSTM_V1	V3	Increased hidden dimension (128 → 256)	150	256	1	0.2	0.003	30	0.8854	0.5972	0.4379	0.9263
LSTM_V1	V4	Increased sequence length (150 → 200)	200	128	1	0.2	0.003	30	0.9117	0.5839	0.4146	0.9460
LSTM_V1	V5	Increased sequence length (200 → 300)	300	128	1	0.2	0.003	30	0.9408	0.5643	0.5031	0.9658
LSTM_V1	V6	Increased sequence length (300 → 400)	400	128	1	0.2	0.003	30	0.9553	0.5711	0.4379	0.9743
LSTM_V1	V7	Increased hidden dim (128 → 256) for long sequence	400	256	1	0.2	0.003	30	0.9561	0.5781	0.4581	0.9745

Trend Analysis

• RNN_V1: Initiated as a baseline without skill embedding. Increasing the learning rate in V2 significantly improved AUC, but the major leap occurred when expanding MAX_SEQ_LEN from 100 to 110 in V3. Scaling the HIDDEN_DIM to 128 did not improve Accuracy alone, but pairing it with EPOCHS=30 in V5 yielded the best result for this family (Accuracy=0.8415, AUC=0.8927).
• RNN_V2: Introduced skill embeddings alongside new features (day, timeTaken, is_weekend). Elevating dropout to 0.4 in V2 improved Recall, but the most substantial gain was achieved by expanding the sequence length to 140 in V3, jumping Accuracy from 0.8390 to 0.8728.
• RNN_V3: Added the log1p transformation on timeTaken to mitigate the impact of massive outlier response times. Reducing dropout back to 0.2 in V2 stabilized the model, allowing a gradual increase of MAX_SEQ_LEN (from 150 up to 500) to continuously push Accuracy and AUC upwards. The absolute best result of the entire project was recorded in RNN_V3 V7 with Accuracy=0.9649 and AUC=0.9799.
• LSTM_V1: Transitioned to an LSTM architecture while maintaining the RNN_V3 feature set. Reducing layers to 1 and boosting the hidden dimension provided minor improvements, but similar to the RNN architectures, the primary performance driver was the sequence length. LSTM_V1 V7 peaked at Accuracy=0.9561 and AUC=0.9745, falling slightly short of the optimal RNN configuration.
• Overall Conclusion: The most impactful hyperparameter in these experiments was undoubtedly MAX_SEQ_LEN. While skill embeddings, temporal features, and log1p(timeTaken) normalization were critical components for improvement, the highest-performing combination ultimately belonged to RNN_V3.

---

Installation & Execution

Setting Up the Environment

Linux/macOS:

python -m venv tf-env
source tf-env/bin/activate
pip install -r requirements.txt

Windows:

python -m venv tf-env
tf-env\Scripts\activate
pip install -r requirements.txt

Running the Models

Always execute the scripts from the project root directory, as relative paths to the dataset are defined as ./Data/assistments2017.csv.

python Src/Archive/Task3_RNN_V1.py
python Src/Archive/Task3_RNN_V2.py
python Src/Task3_RNN_V3.py
python Src/Task3_LSTM_V1.py

Upon execution, the terminal will prompt for hyperparameters. Pressing Enter will apply the default values:

• Sequence Length: 100
• Hidden Dimension: 64
• RNN/LSTM Layers: 1
• Dropout: 0.2
• Skill Embedding Dim: 16 (Only for versions utilizing skills)
• Batch Size: 32
• Learning Rate: 0.001
• Epochs: 10
• Version Number: 1

---

Outputs

After a successful training loop, the following assets are generated dynamically:

• Models/<version>/V<n>.json: Detailed hyperparameters and evaluation metrics.
• Models/<version>/V<n>_best_rnn_model.pth: Optimal weights saved based on validation loss.
• Models/<version>/V<n>_final_rnn_model.pth: Final model weights after the last epoch.
• Plots/<version>/V<n>_confusion_matrix.png: Confusion Matrix visualization.
• Plots/<version>/V<n>_loss_curve.png: Training vs. Validation Loss plot.
• Plots/<version>/V<n>_validation_metrics.png: Validation Accuracy and AUC trajectory.

---

Pipeline Stages

• preprocessing.py: Handles CSV parsing, temporal feature extraction, skill_id construction, sorting, and scaling.
• features.py: Acts as a registry for the feature sets of each version to prevent accidental logic overrides.
• sequences.py: Aggregates individual student interactions and applies padding up to MAX_SEQ_LEN.
• datasets.py: Converts numpy arrays to PyTorch Tensors and orchestrates Train/Validation/Test splits with a fixed seed (42).
• models.py: Houses the core PyTorch class definitions for KT_RNN_V1, KT_RNN_WithSkills, and KT_LSTM_WithSkills.
• training.py: Manages the training loop, validation phases, and Early Stopping mechanics (Patience = 3).
• evaluation.py: Computes Accuracy, Precision, Recall, and AUC, and exports visualizations/logs.

---

Important Notes

• Ensure Data/assistments2017.csv is correctly placed before initiating training.
• Target directories (e.g., Models/RNN_V3/ or Plots/RNN_V3/) must exist prior to saving outputs (the scripts generally handle directory creation, but verify permissions).
• Do not execute scripts from inside the Src/ folder. Relative paths are anchored to the root directory.
• Based on rigorous empirical testing, RNN_V3 represents the most robust and accurate architecture in this repository.