Depthtective: A Depth-Aware Framework for Spatio-Temporal Deepfake Detection

Carmen Bisogni · Luigina Costante · Vincenzo Loia · Michele Nappi · Chiara Pero

Array, vol. 32, 101278, 2026

Official PyTorch implementation of Depthtective, a data-efficient deepfake detector that exposes identity and expression manipulations using only two adjacent frames. Absolute residuals between consecutive aligned RGB frames and their estimated depth maps are fused into a compact four-channel RGBD tensor that is processed by a standard CNN backbone, avoiding long temporal sequences, optical flow and multi-stream architectures.

Table of contents

• Method
• Results
• Repository structure
• Installation
• Data preparation
• Training
• Evaluation
• Inference on a video
• Explainability and analysis
• Hyper-parameter search
• Citation
• License
• Acknowledgments

Method

1. Preprocessing. A pair of consecutive frames is sampled from the video. The face is localised with MediaPipe Face Detection and cropped, and a relative depth map is estimated for each crop with MiDaS v3.0 DPT-Large.

2. Temporal alignment and residual construction. Frame t and its depth map are aligned to frame t+1 through a partial affine transform estimated on the MediaPipe FaceMesh contour landmarks. The absolute residuals are then computed:

   $$ R_{\text{RGB}} = \lvert F(t+1) - F(t)_{\text{aligned}} \rvert, \qquad R_{\text{Depth}} = \lvert D(t+1) - D(t)_{\text{aligned}} \rvert $$

   and concatenated into a four-channel RGBD residual.

3. Classification. The residual is classified by an ImageNet-pretrained Xception or ResNet50 whose first convolution is extended to four channels: the RGB kernels are preserved and the depth kernel is initialised as their average.

4. Contrastive variant (optional). The four-channel Xception encoder is pre-trained with a triplet loss; it is then frozen and an MLP classifier is trained on its embeddings.

Results

Held-out test results (macro-averaged, %) reported in Table 3 of the paper. CL denotes the contrastive learning variant.

Dataset / manipulation	Acc. Xception	Acc. ResNet50	Acc. CL	AUC Xception	AUC ResNet50	AUC CL
FF++ Deepfakes	96.88	95.31	96.35	99.84	99.07	99.07
FF++ Face2Face	95.81	95.29	92.15	98.39	98.65	97.04
FF++ FaceSwap	96.86	95.81	96.34	99.21	98.96	99.48
FF++ NeuralTextures	86.39	87.43	89.01	92.44	93.33	95.31
FF++ All (5-class)	92.05	91.84	91.37	95.89	95.39	94.61
Celeb-DF (v2)	88.17	87.57	92.31	94.42	93.74	97.26
DFDC (subset)	84.49	83.50	97.03	83.29	80.65	98.35

The classifier requires 20.8–23.5 M parameters and 4.17–4.60 GMACs, with an inference latency of about 8–9 ms per frame pair on an NVIDIA RTX 5000 Ada; monocular depth estimation (MiDaS DPT-Large) is the main preprocessing cost.

Repository structure

Depthtective/
├── depthtective/                 # Python package
│   ├── config.py                 # Default preprocessing parameters
│   ├── preprocessing/            # Frame sampling, face analysis, MiDaS depth, RGBD residuals
│   ├── data/                     # Residual dataset, video-level splits, triplet sampling
│   ├── models/                   # 4-channel Xception / ResNet, contrastive encoder and classifier
│   ├── training/                 # Baseline trainer, contrastive stage 1 and stage 2
│   ├── evaluation/               # Metrics (Acc, F1, AUC) and checkpoint loading
│   ├── explainability/           # Grad-CAM
│   ├── inference.py              # Model building and single-video inference
│   ├── losses.py                 # Triplet loss
│   └── utils.py                  # Logging, seeding, CLI helpers
├── scripts/
│   ├── extract_residuals.py      # Videos -> RGB and depth residuals (.npy)
│   ├── train_baseline.py         # Supervised Xception / ResNet training
│   ├── train_contrastive.py      # Two-stage contrastive training
│   ├── evaluate.py               # Test-set evaluation
│   ├── predict.py                # Single-video prediction and profiling
│   ├── gradcam.py                # Grad-CAM visualisations
│   ├── visualize_embeddings.py   # t-SNE of the contrastive embedding space
│   └── check_residuals.py        # Detects corrupted .npy files
├── configs/sweeps/               # Weights & Biases Bayesian sweeps
├── assets/                       # Figures used in this README
├── requirements.txt
├── environment.yml
└── pyproject.toml

Installation

The code was developed and tested with Python 3.10, PyTorch 2.2.2 and CUDA 12.1.

Conda

git clone https://github.com/Luigina2001/Depthtective.git
cd Depthtective
conda env create -f environment.yml
conda activate depthtective

pip

git clone https://github.com/Luigina2001/Depthtective.git
cd Depthtective
python -m venv .venv && source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install torch==2.2.2 torchvision==0.17.2 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
pip install -e .

The MiDaS model is downloaded automatically from torch.hub on first use.

Data preparation

1. Download the datasets

Request access and download the original datasets from their official sources: FaceForensics++ (C23), Celeb-DF (v2) and the Deepfake Detection Challenge. The datasets cannot be redistributed with this repository.

2. Extract the RGBD residuals

scripts/extract_residuals.py samples one pair of adjacent frames per video, computes the residuals and saves them as .npy files. Run it once per class folder:

# FaceForensics++ (C23)
python scripts/extract_residuals.py --input_dir <FF++>/original_sequences/youtube/c23/videos \
    --output_dir data/residuals/FFPP --class_name Original
for m in Deepfakes Face2Face FaceSwap NeuralTextures; do
  python scripts/extract_residuals.py --input_dir <FF++>/manipulated_sequences/$m/c23/videos \
      --output_dir data/residuals/FFPP --class_name $m
done

# Celeb-DF (v2)
python scripts/extract_residuals.py --input_dir <Celeb-DF>/Celeb-real      --output_dir data/residuals/CelebDF --class_name Real
python scripts/extract_residuals.py --input_dir <Celeb-DF>/YouTube-real    --output_dir data/residuals/CelebDF --class_name Real
python scripts/extract_residuals.py --input_dir <Celeb-DF>/Celeb-synthesis --output_dir data/residuals/CelebDF --class_name Fake \
    --sample 890   # balanced 1:1 real/fake subset

For DFDC, first sort the videos into Real/ and Fake/ folders according to the labels in metadata.json, then run the script on each folder.

The resulting layout is:

data/residuals/FFPP/
├── Original/        RGB/000.npy       Depth/000.npy      ...
├── Deepfakes/       RGB/000_003.npy   Depth/000_003.npy  ...
└── ...

Videos in which no face is detected are skipped and listed in failed_videos.txt. Use python scripts/check_residuals.py data/residuals/FFPP to detect corrupted files.

Note

Splits. Each dataset is split 80/10/10 at the source level with a fixed seed (42): all residuals derived from the same source video share the same partition. The source identifier is the first token of the file name (000_003 → 000, id0_id1_0000 → id0), so keep the original file names of the datasets. The Celeb-DF subset (890 real / 890 fake videos) and the DFDC subset (dfdc_train_part0, dfdc_train_part46, deduplicated) are described in Section 4.1 of the paper.

Hard negatives. During contrastive pre-training, a manipulated residual saved with the DF_ prefix (--name_prefix DF_, e.g. DF_000_003.npy) is paired with the pristine residual of its source (000.npy) to build same-source hard negatives; without the prefix, triplets are sampled at random across classes.

Training

The first entry of --classes (label 0) must always be the pristine class. Classes can be given as a comma-separated list or as a JSON dictionary ('{"Original": 0, "Deepfakes": 1}'); several folders can share the same label.

Supervised baseline (Xception / ResNet50)

# One binary detector per FF++ manipulation
python scripts/train_baseline.py --data_dir data/residuals/FFPP --classes Original,NeuralTextures \
    --model_name xception

# Five-class FF++ model ("Real vs. All" metrics are obtained by aggregating the four manipulation classes)
python scripts/train_baseline.py --data_dir data/residuals/FFPP \
    --classes Original,Deepfakes,Face2Face,FaceSwap,NeuralTextures --model_name xception

# Celeb-DF with ResNet50
python scripts/train_baseline.py --data_dir data/residuals/CelebDF --classes Real,Fake --model_name resnet50

The best checkpoint (lowest validation loss) is saved in checkpoints/baseline/. The input-modality ablation of Table 6 is reproduced with --mode RGB or --mode DEPTH (the depth residual is replicated over three channels to reuse the pretrained stem).

Contrastive variant

python scripts/train_contrastive.py --stage all --data_dir data/residuals/CelebDF --classes Real,Fake \
    --embedding_dim 256 --triplet_margin 0.2 --hidden_features 128

Stage 1 saves best_contrastive_model.pth (encoder) and stage 2 saves best_classifier_model.pth (frozen encoder + MLP head) in checkpoints/contrastive/. Use --stage pretrain or --stage classify --contrastive_model_path <encoder.pth> to run a single stage.

Logging to Weights & Biases is disabled by default and can be enabled with --wandb --wandb_entity <entity>. Run any script with --help for the complete list of options.

Evaluation

The test split is rebuilt deterministically from --seed, --val_size and --test_size, which must match the training run.

# Baseline
python scripts/evaluate.py --data_dir data/residuals/FFPP --classes Original,Deepfakes \
    --model_type baseline --backbone xception --checkpoint checkpoints/baseline/<run>_best.pth

# Contrastive variant
python scripts/evaluate.py --data_dir data/residuals/CelebDF --classes Real,Fake \
    --model_type contrastive --checkpoint checkpoints/contrastive/best_classifier_model.pth \
    --embedding_dim 256 --hidden_features 128

Accuracy, macro F1-score, AUC and the confusion matrix are reported; for multi-class models the aggregated Real vs. All metrics are reported as well. Add --full_dataset to evaluate on an entire dataset (cross-dataset testing).

Inference on a video

python scripts/predict.py --video path/to/video.mp4 \
    --model_type contrastive --checkpoint checkpoints/contrastive/best_classifier_model.pth \
    --embedding_dim 256 --hidden_features 128

==================================================
Video      : video.mp4
Model      : contrastive (xception)
Prediction : Fake
Confidence : 98.45%
==================================================

Add --benchmark to report the number of parameters, the GMACs and the preprocessing/inference latency (Section 5.5).

Explainability and analysis

# Grad-CAM on true/false positives and negatives of the test split ("Fake" is the positive class)
python scripts/gradcam.py --data_dir data/residuals/FFPP --classes Original,NeuralTextures \
    --model_type baseline --checkpoint checkpoints/baseline/<run>_best.pth --output_dir results/gradcam

# t-SNE of the contrastive embedding space
python scripts/visualize_embeddings.py --data_dir data/residuals/CelebDF --classes Real,Fake \
    --encoder_checkpoint checkpoints/contrastive/best_contrastive_model.pth --output results/tsne.png

RGB residual, depth residual and Grad-CAM of a NeuralTextures video correctly detected as fake.

Hyper-parameter search

The hyper-parameters reported in Section 4.3 were selected with Bayesian optimisation through Weights & Biases sweeps. Edit data_dir and classes in the configuration files, then run:

wandb sweep --entity <your-entity> configs/sweeps/baseline.yaml
wandb agent <your-entity>/Depthtective-Baseline/<sweep-id>

configs/sweeps/contrastive_stage1.yaml and contrastive_stage2.yaml implement the two-stage search of the contrastive variant.

Citation

If you find this work useful, please cite:

@article{bisogni2026depthtective,
  title     = {Depthtective: A depth-aware framework for spatio-temporal deepfake detection},
  author    = {Bisogni, Carmen and Costante, Luigina and Loia, Vincenzo and Nappi, Michele and Pero, Chiara},
  journal   = {Array},
  volume    = {32},
  pages     = {101278},
  year      = {2026},
  publisher = {Elsevier},
  doi       = {10.1016/j.array.2026.101278}
}

License

The code is released under the MIT License. The paper is published open access under the CC BY-NC-ND 4.0 license. FaceForensics++, Celeb-DF and DFDC are subject to their own terms of use.

Acknowledgments

This work was partially supported by the SINTESI project ("Sistemi cyber-fisici Intelligenti e sicuri per la sanità, l'industria e la società") under the PON Research and Innovation 2021–2027 programme, funded by the European Union, the Italian Ministry of Enterprises and Made in Italy (MIMIT) and the Italian Ministry of University and Research (MUR).

The depth estimation relies on MiDaS, face analysis on MediaPipe and the Xception backbone on timm.

For questions, please open an issue.