Reproduce: Susman & Kimmelman 2024 — Eye Blink Detection in Sign Language

[AmitMY]

Paper

Title: Eye Blink Detection in Sign Language Data Using CNNs and Rule-Based Methods
Authors: Margaux Susman, Vadim Kimmelman
Venue: SignLang 2024 (LREC-COLING 2024 workshop)
ACL Anthology: https://aclanthology.org/2024.signlang-1.40/

Summary

Trained a LeNet-5-inspired CNN classifier of eye openness (open / in-between / closed) on French Sign Language data and combined it with rule-based temporal aggregation to detect linguistically defined blinks (rapid, possibly incomplete closure). 3 conv layers for 64×128 eye crops; 4 conv layers for 256×256 face crops; softmax over 3 openness classes. Trained with PyTorch + Adam + cross-entropy + TrivialAugment, on 1/2/3/4 signers for 100/200 epochs. Two openness detectors (CNN vs MediaPipe-EAR) feed two rules (R1 high-low value diff, R2 U-curve polynomial) over 5-frame windows. Evaluated on Dicta-Sign-LSF-v2 (5 signers, 1565 annotated blinks).

Results / Conclusions

• Best: 4-signer eyes-crop CNN + R1 → F1 0.75–0.97 across signers.
• CNN slightly outperforms EAR; R1 always beats R2.

Code & Data

• Code/data: https://osf.io/dth2q/

Suggested Reproduction

1. Pull the repo from OSF.
2. Reproduce CNN training across 1–4 signers and 100/200 epochs.
3. Run R1 and R2 over CNN and EAR outputs; reproduce per-signer F1.

Bibtex

@inproceedings{susman-kimmelman-2024-eye,
    title = "Eye Blink Detection in Sign Language Data Using {CNN}s and Rule-Based Methods",
    author = "Susman, Margaux  and
      Kimmelman, Vadim",
    editor = "Efthimiou, Eleni  and
      Fotinea, Stavroula-Evita  and
      Hanke, Thomas  and
      Hochgesang, Julie A.  and
      Mesch, Johanna  and
      Schulder, Marc",
    booktitle = "Proceedings of the LREC-COLING 2024 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.signlang-1.40/",
    pages = "361--369"
}