PanoramicWEB Emotion

FACS / Action Unit based facial-expression realization analysis.

PanoramicWEB Emotion is reproducible research software that measures how fully a posed facial expression is actually produced from short video clips. For each participant and each target emotion it extracts frame-level facial Action Units (FACS AUs), computes a single 0–100 realization score, and compares two participant groups with non-parametric statistics. Facial expressions are quantified with two independent engines (Py-Feat and OpenFace 2.0) so results can be cross-validated rather than depending on a single tool.

This is research software (not a commercial product), released under the MIT License. It builds on third-party open-source tools that remain under their own licenses — see NOTICE. Please read the Disclaimer.

---

Table of contents

• Overview
• Key features
• How it works — the pipeline
• Emotion → AU maps
• Realization score methodology
• Two-engine cross-validation
• Statistical analysis
• Installation
• Configuration
• Usage
• Outputs
• Project structure
• Third-party components and licenses
• Disclaimer
• License
• Citation

---

Overview

Conventional emotion-recognition tools answer "is this face happy?" with a probability. PanoramicWEB Emotion targets a different question: "To what extent did the participant manage to produce the intended expression?" It is designed for clinical / behavioral research that cares about the degree of an expression rather than its mere presence — for example, comparing the facial-expression production ability of two participant groups.

The software extracts facial Action Units (FACS AUs) from short video clips, converts the intensity of the emotion-specific AU set into a single 0–100 realization percentage, accounts for a person-specific baseline and temporal dynamics, and statistically tests the difference between the two groups. Every step is reproducible and protected by automatic integrity checks.

Key features

• Realization scoring (0–100): measures how completely an expression is produced.
• Two independent AU engines: Py-Feat (Detectorv2) and OpenFace 2.0 (true 0–5 FACS intensities) run through the same pipeline for cross-validation.
• Person-specific baseline (baseline-delta): each participant is normalized to their own neutral level, reducing face/lighting differences.
• Specificity measure: the gap between target AUs and non-target AUs, assessing whether the expression was produced distinctively.
• Temporal analysis: apex, onset, rise slope, offset, and area under the curve (AUC).
• Inverse scoring for neutral: low overall AU activation = better neutrality (score = 100 − activation).
• Robust face detection: YuNet; when several candidates exist the highest-confidence face is selected (eliminating torso/clothing false positives).
• Automated reporting: summary tables, figures, a per-participant apex face frame, and a comprehensive PDF.
• Integrity checks: protection against silent data loss, online-only (0-byte) file detection, and the dogrula.py validator.
• Streamlit interface: run without writing code.

How it works — the pipeline

1. Scan & manifest: the video folder is scanned; participant, group, and target emotion are parsed from the file name into manifest.csv.
2. AU extraction: each video is processed frame by frame; AU intensities are extracted from frames where a face is detected (frames without a face are excluded). Frame-level AU tables are cached.
3. Feature generation: per-video mean, apex, and temporal features are computed.
4. Scoring: the target emotion's AUs are converted into a 0–100 realization score (inverse scoring for neutral).
5. Summary & statistics: groups (e.g., patient and control) are compared per emotion.
6. Reporting: figures, summary CSVs, and a comprehensive PDF are produced; apex face frames are attached.

Each video is wrapped in try/except and progress is logged; unprocessable videos are reported in a separate list (no silent loss).

Emotion → AU maps

The mappings below are defaults defined in src/duygu_analiz.py; you can edit them for your own study.

Emotion	Target Action Units (AU)
Happiness	AU6, AU12
Sadness	AU1, AU4, AU15
Fear	AU1, AU2, AU4, AU5, AU20, AU25, AU26
Anger	AU4, AU5, AU7, AU23, AU24
Disgust	AU9, AU10, AU15, AU16
Surprise	AU1, AU2, AU5, AU25, AU26
Neutral	Overall mean of all AUs (inverse-scored)

Realization score methodology

An emotion's realization score is derived from the measured intensities of that emotion's target AUs across the video and combines the following components:

• Apex intensity: the target-AU level at the most pronounced moment of the expression.
• Person-specific baseline-delta: the apex is corrected against the baseline activation in the participant's neutral video, so the score answers "how much was the expression produced relative to baseline" rather than "how mobile is the face".
• Specificity: the difference between the mean target-AU and mean non-target-AU activation, measuring that the expression is a correct muscle activation rather than random movement.
• Scaling: raw intensity is scaled to 0–100. OpenFace 0–5 FACS intensities are normalized so they can use the same scoring modules as Py-Feat.
• Neutral (inverse): for neutral, score = 100 − overall AU activation.

For populations whose muscle movements may be very limited, no threshold is applied by design; all values are included in the analysis.

Two-engine cross-validation

The same videos are passed through two independent engines, changing only the AU source:

• Py-Feat (Detectorv2): probability-based AU outputs.
• OpenFace 2.0 (FeatureExtraction): true FACS AU intensities (0–5).

The two engines' scores are compared at the correlation and group-comparison level via karsilastir_motorlar.py, demonstrating that findings are not specific to a single tool.

Statistical analysis

• Mann-Whitney U — non-parametric group comparison (suitable for small/unbalanced samples).
• Cliff's delta — distribution-free effect size.
• Benjamini-Hochberg FDR — false-discovery-rate correction for multiple comparisons.

Installation

Python 3.10+ recommended.

python -m venv .venv
# Windows:
.venv\Scripts\activate
# macOS/Linux:
source .venv/bin/activate

pip install -r requirements.txt

The AU engines are installed separately:

• Py-Feat: pip install py-feat (downloads model weights on first run).
• OpenFace 2.0: installed from the official distribution (Windows binary); its path is given in the configuration. OpenFace is for non-commercial academic use only.

The YuNet face-detector ONNX model (face_detection_yunet_2023mar.onnx) must be present in the project root (downloadable from OpenCV Zoo).

Configuration

Person/data-specific paths are not included in the repository. You can provide them in two ways:

1. src/konfig_yerel.py (ignored by git) — example:

   VERI_KOK = r"D:\data\emotion_expressions"     # group 1 root folder
   KONTROL_KOK = r"D:\data\control_videos"        # group 2 root folder
   OPENFACE_DIR = r"C:\OpenFace_2.2.0_win_x64"    # OpenFace install folder
   VIDEO_KLASORU = r"D:\data\single_folder"       # UI default

2. Environment variables: EMO_VERI_KOK, EMO_KONTROL_KOK, EMO_OPENFACE_DIR.

If none are provided, the fields are left empty and a path is entered from the UI/CLI.

Usage

Streamlit interface (recommended):

streamlit run src/app.py

End-to-end pipeline (CLI):

python src/calistir_full.py

Two-engine comparison and comprehensive PDF:

python src/openface_runner.py        # second pass with OpenFace
python src/karsilastir_motorlar.py   # Py-Feat vs OpenFace
python src/rapor_pdf_kapsamli.py     # comprehensive PDF report
python src/dogrula.py                # integrity validation

Outputs

All outputs are collected under ANALIZ_CIKTILARI/ in the project root:

ANALIZ_CIKTILARI/
├── 00_VERI/          manifest.csv (participant, group, emotion, source)
├── 01_AU_HAM/        frame-level AU tables (cache)
├── 02_OZELLIKLER/    per-video features
├── 03_SKORLAR/       realization scores (detailed)
├── 04_OZET/          per-emotion/group summary and pivot tables
├── 05_ISTATISTIK/    Mann-Whitney / Cliff / FDR results
├── 06_RAPOR/         figures + comprehensive PDF
├── 07_PDF_RAPORLAR/  per-participant PDF
└── 08_KARELER/       apex face frames

Outputs, video frames, and participant-identifying files are not included in the repository (see .gitignore); they are produced locally only.

Project structure

.
├── src/
│   ├── app.py                    # Streamlit interface
│   ├── calistir_full.py          # end-to-end pipeline
│   ├── duygu_analiz.py           # AU extraction + emotion→AU maps
│   ├── ozellik_cikar.py          # feature generation (apex, temporal)
│   ├── skorlama.py               # 0–100 realization scoring
│   ├── istatistik.py             # Mann-Whitney, Cliff, BH-FDR
│   ├── openface_runner.py        # OpenFace 2.0 second engine
│   ├── karsilastir_motorlar.py   # Py-Feat vs OpenFace cross-validation
│   ├── rapor_olustur.py          # summary/figure report
│   ├── rapor_pdf_kapsamli.py     # comprehensive PDF
│   ├── raporla_karsilastirma.py  # engine-comparison report
│   ├── guvenli_yol.py            # readable-path helper
│   ├── dogrula.py                # integrity validator
│   └── oynatici.py               # video/frame reviewer
├── requirements.txt
├── LICENSE
├── NOTICE
├── CITATION.cff
└── README.md

Third-party components and licenses

This software builds on top of the following open-source components; none is bundled in this repository and each remains under its own license. The MIT license of this project covers only our original code and does not extend to these components. For per-component copyright, license identifiers, and reference license texts, see THIRD_PARTY_LICENSES.md (summary in NOTICE).

Component	Role	License
Py-Feat	AU / expression engine	MIT
OpenFace 2.0	AU engine (0–5 FACS)	Academic / non-commercial (CMU)
OpenCV (+ YuNet)	image processing / face detection	Apache-2.0
NumPy, pandas, SciPy	numeric / statistics	BSD-3
Matplotlib	visualization	Matplotlib (BSD-style)
Streamlit	interface	Apache-2.0
PyTorch	engine dependency	BSD-3
ONNX Runtime	ONNX model inference	MIT
statsmodels	optional mixed-model	BSD-3
EmotiEffLib / HSEmotion	optional emotion engine (not used by default pipeline)	Apache-2.0

Important: OpenFace 2.0 is restricted to non-commercial academic/research use. Any use relying on the OpenFace engine must remain within those terms. The Py-Feat engine (MIT) is not subject to this restriction.

Disclaimer

This is academic research software, provided "AS IS" without warranty of any kind (see LICENSE). It is not a medical device and is not intended for clinical diagnosis, treatment, or any decision affecting patient care. Facial Action Unit estimates are produced by third-party models and can be affected by video quality, lighting, head pose, and occlusion; outputs must be interpreted by qualified researchers and validated for the intended use. The authors and Aksan Bilişim accept no liability for any use of this software or any reliance on its results. Use of third-party engines (e.g., OpenFace) is subject to their own licenses and restrictions, which are the user's responsibility.

License

MIT License — see LICENSE. © 2026 Aksan Bilişim (Ulaş Aksan, Hasan Çağdaş Aksan).

The MIT license applies only to the original code authored by Aksan Bilişim (the files under src/). Third-party dependencies are not covered by it and are not bundled here; each is governed by its own license — see THIRD_PARTY_LICENSES.md. In particular, use of the OpenFace engine is restricted to non-commercial academic/research purposes.

Citation

If you use this software or refer to it in a study:

Aksan, Ulaş, & Aksan, Hasan Çağdaş. (2026). PanoramicWEB Emotion: FACS/AU-based facial expression realization analysis software [Computer software]. Aksan Bilişim. https://github.com/panoramicwebvr-cyber/panoramicweb-emotion

The repository includes a CITATION.cff file for GitHub's "Cite this repository" button.