Embedded gesture-recognition firmware built around an Infineon PSoC 6 AI Evaluation Kit and XENSIV BGT60TR13C 60 GHz radar. This work was developed as a gesture-input subsystem for Project Infineon-X, a larger team-built assistive facial-recognition platform.
The firmware performs radar-based gesture classification on the PSoC 6 and exposes detected events to external hardware through GPIO and UART, allowing the radar board to act as a self-contained gesture trigger rather than only displaying inference results locally. I also extended the sensing behavior by collecting radar data and training a custom swipe gesture in DEEPCRAFT Studio, then deploying the resulting model to the PSoC.
This repository is based on Infineon's
mtb-example-ml-deepcraft-deploy-radarexample. The upstream project provides the radar acquisition and DEEPCRAFT inference framework; this repository contains my modifications for system integration and custom gesture recognition.
This radar subsystem was one component of Project Infineon-X, an end-to-end facial-recognition and control platform developed by a six-person team.
The larger project includes:
- a facial-recognition backend and model-training workflow,
- a Flask API for recognition and health checks,
- single-board-computer client software for image capture and continuous operation,
- a web/dashboard frontend, and
- embedded hardware interfaces, including this radar-based gesture input subsystem.
My radar work was intended to give the larger system a simple, hands-free physical input. Rather than sending raw radar measurements into the main application stack, the PSoC performs acquisition and gesture inference locally and emits compact events that other hardware or software can consume.
This separation keeps radar-specific processing isolated from the rest of the system while allowing gesture behavior to be changed through firmware and ML-model updates.
The goal of the radar subsystem was to turn an existing radar gesture-classification demo into a practical system input for Project Infineon-X.
The Infineon example already demonstrates how to:
- acquire data from the XENSIV 60 GHz radar sensor,
- feed radar frames into a DEEPCRAFT-generated machine-learning model,
- classify gestures such as push and circle, and
- report inference results over the debug interface.
For the larger system, simply printing classification results was not enough. A detected gesture needed to become a usable hardware/software event that other system components could react to.
The firmware was therefore modified to provide external signaling through both a digital GPIO output and a UART data stream. I also extended the reference gesture set by collecting training data for a swipe gesture, training a new DEEPCRAFT model, and deploying that model to the embedded target.
My work on this repository focused on adapting and extending the reference implementation for system-level integration.
- Added GPIO signaling in response to gesture detections.
- Added binary UART output so detected gesture events can be consumed by another processor or controller.
- Configured the required PSoC GPIO resources for the external interface.
- Integrated the new signaling behavior into the existing radar inference loop.
- Collected and labeled radar data for a custom swipe gesture.
- Trained and exported a new DEEPCRAFT gesture-classification model that included the swipe gesture.
- Deployed the custom model to the PSoC and tested swipe recognition on the physical radar hardware.
- Integrated the radar-based trigger into the broader Project Infineon-X system concept.
The underlying radar acquisition and DEEPCRAFT inference framework originated from the Infineon reference project linked above.
flowchart LR
A[XENSIV BGT60TR13C\n60 GHz Radar] -->|SPI| B[PSoC 6]
B --> C[Radar frame acquisition]
C --> D[DEEPCRAFT ML inference]
D --> E[Gesture classification]
E --> F[GPIO trigger]
E --> G[UART event output]
F --> H[Project Infineon-X]
G --> H
At a high level, the radar sensor supplies frames to the PSoC 6 over SPI. The embedded inference pipeline evaluates the incoming data and assigns confidence scores to supported gestures. When the application identifies a relevant gesture, the modified firmware converts the classification into an external GPIO/UART event.
This keeps the time-sensitive sensing and ML inference local to the PSoC while giving the rest of Project Infineon-X a much simpler interface.
- Infineon CY8CKIT-062S2-AI PSoC 6 AI Evaluation Kit
- Infineon XENSIV BGT60TR13C 60 GHz radar sensor
- External controller/system receiving GPIO and UART events
- USB connection for programming, debugging, and serial output
- C
- ModusToolbox 3.4+
- DEEPCRAFT Studio / DEEPCRAFT-generated model
- GNU Arm Embedded toolchain
- Git / GitHub
The reference application targets the CY8CKIT-062S2-AI board and uses the GNU Arm Embedded compiler by default.
The reference firmware captures radar data at approximately 200 Hz. Once enough frames have been collected, the data is passed to the embedded inference engine. The model then produces confidence scores for the supported gesture classes.
Conceptually, the pipeline is:
Radar frames
↓
Frame buffering / preprocessing
↓
DEEPCRAFT model inference
↓
Gesture confidence scores
↓
Gesture decision
↓
GPIO + UART event
↓
Project Infineon-X
The model files in models/ can be replaced with compatible models generated by DEEPCRAFT Studio, although changes to the data-feeding pipeline may be required for models with different frame rates, input shapes, or memory requirements.
The stock Infineon example demonstrates gestures such as push and circle. For Project Infineon-X, I wanted an interaction that better matched the intended control behavior, so I trained the system to recognize a swipe as an additional gesture class.
The training workflow was:
- Collect radar data — I recorded examples of the swipe motion using the same radar hardware and sensing configuration used by the embedded target.
- Label the recordings — The captured radar sequences were labeled as the new swipe class so the model could learn the motion pattern associated with that gesture.
- Include contrasting examples — Other gesture and non-gesture data were kept in the training set so the classifier learned to distinguish a swipe from unrelated motion rather than simply detecting movement.
- Train the model in DEEPCRAFT Studio — The expanded dataset was used to train a new gesture-classification model containing the additional swipe class.
- Export the embedded model — After training, the DEEPCRAFT model was exported in a form suitable for deployment to the PSoC application.
- Replace the model in firmware — The generated model files were integrated into the existing ModusToolbox project so the same radar acquisition and inference pipeline could run the new classifier.
- Flash and test on hardware — The updated firmware was programmed onto the PSoC 6 and tested with real swipe gestures in front of the radar.
- Iterate on the dataset/model — Recognition behavior was evaluated on the physical system, with additional training examples or model revisions used as needed to improve the distinction between gestures.
This demonstrated that the radar subsystem was not limited to the gesture classes supplied with the vendor demo. Its sensing behavior could be adapted to the needs of the larger project by collecting application-specific data and retraining the classifier.
A radar classifier does not recognize a new gesture simply because the firmware assigns it a new name. The motion has to be represented in the model's training data so the classifier can learn the radar signature associated with it.
A swipe changes the radar return over time as the hand moves laterally across the sensor's field of view. By training on labeled examples of that motion, the model can learn temporal and spatial characteristics that distinguish it from the existing gesture classes and from background movement.
Perform swipe gestures
↓
Capture radar samples
↓
Label swipe / other motion
↓
Train in DEEPCRAFT Studio
↓
Export embedded model
↓
Integrate generated model files
↓
Build + flash PSoC 6
↓
Test real-time swipe recognition
↓
GPIO / UART event
↓
Project Infineon-X
Keeping the acquisition firmware and hardware configuration consistent between data collection and deployment helps ensure that the data presented to the embedded model resembles the data it saw during training.
A GPIO output was added so gesture detection can produce an immediate hardware-level signal. This is useful when the downstream device does not need the full classifier output and only needs to know that a valid trigger occurred.
Advantages of the GPIO path include:
- very low interface complexity,
- minimal software overhead on the receiving controller,
- deterministic hardware signaling, and
- easy observation with a logic analyzer or oscilloscope during debugging.
UART output was also added to carry gesture-event information digitally. Unlike the GPIO path, UART can communicate more than a simple trigger state and therefore provides a path for distinguishing event types or adding additional metadata later.
Using both interfaces provides flexibility: GPIO can handle a simple fast trigger while UART can provide a richer software-facing event channel.
The most relevant project files are:
PSOC-Trigger1/
├── main.c Main application and gesture-event integration
├── radar_settings.h Radar configuration
├── models/ Embedded DEEPCRAFT model files
├── images/ Reference documentation media
├── deps/ ModusToolbox dependencies
├── Makefile ModusToolbox build configuration
├── project_info.json Project metadata
└── README.md Project documentation
Some of the repository structure and support files are inherited directly from the Infineon ModusToolbox example.
- ModusToolbox 3.4 or later
- CY8CKIT-062S2-AI board support package
- GNU Arm Embedded toolchain
- PSoC 6 AI Evaluation Kit connected through KitProg3
After ModusToolbox and the required board support packages are installed, the application can be built and programmed from the ModusToolbox IDE or command line.
make program TOOLCHAIN=GCC_ARMThe upstream Infineon project contains complete instructions for creating the ModusToolbox workspace, configuring supported IDEs, programming the board, and debugging the reference application:
Infineon DEEPCRAFT Radar Deployment Example →
During development, the system can be observed at several points:
- Radar/inference output — verify that the expected gesture class, including the custom swipe class, is being detected.
- UART output — confirm that the intended event data is transmitted after detection.
- GPIO output — observe the output pin using a logic analyzer, oscilloscope, or receiving controller.
- Receiving system — verify that the downstream Project Infineon-X hardware/software responds correctly to the event.
Breaking the pipeline into these stages makes it easier to distinguish a model-classification problem from an interface or downstream-system problem.
For custom gestures, testing also feeds back into model development. False positives, missed swipes, or confusion with another gesture indicate that the training dataset may need more examples or greater variation.
Keeping radar processing and classification on the sensor-side microcontroller means the rest of Project Infineon-X does not need to process raw radar data. Instead, it sees a small, well-defined event interface.
This reduces:
- inter-processor bandwidth,
- processing requirements on the host,
- coupling between the radar implementation and the rest of the system, and
- the amount of radar-specific code required elsewhere.
The interfaces serve different purposes.
| Interface | Best suited for |
|---|---|
| GPIO | Fast, simple trigger/event signaling |
| UART | Gesture identity, debugging, and extensible event data |
Using both also makes system bring-up easier because the simple hardware trigger can be verified independently of the higher-level serial protocol.
Using a custom swipe class let the gesture interface be designed around the intended user interaction rather than forcing the rest of the system to adapt to the stock demonstration gestures.
It also provided an end-to-end embedded-ML workflow: data collection → labeling → training → model export → firmware integration → hardware testing. That made it possible to change the sensing behavior through both firmware and model development instead of treating the machine-learning component as a fixed black box.
This repository is currently a project integration/prototype rather than a production-ready gesture-recognition subsystem. Useful next steps would include:
- defining a documented UART packet format,
- adding explicit timing/debounce behavior for repeated detections,
- recording per-gesture detection accuracy and false-trigger rates,
- measuring end-to-end trigger latency,
- expanding the custom training dataset across different users, gesture speeds, distances, and orientations,
- building a repeatable validation dataset for comparing model revisions,
- adding automated or repeatable interface tests,
- documenting the exact external GPIO pinout and electrical interface, and
- documenting the complete path from a detected gesture through the downstream Project Infineon-X behavior.
These measurements would make it easier to compare firmware and model revisions quantitatively rather than evaluating them only through manual testing.
- Project Infineon-X — full team system
- Infineon-X GitHub organization
- Infineon DEEPCRAFT Radar Deployment Example
This work is derived from Infineon's DEEPCRAFT Studio Deploy Model: Radar code example:
The reference implementation provides the PSoC 6 radar acquisition, inference framework, example model, and associated support code. The modifications in this repository focus on adapting that example to operate as a gesture-trigger interface within Project Infineon-X and extending the classifier with an application-specific swipe gesture.
See the repository's LICENSE file for licensing information.