Name: LUIZ CARLOS COSMI FILHO
Publication date: 24/06/2026
Examining board:
Name |
Role |
|---|---|
| BRUNO LEGORA SOUZA DA SILVA | Examinador Interno |
| MARIANA RAMPINELLI FERNANDES | Examinador Externo |
| RAQUEL FRIZERA VASSALLO | Presidente |
Summary: Action and gesture recognition is essential for Human-Machine Interaction (HMI) and Human-Robot Interaction (HRI) in Intelligent Spaces, which are physical environments equipped with sensors capable of collecting data, analyzing it, and responding to users’ commands. Gestures can be characterized as short and intentional movements involving a limited set of body parts. In contrast, actions involve coordinated movements of multiple body parts over longer temporal intervals. Among the different data modalities that can be used to understand human motion, skeleton-based representations have gained significant attention in recent years, in which a skeleton sequence models the human body as a set of joints (points), capturing both the spatial configuration of the body and its temporal evolution. In this context, this work investigates action and gesture recognition from 3D skeleton sequences within an Intelligent Space. To this end, a gesture recognition dataset focused on Human-Robot Interaction scenarios was developed, named HGI-IS. Data acquisition was carried out in an environment equipped with multiple calibrated RGB cameras, enabling synchronized multi-camera capture as well as 3D human skeleton reconstruction and tracking. Initially, action and gesture recognition on 3D skeleton sequences was performed using the Swin Transformer model adapted for this data modality. To address the limited availability of labeled data, a self-supervised learning strategy inspired by masked modeling was employed during pre-training to improve performance after fine-tuning. The proposed methodology was evaluated on the action recognition task using the NTU RGB+D and NTU RGB+D 120 datasets, achieving competitive results compared with the state of the art, as well as on the gesture recognition task using the HGI-IS dataset. Next, an approach was proposed to enable action and gesture recognition models trained using complete temporal sequences as input to operate in
real-time. Through a sliding window-based strategy, the proposed methodology supports immediate recognition and integration with robotic control systems. The approach is evaluated on temporal localization and gesture recognition tasks using the HGI-IS dataset, achieving consistent results. Overall, these contributions reinforce the effectiveness of skeleton-based representations for action and gesture recognition in Intelligent Spaces.
