Robotic Arm Handling Based on Real-Time Recognition of the Number of Raised Fingers Using Convolutional Neural Networks

This paper presents a system for operating a robotic arm based on the number of raised fingers detected in a human hand using a camera mounted on the robot. Leveraging advancements in physical human-robot interaction (pHRI), the system utilizes a convolutional neural network (CNN) to interpret hand gestures for intuitive control. Initially, the system uses the MediaPipe Framework to identify 21 landmarks of the hand, which are then used to define the bounding box of the hand. A convolutional neural network (CNN) processes this bounded hand image to determine the number of raised fingers. Implemented using a Jetson Nano, a Logitech Brio 4K webcam, and Python libraries such as OpenCV, I2C tools, and TensorFlow, the model was trained on 30,000 images, achieving a 92.7% accuracy during training and 94% during real-time testing. A voting strategy ensures robust predictions by considering the most frequent result from ten consecutive predictions, mitigating the impact of minor hand movements. The system demonstrates the potential for advanced applications in hand gesture-based robot manipulation and interaction.