Research
Gesture-Controlled Robot Arm
Robotics, Physical Computing, Interactive, Programming, C++, TouchDesigner, OpenFrameworks, Leap Motion, GitHub
Control System – Touchdesigner Network
The Challenge
The challenge was to develop an intuitive, real-time control system for a 4-DOF robot arm capable of mimicking the movements of a human hand. This process involved integrating multiple hardware and software components, accurately translating human gestures into servo commands, and addressing the challenge of managing noisy sensor data.
Leap Motion ‘Pick and Place’ Demo
My Solution & Approach
A complete system to control a Robotis robot arm built with Dynamixel AX-12A smart servos was constructed. A critical finding during development was determining where to execute complex data processing. Using TouchDesigner for all high-level control logic proved significantly more efficient than writing complex math in C++. This allowed me to filter, smooth, and map noisy Leap Motion data visually, as well as the design of keyframe animations, prior to issuing simple commands to an OpenFrameworks application responsible for final low-level communication with the robotic arm.
Keyframe Animation ‘Pick and Place’ Demo
Touchdesigner Network – Animation Timeline of Servo Movements
Technical Highlights & Learnings
-
Versatile Control System: Designed a flexible network in TouchDesigner that can seamlessly switch between three different input methods:
- Real-time Gesture Control: Using a Leap Motion sensor for direct, intuitive manipulation.
- Pre-Programmed Automation: Using TouchDesigner’s Animation COMP for precise, repeatable keyframe sequences.
- Manual Slider Control: A simple UI for direct control over each individual servo.
- Efficient Data Pre-processing: Developed a TouchDesigner network to filter, smooth, and normalize noisy Leap Motion data, ensuring only clean control signals were sent to the robotics application.
- Dual-Software Pipeline: Built a robust system using two specialized applications: TouchDesigner for high-level user input and animation, and OpenFrameworks for low-level C++ servo communication.
- Smart Servo Integration: Controlled Dynamixel AX-12A smart servos via a U2D2 adapter, managing precise movements and feedback.
Project Files
GitHub Repository: The complete TouchDesigner, OpenFrameworks, and C++ project files are available on GitHub.
Robot Arm Construction
Tools & Technologies Used
Hardware: Leap Motion Controller, U2D2 Adapter, Dynamixel AX-12A Servos, Robotis Bioloid Frames
Software: TouchDesigner, OpenFrameworks, Visual Studio
Programming: C++