Select Page

Research

Gesture-Controlled Robot Arm

Tags

Robotics, Physical Computing, Interactive, Programming, C++, TouchDesigner, OpenFrameworks, Leap Motion, GitHub

Control System – Touchdesigner Network

The Challenge

To create an intuitive, real-time control system for a 4-DOF robotic arm that could mimic the nuanced movements of a human hand. This required building a bridge between multiple hardware and software components, translating human gestures into precise servo commands, and effectively managing noisy sensor data.

Leap Motion ‘Pick and Place’ Demo

My Solution & Approach

I built a complete system to control a Robotis robotic arm, built with Dynamixel AX-12A smart servos. A key discovery during development was the importance of where to handle the complex data processing. I found that using TouchDesigner for all high-level control logic was far more efficient than writing complex math in C++. This allowed me to filter, smooth, and map noisy Leap Motion data visually, and to create keyframe animations, before sending clean, simple commands to a dedicated OpenFrameworks application that handled the final, low-level communication with the robotic arm.

Keyframe Animation ‘Pick and Place’ Demo

Touchdesigner Network – Animation Timeline of Servo Movements

Technical Highlights & Learnings

  1. Versatile Control System: Designed a flexible network in TouchDesigner that can seamlessly switch between three different input methods:

    • Real-time Gesture Control: Using a Leap Motion sensor for direct, intuitive manipulation.
    • Pre-Programmed Automation: Using TouchDesigner’s Animation COMP for precise, repeatable keyframe sequences.
    • Manual Slider Control: A simple UI for direct control over each individual servo.
  2. Efficient Data Pre-processing: Developed a TouchDesigner network to filter, smooth, and normalize noisy Leap Motion data, ensuring only clean control signals were sent to the robotics application.
  3. Dual-Software Pipeline: Built a robust system using two specialized applications: TouchDesigner for high-level user input and animation, and OpenFrameworks for low-level C++ servo communication.
  4. Smart Servo Integration: Controlled Dynamixel AX-12A smart servos via a U2D2 adapter, managing precise movements and feedback.

Project Files

GitHub Repository: The complete TouchDesigner, OpenFrameworks, and C++ project files are available on GitHub.

Robot Arm Construction

Tools & Technologies Used

Hardware: Leap Motion Controller, U2D2 Adapter, Dynamixel AX-12A Servos, Robotis Bioloid Frames
Software TouchDesigner, OpenFrameworks, Visual Studio
Programming C++