Hackathon build · 24 hours · solo
Hand-Controlled Robot Arm
Move your hand in front of a camera and the robot arm moves with you. Make a fist and the gripper closes. No gloves, no controller: just a webcam, hand tracking and an Arduino.
5 servos 21 hand landmarks per frame 30 commands / s 115200 baud 1.5 cm floor clearance
Directions
Sample motion. Start the camera or use your mouse to take control.
Drag to rotate. Double-click to reset the view.
Serial packet<90,95,89,90,30>
Hand tracking runs on your device with Google MediaPipe. The arm connection uses Web Serial, available in Chrome and Edge on desktop.
From your hand to a servo
Every camera frame goes through the same six steps. The Python app and the demo above share the same numbers, so what you see in the browser is how the real arm behaves.
- 1
Camera
A webcam or Raspberry Pi camera at 640×480 (320×240 on a Pi). The image is mirrored so it feels like a reflection.
- 2
Hand landmarks
MediaPipe finds 21 points on one hand: the wrist, each knuckle and each fingertip.
- 3
Measurements
Palm position, palm size (how close you are), knuckle-line tilt, and how far the fingertips reach from the wrist.
- 4
Joint angles
Each measurement maps onto one servo's range, then a moving average smooths out jitter.
- 5
Floor limit
The pose is checked against the arm's real dimensions and raised if any part would touch the floor.
- 6
Arduino
Angles go over USB as
<b,s,e,w,g>. The Arduino eases each servo toward its target at 2° per 15 ms.
| Your hand | Arm joint | How it's measured | Servo range |
|---|---|---|---|
| Move left / right | Base turns | Palm x across the frame | 20°–160° |
| Move up / down | Shoulder lifts | Palm y down the frame | 40°–140° |
| Move closer / farther | Elbow reaches | Palm size, wrist to middle knuckle | 40°–140° |
| Tilt your hand | Wrist rotates | Angle of the knuckle line, ±60° | 30°–150° |
| Make a fist | Gripper closes | Fingertip reach ÷ palm size, below 1.15 to close, above 1.40 to open | 30° / 100° |
If your hand leaves the frame, nothing is sent and the arm holds its last position.
The arm never touches the floor
Before any command is sent, the software works out where the elbow, wrist and gripper tips would be, using the arm's measured lengths. If any of them would come within 1.5 cm of the floor, the pose is raised just enough to clear it.
Stop and slide
The elbow bends up first. If the elbow joint itself would hit the floor, the shoulder lifts and then the elbow is corrected. Each correction is the smallest one that works, so pushing your hand down makes the arm stop at the floor and slide along it instead of jumping.
Tested, not assumed
- 11,025 poses checked: none below the floor, none outside a servo limit.
- About 0.2 ms per check, fast enough for every frame.
- The Arduino also clamps every angle to hard limits, as a second layer.
Try it above: in mouse mode, move to the bottom of the pad and set the distance to “farther”. The badge turns on and the model shows where the arm is closest to the floor.
What it's built from
Hardware
- Pre-built 5-servo robot arm (base, shoulder, elbow, wrist, gripper)
- Arduino, receiving packets over USB serial
- Laptop, or a Raspberry Pi running headless
- Webcam or Raspberry Pi Camera Module
- Separate 5–6 V supply for the servos, sharing ground with the Arduino
Software
- Python, MediaPipe Hand Landmarker, OpenCV
- Tkinter desktop app with a hand-drawn 3D arm view
- pyserial and an Arduino sketch with per-servo limits and easing
- This page: MediaPipe for the web, Canvas 2D, Web Serial
Set up in three steps
# 1. Upload arm_receiver/arm_receiver.ino to the Arduino
pip install mediapipe opencv-python pyserial pillow # 2. install
python3 arm_gui.py # 3. run (add --headless on a Pi)Source code: [GITHUB LINK]