Chironomy Recognition As a Touchless Experience
A worker with both hands on a box still has to confirm, skip, or cancel the station. CRATE reads the hand in the camera and answers on the board itself: an icon on the LED matrix and a sound from the buzzer, with no cloud and no touchscreen.
embedded world North America, Anaheim, September 22–24, 2026. Team 3, Industrial Automation.
The prompt was a warehouse station. The worker walks up holding a box and cannot reach a screen. The device has to notice a hand, decide which of three commands it is, and respond in a few hundred milliseconds. It also had to catch as many gestures as it could inside a 30-second window.
| Gesture | Label | Command | What the board does |
|---|---|---|---|
| Open palm | five | Pause / cancel | Pause bars on the matrix. Music pauses. |
| Pointing swipe | skip | Skip | Fast-forward icon. Pixel strip flushes and the next tune starts. |
| Thumbs-up | good | Confirm | Thumbs-up icon. A short confirmation tune plays. |
The tunes are the stand-in for the station. Confirm, cancel, and skip are real commands; the melody and the lights are how the worker knows the command landed without looking at a display. Lower the hand and the matrix clears after one second. The last command stays in effect.
Built in two days with Chaaruhaas Kandregula, Howie Zhang, and Terek J. I deployed the detector onto the Uno Q through App Lab, wrote the gesture-to-response path with Terek, and ran the failure-driven data loop once the pipeline was up.
What actually ran on the board for the demo.
| Piece | Role |
|---|---|
| Arduino Uno Q | Linux MPU runs the camera and the model. The microcontroller drives the matrix, buzzer, and pixels. |
| YOLOv5 via Edge Impulse | 320×320 RGB detector, three classes. Deployed as a Linux aarch64 .eim (impulse v17) through the App Lab video-object-detection brick. |
| Logitech C920 | 640×480 at 10 fps into the brick. The model then sees a 320×320 crop. |
| Python on the MPU | Picks the strongest known box above 0.5 confidence, holds the active gesture, and clears it after 1 s with no hand. |
| Arduino sketch on the MCU | Draws the 8×13 icon, plays the tune on the Modulino buzzer, and streams note color down eight Qwiic pixels. |
| RouterBridge | matrix_draw is a notify. music is a call, so the measured latency includes the MCU acknowledging the command. |
Each choice is a constraint from the challenge, not a default stack.
YOLOv5, one box per hand
A classifier would score the whole frame. The hand is not centered, and a hard case we cared about puts two hands in one image doing different things. A detector localizes each hand and labels it on its own. The published examples show two skip boxes in a single frame, at 0.88 and 0.86.
Still frames of the swipe, not a temporal network
Skip is a motion. A video model would have needed a longer window and more of the latency budget, and we had two days. A swipe is a sequence of pointing poses. If those poses are in the dataset, a per-frame detector can fire on the gesture without waiting for a clip. That is also why a sideways thumb works as a backup skip when a full swipe is awkward while holding weight.
Train on the floor, export .eim, load it in App Lab
Qualcomm’s Uno Q example showed how to get a detector onto this board. Its weights did not include palm, swipe, or thumbs-up. We kept the deployment path and replaced the model. Edge Impulse was the route that turned a labeled set into the aarch64 impulse the video brick already knew how to run.
Python for vision, the sketch for actuators
The Uno Q is a Linux computer attached to a microcontroller. The camera and YOLO belong on Linux. The matrix, buzzer, and pixels belong on the MCU, where the music state machine can run without waiting on Python for every note. Calling across the bridge, instead of only notifying, makes the latency number include the moment the command is actually accepted.
Onboard matrix, Modulino buzzer, Modulino pixels
The worker’s hands are full, and the point of the device is that they do not look down at a panel. The matrix is the command echo (thumbs-up, pause bars, fast-forward). The buzzer is the same echo in sound. The pixel strip is a running display of the note, so skip is visible as the lights flushing to the next tune.
About a 30-minute loop: train, deploy, find the miss, shoot that miss, retrain
Adding random hands did not fix the confusions we could already name. Swipe and thumbs-up collided, so we photographed both in the same frame and the mix-up dropped. Open palm was next: missed entirely, or called swipe, or called thumbs-up. The following batches were palms at new angles, distances, people, and backgrounds. Eleven-plus retrains, not one big collect-and-train.
SSH from the laptop to the Uno Q
We did not have the USB hub the bring-up assumed, and the venue network blocked device-to-device traffic. Terek’s phone hotspot put the laptop and the board on one network so we could deploy and read logs without the hub. That is what kept the 30-minute loop possible on the show floor.
Numbers from the Edge Impulse run and the final hardware log. The architecture figure is the 341-image snapshot; later collection pushed the set past 450.
On the hardware, inference stayed under 200 ms and averaged about 175 ms, which is the number the challenge asked for. In our swipe tests, about one frame in ten was a miss. A sideways thumb still registered as skip when a full swipe was hard to make. The closed loop on the floor was camera, YOLOv5, matrix icon, and buzzer together, not a model score on a laptop.
The same system took first prize in the Industrial Automation track of the EWNA 2026 Edge AI Hackathon.