Interactions in mixed reality are a challenge.Nobody wants to hold bulky controllers and type by clicking on big virtual keys one at a time.But people also don’t want to carry around dedicated physical keyboard devices just to type every now and then.
That’s why a team of computer scientists from China’s Tsinghua University developed interaction technology called Palmpad that enables typing with just a single RGB camera and an Arduino.This isn’t a new idea, but the execution stands out.It works by pointing a camera at the user’s hands.
In mixed reality, the user sees a small keyboard projected on top of their palm.They then use their index finger on their opposite hand to tap the virtual keys.The camera tracks the movement of the index fingertip relative to the palm to determine which virtual keys the user touches.
Previous attempts by others to implement this method have had limited success for two reasons: unreliable fingertip tracking and poor touch detection.The Tsinghua University team solved the first problem with sophisticated video frame processing techniques, such as motion extraction, and with a convolutional neural network (CNN) trained for this purpose.Their system is able to determine the position of the fingertip relative to the palm with great accuracy.
They solved the second problem by using an Arduino UNO Rev3 to monitor electrical signals being passed from one hand to the other.Those signals come from a waveform generator, which produces two very low-current, low-voltage AC signals at 12.5MHz.One is a reference and the other passes from an electrode on one hand to an electrode on the other hand.
The Arduino monitors those signals through an analog multiplier and a low-pass filter, detecting a change in impedance indicating finger-to-palm contact.
Wherever the user’s fingertip is at that moment is the point registered as a “click.” The team reported a touch recognition accuracy of 97.0%, which makes Palmpad quite reliable.More details on the Palmpad can be found in the researchers’ paper.