Play a piano with your eyes - Gaze estimation
In this example, we will estimate the gaze direction of a person and use it to play a virtual piano when the person looks to certain notes.
This examples uses the Google Mediapipe library to calculate the face mask and obtain the position of the eyes. The calculation of the gaze diration is done with OpenCV.
Pipeless: Check the installation guide.
Python OpenCV, NumPy and Mediapipe packages. Install them by running:
pip install opencv-python numpy mediapipe
- Python SimpleAudio` package. Install it by running:
sudo apt-get install libasound2-dev # Required by simpleaudio
pip install simpleaudio
pipeless init my-project --template empty # Using the empty template we avoid the interactive shell
Feel free to change
my-projectby any name you want.
wget -O - https://github.com/pipeless-ai/pipeless/archive/main.tar.gz | tar -xz --strip=2 "pipeless-main/examples/gaze-piano"
The following command leaves Pipeless running on the current terminal
pipeless start --stages-dir .
Open a new terminal and run the following command to provide a stream from your webcam:
pipeless add stream --input-uri "v4l2" --output-uri "screen" --frame-path "gaze-piano"