Learning to Play Guitar with Robotic Hands

Luo, Chaoyi; Tang, Pengbin; Ma, Yuqi; Huang, Dongjin

Learning to Play Guitar with Robotic Hands

Files

v43i8_cgf15166.pdf (1.82 MB)

paper1068.mp4 (126.53 MB)

Date

2024

Authors

Luo, Chaoyi
Tang, Pengbin
Ma, Yuqi
Huang, Dongjin

Publisher

The Eurographics Association and John Wiley & Sons Ltd.

Abstract

Playing the guitar is a dexterous human skill that poses significant challenges in computer graphics and robotics due to the precision required in finger positioning and coordination between hands. Current methods often rely on motion capture data to replicate specific guitar playing segments, which restricts the range of performances and demands intricate post-processing. In this paper, we introduce a novel reinforcement learning model that can play the guitar using robotic hands, without the need for motion capture datasets, from input tablatures. To achieve this, we divide the simulation task for playing guitar into three stages. (a): for an input tablature, we first generate corresponding fingerings that align with human habits. (b): based on the generated fingerings as the guidance, we train a neural network for controlling the fingers of the left hand using deep reinforcement learning, and (c): we generate plucking movements for the right hand based on inverse kinematics according to the tablature. We evaluate our method by employing precision, recall, and F1 scores as quantitative metrics to thoroughly assess its performance in playing musical notes. In addition, we conduct qualitative analysis through user studies to evaluate the visual and auditory effects of guitar performance. The results demonstrate that our model excels in playing most moderately difficult and easier musical pieces, accurately playing nearly all notes.

CCS Concepts: Computing methodologies → Animation; Reinforcement learning

        @article{10.1111:cgf.15166
,
journal = {Computer Graphics Forum},
title = {{Learning to Play Guitar with Robotic Hands
}},
author = {Luo, Chaoyi and 
Tang, Pengbin and 
Ma, Yuqi and 
Huang, Dongjin
},
year = {2024
},
publisher = {The Eurographics Association and John Wiley & Sons Ltd.
},
ISSN = {1467-8659
},
DOI = {10.1111/cgf.15166
}
}