The technology used in this project develops a virtual instrument performance engine combining multi-mode sensors, applying deep learning techniques for finger tracking and professional fingering, and developing various virtual instrument groups suitable for large-scale real-time virtual orchestra performances.

The virtual performance engine proposed in this project combines a multi-mode sensor to realize an empty-handed live performance of each virtual musical instrument. Although researchers studied about virtual instruments and similar virtual instruments, those techniques focus on improving the accuracy of detailed finger tracking.

The proposed virtual orchestra performance engine has successfully verified its possibilities in different performances. Combined with a multi-mode sensor performance finger identification and virtual theater achievements researched by Professor Shi Guojun＇s research team in recent years adds more interactive technology elements to the music performance.