Jason Peng, an assistant professor of computing science at Simon Fraser University, is leading a research team that is pushing the boundaries of motion simulation technology—using tennis as the proving ground to demonstrate just how realistic virtual athletes’ movements can become.
Peng and his team have developed a cutting-edge machine learning system that learns complex tennis skills from broadcast video footage. By analyzing a vast dataset of professional tennis player videos, the system is able to perform intricate shots and string them together to simulate lifelike rallies. The result is realistic matches between two physically simulated characters, complete with accurate racket and ball dynamics.
In contrast to the motion capture methods used in most sports video games, which are limited to movements recorded during specific sessions, Peng’s system generates a broader range of behaviors by learning directly from video footage. This means characters can perform new and responsive movements that weren’t captured in the initial recordings.
Collaborating with researchers from Stanford University, the University of Toronto, the Vector Institute, and NVIDIA, Peng’s team is pioneering a system that enables video game characters to learn real-life movements and automatically simulate new variations on the fly. Their hierarchical control model combines low-level imitation with high-level motion planning to guide character movements based on actual footage of sports games.
One of the challenges the researchers faced was creating lifelike motions from real tennis footage, as early attempts led to unstable and unnatural movements. To overcome this, the team implemented a physics-based motion correction system, which refines the learned movements and ensures smooth, stable animations.
Now, the system can accurately simulate tennis players hitting the ball with a range of strokes—forehands, backhands, spins, and serves—while maintaining the unique playing styles of each athlete. The characters can even mimic left and right-handed players, as well as one-handed or two-handed backhands.
While the system is currently focused on tennis, Peng envisions its application expanding to other sports, such as basketball, hockey, and soccer. “We certainly plan to expand beyond tennis into additional sports,” says Peng.
He also hints at long-term goals beyond gaming, stating, “The system is versatile and could eventually be used for teaching robots new skills by analyzing video footage.”
