Peripheral Labs Uses Self-Driving Tech for 3D Sports Video

Peripheral Labs, a Toronto-based AI research company founded in 2024 by Kelvin Cui and Mustafa Khan, is developing a novel approach to live sports broadcasting by integrating "spatial intelligence" derived from self-driving car technology. This innovation aims to provide viewers with an entirely new, interactive, and immersive experience, allowing them to watch replays and highlights in a fully three-dimensional environment, akin to a video game. The technology enables users to navigate the action from virtually any angle or focus on specific elements within the video, such as a player or the ball.

Kelvin Cui explained that the company is building machine learning models capable of understanding the 3D world, drawing directly from the technological advancements and concepts pioneered in autonomous vehicles, where both he and Khan previously worked. Peripheral Labs applies these principles to 3D video, volumetric video, and video reconstruction, facilitating the creation of photorealistic and navigable video environments. This allows for a level of detail and interactivity previously unattainable in traditional sports viewing. The system can track up to 65 points on individual players, extending down to the joints of their fingers, offering granular detail for analysis and replay.

While other companies like Arcturus and Wild Capture are exploring similar technologies, Peripheral Labs is initially focusing its application on sports. Cui noted that the controlled environments and repeatable actions within sports make them an ideal starting point for refining and demonstrating the technology. The goal is to offer a viewing experience that begins like a conventional TV broadcast but allows viewers to dynamically change camera perspectives at any moment, offering a level of control and engagement that mirrors the experience of playing a sports video game. This technology promises to redefine how fans engage with live sporting events by transforming passive viewing into an active, exploratory experience.

The cofounders' background in autonomous vehicle technology provides a unique foundation for Peripheral Labs' spatial intelligence approach. The sensors and machine learning algorithms developed for self-driving cars, which are designed to perceive and navigate complex, dynamic environments in real-time, are being repurposed and adapted for video analysis and reconstruction. This cross-disciplinary application highlights the growing trend of leveraging AI advancements from one sector to solve challenges in another. The ability to reconstruct entire scenes in 3D from standard video feeds opens up significant possibilities for sports analytics, fan engagement, and content creation, potentially setting a new standard for sports media production.