果冻影院

The 3D Vision team at 果冻影院 developed a new method to synthesise video of photorealistic human faces in speech, resulting in the creation of a highly successful spinout company Synthesia.

The technology behind Synthesia, developed by Professor Lourdes Agapito (果冻影院 Department of Computer Science) and team, enables users to create professional looking videos with photorealistic digital 鈥榓ctors鈥� simply typing a message. The results are indistinguishable from real video but without the need for cameras, actors, or expensive film studios.

Synthesia鈥檚 technology was used to make David Beckham speak nine languages as part of a 2019 global campaign video to raise awareness about malaria, resulting in 700 million online impressions.

A cheaper and easier method for 3D capture 聽

Synthesising photorealistic, expressive human faces in speech has been a long-standing challenge in computer vision and graphics. For decades, this technology has been the exclusive domain of the film and TV industries, with multi-million budgets needed to build specialised and complex multi-camera 3D capture studios to create digital 3D doubles of humans, and for manual post-production by visual effects artists. 聽

Professor Agapito鈥檚 team pioneered weakly supervised methods for 3D human pose estimation from single images that only require 2D image annotations. These are cheaper and easier to harvest than the 3D annotations required by other methods. These breakthrough algorithms for monocular non-rigid 3D reconstruction by Professor Agapito鈥檚 team at 果冻影院 form the underpinning technology that made 3D-driven, photorealistic and low-cost AI video synthesis finally possible and form an integral part of Synthesia鈥檚 technology. 聽

Scoping a range of commercial applications

Professor Agapito co-founded Synthesia with other researchers and entrepreneurs to provide commercial solutions for a range of applications for this new technology, from lip-sync dubbing for content localisation to personalised video messages and corporate training. Synthesia technology allows users to create professional looking videos by simply typing a message, using an automated, 3D-driven AI process to synthesize photorealistic results that are indistinguishable from real video but without the need for cameras, actors, or expensive film studios.

Synthesia has transitioned from offering high profile video-to-video services towards offering a 鈥楽oftware as a Service鈥� platform where users can create videos simply by writing the speech of the digital 鈥榓ctor鈥�. This technology has a wide range of applications, from corporate training to in-house communication to sales. As such, Synthesia鈥檚 services have been used by diverse clients, including Reuters, WPP, Dixa, Just Eat, Tesco, FedEx, Facebook and Google. As a result of its world-leading technology, Forbes magazine named Synthesia one of its 鈥榝earless five鈥� Tech companies.

Public health engagement

Synthesia鈥檚 technology was also crucial in a 2019 Malaria No More campaign that raised $14 billion to help end the world鈥檚 three biggest preventable killer diseases: AIDS, Tuberculosis and malaria. It was used to make David Beckham speak nine聽languages as part of a 2019 campaign video, and because this video localised the campaign to suit specific global audiences, it created 700 million online impressions and resulted in the disease鈥檚 peak awareness in almost three聽years.

Synthesia鈥檚 CEO and co-founder Victor Riparbelli commented:

鈥淭he ability to capture the 3D geometry and appearance of a human face in speech from a single video with algorithms building on Agapito鈥檚 research has been transformational in allowing us to build a low-cost solution to create high fidelity 3D avatars of humans for animation and synthesis.鈥�

Links

• 果冻影院 Computer Science
• 果冻影院 Faculty of Engineering Sciences
• 果冻影院 Engineering REF 2021

滨尘补驳别听

• Image credit: