Mixed Reality Video Calling for Microsoft Teams Together mode

In Spring 2017, we trained our first deep person segmentation model as part of the inaugural AI-611 Advanced Projects class in the Microsoft AI School. We built upon this work to create background blur in Microsoft Teams, which was later released in Spring 2018 based on a refined version of the segmentation model. Ever since, we have continuously optimized and improved the accuracy of this model in close collaboration with Microsoft Research Asia.

Then in Summer 2018, David Zhao and Henrik Turbell created a Microsoft Global Hackathon project called Mixed Reality Video Calling, with the goal to remove the “boxes” from video calls and to enable people to communicate virtually in a shared environment. Walid Boumerdassi, an engineer with the media bots team, joined the project to build the solution using the real-time media bot framework. The Microsoft media bot technology allows for real-time processing of both audio and video in the Azure cloud at scale. The media bot receives the video streams from each participant, applies the segment algorithm and composes them together into one composite stream. Since all processing is done in the cloud, it works on any kind of device without any special hardware requirement.

The Hackathon project showed how everyone on the call can be segmented and placed into the same virtual environment. The project was rewarded the first place in both the Modern Life Challenge and the HACKxR category. The team met with Rajesh Jha, Executive Vice President, who was very encouraging and mentioned that students and teachers would love the ability to place everyone in the same virtual room for online learning.

Fast forward to Spring 2020, when Jaron Lanier, Microsoft Prime Unifying Scientist, father of virtual reality, and friends with a late-night show host, learnt that the host was struggling to perform his monologue to a camera in his living room, without a live audience to react to his jokes. Jaron reached out to Microsoft Teams for a way to have a virtual audience. Over a weekend the team hacked the hack and demonstrated the first Together mode call over Teams.

The team continued working sleepless nights with more colleagues joining the effort, and in record time, transformed the Hackathon prototype code into a production quality feature. Microsoft Teams designers and researchers made crucial design improvements, like adding seats and creating beautiful environments. If you use Teams Together mode, you might also notice a detail that everyone is mirrored. This was something that Jaron insisted on because it makes interaction with others feel more natural, for example, if you want to “tap” the person’s shoulder who is sitting next to you in the Together mode seats.

After three months, the Together mode feature launched worldwide. Those hectic months brought together a diverse set of talents from all around the company, with expertise in design, cloud deployment, computer vision, algorithm optimization, AI/human collaboration, marketing, and even brain activity research. Together mode evolved from prototype to a cross-platform worldwide service during the pandemic. Everyone now can enjoy a new way to hold meetings and interact with each other.