HypeZone, released in December 2017, rapidly gained millions of new users to livestream community Mixer. HypeZone’s secret weapon? The 2017 Hackathon Grand Prize Winner, Watch For, a Microsoft Garage project.
Last month, Microsoft’s fifth annual One Week Hackathon wrapped up with astounding numbers. This year, during the largest private hackathon on the planet, over 23,000 employees registered to hack, and ultimately created 5,800 projects. As judging for this year’s projects begins and eager hackers await the winner announcements, it’s the perfect time to reconnect with last year’s Grand Prize Winner.
Originally called Lookout, the project team now known as Watch For has made tremendous strides in both personal growth and Microsoft business growth. Over the past year, team members Lenin Ravindranath Sivalingam, Matthai Philipose and Peter Bodik have been working as an incubation startup within Microsoft Research with autonomy and ownership to steer their project in a direction they desire.
The team’s original idea, which won the 2017 Hackathon, was an app to monitor live video streams on behalf of a user and notify him or her when specified events occur. Such a seemingly simple idea can be very powerful using artificial intelligence with many different applications.
Hackathon 2017 winning team: Matthai Philipose, Lenin Sivalingam, Yifan Wu, Peter Bodik and Victor Bahl. (Photo by Elizabeth Ong)
As part of Microsoft Research, the project team members previously worked on video analytics for enterprise scenarios in their day jobs. One of their biggest partners was working with the city to monitor and analyze traffic cameras for a better understanding of how pedestrians, bikes, and vehicles crossed intersections.
Not surprisingly, livestreams are big in enterprise settings, and that translates as well to consumer settings. For Lenin, Matthai, and Peter, the most interesting part of working on a hack project was experimenting with how best to apply video analysis to consumer scenarios.
“What attracted me to this hackathon project was the chance to apply AI in large scale and at low-cost to the consumer setting. Our project really pushes the envelope on how efficient the AI systems would need to be, and it’s also meaningful in that my kids and mother can understand it and use it.” Matthai explained, adding, “And I love the idea of working with Lenin and Peter.”
The team took what they learned over the years about video analytics and traffic cams, and created such a compelling project that not only did Microsoft CEO Satya Nadella put his influence behind them, but the senior leadership team took notice and became excited about the possibilities. Ed Essey, principal program manager of Microsoft Garage, helped prepare the team to think and work like a lean startup.
Over the course of several months, they fine-tuned a business strategy for their product – including the team’s special blend of expertise, knowledge, experience, and idea-leadership – that led the team to work on Watch For full time.
In September 2017, a few weeks after the team’s Hackathon win, the Mixer group reached out to the team, having seen their project video. Mixer, acquired by Microsoft in 2016 as Beam, is a next-generation, interactive live streaming platform with a large gaming audience.
Taking a community-first focus on features, Matt Salsamendi, principal software engineering lead, Mixer and Chad Gibson, general manager, Mixer saw huge opportunity to accelerate Mixer’s vision in the computer vision space and were excited to partner with other Microsoft teams working in this area.
The more popular games on Mixer tend to be multiplayer battle-royale style competitions where the last person standing wins. “Games like PlayerUnknown’s Battlegrounds (PUBG) and Fortnite are pretty new. For these games, a very simple thing works very well to light up Mixer scenarios.” Peter explained.
The scenario that Matt and Chad of Mixer wanted to execute on was how best to surface the most interesting parts of streams to a bigger audience. There are thousands of streams at any given time, of which only a couple hundred get viewed by most people. How do the rest of the streamers get any visibility and how do you avoid wasting those assets? How do Mixer fans discover those hidden gems? “The game streaming ecosystem has lots of undiscovered content, people wanting to be discovered, and viewers wanting to discover more compelling moments.”
“The game streaming ecosystem has lots of undiscovered content, people wanting to be discovered, and viewers wanting to discover more compelling moments.”
Lenin, Matthai, and Peter started to work closely with the Mixer team last September, and an ambitious goal organically formed, of launching new channels in winter of 2017 tailored with content discovered by AI models trained to “Watch For” specific events in streams. The timing coincided with PUBG’s release on Xbox One, which was fast becoming one of the most popular games on Mixer.
Mixer already had a front-end design where a single channel could host many different people’s streams continuously – they took advantage of that, and queried Watch For’s backend to determine when to switch between streams for the most interesting content. Thus, HypeZone was born – channels on Mixer using Watch For algorithms to highlight the final, nail-biting rounds of last-person standing games like PUBG that viewers found so engaging to watch.
“Matt already had the idea of HypeZone itself, to switch from stream to stream within a channel – but the experience of HypeZone evolved very quickly during our collaboration.” Lenin recalled. “We met with Matt and Chad early September. Two weeks later we had a prototype that we showed them. Then we kept improving its accuracy. By mid-October we had another prototype that they could use to run their HypeZone experience. We tested it for another 3 weeks. Then, 2 days before release, PUBG changed their UI. 1 day before release, we had to completely change all our models.”
Despite the whirlwind of activity, the Watch For team appreciated Mixer’s style of working fast and friendly. “As a business group, Mixer is very agile and easy to work with. We work close and we work well together.”
“The choice of content for HypeZone is determined by all the analysis Watch For does. Which is one of the reasons why we were able to move so fast,” Peter explained. Peter and team had to tailor their AI models for HypeZone by building core video analytics skills specific to each game.
Over the last several months, HypeZone channels were among the most popular channels on Mixer. “It’s a win-win product. Viewers love it because it shows only the most exciting content, and streamers love it because they get featured on Mixer’s front page and get new followers. They start streaming more because they want to be featured on HypeZone and gain followers.” Game producers can also be counted among the many fans as HypeZone provides more exposure for their games.
The biggest challenge – and the team’s biggest accomplishment – was how to get HypeZone to scale, and at low-cost.
“HypeZone is driven by Watch For’s large-scale video analysis of every stream that’s coming into Mixer. Every stream we try to understand what’s on the screen. We look for various metadata that tell us the game is exciting. Text on the screen, icons that tell you state of the game, player stats and score. Over time we have evolved to understand more and more.” Lenin explained.
The secret sauce is very much a combination of Matthai’s AI expertise and Lenin and Peter’s end-to-end distributive systems knowledge that allows them to deeply and efficiently analyze and understand each stream’s content in real-time.
“This is one of the advantages of being in a company like Microsoft. The Garage and Hackathon gave us visibility, but there was a product group (Mixer) out there looking around who had a great understanding of their customers, and that Watch For might light up their market.” Matthai recalled how it all came together. “There was an element of luck that battle royale type games came into vogue around the same time. It’s a combination of all of these things that made this partnership work so well.”
“It’s one thing to have cool demos and enthusiasm from senior leadership, but it’s another thing to see our customers enjoying, laughing and crying , wanting to see more. That’s what really lit a fire under the whole project, that connection.”
A game-changer for streaming content platforms and how content can be surfaced and consumed – Watch For is a stellar example of using artificial intelligence for consumer scenarios. What’s next for Watch For? The team continues to work with Mixer, and other groups, to create awesome experiences yet to come using the power of AI.
Story by Meixia Huang
Check out HypeZone on Mixer https://mixer.com/
Get videos on the Mixer Channel One on YouTube
Follow Mixer on Twitter: https://twitter.com/WatchMixer
Read more about this Hackathon team:
Artificial intelligence eclipses cloud and mobile projects to win the day at Microsoft 2017 Hackathon
