What if everyday work flows were re-imagined through the power of AI to inspire and delight? What if users could achieve more with a tool that is both easily accessible and familiar?
Millions of Windows users perform the same task every single day. Screenshots are taken all the time, but as technologies progress in leaps and bounds, screen capturing tools haven’t really changed at all.
Snip Insights is a Microsoft Garage intern project focused on injecting AI into the screen capturing process.
The project has three primary goals. One is to revolutionize the way you search, shifting the paradigm from text-based search to image-based search. Another is to increase productivity by providing insights just a snip away. The last is generate insights about the limitless possibilities of AI.
About the team
At Microsoft Vancouver, the curiosities of 7 interns were piqued by a pitch from the Cloud AI team. The intern team, composed of 5 developers, 1 designer, and 1 program manager, was challenged to re-imagine widely used tools that millions of users love through the capabilities of AI. Thus, Snip Insights was born.
Snip Insights team (l to r) David Wu, Mehdi Chaid, Christie Chan, Paulo Aguiar Araujo, coach (on-screen Evgeny Vinnik, coach), Athi Narayanan Jeyaraj, Jeffrey Chang (on-screen Marius Ungureanu, coach), Vivy Wang, Charmaine Lee
What is Snip Insights?
Snip Insights is an open source desktop utility that enables Windows users to retrieve image intelligent insights through a simple screenshot. Screenshots are essentially snapshots of peoples’ lives. Snip Insights helps you achieve more with those stories by leveraging Cloud AI services to convert images to translated text, automatically detect and tag your image content, along with many more smart image suggestions that would improve your workflows while showcasing Azure Cognitive Services’ potential.
Imagine that you have a scan of a textbook or work report. Rather than having to manually type out the information, snipping it will return editable text in just one click. Or maybe you’re scrolling through your social media feed and come across someone wearing a cool pair of shoes, you can simply snip to find out where to purchase them! Snip Insights can show you relevant information based on what you’ve just snipped – including identifying famous people, places, or landmarks.
Building Snip Insights
Injecting the capabilities of AI seemed technically daunting. Finding focus was also a painstaking trial. The intern team was given an arsenal of resources, but they had to forge the path to a product on their own through endless sessions of brainstorming, whiteboarding, and mind mapping.
From a technical standpoint, it came down to building off the Windows Snipping Tool or Snip, a retired Microsoft Garage project that let users annotate a photo or screenshot with voice and ink and share with others.. Ultimately, the team chose Snip for a more straightforward integration of Azure Cognitive Services and for overall ease of releasing as an independent tool. Snip, built on the WPF platform using C#, was a solid foundation for this project.
Quick iterations
An integral part of the experience was conducting user research and user-focused development. The team constantly tested out iterations internally, working with developers, designers, PMs, and stakeholders. It became clear that one of the biggest challenges would be to decide on a target audience.
Any computer user has access to screen capturing tools. Feedback from users varied drastically; each person used her or his tools in diverse ways and for different purposes. The interns figured out a lot of things just by trial and error. They learned to react quickly to change, whether that be a newly discovered visual design problem or a sudden technical obstacle. At one point, just a few days before a key milestone, the team decided to entirely redesign the user interface based on critical user feedback, to further distance the product from the original Snip visual design and bridge the gap to the Fluent design language. The ability to iterate quickly as a team proved to be the most invaluable skill.
Endless possibilities of community-driven development
By releasing the project open-source, the team hopes to inspire developers around the world to innovate and improve on their efforts. The Snip Insights GitHub repository will be maintained by the Cloud AI team. The internship was just a short four months where the interns worked on their own team, developing and iterating on this project as part of The Garage Internship Program in Vancouver. Though they were chartered by the Cloud AI group with a specific challenge, the solution possibilities were many. There were various use cases explored but not realized. On such use-case was to inject intelligence into the actual snipping process. What if the tool were smart enough to recognize object edges and shapes, cutting out additional steps and third-party software completely?
With the power of Microsoft AI and Cognitive Services, any developer can develop intelligent applications that will be loved by users around the world. If you have the motivation, curiosity, and drive, all the information and resources are openly available.
Snip Insights is now on GitHub and this is an invitation for you to contribute your ideas!
Check out other Garage projects built by The Garage Internship Program in Vancouver including Write Ideas, My Workspace, and Your News
