This summer, Microsoft’s AI for Earth team and the office of the AI CTO decided to challenge a group of Garage interns to pair AI with aerial images to make a difference. Today, we announce the release of their summer project: Earth Lens, a Microsoft Garage project, a Xamarin-based, open source project for iPad that identifies, tracks, and analyzes objects in aerial imagery to assist in scenarios such as disaster relief and environmental conservation. Earth Lens releases on the heels of Microsoft’s announcement to commit over $40 Million to a new initiative to leverage AI in efforts to save lives, joining a suite of programs known as AI for Good, including AI for Earth, AI for Accessibility, and now, AI for Humanitarian Action.
Imagine yourself as a data analyst working to identify objects in aerial imagery. Maybe you’re tracking fishing vessels through the ocean to support ocean sustainability and protect marine ecosystems, or perhaps you’re looking at cities that have been hit by a natural disaster to determine where to allocate your relief efforts. In either case, you would need a dedicated team to comb through thousands of satellite images, manually identifying, tracking, and analyzing relevant objects. Extracting insights from imagery data is an expensive process that could take upwards of days, delaying the decision-making process in mission critical situations.
Inspired to make this process more efficient, the two co-sponsors decided to team up to pitch Garage interns on tackling this problem by leveraging AI. The Garage Internship Program offers teams of 5-6 university students the opportunity to design and build their own project in response to a challenge by sponsoring Microsoft groups. The Vancouver-based group of interns loved the AI for Good pitch. “In the start of May, our sponsors came to us with an idea that immediately sparked interest among the six of us: an opportunity to leverage AI to help with environmental efforts headed by researchers and conservationists,” shared Michelle Chen, Program Manager intern for Earth Lens.
AI for Good
The team was especially inspired by the work of the organizations helping victims of natural disasters by providing satellite imagery to hasten relief. She continues, “Our vision for Earth Lens was to use technology and AI to transform the way humanitarian work is conducted. The rise of automated image recognition has empowered humanitarian organizations to triage damage, and prioritized areas that require immediate help.”
Equipped with a trained machine learning model from the AI for Earth team, the interns set out to build a minimally viable product over the course of their remaining 16-week internship. They approached design with a customer obsessed mindset and interviewed prospective users like to the Red Cross, OceanMind, and FarmBeats to identify useful features and iterate on their prototype. The team quickly realized that, in addition to disaster relief, such a tool might be useful in a variety of social impact and sustainability applications such as agriculture, forestry, fishing, and land conservation.
The team built an iPad app that offers a host of features that humanitarians and environmental researchers can leverage to accelerate their analyses:
- Automatically identify, classify, and label objects in satellite images, encircling like objects in color-coded bounding boxes
- Count and aggregate objects identified by the machine learning model
- Toggle classes on/off to focus in on specific objects
- View images over time alongside a data visualization chart to identify trends and patterns in a Time Series mode
- Use the app remotely, without an internet connection
With the Earth Lens source code and corresponding instructions now available on GitHub, researchers will be able to build their own iPad apps that automatically extract information from a large dataset and scan that analysis in a useful and interactive view. As a result, this open source project can be useful in various industrial or agricultural applications to further Microsoft’s commitment to sustainability. In a disaster relief scenario, another minute could mean another life lost.
An Internship Where You Can Make an Impact
This Garage project illustrates how developers can use AI to make a difference and the team hopes they’ve built something that will spark humanitarian creativity in others. “We all had the privilege and opportunity to experience what it means to work at Microsoft and be a part of something larger than us,” notes Seara Chen, a Software Developer intern who worked on model integration. “Our entire team agrees that this internship was an amazing experience filled with learning, surprises and a lot of problem-solving.”
Interested in trying Earth Lens? Check out the source code on GitHub, where you can also find instructions on how to build the iPad application and leverage the machine learning model that powers the project. You can learn more or provide feedback via GitHub.
The Garage Internship Program is hiring. You can learn more about openings and explore past Garage intern projects: Seeing AI, Ink to Code, Mobile Chest X-ray.
