Art is one of the few languages which transcends barriers of country, culture, and time. Most people view art subjectively through a lens shaped by their experiences and environment. Finding commonalities among pieces from different eras and mediums calls for an open mind and a sharp eye. The Mosaic project is taking this intellectual and human approach and augmenting it with the help of artificial intelligence to find connections between works of art from diverse artists. Working with data from The Rijksmuseum – the Museum of the Netherlands and The Metropolitan Museum of Art (The Met), a group of MIT students implemented a new algorithm for visual search that finds similarities across cultures in the museums’ collections as part of their externship project at Microsoft Garage.
Based on a starting image of a Banyan, the algorithm finds similar works of art from different artists in the Rijksmuseum and The Met collections.
The MIT externship with Microsoft Garage takes a traditional semester-long internship experience and condenses it to a month of rapid development where students take an impactful project from concept to reality. Microsoft Garage at NERD was the home base for the externs during the month of January, where a culture of innovation is fostered, and growth mindset is put into action. Some of the externs have since returned to Microsoft as Garage Interns, as part of a virtual summer internship experience where students collaborate online through Microsoft Teams and other productivity tools.
The idea started with long-time Garage collaborator and Mosaic project sponsor Mark Hamilton, Research Engineer pursuing a PhD at MIT alongside his full-time job on the Cognitive Services Research Group at Microsoft’s New England Research and Development Center (NERD). Mark was inspired by a visit to an exhibit at the Rijksmuseum. This exhibit compared works from the studios of Rembrandt and Velazquez, two prolific artists that never met, yet told similar stories in their works. He began thinking of possible ways the same idea could be enhanced by AI, and developed an algorithm which searches for similar works of art based on a starting image and a feature of that image such as the medium, a color, or the culture (e.g. “French”). Last January, Mark began working with The Met’s Open Access collection with the previous cohort of MIT externs, and out of that effort grew Gen Studio, an interactive tool giving people the ability to generate new art from trained generative adversarial networks.
This year’s MIT students had a different goal with Mosaic.
Mosaic website exploring connections in art.
“This project leverages existing collections of art from both The Met and the Rijksmuseum to find connections between various cultures and time periods,” Mark explained. “With Mosaic we can shine a light into areas of the collections that may not be as well-known and discover really interesting things.” Curators and visitors can use Mosaic to explore art in new ways, such as charting trends that underlie works from different cultures and finding evidence of cultural exchanges and influences throughout history. One example Mark described was how the algorithm found similarities between a Ming Dynasty porcelain vase, a Dutch violin, and a banyan from the Netherlands. “The algorithm picked up on commonalities of form between the banyan and a string instrument, but also the color of the blue and white motif which originated in Ming Dynasty and spread through Europe. You can see evidence of cultural exchanges happening through the outputs of this algorithm.”
For a deeper dive into how this new way of conditional image retrieval can help improve GANs, Mark hosted a Microsoft Research Webinar on the topic.
Before the machine learning model could be implemented, however, there was the complex task of cleaning the data associated with each piece of art.
Meet the team behind Mosaic
Students Mindren Lu and Stephanie Fu study computer science at MIT and worked closely to comb through data from both museums. No small feat, the two data scientists organized descriptive metadata like keywords, authors, and tags associated with each artwork into consistent datasets that could be fed into the algorithm. “There was a lot of manual processing since much of the data is written in another language, sometimes there were misspellings or missing information,” said Stephanie. “It took a lot of time to boil all that down to a nice, formatted dataset that the back-end team could work with.”
Marina Rogers was part of the team building the back-end of Mosaic that interfaces with various Azure products and services. “We’d bring the cleaned datasets and collections in and create the machine learning models based on Mark’s research. Then, we’d deploy the models and different APIs for sharing that connect to the front-end website,” explained Marina, who is an electrical engineering and computer science student with a minor in design.
“Being able to share our work with others is definitely a highlight,” said Johnny Bui, also studying electrical engineering and computer science. He reflected on how getting technical pieces and services to “talk” to each other in novel ways behind the scenes was a challenge. Felix Tran and Maggie Wang completed the team working on the back-end systems that provided uninterrupted end-to-end data flow, described by Jean-Yves Ntamwemezi, Software Engineer on the Microsoft Garage at NERD, as “the glue that brings the pieces together.”
Jean-Yves joined the Garage team just before the externship began, coming from a long tenure as a software engineer on the Office Docs team. “It was a very interesting, fast-paced learning environment, jumping in to lead the externs in many ways.” In addition to general mentorship and support, Jean-Yves readied the Azure DevOps environment for the project, set up pull requests so the externs could have their code reviewed, and introduced them to the right tools and engineers with the expertise to help unblock progress. He also leads the Blacks at Microsoft employee resource group local chapter, building culture and leadership within the company.
MIT externs: Stephanie Fu, Mindren Lu, Ben Chen, Felix Tran, Darius Bopp, Maggie Wang, Marina Rogers, Johnny Bui
“The Garage is a perfect mix of working on exciting technologies and leading a team of engineers toward a solution,” said Jean-Yves. “Interns and new hires inform what the culture becomes, affecting what the work looks like, what teams look like, what the focus is in our working environment. Working in a technical space where empathy is a priority in your thought process – that’s a place where I want to grow.” For Jean-Yves, an important part of his day-to-day was making sure the students were building camaraderie throughout the process.
Darius Bopp and Ben Chen were responsible for the front-end website that is a culmination of all the work the team performed packed into a visually engaging web experience that finds connections among art in new ways. “They took this nebulous design of a website and made it into something that looks great and is easy to navigate,” said Mark. “The front-end team was so fast that we were able to spend a lot of time thinking about what was absolutely necessary on the site, making the website less complex and more intuitive.”
Visualizing machine learning
Though the data science team performed a lot of critical work with Mark’s original algorithm, they didn’t stop there. As a stretch goal, Mark introduced the SHAP (SHapley Additive exPlanations) algorithm to Mindren and Stephanie, a game theoretic approach to explain the output of any machine learning model. The two were immediately intrigued, delving further into the research and how to apply it to their ML model for Mosaic. “SHAP is a tool that tries to explain why the ML model made the decisions that it did, why out of all of these paintings it chose this particular one as a match. ML models don’t think like humans, even though it seems like they do,” said Stephanie. What resulted is a fascinating visual representation of what the algorithm is looking at when it finds objects “similar”. This can show what the model “sees”— like a window into the brain of machine learning — but it takes extensive memory and space to run. For that reason, and because their externship was ending, the team could only implement a few rationales to show on the website as an example.
Machine learning algorithm behind Mosaic identifying parts of pictured sculptures to query and find matches in other artworks.
Though the weekly development cycles went quickly, the externs had solid guidance from Mark throughout the entire process. Mark brings his deep appreciation for art to the work and enjoys applying his knowledge of AI to create new contexts in how art is explored and studied. “So many artists are trying to embody the metaphorical in the visual domain, and this algorithm just can’t help but see that and pick up on it when it finds these matching works from all over the world,” Mark explained. “Art is aesthetic and beautiful, but a lot of times artists are trying to say really deep things about the nature of perception, of feelings and emotions.” In most instances one could almost say the AI managed to extract the intent or idea of the paintings, not just the pixels, to find commonalities.
Art also resonated in Marina’s life as she grew up with a mother who studied fine art. Marina found the most important part of the experience was “the culture of improvement and personal growth. The overall environment was one of learning and encouragement.”
Johnny and his peers agreed the structure of the externship experience set them up for success. “Working on a project of this scale as well as interfacing with third party tools, and collaborating with other people, was a great learning experience. We had a running start on the project within the first couple of days,” he explained, referring to the mini-hackathon that kicked off their experience at the Microsoft Garage where each person could try out what data science, back-end, and front-end development work on the project might look like.
“It was my first real work experience in a big company on a fast-paced project team. It helped inform what direction I want to take in my career,” said Mindren, who found the most exciting moments to be researching about the rationale tool and when he received those first matching artworks from their ML model. “You have a lot more fun than expected, and the culture was cool and super friendly.”
Explore more on the Mosaic website
Check out the Microsoft Research Webinar
Read about 2019 MIT externship project Gen Studio
Learn about the Microsoft Garage Internship Program
