To help celebrate the second anniversary of The Metropolitan Museum of Art’s Open Access Program, The Met teamed up with Microsoft engineers and MIT students and faculty to imagine new ways in which their encyclopedic collection could be explored by global audiences.
Chris Hoder speaking at December 2018 hackathon with The Met, MIT, and Microsoft
The collaboration started with a 2-day hackathon in December 2018 at The Garage in Microsoft’s New England Research and Development (NERD) center. Microsoft researchers and creative technologists along with students and faculty from MIT’s Open Learning and Knowledge Futures Group worked alongside a group of curatorial and digital department participants from The Met. At the hackathon, The Met provided a pre-release of their “subject keywords” dataset. The new dataset contains keywords about depicted subject matter and provides a new pathway into the museum’s collection.
Collaborators from MIT included Matthew Ritchie, the Dasha Zhukova Distinguished Visiting Artist at the MIT Center for Art, Science & Technology and computational neuroscientist Sarah Schwettmann from the MIT Department of Brain and Cognitive Sciences and the Knowledge Futures Group. Participants leveraged Microsoft’s AI technologies and built custom models with Azure Machine Learning to parse, explore, and create prototype experiences. At the heart of these multifaceted experiments is something Ritchie, a contemporary artist, explained so simply: “Machine learning makes it possible to begin visualizing the diversity and complexity of artistic creation.”
On February 4th at a private event hosted at The Met, teams revealed five concepts from the hackathon brought to life as prototypes that connect and inspire a global audience using art and AI.
Gen Studio website experience
One experience showcased is Gen Studio, a project based on sophisticated generative adversarial networks (GANs) which allow users to explore, search, and even be immersed within the latent space between art pieces.
Generist Maps, a component of the Gen Studio project, enables users to visually and creatively navigate the shared features and dimensions underlying 5,000+ years of human creation represented in The Met’s collection.
“It’s an interactive map-like model showing how each piece of art relates – in both linear and unexpected ways – based on corresponding datasets and attributes like country, material, and keywords.” Mark Hamilton, software engineer at NERD for cognitive services and co-manager of the project brought his deep learning research and applied AI background to the collaboration. In exploring the connections amongst different art pieces, the spaces between is where users can discover dreamlike images created by the AI. “Each space corresponds to a full painting the network dreams up so you can take two paintings and discover what’s between the space of those paintings. It’s the researcher’s job to figure out what’s going on in the network’s brain – and now you the user can explore the network as an artist in this map-concept.” The project enlists the help of multiple Azure tools including Azure Kubernetes Services, Azure Cognitive Services, and Azure App Services.
Hamilton, alongside Microsoft colleague and program manager, Chris Hoder, mentored five MIT students who built a working prototype during their MIT externship with The Garage at NERD for the entire month of January.
“It’s been fun to work with everyone to get everything up and running, we’ve moved very quickly to take ideas and implement them in hours instead of weeks.” Hoder has been involved with The Garage at NERD since its opening last year and managing this collaborative project since the December hackathon event. “The prototype we built is such an intriguing and interesting experience that I’ve found myself getting lost in and playing in it for hours.”
Generist Maps website experience
Generative Explorations, the second component of the project, is a searchable web experience that uses Azure Search with MML Spark to make it easy to find specific art pieces. “We take all of The Met’s data and add searchable fields so you can query based on properties like tags and medium.” Alonso Salas Infante, sophomore at MIT interested in cyber security, was on point for the search experience. “In some cultures, the artist of the art piece doesn’t matter, so, many pieces did not have authors. There are also differing labels. We had to find a balance between how much information we should make available to the user and what we shouldn’t for the best experience.”
For some of The Garage externs, working with machine learning was a new experience. Elaine Zhang, junior at MIT, explained how the externship was her crash-course on neural networks and deep learning algorithms. “Getting thrown headfirst into ML has been challenging but so rewarding. We didn’t know what project we’d be working on prior to accepting the externship, so this was a very cool project I did not expect.” Zhang focused her efforts on the GAN, inputting data into the machine learning model to fine-tune the results.
MIT externs at The Garage. Left to right, top to bottom: Chris Templeman, Maria Guirguis, Mark Hamilton, Chris Hoder, Linda Thackeray, Alonso Salas Infante, Gillian Belton, Darius Bopp, Diana Nguyen, Elaine Zhang
Studying computer science at MIT and interested in game design and product development, Darius Bopp was responsible for much of the front-end including building the website infrastructure. “I took a class on web development last semester so I’m pretty new to it. That’s where Chris [Hoder] helped me a lot, and the community here has been really supportive. Learning the project workflow from Microsoft’s point of view was great and this is the first project I’ve worked on where the stakeholder wasn’t in the same room as me. We had a lot of autonomy in our day-to-day work.”
Gillian Belton, senior at MIT, drove the visual map-like exploration piece of the app. “The biggest reward for me was dealing with all the things on the back-end, setting up remote computers, using Azure. There’s been three or four different frameworks I’ve never seen before. Now I’m able to go in and feel confident working with all these things. It made me realize I could pick up almost anything and do it.”
Diana Nguyen, a junior at MIT, previously interned with The Garage and worked on an exploratory HoloLens concept that was more under wraps. “I’m really happy to see something I worked on getting into the hands of potential customers and being shown at such a public venue. Seeing how these technologies can be applied here with art, has made me realize machine learning is something I would be interested to work on more.” Nguyen’s focus was integrating the backend ML with the website and deploying models on Azure, ensuring the site could call these models to create more images.
Though currently still in the prototyping phase, the larger goal is to eventually release Generist Maps and Generative Explorations publicly and open-source much of the nuts and bolts of the project so developers can use the APIs in their own apps.
Though Microsoft Garage has internship opportunities in many worldwide locations, the New England location sits in a unique area with an incredibly vibrant STEM ecosystem and student population. So much so, that more organizations from all over are reaching out to partner on hacking opportunities with not only Microsoft engineers, but students who bring fresh perspectives to tough problems. The MIT externship program is a lot like The Garage internship, students can explore the industry they are interested in working in, but a key difference is how long externships last – approximately 4 weeks rather than an entire semester. “Though Microsoft has been participating in externships for a while, this is only the second year that we have brought externs into The Garage.” Maria Guirguis, Garage Software Developer, engineer, and the industry sponsor of the January MIT externs, provided guidance and helped the students settle into their new digs at NERD.
Collage of paintings generated by the network
“Definitely one of the most fun experiences in my time at Microsoft.” Mark Hamilton shares. “I think The Garage really understands the importance of trying something, going out on a limb, it’s been a great space to work in. And you know software development doesn’t have to be a monolithic thing. All you really need is a few passionate people and interesting content to work with.” Hamilton grew up in New York and visited The Met on numerous occasions. “It was amazing to be able to see behind the scenes. The people we worked with at The Met were very open and excited about this opportunity. And the externs never cease to amaze me with how fast they can pick things up and hit the ground running.”
Having all the right tools helps, as Chris Hoder explains. “This collaboration shows what you can do today with Azure and AI without needing a depth of data science and expertise. Azure and our AI tools can make it easier to apply this kind of technology to any project.”
Linda Thackeray, Director of The Garage, is excited to cultivate more partnerships both within and outside of Microsoft. “Having Mark and Chris embedded with the extern team and working so closely – they’re literally right down the hall – has been fantastic. It was meant to be. And we’re excited to see what comes next.”
Learn more about The Met, MIT and Microsoft hackathon and other projects.
