AI-powered Reconstruction of Destroyed Buildings

In the aftermath of destruction, when cities are reduced to rubble and communities displaced, a team of hackers asked a bold question: Could AI help rebuild what was lost?

That question became the foundation for a project that would go on to win a Microsoft Global Hackathon challenge. The project, AI-powered Reconstruction of Destroyed Buildings, quickly drew attention for its ambition and sense of empathy, resonating with both government agencies and Microsoft leadership.

As the conflict in Ukraine continued to ravage the country and displace millions, a team of engineers, designers, data scientists, and program managers felt compelled to act. During Hackathon 2023, they envisioned a solution to help local Ukrainian authorities optimize the planning and execution of reconstruction efforts in affected areas.

The team created a working prototype that used AI models to interpret and visualize 3D blueprints and construction data.

From vision to prototype

The team’s approach was both technically rigorous and creatively challenging.

Using emerging technologies—the latest OpenAI large language models at the time, hosted on Azure OpenAI Service and advanced 3D visualization tools—they interpreted and visualized the reconstruction of the destroyed House of Culture in Irpin. Then, they created an Azure web app where users can ask questions via a copilot-style chat to cross-reference 3D blueprints with building data and receive information about materials, layout, and designs to support informed reconstruction decisions.

During the Hackathon, the team achieved a significant breakthrough when they connected Retrieval Augmented Generation (RAG) and Azure Machine Learning with Universal Scene Description (USD) data, despite uncertainty regarding the feasibility of such an integration. This success enabled the GPT large language model to interact effectively with the data, allowing team members to ask questions and obtain insights through conversational AI.

The prototype was built using key technologies:

• OpenAI GPT Models: Enabled users to ask questions and interact with the data conversationally.
• Azure Machine Learning and RAG pattern: Powered the AI capabilities, supported by a pre-release version of Azure AI Foundry.
• 3D Model: Utilized a .GLB format embedded with .USD data for rich visual representation and .USD-A variant for copilot context.
• Azure AI Search: Semantic search index supported implementation of the RAG technique for a copilot-like chat.
• Azure Web App: Served as the central host for all services with Web UI implemented in WebGL and ReactJS.

Before the Hackathon, the team recognized that their original scope of full reconstruction was overly broad and data-intensive. They made a strategic pivot, shifting their focus to preconstruction optimization, which helped planners make smarter and faster decisions about what to rebuild and how. This adjustment allowed the team to narrow their data requirements, provide more actionable insights, and better align with real-world planning workflows.

This pivot became a turning point, enabling the team to deliver a working prototype by the end of the Hackathon and opening the door for deeper conversations with potential partners.

Real-world traction and internal transformation

Following the Hackathon, the project evolved into a catalyst for broader conversations about technology’s critical role in post-crisis recovery. The team’s innovative solution formed the foundation of a collaborative initiative between Microsoft and Ukraine’s Ministry of Digital Transformation. Ed Essey, Senior Director of Business Value, praised the team’s direct engagement with customers, noting, “This team provided real customer value by working directly with the customer to solve acute scenarios with pragmatic applications of technology.”

The team refined their Hackathon prototype, ultimately featuring it in a joint demonstration by Microsoft and the Ministry at the 2024 World Economic Forum Annual Meeting in Davos. The demo showcased how Industrial AI’s design, build, operate, and optimize principles can significantly accelerate digital reconstruction efforts and restore essential infrastructures.

Furthermore, the team’s solution became integral to “Arina’s Story,” an immersive documentary project owned by the Ministry of Digital Transformation and supported by Microsoft. Built using Unreal Engine and Unreal Pixel Streaming on Azure, the project blends technology, art, and storytelling to raise global awareness of the conflict’s impact on Ukrainian children. Visitors experience the poignant journey of Arina, an 8-year-old dancer from Irpin, within a virtually reconstructed version of the town’s House of Culture.

Where they are now

The AI-powered Reconstruction of Destroyed Buildings project team continues to expand upon their Hackathon innovations, exploring new projects and diverse applications in immersive environments—from historical asset preservation to visionary city planning.

Widely shared across Microsoft, the project’s narrative serves as a powerful example of purpose-driven innovation, demonstrating that even in the darkest moments, technology can illuminate pathways toward hope and recovery.