Apple's Matrix3D: 2D to 3D

Apple's Machine Learning team has unveiled Matrix3D, an AI model capable of generating 3D scenes from as few as three 2D photos. This innovation streamlines photogrammetry by unifying pose estimation, depth prediction, and view synthesis into a single process, unlike traditional methods that require multiple models. Developed in collaboration with Nanjing University and the Hong Kong University of Science and Technology, Matrix3D uses a multimodal diffusion transformer (DiT) architecture and a masked learning strategy, enabling it to fill in missing data and train effectively with smaller datasets.

Matrix3D holds promise for various applications, including enhanced photo editing, AR/VR content creation for devices like the Vision Pro, and improved 3D product visualisation for online shopping. By simplifying the creation of 3D assets, Matrix3D could also benefit game developers. Apple has made the model available on GitHub, potentially allowing other AI companies to develop their own versions. While its integration into Apple Intelligence remains uncertain, Matrix3D represents a significant step forward in AI-driven 3D reconstruction.

The model's ability to estimate camera positions, generate depth maps, and synthesise novel views from limited images addresses key challenges in photogrammetry. This advancement could lead to more efficient and accurate 3D modelling across various industries, reducing the reliance on extensive image collections and complex multi-step processes. The move aligns with Apple's focus on AI research and its potential to enhance user experiences across its product ecosystem.