LegoGPT: AI Turns Text Prompts Into Lego Creations

Researchers at Carnegie Mellon University have introduced LegoGPT, a new AI system that builds real-world Lego creations from written descriptions.

It’s the first AI of its kind that not only follows a text prompt—like “a streamlined, elongated vessel”—but also ensures the resulting structure is physically stable and can be built, brick by brick.

“To achieve this, we construct a large-scale, physically stable dataset of LEGO designs, along with their associated captions,” the team explained in their research paper.

LegoGPT was trained using over 47,000 stable Lego models paired with detailed captions generated by GPT-4o. These were built from 3D shapes, turned into Lego structures, then tested for real-world stability using physics simulations.

Each structure was also described from 24 angles so the AI could learn what various designs should look like in words.

The team used a special technique called “physics-aware rollback,” where unstable parts of a design are removed and rebuilt until the whole structure holds up. This improved build success rates from 24% to 98.8%.

The AI model, based on Meta’s LLaMA-3.2-Instruct, predicts which Lego brick to place next—similar to how ChatGPT predicts the next word. Every suggested brick is checked for placement, size, and potential collisions before being added to the model.

LegoGPT’s creations can be built by both humans and robots. “Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing Lego designs that align closely with the input text prompts,” the researchers wrote.

For now, LegoGPT uses just eight basic brick types and works within a 20×20×20 space, but the team hopes to expand it.

Their full dataset, code, and model are free to access, so others can keep building on this research. Alternatively, you can just play around with their demo.