BrickAnything Converts Any 3D Shape into Building Instructions with Discrete Pieces

Converting an arbitrary 3D mesh into a set of assembly instructions with discrete pieces—the kind someone would use building with LEGO—is a problem that has been stuck between manual heuristics and blind sequence generation for years. A paper published on May 27, 2026 on arXiv (cs.AI) proposes a solution that approaches the problem from a different angle: treating geometry as an explicit model condition, not as a secondary objective.

The system is called BrickAnything and its premise is straightforward: given any 3D representation—whether a mesh, point cloud, or scanned reconstruction—the model generates an ordered sequence of pieces that, when assembled, reconstruct the target shape while respecting physical constraints of fit and stability.

The problem they solve

Previous methods fell into two traps. The first: heuristic optimization, which works well with shapes that fit predefined patterns but collapses as soon as the target geometry isn't "friendly" to those rules. The second: models that generate piece sequences without explicitly modeling the underlying geometry or assembly relationships, producing geometrically plausible structures that are physically unstable or directly impossible to build.

BrickAnything uses point clouds as a unified geometric interface: any 3D representation is first converted to a point cloud, allowing the system to work with heterogeneous sources without needing type-specific preprocessing.

Tokenization with structural awareness

The most interesting technical contribution in the paper is what the authors call structure-aware tree tokenization: instead of representing pieces as a flat list, the system organizes them as a tree of local adjacency relationships. Each piece is encoded in relation to the one above or beside it, not independently.

This has clear practical consequences. First, autoregressive generation is more coherent with the actual physical building process: you add one piece on top of another, not materialize the entire structure at once. Second, it reduces invalid intermediate states, which were one of the main problems in long sequences: a piece misplaced at step 12 could make everything that followed unfeasible.

The model also incorporates preference-based alignment, a fine-tuning technique that steers the model's outputs toward structures that evaluators—human or automated—consider better constructed, more stable, or more faithful to the original geometry. The paper doesn't fully detail this phase in the published abstract, but it's already a common element in systems that need to balance several competing objectives.

Who this is relevant for

The most obvious use case is assisted design in industries working with modular assembly: construction toys, modular architecture, rapid physical prototyping with standard pieces. But there's a second, more technical reading: BrickAnything is fundamentally a system of geometry-conditioned assembly planning, and that has applications in robotics and automated manufacturing where the pieces aren't LEGO but actual components.

For teams working on 3D content generation—an area where Claude is frequently used as a reasoning layer over external tools via MCP—this kind of research points to an interesting direction: it's not enough to generate valid geometry, you need to generate fabricable geometry. The difference between the two is exactly the problem BrickAnything tries to solve.

The research is still in preprint phase and doesn't include publicly available code or a model as of publication. We'll have to wait and see if the authors release reproducible artifacts before we can evaluate performance on their own benchmarks.

---

EP Editorial: BrickAnything is solid work that addresses a real limitation of previous systems with a technically sound solution. If tree tokenization proves as effective in practice as the paper suggests, it could become a standard component for any physical assembly generation system. The logical next step is seeing how it scales when target shapes are truly irregular or asymmetrical.