Undergraduate project · 2025–2026
Mind Reality
Mind Reality is a mixed reality (MR) app for HoloLens 2, built on LLM agents and realtime voice AI. You describe an object out loud. Agents write it as code, check it from 16 views, give it motion and place it in the room, where you can touch and move it.
- Award
- Grade A award in the Department of Communication Engineering's undergraduate project competition, academic year 2025–26
- Team
- 3 people
- Time
- 2025–2026
- Stack
- HoloLens 2
- Unity
- LangGraph
- Three.js
- Realtime API
- TypeScript
- Python
- Node.js
- SQLite
01 / Idea
Speak, and it appears
3D modeling is slow and needs special skills. In Mind Reality, you say what you want.
The system writes the object, checks it, gives it motion and places it in the room. You can touch it, drag it and walk around it. In the demo, users made gears, a valve, a mechanical bird, a windmill and a clock, and each one moves.
02 / My part
What I did
The team had three people. I built the parts below.
- The multi-agent architecture in LangGraph
- Object generation with Three.js and the 16-view check
- The Unity and HoloLens 2 side
- The realtime voice agent
- The report and the demo film
- The poster, with my teammates
03 / System
A voice in front, agents behind
A realtime voice agent talks with you and gives each job to a sub-agent built with LangGraph. Each sub-agent sees only its own task, so its prompt stays small.
04 / Objects
Code as the 3D format
The model writes Three.js code instead of a mesh. Code is text, so the model can fix one part instead of starting over.
A tool server runs the code in a headless browser and exports glTF for Unity.
05 / Quality
Sixteen views before it goes into the room
The tool server takes 16 views of each object from camera positions on a double spiral around it. A multimodal model checks the views for defects such as clipping or bad proportions. A failed object goes back to an agent for a fix, up to a set limit.
Below are 4 of the 16 views of a wizard that the system generated.
06 / Motion
Animation by plan
The animation agent writes a C# planner that may only call a fixed action library. Unity compiles it at run time with Roslyn and attaches it to the object, with no rebuild.
07 / Scene
An agent that looks first
The voice agent can take a photo with the HoloLens camera. It reads the room and suggests objects that fit it.
08 / Credit
Built on the lab's platform
The multi-user mixed reality streaming comes from LARR, the lab's remote rendering platform (Zeng, Huang and Huang, IEEE ICC Workshops 2023). The team built the generation pipeline on top of it.
With LARR, two users see the same objects from both sides of the table.