Infinity's Gate

How It Works:

The user has the option to either flip through portals created by other users, or create their own portal. To create a new portal, the user should enter a prompt which will be processed through two different AI services: Stability.ai (which uses Stable Diffusion) and Microsoft Azure Computer Vision.

The prompt will first be passed to Stability.ai to generate an image based on the text entered. The image that is generated is then sent to Microsoft Azure Computer Vision to segment the image by separating the background from the image. We then have two different versions of the image: the original, and the original with the background removed (the foreground). The original image is applied as the color map texture on the background layer, and the foreground image is applied as the color map texture on the foreground layer, and are both uploaded to Firebase Storage. Both layers create the perception of depth to the scene, similarly to how a pop-up picture book or diorama would.

The user can choose to re-generate an image if they aren't satisfied with the result until they create one they like. On the next page, they can add the name of their creation, and their own name. Once submitted, this information and the links to the images uploaded to Firebase Storage are stored as an entry in Firestore. By using Firebase to collect entries, this VPS installation becomes a collaborative space where users can see others' creations and contribute their own.

Tech Stack:

Needle Engine – Converts Unity scene into R3F Group React Three Fiber (R3F) – Allows me to build my Three.js scene using declarative self-contained components Vite & React Tailwind CSS – Styling library 8th Wall – XR environment where R3F scene gets added Firebase – Used for hosting the site & storing images generated by Stability.ai Stable Diffusion / Stability.ai – Used to generate texture maps via a REST API request including a prompt and pre-populated settings

How Development Works:

This project combines multiple tools together including Needle Engine, React Three Fiber (R3F), React, and 8th Wall. I used self-hosting with 8th wall so that I could create a more custom pipeline.

Needle Engine is being used to visualize the scene in Unity and then convert the scene into a Vite with R3F Three.js code. Within the source code it initially creates, I also add layers of my own Three.js-only code with R3F – such as the portal with the hider material. If the user has debug=true in their url query, it will load up this scene on your desktop as a plain webpage. If the user does not have this, by default, the scene will be loaded with 8th wall's pipeline instead. Within the 8th wall scene initialization code, I pass a reference to the group containing the R3F scene, and add it to the 8th wall scene. I also create some custom rendering and lighting settings that are 8th wall specific.