Wall Bop
Wall Bop is an audiovisual WebAR experience that transforms a mural by Kel Brown in Austin, Texas into an interactive musical memory game. Developed by Lauren Schroeder, the game invites players to tap on different sections of the mural, triggering animations and sounds, and challenging them to repeat a musical sequence correctly. Using Niantic Studio and Niantic VPS, Lauren mapped the game precisely to the mural’s location, blending the real and digital worlds in an engaging new way.
Behind the Build
Written by Lauren Schroeder
February 27, 2025
Introduction
This experience is a WebAR game created using Niantic Studio, November 2024
Project Structure
3D Scene
- Game: This holds the rest of the game components, as well as the main game.js script that handles game logic.
- VPS Location: This is the VPS location that will be scanned to start the experience. The position-dependent game objects are all child components of this location.
- Base Entities: Includes the Perspective AR camera and Ambient/Directional Lights.
- UI Entities: Comprises all user interface elements displayed on the screen, providing information and feedback to the player. A tutorial is shown between each challenge.
Assets
- Includes all 3D models and audio files used throughout the game. The Models folder contains blobs and animations, while the sound folder has sound clips, as well as the final song played when the game enters the win state.
Scripts
- game.js: This script handles game logic - it initializes the puzzle, plays the correct solution, and tracks progress. It triggers winning and restart logic
- blob.js: This script is used on each wall shape. It handles what happens after the blob is touched, by updating animation and playing a specific sound.
Implementation
Blob interaction
This is where the blobs can be pressed in order to animate and make sound.
world.events.addListener(eid, ecs.input.SCREEN_TOUCH_START, click)
First, we set up a listener to detect click events on the blob.
const click = () => {
world.events.dispatch(world.events.globalId, 'submitBlob', {
blob: schemaAttribute.get(eid).blob,
})
Upon click, an event called ‘submitBlob’ is dispatched, in order for the game to register the input
ecs.Audio.mutate(world, eid, (cursor) => {
// Ensure the component's audio sample is playing
cursor.paused = false
})
The Audio component is then set to play so that the registered sound file specific to that blob will play
ecs.ScaleAnimation.set(world, eid, {
autoFrom: true,
toX: originalScale.x * scaleAmount,
toY: originalScale.y * scaleAmount,
toZ: originalScale.z * scaleAmount,
loop: false,
duration: 200,
easeOut: true,
easingFunction: 'Elastic',
})
// Set a callback to scale back to original size
setTimeout(() => {
ecs.ScaleAnimation.set(world, eid, {
autoFrom: true,
toX: originalScale.x,
toY: originalScale.y,
toZ: originalScale.z,
loop: false,
duration: 200,
easeOut: true,
easingFunction: 'Elastic',
})
}, 200)
Game
game.js: setting up the game data
const sequence = ['w1', 'w2', 'w3', 'b1', 'b2', 'b3', 'b2', 'b3', 'r1', 'r2', 'r3']
const messages = ['FIRST ONE!', 'KEEP GOING..', 'IS THAT ALL YOU GOT?', 'DOING GREAT',
'YOU GOT THIS', 'OVER HALFWAY', 'WHAT NOW?', 'DOING GREAT!', 'ALMOST THERE', 'LAST ONE...']
The correct order of blobs is set in the sequence array, so that the game can play the test sequence in the right order. The sequence is also used to check for accuracy during the user input part of the game.
Linking other Game Objects
schema: {
w1: ecs.eid,
w2: ecs.eid,
w3: ecs.eid,
b1: ecs.eid,
b2: ecs.eid,
b3: ecs.eid,
r1: ecs.eid,
r2: ecs.eid,
r3: ecs.eid,
startButton: ecs.eid,
winEntity: ecs.eid,
},
The schema lets you link up game elements to the sequence IDs. It also pulls in the startButton and animated object that should play when you win the game.
Initializing Game State
ecs.defineState('onboarding')
.onEnter(() => {
const onxrloaded = () => {
world.events.addListener(startButton, ecs.input.SCREEN_TOUCH_START, handleStart)
}
window.XR8 ? onxrloaded() : window.addEventListener('xrloaded', onxrloaded)
})
.onExit(() => {
world.events.removeListener(startButton, ecs.input.SCREEN_TOUCH_START, handleStart)
})
.initial()
.onTrigger(startGame, 'gameStarted')
The onboarding state is triggered first, so that the event listener for the in-game UI can be initialized.
ecs.defineState('gameStarted')
.onEnter(() => {
ecs.Ui.mutate(world, startButton, (cursor) => {
cursor.text = 'LISTEN CLOSELY'
})
if (!restarted) {
world.events.addListener(world.events.globalId, 'submitBlob', (e) => {
handleBlobPress(e)
})
The game is then started, and a listener is added to events that get triggered when a blob is pressed
Evaluating Game Status and Input Results
function handleBlobPress(e) {
if (sequence.length > guessIndex + 1) {
if (e.data.blob == sequence[guessIndex]) {
ecs.Ui.mutate(world, startButton, (cursor) => {
cursor.text = messages[guessIndex]
})
guessIndex += 1
} else {
resetGame(world)
ecs.Ui.mutate(world, startButton, (cursor) => {
cursor.text = 'PRESS TO PLAY AGAIN'
})
world.events.addListener(startButton, ecs.input.SCREEN_TOUCH_START, handleStart)
}
} else {
ecs.Ui.mutate(world, startButton, (cursor) => {
cursor.text = 'YOU WIN!'
resetGame(world)
ecs.GltfModel.set(world, winEntity, {
paused: false,
})
ecs.Audio.mutate(world, eid, (Audiocursor) => {
Audiocursor.paused = false
})
})
}
}
- The function handleBlobPress handles the state of the game. It checks for win and lose conditions by keeping track of the current game progress.
- guessIndex is used to see how far into the puzzle the user is and check for sequence ID accuracy
- Once the user makes it to the end of the sequence, the win condition is triggered
- For the win condition, the game components audio sample is triggered, and the Win Entity’s animation is started
Playing the Correct Sequence to the User
const intervalId = setInterval(() => {
if (playIndex < sequence.length) {
try {
ecs.Audio.mutate(world, schemaAttribute.get(eid)[sequence[playIndex]], (cursor) => {
ecs.ScaleAnimation.set(world, schemaAttribute.get(eid)[sequence[playIndex]], {
autoFrom: true,
toX: 1.12,
toY: 1.12,
toZ: 1.12,
loop: false,
duration: 1000,
easeOut: true,
easingFunction: 'Elastic',
})
setTimeout(() => {
ecs.ScaleAnimation.set(world, schemaAttribute.get(eid)[sequence[playIndex]], {
autoFrom: true,
toX: 1,
toY: 1,
toZ: 1,
loop: false,
duration: 200,
easeOut: true,
easingFunction: 'Elastic',
})
}, 1000)
cursor.paused = false
})
playIndex++
} catch (error) {
console.error('Failed to play', error)
}
} else {
clearInterval(intervalId)
ecs.Ui.mutate(world, startButton, (cursor) => {
cursor.text = 'NOW YOU TRY'
})
}
}, 1000)
- A time interval is set in order for the game to play each of the sound samples in the correct order for the user to listen to
- The specific blob is referenced and triggered. This causes the blob’s sound and animation to play when needed.
Initial UI
- In order to use avoid limitations of the current UI Component, you can use Javascript to create a custom UI screen. This screen appears when the game starts and allows for basic instructions on how to play the game
// Start button
const startButton = document.createElement('button')
startButton.textContent = 'LET\'S GO'
startButton.style.marginTop = '20px'
startButton.style.padding = '10px 20px'
startButton.style.fontSize = '16px'
startButton.style.cursor = 'pointer'
startButton.style.backgroundColor = 'white'
startButton.style.color = 'navy'
startButton.style.border = 'none'
startButton.style.borderRadius = '5px'
// Button click event
startButton.addEventListener('click', (event) => {
event.stopPropagation()
document.body.removeChild(instructionsBox)
})
instructionsBox.appendChild(startButton)
document.body.appendChild(instructionsBox)
