This interactive demo lets you experience what it’s like to explore a space using only sound. Each object and terrain feature is translated into unique musical cues that tell you where things are, how big they are, and how to move safely between them.
Pop on your headphones and start listening to the view.
Core Architecture
Three.js 3D Environment
- First-person perspective camera with 75° field of view
- 20×20 meter room with wireframe boundaries and grid floor
- Real-time rendering with fog effects for depth perception
- Mouse-look controls with pointer lock for immersive navigation
- Movement speeds: Walking (60 m/min) and Running (180 m/min)
Audio System Architecture
Tone.js Audio Engine
- Master audio bus with limiter (-8 dB) and compressor
- Five sub-buses: Melodic, Percussion, Proximity, Surface, and Sky
- 130 BPM tempo in 4/4 time signature
- 32-bar musical loop structure
Five Primary Sound Objects
- Human (Category 4)
- Sawtooth synthesis, 8-note patterns per bar
- Position: (-3, 0, 4), Height: 1.6m
- Tom pitch: C4
- Bird (Category 3)
- Triangle synthesis, 4-note patterns per bar
- Animated flight pattern (figure-8 movement)
- Position: (-2, 2.5, 2), Height: 0.4m
- Tom pitch: F4
- Car (Category 2)
- Square wave synthesis, 2-note patterns per bar
- Elliptical movement path
- Position: (6, 0, 0), Height: 0.9m
- Tom pitch: A3
- Tree (Category 1)
- Sine wave synthesis, sustained notes
- Static position: (-5, 0, -5), Height: 3.2m
- Tom pitch: D4
- Pillar (Category 1)
- Sine wave synthesis, sustained notes
- Static position: (0, 2, 6), Height: 4.0m
- Tom pitch: G3
Spatial Audio Features
Multi-Octave Vertical Mapping
- Screen divided into 5 horizontal zones (octaves 2-6)
- Objects spanning multiple zones play in multiple octaves simultaneously
- Dynamic octave calculation based on object height and camera perspective
HRTF-Inspired Processing
- Elevation filter: ±13.5 dB high-shelf based on vertical angle
- Pinna notch filter: Simulates ear-shape effects (8kHz ± 3.6kHz)
- Occlusion filter: Muffles sounds below/behind listener
- 3D panning with inverse distance model
Proximity Warning System
Tom Drum Indicators
- Near zone (≤3m): Regular pattern playback
- Touch zone (≤0.5m): Rapid 16th-note patterns
- Four complementary patterns to avoid congestion
- Individual tom pitches per object
Surface/Terrain Percussion
Four Surface Types with Indicators:
- Road (NW): Snare drum, plays twice per bar
- Path (NE): Rim shot with double-strike pattern
- Natural (SW): Shaker on beat 2
- Water (SE): Sustained cymbal sound
Special Terrain Indicators:
- Incline/Decline: Bongo/Conga plays every 500ms during ramp traversal
- Dynamic ramp visuals: Green (ascending) / Red (descending)
- Edge detection: Hi-hat within 3m of walls
- Drop hazard: Crash cymbal with proximity-based intensity
Dual Sky System
Weather States (Ambient)
- Clear: Sine wave drone at 110Hz
- Wind: Pink noise
- Rain: White noise
- Storm: Brown noise
- Elevation-based filtering and volume
Sky Melodic Items:
- Sun (Day only): Power chords on bars 1, 9, 17, 25
- Octave varies with elevation (4-6)
- Moves across sky during progression
- Clouds (Optional): Tubular bells on bars 3, 11, 19, 27
- Alternates between C and E notes
- Steady 8-note repetitions
Movement-Coupled Bass Drum
Velocity-Based Patterns:
- Stationary: No bass drum
- Walking (60 m/min): Beats 1 and 3
- Running (180 m/min): All four beats
User Interface
HUD Elements:
- Real-time position and rotation display
- Current bar indicator (1-32)
- Movement speed in meters/minute
- FPS counter
- FOV object count
Visual Object Status:
- 🟢 Green: Visible and playing
- 🟡 Yellow: Near (within 3m)
- ⚫ Black: Not in view
Proximity Alerts Panel:
- Lists all objects within 3m
- Color-coded warnings (yellow/red)
- Distance measurements
Control System:
- WASD movement
- Arrow keys for turning
- Shift to run
- Number keys 1-4: Weather states
- Key 5: Day/Night toggle
- Key 6: Clouds toggle
Technical Optimizations
Performance Management:
- Maximum 20 simultaneous objects
- Voice reduction algorithm for dense scenes
- Micro-delays to prevent phase cancellation
- Complementary percussion patterns to avoid clustering
Volume Management:
- Automatic gain reduction for multiple octaves
- Distance-based attenuation (30dB over 15m)
- Category-specific volume balancing
- Proximity-based voice prioritization
This engine creates a rich, multi-layered audio environment where every element serves both musical and navigational purposes, allowing users to build mental maps of 3D space through sound alone.