We’re excited to shine a light on Jesse Austin-Stewart and Music for PlayStation — a fascinating project that taps into our sense of touch to engage with music in an inventive way. While Jesse is not an employee of Sony Interactive or affiliated with the company, his efforts are part of a vast groundswell of momentum for the mission of accessibility in gaming. 

Jesse’s tracks are designed only to be felt, not heard. Whether hearing, hard of hearing, or d/Deaf, audiences engage with the music on equal terms, utilizing vibrations from the innovative haptic feedback capabilities of the DualSense wireless controller. In this profile, we spoke with Jesse about his inspirations for the project, the challenges he’s faced, the future of accessibility within the gaming industry, and more.

The Spark

Jesse started work on what has turned into the project at the tail end of completing his PhD in 2022. While the conversation circled around spatial audio, a topical feature in the commercial and home entertainment industries at the time, his perspective shifted elsewhere. “I was looking at it from the perspective of sound art,” he says. “I was curious how we can make spatial music within sound art more accessible through education and also in terms of disability and hearing.” 

From there he created spatial audio interfaces. One example being a plank to rest your arm on and feel different points of vibration. “The d/Deaf and hard of hearing participants I worked with found that really accessible for them. They found that it broke down barriers they’d previously encountered in the field.”

This was a real success in Jesse’s eyes but also invited more discovery.

Vibrant Discovery

Serendipitously, he stumbled upon an article about using the DualSense wireless controller as an audio interface. He began experimenting a bit with playing music from Spotify through the controller and noticed that, when hitting the right frequencies, you could excite the vibration motors significantly. Once he worked out the proof of concept, the creative spark fanned into a full development flame.

Jesse’s breakthrough in applying his studies to a practical, accessible, and technologically approachable accessory like the DualSense wireless controller is one of his proudest moments. “From a composer and a user perspective,” he notes, “this was the proudest moment. The kind of work that I was doing became more accessible with an easy piece of technology for people to access, whether they be composers or users. The initial prototype was limited but the DualSense [wireless controller] is plug-and-play”. 

With a new outlet for his conceptual application of music using the controller, the challenge would be to refine the hardware to output the precise vibrations and sensations that mirrored the audio components of music. “When you’re listening to a song,” he said. “You have the benefit of different instruments making different palettes of sounds. How does that translate through the context of the DualSense [wireless] controller’s capabilities?” Jesse expanded his train of thought and approached that hurdle by experimenting with varying frequencies, feedback, and feeling the output. This blossomed into what his project is now, a collection of musical vibration pieces developed with intentional collaboration with d/Deaf and hard-of-hearing artists.

Inspiration & the Future 

Inspiration takes many forms, and the music industry itself was an area that motivated Jesse to apply his studies in the direction of accessibility. Citing a range of artists including Christine Sun Kim, a Deaf sound artist who regularly includes studies notation, written language, and American Sign Language in their works, Jesse seeks to join their discussions of sound and how it impacts society. “Outside of Christine, there’s Suranga Nanayakkara, Ene Alicia Søderberg, Blake Johnston. These artists all engage with haptics and/or vibrational hearing in various ways,” he says.

When it comes to the immediate next steps, Jesse is looking forward to bringing his thought process and results to a larger audience. “I am exhibiting at a conference for music in the Faroe Islands soon,” he says. “I’m keen to exhibit in a bunch of places, increasing the exposure so that others can take the idea home to think on it too.” Jesse invites and encourages everyone to innovate if they’re inspired by his work. “That’s where it would become really interesting to me, to see other people’s take on what they would do for the controller in the same vein.”

His desire for more exposure is complementary to work he sees elsewhere, as Jesse spoke glowingly about the work being done by experts who specifically focus on accessibility. He also noted that lengths taken across the industry and at Sony Interactive have been encouraging, highlighting the possibility of pushing forward with a collective and inclusive effort. 

“Generally, we have seen some incredible progress in terms of gameplay and visual accessibility over the last 15 years or so,” he shared.  What Jesse hopes to see is not only more agency given to people who would benefit from additional accessibility options in any format — be it visual, tactile, or audio — but also involving people with disabilities in the invention process. “Agency is critical, so continuing to have advocates involved in discussions and decision-making processes is integral.”