In “Games Agency as Art,” philosopher C. Thi Nguyen posits that games are an art form wherein the game designer creates an environment, obstacles and goals and invites us to assume a new agency to pursue those goals. Just as a painting lets us record sights, music lets us record sounds and stories let us record narratives, games, according to Nguyen, let us record agencies.

According to Nguyen:

In games, we can engineer the world of the game, and the agency we will occupy, to fit us and our desires. Struggles in games can be carefully shaped in order to be interesting, fun, or even beautiful for the struggler.

…

Game designers don’t just create a world; they create who you will be in that world. They tell you what abilities to use and what goals to take on. In other words, games work in the medium of agency.

The essence of a game then is the construction of a world and rules and goals and challenges for us to manipulate with our agency.

But if that is so, the scope of the worlds that designers can create has expanded and continues to blossom since the advent of the computer and video games. Computers gave birth to a new type of game, and as computational resources have evolved, so too have those games. And now (to some degree) in today’s high-tech world, the environments and the challenges and the agencies we can take up are limited only by our imaginations.

And that’s where SIE’s Future Technology Group (FTG) comes in. We specialize in imagination.

FTG looks to a 7- to 10-year horizon for what the future of gaming will be, what it could be. Our job is really twofold. We first want to harness the technology of today to expand the scope of gaming and play. But more than that, we look to imagine what environments and agencies would be “interesting, fun and beautiful” (can we add “cool?”) in order to identify the technological challenges that must be overcome to bring them about.

Alas, clever trailblazers though we may be, we are not likely to produce technological revolutions in these profound areas all by ourselves. Ergo, FTG engages in academic collaborations with the premiere scientists and thinkers of today to guide and pursue the research endeavors that lead to tomorrow’s great games.

So, what are the basic themes of that research?

Well, first, understand that concretely speaking, agency is simply what you can do in the world of the game. In chess, you castle your king or move your knight. In Tetrus you rotate or slide a bunch of falling, square-composed 2D shapes before they clog up your inbox. In Horizon, you, as Aloy, engage in a quest to uncover your past while not getting vaporized by the Stalkers.

What are the domains of modern research – specifically for modern video games – that can make these worlds more realistic or otherwise embellish these agencies?

One of the key themes at FTG focuses on immersion. What are the advances, such as virtual reality or more engaging game controllers or more vivid surrounding audio, that make your environment more realistic, your agency more heart-pounding? What rendering, via technologies such as neural radiance fields and physics-based animation, evoke the awe reaction (we call it “Kando,” by the way – Kando is Sony’s core philosophy, which roughly translates into the power of emotional connection) when you play a game? And further, how can your personal preferences, game level skills, and playing style feed into the game to make the experience unique to you?

Another realm of FTG focus is in cloud computing. As you no doubt know, Sony Interactive Entertainment makes the best gaming console in the world in the PS5 (we’re not here to argue about this). But splendid though this machine may be, the sheer computation that we can imagine needing for massively multi-player games, or for rendering with super-resolution, or for the ballooning applications of machine learning algorithms in games will eventually exceed the capacity of even the best home devices.

Enter the cloud. Plenty of FLOPS there.

But then you’ve got to get moving images and sound to, and controller commands from, your living room seamlessly. And remember, we’re not Netflix. We can’t buffer.

The resulting challenges in configuring parallel computing machines to execute large, highly complex games and to solve the transmission latency problem are at the heart of our cloud gaming research plans.

One final preoccupation of FTG research involves the characters in games that are not actual players – the so-called non-player characters or NPCs. Note that these have been around at least since Pacman (did you know that the four ghosts in Pacman actually have four independent “personalities” and algorithms for chasing you?). In even the best “AAA” games of today, however, NPCs are scripted and their dialogues (mostly monologues) are based on conversation trees recorded by human actors.

Perhaps you’ve noticed, however, that there’s an explosion going on out there in the area of large language models. What that ultimately means is anyone’s guess. But certainly, in the not-too-distant future, not only realistic animation and realistic, emotion-laden voices but also realistic personalities with long-term memory and distinct histories and moods, engaging in open-ended conversation with you or with each other – those, and more, are coming. With the help of FTG!

So that’s a tad bit about what’s going on at FTG.

Allow me to conclude with one final point.

Scientific research for gaming needs to proceed in both a game-to-tech and a tech-to-game direction. By that I mean that games, as we know them today, naturally lead to research directions for improving realism or complexity or immersion. Here, critical resources of inspiration are our fabulous studios and the many award-winning games that they produce.

But in addition, scientific developments in the laboratory, particularly in the areas like machine learning, could themselves provide the foundation for new kinds of games – new kinds of agency in Nguyen’s phrasing – that have not heretofore been created. One thing the past tells us for sure is that fun is flexible. So at FTG we strive to be flexible too.