Pokémon Snap

At the end of April, Nintendo relaunched Pokémon Snap for their Switch console. As an aficionada of the original N64 game, I am absolutely thrilled. It’s hard to find non-violent video-games, let alone ones that legitimately foster artistic skills. Rock Band, Minecraft, and Animal Crossing are our other favorites.

Finch and I spent some time playing this weekend, out in the garden, deploying the motion capture feature that allows you to move the Switch around like a viewfinder — kind of like a lo-fi VR. We mostly talked about how the scoring system works, and we stumbled upon a question we really liked. How does the game see the photos you take?

The scoring system for photos was excellent in the N64 game, so the efficacy of the new edition is no surprise. The goal is to photograph each Pokémon, and each photo submitted earns a score. Points reflect how centered the subject is, how much of the frame they occupy, how fully their body is displayed, and how directly they face the camera. The background and extra features are factored in as well, but to a lesser degree.

The machine needs to know two things: how the photo is composed, and what features it includes. Each Pokémon is already modeled in the system — the machine that generates the scenery is the same machine that photographs it.

We considered here the idea of a blind person’s visual cortex, and how the brain adapts to whatever conditions its presented. This study from Georgetown confirms:

“We can see that in the blind, large parts of the visual cortex light up when participants are engaged in auditory and tactile tasks. This is in addition to the areas in their brain that are dedicated to processing sound and touch,” says the study's lead investigator, Josef P. Rauschecker, PhD, professor in the Department of Physiology and Biophysics at GUMC.

“This shows us that the visual system in the blind retains the functional organization that was anatomically laid out by genetics, but that the brain is plastic enough to use these modules to analyze input they receive from different senses,” he says.

A machine built to accommodate sight that cannot see still sees in its own way. Our best guess of how the game works is similar to how a blind person might take a photo of their home. This is a space they know very well — they arranged all the objects it contains. The objects are arranged to encourage a particular path and flow, accommodating habits and preferences. With an understanding of camera angles, a blind person could likely take photo of their couch just as their dog has jumped up, or a plant in the window when the sun hits. In the game, the static 3D models of the backgrounds, the animated 3D models of the Pokémon, and the player’s point-of-view are all known to the scoring system. The environment is dynamic but limited in scope, nicely balancing complexity for the game developers and players.

I’m so happy this game is back, and finding a new audience. There should be more like it.