MCW Researcher Harnesses AI to Pinpoint Mental Imagery in the Brain

When remembering a wedding you attended, your brain might conjure up a detailed visual scene of food, speeches, and dancing.

Just how your brain creates these images is still a mystery, but researchers are one step closer to understanding which parts of the brain help reconstruct visual scenes from memory.

In a study conducted at the Medical College of Wisconsin (MCW) and published in Science Advances, researchers combined brain images with AI tools to identify which brain network represents visual elements of memories.

Not only did they identify the medial temporal subsystem of the default mode network (MT-DMN) as a key area of the brain where visual memories are summoned, but they also captured differences in brain activation associated with a person’s individual version of a scene.

Understanding how the brain represents scenes from memory could ultimately help researchers characterize differences in mental imagery between people, including clinical populations that are affected by intrusive imagery, says Andrew Anderson, PhD, who led the study.

“We now have direct evidence that this network is involved in visualizing scenes from memory,” says Dr. Anderson, assistant professor of neurology. “And we’ve shown that using these AI tools, we can test hypotheses about the brain in a new way.”

Using AI to Connect Patterns Between MT-DMN and Visualizing Memories

Researchers had suspected that MT-DMN plays an important role in imagining scenes but lacked direct evidence linking its activity to the visual structure of those scenes.

While researchers use fMRI scanners to measure which areas of the brain are active when a person remembers a scene, it is much harder to determine whether those activity patterns reflect the visual content of what that person is imagining.

“It’s something that is difficult to study,” Dr. Anderson says.

To determine whether the visual content of memories is reflected in this network, Dr. Anderson and his team harnessed the power of AI tools that could find patterns within the data and match them across data sets.

For the study, 50 participants were asked to imagine 20 different scenarios, including shopping or going to a wedding, that drew on their own experiences. Each participant gave a brief verbal description of the mental image. Participants then underwent an fMRI scan, where they imagined the same 20 scenarios without speaking. There, researchers recorded the patterns of brain activity associated with the different memories.

But such patterns are difficult to interpret. They “are a patchwork that you’re not going to immediately understand,” Dr. Anderson says.

To connect the patterns with the visual content of participants’ memories, Dr. Anderson and his team inputted participants’ verbal descriptions into Stable Diffusion, an AI model that turns text prompts into images. Stable Diffusion acted as a sketch artist for the memories, generating five images per scenario for each participant.

Though the images were not exact renditions of what participants remembered, they provided visual models that captured plausible features of the scenes they described.

The research team then fed those images into another AI tool called a convolutional neural network (CNN). The CNN analyzed the images to find visual features associated with people, objects, buildings, and landscape elements and translated those features into numerical patterns.

Researchers then used those patterns to measure how visually similar each imagined scenario was to the others. They asked whether the same pattern of similarities appeared in the participants’ fMRI data – for example, whether scenes that were more visually similar according to the AI model also produced more similar patterns of brain activity.

When the pattern of similarities in the image analysis matched the pattern of similarities in the fMRI data, “we’ve got an argument that this region of the brain is reflecting that visual information.”

The researchers also tested the visual model against 16 other brain networks, but the MT-DMN showed the strongest and most selective match to the visual structure of imagined scenes.

“This method does not directly tell us how the brain works, but it provides a means to probe for this visual information in particular brain networks,” Dr. Anderson says.

Laying the Groundwork for Future Communication

The visual models also captured differences in brain activity among individual participants. In the future, this approach could help quantify and compare differences in mental imagery across people, including in clinical populations where visual imagery may differ, such as in conditions involving intrusive imagery.

Dr. Anderson hopes the approach could ultimately be extended to help people with communication disorders, including people with post-stroke aphasia who have difficulty finding the words to express their thoughts and memories.

“If we can eventually learn how to interpret self-generated imagery from brain activity, it could open the door to new means of supporting communication about memories,” he says. “In principle, if we could understand what someone wants to express by interpreting imagined content from patterns of brain activity, we might one day be able to translate this into words for them.”