An AI decoder can rebuild what you're looking at from a brain scan, and needs far less data to do it

A new brain decoder built by researchers at the Weizmann Institute of Science can take an fMRI scan of someone looking at a picture and reconstruct a close visual match: same layout, same colors, same objects in roughly the same place. The system also runs in reverse, predicting what a person's brain activity should look like given an image, a capability the team used to manufacture far more training data than any single brain scanning study could ever collect.

Michal Irani, the computer scientist at Weizmann who led the work with colleagues, presented the results last month at the Cognitive Computational Neuroscience conference in New York. She says the goal is not party-trick mind reading but a tool that could eventually help locked-in patients communicate or let scientists study phenomena like PTSD flashbacks. "Mind reading" is a "cute, jazzy name" for what the system actually does, she says.

Prior attempts to reconstruct seen images from brain scans tended to produce the right general category of object in the wrong place, shape, or arrangement. Irani says a banana shown to earlier systems might come back as a banana, but "it wouldn't have the same structure, the same position." Her team's decoder splits the problem into two branches, one predicting where colors and shapes fall in an image and the other predicting its content, such as a bunch of bananas on a plate. Both predictions feed a diffusion model, the same class of system that powers modern AI image and video generation, which then renders the reconstruction.

The underlying data came from eight volunteers who viewed roughly 9,000 images each while lying in high-resolution fMRI scanners, with each voxel covering about one cubic millimeter of tissue rather than the three cubic millimeters typical of standard scanners. That is still a small dataset by deep learning standards, so Irani's team trained a second model, an encoder, to predict what a person's brain scan would look like for any given image. Running the encoder and decoder together let the researchers bootstrap training on images that were never shown to anyone in a scanner at all; Irani says about 70 percent of the system's training data was generated this way rather than collected directly.

Pooling scans across multiple existing studies also let the team spot brain regions that appear to respond consistently across different people, including one area tied to images of food and another to images of sports. Irani, who trained as a computer scientist, says she is now working with neuroscientists "to see if we can actually use these tools that we've developed to really find out new things about the brain."

A separate practical gain is how little new data the system needs to work on someone it hasn't seen before. Earlier decoding tools generally needed about 40 hours of fMRI scanning on a new subject before they could predict what that person was looking at. Irani says her decoder needs about one hour. Tommy Sprague, a neuroscientist at the University of California, Santa Barbara who was not involved in the research, says that difference matters for working scientists: "None of us can afford 40 hours of imaging for a new subject. It's something like $600 to $1,000 an hour."

The system still fails in recognizable ways. Irani has shown colleagues a reconstructed cake that came out as a pile of three sandwiches, and a dog in a bathtub that the model rendered as a similarly colored goat in the same tub. In head-to-head comparisons against previously published decoders, though, Irani says her team's tool came out well ahead: "All in all, really we outperformed the others by a significant margin." Judy Illes, a neuroethicist at the University of British Columbia who was not part of the study, calls the work "magnificent" and says using the approach to help people with neurological conditions is "tremendously exciting."

What the work has not yet shown is any reconstruction of imagined or remembered content, only images a person was actively looking at during a scan. Irani says extending the method to video, audio, and eventually dreams or mental imagery is the next target, but "that's something we don't have yet." Sprague thinks the current approach would probably work reasonably well on imagined images too, which is part of why he and others are flagging mental privacy concerns well ahead of any such demonstration.

Those concerns sharpen considerably if the method moves from fMRI, which requires a person to lie still inside an expensive scanner, to EEG, the electrode-based measurement of electrical brain activity that can be collected through a cap or even headphones. Irani and other groups are already working on EEG-based decoding. Marcello Ienca, a neuroscientist and philosopher at the Technical University of Munich, says a working EEG version would be a significant shift: once a device is calibrated to one person's brain, a company could potentially extract additional information from that signal without consent, and he can imagine courts eventually weighing mental image reconstructions as evidence. "I have no doubt that this is, you know, well-intentioned research, but I think it's also pretty obvious that it could be co-opted for ethically and societally problematic commercial uses," he says.

Sprague puts the shift in blunter terms: getting a willing, cooperative volunteer to hold still in a scanner for a research study is hard enough, so a decade ago the idea of someone's brain activity being read involuntarily seemed far-fetched. He now believes the field needs to take the ethics more seriously as the underlying models keep improving. Irani acknowledges the misuse risk around EEG but says that for now she is choosing where to point her attention: "I'm trying to think only of good things."