From Our Neurons to Yours

Could brain implants read our thoughts? | Erin Kunz

• Wu Tsai Neurosciences Institute at Stanford University, Nicholas Weiler, Erin Kunz • Season 8 • Episode 8

Share episode

Imagine what it’s like to lose your ability to speak. You know what you want to say, but the connection between your brain and the muscles that form words is no longer functioning. For people with conditions like ALS, or who experience a severe stroke, this is a devastating reality. 

Today's guest is Erin Kunz, a postdoctoral researcher in the Neural Prosthetics Translational Laboratory at Stanford, who is part of a global community of scientists working towards the vision of a brain–computer interface — or BCI — to bypass those broken circuits and restore the ability to speak to people with paralysis.

We discuss how these BCIs work and the inspiring progress the tech has made in recent years, as well as the troubling question of whether a technology designed to decode what people intend to say from their brain activity could one day read out thoughts they never intended to communicate?

Learn More


Send us a text!

Thanks for listening! If you're enjoying our show, please take a moment to give us a review on your podcast app of choice and share this episode with your friends. That's how we grow as a show and bring the stories of the frontiers of neuroscience to a wider audience.

We want to hear from your neurons! Email us at at neuronspodcast@stanford.edu

Learn more about the Wu Tsai Neurosciences Institute at Stanford and follow us on Twitter, Facebook, and LinkedIn.

Nicholas Weiler (00:07):

This is From Our Neurons to Yours from the Wu Tsai Neurosciences Institute at Stanford University, bringing you to the frontiers of brain science. I'm Nicholas Weiler.

(00:24):

Imagine what it would be like to lose your ability to speak. You know what you want to say, but the connection between your brain and the muscles that form words is no longer functioning. For people with conditions like ALS or some people who experience a severe stroke, this is a devastating reality. But what if we could use technology to bypass those broken circuits, allowing people to speak again? Today, we're talking with Erin Kunz, a postdoctoral researcher in the Neural Prosthetics Translational Laboratory here at Stanford. Erin and her colleagues are part of a global community of scientists working towards the vision of a brain-computer interface or BCI that can do just that.

(01:06):

The basic idea is to use tiny electrodes implanted on the surface of the brain to record the activity of the circuits that normally control your speech muscles. Researchers like Erin can train machine learning algorithms to decode those signals in real time and display them as words on a screen or even produce synthesized speech. The ability to directly read out brain signals to translate them into speech is still restricted to research labs here at Stanford and elsewhere. But the field is progressing rapidly and some companies are taking interest in scaling up the technology, raising the possibility that many people with speech impairments could soon speak again using brain implants.

(01:46):

Now, as incredible as this scientific frontier is, it does open the door to a potentially unsettling risk. If we can decode what a person is trying to say, maybe even what they are intending to say, could we someday go further toward a world where computers can access what we are thinking about saying, or even what we don't mean to say? The idea of using technology to read minds has long belonged to science fiction, but it's getting harder to dismiss.

(02:17):

So I started my conversation with Erin today by asking her, "Are we anywhere close to being able to decode people's private thoughts using brain implants?"

Erin Kunz (02:26):

So the short answer is that no, we are not anywhere near that. The longer answer is that one of the questions that our study wanted to address was actually this question exactly. Can a speech BCI really read your thoughts? In this field, thought is a very overloaded term. What even is a thought? But we specifically wanted to answer the question of if your inner monologue is just randomly occurring as it does in whatever form it takes for you, would a speech BCI be able to pick up on those signals? And we found that there are very specific scenarios that it may pick up on those signals, but not to the extent where we would be able to decode freeform thought.

Nicholas Weiler (03:03):

Okay. Well, and I want to get into some of those nuances. And we'll dive more deeply into that study because the results are really fascinating both because of what they say about these concerns about if we're trying to help people who can't speak speak again, is there a risk that we're accidentally going to gain access to something that they didn't want to say? But also, as you said, what even is a thought, right? Is it words? Do we have an inner monologue? But I think maybe first, before we dive into all that, it would be great to do a broader overview of this field of what you're doing and who it's for.

(03:38):

So my understanding is you're working with patients who have difficulty speaking for one reason or another. And you've gotten really good at decoding what those people are trying to say using implants that are placed into the brain in regions of the brain that are actually controlling the movements of the mouth and the throat and the parts of the body that are responsible for speech. Who are some of the people who are volunteering for these studies? And can you give us a sense of the speech difficulties that they experience?

Erin Kunz (04:08):

Yeah. So in this particular paper and in the speech participants that we have broadly across Braingate, most of the participants have the neurodegenerative disorder of ALS. So that is a disease where you slowly lose all your voluntary muscles. And for some people, that can start with your speech muscles. So they lose the ability to speak first, which first makes it harder to speak, where they're either slower or it's not as clear as they once were. And that's called being dysarthric. And then eventually they might lose the ability to move their speech muscles altogether. And so they can't produce sound or move their mouth, speech muscles at all. And that's what we call being anarthric. We also have a participant who had had a stroke. Though she is dysarthric as a result of some of the damages left from the stroke.

Nicholas Weiler (04:50):

And so all of these patients have in common the communication between their brain and their muscles is the fundamental thing that's not working. Is that right?

Erin Kunz (04:59):

Exactly. The connection is either totally or partially cut. Even though those signals are coming from the brain and being sent to the muscle, somewhere along the line, the connection, it's not all the way there.

Nicholas Weiler (05:09):

And how are these people managing to communicate now with their medical team, with their friends, with their loved ones?

Erin Kunz (05:16):

So it varies a lot based on the participant. For example, one participant still has pretty good use of her hands. And so she actually communicates by typing on an iPad. Another participant, before being in the study largely communicated by caretaker interpretation. So his partner would watch his mouth move as he was trying to speak and be able to translate it, not always accurately, but mostly accurately. And that's just the result of knowing someone really well. And that worked better for him than trying to use some of the typical devices that track your eye movements as you're selecting keys on a keyboard.

Nicholas Weiler (05:49):

Yeah, I joined my producer Michael, on his podcast, Famous & Gravy, to talk about Stephen Hawking, who also had ALS, a slow developing version of that. Famously, had to communicate through, ultimately, towards the end of his life, moving the muscle under his eye and using that to select letters and type out what he needed to say that would then be read out by his famous synthesized voice. That's the kind of person who this technology is asking, well, is there a way that we could make it easier for these people to communicate with their friends and family? And you're doing that by having brain implants that are basically standing in in a way for the musculature that normally controls our speech by saying, "Well, the brain is saying do X, Y and Z. That would make this kind of sound, that should be this kind of word, and I'm going to output that on a screen or use a synthesized voice to say that."

Erin Kunz (06:45):

Exactly. So we can record the signals directly from the source, where they have not been, I guess you could say altered in any way yet, where they still represent the full richness of what they're trying to communicate.

Nicholas Weiler (06:56):

Can you tell us a little bit about those implants? How big are they? What do they look like? Where are you currently placing them in the brain?

Erin Kunz (07:03):

Yeah. So the implants we use, they're called Utah Arrays. They're microelectrode arrays. They're 3.4 millimeters square. So you can think of that as the size of a pea. And they have one millimeter silicon probes on them. These devices get placed on the surface of the brain. So during a surgery, a surgeon will-

Nicholas Weiler (07:20):

Like a hairbrush or a pincushion or something?

Erin Kunz (07:22):

Yeah, a very shallow pincushion.

Nicholas Weiler (07:24):

Yeah.

Erin Kunz (07:25):

So a surgeon will place these on the surface of the brain. And multiple can be put on. So between four and eight now. And these are mostly put on the motor cortex. So the part of the brain that controls your muscle movements. Whether that's in a hand area or in this case, the area of the brain that controls the muscles of your face and your larynx and your tongue.

Nicholas Weiler (07:45):

And why are these placed in the motor cortex?

Erin Kunz (07:49):

Yeah. So the history of brain-computer interfaces is largely that they were intended to restore movement where it had been lost. Historically, this was done in the hand knob area. So the part of the brain that controlled hand. And what they found when they would replace these in the hand area of people that had lost the ability to move their hands is that they could still pick up the motor signals of their hand movements when they tried to move their hands. So even though they couldn't move their hands, they were still able to differentiate signals when participants were trying to move their hands. And they could use these signals to control a cursor on a computer screen. So going into speech, it was known that the motor cortex, despite a person being paralyzed, the motor cortex still encoded signals of attempted movement. So the hypothesis was that speech motor cortex would still also encode movements underlying speech even if a person had limited, impaired or no ability to speak.

Nicholas Weiler (08:46):

And so a lot of the work in the field, if I understand correctly, has been trying to figure out, well, those signals are in there and somehow the brain and the muscles have their secret language of nerve impulses that allow us to do amazing feats of physical skill like walking down the street or speaking using all these different muscles to precisely articulate our words. The challenge has been how do we decode that? How do we get a computer to be able to figure out when those muscles aren't working anymore, what is the brain trying to say? So what is the approach to training the computer to interpret those signals coming out of the motor cortex?

Erin Kunz (09:26):

Yeah. So speech is a very complex high dimensional behavior as you could guess. You don't actively think about which muscles are doing what at any given time, and there's lots of muscles involved. The approach that we use is actually very similar to what an Alexa is using. So it's techniques taken from automatic speech recognition. Except, instead of it interpreting sounds, it's interpreting neural signals. So we use decoding algorithms. In this case, we're using a recurrent neural network that is decoding probabilities of phonemes over time. And a phoneme is the individual sound unit of speech. So in English, there's 39 distinct phonemes that all of our words are constructed of. And then we take these sequences of phoneme probabilities and those are fed into a language model. And so that language model can predict the most likely sequence of words given those phoneme probabilities, and it can output the text of the most likely sentence on the screen.

Nicholas Weiler (10:19):

Interesting. So there are a few stages here. So first, you're training this recurrent network on telling the person, "Say the sound, F, say the sound, P, say the sound, D." Right? And looking at the brain activity. And then it should be able to predict in future when it sees a pattern of brain activity, "I'm going to say that's a 80%, they're trying to say D." Or, "70%, they're trying to say F."

Erin Kunz (10:46):

Basically, the idea. Instead of prompting them with individual sounds to start, however, we're actually prompting them with full sentences. So they see a text of a sentence on the screen, they attempt to say it, we record that data, and we do that for several sets of sentences. And that's our training data. And so that we can label those sentences with the sequence of phonemes that they're constructed from. And that's how we train the model. And then going into a lot more detail, there's particular loss functions that you can use that can help with the adjustment of matching the sound unit to the neural data in time so it's flexible.

Nicholas Weiler (11:22):

So it was challenging at first, but now there are ways of adjusting it on the fly or adjusting it automatically.

Erin Kunz (11:28):

Yeah, which is important for this type of work because we don't actually have ground truths of when a person who can't speak is attempting to speak. So having this flexible CTC loss function helps address that issue.

Nicholas Weiler (11:40):

Okay. So it's trained on telling them to say a certain set of sentences, break it down into sounds, figure out this pattern of brain activity probably means this sound. And then reverse engineering that to say, "Okay, we've this pattern of sounds that we think they're trying to make, that ought to be these words." Well, so I guess now that we have a little bit of foundation, what is sort of the state-of-the-art performance of these systems? How good are these systems now for people who have speech difficulties?

Erin Kunz (12:08):

Yeah. So in the best case, the best demonstration today in a man with ALS is able to use a system like this every day for work and talking to his family. And the error rate... So in this sort of technology, we use error rate as one of the performance metrics. And so it means... So lower is better. And so that error rate is somewhere between 5 to 10%. So if it gets 5 to 10% of the words not quite right, it's able to output any of almost 130,000 words. So basically, anything that the participant wants to say, it can get those most of the time. And just to put into context what 5% word error rate is, that is the typical performance of your Alexa system or Siri system. They typically get 5% error rate of what you're trying to say.

Nicholas Weiler (12:52):

I'm now wondering what my speech error rate is because I know that I don't always get out the words that I'm trying to say. That's another topic maybe for another day. And how fast are these? Because I know that one of the challenges is we speak incredibly fast. I try to slow myself down on the podcast, but we speak incredibly fast. And I assume these systems are still trying to catch up to our natural speech rate.

Erin Kunz (13:13):

Yeah. Right now the speed can vary. So the fastest reported system, I think has gotten up to 90 words per minute. For context, conversational speeds can get up to 160 words per minute. And presentation style speech is about 120 words per minute. So 90 words per minute being the fastest is still not quite to the speed of typical conversation. But that being said, it varies a lot across participants. And the reason for this is because largely, the speed is participant driven. So it depends on their ability to actually attempt to produce speech. Some of our participants have reported that the actual act of attempting to move is like moving in quicksand or like you have a weighted blanket. So it's actually difficult for them to attempt speech. And so they're not able to do it as fast as they used to be. For example, this participant that has a really good error rate, his natural rate of speech using the system is closer to 50 words per minute. So it's still quite a bit slower than that conversational 150 words per minute rate.

Nicholas Weiler (14:12):

Well, that's a great transition actually, and a good point to bring up, which is that in most of these previous studies, what you're asking people to do is actually try to speak as best they can, even though the signaling between their brain and their muscles is not what it used to be. And the brain is interpreting what they're literally trying to send to their muscles. So they're trying to tell their mouth to do a certain thing, maybe it doesn't work, but the implant can figure out what they were trying to say. But in your new study, you're taking a different approach, which is what if we asked them not to try to actually speak, but just to imagine what they want to say. Was that the main purpose of this study, to try to see if it was possible to get better performance from these devices by not asking people to actually have to make that effortful quicksand attempt to use the muscles?

Erin Kunz (15:00):

Yeah, exactly. So the main motivation for the study was to see if we could bypass those physical limitations. So if a participant only has to think about what they want to say instead of actually attempt to move their muscles, one, that could be less tiring and more comfortable, and two, it could potentially be faster.

Nicholas Weiler (15:15):

But there's also another angle to this. The paper, it brought up the fact that people have expressed concern in the past that once we are getting into the business of decoding what someone is trying to say, doesn't that get awfully close to decoding things that someone is thinking but not trying to say? Was that also part of the motivation for the study to see is that a real risk? How big of a risk is that?

Erin Kunz (15:42):

Yeah, absolutely. So given that the most recent demonstration with Participant T15 showed basically open-ended conversational performance in terms of decoding anything he wants to say. That also opens up the question of, "Okay, so it can decode anything he wants to say. Would it also decode anything that he is thinking, that he might not want to say?" So that was another motivation was to begin to address this question of would a speech BCI that is now decoding from open-ended vocabularies also be able to decode your open-ended inner monologue?

Nicholas Weiler (16:15):

And that's something I want to get back to, which is this idea of an inner monologue and what that might look like in the brain. How realistic is that that we have this inner monologue? But I'm going to come back to that because first I want to understand how you did this study. So in this study you were trying to understand, "Okay, we know we can decode what a participant is trying to say. And we can do that pretty fast and pretty well with a large vocabulary. Now can we record just what they're thinking?" And you had a range of tasks that you had them do. Can you tell us a little bit about the different variations that you went through in this study trying to figure out where the system was successfully reporting what they were imagining and where maybe it wasn't recording what they were imagining?

Erin Kunz (17:01):

Yeah. So we started off by investigating just decoding between seven single-syllable words. So just really a diagnostic test of can we decode or not? And we did seven different behaviors. So the first of those was just attempt to say the word moving your muscles and vocalizing. Then we also tried dropping the vocalization aspect. So the specific cue was mouth the words like you're mouthing to someone across a room. So you're focused on the actual movement of the articulators and not actually the vocalization, which for a lot of our participants, breath control is impaired. So taking away that vocalization aspect makes it a lot more comfortable.

(17:38):

And then we tried several different types of imagining speech. These strategies were taken from literature on how people report experiencing an inner monologue. So the first is probably what the most common way people report an inner monologue, where they hear their own voice in their head. So we cued participants to imagine saying the word focusing on the sound of the voice in their head. And we also tried an imagined listening one, where you imagine someone else saying the word. So someone you know well. So this could be a participant partner or family member. Imagine them saying the word, focus on the sound of their voice. And the last one we tried was a more motor version. So without moving, imagine moving your articulators like you're saying the word. So focus on the way that your tongue and lips feel as if you were saying that word. And then we also tried some passive behaviors as well of just simply hearing someone say the word and silently reading a word.

(18:31):

And basically, the question we wanted to answer was how similar or different the neural activity during all these behaviors were to each other. If there was representation in this part of the brain, specifically, the motor cortex, which previously was always thought to encode the actual attempt to speech, attempt to move, as opposed to more passive or imaginary behaviors. So one, is this signal actually represented in the motor cortex? And two, if these signals are represented, how different or similar are they to each other?

Nicholas Weiler (19:01):

That's fascinating. I love all the different variations that you did like imagining hearing their own voice, imagining hearing someone else's voice, reading, listening, different versions of speaking. So what were your conclusions? Did all of these evoke activity? Were these things that the computer could decode into speech?

Erin Kunz (19:21):

So yeah, the short answer was that yes, all of these behaviors were represented in the motor cortex. The longer answer is that, it depends where in the motor cortex. So there's a recent finding in several different speech neuroscience labs is that there seems to be two different speech hotspots within the motor cortex. So instead of the motor cortex actually just representing purely muscle movements of coordinated speech, there actually seems to be specific sub-regions that have higher level representations of speech and sounds themselves. And so these higher level regions within the motor cortex were the regions that had representation for the imagined behaviors as well as even listening. So yeah. So basically, hearing the word dog, saying the word dog, imagining saying the word dog, all were represented similarly in these specific speech hotspot regions of the motor cortex.

Nicholas Weiler (20:10):

Interesting. I didn't realize that. So there's some regions that seem to be more directly connected to the muscle movements and some that are what? A plan, an abstraction of what those muscle movements are going to have to be, some echo of a sensory version of the speech. Do we know what those higher order motor regions are?

Erin Kunz (20:29):

Yeah. We don't know for sure, but we do know, for example, that there's the more motoric sub-regions that yeah, mostly focus on the actual movement of muscles. And then these higher level areas, they represent sounds more so than the muscles that make up those sounds. So they're slightly more, you could call it an abstraction of maybe this routine of typical sounds that we say maybe, versus individual coordinated muscle movements.

Nicholas Weiler (20:56):

So in that activity, you can read out not only if the person is trying to say something, but if they're thinking about saying something or imagining themselves saying something or hearing someone else say it or even reading it?

Erin Kunz (21:08):

Mm-hmm.

Nicholas Weiler (21:09):

So how does that affect the ability of the system to actually produce language? Does this suggest, as you were saying before, that maybe people could just imagine what they want to say and have the system do it and that could speed them up quite a bit?

Erin Kunz (21:21):

Yeah. Yeah. So the good implications are that yes, this likely means that people can focus on just imagining saying words. And if we train a system on imagined speech that it would be able to decode someone's imagined speech. The other implications are that because these are all represented in the same area we're recording from is that we need to carefully understand the difference between the signals so that the decoder isn't accidentally decoding something that's either a thought not meant to be said aloud, or for example, something someone's listening to. We also wouldn't want to decode that.

Nicholas Weiler (21:52):

Interesting. Right. So you somehow need to install or program the system somehow to recognize these distinctions and make sure that the person has control over what they're talking about. So tell me about... Listeners may not understand this question when I first ask it, I apologize, but you will in a moment. Tell me about Chitty Chitty Bang Bang.

Erin Kunz (22:12):

Okay. Yeah. So going back a little bit, the first step towards making sure we don't accidentally decode things that we don't want to decode is actually understanding the relationship between all of these signals, listening, imagining, thinking or saying certain words. So that was one thing we investigated was the actual difference between the signals. And we found that even though they all share similarities, they also have distinctions between them. So there is a large amount of overlapping information between all of them, but there are neural dimensions that distinguish between the intention to move or whether it's listening or imagining. And so Chitty Chitty Bang Bang was basically the concept of putting a mental password on a decoding system. So you can think of this as your trigger word for your... It's like saying, "Hey, Alexa." It's basically a, "Hey, Siri." But for your speech BCI. So the person has to think Chitty Chitty Bang Bang before the system will begin decoding anything.

Nicholas Weiler (23:08):

Interesting. So that way whenever they happen to be thinking about or listening to or something, it's not going to decode anything until they give the password, they think Chitty Chitty Bang Bang. And then the system gets primed to, "Okay, now you're about to try to say something. I'm going to decode that." And then presumably, you would say it again to turn it off.

Erin Kunz (23:26):

Exactly. So a keyword password system.

Nicholas Weiler (23:30):

How well did that work in the study? Was that pretty seamless for the patient to-

Erin Kunz (23:34):

Yeah. So it worked. The accuracy was nearly 99% accuracy for detecting the keyword on and off.

Nicholas Weiler (23:41):

I guess that makes sense. You probably picked the word Chitty Chitty Bang Bang for exactly that reason. Hard to mistake it.

Erin Kunz (23:47):

Exactly. It was highly unique and identifiable and not a word that typically comes up in conversation most of the time.

Nicholas Weiler (23:53):

And if someone needs to watch the film, then they'd have to find some workaround.

Erin Kunz (23:57):

Exactly.

Nicholas Weiler (23:58):

Okay. That makes sense. One of the reasons that you did this study was to assess this question of how concerned should we be about these systems picking up what people are thinking about. And to me that raises this question of, well, how much in the course of our mental lives are we thinking in words? Is this concept of an inner monologue how we really think? And is that something that neuroscience can tell us? I imagine that's a pretty difficult question to address.

Erin Kunz (24:29):

Yeah. So as you can imagine, studying someone's inner monologue is quite hard. We don't necessarily have ground truth other than what people report as their experience, which could be very different across individuals. And in fact, people in the course of conducting this, I've had many conversations talking about how people experience inner monologues. There's a full range of people that don't claim to have any inner monologue whatsoever. It's all just abstract feeling. There's people that are like, "Yes, I hear everything in my head. Even when I'm brushing my teeth, I'm hearing, "I am brushing my teeth."" There's people that are like, "Oh, I have multiple voices. So I hear myself and I can also hear family members all the time. I have imaginary conversations in my head." There's people that are visual. So they actually imagine words and letters or even people that are just visual in general, where they imagine images more so than actual words.

Nicholas Weiler (25:21):

And tell us, in what we've described so far, all of the things that you're having the participants do are still intentional behaviors. They're still focused on the task that you're asking them to do, even if it's just listening to someone else speak. Did you get any sense of whether the algorithm might be decoding incidental thoughts that happened to be passing through their heads?

Erin Kunz (25:45):

Yeah. So we did do a couple targeted experiments to try and get at this question, where we had participants do these other types of tasks that weren't speech related but were designed to naturally elicit inner speech. So one of these is basically counting. The participants were presented with a grid of shapes of different colors on a screen, where there was maybe 100 shapes on the screen. And they were instructed to count the occurrence of a specific color-shape combination in that grid. So this was a task where you couldn't necessarily eyeball it. So you had to scan the grid and sequentially count up. And this was just a silent task. And then in a subsequent screen, they would attempt to say the number that they counted. And they did this for several trials. They weren't cued with any type of mental strategy. So they weren't cued to use inner speech specifically or count up by ones or count up by twos or whatever. They were just told to report the final number that they came across.

(26:38):

The goal of this was we thought it might naturally elicit inner speech. And so to analyze this, we took the neural signals while they were counting that grid, and we passed it through the same decoder that had been trained on their intentional inner speech to see if we could pick up on this naturally occurring inner speech. And we found that while weak, we could decode numbers that were increasing above chance. So we think that that means that in this particular case where they naturally use inner speech to accomplish the task that a speech BCI would pick up on those signals, that those signals are similar enough to the intentional signals of trying to communicate with inner speech.

Nicholas Weiler (27:16):

That's the situation, and I guess this is why you picked this, where if you're trying to count stuff and keep track of it, you may be actually talking to yourself in your head to say, "Okay, 1, 2, 3, 4." And count those up and then remember it so that you can report out the result. So in that situation, it was decoding some of those numbers that people were saying?

Erin Kunz (27:36):

Exactly. Yeah.

Nicholas Weiler (27:37):

And so that would suggest that there might be other situations where particularly if someone has a particularly active inner voice, that a system like this might be picking that up as well?

Erin Kunz (27:46):

Yeah. So that was our best attempt so far at trying to decode a naturally occurring inner monologue. We also did try to cue participants with open-ended questions, and not to respond to them out loud, but three different types of questions. So one was a control of, "Clear your mind." So that was intended for them to not be thinking about anything. And then two were more verbal type of questions. So, "Think about the lyrics to your favorite song." Or, "Think about a phrase that someone in your family says often." And then the third type was these autobiographical questions of, "Think about your favorite food." Or, "Think about a vacation you took." Those are less tied to words basically. And so these were all... The only prompt was to think. And then we also analyzed that by passing the recordings through the same decoder that they used for the inner speech BCI. And basically, we attempted to see what outputs would come from that.

(28:37):

The decodings were largely gibberish with maybe some plausible things, but we don't have ground truth of what they were actually thinking. So we can't quantify it at all, of course. But we did quantify basically, the number of words that the system decoded during each of these types of cues. And we found that the cues that were either think about something verbal or think about an experience decoded more words significantly than the clear your mind cues. And then it differed between participants, but in some participants, the specifically, verbal cues resulted in decoding of significantly more words than the experience autobiographical cues.

Nicholas Weiler (29:14):

Suggesting they're telling a story in their heads, potentially, and-

Erin Kunz (29:18):

Yeah, exactly. Basically, the neural signals of what's happening when they're thinking about these specific things are similar enough to trigger the speech BCI to decode something as opposed to silence.

Nicholas Weiler (29:32):

It's really interesting. And I love again, that you have this approach of doing the spectrum of what are the different kinds of things people might be thinking about? And are we going to be more likely to accidentally decode some words associated with a verbal story or a narrative about your life compared to some other thing that you might be thinking about? I feel like the fear that comes up with this kind of research is a science fiction scenario, where somehow this brain technology is used to surveil people and create some sort of literal thought police. And I guess given what you're finding, does that feel like a plausible thing people should be worried about? Or is that pretty far distant from what we're actually capable of doing?

Erin Kunz (30:18):

I think at this point in time, that's pretty far distant from where we are. To reflect, we're using the highest resolution recording devices available. And at this point we're showing above chance decoding of numbers during a specific task.

Nicholas Weiler (30:35):

So we can hear people counting to themselves?

Erin Kunz (30:37):

Yes, that's where we're at. But I guess this is something that needs to be a question as the field progresses. And again, we're specifically in the motor cortex here. So we're zoomed in on one part of the brain. And there's this question of the limitation of the representation in the motor cortex. So it does seem like the more verbal, the more word-like or speech-like a thought is, maybe the more similar it is to actually volitional inner speech or outer speech, even where that starts to blend more and maybe picked up by a speech BCI. But we still don't know that at all.

Nicholas Weiler (31:11):

So you're in the motor cortex, you're still picking up the echoes of what people might plausibly say out loud, whether they're actually saying it or not. But the fear we need to think about, well, what happens if we were to put these implants in other parts of the brain? What else might be decodable? But I think, maybe this is obvious or maybe it isn't, this requires neurosurgery, right? These are implants that you're putting into people because they have a serious medical condition and you're able to put these implants in the lab to try to develop this technology. But without an actual implant, this is pretty much impossible, right? We're not getting this from an electrode on the surface of the scalp. That would be pretty challenging to imagine.

Erin Kunz (31:54):

No. Yeah, this requires a surgery and absolutely everything you said.

Nicholas Weiler (32:01):

I raised that just because I think it's important to remember that this is not something that someone could be passively recording brain activity and reading what words you're thinking about.

Erin Kunz (32:14):

No. Absolutely not. We're not at that. No.

Nicholas Weiler (32:14):

I know this is all work in progress, but I'd love to get your take on where you see this technology being in the next 5, 10 years. Are we going to start being able to read minds anytime in the near future? Or are we going to be running up against the limits of brain recording and machine learning technology and get a bunch of nonsense because who knows what is actually going on when people are imagining a past experience or thinking about a loved one or something like that?

Erin Kunz (32:43):

Yeah. I think there's many, many questions on many access that need to be answered. So one is the recording technology. Right now we're using the highest resolution currently available. If the limitation is resolution, we're not there right now. Then another access is the actual neuroscience function mapping. So is this a question about where these types of inner monologue thoughts are actually represented in the brain? We think that the brain speech motor cortex has these speechy hot spots that are somewhere above articulatory muscle control, but maybe below abstract thought. So something verbal or speech-like in nature that if you're having a thought that looks a lot more like speech, that it might be similarly represented enough that a speech decoder would pick up on it. The other question is also there's a lot of variation across brains that we're finding. So even across our four participants in this study, we found a lot of variation in where inner speech was represented along the motor cortex. And the exact anatomy of a person's brain and where you find that little hot spot in the brain is actually a very difficult problem too.

(33:55):

So someone wouldn't be able to easily just go in and find that hot spot and be like, "Okay, that's where I can decode this from." There's a lot of pre-processing and planning that has to go into targeting those precise spots within the cortex. And broader, as we move outside the motor cortex, it's unknown territory of really what's going to be represented where.

Nicholas Weiler (34:18):

So it's interesting to speculate and important for the future of the field to think through the potential more distant ramifications of this technology. But it's also important to remember who this is for today, who this is for right now. And these are patients who fundamentally are not able to communicate very effectively with their loved ones, with their families and so on. And finding the right balance where we're actually helping those people while staying aware of the fact that we need to be thinking about these privacy issues, these control issues, making sure this is serving patients and not being used in any nefarious way.

(34:53):

And so the last thing I wanted to ask you about is what is the status or what is the progress of this kind of technology for clinical use for patients? Is this the kind of thing where we should expect to see this in people with ALS or stroke or other kinds of disorders outside of the lab and into the clinic in the next few years?

Erin Kunz (35:12):

Yeah, that's a great question. I think this field has made a lot of progress in the lab setting and especially, in the last five years. And I think the neurotechnology industry has seen that and is really starting to grow rapidly. So there are several startups out there now that are running their own clinical trials. I think it'll still be quite a few years yet before these are actual consumer devices that someone can pick up in the clinic for non-research purposes. But for now, there's several different clinical trials that people with ALS or stroke can enroll in for continuing to develop this technology.

Nicholas Weiler (35:48):

It's very promising and exciting times and great to imagine a day when someone is still able to communicate despite having a disorder like this.

(35:59):

Well, Erin Kunz, thank you so much for coming on From Our Neurons to Yours. This was a fascinating conversation. As often happens, I feel like we've only scratched the surface here.

Erin Kunz (36:07):

Yeah. Thanks for having me.

Nicholas Weiler (36:10):

Thanks so much again to our guest, Erin Kunz, a postdoctoral researcher in the Neural Prosthetics Translational Laboratory in Stanford Medicine's Department of Neurosurgery, co-directed by Frank Willett and Jaimie Henderson. Erin was also formerly, a Ketterer-Vorwald Graduate Fellow here at the Wu Tsai Neurosciences Institute. To read more about her work, check out the links in the show notes.

(36:32):

If you're enjoying the show, please subscribe and share with your friends. It helps us grow as a show and bring more listeners to the frontiers of neuroscience. We'd also love to hear from you. Tell us what you'd love or what you'd like to hear more of on the show in a comment on your favorite podcast platform, or send us an email at neuronspodcast@stanford.edu. From Our Neurons to Yours is produced by Michael Osborne at 14th Street Studios with sound design by Mark Bell. I'm Nicholas Weiler. Until next time.