[00:00:01] Speaker A: Noise is really sort of signal we don't understand yet, right? For the most part, I think most of the noise that we see in the brain is not true noise in the sense of Brownian motion of molecules determining some, you know, baseline physical noise limit, but instead is processes that are going on in the brain that we just aren't controlling or aren't aware of. The key intuition it gave me was that some of the things that we might ignore in single neurons are actually really interesting parts of a population's behavior. The sort of classic difference is how flexible humans are, how much they can sort of generalize between different situations. And I know a lot of people are working on this issue in AI, right? But I think this one is going to be a vexingly hard one. And I think the kind of generality that they're building is small compared to the true generality of our intelligence.
[00:00:57] Speaker B: How often do you walk away from a conversation about what's going to happen next and just think, well, that's not going to work.
This is brain inspired.
The following is a reenactment of a.
[00:01:27] Speaker A: Purely fictional event that never happens in neuroscience.
[00:01:31] Speaker B: Juul.
Juul, wake up in there.
We just need a few more trials today.
Juul, no, not that awake. Settle down a little bit. Again, purely fiction. Hey everyone, it's Paul. And today I have Matt Smith on. Matt runs his lab at Carnegie Mellon University where he studies how we process visual information and how that information eventually transforms into motor decisions. To do that, he uses multielectrode recordings of tens and or hundreds of neurons, while non human primates perform tasks. And today we talk about two main things. One, we talk about the nature of variability in neuron activity, especially the variability during a task from one trial to the next. A neuron might respond a lot in one trial and a little in the next. Even though the animal made the same decision on both trials. This variability is often chalked up to noise that's irrelevant for the mental function we're interested in understanding. But as Matt describes, recording so many neurons at a time has dispelled that notion of variability as just being noiseless. The other main thing we discuss is variability on a slightly longer time scale, something Matt calls slow drift, which was alluded to in the award winning fictional reenactment a second ago.
We all experience ebbs and flows in our internal cognitive state as we move throughout the day. And Matt and his lab tracked this kind of thing behaviorally and found there's a global neural signature for it as well. And they have Some ideas for what that means and how the brain deals with it. And of course, we talk a bit about how all this might relate to intelligence and AI in general, and lots more with a few guest questions as well. This episode is an example where Patreon supporters hear an extra big chunk, where we go more in depth about Matt's experiences and challenges, getting to where he is now in his career, and a few other things. So if that kind of discussion interests you, consider a small monthly contribution on Patreon. You can find Matt on Twitter, Mithlab Neuro, and I will link to that and all the other relevant Stuff at BrainInspired Co podcast 89. And as you'll hear in a second, if you're thinking about going into neuroscience, maybe applying to neurophysiology labs, you'd be lucky to call Matt your mentor.
How long has it been since we actually spoke to each other? Do you recall?
[00:04:14] Speaker A: In person?
[00:04:15] Speaker B: Yeah. I mean, we've texted and we're always sending each other funny emojis and stuff.
[00:04:18] Speaker A: Yeah, yeah, I do like the funny emojis.
Maybe four years ago, maybe.
[00:04:25] Speaker B: It might be. But it's like seeing you like, you know, yesterday. I guess the emails and texts help as well.
[00:04:30] Speaker A: Right?
[00:04:30] Speaker B: Thanks for coming on my show. Of course. It's very good to see you.
[00:04:33] Speaker A: Yeah, good to see you too. And thanks for having me.
[00:04:35] Speaker B: I don't ever start an episode this way, but I recruited a few people to ask a question of you. I'm just going to start off the episode playing you a question from one of these people. Hi, Paul.
[00:04:49] Speaker A: My question for Matt is if he.
[00:04:51] Speaker B: Has any advice on mentorship and running a lab.
[00:04:54] Speaker A: He made it look so effortless, but.
[00:04:56] Speaker B: I'm finding it close to impossible. So any tips he might share would be appreciated. Thanks. That would be one Adam Snyder, whom I'm sure you recognized, but I enlisted him to ask you a question and I was surprised that. I mean, I knew that he was kind of recently faculty, but it's always good to start by buttering up my guest. And it's a genuine question. Hey, Matt, how are you? So great. How do you do it?
[00:05:28] Speaker A: Certainly not by thinking I'm great in any is the first answer. Right.
[00:05:33] Speaker B: We all knew that.
[00:05:35] Speaker A: Right? You're very, very well in tune to my not greatness, Paul. That's. That's good.
[00:05:41] Speaker B: I am very in tune to your humility. That's true.
[00:05:45] Speaker A: Okay, okay, fair enough. Yeah. Running a lab, I mean, I would say it was maybe all the things I expected and at the same time, you know, so many things that I didn't. Right. So certainly getting to do the science that I love and getting to work with people really closely on that and getting to sort of come up with the next question and then follow them. Those are all the things I knew it would be. Right. And it is. And those things are great.
Some things are a little bit hard, but I knew they were coming. Working on grants and sort of drudging through sort of maybe committees or things that you kind of knew you would have, like administrative work that you might not necessarily always want to do, but you knew would happen or had heard would happen. Those are true, too.
But I think the things that were not expected at all were how much, I don't know, managing time and people thinking about how to interact with people optimally and how to arrange your schedule and their schedules and how to get the most out of a meeting and how to move between different topics so quickly. All of those are not things that I thought a lot about. And as a postdoc, you certainly aren't really prepared for that. In fact, you're sort of the opposite of prepared for that often because you're so focused on one topic and one project, and then you move from that to suddenly having to be so focused on many projects. Right. And that's very different. So I don't know how I prepared for it, but I would say it was very different than what I thought at that level.
[00:07:18] Speaker B: I'm going to repeat his question to you, but before that, I want to ask you if that is so. That's one thing I have a little fear of missing out or regret of missing out, maybe by not trying to pursue a faculty position. Not that I was really ready for it, but I feel like I would have been a lot happier with multiple projects and in that maybe role. And I don't know if I was ready for it. But is that something that you're enjoying, surprisingly, or because you said it's kind of a new thing?
[00:07:47] Speaker A: Yeah, I think that's something that I hadn't fully appreciated how much I would like.
And I guess I've seen a lot of different labs. Some there's maybe the archetype of the absent PI, who's never around and kind of bumbles into a meeting and says something and walks out. And then there's some other archetype of the early PI who is basically sitting there with their student doing the project together or something like that. And I guess I have found a lot of joy in this sort of intermediate level where you're somewhat abstracted from the exact motions of the research. Not totally. I like to stay connected. But somewhat. You don't have to do it every day. You're not the person who does each step every day. But at the same time you get to move between different topics and think about how they connect and talk to other faculty and read other research and think about bigger connections that you maybe didn't have time to think about.
[00:08:43] Speaker B: Before Adam's specific question, I'm going to just play it again.
[00:08:47] Speaker A: Sure.
[00:08:48] Speaker B: Just to revisit, my question for Matt is if he has any advice on mentorship and running a lab. He made it look so effortless, but I'm finding it close to impossible. So I reached out to him and asked him if he had any question for you. And I told him what we were going to be talking about. And he probably, I don't know, 30, 35 seconds. I mean, this was a 12 minute long clip that was full of crying and tears of joy that I whittled down to this thing. But he really. And then I asked him just to follow up, what do you think Matt's going to say? And he said, I'm reading what he responded to me. I'm sure he's going to say he got lucky with a great team, which is probably true, but I'm sure that takes some skill also. And then I have this long paragraph about really just how you made him feel in the lab. Like he never felt like he was imposing on you. You always seem to have time for him and you seem to have it all together, which we both know isn't true.
[00:09:42] Speaker A: I was crying right before the start of this.
[00:09:46] Speaker B: But he is, like he said, sort of mentally struggling to think about how to do these things as well as you seemingly did. And that means you did do them well. And so he's wondering if there's some magic. What's the magic dust that you consumed?
[00:10:01] Speaker A: Yeah, I would say, okay, so something that served me well, maybe, I think as a PI, is that I think I'm a pretty low key guy. I don't go too much with the highs and lows of the moment. And I think that helps in a mentor. I've had mentors and experiences with people where you show them one data slide and they're jumping to huge conclusions or you tell them about some result that didn't work out and they're, oh, no, this is a travesty. How will we go on? I feel like that's not, for the most part, that level of either Excitement or depression after anything that happens in a lab is just not called for. Right.
We're all doing this. This is our job. It's important to all of us. But we're also people who have to interact with each other and get through our days and work together. And there's some sort of maintaining an even keel that I think is part of my personality and how I get through life. And maybe that serves me well in a lab too.
[00:11:00] Speaker B: Well, you have to show I agree with you about the even keel. See, knowing you a little bit, I can agree or disagree with you about your own personality traits, which is not usually the case on an episode. So that's fun. Fair enough.
But if someone comes into your office, I mean, you need to show some level, and we're not going to talk about this for very long, and we'll come back to mentorship and stuff later. But you're not so even keel that you don't show support and excitement and encouragement in that respect. There's some balance in there.
[00:11:28] Speaker A: Yep. So totally agreed. So I think the attitude that I take into meeting with students is sort of, you know, you certainly want to be. To feel like we're in this together and we'll solve this together. Right. And I'm always optimistic that I'll solve problems and we'll solve problems. Right. So then I start with, someone comes in, they have a result. Okay. It's either good or bad. I mean, either it's exciting and we want to do what's next, or it's not exciting or didn't work out the way we thought, and we want to think what to do instead or how to change things.
And the feeling is we're on the same side in this.
It's not your project and my lab. It's our work that we're doing together. And I think if you have that attitude, people will enjoy talking about their science more than if you put it more on them. Like it's some sort of adversarial thing.
[00:12:23] Speaker B: Yeah, yeah. The adversarial thing, I don't get. I don't understand how that dynamic ever works anyway.
[00:12:30] Speaker A: Right.
[00:12:30] Speaker B: But one of the things that he said specifically is that you seem to give him free reign to do the science that he wanted, but were very supportive when he needed you and otherwise gave him free rein. So it's this really nice balance. And I had this as a graduate student, and I should say he was a postdoc in your lab, correct?
[00:12:48] Speaker A: That's right.
[00:12:49] Speaker B: And that's kind of what you should Be as a postdoc, I would imagine. I think.
[00:12:52] Speaker A: Yeah, I think that's a good. So, yeah, Adam wasn't a unique case. He was terrific. And he both came with a lot of.
So it was a really great combination, I think, for someone to be my first postdoc because he had a lot of background in things like EEG that I didn't. And so a lot to show me and teach me about things that were different. But he had never worked in an animal lab in a serious way and done that kind of neuroscience.
So all the physiology techniques were new to him, or at least most of them.
[00:13:29] Speaker B: So he had never done real science.
[00:13:31] Speaker A: I didn't say that.
You said that.
[00:13:35] Speaker B: I had a PI that said that once.
[00:13:38] Speaker A: Yeah. So it was good in that there was a lot that we could each bring to it.
And also, frankly, I've never enjoyed the thought of saying, okay, here's my grant. You're going to do AIM one. Go ahead. That doesn't seem like a fun way for anyone. It's not the way I did my PhD either. I think in my PhD, maybe a third of my thesis was something that my advisor had gave me at the beginning. And then the other two thirds were things that we worked out along the way and projects that we did what was exciting and what seemed like the right thing to do. And I kind of like that balance as a trainee and I hope that I can give that kind of balance to my trainees.
[00:14:17] Speaker B: Yeah, I mean, all of these things are very dependent on both the trainer, the mentor and the mentee as well. And those are difficult dynamics.
[00:14:24] Speaker A: That's right.
[00:14:25] Speaker B: So we'll come back to that later in the show. Hopefully that was satisfying to Adam. We'll see. I'm sure you'll be hearing from him. He is an author on the paper that we're going to be talking about in a little bit.
[00:14:36] Speaker A: That's right, he is.
[00:14:37] Speaker B: Before we get. So speaking of even keel, would you say that you have a slow drift people make that joke yet in your lab?
[00:14:45] Speaker A: No. Well, my joke about slow drifts that I've used in a couple talks is that it either sounds like a brand of canoe.
[00:14:53] Speaker B: Oh, I like that.
[00:14:54] Speaker A: Or like, you know, it's a certain canoe. Or it's like a particular type of marijuana you can buy in a store in Colorado.
[00:15:01] Speaker B: Here in Colorado, where I am, that's. It's a canoe company that also sells marijuana.
[00:15:06] Speaker A: Right, right. It could be that too. Maybe it's both. Maybe it's both.
[00:15:08] Speaker B: Yeah. For the non adventurous high Canoe work, I suppose. Yeah. Anyway, we can, we can work on that, but I want to talk about a little bit about your other work and kind of your past work leading up to this because this is fairly new for you, right? The slow drift story.
[00:15:25] Speaker A: That's right.
[00:15:26] Speaker B: You're Mr. Correlations and sort of noise in high electrode density recordings in non human primates.
[00:15:35] Speaker A: It's kind of a mouthful for a name, isn't it?
[00:15:37] Speaker B: I tried slow drift, but you didn't take it. Okay. I mean, you've been recording with using Utah arrays for a long time and you sort of made your career studying correlations in the brain over different spatial scales and different timescales. And there's just this slew of studies in various parts of visual cortex and how the neural activity is correlated. I want to talk a little bit about that and just recording with lots of electrodes for a few minutes because now we're getting to more and more electrodes and it used to be the Utah array, 100 electrodes was like, oh man, how are we ever going to do anything with this?
It's too much data. Right.
[00:16:15] Speaker A: We're going to fill up the hard drive.
[00:16:16] Speaker B: Yeah. And who knew of a terabyte back in the day?
[00:16:20] Speaker A: Right, right.
[00:16:21] Speaker B: I mean, so there's a temptation to just keep putting more and more electrodes in the brain. And didn't you start with single electrodes in Motion's lab? Okay. So you kind of went up to like tetrodes probably, and then our mini, and then boom, Utah race and you're recording 100 using 100 electrodes. Has that been a smooth transition? What are some of the advantages and disadvantages of just recording more and more and more neurons?
[00:16:47] Speaker A: Right. So it's a great question. As a physiologist, I want to say there's no drawbacks, there's only advantages. Right. Because I just want more data and more data is better. So I think that's maybe a too glib an answer. Classically, I can think there's two advantages, rare signals and small signals.
So if there's something that's only present in one of a very large number of neurons or something that is very small in each individual neuron, then adding neurons really helps you a lot. Right. And I think we know there are signals like that in the brain that are present in small numbers of neurons or have a very small impact on the neurons, and those are big advantages for large scale recordings, especially, I suppose.
[00:17:32] Speaker B: In frontal cortex, where everything's multiplexed anyway.
[00:17:35] Speaker A: Yeah, I think that's exactly what we're thinking is that in areas like frontal cortex, where there's so much going on, it may be that a lot of the mysteries of those areas we can't unlock unless we have access to a lot of neurons.
[00:17:49] Speaker B: Thinking about, like, a small signal distributed among a lot of neurons makes you think that everything else is noise. One of the things that you have found and that you can do is look at the correlations between all the different signals that you're getting. And there's interesting stories to be told there, but. And among those correlations, then there's also a bunch of noise. Or is there? I mean, how much of the brain is noise and how much is signal that we just call noise?
[00:18:14] Speaker A: Right. I think that's the way you worded it is often the way I think, which is the first time you look at the brain, at the responses of a neuron, you think, wow, why is it so different every time? And maybe people who record in the auditory brainstem would think differently. But mainly I'm thinking about cortex when I say that. And when you look at cortical neurons, they seem to be so variable and so noisy that it almost seems just daunting. But the way you put it, I think, is right, or at least it's the way I think about it, which is that noise is really sort of signal we don't understand yet. Right? For the most part, I think most of the noise that we see in the brain is not true noise in the sense of Brownian motion of molecules determining some baseline physical noise limit, but instead is processes that are going on in the brain that we just aren't controlling or aren't aware of. And so I think that's the great thing that intrigues me about studying variability in noise, is that we could, in that noise, sort of uncover hidden mysteries about what the brain's doing.
[00:19:17] Speaker B: I mean, how has recording more and more neurons made you think differently about the processing that goes on in the brain?
Or has it changed it at all? But like you said the first time, even with a single neuron, you go in and you see two completely different responses from one trial to the next, even within the same behavior. And you realize, how does the brain give rise to such repetitive behaviors? But then, I mean, you stick in 100 electrodes and that's 100 things firing variably.
[00:19:47] Speaker A: Right? I think the key intuition it gave me was that some of the things that we might ignore in single neurons are actually really interesting parts of a population's behavior. For instance, in visual cortex, I Spent a lot of my time in grad school recording from different parts of visual cortex. And we know a lot of things about visual neurons, but one of the things we know is that in V1, most neurons are concerned with contrast. So if you turn on a low contrast stimulus, it maybe gets them to respond a little. If you turn on high contrast stimulus, it gets them to respond more. And great. So we think we understand that. But there's plenty of neurons that turn off when you turn on high contrast stimuli, right? So that seem to be oddballs, right. That sort of do different things.
And those oddball neurons and those oddball trials where things are different. I think you look through them, you look at them with a different lens when you see the population. So that sort of diversity, like one neuron turning off, could be maybe that's just an oddball that the brain's ignoring. That's maybe the view that I might have had, or at least that I want to ignore because it doesn't conform to the type of neuron that I want to study.
When you look at the population taking the perspective of, okay, well I'm going to just show the stimulus that I show and I'm going to take the responses that I get. I think is a powerful change in how we think about the brain.
[00:21:15] Speaker B: That is a really big difference from even when I started in graduate school and when I finished, I started where if you didn't have a single neuron absolutely isolated for three hours and feel confident about it, you couldn't keep it. And it needed to also be task related in the way you wanted it to be task. It needed to respond to visual stimuli or to an eye movement or something. And by the end of my postdoc we were putting these multielectrodes down and you know, up to I think we had like 32 channels or something on an electrode. So. But then just taking every. Because you can't isolate a cell, right? You have to just stick it down and take what it gives you anyway and then you might as well. And then sell. I mean, I know you guys use some neural networks to do spike sorting, but you can even throw out the spike sorting and just take the multi unit activity and there's still like these massive high dimension state spaces that then you can start using and then everything is part of the story and it seems to be a much.
Oh, what's the word I'm looking for more admirable way. Is that the right word? True.
[00:22:18] Speaker A: Yeah.
[00:22:19] Speaker B: Inclusive.
[00:22:21] Speaker A: Inclusive, sure. Let's go with inclusive.
[00:22:23] Speaker B: Okay. Inclusive not exclusive, I don't know. But, you know, because then you're not just. You're not. You don't have the bias of just taking the neurons that are helping you tell your story. You have to account for all these different neurons that, en masse, probably do help tell the story.
[00:22:38] Speaker A: Right. I think it's unlikely that there are neurons in the brain that are just sitting, doing, you know, random stuff that doesn't matter to the rest of the brain. I think we could start with that assumption that's pretty good, that there's lots of neurons in the brain just doing random stuff and they don't matter, and they don't affect our decisions in any way or play an important role in anything. Right.
[00:22:58] Speaker B: You've not met my dog.
[00:22:59] Speaker A: But okay, so if we start with that, then it tells us that the responses of those neurons might matter to what task you're looking at. And so then it sort of gives you this perspective of saying, okay, well, let's see if they do. Let's look at them together. And I think with the shift that the field has made toward more powerful computational techniques that can leverage these large population recordings, it has started to reveal interesting things about what those few oddball neurons are doing. And it's not always just oddballs. But my point is what interesting mixtures of neurons can do compared to single neurons. And I don't, you know, I certainly haven't moved away. My lab still does single neuron recording. We still, you know, I think I wouldn't place myself as someone who says, oh, yeah, let's throw out all the single neuron techniques. But I would say that it's a very different perspective and valuable complement if.
[00:23:56] Speaker B: In a week you could purchase a system that there are little nanoparticles that attach to membranes and can record the spikes of every single neuron in the brain and send them wirelessly. Would you want that? So it would record 86 billion channels.
[00:24:14] Speaker A: Can I get rid of the cerebellum?
[00:24:16] Speaker B: Well, sure. So that cuts it more than half.
[00:24:18] Speaker A: Sorry, I just have to make fun of cerebellum. I'm sorry.
[00:24:21] Speaker B: I've started enjoying the cerebellum. But I used to be in your camp.
[00:24:26] Speaker A: Now I'm more joking that people try to get rid of the cerebellum than I would want to.
[00:24:31] Speaker B: Yeah. You were part of the cerebellum journal club back in the day.
[00:24:34] Speaker A: I was. I was.
[00:24:35] Speaker B: For a little while. I scoffed at you guys wasting your time with that massive lobe.
[00:24:41] Speaker A: Right. But would I want to do that? Of Course I would want that. But I will say if I had all of those neurons sitting on some instantly accessible data storage media, the first thing I would do was just select 1000 of them.
So in other words, I don't think I would try. I think there are computational hurdles, not just things like disk storage and processing speed, but sort of conceptual computational hurdles that would keep us from just taking all of those neurons and understanding what to do with them right away.
[00:25:14] Speaker B: You have one cell you can trial average and you can't do much single trial stuff. You have 100 channels from Utah array or two because you got for you one in V4, one in PFC or something. And well, you analyze those separately. You could analyze them all together though, and that would be 200 channels. That'd be okay, yes. 1,000 would be okay, yeah.
[00:25:35] Speaker A: I think an important limit that we're confronting as a field is that as neurons grow, the relative importance of the neuron limit shrinks and the relative importance of things like the trial limit or the amount of data we can get in a session grows. So I think if we had a situation where you said, I'll give you a thousand neurons and 1000 trials of some task, I would say, great, that sounds good. And if you said to me, okay, wait, I'll give you a million neurons in a thousand trials of a task, I would say I'd rather have more trials than more neurons at that point.
[00:26:10] Speaker B: Yeah, I guess that's the big question is what is the limit of neurons that would be even useful beyond which would be not useful anymore to link to behavior within some region of your.
Not brain wide, but like a sort of local recording or something? Because it's got to be pretty low, I would imagine. We don't have to go on about this, but these are just things I don't ever think about. And you probably do.
[00:26:35] Speaker A: Yeah. What would be the limit of how far you could go? I mean, it's maybe self serving also because I don't have access to a million neurons to say, oh, I think 1,000 neurons is enough. But what I would say is in the questions I've asked, I've tried to.
Within the limit that I can rigorously assess. Oh, would my answer change if I had 10 more neurons?
[00:27:00] Speaker B: Right.
[00:27:00] Speaker A: Or 100 neurons instead of 50? And to a large extent the answers don't seem to change. So at least I feel like I'm operating in a regime where I have enough neurons to ask the questions I'm trying to ask. And I don't feel largely neuron limited. If I had 200 instead of 50, sure, I would definitely take that. And if I had a 'Thousand instead of 200, I would definitely take that. But if you asked me, would I rather have another thousand neurons or more trials or more time with those fewer neurons? I'd rather have more time.
[00:27:35] Speaker B: I'm going to come back to this question because there's a, there's kind of a follow up question, but we should talk about Slow Drift a little bit and debated whether to play this before or after. But I'm going to play it before.
And so it's jumping the gun. But this is someone else whose voice I believe you'll recognize. So before we talk about Slow Drift, here's a question, a meta question about it. Hi Matt, your recent Slow Drift paper was in a high profile journal.
[00:28:01] Speaker A: Do you feel like this is your.
[00:28:02] Speaker B: Biggest result so far in whatever way you want to quantify? Biggest.
Thank you, Patrick, for that. Patrick Mayo, previous guest on the show.
[00:28:11] Speaker A: Yes, it's a good question.
[00:28:14] Speaker B: You haven't considered this yet, huh?
[00:28:17] Speaker A: I don't know. I don't like to rank my scientific greatest hits and it feels like that's something that at least I should wait a little bit longer to do.
[00:28:28] Speaker B: I can rephrase the question, but let me, because I asked him for a follow up, I said, what do you think Matt will say in the first sentence he wrote is that he doesn't think in terms of biggest, quote, unquote.
[00:28:37] Speaker A: Right.
[00:28:38] Speaker B: Which is paraphrasing what you just said, which I think a lot of people would rightfully reply that way. But he also said in whatever way you think of as biggest and maybe most impactful, career wise might be a different way to look at it.
[00:28:51] Speaker A: So here's a way I can answer that I think is fair in some sense. Maybe the biggest scientific result for me would be one that changes the way I think and do science in the future.
[00:29:03] Speaker B: I like that.
[00:29:04] Speaker A: So for me, a big result would be one that makes me want to, I don't know, write a different grant or do a different experiment or change how I analyze some data.
From that context, I would put it among the biggest scientific results I've had in that it's really changed how I think about the experiments I'm going to do and how I think about the other science that I'm currently doing.
[00:29:30] Speaker B: Do you want to hear what Patrick believes?
[00:29:33] Speaker A: Yes.
[00:29:34] Speaker B: You can say no.
[00:29:36] Speaker A: I always want to hear what Patrick believes.
[00:29:38] Speaker B: I know that's not true.
He didn't know whether you might think or whether he thinks that the slow drift work is your biggest biggest. Or you're finding about the correlation changing over different spatial scales so that two neurons would be more highly correlated when they're physically closer together than when they're further apart. He found that to be one of your biggest, at least. So I don't know, how would you rank that?
[00:30:08] Speaker A: Yeah, I think that result to me was more what I expected.
I mean, there are many parts of that study that had things that I hadn't expected or predicted or known. But I guess to me that felt like some of the science I do. I feel like what I'm contributing is, I don't know, taking a really systematic view on something that maybe isn't necessarily the most novel question or the most different question, but that I couldn't do a good job at it. And I guess I felt like that study was me and Adam Cohen doing a really good job at a question that was not, you know, quite as surprising or quite as like, oh, wow, no one expected this. I think there was. It's a little weird to think no one expected this. Right? I mean, in the introduction we cite other studies that show that distance matters. And so of course, we weren't the first to show that. And so in that sense, I don't think of that as bigger if I were to try to quantify that somehow.
[00:31:11] Speaker B: So in some sense there's not before slow drift and drift and after slow drift. But the slow drift stuff is more of a foray into, I don't want to say behavioral work, but our higher cognitive work. But a lot of your previous work has been really in the nuts and bolts of the neurons and their activity and the statistics among the population of neurons while showing graded bars, which you also do in this slow, slow drift paper. But not necessarily during like, you know, interesting decision making tasks or whether you find that interesting. That's a completely subjective thing.
I don't know how you think about that. If the slow drift is sort of a foray into. It's a different sort of question anyway.
[00:31:57] Speaker A: Right. So that's absolutely true. So I guess that part of it, my lab maybe, since I've had.
I would date it back to when I was a postdoc with Mark Sommer, which was how I came to know you best.
I had this idea that I should take some of the skills that I had learned in visual cortex or basic visual processing, visual neuroscience, and kind of move forward in the brain. And that was an idea that was what led me to want to work with Mark, at the time, it was the interaction between vision and eye movements.
[00:32:32] Speaker B: Specifically frontal eye field. Guy.
[00:32:34] Speaker A: Right, frontal eye fields. But more generally it was moving into something, you know, away from, let's say, static vision or fixation into something a little more active and a little closer. And I guess as I started an independent faculty job, I moved further with that. And I would attribute some of that to the ideas that Adam Snyder and I were talking about as he was a postdoc and that was sort of pushing forward into thinking about attention and cognition more broadly. So I guess I wouldn't say that the Slow Drift paper was the thing that started this, but it is maybe a culmination of me sort of moving in the direction of thinking more about cognition and decision making.
[00:33:16] Speaker B: Okay, so let's actually talk about the work so that people will know what we're talking about. So you're doing an experiment and you have an animal in the lab performing a task. And we've all had any of us who have undergone this experimental situation know that the animal sort of waxes and wanes, can fall asleep for a while, seems overly aroused sometimes. And with that, and even besides that and with that, you just see these sort of patterns of like terrible performance on a task for 30 trials or something and you don't understand why. And then all of a sudden it gets like a little better and you think they're doing it right. And it's a super aggravating thing. Actually, in at least it was for me, it was aggravating on a regular basis because what you need is for them to be these sort of statically on and aroused and engaged animals performing this task so that you can get the perfect data and what you have found and you can tell me how you came to ask this question. But what you found is as this behavioral drift, which is related to an animal's quote, unquote internal state. As this occurs, as it kind of waxes and wanes, so does the brain, which you would expect in interesting ways. And you call this minute to 10 minute ish kind of oscillation over the, over the course of a few hours of an experiment, the Slow drift. So what did make you start looking into this drift within an experimental session?
[00:34:53] Speaker A: Right. So I should start by saying the project there was led by Ben Cowley, who was a PhD student working jointly with Byron Yu's group and my own. And so I give all the credit to Ben for sort of discovering this. This certainly wasn't a case of us saying, hey Ben, why don't you analyze how things varied across trials. And then Ben said, okay, and then plotted a plot. And we said, great, that's exactly what we thought was going to happen. So we were interested in a thing that we're talking about more generally, which is variability and neurons and the connection to behavior. And I think a motivating thought we had was that a lot of people have considered the connection between variability and behavior, but not thought about whether the brain has mechanisms or sort of awareness of its own variability, let's say. So maybe. Let me unpack that sort of a weird statement.
What I mean is noise might, in some ideal sense, limit your perception or your behavior. Right. If you have a set of noisy things, you can check how you might be able to extract information from them, and you can mathematically derive answers about how good some ideal subject could do. And we were really interested in this idea of sort of, what if the subject isn't ideal? Or what if they're doing, you know, different things at different times? And so Ben, you know, not out of our direction, but out of his own cleverness, decided to sort of plot some of the extracted statistical measures he looked at in the population across the session. And then we just saw, wow, look, they're all moving up and down. And then we said, oh, that must be. That's weird. That must be some artifact or something. And so then he plotted individual neurons. And we saw that individual neurons that at the beginning of the day were responding with 10 spikes to some particular stimulus, sometimes at the end of the day would be responding with 30 spikes to that same stimulus.
So then we started thinking, oh, okay, this looks like a really cool signal. And we started thinking, what is happening to the behavior at the same time, which we kind of knew was variable, but hadn't plotted quite in this way either. And they sort of magically lined up. And then we dug into that deeper and deeper.
[00:37:09] Speaker B: Okay, so this is why you were basically record. You were already recording in prefrontal cortex in V4. So you have two Utah arrays implanted in V4, which is a higher visual area, but kind of in the crux of a lot of different pathways. And prefrontal cortex, which is a high, higher cognitive functioning area, so you could look at these signals simultaneously. So you didn't. So you. This is analyzed from performing other tasks dedicated to other experiments.
[00:37:40] Speaker A: That's right. So this project was really a reanalysis of a set of data that, in fact, Adam Snyder, who we heard from before, collected on us. And the central question we were interested in was, how do Top down influences affect our visual perception during attention. So when we're paying attention or not paying attention or shifting our attention, let's say, so what happens? You know, one advantage of the way we were studying that task is we're considering the idea of what if an animal is making more or less the same perceptual decision all day long? Very naturalistic. So they're trying to tell.
[00:38:22] Speaker B: Yeah, go ahead. What do you mean, making the same? What does that mean, making the same?
[00:38:25] Speaker A: What that means is.
[00:38:26] Speaker B: Oh, you mean if they're doing the same task all day long?
[00:38:29] Speaker A: Exactly. What if they've got a vertical stimulus and they want to tell you when it's not vertical anymore? And they have to do that over and over again. And that sounds really boring. And so it would be natural all of us would wax and wane in our ability to perform that, just as you said, animals do and they do. But the advantage of it is that then the things that we think of as noise might sort of reveal themselves because we have so many repeats of the same thing that we can start to find structure in that noise. And so that was the motivation of studying it that way. And I think that was. The task was about PFC and V4 and attention. But then Ben was interested in these sort of population activity structures and looking at how things progressed over the course of the session.
[00:39:17] Speaker B: And what did you find specifically in V4 and in prefrontal cortex and how they relate to each other?
[00:39:23] Speaker A: Right.
So as I mentioned, the sort of core of the observation if you think of one neuron is that even with the same stimulus, if you just go from minute to minute, the amount that that neuron responds to that stimulus seems to change.
And this seems really counterintuitive, right? Because if you think of our visual system as kind of being, I don't know, some sort of processing module, right, that sort of takes the input stimulus and then converts it to some output signal. You certainly don't want it to be giving a different output signal at 2:00 than it did at 130. Right. Because that wouldn't seem to be how you'd design any such system.
[00:40:01] Speaker B: But I know from personal experience just, I mean, not. I'll just interrupt you because, I mean, just reading this paper, like every moment I started drifting off and remembering just being in the lab. And for instance, with these multi electrode, single electrode, like, but multi contact, you'd have to put it in and then let it kind of rest for an hour if you were smart, or 12 hours if you were really smart, you know, or just go as long as you could because, you know, you'd have cortex kind of moving up on the electrode, which changes the signal. But even once I started the task, I would find these beautiful responses all over the place. I start the task and then, like, half of them kind of go away.
[00:40:39] Speaker A: Right.
[00:40:40] Speaker B: And it's not because the electrode's drifting, it's because the neurons are changing their responses. And it's just infinitely frustrating in that. In that setting. Yep.
[00:40:49] Speaker A: And I think it. It remains frustrating to me.
So that's not.
[00:40:55] Speaker B: But it gives rise to this story.
[00:40:57] Speaker A: So the thing that we started thinking more deeply about was if some neuron is giving a signal of 10 spikes at, you know, 1:30 and 30 spikes at 2:00 in the afternoon, maybe that doesn't just happen for reasons of noise, but it reflects something about how the animal is changing as it's making those decisions over and over again. So we looked in V4, and then, because we were recording many neurons at the same time, we could extract out some signal that seemed to represent what the whole of the neurons were doing. And using some statistical approaches, we could see that, yeah, it does look like they're all moving together as a group in a very coherent way.
So then from V4, we started saying, oh, that seems like a really undesirable thing for your visual system. What's going on in this other part of the brain, prefrontal cortex, maybe it's not drifting and it gets rid of this signal. Wouldn't that be kind of cool?
But I guess what we found was maybe. I don't know if it's cooler, but it's certainly more perplexing or leads to. Yeah, it's cooler. It's cooler.
Was that PFC is doing the same thing at the same time. So if you independently take a mixture of PFC neurons, you can map the same kind of signal, and it wouldn't have to by chance be doing anything.
[00:42:12] Speaker B: Like this at the same oscillatory frequency in the same phase. Yeah.
[00:42:17] Speaker A: Yep, Yep. Yeah. Well, let me make a small correction. I wouldn't say oscillatory in the sense that it looks more like what you'd call a random walk or something. Right. Which is to say it's not that it's going up and down at a certain. It does go up and down, but it doesn't do it at a characteristic frequency, even though it does have a characteristic timescale.
[00:42:37] Speaker B: So what did you think? So when you saw that it was also happening in Prefrontal cortex. Immediately you think, oh, it seems to be a global signal then.
[00:42:44] Speaker A: Right. So that led us to ask, okay, this gets back to the question about what's the role of noise in making decisions? And I think most of the literature thinking about noise in the brain has either treated it as. Either ignored it. Right. They record from one neuron or otherwise don't measure it or whatever, which is fine.
[00:43:08] Speaker B: Average it out.
[00:43:09] Speaker A: Yep, average it out. Or treat it as a thing that sets a limitation on us. Right. We could perform better if not for this noise, which sets a fundamental lower bound on our performance, let's say, or upper bound, rather. The thing that we started thinking was, well, wait, maybe this is an interesting way to flip this and ask does the noise really have to limit your behavior? Because if the decision making area, if there is a magical decision making area, if the magical decision making area knows what the source of the noise is, then it doesn't have to be limited.
[00:43:48] Speaker B: By it, because it could get rid of it.
[00:43:50] Speaker A: Right, exactly. So in other words, if you're waiting for a neuron to fire 10 spikes, because that means you saw a picture of your dog and the neuron fires 30 spikes. So you say, no, no, that's not my dog.
But you know, that now is a minus 20 trial because, you know, you've got the magic number, the magic offset. Then you would just subtract and you'd say, oh, right, that's really what I thought it was. Right. So I think the intriguing thing about noise in the brain, for me, and how it's dealt with by the brain is trying to think what the, you know, the magical decision making areas, or let's be more specific, you know, the decoding algorithm or the projection weights of the neurons or whichever metaphor we want to use, how does the brain deal with that noise?
Does it actually reach the level of influencing the decision? Is it actively accounted for or removed?
Does it set a fundamental upper bound? Or is it maybe something that the brain knows about? Right. Because the brain has access to it and might account for it.
[00:44:55] Speaker B: And finding the same drift in prefrontal cortex, which is one of those magical decision areas, decision making areas, was surprising because then you think, well, you'd think that PFC would have gotten rid of the noise.
[00:45:07] Speaker A: Right. So that was surprising to us.
And why that or how that could happen and still have the brain kind of quote unquote work is still a mystery, I think, for us. Sorry, yeah, go ahead.
[00:45:23] Speaker B: No, I was just going to say, like you talk about in your paper, it Means that. So you have these early sensory areas, let's say like V4, higher sensory kind of area, and if the signal is messed up in V4 because of this drift and it propagates downwards, then you'd think you'd be making bad decisions, right. And it would just completely mess up your decision making. I'm probably just repeating what you said, but that is not, in fact what you guys found. I mean, so the decisions, the perceptual sensitivity was basically unchanged, correct?
[00:45:56] Speaker A: That's right. So what we did was try to ask, in sort of a relatively classic approach that's been used in the field for a while, how well can you predict the animal's decision from the sensory cortex activity?
And the reasoning there is that if you have some noise in your sensory signal and it flickers in just the right way, maybe you think that you saw a change and that's not, you know, that's not quote unquote, the animal's brain's fault, or at least not the decision maker part of the brain's fault. It's just a fundamental limit, right? It came in, the noise came in on the inside. You can't deal with it. It's just noise, right. And it affects your decision. If, on the other hand, the noise comes from some other thing the brain's doing, then you might be, the brain might be able to account for it, right? It might say, oh, you know, the sensory signal said X. But I know that I was more alert then, so I need to make this, you know, compensation in order to make my decision. And we seem to be able to make a claim for the latter, which is to say that it looks like not all of the noise that we saw, specifically the slow drift that we saw, seemed to influence the behavior directly by its effect on sensory neurons. So we could better predict what the animal is going to do if we removed the noise from the sensory neurons.
[00:47:17] Speaker B: As if the decision making area was getting a clean sensory signal.
[00:47:20] Speaker A: Exactly right. Or getting a dirty sensory signal that.
[00:47:23] Speaker B: It knew how to wash. And how would it wash it?
[00:47:27] Speaker A: Right. So that's a tough question.
Sometimes I like to work backwards when I think about it, and I'm not going to give you a very satisfying answer, but we know that this signal shouldn't be present at the level of the muscles. Right? So in other words, if it was present in the muscles, then you would just move. So we have to somehow have a way to remove it from there. Okay, so now that's. So let's say that's maybe, let's walk back from the muscles for the eyes to the spirit colliculus and say, okay, maybe at the level of the colliculus we need to, you know, this eye movement part of the brain that we need to sort of remove that signal there. So I think one intriguing idea is that in the sort of sensorimotor transformation the noise gets removed, that it stays there present in all these sort of decision making areas.
But the process of actually converting the sensory signal to motor outputs or transforming the sensory signal to a motor output is where it gets removed. I don't know that yet, but we're interested in that possibility.
[00:48:32] Speaker B: And that would be helped by the ubiquitous efference copy or corollary discharge from these earlier areas or even from I guess the higher cortical areas from pfc. I don't know where the efference copy would come from in that case.
[00:48:48] Speaker A: I think that's right. So I don't know either where the reference copy is there. But thinking of efference copy is exactly how I think about it as well. Right. It's a copy of, not necessarily a movement command, but it's a copy of a decision signal. Right. Or a state signal. And the brain, when it makes its ultimate decisions or when it makes its ultimate actions actually, let's say, needs to, I think, take into account all of those states. And some of them directly influence the action, but some of them are only indirect in how they can influence the action.
[00:49:21] Speaker B: In the paper you talk about either subtraction being a mechanism or this sort of divisive normalization or, you know, dividing out the signal. And I mean, this is all kind of hand wavy at this point, but it gives rise to like, you know, further questions to ask and places to stick electrodes and do experiments.
[00:49:39] Speaker A: Maybe my answer isn't going to be so totally satisfying for this one. But I will say the mechanism of how the noise gets removed I think is a tricky one. You're right to say that divisive normalisation or subtraction there are very hand wavy because we don't know a lot about how cortical circuits would do this kind of, or subcortical circuits would do this kind of computation.
One way is that it's not divisive normalization or subtraction, it's just linear algebra. You just have a weighted sum of inputs and those things you weight the sum just in a way that makes this thing go away. Right. But beyond that, I guess I would say that the thought of it being a signal that needs to get removed or may get removed at some point has led us to think, okay, well where is it coming from and where is it going and what are the limits of it? And that has led us to a bunch of really cool follow up questions where we're exploring those kind of directions.
[00:50:37] Speaker B: We'll see. This goes back to whether the biggest result means the one that leads to new grants and new experiments. Because you've got multiple papers on was it Biorxiv or Arxiv that have just already been follow ups. And I can just see this is like a decade's worth of work. I mean, it reminds me of. So the quenching of variability as an animal is getting closer and closer to inactuating a decision. The variability in neuronal responses goes down and down. And you contributed. This was like a huge collaboration between tons of different labs that gave their data and put it all together.
I don't know. Was this a Shinoy lab?
[00:51:20] Speaker A: Mark Churchland. It was the Shenoy. Mark Churchill, when he was in the Shenoy group, when he was in the.
[00:51:23] Speaker B: Shenoy group and put this all together. So it's kind of a ubiquitous thing. And this slow drift is going to be a ubiquitous thing as well.
[00:51:31] Speaker A: I sort of maybe hope so and also feel bad about it because the part where I feel bad about it is it's made me sort of rethink other studies and other data analysis in my own lab. And I hope it does that for other people, but it does kind of throw a wrench in things. Right. If you were sort of treating all your trials as the same and doing some analysis, you might start to think, oh wait, I shouldn't be treating them all as the same.
[00:51:56] Speaker B: I mean, there is a level to which averaging out would still average out.
[00:52:00] Speaker A: Sure. And it depends on the question you're asking. Many questions might sort of be totally fine to ignore this, but it falls into the category of neuroscience results where it's like, oh wait, that happens. I didn't know that happened. That's going to be problem when I'm trying to do the thing that I was planning to do.
[00:52:19] Speaker B: I can't wait in three or four years when you have the review out and it's titled it's totally fine to ignore this, but ellipses.
Or you may think it's totally fine to ignore this, but yeah.
[00:52:32] Speaker A: Yes. Yeah. And as you said, we're following up. And part of the following up that was so natural was because you could then say, oh wait, let's look at this other experiment we had done and see what it was there. So we have lots of data we already collected where we could perform the same kinds of analysis.
[00:52:48] Speaker B: One of the things that you guys recorded is. And that it seems like everyone records now, I guess we've always had the capability to record it, but we've always ignored pupil diameter. And it seems like there's always a pupil story in every eye movement. Well, every paper now. Right. So you saw the same fluctuation happening with the pupil as well. What does that mean?
[00:53:10] Speaker A: Right. So this is something we're following up now. But one of the differences in the way we analyze the pupil compared to what a lot of people do, although not everyone, is that we're looking at kind of a tonic pupil size. So a common way to analyze the pupil is to, you know, flash a stimulus or have some event and then look at how much the pupil changes in response to that flash. It's called the pupillary evoked response.
What's different here is we weren't looking at that. Instead, we were looking at sort of the baseline value of the pupil before the start of the trial, although we looked at different intervals.
So we think that what's changing here in the pupil is not something about how much it's responding, but more of a reflection of maybe a push pull between sympathetic and parasympathetic systems. That reflects something about kind of our balance between maybe a more active, aroused, alert state and a more passive, quiet, patient state.
[00:54:10] Speaker B: I mean, I buried the lead by.
We didn't even talk about how you interpret what this slow drift means. So we talked about how it seems that it's not really affecting the perceptual sensitivity and the perceptual performance.
And yet it does change the behavioral performance. And what you guys concluded is that it's an indicator of what you call impulsivity and what other people might call arousal. But you settled on impulsivity, and maybe you can just talk about that for a second.
[00:54:46] Speaker A: Right. I think the thing that this study, as we move. You mentioned before that I have sort of shifted into thinking more about cognition and decisions in maybe the last few years of my research. And one of the things that I've become aware of or thought a lot about as I've done that is how many things go into making a decision. Right. And so I think one of the things that we reflect on when we're studying animals making these decisions is our first approach might be sort of a simple one like, okay, they have to say yes or no, or I saw something or I didn't, and Then we interpret. If they say yes, that means they saw it, and if they say no, that means they didn't. But they may also say yes because they were just kind of tired of looking at the dots. Or they may also say yes because they thought they saw something. Or they may say yes because they're looking for a reward and just hoping that without expending much cognitive effort, they could get that reward.
[00:55:45] Speaker B: Or they may have been surprised by hearing in the next room another experimenter trying to wake up their animals so that they'll keep behaving, perhaps.
[00:55:53] Speaker A: No, no, that never happens.
[00:55:55] Speaker B: Never happens.
A little insider knowledge there.
[00:56:01] Speaker A: Yes. Particularly annoying rigmates who play loud music. Music.
[00:56:05] Speaker B: No, that never. Oh, wait, that was me. Oh, no.
Yeah, but. So there's a lot of things that go into a decision.
[00:56:14] Speaker A: So there's a lot of things that go into a decision. And in the kind. The way we set up our task, the animal couldn't be, you know, it was a challenging one. So there were a lot of trials where it was a very small change, and they couldn't be sure that a change happened because it was so small. And also the task involved multiple flashes, and they could never be sure which one was going to be the change. I don't know if you'd call it more realistic, but the point is that it's a decision that's made at a variable time where you have a lot of uncertainty.
And because of that, I think that played into this notion that people have studied a lot in humans.
Response inhibition, where the notion is that you sometimes have to make decisions where your job is to hold back your response until a certain time when it's time to go. And I think that was its own phenomena that was manifesting in our subjects.
That's why we called it impulsivity, because people think of that as impulsivity control. But I totally agree. I think you could label it arousal, you could label it alertness. All of those are fine. And I wouldn't make a strong case that one was a particularly better label than the other.
[00:57:33] Speaker B: I mean, there's work being done. Like you guys cite David McCormick, I think, and there are all these internal state terms now. And you think of the state space of internal state and moving among restfulness, wakefulness, sleepiness, alertness, impulsivity, and I don't know all the terms. So. But this is, you know, this is happening during the slow drift. Right. And you guys noted this impulsivity aspect of the behavior.
[00:58:03] Speaker A: Yep.
[00:58:03] Speaker B: Where does the drift come from? How does it happen?
[00:58:07] Speaker A: So I think a really good candidate is a global neuromodulator.
And we know that there's multiple neuromodulatory systems operating in the brain. And most of them don't, you know, don't directly lead to spiking in neurons. Right. The activation of these systems doesn't directly cause a neuron to spike. I mean, some do, but it depends, depends on the system.
And then the question is, well, what are they there for and what do they do? And I think a lot of cognitive processes have been linked to neuromodulators, right? Acetylcholine and norepinephrine and serotonin and all of these different neuromodulators. So I think at the same time, most models of how vision works are operating basically on glutamate and GABA excitation and inhibition and aren't thinking much about neuromodulators.
That's not strictly true. There are plenty of people doing that. But predominantly we're just thinking about these two canonical.
[00:59:03] Speaker B: When you learn the story, that's what you learn.
[00:59:05] Speaker A: That's what you learn. Exactly. So I think a good question is whether this kind of slow drift could be generated by a neuromodulatory system. There's good candidates. Norepinephrine in the locus coeruleus is one good candidate, but there's others. And I think the question now is, can we link it to one particular one? It may be that it really is okay, just inputs from this one brain area and this one neuromodulator are going across the brain and leading to this signal. Alternatively, maybe that's not the right approach. Maybe it's a different neuromodulator that's sending inputs just to visual cortex and the drift gets passed on everywhere. I think there's good options to explore and I don't know yet which one is going to be the answer.
[00:59:48] Speaker B: I mean, so the brain is complicated. This is a problem. It seems to be a problem that the brain is complicated.
I mean, you have, we were talking earlier and you know, cortical neurons and as you, as you go up the hierarchy, the quote unquote hierarchy toward frontal cortex, this happens more and more where you're getting these multiplex signals and they're involved in these higher level decision making components. And then stacked on top of that you've got these neuromodulators that are sort of drifting this way and that and they're mixed up. So sometimes it's just one neuromodulator in an area. Sometimes it's multiple and it's. It's going to be hard to track that.
I don't remember McCulloch and Pitts when they were developing, you know, I don't remember reading about them when they're developing their logic gate first. Artificial neurons. Worrying about global neuromodulators.
I also like. I don't remember Alan Turing writing about the complexity of the slow drift and how to map that on. I bet von Neumann did. If I had to guess, he might have.
So we talk a lot on this show about. I harp on about how AI doesn't pay enough attention to the brain. There are some things that maybe should be paid attention to, some things that shouldn't. I mean, does slow drift tell us anything about building AI? Are these like neuromodulatory ingredients? For instance, are these global sort of dynamic slow dynamics? Are those necessary ingredients in complex systems? Or, you know, are they just byproducts of, like, metabolism and needing to stay alive?
Is there anything that AI could take from this sort of result? Yeah, I think not your particular result, but just the broader context of these sort of slow dynamic things.
[01:01:38] Speaker A: I think there's a couple of lessons. I think one is, for the most part, I think of AI as mostly handling static phenomenon. Or let's say at the very least, that the dynamics are a second thought rather than the first thought.
[01:01:56] Speaker B: But I'm talking about ambitious AI, not current AI. But we can talk about current AI.
[01:02:00] Speaker A: Yeah, that's right. So one key step is to consider how the brain solves problems dynamically, which involves both very fast timescales and slow ones like we're talking about now. And if we want to think about how AI could be informed by the brain, we have to at the very least move beyond making a comparison of a static system to a dynamic one.
Maybe a second point is that when we make comparisons between AI and the brain and try to ask whether they should inform each other, one of the big areas that you know about that's been done is in visual object recognition, right, where people compare things like what's the AI doing to what is area it doing, or something like this that's.
[01:02:46] Speaker B: Kind of like that and reinforcement learning are the two. Two big stories about how brain and AI seem to jibe.
[01:02:53] Speaker A: Right?
So those are great. And I don't have anything particularly critical to say about that. But what I would say is if we're going to move beyond the first order comparison there, we have to remember that the brain is Doing a lot of things other than just categorizing objects. Right. So even if we think we've got a task where we've got some brain activity where objects were being categorized by the animal or whatever, there's a lot of other things going on, right. The animal's impulsive or not impulsive, tired or hungry or thirsty or all of those things are all happening. And if we're trying to make a match and we don't see a match, or we don't, you know, the match isn't clear, or it seems like a match, it may not be because of all these other things the brain is doing at the same time.
[01:03:40] Speaker B: But do we? For instance, three out of the things that you mentioned that might be going on are related to just staying alive. And an AI system doesn't have to worry about just staying alive. But I guess the question is, and bringing noise and variability into this as well, are these things that are inherent to a complex system and almost a necessary byproduct of all of the different dynamics that are going on and like a byproduct through evolution of these are things that occur because of our metabolism and because we have to stay alive and then get co opted for all these beautiful decision area making things. The cognitive processes that we want to build into AI, or are there things that we can ignore and move on with just the computational drive?
[01:04:34] Speaker A: I think of the same question when it comes to the slow drift. So one possibility for slow drift. So you can ask, if it just gets removed, why is it there? Right.
If you just need to remove it, why was it there in the first place?
[01:04:51] Speaker B: It seems costly, energetically.
[01:04:53] Speaker A: Right. So I think there's a couple of answers. One is that it's useful for something that we're not studying. That there's a situation in which it's really important for this to be there. And that's the kind of one you're comparing with the AI, Right? There's something the brain's doing, but the AI doesn't have to. So it's there because the brain has to do these other things. Another possibility is that it's sort of there as an evolutionary artifact of how things were built. Right? Oh, the brain evolved, it evolved a certain way at some point it was useful or evolved along some path. And the path it's not the most, let's say simple or straightforward or from first principles way you might build it, but it just happens to be the way the brain was built or evolved.
[01:05:38] Speaker B: Evolution doesn't care about first principles for sure.
[01:05:41] Speaker A: Right, right. So, I don't know. I think an important lesson, certainly, in the AI brain comparison is to not.
Is to be aware that mismatches may not be just because. Well, let me say it a different way. If you observed a mismatch between AI and the brain, and even if you had developed an AI that had to, you know, that got tired and felt hungry and all of those things. Right. It wouldn't necessarily do them at exactly the same time as the brain you were comparing it to would. So, you know, searching for the perfect match is probably not the right way to compare the AI to the brain. And I think a better approach, which is, you know, by all, to be totally fair, exactly what's been done in the visual object recognition category is to search for canonical computations or themes by which the brain works and try to make them match at that level. So in that sense, I would applaud that effort because I would say, okay, they're thinking about sort of canonical ways vision works and how you can make an AI that works in some similar ways. But you don't necessarily want the AI to be some sort of automaton that exactly replicates an individual being because that's not what its purpose is.
[01:06:58] Speaker B: Yeah. I mean, this goes back to the function of what we want AI for and.
Yeah, yeah. I mean, these are fun and difficult questions. Moving forward. I think of you. I used to think of you in my early days of knowing you as like a hardcore statistics computer vision guy. And the 2012. I guess I had left.
I guess I had a PhD by then and I had left you high and dry.
[01:07:25] Speaker A: It was terrible of you.
[01:07:26] Speaker B: I know.
Letters we kept sending, but that's kind.
[01:07:31] Speaker A: Of when the tear stains on the paper. Tear stains on the paper. Little smear. The.
[01:07:36] Speaker B: I fused out a bunch of keyboards from my tears. I stopped using paper a long time ago. I'm not as old as you, that's true. But the AI, the deep learning revolution happened in 2012. Right. With the ImageNet competition and convolutional neural networks performing the best on that. I'm curious, like, just what your take is on the recent hype and explosion of deep learning. And, you know, it's obviously changing the financial world and. But I'm wondering what you think of it in terms of how much we're learning about the brain from it and how much the brain can contribute to deep learning and AI. How's that for broad question?
[01:08:23] Speaker A: That's broad. Well, I certainly wouldn't place myself as an expert in statistics. Or computer vision. But I do like to work with those tools.
So take my response for what it's worth. But I would say the development of AI as a tool has been quite amazing. The development of AI as a tool for understanding neuroscience, or comparing to the brain has been useful and intriguing to me. It's not clear how far that will go. The development of AI is a tool for analyzing, making your phone better and analyzing data and all of that.
There's no question of the value there. I will say one, one problem with AI as a tool is that there's still a little bit of black box ness to it. Right. And I would say the thing that holds me back in understanding the brain is not, oh, do I have a more powerful statistical method? It's like, can I deeply understand that statistical method and how it works and what it tells me about the neurons? Right. In a way that I can write a scientific paper and communicate it to others. So I do have some, let's say, hesitation in sort of accepting the AI revolution as a tool for neuroscience to the extent that it moves faster than our intuition does.
[01:09:49] Speaker B: Do you consider it a revolution?
[01:09:51] Speaker A: Well, it's certainly a revolution in how it works on my phone, what my phone can do, what my computer can do.
It's a big change in that. I'm not saying a revolution in neuroscience.
[01:10:02] Speaker B: Got it. Going back to your trending, I suppose, toward more cognitive behaviors and cognitive functions with your research, what is your take on this recent. What I consider a recent anyway, sort of push to ban all non ecologically valid tasks and only study tasks that you would see in the wild, because otherwise you're not actually what you're recording in the brain doesn't match up necessarily with the actual cognitive function.
[01:10:34] Speaker A: I would characterize my recent push as one in which I try to think about cognitive function and decision making with a strong grounding in early vision and visual processing. So what I'm trying to say is that I'm dipping one toe into the deep waters while keeping the other foot firmly on the ground. That I understood.
[01:10:57] Speaker B: Yeah, in your.
[01:10:59] Speaker A: Right. In my reality.
[01:11:00] Speaker B: And your deep expertise really is what it is too, perhaps.
[01:11:04] Speaker A: So I would say I'm old enough to remember being in grad school in the 90s at a time when there was a push for natural images and vision, and that was the answer for vision. The idea was that we're going to have to switch to natural images because studying gratings is not going to tell us about how the brain works.
I wouldn't want it to Paint too. Broadest stroke, because that's not my tendency. But I would say if you look at the last 10 to 15 years, it certainly doesn't seem to me that natural images are the only way people study things, and we haven't learned things without them. So what I would say is that we should remember that these ideas are kind of like tides. Right. And they come and go. And as a scientist, if I'm really invested in the question that I'm studying, I would want to hold my position and study it the way I think is best, rather than being pushed around as the tides ebb and flow. It doesn't mean we don't adapt. I'm not saying that there's not a balance there. I think using ecological frameworks and behaviors is great. It can teach us different things. But there's a gain to switching to more naturalistic stimuli and paradigms. But there's also a cost. And one of the costs is the loss of the ability to replicate a particular situation to the extent that you can. And I think this issue of trying to replicate things and think about noise and its impact on behavior is really counter to the sort of way I study things. So I think switching to something like free viewing with free moving would be really hard for the way I think or the issues I'm trying to study in the brain, because I think it would be really hard to isolate the signal from the noise and its influence on coding.
[01:12:46] Speaker B: Yeah, I mean, I suppose it also does matter how much you want to make claims about a cognitive function based on your behavioral, based on your recordings, versus how much you want to study the inherent worth of the neural activity that you're recording. Does that make sense?
[01:13:06] Speaker A: Yeah, I mean, I guess we all want to generalize. No one wants to think that what they're, you know, the slow drift I was studying in the lab just doesn't even happen when you're out and walking down the street. Right. That would be a. I mean, scientifically, that would be sort of a travesty, right. That you were studying something that was so obscure that it only happened in some, you know, random situation, just had no meaning ever in another circumstance.
[01:13:29] Speaker B: Our neurons never fire out in the wild, right?
[01:13:33] Speaker A: Yeah, they're. They're too scared. They don't. It's scary out there. You don't want to fire.
So, yeah, I think, to me, the chance of that is very small. Right. The chance that somehow the phenomena we see in the lab just don't act in even the same way or remotely the same way. Under other circumstances just doesn't fit with me as a likely scenario. I'm not saying that you don't gain things by studying ecologically relevant behaviors. There's lots of different insight that that kind of richness can give you. But I don't think that it's required. And in fact, I would say to me, it's sort of required that you don't just do that. Right. So if we just move to a situation where we don't repeat anything and we just observe, then that's only a very small part of the scientific method. Right. So, I mean, in some extreme, we could say, if I'm studying ecological really relevant stuff, what I should do is just have some data collection device attached to some animal and just let it go and watch what it does. Right. I mean, that's a cartoonish characterization. But if I move to that, then I think that I would just call that observation. And observation is only a small part of science.
[01:14:40] Speaker B: Given that you have been more on the implementation level in Mars, levels of figuring out what's going on in visual cortex and now prefrontal cortex and studying really at that mechanistic implementational level of detail, has it changed the way that you think about how the brain works and gives rise to mind and cognition?
[01:15:10] Speaker A: Yeah, learning about noise in neurons and thinking about noise in neurons, which is, I think, what you're referring to when looking at the implementation level, that's really at the implementation level, that has made me think a lot harder about things like how different brain areas might interact and communicate. Right. Where. So there you've got maybe an algorithm for how do you pass a signal between two different brain areas, if that's what we're doing? Or how does a signal sort of projected across the brain commonly influence two brain areas? So I guess you might call that implementation still. But I guess I'm trying to move.
[01:15:49] Speaker B: It's getting in between.
[01:15:50] Speaker A: Right?
[01:15:50] Speaker B: Yeah.
[01:15:52] Speaker A: And I think so if you were to say functional computation there, I would say one of the key functional computations we really need to understand is how a group of neurons in brain area A interact with and influence a group of neurons in brain area B. And I think that is almost a wasteland in terms of actual real, deep scientific discoveries. And I'm not trying to cast aspersions on any particular field. I'm just saying we know very little about how different parts of the brain communicate. We really do.
[01:16:28] Speaker B: And that's such a fundamental thing.
There are questions like that that are so fundamental, and that is one in particular that it's really no wonder that people. It's no wonder that critics of neuroscience point to things like that and say, look how little neuroscience has done. I mean, the antidote to that is like, it's still really young and the brain is really complicated. Like, get off my back. You know, we just graduated from recording a single electrode at a time, you know, and you and I were talking about neurophysiology, but of course there's FMRI and the rest of the world of that as well. Right, right. So, you know, you don't want to cast aspersions, but I mean, it's like looking in the mirror a little bit. That is a very fundamental thing, and it's a big problem.
[01:17:09] Speaker A: Right. And I think that's an area that, you know, partly motivated our study is thinking, okay, let's record from different areas at the same time while animals are doing tasks so that we can start to understand interactions. But being fairly critical, I would say the idea of how those interactions work, the conceptual model of how they work, is really in its infancy in my own mind, not just elsewhere.
[01:17:35] Speaker B: Is it something where we're going to have to get down to the really nitty gritty microcircuitry details and understand how neuron A is connected to neuron B and.
And how neuron Z through double X or whatever double X is, you know, they're all impinging and consider their dynamics, you know, and computationally account for all that stuff. Are we going to have to get down to that level to understand it, or is there a level just above that where we're going to be able to talk about it in descriptive terms and feel like we understand what's going on, let's say, in the example, from one neural area to another neural area?
[01:18:15] Speaker A: I certainly hope that it's the latter. I hope that we're able to talk in sort of descriptive terms about modes of communication between brain areas and interactions. Yeah, I think right now the major sort of, let's call it inter area work has all been of the former type. Right. So people doing orthodromic anadromic experiments where they look for connected neurons and they find certain types of neurons are connected, and those have taught us really cool things about how the brain is wired up.
But I think for computations that involve lots of brain areas and lots of neurons all at once, I have a hope that we can sort of use statistical tools to kind of extract things from that group of neurons and things from another group of neurons and learn some of the principles about how they interact at the level of, of those, let's call them modules or motifs or that you used. Right. The hope is that maybe this kind of low dimensional analysis might save us a bit from understanding the nitty gritty details or needing to record every neuron.
[01:19:21] Speaker B: Yeah, the manifolds go a long way toward doing that. Talks, talk about manifolds. I mean, I just Talked with Randy O'Reilly and he has like this deep predictive learning mechanism that he maps onto like an early cortical area to thalamus and then from a later cortical area back to thalamus. And it's pretty detailed circuitry and it's a monster of a paper to read. And I'm reading the thing and I'm thinking, man, this thing needs a name. Like you need like a motif name or something, because it's really hard to keep everything in mind and I don't know. So it's like we're on the cusp of these motifs and a manifold, like I mentioned, may be one of those sorts of things that will help.
[01:20:01] Speaker A: Yeah, I wish I could give better, you know, have some magic guidance like, oh, if you just study it this way, I'm sure that's kind to work out. But I think the truth is that to me is one of the cutting edge questions in systems neuroscience. And. Fair enough. You said we've been working at this a long time and we haven't gotten to an answer to those kind of what you think of as elemental questions. And I guess I would argue, yeah, we've been working at a long time. But name me the studies that have been looking at the level of groups of single neurons in two different areas of the brain that had access to, let's say, at least dozens in each area.
And I think you can name those on a small number of digits, relatively. It's growing, it's absolutely growing and it's changing. But it's not decades that we've been doing that. It's this decade that really such studies have started to really pop out. So it's not been that long that we've been thinking about this. And my hope is that our thinking will mature rapidly in the next years.
[01:21:03] Speaker B: So you're optimistic.
[01:21:04] Speaker A: Yes, always.
[01:21:06] Speaker B: That's true. Yeah. That's another thing about you. Just chalk that one up and Matt's personality column in the grid going back to AI for just a minute. And then I have some career questions for you.
[01:21:19] Speaker A: Absolutely.
[01:21:19] Speaker B: Do you find. I mean, I don't know if you think about this, because we really haven't talked about this, but are there things about our intelligence that you can point to that you think, oh, that's going to be hard to build?
[01:21:32] Speaker A: Yeah. I mean, maybe this is similar to my other answer, but maybe not. I think that a thing that's going to be really hard to build is the kind of generality that we have to our intelligence. Right. When I think about the difference between different animals, you know, like, when I compare how my dog behaves to how people behave, I feel like the sort of classic difference is how flexible humans are, how much they can sort of generalize lies between different situations. And I know a lot of people are working on this issue in AI, Right? But I think this one is going to be a vexingly hard one. And I think the kind of generality that they're building is small compared to the true generality of our intelligence.
[01:22:15] Speaker B: Yeah. Generality of being able to play 40 Atari games is different than the generality of being able to go to the grocery store and go to the bank and then raise a child. Yep.
[01:22:25] Speaker A: Or play an Atari game of Pong and then go over and play ping pong. Right. You know, I mean, this. Like, this is not.
I think the scale of that generality really just dwarfs what AI can do. So I think that's. That's a thing that's really hard, and I don't think it will be solved anytime soon.
[01:22:43] Speaker B: So we're not going to end on such a pessimistic note.
So I have two more questions, and I promise I'll let you go. Do you have some. Something that sticks out in your mind as one of your best scientific, like, moments? And we talked about, you know what, maybe, like, Patrick Mayo's question to you. Is this the biggest thing you've done, but just as a satisfying scientific moment, do you have one of those that you can share?
[01:23:06] Speaker A: Yeah. So I feel like the thing I didn't anticipate, this maybe links to your other questions about running a lab versus being a postdoc or a grad student. The thing I didn't anticipate is how much fun it would be to take joy in the scientific discoveries of others or working with others on their stuff. And so I feel like, for me, the thing that is the absolute best thing about running a lab is when someone else discovers something that you were working with them discovering, but you didn't really do it. They did it. You helped them think about it and frame it. So, yeah, things like graduating my first student, and all of that feels like the greatest moments. In science for me. And when a student gets a result or presents something at a conference, they feel like they really got the intuition for something. Those feel like the greatest moments.
[01:23:55] Speaker B: I mean, it really is. Science really is about the people. That's what I miss. And I get that a little bit through the podcast. But I mean, and we're talking right now, we haven't seen each other face to face in four years or whatever you said. But it's genuinely so nice to see you, even though we're doing this in a podcast format. So anyway, I miss you, man.
[01:24:15] Speaker A: Yeah, I miss you too.
[01:24:16] Speaker B: Miss hanging out.
And I was hoping to cut you off before you return the compliment there.
Finally, Matt, you may or may not have an answer to this. What is something that you used to believe that you now consider naive?
[01:24:33] Speaker A: That I can just average across neurons and trials and figure out how everything works?
[01:24:39] Speaker B: I believe that in my entire academic career. In fact, I'm not sure if I've let it go yet. So that's naive. Is that what you're telling me?
[01:24:47] Speaker A: It depends on your question. It depends on your question. But yeah, thinking about averaging across, just sort of taking a bunch of things and pulling them together and solving questions about the brain. I don't mean that that's naive that there's not questions that can be solved that way, but I used to think more in that direction and now all the questions that I want to solve require that I not think that way at all.
So in that sense, my thinking has been very changed by.
[01:25:16] Speaker B: Yeah, okay, that's a great answer and humbling to me.
Matt, thank you so much. This has been a lot of fun. Let's not wait another four years. And I just really continued success to you and I'm really happy for you.
[01:25:28] Speaker A: Thank you, Paul, and best wishes to you as well. And thank you for having me on inviting me. It's been really fun to chat and I'm really impressed with all you've accomplished on your podcast too.
[01:25:51] Speaker B: Brain Inspired is a production of me and you. I don't do advertisements. You can support the show through Patreon for a trifling amount and get access to the full versions of all the episodes, plus bonus episodes that focus more on the cultural side, but still have science. Go to BrainInspired Co and find the red Patreon button there to get in touch with me. Email Paul. BrainInspired co. The music you hear is by the New Year. Find
[email protected] thank you for your support. See you next time.
