[00:00:01] Speaker A: Keeping in mind that the point of neuroscience is not to engineer something is important, and that allows us to sort of identify like, well, what aspects of it are important. I think it will continue to be important though, because the brain continues to be sort of like the only example of this kind of general intelligence that we want to construct in AI.
Most likely the most exciting kinds of AGI that we're going to come up with is going to be tightly interconnected with humans and is sort of going to be interwoven throughout our society. And so we'll need to have a good understanding of what induces cooperative behavior.
There's always challenges, there's always sort of internal conflicts between our altruistic impulses and also like acting in our own self interests.
This is brain inspired.
[00:01:17] Speaker B: Hey guys, it's Paul. Who do you think would win in an altruism competition between you and an AI agent? If that AI agent was created by Jane Wong and her team at DeepMind, you would have some stiff competition. Today my guest is Jane Wong from DeepMind, and that is one thing that she's been working on lately. We talk about her multi agent reinforcement learning work to explore how cooperation can emerge from interacting reinforcement learning agents, which will become more important as AI becomes even more ubiquitous than it is now and has to work well with us and with other AI agents. It's also a potential way for us to learn about our own altruistic potential and how we might better reach that potential. The altruism or cooperation work is actually the last topic that we discuss. To get there, we review the meta reinforcement learning framework she helped develop, where a recurrent neural network's weights were slowly trained on a set of tasks such that the recurrent network could then by itself learn other related tasks without any more synaptic weight changes. And there's a story there about how these processes map pretty well onto what we know about how dopamine and the prefrontal cortex operate in brains.
And before any of that, we discuss at length her experience working at DeepMind, having come from an academic background in cognitive science, complexity science and neuroscience. And we focus a lot on the relationship and exchange between neuroscience and AI, since Jane has plenty of insight about both sides. Show notes@BrandInspired Co Podcast 83 if you value this podcast and you want to support it and hear the full versions of all the episodes and occasional separate bonus episodes, you can do that for next to nothing through patreon, go to BrainInspired Co and click the red Patreon button there. My thanks to Jane, for a very enjoyable discussion. These are exciting times in the study of and in the engineering of intelligence. Thank you for listening.
Was it over a year ago that I had asked you to come on?
[00:03:39] Speaker A: Yes, and yeah, and I'm really glad that we were finally able to connect on this. I was going back through the recent episodes and I noticed that almost all of the people that you've had on are people that I really love their work, I really love to follow their, their work and they're scientists that I really respect. And so it just kind of feels like it's all the kinds of topics that I'm most interested in. So I'm really happy to be able to come on.
[00:04:10] Speaker B: Well, I mean, you fit right in, of course, but it was funny because you seemed surprised that I reached out again. You thought I'd maybe forgotten about you or something, but no, not me.
[00:04:22] Speaker A: Oh, well, I'm very happy that you did.
[00:04:25] Speaker B: Anyway, it's good to have you on and we're here today, of course we're going to solve intelligence. Is that DeepMind's monster?
[00:04:33] Speaker A: Yes, it is. Technically our mission is to solve intelligence and then use it to solve everything else.
[00:04:38] Speaker B: Ah, right.
[00:04:39] Speaker A: And of course that last bit is quite overloaded, but I think it's very important.
[00:04:44] Speaker B: Well, you had a, I guess, brief career in academia. Right. And then transitioned to DeepMind. So you've been on. Of course, DeepMind is not really not academia. You wouldn't call DeepMind industry, but I want to talk just for a few minutes about what the experience of transitioning was like and what DeepMind is like relative to academia.
So what was your experience?
[00:05:10] Speaker A: I mean, you say brief, but it doesn't feel like academia was very brief for me because if you think about it, I've really been in some form of academic environment for like 20 plus years at the point. So I did a postdoc before going to DeepMind.
[00:05:27] Speaker B: In 20 years, that's going to look pretty brief, I think.
[00:05:31] Speaker A: I guess so, yeah. I guess maybe relatively speaking it's a bit brief. I think one of the main things, that's one of my main experiences about transitioning to industry from academia was at first just kind of being unsure and a bit scared about what it was going to be like. Just leaving academia is quite a daunting thing when the whole time that I did my PhD, my postdoc, it's sort of all that I know and sort of the whole point of being there was kind of to drive you towards getting like a faculty position.
[00:06:09] Speaker B: Yeah, well, let's Pause just for a second. So people know. I mean, you have a background in complexity and in physics, and that kind of led you into the neuroscience work, Right?
Is that the train of introduction?
[00:06:22] Speaker A: Yeah, I've had a pretty circuitous path to get to where I'm at.
[00:06:26] Speaker B: I think everybody says that, Jane. Everybody says that.
[00:06:30] Speaker A: Okay. It's probably true for everybody. It's not. Yeah, yeah. Maybe it's not that circuitous, but I started out in physics, so I did my PhD in Applied Physics, where I was more doing computational neuroscience.
So simulating complex dynamic networks, which are supposed to mimic, like, memory consolidation and memory interaction within the brain.
And so from that, I also worked a bit in complex systems, in graph theory and more on the dynamical system side.
[00:07:07] Speaker B: Yeah.
[00:07:09] Speaker A: But I didn't get into cognitive neuroscience until my postdoc.
[00:07:13] Speaker B: Oh, okay. But you were working with.
Well, I'd have to look at your CV again in your publication record, but I thought you did a little bit of neuroscience work also in graduate school. No matter.
[00:07:25] Speaker A: Well, I would say it was compute. I would call it computational neuroscience. It was setting up, spiking neural network models, but it wasn't that experimental. I did a bit of experiment with cell cultures and things like that.
[00:07:42] Speaker B: See, I want to change this notion of what people think of when one says neuroscience. When I hear neuroscience, I don't think experiment. I think of all of it. I think of the theory and the experimentation, the modeling. To me, computational, cognitive, it's all neuroscience to me. And I don't know why it's not that way for everyone else as well.
[00:08:03] Speaker A: Yeah, it is kind of weird. I think once you get into the field, people love to put labels on things and say, oh, that's not neuroscience. That's cognitive science. Or that's psychology, or. That's right. But I'm sure, like, yeah, from the outside, it all just looks the same. So, I mean, I definitely did work in neuroscience in grad school, but I say it's a completely different kind of neuroscience. So I wasn't, you know, super.
I wouldn't say I was an expert on sort of the brain regions and more of the cognitive aspects. Like, I had very specific knowledge about the hippocampus and how to model it, and that was sort of my experience of it. And then in my postdoc, I got a much broader sort of perspective about neuroscience and cognitive science and doing human experimentation, which, I mean, maybe this is a bit of a tangent, but to me, I feel like the cognitive science is a bit closer to AI than more of the maybe like electrophysiological type experiments in neuroscience.
[00:09:12] Speaker B: Yeah, yeah, I want to talk about this too in terms of like Mars levels and things, because I think that when people think of neuroscience, they also think of Mars implementation level, as if neuroscience maps directly onto the implementation level. And what you're saying is that cognitive science would be more like the algorithmic and computational levels of analysis. But I don't know, maybe this is a fault of mine. I see them all as spanning everything. When I think of neuroscience, I don't just think of recording single neurons, you know, so maybe that's a fault of mine.
[00:09:41] Speaker A: I wouldn't. I mean, I think everybody could interpret it the way that they, that they prefer. I think it's more just a matter of scale and also maybe the thing that you're studying. So in cognitive science and more like related to psychology, you're kind of studying the cognitive function itself.
You're trying to study what are people actually doing in this particular situation, what are they actually learning.
Whereas if you are moving more to molecular neurobiology or things like that, you're looking at the mechanisms. And so it's kind of like a slight difference in terms of the thing that you're studying and your motivation. And I think for me, AI is more about the cognitive function itself rather than the implementation, at least with regard to neuroscience.
[00:10:31] Speaker B: Yes, I can see that. I just think that one should always have the big picture across level in mind. Right. While you're.
[00:10:42] Speaker A: It's good to keep the context in mind. Yeah, yeah. Why are we doing this? Yeah, for sure.
[00:10:47] Speaker B: So anyway, so then you had this background and you went and you did a postdoc.
[00:10:51] Speaker A: Yeah, I did. So I performed, you know, psychological experiments with humans looking at learning and decision making and memory and also doing neuroimaging. So with FMRI and EEG and also a bit of work with transcranial magnetic stimulation, which is a kind of non invasive stimulation that you can do with humans.
[00:11:15] Speaker B: Have you ever been stimulated?
[00:11:17] Speaker A: I have, yes, of course.
[00:11:19] Speaker B: I hear it's like a rubber band pop.
[00:11:21] Speaker A: It depends on the kind of stimulation protocol, but. Well, I was gonna say we're all involved in each other's experiments. So of course, other researchers in the lab, I'm in their experiments and they're mine. So yeah, I've had TMS done plenty of times. So I was doing repetitive stimulation for the most part. And in that case you're stimulating at maybe like 10 or 20 hertz per second. So it's kind of a little bit of a buzzing and if you do that long enough, it can start to hurt. But if you just do a single pop, it doesn't hurt at all.
That can do crazy things like move your thumb or maybe even move your entire arm.
[00:12:01] Speaker B: Have you had the. Well, first of all, we should just briefly say what TMS actually is. So there's a thing that's sitting just outside of your head, and there's a magnetic pulse that you can use to target very specific locations in the brain and stimulate neural activ. Or ablate neural activity. I don't know what. The most recent.
[00:12:18] Speaker A: Yeah, exactly. Yeah. So depending on the kind of protocol that you're using, you can either do inhibition or excitation.
And you're essentially just, like, inducing electrical activity in a very focal part of the cortical surface of your brain.
And. Yeah, and you can find all sorts of effects from affecting people's judgments to affecting memory to maybe you can also even induce visual effects if you do it in occipital cortex.
So, yeah, it's really interesting, and it's a hugely kind of growing and exciting field of study right now.
[00:12:58] Speaker B: Did you feel like. So did you have a thumb twitch? What was your experience? Did it elicit a behavior in you when you did it?
[00:13:04] Speaker A: Well, so the thumb twitching is actually what we do to try and get a motor threshold. Because it turns out for every single person, depending on, you know, just the thickness of your skull or other biological aspects, you will have a different threshold.
[00:13:19] Speaker B: Laziness.
[00:13:19] Speaker A: Yeah, yeah, maybe. Yeah, yeah. Just, you know, every person is different. So I'm gonna have a different sort of threshold at which my. My thumb is gonna react, for instance. So we sort of use that to calibrate the amount of induction or, like, you know, energy that we want to be putting in. And then we use that, and then we put it over a part of your brain that we are interested in studying.
So for us, it was to try and induce memory formation or to enhance memory, just like associative memory. And you can also do things like try or look at the effect on functional connectivity, which is what I did, because we understand that you have these different kinds of networks, like functional brain connections in the brain, where different regions of your brain are sort of functionally connected to each other, meaning that they're sort of. They seem to be interacting if you analyze the activity.
So we can detect changes in that network as a result of doing stimulation.
[00:14:25] Speaker B: Oh, that's cool. But anyway, okay, so you're a veritable neuroscientist at that point. And then DeepMind steals you away. But you were starting to say that you were a little wary of leaving academia.
[00:14:38] Speaker A: Yeah, I mean, I think the main thing that sort of gets like drilled into you in academia is that like, if you leave, you'll never be able to go back. You know, like there's kind of this like weird culture where it seems like if you leave then, oh, you've just given up or something and then like you'll never be able to come back. And it feels like you're closing a door. But that wasn't my experience at all.
I feel like maybe it's just because it's the kind of place that I went.
But DeepMind is, I would say it's like 75% research environment.
So I publish now more than I ever did.
I'm able to be fully in the academic world and to keep up to date on whatever kind of literature that I'm interested in. I mean, I guess the kind of research that I'm doing is definitely different.
I'm not doing so much of the experimental neuroscience side of it. Although we do have a lot of collaborations with academic labs. So we can publish on that kind of stuff or work with people that are doing those kinds of experiments. And a lot of those are coming out as well from our, from our team.
[00:15:51] Speaker B: So you haven't really worked in industry then? I don't know. Because DeepMind is this weird in between kind of place. And so I don't know if it's even fair to ask you what it's like outside of academia, because it's like you're halfway outside or something. I don't know. How would you characterize it?
[00:16:05] Speaker A: Yeah, I think that's right. Yeah. Our sort of model is more like Bell Labs, I think. Yeah, I think we try to model it after those kinds of research institutes that were like primary research institutes. And I say the main differences to academia are essentially that we don't have to worry about funding and grants, which is amazing. I never really enjoy that part of it.
And we can also collaborate much more readily, I think. So we don't have this notion of sort of like individual labs that work very closely within themselves and then like kind of sometimes collaborate with other labs. But it has to be sort of set up and they can take a while for us. I can reach out to somebody else on a different team and just start work on a project just immediately.
And there are so many people that do different things at DeepMind that it's really easy to reach across fields or across or to find somebody else with a different field of expertise that can help me do something that's almost like, in some cases, it can be a difficulty to kind of just stop forming new collaborations because at some point I feel like I have too many things on my plate and I need to sort of like, narrow down and just say, okay, no, I got to focus here.
[00:17:31] Speaker B: Decadent.
[00:17:32] Speaker A: Yeah, exactly.
[00:17:33] Speaker B: What's the general atmosphere like? Everybody happy DeepMind? I mean, is everyone optimistic?
And we'll come back to this in a little bit when we talk about the progress of AI, but relative to an academic lab, let's say, and we all have only had experience with a few different labs, so, you know, we have a small in to say, can some labs have very different character and flavor than other labs? But does it feel looser? Does it feel more business like than an academic lab? Or about the same? I imagine it's about the same.
[00:18:07] Speaker A: The only way I can describe is it feels like it moves faster. But I don't know if I'm conflating that with just like the fact that I've switched from neuroscience to AI, because it could be that.
[00:18:18] Speaker B: Yes.
[00:18:19] Speaker A: Yeah, AI just seems to move fast everywhere. I can't imagine what it's like to be a grad student in AI right now and to have to submit to conferences every few months. It just seems like, yeah, it seems so intense, but it does feel like it moves faster. Papers can, you know, projects can be done and out in a few months, as opposed to, you know, before I'd work on something for a couple years. There still are those kinds of projects at DeepMind that, you know, are large scale and we want to do right. And so we'll work on them for a while and, you know, there'll be like a large piece of work, you know, like AlphaGo. When that came out, a lot of people worked on that for a long time.
[00:19:00] Speaker B: But the rewards, I mean, you work a lot on reinforcement learning, and the rewards come much quicker in AI than in neuroscience.
[00:19:07] Speaker A: Oh, I don't know what you mean by rewards.
[00:19:09] Speaker B: Well, your project can be done in a month or two months relative to a year or two years. So there's a longer delay in neuroscience between when you begin and when you are quote, unquote, rewarded with a publication or whatever.
[00:19:25] Speaker A: I guess. Yeah, that's true. And in fact, in AI, you can just sort of release your work on Arxiv. You don't even need to wait for the whole five cycles of review, like submitting to different journals and things like that which we've all gone through.
[00:19:38] Speaker B: Yeah, that's happening in neuroscience now, too. And I don't know if maybe AI has led the way there. So it sounds like the transition was fine. You had slight reservations, but I'm sure you're glad you did.
[00:19:51] Speaker A: Oh, yeah, yeah. No regrets whatsoever on that front.
[00:19:56] Speaker B: So science is this beautiful thing, but it's also frustrating in that the more you learn, the more you understand how little you actually know how much more there is to learn. Essentially, I'm wondering how your experience in neuroscience and now in AI has changed your perspectives on either neuroscience and. Or AI. You already talked about how nice it is that AI goes faster, which, yeah, I just was jealous of even when I was still in neuroscience.
[00:20:31] Speaker A: Well, I'm not going to make a value judgment about that. I don't know if it's nicer or not nicer, it's just faster. Sometimes I miss being able to spend five months on one paper and just think very thoroughly about it.
[00:20:45] Speaker B: But has it changed your.
Do you have less reverence, more reverence for the brain, for instance, things like that.
[00:20:53] Speaker A: So I'm definitely much more informed about AI and machine learning than I was a few years ago. And I think having that kind of background allows me to see neuroscience in a new light because you can sort of compare the two. In particular, the goals and motivations of AI are almost like the opposite of what they are in neuroscience. Maybe not the opposite, but it's coming at it from a different perspective. So in AI, the point of it is to try and figure out how can I engineer a system such that I can create learning that's either human like, or something that can do superhuman cognition, like play go or something. Whereas with neuroscience, you're taking a system that already can do that, you know, human level intelligence or like even animal level intelligence. And you're trying to figure out, you're trying to dissect it down and see, like, well, what is that system? You're trying to characterize that system. So there's sort of like two sides of the same coin to me.
And it's very interesting for me to think about it like that because then that can tell you much more, you know, what can I take away from neuroscience research for AI? What should I take away from neuroscience research?
And what are the limitations in what I can take away? So, yeah, so I think that that's, you know, allows me to, I guess, form, like, manage my expectations of what. What you can take away from neuroscience. I think a lot of, you know, AI Researchers that I've spoken to and have talked to over the years maybe expect, like, they're a little bit disappointed that neuroscience just doesn't give them an answer about, like, well, how do you. How do you program this particular aspect of cognition? Right. You know, why can't we just take, like, what we know in neuroscience and just sort of, like, implement it? Well, the reason is that it's not. Neuroscience isn't trying to engineer anything. All it can do is sort of tell you this is how one system did it, or not even that. You know, like, it's such a. It's such a hard thing to dissect down because there are so many levels that you can figure out a neural system at. You know, you can talk about the cellular level, you can talk about the neuronal level, talk about the biological level, you can talk about the functional level. Like, functionally, what is it trying to do? And so you have to not only manage your expectations, but also identify the right level that you want to be taking these lessons from. So I don't think it particularly helps you to be looking at action potentials and specifically how signals are transducted from neuron to neuron if you're trying to set up an AI system.
[00:23:52] Speaker B: Well, part of the problem is we don't even know yet what is actually important, right? Yes. For a given cognitive function, like, what is the right level to even examine it. And something that I have come to appreciate more and more the older I get is how young neuroscience still is. I mean, it doesn't seem like it is, but relative to something like physics, neuroscience is really new. And I mean, it's just when I started in neuroscience, looking at the classic papers, right, Recording single units and just characterizing a single neuron's activity, you know, and it had been around already for a little while, but it was incredibly young at that point still, and it really still is. So neuroscience is slower, inherently slower than AI cycles, goes on, you know, slower cycles.
And so at this point, at this, like, right now, is neuroscience informative for AI still? Or has AI just en masse said, all right, we can't wait for neuroscience, we got to move forward, because obviously, AI is beneficial to neuroscience.
[00:25:02] Speaker A: Well, people might not, like, neuroscientists might not think that that's quite so obvious.
[00:25:09] Speaker B: Well, I just meant as a tool, but as theory generators, as theory generation, I think it's pretty well accepted, at least that there are interesting ideas coming out of AI. Maybe not that it's so helpful yet.
[00:25:26] Speaker A: Well, actually, I have a Lot of thoughts on that that maybe we can get into later. But in terms of the speed of neuroscience and how useful it can be for AI, I think it's got to move at its own pace. I think that keeping in mind that the point of neuroscience is not to engineer something is important, and that allows us to sort of identify, like, well, what aspects of it are important. I think it will continue to be important, though, because the brain continues to be sort of the only example of this kind of general intelligence that we want to construct in AI. And I think that we can continue to learn from it, and we can continue to always take lessons from it. And I'm really grateful that there are people out there that are willing to continue to do experiments on it, on the brain. And to me, they're sort of doing all the hard work, and I can just sort of reap the rewards and the lessons that they learn for my work.
[00:26:34] Speaker B: AI is where it is today. Deep learning is because of neural networks which are made up of artificial neurons, units which, you know, emulate the way, you know, really, really crudely emulate the way that brains are connected and function really crudely. But so that that basis was already there. And it's interesting to me to think about how much we still have yet to learn about how brains operate and how they collectively give rise to higher cognitive functions, and whether there'll be some discovery, some advance, that will then really impactfully inform AI. And I don't know when that will be, but it feels like it's really ramping up. So I don't know what is your sense of that?
[00:27:22] Speaker A: I think that it's. So some pretty fascinating areas of advancements in the cognitive science side and the neuroscience side is. And be able to essentially quantify how humans can most efficiently learn about task structure, learn about decision making. So there are things like Bayesian inference or hierarchical Bayesian learning. There are these kinds of accounts that are starting to be constructed that show that humans will perform optimal inference under bounded rationality or different kinds of resource constraints that allow them to act most optimally in the situations that we most naturally find ourselves.
And so I think that kind of work is, to me, most related to the kinds of things we might want to think about in AI.
And that's, incidentally, also related to meta learning, which is my favorite topic in AI is thinking about, you know, well, how can we set up these systems such that they can learn for themselves these, like, optimal. Like close to Bayes, optimal ways of performing inference, given a particularly, like, structured set of environments that it can be encountered with or particular set of tasks.
So, yeah, so I think that there's definitely parallels in the way that we can think about human information processing and cognition and the ways that we might want to structure our artificial intelligence agents.
[00:29:05] Speaker B: Would you say, as you continue your career, that you take less and less inspiration from neuroscience or more and more? Or is it just kind of a rotating cycle?
Sorry to harp on this.
To me, I mean, that's what the show is about. It's like the balance between AI and neuroscience. And so I'm trying to dig down a little bit.
[00:29:26] Speaker A: It's definitely a very interesting question and it is one that I tend to get more often than not. I feel like I definitely want to continue to be abreast of the latest developments in neuroscience, but I'm quite selective about the kinds of things that I pay attention to. And I think that we're now actually getting to a point where we can start seeing more of an interactive bidirectional flow between AI and neuroscience, the more and more people that are at the intersection. And because of that, I think we're going to start to get more work coming out that's going to be relevant for both fields. And to me, that's very exciting.
I mentioned a bit earlier about talking about this feedback from AI to neuroscience, where you can think about.
So one exciting thing that you can do is you can take. You can take it like an RL sort of setup that you have, or an agent that is learning on a particular set of tasks that you constructed, and you can then analyze it and see, well, what kinds of things has it learned?
What are the sort of like, biases, inductive biases that it's picked up from the environments. And then from that you can infer, you can say something about the task requirements that you've presented the agent with, and then you can then imagine taking an animal and then training that animal on the same set of tasks and then seeing, well, does it have the same biases that agents have? Can it perform to the same level? Can it learn the same kinds of structure that an agent can learn, or does it have a lot of its already like sort of preset priors that it's going into this task with? And I think that, you know, this is something that isn't as well, hasn't been studied as much in neuroscience and with animals is thinking about, well, what kinds of priors and inductive biases already exist. You know, a lot of laboratory task paradigms make the assumption that an animal just sort of comes in knowing nothing or with the uniform prior, and then you train it sort of up to ceiling and then you analyze the neuroactivity or something like that.
[00:31:55] Speaker B: Well, that's kind of what the most vanilla neural networks connectionism used to assume as well by not structuring it. That's changed with CNNs and these more structured networks, which I don't know why I'm saying that, because you know all about that.
[00:32:10] Speaker A: Well, yeah, I mean, networks also come with these biases as well. Yeah, CNN is a kind of an inductive bias. You build in biases all throughout by your choice of architecture and your learning algorithm and so forth. But the thing is with AI is that we know that with animals we don't know what kinds of biases are already existing.
We know that there tend to be certain things like they tend to maybe think of color as more of an irrelevance factor, like, whereas shape tends to be more task relevant in particular, like categorization tasks. But there hasn't been as much work done in just characterizing the priors that biological learning systems have. So I think that's one thing that having done work in AI, that's one thing that I've taken away from doing that kind of work that I think can be applied to neuroscience.
[00:33:11] Speaker B: You might get docked depending on how you answer this, but there's lots of talk these days about deep learning coming to a standstill, reaching its limits. Right. And I'm wondering, do you think so? Or again, maybe we can go off the record so you won't get docked in pay. But I imagine I know what your answer is going to be. What do you think?
[00:33:34] Speaker A: I don't think so. I mean, I think that there's a danger in getting too wrapped up in any one approach or to jumping on the bandwagon of deep learning or like this particular method of training and we could just sort of continue to throw more and more data at it.
But I think that the field of AI, you know, just even beyond deep learning is so rich and that there are still so many unexplored regions, areas of research that we haven't even tapped that.
Yeah, I think that we're at a position where we have so much data, we have so much compute that I think at any moment we can have a really amazing new algorithm that pops up and it shows us that it opens up an entire new way of thinking about deep learning and AI.
Recently there's been a lot of, a lot of excitement around like GPT3 stuff coming out of OpenAI, these large scale language models that are just, you know, they have tons of parameters, they have tons of data, and it's just amazing the kinds of, you know, like, structure that they've learned and the kinds of things that you can get them to do with the proper prompt.
And so a lot of people are now sort of thinking about, well, how do we characterize the limits of that, how do we characterize the kinds of representations that they've learned and how do we sort of leverage that towards building bigger things and better models? So I think we're going to continue to see these kinds of things where these little breakthroughs can help us to bootstrap and to get to even better things.
[00:35:30] Speaker B: It sounds like you might actually get a bonus. I thought you might get docked pay, but maybe if someone listens to this, they give you a bonus. All right, well, I mean, I've almost taken you in an hour now just talking about the deep mind and. But I appreciate you going down that road with me. So let's talk about what was the next big thing and then you published it, and then we're going to talk about some meta reinforcement learning and then we'll talk about some more of your recent work. So I talked about meta learning and meta reinforcement learning when I had Matt Botvinnik on the show long time ago, your colleague, and.
But this is work that you headed up as well, and you just. I didn't realize that meta learning is like your favorite topic. So it's good that I'm going to ask you about it, I suppose. But maybe we can recap just the story of meta reinforcement learning as it pertains to, as it maps onto the prefrontal cortex and the cortico basal ganglia, thalamic loops.
Maybe you could just summarize what you guys found because that'll lead into this more recent work as well.
[00:36:37] Speaker A: Yeah, so, I mean, this work is actually kind of a series of two papers, I would say, because we had a paper that was presented at COGSCI a couple years before that Learning to Reinforcement learn, which essentially is looking at meta learning from a reinforcement learning context. So the idea of meta learning is that you have these multiple nested scales of learning and that you can have sort of like an outer loop of learning, which is tuning an inner loop of learning. So essentially you can learn the learning algorithm that in the inner loop can sort of perform more quickly or like adapt more quickly to new situations. And we call it meta reinforcement learning because both sort of learning Loops are using reinforcement learning.
In particular, we use a deep neural network, a deep RL agent which is learning through just like policy gradient methods to optimize for reward. Over the course of an episode and every episode you can sample like a new task with different task parameters.
And over the course of seeing enough of these episodes, the inner loop is sort of able to quickly adapt. So the sort of. The simplest example that we gave in our paper is bandit. It's a two armed bandit. So a bandit task is sort of like the most the smallest unit of a reinforcement learning problem that you can have. Which is still interesting because it's just a single step sort of trial where you're asked to choose from one of two arms. It's just like imagine like Uranus, you're at a in Vegas in front of two slot machines. You have to pick one to pull, but each arm has some fixed but unknown probability of giving you a reward. And you have to just sort of over the course of say a hundred of these trials, figure out which one is the better one to pull.
But you need to sample both of them because you don't know what the probabilities are because they're hidden from you. And so you need to take your past history of observations and then from that figure out how can I best perform sort of exploration versus exploitation to get the most amount of reward over the course of this episode of my set of experiences with, with this one.
[00:39:12] Speaker B: Parameter set with the one bandit machine. Right?
[00:39:16] Speaker A: Yeah. And so, you know, it turns out that through just using a, just a deep RL agent that has, you know, just a recurrent memory. So the recurrence is quite important because it needs to be able to integrate information over time. So this past history of what it's seen and what it's done and so forth, and the kinds of rewards that it's gotten, all that's really important.
[00:39:42] Speaker B: You guys use an LSTM for that, right?
[00:39:45] Speaker A: Yeah, yeah, an lstm. And from that it can then map onto an appropriate policy to tell it, you know, well, I should be now I should pull arm A, now I should pull arm B. And then it turns out that you can actually take the system and it can learn to do something that looks close to Bayes optimal. So if I can just define this is why we use a band edit at first is because these sort of Bayes optimal solutions already exist and so that you can compare against them. So these called like these Gittins indices. And so yeah, over the course of sort of many training episodes you can, you can get an algorithm, you can learn a policy that can perform sort of close to these Bayes optimal solutions.
And it turns out that also if you apply it to even like to harder task distribution, to harder tasks that maybe are not just a single step bandit, but are some sort of MDP or some sort of task where you need to first gain information and then exploit it. So if you. So these kinds of tasks sort of can have arbitrary structure. So whatever kind of structure that you want your task distribution to have, the Meta RL agent will eventually be able to learn that and it will eventually be able to leverage that in order to perform really well within the span of an episode.
[00:41:23] Speaker B: But you haven't really talked about, because you have to present multiple different tasks to this network for it to be able to meta learn essentially. But those tasks have to have some similar structure among them. So there is that constraint.
[00:41:38] Speaker A: Yes. Yeah. And this is related to kind of the free, like there's no free lunch theorem.
[00:41:45] Speaker B: Right.
[00:41:46] Speaker A: So there, so this is a theorem that's essentially saying that, you know, a given algorithm is never going to be better than another one if you're sort of testing on all possible problems. So the key is that you need to have some kind of structure in the problems, and if your algorithm sort of can pick up on the biases that match that particular structure that exists in your problems, then you're going to do better than another given sort of solution.
So that's the whole key to meta learning, is that you're able to pick up on these inductive biases and these priors. You can learn them as opposed to, for instance, you can handcraft them in, you can build them in, which is the case with more sort of like Bayesian solutions. So I can define a Bayesian solution which is going to perform optimally on this particular task, but then I need to do that for every single kind of task. Whereas meta learning, like Meta RL is sort of like a general purpose method for you to say I have a task distribution, I'm just going to try and learn that structure and then I don't need to sort of define it by hand. Yeah. And so the relationship of this to like the neuroscience aspect of it, which is in the Nature Neuroscience paper that we published in 2018, is that it turns out that you can find signatures of this in the brain as well. So we had a series of, I believe it was like five or six neuroscience experiments that are just taken directly from the neuroscience literature. They've already been published and we can look at the behavior and the sort of like neural activity of animals that were trained and also humans that were trained in these tasks. And then we can train our RL algorithm, our meta RL agents on those same tasks. And it turns out that we can find sort of similar signatures in our meta RL agents that exists in humans or in animals, and the same kinds of behaviors as well.
[00:43:57] Speaker B: And to me, the magic in the story, because I actually still kind of struggle to think about this, is you have the slow outer loop training the faster inner loop, and then at some point you turn off the training, you turn off the wait updates. So then you just have this recurrent network and learning can still take place in the recurrent network, in the dynamics of the recurrent network without the weights being updated. And it's like magic because.
So then there are two things that are driving the learning in the dynamics. There's the rewards still coming in from doing the tasks, and there's the internal hidden state history that can just get shifted around in conjunction with incoming rewards to alter its behavior. And that seems like magic to me.
[00:44:49] Speaker A: Yeah, I mean, it's magic, but it's also very smart for evolution and biology to have come across. Right. Because a lot of times you have things that change in the moment. You have sort of, you know, constantly like new observations that you sort of need to adapt to.
And yeah, the thing about this work is that we're able to sort of point to prefrontal cortex as being the place where sort of this fast inner loop of adaptation is happening, that we can like find signatures of this, of being able to adapt to sort of changing context or changing incoming rewards and observations.
That's what allows us to sort of within minutes be able to adapt to a new situation. Whereas we know that sort of synapsic plasticity and things that require maybe like protein synthesis or long term plasticity, these are things that happen on a timescale of much longer. Yeah. So this is that more outer loop adaptation or learning process that we think sort of like is mediated by maybe dopamine and by synaptic weight changes in basal ganglia and things like that. Yeah. So it's nice that we can sort of, in a rough sense, I wouldn't say that we have an exact sort of mapping to the brain, but it's nice that we can sort of see the same sort of a system in a biological system that we find in an RL agent, in a meta RL agent that sort of like naturally emerges if you have these ingredients Set up if you have a task distribution that has structure in it, if you have recurrence, if you have this model free form of reward based learning and that you just get the meta RL to emerge from that.
[00:46:45] Speaker B: Yeah, it's really cool. And those two papers that you were just talking about, that we were just talking about are among a few others that are highlighted in this recent review that you guys have written about deep reinforcement learning. And deep meta reinforcement learning is one part of that story. And so I'll point to that review as I'll, you know, point to both the papers that we just talked about and this recent review about deep reinforcement learning. And so we don't need to talk about deep reinforcement learning. But that's the big rage right now, I suppose. And a lot of what you guys are working on, because I want to make sure we get to your newer work here. But are you still working on the meta reinforcement learning PFC dopamine story? Is there more coming or is that now a thing of the past?
[00:47:36] Speaker A: Well, I mean, I wouldn't say it's a thing of the past. So I'm still working on. On meta learning. Yeah, it's, it's not like I'm done with it. You know, I'm still quite like interested.
[00:47:47] Speaker B: Yeah.
[00:47:48] Speaker A: So I guess the, one of the things that we're interested in is trying to test some of the hypotheses that were made in the original paper to see if you can. That's the ultimate test of a good hypothesis or theories that you can test it at some point and falsifiable. Confirm. Yeah. Confirm or deny it. So there's, you know, we're interested in looking at how we can do that. I think one challenge to try and do that in particular is that, well, it turns out that I mentioned earlier as well, but in neuroscience there's not as much work that's being done in looking at the learning process itself. So a lot of times if you have an animal and you're, and you have a particular task paradigm, usually you train the animal first to ceiling, like to perfect performance or some kind of like, you know, steady state performance. And then you start doing neural recordings. Most people don't do neural recordings during. And so that's sort of one challenge in trying to like falsify this theory. But in general it's also just interesting to think about the kinds of.
So I've been reached out to a lot by neuroscientists that are curious as to, well, what can I do with this?
How can I incorporate the learnings from this into my work. And what kind of lessons can I take away in particular? How can you close that. That loop back from AI to neuroscience?
[00:49:31] Speaker B: And you tell them you can't because you're too slow, right?
[00:49:35] Speaker A: No, no, I encourage them to maybe, yeah, to think about looking at in particular that learning process itself.
And to look at, you know, what are the signatures of meta learning as opposed to just sort of learning in that inner loop. And can we identify those. Can we look at some. An animal that is at the beginning of learning and compare that to an animal that's at the end of learning? When I say learning, I mean sort of, I guess, training, where if you train an animal that can take, you know, months. And so, you know, I think it'd be really interesting to look at, you know, what's happening at the beginning of training an animal where I'm sure it's not at that point. It's probably just exploring, you know, it's not, not really like doing much that's of quote unquote interest, maybe from the task paradigm's perspective. But I think that it probably is quite interesting if you look at it from a meta learning perspective, because this is what meta learning is doing is that it's sort of like needing to construct that inner loop.
[00:50:43] Speaker B: Yeah, it is interesting. When I was in my, well, graduate school and for my postdoc, I had trained monkeys, but in particular in graduate school, I trained them on a metacognition tasks. So they had to keep track of their decisions and then make a bet whether they think they made the decision right or wrong. And that's really hard to train in a monkey. You know, it took months and so you'd spend those months without recording any neurons. And then finally you'd get them all trained up and then you'd stick the electrodes in and record the neurons. And we even published like a little learning, behavioral learning graph over time. But yeah, it'd be super interesting to have neural recordings and come up with a story about, I mean, in this case it wouldn't have been meta learning, but come up with a story about the progress of that training regimen. That's a lot of work too. And I can imagine it would be messy and not an easy story to tell, perhaps.
[00:51:37] Speaker A: Yeah, and I can totally sympathize with neuroscientists that they don't want to do that because it is just so messy. How do you even analyze that? And also recording for that long, chronically over months, that's also Just its own special challenge.
[00:51:54] Speaker B: Oh, yeah.
Less so these days, but it's getting better and better. All right, we don't have that much time left, so I really want to talk about altruism. And this isn't even your most recent work. I could reach into a hat and pull out any of your publications that we could have talked about, but I don't know. This one's interesting to me. So the title of the paper is Evolving Intrinsic Motivations for Altruistic Behavior. Why is Altruism Important? I should back up. So this is part of multi agent reinforcement learning. That's pretty big these days in multi agent game playing. And multi agent meaning multiple AI agents that have to work together to, you know, play a game, perform a task, and. But that doesn't necessarily. So that means they have to cooperate. But altruism is, well, to my understanding, and maybe you maybe have a different understanding than I do. Altruism isn't just about cooperation. So we'll talk about the importance of evolving the cooperative or altruistic behavior. But I'm just wondering, you know, why altruism is important. I'm not sure that.
[00:53:05] Speaker A: Oh, okay.
[00:53:06] Speaker B: Humans are terribly altruistic anyway, why I.
[00:53:09] Speaker A: Got interested in it. Yeah, well, this is sort of my first foray into multi agent rl, But I've always been just interested in how actors can sort of learn to coordinate behavior to accomplish something bigger.
And in particular, how can nominally sort of self interested actors such as RL agents, how can they come to coordinate, to be able to, you know, perform altruistic actions and things that are sort of good for the whole. For everybody? Altruism is, to me, you know, a very important part of human intelligence and an important part of why we are as, you know, successful as a species, as we are, because we are able to sort of act for the betterment of everybody as opposed to just ourselves. You know, humans are intrinsically social in nature.
[00:54:14] Speaker B: I don't know this. I don't know if Covid stands that passes that test.
[00:54:19] Speaker A: I mean, I didn't say we're perfect at it, but I think, you know, it's one of the reasons why we got to the level that we're at. And of course there's always challenges. There's always sort of internal conflicts between our altruistic impulses and also like acting in our own self interests. But that's also the interest, you know, that's the source of my fascination with it, because we do have these two sort of competing instincts. And how is it that we Got to the point where we were able to come together and build a society and to build all that we had built and to be able to act. Yeah, I think a lot of people are, or at least they want to be altruistic and to act for the betterment of others and aren't just sort of in it for themselves.
[00:55:14] Speaker B: Living in London, you're losing your American roots, I can tell.
No, I'm just.
[00:55:20] Speaker A: Yeah, I don't know. I mean it's. Yeah, I think maybe we spent, or at least I spent too much time on Reddit and Twitter and just getting too bummed out by all the bad actors out there. But I think in general people try to be good.
[00:55:37] Speaker B: Okay, all right, well, so you took a different approach because previous multi agent reinforcement learning approaches, they have hand engineered the altruism into the agents, right into the way that the agents behave. But you decided to evolve it. So tell me about that. Why was it important to evolve it And. Yeah, and we'll go from there.
[00:56:01] Speaker A: Yeah, so I mentioned before that there's kind of this puzzle as to how altruistic behavior even came about. Because if you think about.
Yeah. From an evolutionary perspective, why is it that we should act in like altruistically or for the betterment of somebody else if it doesn't benefit myself, if it doesn't allow me to sort of reproduce or to pass on my genes.
And so there's been a lot of different research into this and suggestions as to how it might happen. You can, you know, some people or like there's been suggestions that is this notion of reciprocity that you do something for somebody else with the assumption that then they will do something for you. Or there's also like kinship selection or even the fact that, you know, if you are in a closely knit group and you act sort of altruistically just within that group, then overall your group will tend to out compete out of the other groups that don't do that. And eventually your group as a whole will benefit from that. So I should preface this by maybe explaining what is the actual dilemma that we're even trying to solve.
At DeepMind, we always work in games. And so we came up with a game for agents to play that for us typifies this idea of, of a dilemma between acting in your own self interest and for the benefit of everybody. So these are called intertemporal social dilemmas. And these are just generalizations of matrix games, which the most famous example would be the prisoner's dilemma and Prisoner's Dilemma for people who haven't heard of it is essentially just this example of where you imagine that two criminals have been caught and they're being interrogated in two different rooms. There's no evidence or there's very little evidence to sort of convict either one of them. And so the cops really want, like, want a confession in order to be able to convict. And each of them have their own sort of decision points to make where they can either choose to cooperate or to defect. If they cooperate, meaning that they stay silent, they don't sort of rat the other person out, then if they both do that, then they sort of both will benefit in the long run. Or like they'll serve sort of a minimal amount of a sentence, short sentence, yeah, but if they rat the other person out, sorry, if prisoner A defects but prisoner B cooperates, then prisoner A would go free and then prisoner B would have like the maximum sentence. But then if they both defect, then they both will serve sort of a medium amount of sentence. So if you're acting rationally, you actually would always defect because depending on like whether or not the other person cooperates or defects, it's always better for you to defect. But then if both of you defect, then you're both worse off than if both of you cooperated. So this is just sort of setups to where if you just act just strictly in your own best interest, then everybody is worse off. And so the intertemporal social dilemma essentially generalizes that to multiple time steps. So these matrix payout games is always just a single decision that you make, you choose to cooperate or to defect. Intertemporal social dilemmas sort of are just like these grid world games where the agents can choose to sort of walk around and they can collect apples, or they can choose to, for instance, clean a river and contribute to the public good and things like that. So they take multiple time steps to sort of do.
And then they also have this notion of that there's this tension between acting in your own self interest in the short term. So if I collect this apple, then it's sort of I get a lot of reward in the short term, but in the long term it's worse off for everybody because I have now depleted our source of apples. And so this is the tragedy of the commons, where if everybody sort of raids the common good, then it's just going to be gone for everybody forever. And this kind of problem seems a little bit maybe contrived, but it actually typifies a lot of real world problems that we're Facing now, such as climate change, where we, a lot of people are sort of unable to act in the long term interest of everybody. Because in the short term we want to be able to continue using as much resources as we want. We want to be able to have short term rewards, even though in the long term we know that it's going to be worse off for everybody.
So we know that this is an issue that I guess humans already are not very good at solving. And so that's for me.
[01:01:13] Speaker B: Yeah, go ahead.
[01:01:15] Speaker A: Yeah, well, so that's one of the motivations for me to study this in these kind of multi agent RL settings. The most important point is that you have these sort of two different timescales. Because you have these two different timescales. It actually is a pretty perfect setting for meta learning, because meta learning sort of naturally has these like multiple timescales of learning. And this is where evolution comes in. Because evolution can be thought of as sort of the ultimate meta learner in my mind. It operates on a very slow timescale, but within, say, a single lifetime, an organism learn like it has a lot of different learning strategies that has presumably been evolved or been sort of instilled in developmentally. And so this is, you know, why the decision was made to address the evolution of altruistic behavior using evolution.
And the way that we do it is we essentially are evolving like social signals that can be passed from one agent to another. So we have essentially a, we would call a reward network which constructs the intrinsic reward that an agent can derive from social signals that are given by other agents.
For this first work, for the social signals, we just used sort of the observable external rewards that other agents are getting, like the number of apples that they're picking and so forth. But you can think of it as like, I can observe other agents being happy or sad. Like, are they smiling? Are they not smiling? Are they getting reward?
[01:02:57] Speaker B: Reward, yeah, okay, yeah, yeah.
[01:02:58] Speaker A: And they're social because they're sort of like passed between agent to agent. And so if I can evolve over this kind of signal and subject to certain constraints such that we all sort of need to be playing by the same rules, we all need to be valuing these social signals the same way, then it turns out that we can solve these intertemporal social dilemmas and we can get the evolution of altruistic behaviors, such as agents that refrain from collecting too many apples so that there's going to be more in the long term, or agents that will perform a public good action. Such as cleaning a river so that more apples will grow for everybody.
[01:03:41] Speaker B: So it's interesting, the idea of creating and evolving agents that behave in some sort of optimally altruistic manner.
It's interesting that they could far outstrip our own ability to do that.
I mean, is that the goal?
So there are two things. One, it's just we could use what you're learning about how to create artificial agents. We could use that to sort of shape policy. Right.
And our own behaviors. But the other issue, and I don't know what's more important to you, is that in the future, when AI is just, I mean, it's already all around us, but when we have these multi agents, AI is interacting with each other, well, they're going to need to be constructed and constrained in such a manner as to be able to cooperate.
[01:04:33] Speaker A: Yeah. So I think that this work can address sort of both of those aspects because, yeah, you can imagine a field of work where you take these artificial sort of RL agents and you try and figure out what kind of system or what kinds of like, intrinsic rewards can you give them such that then they're more likely to cooperate or they're more likely to sort of coordinate with each other. And then you can try and see if, like, maybe that would sort of generalize to the way that humans act or behave and see if you can sort of induce more cooperative behaviors. But similarly, it's also true that, you know, I don't think we can get away from the fact that humans are going to always be having to interact with AI agents. Like, I don't think that any AI agent that we could come up with is just going to be this monolithic sort of separate entity that will not need to be interacted with at all. I think that most likely the most exciting kinds of AGI that we're going to come up with is going to be tightly interconnected with humans and is going to be interwoven throughout our society.
And so we'll need to have, yeah, we'll need to have a good understanding of what induces cooperative behavior and what kinds of learning mechanisms or what kinds of loss functions are going to be most important for us to get the behavior that we value and that allows us to get ethical behavior from both these agents and from ourselves. So the main takeaway is that having these evolved reward networks, which are sort of the important thing is that it's the same reward network that every agent plays with. So if they're playing together, we call it like playing together. But if we put together five agents into one of these environments, then they all need to have the same reward network.
And then if we allow evolution of these reward networks over the course of many sort of interactions between these different agents, then you start to see cooperative behavior emerge. And then depending on the task, depending on if it's a public goods game or if it's a tragedy of the commons kind of game, then you get these different reward networks that are useful for these different situations because they're different kinds of tasks.
So this is kind of the power of evolving and letting it sort of like meta learn in a sense what should be the. The correct sort of set of rules. So to me, I can broadly sort of interpret these reward networks as being the rules of play. So if I'm going to be engaging with a group of other agents or people, then we all need to sort of agree beforehand. These are sort of how we want to be interacting. These are the rules. And if they're a good set of rules, then it will allow us to sort of get good behavior out of everybody.
[01:07:45] Speaker B: How many different algorithms are out there are in our heads for different situations?
[01:07:51] Speaker A: Well, I mean, infinite, right?
Anytime you come into a different situation, I think you're always sort of negotiating the rules of engagement. And maybe just through communication, you can sort of implicitly agree on how we should be interacting with each other. Every social network that you go on, like Twitter and, you know, Reddit, and like they all come up, come with their own sort of set of rules for how you interact with each other and so forth. So I think it's really interesting to think about these kinds of things to different systems that people can interact with each other under. And it induces different kinds of behavior.
[01:08:31] Speaker B: Why do I hate Reddit so much? Why can I not get in? Everyone tells me I need to do I need to be more into Reddit and I get nothing from it. I don't understand why anyone uses it.
[01:08:41] Speaker A: I like Reddit. I think it's super fascinating. And also Twitter, like the kinds of information that you can get from it. I mean, maybe you just need to follow different subreddits. I don't know.
[01:08:53] Speaker B: I'm trying, but I get next to nothing from Twitter as well. I just don't want to. This is too long of an aside, but I can see why they'd be useful to people. Go ahead.
[01:09:04] Speaker A: I will say I think we are just, we're very embryonic in terms of our understanding what kinds of social networks are most useful for communication. And, you know, I think right Now, a lot of social networks are built to sort of generate advertisement revenue or to increase engagement, most like, first of all. So I don't think that they're necessarily built to incentivize good communication, and that's probably why a lot of times you would find them sort of upsetting or that they're very distracting because they're not right now, I guess, built to do that. And I think that it is important for us to think hard about what kinds of systems would allow us to engage best with each other and to come up with ways to, I don't know, better coordinate and to form useful collective action and to coordinate with each other and altruistic ways and so forth.
I mean. Yeah, So I definitely think that there's a lot of work to be done on this front.
[01:10:11] Speaker B: Yeah, agreed. Well, I know I've already taken you over time here and we had planned because I know that you're interested in AI ethics, and we plan to talk a little bit about that. But instead I'm going to have to. I'll email you in a couple weeks and then you can say you're too busy and then I'll wait another year and then I'll email you again and maybe you can come back home because we really never talk about ethics on the show. And I know that's like a big topic of interest publicly as well. For now, my last question, random Patreon supporter question here. What's one? Before I let you go, what's one book that you would recommend people read and it could be that's been important to you or that it doesn't have to be even a science book?
[01:10:57] Speaker A: Yeah. One book that I read recently that is related to AI that I quite liked is called Human Compatible by Stuart Russell, which is essentially asking the question of how can we ensure that an advanced artificial intelligence is going to be able to, well, align with human values and to be safe to interact with? Oh, another book that I read, which I think is very good, is Algorithms to Live By.
[01:11:30] Speaker B: They're making a second part and interviewing a bunch of people because I had such success with the audio version of that one. I had Tom on the show, so.
[01:11:40] Speaker A: I like that book too.
[01:11:41] Speaker B: I can recommend that as well.
[01:11:43] Speaker A: Yes. Yeah, that one's definitely one of my favorite recent books.
[01:11:47] Speaker B: So thanks, Jane. It's been a long time coming. I really appreciate you coming on the show and continued fortune at DeepMind.
[01:11:54] Speaker A: Thank you very much. Yeah. And I look forward to coming back in a year and a half.
[01:11:59] Speaker B: Good, good.
Brain Inspired is a production of of Me and you. I don't do advertisements. You can support the show through Patreon for a trifling amount and get access to the full versions of all the episodes, plus bonus episodes that focus more on the cultural side but still have science. Go to BrainInspired Co and find the red Patreon button there to get in touch with me. Email Paul BrainainSPIRED co. The music you hear is by the New Year. Find
[email protected] thank you for your support. See you next time.
