Dr. Bevil Conway — Coloring Inside the Minds

>> Diego (narration): Over the last couple of weeks, I’ve been shopping around for a new color to paint the walls in my bedroom. I’ve looked at what honestly feels like countless swatches, each with their own color ordering system and roster of clever names, which I’ll admit I do get a kick out of. Shout out to “Chrome your own way” and “Positive Slate of Mind.” But narrowing down the actual color options is proving to be more difficult than picking out my favorite name.

And that’s because depending on where I put the samples, the colors seem to change. The chambray I picked out under the fluorescent lights of the hardware store, seemed to lose some of its blueness by the time I get back home and hold it up natural light coming from my window. And depending on what other colors I put it next to, it appears to change yet again. Color me confused, I guess.

The whole situation is very reminiscent of the viral photo of “the dress” from 2015 that some people saw as black and blue, and others swore was white and gold and it’s enough to make you question whether we’re really living in a simulation and if reality is just a construct.

Existential crisis aside, it’s clear that the way we perceive color isn’t so… black and white. So why is our experience of color so mutable and complicated? In looking for answers, I turned to Dr. Bevil Conway, a senior investigator of the Sensation, Cognition and Action Section at the National Eye Institute. Dr. Conway’s lab focuses on the neural structures and operations that give rise to perception and cognition. So in other words, how the brain receives and generates knowledge about stuff in the world. And he’s using color perception as a model to gain insight into these fundamental processes.

Recently his team published a study that looked at how different colors stimulate different patterns of brain activity. Using a technique called magnetoencephalography or MEG, that captures electrical impulses from firing neurons, Dr. Conway was able to organize colors based on how our brains react to them.

During my conversation with Dr. Conway, he explained the results of the study and what it means in the larger context of how our visual system colors our perception of the world.

>> Diego (interview): So, Dr. Conway, when I first started to read up on your research, and I saw that you work to decipher the neural basis of color, I flashed back to being a kid on a playground looking up at the sky and asking my friends to describe what they saw. Of course, most of them were like, “blue, duh.” But I was frustrated not knowing exactly if their blue was what I saw as my blue. And I remember wishing I could kind of look inside their heads and compare what we saw, to see if it was indeed the same thing. So, I was really excited to see that you're actually doing that. You're peering into people's brains to see if and how different colors kind of induce the same neural or brain activity. So can you kind of describe that line of work?

>> Dr. Conway: Sure. So, we're interested, as you've described, in trying to understand what the mechanisms are in the brain that enable us to see color. So, your question, you know, do you experience color the same way I experience color—I think, you know, peeling back the layers of the onion, as it were—I think it really does get at this question of, like how do we transform sensory information from out there into perceptions and thoughts? And is that process a common process for you and for me? And you know on first principles we would think it should be, right? Because there's no reason to think it should be any different in you and me. I mean, the evolution hasn't gone out of its way to craft individual brains, you know, out of whole cloth. It sort of works within a set of constraints.

>> Diego (interview): Yeah.

>> Dr. Conway: We have similar genetics, we have similar brain anatomy, our neurons work similarly. Why would your experience be any different than mine? But, I think, you know, someone could come along and make the alternative argument and say well, “you look different than I do, you dress differently, you speak a different language or you have different sort of cultural history, different experiences, so of course your experience is going to be very different from mine.” And why should that difference not extend to your experience of color?

>> Dr. Conway: So, the question is, ultimately an empirical one, to what extent is your experience of color similar or different? And the way we've taken up this problem is to try and sort of crawl inside the head too—to watch the brain work directly unmediated, as it were, by language. So, we can't always use color language as the objective metric of what you're seeing. People have different numbers of color words. We go to different languages and different languages around the world have different numbers of color words.

>> Diego (interview): Right.

>> Dr. Conway: But new techniques in brain imaging like magneto encephalography and functional magnetic resonance imaging and electroencephalography, EEG, are allowing us to noninvasively watch the living human brain work, while people perform tasks or look at stuff. And we use this technique, magneto encephalography, MEG, to watch the brain in action while people were shown colors. And the logic of our experiment actually is quite simple. You know, it's very hard for me to ask, do you see the same blue as I do. But what we can ask in psychology is if I gave you a color, can you pick colors that are similar to that one? And then, we can look to see whether or not those similar colors are the same in me and you. So do we pick, you know, the same kinds of sets of colors to go together. And so, the question we asked is, is the pattern of brain response between two colors more similar if we think of those colors as more similar perceptually?

>> Diego (interview): Oh I see. So you were looking to see if the brain categorizes colors the same way our conscious selves do. So for example, is the brain pattern cast by orange more similar to red than it is, to say, blue, since that’s how we tend to group them together.

>> Dr. Conway: Right. We can think of a psychological representation of color as, you know, putting red next to orange next to yellow, you know, kind of structuring color chips from the paint store or something. We can think of that as a psychological color space, how do you organize them. And then we can say, looking at the brain, how does the brain organize these colors? And it turns out that there are some striking similarities between the way your brain does it and mine. And those similarities then get manifest in a kind of neural geometry of that looks in many ways, a lot like perceptual color space. So, for example, we think of warm colors as being more similar to each other than they are to cool colors. So, reds are more similar to oranges and yellows, than they are to greens and blues and greens and blues are more similar to each other than they are to the warm colors.

>> Diego (interview): Gotcha. And that neural geometry was detailed enough that you could tell what color people were seeing based on the pattern of activation in the brain? So they are kind of similar across individuals?

>> Dr. Conway: Absolutely, yes. So the first thing we discovered was that actually the spatial pattern of responses picked up with magneto encephalography, is sufficient for us to kind of mind read, to decode what color you're looking at. And I have to say, that I was very surprised by that. Because, you know, magnetoencephalography is the surface electrodes. Each electrode is picking up activity from hundreds of thousands, if not millions of neurons in the brain. The device itself looks like one of those old-fashioned hair dryers that sort of sits down over the head with a little, you know, opening where people can look out and then look at the screen. So, the idea that you would have a spatial pattern of response across those electrodes that would be sufficient to decode color just seemed impossible to me. It’s a fun backstory there. I was persuaded to try the experiment, because I had a brilliant young postdoc fellow in my lab named Katherine Herman back in 2014. And she was very interested in these powerful tools that were being used in machine learning and artificial intelligence and being deployed and analyzing brain imaging data. And she sort of said, you know, maybe there's some signal there if we looked in a kind of multivariate way at the pattern of response across the brain. And I thought, well high risk, high reward, that would be amazing if it were true. It probably isn't, but let's give it a shot. And lo and behold there were the signals. We could do this decoding.

I think when we’re doing science, we have to give ourselves the freedom to be broad-minded or think outside of the box. So the description that I provided earlier about the geometry of that neural representation of color, that really fell as a kind of next step that emerged as a result of now being able to decode colors using MEG.

>> Diego (interview): So, being able to match these neural patterns with a particular color then led you to build a neurological Pantone of sorts I guess that’s what I've seen other people describe it as.

>> Dr. Conway: Right. Yeah. So that's one way of thinking about it. People have invented dozens of different kinds of color spaces. The history of color spaces goes back to Aristotle. This is thousands of years ago, where he built up kind of picture of what colors meant what and how they related to each other and so on. And now understanding color order systems and inventing color order systems is a multi, literally, multibillion dollar business. And, you know, there’s so many. There's like the Munsell system and Pantone, which is related. And then there's, you know, your computer graphics cards have their own color systems. And then there's the International Commission on Lighting has its own system. And so, one way that you might think about what we discovered, is an organizational structure kind of color ordering system that's really written into the hardware in the brain.

But you know, it isn’t static. That the geometry that we see in the brain is actually dynamic. It's changing as a function of time. So, what I mean by that is, if I show you a color, the brain's response to that color isn't sort of a binary switch on, switch off, you know, there I saw pink and there I turned it off. It's a kind of dynamic evolving response. And those temporal dynamics are themselves different for different colors. Which means that the relationship between the spatial patterns in the brain is going to evolve and is different as a function of time after you see a color.

>> Diego (interview): That's interesting. So there’s no one specific brain region for say blue or yellow. If someone were to suffer head trauma or injury to a specific location in the brain, it’s not like they wouldn’t be able to distinguish a color that they previously could before the damage, because like you said, it’s a time-dependent coordination between different parts in the brain?

>> Dr. Conway: Exactly. That's right. A long history, shows that almost every brain area that is involved in vision, incorporates color information to some extent. But what's true is color is this kind of distributed, almost like a kind of hidden operating system in the brain, that governs a lot of visual computations. And so, it almost makes sense that color would be dynamic. Because these different brain areas are going to be related by different numbers of synapses to the rods and cones in your eye. And are going to be related to each other and the feedback mechanisms and the output mechanisms to a different extent. And so, it almost stands to reason that color would be a kind of dynamic evolving property in the brain.

>> Diego (interview): Yeah. It’s all so interesting to think about. How the notion of color really lies inside the brain not so much the eyes. From what you're telling me, it's like color isn’t intrinsic to wavelengths of light. I mean it matters how light bounces off an object or whatever wavelengths get reflected or absorbed. But color isn't just how light hits our retinas. And I now guess I'm thinking about those like optical illusions where you think two blocks on a different background are different colors, but when you put them side by side, they're actually the same.

>> Dr. Conway: Right.

>> Diego (interview): Do we know why the brain processes those blocks that way, as different when they're actually the same?

>> Dr. Conway: Yeah. So, if color were simply tied to wavelength, then it wouldn't be very useful. And the reason why is that the lighting conditions change constantly. So, as you walk around the world, you know, if you're looking at a scene that is illuminated by bright blue sky where there's no direct light, then everything is going to be kind of infused with that short wavelength, the blueness of the light source. And somehow your brain has to color correct, has to remove that blue confound in order to be able to see, you know, colors in a stable way, in a constant way with a change. So as that object then moves, or as the cloud moves away, or as the sun becomes direct sunlight, the spectral properties of the light coming off a given object will then change. And so if color were simply tied to the wavelengths of light entering your eye off of a given object, then the colors of objects would change with different lighting conditions. So a given object would look a very different color under broad daylight versus sky light, versus shadow, versus indoor tungsten light bulbs, or all of the LED lights that we now have. And that just wouldn’t be very useful.

>> Diego (interview): Right.

>> Dr. Conway: And so evolution has worked very, very hard to give us a situation, a system, a brain system that renders color relatively stable in spite of changes in light conditions. And the consequence of that system is that the colors are now no longer simply physical properties entering your eye. They're now a kind of cognitive or neural computation which then leads us to ask this question whether or not color is a property in our heads or a property out there in the world. So the very feature of the system that made it such that we can experience color to be stable in the world is itself causing us to ask whether or not color is not a property of the world. So those cubes you're describing, where you have a little surface patch that looks a given color under one context and looks at different color under a different context, all that illustrating really is that your brain isn't calculating color based solely on, you know, the light coming from a given patch. It's basing it on the context in which you see that patch.

>> Diego (interview): Right. That makes sense, this is all so interesting, just like our experience or perception of color, how complicated it is. And also it’s not completely rooted in our biology. I mean there is like a cultural aspect.

>> Dr. Conway: Yeah, I mean I think the question about culture and biology, the interplay between them, is really very fascinating. And that is one of the reasons why I think color is a powerful system to study how the brain works. Because those so-called cultural influences, which are, you know, either embedded in your biology as a result of your evolutionary history. Or because of your development, you know, the exposure to the lights in the world, or the exposure to your language, or your food, or your local environment, or the exposure to artificial colored objects, other kinds of technology. All of these influences are not bugs, they're features of the system that really allow us to use color to understand those interactions, which is important because those interactions are really at the heart of how brains work. What is a hallmark of what makes a human brain such a powerful organ is it 's ability to integrate and to adapt to changing circumstances. You know, I often say what's fascinating to me about, color and the color visual system, is that it’s a really nice, tractable model system for understanding fundamental processes of brain function that underlie, you know, many aspects of perception and cognition. And so in this way, color kind of extends to being really a kind of playground for understanding how does the brain do memory and emotion and thought. And I like to underscore that, because I think there's a kind of pervasive feeling among many that color is sort of just that thing where you think, oh that's kind of neat. But it's not really the real thing. It's not sort of a thing of real substance and consequence. And I think a lot of my lab's work has been to try and uncover the ways in which color really is of fundamental importance to human experience.

>> Diego (interview): Yeah, for sure. Well, your passion for color isn't just from a scientific perspective, you're also a visual artist. Would you say that your science informs your art, or vice versa?

>> Dr. Conway: I've been asked this question before. And every time it stumps me. Because I don't see the science I do and the art that I do as independent pursuits. I see them as very much grounded in the same sets of questions, or the same kind of curiosity about how does the brain work. An artists working in the studio—and here I mean me—when I'm painting, I'm doing science. There's a kind of empirical experimentation that's taking place. You know, I sit there with a palette of dozens of different watercolor paints, for example and I'm squirting them out. Or different spools of brilliantly colored threads. And I'm trying out to see how different colors might work together. And the goal at the end of the day is to find something that's sort of effective. And by effective, I mean it resonates with something either my expectation or my prediction or it generates some kind of surprise. But I'm also looking for something that's reproducible. I want to be able to do this thing again. You know, art doesn't work if it's only, always accidents. You know, accidents are helpful because they may direct you to ideas that you hadn't considered before. But then the idea is that you've got a prepared mind that can take advantage of those accidents and exploit them and then be able to make them reproducible. And that's the same spirit as what guides me in the laboratory. It's just the resources are different, the tools are different. But the end goal to understand how the mind and the brain work that's pretty much the same thing.

>> Diego (interview): Right. Is there like a moment that you remember where your fascination for color originated? Is it something that was based more in the art, like the visual, almost like spiritual seeing color? Or is it more of like the curiosity behind what happens inside the brain? I guess you're saying it's both. But is there one that kind of led to the other?

>> Dr. Conway: Yeah, I mean that's an easy question for me because I, you know, I didn't have the scientific resources until relatively late in my career. And my interest in color dates to, you know, my very earliest memories. My family's originally from Southern Africa. I was born in Zimbabwe. And we would go down from Zimbabwe to South Africa to spend time on my mother's family's farm. They were a sugarcane farmer in a little corner of South Africa, not far from the Swaziland border. And I had lots and lots of first cousins, but most of them were girls. And my older brother, and the only other boy in the clan, would go off into the bush and leave me alone. And my aunt Anne, she was a potter, she took me sort of under her wing and gave me a little corner in her art studio on the farm. And I just remember being mesmerized by the glazes on pots. You know, there is really a kind of alchemy in glazing where you paint on the outside color on this pot and then you fire it. You put it in a very hot oven. And it changes color and becomes these brilliant blues and reds and rusts and oranges. And I just thought that was absolutely magic. It's like, how does that happen? That's so cool. I mean you've really transformed what this thing was by changing its color. I mean it's gone from being this drab, you know, pot that I don't really care about into this jewel. And I couldn't understand why everybody didn't think that was the most amazing thing. And that's why I got interested in making art. And that same aunt gave me my first set of real watercolors, you know, the little ones in tubes. And still to this day, I will squeeze out these brilliantly saturated pigments, and then make puddles of color and just watch them like a kind of movie spilling into each other on a good sheet of, you know, of red cotton rag paper and you just watch. It's like it's thrilling.

>> Diego (interview): Well, I have to ask though, what is your favorite color?

>> Dr. Conway: I have many favorite colors. I would say it depends a lot on the context. So for example, if we're talking about my favorite color of a sky, I actually really like when the sky is covered in thick low hanging heavy rain cloud and then the sun dips right below the cloud and it illuminates the whole sky to be this kind of orangey brown. That's one of my favorite colors. But you know, I like it in that context. But I don't like it sort of on a plate of food.

>> Diego (interview): Yeah.

>> Dr. Conway: And so you know, I think again the question, what is your favorite color is a fascinating one. Because there is a hidden question which relates to the context. It's like favorite color for what? And again, I turn it back and say, why do we ask that question about favorite colors? I think what it tells us is at some level we do actually distill out color to be an abstraction removed from any object, from any context. And color then becomes, in some sense, a kind of pure idea. So when you say what is your favorite color, that's really evidence that we're really doing that. We really do, at some point, take color all the way through, and our brains pull out all of the context and end up with this almost platonic pure thing. It's just an idea. And we're then probing what that thing is. And in that framework, you know, my favorite color is orange [laughter].

>> Diego (interview): [Laughter]. Well now that you’re saying it, favorite color to me is usually tied to emotion. So, it's something that, like you said, kind of see in your mind's eye. And just kind of completely isolate from everything else, depend on the context. Like clothes I usually like darker clothes. But I know that for me my favorite color, or the one I'm most mesmerized by, a greenish blue. It is something that is very calming to me.

>> Dr. Conway: Yeah, it's interesting. We did this little poll in my lab recently, asking everybody what people's favorite colors were. And a kind of greenish blue and turquoise, you know, that seems to be a very common popular color right now. I mean these things are also subject to change. But it's fascinating to me and I've wondered where that comes from. I've wondered whether or not it's a response right now to the geopolitical global health, all the numbers of crises we're experiencing. You know, COVID and Afghanistan and so on, whether there's all of this instability, and we're sort of seeking out those colors that leave us grounded and stable and connected to nature. I'm curious whether or not your favorite color has changed over time, or whether you have any insight where it comes from?

>> Diego (interview): Yeah, I remember being a little kid and saying that red was my favorite color. And obviously they're diametrically opposed: green and red. So I think it is something that evolves probably with your world perception and also influenced by the world kind. Like I'm sure, you know, media has influenced what colors I prefer. I know millennial pink was a thing a couple years ago and everyone loved that color. And you saw it everywhere. So I'm sure it’s not just your isolated brain picking your favorite color.

>> Dr. Conway: Absolutely. And I think that points to this other key feature of color, which is its role in social communication. As visual neuroscientists, we sort of think of colors as, oh that's a cue that tells me what objects are out there, or where object boundaries are or something. And it becomes this, we call a low level visual cue. Which I've always found very dismissive. Like it's just that low level stuff that sits in the gutters, you know, it doesn't really do much. Whereas I think color is actually this fascinatingly high level cognitive thing that governs so much of our social interactions. The very fact that Pantone puts out a color every year, this is years in the making. At least 18 months before. And it's a marketing tool. It's like we're going to try and figure out how we're going to make a color that will be everyone's favorite color that year. And we want it to be a color that they all think they chose themselves.

>> Diego (interview): Yeah, yeah, yeah.

>> Dr. Conway: I don't mean to be cynical, but this is the power of color. It’s serving this role for nonverbal social communication.

>> Diego (interview): Yeah, I think it definitely speaks to the idea of color being a mode of thought, even if it's maybe not your own thought.

>> Dr. Conway: Yeah.

>> Diego (interview): Ok, well thank you so much for a great conversation. This was honestly super fascination. I could keep talking to you about color, but I know you have a full day’s work.

>> Dr. Conway: This has been super fun. Thank you so much.