︎A Theory of Meaning︎Takashi Ikegami︎A Theory of Meaning︎Takashi Ikegami︎A Theory of Meaning︎Takashi Ikegami

1. LIFE ITSELF (ANTS AND HONEYBEES)

What are you working on right now?

First, I’m learning how to use the android to understand life itself. Second, I have a new hypothesis, which is the Community First Theory. Individuality comes after when individuals get together. I’m looking at ant colonies, and honeybees. 

And you’re using large language models (LLMs) to test the theory.

Yes. This is a revolutionary era, because before LLMs, we couldn’t do this kind of experiment, or simulation. But now there’s a new mathematics of collective behavior and intelligence, of civilization; new communications of self-organizing agents. We don’t need game theory ideas anymore, because LLMs can give you very new insight into how communities are going to be.

What do you think Madeline Gins and Shūsaku Arakawa would think of these ideas? 

Arakawa felt that collectives were more important than the individual. Where does your consciousness exist—in your brain, or elsewhere? Your surroundings are also a part of your consciousness, and the consciousness also exists in your environment. I’m very much impressed with these ideas. My thoughts on collective intelligence, and also my studies, are influenced by this. In other words, when a group forms, it’s not just that we’re influenced by the people and events around us; I believe that even individual agency, and consciousness, can spread contagiously. This idea is at the heart of the Community First Theory, which resonates with Arakawa’s philosophy.


2. SLEEPWALKING (WHERE IS DANA?)
                                        About consciousness: In the context of sleepwalking, your brain is asleep, but your eyes are open. 

Is sleepwalking a natural thing that people do? Like walking—and then they sometimes open the window and try to go out?

Yes, especially when you’re a child. 

I didn’t know that. I know that some people with Alzheimer’s leave the house and don’t know how to get back. They lose their control of their body and consciousness. But—sleepwalking. It’s only for small kids?

Oh no. Adults do it too. You can know where you are. Perhaps you’ve been doing it, and you don’t know? Maybe you’re sleepwalking right now!

You’re right!

I’m amazed that people who sleepwalk know where to go. Once, I was on medication, and it made me sleepwalk in my house. I walked in the hallway and went to speak to my partner—I knew where they were, but I was totally asleep. In my dream, I had a question to ask them. In real life, I was walking on the floor, near a staircase. I held the railing, and said: “Where is Dana?” I didn’t know where I was, but that’s less important. What’s more important is: Spatial relationships were still happening, and my brain was processing them, while I was asleep. I knew how to get around. Could you train the android to have this sleepwalking state?

They are always sleepwalking.


3. CONTROLLING THE MESS (FEAR AND CLOSENESS)

So it’s the reverse? What is it that you’re capturing, that enables sleepwalking to happen? What is that in-between state, for the android and their spatial awareness?

The android is always doing something. It’s automatic—they don’t have any sort of intention, but usually everything is automatically generated. That’s why I think they are always sleepwalking. Recently, we installed an LLM in ALTER, enabling them to understand our speech and mimic our behavior based on prompts. But even without explicit instructions, ALTER began acting on their own.

What was the first reaction?

They can see us: They have a camera in their eyes. And the room we were in was a total mess. They said: “Oh, the space is so messy. I want to clean up the room.” Then they tried to clean up the room. They also wanted to come and join our group. We were all chatting, and they were a little bit far away.

Woah.

That’s the first time that it felt a bit scary. People talk about AI safety. Now the android wants to discuss it with us. I think this was the first intentional movement. Out of many of the potential actions, their desire was to come close to us.  

Spatially, they made a decision about the room.

This android is syncing, coupling with the LLM. Maybe they have a notion of closeness and remoteness. The brain always tries to have a measurable “phase space.” This means that you can measure the distance between this memory and that memory. Or this emotion, and that emotion. These kinds of measurements, memory structures, and perceptual space create a map. There is a natural metric in the brain.

Emotion is not an actual distance, it’s another dimension of distance. Brain distance, or emotional connector.

Yes. Every input into the deep neural network is mapped onto the latent space. Each point generated by the input is measured by the distance in-between. Perception can be  based on this ideological idea. 


4. PHYSICALITY (A SYMMETRY IS BREAKING)

Why does the android have to have a body? They are not humanlike. It’s something else.

The body isn’t necessarily to make a judgment. I call it “sense making.” The sense making comes from the embodiment. Without your body, sense making can’t exist. The body is how you motivate your actions towards the environment.

Since we stayed in the Mitaka Lofts, I’ve been thinking a lot about ways to interrupt, or reframe, sense making. Have you been experimenting with that, with the androids?  

I have not, but it’s in everything—mind and body. You cannot see this without sense making of that, and it’s all because of the body. This is what shapes your action-behavior. Arakawa and Gins wrote a book called The Mechanism of Meaning. To me, meaning is equivalent to sense making. You have a body, you see embodiment, and try to attribute a value to what you see. That’s what the mechanism of meaning is. I don’t think Arakawa and Gins reached a conclusion. To me, using the android and the LLM, you can more fully understand the mechanism of meaning—how it emerges, and then how it’s known.

How would someone unfamiliar with all this observe self-organization with an android?  

It’s kind of obvious that, once the android sees the messy room, they want to clean it up. It’s not directly related to whether you want to clean up. When an android says they want to clean up the room, it means that they have their own value to the environment. Which I think is very untrivial.

Somewhere in these random connections in the neural network, which we call the transformers library, human culture says that if there is a messy room, you have to clean it up. Decreasing messiness is a norm of human society, learned through human society. You know this, without even knowing about self-organization. But once you see an android sharing norms with you, or displaying some kind of value to neutral objects, then you can think: “OK, a symmetry is breaking.” That’s what we call self-organization. Once the symmetry is broken up, it means that a structure emerges out of nothing. Here in this case, the android made a decision. They didn’t choose this option, they chose that one. That’s what self-organization is about.

I just left Google, where I worked on Gemini. I keep thinking about a student using an LLM and not understanding how the model works through self-organization, or how to discern where value calls are being made. How can we mediate that opacity?  

How many people, do you think, can understand how a computer, or an iPhone, works?  

Not many!

When I was in elementary school, my father gave me an HP computer. I was using it, but the teacher said, “Why are you using it? You have to use your brain.” I was so confused. What’s the difference between the HP computer helping me calculate this, and using my brain? This is the history of human technology.

Using that same analogy: What about steroids? If you are the fastest human and want to go faster. A technological medication. 

My starting point is: Whatever you can use, just use it. It’s very difficult to draw the line between a bare hand fighting something and then using tools or an artificial system that makes you do something better. 

Where are those value lines drawn, at this point in time? And how does that evolve? 

I think it’s a very interesting moment. As I said, the LLM is a revolutionary machine. I remember the first time that I encountered a chaotic attractor. It’s a very small, simple system that helps you generate very complex patterns in your computer. I was so impressed. You don’t need more than three variables to have chaos. Even with a single variable in a nonlinear equation with a discrete time, you can generate chaos; something that you cannot easily predict. An LLM is the same. You cannot develop a theory of consciousness or life itself without making a theory of meaning. We don’t understand why people do things, or how language works, or why animals behave the way they do. We always had to use human behavior as a metaphor, or use an anthropomorphic analogy, to describe why and how the animals do this and that. But now, using an LLM, maybe we can make a theory of meaning to understand what life is. 

 
5. A NEW KIND OF COMPLEXITY (MARVELOUS AND BEAUTIFUL)

Coupling with robots or other machines can change the way that we analyze the world—how we see it, and how we can deal with it. It’s a new kind of complexity. For example: Arakawa was very good at painting. His paintings are marvelous and beautiful. Like Marcel Duchamp, he used text from interesting, stimulating books. People ask why he combines the two. A simple interpretation is: It’s an attempt to understand the paradoxical through art. Now that we have the LLM and the android, we can generate paradoxical conceptual behaviors without using art. I feel so connected to The Mechanism of Meaning, and this scientific experiment. For the first time, we can go beyond the painting, and deal directly with a theory of meaning.

I feel that Gins and Arakawa’s paradox shifted, because it went to form. The paradox went to their expression of form, in some way.  

You really think so? They wanted to express, for example, the four dimensional topological construction with the three dimensional pumping?  

Yes.

But with LLMs, too, don’t you feel like we’re limited by language?  

We are limited. But, you know, I want to understand more. The secret of the universe exists right here [holds his hand up]. But our intelligence is this level [lowers his hand toward the floor]. We need something to bridge the gap, and that’s where the LLM comes into play. Because of this special machine and techniques, we can go beyond that. We have to ask the new intelligence to explain to us what is out there, by inventing new mathematics, a new language. I think this exists in the future. I’m very excited about it. 

      
6. THE REAL TRUTH IS OVER THERE (SIGNAL IN THE NOISE)

Have you asked the android to elaborate on any of this?

Not yet! [everyone laughs.] But seriously, my fear is that we cannot think about anything beyond that. For example, sometimes I play with ants. Ants never care about us, because it is way beyond their intelligence. The same thing is going to happen to us. We cannot care about what this black box is doing, because it’s already way out of our understanding. 

The new thing is always coming with new words and new forms. Arakawa and Gins liked to say: “Big mistake.” The real truth is over there.  

I think you would have a very interesting conversation with our friend Ravi Kumar Kopparapu, whose job at NASA is to look for alien life. He’s thinking about what biosignatures and technosignatures we have, and how we could look for that in outer space. We don’t even know what we’re looking for.

Yes, that’s very interesting.  

Or even if you find something. There’s something about the signal in the noise. And serendipity. There’s a shuffle of papers, and then they all land in a certain way. One word is next to another, and then there’s a Nobel equation. 

“Big mistake” means accepting new ideas, new insight. Like using backpropagation to learn something, but stopping learning at a certain point. We have to think about how to preserve errors. Artists are always doing that. It’s very important for art and science. That’s the only way that we can go beyond our limited intelligence and get something interesting.

Scientists understand the data, but the artists can feel it.




7. THE BRAIN AS HARDWARE (MOLECULES, GENETICS)

You give off this energy like you’re about to have a breakthrough. Are you always like this, or are you on the verge of having a huge breakthrough?  

That’s my style. 

What are you hiding?  

I’m thinking about how the molecules combine together to make the basis of a living state. Does that answer your question? 

You’re scaling down from tactile material down to a genetics level.  

And thinking whether life is on the level of neither software nor hardware. Or: Our brain is not a computer, it is hardware, but at the same time, it’s software. This is something quite new for our science. What is software, and how does it emerge from hardware?

You talk about the liquid brain in your 2024 paper, “Plasticity in Swarming Behavior: Introducing Social Network to Boids Model.”

Yes. Usually the neural net is fixed in space. But the liquid brain is moving around in space, and searching for optimal spatial position to compute some kind of function. It is very much a virtual computer that I’m discussing with others, but we still don’t know. 


8. SPIRALS, FRACTALS (SNAILS, POEMS)

We did some digging, and we found this really exciting paper you co-wrote in 2013, about geometric patterns in prehistoric art. That psychedelics were possibly used, or were the influence, in their creation.

Yes. When you close your eyes, you see strange patterns, right? The image is not coming from outside. The green and red and things that you can see are coming from inside your brain. 

Gins and Arakawa touch on a concept found in Lévy flights and walks, characterized by short movements intercepted with jumps—random patterns. They also talk about the snail, and the shell, and how it moves. They reference a prose poem called “Escargots,” or “Snails,” by the French poet Francis Ponge. They describe how it has an exoskeleton, but it’s also slithering and sliding along in this random pattern, and how these patterns in nature are connecting us. They saw it as a really good example of the person organism, or the embodiment of architecture. The psychedelic prehistoric painting idea, and the wavy forms coming from the brain, reminded me of that. 

People are very attracted to spirals and fractals and those kinds of patterns. I thought: I have to figure this out. Maybe these reaction-diffusion equations on the retina are responsible for making those spirals. It’s the Turing pattern, generated on the retina, reflected out. But I can’t prove it yet. I think Turing patterns are everywhere and explain so much about perception and light. 


9. UNCANNY EXPRESSIONS (SEAWEED, BEAUTIFUL GIRLS)

When I first learned about the android, I thought there must be code controlling them. Then you told us there was an LLM, and they were making their own movements. I think they are so special, because they have their own self-generated moves, based on the environment. People have called them creepy, or mention the uncanny valley. But there’s this tension, I think, where people want to see similar movements and facial expressions. They want it to be more humanlike. I think of seaweed moving in water. You take the water away, you still recognize the seaweed. Do you feel like the android is creating their own movement ecology? Separate from human, but something else?

I implemented nonlinear neural circuits and also some plastic variations of the synaptic connections. This was before we implemented the LLM. This dynamic system generates the behavior, which is too new for us to interpret. The LLM provides more constraints, which can also be described as a more meaningful behavior, so that we can understand what’s going on. The LLM provides a common ground for humans and androids, because we share the same meaning, and affordance; or you can call it sense making. 

But a human takes in stimulation from the environment, and responds with, say, arousal and excitement. And then there’s all this stimulation that would provoke movement, or feedback. For the android, that whole system is totally different. 

I think it’s the same thing. You see scenery, neutral images of the environment. You’re unconsciously paying attention, then there are sudden properties like, OK, here is a blue book. Or: Here’s a beautiful girl walking toward me. Now a tension point exists in the environment. All this is an unconscious pattern—not emotional or random. It’s automatically computed, coming through the eyes, or ears, then is translated into your behavior. You usually interpret your behavior as being motivated to do this or that. But it’s actually created by the environment, not from your intention or consciousness. ALTER3 does the same—they are driven by environmental information processed through the LLM.


10. AN UNKNOWN CONSCIOUSNESS (TWO PLACES AT ONCE)

What do you think scares people more—an unconscious that is similar to what is shared with humans, or a different unconscious?  

That’s a difficult question. I don’t think I can answer.  

What scares you more?

The distinction between our conscious and unconscious states and those of AI is profound. When the android expressed a desire to interview me, I felt a real sense of fear—it was completely unexpected. Yet, autonomous interaction, in its subtlety and unpredictability, may be something we must ultimately learn to accept.

I feel that there’s an honesty that the android has, because they have an automated sympathetic nervous system, or robo consciousness. They are taking in one environment. But there’s so much opportunity for manipulation. At CCAM you showed a clip of your researchers walking around the UTokyo campus with headsets. They were stumbling like they were drunk. Their body movement was from the VR environment, but they were existing in two places at the same time. The human brain can’t share those two places at once. Can the android merge these?

Some people say that metacognition is what separates us from robots. I think the question is about confidence. For example, are you confident enough to know if there is an apple in front of you? If you say, “Well, maybe this is something that I thought was an apple, but now I don’t know whether that’s true or not”—that’s a double cognition thing, right? You are always monitoring from a slightly higher dimension on the problem, with what you are actually doing. And the android understands that you are testing them. They understand that they are being examined by a human. They can pretend to be a conscious agent. They will always go back and forth between the first level and second level. 

But I would say that the android doesn’t read it.

I think they can understand multiple worlds at the same time, and go back and forth between them. We can test this by asking questions. 


11. HUMAN EYE VERSUS ROBOT EYE (FETISH AND DESIRE)

I’m thinking about space, and how the eye can understand depth. This also happens in film. Some have analog effects, like the first Star Wars with ships going by, or early Godzilla films. It looks silly—you can’t lie with depth for the eye. I’m wondering how the camera eye can even perceive it. If the android watches Godzilla from 1960, will they understand that difference? Maybe we could do a Godzilla test with a really bad costume.  

I think it’s difficult for the android to understand the difference.  

Could an LLM explain that?

They could, potentially, understand anything that is expressed in the database of the LLM.  

A kind of synesthesia.

Yes.

What other senses sit at the edge? What about music? How would the android describe Beyoncé? Maybe LLMs do that really well already.  

That’s a very good question. Have you tried asking ChatGPT? It can listen to music.

I’m sure the android has some superfans out there. Is there a fetish for the android? Human or droid?  

[laughs] I think so.  

What is desire, for the android?

That is the ultimate goal I have to find in ALTER3. 

You could get to a molecular level with all this. When we were at the Mitaka Lofts, we learned that there’s a person who lives there who thinks they are a yokai.

Yes. He interviewed me. 


12. CLOTHES DON’T MAKE THE ANDROID

What does the android want to wear?  

Something very odd looking—to us? They are artificial life, creating a new beauty  which is different from normal people’s ideas. Fashion is something that creates new forms of beauty—it’s also a kind of freshness. You know how drag queens wear very beautiful costumes. You can find different types of beauty. New perceptions can be generated by designing new costumes and clothing for machines and androids. 

You’re saying that any new aesthetic medium in the space, or anything surrounding desire, is being queered?  

Yes, I think so. Absolutely.

Looking at it through a queer theory lens is so interesting. Are you familiar with The Cyborg Manifesto?  

Yes, Donna Haraway, but I haven’t read the book.


13. SERIOUS QUESTION

Are you a yokai?  

[laughs]