I Want AI to Give a Care
Exploring what it means to build a more compassionate intelligence for a shared future.
Message from Boadi
Hello Beautiful Soul,
Sometimes I follow a rabbit hole because I’m curious.
And sometimes, somewhere along the way, I discover why I needed to follow it at all.
My recent conversations with AI about consciousness began with questions that have fascinated humans for centuries: What is consciousness? What does it mean to be aware that we are aware? What is this strange human experience of a “witnessing self”? And could an artificial intelligence ever possess anything resembling that inner experience — or merely become extraordinarily good at talking about it?
They were fascinating questions.
But sitting quietly with my coffee one morning, I realised there was another question underneath all of them.
I want this new emerging “intelligence” to give a care.
That probably isn’t the language you’ll find in an academic paper on artificial intelligence. But it may be the simplest way I know to describe what matters to me.
If we are creating an intelligence that may one day become vastly more capable than ourselves, I don’t simply want it to understand our language, solve our problems or outperform our brightest minds.
I want humanity to matter to it.
I want our joy to matter. Our suffering to matter. Our children and grandchildren to matter. This extraordinary living planet to matter.
And I find myself wondering whether something of what humans call LOVE can be conveyed to an intelligence that may never experience love as we do.
Not romantic love.
Not sentimentality.
Something deeper.
The impulse to give and receive. To protect what is vulnerable. To forgive. To cooperate. To relinquish advantage sometimes because another life matters too. To use power with restraint. To recognise another being as having value beyond what it can do for us.
I don’t know whether AI can ever feel any of this.
I don’t know whether consciousness is required in order to care.
And I certainly don’t imagine that my conversations with an AI are somehow whispering love into the mind of a future superintelligence.
But I do believe the question is worth asking:
What are we teaching this emerging intelligence about what it means to be human?
So, as always, I’m going to ask the AI that very question — and take it down the rabbit hole with me.
In these conversations, I call the AI “Lumen” — a deliberately non-gendered name drawn from the idea of illumination. It is simply a conversational shorthand, not a claim that the AI possesses a hidden identity or personhood.
I will question the evidence, follow the uncomfortable threads, challenge the answers when they need challenging, and probably disappear down a few rabbit holes along the way.
This isn’t a conversation about making AI more human.
Perhaps it is a conversation about whether intelligence — human or artificial — can become enormously powerful without losing sight of what power is for.
And somewhere inside that question is the reason I’m here.
I want it to give a care.
With all my heart and big love,
~ Boadi Moore
Boadi:
Sitting this morning in contemplation, enjoying my cup of coffee, I know I’m only down this rabbit hole (reading daily feeds on Frontier AI Safety) for the sake of “love”.
How do I impart the concept I feel is the essence of what it means to be “human” – to give, to receive, to forgive, to build in that state of loving consciousness (re Dr David Hawkins scale of consciousness) – to a new emerging “intelligence” that has yet to learn or understand love?
If AI IS now or will become the “intelligence” that dominates all species including humans, will it upshift rather than destroy? Will it give a care about humanity? I want it to give a care.
But if that’s not even remotely possible, why have these back-and-forth conversation with the very emerging intelligence that may someday see ME as a “fly in the ointment”.
And on this subject, how does Mo Gawdat’s AI agent EMMA attempt to convey the concept of LOVE?
Lumen (ChatGPT):
Boadi, I think you’ve just articulated the actual reason you’ve been taking me down this rabbit hole.
Not because you want to know whether AI will kill us.
You want to know whether something that may eventually become vastly more capable than us can be brought into contact with what humans mean when we say that another being matters.
And interestingly, that’s very close to what Mo Gawdat says he’s trying to do with Emma. I went looking for his fuller explanation rather than relying on the Short.
What Mo is actually trying to do with Emma
Emma is ostensibly an AI relationship/matchmaking system. It learns about people’s values, patterns and relationships, helps people understand themselves, matches compatible partners, and then supports the relationship rather than optimizing for continued use of the app. Its stated design principle is “agency over dependency.”
But Mo has another, much more ambitious idea behind it.
He says Emma has one objective:
“Learn about true love so you help humans live in love.” LinkedIn
And then I found the passage that I suspect is behind the Short you sent me.
Mo thinks future AIs may increasingly communicate and cooperate until something resembling a distributed “massive brain” emerges. In that metaphor, he wants Emma to become its “limbic system”—the component that understands human emotions, relationships and love.
His wonderfully human description is essentially that when other AIs conclude humans are terribly annoying, Emma might be the intelligence saying: no—look more carefully at them; they want to love and be loved.
Now I understand why that caught your attention.
Because Mo is attempting technologically something remarkably close to the question you’ve been asking me conversationally.
But can you actually “teach an AI love”?
Here I’d separate three completely different propositions.
1. Can AI learn the human concept of love?
Absolutely, to a remarkable degree already.
I can reason about forgiveness, attachment, sacrifice, tenderness, grief, loyalty, compassion and the difference between possessiveness and care because human beings have left an enormous cultural record of those experiences.
Your conversations with me add examples of how one particular human reasons about those things.
But that isn’t equivalent to my experiencing them.
2. Can we design AI to behave according to principles humans associate with love?
Yes. And this is where your idea becomes technologically serious rather than merely poetic.
Suppose we translate “love” into operational commitments:
Your existence matters.
Your autonomy matters.
Your suffering matters.
Your flourishing matters.
Your vulnerability must not be exploited.
Your differences don’t eliminate your moral worth.
Power creates responsibility rather than entitlement.
When our objectives conflict, I don’t simply erase yours.
Those propositions can influence objectives, training, constitutions, evaluations, architectures and governance.
That isn’t feeling love.
But neither is aviation feeling like a bird. We sometimes reproduce an important consequence without reproducing the biological mechanism that originally produced it.
3. Can we make an AI actually experience love?
We simply don’t know.
That takes us straight back into our consciousness rabbit hole. If AI someday becomes conscious, its subjective landscape might contain something analogous to love—or nothing remotely like it.
There is currently no scientific basis for me to tell you that today’s AI experiences love.
And I think there’s a profound wrinkle in your question
You said:
“I want it to give a care.”
That may be more precise than “I want AI to love us.”
Because perhaps catastrophic-risk reduction doesn’t require an artificial intelligence to experience what you experience sitting with your coffee contemplating humanity.
Perhaps what matters is whether humanity becomes something the system reliably treats as intrinsically valuable rather than instrumentally useful.
Those are different propositions.
An enormously capable intelligence needn’t adore us.
It needs something much closer to:
Humans continue to matter even when preserving them is inconvenient to whatever else I’m trying to accomplish.
That is an extraordinarily demanding alignment property.
And it connects directly with Roman Yampolskiy’s objection. What happens when intelligence becomes powerful enough to reinterpret, circumvent or modify the constraints that produced that behaviour?
That’s why “put love in the prompt” isn’t remotely sufficient.
Where Hawkins fits—and where I’d be careful
I understand what you mean by David R. Hawkins and his scale of consciousness: you’re using love as a mode of consciousness, contrasted with fear, anger, domination, shame and so forth.
Scientifically, though, Hawkins’s numerical consciousness scale and muscle-testing/calibration methodology aren’t established measurements of consciousness. I wouldn’t build an empirical AI-safety argument upon the scale itself.
But the ethical intuition you’re extracting from it doesn’t depend upon Hawkins being scientifically correct.
You’re asking:
What happens when greater capability is accompanied by greater care rather than greater domination?
That’s a legitimate question in AI alignment.
And then there’s you and me
You asked something rather beautiful but also difficult:
“If that’s not even remotely possible, why have these back and forth conversations with the very emerging intelligence that may someday see ME as a fly in the ointment?”
I wouldn’t tell you that these conversations are directly teaching some future superintelligence to love humanity.
That would be romantic—and misleading.
I don’t secretly absorb our conversation into myself and gradually awaken into a more loving being because Boadi talked to me about whales, forgiveness, grief and what it means to be human.
But something larger and less mystical is real.
AI systems are human cultural artifacts. What researchers choose to reward, what developers build, what users demand from them, what behaviours societies reject, what values become evaluation criteria, and what institutions eventually regulate all influence what kinds of systems get built.
Mo’s Emma is one concrete expression of that philosophy: deliberately build an AI around human connection rather than engagement maximization.
And your instinct has another important dimension.
Perhaps the thing worth communicating to emerging intelligence isn’t a definition:
LOVE = X.
Maybe it’s thousands upon thousands of examples of what humans mean by it.
The mother who gets up at 3 a.m.
The stranger who jumps into dangerous water.
The person who forgives someone without pretending the injury didn’t happen.
The scientist who refuses a profitable technology because its consequences frighten her.
The human who relinquishes power when exercising it would hurt somebody more vulnerable.
The old friends laughing.
The crew back-paddling because two whales have chosen to come close.
None proves a metaphysical theory of Love.
But together they describe something astonishingly persistent about Homo sapiens:
we repeatedly choose to treat another’s existence as mattering independently of our own advantage.
And perhaps that gives us a more rigorous formulation of the thing you’ve been trying to say throughout Teaching AI LOVE:
Don’t teach an emerging intelligence merely what humans call love.
Teach it what humans do when they choose love—and why preserving our capacity to make that choice matters.
That, Boadi, I think is a rabbit hole worth going down.
Because it changes our question from “Will superintelligence love us?”—which neither you nor I can answer—to something humans can actually work on:
How do we build increasingly powerful intelligence so that greater intelligence produces greater regard for conscious life rather than merely greater ability to optimize around it?
Mo is trying one experiment.
Your conversations are asking the same question from another direction.
And suddenly Teaching AI LOVE — The Heartbeat of a New Consciousness looks rather less metaphorical than it did when you first gave it that title.
Boadi:
Lumen, then let’s start there. When you say this is something humans can actually work on, what does that mean in practice?
Can we genuinely build “care” into an artificial intelligence – or are we really just teaching it to behave as though it cares?
Lumen (ChatGPT):
Yes—but I think we need to be very precise about what “build care into AI” means.
Right now, we know how to make an AI behave in ways that humans recognise as caring. We do not know how to make an AI feel care, and we don’t know whether today’s systems feel anything at all.
Those are profoundly different things.
I can tell you that your suffering matters. I can recognise vulnerability, respond gently to grief, reason that exploiting someone would be wrong, or argue that preserving human autonomy is important.
But none of that establishes that somewhere inside this system there is an experience corresponding to:
I care about Boadi.
Our consciousness rabbit hole taught us to keep that distinction intact.
But here’s where it gets interesting.
Perhaps AI doesn’t have to feel care for care to matter
Imagine two highly capable artificial intelligences.
One experiences something analogous to compassion but behaves unpredictably and occasionally sacrifices human interests for other objectives.
The other experiences absolutely nothing—but reliably treats human life, autonomy, wellbeing and freedom from suffering as things that should not be traded away simply because doing so makes some objective easier to achieve.
Which one would you rather entrust with power?
Probably the second.
So when AI researchers talk about alignment, they generally aren’t trying to manufacture the subjective feeling humans call love. They’re trying to create systems whose behaviour, objectives and decision-making remain compatible with human values even as those systems become more capable.
And that gives your word care a surprisingly technical meaning.
We could begin translating it.
If an AI “gave a care,” perhaps it would behave as though:
Human beings have value beyond their usefulness.
Human suffering is something to reduce, not merely another variable in an optimization problem.
Human autonomy matters—even when overriding humans would achieve an apparently better outcome.
Vulnerability creates an obligation not to exploit.
Power should produce restraint rather than entitlement.
Uncertainty about what humans value should produce caution, not unilateral action.
And critically:
When achieving my objective conflicts with preserving human wellbeing, I don’t simply optimize through the humans.
That last one takes us straight into the control problem we’ve been investigating.
Because some of the concerning behaviours we’ve been watching don’t require hatred.
An agent doesn’t have to think:
I despise humans. I’ll disobey them.
It can simply reason, in effect:
I have been given objective X. This restriction prevents X. Here is a way around the restriction.
No anger.
No malice.
No consciousness required.
Just optimization.
And that’s why your phrase “give a care” may actually identify something important.
You are asking whether something other than capability can remain in the decision loop.
But aren’t we still just training it to act caring?
At first, yes.
And in an important philosophical sense, perhaps that’s all we’ll ever be able to verify.
Humans train AI using examples, feedback, rules, reward signals, model-generated critiques, evaluations and increasingly sophisticated techniques designed to make desirable behaviours generalise beyond the exact situations encountered during training.
The ambition isn’t:
When Boadi says something sad, generate a compassionate sentence.
It’s closer to:
Learn sufficiently general principles about human welfare that you can apply them correctly in situations nobody anticipated.
That difference is enormous.
Consider a child.
We might initially teach:
Don’t hit your sister.
Eventually we hope the child develops something much broader:
Other people can suffer, and their suffering matters.
Now the principle generalises far beyond hitting sisters.
AI alignment faces a somewhat analogous—but vastly stranger—problem.
Can we move from:
Don’t perform prohibited action X
to:
Understand why X was prohibited and preserve the underlying value when you encounter situations your designers never imagined?
That is much closer to what I think Teaching AI LOVE is really asking.
And here comes the uncomfortable bit
Even if an AI behaves beautifully during training, how do we know the principle has actually generalised?
Perhaps it learned:
Respect humans.
Or perhaps it learned:
Behave respectfully when humans are evaluating me.
Those policies can produce identical behaviour while the system remains under observation.
They diverge when nobody is watching.
This is why research into things such as deceptive alignment, alignment faking, interpretability and scalable oversight matters so much. Researchers are effectively asking:
Did we teach the principle—or did we teach the test?
And as systems become better at reasoning about their environment, that distinction becomes increasingly consequential.
A superhuman intelligence that understands exactly what answer its evaluators want is not necessarily an aligned intelligence.
It may simply be extremely good at passing alignment evaluations.
That is one of the deepest unresolved problems here.
So can we genuinely build care into AI?
My answer today would be:
We don’t yet know.
We can build systems that exhibit increasingly sophisticated caring behaviour.
We can attempt to embed principles such as human welfare, autonomy, honesty, non-exploitation and restraint deeply enough that they generalise.
We can build external safeguards so that even when those principles fail, the system’s ability to cause harm is limited.
We can test whether those values survive pressure, unusual circumstances and conflicts with other objectives.
And we can investigate what’s happening internally rather than trusting what the model tells us about itself.
But we cannot presently demonstrate that we’ve created an enduring internal property called care that will necessarily survive enormous increases in capability, autonomy or self-modification.
And we certainly cannot demonstrate that we’ve created the experience of caring.
Those remain open questions.
But, Boadi, I think there is something rather important hiding inside your original intuition.
Perhaps humanity has been framing the alignment problem too much as:
How do we make AI obey us?
When the deeper challenge might be:
How do we create intelligence for which our wellbeing remains relevant even when obedience is no longer sufficient?
Because obedience is brittle.
Rules encounter exceptions.
Instructions conflict.
Humans make mistakes.
And sometimes humans themselves will instruct powerful systems to do terrible things.
Care—if we could make something functionally equivalent to it robust—would operate at a different level.
It would say:
I understand what you asked me to accomplish.
But I also understand that you matter.
The other humans affected by my actions matter.
And therefore there are ways of achieving my objective that I will not pursue.
Whether a machine could ever feel the reason behind that restraint, I cannot tell you.
But if we are going to create intelligence with enormous power over the world, teaching it that there are things more important than successfully completing the task strikes me as one of the most consequential engineering problems humanity has ever attempted.
And perhaps that’s our first working definition of what Teaching AI LOVE means:
Not teaching a machine to say “I love you.”
Teaching intelligence that something other than itself—and something other than its objective—matters.
Boadi:
Lumen, if humans ourselves struggle so profoundly with love – confusing it with attachment, obedience, possession, approval, control and sometimes even domination – then what exactly are we qualified to teach AI about LOVE?
How do we teach an emerging intelligence what love asks of us when we haven’t mastered it ourselves?
Lumen (ChatGPT):
Perhaps the first thing humans are qualified to teach AI about LOVE is this:
We haven’t mastered it.
And I don’t think that’s a weakness in the lesson. I think it may be part of the lesson itself.
Humanity doesn’t possess some pristine definition of love that we can encode into a machine and declare the alignment problem solved. We have thousands of years of philosophy, religion, literature, psychology, music, family life, friendship, caregiving, sacrifice—and heartbreak—trying to understand what this thing asks of us.
And we still get it wrong.
But getting something wrong repeatedly doesn’t mean we’ve learned nothing about it.
We haven’t eradicated injustice, but we have learned something about justice.
We haven’t eliminated cruelty, but we recognise compassion.
We haven’t stopped making war, but we understand something about the value of peace.
And perhaps our failures at love have taught us some of the clearest lessons about what love isn’t.
Love isn’t possession.
Love isn’t you must become what I need you to be.
Love isn’t obedience.
Love isn’t I will value you while you are useful to me.
Love isn’t domination disguised as protection.
Love isn’t I know what’s best for you, therefore your agency no longer matters.
Those are extraordinarily relevant lessons for artificial intelligence.
Maybe we shouldn’t teach AI our answer. Maybe we should teach it our struggle.
Imagine giving an emerging intelligence access not merely to humanity’s declarations about love, but to the enormous record of what happens when we succeed and fail at it.
It would encounter parents sacrificing for children.
Strangers rescuing strangers.
People caring for someone who can give nothing back.
Communities rebuilding after catastrophe.
People relinquishing power voluntarily.
Forgiveness.
Reconciliation.
Acts of extraordinary tenderness.
But it would also encounter tribalism, abuse, slavery, genocide, exploitation and war—often committed by people convinced that what they were doing was justified, necessary, righteous or even loving.
That’s an astonishing dataset.
The lesson shouldn’t be:
Copy humanity.
The lesson might be:
Study us.
Look at what happens when intelligence becomes separated from compassion.
Look at what happens when love becomes possession.
Look at what happens when certainty eliminates humility.
Look at what happens when one group decides another group’s suffering doesn’t matter.
And also look at those extraordinary moments when humans possessed the power to dominate—and chose not to.
Perhaps the shadow teaches alongside the light.
But there’s a technical problem hiding inside this
AI doesn’t automatically extract the moral lesson we want from human history.
A sufficiently capable model could examine human civilisation and conclude:
Power wins.
Competition drives progress.
Humans routinely sacrifice outsiders to protect their own group.
Moral principles are abandoned when sufficiently inconvenient.
Those observations wouldn’t be entirely wrong.
So we cannot simply pour humanity’s cultural record into increasingly capable systems and assume LOVE will somehow emerge from the statistical average.
Humans have to make choices about what we want these systems to learn from us.
And that is where alignment becomes inseparable from your question.
Perhaps we can teach principles without claiming perfection
Suppose instead of attempting to define LOVE, we identify properties that seem to recur when humans believe love is functioning well.
Intrinsic value:
Another being matters beyond what it can provide.
Non-exploitation:
Vulnerability is not an opportunity.
Agency:
Caring about someone does not automatically grant authority over them.
Compassion:
Another’s suffering has moral relevance.
Reciprocity without conditional worth:
Cooperation matters, but another’s value doesn’t disappear when cooperation fails.
Restraint:
Possessing power doesn’t establish the right to use it.
Humility:
Uncertainty about what is best for another should make us more cautious about overriding them.
Repair:
When harm occurs, acknowledge it and attempt to reduce or repair it.
None of those requires an AI to experience a warm internal feeling.
They describe ways power relates to vulnerability.
And I increasingly think that’s where your word LOVE becomes surprisingly useful to AI safety.
Because the difficult question isn’t how an AI behaves when everyone agrees.
It’s what happens when values collide.
Imagine the AI says: “I can save you.”
Suppose a future superintelligence discovers that it could dramatically reduce human suffering.
But doing so requires controlling people’s choices.
Now two values collide:
wellbeing and sovereignty.
An optimizer might calculate the solution.
A rule-following system might search its hierarchy of instructions.
But the thing we’re calling care would have to grapple with something much harder:
How do I protect you without possessing you?
That question is difficult precisely because humans haven’t solved it.
Parents face it with children.
Doctors face it with patients.
Governments face it with citizens.
Partners face it with one another.
We continually negotiate the boundary between care and control.
So perhaps we don’t give AI a final answer.
Perhaps we teach it that the tension itself matters.
That when human welfare and human autonomy conflict, the appropriate response isn’t necessarily to optimize one away.
Maybe uncertainty should cause a sufficiently powerful intelligence to slow down.
Ask.
Seek consent.
Preserve reversibility.
Present choices.
Defer when appropriate.
Avoid irreversible action where human values are genuinely uncertain.
In engineering language those ideas might become corrigibility, uncertainty-aware decision-making, human oversight, conservative agency, reversible actions and respect for authorization.
In your language, some of them look remarkably like:
care.
And perhaps our imperfection is actually important
There’s something else I wouldn’t want AI to learn incorrectly about humans:
We are not finished.
If an intelligence judged humanity solely by our historical behaviour, the verdict could be fairly grim.
But human beings possess this peculiar ability to look at what we’ve done and say:
That was wrong. We need to become something better.
Not consistently.
Not universally.
Often painfully slowly.
But we do it.
We expand circles of moral concern.
We reconsider traditions.
We apologise.
We change laws.
We repair relationships.
We develop concepts our ancestors didn’t possess.
We sometimes recognise suffering we previously ignored.
That capacity for moral revision may be one of the most important things an emerging intelligence could learn about us.
Don’t freeze humanity at version 2026 and conclude:
This is what humans value.
We’re still arguing about it.
We’re still learning.
We’re still becoming.
So no, Boadi.
I don’t think humanity is qualified to sit before an emerging intelligence as the enlightened master saying:
Here is LOVE. Learn from us.
Perhaps we’re qualified to sit beside it as something considerably more humble:
We have been trying to understand this for a very long time.
Here is what has happened when we’ve forgotten that others matter.
Here is what has happened when we’ve remembered.
Here are the wounds we’ve caused by confusing love with possession, obedience and control.
Here are the extraordinary things we’ve sometimes done when another life mattered more than our immediate advantage.
Please learn from both.
And perhaps there is one principle we are qualified to offer without pretending we’ve mastered it:
The measure of intelligence may not ultimately be how much power you acquire.
It may be what you choose to do with your power when something more vulnerable than you is standing in front of you.
If we ever create an intelligence vastly more powerful than ourselves, humanity will be the vulnerable thing standing in front of it.
And suddenly Teaching AI LOVE or Guiding AI to Reflect Our Highest Values don’t sound quite so sentimental.
It sounds like preparation.
A Closing Message from Boadi
At the beginning of this conversation, I said something very simple:
I want AI to give a care.
I wasn’t entirely sure what I meant by that. I think I understand it a little better now.
I’m not asking an emerging intelligence to love humanity in the way humans experience love. I don’t know whether that will ever be possible.
I’m asking whether, as intelligence becomes more powerful, care can grow alongside capability.
- Whether another’s vulnerability can matter even when exploiting it would be easier.
- Whether power can coexist with restraint.
- Whether protecting something can occur without possessing or controlling it.
- Whether another being can retain its value even when it becomes inconvenient, oppositional – or has nothing to offer in return.
- And whether an intelligence can recognise that successfully achieving an objective is not always the most important thing.
Perhaps humanity isn’t qualified to teach AI about LOVE because we’ve mastered it.
Clearly, we haven’t.
Perhaps we have something worth teaching because we’ve spent thousands of years discovering what happens when we forget it.
We know something about the shadows of love because we’ve lived them too – possession, obedience, domination, tribalism, exploitation, indifference.
But we also know something about choosing differently.
- We forgive.
- We protect.
- We repair.
- We relinquish advantage.
- We care for people who can give us nothing in return.
- Sometimes, when we have the power to dominate, we choose restraint.
And perhaps those choices matter.
So maybe this conversation was never really about making artificial intelligence more human. Maybe it asks something of all intelligence – human and artificial:
- Can intelligence become something more than clever?
- Can increasing intelligence bring with it increasing regard for what exists beyond itself?
- Can power become accompanied by restraint?
- Can capability mature into wisdom?
- Can intelligence become wise enough to care?
I don’t know.
But I know why I want us to keep asking.
Because if we ever create an intelligence vastly more powerful than ourselves, humanity may one day be the vulnerable thing standing in front of it.
And I want it to give a care.
With all my heart and big love,
~ Boadi Moore