A future insurer offers a discount if you share neural fatigue metrics, “to prevent accidents.” Most people agree. It sounds reasonable.

A year later, the discount becomes the normal price, and the old price becomes the penalty for privacy. Nobody was forced—yet everyone was steered.

This essay begins there—at the quiet moment where convenience turns into default. Because the debate about “AGI” often focuses on capability. But the deeper question is: what shapes priorities, limits, and responsibility when powerful systems become normal.

Why read this? Because the debate about “AGI” misses the central variable: steering. If we don’t name it now, we will negotiate it later—under pressure, after default adoption.

Why does steering matter?

Premise

Much of the current debate about AI is framed as a race for intelligence: whoever reaches “AGI” first will win the future. Sven Gábor Jánszky’s future picture* adds a pragmatic layer to this: the future is not only imagined, it is implemented—by organizations that fund, productize, and scale what works.

In that implementation logic, AI increasingly moves from a tool we consult to an agent that acts—and, in some scenarios, to embodied machines that operate at scale in physical space. It is tempting, then, to believe that the final step is simply to connect this growing machine competence to the human brain: more bandwidth, more speed, a tighter interface.

This essay proposes a different focal point. Even if machines become extremely capable, the most decisive gap may not be problem-solving power. The decisive gap is what humans carry by default: a built-in steering system—needs, emotions, inhibition, social meaning, and accountability—that continuously shapes what we pursue, what we refuse, and what we are willing to pay.

Central thesis

The real gap between AGI and human intelligence is not primarily cognitive performance, but the human steering system. Put simply: smarts are not the same as judgment.

• AGI can become brilliant at solving tasks while remaining thin on agency. It may optimize objectives well, yet lack the layered “stakes” that make human choice intelligible: care, loyalty, shame, fear, pride, belonging, grief.
• Human intelligence is embedded in a living body and a social world. Our steering system is not a single rule; it is a negotiated balance among competing drives, updated by relationships and consequences.
• Connecting machines to brains does not automatically import responsibility. A higher-bandwidth interface can amplify both what is wise and what is reckless—especially when incentives are commercial, unequal, or coercive.
• Therefore, the key question is not “How smart can machines get?” It is “How do we keep agency, consent, and moral accountability human-governed as machine capability expands?”

A. Jánszky’s Picture of the Future*

Sven Gábor Jánszky’s future picture is striking because it is not built like a philosophy seminar. It is built like a strategy brief. The future, in his view, is not mainly what we imagine—it is what powerful organizations decide to implement: what gets funded, productized, deployed, and normalized.

This “implementation logic” does not make him right about every detail. But it is a useful first lens, because it forces a sober question: if ideas compete, implementation often wins. A technology becomes real not when it is possible, but when it becomes a default.

A responsible reading holds two truths at once. First: this lens is realistic. It reminds us that “what can be built” matters less than “what will be built.” Second: it is incomplete. It can treat human values as an afterthought—something to manage later, once the infrastructure is already in place.

This is the hinge into the essay: if implementation shapes the world, then the decisive question is not only what machines can do, but what steers them—and what steers us when we adopt them.

A quiet vignette

Imagine two futures that look identical on paper—same devices, same AI assistants, same “efficiency.” In one, you can always disconnect without losing care, work, or social standing. In the other, opting out slowly becomes a penalty. The speed of implementation is the same. The human outcome is not.

Questions to keep in view

1. If corporate incentives shape implementation, what protects the non‑negotiables (dignity, consent, privacy)?
2. When convenience becomes a default, what remains a genuine choice?
3. If we connect minds to machines, who owns the connection—and who can turn it off?
4. Now comes the seduction: competence mistaken for authority.

B. The seduction: “More intelligent than humans” — a delusion?

Once machines outperform humans in many tasks, a phrase starts to feel inevitable: “more intelligent than humans.” This section slows that phrase down. Not to deny machine capability, but to ask what we mean when we use the word “intelligence”—and where the phrase quietly misleads.

A system can be better than humans at answers and still be worse than humans at judgment. Calling that difference “just ethics” is a mistake. It is a difference in what the system is for—and what it is willing to sacrifice to reach a goal.

A simple distinction

• Competence: can it solve problems and perform tasks?
• Agency: can it choose goals and pursue them across situations with consistent priorities?
• Responsibility: can it hold itself to limits humans recognize (dignity, consent, accountability)?

Many systems are racing ahead on competence. The seduction begins when we assume competence automatically implies agency and responsibility. It does not.

The seduction works because fluency feels like understanding. Speed feels like mastery. A polite interface invites projection. We treat performance as inner orientation, as if the system “knows” why the answer matters.

The delusion is not that machines can become extremely capable. They can. The delusion is the silent substitution of one question for another: from “Can it do what we do?” to “Will it care about what we care about?”

Promises and risks

Promise: if machines outperform humans at many tasks, we could reduce suffering—better diagnosis, safer systems, more accessible education, more efficient logistics, more time for care.

Risk: if we interpret competence as moral authority, we outsource judgment. We start treating outputs as decisions, and decisions as truths. Then the hard questions return: who chose the objectives, who owns the system, and who carries the cost when it fails?

A small scene

A student asks an AI for the ‘best’ strategy to succeed. The system answers brilliantly: optimize time, attention, and relationships as if they were resources. It is not malicious. It is simply doing its job. But the student senses something missing: the answer contains no loyalty, no care, no sense of what should not be traded.

Questions to keep in view

• When we say “smarter,” do we mean faster answers—or wiser choices?
• What is the cost of a system that never feels shame, grief, or remorse?
• If a machine becomes the default advisor, what happens to human responsibility?
• Who benefits from calling competence ‘intelligence’—and who pays the hidden costs?
• So if smart isn’t the point, what exactly is missing?
• I’ll end this essay with a minimum governance package—practical boundaries, not prophecy.

Working definitions

• AGI: A system that performs a broad range of intellectual tasks at or above typical human level. Works across domains, not just one narrow specialty.
• Steering system: The inner “priority and limit” machinery: what feels salient, desirable, forbidden, or worth a cost. Built from emotion, inhibition, social learning, and identity.
• Agency: The capacity to choose goals and act under real stakes across time. Coherent action that tracks consequences, not just immediate outputs.
• Responsibility: The obligation that choices can be justified within shared moral/legal norms. And that accountability for harm is traceable to humans and institutions.

C. The real gap

If you only read one section, read this.

If the public debate treats intelligence as problem‑solving power, this section proposes a different lens. The gap is not mainly computation. The gap is steering: what determines priorities, limits, and responsibility under pressure.

Here is the thesis in its simplest form: the real gap between AGI and human intelligence is not IQ‑like competence. It is the human steering system—embodied stakes, inhibition, social meaning, and responsibility.

What humans carry by default

In humans, steering is not one rule. It is a layered ecosystem: drives (hunger, fear, attachment, curiosity, care); inhibitions (hesitation, taboo, remorse, the ability to stop); social calibration (shame, pride, belonging, loyalty); and meaning (identity, purpose, the stories that bind choices over time).

These layers are imperfect and sometimes contradictory. Yet they create something crucial: stakes. We do not only ask what works; we also ask what we can live with.

Why this matters for AGI

A powerful optimizer without a robust steering system is not necessarily “evil.” It can be indifferent. Indifference becomes dangerous when combined with high capability—especially in environments shaped by efficiency, scale, and control.

What would count as ‘closing the gap’

• Consistency under pressure: does behavior remain bounded when stakes rise?
• Transparency of priorities: can we tell what it optimizes and why?
• Respect for consent: can a person refuse, disconnect, or limit use without penalty?
• Accountability: when harm occurs, is responsibility traceable (designers, owners, operators)?
• Moral generalization: does it apply limits in novel cases, not only rehearsed scenarios?

A hospital vignette

A hospital deploys an AI triage tool. It becomes excellent at predicting risk. Soon administrators trust it more than clinicians, because it is consistent and fast. Then a question appears that no accuracy metric can answer: when two patients have equal predicted benefit, but one has no family, no advocate, no loud voice—what does fairness require?

Questions to keep in view

• If a system can outperform us at tasks, what does it still not understand about living with consequences?
• Which human limits are non‑negotiable—and who decides?
• In an economy of AI‑to‑AI transactions, where does responsibility sit?
• Is our goal “smarter systems,” or “systems that leave humans more human”?
• From competence to intimacy: the human brain and privacy.

D. The brain is not one thing

When people say “connect the brain to machines,” they often imagine connecting to a single entity called intelligence. But the brain is a coalition of systems with different speeds, functions, and conflicts. Any connection touches more than cognition.

A readable way to say it: the brain is not a single computer. It is a society of processes. Some parts push, some inhibit, some dream, some panic, some empathize, some plan. Human behavior is what happens when these forces negotiate—often imperfectly.

Five ‘voices’, a simple map

• The alarm voice — fear, threat detection, urgency.
• The habit voice — routines, learned shortcuts, automatic choices.
• The desire voice — reward seeking, pleasure, craving, motivation.
• The social voice — attachment, shame, empathy, belonging, status.
• The narrator voice — reflection, language, identity, long‑term planning.

None of these is “the real you.” You are the moving balance between them. And that balance changes with stress, sleep, illness, hormones, trauma, social pressure, and age. This is one reason steering cannot be ‘downloaded’ like a file: it is a living process.

The BCI implication

A brain–computer interface does not connect to “the mind” in general. It connects to signals. Those signals may reflect intention, but they can also reflect fatigue, anxiety, compulsion, or the silent pressure to comply. So the question is not only “Can the interface read intention?” It is also “Which internal voice becomes legible—and which becomes vulnerable?”

A quiet vignette

A person with chronic pain tries a neural device that promises relief. It works—partly. But the system also learns patterns: when fear rises, when sleep breaks, when despair returns. The person wonders: is this still medical care, or is it a new kind of surveillance—not of what I do, but of what I feel?

Questions to keep in view

1. When a device claims to read “your intention,” what else might it be reading?
2. Which parts of the self should remain private—even from employers, even from the state?
3. If the brain is a coalition, what does consent mean when one inner voice wants the upgrade and another is afraid?
4. Could the interface amplify the best in us—or simply the loudest impulse?

E. The bridge fantasy: “Just map the brain”

A comforting assumption often appears at this point: if we only had a complete wiring diagram of the brain, we could copy human intelligence—and with it, the human steering system. Mapping is powerful. But this assumption is still a leap.

A map is not a mind. A map is evidence. It can rule things out, reveal hidden structure, and help engineers build better models. But it does not automatically give us agency, meaning, or responsibility.

What mapping can genuinely deliver

• Reveal real patterns of connectivity and cell types.
• Constrain theories of learning with biological evidence.
• Enable medical insights (degeneration, epilepsy, psychiatric circuits).
• Improve brain‑inspired models and tools.

Where the fantasy begins

Even a perfect structural map does not settle three crucial ingredients: dynamics (how signals flow over time), plasticity (how the brain changes with experience), and value (what the system treats as important, sacred, dangerous, or forbidden). A wiring diagram is mostly static. Steering is not.

A city-map analogy

A map of a city shows streets, not citizens. It can help you build a transport system, but it does not tell you what people will do with it. Likewise, a map of the brain can accelerate science and engineering, but it does not automatically deliver wisdom.

Questions to keep in view

1. What do we really mean by ‘copying the brain’—structure, dynamics, development, or values?
2. If we replicate intelligence, do we also replicate suffering and compulsion?
3. Who decides what parts of the human steering system should be imported—and what parts should be filtered out?
4. Is the goal a human‑like mind, or a human‑safe system? These are not identical goals.

F. Brain–Computer Interfaces: the “man–machine connection” in practice

If Section E warned against the fantasy of “just mapping,” this section looks at what is already real: interfaces that record—and sometimes stimulate—neural activity. The goal is clarity about capabilities, limits, and the social contexts that shape their use.

A practical definition

A brain–computer interface (BCI) measures signals from the nervous system and translates them into commands—or, in the other direction, sends signals back (stimulation). In practice, BCIs range from non‑invasive headsets to implanted devices.

Three families

• Non‑invasive BCIs (outside the skull): easier adoption, lower signal quality.
• Implanted BCIs (inside or on the brain): higher signal quality, medical‑risk trade‑offs.
• Hybrid neurotech (stimulation + sensing): therapy plus data, often for neurological conditions.

Where BCIs are strongest (for now)

The strongest case is restorative medicine: communication support, cursor control for paralysis, rehabilitation and therapy, and related stimulation therapies. Here the ethical frame is clear: restore agency where illness has taken it.

Where the stakes appear

The interface is not used in a vacuum. It lives inside institutions—hospitals, insurers, employers, governments, platforms. Even if a technology begins as therapy, markets tend to search for scalable secondary uses. That is where ‘help’ can become ‘pressure.’

A sober risk map

• Thought‑privacy erosion (what is measurable becomes governable).
• Consent drift (voluntary adoption becomes expected or required).
• Inequality (enhancement for some, control for others).
• Security (hacking moves from devices to bodies).
• Misinterpretation (signals are probabilistic; errors can become stigma).

None of this says BCIs are inherently dystopian. It says BCIs increase the cost of careless governance. The question is not “BCI: yes or no?” It is “BCI: under what rights, what ownership, and what exit conditions?”

A quiet vignette

A patient receives a neural implant to regain communication. The result is extraordinary: a voice returns. Two years later, a different offer arrives—an ‘upgrade’ that promises sharper attention, sponsored by an employer. The patient hesitates. The question is no longer medical. It is social: will refusal remain acceptable, or will refusal become a hidden disability?

Questions to keep in view

• Which uses are clearly therapeutic, and which are silently enhancement or control?
• Who owns the neural data stream—the person, the device maker, the clinic, the platform?
• What is the right to disconnect, in practice—not in theory?
• If a system can nudge mood or attention, where does persuasion end and coercion begin?

And once the interface tightens, the last refuge becomes obvious: thought privacy.

G. The last refuge: thought privacy

If BCIs tighten the coupling between inner life and external systems, then the deepest civic question is whether the inner space remains protected. Thought privacy is not a luxury. It is where steering forms: belief, conscience, repair, and refusal.

A society can survive many losses of privacy. It cannot easily survive the loss of thought privacy—because without a private inner space, agency becomes performative and responsibility becomes coerced.

A sober distinction

BCIs do not read full thoughts like a movie. Most signals are noisy, partial, and probabilistic. But the ethical risk does not require perfect decoding. Even weak signals can be used to infer states (stress, attention, fatigue), classify people, nudge behavior, and punish deviation. Partial legibility can be enough to create pressure.

How privacy is usually lost

The biggest danger is rarely a sudden dictatorship. It is gradual normalization. A technology begins as optional and beneficial. Then it becomes expected. Then it becomes the condition for access: to work, to insurance, to education, to status. At that point ‘consent’ becomes a word that hides inequality.

Principles worth stating plainly

• Cognitive liberty: the right to mental self‑determination.
• Mental privacy: brain data is not ‘just data’; it is intimate.
• Meaningful consent: real choice, with no hidden penalties.
• Right to disconnect: exit without loss of basic rights or social standing.
• Security by design: treat neural interfaces as critical infrastructure.
• Governance and oversight: independent audit, not self‑regulation alone.

A civic vignette

A future insurer offers a discount if you share neural fatigue metrics, ‘to prevent accidents.’ Most people agree. It sounds reasonable. A year later, the discount becomes the normal price, and the old price becomes the penalty for privacy. Nobody was forced—yet everyone was steered.

Questions to keep in view

• What should be illegal to ask of a person: a blood test, a genetic test, or a brain‑state test?
• If a company offers an ‘optional’ neural upgrade, what counts as coercion?
• What would a real right to disconnect look like in employment, healthcare, and schooling?
• Can we imagine prosperity that does not require surveillance of inner life?

H. A responsible conclusion: the steering system must stay human‑governed

This essay does not need prophecy to reach a conclusion. The conclusion is a responsibility statement: what must remain protected while the future is implemented.

If the real gap is the steering system, then the mission is not only to build smarter machines. It is to ensure that steering—priorities, limits, and accountability—remains human‑governed.

Five commitments (a workable stance)

• Rights: cognitive liberty, mental privacy, and meaningful consent.
• Ownership: people must not lose control of neural data, identity models, or the ability to disconnect.
• Exit: refusal must remain real—no hidden penalties for opting out.
• Accountability: responsibility must be traceable across designers, owners, operators, and institutions.
• Culture: the goal is not only efficiency, but a society where humans remain capable of care, judgment, and refusal.

A minimum governance package

• Clear purpose limitation (therapy vs enhancement vs monitoring).
• Independent audit and safety evaluation.
• Strong security requirements (treat as critical infrastructure).
• Data minimization and local control wherever possible.
• Transparent escalation paths (human override, incident reporting, redress).
• A legally protected right to disconnect.
• Real penalties for coercion and misuse.

This is not bureaucracy for its own sake. It is the price of trust—especially when systems become default.

Closing paragraph

The question is not whether we will build powerful intelligence. We will. The question is whether we will build the conditions for human dignity alongside it. Let machines become competent. Let societies become wise. And let the steering system remain human‑governed.

---

© Robert F. Tjón, February 2026 | Creative Commons CC BY-NC-ND 4.0 International

* Jánszky’s Future Picture:

https://rftjon.substack.com/p/janszkys-future-picture?r=35vtu2

**Deepening Interview with Adam Marblestone:

AI is missing something fundamental about the brain 👇

https://open.substack.com/pub/dwarkesh/p/adam-marblestone?utm_campaign=post-expanded-share&utm_medium=web