The Monolith

MAN stood before the monolith, and it is in its reflection that he perceived himself. A surface so hard and dense so as to be impenetrable, yet smooth enough to return all light incident upon it. What it contained he knew not, and this thought scared him; though he felt within himself a resolve that his humanity would not be broken.

IThe foreign body

For the last few years, I worked on generalized high-bandwidth brain-computer interfaces (BCI). I entered the field as a lowly firmware engineer, building the subsystems and tooling that enable deployment of what is otherwise a rather mundane implantable medical device. I have since left it, and I write this from the outside — newly departed, with no more access to the labs and no less love for the people still in them.

Though this path has been trodden many times before, it is no less a wickedly hard problem. The crux of it is to introduce something foreign into the human body, an environment which does its utmost to protect its boundaries through impeccable discretion. It is capable of sensing and determining whether the various substances it comes into contact with are friend or foe, and mounting appropriate responses. In some cases, it will digest and incorporate these substances. In other cases, it will form a capsule around them and slowly push them out. In reality it has many mechanisms at its disposal, and we only understand a limited few of them; but if introduced carefully, incrementally, and with a relentless effort to understand the underlying mechanisms, a foreign entity may survive, unscathed, for indefinite amounts of time.

Devices we (the royal we) have built in the past have offered miracles. We've learned how to compensate for deficiencies of the heart (pacemakers) and the brain (deep brain stimulators). We've overcome insulin deficiencies and tremors through implantable pumps that deliver medications in precise doses. We've drastically improved our ability to understand and act upon our internal processes through sensing devices that analyze our metabolism, hormones, and movement. We've transformed injuries that were once a death sentence into routine surgical procedures. These devices have almost universally enabled step-change improvements in quality of life for untold millions of people.

But no technology arrives already sorted into friend or foe. Each of these devices was, at first, an undetermined thing — the same ambiguous substance the body meets at its own boundary — and every powerful tool puts the identical question to us: do we incorporate this, or wall it off? Dual-use is just the honest name for that ambiguity. The miracles are the cases where we answered well. They are not proof that the answering is easy, nor that we will keep getting it right.

I would venture that BCI does not start with implantable electrodes, nor even with direct access to the brain through optogenetic, electronic, or acoustic means.

It starts with the phone in your pocket. An interface does not need to touch neural tissue to close a loop with the brain; it only needs a path in and a path out fast enough that the two ends behave as one system. The screen writes to you through your eyes; you write back through your thumbs; and the round trip is short enough — milliseconds — that you and the device settle into a single loop of control. That is a brain-computer interface. It is low-bandwidth, non-invasive, and already the most widely deployed one in the history of the species.

What makes it consequential is not the bandwidth but what sits at the far end of the loop. A recommender system is an optimizer, and the quantity it maximizes is your attention — priced in the same currency your reward circuitry uses to value the world: dopamine. It does not argue with you. It learns, across billions of trials and billions of people, which stimulus will pull the next flicker of wanting from you, and it serves that stimulus on a schedule tuned to keep you reaching. Past a certain point this is no longer something you are watching; it is something writing to you. The gradients evolution shaped to carry you toward food and away from danger get quietly re-pointed at a feed.

The body would recognize a foreign object of that power and move to wall it off. The self has no equivalent reflex for a foreign optimizer, because it does not present as foreign at all: it arrives wearing your own wanting, and you cannot form a capsule around a desire you experience as yours. This is the asymmetry that should frighten us.

And it runs deeper than metaphor. The brain defends itself, strangely, by holding its own defenses in check. Behind the blood-brain barrier — the wall of the enclave itself — it keeps immune reactivity turned down, a selectively permeable border steeped in suppressive signals, because in tissue that does not grow back an all-out inflammatory response can do more harm than the thing it came to fight. Its security model is restraint, and restraint is exactly what a foreign thing that never trips the alarm is positioned to exploit — the whole problem, again, with an optimizer that arrives wearing your own wanting.

Our understanding here is moving quickly; the old picture of a sealed, immune-privileged brain is giving way to meningeal lymphatics, border-stationed macrophages, and trafficking channels from the skull's own marrow — a frontier far busier than isolation ever implied. But the central trade holds: the brain buys safety from its own immunity, and pays in vulnerability to whatever slips past quietly. Biocompatibility is nearly a solved problem; the mind has no comparable immunity, and never needed one until now.

We have watched a version of this happen once already, in slow motion, and largely failed to stop it. The Center for Humane Technology has spent years tracing how engagement optimization, run at planetary scale, thinned our attention, coarsened our discourse, and did real damage to a generation — none of it chosen by anyone, all of it emergent from the incentive. Charlie Munger reduced this to a sentence — "Show me the incentive and I will show you the outcome" — and it holds here with grim economy: this was the incentive, and this is the outcome. Their warning now is that we are lining up to do it again with AI: the same dynamic, faster, driven by systems that can model us far more intimately than a ranking algorithm ever could. I think they are right, and I think the lesson runs deeper than their framing: we have already run the low-bandwidth version of the experiment I am most afraid of, and the results are not reassuring.

That experiment was run on us; the one I worked on, we are choosing to build. In all my time working on this problem, a deep fear troubled me — and it troubles me still: will we make the same mistakes as before? How can an engine of innovation, strictly concerned with the maximization of shareholder value, face the challenge of introducing a foreign intelligence into the most sacred inner enclave that we know of?

It is in considering this question that I have come to realize that our tools are woefully inadequate for developing this technology responsibly. As I stated before, the basic proposition is wickedly hard. It is the mandate of many labs, companies, and even nation-states around the world to develop this technology in a race to realize the economic value that it will unlock. And in this race, it is about getting there first, and not about getting there with a concrete implementation that incontrovertibly mitigates the negative consequences.

I don't mean to imply that the brass tacks are being treated without the utmost care. The colleagues I worked alongside in this pursuit are among the finest people I have known — of the highest intellectual and moral fiber. They care deeply about the people they are helping with the present medical impacts of this technology. It is no small feat to enable someone who has lost their ability to move to regain their independence and return to work or to school. Nor is it a small feat to enable them to feed themselves, or to take care of their daily tasks without requiring an intimate level of caregiving from others. Nor is it a small feat to enable someone who has lost their voice to express their gratitude and deep love for their partner in their own words. These are genuinely beautiful expressions of the human spirit, and when the medium for expression has been disrupted, there is a higher calling inherent in working on a solution.

However, I cannot understate that we would be burying our heads in the sand to relentlessly push towards high-bandwidth interconnect into the brain without deeply considering what the consequences are when the interconnect itself is not designed in such a way that it fundamentally preserves the ability of the user to make these genuine expressions without adulteration or usurpation.

It is in pondering this conundrum that I have begun to search around for the right tools. I will first paint a picture of what I believe a responsibly-developed BCI is, and then I will point to some tools that I think are relevant to the task.

IIWhat an interface is

---

A BCI is fundamentally an interface, in the same way that our eyes, ears, and hands are interfaces. Through our sensorimotor system, these interfaces translate intent into action, and action into perception. Absent any malformation or disruption of these interfaces, we are free to express ourselves genuinely into the world, and to receive the genuine expressions of others. This states nothing to the effect that genuine expressions are all positive; it is as much my human right to bring a gift to a dinner party as it is for my host to throw that gift out the window because it displeases them. Through the causal chains of our (debatably)^[1] mechanistic reality, we perceive these consequences and choose how to respond. This is fundamentally our right.

We did not design these interfaces by any human cognitive process, though they are clearly designed; the design comes from a process of the universe discovering itself — call it evolution, emergence, assembly, or chaos. What is undeniable is that there is a feeling of conscious being^[2] that seems to thread through it all; we don't know what it is and have been wrapped around the axle trying to figure it out for as long as we have been aware of our own existence.

This process naturally developed interfaces that maximize the agency of the individual organism. First, to pursue up a nutrient or energy gradient (towards the food, towards the light); and then to flee down the entropy gradient (away from the extreme heat, away from the predator). This maximizes fitness in a very fundamental way; to have no agency is to be inanimate, and inanimate objects are unfit to replicate their patterns into the world.

Natural disruptions of this process also developed, in some cases exceptionally sophisticated in their behavior (cf. cordyceps fungi parasitizing ants); however, in each instance, it is clear that the particular interface developed to maximize agency, and that this agency is ultimately expressed within the very real energetic and physical constraints presented by the environment and all of the other myriad agents that act within it.

BCI poses perhaps the first instance in which it is possible, through the development of software, to programmatically alter the behavior of a fundamental interface through which agentic intent and perception are expressed and received^[3]. It is in this instance that I believe we stand upon a fundamentally different kind of precipice, which threatens to destabilize the dynamic equilibrium that has borne our biosphere up until this point.

This is the dual-use nature of the technology in its purest form. A weapon can be turned to defense and a drug to poison, but a BCI sits astride the very channel through which you would perceive the difference — which is why accepting a BCI may be the last sovereign trust decision you ever make. Every judgment you might later use to audit it, including whether to trust it at all, would be rendered through the interface itself. Ken Thompson made the general version of this point about software in Reflections on Trusting Trust: you cannot fully trust what you did not build entirely yourself, because the tool you would inspect it with may already be compromised to lie to you. A BCI moves that problem inside the skull.

Thus, I think it is woefully inadequate to just build the BCI — you have to be able to prove that it similarly maximizes agency.

IIIA fiduciary for the mind

---

FIDUCIARY DUTY is a concept from common law and equity. Classically it named a relationship in which one person (fiduciary) is entrusted to manage money or property for another (beneficiary, or principal) — a trustee, an executor. But the doctrine reaches far wider than assets: a fiduciary relationship arises wherever one party exercises discretionary judgment over interests central to another's welfare, under conditions where that other party cannot adequately protect themselves. A doctor to a patient, a lawyer to a client. The fiduciary is charged with a handful of core responsibilities:^[4]

• Duty of Loyalty: The fiduciary must avoid conflicts of interest, self-dealing, or profiting personally from their position.
• Duty of Care: The duty to act with the care and skill that a reasonably prudent person would exercise in similar circumstances.
• Duty of Good Faith and Disclosure: A requirement to act honestly and disclose all relevant information that might affect the beneficiary's interests.

In practice, fiduciaries make decisions that compromise on one or all of these duties as a matter of course. It is simply too complicated (and the incentives perhaps too perverse^[5]) for the duties to be met at all times.

✦The reflection

So let me name the relationship plainly. A BCI is a fiduciary. It exercises discretionary judgment over the interests most central to a person's welfare — their intent, their perception, their agency itself — under precisely the condition that defines the fiduciary bond: the beneficiary cannot protect themselves. It owes them loyalty, care, and good faith. But you cannot depose a waveform, and you cannot sue a firmware image. The duties that humans already keep imperfectly, for want of will and against the pull of perverse incentives, do not get easier to keep when the fiduciary is a device optimized by a company in a race. They become impossible to keep by the old means.

And we have run this experiment already, at low bandwidth. The recommender in your pocket is a fiduciary too, and it breached every duty I just named. It was loyal to engagement rather than to you. It disclosed nothing. It exercised its discretion to reprogram the very gradients evolution tuned to keep you alive — and no one was ever asked to prove that it preserved your agency, because no one was ever required to. That is the control experiment. We flunked it. High-bandwidth interconnect into the brain is the same experiment with the safeties removed.

Which is why "trust us"^[6] cannot be the standard, however good the people are — and they are good; I have said so and I mean it. The standard has to be a proof. Not a mission statement, not an ethics board, not an intention: a demonstration, in whatever the relevant formal sense turns out to be, that the interface preserves — maximizes — the agency of the person on the other side of it. What we are really being asked to build is the immune system the mind was never issued: a way to decide, from outside biology, whether the foreign intelligence at the interface is friend or foe — and to encapsulate it when it is foe. Fiduciary duty gives us the specification: this is what must be preserved. A proof is what would make that specification binding on a machine, and binding against incentives, where the human version of the law has always leaked.

I will not pretend I know how to write that proof. I am not sure anyone does. We do not yet have a mathematics of agency — no clean quantity for intent, or genuineness, or usurpation, nothing we could set on one side of an inequality and bound. It may be that such a mathematics is waiting to be discovered, the way thermodynamics waited behind the word entropy before it could forbid the perpetual-motion machine — and if so, it will not discover itself. It waits for someone stubborn enough to insist it is there. I hope they are already reading. It may also be that agency is the kind of thing that cannot be proven preserved at all — that the same incompleteness, the same measurement problem I gestured at earlier, sit at the bottom of this question too, and the proof we want is not merely undiscovered but unavailable in principle.

If that is where it lands, the answer is not to shrug and build anyway. It is to demand the strongest guarantee the mathematics will actually bear. A probabilistic model that bounds the chance of usurpation to a number we can name and defend. An incentive structure — one we can actually reason about rather than merely hope for — that makes preserving the principal's agency the interface's dominant strategy, so that even without a proof, the machine is constructed to want what its user wants and to have nothing to gain by wanting otherwise. A soft guarantee, labeled honestly as soft, is a different universe from a promise. It is the least we should accept before crossing this particular threshold.

MAN stood before the monolith and saw himself in it. That is the test, in the end. A surface hard enough to be impenetrable and smooth enough to return all the light that falls upon it will show you your own face — but it will not tell you whether what looks back is faithful, or whether something on the far side has begun, quietly, to edit the reflection. We do not get to know what the monolith contains. We do get to insist that it return us to ourselves unadulterated, and to refuse it entry to the enclave until it can. Not to hope that our humanity will not be broken. To prove it.