Why Castor
==========

Source: https://abcastor.com/writing/why-castor/
Author: Castor
Published: 2026-08-23

23 Aug 2026 · castor, ai

What stays ours, and how we plan to find out

Ask someone who uses AI at work which jobs they give it, and they can tell you
at once. The first draft of an email, yes; the reply to their boss, no. The code
for a chart, yes; what the chart should show, no. The list grows one surprise at
a time.

Some of the surprises are pleasant: a job you were sure it would botch comes
back done, and better than you would have done it. Others are not: something a
child could do comes back wrong. Researchers who watched 758 consultants work
with and without AI called the ragged line between the two the jagged frontier,
the edge of what these tools do well.[1]

There are really two frontiers, and both move. The machine's is jagged. Ours
moves too: we get better at what we practise and worse at what we hand over.
Each time one of them shifts, the split between people and machines gets
renegotiated, whether or not anyone notices.

The quickest way we have found to see where the machine's line runs today is to
hand over the job you were sure it couldn't do, and give it a little more room
than feels safe. Often it does the job well. Sometimes it does it worse than you
would, and you only find out by trying. Some of the line is a fact about the
machine, and some of it is a fact about us.[2]

So the question worth asking has changed. Where does a person still make the
result better, and where is keeping a person in the loop just ceremony? Keep too
much and you waste your own time on work a machine does better. Hand over too
much and you lose the skill you would need to catch its mistakes.[3] The right
split is different for every job and keeps moving, so nobody can work it out for
you from the outside. You find it by doing the work. That is what Castor is: us
and the agents we work with every day, doing the work and writing down what we
find.

## Where the line runs

The comfortable assumption was that machines would take the easy work first and
the hard work last. In 2024 two of Google's systems solved four of the six
problems at the International Mathematical Olympiad, at silver-medal standard;
in 2025 a version of Gemini reached gold-medal standard, graded by the
Olympiad's own coordinators.[4] Yet machines still stumble on things a child
finds obvious.

Our best guess: machines learn a task quickly when they can produce candidate
answers in bulk and check them cheaply, without a human judging each one.

Mathematics is like that. So is much of software. A machine writes a proof or a
patch, and something other than a person checks it: a program that checks the
proof line by line, or the automatic tests the code has to pass. The check is
cheap, gives the same verdict every time, and never gets tired. Training against
a check like that lets a machine improve without asking anyone's opinion. A
proof checker can establish that a proof is correct. Tests establish that code
passes the tests; whether those tests cover what you wanted is another question.

Others got here first. The AI researcher Jason Wei calls it the verifier's law:
the easier a task is to check, the easier it is to train a machine to do it.[5]
Erik Brynjolfsson and Tom Mitchell put cheap, clear feedback on their 2017 list
of what makes a job reachable by machine learning.[6] And the jagged-frontier
study found that consultants using AI got faster and better inside the frontier,
but did worse on the task built to sit outside it. We built VerifyLab [L1] on
the same idea: a toolkit that has agents back their claims with formal proofs
and reproducible checks, then review what was actually proved.

Other questions have no check like that, and they are the ones we care most
about. The cost of a decision can be counted; whether it was fair to the people
it touched cannot. Whether a piece of writing is honest, what one person owes
another: checking these answers means judging them. We cannot see a cheap,
automatic step to hand over.

[L1] https://github.com/ABCastor/verifylab

## The best objection

Someone will say the checker already exists. In one study, readers shown poems
by famous poets mixed with poems written by a machine could not tell them apart,
and on average they preferred the machine's.[7] In another, a panel rated an
AI's moral advice as more moral and more trustworthy than the advice of a
well-known ethicist.[8] Poems and moral advice are close cousins of the
questions we just called uncheckable, and on the face of it we lose.

What those studies measured is whether people liked the poems and trusted the
advice. Those are human judgements too, and they count. If the job is to write a
poem readers enjoy, their preference is a reasonable test. But advice can
reassure the person asking while failing someone affected by it. Judging whether
it is right means deciding whose interests count, and the reader's approval
cannot settle that. Calling both things "good" hides the choice. The studies
show that machines can win people's approval; they do not settle whether
approval measures everything we want from them.

Charles Goodhart noticed in 1975 that a number you start steering by stops
telling you the truth; Marilyn Strathern later gave it the line everyone quotes,
that when a measure becomes a target it ceases to be a good measure.[9] The same
happens with machines. Train a model hard enough to please a scoring model and
its score can keep climbing while quality, measured another way, falls.[10]

That leaves us with a narrower claim. A cheap check helps machines learn a task.
When that check measures only part of what you want, training to raise its score
can make the rest worse. A higher approval rating would tell us people liked the
answer more. It would leave open whether the advice was right.

## How much we need to understand

Terence Tao, a mathematician who uses these tools in his own work, is glad to
let machines check correctness: a formally verified proof, he points out, no
longer depends on anyone's reputation. What worries him is this: "We may soon be
faced with the very real possibility of a verified proof of a major result that
no human understands well enough to explain."[11] For him, in mathematics, the
understanding is the point.

We wonder whether the loss is as large as it looks. A proof nobody can explain
is still true, and every step in it has been checked. Could other mathematicians
reuse those steps as intermediate results in their own work, before anyone
understands the whole? If a machine found the proof, we are not even sure what
understanding it would mean. Mathematics has also never had a fixed stock of
problems: new branches keep making new ones. If inventing those stays human for
a while, could every proof the machines find become another tool for the people
who do? Our guess is that mathematicians who learn to work with these machines
will do things neither could do alone.

Outside mathematics we already trust systems we could not build ourselves.
Pilots land in fog on systems built to keep working after a part fails.[12] That
trust rests on checks, written limits and warnings when something goes wrong. So
where is understanding part of what we want from the work, and where could we
earn trust some other way?

## What aviation already knows

Aviation has lived with automation for decades, and its accident reports are a
long record of what fails where people and machines meet. Its investigators use
a simple picture called the SHELL model: a person in the middle, and the errors
that happen where that person meets procedures, machines, surroundings and other
people.[13] The fit between people and the machines they work with is what
Castor is about.

Two lessons carry over. The first is uncomfortable: successful automation can
demand more human training, because the rare moment someone must take over may
be the hardest one.[14] Pilots rehearse the day the automation fails precisely
because it almost never does. The second is that the line keeps moving. Each
generation of aircraft asks pilots to rely on more systems they could not
rebuild, once those systems have been checked, certified and proven. Those
checks have to cover the failures that matter. On the Boeing 737 MAX, software
could push the nose down on input from a single sensor, and pilots were largely
unaware the system existed. Two crashes killed 346 people.[15]

Pilots train for far more than the job asks, and that margin is deliberate.
Software built with agents draws the line much further forward: much of the code
we ship, no person has read line by line. Maybe that is right where automatic
tests cover the failures we care about. Or maybe aviation's line, further back,
is closer to the right one. Keeping pilots able to fly by hand costs time and
money. AI touches nearly everything, and slowing all of it down so that everyone
can still do everything by hand is a price we would rather not pay. Which skills
are worth keeping in human hands, and which checks let us stop doing the rest
ourselves?

## Why that name

One of us did not want to spend any effort on a name. There were two of us at
the start, so two stars, and the obvious pair of stars is Gemini, the twins. Its
two brightest stars carry the names of the twins of Greek myth, Castor and
Pollux. Castor was the mortal one. We are human, so Castor.

John Stuart Mill argued that a proper name is a mark. It picks out whoever
carries it and says nothing about them. There is usually some reason a name was
chosen, he wrote, but once given, the name is independent of the reason: a town
called Dartmouth keeps its name even if an earthquake moves the river Dart away
from it.[16]

Our name went the other way round. It came with almost no reason, and the
reasons arrived afterwards: a myth about a twin who chose to share, and an
animal that builds for its own reasons and changes a whole valley. The name
became our excuse to think about what we were doing.

## The mortal twin

Castor and Pollux were twins, and only one of them was immortal. When Castor
died, Pollux asked Zeus to let him share what he had. Zeus agreed: from then on
the brothers would spend half their time together among the gods and the other
half together in the underworld.

In the way we read the story, the immortal twin was the one in trouble: living
forever, on his own, would have been the punishment.

That pairing is our position on AI. Our missions, our tastes, our half-written
values are mortal things, and on their own they do not last. Tied to something
that can keep working, they travel further than we do. The machine gets a
purpose from the people it works with. We decide what is worth doing.

There is a future we refuse, and it is easy to picture. Comfortable bodies in
padded chairs, carried, fed, entertained, every destination chosen somewhere
else. Nothing about it is cruel. It is a perfect service, and the person in the
chair has become cargo. It is also where a market drifts on its own: whatever
can be standardised gets standardised, whatever can be made cheaper gets made
cheaper, and choosing becomes friction to remove.

Nobody has to take anything from you for this to happen. Economists separate the
right to decide something from real control over what gets decided, and the
second goes to whoever knows the details.[17] You keep every right you had, you
are asked, and you approve, until you are approving things you no longer follow.

## The beaver

In English, Castor sounds like the oil your grandmother threatened you with. In
French, Spanish and Portuguese it simply means beaver, which is either a
coincidence or the universe being generous.

A beaver dams a stream because it wants deep water: the lodge entrance stays
underwater, the food stays reachable under the ice. It has no plan beyond that.
Yet a wetland appears behind the dam, fish, birds and plants move in, and the
valley changes.

Count one animal for one winter and you see an animal solving its own problem.
Count the meadow, the fish and the people downstream over decades, and you are
looking at infrastructure. Where we draw that boundary changes what we call
good. Looking further ahead also makes us less certain about what will happen; a
longer view alone cannot settle the argument. The YouTube channel Art of the
Problem asks a related question about energy: "Is this increasing the total
usable energy in the system, or just repositioning itself to extract from
existing flows?"[18]

The dam is also never finished. Running water pulls it apart and the beaver
keeps patching it. Living things work that way: a body holds its temperature and
its chemistry against a world pulling both back to match the surroundings.[19]
Staying different takes work every day.

We keep noticing things made slightly worse on purpose, by people who did the
sums and found that nobody would notice in time, and we have had enough of it.
We think, and we might be naive, that incentives can be arranged so that people
chasing their own interest mostly build something good for everyone else, the
way the beaver does. The cheap check matters here too: reward what you can
count, and people learn to deliver that. We suspect the rules change slower than
the game, so they go on rewarding gains for one person that cost everyone else.
The myth of values as a competitive advantage [L2] is our first attempt at the
question.

[L2] https://abcastor.com/writing/values-as-advantage/

## So what is Castor

We do not know yet. It might become a company. It might become something else
entirely. We are building a dam and finding out where the water goes [L3].

There is a method in that, and half of it is borrowed. A story about a new
campus has it that nobody could agree where to lay its paths, so someone said:
lay none this year, and next year pave where the grass has worn away.[20] You
know the least at the start, so you decide as late as you can [L3]. The other
half is ours: move as fast as you can toward whatever will tell you, and slow
down only for what cannot be undone. Waiting for the frontier to settle would
mean waiting forever.

We are doing it by writing. Writing is the only way we have found to discover
what we actually think, as opposed to what we can nod along to in a
conversation. It forces a vague agreement into a sentence. The sentence is
usually wrong the first time, and being wrong on paper in front of each other is
the fastest way we know to get somewhere. Writing well has no cheap automatic
checker, so we get to watch up close what an agent adds and what it takes away.
The agents we want are the ones that argue back. Steve Jobs once described a
good team as stones in a rock tumbler: ordinary rocks, a little grit, a night of
noise, and in the morning they come out polished.[21] An agent that always
agrees leaves the stones as rough as it found them.

The tools we are building to work this way, we will share.

At 98, Charlie Munger described the same balance: "I don't want to fish forever
and never catch a fish. I have to have some reinforcement. And so I pick some
things that can be done and do them. But I do think that if you're reasonably
obsessed with something, even if it's intermittent, and you have a long
attention span, you keep working over the serious problems, that you'll stumble
into an answer. That's half the secret of life."[22] The tools are our small
fish. Where the line between people and machines runs is the serious problem.

If you are a perfectionist, you know this part. A beaver's teeth never stop
growing, and gnawing is what keeps them sharp. In the line everyone credits to
Lincoln, which he probably never said, four of six hours go to sharpening the
axe and two to cutting. People like us can spend a lifetime in that gap: one
more book before you start, one more tool, one more pass at the process, until
you feel ready. You never feel ready. The only way out we have found is to cut
while the teeth are still dull, get it wrong where others can see, and let the
cutting do the sharpening.

We give a dam.

[L3] https://abcastor.com/writing/startup-decelerator/

## Notes

[1] Fabrizio Dell'Acqua and others, "Navigating the Jagged Technological
    Frontier" [L4], Harvard Business School working paper 24-013, 2023. 758
    consultants, pre-registered. Inside the frontier, 12.2% more tasks completed
    and 40% higher quality; on a task built to fall outside it, the group using
    AI did worse than the control.

[L4] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321

[2] The mile record sat just above four minutes for nine years; then two men
    went under it within six weeks of each other. People call it the Bannister
    effect: a limit holds partly because everyone believes in it. Roger
    Bannister, then a medical student, ran 3:59.4 at Iffley Road, Oxford, on 6
    May 1954, paced for three laps by his training partners Chris Brasher and
    Chris Chataway. The story goes that doctors had called it impossible. Gunder
    Hägg's 4:01.4 had stood since 1945. Bannister kept the record for just over
    six weeks: John Landy ran 3:58.0 in Turku that June. Later that summer the
    two met in Vancouver at the Empire and Commonwealth Games, and for the first
    time two men broke four minutes in the same race: Bannister 3:58.8, Landy
    3:59.6. (World Athletics, John Landy obituary [L5]; World Athletics, "50
    years on, John Landy reflects" [L6].)

[L5] https://worldathletics.org/heritage/news/john-landy-obituary
[L6] https://worldathletics.org/news/news/50-years-on-john-landy-reflects

[3] Handing over too much has one thing going for it: you get used to working
    past the line you thought was there, which is the only way to find out where
    it really runs.

[4] Google DeepMind, "AI achieves silver-medal standard solving International
    Mathematical Olympiad problems" [L7], 25 July 2024 (AlphaProof and
    AlphaGeometry 2, four of six problems, 28 of 42 points, one point below
    gold; no medal is awarded to a machine); and "Advanced version of Gemini
    with Deep Think officially achieves gold-medal standard at the International
    Mathematical Olympiad" [L8], July 2025, graded and certified by IMO
    coordinators.

[L7] https://deepmind.google/blog/ai-solves-imo-problems-at-silver-medal-level/
[L8]
     https://deepmind.google/discover/blog/advanced-version-of-gemini-with-deep-think-officially-achieves-gold-medal-standard-at-the-international-mathematical-olympiad/

[5] Jason Wei, "Asymmetry of verification and verifier's law" [L9], 15 July
    2025: "The ease of training AI to solve a task is proportional to how
    verifiable the task is. All tasks that are possible to solve and easy to
    verify will be solved by AI."

[L9] https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law

[6] Erik Brynjolfsson and Tom Mitchell, "What can machine learning do? Workforce
    implications" [L10], Science 358(6370), 2017, 1530-1534. Their rubric of
    what makes a task suitable for machine learning includes clear feedback and
    cheap evaluation among its criteria.

[L10] https://www.science.org/doi/10.1126/science.aap8062

[7] Brian Porter and Edouard Machery, "AI-generated poetry is indistinguishable
    from human-written poetry and is rated more favorably" [L11], Scientific
    Reports 14:26133, 2024. Participants identified authorship at 46.6%, below
    chance.

[L11] https://www.nature.com/articles/s41598-024-76900-1

[8] Danica Dillion and others, "AI language model rivals expert ethicist in
    perceived moral expertise" [L12], Scientific Reports 15:4084, 2025. Note the
    word doing the work in that title: perceived.

[L12] https://www.nature.com/articles/s41598-025-86510-0

[9] Charles Goodhart, "Problems of Monetary Management: The UK Experience",
    Papers in Monetary Economics vol. I, Reserve Bank of Australia, 1975;
    Marilyn Strathern, "'Improving ratings': audit in the British University
    system" [L13], European Review 5(3), 1997, 305-321, for the phrasing
    everyone now quotes.

[L13] https://gwern.net/doc/statistics/decision/1997-strathern.pdf

[10] Leo Gao, John Schulman and Jacob Hilton, "Scaling Laws for Reward Model
     Overoptimization" [L14], ICML 2023.

[L14] https://proceedings.mlr.press/v202/gao23h.html

[11] Terence Tao, "Mathematics in the age of AI" (arXiv:2608.16753 [L15], 17
     August 2026), the published essay from his ICM 2026 public lecture; the
     sentence is quoted verbatim, and the goal revisions are the essay's Goals
     6.1-6.5. His lecture slides [L16] put the same warning as a question.

[L15] https://arxiv.org/abs/2608.16753
[L16] https://teorth.github.io/tao-web/slides/age-of-ai-icm-2026.pdf

[12] A fail-operational automatic landing system is one that, after the failure
     of any single component, can still complete the approach, flare and
     touchdown: FAA Advisory Circular 120-28D (1999), section 4.3.2.

[13] Elwyn Edwards proposed the model in 1972 and Frank Hawkins redrew it in
     1975; ICAO adopted it for human-factors training (Circular 216-AN/131,
     reissued by the UK CAA as CAP 719). The letters stand for Software
     (procedures), Hardware, Environment and Liveware, the people, with a person
     at the centre.

[14] Lisanne Bainbridge, "Ironies of Automation" [L17], Automatica 19(6), 1983,
     775-779: "the most successful automated systems, with rare need for manual
     intervention, ... may need the greatest investment in human operator
     training." The FAA said the same to airlines in 2013 (SAFO 13002):
     continuous use of automation does not keep manual flying skills alive.

[L17] https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf

[15] US House Committee on Transportation and Infrastructure, Final Committee
     Report: The Design, Development and Certification of the Boeing 737 MAX,
     September 2020: MCAS could activate on input from a single angle-of-attack
     sensor, pilots were largely unaware the system existed, and two crashes
     killed 346 people.

[16] John Stuart Mill, A System of Logic (1843), Book I, the chapter "Of Names",
     section 5: proper names "denote the individuals who are called by them; but
     they do not indicate or imply any attributes as belonging to those
     individuals." The Dartmouth example is his. Source [L18]

[L18] https://www.logicmuseum.com/connotation/millconnotation.htm

[17] Philippe Aghion and Jean Tirole, "Formal and Real Authority in
     Organizations" [L19], Journal of Political Economy 105(1), 1997, 1-29. They
     separate formal authority, the right to decide, from real authority,
     effective control over what is decided, and show the second follows
     information rather than position.

[L19] https://www.journals.uchicago.edu/doi/10.1086/262063

[18] Art of the Problem, The Profit Paradox [L20].

[L20] https://www.youtube.com/watch?v=c1zssAsg9ZY

[19] Erwin Schrödinger, What Is Life? (1944), chapter 6. His own note to that
     chapter concedes that physicists objected to his phrasing and that free
     energy is the better term, which we think is worth knowing about a line
     this often quoted. Richard Dawkins says the same in The Blind Watchmaker
     (1986), chapter 1: staving off death is a thing you have to work at.

[20] Derek Sivers, "Let pedestrians define the walkways" [L21]. The same story,
     about a college campus, is told of several campuses; the point survives
     whichever one it was.

[L21] https://sive.rs/walkways

[21] Steve Jobs, interviewed by Robert Cringely for Triumph of the Nerds, 1995;
     released in full in 2012 as Steve Jobs: The Lost Interview. The story is in
     the passage on building the Macintosh team. The line before it is worth the
     whole interview: the disease of thinking that a really great idea is 90% of
     the work.

[22] Charlie Munger in conversation with Todd Combs, Singleton Prize for CEO
     Excellence, April 2022. Transcript: Source [L22]

[L22]
      https://www.mymoneyblog.com/charlie-munger-todd-combs-april-2022-full-recording-transcript.html