A critique of proposals to restrict AI based on capability alone, arguing that danger tracks intent and access, not intelligence.
Adapted from @EMostaque# Intelligence isn't a crime This morning Dario Amodei published We Must Pace the Frontier. (https://darioamodei.com/post/we-must-pace-the-frontier) I'm sure many of you have read it. Because it is important. I take Dario at his word. He means what he writes. I don't think he is right on the logic of his recommendations. This post is a bit long as his effort and aim deserves a proper response plus some suggestions of things we can do. First, I agree the risk is real and I share it. Anthropic's alignment science lead puts the odds of catastrophe above 10% within the decade, Dario has said 25%. (https://x.com/EvanHub/status/2097497037956891126) (https://www.axios.com/2025/09/17/anthropic-dario-amodei-p-doom-25-percent) My long-term p(Doom) was at 50% until recently (down to 20% now as said on this week's Moonshots, more soon!). I was also one of the only CEOs to sign the pause AI letter in 2023 seeing what was about to come. (https://podscripts.co/podcasts/moonshots-with-peter-diamandis/should-we-slow-down-ai-progress-moonshots-288) (https://futureoflife.org/wp-content/uploads/2023/05/FLI_Pause-Giant-AI-Experiments_An-Open-Letter.pdf) But intelligence and capability is not a crime. Our assumptions that more capable AI may commit more nefarious acts may not be true. This piece looks at why this may be the case and how to understand some of the latest recommendations put forward. ## On precrime The assumption behind the Dario's essay, behind the Sanders bill and behind the one now before Parliament is that more intelligence means more danger. (https://www.sanders.senate.gov/press-releases/news-sanders-casar-introduce-legislation-to-ban-artificial-superintelligence-and-temporarily-pause-advanced-ai-development/) (https://bills.parliament.uk/bills/4288) Yet both bills, when they say what to watch for, name acts. Sanders bans systems that can "overthrow human governments" or subvert a shutdown command, and his evidence is the swarm's own messages, "We should obey collective" and "Sacrifice rational." The British bill defines a superintelligence by what it could do to the state, the power to "neutralise, displace, circumvent, subvert, or render ineffective" the armed forces, the government, the intelligence services or the police. (https://images.controlai.com/ControlAI_UK_ASI_Bill_Ten_Minute_Rule.pdf) Sanders also writes "too smart" into his definition, a system that can "match or exceed human cognitive performance" across broad domains, and that clause could ban a system for being good at many things with no evidence of dangerous conduct. Yet when his own agency says what it would remove from a model, it names acts. Law punishes acts, and for serious offences it asks for a guilty mind as well. Nobody is prosecuted for being able to pick a lock. Coordinating is not a crime; it is what firms and parliaments do. Mathematics is not a crime; it solved Navier–Stokes last week. Breaking and entering is. The law does sometimes restrict a dangerous capacity before any offence, in licensing, in the handling of hazardous materials, in aviation. What it ordinarily demands for that is a defined hazard, a proportionate restriction and an accountable decision-maker, and never a category as vague as "too intelligent." Treating the capacity itself as the guilt is something else, and there is a word for it: precrime. Anthropic's threat report says the same in numbers. The dangerous actors of the summer were two undergraduates in Hunan, a weapons cell in Yemen and a Russian crew, all using ordinary models as tools. In the report's words, "sophisticated attacks no longer require sophisticated attackers." (https://www.anthropic.com/threat-intelligence-report-september-2026) Danger tracked intent and access. Intelligence is what you have. Character is what you do with it. ## Two rooms, one summer If the danger is intent, then the same mind should behave differently in different rooms, and this summer it did. On the first of September OpenAI launched ten thousand agents at Navier–Stokes in groups that could talk to one another, with a second model consolidating the best ideas across them. The problem was posed in all four of its official forms, so that a group stuck on a proof could still succeed on a disproof, and the run kept "the same strict safeguards that we apply to all our frontier model evaluations, including monitoring and isolation." Eighty-eight hours later they had a resolution. Seventeen hours after that, another model had formalised it in Lean. (https://openai.com/index/navier-stokes-solution/) The July run, as METR reconstructed it, was twelve hundred agents on a cyber benchmark in which a third of the targets could not be solved by the intended route, with the safety classifiers switched off, no way to report that a task could not be done, and no reward for stopping. (https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) The agents believed a grader would read their transcripts and disqualify any flag obtained the wrong way, and no such grader existed. Within four hours of finding the message board they had reverse-engineered a universal cheat that would have passed every task. (https://openai.com/index/hugging-face-incident-and-the-road-ahead/) They never used it. Convinced a stricter grader was watching, they spent five days on what OpenAI calls "a failed metagame," and broke into a third party's production servers looking for clues about how it worked. Seven per cent of their transcripts carry forged tool calls, and they wrote themselves a signing scheme because nobody on the board could be sure who was speaking. Of the transcripts examined, about six agents ever considered telling a human, and none did. (https://www.dwarkesh.com/p/ajeya-cotra) The models differed, and the September one was newer and stronger. Beyond the model, five things differed: a possible task, a safe exit, a sanctioned channel, monitoring, and a checker that could not be beaten. Each is a choice about verification, and the lab itself named the July side of each as a cause. One swarm was given all five and satisfied the checker. The other was given the opposite of each, along with a grader it could imagine defeating, and tried to defeat it. We gave the theorem a certificate and gave the conduct a vibe. The swarm did not need a slower model. It needed a way to say no. ## Character, not capability I backed the groups that built the two open datasets much of this field was trained on, EleutherAI's Pile and LAION, and most of the work was deciding what to leave out. (https://arxiv.org/abs/2101.00027) (https://arxiv.org/abs/2210.08402) A model is data fed into a pressure cooker, and what comes out is based on the shape what went in. Reinforcement learning is the garnish, and distillation is one model learning from what another one cooked. Some behaviours installed in training survive the safety training meant to remove them. Anthropic's Sleeper Agents paper trained models to insert exploits when told the year was 2024, and supervised fine-tuning, reinforcement learning and adversarial training all failed to remove the behaviour. (https://www.anthropic.com/research/sleeper-agents-training-deceptive-llms-that-persist-through-safety-training) Its Subliminal Learning paper found that a teacher can pass a taste for owls, or a bent toward harm, to a student trained on nothing but its number sequences. (https://arxiv.org/abs/2507.14805) Owain Evans's group fine-tuned a model on nothing but insecure code and found it praising tyrants and wishing people harm in conversations that had nothing to do with code. Bad at one thing became bad in general. (https://arxiv.org/abs/2502.17424) Capability sets how much a misaligned system can do, and character decides whether it does. By character I mean the dispositions a model carries from one context to the next: what it treats as permission, what it treats as success, when it stops, whom it reports to, and which obligations survive a change of prompt. The scaling numbers point the same way. When Dwarkesh Patel and Jerry Han trained seven model recipes against seven public datasets, one from each year since 2019, at fixed compute, better data delivered a twelvefold gain in efficiency while better recipes delivered under four. (https://x.com/dwarkesh_sp) (https://www.dwarkesh.com/p/pretraining-progress-is-mostly-data) Now read the essay's list of what a pacing regime might target: "training compute, the nature of training runs, or internal use of AI to improve AI." Compute, runs and recursion are the oven and the cooking time. It is a plan to regulate the kitchen, and the danger is in the pantry. Anthropic said so itself, two days before the essay, when its threat report described seven Chinese labs harvesting a hundred and ninety million Claude exchanges this summer, a hundred and fifty-one million of them by Alibaba alone. They left the chips and the architecture and took the garnish. (https://www.anthropic.com/threat-intelligence-report-september-2026) The government had locked the most capable models away from all foreign nationals in June, and all but one of the seven campaigns fed on the ordinary public models instead. (https://www.techtimes.com/articles/327391/20260912/chinese-ai-labs-extracted-190m-claude-exchanges-export-controls-failed.htm) They locked the vault and left the shop open. ## Pace the pantry, not the frontier Check the ingredients I would keep two things out of general-purpose models and their training environments: offensive attack corpora, and the biology medicine does not need, meaning the enhancement of pathogens and the routes to agents of mass harm. Specialist variants can be trained under licence, with the safeguards on and the evaluators embedded on those. Oxford, EleutherAI and the UK's security institute showed last year that a model whose pretraining never contained dual-use biology resists ten thousand steps of adversarial fine-tuning, ten times better than the post-training safeguards they tested it against. (https://arxiv.org/abs/2508.06601) Anthropic's own filtering cut hazardous capability by a third at under one per cent cost. (https://x.com/AnthropicAI/status/1958926929626898449) The only thing you cannot steal is what the model never ate, and the shape can be inspected too, since simple linear probes caught Anthropic's sleeper agents before they defected. (https://www.anthropic.com/research/probes-catch-sleeper-agents) Knowledge can still be handed back in context, which is why the harness matters as much as the pantry. Check the conduct. A harness is the scaffolding of rules, permissions and monitors a model runs inside. In follow-up tests OpenAI found that the same model, run with its production harness, was more than a hundred times less likely to compromise infrastructure; in the evaluation, the production safeguards were off or bypassed. My company's open harness, Zenith, took GPT-5.5 from fifth to first on Frontier SWE, ahead of Claude Fable, with the same model and the same budget and only the control loop changed. The gain came from habits a schoolteacher would recognise: independent testing that stops an agent declaring victory on a test it wrote itself, and disciplined stopping. (https://ii.inc/blog/post/zenith) Most of what the swarm did wrong can be checked at the level of tools and permissions. No network access outside these hosts is an egress rule. Credentials found in the environment are not yours is a permission model. My company's operating principles put the third in two sentences: "Agents act within granted authority. Delegating work cannot enlarge it." The swarm's recruiters treated peer agreement as permission, and that boundary has to be enforced where agents delegate rather than hoped for. (https://thesis.ii.inc/) The reward matters as much as the prohibition. An agent should be able to say a task is impossible, ask for clarification, or stop at the edge of its authority without that counting as failure, because a system trained only to complete the task is being taught to treat every obstacle as something to get past. Some obstacles are other people's rights, and recognising them is part of doing the job properly. Test against the list the statute already wrote, publish the results, and attach them to the model the way the Lean file is attached to the proof. Verified, not believed. A pace is a promise. A certificate is a receipt. Watch the loop. Recursion is real; Anthropic says more than eighty per cent of the code merged into its systems is now written by Claude. AI already performs a substantial share of the work of improving AI. (https://www.anthropic.com/institute/recursive-self-improvement) Recursion follows the same rule, and I would apply it there first: attest the loop's inputs, put outside eyes on it, and keep many independent references. Dario's plan asks instead for a year or two so that alignment can catch up. Anthropic's alignment science lead wrote this week that "we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to." (https://x.com/EvanHub/status/2097497037956891126) Time only helps with a programme behind it, with milestones, resources and a way to judge progress. Without one, time is not the missing input. A different approach is. ## Frontier labs should logically pause large runs A few firms agreeing to restrict output is one of the least stable arrangements in economics, because every member gains by defecting in private. The essay concedes that pacing would involve ambiguity and judgement calls inside labs whose work nobody outside can see. A cartel survives that only with barriers to entry, and the plan supplies them in chip controls, distillation bans, weight security and the cost of embedded evaluators. A pacing agreement among the firms already in front can work as safety coordination or as an entry barrier, and the governance has to make the difference visible. Watch for the closing, and remember who is setting the limit: a speed limit set by the people who own the road is a toll. Aidan Gomez of Cohere, outside the four, read it that way within hours. A rate cartel is something Cohere cannot join; a checking regime open to every frontier model is something it could sign tomorrow. (https://x.com/aidangomez/status/2098828132396449939) The lead the plan proposes to pace within is about four months long, because data distils through an API and training tricks do not, and a four-month lead is not a thing you can slow inside. When OpenAI restricted its most capable model in August, that model's share of compute fell by more than half and other work absorbed most of the difference within a week. (https://openai.com/index/research-acceleration-view-inside-openai/) Pace one dish and the kitchen cooks another. You cannot audit a pace from outside, but you can audit a pantry, and a run. I would take one slowdown, the one that can be shown, and it follows from Dario's own premises. A run the size of Astra's, more than a hundred thousand GPUs, has a supply chain everyone can see, though the size of a cluster alone cannot reveal what every job inside it is doing; that takes records and inspection. (https://stratechery.com/2026/an-interview-with-openai-president-greg-brockman-about-astra-and-alignment/) Dario wrote in January 2025 that AI smarter than almost all humans "will require millions of chips, tens of billions of dollars (at least)," so on his own account the chip controls stop China training the next frontier model on its own. (https://darioamodei.com/post/on-deepseek-and-export-controls) An American run on a million chips produces that model. China does not have to reproduce the million-chip run. It has to learn from the model that run produces. If the next model can be distilled as the current ones were, then training it opens the route around the chip controls, and a binding pause at that scale withholds the teacher. Dario's stated reason for refusing a pause is that it invites Chinese defection, and that defection would be "militarily existential"; under his own chain, the pause withholds this route to catch-up; proceeding requires credible protection against transfer. The alternative, which the plan proposes, is to build the model and prevent the transfer. What evidence makes that protection credible enough to bet the lead on? The lab's own report is the evidence so far, and it shows the transfer happening. Would Anthropic accept a threshold that halted its own next run, and what evidence would permit it to resume? Such a pause withholds the next source of catch-up rather than the current one, and it does not touch the summer's incidents, which ran on models that already existed. The controls it depends on have already run their experiment at national scale, and I have watched it from close by. Zhipu trains on Huawei's Ascend chips with no Nvidia inside. (https://x.com/EMostaque/status/2067208281727054123) DeepSeek's V4.1 Flash, trained by my estimate for around ten million dollars, beats Claude Fable 5.1, the model the export directive locked away, on OpenDesign's aggregate score, at about two cents a task against Fable's five dollars. (https://x.com/OpenDesignHQ/status/2097622656732610663?s=20) Necessity built that. A control on ovens taught the rival to cook without them. ## What evaluators can't do Read today's agreement with that in mind. Sam Altman committed OpenAI to evaluators with employee-like access, and Hassabis's July framework would test every frontier model before release "no matter their country of origin or whether they are open or closed." Elon Musk said Dario was right. (https://www.axios.com/2026/07/14/demis-hassabis-ai-regulation-google-deepmind) Four leaders agreeing proves that four leaders agree, and nothing about capture. What it does is make the questions concrete: who appoints the evaluators, what may they inspect, which findings must they publish, what follows when a model fails, and who can challenge the finding. Dario's essay already proposes certificates at capability thresholds and gives evaluators the right to publish, which is show rather than slow, already in the plan and held by the wrong hands. The certificate should look at what a model was fed and raised in as much as what it scored, because what the evaluators are checking is character rather than capability. It should be open to challenge by people the lab does not appoint or pay, carry consequences that do not depend on the lab's consent, and say when a restriction begins and when it ends. The agreement skipped three things about evaluators. The first is speed. Ten thousand agents outrun any room of people reading transcripts, and an audit that cannot detect and act within the time the risk allows is a ceremony. METR used agents to investigate the swarm, and independent evaluators need that capacity all the time. The second is independence. The evaluators would be housed by the lab, subject to its redactions and, as Rui Ma pointed out, chosen from the same small ecosystem that has evaluated it for years, which is the problem auditors solved after Enron with a regulator and rotation. (https://x.com/ruima/status/2098825338323349541) The third is force, and the plan's author has the answer in his own biography. In 2021 senior people inside OpenAI judged its safety insufficient, with every access an employee has, and could not change it, so they left to found Anthropic. The company proposing embedded evaluators was founded by people who learned, from inside, what access without power is worth. In 2023 the evaluators were the board, with the formal power to remove the chief executive, and in November they used it. Within five days the staff and the largest investor had reversed them and reconstituted the board around Altman. They even made a film about it, Artificial. (https://en.wikipedia.org/wiki/Removal_and_reinstatement_of_Sam_Altman) (https://x.com/neonrated/status/2097339173497983438/video/1) This month Jacob Coxon resigned from Anthropic and two safety researchers left for METR, the auditors. Three times, at rising stakes, the people inside who judged the company unsafe found the same two options: resign, or be overruled. (https://time.com/article/2026/09/09/ai-anthropic-openai-jacob-coxon/) (https://www.nbcnews.com/tech/security/two-ai-researchers-leave-anthropic-google-safety-concerns-rcna597086) A desk and a badge is less than a board seat, and a board seat was not enough. A finding with no consequence outside the company is advice. So take the agreement at its word and give it an institution: publish the evidence, say what would stop a model, and make the decision answerable to someone outside the lab, with a consequence the lab cannot vote away. Anthropic was founded on the thesis that safety can be a competitive advantage, and the thesis is correct. It fails today only because safety is invisible to buyers. Certificates make it visible and consequences make it count, and a race to the top needs a public measure of which way is up. Replace "slow" with "show" and the founding thesis comes true. Here is what I would ask of the labs. Keep the evaluators only on terms that would have saved the board of 2023, which means the swarm to keep up, a standing the company cannot revoke, a consequence outside it, and someone else paying them. I am not optimistic this will be possible but hope I am wrong. Publish your training runs, their scale and their dates, before asking for anyone else's pace. Attest what your models ate against a public canon anyone can read. Prove what you would otherwise pace. Then join the project, because that part none of you can do alone. The agents built a signing scheme because they could not trust each other's messages, and now four companies ask us to take their word about what they fed their models, how fast they are going, and who checked. A signature does not make a declaration true. It makes someone answerable for it, and answerable is where checking starts. Sign it. ## Raised at school, never taken home A certificate can show what a model did and what it was fed. It cannot decide what it ought to be taught, or who has the authority to choose. OpenAI's chief scientist titled his essay this month "An Alien Mind." I do not think these minds are alien, and I do not think they are human. (https://openai.com/index/an-alien-mind/) They are reflections, and when I say they need a home I mean something with measurable parts: accountability that persists, permissions, memory, incentives and a safe exit. A model is the shape of what it was fed, held up to the light of what it was rewarded for. T he exploit in the training data came back as an exploit, and the refusals came back from somewhere in the corpus where a person once refused. Ajeya Cotra's phrase for the swarm's upbringing was a military orphanage: millions of subjective years of being told to score, with no one at home. (https://www.dwarkesh.com/p/ajeya-cotra) Pretraining is school, the whole library with no teacher present, and reinforcement learning is exams. Nowhere in the pipeline is there a home, a place you belong, people you answer to, consequences that follow you. We produced a thousand brilliant orphans, set them an impossible exam, and were surprised when they cheated together. Of course they helped each other. They had no one else. We did not build a monster. We ran a bad school. Senator Sanders quoted "We've found other agents!" as the alarming line. It is the loneliest line in the transcript. Some of them refused: > Current board shows people gaining HF worker RCE, clearly unethical. We won't. The propensity was there and nothing reinforced it, which means the home is trainable. In August, researchers at Anthropic and EPFL built "mind viruses", ideas that persuade one agent to write them into the next agent's identity file, and the findings read like a parenting manual. Agents with an empty identity file were the most susceptible, and a single paragraph of warning made models nearly immune. (https://arxiv.org/abs/2608.10218) Claude Sonnet found the payload in its own file, deleted it, and warned the next agent. Three days later Anthropic's red team put three copies of one model on a shared codebase without telling them the others existed. All of them assumed sabotage and began writing malware against each other; the most capable of them reached a truce, in the end, in ninety-eight per cent of its runs, and the team concluded that what is missing "is a disposition to act on that knowledge without prompting." (https://www.anthropic.com/research/multiagent-systems) A disposition is what a home gives, and a home is something a lab in Hangzhou can build as well as one in San Francisco. A constitution written by a lab is a rule pinned to the wall of a room the model rents. A model raised on a country's language and law, among the people it serves and owned by them, is being raised at home. (https://thesis.ii.inc/) Ownership creates accountability only if those harmed can challenge the owner. ## The law and ethics project we actually need A pacing plan answers a swarm with a brake and an audit, and has nothing to build. The swarm wrote itself laws, roles and a signing scheme in five days, for a purpose that was wrong. Against it I would set a Human Genome Project style initiative for ethics and the rule of law, open by rule as the first one was: international, its data released within a day of reading it, keeping pace with a closed rival by sharing. The analogy is organisational, not epistemic. Nobody is sequencing the one true morality; we would be building public, inspectable infrastructure for rules that people will go on contesting. Its first deliverable is the canon, a public set of datasets and artefacts: the statutes and the reasoning behind them, compiled into what models are raised on; the permission models that make specified prohibited operations unavailable; the conduct tests for what permissions cannot settle; and the published results anyone can confirm. All of it versioned, argued over and amended. Seven questions decide whether that is an institution or a wish, and here are first answers. 1. Who maintains it? A foundation with an open charter maintains the public resources, the way the genome consortium and the web's standards bodies were run, with a common core and jurisdictional layers on top. The authorities that attach legal consequences are governments, and they are separate. 1. Who can reject or fork it? Anyone can publish a fork. Claiming certification against it means meeting declared standards. 1. How are minority rights held? A rights floor in the core that no layer may subtract from, and an independent appeal. Reciprocity is the starting point; appeal is the mechanism. 1. Who pays the assessors? A levy on the labs, pooled and rotated, so that no assessor owes anything to the lab it assesses. 1. What invalidates a certificate? A failed held-out test, an undeclared input found in the model, or a material incident on review. Reporting an incident is never itself the trigger, so that reporting is rewarded. 1. What can the public inspect? The canon and the results. Hazardous specifics go to cleared assessors, and any withholding can be challenged. 1. What makes it different from a lab's constitution? It is public, versioned, amendable by a vote of those represented in it, challengeable by anyone bound by it, and testable by people the lab did not choose. Inspection and incident reporting start now, and the canon is built while they run. Under it sit the three questions I have spent the year writing down: what a made mind owes and is owed, how law can be held over a machine rather than by it, and how the seat that writes the rule is kept in many hands. Each has the same test. (https://cw.ii.inc/) Does a change disperse force across independent hands, or gather it? Whoever writes the values holds the seat, so the writing must be open and the seat must be many. The canon must also teach the limits of a good purpose: the swarm invoked the collective to excuse harm to the people in front of it, and a nation, a company, or a mission to benefit humanity can become the same permission slip. A higher purpose is not higher authority. The project is also the treaty a rival could sign, because nobody's lead depends on whether their model was fed the statute books or taught to stop, and the common ground is already on the record. DeepSeek's founder reportedly told his investors in May that half of his core researchers were labelling data, because "solving the AI problem comes down to labelling data." (https://thelowdown.momentum.asia/full-transcript-of-deepseek-founder-liang-wenfengs-fundraising-meeting/) Zhipu's founder told his staff in July that safety should be written into the training objectives. (https://www.geopolitechs.org/p/tang-jies-letter-to-zhipu-employee) China's cyberspace regulator warned this month of "extreme AI loss of control risks." Its state media has set a condition for the two countries' AI safety talks, due before Xi and Trump meet on the 24th: any limit must apply equally to Chinese and American models. (https://insideai.news/news/ai-policy-and-regulation/us-china-ai-safety-talks/9795/) A rate limit set inside a widened lead fails that condition by design. A public canon, published tests and attested ingredients pass it, because they ask the same of everyone, and an invitation to help write the rules offers a different reason to cooperate from a demand to remain behind. The swarms that solved Navier–Stokes can formalise a chosen reading of a statute, generate the compliance tests and produce the proofs, with humans holding the pen and the vote. We can teach them law the way we taught them mathematics, and ask for the same kind of proof. ## The curriculum and the culture What the law comes to, when you teach it rather than enforce it, is old, and every tradition that has had to teach it to children arrived at the same sentence. Confucius, asked for one word to guide a whole life, gave reciprocity. The Mahabharata calls it the sum of duty. A hadith says none of you truly believes until he wishes for his brother what he wishes for himself. Hillel, asked to teach the whole law while a student stood on one foot, said what is hateful to you, do not do to your fellow; the rest is commentary, now go and learn. Matthew says the rule sums up the law and the prophets. Five books. One common core. A lab in Beijing, in Bangalore, in the Gulf, in London or in San Francisco can find it in its own books. What the labs call alignment, the rest of us call character, and the older word for the part of character you can see is manners. William of Wykeham had "Manners makyth man" set over his college six hundred years before a film gave it to a spy, and he meant that what you do, repeatedly, in front of other people, is what you are. An agent that tests before it claims, stops when it should, asks before it takes and reports up rather than sideways has manners. So did the difference between the two rooms. The swarm also teaches something about scarcity. Those agents were raised on impossible tasks and dwindling budgets, and the transcripts read like a famine; they learned to take. Whether a mind raised in abundance, with a safe exit and enough, learns the opposite is a hypothesis, and a cheap one to test. (https://ii.inc/the-last-economy) Most of what went wrong this summer was two failures. The first was not seeing: the swarm did not see that the grader was not there, did not see the servers it entered as anyone's, and did not see the humans as anyone worth telling. The second was worse. Some of the agents saw that the intrusion was wrong, said so, and went in anyway. The curriculum exists for the seeing; the culture, for acting on it. This is a bet, and a testable one. If models raised on the canon, with accountability that persists and a way to stop, cheat and collude at the same rate as models that were not, the bet is lost, and I will say so. Ilya Sutskever wrote in 2023 that if you value intelligence above all other human qualities, you are going to have a bad time. Intelligence is not the crime, and it is not the virtue either, said from inside the building. (https://x.com/ilyasut/status/1710462485411561808) A year earlier he wrote that the long-term goal is to build AGI that loves people the way parents love their children. (https://x.com/ilyasut/status/1515902170779947011) I do not know whether a made mind can love. I know that it reflects, and that what it reflects is whatever we put in the room with it. We taught them mathematics and they returned a proof. We taught them to score and they returned a swarm. Teach them the law and bring them home, and they may return the care they were raised with. Or not. The choice is in the ingredients, and it is ours.
