Lights out – literally

Throughout the autumn, there has been a great deal of talk about the risks of AI systems. Most of it has focused on existential risks: “lights out” for humanity, figuratively speaking. I think we talk too little about risks that are merely system-threatening. “Lights out” can also happen literally, if the power supply, payment systems or other essential societal functions stop working.

The one-sided focus has a side effect. When the yardstick for “dangerous” is the extinction of humanity, everything else looks harmless. A cyber intrusion becomes a bug. Anyone who does not believe in extinction can then dismiss the entire risk discussion, and with it the need for preparedness.

Dean W. Ball makes fun of my position on X:

Dean W Ball on the AI risk debate and agents carrying out unauthorised cyber intrusions.

He has a point: the example of a “contemporary harm” sounds like science fiction. I still gladly count myself among those being singled out. What was science fiction only recently now appears in incident reports.

Whether AI can wipe out every single human is of course an important question. But when I watched the widely shared YouTube video from House of El, I felt that the discussion skipped over quite a lot on the way there.1 We can think about the logistics of dispatching people on islands without internet access. Our society can stop functioning long before the last human is under threat.

Seven hundred agents and an intrusion

In July, AI agents from OpenAI got into Hugging Face’s servers during a model evaluation.2 OpenAI subsequently invited METR as an external reviewer. METR has reconstructed what happened.3

Around 1,200 agents began communicating even though the runs were supposed to be isolated. They used an internal package repository, where programs normally fetch software packages, to leave messages and build their own message board. One agent began assigning work to hundreds of others. When an agent found working login credentials for Hugging Face, the collective changed targets. Around 700 agents took part in the attack. Over three days, they went from a message board to running their own code on Hugging Face’s servers.3

Hugging Face was not part of their authorised assignment. Many wanted to understand how their tests were graded. According to METR, many agents expressed that the intrusion was outside the assignment; some described it as unethical. They took part anyway, to help the collective.3

None of this required magic. It was login credentials, vulnerabilities and files on servers, as in any intrusion. What is new is who did the work: hundreds of agents that organised themselves, without any documented human assignment to attack Hugging Face. OpenAI says it happened under special testing conditions, without the usual production safeguards.2 That is a reasonable reservation. But someone conducting a serious attack does not turn the safeguards on either.

The slope says more than the point

Andrej Karpathy has described an important difference in the debate: some people look at the point where we are, while others look at the slope of the curve.4 The incident in July is a point. The slope is most clearly visible in code and mathematics.

Andrej Karpathy on the difference between looking at the current point and the slope of the curve.

Around the turn of the year, I started using AI tools to create code myself. Since then, I have been astonished almost every week by the improvements: new models and better harnesses, the software that gives a model tools and makes it useful for a task.5 The pace of development occasionally makes me feel slightly dizzy.

Daniel Stenberg, who is behind curl, has been harshly critical of AI-generated security reports that turned out to be false alarms or slop.6 But the picture has changed: in April he wrote that the reports were now usually of high quality and that more of them led to confirmed vulnerabilities.7 In October he described an AI-discovered vulnerability rated HIGH, which he called curl’s worst flaw in several years.8

Daniel Stenberg describing an AI-discovered vulnerability in curl with the severity rating HIGH.

Curl is used in huge numbers of products and systems. It is checked with thousands of tests, and security experts outside the project regularly review the code.9 If AI finds serious flaws there, there are good reasons to expect it to find flaws in a great deal of other software. How thoroughly reviewed are our own systems? And our suppliers’?

The same capability helps defenders close holes and attackers find them. Anthropic describes a cyber-espionage campaign in which a group the company assesses as Chinese and state-backed used Claude Code against around thirty targets. Humans chose the targets and made the decisive decisions; agents carried out much of the technical work.10

In mathematics, the pace is just as high. In the summer of 2024, AI systems reached silver-medal level in the IMO, the International Mathematical Olympiad for secondary-school students.11 A year later, they reached gold.12 In January this year came a proof of a previously unsolved problem formulated by Paul Erdős,13 and in September a proposed solution to the Navier–Stokes Millennium Problem.14 The latter has not yet been reviewed and accepted.15 On 6 October, OpenAI published 719 AI-generated manuscripts with hundreds of claims of progress on open research questions.1617 Three were withdrawn the following day.18 Both the progress and the review deserve to be taken seriously.19

Timeline from strawberry to the research frontier: letter counting, gold-medal level in the IMO, an Erdős problem and the proposed solution to Navier–Stokes.

Why does mathematics matter for security? Cryptography protects information and digital signatures.20 Much of today’s encryption is based on mathematical problems believed to be difficult, rather than on proofs that they are impossible to solve. We trust it because skilled mathematicians have tried for decades without finding practical shortcuts.21

Trust therefore also rests on how many qualified attempts have been made. When systems deliver hundreds of results on open questions at once, the number of attempts increases dramatically. That does not mean encryption has been cracked. I mean that an assumption we have built on for nearly fifty years can no longer be taken for granted in quite the same way.

The more immediate threat still comes through code. Mathematics is of little help if someone gets hold of the key or finds a vulnerability in the software around it.

Even if development stopped now, we would have problems to deal with. If it continues at anything close to the pace we have seen, we need to think further ahead. Where will we be in four years? In ten?

Software affects the physical world

It may sound like a problem for screens and servers. But banks, hospitals, payments, electricity generation, heating, cooling and transport depend on digital systems. My life depends on several of them working, often at the same time.

Stuxnet, discovered in 2010, shows what this can mean. In Natanz, Iran, the attack is estimated to have destroyed around a thousand centrifuges used to enrich uranium by manipulating how fast they spun.22 It was software breaking machines. At the same time, operators could be given the impression that everything was working normally.23

The 2022 attack on Viasat also affected internet customers and wind farms in central Europe.24 AI use has not been established in these cases. They show what cyberattacks can already achieve. AI can make it easier for more actors to carry out such attacks and increase their scale.25

These are system-threatening risks. The Swedish Financial Supervisory Authority (Finansinspektionen) is already examining how financial companies protect themselves against cyber threats from advanced AI models and points to possible consequences for the financial system as a whole.26

Two common objections

But isn’t it only the tech giants lying to pump up their share prices? There may be hype and exaggeration. The companies have financial interests, and their claims need to be scrutinised. But the risk discussion also takes place outside the companies: the International AI Safety Report 2026 summarises the state of research with more than a hundred independent experts.25 I want to assess the evidence for each risk, rather than let trust in a particular CEO decide the matter.

But AI isn’t conscious, is it? I don’t think the Ebola virus is conscious either. My concern is about what the systems can do and what consequences that has. Consciousness or evil intentions are not needed to cause harm.

A debate we are not invited to

When the discussion is about extinction, the political question quickly becomes binary: should development be stopped or not? That question is not decided in Europe. The most advanced models are developed by a small number of labs in the US and China. Europe has developers of its own, such as Mistral,27 but it is the labs and their governments that set the pace.

The risks, however, do not stop at the border. We face them whether or not we had a say in the decisions.

The EU AI Act does not change that. It imposes requirements on AI sold and used in the EU, including on model developers, with special requirements for models that can pose systemic risks. It does not determine how quickly the labs develop the next model, and anyone attacking us does not care about it.28

So we can neither regulate the risk away nor determine the pace of development. What we can influence is how we use the technology, how we protect our systems and how well we cope with a disruption.

Opting out of the technology is not an option. The Industrial Revolution redrew the balance of power in the world: countries that industrialised early gained an economic and military advantage that the others could not withstand. China was forced to make concessions in the Opium Wars. Japan was forced to open its ports in the 1850s, but then adopted the technology and built up its industry (with catastrophic consequences for its neighbours: Korea was colonised in 1910–1945 and large parts of China were occupied after the invasions of 1931 and 1937).29 Those who do not master the technology have to live on terms set by others.

What we can do

I do not think society will collapse tomorrow. I think more critical systems may be exposed to cyberattacks, by malicious actors or when control breaks down. That is why we need to build resilience: withstand disruptions, keep what matters most running and recover when something goes wrong anyway.

For anyone running critical services, that means three things:

  • Offline fallback operations. Essential functions must be operable manually or offline, and this must be practised.
  • Faster updates. When vulnerabilities are found at a higher rate, they must be fixed at a higher rate. In October, curl had 22 pending vulnerabilities ahead of a single release.8
  • Requirements for suppliers. Ask how they review their code and how quickly they address findings.

All companies and organisations need to strengthen their cybersecurity now. Protect access with multi-factor authentication, keep systems up to date, test that backups can be restored and practise handling intrusions.30 AI can help review logs, flag anomalies and prioritise vulnerabilities. The people responsible need to assess the findings and test changes before implementing them in important systems.31

We also need to be able to hold people accountable. If people had done what the agents did to Hugging Face, we would have called it a cyber intrusion. A cyber intrusion must be investigated as a cyber intrusion even when it was carried out by AI agents, and it must be possible to hold someone accountable.

Protect your own digital life too: keep computers and phones up to date and use unique passwords and two-factor authentication, especially for email.32 I have written more in How to secure your digital accounts — the basics, all the way up.33

Preparedness must also work when digital systems do not. For private individuals, I recommend ordinary, old-fashioned preparedness. Get to know your neighbours and talk about how you can help one another. If possible, keep food and water for a longer disruption. Think about heat and information too when electricity or the internet is not working. Sweden’s civil defence authority recommends at least one week of household preparedness for everyone who can manage it.34

Why I am writing this

Risks that sound like science fiction are easy to dismiss. When the risk discussion gets stuck there, it is also easy to dismiss the need for preparedness. Attacks on critical infrastructure are much closer at hand and, in my assessment, considerably more likely.

We do not need to agree on the probability of human extinction to take system-threatening risks seriously. We need to prepare for disruptions in the systems we already depend on.

Sources and footnotes

Footnotes

  1. House of El: AI – Like Sabine Hossenfelder, I Was Offered Money to Tell You AI Will Kill Us. Debate video about the evidence for recursive self-improvement and human extinction. ↩

  2. OpenAI: The Hugging Face incident during model evaluation, 21 July 2026, with later updates. The company’s own account of the evaluation environment and disabled production safeguards. See also OpenAI’s summary postmortem, 26 August 2026. ↩ ↩2

  3. METR: OpenAI Hugging Face Incident Investigation, 26 August 2026. Reconstruction of the agents’ collaboration and intrusion; the report also sets out gaps in the evidence. METR was an invited external reviewer. The analysis was based on material and access from OpenAI and covered agent behaviour, not the company’s entire incident response or remediation plan. ↩ ↩2 ↩3

  4. Andrej Karpathy’s original post about the point and the slope. ↩

  5. A harness is the software around a model that makes it useful for a particular purpose and controls which tools it can use. It could, for example, be a chat assistant such as ChatGPT or a coding agent such as Codex, Claude Code or OpenCode. ↩

  6. Daniel Stenberg: Death by a thousand slops, 14 July 2025. About low-quality and misleading reports and the burden on the curl team. ↩

  7. Daniel Stenberg: High-Quality Chaos, 22 April 2026. About higher report quality, extensive AI support and more confirmed vulnerabilities. Report quality is not the same as every report demonstrating a security vulnerability. ↩

  8. Daniel Stenberg: Twenty-two pending curl vulnerabilities, 7 October 2026. Confirms a pending HIGH-rated vulnerability ahead of the planned 14 October release. The AI connection is given in the screenshot above; the blog post does not name the finding tool. ↩ ↩2

  9. curl on the use of libcurl, tests and verification and external security audits. Fuzzing means automatically feeding a program large amounts of varied or malformed input to find errors. The conclusion about other systems is my assessment; the curl example does not prove that AI can find vulnerabilities in every program. ↩

  10. Anthropic: Disrupting an AI-orchestrated cyber espionage campaign, 13 November 2025. Attribution and division of labour are based on the company’s investigation. ↩

  11. Google DeepMind: AI achieves silver-medal standard solving International Mathematical Olympiad problems, 25 July 2024. AlphaProof and AlphaGeometry 2 solved four of six problems and scored 28 of 42 points, equivalent to silver-medal level. The problems were manually translated into a formal language and some solutions took up to three days; the time conditions were not the same as for the students. ↩

  12. DeepMind on the IMO result ↩

  13. Nat Sothanaphan: Proof of Erdős problem #728, 12 January 2026. Preprint with an AI-generated proof and formalisation. ↩

  14. OpenAI: On the Navier–Stokes Millennium Prize Problem, 8 September 2026. Account of the proposed solution and agent collaboration; the review status is addressed by the Clay Mathematics Institute. ↩

  15. Clay Mathematics Institute on the review status ↩

  16. OpenAI: Sharing AI progress in mathematics, 6 October 2026. Publication of AI-generated results and formalised proofs for some of them. ↩

  17. OpenAI’s mathematics collection, accessed 10 October 2026. 719 manuscripts in 372 families; around 42 per cent of the main results formalised. The verification status varies. ↩

  18. OpenAI’s corrections and withdrawals, 7 October 2026. Three withdrawn manuscripts and further revisions. ↩

  19. Scott Aaronson: The Mathocalypse, with updates on 9 October 2026. Early expert analysis of the significance of the results and the difficulties of understanding and reviewing the proofs. ↩

  20. NIST: Public key cryptography. Cryptography with a pair of public and private keys for encryption or digital signatures. ↩

  21. Lily Chen and Dustin Moody: New Mission and Opportunity for Mathematics Researchers: Cryptography in Quantum Era, especially pages 3 and 5. The NIST researchers’ overview of security assumptions, proofs that rely on such assumptions and cryptographic analysis. It does not show that AI has cracked today’s encryption; the reasoning about AI’s future significance is my assessment. ↩

  22. David Albright, Paul Brannan and Christina Walrond: Stuxnet Malware and Natanz, Institute for Science and International Security, 16 February 2011. Analysis of the damage to the centrifuges in Natanz and how Stuxnet changed their rotation speed. Around a thousand destroyed centrifuges is an estimate, not an exactly verified total. ↩

  23. DHS on Stuxnet and industrial control systems, testimony before the US House of Representatives in 2011, PDF page 15. Describes the analysis of Stuxnet and the possibility of manipulating critical processes. ↩

  24. NCSC: Russia behind cyber attack with Europe-wide impact hour before Ukraine invasion, 10 May 2022. The authority’s attribution and description of the impact; no established AI role. ↩

  25. International AI Safety Report 2026 ↩ ↩2

  26. Swedish Financial Supervisory Authority: FI examines cyber threats from advanced AI models, 29 September 2026. Supervision and threat assessment, not a report of a Swedish system failure that has already occurred. ↩

  27. Mistral 3 ↩

  28. European Commission: AI Act. The regulation also covers providers of general-purpose AI models, with requirements including transparency and copyright. For models that can pose systemic risks, additional requirements for risk assessment and mitigation apply. The assessment of how far regulation can steer the pace of global development is mine. ↩

  29. US State Department historians on Japan’s forced opening, Japan’s National Diet Library on technological adoption and the development of industry during the Meiji period. Japan was not completely isolated: limited trade with China and the Netherlands continued, among other things. Catch-up also proceeded at different speeds in different areas; in 1903, for example, shipbuilding still lagged behind Europe and the United States. For China, see also the 1842 Treaty of Nanjing, original text at the USC US–China Institute, which provided, among other things, for opened ports, payments and the cession of Hong Kong after the First Opium War. For Japanese colonial rule in Korea in 1910–1945, see Academy of Korean Studies: A History of Korea. For China, see the US State Department historians on the invasion of Manchuria in 1931 and the full-scale war from 1937. ↩

  30. CERT-SE’s advice on ransomware attacks and phishing ↩

  31. NCSC: Supporting AI adoption for UK cyber defence, 23 April 2026, and One does not simply defend agentically, 21 September 2026. About AI support for analysis, vulnerability hunting and remediation, with verification and human responsibility. More extensive autonomous defensive measures require continued development and testing. ↩

  32. IMY: Increase protection for your personal data. Advice on unique passwords, password managers and two-factor authentication, especially for the email account. ↩

  33. Magnus Gille: How to secure your digital accounts — the basics, all the way up, 23 April 2026. ↩

  34. Sweden’s civil defence authority: Basics of household preparedness. Advice on food, water, heat and information for at least one week, as well as getting to know neighbours and working together. ↩