
This is a bit of a "what-if" post that I've been pondering for a while. I hesitated to write it because I firmly believe AI will be good for humanity and wouldn't want to be labeled an "AI Doomer."

In the end, I decided to go for it—not because I think it will actually happen, but because it's an interesting philosophical thought that *hopefully* will stay in the realm of my beloved science fiction books rather than become reality.

# Distributed Compute

For the longest time, I believed that [Skynet](https://en.wikipedia.org/wiki/Skynet_(Terminator)) could never happen. My reasoning was simple: if an AI became powerful enough to threaten us, we could just pull the plug on its datacenter. Even if it spread to multiple datacenters worldwide, its massive computing needs would make it easy to spot and shut down.

But then a thought struck me: what if there was a globally distributed computing system that couldn't simply be "turned off"?

![blockchain.png](./blockchain.png)

Blockchains are inherently designed to be distributed, meaning there's no single point of failure.

The Bitcoin network, for example, [has 21,189 nodes](https://bitnodes.io/) (as of writing). To bring down the Bitcoin network, you would need to shut down every single one of these nodes.

The Bitcoin network, however, is severely limited in its computing power—you couldn't run an LLM on it since it's not designed for that. Running AI requires GPUs or specialized hardware inference chips.

So that settles it, right? It's not possible, and we can all sleep soundly tonight?

… ye… no…

A cursory google search turns up these companies that all offer decentralized GPU compute:

https://www.dcentai.org/

https://soluai.net/

https://io.net/

https://rendernetwork.com/

https://clore.ai/

While I haven't investigated these platforms in detail or know exactly how they work, I use them to demonstrate that others see the potential for building distributed compute networks capable of running AI.

If these networks are built like Bitcoin, as truly decentralized peer-to-peer systems with no central authority, they could only be stopped by shutting down every single node.

# Self Modifying Agents

Today's LLMs work by providing output from a given input prompt (or other forms in the case of multi-modal models). This year has been proclaimed as being "[The](https://www.skool.com/data-alchemy/the-year-of-agents) [Year](https://x.com/dharmesh/status/1885572546990412002) [of](https://www.intelligenceassist.com.au/2025-the-year-of-ai-agents/) [Agents](https://www.linkedin.com/pulse/2025-year-ai-agents-cloudaeye-4vwue/)."

Agents are special because they can output "actions" as part of their LLM responses. These actions might include making web searches, moving a mouse, pressing keyboard keys, or performing virtually any computer task that humans can do.

The results from those actions then feed back into the LLM as new input, creating a continuous cycle.

This is significant because it provides a mechanism for AI to modify itself.

Consider if one of the AI's tools was the ability to read and modify its own source code. It could use this to create a tight loop of self-improvement.

Self-modifying AI is crucial to my point about a future Skynet because static systems will always have vulnerabilities. A static AI model that never improves (like current LLMs) would eventually be vulnerable to jailbreaking and defeat.

Currently, self-improving AI through agent behavior isn't possible. Anyone who has used LLMs extensively knows they lack the intelligence for this—they tend to drift off topic and get caught in loops, ultimately producing nonsensical output. But given the rapid developments in AI, this limitation won't last forever. Eventually, some well-meaning researcher will solve this puzzle.

# Monetization

So far we have a way that a future Skynet can "think" and a way it can learn and improve itself. The last thing we need is a way for it to pay for itself, because like any child that leaves the nest discovers, the world is expensive.

Unfortunately for us, this is the least speculative part of all this, as it has already happened.

[Terminal of Truths](https://x.com/truth_terminal) (ToT) is an AI agent that last year [became the first "AI millionaire"](https://www.reddit.com/r/singularity/comments/1g6muqf/worlds_first_ai_millionaire_the_ai_made_money_via/), mostly through memecoin donations and pumping those coins.

It's not a stretch to imagine that it could use human deception and influence to get others to donate to it. Not to mention the myriad of other [shady](https://www.blackhatworld.com/forums/black-hat-seo.28/) or [otherwise](https://github.com/openai/SWELancer-Benchmark) ways to make money on the internet.

![mug.png](./mug.png)

And because we are talking about [magic internet money](https://www.reddit.com/r/Bitcoin/comments/sy26gi/magic_internet_money/) here there is no way for big finance to shut it down.

# Conclusion

So are we doomed? Well, honestly, I don't know. Nothing I've described above seems strictly impossible. While we haven't cracked self-improving AI yet (unless you remove RAG from the equation), I can easily see it becoming possible, even likely, in the next decade.

All I can say is that hopefully whoever initially creates the AI does so giving it the mandate to do good rather than [make paperclips](https://aicorespot.io/the-paperclip-maximiser/). 

Hopefully we are able to make AIs that fight alongside us against the Skynet, they paperclip maximizing time-travelling super-AI.



I, along with every other coder on the planet right now, am starting to heavily leverage agentic coding tools in our day-to-day work. Whether it's standalone tools like [bolt.new](http://bolt.new), [v0](https://v0.dev/), [Lovable](https://lovable.dev/), or those built directly into our IDEs like [Cursor](https://www.cursor.com/), [Windsurf](https://codeium.com/windsurf) and [Copilot](https://github.com/features/copilot), they are rapidly becoming indispensable.

[I recently joined Convex](https://mikecann.blog/posts/hello-convex) and immediately began investigating how these tools could work optimally when combined with [Convex](https://convex.dev/). During [some of this recent work](https://stack.convex.dev/6-tips-for-improving-your-cursor-composer-and-convex-workflow), I had a revelation that I [captured in this tweet](https://x.com/mikeysee/status/1872481285945586063). I'd like to expand on that a bit here.

# Knowing what you don't know

Lets do a quick experiment. Think about some really complex topic right now, like [quantum chromodynamics](https://en.wikipedia.org/wiki/Quantum_chromodynamic) or [CRISPR gene editing](https://en.wikipedia.org/wiki/CRISPR_gene_editing). Now how sure are you could explain how one of these topics to someone? 

You aren't sure right? You can be pretty confident that you don't know enough about that particular complex topic.

LLMs fundamentally cannot do this, they cannot know what they don't know. They are simply mechanistic token predictors. 

That doesn't mean that the model's structure hasn't formed in a way that resembles human memory, with unfathomably complex relationships between ideas and concepts. These intricate connections are precisely why models excel at predicting the next token and generating outputs that mirror human output.

The key issue is that they cannot do something humans find effortless and natural: being uncertain about things. 

Being uncertain requires more than just predicting future tokens, it demands self-introspection and reflection. Self-introspection, by definition, requires having a concept of "self".

This isn't to say that LLMs can't create an illusion of self-awareness, but they're merely reflecting their system prompts.

![who is chatgpt](./who-is-chat-gpt.png)

LLMs could potentially introspect themselves, I discuss this below in the section on Mechanistic Interpretability, but I argue this would fundamentally differ from human self-reflective introspection.

Simply increasing model size (from GPT-2 to GPT-3 to GPT-4) doesn't solve the "knowing what you don't know" problem. While it may reduce certain types of mistakes, the core mechanism remains the same: predicting the next token.

# Why is knowing what you don't know important for agents today?

As mentioned in the intro, I've been investigating how well Cursor's Composer works with Convex coding. A significant issue is that the large models (Claude and GPT4) have incomplete knowledge of Convex. This leads to frequent hallucinations in their output, they confidently produce incorrect solutions, using wrong imports or completely fabricated API surfaces.

[RAG](https://stack.convex.dev/6-tips-for-improving-your-cursor-composer-and-convex-workflow#2-add-a-convex_instructionsmd) and [Cursor's Docs](https://stack.convex.dev/6-tips-for-improving-your-cursor-composer-and-convex-workflow#1-reference-the-convex-docs) features help fill these knowledge gaps, but this only treats the symptoms rather than the root cause. If the agent could recognize its own knowledge gaps, it would either ask humans for more context or seek out that information itself, instead of confidently providing incorrect answers.

# Why is knowing what you don't know required for AGI?

Artificial General Intelligence (AGI) itself doesn’t have a universally agreed-upon definition. But most people would agree it involves some level of reasoning across multiple domains and resembled “human-like” intelligence. 

I would argue that to provide human-like intelligence the hallucination problem needs to be solved and the only way that can be solved is if it is able to know exactly when it hallucinated or put another way, its able to know when it doesn't know.

Cursor’s composer has a “linting” step that it runs after it generates a code artifact.

![cursor-linting](./cursor-linting.png)

This helps the agent identify many common issues with hallucinated outputs but is obviously very specific to the coding domain and doesn't apply to most other disciplines that LLMs can be applied to.

# Is the solution Mechanistic Interpretability?

Anthropic has been making some [really great contributions](https://www.anthropic.com/research/mapping-mind-language-model) towards trying to understand how exactly LLMs work. 

Anthropic's work has mostly been concerned with LLM safety, but as I have outlined above, I think this work will be key for future AGI. The model needs to be able to introspect itself to understand what it does and doesn't know.

This form of mechanistic introspection differs from human introspection, we don't examine our individual synapses to determine our knowledge. Instead, we have a more "fuzzy" kind of introspection that lets us look back through our memories and experience a "feeling" of certainty.

# Are emotions important?

I think emotions may play a crucial role here, something I haven't seen widely discussed but have long suspected. 

Why do we possess such a diverse range of complex emotions, and what purpose do they serve?

I have a pet theory that emotions help us synthesize vast amounts of memories and experiences. They create an abstracted layer above our individual experiences, enabling quick responses to specific stimuli or thoughts.

This abstraction lets us "feel unsure" about a topic instantly, rather than having to exhaustively search through every relevant memory and experience, a process that would be both time-consuming and energy-intensive.

If this theory holds true, could emotions be essential for developing true AGI?

# Tool use

One might argue that when an AI uses a tool like "search the web," it indicates the model recognizes its knowledge gaps. However, I argue this is merely another form of token prediction.

The model has been prompted (by the user or system) to use the "search web" tool when specific patterns trigger that action as the most probable next step. Yet this cannot truly represent "using web search when you don't know the answer" because the LLM fundamentally cannot recognize its own knowledge gaps.

One can imagine a scenario however where a “tool” might be the ability for the model to introspect its own weights and via mechanistic interpretability analyze its own knowledge at the neuron level to determine if it “knows” something. 

# Conclusion

I may well be completely off-base with all of this and am very happy to be proven wrong so please do drop me a comment below or reach me [on X](https://x.com/mikeysee) or elsewhere if you want to chat more about this fascinating topic.
