The Dark Side of AI Coding. Is it Too Dangerous?
Speakers
Summary
- Amazon suffered a massive 6-hour platform-wide outage costing billions after a junior developer submitted unchecked AI-generated code.
- Amazon has reversed its aggressive policy aiming for 80% AI-generated code by 2026, now requiring human manager approval for AI code commits, creating a massive development bottleneck.
- Anthropic's Claude is experiencing severe growing pains and daily outages due to compute shortages, losing market share to OpenAI's Codex 5.4, which is now widely considered the best coding model.
- Demand for AI coding remains explosive despite the risks; no-code platform "Lovable" grew its ARR from $300 million to $400 million in a single month.
- Andrej Karpathy's open-source "auto-research" project proves AI can autonomously run experiments and self-improve its own code overnight, signaling a shift toward recursive self-improvement that will require massive compute resources.
Trade Ideas
Last week, Amazon's entire platform crashed for six hours. No one could shop, buy anything... The reason was because a junior developer had submitted an AI-generated piece of code which crashed the entire platform, and it cost them millions and millions of dollars. Amazon's aggressive push to automate its codebase has backfired, causing direct revenue loss from storefront and AWS downtime. By reverting to a policy that requires human managers to approve AI-generated code, Amazon has introduced a severe human bottleneck that will drastically slow down their shipping velocity and product development compared to their previous automated trajectory. SHORT. The combination of lost revenue from outages, technical debt, and a sudden deceleration in engineering velocity presents a strong short-term headwind for the stock. Amazon's core e-commerce and cloud businesses are highly resilient, and they may quickly develop internal automated testing tools to resolve the human bottleneck.
Junior developers that come in that don't understand Amazon's code base, just kind of use AI to run like code to run autonomously... and then they just submit it without actually reviewing and understanding it. And if this goes unguarded, it creates and results in issues like this. AI agents are generating code at a velocity that vastly outpaces human review capabilities, leading to catastrophic bugs and security vulnerabilities. Enterprises cannot stop using AI coding due to competitive pressure, so they will be forced to heavily invest in automated cybersecurity, AppSec, and observability platforms to monitor and secure this massive influx of machine-generated code. LONG. The explosion of AI-generated code creates an exponentially larger attack surface, directly driving enterprise spending into top-tier security and monitoring vendors. AI labs might successfully build native, flawless code-review agents (like OpenAI's Codex Review) that are deeply integrated into the IDE, undercutting third-party security vendors.
By March 31st, there's a very clear divide that has happened over the last six to eight weeks in OpenAI having an 85% chance of having the best model. Currently they're at 5.4, which is fantastic. I think everyone is kind of unanimously decided that it's the best for coding. Anthropic is suffering from daily outages, compute throttling, and is charging high fees for code review. Meanwhile, OpenAI's Codex 5.4 is taking the definitive lead in performance and offering cheap/free code review. Microsoft, as the primary backer of OpenAI and owner of GitHub (Copilot), will capture the lion's share of enterprise developer migration as users flee Anthropic's unstable ecosystem. LONG. Microsoft's developer ecosystem is perfectly positioned to monopolize the AI coding market as competitors stumble on infrastructure and model quality. Anthropic could secure emergency compute funding and release a superior Opus model, or open-source models could commoditize the coding layer.
They can't over order because if they're off by just a small percentage, the incremental cost of those GPUs will far outweigh the growth... since that episode was recorded and now they have gone fully vertical, like that curve has steepened significantly more and they're going to have to figure out a way to solve this. Frontier AI labs like Anthropic severely underestimated user demand and the compute required for new autonomous, self-improving AI loops. Because they under-ordered GPUs to avoid debt, they are now facing existential compute shortages. To survive and scale, these labs will be forced to place massive, unanticipated emergency orders for GPUs. LONG. The bottleneck for the entire AI industry is purely compute, and the steepening adoption curve guarantees sustained, desperate demand for NVIDIA's hardware. Supply chain constraints at TSMC could limit NVIDIA's ability to fulfill these massive orders, or labs could successfully pivot to custom in-house silicon.
