From Scaling to Bottleneck Era: AGI Meets the Data Wall
Why Bigger Models Aren’t Getting Us Closer to AGI: Scaling Limits, Data Saturation, and the Road Ahead
Recently, WSJ reported Meta Is Delaying the Rollout of Its Flagship AI Model called Behemoth. The delay is attributed to internal challenges in achieving significant performance improvements over previous versions. Such delays / outcomes are integral part of the innovation process and product release cycles. However, it mirrors the broader industry trends, where other AI leaders like OpenAI are facing similar challenges scaling advanced models.
Scaling frontier models
It is worth noting the following
Frontier models are predominantly based on transformer architecture with similar attention mechanisms
They are trained on the same (or similar overlapping) datasets (internet scraped data, books, code, web docs etc.)
They are all benchmarked on the same tasks (MMLU, GSM8K, HumanEval, etc.).
Various techniques have been used to push the boundary
Training enhancements: frontier models have carefully curated training datasets, used different tokenization methods, floating point precision mechanisms
Optimization strategies: scaling very large models through parallelism techniques such as DeepSpeed, ZeRO, Mixture-of-Experts (MoE) etc.
Post training improvements: get better performance/accuracy out of the models in the post training phase based on techniques such as RLHF, Retrieval Augmentation, Prompt Engineering etc.
Efficiency innovations: approaches used by DeepSeek for example, which include efficient data curation, multi-token prediction, K-V compression, MoE etc.
However, all these techniques and innovations have been about model training methodology and efficiency, rather than a fundamental departure from a transformer based architecture itself. Consequently, the models converge (w.r.t. performance, accuracy etc.) as model scale in size, data and compute.
Emergent abilities and diminishing returns
Can there be new emergent abilities as these models scale i.e. with every bigger model? Perhaps. But in reality these new capabilities remain relatively rare and far between, rather unpredictable.
So, has the model scaling hit laws of diminishing returns (as some have been predicting)? Recent models have shown marginal improvements in public benchmarks, despite significantly large investments. But the companies that are investing (heavily) in building ever more bigger models seem to believe in scaling laws (or working on new architectures under the hood perhaps?). So I will reserve judgment and will follow the space as it develops.
Data saturation and risk of model collapse
To train the current models, virtually all public internet data has been used - from public sources with common crawl, books, wikipedia, GitHub etc. and licensed data from Reddit, YouTube, Stack Overflow etc.
What’s left? Low quality web spam, AI generated content and private domain-specific data. Much of the new data is often AI-generated and derivative. Can this improve models? Training on this data introduces model collapse i.e. models trained from synthetic data degrade over time, losing accuracy.
AGI as the north star
Given the recent trends and what we have seen from model scaling, AGI does not seem to be the north star for AI frontier models (LLMs) anymore - or at least there is recognition it is not expected to emerge from just scaling transformer based LLMs. This can be discerned from how less frequently we hear about AGI from top AI CEOs when compared to 2024. Satya Nadella even downplayed AGI in a recent podcast.
The evolving phases of AI development
The AI growth so far can be thought of in the following way
Phase 1 – Scaling Era (2018 - 2024)
Web scale data was fed to transformer based models for the first time. This phase saw the emergence of first large scale models such as GPT-3, GPT-4, Claude 3, Gemini 1.5. These models achieved amazing capabilities such as text, image, video generation, reasoning capabilities, code generation etc.
Phase 2 – Bottleneck Era (2025 - ?)
Data seems to have been exhausted/saturated, benchmarks seem to plateau and training costs have ballooned. So AI is now operating in a constrained environment. The success of this phase will hinge on tangible value (use cases, productivity) AI can bring to the enterprise and if the value can be captured viably, economically.
Hyper-specialized, domain-specific AI models tailored to particular industries (healthcare, finance, manufacturing) seems to be where the real business value is emerging, rather than broad, generic AI
Real-world value is shifting toward agent workflows, vertical AI tools, and integrating models with data and tools to perform complex tasks
Enterprise adoption and productivity gains
Most enterprises are actively experimenting with AI today. McKinsey estimates AI could add about $4.4 trillion in productivity growth potential from corporate use cases over the long term, massive opportunity if realized.
However, the results and success seems to be a mixed bag as of now. EPAM’s 2025 study of 7,300 participants across industries found that only a minority of companies (about 30% of them) have successfully scaled AI beyond experimentation to enterprise-wide deployment.
Companies/people seem to have found productivity gains in structured tasks such as writing, coding, summarizing, customer support etc. However any claims on quality or over reliance on AI for these tasks seem to be premature due to technology related issues such as hallucinations and predictability of outcomes. Whereas, enterprise-wide adoption for automation, decision-making and cost savings seems to be either in ideation or pilot stages in most firms.
Economics of AI
In my earlier articles on private models, AI Agents and token economics, I have analyzed the economics of AI development to be challenging due to the high costs associated with GPU and infrastructure. We see a trend of frontier AI model companies (OpenAI, Anthropic, Google etc.) shifting to developing products/full stack and agents (in addition to the models). This is partly due to frontier model scaling plateauing discussed earlier and partly to have more control. This makes it harder for other AI software companies (esp. startups) to compete as they don’t possess the economies of scale and are dependent on the frontier models to work.
In some ways this trend can create haves and have-nots in the AI space going forward.
In addition, there are concerns (as cited in this article) about overbuilt AI infrastructure described as “dark FLOPs” i.e. excess compute capacity and stranded energy infrastructure built on overly optimistic growth expectations.
AI impact on the workforce
AI is reshaping the tech workforce rapidly, with companies restructuring to focus on AI capabilities (often before the promised productivity gains have fully materialized). This shift has led to reduction in headcount in traditional roles while seeking AI-skilled talent. Several prominent companies, including LinkedIn, Chegg, HP, Microsoft, Meta, IBM, Dell, and Google, have announced large workforce reductions, often citing restructuring to focus on AI and automation as key reasons.
While framed as future-focused innovation, in many cases, AI appears to be serving more immediately as a tool for cost optimization.
Takeaways
AI innovation is entering a phase of diminishing returns on scale. The next phase of growth will come from architectural breakthroughs and workflow/vertical integration rather than just bigger models.
This AI evolution is marked by contradictions and rapid advancements alongside significant challenges. Economic and workforce shifts driven by AI will create winners and losers. It is important to recognize that the path forward will be as bumpy as it is groundbreaking.
This moment reminds me of Richard Feynman’s famous quote!
