

Metaverse
Large language models are getting bigger and better – Crypto News
That hunger for the new has only accelerated. In March Anthropic launched Claude 3, which bested the previous top models from OpenAI and Google on various leaderboards. On April 9th OpenAI reclaimed the crown (on some measures) by tweaking its model. On April 18th Meta released Llama 3, which early results suggest is the most capable open model to date. OpenAI is likely to make a splash sometime this year when it releases GPT-5, which may have capabilities beyond any current large language model (LLM). If the rumours are to be believed, the next generation of models will be even more remarkable—able to perform multi-step tasks, for instance, rather than merely responding to prompts, or analysing complex questions carefully instead of blurting out the first algorithmically available answer.
For those who believe that this is the usual tech hype, consider this: investors are deadly serious about backing the next generation of models. GPT-5 and other next-gen models are expected to cost billions of dollars to train. OpenAI is also reportedly partnering with Microsoft, a tech giant, to build a new $100bn data centre. Based on the numbers alone, it seems as though the future will hold limitless exponential growth. This chimes with a view shared by many AI researchers called the “scaling hypothesis”, namely that the architecture of current LLMs is on the path to unlocking phenomenal progress. All that is needed to exceed human abilities, according to the hypothesis, is more data and more powerful computer chips.
Look closer at the technical frontier, however, and some daunting hurdles become evident.
Beauty’s not enough
Data may well present the most immediate bottleneck. Epoch AI, a research outfit, estimates the well of high-quality textual data on the public internet will run dry by 2026. This has left researchers scrambling for ideas. Some labs are turning to the private web, buying data from brokers and news websites. Others are turning to the internet’s vast quantities of audio and visual data, which could be used to train ever-bigger models for decades. Video can be particularly useful in teaching AI models about the physics of the world around them. If a model can observe a ball flying through the air, it might more easily work out the mathematical equation that describes the projectile’s motion. Leading models like GPT-4 and Gemini are now “multimodal”, capable of dealing with various types of data.
When data can no longer be found, it can be made. Companies like Scale AI and Surge AI have built large networks of people to generate and annotate data, including PhD researchers solving problems in maths or biology. One executive at a leading AI startup estimates this is costing AI labs hundreds of millions of dollars per year. A cheaper approach involves generating “synthetic data” in which one LLM makes billions of pages of text to train a second model. Though that method can run into trouble: models trained like this can lose past knowledge and generate uncreative responses. A more fruitful way to train AI models on synthetic data is to have them learn through collaboration or competition. Researchers call this “self-play”. In 2017 Google DeepMind, the search giant’s AI lab, developed a model called AlphaGo that, after training against itself, beat the human world champion in the game of Go. Google and other firms now use similar techniques on their latest LLMs.
Extending ideas like self-play to new domains is hot topic of research. But most real-world problems—from running a business to being a good doctor—are more complex than a game, without clear-cut winning moves. This is why, for such complex domains, data to train models is still needed from people who can differentiate between good and bad quality responses. This in turn slows things down.
More silicon, but make it fashion
Better hardware is another route to more powerful models. Graphics-processing units (GPUs), originally designed for video-gaming, have become the go-to chip for most AI programmers thanks to their ability to run intensive calculations in parallel. One way to unlock new capabilities may lie in using chips designed specifically for AI models. Cerebras, a chipmaker based in Silicon Valley, released a product in March containing 50 times as many transistors as the largest GPU. Model-building is usually hampered by data needing to be continuously loaded on and off the GPUs as the model is trained. Cerebras’s giant chip, by contrast, has memory built in.
New models that can take advantage of these advances will be more reliable and better at handling tricky requests from users. One way this may happen is through larger “context windows”, the amount of text, image or video that a user can feed into a model when making requests. Enlarging context windows to allow users to upload additional relevant information also seems to be an effective way of curbing hallucination, the tendency of AI models to confidently answer questions with made-up information.
But while some model-makers race for more resources, others see signs that the scaling hypothesis is running into trouble. Physical constraints—insufficient memory, say, or rising energy costs—place practical limitations on bigger model designs. More worrying, it is not clear that expanding context windows will be enough for continued progress. Yann LeCun, a star AI boffin now at Meta, is one of many who believe the limitations in the current AI models cannot be fixed with more of the same.
Some scientists are therefore turning to a long-standing source of inspiration in the field of AI—the human brain. The average adult can reason and plan far better than the best LLMs, despite using less power and much less data. “AI needs better learning algorithms, and we know they’re possible because your brain has them,” says Pedro Domingos, a computer scientist at the University of Washington.
One problem, he says, is the algorithm by which LLMs learn, called backpropagation. All LLMs are neural networks arranged in layers, which receive inputs and transform them to predict outputs. When the LLM is in its learning phase, it compares its predictions against the version of reality available in its training data. If these diverge, the algorithm makes small tweaks to each layer of the network to improve future predictions. That makes it computationally intensive and incremental.
The neural networks in today’s LLMs are also inefficiently structured. Since 2017 most AI models have used a type of neural-network architecture known as a transformer (the “T” in GPT), which allowed them to establish relationships between bits of data that are far apart within a data set. Previous approaches struggled to make such long-range connections. If a transformer-based model were asked to write the lyrics to a song, for example, it could, in its coda, riff on lines from many verses earlier, whereas a more primitive model would have forgotten all about the start by the time it had got to the end of the song. Transformers can also be run on many processors at once, significantly reducing the time it takes to train them.
Albert Gu, a computer scientist at Carnegie Mellon University, nevertheless thinks the transformers’ time may soon be up. Scaling up their context windows is highly computationally inefficient: as the input doubles, the amount of computation required to process it quadruples. Alongside Tri Dao of Princeton University, Dr Gu has come up with an alternative architecture called Mamba. If, by analogy, a transformer reads all of a book’s pages at once, Mamba reads them sequentially, updating its worldview as it progresses. This is not only more efficient, but also more closely approximates the way human comprehension works.
LLMs also need help getting better at reasoning and planning. Andrej Karpathy, a researcher formerly at OpenAI, explained in a recent talk that current LLMs are only capable of “system 1″ thinking. In humans, this is the automatic mode of thought involved in snap decisions. In contrast, “system 2″ thinking is slower, more conscious and involves iteration. For AI systems, that may require algorithms capable of something called search—an ability to outline and examine many different courses of action before selecting the best one. This would be similar in spirit to how game-playing AI models can choose the best moves after exploring several options.
Advanced planning via search is the focus of much current effort. Meta’s Dr LeCun, for example, is trying to program the ability to reason and make predictions directly into an AI system. In 2022 he proposed a framework called “Joint Embedding Predictive Architecture” (JEPA), which is trained to predict larger chunks of text or images in a single step than current generative-AI models. That lets it focus on global features of a data set. When analysing animal images, for example, a JEPA-based model may more quickly focus on size, shape and colour rather than individual patches of fur. The hope is that by abstracting things out JEPA learns more efficiently than generative models, which get distracted by irrelevant details.
Experiments with approaches like Mamba or JEPA remain the exception. Until data and computing power become insurmountable hurdles, transformer-based models will stay in favour. But as engineers push them into ever more complex applications, human expertise will remain essential in the labelling of data. This could mean slower progress than before. For a new generation of AI models to stun the world as ChatGPT did in 2022, fundamental breakthroughs may be needed.
© 2024, The Economist Newspaper Limited. All rights reserved.
From The Economist, published under licence. The original content can be found on www.economist.com
-
others1 week ago
Here’s What Bitcoin Needs To Do To Confirm Bullish Breakout, According to Trader Who Nailed 2024 BTC Correction – Crypto News
-
Blockchain7 days ago
XRP MVRV Ratio Dips Below The 200-Day MA – Trend Shift Underway? – Crypto News
-
others6 days ago
Bybit Shuts Down Its NFT Marketplace As Crypto Sector Struggles To Recover – Crypto News
-
Business1 week ago
Can Pi Coin Price Hit $1 Soon? – Crypto News
-
others1 week ago
Here’s What Bitcoin Needs To Do To Confirm Bullish Breakout, According to Trader Who Nailed 2024 BTC Correction – Crypto News
-
others1 week ago
Here’s What Bitcoin Needs To Do To Confirm Bullish Breakout, According to Trader Who Nailed 2024 BTC Correction – Crypto News
-
others7 days ago
XRP and Three Other Altcoins Could Witness Another Sell-Off Event, According to Crypto Strategist – Crypto News
-
Cryptocurrency6 days ago
Sony Singapore Now Lets Shoppers Pay in USDC Through Crypto.com – Crypto News
-
Blockchain6 days ago
Dogecoin Breaking These Levels Could Be The Catalyst For Next Bull Run, Analyst Says – Crypto News
-
Business1 week ago
XRP, BTC, ETH Price Prediction As Inflation Data Sparks Downturn in U.S. Stocks – Crypto News
-
Blockchain1 week ago
zkLend hacker claims losing stolen ETH to Tornado Cash phishing site – Crypto News
-
others1 week ago
Gold extends bullish trend amid rising trade tensions; fresh record high and counting – Crypto News
-
Blockchain1 week ago
SpaceX flight bankrolled by crypto investor launches first manned polar orbit – Crypto News
-
Blockchain1 week ago
Will Bitcoin Downtrend Continue? This Metric Suggests Yes – Crypto News
-
Blockchain1 week ago
Binance ends Tether USDT trading in Europe to comply with MiCA rules – Crypto News
-
Blockchain1 week ago
Binance ends Tether USDT trading in Europe to comply with MiCA rules – Crypto News
-
others1 week ago
Austria Unemployment fell from previous 347.4K to 316.3K in March – Crypto News
-
Blockchain7 days ago
Analyst Calls Dogecoin Chart A ‘Beauty’ As Key Indicators Align – Crypto News
-
Technology6 days ago
XRP Price Predicted to Reach $10 in April if US Congress Stablecoin Bill Promotes Ripple’s RLUSD – Crypto News
-
Business6 days ago
XRP Price Predicted to Reach $10 in April if US Congress Stablecoin Bill Promotes Ripple’s RLUSD – Crypto News
-
Technology6 days ago
Apple Intelligence debuts on Vision Pro with visionOS 2.4 update: AI-powered features, spatial content and more – Crypto News
-
Cryptocurrency6 days ago
US equities slip after job openings disappointment – Crypto News
-
others6 days ago
Will BNB Price Rally to ATH After VanEck BNB ETF Filing? – Crypto News
-
others6 days ago
Pound Sterling consolidates against US Dollar ahead of Trump’s tariffs announcement – Crypto News
-
Cryptocurrency6 days ago
FLOKI price poised for 20% rally, Here’s why – Crypto News
-
others6 days ago
PENDLE Price Jumps 8% Today Amid Huge Whale Accumulation – Crypto News
-
others6 days ago
Fundstrat’s Tom Lee Calls for Imminent Stock Market Reversal, Says US Has the ‘Right Pieces’ for a Bottom – Crypto News
-
Technology6 days ago
European regulators warn of financial risks from US crypto integration – Crypto News
-
Cryptocurrency1 week ago
Is Bitcoin’s rebound near as key area rises? Assessing… – Crypto News
-
Business1 week ago
Builder.ai Announces Third-Party Audit After Allegations – Crypto News
-
others1 week ago
Will the RBA hint at further interest rate hikes at its policy meeting? – Crypto News
-
Technology1 week ago
Over 60 pc broadband, fiber, DSL users surveyed flag problems with connection: LocalCircles poll – Crypto News
-
Cryptocurrency1 week ago
Tether Boosts Bitcoin Holdings By 8,888 BTC In Q1 2025 – Crypto News
-
Technology1 week ago
Bumper discounts in Amazon Gaming Fest! Up to 70% off on gaming laptops, monitors, vlog cameras and more – Crypto News
-
Blockchain1 week ago
Hayes Predicts $250,000 Bitcoin As Fed Caves To QE Pressure – Crypto News
-
Cryptocurrency1 week ago
XRP, DOGE Shoot up as BTC Price Reclaims $84K Level (Market Watch) – Crypto News
-
Technology7 days ago
Best washing machines under ₹10000 in April 2025 to boost your laundry routine without overspending – Crypto News
-
Blockchain6 days ago
Bitcoin traders are overstating the impact of the US-led tariff war on BTC price – Crypto News
-
Blockchain6 days ago
Bitcoin Price Bounces Back—Can It Finally Break Resistance? – Crypto News
-
Cryptocurrency6 days ago
US equities slip after job openings disappointment – Crypto News
-
Blockchain6 days ago
UK trade bodies ask government to make crypto a ‘strategic priority’ – Crypto News
-
Blockchain6 days ago
Several Altcoins Crash Up To 50% On Binance, What’s Going On? – Crypto News
-
Business6 days ago
What to Expect From XRP Price as Trump’s ‘Liberation Day’ Tariffs Go Into Effect Today – Crypto News
-
Technology6 days ago
Franklin Templeton Eyes Crypto ETP Launch In Europe After BlackRock & 21Shares – Crypto News
-
Technology1 week ago
ChatGPT finally allows free users to create Ghibli-style AI images: Check our step-by-step guide – Crypto News
-
Technology1 week ago
Elon Musk’s Grok AI calls him ‘top misinformation spreader’, sparks debate on ‘AI freedom’ – Crypto News
-
Cryptocurrency1 week ago
Will the $80K Support Level Hold BTC After Recent Rejection? – Crypto News
-
Metaverse1 week ago
Mint Primer | Resistance is futile: AI is now writing code – Crypto News
-
Metaverse1 week ago
The Tools of Tomorrow: What Lies Ahead with the AI Revolution – Crypto News
-
Business1 week ago
MicroStrategy Acquires 22,048 Bitcoin For $1.92 Billion – Crypto News