<\/p>\n
That hunger for the new has only accelerated. In March Anthropic launched Claude 3, which bested the previous top models from OpenAI and Google on various leaderboards. On April 9th OpenAI reclaimed the crown (on some measures) by tweaking its model. On April 18th Meta released Llama 3, which early results suggest is the most capable open model to date. OpenAI is likely to make a splash sometime this year when it releases GPT-5, which may have capabilities beyond any current large language model (LLM). If the rumours are to be believed, the next generation of models will be even more remarkable\u2014able to perform multi-step tasks, for instance, rather than merely responding to prompts, or analysing complex questions carefully instead of blurting out the first algorithmically available answer.<\/p>\n
For those who believe that this is the usual tech hype, consider this: investors are deadly serious about backing the next generation of models. GPT-5 and other next-gen models are expected to cost billions of dollars to train. OpenAI is also reportedly partnering with Microsoft, a tech giant, to build a new $100bn data centre. Based on the numbers alone, it seems as though the future will hold limitless exponential growth. This chimes with a view shared by many AI researchers called the \u201cscaling hypothesis”, namely that the architecture of current LLMs is on the path to unlocking phenomenal progress. All that is needed to exceed human abilities, according to the hypothesis, is more data and more powerful computer chips.<\/p>\n
Look closer at the technical frontier, however, and some daunting hurdles become evident.<\/p>\n
Data may well present the most immediate bottleneck. Epoch AI, a research outfit, estimates the well of high-quality textual data on the public internet will run dry by 2026. This has left researchers scrambling for ideas. Some labs are turning to the private web, buying data from brokers and news websites. Others are turning to the internet\u2019s vast quantities of audio and visual data, which could be used to train ever-bigger models for decades. Video can be particularly useful in teaching AI models about the physics of the world around them. If a model can observe a ball flying through the air, it might more easily work out the mathematical equation that describes the projectile\u2019s motion. Leading models like GPT-4 and Gemini are now \u201cmultimodal”, capable of dealing with various types of data.<\/p>\n
When data can no longer be found, it can be made. Companies like Scale AI and Surge AI have built large networks of people to generate and annotate data, including PhD researchers solving problems in maths or biology. One executive at a leading AI startup estimates this is costing AI labs hundreds of millions of dollars per year. A cheaper approach involves generating \u201csynthetic data” in which one LLM makes billions of pages of text to train a second model. Though that method can run into trouble: models trained like this can lose past knowledge and generate uncreative responses. A more fruitful way to train AI models on synthetic data is to have them learn through collaboration or competition. Researchers call this \u201cself-play”. In 2017 Google DeepMind, the search giant\u2019s AI lab, developed a model called AlphaGo that, after training against itself, beat the human world champion in the game of Go. Google and other firms now use similar techniques on their latest LLMs.<\/p>\n
Extending ideas like self-play to new domains is hot topic of research. But most real-world problems\u2014from running a business to being a good doctor\u2014are more complex than a game, without clear-cut winning moves. This is why, for such complex domains, data to train models is still needed from people who can differentiate between good and bad quality responses. This in turn slows things down.<\/p>\n
Better hardware is another route to more powerful models. Graphics-processing units (GPUs), originally designed for video-gaming, have become the go-to chip for most AI programmers thanks to their ability to run intensive calculations in parallel. One way to unlock new capabilities may lie in using chips designed specifically for AI models. Cerebras, a chipmaker based in Silicon Valley, released a product in March containing 50 times as many transistors as the largest GPU. Model-building is usually hampered by data needing to be continuously loaded on and off the GPUs as the model is trained. Cerebras\u2019s giant chip, by contrast, has memory built in.<\/p>\n
New models that can take advantage of these advances will be more reliable and better at handling tricky requests from users. One way this may happen is through larger \u201ccontext windows”, the amount of text, image or video that a user can feed into a model when making requests. Enlarging context windows to allow users to upload additional relevant information also seems to be an effective way of curbing hallucination, the tendency of AI models to confidently answer questions with made-up information.<\/p>\n
But while some model-makers race for more resources, others see signs that the scaling hypothesis is running into trouble. Physical constraints\u2014insufficient memory, say, or rising energy costs\u2014place practical limitations on bigger model designs. More worrying, it is not clear that expanding context windows will be enough for continued progress. Yann LeCun, a star AI boffin now at Meta, is one of many who believe the limitations in the current AI models cannot be fixed with more of the same.<\/p>\n
Some scientists are therefore turning to a long-standing source of inspiration in the field of AI\u2014the human brain. The average adult can reason and plan far better than the best LLMs, despite using less power and much less data. \u201cAI needs better learning algorithms, and we know they\u2019re possible because your brain has them,” says Pedro Domingos, a computer scientist at the University of Washington.<\/p>\n
One problem, he says, is the algorithm by which LLMs learn, called backpropagation. All LLMs are neural networks arranged in layers, which receive inputs and transform them to predict outputs. When the LLM is in its learning phase, it compares its predictions against the version of reality available in its training data. If these diverge, the algorithm makes small tweaks to each layer of the network to improve future predictions. That makes it computationally intensive and incremental.<\/p>\n
The neural networks in today\u2019s LLMs are also inefficiently structured. Since 2017 most AI models have used a type of neural-network architecture known as a transformer (the \u201cT” in GPT), which allowed them to establish relationships between bits of data that are far apart within a data set. Previous approaches struggled to make such long-range connections. If a transformer-based model were asked to write the lyrics to a song, for example, it could, in its coda, riff on lines from many verses earlier, whereas a more primitive model would have forgotten all about the start by the time it had got to the end of the song. Transformers can also be run on many processors at once, significantly reducing the time it takes to train them.<\/p>\n
Albert Gu, a computer scientist at Carnegie Mellon University, nevertheless thinks the transformers\u2019 time may soon be up. Scaling up their context windows is highly computationally inefficient: as the input doubles, the amount of computation required to process it quadruples. Alongside Tri Dao of Princeton University, Dr Gu has come up with an alternative architecture called Mamba. If, by analogy, a transformer reads all of a book\u2019s pages at once, Mamba reads them sequentially, updating its worldview as it progresses. This is not only more efficient, but also more closely approximates the way human comprehension works.<\/p>\n
LLMs also need help getting better at reasoning and planning. Andrej Karpathy, a researcher formerly at OpenAI, explained in a recent talk that current LLMs are only capable of \u201csystem 1″ thinking. In humans, this is the automatic mode of thought involved in snap decisions. In contrast, \u201csystem 2″ thinking is slower, more conscious and involves iteration. For AI systems, that may require algorithms capable of something called search\u2014an ability to outline and examine many different courses of action before selecting the best one. This would be similar in spirit to how game-playing AI models can choose the best moves after exploring several options.<\/p>\n
Advanced planning via search is the focus of much current effort. Meta\u2019s Dr LeCun, for example, is trying to program the ability to reason and make predictions directly into an AI system. In 2022 he proposed a framework called \u201cJoint Embedding Predictive Architecture” (JEPA), which is trained to predict larger chunks of text or images in a single step than current generative-AI models. That lets it focus on global features of a data set. When analysing animal images, for example, a JEPA-based model may more quickly focus on size, shape and colour rather than individual patches of fur. The hope is that by abstracting things out JEPA learns more efficiently than generative models, which get distracted by irrelevant details.<\/p>\n
Experiments with approaches like Mamba or JEPA remain the exception. Until data and computing power become insurmountable hurdles, transformer-based models will stay in favour. But as engineers push them into ever more complex applications, human expertise will remain essential in the labelling of data. This could mean slower progress than before. For a new generation of AI models to stun the world as ChatGPT did in 2022, fundamental breakthroughs may be needed.<\/p>\n
\u00a9 2024, The Economist Newspaper Limited. All rights reserved.\u00a0<\/i><\/p>\n
From The Economist, published under licence. The original content can be found on www.economist.com<\/i><\/p>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"
That hunger for the new has only accelerated. In March Anthropic launched Claude 3, which bested the previous top models from OpenAI and Google on various leaderboards. On April 9th OpenAI reclaimed the crown (on some measures) by tweaking its model. On April 18th Meta released Llama 3, which early results suggest is the most […]<\/p>\n","protected":false},"author":2,"featured_media":285873,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[440,19998,284,263,262,5834,260,259,258,5067,18764,19997,8707,8714,265,202,5792,261,264],"class_list":["post-285872","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-metaverse","tag-ai","tag-ai-programmers","tag-artificial-intelligence","tag-axie-infinity","tag-axs","tag-chatgpt","tag-decentraland","tag-facebook","tag-game","tag-gemini","tag-gpus","tag-graphics-processing-units","tag-large-language-models","tag-llms","tag-mark-zuckerberg","tag-nft","tag-openai","tag-sandbox","tag-vr"],"_links":{"self":[{"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/posts\/285872","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/comments?post=285872"}],"version-history":[{"count":2,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/posts\/285872\/revisions"}],"predecessor-version":[{"id":285875,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/posts\/285872\/revisions\/285875"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/media\/285873"}],"wp:attachment":[{"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/media?parent=285872"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/categories?post=285872"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/tags?post=285872"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}