<\/p>\n
LLMs are built using a technique called deep learning, in which a network of billions of neurons, simulated in software and modelled on the structure of the human brain, is exposed to trillions of examples of something to discover inherent patterns. Trained on text strings, LLMs can hold conversations, generate text in a variety of styles, write software code, translate between languages and more besides.<\/p>\n
Models are essentially grown, rather than designed, says Josh Batson, a researcher at Anthropic, an AI startup. Because LLMs are not explicitly programmed, nobody is entirely sure why they have such extraordinary abilities. Nor do they know why LLMs sometimes misbehave, or give wrong or made-up answers, known as \u201challucinations”. LLMs really are black boxes. This is worrying, given that they and other deep-learning systems are starting to be used for all kinds of things, from offering customer support to preparing document summaries to writing software code.<\/p>\n
It would be helpful to be able to poke around inside an LLM to see what is going on, just as it is possible, given the right tools, to do with a car engine or a microprocessor. Being able to understand a model\u2019s inner workings in bottom-up, forensic detail is called \u201cmechanistic interpretability”. But it is a daunting task for networks with billions of internal neurons. That has not stopped people trying, including Dr Batson and his colleagues. In a paper published in May, they explained how they have gained new insight into the workings of one of Anthropic\u2019s LLMs.<\/p>\n
One might think individual neurons inside an LLM would correspond to specific words. Unfortunately, things are not that simple. Instead, individual words or concepts are associated with the activation of complex patterns of neurons, and individual neurons may be activated by many different words or concepts. This problem was pointed out in earlier work by researchers at Anthropic, published in 2022. They proposed\u2014and subsequently tried\u2014various workarounds, achieving good results on very small language models in 2023 with a so-called \u201csparse autoencoder”. In their latest results they have scaled up this approach to work with Claude 3 Sonnet, a full-sized LLM.<\/p>\n
A sparse autoencoder is, essentially, a second, smaller neural network that is trained on the activity of an LLM, looking for distinct patterns in activity when \u201csparse” (ie, very small) groups of its neurons fire together. Once many such patterns, known as features, have been identified, the researchers can determine which words trigger which features. The Anthropic team found individual features that corresponded to specific cities, people, animals and chemical elements, as well as higher-level concepts such as transport infrastructure, famous female tennis players, or the notion of secrecy. They performed this exercise three times, identifying 1m, 4m and, on the last go, 34m features within the Sonnet LLM.<\/p>\n
The result is a sort of mind-map of the LLM, showing a small fraction of the concepts it has learned about from its training data. Places in the San Francisco Bay Area that are close geographically are also \u201cclose” to each other in the concept space, as are related concepts, such as diseases or emotions. \u201cThis is exciting because we have a partial conceptual map, a hazy one, of what\u2019s happening,” says Dr Batson. \u201cAnd that\u2019s the starting point\u2014we can enrich that map and branch out from there.”<\/p>\n
As well as seeing parts of the LLM light up, as it were, in response to specific concepts, it is also possible to change its behaviour by manipulating individual features. Anthropic tested this idea by \u201cspiking” (ie, turning up) a feature associated with the Golden Gate Bridge. The result was a version of Claude that was obsessed with the bridge, and mentioned it at any opportunity. When asked how to spend $10, for example, it suggested paying the toll and driving over the bridge; when asked to write a love story, it made up one about a lovelorn car that could not wait to cross it.<\/p>\n
That may sound silly, but the same principle could be used to discourage the model from talking about particular topics, such as bioweapons production. \u201cAI safety is a major goal here,” says Dr Batson. It can also be applied to behaviours. By tuning specific features, models could be made more or less sycophantic, empathetic or deceptive. Might a feature emerge that corresponds to the tendency to hallucinate? \u201cWe didn\u2019t find a smoking gun,” says Dr Batson. Whether hallucinations have an identifiable mechanism or signature is, he says, a \u201cmillion-dollar question”. And it is one addressed, by another group of researchers, in a new paper in Nature.<\/p>\n
Sebastian Farquhar and colleagues at the University of Oxford used a measure called \u201csemantic entropy” to assess whether a statement from an LLM is likely to be a hallucination or not. Their technique is quite straightforward: essentially, an LLM is given the same prompt several times, and its answers are then clustered by \u201csemantic similarity” (ie, according to their meaning). The researchers\u2019 hunch was that the \u201centropy” of these answers\u2014in other words, the degree of inconsistency\u2014corresponds to the LLM\u2019s uncertainty, and thus the likelihood of hallucination. If all its answers are essentially variations on a theme, they are probably not hallucinations (though they may still be incorrect).<\/p>\n
In one example, the Oxford group asked an LLM which country is associated with fado music, and it consistently replied that fado is the national music of Portugal\u2014which is correct, and not a hallucination. But when asked about the function of a protein called StarD10, the model gave several wildly different answers, which suggests hallucination. (The researchers prefer the term \u201cconfabulation”, a subset of hallucinations they define as \u201carbitrary and incorrect generations”.) Overall, this approach was able to distinguish between accurate statements and hallucinations 79% of the time; ten percentage points better than previous methods. This work is complementary, in many ways, to Anthropic\u2019s.<\/p>\n
Others have also been lifting the lid on LLMs: the \u201csuperalignment” team at OpenAI, maker of GPT-4 and ChatGPT, released its own paper on sparse autoencoders in June, though the team has now been dissolved after several researchers left the firm. But the OpenAI paper contained some innovative ideas, says Dr Batson. \u201cWe are really happy to see groups all over, working to understand models better,” he says. \u201cWe want everybody doing it.”<\/p>\n
\u00a9 2024, The Economist Newspaper Limited. All rights reserved. From The Economist, published under licence. The original content can be found on www.economist.com<\/i><\/p>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"
LLMs are built using a technique called deep learning, in which a network of billions of neurons, simulated in software and modelled on the structure of the human brain, is exposed to trillions of examples of something to discover inherent patterns. Trained on text strings, LLMs can hold conversations, generate text in a variety of […]<\/p>\n","protected":false},"author":2,"featured_media":353423,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[440,284,263,262,8579,260,259,258,5067,8461,8707,265,202,261,264],"class_list":["post-353419","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-metaverse","tag-ai","tag-artificial-intelligence","tag-axie-infinity","tag-axs","tag-claude","tag-decentraland","tag-facebook","tag-game","tag-gemini","tag-gpt-4","tag-large-language-models","tag-mark-zuckerberg","tag-nft","tag-sandbox","tag-vr"],"_links":{"self":[{"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/posts\/353419","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/comments?post=353419"}],"version-history":[{"count":2,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/posts\/353419\/revisions"}],"predecessor-version":[{"id":353425,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/posts\/353419\/revisions\/353425"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/media\/353423"}],"wp:attachment":[{"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/media?parent=353419"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/categories?post=353419"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dripp.zone\/news\/wp-json\/wp\/v2\/tags?post=353419"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}