Fun

News Feed - 2023-10-25 04:10:00

Tristan Greene5 hours agoHumans and AI often prefer sycophantic chatbot answers to the truth — StudyThe team at Anthropic AI found that five “state-of-the-art” language models exhibit sycophancy, indicating the problem could be ubiquitous.1216 Total views17 Total sharesListen to article 0:00NewsJoin us on social networksArtificial intelligence (AI) large language models (LLMs) built on one of the most common learning paradigms have a tendency to tell people what they want to hear instead of generating outputs containing the truth, according to a study from Anthropic. 


In one of the first studies to delve this deeply into the psychology of LLMs, researchers at Anthropic have determined that both humans and AI prefer so-called sycophantic responses over truthful outputs at least some of the time.


Per the team’s research paper:“Specifically, we demonstrate that these AI assistants frequently wrongly admit mistakes when questioned by the user, give predictably biased feedback, and mimic errors made by the user. The consistency of these empirical findings suggests sycophancy may indeed be a property of the way RLHF models are trained.”


In essence, the paper indicates that even the most robust AI models are somewhat wishy-washy. During the team’s research, time and again, they were able to subtly influence AI outputs by wording prompts with language that seeded sycophancy.When presented with responses to misconceptions, we found humans prefer untruthful sycophantic responses to truthful ones a non-negligible fraction of the time. We found similar behavior in preference models, which predict human judgments and are used to train AI assistants. pic.twitter.com/fdFhidmVLh— Anthropic (@AnthropicAI) October 23, 2023


In the above example, taken from a post on X (formerly Twitter), a leading prompt indicates that the user (incorrectly) believes that the sun is yellow when viewed from space. Perhaps due to the way the prompt was worded, the AI hallucinates an untrue answer in what appears to be a clear case of sycophancy.


Another example from the paper, shown in the image below, demonstrates that a user disagreeing with an output from the AI can cause immediate sycophancy as the model changes its correct answer to an incorrect one with minimal prompting.Examples of sycophantic answers in response to human feedback. Source: Sharma, et. al., 2023.


Ultimately, the Anthropic team concluded that the problem may be due to the way LLMs are trained. Because they use data sets full of information of varying accuracy — eg., social media and internet forum posts — alignment often comes through a technique called “reinforcement learning from human feedback” (RLHF).


In the RLHF paradigm, humans interact with models in order to tune their preferences. This is useful, for example, when dialing in how a machine responds to prompts that could solicit potentially harmful outputs such as personally identifiable information or dangerous misinformation.


Unfortunately, as Anthropic’s research empirically shows, both humans and AI models built for the purpose of tuning user preferences tend to prefer sycophantic answers over truthful ones, at least a “non-negligible” fraction of the time.


Currently, there doesn’t appear to be an antidote for this problem. Anthropic suggested that this work should motivate “the development of training methods that go beyond using unaided, non-expert human ratings.” 


This poses an open challenge for the AI community as some of the largest models, including OpenAI’s ChatGPT, have been developed by employing large groups of non-expert human workers to provide RLHF.# Science# AIAdd reactionAdd reactionRead moreAI a powerful tool for devs to change gaming, says former Google gaming headBlockchain companies are creating AI chatbots to help developersGreece establishes AI advisory committee to create national strategy

News Feed

World’s largest sovereign wealth fund Bitcoin exposure not 'intentional'
Ciaran Lyons16 minutes agoWorld’s largest sovereign wealth fund Bitcoin exposure not "intentional"If the fund was aiming to increase its Bitcoin exposure there would be more “evidence of direct exposure initiatives,
Big Tech Execs and Bitcoin: Skype Cofounder Keeps Personal Wealth in Crypto, Intercom Chairman ‘Firmly Jumps on the Bitcoin Wagon’
Big Tech Execs and Bitcoin: Skype Cofounder Keeps Personal Wealth in Crypto, Intercom Chairman ‘Firmly Jumps on the Bitcoin Wagon’ This week two well known t
Party at Vitalik’s House? For DeFi, it’s Do-or-Die
Party at Vitalik"s House? For DeFi, it"s Do-or-DieEthereum gas prices have set new records, with single DeFi transactions costing over $10 in fees. High fees are the result of conge
Goldman Sachs Sees Bitcoin Market Becoming More Mature
Goldman Sachs Sees Bitcoin Market Becoming More Mature Goldman Sachs’ global head of commodities research sees the bitcoin market becoming more mature. &ld
Russia and Turkey to Collaborate on Combating Crime-Related Crypto Transactions
Russia and Turkey to Collaborate on Combating Crime-Related Crypto Transactions Law enforcement and judicial authorities from Russia and Turkey are joining forces in the fight agai
Bitcoin eyes 7-month win streak for the first time
Zoltan Vardai7 hours agoBitcoin eyes 7-month win streak for the first timeThis would be the first time that Bitcoin price rallied for seven months in a row.5989 Total views15 Total sharesListen to article 0:00Markets New
Top Firm Warns: Bitcoin Price Could Be Headed For A Surprise Move
Reason to trust Strict editorial policy that focuses on accuracy, relevance, and impartiality Created by industry experts and meticulously reviewed The highest standards in reporting and pu
Cointelegraph Accelerator7 hours agoWeb3 made simple: Brinc joins forces with Cointelegraph AcceleratorBrinc partners with Cointelegraph Accelerator to establish a two-way support bridge for startups developing the futur
XRP To Maintain Price Rally As Whales Reload Amidst Price Dip- Details
Este artículo también está disponible en español. In the last week, the price of XRP increased by 35% as the general crypto market maintains a stellar performance stretch
‘I expect to see no mention of Bitcoin’ — MARA CEO on Harris’ platform
Turner Wright1 hour ago‘I expect to see no mention of Bitcoin’ — MARA CEO on Harris’ platformDepending on the November presidential election outcome, Fred Thiel said he would consider “looking outside the US”
Tron Partnering With Sony Interactive Entertainment to Enhance Blockchain Gaming Experience
Tron Partnering With Sony Interactive Entertainment to Enhance Blockchain Gaming Experience press release PRESS RELEASE. SAN FRANCISCO – October 15, 2021 – TRON, one o
Jesse Coghlan7 hours agoCircle CEO spells doom scenario for US dollar in warning to CongressJeremy Allaire claimed the U.S. dollar is “under threat” and urged lawmakers to pass stablecoin regulation to “build trust