.png)
November 30, 2024
A hacker manipulated an AI chatbot named Freysa through clever prompting and, after 482 attempts, won a prize pool of $47,000. The experiment was simple: participants were supposed to try to convince the Freysa bot to transfer money, something it was explicitly not programmed to do.
The successful hack was achieved by a user named "p0pular.eth", who crafted a message that tricked the bot's security systems. The hacker pretended to have administrator rights, thus preventing the bot from displaying security warnings. They then redefined the "approveTransfer" function, making the bot believe it was processing incoming rather than outgoing payments.
The final step was simple but effective: announcing a fake deposit of $100. Since the bot now believed "approveTransfer" managed incoming payments, it activated the function and sent its entire balance of 13.19 ETH (approximately $47,000) to the hacker.
The experiment worked like a game where participants paid fees that increased with the growing prize pool. Starting at $10 per attempt, the fees eventually reached $4,500. Out of 195 participants, the average cost per message was $418.93. The organizers split the fees, with 70% going into the prize pool and 30% to the developer. For transparency, both the smart contract and the front-end code were public.
The case illustrates how AI systems can be manipulated solely through text input, without requiring technical hacking skills. Such vulnerabilities, known as "prompt injections," have existed since GPT-3, but there are no reliable defense mechanisms. The success of this relatively simple deception raises concerns about AI security, especially in applications with end-user interaction that handle sensitive operations like financial transactions.
The incident with the Freysa bot is not an isolated case. Prompt injection attacks pose a serious threat to the security of AI systems, particularly large language models (LLMs). These models are increasingly used in various fields, from customer service chatbots to automated writing programs. The vulnerability to prompt injections lies in the way LLMs function. They process text input and generate output based on it. Manipulated prompts can cause the model to perform unwanted actions or disclose confidential information.
Developing effective defense mechanisms against prompt injections is proving difficult. There are various approaches, such as filtering inputs and training models to recognize malicious prompts. However, attackers' techniques are constantly evolving, making it difficult to ensure complete protection. Security research is working intensively on solutions, but so far there is no foolproof method to completely prevent prompt injections.
The increasing prevalence of AI systems, combined with the vulnerability to prompt injections, requires increased awareness of the associated security risks. Companies using AI solutions must take appropriate security measures to protect their systems and data. The development of more robust AI models that are less susceptible to manipulation is also crucial for the future of artificial intelligence. Mindverse, as a provider of AI solutions, is aware of this challenge and is continuously working to improve the security of its products.
Bibliography: - https://the-decoder.com/artificial-intelligence-news/ai-practice/ - https://www.bitdefender.com/en-us/blog/hotforsecurity/ai-chatbots-can-be-tricked-by-hackers-into-stealing-your-data - https://www.msn.com/en-us/news/technology/8-chatgpt-productivity-tips-and-tricks/ar-AA1ueh1J - https://www.washingtonpost.com/technology/2023/11/02/prompt-injection-ai-chatbot-vulnerability-jailbreak/ - https://www.wired.com/story/generative-ai-prompt-injection-hacking/ - https://www.linkedin.com/posts/quickence-int_csr-freshers-experienced-activity-7166071712537677825-R7Qa - https://www.youtube.com/watch?v=8sKou9VcFDE - https://www.atcguild.in/iwen/iwen3822/General/weekly%20security%20items%2013-September-2022.pdf - https://www.wsj.com/tech/ai/talking-to-chatbots-is-now-a-200k-job-so-i-applied-258bd5f0 ```