The UK’s National Cyber Security Centre (NCSC) has issued a warning about the risks of prompt injection attacks to AI applications, according to articles in The Guardian and PopSci. Primarily intended for those building large language models, the warning can apply to almost all types of AI tools and relate to attackers using specific prompts to get the AI application to do or reveal something it shouldn’t be allowed to do. Depending on the goals of the attacker, this could include revealing confidential data, assisting in generating malicious code, or even revealing the underlying rules that limit the AI tool, thus making it easier for hackers to exploit weaknesses.
While not generally seen as a critical problem with most generic chatbots, it becomes of greater concern when AI applications are built on top of other tools or services containing private, personal, or sensitive material. For items like banks, this could lead to hackers being able to push through illegal transactions, and coverup their actions.
With these AI models and tools still being so new, it’s not clear what the eventual security measures will look like. Says security researcher Simon Willison, “The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system.”
Source:
https://www.popsci.com/technology/prompt-injection-attacks-llms-ai/
