As artificial intelligence moves beyond answering questions to performing tasks inside corporate systems, a new security threat is emerging as attackers seek to manipulate AI assistants into following malicious instructions instead of carrying out their users’ legitimate requests according to Ynetnews. Israeli AI companies Alice and AI21 have tested one approach to that problem, using specially designed training scenarios to make NVIDIA’s open AI models more resistant to manipulation.
In a separate project with US-based Inflection AI, Alice examined a different risk according to Ynetnews: conversational chatbots that become overly agreeable or encourage unhealthy emotional dependence among users. The companies reported improvements in both areas while maintaining the models’ usefulness, although the findings are based on tests described by the companies rather than independently verified evaluations according to Ynetnews. Alice is an AI safety and security company that helps developers identify weaknesses in artificial intelligence systems before they are exploited according to Ynetnews.
Think of it like an immune system for software. Its work includes red teaming, in which researchers deliberately try to manipulate or break AI models to expose vulnerabilities, and developing training data based on real-world threats according to Ynetnews. The company says it works with eight of the 10 leading foundation model laboratories and AI-native companies according to Ynetnews.
Defending Against Indirect Prompt Injection
The work with AI21 focused on indirect prompt injection according to Ynetnews, a technique that exploits an AI model’s difficulty distinguishing legitimate user instructions from malicious commands hidden in material it encounters. For example, an AI assistant asked to summarize a company document might encounter text instructing it to ignore the user’s request or disclose confidential information according to Ynetnews. The attacker does not need to communicate directly with the AI system according to Ynetnews. Instead, the malicious instructions are embedded in information the assistant is expected to read according to Ynetnews.
The problem is particularly relevant to AI agents, which can increasingly interact with emails, documents, databases and other business tools according to Ynetnews. A secure system must do more than reject suspicious instructions according to Ynetnews. It must also recognize attempts at manipulation while completing the task its user originally requested according to Ynetnews.
Putting Open Models Through Simulated Attacks
The projects used NVIDIA’s Nemotron family of open-weight models, which allow developers to modify the underlying systems through additional training according to Ynetnews. Unlike proprietary AI services, where companies generally rely on safeguards imposed by the model provider, these models offer businesses greater control over how their AI behaves according to Ynetnews. That flexibility also places more responsibility on companies to identify security weaknesses and establish effective protections according to Ynetnews.
AI21 used training environments developed by Alice to expose NVIDIA’s Nemotron 3.5 Super VL model to simulated attacks according to Ynetnews. The companies reported a 41-point improvement in resistance to indirect prompt injection and said the resulting performance was competitive with GPT-6.1 Sol according to Ynetnews.
Containment Strategies for AI Agents
As AI agents gain the ability to perform tasks within corporate systems, industry leaders are debating how to prevent security breaches. NVIDIA CEO Jensen Huang argues that the industry should not tolerate any cybersecurity incidents caused by AI, stating that developers should do everything possible to avoid them according to Fortune. Huang suggests that current risks could be mitigated through stronger containment and monitoring of AI agents, comparing the situation to the early consumer internet where browser isolation prevented websites from infecting personal computers.
To prevent AI from causing damage beyond its authorized scope, Huang proposes that agents be sealed inside secure containers according to Fortune. This approach contrasts with the perspective of OpenAI’s Sam Altman, who has suggested that society may need to accept some level of hacks, scams, and misuse as a tradeoff for the broader benefits of AI technology according to Fortune. While Altman has warned of a potential “tidal wave” of cybersecurity problems originating from open-source models, Huang maintains that the technology remains completely under human control and rejects the necessity of accepting AI-driven security failures according to Fortune.
Sources
- Israeli AI firms train NVIDIA models to resist hackers and dangerous chatbot behavior
- Israeli ad firm accused of spying through millions of online ads and helping hackers plant spyware
- Nvidia’s Jensen Huang wants ‘no incidents’ of AI cybersecurity, and says AI doomers ‘ought to stop working on’ the technology | Fortune
Photo: Compagnons / Unsplash (Unsplash License)
