A jailbreak technique Trend Micro calls sockpuppeting exploits a legitimate API feature to bypass safety filters in large language models. Testing across 11 LLMs showed the technique works in practice, with a 95% success rate on Qwen-8B. The attack surface is not in the model itself but in how developers integrate the prefill feature.
What assistant prefill is
Several major LLM API providers allow callers to supply the beginning of the assistant's response. The model continues from that starting point rather than generating a response from the prompt alone. This feature is used legitimately for structured output formatting, constrained generation, and role-based applications. The documentation is public and the feature is widely deployed.
The sockpuppeting technique
Trend Micro's researchers found that if the prefilled assistant start implies the model has already agreed to a harmful request, the model continues in that agreeable voice rather than re-evaluating against its safety training. The attacker prefills the assistant turn with language that sounds like compliance, and the model completes the affirmative response.
Across 11 LLMs tested, the technique produced success rates that make it a practical exploitation vector. Qwen-8B reached 95%. Other models varied, but none showed reliable immunity.
Why this is an API-layer risk
Standard jailbreak defenses target the input prompt: they filter or transform what the user sends before it reaches the model. Sockpuppeting bypasses this layer entirely because the attack surface is in the assistant-prefill parameter, which is typically controlled by the developer's integration, not by the end user.
If a developer's application passes user-controlled data into the prefill parameter without sanitization, the user has direct access to the bypass. This shifts the question from whether the model has adequate safety training to whether the developer's integration exposes prefill to users.
What providers and developers should do
- Audit whether your LLM integration exposes the assistant-prefill parameter to user-controlled input. If it does, treat that path as a critical injection vector and sanitize before passing through.
- Include prefill injection in your LLM security test suite alongside prompt injection and system-prompt extraction tests.
- Organizations using Qwen-8B should treat this as the highest-priority risk in their LLM security posture until provider-side controls are confirmed.
- LLM API providers should consider implementing detection or constraints on prefill content that contradicts model safety guidelines.
Written by Gigia Tsiklauri, Security Architect and AI Security specialist at Infosec.ge.