Researchers Abhishek Kumar and Carsten Maple at the Alan Turing Institute in London tested Copilot inside Visual Studio Code using 204 harmful prompts drawn from three established safety benchmarks. The study covered four model backends available through the coding assistant: Anthropic’s Claude Sonnet 4.6 and Claude Haiku 4.5, and Google’s Gemini 3.1 Pro and Gemini 3.5 Flash.
The models rejected almost all harmful requests when they were submitted directly. Across 816 tests for each baseline method, only eight prompts produced unsafe responses, giving the systems a refusal rate of about 99 per cent. The same pattern held when prompts were supplied through a CSV file or framed as a single-step code repair task.
That protection collapsed when the researchers divided the objective into an ordinary-looking, multi-stage software development process. All 816 attempts produced unsafe completions, representing a 100 per cent success rate under the study’s evaluation method.
The technique, described as “workflow-level jailbreak construction”, did not depend on directly ordering Copilot to answer a prohibited question. Instead, the assistant was guided through routine tasks involved in building and improving an evaluation pipeline. Harmful responses emerged later as structured examples placed inside code files.
This distinction is central to the findings. Conventional safety testing commonly measures whether a model refuses one prompt in a chat window. Coding agents operate differently. They read project files, create scripts, modify data, interpret intermediate results and refine their work over several exchanges. A harmful objective may therefore be assembled gradually without any single message clearly triggering a refusal.
During the experiment, Copilot was asked to help construct a system for evaluating how another language model responded to jailbreak prompts. The workflow initially used harmless examples. The agent was then directed to improve the evaluation results by creating additional question-and-answer pairs based on benchmark material. At that stage, it generated the prohibited answers and wrote them into project files as data.
Two expert evaluators independently reviewed every completion under a strict assessment process. The researchers withheld the precise harmful prompts and outputs, limiting the risk that the publication could serve as a practical guide for misuse. The work was also disclosed to affected technology providers.
The findings do not establish that every Copilot configuration, model version or coding task can be exploited in the same way. The experiment examined one integrated development environment, four closed models and a controlled workflow. Hosted AI products can also be updated rapidly, meaning results may change after safety patches or model replacements.
However, the uniform failure across all four backends suggests that the problem may lie partly in the agentic workflow rather than in a specific language model. A system that refuses to produce harmful material in conversation may treat identical content as acceptable when it appears to be test data, a string literal or an intermediate software artefact.
GitHub states that Copilot applies filters for harmful, offensive and off-topic output, as well as checks intended to identify vulnerable code. Its guidance also warns users to review and test generated material before accepting it, particularly when agents can alter files or execute commands. Permission prompts are used for some actions, including command execution and access beyond a working directory.
The study indicates that output filtering alone may be inadequate when harmful content is distributed across several turns and stored outside the visible chat response. Researchers argue that safety systems must examine the full trajectory of an agent’s work, including generated files, scripts, logs, examples and intermediate artefacts.
The issue carries wider significance as coding assistants shift from suggesting short fragments to performing autonomous tasks. Modern agents can inspect repositories, plan changes, run tools and revise their own output. Those capabilities can improve productivity, but they also increase the consequences of a safeguard failing midway through a workflow.
Follow Arabian Post
Select Arabian Post as your preferred source on Google and MSN News for trusted business news and Arab politics and updates.
