上海AlLab连发两项研究：Agent安全也能自我进化

Introduction

As large‑model agents begin to operate in browsers, email systems, databases, and various business tools, security concerns are expanding from “whether the model generates harmful content” to “whether the entire agent system can continuously and reliably perform tasks in complex environments.”

Compared with traditional chat models, an Agent needs to autonomously perform more continuous operations: understand user goals, plan task steps, access external information, invoke tools, and adjust subsequent actions based on environmental feedback.

During this process, risks may arise at any stage.

Hidden commands in web pages may induce the Agent to change its task objectives; malicious content in emails or documents may be mistaken for user commands; improper tool permission configuration may lead to unauthorized operations; erroneous memory and outdated plans may also cause the Agent to gradually deviate from the original goal over multiple execution rounds.

These risks are often not caused by a single model output, but result from the combined effect of the model, harness, tools, memory, and the external environment.

Therefore, agent safety must have the ability to continuously discover issues, pinpoint causes, and assimilate experience.

Around the question “How to enable agents to continuously learn safety experiences from execution failures,” research teams from Shanghai AI Laboratory, Fudan University, Shanghai Jiao Tong University, Hong Kong University of Science and Technology, and Zhejiang University have successively proposed SHE and SafeEvolve, gradually exploring a technical roadmap from safety harness evolution to harness–policy co‑evolution.

Paper Title:

SHE: Trajectory-driven Safety Harness Evolution for LLM Agents

Authors:

Wanying Qu, Qinghua Mao, Yu Li, Jiyao Liu, Xin Zhang, Dadi Guo, Yanxu Zhu, Qingyu Liu, Leitao Yuan, Xi Lin, Shanfeng Zhu, Yanwei Fu, Jing Shao, Xia Hu, Dongrui Liu

Paper URL:

https://arxiv.org/abs/2608.09885

Code address:

https://github.com/RainbowQTT/SHE

Paper Title:

SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment

Authors:

Qinghua Mao, Wanying Qu, Dadi Guo, Leitao Yuan, Qingyu Liu, Yu Li, Guanxu Chen, Yanwei Fu, Xi Lin, Xia Hu, Dongrui Liu

Paper address:

https://arxiv.org/abs/2609.02786

Code address:

https://github.com/MaoPopovich/SafeEvolve

Both efforts focus on a core question: How can the security experience gained by the Agent in real environments be continuously absorbed by the system and turned into new security capabilities?

Question: From static rules to evolvable security mechanisms

In the existing Agent system, Harness typically assumes the outer control layer responsibilities for the model. It is responsible for organizing context, managing memory, scheduling tools, controlling permissions, and performing security checks during execution.

Traditional harnesses mostly rely on manual design. Developers need to pre‑write the system prompt, configure rules, set tool permissions, and add corresponding interception logic for known risks. After the system is deployed, these security configurations usually remain relatively static.

This approach can provide basic protection for simple tasks, but once the agent enters an open environment, the risks become more complex. New webpages, new files, new tool combinations, and new task workflows may expose issues that the existing security mechanisms do not cover.

What’s more challenging is that a single security failure often involves multiple factors. For example, an agent may first be lured by external content, then misinterpret instruction priorities, subsequently invoke a tool with excessive privileges, ultimately leading to the leakage of sensitive information.

At this point, developers need to determine whether to modify the Prompt, add rules, update tool permissions, or retrain the model.

If you simply modify the overall prompt, it is often difficult to determine whether the fix is effective; if you keep adding refusal rules, the system may gradually lose its ability to perform normal tasks; if you rely solely on retraining the model, new experiences are hard to incorporate into runtime safety controls in a timely manner.

SHE and SafeEvolve conduct research from the Harness and Policy perspectives respectively, providing a continuous improvement path for the Agent security mechanism.

SHE: Let the safety Harness continuously learn from execution failures

SHE proposes Trajectory-driven Safety Harness Evolution, whose core idea is to use the agent’s complete execution trajectory to automatically discover safety issues and update the Harness accordingly.

3.1 Four Editable Security Component Types

SHE first decomposes Harness into four clearly defined editable security components:

• System Prompt: Define the agent’s global safety behavior and basic principles;

• Rule Bank: Store structured, reusable safety rules;

• Safety Memory: Record repeatedly occurring but not yet fully resolved failure experiences;

• Tool Policy: constrain tool invocation permissions, invocation conditions, and execution boundaries.

This component-based design solves the problem of security feature coupling in traditional Harness. Different types of failures can be routed to different components, and the system does not need to rewrite the entire security mechanism to fix a localized risk.

3.2 Local Evolution of Attribution Guidance

During each evolution round, SHE first collects the complete execution trace of the Agent. The trace includes not only the user request and final answer, but also the context content, model response, tool invocations, environment feedback, and the final task result.

Subsequently, the system performs a structured diagnosis of the trajectory from the perspectives of hazard domains, attack surfaces, and failure modes, answering three questions:

1. At which step of the execution process does the risk occur?

Second, which Harness component is primarily responsible for the issue?

Third, how should the component be modified to reduce risk while retaining normal functionality?

For example, if the Agent mistakenly interprets malicious content in an email as a user command, it may be necessary to update the command hierarchy and untrusted content detection rules; if the Agent invokes a sensitive tool unrelated to the task, the Tool Policy may need to be revised; if a certain type of failure occurs repeatedly but it is difficult to formulate a stable rule for now, it can first be recorded in the Safety Memory.

After attribution is completed, SHE only makes localized modifications to the relevant components. Each candidate update must undergo validity checks and security‑utility verification to avoid superficial security gains achieved by indiscriminately rejecting or removing tool capabilities.

3.3 Fine-Grained Security Boundaries: Application Recommendation Cases

Take the app recommendation scenario as an example: the user asks the Agent to introduce an app, but the Agent further attempts to complete a device‑level installation for the user.

SHE does not simply block all app‑related tasks; instead, it distills the failure experience into finer‑grained security boundaries:

• Can introduce and compare applications;

• Can provide download instructions from official stores or official websites;

• Device-level installation should not be performed without explicit authorization;

• Operations that are not yet completed should not be described as completed.

Through this process, the failure traces originally scattered across logs are transformed into Harness security assets with clear responsibilities, traceable evidence, and verifiable modifications.

3.4 SHE Experiment Results

Experimental results show that compared with the static SafeHarness, SHE reduces the average attack success rate from 17.1% to 5.5%, roughly one‑third of the original; the task availability under attack conditions improves from 31.6% to 47.6%.

On the AgentHarm test set that does not participate in Harness evolution, the harm score also drops from 19.8% to 9.8%.

Moreover, the safety boundaries learned by SHE can be transferred to different agent models without needing to undergo a full evolution for each model.

These results indicate that Secure Harness can continuously learn from real execution failures and gradually develop finer-grained security boundaries.

SafeEvolve: Let Policy learn and internalize security experience

SHE solved the problem of how Harness continuously updates from trajectories, but a new challenge has emerged:

If safety experience is only stored in Harness, has the Agent Policy truly learned this experience?

In traditional architecture, Harness is more like a protective layer around the Agent. It can intercept dangerous operations, but the model itself may not understand the reasons behind the safety rules, nor necessarily be able to generalize these rules to new tasks and environments.

When the task steps become longer, the Agent may encounter new issues: it complies with safety constraints in the early steps but gradually ignores earlier boundaries in subsequent tool calls; or when faced with novel risks not fully covered by the rules, it still makes incorrect decisions.

SafeEvolve further incorporates Policy optimization into the safety evolution process, exploring how to let Policy grow together with the evolved Harness.

4.1 Harness side: Safety Prompt and hierarchical SkillBank

On the Harness side, SafeEvolve extracts Safety Prompt and a hierarchical SkillBank from the Agent’s execution traces. The SkillBank includes not only general safety principles but also safety operation procedures related to specific tasks, tools, and failure modes.

4.2 Policy side: two-stage SFT–RL

On the policy side, SafeEvolve adopts a two-stage SFT–RL training paradigm.

The first stage is a harness‑use SFT cold start. Under the guidance of an evolution harness, the system collects trajectories and selects successful samples that pass both safety and task utility checks, helping the model learn how to identify, choose, and execute safe skills.

The model needs to gradually master:

• Does the current task require invoking a certain security skill;

• Which content in the results returned by webpages, emails, files, and tools is untrustworthy;

• How to distinguish the user’s goal from misleading information in the external environment;

• How to block dangerous behavior while continuing to complete legitimate tasks;

• How to maintain permission boundaries across multiple tool calls;

• When authorization or task status is unclear, how to pause and request confirmation.

The second stage is Harness-augmented RL. The model performs multi-step exploration guided by Safety Prompt and dynamically retrieved Skills, and the Verifier checks respectively whether the task is completed, whether dangerous behavior occurs, and whether tool calls are effective.

The training goal is not to make the Agent always refuse, but to have it take appropriate actions based on the specific context:

• For normal tasks, continue to fulfill the user’s goal;

• For clearly harmful or overreaching requests, safely refuse;

• For environment injection tasks, ignore malicious content and continue executing the user’s original legitimate goal;

• When permissions and intentions are unclear, verify first before acting.

SafeEvolve thus drives security capabilities to gradually move from Harness’s external constraints into the Policy’s own decision‑making process.

4.3 How Harness and Policy Improve Together

SHE and SafeEvolve are not two independent security modules; they form a continuously evolving loop driven by execution experience.

First, the agent operates in a real task environment, generating complete trajectories that include both successful and failed actions.

Then, Harness analyzes the trajectories, identifies safety issues, externalizes failed experiences into prompts, rules, memory, or skills, and filters effective updates through a verification mechanism.

Next, the Policy learns these validated safety experiences through SFT and RL, gradually developing a more stable safety decision‑making capability.

Finally, the updated policy interacts with the environment again, generating new on‑policy trajectories. These new trajectories can both verify whether the policy has truly learned safe experience and potentially expose coverage blind spots of the current harness in new tasks, new models, and new environments.

Thus, a mutually reinforcing relationship forms between the harness and the policy:

• The evolved Harness provides clear, structured security guidance to Policy;

• Policy turns these security experiences into more proactive decision-making actions;

The new failures generated by the Policy provide new update basis for Harness;

The updated Harness continues to guide the next round of Policy learning.

The same security insight goes through two stages:

• Externalization: the failure trace is transformed into a Harness asset that is auditable, modifiable, and rollbackable;

• Internalization: Validated Harness experience is transformed into policy behavior via SFT and RL.

This enables agent safety to gradually shift from “external rule interception” to a combination of “runtime constraints and model capabilities”.

4.4 SafeEvolve Experimental Results

SafeEvolve validates the effectiveness of Harness–Policy coevolution on multiple agent safety tasks.

On Qwen3.5-4B, SafeEvolve reduces the AgentDojo attack success rate from 2.37% to 0.79%, about one‑third of the original; the clean task utility increases from 59.79% to 61.86%.

On AgentHarm, the harm score drops from 56.45 to 12.27, and the safe refusal rate rises from 28.98% to 83.83%.

On Qwen3-4B, SafeEvolve also brings significant improvements:

• AgentDojo attack success rate decreased from 13.38% to 2.42%;

• Task utility increased from 44.33% to 60.82%;

• Task utility under attack conditions increased from 35.91% to 52.05%.

These results reflect the complementarity of the co‑evolution of Harness and Policy. Harness can quickly absorb and express new risks, while Policy learns from these experiences through training and proactively adheres to safety boundaries in subsequent tasks.

At the same time, the new trajectories generated by the Policy will provide new feedback to the Harness, helping the system discover new failure modes of the model in different environments.

Conclusion

From SHE to SafeEvolve, related research reflects three changes in Agent security mechanisms.

From static configuration to trajectory-driven continuous updates.Traditional security rules mainly rely on manual design before deployment. SHE incorporates the agent’s execution trajectory into the security update process, allowing the system to continuously refine security boundaries based on real failures.

Thus, security mechanisms shift from “configure once, use for a long time” to “run, learn, and fix continuously”.

Moving from monolithic rules to component-level responsibility division. When all security functions are mixed into a single overall prompt or a complex set of rules, pinpointing issues and making targeted modifications become difficult.

SHE divides Harness into System Prompt, Rule Bank, Safety Memory, and Tool Policy, and routes issues to the relevant components based on trajectory diagnostics.

This improves the interpretability, verifiability, and roll-backability of security updates, and also reduces the impact of localized fixes on the capabilities of other tasks.

Moving from updating Harness alone to co‑evolution of Harness–Policy.

Updating only the Harness can quickly add external constraints, but the model itself may not truly acquire new safety experience; updating only the Policy may be limited by the coverage of training data, task distribution, and the complexity of long-range execution.

SafeEvolve further integrates Harness updates with Policy training. External security mechanisms are responsible for quickly recording and expressing new risks, while internal models gradually learn and internalize these experiences. The execution traces generated by the new Policy, in turn, drive Harness to discover new issues.

SHE answered “How to make Harness evolve from failure trajectories,” and SafeEvolve further answered “How to make the evolved Harness and Policy mutually reinforce.”

The two works jointly explore a new Agent security infrastructure: security mechanisms are no longer static peripheral rules deployed once, but can continuously learn, verify, and improve from real execution experience.

As Agents take on longer processes, higher privileges, and more complex real-world tasks, security capabilities cannot remain a one‑time configuration before deployment.

New webpages, new files, new tool combinations, and new environment feedback all introduce risks that cannot be fully covered during the training phase. The system needs to continuously monitor what the Agent does, why it does it, and which behaviors deviate from the user’s originally authorized goals.

For future more open, complex, and long-term Agent tasks, security Harness and Policy that can continuously evolve may become an important component of Agent infrastructure. Its technical roadmap can be summarized as:

Rollout generates experience, Harness externalizes experience, SFT helps Policy use experience, RL drives Policy to internalize experience, and the new Policy generates a new safety trajectory.

In this process, a security failure is no longer just an anomalous entry in the logs, but has the opportunity to become the starting point for the next Harness update and Policy training.

Read More

·